Cannot connect to an OPDS server (custom)

Rishi_Ghan · September 20, 2021, 6:55pm

Hello,

I’m developing a comic book curation app and it has OPDS capability. For local development, I run the app locally. It is a React-based frontend, with a nodejs/express backend.

The OPDS server is a route configured in Express. I am testing this server against Panels, specifically,

Support Info:
Version: 2.8.2 Build: 202109171623 Env: Production
Support Identifier: [redacted]

Anytime I try to add a source via the OPDS option in Panels, it complains that it can’t find it.

Weirdly, my sever endpoint is been hit by Panels, I guess there is some error that I can’t see on my end. Is there a way I can debug this?

It was able to connect till the last update. Also I changed the route from http://192.168.1.229:8050/opds

To

http://192.168.1.229:8050/api/opds

Still get the same error.

Can you help me with this?
Thanks in advance!!

dani · September 20, 2021, 8:32pm

Hey @Rishi_Ghan

Awesome stuff!!

Could you send us a couple of examples of your OPDS feeds? After authenticating with the server, Panels tries to parse the root feed. Sounds like that is what is causing the problem.

Maybe we are requiring a field that should be optional or something like that…

If you send us some of your feeds, we’ll make sure to write tests with them and make it work.

Rishi_Ghan · September 20, 2021, 9:10pm

Sure thing

 <feed>
 <title>title</title>
 <subtitle>subtitle</subtitle>
 <icon>/favicon.ico</icon>
 <updated>2021-09-20T21:07:51.912Z</updated>
<entry>
<title>Iron Man - V1 193</title>
<link href="/file/Iron%20Man%20-%20V1%20193.cbz" rel="http://opds-spec.org/acquisition"   type="application/x-cbr"/>
</entry>
<entry>
<title>Iron Man - V1 194</title>
<link href="/file/Iron%20Man%20-%20V1%20194.cbz" rel="http://opds-spec.org/acquisition"  type="application/x-cbr"/>
 </entry>
<entry>
<title>Iron Man - V1 195</title>
<link href="/file/Iron%20Man%20-%20V1%20195.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
</entry>
<entry>
<title>Iron Man - V1 196</title>
<link href="/file/Iron%20Man%20-%20V1%20196.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
</entry>
</feed>

dani · September 21, 2021, 11:00am

Thanks @Rishi_Ghan!

This is what is missing:

feed.id
feed.link.self
entry.id

An OPDS feed is an Atom feed. According to the Atom RFC and the OPDS documentation

atom:feed elements MUST contain exactly one atom:id element.
atom:entry elements MUST contain exactly one atom:id element.

The feed self link is not required, but recommended

atom:feed elements SHOULD contain one atom:link element with a rel
attribute value of “self”. This is the preferred URI for
retrieving Atom Feed Documents representing this Atom feed.

I think it’s better if we keep it.

https://datatracker.ietf.org/doc/html/rfc4287#section-4.1.1

<?xml version="1.0" encoding="UTF-8"?>
<feed>
   <id>urn:uuid:2853dacf-ed79-42f5-8e8a-a7bb3d1ae6a2</id>
   <link rel="self" href="/opds-catalogs/root.xml" type="application/atom+xml;profile=opds-catalog;kind=navigation" />
   <title>title</title>
   <subtitle>subtitle</subtitle>
   <icon>/favicon.ico</icon>
   <updated>2021-09-20T21:07:51.912Z</updated>
   <entry>
      <id>1</id>
      <title>Iron Man - V1 193</title>
      <link href="/file/Iron%20Man%20-%20V1%20193.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr" />
   </entry>
   <entry>
      <id>2</id>
      <title>Iron Man - V1 194</title>
      <link href="/file/Iron%20Man%20-%20V1%20194.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr" />
   </entry>
   <entry>
      <id>3</id>
      <title>Iron Man - V1 195</title>
      <link href="/file/Iron%20Man%20-%20V1%20195.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr" />
   </entry>
   <entry>
      <id>4</id>
      <title>Iron Man - V1 196</title>
      <link href="/file/Iron%20Man%20-%20V1%20196.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr" />
   </entry>
</feed>

Let me know if that makes sense.

dani · September 21, 2021, 11:01am

Also, we’ll be improving our OPDS parser to log the errors and provide better information. Future versions of Panels will record these parsing erros in the Errors section in Settings.

Rishi_Ghan · September 22, 2021, 3:55am

Thanks so much Dani!
I think I’ll try to add those field to the feed and the entries and try again.

Rishi_Ghan · September 23, 2021, 3:06pm

Hello, I was able to add the necessary nodes to the XML. Took me some finagling.
I tried serving it from HTTPS, no dice. My feed URL looks like http://localhost:8050/api/opds

I am still not able to get Panels to connect to the server.

This is currently what my feed looks like:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/terms/" xmlns:opds="http://opds-spec.org/2010/catalog">
  <id>urn:uuid:2853dacf-ed79-42f5-8e8a-a7bb3d1ae6a2</id>
  <title>title</title>
  <subtitle>subtitle</subtitle>
  <icon>/favicon.ico</icon>
  <updated>2021-09-23T15:23:57.373Z</updated>
  <entry>
    <id>0</id>
    <title>Iron Man - V1 193</title>
    <link href="/file/Iron%20Man%20-%20V1%20193.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
  </entry>
  <entry>
    <id>1</id>
    <title>Iron Man - V1 194</title>
    <link href="/file/Iron%20Man%20-%20V1%20194.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
  </entry>
  <entry>
    <id>2</id>
    <title>Iron Man - V1 195</title>
    <link href="/file/Iron%20Man%20-%20V1%20195.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
  </entry>
  <entry>
    <id>3</id>
    <title>Iron Man - V1 196</title>
    <link href="/file/Iron%20Man%20-%20V1%20196.cbz" rel="http://opds-spec.org/acquisition" type="application/x-cbr"/>
  </entry>
  <link rel="self" href="/opds-catalogs/root.xml" type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
</feed>

dani · September 24, 2021, 9:42am

Thanks for the update @Rishi_Ghan

I wrote a test and our parser can handle that feed correctly. So must be something else.

Is there any chance you could give us access to a testing server that we can make requests to?

Rishi_Ghan · September 24, 2021, 4:40pm

Sure I can email you the link, if that’s okay with you.

dani · September 24, 2021, 5:30pm

absolutely. reach me out at dani@produktstudio.com

dani · September 26, 2021, 8:01pm

I found the problem! and it’s on our side.

Panels is expecting to always store some credentials, and your OPDS server allows public access.

We’ll fix this in the next release. If you want to test it, set a username and password in Panels and it should work.

Just a heads up:

Panels does not require thumbnails to parse the file items. But they won’t appear in the “external library”.
We’ll probably reconsider this decision in the future but, for now, they won’t appear.

Rishi_Ghan · September 26, 2021, 9:11pm

That’s really awesome @dani !! Thank you for taking the time to debug the problem!

I’ll set the credentials and re-test!

Rishi_Ghan · September 27, 2021, 4:26am

Also do you mean set mock credentials in Panels in the OPDS source setup pane?

dani · September 27, 2021, 9:31am

Yeah! simply add any username + password.

like username: test ,password: test. Panels will send the credentials in the Authentication header, but the server should ignore it if auth is not required.

Rishi_Ghan · September 27, 2021, 2:15pm

Thanks @dani it worked!

dani · September 27, 2021, 2:16pm

Amazing! thanks for confirming