OPDS Downloads Failing

dani · January 15, 2023, 4:52pm

Hey @Clay , we just released a new Testflight with some fixes in the OPDS integration. Could you try it and let us know if you can download files as expected?

I tested the integration with your test account and could download several files without a problem.

Hopefully, this version works better for you.

timbroder · January 17, 2023, 9:44pm

I’m having the same problem on Komega v0.157.5-master. I’ve tried the testflight but it did not alleviate

The steps I’m taking are

Go into the remote library
Go into a folder that has 25 manga issues in it. Each issue is 5-30 megs
Select all
copy to iPad

I can provide a test account and more info if that’s helpful. Just let me know where to send it

Thanks

timbroder · January 17, 2023, 9:45pm

I should note that I don’t have this problem with Mylar

timbroder · January 17, 2023, 9:48pm

Interesting. I’m noticing I’m having issues with a specific file that’s locking it up. What’s the best way to get you logs or access?

Clay · January 18, 2023, 4:09am

Yep, same problem/behavior with the TestFlight version.

Should note that it only happens if I queue up several (like 6+) for download.

Small numbers it handles just fine.

dani · January 19, 2023, 4:50pm

We are kind of at a dead end with this, folks. I’ve tested the downloads using @Clay 's testing account, and I can download many comics at once without a problem. I just tested it again and could download 54 files in one go.

Could you find some diagnostic logs in your device(s) and send them to us? Sending App Diagnostics

Maybe we find something that helps us understand what’s going on.

Clay · January 20, 2023, 6:21pm

I sent one - hopefully it helps! Also send the crash reports from TestFlight

dani · January 21, 2023, 2:22pm

Thank you so much @Clay

We found something in your logs, and we have made some changes. We weren’t getting any crash reports because the app wasn’t crashing, but iOS was killing it. It looks like Panels was locking the main thread on your device for a few seconds, and that caused the system to kill the app.

Hopefully, the new Testflight works better. Let us know if it made any difference.

Clay · January 22, 2023, 3:08am

So it no longer seems to freeze the app, but the app still gets stuck at 98%, and the menus stop responding. See screenshots.

Unfortunately it’s no longer crashing/getting killed so I can’t find additional logs to send

dani · January 22, 2023, 10:50am

WTF

I’m starting to think the problem has nothing to do with the OPDS download but maybe with something in your local library. The last 2% of the download process corresponds to indexing the file locally. If it reaches 98%, the download is completed correctly, and it is tracking it in the local database.

There is additional information you can send to us. If you go to the app settings/profile, there is an Email Us button in the Contact Us section. If you tap and hold for a few seconds, the app will attach additional debug information to the email. Could you please send it to us? The debug data includes statistics about your local library.

We can try reproducing a similar scenario on our end and see if the app gets stuck.

Clay · January 22, 2023, 7:34pm

Ok, sent you a few emails.

recording of seeing it happen with the 01/22 build
debug output after clean install
debug output after adding the OPDS source and creating a folder to download into
debug output after failed downloads. Whatever impacts the menus seems to prevent generating the debug email, so i had to kill the stuck files and deleted the queued ones, as well as restarting. Not sure if that will still give you useful details, but it was the only option I’ve got.

Vandor · January 23, 2023, 3:27am

Hi! Not sure if this is helpful but I have the EXACT same app behavior except I source from Dropbox.

Noticed for first time 3-4 weeks ago. I had not had any issues prior.
Only happens when downloading multiple titles. Sometimes I can get away with 5-6 but at some point one will stick at 98%
Once one sticks the app becomes unusable - even after killing it, restarting my iPad, etc. Only clean uninstall allows it to be usable again.
I agree something seems to be happening after the download but before it gets marked complete

I’ll keep watching this thread, will send debug info after failed downloads.

Thanks!

Clay · January 23, 2023, 4:07am

One of us!

dani · January 23, 2023, 5:38pm

Could you install the latest Testflight 2.13.13 (202301231515) and let us know if it made any difference?

Vandor · January 23, 2023, 5:54pm

My first test went great! Bulk import of 109 comics no issue.

You removed a checksum?

Vandor

dani · January 23, 2023, 7:28pm

Amazing, that’s great to read @Vandor .

You removed a checksum?

Not at all. We re-implemented it using different APIs.

For those interested in technical details:

Until now, we have been using the open-source library Checksum to calculate them.
Looking at the implementation Checksum/Extensions.swift at develop · rnine/Checksum · GitHub, we noticed that, even though checksums are returned synchronously, it uses DispatchGroups that internally use thread locks.
This is unnecessary in our use case because we already have our own thread model and can wait for long-lasting tasks if needed.

I assume that those thread locks are causing deadlocks on some devices.
On top of that, Checksum uses CommonCrypto, an open-source cryptographic library by Apple that has been deprecated in favor of a more modern CryptoKit library.

We used Checksum for a long time because CommonCrypto is not very ergonomic in Swift, and we preferred using an open-source wrapper that many people would use, to make sure our usage of CommonCrypto was correct and safe. CommonCrypto provides a C API which is more finicky in terms of memory management and things like that.

The last build we released today contains our own implementation for calculating the file checksums using CryptoKit, which provides a nice Swift API.
Our new implementation doesn’t use thread locks; hopefully works the problem around.

I will wait for @Clay 's feedback before assuming this issue is fixed. But it makes sense and matches what I saw in one of Clay’s logs.

Vandor · January 23, 2023, 7:57pm

Awesome - thanks for the info!

Clay · January 24, 2023, 1:19am

BY JOVE I THINK YOU’VE GOT IT

pulled 2 separate 20+ downloads. no problems.

THANK YOU

dani · January 24, 2023, 9:33am

YES !!!

I’m glad it works now! Thank you so much for the cooperation and patience

Telecart · February 17, 2024, 10:04pm

I downloaded the app for the first time today
I just sent a refund request for the $20 since I thought it must be a scam. Glad I googled it but yeah I see exactly this latest problem of everything getting stuck at 98% progress and all the buttons starting to misbehave and be unresponsive, only I’m not doing it over OPDS but over SMB connection
Otherwise exactly as described here