2.9.4 - We've screwed it up - Empty Library bug post-mortem

Panels
#1

Some bugs are more annoying than others. Some affect a small group of users and some others affect mostly everyone. But one of the worse bugs an app can have is those that involve data loss. And our latest version 2.9.4 contains one of those.

Here’s what happened:

For quite a long time now, we’ve been working on rebuilding the data storage foundations of Panels. We are aiming to open the app sandbox and unlock access to the files you added to your library. This requires quite a lot of changes. And for the last 9 months, we’ve been making those changes under the hood, using the most rigorous testing practices to ensure it doesn’t affect the normal operation of the app.

Last week we made a necessary change and we screwed up. By mistake, we moved the system folder into the caches container.

Let me give you some context:

iOS sandbox gives every app access to a small portion of the mass storage drive. To keep the explanation simple, iOS expects you to use one of 4 different folders inside the sandbox (there are actually more).

  • The Temporary folder: meant to be used for temporary information.
  • The Library folder: meant to be used to store information generated by the app.
  • The Library/Caches folder: meant to be used to store information that can be re-generated.
  • The Documents folder: meant to be used to store user documents and user-generated data.

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html

The information stored in those folders has a different lifespan. iOS can delete the files in the temporary folder at any time, but never when the app is open. The content in the caches folder will much longer than the temporary folder, but the system can decide to delete it if the device is running low in disk space.

What did we do wrong?

Historically, the system folder that contains, amongst other things, the app database, was stored in the Documents folder. For no particular reason.
But now that we are planning to open the sandbox container and users will have access to it, we needed to move the system folder to another place, to prevent users from accidentally deleting their database.

The plan was to move the system folder from Documents/system to Library/Application Support/system. But accidentally we moved it into Library/Caches.

If you install 2.9.4 and you are running low on disk space, chances are that iOS will prune the folder and delete the database, containing meta-information like manual sorting, collections, and local copies of the reading sessions.

Does that mean that everyone who installed 2.9.4 lost their library information?

No. It depends on how frequent/infrequent they use their app or the amount of free space available on disk.

Do you have a fix?

This morning we released 2.9.5 that solves this issue for new users and mitigates the data loss for affected users.

What happens if I’m an affected user?

If your library appears empty after installing 2.9.4, I am afraid that your collections and manual sorting have been lost. If you have a Panels account, your reading session get automatically synchronized, so that won’t be lost.
2.9.5 restores the Library by scanning the comics folder and rebuilding the index. Unfortunately, the comics folder does not contain information about collections. The content will re-appear in the root of the library but, unfortunately, collections are impossible to be restored. :sweat:

Have you put any measures in place to prevent this from happening again?

Yes. We’ve fixed the code but also added a set of tests to ensure the system folder is protected in the correct container.

We are also planning to add additional measures to back your database up periodically so, in case of an improbable regression, we can have a backup.

Also, the imminent switch to the open sandbox approach means that the collections are represented with folders, allowing us to re-build the whole library at any time.

How many people have been potentially affected by this bug:

Since some time ago, we make our releases using phased rollout. Meaning that a release is not made available to 100% of users at once but progressively. Right after we know something wrong was happening we paused the rollout to minimize the impact.
According to our metrics, 22% of our users have 2.9.4 installed at the moment of writing this. It might grow up a bit in the upcoming hours, but all new users will download 2.9.5, and users that haven’t upgraded yet will skip it and upgrade to 2.9.5 directly.

If you have 2.9.4 installed and you haven’t lost any data, please upgrade to 2.9.5 ASAP.

Timeline of events

  • Nov 17, 2021 at 12:18 PM - 2.9.4 is released on the AppStore
  • Nov 17, 2021 at 8:51 PM - We receive the first report from a user
  • Nov 18, 2021 at 2:23 PM - We acknowledge the report and pause the release and start investigating the issue.
  • We keep testing and trying to reproduce the problem without any luck. The release is still paused.
  • Nov 21, 2021 at 3:48 PM - We receive another report by email asking if it is possible that running short in disk space could cause it. This hint gave us the idea and pointed us in the right direction.
  • We investigate the problem, found the error, and start working on a fix.
  • Nov 22, 2021 at 1:06 AM - We release a new Testflight (2.9.5 - 202111220051) including the fix and start testing it.
  • Nov 22, 2021 at 8:00 AM - In the morning I realized that the fix solves the problems for non-affected users but leaves the affected ones without a good solution. I start working on recovering the Library using the files in the documents/comics folder.
  • Nov 22, 2021 at 10:45 AM - We release a new Testflight (2.9.5 - 202111221030) with the additional logic to try to restore as much information as possible.
  • Nov 22, 2021 at 10:51 AM - We send 2.9.5 - 202111221030 for AppStore Review.
  • Nov 22, 2021 at 11:49 AM - Apple Starts reviewing the app
  • Nov 22, 2021 at 12:54 PM - 2.9.5 - 202111221030 Is approved
  • Nov 22, 2021 at 12:55 PM - The app is released to the App Store.

Wrap up

We know this post does not solve anything for those who’ve lost their work organizing their Library. But
hopefully will explain what happened and what we’ve done to prevent this from happening again.

Our sincere apologies for the error.
If you have been affected and lost some information to this bug, please contact us. We’ll try to find a way to make up to you.

Dani.

4 Likes
#2

This is the way to handle things.
Publish what happened, why and how it happened and what was done to fix it.

Keep up the good work.

1 Like
#3

At least I’m happy I was the whistleblower! You’ve handled it professionally, guys!
Keep up the good work!

1 Like
#4

I was affected by this too. No worries, I just loaded up a new batch of comics to read instead.

Shit like this happens from time to time; own up to it, fix what you can and learn from it :upside_down_face:

1 Like
#5
#6

“Also, the imminent switch to the open sandbox approach means that the collections are represented with folders, allowing us to re-build the whole library at any time.”

Does this mean that at some point all the gorgeous stacks of cover art will be replaced with folder icons? I really really really hope not. Panels would go from being alive and colorful to dull and colorless and I would have to read the text under the folder icons to tell what’s what. That’s a worst case UI scenario for Panels, imho.

I have almost a TB of comics in Panels. Countless collections that scroll for a country mile. I have no trouble finding things (or knowing how far I’ve scrolled) because of its cover-art-as-interface. I’ve made careful choices as to which comic is on top of every stack. If it all turned to folders, my carefully constructed comics ship would be sunk. Please tell me this is not what you meant.

Shit. -Joe Cunningham

1 Like
#7

Nothing will change in that regard. I didn’t phrase that correctly, sorry. I was referring to the data structure, not the visual representation. Even though collections will be folders in the file system, they will maintain their visual aspect inside Panels.

#8

That’s quite a relief. :sweat_smile: Thank you for clarifying.

Best, Joe

1 Like