As you may be aware, last month, a few users reported seeing some of their snaps’ metadata (description, name, summary) being overwritten by what seemed to be old information. On closer inspection, this affected snaps for which the authors had modified the metadata via the Web publisher interface, and the metadata had reverted to the version included within the snap, which may be shorter or out-of-date.

Upon investigation, we found that a feature that was announced in June and actually enabled on August 25th is to blame for this behavior. This feature allows publishers to use the metadata included in the snap, when a build of the snap is released to the stable channel. However, it does not (or should not) upload any new metadata information if said metadata has been manually updated by the publisher through the Web interface of the Snap Store.

Why it happened

What we found is that a mistake was made when enabling the feature. The “update metadata on releasing a new build” flag was actually set to “Yes” for all snaps, whereas it should have been set to “No” for any snaps for which metadata had been previously updated via the web UI.

Once we identified the situation, we took mitigation measures, by globally disabling the “Update metadata on release” feature, to prevent any further snaps metadata from being overwritten.

What we’re doing about it

Our recovery plan will have three principal stages.

Data recovery

The main reported impact of this incident was an inadvertent modification of snaps’ metadata, which affects their visibility and discoverability.  We set out to try and recover the metadata for each affected snap. We identified a set of 428 snaps that had a release in the period of August 25th to September 3rd (when the feature was globally disabled), and thus may have been affected. 

Unfortunately, we found that our daily database backups were only kept for two days, which is great for a recovery from total database failure, but doesn’t help when the potentially destructive event happened a few days earlier. Since, we have corrected this omission, and now maintain several historical copies of backups for such an eventuality, as well as any other potential loss-of-data situations or scenarios.

We were able to recover some old snapshots of the affected snaps’ metadata from various sources and from various points in time, and we will make this information available for publishers to recreate their snaps’ metadata as needed.

Still, this information may be outdated, so we do advise publishers to carefully review and edit the metadata as appropriate. Further, we were only able to recover information for 297 of the affected snaps, and found no historical data on any of the others.

The recovered information is available as a read-only Google spreadsheet, as well as a machine-readable JSON file, and will remain accessible online until September, 2022.

Setting “Update metadata on release” correctly for each snap

For snaps whose metadata has been updated via the Web interface at any point, the individual, per-snap “Update metadata on release” flag will be set to “No”. This is the crucial step that was missed in the initial enablement of this feature. Publishers are then able to control this flag for their snap, as described in the blog post.

Re-enabling “Update metadata on release” behavior globally

Once the two measures mentioned above are in place, we will turn on the global “Update metadata on release” behavior as we did back in August. Note that the behavior is also controlled by each snaps’  individual flag. Re-enabling the global behavior will not result in all snaps being overwritten again.

Some closing thoughts

We recognize this is not optimal, and we could have executed better when it comes to keeping backups, which would have significantly reduced the impact of this issue. We know you expect better from us, and we are truly sorry we didn’t live up to your expectations in this case. We have learned from this experience and put improvements in place to ensure we have good backup coverage going forward.

If you have any questions, comments or concerns, please post in the Snapcraft forum, and we will be happy to answer them.

-The Snap Store team at Canonical.

Photo by Sarah Kilian on Unsplash.