Verification Builds and Snapshots For Debian

[Vagrant Cascadian]

On 2023-09-30, Vagrant Cascadian wrote:
> On 2023-09-20, Lucas Nussbaum wrote:
>> On 19/09/23 at 13:52 -0700, Vagrant Cascadian wrote:
>>> * Looking forward and backwards at snapshots
>>> 
>>> I do think that a more complete snapshot approach is probably better
>>> than package-specific snapshots, and it might be worth doing
>>> forward-looking snapshots of ftp.debian.org (and security.debian.org and
>>> incoming.debian.org), in addition to trying to fill out all the missing
>>> past snapshots to be able to attempt verification builds of older
>>> packages, such as all of bookworm.
>>> 
>>> Snapshotting the archive(s) multiple times per day, today, tomorrow, and
>>> going forward will at least enable doing verification rebuilds of
>>> packages starting from this point, with less immediate overhead than
>>> trying to replicate the entire functionality or more complete history of
>>> snapshot.debian.org.
>
> In the meantime, I worked on a naive implementation of this, using
> debmirror and btrfs snapshots (zfs or xfs are other likely candidates
> for filesystem-level snapshots). It is working better than I expected!

xfs seems to work well enough too, by using "cp --archive --reflink" to
produce snapshots:

  http://snapshot.reproducible-builds.org/snapshot-experiment/archive/debian/

A little klunkier than btrfs snapshots, but workable.

It would also be possible with btrfs or xfs (and presumably zfs somehow)
to make by-checksum reflinked copies, which might make it possible to
(at least partially) reassemble a repository from the contents if you
have the relevent Release, Packages and Sources files...

I have not migrated all of the snapshots captured on btrfs ... because
it is a bit tricky to sync the files efficiently... What becomes tricky
is efficiently transferring those files to another machine or making
backups... rsync can break the reflinked files if you are not
careful. There are patches for rsync to support efficiently reflink:

  https://github.com/WayneD/rsync/issues/119
  https://github.com/WayneD/rsync/issues/153


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20231006/04f473d6/attachment.sig>