Rebuilding the official Debian live images -> nearly reproducible

[Roland Clobus]

Hello all,

I've previously reported that the official Debian live images are 
reproducible, with the remark that such statement is only valid within 
the same DAK run (i.e. within the same 6 hour time slot).

Now I've started to investigate whether long-term reproducible images 
are possible too.

Because the bookworm section is frozen until the next point release, I 
can avoid using snapshot.debian.org and work directly on deb.debian.org.

So far I've looked at the standard image and recently started looking at 
the gnome image.

I've using the same command line as in live-setup [1] and encounter a 
few differences in the generated files...

Symptoms:
1) The sorting order inside the checksum files (md5sum.txt and 
sha256sum.txt) is different
2) The file .disk/archive_trace contains a different timestamp
3) The timestamp of boot/grub/live-theme/theme.txt is different, but the 
content is the same
4) The timestamps in the source tar are the 'now' of the generation of 
the image
5) In the GNOME image, the live/filesystem.squashfs contains a 
difference in /var/cache/swcatalog/cache/C-local-metainfo.xb

Diagnosis:
1) On my test computer I have a locale set, adding LC_ALL=C before the 
invocation of the rebuild script fixes the leak from the host to the 
build environment
2) The archive trace is the timestamp of the last DAK run, for the whole 
Debian repository and will always be newer than the moment the live 
images were generated
3) When using the rebuild script, this file is copied from the git 
checkout. live-setup uses caching of the previous checkout and if there 
are no changes to this file, the timestamp of this file stays identical 
to the cached timestamp, which is older than SOURCE_DATE_EPOCH and will 
be used unchanged in the image
4) For the source image, up till now, there has been no focus on 
reproducibility
5) fonts-nanum and net.thunderbird.Thunderbird have swapped their order. 
The file C-local-metainfo.xb is probably generated by 'appstream 
refresh-cache --force'. I'll look into this later

Remedy:
1) Ensure LC_ALL=C for all sort commands on the host, fixed by [2]
2) Proposal: stop copying archive_trace into the image. The information 
that is required for rebuilding the image is already found in 
.disk/generator, .disk/info and .disk/mkisofs
3) Proposal: treat theme.txt as a configuration file (all other 
configuration files in the bootloader directory are touched)
4) This is now fixed by [3], which clamps to SOURCE_DATE_EPOCH for new 
files and directories

I've confirmed that the remedies 1 and 4 work as intended by setting 
LIVE_BUILD before invoking rebuild.sh, which results in two expected 
differences: the isoinfo 'Data preparer id' field and the .disk/mkisofs 
file refer to the current live-build version.

With kind regards,
Roland Clobus

--
[1] /home/roland/git.nobackup/live-build/test/rebuild.sh --configuration 
standard --debian-version bookworm --debian-version-number 12.1.0 
--timestamp archive --installer-origin archive --disk-info "Official 
Debian GNU/Linux Live 12.1.0 standard" --generate-source
[2] 
https://salsa.debian.org/live-team/live-build/-/commit/f38a906715d68d88d14aa670231163f7923a33f1
[3] 
https://salsa.debian.org/live-team/live-build/-/commit/d6e7b80ea0f260a21434269ae63519467e4cff6b
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20230816/371faef9/attachment.sig>