Does diffoscope compares disk partitions -> The German word is 'jein' (yes and no)

[Roland Clobus]

Hello,

On 01/03/2023 22:59, Thomas Schmitt wrote:
> Mattia Rizzolo wrote:
...
> ... The file tree and the data files' content is only
> a part of a bootable ISO 9660 image. There's executable code in the
> blind spots of such a view: MBR legacy BIOS boot code, EFI programs in
> the EFI partition, "hidden" El Torito boot images, ...
> 
>> I know that thing has been used to work on reproducible bootable image,
> 
> Possibly because typical pitfalls aren't tested. Like FAT timestamps in
> the EFI partition.

In live-build, the FAT timestamps in the EFI partition are carefully 
ironed out [1][2]. Before the fixes, they did show up in the diffoscope, 
but only in the hex dump.
After I realised that the differences in the hex dump were actually 
timestamps (or timestamp related), I could remove the differences.

 From the original mail:
** 2023-02-27 14:09:12 D: diffoscope.comparators.utils.command: 
Executing xxd {}
** Killed

I've seen this too. You'll need loads of memory and temporary disk 
space, especially in the initial stages of finding the root cause of the 
differences (comparing 2 xxd outputs of 6GB files can produce lots of 
output)
The original mail also hint at a possible root cause: the timestamp 
inside each partition have not been clamped to SOURCE_DATE_EPOCH. See 
e.g. [3] how it is done in live-build.

 > It is possible to make reproducible ISOs by xorriso and 
SOURCE_DATE_EPOCH.
 > But the input files for the ISO production have to be identical, too.

Indeed, before the meta-content of ISO files can be considered, its 
payload should have been harmonized. Live-build does it in [3].

 > Both, the use of SOURCE_DATE_EPOCH and care for identical input are 
not tradition with bootable ISO production.
 >
 > So another reason for no protests might be that bootable ISOs aren't 
often challenged by reproducibility tests which use diffoscope.

I agree that having a specialised comparator would be nice, but even the 
hex dump has fulfilled its purpose. The live ISO images are now 100% 
reproducible (the header is containing a lot of the magic in the 
previous mails). The ISO images are tested with diffoscope on a daily 
basis in Jenkins [4].

With kind regards,
Roland

[1] 
https://sources.debian.org/src/live-build/1:20230131/scripts/build/binary_grub-efi/?hl=262#L262
[2] 
https://sources.debian.org/src/live-build/1:20230131/scripts/build/efi-image/?hl=79#L79
[3] 
https://sources.debian.org/src/live-build/1:20230131/scripts/build/binary_iso/?hl=180#L180
[4] https://jenkins.debian.net/view/live/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20230302/8f35da2a/attachment.sig>