[rb-general] Debian Buster will only be 54% reproducible (while we could be at >90%)

[Holger Levsen]

hi,

disclaimer: this has not yet been verified by anyone other than myself,
so I could very well be wrong. Reproducible builds are about enabling
anyone to independently verify that... ;p


== Reproducibility in theory ==

According to https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html
we have 26476 source packages (92.8%) which can be built reproducibly in
buster/amd64, out of 28523 source packages in total. 
(These 28523 source packages build 57448 binary packages.)

But these tests are done without looking at the actual .deb files distributed
from ftp.debian.org (and we always knew that and pointed it out: 
"93% reproducible _in our current test framework_".)


== Looking at binary packages Debian actually distributes ==

So, Vagrant came up with an idea [1] to check buildinfo.debian.net for
.deb files for which 2 or more .buildinfo exist (where "exist" means
that the .deb files sha1sum is listed in the .buildinfo file) and I
turned that into a jenkins job doing this check for all 57448 binary
packages in amd64/buster/main (incl downloading all those .deb files from
ftp.d.o). 

The current main results (from this job [2]) are:

reproducible packages in buster/amd64: 30885: (53.7600%)
unreproducible packages in buster/amd64: 26543: (46.2000%)

and

reproducible binNMUs in buster/amd64: 0: (0%)
unreproducible binNMU in buster/amd64: 7423: (12.9200%)


== why are binNMUs unreproducible? ==

Because of their design, binNMUs are unreproducible, see #894441 [3] for
the details (in short: binNMUs are not what they are ment to be: the source
is changed and thrown away) and our proposed solution: 'binNMUs should
be replaced by easy "no-change-except-debian/changelog-uploads'.

So that accounts for 12%, but 12% are not enough to explain the
difference between 54% and 93%...


== packages which have not been rebuilt since December 2016 ==

And today I remember a thread I started last year in May, titled
"packages which have not been rebuilt since December 2016" [4] (because
these packages were build with an old dpkg not producing .buildinfo
files, which Chris turned into #900837 [5] "release.debian.org: 
Mass-rebuild of packages  for reproducible builds" and so today I ran
Chris' script [6] again on coccia.d.o, and today it showed that 'only'
6804 source packages need a rebuild (compared to 9192 eight months ago).

6804 of of 28523 is 23.9%. And 54%+12%+24% equals 90%. Bingo. Bummer.

(While #900837 was only filed in 2018 we knew about this issue since
2015 or so... probably earlier. Sigh.)


== After the release is before the release. ==

So, as we first need to fix #894441 before we can sensibly fix #900837 and
because Buster is practically frozen, I think we can just conclude that
Buster is quite reproducible in theory (similar but better than
Stretch...)  and that we need to make sure to address #894441 ASAP, which 
means for Bullseye, the release after Buster.

Fur future reference, a summary of the current status of Debian's
reproducibiliy is available at
https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues [7]

Happy hacking and many many thanks to everyone who has contributed so far!


[1] https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001239.html
[2] https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/103/console
[3] https://bugs.debian.org/894441
[4] https://lists.debian.org/debian-devel/2018/05/msg00499.html
[5] https://bugs.debian.org/900837
[6] https://lists.debian.org/debian-devel/2018/06/msg00007.html
[7] https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20190305/2bac5a3d/attachment.sig>