Real World Reproducibility in Debian (was Re: Debian and reproducible-builds.org incoherence?)

Thu Apr 13 21:43:43 UTC 2023

On 2023-04-13, David A. Wheeler wrote:
>> On Apr 12, 2023, at 11:46 AM, Chris Lamb <chris at reproducible-builds.org> wrote:
>> This is, unfortunately, a little misleading. To clarify, this
>> statement only means that *tests.reproducible-builds.org* believes
>> that the fbreader source package is reproducible — it doesn't promise
>> that the binary packages on the official Debian mirrors are
>> bit-for-bit identical with anything.
>> 
>> This is, of course, not ideal. Still, this is what folks on this list
>> are getting at when they say they "want to make Debian 'really'
>> reproducible".
>
> Any progress on that front? What can be done to change things so that the
> packages people normally *use* are reproducible?

I think it is not nearly as bad as people think, and we undersell
ourselves when we say we do not have "real" reproducibility testing for
Debian. The work we have done and continue to do has made significant
real-world reproducibility possible!

The numbers actually looked quite good (~93%) for comparing against
packages actually in Debian Bullseye (a.k.a. the current stable release)
the last time it was done:

  https://beta.tests.reproducible-builds.org/debian_bullseye.html#debian-bullseye-amd64+all

I am not sure exactly how up-to-date it is, but the changes in Debian
stable releases are very minimal; I would not expect much has changed
since the last run.

That is very similar to our "fuzzed" build testing (~96%), though I am
somewhat surprised that it was not consistently better than the fuzz
testing numbers!

... Unfortunately, beta.tests.reproducible-builds.org testing against
packages in Bookworm and Unstable are currently lagging, with most of
the results stuck in a pending state... although of the packages that
were tested, it follows a similar curve. (e.g. vast majority are
reproducible).

The main blockers to making consistent comparisons against the packages
in the Debian archive more prominent is getting a reliable replacement
for snapshot.debian.org, to be able to rebuild with the same packages as
the packages that were used at the time of the build.

There was a talk a couple years ago that touches on many of those
challenges:

  https://debconf21.debconf.org/talks/22-making-use-of-snapshotdebianorg-for-fun-and-profit/

I've also read some recent reports suggesting that snapshot.debian.org
is missing quite a few archive states, too:

  https://bugs.debian.org/1031628

And it was marked as "wontfix" and closed...

So I do feel we really need an alternative. Or a significant turn of
events to make snapshot.debian.org *much* better.

There is significant work in progress on getting an alternative for
snapshot.debian.org going, but it can take several months to sync that
much data... and every time it gets caught up, there is more data
accumulated in the meantime... so it is not a "just do it already" kind
of thing.

Obviously, in the meantime, we are not sitting there watching the
progress bars... :)

Another approach might be to try and keep on top of the packages as soon
as they are built and we have a .buildinfo file for them, schedule a
build and to match the build environment before the package versions in
the archive change too much... but that is still going to be not
entirely systematic... but might be possible (at least for amd64/x86_64)
with a huge amount of builders and a bit more explicit coordination with
various infrastructure teams to make that more realistic.

For example, right now we sync the .buildinfo files several times a day
to https://buildinfos.debian.net ... but if we could get the .buildinfo
files to land in https://incoming.debian.org in real-time and/or some
coordination from the https://buildd.debian.org machines that perform
the builds (or the wanna-build infrastructure that coordinates
buildd.debian.org) to inform us of a new .buildinfo file... we would be
more likely to be able to get a sufficiently identical build environment
in nearly real-time.

My long-term fantasy of course, would be that https://buildd.debian.org
performs two builds and only uploads them to the archive if they are
successfully built reproducibly. We have significant enough coverage of
the archive that at least making that an opt-in feature on a per-package
basis would be a good start. (e.g. if you normalize build paths, we
should be able to reach 90-95% reproducible).

> I think it'd be better to have package build process insert tools like
> strip-nondeterminism, and then have *actual* packages reproducible, than
> be strongly confident in the packages no one uses.

Well, strip-nondeterminism is already used on the vast majority of
packages already, as it is integrated with debhelper, the main packaging
framework used in debian.

There is some academic work in what I would describe as normalized
builds(even normalizing the order of system calls and cpu ticks, if I
remember correctly), but for that to be useful, you have to normalize
the original build as well in the same way... and it is SLOW.

Surely there is broad agreement that testing against packages actually
used by people is tremendously desireable, but not having a reliable way
to consistently get historical package versions is unfortunately making
that very challenging for Debian.

I would like to add that the fuzz testing is still *extremely* useful to
test for regressions in toolchains used to build packages, and without
constantly keeping on top of that, we will never even approach practical
reproducibility. One toolchain regression can affect hundreds or
thousands of packages, and it is ideal to catch those regressions very
early!

As mentioned earlier, the fuzz testing does fairly closely mirror
real-world testing at the end of the day... so it is a reasonable proxy
while we sort out the technical issues to be able to meaningfully do
real-world comparisons (mainly, being able to recreate a build
environment with the same package versions).

So, we have a lot of moving parts.
It is not so bad, probably well over 90% for real!
It is pretty clear what we need next,
historic package archive,
even if it is not exactly clear
the exact steps to get that!

live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20230413/e57d3563/attachment.sig>