[EXTERNAL] Re: Reproducible Builds Verification Format

[kpcyrd]

On Fri, May 15, 2020 at 08:51:50PM +0000, Jason Zions via rb-general wrote:
> kpcyrd:
> > The argument was that a debian/arch rebuilder *always* needs to take
> > the buildinfo file as a rebuild input. That's the reason the buildinfo is
> > shipped inside the arch package, collecting detached buildinfo files is a
> > debian thing, but only the buildinfo file for the build that was actually
> > uploaded into the archive is useful for anything.
> 
> This is one of the challenges we face today. The buildinfo file is required to rebuild a package. That's fine, most of the time. When an upstream team issues a patch (e.g. a fix for a security issue), I need to build the updated package immediately and get it into the hands of my users. It's often the case that, when I build that patched package, there's no buildinfo file yet because a build hasn't yet appeared in the Debian repo. It might be 24-48 hours before that package appears. For security patches, that delay is troublesome.

This is a very interesting aspect that I didn't consider before, but I'm
not sure if that system is technically a debian rebuilder. You're also
building ahead of the debian source upload if I'm reading this right, so
debian might end up patching something differently, resulting in a
different binary. :)

In debian you'd (usually) need to wait until the source upload happened
before you're able to know which value you need to set SOURCE_DATE_EPOCH
to (by looking at debian/changelog). In arch you need the buildinfo file
inside the package to set S_D_E correctly.

The delay for security updates is indeed a problem that no distro has a
solution for, as far as I know. Ideally source and binary would be
released at the same time and then it takes $TIME_TO_BUILD_THE_PACKAGE
until the first rebuilder confirmations become available.

> In Marrakesh, we talked about the distro maker just being one among multiple rebuilders; often the "first one in", but not required to be first in. It seems to me that we'd want each rebuilder to
>  - build without an input buildinfo for a given source if none is available from the clearinghouse
>  - record its output buildinfo and checksum information in the clearinghouse
>  - receive an automated notification if another rebuilder recorded a different buildinfo/checksum for the same source
> 
> Most of the time, the Debian build process would be first-in; it would build without an input buildinfo and record the buildinfo and checksums in the clearinghouse. Rebuilders would then rebuild, using the recorded buildinfo, and record the checksum they got. Any differences would trigger email to all the builders.
> 
> If rebuilders got out ahead of the Debian process, they would build (without buildinfo), then record their buildinfo and checksums. Multiple rebuilders in parallel might do this; as soon as the second rebuilder completes, conflicts would be detected and raised to human eyes for resolution.

Clearinghouse is a term I didn't run into yet, I'm assuming this
describes a central system that collects the package and ensures the
rebuilders confirmed it before releasing the package to the users
(please let me know if I guessed incorrectly).

Building ahead of the debian builders is an optimization that could
potentially cause complexity down the road, the build environment is
more likely to change in the meantime if you're rebuilding packages in
sid. It's also a distro specific approach because you need to know in
advance which value SOURCE_DATE_EPOCH is going to be set to and not all
distros can do that.