Reproducible Builds Verification Format

[Paul Spooren]

Hi Richard,

On Tue, 2020-05-12 at 22:14 +0100, Richard Purdie wrote:
> I'm not sure how relevant this is but I can mention what the Yocto
> Project is doing. We're not a traditional distro in that we don't ship
> binaries. We do however care a lot about getting consistent results.

> Whilst we don't ship binaries, we do cache build artefacts in our
> "sstate". This means something can be reused if its in the cache rather
> than building it again.
> 
> We thought long and hard about how to prove reproducibility and we
> ended up adding a new "selftest" to our autobuilder:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oeqa/selftest/cases/reproducible.py
> 
> This takes one set of artefacts from our "sstate" binary cache (if
> available) and builds another set locally, both in different build
> directories. It them compares the build results and flags up
> differences, saving the diffoscope html output and the differing
> binaries somewhere we can analyse them.

Perfect, that sounds like exactly what the format is supposed to handle.
Developer information (e.g. diffoscope) to solve problems and information to
end-users (or builders) to be sure the downloaded "sstates" are what they claim!

> The sstate can be built and come from any worker in our cluster so many
> different host distros and in arbitrary paths.

Such information could be saved in some Yocto specific dict/keyvalue field or
file provided next to the results. 

> We have these tests passing for our deb and ipk package backends for
> the 'core-image-minimal', 'core-image-sato' and 'core-image-full-
> cmdline' images. Sato is an X11 based desktop target.
> 
> I'd say these count as "rebuilds", even if we are testing them against
> ourselves and not worrying about signing (sstate can be signed but its
> not particularly relevant here as we trust ourselves). Not sure they're
> useful from a statistics perspective but we are running this quite
> heavily, day in, day out.

The goal of having such format would to have multiple entities rebuilding
whatever kind of artifacts and thereby making sure it's trustworthy. Trusting
your own infrastructure is great but in case you publish it other entities
should join in trying to reproduce whatever you build. And that's where
signatures become important.

I'm unfamiliar with the Yocto build process but would imagine that my local
builder only uses "sstates" provided by upstream Yocto server when it is
verified by some independent organization to be what it claims. Ideally avoiding
any maliciously distributed files by a hacked upstream server etc.

-- 
Paul Spooren <mail at aparcar.org>