citests vs. (verification |re)builds

[Vagrant Cascadian]

On 2022-11-13, kpcyrd at archlinux.org wrote:
> On 11/13/22 22:59, Vagrant Cascadian wrote:
>> I'm not sure how exactly to structure a rewording or adjustment of the
>> website and whatnot, but would like to start the conversation, at least!
>
> Thanks for bringing this up, maybe we should be more explicit what's 
> being tested, this is currently not clear when looking at 
> https://reproducible-builds.org/citests/.

Yeah, clarifying what type of builds are performed would be very
helpful...


> I'd suggest having a page (and also place it more prominently) that is 
> more explicit around this:
>
> -- 8< --
>
> ## Verification Builds (imo this is the only true reproducible builds)
>
> Binary artifacts are downloaded and compared to binaries built from 
> source (using the official buildinfo file as additional build input, if 
> the projects needs one for reproducible builds).
>
> https://reproducible.archlinux.org/ (Arch Linux)
> https://beta.tests.reproducible-builds.org/ (Debian, Qubes)
> https://r-b.engineering.nyu.edu/ (Arch Linux)
> https://rebuilderd.dustri.org/ (Tails)
>
> ## Build Environment Fuzzing
>
> The source code is downloaded and built 2+ times in a diverse set of 
> environments.
>
> https://tests.reproducible-builds.org/archlinux/
> https://tests.reproducible-builds.org/coreboot/
> https://tests.reproducible-builds.org/debian/
> https://tests.reproducible-builds.org/freebsd/
> https://tests.reproducible-builds.org/netbsd/
> https://tests.reproducible-builds.org/openwrt/
> https://reproducible-builds.openeuler.org/
> https://www.yoctoproject.org/reproducible-build-results/

This seems like a reasonable start, thanks!


> ## Unclear
>
> I don't know what these services are doing, can somebody help categorize 
> them?
>
> https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility

This is a bit hard to place... neither really "verification" nor
"fuzzing" builds.

It is comparing builds from (at least) two different build farms
(a.k.a. substitute servers). The build environment is largely
normalized, but some differences do reveal themselves (e.g. differing
cpu implementations).

Any user of guix could build a given package, and compare the results to
publicly available substitute servers. Guix is fundamentally a
source-based distro; binary substitutes of packages are an opt-in
(opt-out?) performance optimization...

I'm not sure there is an equivalent to a .buildinfo; there is a
derivation file, which describes the inputs used in the build process,
and which are used to define a given build as essentially a hash of the
inputs. The .narinfo might contain the derivation file and I think the
signed hashes of the build result, which starts to look almost like a
.buildinfo file... but I'm not sure.

There is no "original" build in the guix context to verify against, but
it is (at least) two builders comparing against each other... so is
arguably not a "verification build" but the build environment is
normalized as much as possible instead of intentionally fuzzing the
build environment.


> https://r13y.com/

I would guess Nix is similar to Guix here, but do not know for sure.


> They both serve different purposes, Build Environment Fuzzing helps 
> detect issues before they show up during Verification Builds but can 
> also mislead, if you already have a diverse set of Verification Builders 
> and they never run into the issue, is there an issue to begin with?

With normalized build environments, this is significantly less of an
issue... so I can see what you are getting at!

Here are some other angles to consider...

Doing two consecutive builds and comparing them is really helpful to see
if toolchain fixes actually worked or not, without re-uploading all the
packages to the "official" archive; doing verification builds would
necessarily use the unfixed toolchains of the original .buildinfo file.

Fuzzing does sometimes reveal bugs that would not be revealed in a
normalized build environment; for example, I have seen error messages
reproducibly getting embedded in manpages, but fuzzing with a different
locale actually revealed the issue(or some other case), and actually
lead to removing what triggered the embedded error in the first
place. Fuzzing in this way can actually reveal papered-over or hidden
issues.

Fuzzing style building can gain greater confidence in the results with a
smaller number of builders. It does not rely as much on unintentional
subtle differences between a diverse pool of builders discovering
issues, because you are actively seeking out issues through intentional
variations.

In conclusion, I still think they're all valuable; let us all scratch
our own particular itches. :)


> I also think the page listing this should be placed higher than "Who is 
> involved?" on the website, having results to show is a much higher 
> involvement than having a manual somewhere.

Sounds reasonable to me.


> PS: vagrant, please get an irc bouncer.

No thanks. I used to keep a permanent irc connection going and was very
happy to discover the value of logging off when I am not actively paying
attention. :P


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20221113/0c414f15/attachment.sig>