citests vs. (verification |re)builds

[kpcyrd]

On 11/13/22 22:59, Vagrant Cascadian wrote:
> I'm not sure how exactly to structure a rewording or adjustment of the
> website and whatnot, but would like to start the conversation, at least!

Thanks for bringing this up, maybe we should be more explicit what's 
being tested, this is currently not clear when looking at 
https://reproducible-builds.org/citests/.

I'd suggest having a page (and also place it more prominently) that is 
more explicit around this:

-- 8< --

## Verification Builds (imo this is the only true reproducible builds)

Binary artifacts are downloaded and compared to binaries built from 
source (using the official buildinfo file as additional build input, if 
the projects needs one for reproducible builds).

https://reproducible.archlinux.org/ (Arch Linux)
https://beta.tests.reproducible-builds.org/ (Debian, Qubes)
https://r-b.engineering.nyu.edu/ (Arch Linux)
https://rebuilderd.dustri.org/ (Tails)

## Build Environment Fuzzing

The source code is downloaded and built 2+ times in a diverse set of 
environments.

https://tests.reproducible-builds.org/archlinux/
https://tests.reproducible-builds.org/coreboot/
https://tests.reproducible-builds.org/debian/
https://tests.reproducible-builds.org/freebsd/
https://tests.reproducible-builds.org/netbsd/
https://tests.reproducible-builds.org/openwrt/
https://reproducible-builds.openeuler.org/
https://www.yoctoproject.org/reproducible-build-results/

## Unclear

I don't know what these services are doing, can somebody help categorize 
them?

https://data.guix.gnu.org/repository/1/branch/master/latest-processed-revision/package-reproducibility
https://r13y.com/
http://rb.zq1.de/compare.factory/
https://qubesos.gitlab.io/qubes-g2g-report/

-- >8 --

They both serve different purposes, Build Environment Fuzzing helps 
detect issues before they show up during Verification Builds but can 
also mislead, if you already have a diverse set of Verification Builders 
and they never run into the issue, is there an issue to begin with?

I also think the page listing this should be placed higher than "Who is 
involved?" on the website, having results to show is a much higher 
involvement than having a manual somewhere.

PS: vagrant, please get an irc bouncer.

cheers,
kpcyrd