Minimal Reproducible Arch Linux (4(+2) unreproducible, January 2025 status update)

[kpcyrd]

On 1/22/25 5:01 PM, David A. Wheeler via rb-general wrote:
> 
>> On Jan 22, 2025, at 6:50 AM, kpcyrd <kpcyrd at archlinux.org> wrote:
>>
>> Dear list,
>>
>> I rechecked my VM that I tried to build with "reproducible only" Arch Linux packages, last year there was only the Linux kernel missing, there have been some regressions that I've investigated.
> 
> Kudos on these investigations!

Thanks!

> In my mind, the best long-term solution is to make testing for reproducibility *part* of the
> tests for these systems. That obviously at-least-doubles the CPU for testing, but it
> would eliminate the whack-a-mole process of fixing regressions endlessly.
> Ideally such tests would be in upstream as well.

It's very difficult to predict how things will go once a "minimal 
reproducible" install is viable enough for a niche to form around it.

I could imagine some things would "self-regulate", so if some software 
regresses to often, some alternative with a proper build system will 
show up.

Or, since Python is likely not going to be reproducible in the 
foreseeable future in Arch Linux, I could see people in that 
niche-reproducible-only-community switch from Python-based tools to 
Rust-based alternatives, to avoid having that dependency on their system.

For example, when I initially setup this "reproducible only" VM last 
year, I explicitly picked something that wasn't grub, because it wasn't 
reproducible back then.

---

About 7 years ago I did in fact add a repro-test based integration test 
to one of the projects I'm upstream for, back when I had my first few 
appearances in the Reproducible Builds scene:

https://github.com/kpcyrd/sniffglue/commits/main/ci/reprotest.sh

I stopped adding this to other projects though because, generally, 
reproducible builds for my software "just works" without me needing to 
do anything.

The projects that care enough about Reproducible Builds to add something 
like this are likely already reproducible to begin with, but of course I 
appreciate if projects make efforts in this direction.

Building twice on Github Actions and checking artifacts would've caught 
the missing `gzip -n` in kbd (if started with a few seconds delay), but 
likely wouldn't have caught the terminal-width one in curl. Many things 
we consider "acceptable deviations" are only caught in the wild because 
servers operated by the same org are usually still too homogenous to 
catch _everything_.

Besides having a domain set, the wahrwolf operated rebuilder also 
noticed that grub probes how the build system was booted if no default 
is explicitly configured. The archlinux.org operated servers were both 
EFI booted, but wahrwolf's server wasn't.

> Perhaps there's a way to require that, at least before accepting an update
> (and allowing exceptions if necessary)? At least for widely-used important
> packages like curl?

There's currently some Valve sponsored work rewriting how packages are 
built/released in Arch Linux, maybe after that's done something like 
this could be integrated.

As far as I know progress is tracked over here:
https://gitlab.archlinux.org/archlinux/buildbtw/-/milestones

cheers,
kpcyrd