Verified vs. Verifiable Reproducible Builds

[Vagrant Cascadian]

On 2025-04-29, Larry Doolittle wrote:
> On Tue, Apr 29, 2025 at 11:55:40AM -0400, Matthew Suozzo via rb-general wrote:
>> * A single example of a reproduction does not convey the ability for anyone
>> to do the same, even given the same exact environment. Non-determinism in
>> some input or in the build itself could spuriously cause a success in spite
>> of the build not being reproducible to another party. I think the concept
>> of reproducibility becomes blurry when we start considering many parties
>> and diverse environments and so the idea of validating the property should
>> reflect that.
>
> Right.  Special case of the general rule "you can't prove a negative".

Simplest example that I have seen in the wild is ... a randomized list
of two elements ... will appear reproducible about 50% of the time!

There are some things that come out reproducible 99% of the time ... is
that verifiably reproducible?

The difference between something appearing reproducible once, twice,
... a hundred, a thousand ... times is significant.

There have been time bombs in test suites (e.g. expired certs) that
cause unreproducibility at a future date, but seem reproducible up till
that trigger point... (and rarely get discovered until it is too late)

It is always a matter of degrees of confidence.

But from a marketing perspective, it feels pretty weak to say "We are
pretty sure this is reproducible" ... and quite stronger to say "In our
ten-thousand tests in hundreds of subtly different build evironments, it
built reproducibly!"  ... vs. the simplicity of "It is reproducible! Yay!"

Simplistic messaging is all too powerful, even if it might be wrong. :/

So, we can lay out the conditions in which something was reproducible (a
.buildinfo file), define a confidence interval (number of builds, tried
across some date ranges, on different CPUs, built on different
filesystems, etc.), but still be tempted to use a more simplistic
message ... and wait for a counter-example? Use a simplistic message but
include the more detailed explanation in the fine print? :)


>> * Whether it is practical to reproduce a build may (and, likely, will)
>> change over time. Build inputs / environments may become unavailable or
>> costly to acquire so it may not be feasible to verify a given artifact. The
>> ability of someone to actively demonstrate a reproduction (maybe
>> "verifiable" versus "verified") seems an equally valuable asset as the
>> point-in-time measure that's currently proposed.

If it becomes impossible to recreate the build environment, I daresay it
is no longer verifiably reproducible... although having a history of
past verifications is still better than not... but not very practically
useful anymore. Hrm.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20250429/736dcda6/attachment.sig>