Call for real-world scenarios prevented by RB practices

[Dan Shearer]

On 22/03/2022 13.46, Chris Lamb wrote:

> Just wondering if anyone on this list is aware of any real-world
> instances where RB practices have made a difference and flagged
> something legitimately "bad"?

This is an interesting pointi Chris. I too have looked for such instances and I
concluded they are rare, and that is why reproducibility is not high on most
people's priority lists.

Reproducibility feels to me a bit like Y2K. It has driven code reviews and many
bugfixes, and there is a boolean pass/fail way of measuring reproducibility.
But reproducibility doesn't improve security any more or less than better
coding practices, chain of custody efforts or commit signing - and, for
example, none of these are a defence again Ken Thompson's famous 1984 paper
despite some of the best minds in computer science working on it. And also
like Y2K, person-centuries of effort can be put into improving every aspect of
the code ecosystem and it can never be proved that it prevented planes falling
out of the sky (although there is plenty of evidence that many bad things were
much less likely after Y2K mitigation.)

For that reason, I don't think reproducibility is a good single-purpose driver
for general-purpose projects. In the same way that OpenBSD produced security
results in a handful of applications that prompted better practices everywhere,
I think NixOS has helped highlight the benefits of reproducibility. But I am
unlikely to introduce either of these into an organisation because I see no
evidence or even good arguments that it will improve the quality, reliability
or privacy aspects of critical infrastructure.

Is this reasonable?

--
Dan Shearer
dan at shearer.org