Introducing: Semantically reproducible builds

[Vagrant Cascadian]

On 2023-05-28, David A. Wheeler wrote:
> On Sun, 28 May 2023 13:04:40 +0100, James Addison via rb-general <rb-general at lists.reproducible-builds.org> wrote:
>> Thanks for sharing this.
>> 
>> I think that the problem with this idea and name are:
>> 
>> - That it does not allow two or more people to share and confirm that
>> they have the same build of some software.
>
> Sure they can, they just use the same process (e.g., use the same tool to
> verify it). E.g., if you rebuild it, and the two builds are the same EXCEPT
> for the datetime stamps, it's semantically reproducible (not fully reproducible).

Do such tools actually exist, or are we talking about something
theoretical here?  I am nervous about investing too much energy in
something without a specific, precise, working proof of concept.

In your earlier mention of OSSGadget, it was not immediately clear that
anything in there could actually do this sort of analysis... ?


>> - That it does not allow tests to fail-early, catching and preventing
>> reproducibility  regressions (semantic or otherwise).
>
> It's *possible* to fail early, though the CPU requirements are admittedly
> higher (because you have to do much more than a bit-for-bit test).
>
> But I expect that in practice, the use of "semantically reproducible builds"
> is long *after* any CI/CD process of the package being analyzed.
> The problem, in many cases, is that the package was not created in a way
> that supports reproducible builds, so the goal is to try to estimate the
> risk of the package when it is *not* a reproducible build.

I still expect it will be harder to actually do "semantically
reproducible builds" than "fully reproducible builds".

To be honest, it sounds like a lot of extra work to avoid fixing things
properly...

I find it hard to believe it could so close that you can programatically
determine something is (probably!) mostly harmless and yet still have it
be implausible to go all the way to make a properly reproducible build.

That flys in the face of the thousands of packages I have personally
reviewed, and submitted patches for hundreds of them... sometimes only
partially successful, but in the vast majority of cases I end up staring
at megabytes of leftover gibberish... or something bit-for-bit
reproducible.

The main type of issue you have mentioned, timestamps, is usually the
easiest one to identify and actually remove from resulting builds.

The other one, arbitrary files that end up in the package? That also
sounds like it ought to be something easy to fix; part of the build
process should remove files known to arbitrarily appear if they are
truely non-functional and not intended to be part of the built
artifacts. Although why these files appear arbitrarily and unpredictibly
is a bit of a concerning situation...

The only solid case I can think of off the top of my head would be
embedded cryptographic signatures, but as I understand it, there are
projects (android apks used by f-droid? rpm?) that have some ways to
deal with this, even if they are less than ideal.

If there are still outstanding issues on a given piece of software (or
toolchain), it should be tracked in some sort of issue tracker...


>> - That the naming terminology conflates with true reproducible builds,
>> therefore creating the potential for misunderstanding to consumers.
>
> Naming is hard. As long as the term is carefully defined I think it works.
> You can use "fully reproducible build" when you want to contrast, and
> that makes it clear that a normal "reproducible build" is the stronger test
> (at the cost of being sometimes harder to achieve).

I do not like the idea of calling this thing using the Reproducible
Builds name... it is hard enough to get across the bit-for-bit identical
meaning of Reproducible Builds, and using it to mean something else will
inevitibly dilute that meaning, no matter how many other very clearly
and appropriately selected words you prefix it with.

I very much worry that the meaning of Reproducible Builds may gradually
get whittled down by adding more exceptions up to the point of being
little more than a checkbox in a compliance checklist... with little
actual benefit to the world at large.

The primary benefit of Reproducible Builds is that it is easy to
actually verify the result is in fact reproducible... or not.


Well, I appear to have strongly held opinions!


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20230528/ea23e050/attachment.sig>