From chris at reproducible-builds.org  Fri May  1 20:20:52 2026
From: chris at reproducible-builds.org (Chris Lamb)
Date: Fri,  1 May 2026 16:20:52 -0400 (EDT)
Subject: =?utf-8?q?diffoscope_318_released_=F0=9F=92=A0?=
Message-ID: <177766671959.1102845.6013222625229804724@bigcat>

Hi,

The diffoscope maintainers are pleased to announce the release of
version 318 of diffoscope.

diffoscope tries to get to the bottom of what makes files or
directories different. It will recursively unpack archives of many
kinds and transform various binary formats into more human-readable
form to compare them. It can compare two tarballs, ISO images, or PDF
just as easily.

Version 318 includes the following changes:

  [ Chris Lamb ]
  * Upload to test PyPI integration.
  * Bump Standards-Version to 4.7.4.

  [ Manuel Jacob ]
  * Remove a misleading comment.

## Download

Version 318 is available from Debian as well as PyPI, and will
shortly be available on other platforms shortly. More details can be
found here:

   https://diffoscope.org/

??? but source tarballs may be located here:

  https://diffoscope.org/archive/

The corresponding Docker image may be run via (for example):

  $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \
      registry.salsa.debian.org/reproducible-builds/diffoscope a b


## Contribute

diffoscope is developed within the "Reproducible builds" effort.

  - Git repository
    https://salsa.debian.org/reproducible-builds/diffoscope

  - Docker image, eg.
    registry.salsa.debian.org/reproducible-builds/diffoscope
    https://salsa.debian.org/reproducible-builds/diffoscope

  - Issues and feature requests
    https://salsa.debian.org/reproducible-builds/diffoscope/issues

  - Contribution instructions (eg. to file an issue)
    https://reproducible-builds.org/contribute/salsa/


Regards,

-- 
      o
    ???   ???      Chris Lamb
   o     o     reproducible-builds.org ????
    ???   ???
      o

From imbacen at gmail.com  Fri May  1 21:56:17 2026
From: imbacen at gmail.com (cen)
Date: Fri, 1 May 2026 23:56:17 +0200
Subject: Grep not actually reproducible on Arch but do we have a bigger issue?
Message-ID: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>

Hi,

today I found an interesting case of time based non-reproducibility.

Arch rebuilderd has grep as reproducible: 
https://reproducible.archlinux.org/api/v0/builds/787931/log (build was 
in 2025)

Mine is not: https://rebuilderd.xpam.pl:2096/api/v1/builds/416039/log


Diffoscope shows differences in translations. It seems that translations 
are fetched from a remote location during bootstrap

and it just so happens that remote translations have changed since last 
year and thus introduced a diff.

The obvious fix is that translations should not be dynamically fetched 
but it got me thinking more generally.


Perhaps rebuilderd needs a feature where GOOD packages are also 
periodically rebuilt in exponential back-off style and compared against 
current upstream build and also our last GOOD build. This would confirm 
whether a package is reproducible if built in a short time window but 
also help uncover longer time window issues that are currently hidden.

This won't solve the issue entirely but at least uncover them eventually.

Such edge cases will also cause weird issues like all rebuilders 
agreeing that package is GOOD but repro-treshold failing.

Another feature that might be useful would be to add a list of 
rebuilders repro-treshold style to rebuilderd instance and upon package 
being marked GOOD, daemon could fetch hashes via API from other 
independent instances and if there is a mismatch, flag such packages for 
further inspection. Right now, even if I independently managed to match 
the exact reproducibility % of another rebuilder, I have no idea if the 
package list is the same between instances.


Best regards, cen




From kpcyrd at archlinux.org  Fri May  1 22:35:26 2026
From: kpcyrd at archlinux.org (kpcyrd)
Date: Sat, 2 May 2026 00:35:26 +0200
Subject: Grep not actually reproducible on Arch but do we have a bigger
 issue?
In-Reply-To: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
References: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
Message-ID: <dc716d26-442a-42f8-9ba9-3b5bea4382dd@archlinux.org>

On 5/1/26 11:56 PM, cen wrote:
> Hi,
> 
> today I found an interesting case of time based non-reproducibility.
> 
> Arch rebuilderd has grep as reproducible: https://reproducible.archlinux.org/ 
> api/v0/builds/787931/log (build was in 2025)
> 
> Mine is not: https://rebuilderd.xpam.pl:2096/api/v1/builds/416039/log
 >
 > Diffoscope shows differences in translations. It seems that translations are
 > fetched from a remote location during bootstrap
 >
 > and it just so happens that remote translations have changed since last year and
 > thus introduced a diff.
 >
 > The obvious fix is that translations should not be dynamically fetched but it
 > got me thinking more generally.

This is an ongoing issue with GNU software, there's more details on the page 
that tried to address some of this:

https://archlinux.org/todo/unstable-gnu-translations/

The core issue is that their git repository does not contain everything you 
need, when building from git it downloads the latest translations from some 
network server.

This problem started showing up in Arch Linux when we started implementing this 
RFC, in response to the Jia Tan XZ autotools incident:

https://rfc.archlinux.page/0046-upstream-package-sources/#transparency

For many GNU projects, the pre-processed tarballs contain both a snapshot of the 
git repository (that we can trace through the git tag), and a snapshot from the 
translations server, which we can not.

To work around this, we now specify both the git tag, and the pre-processed 
tarball as source code inputs, taking the source code from git, and the 
(versioned) translation files from tar.

You can see there's quite a difference between git and tar (also which distro is 
using what):

# grep-3.12.tar.xz
https://whatsrc.org/artifact/sha256:2649b27c0e90e632eadcd757be06c6e9a4f48d941de51e7c0f83ff76408a07b9

# grep.git#tag=v3.12
https://whatsrc.org/artifact/sha256:9543190d9ca2201ea46fddaeb39031a0acde1f6aa4351a72f33ef3455e6dd41e

# diff between them

https://whatsrc.org/diff-right-trimmed/sha256:9543190d9ca2201ea46fddaeb39031a0acde1f6aa4351a72f33ef3455e6dd41e/sha256:a1df6ede939dccc06abe879ca55dd1f793042e077165938cf63b26648343a06a

The Arch Linux grep PKGBUILD has been fixed in this commit:

https://gitlab.archlinux.org/archlinux/packaging/packages/grep/-/commit/e7a24b27b5b9a0d3c25c8e140e7f5ba6b6d80bd0

But there has been no release since.

> Perhaps rebuilderd needs a feature where GOOD packages are also periodically 
> rebuilt in exponential back-off style and compared against current upstream 
> build and also our last GOOD build. This would confirm whether a package is 
> reproducible if built in a short time window but also help uncover longer time 
> window issues that are currently hidden.
> 
> This won't solve the issue entirely but at least uncover them eventually.

That would indeed help discover problems like this, +1.

> Such edge cases will also cause weird issues like all rebuilders agreeing that 
> package is GOOD but repro-treshold failing.
repro-threshold only acts as a rebuilderd client, if you configure "I want two 
parties to confirm they could reproduce the build server output from the alleged 
source code", and two if the rebuilders you've configured as trusted could 
indeed reproduce the binaries immediately after they've been released, 
repro-threshold would accept the package as valid - unless there's a collusion, 
the security guarantees still hold.

The package eventually becoming unreproducible is "only" a problem if you later 
want to prove this to an auditor (or yourself), but can't.

> Another feature that might be useful would be to add a list of rebuilders repro- 
> treshold style to rebuilderd instance and upon package being marked GOOD, daemon 
> could fetch hashes via API from other independent instances and if there is a 
> mismatch, flag such packages for further inspection. Right now, even if I 
> independently managed to match the exact reproducibility % of another rebuilder, 
> I have no idea if the package list is the same between instances.

I did this manually in the past using the *-repro-status[0][1] tools. :) Right 
now you need to run the tool multiple times, there's no flag (yet) to query 
multiple at the same time and compare results.

[0]: https://github.com/archlinux/arch-repro-status
[1]: https://github.com/kpcyrd/debian-repro-status

cheers,
kpcyrd

From imbacen at gmail.com  Fri May  1 23:15:57 2026
From: imbacen at gmail.com (cen)
Date: Sat, 2 May 2026 01:15:57 +0200
Subject: Grep not actually reproducible on Arch but do we have a bigger
 issue?
In-Reply-To: <dc716d26-442a-42f8-9ba9-3b5bea4382dd@archlinux.org>
References: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
 <dc716d26-442a-42f8-9ba9-3b5bea4382dd@archlinux.org>
Message-ID: <79b4fcd5-7caa-4a9c-bc24-1e8ffd9ef846@gmail.com>

On 02/05/2026 00:35, kpcyrd wrote:
> repro-threshold only acts as a rebuilderd client, if you configure "I 
> want two parties to confirm they could reproduce the build server 
> output from the alleged source code", and two if the rebuilders you've 
> configured as trusted could indeed reproduce the binaries immediately 
> after they've been released, repro-threshold would accept the package 
> as valid - unless there's a collusion, the security guarantees still 
> hold.
>
> The package eventually becoming unreproducible is "only" a problem if 
> you later want to prove this to an auditor (or yourself), but can't. 

That seems correct, I was wrongly thinking about a scenario where A and 
B agree on a package but then the package would be rebuilt upstream 3 
months later for some reason and become non-reproducible.? But in 
reality that never happens because upstream never rebuilds the same 
exact thing, there is a version bump involved.


> I did this manually in the past using the *-repro-status[0][1] tools. 
> :) Right now you need to run the tool multiple times, there's no flag 
> (yet) to query multiple at the same time and compare results.
>
> [0]: https://github.com/archlinux/arch-repro-status
> [1]: https://github.com/kpcyrd/debian-repro-status 

 ?Right, so some kind of a daily scheduler that basically does the same 
thing and flags the differences. Then I can add a nice shiny badge to 
each package either saying "This package has been confirmed by N 
different rebuilders" or one that says "Rebuilders disagree about this 
package" or something like that and that's a giant red flag to 
investigate it.

It makes more sense to do it periodically and not live because the 
agreement can change over time.


As always, thank you for comprehensive response.

Best regards



From pol.dellaiera at gmail.com  Sat May  2 06:47:00 2026
From: pol.dellaiera at gmail.com (Pol Dellaiera)
Date: Sat, 2 May 2026 06:47:00 +0000
Subject: Grep not actually reproducible on Arch but do we have a bigger
 issue?
In-Reply-To: <79b4fcd5-7caa-4a9c-bc24-1e8ffd9ef846@gmail.com>
References: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
 <dc716d26-442a-42f8-9ba9-3b5bea4382dd@archlinux.org>
 <79b4fcd5-7caa-4a9c-bc24-1e8ffd9ef846@gmail.com>
Message-ID: <420437fc-00ca-4d88-a3b6-39c41bf59ce5@gmail.com>

Hi,


I think this is very similar to what happened in `NixOS/nixpkgs` with 
grub2: https://github.com/NixOS/nixpkgs/issues/245356

In that case, it was reported as unreproducible because localisation 
resources were coming from a nondeterministic translation source 
(network server). The fix by Arnout was to make those resources 
deterministic by taking only the localisations from the release tarball 
instead: https://github.com/NixOS/nixpkgs/pull/316655

The issue was definitely an issue, but it was gracefully fixed once the 
non-deterministic input path was identified.


Best regards

-- 
Pol Dellaiera

On 2026.05.02 01:15, cen wrote:
> On 02/05/2026 00:35, kpcyrd wrote:
>> repro-threshold only acts as a rebuilderd client, if you configure "I 
>> want two parties to confirm they could reproduce the build server 
>> output from the alleged source code", and two if the rebuilders you've 
>> configured as trusted could indeed reproduce the binaries immediately 
>> after they've been released, repro-threshold would accept the package 
>> as valid - unless there's a collusion, the security guarantees still 
>> hold.
>>
>> The package eventually becoming unreproducible is "only" a problem if 
>> you later want to prove this to an auditor (or yourself), but can't. 
> 
> That seems correct, I was wrongly thinking about a scenario where A and 
> B agree on a package but then the package would be rebuilt upstream 3 
> months later for some reason and become non-reproducible.? But in 
> reality that never happens because upstream never rebuilds the same 
> exact thing, there is a version bump involved.
> 
> 
>> I did this manually in the past using the *-repro-status[0][1] 
>> tools. :) Right now you need to run the tool multiple times, there's 
>> no flag (yet) to query multiple at the same time and compare results.
>>
>> [0]: https://github.com/archlinux/arch-repro-status
>> [1]: https://github.com/kpcyrd/debian-repro-status 
> 
>  ?Right, so some kind of a daily scheduler that basically does the same 
> thing and flags the differences. Then I can add a nice shiny badge to 
> each package either saying "This package has been confirmed by N 
> different rebuilders" or one that says "Rebuilders disagree about this 
> package" or something like that and that's a giant red flag to 
> investigate it.
> 
> It makes more sense to do it periodically and not live because the 
> agreement can change over time.
> 
> 
> As always, thank you for comprehensive response.
> 
> Best regards
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20260502/2cf0fa0e/attachment.sig>

From imbacen at gmail.com  Mon May  4 19:00:47 2026
From: imbacen at gmail.com (cen)
Date: Mon, 4 May 2026 21:00:47 +0200
Subject: Grep not actually reproducible on Arch but do we have a bigger
 issue?
In-Reply-To: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
References: <454b7074-e4c0-4c56-a9af-e1c3ff3850d3@gmail.com>
Message-ID: <b773e113-3b86-4a03-bdb4-e16336072af0@gmail.com>

On 01/05/2026 23:56, cen wrote:
> Another feature that might be useful would be to add a list of 
> rebuilders repro-treshold style to rebuilderd instance and upon 
> package being marked GOOD, daemon could fetch hashes via API from 
> other independent instances and if there is a mismatch, flag such 
> packages for further inspection. Right now, even if I independently 
> managed to match the exact reproducibility % of another rebuilder, I 
> have no idea if the package list is the same between instances. 

Follow up, I hacked together this feature over the weekend as a 
scheduled task and storing results in a new table.


Arch results: 
https://rebuilderd.xpam.pl:2096/?distro=archlinux&release=&arch=x86_64&status=GOOD&has_disagreement=true&peer=reproducible.archlinux.org%2Creproducible.crypto-lab.ch

Debian results: 
https://rebuilderd.xpam.pl:2096/?distro=debian&release=sid&arch=amd64&status=GOOD&has_disagreement=true&peer=reproduce.debian.net


I have 29 more good packages on Arch and 36 on Debian.

The reverse, locally-bad-upstream good are much more but most of those 
are likely infra related issues I still need to go through on my end.


-?"Show peer disagreements" checkbox lists packages where my rebuilderd 
disagrees with any of the peers

- Individual peers can be toggled on/off

- Clicking on the package name shows you remote peer status and direct 
remote log links for inspection.


Initially I also fetched all hashes via attestations to try and see if 
there is a wild existence of good-good-different-hash in the wild but 
found none, so I removed it.



Best regards






From chris at reproducible-builds.org  Tue May  5 19:20:21 2026
From: chris at reproducible-builds.org (Chris Lamb)
Date: Tue, 05 May 2026 12:20:21 -0700
Subject: Please review the draft for April's report
Message-ID: <177800872953.1786570.15771353399313547868@bigcat>

Hi all,

Please review the draft for April's Reproducible Builds report:

  https://reproducible-builds.org/reports/2026-04/?draft

? or, via the Git repository itself:

  https://salsa.debian.org/reproducible-builds/reproducible-website/blob/master/_reports/2026-04.md

I intend to publish it no earlier than:

  $ date -d 'Thu, 07 May 2026 20:00:00 +0000'

  https://time.is/compare/2000_07_May_2026_in_UTC

?

Please feel free and commit/push to drafts directly without the overhead of
sending patches or merge requests. You should make your changes to the
"_reports/2026-04.md" file in the "reproducible-website" repository:

  $ git clone https://salsa.debian.org/reproducible-builds/reproducible-website
  $ cd reproducible-website
  $ sensible-editor _reports/2026-04.md

I am happy to reword and/or rework additions prior to publishing. If you
currently do not have access to the above repository, you can request access
by following the instructions at:

  https://reproducible-builds.org/contribute/salsa/


Regards,

-- 
      o
    ?   ?      Chris Lamb
   o     o     reproducible-builds.org ?
    ?   ?
      o

