Reproducible Builds in June 2025🔹
Chris Lamb
chris at reproducible-builds.org
Mon Jul 14 17:15:30 UTC 2025
--------------------------------------------------------------------
o
⬋ ⬊ June 2025 in Reproducible Builds
o o
⬊ ⬋ https://reproducible-builds.org/reports/2025-06/
o
--------------------------------------------------------------------
Welcome to the 6th report from the Reproducible Builds project in
2025.
Our monthly reports outline what we've been up to over the past
month, and highlight items of news from elsewhere in the increasingly-
important area of software supply-chain security. If you are interested
in contributing to the Reproducible Builds project, please see the
Contribute [0] page on our website.
[0] https://reproducible-builds.org/contribute/
§
In this report:
* Reproducible Builds at FOSSY 2025
* Distribution work
* diffoscope
* OSS Rebuild updates
* Website updates
* Upstream patches
* Reproducibility testing framework
§
Reproducible Builds at FOSSY 2025 [2]
-------------------------------------
On Saturday 2nd August, Vagrant Cascadian and Chris Lamb will be
presenting at this year's FOSSY 2025 [3]. Their talk, titled "Never Mind
the Checkboxes, Here's Reproducible Builds!" [4], is being introduced
as follows:
> There are numerous policy compliance and regulatory processes being
> developed that target software development... but do they solve
> actual problems? Does it improve the quality of software? Do
> Software Bill of Materials (SBOMs) actually give you the information
> necessary to verify how a given software artifact was built? What is
> the goal of all these compliance checklists anyways... or more
> importantly, what *should* the goals be? If a software object is
> signed, who should be trusted to sign it, and can they be trusted
> ... forever?
The talk will introduce the audience to Reproducible Builds as a set of
best practices which allow users and developers to verify that software
artifacts were built from the source code, but also allows auditing for
license compliance, providing security benefits, and removes the need to
trust arbitrary software vendors.
Hosted by the Software Freedom Conservancy [5] and taking place in
Portland, Oregon, USA, FOSSY aims to be a community-focused event:
"Whether you are a long time contributing member of a free software
project, a recent graduate of a coding bootcamp or university, or just
have an interest in the possibilities that free and open source software
bring, FOSSY will have something for you". More information on the event
is available on the FOSSY 2025 website [6], including the full programme
schedule [7].
Vagrant and Chris will also be staffing a table this year, where they
will be available to answer any questions about Reproducible Builds and
discuss collaborations with other projects.
[2] https://2025.fossy.us/schedule/presentation/327/
[3] https://2025.fossy.us/
[4] https://2025.fossy.us/schedule/presentation/327/
[5] https://sfconservancy.org/
[6] https://2025.fossy.us/about/
[7] https://2025.fossy.us/schedule/
§
Distribution work
-----------------
In Debian [8] this month:
* Holger Levsen has discovered that it is now possible to bootstrap a
minimal Debian 'trixie' using 100% reproducible packages. This result
can itself be reproduced, using the debian-repro-status tool and
mmdebstrap's support for hooks:
$ mmdebstrap --variant=apt --include=debian-repro-status \
--chrooted-customize-hook=debian-repro-status \
trixie /dev/null 2>&1 | grep "Your system has"
INFO debian-repro-status > Your system has 100.00%
been reproduced.
* On our mailing list [9] this month, Helmut Grohne wrote an extensive
message raising an issue related to "Uploads with conflicting
buildinfo filenames" [10]:
> Having several .buildinfo files for the same architecture is
> something that we plausibly want to have eventually. Imagine
> running two sets of buildds and assembling a single upload
> containing buildinfo files from both buildds in the same upload.
> In a similar vein, as a developer I may want to supply several
> .buildinfo files with my source upload (e.g. for multiple
> architectures). Doing any of this is incompatible with current
> incoming processing and with reprepro.
* 5 reviews of Debian packages were added, 4 were updated and 8 were
removed this month adding to our ever-growing knowledge about
identified issues [11].
[ 8] https://debian.org/
[ 9] https://lists.reproducible-builds.org/pipermail/rb-general/
[10] https://lists.reproducible-builds.org/pipermail/rb-general/2025-June/003803.html
[11] https://tests.reproducible-builds.org/debian/index_issues.html
In GNU Guix [12], Timothee Mathieu reported that a long-standing issue
with reproducibility of shell containers across different host operating
systems has been solved. In their message, Timothee mentions:
> I discovered that pytorch (and maybe other dependencies) has a
> reproducibility problem of order 1e-5 when on AVX512 [13] compared
> to AVX2 [14]. I first tried to solve the problem by disabling AVX512
> at the level of pytorch, but it did not work. The dev of pytorch
> said that it may be because some components dispatch computation to
> MKL-DNN, I tried to disable AVX512 on MKL, and still the results
> were not reproducible, I also tried to deactivate in openmpi without
> success. I finally concluded that there was a problem with AVX512
> somewhere in the dependencies graph but I gave up identifying where,
> as this seems very complicated.
[12] https://guix.gnu.org/
[13] https://en.wikipedia.org/wiki/AVX-512
[14] https://en.wikipedia.org/wiki/Advanced_Vector_Extensions
The IzzyOnDroid [15] Android APK repository made more progress in
June. Not only have they just passed 48% reproducibility coverage
[16], Ben started making their reproducible builds more visible, by
offering rbtlog shields [17], a kind of badge that has been quickly
picked up by many developers who are proud to present their
applications' reproducibility status.
[15] https://apt.izzysoft.de/fdroid/
[16] https://apt.izzysoft.de/fdroid
[17] https://shields.rbtlog.dev/
Lastly, in openSUSE [18] news, Bernhard M. Wiedemann posted another
monthly update [19] for their work there.
[18] https://www.opensuse.org/
[19] https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/U55TFJTYPTDALD4NB7KV4SRFSLGGJKRV/
§
diffoscope
----------
diffoscope [20] is our in-depth and content-aware diff utility that can
locate and diagnose reproducibility issues. This month, Chris Lamb made
the following changes, including preparing and uploading versions 298,
299 and 300 to Debian:
* Add python3-defusedxml to the Build-Depends in order to include it in
the Docker image. [22]
* Handle the RPM format's HEADERSIGNATURES and HEADERIMMUTABLE as a
special-case to avoid unnecessarily large diffs. Thanks to Daniel
Duan for the report and suggestion. [23][24]
* Update copyright years. [25]
[20] https://diffoscope.org
[22] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f430bec0
[23] https://salsa.debian.org/reproducible-builds/diffoscope/commit/1e9f288d
[24] https://salsa.debian.org/reproducible-builds/diffoscope/commit/5c93c759
[25] https://salsa.debian.org/reproducible-builds/diffoscope/commit/ec0d9315
In addition, @puer-robustus [26] fixed a regression introduced in an
earlier commit [27] which resulted in some differences being
lost. [28][29]
[26] https://salsa.debian.org/puer-robustus
[27] https://salsa.debian.org/puer-robustus/diffoscope/-/commit/5b187ad563526412fb5a5b328464f13047a49eff
[28] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c8426f05
[29] https://salsa.debian.org/reproducible-builds/diffoscope/commit/0c2b31a4
Lastly, Vagrant Cascadian updated diffoscope in GNU Guix [30] to
version 299 [31][32] and 300 [33][34].
[30] https://guix.gnu.org/
[31] https://codeberg.org/guix/guix/pulls/561
[32] https://codeberg.org/guix/guix/commit/0a5d93cdfed44937f3b97196ce4c2af1e58a1d61
[33] https://codeberg.org/guix/guix/pulls/886
[34] https://codeberg.org/guix/guix/commit/dd7e39ccfdd23a388dfa6b7665de466691bc6cda
§
OSS Rebuild updates
-------------------
OSS Rebuild [35] has added a new network analyzer [36] that provides
transparent HTTP(S) interception during builds, capturing all network
traffic to monitor external dependencies and identify suspicious
behavior, even in unmodified maintainer-controlled build processes.
The text-based user interface now features automated failure clustering
[37] that can group similar rebuild failures and provides natural
language failure summaries, making it easier to identify and understand
patterns across large numbers of build failures.
OSS Rebuild has also improved the local development experience with a
unified interface for build execution strategies [38], allowing for more
extensible environment setup for build execution. The team also designed
a new website [39] and logo [40].
[35] https://github.com/google/oss-rebuild
[36] https://github.com/google/oss-rebuild/pull/545
[37] https://github.com/google/oss-rebuild/pull/513
[38] https://github.com/google/oss-rebuild/pull/575
[39] https://oss-rebuild.dev/
[40] https://github.com/google/oss-rebuild?tab=readme-ov-file#oss-rebuild
§
Website updates
---------------
Once again, there were a number of improvements made to our website this
month including:
* Arnaud Brousseau added Stageˣ [41], a new Linux distribution, to our
"Tools" [42] page.
* Chris Lamb improved the docker instructions on the diffoscope
website [43]. [44]
[41] https://stagex.tools/
[42] https://reproducible-builds.org/tools/
[43] https://diffoscope.org/
[44] https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/fee0467
§
Upstream patches
----------------
The Reproducible Builds project detects, dissects and attempts to fix as
many currently-unreproducible packages as possible. We endeavour to send
all of our patches upstream where appropriate. This month, we wrote a
large number of such patches, including:
* Chris Riches:
* rpm [45]
[45] https://github.com/rpm-software-management/rpm/commit/bc0b94026bc5651435819043394cbe9a766a4fd5
* Bernhard M. Wiedemann:
* arandr [46], curl [47], dpdk [48], dpdk [49], eww [50], gnucash
[51], gramps [52], latex2html [53], llvm20 [54], mp [55], nvidia-
open-driver-G06-signed [56], obs [57], ovmf [58], pcre2 [59],
perl-XML-LibXML [60], perl [61], python-reportlab [62], python313
[63], qt6-datavis3d [64], qt6-declarative [65], qt6-sensors [66],
qt6-virtualkeyboard [67], rage-encryption [68], scummvm [69],
timescaledb [70] & zoxide [71].
* Plus several issues with how rpmbuild expands the %jobs variable
in the .src.rpm header, including: chromium [72], cmake [73],
edk2 [74], firefox-esr [75], gnome-keyring-sharp [76],
gtk2-engine-aurora [77], gtk2-engine-cleanice [78], gtk2-engines
[79], libqt5-qtlocation [80], libreoffice [81], luabind [82],
lxmenu-data [83], mozc [84], MozillaThunderbird [85], perl-
DateTime-Calendar-Mayan [86], perl-Getopt-ArgvFile [87], perl-
MooseX-Meta-TypeConstraint-ForceCoercion [88], perl-XML-Entities
[89], python-convertdate [90], suitesparse [91] &
webkit2gtk3 [92]
[46] https://build.opensuse.org/request/show/1286168
[47] https://github.com/curl/curl/pull/17665
[48] https://bugs.dpdk.org/show_bug.cgi?id=1718
[49] https://build.opensuse.org/request/show/1283587
[50] https://github.com/elkowar/eww/issues/1334
[51] https://bugs.gnucash.org/show_bug.cgi?id=799623
[52] https://github.com/gramps-project/gramps/pull/2081
[53] https://build.opensuse.org/request/show/1287226
[54] https://build.opensuse.org/request/show/1284969
[55] https://build.opensuse.org/request/show/1281890
[56] https://build.opensuse.org/request/show/1284004
[57] https://github.com/openSUSE/obs-build/pull/1076
[58] https://bugzilla.suse.com/show_bug.cgi?id=1244218
[59] https://build.opensuse.org/request/show/1284321
[60] https://build.opensuse.org/request/show/1288338
[61] https://build.opensuse.org/request/show/1284187
[62] https://build.opensuse.org/request/show/1284762
[63] https://bugzilla.opensuse.org/show_bug.cgi?id=1244680
[64] https://build.opensuse.org/request/show/1282734
[65] https://build.opensuse.org/request/show/1283382
[66] https://build.opensuse.org/request/show/1283386
[67] https://build.opensuse.org/request/show/1283396
[68] https://build.opensuse.org/request/show/1285623
[69] https://build.opensuse.org/request/show/1284696
[70] https://build.opensuse.org/request/show/1282546
[71] https://build.opensuse.org/request/show/1283367
[72] https://src.opensuse.org/chromium/chromium/pulls/1
[73] https://build.opensuse.org/request/show/1283475
[74] https://build.opensuse.org/request/show/1284200
[75] https://build.opensuse.org/request/show/1283964
[76] https://build.opensuse.org/request/show/1283991
[77] https://src.opensuse.org/lxde/gtk2-engine-aurora/pulls/1
[78] https://build.opensuse.org/request/show/1284149
[79] https://build.opensuse.org/request/show/1284006
[80] https://build.opensuse.org/request/show/1284198
[81] https://build.opensuse.org/request/show/1284193
[82] https://src.opensuse.org/lua/luabind/pulls/1
[83] https://src.opensuse.org/lxde/lxmenu-data/pulls/1
[84] https://build.opensuse.org/request/show/1284192
[85] https://build.opensuse.org/request/show/1283963
[86] https://build.opensuse.org/request/show/1284224
[87] https://build.opensuse.org/request/show/1284223
[88] https://build.opensuse.org/request/show/1284221
[89] https://build.opensuse.org/request/show/1284220
[90] https://build.opensuse.org/request/show/1283764
[91] https://build.opensuse.org/request/show/1283654
[92] https://build.opensuse.org/request/show/1283990
* Robin Candau:
* gramps [93] (use SOURCE_DATE_EPOCH when compressing man pages)
[93] https://github.com/gramps-project/gramps/pull/2078
* Chris Lamb:
* #1108273 [94] filed against tree-puzzle [95].
* #1108281 [96] filed against cctools [97].
* #1108532 [98] filed against python-django-import-export [99].
[94] https://bugs.debian.org/1108273
[95] https://tracker.debian.org/pkg/tree-puzzle
[96] https://bugs.debian.org/1108281
[97] https://tracker.debian.org/pkg/cctools
[98] https://bugs.debian.org/1108532
[99] https://tracker.debian.org/pkg/python-django-import-export
§
Reproducibility testing framework
---------------------------------
The Reproducible Builds project operates a comprehensive testing
framework running primarily at tests.reproducible-builds.org [100] in
order to check packages and other artifacts for reproducibility. In
June, however, a number of changes were made by Holger
Levsen, including:
* reproduce.debian.net [101]-related:
* Installed and deployed rebuilderd version 0.24 from Debian
unstable in order to make use of the new compression feature
added by Jarl Gullberg for the database. This resulted in massive
decrease of the SQLite [102] databases:
* 79G → 2.8G (all)
* 84G → 3.2G (amd64)
* 75G → 2.9G (arm64)
* 45G → 2.1G (armel)
* 48G → 2.2G (armhf)
* 73G → 2.8G (i386)
* 72G → 2.7G (ppc64el)
* 45G → 2.1G (riscv64)
… for a combined saving from 521G → 20.8G. This naturally reduces
the requirements to run an independent rebuilderd instance and
will permit us to add more Debian suites as well.
* During migration to the latest version of rebuilderd, make sure
several services are not started. [103]
* Actually run rebuilderd from /usr/bin. [104]
* Raise temperatures for NVME devices on some riscv64 nodes that
should be ignored. [105][106]
* Use a 64KB kernel page size on the ppc64el architecture (see
#1106757 [107]). [108]
* Improve ordering of some "failed to reproduce" statistics. [109]
* Detect a number of potential causes of build failures within the
statistics. [110][111]
* Add support for manually scheduling for the any
architecture. [112]
[100] https://tests.reproducible-builds.org
[101] https://reproduce.debian.net
[102] https://www.sqlite.org/
[103] https://salsa.debian.org/qa/jenkins.debian.net/commit/25bab2166
[104] https://salsa.debian.org/qa/jenkins.debian.net/commit/2176ebbd2
[105] https://salsa.debian.org/qa/jenkins.debian.net/commit/bd5ff0280
[106] https://salsa.debian.org/qa/jenkins.debian.net/commit/dfd191024
[107] https://bugs.debian.org/1106757
[108] https://salsa.debian.org/qa/jenkins.debian.net/commit/a42f1a078
[109] https://salsa.debian.org/qa/jenkins.debian.net/commit/f45c1e40f
[110] https://salsa.debian.org/qa/jenkins.debian.net/commit/67e7b1084
[111] https://salsa.debian.org/qa/jenkins.debian.net/commit/87246a8ad
[112] https://salsa.debian.org/qa/jenkins.debian.net/commit/661fa05e6
* Misc:
* Update the Codethink [113] nodes as there are now many kernels
installed. [114][115]
* Install linux-sysctl-defaults on Debian trixie systems as we
need ping functionality. [116]
* Limit the fs.nr_open kernel turnable. [117]
* Stop submitting results to deprecated buildinfo.debian.net
service. [118][119]
[113] https://www.codethink.co.uk/
[114] https://salsa.debian.org/qa/jenkins.debian.net/commit/06f4ef6fa
[115] https://salsa.debian.org/qa/jenkins.debian.net/commit/cb29e917f
[116] https://salsa.debian.org/qa/jenkins.debian.net/commit/0dfc74251
[117] https://salsa.debian.org/qa/jenkins.debian.net/commit/0fae0daf0
[118] https://salsa.debian.org/qa/jenkins.debian.net/commit/1c8c0361c
[119] https://salsa.debian.org/qa/jenkins.debian.net/commit/ef7e702cf
In addition, Jochen Sprickerhof greatly improved the statistics and the
logging functionality, including adopting to the new database format of
rebuilderd version 0.24.0 [120] and temporarily increasing maximum log
size in order to debug a nettlesome build [121]. Jochen also dropped the
CPUSchedulingPolicy=idle Systemd flag on the workers. [122]
[120] https://salsa.debian.org/qa/jenkins.debian.net/commit/9fd922a3a
[121] https://salsa.debian.org/qa/jenkins.debian.net/commit/c61c40cc8
[122] https://salsa.debian.org/qa/jenkins.debian.net/commit/54d37b5ef
§
Finally, if you are interested in contributing to the Reproducible
Builds project, please visit our "Contribute" [123] page on our website.
However, you can get in touch with us via:
* IRC: #reproducible-builds on irc.oftc.net.
* Mastodon: @reproducible_builds at fosstodon.org [124]
* Mailing list: rb-general at lists.reproducible-builds.org [125]
[123] https://reproducible-builds.org/contribute/
[124] https://fosstodon.org/@reproducible_builds
[125] https://lists.reproducible-builds.org/listinfo/rb-general
--
o
⬋ ⬊
o o reproducible-builds.org 💠
⬊ ⬋
o
More information about the rb-general
mailing list