From chris at reproducible-builds.org Mon Feb 3 12:19:34 2025 From: chris at reproducible-builds.org (Chris Lamb) Date: Mon, 03 Feb 2025 12:19:34 +0000 Subject: Please review the draft for January's report Message-ID: <173858512019.64318.2343551772913528002@copycat> Hi all, Please review the draft for January's Reproducible Builds report: https://reproducible-builds.org/reports/2025-01/?draft ? or, via the Git repository itself: https://salsa.debian.org/reproducible-builds/reproducible-website/blob/master/_reports/2025-01.md I intend to publish it no earlier than: $ date -d 'Wed, 05 Feb 2025 12:00:00 +0000' https://time.is/compare/1200_05_Feb_2025_in_UTC ? Please feel free and commit/push to drafts directly without the overhead of sending patches or merge requests. You should make your changes to the "_reports/2025-01.md" file in the "reproducible-website" repository: $ git clone https://salsa.debian.org/reproducible-builds/reproducible-website $ cd reproducible-website $ sensible-editor _reports/2025-01.md I am happy to reword and/or rework additions prior to publishing. If you currently do not have access to the above repository, you can request access by following the instructions at: https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From ludovic.courtes at inria.fr Tue Feb 4 18:07:06 2025 From: ludovic.courtes at inria.fr (=?utf-8?Q?Ludovic_Court=C3=A8s?=) Date: Tue, 04 Feb 2025 19:07:06 +0100 Subject: Does Functional Package Management Enable Reproducible Builds at Scale? Yes. In-Reply-To: <87ed0lu3jn.fsf@telecom-paris.fr> (Julien Malka's message of "Wed, 29 Jan 2025 14:56:28 +0100") References: <87ed0lu3jn.fsf@telecom-paris.fr> Message-ID: <875xlpwpmd.fsf@gnu.org> Hi Julien, Julien Malka skribis: > ? Does Functional Package Management Enable Reproducible Builds at Scale? Yes. ? [1] > > The article explores the proportion of bitwise reproducible packages in the Nix package repository and its evolution between 2017 and 2023. Thanks for sharing this insightful piece of work! Regarding historical data, fellow hacker Christopher Baines has been developing and running the Guix Data Service, which collects all sorts of data for each Guix commit (actually each commit that was pushed; there are holes in between pushes), in particular build information from the two project build farms, which are independent. It can thus provide reproducibility info for each commit, for each package that was successfully built on both build farms (the page below loads quite slowly): https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-reproducibility On this instance we see 5% of non-reproducible packages on x86_64-linux. Regarding RQ0 (which packages can still be built), it would be interesting to somehow weigh unbuildable packages by their number of dependents. For example, failing to build OpenSSL would in practice mean (for example, if/when cache.nixos.org is pruned) that thousands of packages become unavailable. We worked a bit on countermeasures: https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/ I have yet to digest the entire paper; every plot provides new insight into all this. Thank you! Ludo?. From ludovic.courtes at inria.fr Tue Feb 4 18:08:58 2025 From: ludovic.courtes at inria.fr (=?utf-8?Q?Ludovic_Court=C3=A8s?=) Date: Tue, 04 Feb 2025 19:08:58 +0100 Subject: Does Functional Package Management Enable Reproducible Builds at Scale? Yes. In-Reply-To: <87ed0lu3jn.fsf@telecom-paris.fr> (Julien Malka's message of "Wed, 29 Jan 2025 14:56:28 +0100") References: <87ed0lu3jn.fsf@telecom-paris.fr> Message-ID: <874j19wpj9.fsf@gnu.org> Hi Julien, Julien Malka skribis: > ? Does Functional Package Management Enable Reproducible Builds at Scale? Yes. ? [1] > > The article explores the proportion of bitwise reproducible packages in the Nix package repository and its evolution between 2017 and 2023. Thanks for sharing this insightful piece of work! Regarding historical data, fellow hacker Christopher Baines has been developing and running the Guix Data Service, which collects all sorts of data for each Guix commit (actually each commit that was pushed; there are holes in between pushes), in particular build information from the two project build farms, which are independent. It can thus provide reproducibility info for each commit, for each package that was successfully built on both build farms (the page below loads quite slowly): https://data.guix.gnu.org/revision/20dbf225f332ccc707578263ed710dcf2a8fb78e/package-reproducibility On this instance we see 5% of non-reproducible packages on x86_64-linux. Regarding RQ0 (which packages can still be built), it would be interesting to somehow weigh unbuildable packages by their number of dependents. For example, failing to build OpenSSL would in practice mean (for example, if/when cache.nixos.org is pruned) that thousands of packages become unavailable. We worked a bit on countermeasures: https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/ I have yet to digest the entire paper; every plot provides new insight into all this. Thank you! Ludo?. From chris at reproducible-builds.org Wed Feb 5 12:01:38 2025 From: chris at reproducible-builds.org (Chris Lamb) Date: Wed, 05 Feb 2025 12:01:38 +0000 Subject: Please review the draft for January's report In-Reply-To: <173858512019.64318.2343551772913528002@copycat> References: <173858512019.64318.2343551772913528002@copycat> Message-ID: <34f73849-5cef-457c-995f-4aed70f0e375@app.fastmail.com> Chris Lamb wrote: > Please review the draft for January's Reproducible Builds report: This has now been published ? thanks to all who contributed. If possible, please share the following link: https://reproducible-builds.org/reports/2025-01/ .. and also consider retweeting: https://x.com/ReproBuilds/status/1887108404889411917 Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From chris at reproducible-builds.org Wed Feb 5 12:20:04 2025 From: chris at reproducible-builds.org (Chris Lamb) Date: Wed, 05 Feb 2025 12:20:04 +0000 Subject: Reproducible Builds in January 2025 Message-ID: <91737027-c4ea-4dfa-868b-4e46f00a2f74@app.fastmail.com> -------------------------------------------------------------------- o ? ? January 2025 in Reproducible Builds o o ? ? https://reproducible-builds.org/reports/2025-01/ o -------------------------------------------------------------------- Welcome to the first report in 2025 from the Reproducible Builds project. Our monthly reports outline what we've been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security when relevant. As usual, though, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute [1] page on our website. ? Table of contents: * reproduce.debian.net * Two new academic papers * Distribution work * On our mailing list? * Upstream patches * "diffoscope" * Website updates * Reproducibility testing framework [1] https://reproducible-builds.org/contribute/ ? reproduce.debian.net -------------------- The last few months saw the introduction of reproduce.debian.net [3]. Announced at the recent Debian MiniDebConf in Toulouse [4], reproduce.debian.net is an instance of rebuilderd [5] operated by the Reproducible Builds project. Powering that is rebuilderd, our server designed monitor the official package repositories of Linux distributions and attempt to reproduce the observed results there. This month, however, we are pleased to announce that in addition to the existing amd64.reproduce.debian.net [6] and i386.reproduce.debian.net [7] architecture-specific pages, we now build for a three more architectures (for a total of five) ? arm64 [8] armhf [9] and riscv64 [10]. [3] https://reproduce.debian.net [4] https://toulouse2024.mini.debconf.org/ [5] https://github.com/kpcyrd/rebuilderd [6] https://amd64.reproduce.debian.net [7] https://i386.reproduce.debian.net [8] https://arm64.reproduce.debian.net/ [9] https://armhf.reproduce.debian.net/ [10] https://riscv64.reproduce.debian.net/ ? Two new academic papers ----------------------- Giacomo Benedetti, Oreofe Solarin, Courtney Miller, Greg Tystahl, William Enck, Christian K?stner, Alexandros Kapravelos, Alessio Merlo and Luca Verderame published an interesting article recently. Titled "An Empirical Study on Reproducible Packaging in Open-Source Ecosystem" [11], the abstract outlines its optimistic findings: > [We] identified that with relatively straightforward infrastructure > configuration and patching of build tools, we can achieve very high rates of > reproducible builds in all studied ecosystems. We conclude that if the > ecosystems adopt our suggestions, the build process of published packages can > be independently confirmed for nearly all packages without individual > developer actions, and doing so will prevent significant future software > supply chain attacks. The entire PDF [12] is available online to view. [11] https://www.cs.cmu.edu/~ckaestne/pdf/icse25_rb.pdf [12] https://www.cs.cmu.edu/~ckaestne/pdf/icse25_rb.pdf In addition, Julien Malka, Stefano Zacchiroli and Th?o Zimmermann of T?l?com Paris? in-house research laboratory, the Information Processing and Communications Laboratory [13] (LTCI) published an article asking the question: "Does Functional Package Management Enable Reproducible Builds at Scale?" [14]. Answering strongly in the affirmative, the article's abstract reads as follows: > In this work, we perform the first large-scale study of bitwise > reproducibility, in the context of the Nix functional package > manager [15], rebuilding 709,816 packages from historical snapshots > of the nixpkgs [16] repository[. We] obtain very high bitwise > reproducibility rates, between 69 and 91% with an upward trend, and > even higher rebuildability rates, over 99%. We investigate > unreproducibility causes, showing that about 15% of failures are due > to embedded build dates. We release a novel dataset with all build > statuses, logs, as well as full diffoscopes [17]: recursive diffs of > where unreproducible build artifacts differ. As above, the entire PDF [18] of the article is available to view online. [13] https://www.telecom-paris.fr/en/research/labs/information-processing-ltci [14] https://hal.science/hal-04913007 [15] https://nixos.org/ [16] https://github.com/NixOS/nixpkgs [17] https://diffoscope.org/ [18] https://hal.science/hal-04913007v1/file/2025-MSR-reproducibility.pdf ? Distribution work ----------------- There as been the usual work in various distributions this month, such as: * Arch Linux [19] developer kpcyrd has provided a status report for January 2025 [20] related to Arch's progress towards full reproducibility [21]. kpcyrd notes in particular progress towards to making a "minimal reproducible bootable system" ? that is, an Arch installation containing only reproducible packages. [19] https://archlinux.org/ [20] https://lists.reproducible-builds.org/pipermail/rb-general/2025-January/003641.html [21] https://reproducible.archlinux.org/ * 10+ reviews of Debian packages were added, 11 were updated and 10 were removed this month adding to our knowledge about identified issues [22]. A number of issue types were updated also. [22] https://tests.reproducible-builds.org/debian/index_issues.html * The FreeBSD [23] Foundation announced that "a planned project to deliver zero-trust builds has begun in January 2025". Supported by the Sovereign Tech Agency [24], this project is centered on the various build processes, and that the "primary goal of this work is to enable the entire release process to run without requiring root access, and that build artifacts build reproducibly ? that is, that a third party can build bit-for-bit identical artifacts." The full announcement [25] can be found online, which includes an estimated schedule and other details. [23] https://www.freebsd.org/ [24] https://www.sovereign.tech/ [25] https://freebsdfoundation.org/blog/zero-trust-builds-for-freebsd/ * Finally, for openSUSE, Bernhard M. Wiedemann published another report [26] for that distribution. [26] https://lists.opensuse.org/archives/list/factory at lists.opensuse.org/thread/PPDVJBF22DYFYI6BT7ONGHQLHUUJU7W3/ ? On our mailing list? -------------------- On our mailing list [27] this month: * Following-up to a substantial amount of previous work pertaining the Sphinx [28] documentation generator, James Addison asked a question [29] pertaining to the relationship between SOURCE_DATE_EPOCH [30] environment variable and testing that generated a number of replies. [27] https://lists.reproducible-builds.org/listinfo/rb-general/ [28] https://www.sphinx-doc.org/en/master/ [29] https://lists.reproducible-builds.org/pipermail/rb-general/2025-January/003623.html [30] https://reproducible-builds.org/docs/source-date-epoch/ * Adithya Balakumar of Toshiba asked a question about whether it is possible to make ext4 [31] filesystem images reproducible. Adithya's issue is that even the smallest amount of post-processing of the filesystem results in the modification of the "Last mount" and "Last write" timestamps. [31] https://en.wikipedia.org/wiki/Ext4 * James Addison also investigated an interesting issue [32] surrounding our disorderfs [33] filesystem. In particular: > FUSE (Filesystem in USErspace) [34] filesystems such as disorderfs do not > delete files from the underlying filesystem when they are deleted from > the overlay. This can cause seemingly straightforward tests ? for > example, cases that expect directory contents to be empty after deletion > is requested for all files listed within them ? to fail. [32] https://lists.reproducible-builds.org/pipermail/rb-general/2025-January/003637.html [33] https://salsa.debian.org/reproducible-builds/disorderfs [34] https://en.wikipedia.org/wiki/Filesystem_in_Userspace ? Upstream patches ---------------- The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including: * Bernhard M. Wiedemann: * Komikku [35] (nocheck) * abseil-cpp [36] (race) * dunst [37] (date) * eclipse-egit [38] (jar-mtime minor) * exaile [39] (race) * gawk [40] (bug) * gimp3 [41] (png date) * intel [42] (ASLR [43]) * ioquake3 [44] (debugsource contains date and time) * joker [45] (sort) * libchardet [46] * llama.cpp [47] (random) * llama.cpp [48] (-march=native-related issue) * nethack [49] (race) * netrek-client-cow [50] (date) * nvidia-modprobe [51] (date) * nvidia-persistenced [52] (date) * obs-build [53] (toolchain bug, mis-parses changelog) * perl-libconfigfile [54] (race) * pgvector [55] (CPU) * python-Django4 [56] (FTBFS-2038) * python-python-datamatrix [57] (FTBFS) * qore-ssh2-module [58] (GIGO-bug) * rpm [59] (UID in cpio header from rpmbuild) * zig [60] (CPU-related issue) [35] https://build.opensuse.org/request/show/1238506 [36] https://bugzilla.opensuse.org/show_bug.cgi?id=1235867 [37] https://github.com/dunst-project/dunst/pull/1435 [38] https://build.opensuse.org/request/show/1239889 [39] https://github.com/exaile/exaile/pull/956 [40] https://build.opensuse.org/request/show/1240443 [41] https://gitlab.gnome.org/GNOME/gimp-data/-/issues/7 [42] https://github.com/intel/intel-graphics-compiler/issues/359 [43] https://en.wikipedia.org/wiki/Address_space_layout_randomization [44] https://github.com/ioquake/ioq3/pull/704 [45] https://build.opensuse.org/request/show/1240514 [46] https://build.opensuse.org/request/show/1240682 [47] https://github.com/ggerganov/llama.cpp/issues/11306 [48] https://github.com/ggerganov/llama.cpp/pull/11366 [49] https://build.opensuse.org/request/show/1234705 [50] https://build.opensuse.org/request/show/1234274 [51] https://build.opensuse.org/request/show/1239739 [52] https://build.opensuse.org/request/show/1239742 [53] https://github.com/openSUSE/obs-build/pull/1047 [54] https://build.opensuse.org/request/show/1236852 [55] https://github.com/pgvector/pgvector/pull/764 [56] https://build.opensuse.org/request/show/1240318 [57] https://bugzilla.opensuse.org/show_bug.cgi?id=1236437 [58] https://build.opensuse.org/request/show/1240681 [59] https://github.com/rpm-software-management/rpm/discussions/3547 [60] https://github.com/ziglang/zig/issues/22663 * Chris Lamb: * #1092251 [61] filed against kmetronome [62]. * #1092917 [63] filed against rust-xh [64]. * #1093198 [65] filed against parser [66]. * #1093199 [67] filed against parser [68]. * #1093201 [69] filed against rsync [70]. * #1094611 [71] filed against wasistlos [72]. [61] https://bugs.debian.org/1092251 [62] https://tracker.debian.org/pkg/kmetronome [63] https://bugs.debian.org/1092917 [64] https://tracker.debian.org/pkg/rust-xh [65] https://bugs.debian.org/1093198 [66] https://tracker.debian.org/pkg/parser [67] https://bugs.debian.org/1093199 [68] https://tracker.debian.org/pkg/parser [69] https://bugs.debian.org/1093201 [70] https://tracker.debian.org/pkg/rsync [71] https://bugs.debian.org/1094611 [72] https://tracker.debian.org/pkg/wasistlos * Egbert Eich: * apptainer [73] (randomness) * spack [74] (core-count and date) [73] https://github.com/apptainer/apptainer/pull/2699 [74] https://build.opensuse.org/request/show/1235522 * Valentin Lefebvre: * uki-tool [75] (toolchain) [75] https://build.opensuse.org/request/show/1234742 * Marvin Friedrich: * cargo-packaging/rusty_v8 [76] (upstream [77] toolchain bugfix [78]) [76] https://build.opensuse.org/request/show/1235463 [77] https://github.com/openSUSE-Rust/cargo-packaging/pull/10 [78] https://bugzilla.opensuse.org/show_bug.cgi?id=1231548 * James Addison: * #1092870 [79] filed against binutils [80]. [79] https://bugs.debian.org/1092870 [80] https://tracker.debian.org/pkg/binutils * Pol Dellaiera: * PHP Ecosystem: composer/composer#12090 [81] which was then gracefully fixed by Jordi Boggiano [82] at composer/composer#12263 [83]. [81] https://github.com/composer/composer/pull/12090 [82] https://github.com/seldaek [83] https://github.com/composer/composer/pull/12263 ? diffoscope ---------- diffoscope [85] is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading versions 285, 286 and 287 to Debian: * Security fixes: * Validate the --css command-line argument to prevent a potential Cross-site scripting [86] (XSS) attack. Thanks to Daniel Schmidt from SRLabs for the report. [87] * Prevent XML entity expansion attacks. Thanks to Florian Wilkens from SRLabs for the report.. [88][89] * Print a warning if we have disabled XML comparisons due to a potentially vulnerable version of pyexpat. [90] [85] https://diffoscope.org [86] https://en.wikipedia.org/wiki/Cross-site_scripting [87] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a36ee4eb [88] https://salsa.debian.org/reproducible-builds/diffoscope/commit/889597c9 [89] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c8cd8ee4 [90] https://salsa.debian.org/reproducible-builds/diffoscope/commit/53ac5370 * Bug fixes: * Correctly identify changes to only the line-endings of files; don't mark them as "Ordering differences only". [91] * When passing files on the command line, don't call specialize(?) before we've checked that the files are identical or not. [92] * Do not exit with a traceback if paths are inaccessible, either directly, via symbolic links or within a directory. [93] * Don't cause a traceback if cbfstool extraction failed.. [94] * Use the surrogateescape mechanism to avoid a UnicodeDecodeError and crash when any decoding zipinfo output that is not UTF-8 compliant. [95] [91] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2236701a [92] https://salsa.debian.org/reproducible-builds/diffoscope/commit/5b187ad5 [93] https://salsa.debian.org/reproducible-builds/diffoscope/commit/a5486ebd [94] https://salsa.debian.org/reproducible-builds/diffoscope/commit/e2c21172 [95] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9a0faeed * Testsuite improvements: * Don't mangle newlines when opening test fixtures; we want them untouched. [96] * Move to assert_diff in test_text.py. [97] [96] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9fa3171f [97] https://salsa.debian.org/reproducible-builds/diffoscope/commit/e20a5055 * Misc improvements: * Drop unused subprocess imports. [98][99] * Drop an unused function in iso9600.py. [100] * Inline a call and check of Config().force_details; no need for an additional variable in this particular method. [101] * Remove an unnecessary return value from the Difference.check_for_ordering_differences method. [102] * Remove unused logging facility from a few comparators. [103] * Update copyright years. [104][105] [98] https://salsa.debian.org/reproducible-builds/diffoscope/commit/5f3df08f [99] https://salsa.debian.org/reproducible-builds/diffoscope/commit/28a9d61f [100] https://salsa.debian.org/reproducible-builds/diffoscope/commit/061514a1 [101] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d4fb5f17 [102] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f9aced5c [103] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2836c788 [104] https://salsa.debian.org/reproducible-builds/diffoscope/commit/82467745 [105] https://salsa.debian.org/reproducible-builds/diffoscope/commit/2343ac8f In addition, fridtjof added support for the ASAR [106] .tar-like archive format. [107][108][109][110] and lastly, Vagrant Cascadian updated diffoscope in GNU Guix [111] to version 285 [112][113] and 286 [114][115]. [106] https://github.com/electron/asar [107] https://salsa.debian.org/reproducible-builds/diffoscope/commit/9b426d27 [108] https://salsa.debian.org/reproducible-builds/diffoscope/commit/92a2e60e [109] https://salsa.debian.org/reproducible-builds/diffoscope/commit/b8b99410 [110] https://salsa.debian.org/reproducible-builds/diffoscope/commit/01f0189b [111] https://guix.gnu.org/ [112] https://debbugs.gnu.org/75769 [113] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=47594eb5f6b369aa3f0d56fa62b70d7ae9c0431f [114] https://debbugs.gnu.org/75813 [115] https://git.savannah.gnu.org/cgit/guix.git/commit/?id=77603927fba0edc2c4d9de122aa132b968a051e5 strip-nondeterminism [116] is our sister tool to remove specific non- deterministic results from a completed build. This month version 1.14.1-1 was uploaded to Debian unstable [117] by Chris Lamb, making the following the changes: * Clarify the --verbose and non --verbose output of bin/strip- nondeterminism so we don't imply we are normalizing files that we are not. [118] * Bump Standards-Version to 4.7.0. [119] [116] https://salsa.debian.org/reproducible-builds/strip-nondeterminism [117] https://tracker.debian.org/news/1607484/accepted-strip-nondeterminism-1141-1-source-into-unstable/ [118] https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/17a5bed [119] https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/b9e5fcb ? Website updates --------------- There were a large number of improvements made to our website this month, including: * Arnout Engelen: * Update the link to NixOS' reproducibility-related issue template [120] on the NixOS-specific contribute page [121] [122] and remove an outdated link. [123] [120] https://github.com/NixOS/nixpkgs/issues/new?template=10_unreproducible_package.yml [121] https://reproducible-builds.org/contribute/nixos/ [122] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2f3a8adf [123] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/791c3bec * Holger Levsen: * Check, deduplicate, update and generally cleanup a number of presentations linked on our "Talks & Resources" [124] page. [125][126][127][128] [124] https://reproducible-builds.org/resources/ [125] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2282c7bb [126] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9c9e6a0b [127] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5a3582e0 [128] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/439f6234 * James Addison: * Add some file permissions hints and guidance on the "Archive metadata" [129] page. [130] [129] https://reproducible-builds.org/docs/archives/ [130] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/35ed63bc * Michael R. Crusoe: * Add an R [131]) example to the SOURCE_DATE_EPOCH documentation [132]. [133] * Update the website's README [134] to make the setup command copy & paste friendly. [135] [131] https://en.wikipedia.org/wiki/R_(programming_language [132] https://reproducible-builds.org/docs/source-date-epoch/ [133] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/2d425f72 [134] https://salsa.debian.org/reproducible-builds/reproducible-website#readme [135] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4cbc0195 ? Reproducibility testing framework --------------------------------- The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org [136] in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen, including: * reproduce.debian.net-related: * Add support for rebuilding the armhf architecture. [138][139] * Add support for rebuilding the arm64 architecture. [140][141][142][143] * Add support for rebuilding the riscv64 architecture. [144][145] * Move the i386 builder to the osuosl5 node. [146][147][148][149] * Don't run our rebuilders on a public port. [150][151] * Add database backups on all builders and add links. [152][153] * Rework and dramatically improve the statistics collection and generation. [154][155][156][157][158][159] * Add contact info to the main page [160] [161], thumbnails [162] as well as the new, missing architectures. [163] * Move the amd64 worker to the osuosl4 and node. [164] * Run the underlying debrebuild script under nice [165]). [166] * Try to use TMPDIR when calling debrebuild. [167][168] [136] https://tests.reproducible-builds.org [137] https://reproduce.debian.net [138] https://salsa.debian.org/qa/jenkins.debian.net/commit/9cfe1429a [139] https://salsa.debian.org/qa/jenkins.debian.net/commit/6a101d8b5 [140] https://salsa.debian.org/qa/jenkins.debian.net/commit/2da411c81 [141] https://salsa.debian.org/qa/jenkins.debian.net/commit/619b476e5 [142] https://salsa.debian.org/qa/jenkins.debian.net/commit/412559291 [143] https://salsa.debian.org/qa/jenkins.debian.net/commit/89f472e34 [144] https://salsa.debian.org/qa/jenkins.debian.net/commit/6f78d2dac [145] https://salsa.debian.org/qa/jenkins.debian.net/commit/6abdc5f61 [146] https://salsa.debian.org/qa/jenkins.debian.net/commit/b4b78f803 [147] https://salsa.debian.org/qa/jenkins.debian.net/commit/c547e8fb7 [148] https://salsa.debian.org/qa/jenkins.debian.net/commit/97552fbae [149] https://salsa.debian.org/qa/jenkins.debian.net/commit/c03f53db8 [150] https://salsa.debian.org/qa/jenkins.debian.net/commit/c1c34d03a [151] https://salsa.debian.org/qa/jenkins.debian.net/commit/8a2b0507c [152] https://salsa.debian.org/qa/jenkins.debian.net/commit/5e4605e9b [153] https://salsa.debian.org/qa/jenkins.debian.net/commit/c673acf32 [154] https://salsa.debian.org/qa/jenkins.debian.net/commit/8fc2409ce [155] https://salsa.debian.org/qa/jenkins.debian.net/commit/bb79085d4 [156] https://salsa.debian.org/qa/jenkins.debian.net/commit/38d5f77ef [157] https://salsa.debian.org/qa/jenkins.debian.net/commit/816dd47ae [158] https://salsa.debian.org/qa/jenkins.debian.net/commit/4b88bc73e [159] https://salsa.debian.org/qa/jenkins.debian.net/commit/61f2f9a7e [160] https://reproduce.debian.net [161] https://salsa.debian.org/qa/jenkins.debian.net/commit/ef2a8456c [162] https://salsa.debian.org/qa/jenkins.debian.net/commit/a7b01e978 [163] https://salsa.debian.org/qa/jenkins.debian.net/commit/d46b1b0ba [164] https://salsa.debian.org/qa/jenkins.debian.net/commit/e2ab2a608 [165] https://en.wikipedia.org/wiki/Nice_(Unix [166] https://salsa.debian.org/qa/jenkins.debian.net/commit/6b10da810 [167] https://salsa.debian.org/qa/jenkins.debian.net/commit/ce8f11462 [168] https://salsa.debian.org/qa/jenkins.debian.net/commit/6e512cd04 * buildinfos.debian.net-related: * Stop creating buildinfo-pool_${suite}_${arch}.list files. [170] * Temporarily disable automatic updates of pool links. [171] [170] https://salsa.debian.org/qa/jenkins.debian.net/commit/43111425b [171] https://salsa.debian.org/qa/jenkins.debian.net/commit/730869a41 * FreeBSD [172]-related: * Fix the sudoers to actually permit builds. [173] * Disable debug output for FreeBSD rebuilding jobs. [174] * Upgrade to FreeBSD 14.2 [175] and document that bmake was installed on the underlying FreeBSD virtual machine image [176]. [172] https://www.freebsd.org/ [173] https://salsa.debian.org/qa/jenkins.debian.net/commit/e6e206c61 [174] https://salsa.debian.org/qa/jenkins.debian.net/commit/275c545ff [175] https://salsa.debian.org/qa/jenkins.debian.net/commit/0a1a6f9f4 [176] https://salsa.debian.org/qa/jenkins.debian.net/commit/572110855 * Misc: * Update the 'real' year to 2025. [177] * Don't try to install a Debian 'bookworm' kernel from 'backports' on the infom08 node which is running Debian 'trixie'. [178] * Don't warn about system updates for systems running Debian 'testing'. [179] * Fix a typo in the ZOMBIES definition. [180][181] [177] https://salsa.debian.org/qa/jenkins.debian.net/commit/38492eca6 [178] https://salsa.debian.org/qa/jenkins.debian.net/commit/6cbac8da4 [179] https://salsa.debian.org/qa/jenkins.debian.net/commit/154a6b4cb [180] https://salsa.debian.org/qa/jenkins.debian.net/commit/a000d82d1 [181] https://salsa.debian.org/qa/jenkins.debian.net/commit/5cffecd97 In addition: * Ed Maste modified the FreeBSD [182] build system to the clean the object directory before commencing a build. [183] [182] https://www.freebsd.org/ [183] https://salsa.debian.org/qa/jenkins.debian.net/commit/f0497b83a * Gioele Barabucci updated the rebuilder stats to first add a category for network errors [184] as well as to categorise failures without a diffoscope [185] log [186]. [184] https://salsa.debian.org/qa/jenkins.debian.net/commit/c67beb170 [185] https://diffoscope.org [186] https://salsa.debian.org/qa/jenkins.debian.net/commit/05c2495fa * Jessica Clarke also made some FreeBSD [187]-related changes, including: * Ensuring we clean up the object directory for second build as well. [188][189] * Updating the sudoers for the relevant rm -rf command. [190] * Update the cleanup_tmpdirs method to to match other removals. [191] [187] https://www.freebsd.org/ [188] https://salsa.debian.org/qa/jenkins.debian.net/commit/08542a237 [189] https://salsa.debian.org/qa/jenkins.debian.net/commit/9741a36b0 [190] https://salsa.debian.org/qa/jenkins.debian.net/commit/0b246f80e [191] https://salsa.debian.org/qa/jenkins.debian.net/commit/259366b4e * Jochen Sprickerhof: * Fix logic for old files saved on buildinfos.debian.net [192]. [193] * Rework and simplify the generation of statistics linked from reproduce.debian.net. [194][195][196][197][198] [192] https://buildinfos.debian.net/ [193] https://salsa.debian.org/qa/jenkins.debian.net/commit/554b65a01 [194] https://reproduce.debian.net [195] https://salsa.debian.org/qa/jenkins.debian.net/commit/293129600 [196] https://salsa.debian.org/qa/jenkins.debian.net/commit/9b2d37718 [197] https://salsa.debian.org/qa/jenkins.debian.net/commit/d6934d2a5 [198] https://salsa.debian.org/qa/jenkins.debian.net/commit/54cdd5ac8 * Roland Clobus: * Update the reproducible_debstrap job to call Debian's debootstrap with the full path [199] and to use eatmydata as well [200][201]. * Make some changes to deduce the CPU load in the debian_live_build job. [202] [199] https://salsa.debian.org/qa/jenkins.debian.net/commit/43cf9ff6a [200] https://salsa.debian.org/qa/jenkins.debian.net/commit/8d3c7dc56 [201] https://salsa.debian.org/qa/jenkins.debian.net/commit/df5091791 [202] https://salsa.debian.org/qa/jenkins.debian.net/commit/253bfa5e1 Lastly, both Holger Levsen [203] and Vagrant Cascadian [204] performed some node maintenance. [203] https://salsa.debian.org/qa/jenkins.debian.net/commit/9170361b4 [204] https://salsa.debian.org/qa/jenkins.debian.net/commit/665bd43de ? If you are interested in contributing to the Reproducible Builds project, please visit our Contribute [205] page on our website. However, you can get in touch with us via: * IRC: #reproducible-builds on irc.oftc.net. * Mailing list: rb-general at lists.reproducible-builds.org [207] * Twitter: @ReproBuilds [208] [205] https://reproducible-builds.org/contribute/ [207] https://lists.reproducible-builds.org/listinfo/rb-general [208] https://twitter.com/ReproBuilds -- o ? ? o o reproducible-builds.org ? ? ? o From chris at reproducible-builds.org Fri Feb 7 10:48:30 2025 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 07 Feb 2025 10:48:30 +0000 Subject: =?UTF-8?Q?diffoscope_288_released_=F0=9F=92=A0?= Message-ID: <173892334758.444211.11165958969139943969@copycat> Hi, The diffoscope maintainers are pleased to announce the release of version 288 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 288 includes the following changes: [ Chris Lamb ] * Add 'asar' to DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS. (Closes: #1095057) * Update minimal 'black' version. * Update copyright years. ## Download Version 288 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From bernhardout at lsmod.de Tue Feb 18 12:22:00 2025 From: bernhardout at lsmod.de (Bernhard M. Wiedemann) Date: Tue, 18 Feb 2025 13:22:00 +0100 Subject: RBOS / reproducible-openSUSE Message-ID: Hello fellow R-B-ings, I have some good news to share about my small openSUSE fork project https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ that was sponsored by a grant from NLnet. Of the packages in ring1 [1] , emacs and my VM image ('altimagebuild') probably needed the most effort. For some difficult cases, I just used workarounds, e.g. gcc/bash/python3 build without PGO and a few others build with -j1 to avoid unsolved race-conditions. There is still a chance that some rare issue slipped through all my tests, but at least I was able to reproduce bit-identical binaries on 5 different machines. In the process, I noticed that there are two different variants of reproducible builds that we commonly mix together. There is the first variant where you have an official build and with the help of some buildinfo data, people are able to independently produce bit-identical binaries. And then there is the second variant where all required information is part of the source, so you can do rebuilds without any official build happening somewhere. My RBOS is of the second kind. All packages in RBOS were built with the packages in RBOS, plus an extra bootstrap-snapshot of openSUSE-Tumbleweed for missing pieces. In OBS, we normally have auto-incrementing counters for checkin+rebuild but in this project, they are fixed at 1.1 so that I can produce identical binaries in OBS and outside (only the rpm sig needs to be removed with rpm --delsign (some exceptions apply because of an unmerged patch [2] and pesigning-magic - but here, OBS is not the primary artifact - the external builds are)) Do we have some existing terminology to distinguish these two kinds of r-b? Ciao Bernhard M. [1] https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:distribution:ring1 [2] https://github.com/openSUSE/obs-build/pull/1037 From bernhardout at lsmod.de Wed Feb 19 07:00:18 2025 From: bernhardout at lsmod.de (Bernhard M. Wiedemann) Date: Wed, 19 Feb 2025 08:00:18 +0100 Subject: RBOS / reproducible-openSUSE In-Reply-To: References: Message-ID: On 18/02/2025 14.32, Justin Cappos wrote: > This is really awesome!? ?One thing I am curious about is what sort of > issues testing on 5 machines helped you to find?? ?Did most issues show > up when you tested on 2 machines?? ?What sort of things were missed? Yes. Most issues were even found with 1 machine, rebuilding in 2 different directories there to make comparing easier. I did different kinds of tests. First just building twice on one machine found the worst offenders. Then I did a pass with my 'rbk' (= rebuild with KVM)-script that adds variations for year, CPU, #cores, hostname that found an issue with copyright year in qt6-webengine. I also did a compilation round with 14-core-VMs and found that one package only produced different binaries with 5 or more cores. 3 of the 5 machines were the same model of EPYC/Rome, but when rebuilding on a newer Ryzen, I found one or two race-conditions that did not cause variations before. The 5th machine was a 2010 Intel machine - there I found that some packages required x86_64-v3 to build. In the past I also had one issue that was hard to reproduce that only occurred on an older machine with HDD storage, so races / timing issues seem to be the most likely to slip through. Some packages also have non-determinism with 1-bit entropy (e.g. 2 elements in a hash with random iteration order) - those will give 2 identical results in 50% of cases. And if there is an auto-detection for the AVX512 CPU-flag, you cannot find that on older machines. Ciao Bernhard M. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 236 bytes Desc: OpenPGP digital signature URL: From jc4946 at nyu.edu Tue Feb 18 13:32:54 2025 From: jc4946 at nyu.edu (Justin Cappos) Date: Tue, 18 Feb 2025 08:32:54 -0500 Subject: RBOS / reproducible-openSUSE In-Reply-To: References: Message-ID: This is really awesome! One thing I am curious about is what sort of issues testing on 5 machines helped you to find? Did most issues show up when you tested on 2 machines? What sort of things were missed? Thanks, Justin On Tue, Feb 18, 2025 at 7:22?AM Bernhard M. Wiedemann via rb-general < rb-general at lists.reproducible-builds.org> wrote: > Hello fellow R-B-ings, > > I have some good news to share about my small openSUSE fork project > https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ > that was sponsored by a grant from NLnet. > > Of the packages in ring1 [1] , > emacs and my VM image ('altimagebuild') probably needed the most effort. > > For some difficult cases, I just used workarounds, e.g. gcc/bash/python3 > build without PGO and a few others build with -j1 to avoid unsolved > race-conditions. > > There is still a chance that some rare issue slipped through all my > tests, but at least I was able to reproduce bit-identical binaries on 5 > different machines. > > > In the process, I noticed that there are two different variants of > reproducible builds that we commonly mix together. > > There is the first variant where you have an official build and with the > help of some buildinfo data, people are able to independently produce > bit-identical binaries. > > And then there is the second variant where all required information is > part of the source, so you can do rebuilds without any official build > happening somewhere. > > > My RBOS is of the second kind. All packages in RBOS were built with the > packages in RBOS, plus an extra bootstrap-snapshot of > openSUSE-Tumbleweed for missing pieces. > In OBS, we normally have auto-incrementing counters for checkin+rebuild > but in this project, they are fixed at 1.1 so that I can produce > identical binaries in OBS and outside (only the rpm sig needs to be > removed with rpm --delsign (some exceptions apply because of an unmerged > patch [2] and pesigning-magic - but here, OBS is not the primary > artifact - the external builds are)) > > > Do we have some existing terminology to distinguish these two kinds of r-b? > > Ciao > Bernhard M. > > [1] > > https://build.opensuse.org/project/show/home:bmwiedemann:reproducible:distribution:ring1 > [2] https://github.com/openSUSE/obs-build/pull/1037 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at reproducible-builds.org Fri Feb 21 12:55:45 2025 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 21 Feb 2025 12:55:45 +0000 Subject: =?UTF-8?Q?diffoscope_289_released_=F0=9F=92=A0?= Message-ID: <174014174567.12833.1258849679018292243@copycat> Hi, The diffoscope maintainers are pleased to announce the release of version 289 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 289 includes the following changes: [ Chris Lamb ] * Catch CalledProcessError when calling html2text. * Update copyright years. ## Download Version 289 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From bernhardout at lsmod.de Tue Feb 25 10:03:17 2025 From: bernhardout at lsmod.de (Bernhard M. Wiedemann) Date: Tue, 25 Feb 2025 11:03:17 +0100 Subject: RBOS / reproducible-openSUSE In-Reply-To: References: Message-ID: On 18/02/2025 14.32, Justin Cappos wrote: > This is really awesome!? ?One thing I am curious about is what sort of > issues testing on 5 machines helped you to find?? ?Did most issues show > up when you tested on 2 machines?? ?What sort of things were missed? There was one different issue I found when re-building on my home machine: https://github.com/openSUSE/obs-build/pull/1047 Apparently the parsing of the changelog happened before entering the normalized build env so mis-parsing happened only there. The other servers were set to TZ=UTC so did not encounter this variation. This kind of variation would certainly have gone unnoticed for longer, had I not aimed at bit-reproducible results for RBOS. Ciao Bernhard M. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 236 bytes Desc: OpenPGP digital signature URL: From bernhardout at lsmod.de Tue Feb 25 14:00:59 2025 From: bernhardout at lsmod.de (Bernhard M. Wiedemann) Date: Tue, 25 Feb 2025 15:00:59 +0100 Subject: RBOS / reproducible-openSUSE In-Reply-To: References: Message-ID: On 25/02/2025 13.34, James Addison wrote: > Hi Bernhard, > > On Tue, 18 Feb 2025 at 12:22, Bernhard M. Wiedemann via rb-general > wrote: >> [ ... snip ... ] >> I have some good news to share about my small openSUSE fork project >> https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ >> that was sponsored by a grant from NLnet. > > Many congratulations on building a 100% bit-for-bit reproducible Linux > distribution! Thanks. > I'd been preparing to ask whether it's possible to uniquely identify > the resulting distro build -- however, if I understand correctly, the > pbuild process already does that[1], by emitting a > checksum-of-checksums of the (locale-agnostically) sorted RPM > packages? That is not part of pbuild but my wrapper script around it: https://build.opensuse.org/projects/home:bmwiedemann:reproducible:distribution:ring0/packages/000pbuildconf/files/build.sh that calls sha256sums to do exactly that. Via the sha256sums.src there is also an ID of the source files. So we have a bijective (one-to-one) mapping for srchash => binhash With the current split of ring0 and ring1, it is a bit less nice, because they provide toolchain packages for the other, that is not part of the hash input atm. Ciao Bernhard M. From jay at jp-hosting.net Tue Feb 25 12:34:06 2025 From: jay at jp-hosting.net (James Addison) Date: Tue, 25 Feb 2025 12:34:06 +0000 Subject: RBOS / reproducible-openSUSE In-Reply-To: References: Message-ID: Hi Bernhard, On Tue, 18 Feb 2025 at 12:22, Bernhard M. Wiedemann via rb-general wrote: > [ ... snip ... ] > I have some good news to share about my small openSUSE fork project > https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ > that was sponsored by a grant from NLnet. Many congratulations on building a 100% bit-for-bit reproducible Linux distribution! I'd been preparing to ask whether it's possible to uniquely identify the resulting distro build -- however, if I understand correctly, the pbuild process already does that[1], by emitting a checksum-of-checksums of the (locale-agnostically) sorted RPM packages? > In the process, I noticed that there are two different variants of > reproducible builds that we commonly mix together. > > There is the first variant where you have an official build and with the > help of some buildinfo data, people are able to independently produce > bit-identical binaries. > > And then there is the second variant where all required information is > part of the source, so you can do rebuilds without any official build > happening somewhere. > [ ... snip ... ] > Do we have some existing terminology to distinguish these two kinds of r-b? Personal opinion: in cases where the dependencies listed in buildinfo files are themselves FOSS and reproducible, then the two variants are more similar than they are different. Under those conditions, the buildinfo files are similar to links in a graph; and when completely expanded/flattened, that graph should be equivalent to the complete source code used during variant-two style builds If a distro constructed using the second variant method is self-hosting -- that is, after building it from source, it can be used to rebuild itself again, producing the same bit-for-bit identical outcome -- then I would suggest calling that a bootstrapped reproducible distro. Regards, James [1] - https://build.opensuse.org/projects/home:bmwiedemann:reproducible:distribution:ring1/packages/000pbuildconf/files/sha256sums From kpcyrd at archlinux.org Thu Feb 27 15:17:07 2025 From: kpcyrd at archlinux.org (kpcyrd) Date: Thu, 27 Feb 2025 16:17:07 +0100 Subject: =?UTF-8?Q?rebuilderd_0=2E22=2E0_released_=F0=9F=93=A6?= Message-ID: Dear list, I tagged and released a new version of rebuilderd, namely 0.22.0. It's mostly maintenance work, however there's now also an experimental Fedora integration that was first started in 2023 and recently picked up again by Jelle van der Waa (Red Hat). https://github.com/kpcyrd/rebuilderd/releases/tag/v0.22.0 --- Github release notes: * Migrate diesel dependency from 1.x to 2.x by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/156 * Port argument parsing from structopt (clap 2) to clap 4 by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/157 * Refactor dependency on http client, replace dependency on openssl with memory-safe implementation by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/158 * Update dependencies (in-toto, remove atty, dotenv -> dotenvy) by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/159 * README: Update Debian status and worker docker image by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/160 * README: the development dashboard is served over http by @jelly in https://github.com/kpcyrd/rebuilderd/pull/165 * Run cargo fmt on codebase by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/166 * Experimental Fedora support by @jelly in https://github.com/kpcyrd/rebuilderd/pull/167 * Update dependencies by @kpcyrd in https://github.com/kpcyrd/rebuilderd/pull/169 --- The upload to Debian has made a lot of progress the past few weeks, there's now only one missing dependency (actix-web). cheers, kpcyrd