Reproducible Builds in March 2024

Chris Lamb chris at reproducible-builds.org
Fri Apr 12 11:42:59 UTC 2024


--------------------------------------------------------------------
        o
      ⬋   ⬊      March 2024 in Reproducible Builds
     o     o
      ⬊   ⬋      https://reproducible-builds.org/reports/2024-03/
        o
--------------------------------------------------------------------

Welcome to the March 2024 report from the Reproducible Builds [0]
project. In our reports, we attempt to outline what we have been up to
over the past month, as well as mentioning some of the important things
happening more generally in software supply-chain security.

As ever, if you are interested in contributing to the project, please
visit our Contribute [1] page on our website.

 [0] https://reproducible-builds.org
 [1] https://reproducible-builds.org/contribute/


Table of contents:

 * Arch Linux minimal container userland now 100% reproducible
 * Validating Debian’s build infrastructure after the XZ backdoor
 * Making Fedora Linux (more) reproducible
 * Increasing Trust in the Open Source Supply Chain with Reproducible
   Builds and Functional Package Management
 * Software and source code identification with GNU Guix and
   reproducible builds
 * Two new Rust-based tools for post-processing determinism
 * Distribution work
 * Mailing list highlights
 * Website updates
 * Delta chat clients now reproducible
 * diffoscope updates
 * Upstream patches
 * Reproducibility testing framework
 * Contacting us


                                    §


Arch Linux minimal container userland now 100% reproducible
-----------------------------------------------------------

In remarkable news, Reproducible builds developer kpcyrd reported that
that the Arch Linux [2] "minimal container userland" is now 100%
reproducible [3] after work by developers dvzv and Foxboron on the one
remaining package. This represents a "real world", widely-used Linux
distribution being reproducible.

Their post, which kpcyrd suffixed with the question "now what?",
continues on to outline some potential next steps, including
validating whether the container image itself could be reproduced
bit-for-bit. The post, which was itself a followup for an Arch Linux
update earlier in the month [4], generated a significant number of
replies [5].

 [2] https://archlinux.org/
 [3] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003301.html
 [4] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003291.html
 [5] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/thread.html#3301

                                    §


Validating Debian's build infrastructure after the XZ backdoor
--------------------------------------------------------------

>From our mailing list [6] this month, Vagrant Cascadian wrote about [7]
being asked about trying to perform concrete reproducibility checks for
recent Debian security updates, in an attempt to gain some confidence
about Debian's build infrastructure given that they performed builds in
environments running the high-profile XZ vulnerability [8].

Vagrant reports (with some caveats):

> So far, I have not found any reproducibility issues; everything I
> tested I was able to get to build bit-for-bit identical with what is
> in the Debian archive.

That is to say, reproducibility testing permitted Vagrant and Debian to
claim with some confidence that builds performed when this vulnerable
version of XZ was installed were not interfered with.

 [6] https://lists.reproducible-builds.org/listinfo/rb-general/
 [7] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003321.html
 [8] https://lwn.net/Articles/967866/

                                    §


Making Fedora Linux (more) reproducible
---------------------------------------

In March, Davide Cavalca gave a talk at the 2024 Southern California
Linux Expo [9] (aka SCALE 21x) about the ongoing effort to make the
Fedora Linux distribution reproducible [10].

Documented in more detail on Fedora's website [11], the talk touched on
topics such as the specifics of implementing reproducible builds in
Fedora, the challenges encountered, the current status and what's coming
next. (A YouTube video [12] is available)

 [9] https://www.socallinuxexpo.org/scale/21x
 [10] https://www.socallinuxexpo.org/scale/21x/presentations/making-fedora-linux-more-reproducible
 [11] https://docs.fedoraproject.org/en-US/reproducible-builds/
 [12] https://www.youtube.com/watch?v=5c4gfXVPAbU

                                    §


"Increasing Trust in the Open Source Supply Chain with Reproducible Builds and Functional Package Management"
-------------------------------------------------------------------------------------------------------------

Julien Malka published a brief but interesting paper in the HAL open
archive [13]) on "Increasing Trust in the Open Source Supply Chain with
Reproducible Builds and Functional Package Management" [14]:

> Functional package managers (FPMs) and reproducible builds (R-B) are
> technologies and methodologies that are conceptually very different
> from the traditional software deployment model, and that have
> promising properties for software supply chain security. This thesis
> aims to evaluate the impact of FPMs and R-B on the security of the
> software supply chain and propose improvements to the FPM model to
> further improve trust in the open source supply chain.

Full PDF: [15]

Julien's paper poses a number of research questions on how the model of
distributions such as GNU Guix [16] and NixOS [17] can "be leveraged to
further improve the safety of the software supply chain", etc.

 [13] https://en.wikipedia.org/wiki/HAL_(open_archive
 [14] https://hal.science/hal-04482192
 [15] https://hal.science/hal-04482192/document
 [16] https://guix.gnu.org/
 [17] https://nixos.org/
 [18] https://guix.gnu.org/

                                    §


Software and source code identification with GNU Guix [18] and reproducible builds
----------------------------------------------------------------------------------

In a long line of commendably detailed blog posts, Ludovic Courtès,
Maxim Cournoyer, Jan Nieuwenhuizen and Simon Tournier have together
published two interesting posts on the GNU Guix blog [19] this month. In
early March, Ludovic Courtès, Maxim Cournoyer, Jan Nieuwenhuizen and
Simon Tournier wrote about software and source code identification [20]
and how that might be performed using Guix, rhetorically posing the
questions: "What does it take to 'identify software'? How can we tell
what software is running on a machine to determine, for example, what
security vulnerabilities might affect it?"

Later in the month, Ludovic Courtès wrote a solo post describing
adventures on the quest for long-term reproducible deployment [21].
Ludovic's post touches on GNU Guix's aim to support "time travel", the
ability to reliably (and reproducibly) revert to an earlier point in
time, employing the iconic image of Harold Lloyd hanging off the clock
in "Safety Last!" (1925) [22] to poetically illustrate both the
slapstick nature of current modern technology and the gymnastics
required to navigate hazards of our own making.

 [19] https://guix.gnu.org/en/blog/
 [20] https://guix.gnu.org/en/blog/2024/identifying-software/
 [21] https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/
 [22] https://en.wikipedia.org/wiki/Safety_Last!

                                    §


Two new Rust-based tools for post-processing determinism
--------------------------------------------------------

Zbigniew Jędrzejewski-Szmek announced "add-determinism" [23], a work-in-
progress reimplementation of the Reproducible Builds project's own
strip-nondeterminism [24] tool in the Rust programming language [25],
intended to be used as a post-processor in RPM-based distributions such
as Fedora [26]

In addition, Yossi Kreinin [27] published a blog post titled "refix:
fast, debuggable, reproducible builds" [28] that describes a tool that
post-processes binaries in such a way that they are still debuggable
with gdb [29], etc. Yossi post details the motivation and techniques
behind the (fast) performance of the tool.

 [23] https://github.com/keszybz/add-determinism
 [24] https://salsa.debian.org/reproducible-builds/strip-nondeterminism
 [25] https://www.rust-lang.org/
 [26] https://fedoraproject.org/
 [27] https://yosefk.com/
 [28] https://yosefk.com/blog/refix-fast-debuggable-reproducible-builds.html
 [29] https://sourceware.org/gdb/

                                    §


Distribution work
-----------------

In Debian this month, since the testing framework no longer varies the
build path [30], James Addison performed a bulk downgrade of the bug
severity [31] for issues filed with a level of normal to a new level of
wishlist. In addition, 28 reviews of Debian packages were added, 38 were
updated and 23 were removed this month adding to ever-growing knowledge
about identified issues [32]. As part of this effort, a number of issue
types were updated, including Chris Lamb adding a new
ocaml_include_directories toolchain issue [33] and James Addison adding
a new filesystem_order_in_java_jar_manifest_mf_include_resource issue
[34] and updating the random_uuid_in_notebooks_generated_by_nbsphinx to
reference a relevant discussion thread [35].

 [30] https://reproducible-builds.org/docs/build-path/
 [31] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003257.html
 [32] https://tests.reproducible-builds.org/debian/index_issues.html
 [33] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/a052c30f
 [34] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/cc94c935
 [35] https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/55497f89

In addition, Roland Clobus posted his 24th status update of reproducible
Debian ISO images [36]. Roland highlights that the images for Debian
unstable often cannot be generated due to changes in that distribution
related to the 64-bit time_t transition. And, lastly, Bernhard M.
Wiedemann posted another monthly update [37] for his reproducibility
work in openSUSE.

 [37] https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/NAST5PZPQGJ5JTHYAM6CWB7PCNCLLK6P/
 [36] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003327.html

                                    §


Mailing list highlights
-----------------------

Elsewhere on our mailing list [38] this month:

* Alexander Railean of Siemens asked the list to aid in
  understanding how one can independently verify the reproducibility
  of Java projects [40] from the Maven Central [41] repository. Having
  explored those repositories, Alexander could not find examples where
  the buildinfo file was present. Arnout Engelen responded with some
  details [42].

 [38] https://lists.reproducible-builds.org/listinfo/rb-general/
 [40] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003311.html
 [41] https://central.sonatype.com/
 [42] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003312.html

* Fay Stegerman resuscitated a long-dormant thread [43] to report that
  she added support in her diff-zip-meta.py tool [44] to expose extra
  timestamps embedded in .zip and .apk metadata.

 [43] https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003278.html
 [44] https://github.com/obfusk/reproducible-apk-tools#diff-zip-metapy

                                    §

Website updates
---------------

There were made a number of improvements to our website this
month, including:

* Pol Dellaiera noticed the frequent need to correctly cite the website
  itself in academic work. To facilitate easier citation across
  multiple formats, Pol contributed a Citation File Format [45] (CIF)
  file. As a result, an export in BibTeX [46] format is now available
  in the Academic Publications [47] section. Pol encourages community
  contributions to further refine the CITATION.cff [48] file. Pol also
  added an substantial new section to the "buy in [49]" page
  documenting the role of Software Bill of Materials (SBOMs) and
  ephemeral development environments. [50][51]

   [45] https://github.com/citation-file-format/citation-file-format
   [46] https://www.bibtex.org/
   [47] https://reproducible-builds.org/docs/publications/
   [48] https://salsa.debian.org/reproducible-builds/reproducible-website/-/blob/master/CITATION.cff
   [49] https://reproducible-builds.org/docs/buy-in/
   [50] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/74e44740
   [51] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d9996e2d

* Bernhard M. Wiedemann added a new "commandments [52]" page to the
  documentation [53][54][55] and fixed some incorrect YAML [56]
  elsewhere on the site [57].

   [52] https://reproducible-builds.org/docs/commandments/
   [53] https://reproducible-builds.org/docs/
   [54] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4e97c225
   [55] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/37e81323
   [56] https://yaml.org/
   [57] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/569cf016

* Chris Lamb add three recent academic papers to the publications [58]
  page of the website. [59]

   [58] https://reproducible-builds.org/docs/publications/
   [59] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a40c7422

* Mattia Rizzolo and Holger Levsen collaborated to add Infomaniak [60]
  as a sponsor of amd64 virtual machines. [61][62][63]

   [60] https://www.infomaniak.com/
   [61] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5d91f1e8
   [62] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ac7af0ee
   [63] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ba7e9d00

* Roland Clobus updated the "stable outputs [64]" page, dropping
  version numbers from Python documentation pages [65] and noting that
  Python's set data structure is also affected by the PYTHONHASHSEED
  functionality. [66]

 [64] https://reproducible-builds.org/docs/stable-outputs/
 [65] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6fbbb2b3
 [66] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/e9cae80b

                                    §


Delta chat clients now reproducible
-----------------------------------

Delta Chat [67], an open source messaging application that can work over
email, announced this month that the Rust-based core library underlying
Delta chat application is now reproducible [68].

 [67] https://delta.chat
 [68] https://chaos.social/@delta/112047758353026678
 [69] https://diffoscope.org

                                    §


diffoscope
----------

diffoscope [70] is our in-depth and content-aware diff utility that can
locate and diagnose reproducibility issues. This month, Chris Lamb made
a number of changes such as uploading versions 259, 260 and 261 to
Debian and made the following additional changes:

 [70] https://diffoscope.org

* New features:

    * Add support for the zipdetails [71] tool from the Perl
      distribution. Thanks to Fay Stegerman and Larry Doolittle et al.
      for the pointer and thread about this tool. [72]

     [71] https://perldoc.perl.org/zipdetails
     [72] https://salsa.debian.org/reproducible-builds/diffoscope/commit/d9dfe40d

* Bug fixes:

    * Don't identify Redis database dumps as GNU R [73]) database files
      based simply on their filename. [74]
    * Add a missing call to File.recognizes so we actually perform the
      filename check for GNU R data files. [75]
    * Don't crash if we encounter an .rdb file without an equivalent
      .rdx file. (#1066991 [76])
    * Correctly check for 7z being available—and not lz4—when testing
      7z. [77]
    * Prevent a traceback when comparing a contentful .pyc file with an
      empty one. [78]

     [73] https://en.wikipedia.org/wiki/R_(programming_language
     [74] https://salsa.debian.org/reproducible-builds/diffoscope/commit/28165345
     [75] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c6aa6ec8
     [76] https://bugs.debian.org/1066991
     [77] https://salsa.debian.org/reproducible-builds/diffoscope/commit/bd13f8bb
     [78] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c885c24a

* Testsuite improvements:

    * Fix .epub tests after supporting the new zipdetails tool. [79]
    * Don't use parenthesis within test "skipping…" messages, as PyTest
      adds its own parenthesis. [80]
    * Factor out Python version checking in test_zip.py. [81]
    * Skip some Zip-related tests under Python 3.10.14, as a potential
      regression may have been backported to the 3.10.x series. [82]
    * Actually test 7z support in the test_7z set of tests, not the lz4
      functionality. (Closes: reproducible-builds/diffoscope#359). [83]

     [79] https://salsa.debian.org/reproducible-builds/diffoscope/commit/c598dfa7
     [80] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f30387bd
     [81] https://salsa.debian.org/reproducible-builds/diffoscope/commit/71019a8d
     [82] https://salsa.debian.org/reproducible-builds/diffoscope/commit/f8270ba8
     [83] https://salsa.debian.org/reproducible-builds/diffoscope/commit/529d0ae3

In addition, Fay Stegerman updated diffoscope's monkey patch [84] for
supporting the unusual Mozilla ZIP file format after Python's zipfile
module changed to detect potentially insecure overlapping entries within
.zip files [85]. (#362) [86]

 [84] https://en.wikipedia.org/wiki/Monkey_patch
 [85] https://github.com/python/cpython/pull/110016
 [86] https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/362

Chris Lamb also updated the trydiffoscope command line client, dropping
a build-dependency on the deprecated python3-distutils package to fix
Debian bug #1065988 [87] [88], taking a moment to also refresh the
packaging to the latest Debian standards [89]. Finally, Vagrant
Cascadian submitted an update for diffoscope version 260 in GNU Guix
[90]. [91]

 [87] https://bugs.debian.org/1065988
 [88] https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/d217e92
 [89] https://salsa.debian.org/reproducible-builds/trydiffoscope/commit/f0fcf75
 [90] https://guix.gnu.org/
 [91] https://issues.guix.gnu.org/69656

                                    §


Upstream patches
----------------

This month, we wrote a large number of patches, including:

* Bernhard M. Wiedemann:

    * helm [92] (SSL-related build failure)
    * java-21-openjdk [93] (parallelism)
    * libressl [94] (SSL-related build failure)
    * nfdump [95] (date issue)
    * python-django-q [96] (avoid stuck build)
    * python-smart-open [97] (fails to build on single-CPU machines)
    * python-stdnum [98] (fails to build in 2039)
    * python-yarl [99] (regression)
    * qemu [100] (build failure)
    * rabbitmq-java-client [101] (with Fridrich Strba; Maven
      timestamp issue)
    * rmw [102] (build fails in 2038)
    * warewulf [103] (with Egbert Eich; cpio modification time and
      inode issue)
    * wxWidgets [104] (fails to build in 2038)

     [92] https://github.com/helm/helm/issues/12880
     [93] https://bugzilla.opensuse.org/show_bug.cgi?id=1221224
     [94] https://github.com/libressl/portable/issues/1018
     [95] https://build.opensuse.org/request/show/1163778
     [96] https://build.opensuse.org/request/show/1158939
     [97] https://bugzilla.opensuse.org/show_bug.cgi?id=1221663
     [98] https://github.com/arthurdejong/python-stdnum/issues/431
     [99] https://build.opensuse.org/request/show/1157151
     [100] https://bugzilla.opensuse.org/show_bug.cgi?id=1221340
     [101] https://build.opensuse.org/request/show/1155067
     [102] https://github.com/theimpossibleastronaut/rmw/pull/444
     [103] https://build.opensuse.org/request/show/1162930
     [104] https://github.com/wxWidgets/wxWidgets/issues/24414

* Chris Lamb:

    * #1066042 [105] filed against python-quantities [106].
    * #1066083 [107] filed against gnome-maps [108].
    * #1066084 [109] filed against tox [110].
    * #1066085 [111] filed against q2cli [112].
    * #1067098 [113] filed against mpl-sphinx-theme [114].
    * #1067099 [115] filed against woof-doom [116].
    * #1067100 [117] filed against bochs [118].
    * #1067101 [119] filed against storm-lang [120].
    * #1067102 [121] filed against librsvg [122].
    * #1067218 [123] filed against gretl [124].
    * #1067483 [125] filed against postfix [126].
    * #1067484 [127] filed against node-function-bind [128].
    * #1067485 [129] filed against python-pysaml2 [130].
    * #1067947 [131] filed against golang-github-stvp-tempredis [132].

     [105] https://bugs.debian.org/1066042
     [106] https://tracker.debian.org/pkg/python-quantities
     [107] https://bugs.debian.org/1066083
     [108] https://tracker.debian.org/pkg/gnome-maps
     [109] https://bugs.debian.org/1066084
     [110] https://tracker.debian.org/pkg/tox
     [111] https://bugs.debian.org/1066085
     [112] https://tracker.debian.org/pkg/q2cli
     [113] https://bugs.debian.org/1067098
     [114] https://tracker.debian.org/pkg/mpl-sphinx-theme
     [115] https://bugs.debian.org/1067099
     [116] https://tracker.debian.org/pkg/woof-doom
     [117] https://bugs.debian.org/1067100
     [118] https://tracker.debian.org/pkg/bochs
     [119] https://bugs.debian.org/1067101
     [120] https://tracker.debian.org/pkg/storm-lang
     [121] https://bugs.debian.org/1067102
     [122] https://tracker.debian.org/pkg/librsvg
     [123] https://bugs.debian.org/1067218
     [124] https://tracker.debian.org/pkg/gretl
     [125] https://bugs.debian.org/1067483
     [126] https://tracker.debian.org/pkg/postfix
     [127] https://bugs.debian.org/1067484
     [128] https://tracker.debian.org/pkg/node-function-bind
     [129] https://bugs.debian.org/1067485
     [130] https://tracker.debian.org/pkg/python-pysaml2
     [131] https://bugs.debian.org/1067947
     [132] https://tracker.debian.org/pkg/golang-github-stvp-tempredis

* James Addison:

    * #1065124 [133] filed against matplotlib [134].
    * #1066014 [135] filed against pathos [136].
    * #1066016 [137] filed against rdflib [138].
    * #1066017 [139] filed against xonsh [140].
    * #1066045 [141] filed against maven-bundle-plugin [142]. (This
      patch was then uploaded by Mattia Rizzollo [143])

     [133] https://bugs.debian.org/1065124
     [134] https://salsa.debian.org/python-team/packages/matplotlib
     [135] https://bugs.debian.org/1066014
     [136] https://tracker.debian.org/pkg/pathos
     [137] https://bugs.debian.org/1066016
     [138] https://tracker.debian.org/pkg/rdflib
     [139] https://bugs.debian.org/1066017
     [140] https://tracker.debian.org/pkg/xonsh
     [141] https://bugs.debian.org/1066045
     [142] https://tracker.debian.org/pkg/maven-bundle-plugin
     [143] https://bugs.debian.org/1066045#35

* Jiří Techet:

    * geany [144] (toolchain-related issue for glfw)

     [144] https://github.com/geany/geany/pull/3785


Bernhard M. Wiedemann used reproducibility-tooling to detect and fix
packages that added changes in their %check section, thus failing when
built with the --no-checks option. Only half of all openSUSE packages
were tested so far, but a large number of bugs were filed, including
ones against caddy [145], exiv2 [146], gnome-disk-utility [147], grisbi
[148], gsl [149], itinerary [150], kosmindoormap [151], libQuotient
[152], med-tools [153], plasma6-disks [154], pspp [155], python-
pypuppetdb [156], python-urlextract [157], rsync [158], vagrant-libvirt
[159] and xsimd [160].

 [145] https://build.opensuse.org/request/show/1163784
 [146] https://build.opensuse.org/request/show/1155606
 [147] https://build.opensuse.org/request/show/1157126
 [148] https://build.opensuse.org/request/show/1157756
 [149] https://rb.zq1.de/compare.factory-20240228/diffs/gsl-compare.out
 [150] https://build.opensuse.org/request/show/1157317
 [151] https://build.opensuse.org/request/show/1157313
 [152] https://build.opensuse.org/request/show/1157763
 [153] https://build.opensuse.org/request/show/1156899
 [154] https://build.opensuse.org/request/show/1157315
 [155] https://bugzilla.opensuse.org/show_bug.cgi?id=1221321
 [156] https://build.opensuse.org/request/show/1155640
 [157] https://build.opensuse.org/request/show/1161342
 [158] https://build.opensuse.org/request/show/1157116
 [159] https://build.opensuse.org/request/show/1157910
 [160] https://build.opensuse.org/request/show/1157141

Similarly, Jean-Pierre De Jesus DIAZ employed reproducible builds
techniques in order to test a proposed refactor of the ath9k-htc-
firmware [161] package. As the change produced bit-for-bit identical
binaries to the previously shipped pre-built binaries:

> I don't have the hardware to test this firmware, but the build
> produces the same hashes for the firmware so it's safe to say that
> the firmware should keep working.

 [161] https://issues.guix.gnu.org/69476

                                    §


Reproducibility testing framework
---------------------------------

The Reproducible Builds project operates a comprehensive testing
framework running primarily at tests.reproducible-builds.org in
order to check packages and other artifacts for reproducibility.

In March, an enormous number of changes were made by Holger Levsen:

* Debian [163]-related changes:

    * Sleep less after a so-called "404" package state has
      occurred. [164]
    * Schedule package builds more often. [165][166]
    * Regenerate all our HTML indexes every hour, but only every 12h
      for the released suites. [167]
    * Create and update unstable and experimental base systems on
      armhf again. [168][169]
    * Don't reschedule so many "depwait" packages due to the current
      size of the i386 architecture queue. [170]
    * Redefine our scheduling thresholds and amounts. [171]
    * Schedule untested packages with a higher priority, otherwise slow
      architectures cannot keep up with the experimental distribution
      growing. [172]
    * Only create the stats_buildinfo.png graph once per
      day. [173][174]
    * Reproducible Debian dashboard: refactoring, update several more
      static stats only every 12h. [175]
    * Document how to use systemctl with new systemd-based
      services. [176]
    * Temporarily disable armhf and i386 continuous integration tests
      in order to get some stability back. [177]
    * Use the deb.debian.org CDN everywhere. [178]
    * Remove the rsyslog logging facility on bookworm
      systems. [179]
    * Add zst to the list of packages which are false-positive
      diskspace issues. [180]
    * Detect failures to bootstrap Debian base systems. [181]

     [162] https://tests.reproducible-builds.org
     [163] https://debian.org/
     [164] https://salsa.debian.org/qa/jenkins.debian.net/commit/ba9df5b85
     [165] https://salsa.debian.org/qa/jenkins.debian.net/commit/ae6271021
     [166] https://salsa.debian.org/qa/jenkins.debian.net/commit/f909f0353
     [167] https://salsa.debian.org/qa/jenkins.debian.net/commit/3fc007604
     [168] https://salsa.debian.org/qa/jenkins.debian.net/commit/abc342ba0
     [169] https://salsa.debian.org/qa/jenkins.debian.net/commit/f488c2caf
     [170] https://salsa.debian.org/qa/jenkins.debian.net/commit/3ae7321bb
     [171] https://salsa.debian.org/qa/jenkins.debian.net/commit/052a886b9
     [172] https://salsa.debian.org/qa/jenkins.debian.net/commit/2dea4493d
     [173] https://salsa.debian.org/qa/jenkins.debian.net/commit/57163cee7
     [174] https://salsa.debian.org/qa/jenkins.debian.net/commit/6e792692d
     [175] https://salsa.debian.org/qa/jenkins.debian.net/commit/10a894d08
     [176] https://salsa.debian.org/qa/jenkins.debian.net/commit/141722598
     [177] https://salsa.debian.org/qa/jenkins.debian.net/commit/bec1b358f
     [178] https://salsa.debian.org/qa/jenkins.debian.net/commit/b9e5d80d3
     [179] https://salsa.debian.org/qa/jenkins.debian.net/commit/de6929151
     [180] https://salsa.debian.org/qa/jenkins.debian.net/commit/4df0658d6
     [181] https://salsa.debian.org/qa/jenkins.debian.net/commit/a660d7b7f

* Arch Linux [182]-related changes:

    * Temporarily disable builds because the pacman package manager
      is broken. [183][184]
    * Split reproducible_html_live_status and split the scheduling
      timing . [185][186][187]
    * Improve handling when database is locked. [188][189]

     [182] https://archlinux.org/
     [183] https://salsa.debian.org/qa/jenkins.debian.net/commit/d3ceee116
     [184] https://salsa.debian.org/qa/jenkins.debian.net/commit/d271f4936
     [185] https://salsa.debian.org/qa/jenkins.debian.net/commit/fe098a261
     [186] https://salsa.debian.org/qa/jenkins.debian.net/commit/7d3b8df0f
     [187] https://salsa.debian.org/qa/jenkins.debian.net/commit/75916ab3a
     [188] https://salsa.debian.org/qa/jenkins.debian.net/commit/d9eb99846
     [189] https://salsa.debian.org/qa/jenkins.debian.net/commit/bb69f8d03

* Misc changes:

    * Show failed services that require manual cleanup. [190][191]
    * Integrate two new Infomaniak [192] nodes. [193][194][195][196]
    * Improve IRC notifications for artifacts. [197]
    * Run diffoscope in different systemd slices [198]. [199]
    * Run the node health check more often, as it can now repair some
      issues. [200][201]
    * Also include the string Bot in the userAgent for Git. (Re:
      #929013 [202]). [203]
    * Document increased tmpfs size on our OUSL nodes. [204]
    * Disable memory account for the reproducible_build
      service. [205][206]
    * Allow 10 times as many open files for the Jenkins service. [207]
    * Set OOMPolicy=continue and OOMScoreAdjust=-1000 for both the
      Jenkins and the reproducible_build service. [208]

 [190] https://salsa.debian.org/qa/jenkins.debian.net/commit/ad33f4824
 [191] https://salsa.debian.org/qa/jenkins.debian.net/commit/46cf49e7f
 [192] https://www.infomaniak.com/
 [193] https://salsa.debian.org/qa/jenkins.debian.net/commit/5ea2d1f4a
 [194] https://salsa.debian.org/qa/jenkins.debian.net/commit/772274e4b
 [195] https://salsa.debian.org/qa/jenkins.debian.net/commit/e55af4ea1
 [196] https://salsa.debian.org/qa/jenkins.debian.net/commit/5ea8a9875
 [197] https://salsa.debian.org/qa/jenkins.debian.net/commit/ad5938903
 [198] https://www.freedesktop.org/software/systemd/man/latest/systemd.slice.html
 [199] https://salsa.debian.org/qa/jenkins.debian.net/commit/a46517dd6
 [200] https://salsa.debian.org/qa/jenkins.debian.net/commit/c0edd5a46
 [201] https://salsa.debian.org/qa/jenkins.debian.net/commit/f2d6b1ec0
 [202] https://bugs.debian.org/929013
 [203] https://salsa.debian.org/qa/jenkins.debian.net/commit/285c1de95
 [204] https://salsa.debian.org/qa/jenkins.debian.net/commit/4ced9a545
 [205] https://salsa.debian.org/qa/jenkins.debian.net/commit/b04767ab2
 [206] https://salsa.debian.org/qa/jenkins.debian.net/commit/698721665
 [207] https://salsa.debian.org/qa/jenkins.debian.net/commit/0f283b17a
 [208] https://salsa.debian.org/qa/jenkins.debian.net/commit/1c92680fd

Mattia Rizzolo also made the following changes:

* Debian-related changes:

    * Define a systemd slice [210] to group all relevant
      services. [211][212]
    * Add a bunch of quotes in scripts to assuage the shellcheck
      tool. [213]
    * Add stats on how many packages have been built today so
      far. [214]
    * Instruct systemd-run to handle diffoscope's exit codes
      specially. [215]
    * Prefer the pgrep tool over grepping the output of ps. [216]
    * Re-enable a couple of i386 and armhf architecture
      builders. [217][218]
    * Fix some stylistic issues flagged by the Python flake8
      tool. [219]
    * Cease scheduling Debian unstable and experimental on the
      armhf architecture due to the time_t transition. [220]
    * Start a few more i386 & armhf workers. [221][222][223]
    * Temporarly skip pbuilder updates in the unstable distribution,
      but only on the armhf architecture. [224]

     [210] https://www.freedesktop.org/software/systemd/man/latest/systemd.slice.html
     [211] https://salsa.debian.org/qa/jenkins.debian.net/commit/ec93ea707
     [212] https://salsa.debian.org/qa/jenkins.debian.net/commit/9e44007a6
     [213] https://salsa.debian.org/qa/jenkins.debian.net/commit/16517685d
     [214] https://salsa.debian.org/qa/jenkins.debian.net/commit/f8be35075
     [215] https://salsa.debian.org/qa/jenkins.debian.net/commit/2913cf26a
     [216] https://salsa.debian.org/qa/jenkins.debian.net/commit/ad47ae3aa
     [217] https://salsa.debian.org/qa/jenkins.debian.net/commit/790aea956
     [218] https://salsa.debian.org/qa/jenkins.debian.net/commit/095a73f81
     [219] https://salsa.debian.org/qa/jenkins.debian.net/commit/094f7cda1
     [220] https://salsa.debian.org/qa/jenkins.debian.net/commit/c9027bad6
     [221] https://salsa.debian.org/qa/jenkins.debian.net/commit/fcbcc67dd
     [222] https://salsa.debian.org/qa/jenkins.debian.net/commit/2a4454e6c
     [223] https://salsa.debian.org/qa/jenkins.debian.net/commit/44e07b0e2
     [224] https://salsa.debian.org/qa/jenkins.debian.net/commit/308338319

* Other changes:

    * Perform some large-scale refactoring on how the systemd service
      operates. [225][226]
    * Move the list of workers into a separate file so it's accessible
      to a number of scripts. [227]
    * Refactor the powercycle_x86_nodes.py script to use the new IONOS
      [228] API and its new Python bindings. [229]
    * Also fix nph-logwatch after the worker changes. [230]
    * Do not install the stunnel [231] tool anymore, it shouldn't be
      needed by anything anymore. [232]
    * Move temporary directories related to Arch Linux [233] into a
      single directory for clarity. [234]
    * Update the arm64 architecture host keys. [235]
    * Use a common Postfix [236] configuration. [237]

     [225] https://salsa.debian.org/qa/jenkins.debian.net/commit/dc5fd4433
     [226] https://salsa.debian.org/qa/jenkins.debian.net/commit/622278920
     [227] https://salsa.debian.org/qa/jenkins.debian.net/commit/fea359c67
     [228] https://www.ionos.co.uk/
     [229] https://salsa.debian.org/qa/jenkins.debian.net/commit/e75326235
     [230] https://salsa.debian.org/qa/jenkins.debian.net/commit/d96f48648
     [231] https://www.stunnel.org/
     [232] https://salsa.debian.org/qa/jenkins.debian.net/commit/080296849
     [233] https://archlinux.org/
     [234] https://salsa.debian.org/qa/jenkins.debian.net/commit/ad4f23245
     [235] https://salsa.debian.org/qa/jenkins.debian.net/commit/842f1e056
     [236] https://www.postfix.org/
     [237] https://salsa.debian.org/qa/jenkins.debian.net/commit/14c62ff18

The following changes were also made by:

* Jan-Benedict Glaw:

    * Initial work to clean up a messy NetBSD [238]-related
      script. [239][240]

     [238] https://www.netbsd.org/
     [239] https://salsa.debian.org/qa/jenkins.debian.net/commit/bd35d1bf7
     [240] https://salsa.debian.org/qa/jenkins.debian.net/commit/8bd707382

* Roland Clobus:

    * Show the installer log if the installer fails to build. [241]
    * Avoid the minus character (i.e. -) in a variable in order to
      allow for tags in openQA [242]. [243]
    * Update the schedule of Debian live image builds. [244]

     [241] https://salsa.debian.org/qa/jenkins.debian.net/commit/1fad936a1
     [242] https://openqa.debian.net/
     [243] https://salsa.debian.org/qa/jenkins.debian.net/commit/48221035b
     [244] https://salsa.debian.org/qa/jenkins.debian.net/commit/c2e9625bc

* Vagrant Cascadian:

    * Maintenance on the virt* nodes is completed so bring them back
      online. [245]
    * Use the fully qualified domain name in configuration. [246]

     [245] https://salsa.debian.org/qa/jenkins.debian.net/commit/8fafb3af5
     [246] https://salsa.debian.org/qa/jenkins.debian.net/commit/68572764e

Node maintenance was also performed by Holger Levsen, Mattia Rizzolo
[247][248] and Vagrant Cascadian [249][250][251][252]

   [247] https://salsa.debian.org/qa/jenkins.debian.net/commit/7b35add28
   [248] https://salsa.debian.org/qa/jenkins.debian.net/commit/554d7eb2b
   [249] https://salsa.debian.org/qa/jenkins.debian.net/commit/114758a02
   [250] https://salsa.debian.org/qa/jenkins.debian.net/commit/4db2f0b58
   [251] https://salsa.debian.org/qa/jenkins.debian.net/commit/ce64261bb
   [252] https://salsa.debian.org/qa/jenkins.debian.net/commit/cb768d67f

                                    §

If you are interested in contributing to the Reproducible Builds
project, please visit our Contribute [253] page on our website.

However, you can also get in touch with us via:

 * IRC: #reproducible-builds on irc.oftc.net.

 * Twitter: @ReproBuilds [254]

 * Mastodon: @reproducible_builds at fosstodon.org [255]

 * Mailing list: rb-general at lists.reproducible-builds.org [256]

 [253] https://reproducible-builds.org/contribute/
 [254] https://twitter.com/ReproBuilds
 [255] https://fosstodon.org/@reproducible_builds
 [256] https://lists.reproducible-builds.org/listinfo/rb-general


-- 
      o
    ⬋   ⬊
   o     o     reproducible-builds.org 💠
    ⬊   ⬋
      o


More information about the rb-general mailing list