Reproducible Builds in October 2023

Mon Nov 13 14:17:40 UTC 2023

        o
      ⬋   ⬊      October 2023 in Reproducible Builds
     o     o
      ⬊   ⬋      https://reproducible-builds.org/reports/2023-10/
        o

Welcome to the October 2023 report from the Reproducible Builds
project. In these reports, we outline the most important things that we
have been up to over the past month. As a quick recap, whilst anyone may
inspect the source code of free software for malicious flaws, almost all
software is distributed to end users as pre-compiled binaries.

                                 §§§

## Reproducible Builds Summit 2023

Between October 31st and November 2nd, we held our seventh Reproducible
Builds Summit [1] in Hamburg, Germany!

Our summits are a unique gathering that brings together attendees from
diverse projects, united by a shared vision of advancing the
Reproducible Builds effort, and this instance was no different.

During this enriching event, participants had the opportunity to engage
in discussions, establish connections and exchange ideas to drive
progress in this vital field. A number of concrete outcomes from the
summit will documented in the report for November 2023 and elsewhere.

Amazingly the agenda and all notes from all sessions are already
online [2].

The Reproducible Builds team would like to thank our event sponsors who
include Mullvad VPN [3], openSUSE [4], Debian [5], Software Freedom
Conservancy [6], Allotropia [7] and Aspiration Tech [8].

 [1] https://reproducible-builds.org/events/hamburg2023/
 [2] https://reproducible-builds.org/events/hamburg2023/agenda/
 [3] https://mullvad.net/
 [4] https://www.opensuse.org/
 [5] https://www.debian.org/
 [6] https://sfconservancy.org/
 [7] https://www.debian.org/
 [8] https://aspirationtech.org/

                                 §§§

## Reflections on "Reflections on Trusting Trust"

Russ Cox [9] posted a fascinating article on his blog [10] prompted by
the fortieth anniversary of Ken Thompson's award-winning paper,
"Reflections on Trusting Trust" [11]:

> […] In March 2023, Ken gave the closing keynote [and] during the Q&A
> session, someone jokingly asked about the Turing award lecture,
> specifically “can you tell us right now whether you have a backdoor into
> every copy of gcc and Linux still today?”

Although Ken reveals (or at least claims!) that he has no such
backdoor, he does admit that he has the actual code… which Russ requests
and subsequently dissects in great but accessible detail.

 [9] https://swtch.com/~rsc/
 [10] https://research.swtch.com/nih
 [11] https://dl.acm.org/doi/pdf/10.1145/358198.358210

                                 §§§

## Ecosystem factors of reproducible builds

Rahul Bajaj, Eduardo Fernandes, Bram Adams and Ahmed E. Hassan from the
Maintenance, Construction and Intelligence of Software (MCIS) [12]
laboratory within the School of Computing [13], Queen's University [14]
in Ontario, Canada have published a paper on the "Time to fix, causes
and correlation with external ecosystem factors" of
unreproducible builds.

The authors compare various response times within the Debian [15] and
Arch Linux [16] distributions including, for example:

> Arch Linux packages become reproducible a median of 30 days quicker
> when compared to Debian packages, while Debian packages remain
> reproducible for a median of 68 days longer once fixed.

A full PDF of their paper [17] is available online, as are many other
interesting papers on MCIS' [18] publication page.

 [12] https://mcis.cs.queensu.ca
 [13] https://cs.queensu.ca/
 [14] https://www.queensu.ca/
 [15] https://debian.org/
 [16] https://archlinux.org/
 [17] https://mcis.cs.queensu.ca/publications/2023/emse_rahul.pdf
 [18] https://mcis.cs.queensu.ca/publications

                                 §§§

## NixOS installation image reproducible

On the NixOS Discourse instance [19], Arnout Engelen (raboof)
announced that NixOS have created an independent, bit-for-bit identical
rebuilding of the nixos-minimal image that is used to install NixOS.
In their post [20], Arnout details what exactly can be reproduced, and
even includes some of the history of this endeavour:

> You may remember a 2021 announcement [21] that the minimal ISO was
> 100% reproducible. While back then we successfully tested that all
> packages that were needed to build the ISO were individually
> reproducible, actually rebuilding the ISO still introduced
> differences.  This was due to some remaining problems [22] in the
> hydra cache and the way the ISO was created. By the time we fixed
> those, regressions had popped up (notably an upstream problem in
> Python 3.10), and it isn’t until this week that we were back to having
> everything reproducible and being able to validate the complete chain.

Congratulations to NixOS team for reaching this important milestone!
Discussion about this announcement can be found underneath the post [23]
itself, as well as on Hacker News [24].

 [19] https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756
 [20] https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756
 [21] https://discourse.nixos.org/t/nixos-unstable-s-iso-minimal-x86-64-linux-is-100-reproducible/13723
 [22] https://github.com/NixOS/nixpkgs/issues/125380
 [23] https://discourse.nixos.org/t/nixos-reproducible-builds-minimal-installation-iso-successfully-independently-rebuilt/34756#post_2
 [24] https://news.ycombinator.com/item?id=38057591

                                 §§§

## CPython source tarballs now reproducible

Seth Larson [25] published a blog post investigating the reproducibility
of the CPython source tarballs [26]. Using diffoscope [27],
reprotest and other tools, Seth documents his work that led to a pull
request to make these files reproducible [28] which was merged by Łukasz
Langa [29].

 [25] https://sethmlarson.dev/
 [26] https://sethmlarson.dev/security-developer-in-residence-weekly-report-14
 [27] https://diffoscope.org/
 [28] https://github.com/python/release-tools/pull/62
 [29] https://lukasz.langa.pl/

                                 §§§

## New arm64 hardware from Codethink

Long-time sponsor of the project, Codethink [30], have generously
replaced our old "Moonshot-Slides", which they have generously hosted
since 2016 with new KVM [31]-based arm64 hardware. Holger Levsen
integrated these new nodes to the Reproducible Builds' continuous
integration [32] framework.

 [30] https://www.codethink.co.uk/
 [31] https://linux-kvm.org/page/Main_Page
 [32] https://tests.reproducible-builds.org/

                                 §§§

## Community updates

On our mailing list during October 2023 [33] there were a number of
threads, including:

* Vagrant Cascadian continued a thread about the implementation details
  of a "snapshot" archive server required for reproducing previous
  builds. [34]

* Akihiro Suda shared an update on BuildKit [35], a toolkit for
  building Docker [36] container images. Akihiro links to a interesting
  talk they recently gave at DockerCon [37] titled Reproducible builds
  with BuildKit for software supply-chain security [38].

* Alex Zakharov started a thread discussing and proposing fixes for
  various tools that create ext4 [39] filesystem images. [40]

 [33] https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/thread.html
 [34] https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/003086.html
 [35] https://github.com/moby/buildkit
 [36] https://www.docker.com/
 [37] https://dockercon.com/
 [38] https://medium.com/nttlabs/dockercon-2023-reproducible-builds-with-buildkit-for-software-supply-chain-security-0e5aedd1aaa7
 [39] https://en.wikipedia.org/wiki/Ext4
 [40] https://lists.reproducible-builds.org/pipermail/rb-general/2023-October/003098.html

Elsewhere, Pol Dellaiera made a number of improvements to our website,
including fixing typos and links [41][42], adding a NixOS "Flake" file
[43] [44] and sorting our publications page [45] by date [46].

Vagrant Cascadian presented Reproducible Builds All The Way Down [47]
at the Open Source Firmware Conference [48].

 [41] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/7f3e9550
 [42] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/8ab7459c
 [43] https://nixos.wiki/wiki/Flakes
 [44] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/0c1a61eb
 [45] https://reproducible-builds.org/docs/publications/
 [46] https://salsa.debian.org/reproducible-builds/reproducible-website/commit/d052569b
 [47] https://www.osfc.io/2023/talks/reproducible-builds-all-the-way-down/
 [48] https://www.osfc.io/

                                 §§§

## Distribution work

distro-info is a Debian-oriented tool that can provide information
about Debian (and Ubuntu) distributions such as their codenames (eg.
bookworm) and so on. This month, Benjamin Drung uploaded a new version
of distro-info that added support for the SOURCE_DATE_EPOCH
environment variable [49] in order to close bug #1034422 [50]. In
addition, 8 reviews of packages were added, 74 were updated and 56 were
removed this month, all adding to our knowledge about identified
issues [51].

Bernhard M. Wiedemann published another monthly report about
reproducibility within openSUSE [52].

 [49] https://reproducible-builds.org/specs/source-date-epoch/
 [50] https://tracker.debian.org/distro-info
 [51] https://tests.reproducible-builds.org/debian/index_issues.html
 [52] https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/4QTSQCYBMF6QZYWIB63T46ILLTVGVMMJ/

                                 §§§

## Software development

The Reproducible Builds project detects, dissects and attempts to fix as
many currently-unreproducible packages as possible. We endeavour to send
all of our patches upstream where appropriate. This month, we wrote a
large number of such patches, including:

 * Bernhard M. Wiedemann:

    * edje_cc [53] (race condition)
    * elasticsearch [54] (build failure)
    * erlang-retest [55] (embedded .zip timestamp)
    * fdo-client [56] (embeds private keys)
    * fftw3 [57] (random ordering)
    * gsoap [58] (date issue)
    * gutenprint [59] (date)
    * hub/golang [60] (embeds random build path)
    * Hyprland [61] (filesystem issue)
    * kitty [62] (sort-related issue, .tar file embeds
      modification time)
    * libpinyin [63] (ASLR)
    * maildir-utils [64] (date embedded in copyright)
    * mame [65] (order-related issue)
    * mingw32-binutils [66] & mingw64-binutils [67] (date)
    * MooseX [68] (date from perl-MooseX-App)
    * occt [69] (sorting issue)
    * openblas [70] (embeds CPU count)
    * OpenRGB [71] (corruption-related issue [72])
    * python-numpy [73] (random file names)
    * python-pandas [74] (FTBFS)
    * python-quantities [75] (date)
    * python3-pyside2 [76] (order)
    * qemu [77] (date and Sphinx issue)
    * qpid [78] (sorting problem)
    * rakudo [79] (filesystem ordering issue)
    * SLOF [80] (date-related issue)
    * spack [81] (CPU counting issue)
    * xemacs-packages [82] (date-related issue)

* Chris Lamb:

    * #1053353 [83] filed against dacite [84].
    * #1053356 [85] filed against rtpengine [86].

 [53] https://git.enlightenment.org/enlightenment/efl/issues/41
 [54] https://github.com/elastic/elasticsearch-py/issues/2320
 [55] https://build.opensuse.org/request/show/1116208
 [56] https://bugzilla.opensuse.org/show_bug.cgi?id=1216293
 [57] https://github.com/FFTW/fftw3/issues/337
 [58] https://sourceforge.net/p/gsoap2/patches/185/
 [59] https://sourceforge.net/p/gimp-print/source/merge-requests/9/
 [60] https://github.com/golang/go/issues/63851
 [61] https://github.com/hyprwm/Hyprland/pull/3550
 [62] https://github.com/kovidgoyal/kitty/pull/6685
 [63] https://github.com/libpinyin/libpinyin/issues/162
 [64] https://github.com/djcb/mu/pull/2569
 [65] https://github.com/mamedev/mame/pull/11651
 [66] https://build.opensuse.org/request/show/1116036
 [67] https://build.opensuse.org/request/show/1116040
 [68] https://github.com/maros/MooseX-App/pull/71
 [69] https://build.opensuse.org/request/show/1119524
 [70] https://build.opensuse.org/request/show/1118201
 [71] https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/3675
 [72] https://gitlab.com/CalcProgrammer1/OpenRGB/-/merge_requests/2103
 [73] https://bugzilla.opensuse.org/show_bug.cgi?id=1216458
 [74] https://build.opensuse.org/request/show/1117743
 [75] https://build.opensuse.org/request/show/1117898
 [76] https://bugreports.qt.io/browse/PYSIDE-2508
 [77] https://build.opensuse.org/request/show/1121011
 [78] https://github.com/apache/qpid-proton/pull/411
 [79] https://github.com/rakudo/rakudo/pull/5426
 [80] https://gitlab.com/qemu-project/SLOF/-/merge_requests/1
 [81] https://build.opensuse.org/request/show/1118130
 [82] https://build.opensuse.org/request/show/1119260
 [83] https://bugs.debian.org/1053353
 [84] https://tracker.debian.org/pkg/dacite
 [85] https://bugs.debian.org/1053356

In addition, Chris Lamb fixed an issue in diffoscope [87], where if
the equivalent of "file -i" returns "text/plain", fallback to comparing
as a text file. This was originally filed as Debian bug #1053668 [88])
by Niels Thykier. [89] This was then uploaded to Debian (and elsewhere)
as version 251.

 [87] https://diffoscope.org
 [88] https://bugs.debian.org/1053668
 [89] https://salsa.debian.org/reproducible-builds/diffoscope/commit/81c68d7b
 [86] https://tracker.debian.org/pkg/rtpengine

                                 §§§

## Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing
framework (available at tests.reproducible-builds.org [90]) in order to
check packages and other artifacts for reproducibility. In October, a
number of changes were made by Holger Levsen:

* Debian-related changes:

    * Refine the handling of package blacklisting, such as sending
      blacklisting notifications to the #debian-reproducible-changes
      IRC channel. [91][92][93]
    * Install systemd-oomd on all Debian bookworm nodes (re. Debian
      bug #1052257 [94]). [95]
    * Detect more cases of failures to delete schroots. [96]
    * Document various bugs in bookworm which are (currently) being
      manually worked around. [97]

* Node-related changes:

    * Integrate the new arm64 machines from Codethink
      [98]. [99][100][101][102][103][104]
    * Improve various node cleanup routines. [105][106][107][108]
    * General node maintenance. [109][110][111][112]

* Monitoring-related changes:

    * Remove unused Munin [113] monitoring plugins. [114]
    * Complain less visibly about "too many" installed kernels. [115]

* Misc:

    * Enhance the firewall handling on Jenkins
      nodes. [116][117][118][119]
    * Install the fish shell everywhere. [120]

In addition, Vagrant Cascadian added some packages and configuration for
snapshot experiments. [121]

                                 §§§

## And finally…

If you are interested in contributing to the Reproducible Builds
project, please visit our Contribute [122] page on our website.
However, you can get in touch with us via:

 * IRC: <#reproducible-builds> on <irc.oftc.net>

 * Mailing list: <rb-general at lists.reproducible-builds.org> [123]

 * Mastodon: @reproducible_builds [124]

 * Twitter: @ReproBuilds [125]

 [122] https://reproducible-builds.org/contribute/
 [123] https://lists.reproducible-builds.org/listinfo/rb-general
 [124] https://fosstodon.org/@reproducible_builds
 [125] https://twitter.com/ReproBuilds

-- 
      o
    ⬋   ⬊
   o     o     reproducible-builds.org 💠
    ⬊   ⬋
      o