From chris at reproducible-builds.org Fri Jul 2 16:35:39 2021 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 02 Jul 2021 17:35:39 +0100 Subject: Please review the draft for June's report Message-ID: <162524363873.767378.12451745362785428184@tinycat.chris-lamb.co.uk> Hi all, Please review the draft for June's Reproducible Builds report: https://reproducible-builds.org/reports/2021-06/?draft ? or, via the Git repository itself: https://salsa.debian.org/reproducible-builds/reproducible-website/blob/master/_reports/2021-06.md Note that there is still one 'FIXME' section which I plan to address soon and definitely prior to publication (!), but I wanted to get this draft out today. I intend to publish it no earlier than: $ date -d 'Sun, 04 Jul 2021 16:00:00 +0100' https://time.is/compare/1600_04_Jul_2021_in_BST ? Please feel free and commit/push to drafts without the overhead of sending patches or merge requests. You should make your changes to the "_reports/2021-06.md" file in the "reproducible-website" repository: $ git clone https://salsa.debian.org/reproducible-builds/reproducible-website $ cd reproducible-website $ sensible-editor _reports/2021-06.md I am happy to reword and/or rework additions prior to publishing. If you currently do not have access to the above repository, you can request access by following the instructions at: https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From dwheeler at dwheeler.com Fri Jul 2 16:59:54 2021 From: dwheeler at dwheeler.com (David A. Wheeler) Date: Fri, 2 Jul 2021 12:59:54 -0400 Subject: Please review the draft for June's report In-Reply-To: <162524363873.767378.12451745362785428184@tinycat.chris-lamb.co.uk> References: <162524363873.767378.12451745362785428184@tinycat.chris-lamb.co.uk> Message-ID: > On Jul 2, 2021, at 12:35 PM, Chris Lamb wrote: > > Hi all, > > Please review the draft for June's Reproducible Builds report: > > https://reproducible-builds.org/reports/2021-06/?draft Thanks again for doing this! The report has an Alpine entry, but there?s a new Alpine blog entry that also discusses reproducible builds: https://ariadne.space/2021/07/01/bits-relating-to-alpine-security-initiatives-in-june/ I proposing adding, after "Elsewhere in Alpine news,...? Text like the following: Adriadne Conill later posted hi, this is a reminder for our next IRC meeting on #reproducible-builds on the last Tuesday of the month (so 31st of August) at 15 UTC on irc.oftc.net. The meeting is supposed to last between 1-2h, maybe rather an hour, though we have lots of time (just after 23-42m on one topic we move on anyway), though of course we aim to keep it short. The meetings are logged via https://meetbot.debian.net/reproducible-builds The agenda is at https://pad.riseup.net/p/rb-irc-meetings-keep and currenly looks like this. If you add topics please add your nick as well. Agenda for the meeting on 31st of August 2021 ------------------------------------------- welcome to this monthly meeting, please briefly introduce yourself or update us on recent or planned work short time slot for checkins from various projects: Debian: snapshot.d.o mirror status update (fepitre) Debian: rebuilder status update (h01ger) ... [...]: your topic here. Any Other Business (AOB) Looking forward to talk to you then! -- cheers, Holger ??????? ??????? holger@(debian|reproducible-builds|layer-acht).org ??????? OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ??? Bottled water companies don't produce water, they produce plastic bottles. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From holger at layer-acht.org Thu Jul 22 10:29:21 2021 From: holger at layer-acht.org (Holger Levsen) Date: Thu, 22 Jul 2021 10:29:21 +0000 Subject: #r-b irc meeting, this coming Tuesday, 15 UTC In-Reply-To: <20210722090747.GF12062@layer-acht.org> References: <20210722090747.GF12062@layer-acht.org> Message-ID: <20210722102921.GB21871@layer-acht.org> Hi, somehow I thought it was August already... which of course it isn't. Sorry for the confusion. So, this is correct: this is a reminder for our next IRC meeting on #reproducible-builds on the last Tuesday of the month (so 27th of July) at 15 UTC on irc.oftc.net. The meeting is supposed to last between 1-2h, maybe rather an hour, though we have lots of time (just after 23-42m on one topic we move on anyway), though of course we aim to keep it short. The meetings are logged via https://meetbot.debian.net/reproducible-builds The agenda is at https://pad.riseup.net/p/rb-irc-meetings-keep and currenly looks like this. If you add topics please add your nick as well. Agenda for the meeting on 27th of July 2021 ------------------------------------------- welcome to this monthly meeting, please briefly introduce yourself or update us on recent or planned work short time slot for checkins from various projects: Debian: snapshot.d.o mirror status update (fepitre) Debian: rebuilder status update (h01ger) ... [...]: your topic here. Any Other Business (AOB) Looking forward to talk to you then! -- cheers, Holger ??????? ??????? holger@(debian|reproducible-builds|layer-acht).org ??????? OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C ??? The upcoming clima apocalypse is the big elephant in every room now. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From chris at reproducible-builds.org Fri Jul 23 09:48:29 2021 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 23 Jul 2021 10:48:29 +0100 Subject: Help us map the reproducible builds ecosystem Message-ID: Hi all, (Scroll down the link, but please do read the rest of the email before jumping in!) In order to better grow and understand the Reproducible Builds project and community, a handful of us are in the process of building an "ecosystem map". (For those not familiar with the concept of "ecosystem", please feel free to check out https://en.wikipedia.org/wiki/Digital_ecosystem) To accomplish this, h01ger, gunner and I have brainstormed a big messy list of "ecosystem participants", and we would now love folks on this mailing list to build a "clean room" version of the same. In other words, we are asking you to join in a brainstorm of who and what we collectively consider or know to be participants, contributors or active adopters in and around the RB ecosystem. We would then take "your" list and unified it with our own; we aren't disclosing our version just yet in order to give you a chance for a fresh start and to locate any blind spots. (Thanks to Gunner for this excellent idea.) We have created an Etherpad here to invite your ideas: https://pad.riseup.net/p/rbecosystemmapping-keep In particular, we're looking for: * Entities that are participating in some way in reproducible builds or have done so in the past. This could be technology projects, organisations, individuals, etc. and any type of entity you consider to be associated with us. * Entirely new categories that are not in the list, as well as members of any newly added categories. Please let us know if this request isn't clear, but otherwise please head on over to the pad linked above and share your thoughts on what makes up the Reproducible Builds ecosystem! Thanks again. :) Best wishes, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From rclobus at rclobus.nl Sun Jul 25 12:05:44 2021 From: rclobus at rclobus.nl (Roland Clobus) Date: Sun, 25 Jul 2021 14:05:44 +0200 Subject: Second status update about reproducible live-build ISO images in Jenkins Message-ID: <66f20fa0-6186-1353-1c03-193d001cc568@rclobus.nl> Hello lists, here is a second update of the status for reproducible live-build ISO images [1]. * All major configurations are now built on a daily basis using live-build [2] * All major configurations (except for one) are reproducible * Only the cinnamon image has shown non-reproducible builds, but... ** Diffoscope has issues when comparing these ISO files [3] The ISO images were captured, but on my computer diffoscope is able to finish without a crash (taking more than 2 hours and a lot of space on /tmp though) So this crash might be 'unique' to the current Jenkins setup ** The Perl script /usr/share/perl5/XML/SAX/Debian.pm of libxml-sax-perl contains a foreach on a hash, which _occasionally_ results in a different sort order [4] (a patch is pending, to be added to the Jenkins script, [1] and a new bug report) * While generating the artifacts for later retrieval, I missed a cleanup step, which resulted in /tmp on the Jenkins master node to fill up. Sorry about that... ** As an emergency step, the generation of artifacts is disabled ** A merge request (containing several modifications) is planned, which prevents such possible scenarios You can stop reading here if you want... Future plans: * The building of the live-build images will be spread more evenly, to avoid heavy spikes [5] * The new snapshot service will be used [6] * Reprotest might be used instead of just 2 builds without a short time frame, to capture more variations * Reprotest does not appear to set PERL_HASH_SEED, which might trigger some more non-reproducible cases * The reporting page of the Jenkins job is still rather minimal * The generated ISO files will be stored again (for 24 hours), when it can be assured that Jenkins will not be filled up again * I would like to test the functionality of the generated ISO image. ** I've read about the approach by Tails, that looks really promising (and cool) [7] ** There is also OpenQA, which already tests the current daily images [8] ** Running tests of the functionality of the installer images would reduce a lot of stress during release times * When live-build images are working fine, the work could be extended to other images, e.g. the live-wrapper images, the netinst images or perhaps even Docker images With kind regards, Roland Clobus [1] https://wiki.debian.org/ReproducibleInstalls/LiveImages [2] https://jenkins.debian.net/view/live/ [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=991059 [4] https://reproducible-builds.org/docs/stable-outputs/ [5] https://jenkins.debian.net/munin/debian.net/osuosl173-amd64.debian.net/index.html [6] https://debian.notset.fr/snapshot [7] https://tails.boum.org/contribute/release_process/test/automated_tests/ [8] https://openqa.debian.net/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From kpcyrd at rxv.cc Sun Jul 25 18:21:38 2021 From: kpcyrd at rxv.cc (kpcyrd) Date: Sun, 25 Jul 2021 18:21:38 +0000 Subject: Seeking Help: grub man pages not reproducible Message-ID: <20210725182137.GH32120@localhost> Hello! One of the rebuilderd based Arch Linux rebuilders flagged a design issue in grub that likely affects all distros: ? ??? usr/share/man/man8/grub-install.8.gz ? ? ??? grub-install.8 ? ? ? @@ -112,15 +112,15 @@ ? ? ? is only available on EFI. ? ? ? .TP ? ? ? \fB\-s\fR, \fB\-\-skip\-fs\-probe\fR ? ? ? do not probe for filesystems in DEVICE ? ? ? .TP ? ? ? \fB\-\-target\fR=\fI\,TARGET\/\fR ? ? ? install GRUB for TARGET platform ? ? ? -[default=i386\-pc]; available targets: ? ? ? +[default=x86_64\-efi]; available targets: ? ? ? arm\-coreboot, arm\-efi, arm\-uboot, arm64\-efi, ? ? ? i386\-coreboot, i386\-efi, i386\-ieee1275, ? ? ? i386\-multiboot, i386\-pc, i386\-qemu, i386\-xen, ? ? ? i386\-xen_pvh, ia64\-efi, mips\-arc, mips\-qemu_mips, ? ? ? mipsel\-arc, mipsel\-loongson, mipsel\-qemu_mips, ? ? ? powerpc\-ieee1275, riscv32\-efi, riscv64\-efi, ? ? ? sparc64\-ieee1275, x86_64\-efi, x86_64\-xen Note this is inside a man page, but not the binary. Searching the previous line through the whole codebase leads to this location, in a C file: % rg 'install GRUB for TARGET platform' util/grub-install.c 258: 0, N_("install GRUB for TARGET platform [default=%s]; available targets: %s"), 2}, It seems the man page is generated from the --help text. The default is coming from get_default_platform() which calls grub_install_get_default_x86_platform(): const char * grub_install_get_default_x86_platform (void) { if (is_efi_system()) { if (read_platform_size() == 64) return "x86_64-efi"; else return "i386-efi"; } grub_util_info ("Looking for /proc/device-tree .."); if (is_not_empty_directory ("/proc/device-tree")) { grub_util_info ("...found"); return "i386-ieee1275"; } grub_util_info ("... not found"); return "i386-pc"; } This effectively means `grub-install --help` tests for /sys/firmware/efi among other things, I've confirmed this on two different systems: $ grub-install --help | rg 'available targets' [default=i386-pc]; available targets: and % grub-install --help | rg 'available targets' [default=x86_64-efi]; available targets: The problem now is that `grub-install.8` is a snapshot of that help-text from the build-system, including the system specific defaults that might be different on the users system. I'm calling this a design issue because this means grub is shipping incorrect man pages on some systems: % man grub-install | rg 'available targets' install GRUB for TARGET platform [default=i386-pc]; available targets: arm-coreboot, arm-efi, % grub-install --help | rg 'available targets' [default=x86_64-efi]; available targets: Any ideas? Thanks! From chris at reproducible-builds.org Mon Jul 26 08:35:27 2021 From: chris at reproducible-builds.org (Chris Lamb) Date: Mon, 26 Jul 2021 09:35:27 +0100 Subject: Seeking Help: grub man pages not reproducible In-Reply-To: <20210725182137.GH32120@localhost> References: <20210725182137.GH32120@localhost> Message-ID: Hey kpcyrd, > The problem now is that `grub-install.8` is a snapshot of that help-text from > the build-system, including the system specific defaults that might be > different on the users system. > > I'm calling this a design issue because this means grub is shipping > incorrect man pages on some systems: > > % man grub-install | rg 'available targets' > install GRUB for TARGET platform [default=i386-pc]; > available targets: arm-coreboot, arm-efi, > % grub-install --help | rg 'available targets' > [default=x86_64-efi]; available targets: > > Any ideas? Just to zoom out for a second, manual pages that embed properties of the build system (via the output --help or analogous mechanism) is surprisingly common. I have addressed 100s of similar (if not identical issues) in Debian over the years, although this has something of an observation bias to it. These issues are very easy to locate in the output of diffoscope and, as low hanging fruit, they might have been addressed before others. :) Perhaps the most common property that leaks into manpages is the number of CPUs the build system. This typically ends up in the manual page because the number of CPUs is often used for some default amount of concurrency (e.g. "--num-threads=X [default: 16]"). I mention this in particular as I believe the most user-friendly and upstream acceptable approach to autodetected values like this is to patch the output of --help with fixed values that indicate that they are autodetected. For example, "--num-threads=X [default: num cpus]" or "default: autodetected", instead of the actual number. (*) In fact, I believe I could make a case that some fixed "autodetected" string is superior. After all, it conveys, completely accurately, that the value is autodetected and not some seemingly arbitrary number So for Grub here I would probably experiment with patching the "x86_64-efi" bit with some kind of fixed, metasyntactic variable that makes the best sense in context. "CURRENT", "" might be a place to start, but the answer will _always_ come from looking at the manpage as a whole and seeing what best fits the scenario and trying to put yourself in the mindset of a user reading the output in a rush and just trying to get something done. Hope that braindump helps... (*) Of course, sometimes this requires more patching to make it actually autodetected by the runtime. Sometimes --help will claim it is autodetected, but the value was actually detected long ago on some _build_ system and then fixed. (Which can be interesting if you build system has 128 CPUs and you then run it on some poor embedded device....) Best wishes, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From chris at reproducible-builds.org Tue Jul 27 07:24:31 2021 From: chris at reproducible-builds.org (Chris Lamb) Date: Tue, 27 Jul 2021 08:24:31 +0100 Subject: #r-b irc meeting, this coming Tuesday, 15 UTC In-Reply-To: <20210722102921.GB21871@layer-acht.org> References: <20210722090747.GF12062@layer-acht.org> <20210722102921.GB21871@layer-acht.org> Message-ID: <843888f5-0c80-4230-a630-c3648c28477e@www.fastmail.com> Holger Levsen wrote: > this is a reminder for our next IRC meeting on #reproducible-builds on the > last Tuesday of the month (so 27th of July) at 15 UTC on irc.oftc.net. Just a quick reminder that this is later today. Best wishes, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From dan at shearer.org Tue Jul 27 16:23:40 2021 From: dan at shearer.org (Dan Shearer) Date: Tue, 27 Jul 2021 17:23:40 +0100 Subject: Announcement of LumoSQL Phase II Message-ID: Announcing LumoSQL Database Phase II ==================================== July 27, 2021 LumoSQL is a derivative of SQLite, the most-used software in the world. Our focus is on privacy, at-rest encryption, reproducibility and the things needed to support these goals. The NLNet Foundation continues to support LumoSQL, this time via the NGI Assure fund linked below. In LumoSQL Phase II, the team is focussing on: * Implementing a small subset of the Postgresql 13 RBAC permissions model via the SQL statements CREATE ROLE/GRANT/REVOKE etc. An important addition to Postgres is to allow per-row permissions as well as per-table. * An extension of Phase I's hidden column mechanism, now to include hidden tables. When using the native SQLite file format, hidden columns (similar to ROWID) and tables are intended to be invisible to unmodified SQLite binaries. These columns and tables implement row-based and table-based encryption and more. * Further integration of the LMDB key-value store as an optional backend, from version 0.9.11 onwards, and also the LMDBv1.0 pre-release. This work aims to implement remaining SQLite and LMDB API features, and to prepare for the page-level database encryption that is coming with LMDBv1.0. * Improved documentation, assembling the knowledge we gained in Phase I, discussing the infrastructure as we now understand it, and covering the new features for privacy and encryption. We do realise there is plenty of work to do on this front :-) LumoSQL retains and builds on Phase I features including: * The build, benchmark and test tools to measure unmodified SQLite and LumoSQL in various configurations * The Not-Fork tool which allows multiple upstream codebases to be combined across multiple versions without forking Further info from these URLs: LumoSQL Database: https://lumosql.org/src/lumosql Not-Forking reproducibility tool: https://lumosql.org/src/not-forking NLNet NGI Assure fund: https://nlnet.nl/assure/ Permalink for this announcement: https://lumosql.org/src/lumosql/doc/trunk/doc/LumoSQL-PhaseII-Announce.md -- Dan Shearer dan at lumosql.org From chris at reproducible-builds.org Fri Jul 30 08:11:35 2021 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 30 Jul 2021 09:11:35 +0100 Subject: =?UTF-8?Q?diffoscope_179_released_=F0=9F=92=A0?= Message-ID: <162763268319.66187.16286532250828217331@copycat> Hi, The diffoscope maintainers are pleased to announce the release of version 179 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 179 includes the following changes: [ Chris Lamb ] * Ensure that various LLVM tools are installed, even when testing whether a MacOS binary has zero differences when compared to itself. (Closes: reproducible-builds/diffoscope#270) ## Download Version 179 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o