[Git][reproducible-builds/reproducible-presentations][master] 2 commits: dc24 talk: more wip, almost there
Holger Levsen (@holger)
gitlab at salsa.debian.org
Sun Jul 28 08:32:10 UTC 2024
Holger Levsen pushed to branch master at Reproducible Builds / reproducible-presentations
Commits:
6470a37e by Holger Levsen at 2024-07-28T15:53:27+09:00
dc24 talk: more wip, almost there
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
583b164e by Holger Levsen at 2024-07-28T17:31:36+09:00
dc24 talk: maybe done
Signed-off-by: Holger Levsen <holger at layer-acht.org>
- - - - -
2 changed files:
- 2024-07-22-R-B-the-first-11-years/index.html
- 2024-07-22-R-B-the-first-11-years/todo
Changes:
=====================================
2024-07-22-R-B-the-first-11-years/index.html
=====================================
@@ -986,10 +986,19 @@ Warpforge.
<h3>Short summary of Reproducible Debian</h3>
</section>
+ <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
+ <h3>CI results for Debian unstable, 20240727</h3>
+ <img src="images/stats_pkg_state_20240727.png">
+ </section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>CI results for Debian trixie, 20240727</h3>
- <img src="images/stats_pkg_state_trixie_20240727.png">
+ <h3>Debian unstable, 20150131</h3>
+ <img src="images/stats_pkg_state_20150131.png">
+ </section>
+
+ <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
+ <h3>CI results for Debian unstable, 20240727</h3>
+ <img src="images/stats_pkg_state_20240727.png">
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
@@ -1068,11 +1077,25 @@ Warpforge.
</table>
</section>
+
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>CI results for Debian unstable, 20240727</h3>
- <img src="images/stats_pkg_state_20240727.png">
+ <h3>CI results for Debian trixie, 20240727</h3>
+ <img src="images/stats_pkg_state_trixie_20240727.png">
</section>
+ <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
+ <h2>Debian <em>testing</em> migration</h2>
+ <ul style="font-size: 94%">
+ <li>Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.</li>
+ <li>In 2025 for Debian 14 "<em>forky</em>" however there should penalties for violating:
+ <ul>
+ <li class="fragment">reproducible packages <em>must not</em> regress (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
+ <li class="fragment">NEW packages <em>must</em> build reproducibly (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
+ </ul>
+ <li class="fragment">At first there could be whitelisting of some needed packages, but less over time until we can drop those exceptions.</li>
+ </ul>
+ </section>
+
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
<h2>Debian policy</h2>
<ul>
@@ -1090,6 +1113,7 @@ Warpforge.
<li class="fragment">We need to change <code>debian-policy</code>!</li>
<li class="fragment">We can work around 'must-have-offenders' using whitelists in the beginning.</li>
<li class="fragment">The goal is still 100%, whitelists are just a way to achieve that goal eventually.</li>
+ <li class="fragment">Penalizing testing migration IMO is a means to enforce <code>debian-policy</code> though it can be done before it's policy.</li>
</ul>
</section>
@@ -1120,16 +1144,16 @@ Warpforge.
</tr><tr>
<td>forky</td>
<td>40000</td>
-<td><span style="color: #ff0000">77</span></td>
+<td><span style="color: #ff0000">128 (but no regressions or new pkgs)</span></td>
</tr><tr>
</tr><tr>
<td>forky+1</td>
<td>45000</td>
-<td><span style="color: #ff0000">42</span></td>
+<td><span style="color: #ff0000">42 policy violations left</span></td>
</tr><tr>
<td>forky+2</td>
<td>50000</td>
-<td><span style="color: #00ff00">0</span></td>
+<td><span style="color: #00ff00">0 (?!?!!!)</span></td>
</tr><tr>
</tr>
@@ -1137,35 +1161,18 @@ Warpforge.
</section>
- <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h2>Debian <em>testing</em> migration</h2>
- <ul style="font-size: 94%">
- <li>Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.</li>
- <li>In 2025 for Debian 14 "<em>forky</em>" however there should penalties for violating:
- <ul>
- <li class="fragment">reproducible packages <em>must not</em> regress (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
- <li class="fragment">NEW packages <em>must</em> build reproducibly (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
- </ul>
- <li class="fragment">At first there could be whitelisting of some needed packages, but less over time until we can drop those exceptions.</li>
- </ul>
- </section>
-
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
<h2>Debian <em>testing</em> migration, an even better outlook</h2>
<ul style="font-size: 94%">
- <li>Since the end of 2023, CI reproducible-builds results are included in the excuses output for Debian testing migration, but there is no penalty nor bonus yet.</li>
- <li>In 2025 for Debian 14 "<em>forky</em>" however there should penalties for violating:
- <ul>
- <li>reproducible packages <em>must not</em> regress (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
- <li>NEW packages <em>must</em> build reproducibly (to be allowed into <code>testing</code> and therefore into <code>stable</code>).</li>
- </ul>
- <li class="fragment"><code>snapshot.debian.org</code> got fixed and thus it seems we'll finally be able to base this on <b>rebuilder!</b></li>
+ <li>2023: CI reproducible-builds results included in excuses output for Debian testing migration, but there is no penalty nor bonus yet.</li>
+ <li>2025: (for Debian 14 "<em>forky</em>"): testing migration penalities for reproducibility regressions or new unreproducible packages.
+ <li class="fragment">July 2024: <code>snapshot.debian.org</code> got fixed and thus it seems we'll finally be able to base this on <b>rebuilder</b> instead of CI builds.</li>
</ul>
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
<h2><code>snapshot.debian.org</code> got fixed!</h2>
- /me dances
+ <h1>🥳</h1>
<p class="fragment">huge thanks to Linux Nordberg and DSA!</p>
</section>
@@ -1175,41 +1182,50 @@ Warpforge.
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h4>rebuilders https://beta.tests.reproducible-builds.org/</h4>
+ <h4>https://beta.tests.reproducible-builds.org/</h4>
<img src="images/bookworm_full.amd64+all_borked.png">
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>100% reproducibility in theory is not enough,<span style="font-size: 140%"> by far.</span></h3>
+ <h3>reproducibility in theory is not enough</h3>
<ul>
<li>Then we need rebuilders. Remember: we only have had CI builders for the last 10 years, but what we really want is to be able to rebuild what is distributed on ftp.debian.org.</li>
<li class="fragment">Thus we need a working <code>snapshot.debian.org</code> service.</li>
<li class="fragment">Without snapshot.d.o we cannot recreate the exact same environments...</li>
- <li class="fragment">but snapshot is buggy: #1050815, #1031628, #1029744, #1034000, #1012559, #979115, #969603… </li>
- <li class="fragment">And there we <b>had</b> been stuck for more than five years...</li>
+ <li class="fragment">but snapshot was buggy: <strike>#1050815, #1031628, #1029744, #1034000</strike>, #1012559, #979115, <strike>#969603</strike>… </li>
+ <li class="fragment">And there we <b>had</b> been stuck for more than five years... (as the bugs above weren't fixed until last month.)</li>
</ul>
</section>
-
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>fixing snapshot.d.o</h3>
- <ul>
- <li>150 TB of data for 20 years</li>
- <li>4 pushes per day, adding 70 GB each.</li>
- <li class="fragment">a fun project to fix, so they gave me git commit access and made me a member of the Debian snapshot LDAP group. 🙈</li>
- <li class="fragment">I was honored but we needed something soon.</li>
- <li class="fragment">and then Linus Nordberg stepped in and fixed some bugs.</li>
- </ul>
+ <h2><code>snapshot.debian.org</code> got fixed!</h2>
+ <h1>🥳</h1>
+ <p>huge thanks to Linux Nordberg and DSA!</p>
+ <p class="fragment">IMO this deserves a proper announcement with the technical details...!</p>
</section>
+
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h4>rebuilders https://beta.tests.reproducible-builds.org/debian</br>(these are not CI builds anymore)s</h4>
- <img src="images/bookworm_full.amd64+all.png">
+ <h3>debootsnap and debrebuild from src:devscripts</h3>
+ <ul>
+ <li class="fragment">wget https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/2024/01/16/crun_1.13-1_amd64.buildinfo</li>
+ <li class="fragment">debrebuild --builder=sbuild libaacs_0.11.1-3_amd64-source.buildinfo</li>
+ <li class="fragment">debootsnap and debrebuild need a working snapshot.debian.org thus this didn't really work until last month.</li>
+ <li class="fragment">Please try it out and report bugs the BTS.</li>
+ </ul>
</section>
+ <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
+ <h4>https://beta.tests.reproducible-builds.org/debian</h4>
+ <img src="images/bookworm_full.amd64+all.png">
+ <ul>
+ <li>needs re-setup...</li>
+ <li class="fragment">archlinux rebuilderd could also be used</li>
+ </ul>
+ </section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>an idea at the summit 2023: do we need all of snapshot.d.o?</h3>
+ <h3>another quick side quest, an idea from the summit 2023: do we need all of snapshot.d.o?</h3>
<ul>
<li>70000 binary packages in Debian $suite</li>
<li>these build-depend on only 30000 packages, so 40000 packages are never used as build-depends.</li>
@@ -1224,31 +1240,19 @@ Warpforge.
<h3>https://rebuilder-snapshot.debian.net</h3>
<ul>
<li>a cache for snapshot.debian.org, which stores only the packages used as build-depends <em>today</em> and makes them available via SHA256, path and an API.</li>
- <li class="fragment">each arch takes roughly a week to seed from snapshot.d.o</li>
- <li class="fragment">each arch only takes hours to seed from another rebuilder-snapshot instance</li>
- <li class="fragment">we already run two instances and our goal is to allow many instances</li>
- </ul>
- </section>
-
-
- <section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>https://rebuilder-snapshot.debian.net</h3>
- <ul>
- <li>a cache in for snapshot.debian.org, which stores only the packages used as build-depends <em>today</em> and makes them available via SHA256, path and an API.</li>
- <li class="fragment">so <em>debrebuild</em> can use <em>debootsnap</em> together with <em>metasnap</em> to establish trust.</li>
- <li class="fragment">one blocking bug currently: issue #40</li>
- <li class="fragment">hopefully usable RSN. Many thanks to lynxis and josch!</li>
- <li class="fragment">also: snapshot.d.o is still awesome, despite it's flaws today. We need (to fix) it! Many thanks weasel & DSA!</li>
+ <li class="fragment">1 TB instead of 150 TB (for all release archs and stable+testing+unstable)</li>
+ <li class="fragment">needs <code>metasnap.debian.net</code></li>
+ <li class="fragment">still in development</li>
+ <li class="fragment">will be useful to take load from snapshot.d.o and for offline rebuilds</li>
</ul>
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
- <h3>Debian 2024</h3>
+ <h3>more Debian Reproducible Builds successes</h3>
<ul>
- <li>testing migration can and will be used to enforce policy also in regards of reproducible builds (probably only enforcing for real in 2025...)</li>
- <li class="fragment">for a sensible setup of that, we need real rebuilders, aiming to rebuild what Debian distributes.</li>
- <li class="fragment">for that, we need a better working snapshot.d.o, which with rebuilder-snapshot finally is there.</li>
- <li class="fragment">CI builds will stay, to find issues. Rebuilders are needed to show the absence of issues.</li>
+ <li class="fragment">reproducible docker/podman images: docker.debian.net</li>
+ <li class="fragment">reproducible live images: cdimage.debian.org</li>
+ <li class="fragment">reproducible debian-installer (in theory, not tested atm)</li>
</ul>
</section>
@@ -1360,123 +1364,20 @@ Arch Linux is 86.4% reproducible with 1701 bad and 10849 good packages.
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
<h3>
- unreproducible packages out of the 1337 most popular
+ 22 unreproducible src packages out of the 1337 most popular
</h3>
<p style="font-size: 100%">
-bind9
-bluez
-ffmpeg
-gegl
-graphviz
-grub2
-guile-3.0
-libdmapsharing
-libjcat
-libu2f-host
-libzstd
-linux
-lynx
-nss
-numpy
-python3.11
-qtdeclarative-opensource-src
-qtquickcontrols2-opensource-src
-qtsensors-opensource-src
-qtspeech-opensource-src
-qtsvg-opensource-src
-qttools-opensource-src
-qtwayland-opensource-src
-qtwebchannel-opensource-src
-underscore
-vlc
-wireplumber
+ rdma-core jpeg-xl graphviz qtwebengine-opensource-src pipewire tracker colord nss qtbase-opensource-src grub2 libu2f-host apg libdmapsharing libjcat bluez vlc coinor-cgl lynx underscore gegl ffmpeg bind9
<p>
</section>
<section data-background="images/debconf24.svg" data-background-size="10%" data-background-position="93% 9%" data-transition="none">
<h3>
- build-essential-depends unreproducible source packages
+ 96 build-essential-depends unreproducible source packages (out of 5500)
</h3>
<p style="font-size: 77%">
-black
-bluez
-codenarc
-cxxtest
-dejagnu
-eccodes
-eckit
-efl
-emacs
-emoslib
-ffmpeg
-fish
-fltk1.3
-freetds
-gdb
-ghc
-gmetrics
-graphviz
-groovy
-guile-3.0
-h2database
-hevea
-javaparser
-ldc
-libcamera
-libzstd
-linux
-linux86
-lombok
-lucene4.10
-lucene8
-lynx
-mpich
-mrmpi
-mypy
-nbsphinx
-nss
-numpy
-odc
-oxygen-icons5
-pandas
-parallel
-pmix
-pstoedit
-python3.11
-python3.12
-python-django
-python-jsonschema
-qemu
-qt6-5compat
-qt6-declarative
-qt6-multimedia
-qt6-quick3d
-qt6-remoteobjects
-qtconnectivity-opensource-src
-qtdeclarative-opensource-src
-qtremoteobjects-everywhere-src
-qtsensors-opensource-src
-qtserialport-opensource-src
-qtspeech-opensource-src
-qtsvg-opensource-src
-qttools-opensource-src
-qtwayland-opensource-src
-qtwebchannel-opensource-src
-qtwebsockets-opensource-src
-r-base
-ruby-pygments.rb
-scikit-learn
-scons
-secilc
-sqlalchemy
-statsmodels
-stunnel4
-sympy
-systemtap
-underscore
-valgrind
-vlc
+ maven-shared-utils rdma-core rustc php8.2 jpeg-xl subversion systemtap graphviz r-base pipewire pandas camlp-streams statsmodels colord chromium qt6-declarative nss qtbase-opensource-src python3.12 python3.13 cdi-api gdb pupnp pacemaker scim gcc-13-cross qemu efl bsh cxxtest codenarc fop jsoup hevea libnative-platform-java javaparser black pstoedit lucene4.10 gmetrics emoslib combblas node-function-bind gdcm xmlbeans yarl nbsphinx groovy fltk1.3 ruby-pygments.rb libapache-poi-java ldc doxygen ghc lombok h2database freetds jsch jboss-jdeparser2 dejagnu jts python-nacl servlet-api jaxb ocaml-topkg odc qtremoteobjects-everywhere-src mpich lucene8 secilc bluez vlc valgrind linux86 golang-golang-x-net coinor-cgl parallel lynx underscore asmtools rocm-hipamd libcamera nbconvert mypy petsc node-d3 qtconnectivity-opensource-src eckit eccodes frozenlist extlib emacs ffmpeg bind9 meson-python ipyparallel
<p>
</section>
=====================================
2024-07-22-R-B-the-first-11-years/todo
=====================================
@@ -1,20 +1,6 @@
-update snapshot slides
- debrebuild / linus
-update rebuilder slides
-update popcon1337 and build-essential stats
-emphasize small number of unreproducible packages left
include paper by lamby & zack
-list more debian successes
- live-images
- docker/podman images: docker.debian.net
- d-i (in theory, not tested atm)
-
-explicit community slide?
- /who/projects
ask projects about their stati! now is time.
-what is success?
-"theoretical?" & for users?
- help wanted
+
View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-presentations/-/compare/a6028dcb9fdf1f3c680f328514b6ac1d98f882d6...583b164e8026015c212a908220c48243745b7ad6
--
View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-presentations/-/compare/a6028dcb9fdf1f3c680f328514b6ac1d98f882d6...583b164e8026015c212a908220c48243745b7ad6
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-commits/attachments/20240728/05714986/attachment.htm>
More information about the rb-commits
mailing list