[Git][reproducible-builds/reproducible-presentations][master] drop (outdated) script

Mon Aug 31 10:48:19 UTC 2020


Holger Levsen pushed to branch master at Reproducible Builds / reproducible-presentations


Commits:
ff682b48 by Holger Levsen at 2020-08-31T12:48:12+02:00
drop (outdated) script

Signed-off-by: Holger Levsen <holger at layer-acht.org>

- - - - -


1 changed file:

- − 2020-08-27-Reproducing-bullseye-in-practice/script.txt


Changes:

=====================================
2020-08-27-Reproducing-bullseye-in-practice/script.txt deleted
=====================================
@@ -1,207 +0,0 @@
-intro
-	there's hardly any intro, i expect you have heard about reproducible builds
-	if not, look for talks
-	to repeat: reproducible builds is to enable anyone to indepently verify that a given binary is coming for the source its said to be coming
-	there's a lot to say about why we want this, our motivation.
-	security, reliability, trustworthy computing, trustable computing, building sustainable building blocks for societies. i believe we must.
-	but i'll skip this here.
-
-today i will talk about reproducible Debian bullseye.
-and buster, but that story is very short, so for this talk we'll imagine
-schrödinger's bullseye! it's both released and unreleased at the same time.
-
-released, cause some things currently only work for released debian distros (and unstable),
-and unreleased, because thankfully we can still fix things in buster.
-
-warning: this talk / topic is (too?) rich on details yet it probably misses some special cases like bootstrapping architectures. Please accept my apologies and send corrections where needed!
-
-so, where are we?
-
-we have installed debian from deb.debian.org.
-we choose bullseye, it's a very nice release,
-probably released some time in 2021.
-
-so, let's go to
-https://tests.reproducible-builds.org/debian
-
-nah, that's cool, but "just" based on CI runs.
-as are the results on tracker.do
-
-
-and what is shown on isdebianreproducibleyet.com
-uses that data as well.
-
-
-sid $ reproducible-check
-72/3141 (2.29%) of installed binary packages are unreproducible.
-
-That's too good to be true.
-
-this is #123456
-
-So?
-
-Seems noone has done this. Well, this is not true.
-in-toto folks, but too much integration with in-toto
-and they are actually rebuilding our CI builds. 
-(AIUI, ask sangy to review this claim. and foxboron/kpcyrd too)
-
-so, anyhow, how can one independently verify that a given binary is 
-coming for the source its said to be coming.
-( dpkg -l | verify-the-world ) && echo success
-I wish it was that easy.
-
-So, we have buildinfos.debian.net
-(not to be confused with buildinfo.debian.net...)
-
-which is not available from .debian.org but ok.
-
-this is #123456
-
-it's sorted by release date
-but Holger also created a pool structure
-
-which is a bit unusable, as the filename in the date structure doesnt reflect the content,
-so the filenames in the pool structure indicate the content:
-
-$pkg_$version_amd64+all+source.buildinfo
-
-so this is mess.
-(more soon)
-
-but it's also incomplete:
-
-Ivo de Decker thankfully scheduled thousands of binNMUs
-*and* the release team changed the policy so that now
-only binaries on the built on the buildds are in testing.
-
-That is great (and is why this talk is about bullseye
-and not buster) but then we still have 4447 arch:all packages
-in bullseye which were uploaded before December 2016
-(and thus have no .buildinfo file at all cause dpkg lacked support)
-or where uploaded after (but before 2019) but without
-.buildinfo file.
-
-And then, we still miss all the security updates since
-bullseyes (imagined) release.
-
-because .buildinfo files for security updates currently are lost.
-(and probably only easily reachable at/after point releases
-because embargoed.)
-
-and then... well, then, we could parse the files from
-buildinfos.debian.net, if we have a local copy.
-
-haha.
-
-or we could setup builtin-pho from David Bremner,
-which also needs a local copy from buildinfos.debian.net,
-but which I have done, cause I have one, and now
-I also have a nice postgresql database with all the information
-about .buildinfo files.
-
-so, yay,
-
-lets reproduce a package.
-
-
-so, I thought, this should be easy. I knew it would not be but I still think it should.
-
-So I was pushing to get srebuild into src:devscripts
-
-And since then I filed 15 bugs against srebuild in src:devscripts.
-
-Let me explain, I wrote a prototype shell script to get a feel on
-how to rebuild the debian archive based on .builfinfo files
-and called it bin/reproducible_debian_rebuilder_prototype.sh
-in jenkins.debian.net.git and I would like to think I'm as bad as bremner
-in naming things but the truth is probably that i'm worse. though bremner 
-is very hard competition here ;)
-
-So I pick a random binary package and use grep-available to
-get the source package the binary package belongs too, easy:
-
-ARCH=amd64
-SUITE=bullseye
-DISTNAME="$SUITE-$ARCH"
-PACKAGES=$(ls $CHPATH/$DISTNAME/var/lib/apt/lists/*_dists_${SUITE}_main_binary-${ARCH}_Packages)
-SOURCES=$(ls $CHPATH/$DISTNAME/var/lib/apt/lists/*_dists_${SUITE}_main_source_Sources)
-BINARY_PKG=$(grep-available . -s Package $PACKAGES | sort -R | head -1 | cut -d ' ' -f2)
-PKG=$( ( grep-available -X -S $BINARY_PKG -s Package $SOURCES || grep-available -X -P $BINARY_PKG -s Source $PACKAGES ) | cut -d ' ' -f2)
-VERSION=$(grep-available -X -S $PKG -s Version $SOURCES | head -1 | cut -d ' ' -f2)
-
-and query builtin-pho for the .buildinfo file for arch=amd64 or arch=all
-(I *believe* this doesnt work as well for other archs than amd64 as most arch:all
-binaries are build on amd64 thus...)
-
-as expressed before, there are quite some binaries we cannot find .buildinfo files for.
-let's ignore that. (though I have a question: do you think I should automate this and do
-100 NMUs per day, for 30 days? (And probably rather rather do 1-200, then 7x100, then 21x100.)
-
-so, we have found a .buildinfo file.
-
-if it's signed, we strip the signature because #955050 in devscripts: debrebuild: please accepted signed .buildinfo files
-
-and also because even if it's signed, the key might be expired or not on the keyservers
-or not being signed by other keys, so what's the point.
-
-and, if the .buildinfo file contains the hashes of the sources
-and the hashes of the binaries and the hashes of the binaries
-match, then this should be good enough, or?
-
-well, maybe, but... the Debian .buildinfo files dont contain the
-hashes of the sources (contrary to our design in 2015).
-
-but lets move on and ignore this for now.
-
-but anyway it would be nice if debrebuild would download the source.
-
-because, lets assume we found a binNMU. there's a .buildinfo file, there's a source,
-but there's no tooling to create a source from this source and the .buildinfo file, which
-we want to rebuild. and srebuild has all the information already anyway.
-
-this is #123456
-
-so then we have grepped our sbuild command from srebuild output
-
-this is #123456
-
-and grepped the environment and the base release as well
-
-this is #123456
-
-(and then one needs to create a base chroot with sbuild, which is easy, but still it would be nice if there would be one do-it-all-at-once mode - this is #123456
-
-and then srebuild fails to recreate the environment way too often.
-
-this is #123456
-
-and then, sometimes, we actually have rebuilt something.
-
-and sometimes it matches what ftp.debian.org publishes.
-
-Yay.
-
-Help welcome. debrebuild as sbuild is written in perl.
-
-the in-toto stuff is written in python, so is apt-transport-into.
-
-i'm not sure where exactly this is going to once we can easily independently verify that a given binary comes from a certain source.
-
-cause then we will want others to build for us (as well as maybe us too)
-and then place trust on that.
-
-for now, I would like Debian to be able to say:
-
-a.) this is binary A with version B for architecture C which was build with .buildinfo file from URL D from source package E.
-b.) as gpg signing is hard, I'm considering to release this information in a very simple blockchain^wmerkel tree, append every day and share this with the world. Have 23 mirrors^wgit clones.
-c.) to rebuild and verify all packages installed on your system, you can run this loop: for i in $list_of_packages ; do debrebuild $i ; done
-d.) we released this set of binaries (with these version numbers) build from these .buildinfo files. And *we*, Debian, managed to rebuild this sub set identically. (And this we didn't.)
-
-and then, I want e.) other rebuilders. But I think we need to do a-d first ourselves.
-
-For other rebuilders we have a nice new verification format.
-
-Finally:
-
-this is list of all bugs mentioned in this talk: #123456 #123457 #123458 #123456



View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-presentations/-/commit/ff682b48a40fa11102d684af14ed440ac1966239

-- 
View it on GitLab: https://salsa.debian.org/reproducible-builds/reproducible-presentations/-/commit/ff682b48a40fa11102d684af14ed440ac1966239
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-commits/attachments/20200831/fdf9f9ec/attachment.htm>
