From leo at LeoWandersleb.de Sat Aug 1 05:51:31 2020 From: leo at LeoWandersleb.de (Leo Wandersleb) Date: Sat, 1 Aug 2020 01:51:31 -0400 Subject: I would love to expand WalletScrutiny to Linux but how? Message-ID: <876d6cc0-4fd3-ada4-6b8a-1ae86b70e969@LeoWandersleb.de> Hi list, since weeks I have ideas on how to expand WalletScrutiny.com from only monitoring Android Bitcoin wallets to also monitoring Linux wallets for reproducibility. On general Linux, reproducibility is much more a topic than on Android so I hope to find it much more here but most see reproducibility as a tool a user can use to reassure himself that a binary he downloaded was compiled from a certain source code. The scope of WalletScrutiny is slightly more ambitious: I hope to catch as early as possible any rogue update that would have a huge impact else. Protect the user that is ignorant to the topic of reproducibility. On Android there is: * effectively one repository of binaries (Google Play). Alternative repositories (FDroid, Amazon, ...) account for probably still less than 1% of the market. * updates are pinned to a release manager's private key as the user cannot install an update that is not signed by the same key as all the previous versions. On Linux there is: * The provider's website's download link. * Significantly more than one relevant Linux, each with their own app repository and maintainers. * Snap as an attempt at providing apps to multiple Linux flavors. * Secondary websites. * No pinning to a signing key unless the app has a self-update feature which is scary, too. On Android many developers reacted allergic to their app not being reproducible. "After all they develop it for free as open source." and when I say "proof it!", they usually don't turn more friendly but I try to distinguish between the developer who owes me nothing and the release maintainer who should have some accountability when providing a wallet application to the general public. In that sense, I don't care too much about the developer but about thee who press the compile button to then share that binary with users to safe-guard funds. The project would have to list wallets sliced by release managers and repositories. "wallet X as obtained from from walletx .org" vs. "... as obtained from debian" ... There is a long way ahead to cover any significant amount of wallet users as that requires learning about accountability in the various repositories, poisoning detection, etc. If you think you know how to spread the word about reproducibility in the context of Bitcoin wallets through WalletScrutiny, your contributions are highly welcome on [this PR](https://gitlab.com/walletscrutiny/walletScrutinyCom/-/merge_requests/68) or its repository in general. Kind Regards, Leo Wandersleb From chris at reproducible-builds.org Sat Aug 1 15:00:05 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Sat, 01 Aug 2020 15:00:05 -0000 Subject: I would love to expand WalletScrutiny to Linux but how? In-Reply-To: <876d6cc0-4fd3-ada4-6b8a-1ae86b70e969@LeoWandersleb.de> References: <876d6cc0-4fd3-ada4-6b8a-1ae86b70e969@LeoWandersleb.de> Message-ID: Hi Leo, > On Android many developers reacted allergic to their app not being reproducible. > "After all they develop it for free as open source." and when I say "proof it!", > they usually don't turn more friendly but I try to distinguish between the > developer who owes me nothing and the release maintainer who should > have some accountability when providing a wallet application to the general > public. (This does not address your question about extending to Linux, but...) Could you describe this phenomenon and interaction pattern with Android developers in more detail? I would particularly interested in what incentives, thought patterns or obstacles you believe you are encountering here. I ask because a big part of the purpose of "Reproducible Builds" existing as a community/project is education, ie. to reach out to other communities so we can work together towards common goals. Understanding what other communities' incentives are is therefore highly instructive in working out how we can collaborate, and I don't believe I have a firm grasp on the Android perspective. Best wishes, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From leo at LeoWandersleb.de Sat Aug 1 16:31:58 2020 From: leo at LeoWandersleb.de (Leo Wandersleb) Date: Sat, 1 Aug 2020 12:31:58 -0400 Subject: I would love to expand WalletScrutiny to Linux but how? In-Reply-To: References: <876d6cc0-4fd3-ada4-6b8a-1ae86b70e969@LeoWandersleb.de> Message-ID: <0441c659-9e6b-e7db-5584-c469445f03f3@LeoWandersleb.de> Hi Chris, > Could you describe this phenomenon and interaction pattern with > Android developers in more detail? I would particularly interested in > what incentives, thought patterns or obstacles you believe you are > encountering here. For a big part I think there is ignorance to the concept of reproducibility, even among developers. It just doesn't occur to them that the release manager under distress might do terrible things. At WalletScrutiny I try to get the reproducible wallets on board to promote the concept. They should advertise the fact but kind of don't yet. Wallets that are open source but not reproducible (if buildable at all) I poke with GitHub issues every now and then and there are some few that show phases of interest to make their builds reproducible but it usually fizzles out. Not sure, why. I keep poking every other month but they usually have other urgent things. At the wallet where I am the release manager, the CEO/CTO did not give it sufficient priority to investigate it for a year of me pushing for it but once I got to look into it, it turned out to be almost reproducible already. Only much later did we have to add disorderfs to the build instructions. > I ask because a big part of the purpose of "Reproducible Builds" > existing as a community/project is education, ie. to reach out to other > communities so we can work together towards common goals. > Understanding what other communities' incentives are is therefore > highly instructive in working out how we can collaborate, and I don't > believe I have a firm grasp on the Android perspective. Native Android is mostly reproducible out of the box but those cross-platform frameworks and stuff developed in C++ are more tricky to get right. From julien at lepiller.eu Sat Aug 1 17:32:57 2020 From: julien at lepiller.eu (Julien Lepiller) Date: Sat, 01 Aug 2020 13:32:57 -0400 Subject: You don't need reproducible builds Message-ID: <21EE8050-0056-4847-BF20-D3301BB33859@lepiller.eu> Hi, With a subject like that, I know I have your attention :). This is actually the title of a blog post I found on my social media, and I wanted to share in case you hadn't read it yet: http://blog.cmpxchg8b.com/2020/07/you-dont-need-reproducible-builds.html?m=1 I think it is important to listen to criticism, as it will help us to better explain and educate about reproducible builds. Thoughts on this? Having met some of you at rb-summit in Paris, I know you're adorable people, but remember to be polite if you decide to comment on his blog. We don't want to sound like we're harassing people :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From foxboron at archlinux.org Sat Aug 1 17:41:34 2020 From: foxboron at archlinux.org (Morten Linderud) Date: Sat, 1 Aug 2020 19:41:34 +0200 Subject: You don't need reproducible builds In-Reply-To: <21EE8050-0056-4847-BF20-D3301BB33859@lepiller.eu> References: <21EE8050-0056-4847-BF20-D3301BB33859@lepiller.eu> Message-ID: <20200801174134.ba2ypdimjlqph3p6@anathema> On Sat, Aug 01, 2020 at 01:32:57PM -0400, Julien Lepiller wrote: > Hi, > > With a subject like that, I know I have your attention :). This is actually > the title of a blog post I found on my social media, and I wanted to share in > case you hadn't read it yet: > > http://blog.cmpxchg8b.com/2020/07/you-dont-need-reproducible-builds.html?m=1 > > I think it is important to listen to criticism, as it will help us to better > explain and educate about reproducible builds. > > Thoughts on this? > > Having met some of you at rb-summit in Paris, I know you're adorable people, > but remember to be polite if you decide to comment on his blog. We don't want > to sound like we're harassing people :) The article misses the point, which is quite apparent when you read the following section. > Now if the vendor is compromised or becomes malicious, they can?t give the > user any compromised binaries without also providing the source code. This > ignores some complexities, like ensuring security updates are delivered even > if one vendor is compromised, what to do if the reproducers stop working, or > how to reach consensus if the reproducers and your vendor disagree on what > software or fork you should be using. Tavis only looks at reproducible builds from the standpoint of a proprietary vendor. Which is obvious considering he works at Google. The section above outlines that the vendor providing the binaries, also provides the source code. But this is not the case for reproducible builds in the context of Linux distributions, or Free and Open-Source software in general. In our case (if I'm allowed to say that :p) the pristine source is separate from the distributor of the binary, and you don't need to have the distributor provide the source, you can fetch it yourself. Tavis also includes the "god argument" of bugdoor, which Reproducible Builds simply can't protect against, thus outside the scope of the project. In short: It's a nice criticism of reproducible builds in the context of proprietary vendors, but it doesn't hold up very well when we look at the Free and Open-Source software communities. -- Morten Linderud PGP: 9C02FF419FECBE16 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From chris at reproducible-builds.org Mon Aug 3 15:32:53 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Mon, 03 Aug 2020 16:32:53 +0100 Subject: =?UTF-8?Q?diffoscope.org_not_resolving_(was:_"Re:_[diffoscope]_Command_`?= =?UTF-8?Q?unsquashfs_-n_-f_-no_-li_-d_._a/yakshaveinc=5F73.snap`_failed?= =?UTF-8?Q?")?= In-Reply-To: References: Message-ID: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> [adding rb-general to CC] Hi Anatoli, > Anyway, writing to let you know that http://try.diffoscope.org/ and > http://diffoscope.org/ are down. Ah, interesting. This is, at least, some kind of domain/DNS issue. For example, try.diffoscope.org should resolve to 46.43.2.47 but it is currently not resolving to anything right now. I don't know what diffoscope.org itself should resolve to but I suspect this is a parallel, if not identical, issue. I've added rb-general to this so that the more sysadmin folks that do Reproducible Builds (Holger, Mattia, Vagrant, etc.) are definitely aware of it. I cannot help any further I'm afraid. ? > It appears I need Salsa account for this. Maybe later I will register. Please do so. It's a bit of a pain but I put together some hopefully foolproof instructions for this here: https://reproducible-builds.org/contribute/salsa/ Let us know if you have suggestions if they could be improved. Best wishes, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From anatoli at rainforce.org Mon Aug 3 16:12:06 2020 From: anatoli at rainforce.org (Anatoli Babenia) Date: Mon, 3 Aug 2020 19:12:06 +0300 Subject: diffoscope.org not resolving (was: "Re: [diffoscope] Command `unsquashfs -n -f -no -li -d . a/yakshaveinc_73.snap` failed") In-Reply-To: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> References: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> Message-ID: Registering on Salsa first gave some server error page, but from the second try it worked. As for the DNS issue, it looks like the domain expired. On Mon, 3 Aug 2020 at 18:33, Chris Lamb wrote: > [adding rb-general to CC] > > Hi Anatoli, > > > Anyway, writing to let you know that http://try.diffoscope.org/ and > > http://diffoscope.org/ are down. > > Ah, interesting. This is, at least, some kind of domain/DNS issue. > > For example, try.diffoscope.org should resolve to 46.43.2.47 but it is > currently not resolving to anything right now. I don't know what > diffoscope.org itself should resolve to but I suspect this is a > parallel, if not identical, issue. > > I've added rb-general to this so that the more sysadmin folks that do > Reproducible Builds (Holger, Mattia, Vagrant, etc.) are definitely > aware of it. I cannot help any further I'm afraid. > > ? > > > It appears I need Salsa account for this. Maybe later I will register. > > Please do so. It's a bit of a pain but I put together some hopefully > foolproof instructions for this here: > > https://reproducible-builds.org/contribute/salsa/ > > Let us know if you have suggestions if they could be improved. > > > Best wishes, > > -- > o > ? ? Chris Lamb > o o reproducible-builds.org ? > ? ? > o > -- Anatoli Babenia +1 (650) 605-3365 +375 (29) 320-4241 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mattia at mapreri.org Mon Aug 3 18:47:59 2020 From: mattia at mapreri.org (Mattia Rizzolo) Date: Mon, 3 Aug 2020 20:47:59 +0200 Subject: [diffoscope] diffoscope.org not resolving (was: "Re: Command `unsquashfs -n -f -no -li -d . a/yakshaveinc_73.snap` failed") In-Reply-To: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> References: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> Message-ID: Hi, Afaik only Holger has access to the DNS management interface (including payments, renewal, etc) at Gandi. I'm writing to him explicitly in the hope he'll see this mail earlier. On Mon, 3 Aug 2020, 5:33 pm Chris Lamb, wrote: > [adding rb-general to CC] > > Hi Anatoli, > > > Anyway, writing to let you know that http://try.diffoscope.org/ and > > http://diffoscope.org/ are down. > > Ah, interesting. This is, at least, some kind of domain/DNS issue. > > For example, try.diffoscope.org should resolve to 46.43.2.47 but it is > currently not resolving to anything right now. I don't know what > diffoscope.org itself should resolve to but I suspect this is a > parallel, if not identical, issue. > > I've added rb-general to this so that the more sysadmin folks that do > Reproducible Builds (Holger, Mattia, Vagrant, etc.) are definitely > aware of it. I cannot help any further I'm afraid. > > ? > > > It appears I need Salsa account for this. Maybe later I will register. > > Please do so. It's a bit of a pain but I put together some hopefully > foolproof instructions for this here: > > https://reproducible-builds.org/contribute/salsa/ > > Let us know if you have suggestions if they could be improved. > > > Best wishes, > > -- > o > ? ? Chris Lamb > o o reproducible-builds.org ? > ? ? > o > _______________________________________________ > diffoscope mailing list > > To change your subscription options, visit > https://lists.reproducible-builds.org/listinfo/diffoscope. > > To unsubscribe, send an email to > diffoscope-unsubscribe at lists.reproducible-builds.org. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at layer-acht.org Tue Aug 4 11:33:22 2020 From: holger at layer-acht.org (Holger Levsen) Date: Tue, 4 Aug 2020 11:33:22 +0000 Subject: [diffoscope] diffoscope.org not resolving (was: "Re: Command `unsquashfs -n -f -no -li -d . a/yakshaveinc_73.snap` failed") In-Reply-To: References: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> Message-ID: <20200804113322.GA29817@layer-acht.org> Hi, On Mon, Aug 03, 2020 at 08:47:59PM +0200, Mattia Rizzolo wrote: > Afaik only Holger has access to the DNS management interface (including > payments, renewal, etc) at Gandi. not according to the Gandi (I just logged in and checked) and 'whois diffoscope.org' returns NS1.POTAGER.ORG as nameserver, thus I will mail the potager.org folks now. -- cheers, Holger ------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From holger at layer-acht.org Tue Aug 4 12:05:36 2020 From: holger at layer-acht.org (Holger Levsen) Date: Tue, 4 Aug 2020 12:05:36 +0000 Subject: [diffoscope] diffoscope.org not resolving (was: "Re: Command `unsquashfs -n -f -no -li -d . a/yakshaveinc_73.snap` failed") In-Reply-To: References: <9ef9bb70-0882-438e-bf77-4887cfb1b464@www.fastmail.com> Message-ID: <20200804120536.GB31143@layer-acht.org> On Mon, Aug 03, 2020 at 07:12:06PM +0300, Anatoli Babenia wrote: > As for the DNS issue, it looks like the domain expired. thankfully this has been fixed by Lunar now. -- cheers, Holger ------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From chris at reproducible-builds.org Thu Aug 6 14:52:31 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Thu, 6 Aug 2020 10:52:31 -0400 (EDT) Subject: Please review the draft for July's report Message-ID: <159672551720.2697252.14371979042246937759@tinycat.chris-lamb.co.uk> Hi all, Please review the draft for July's Reproducible Builds report: https://reproducible-builds.org/reports/2020-07/?draft ??? or, via the Git repository itself: https://salsa.debian.org/reproducible-builds/reproducible-website/blob/master/_reports/2020-07.md I intend to publish it no earlier than: $ date -d 'Sat, 08 Aug 2020 16:00:00 +0100' https://time.is/compare/1600_08_Aug_2020_in_BST ?? Please feel free and commit/push to drafts without the overhead of sending patches or merge requests. You should make your changes to the "_reports/2020-07.md" file in the "reproducible-website" repository: $ git clone https://salsa.debian.org/reproducible-builds/reproducible-website $ cd reproducible-website $ sensible-editor _reports/2020-07.md I am happy to reword and/or rework additions prior to publishing. If you currently do not have access to the above repository, you can request access by following the instructions at: https://reproducible-builds.org/contribute/salsa/ Regards, -- o ??? ??? Chris Lamb o o reproducible-builds.org ???? ??? ??? o From chris at reproducible-builds.org Fri Aug 7 11:26:22 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 07 Aug 2020 12:26:22 +0100 Subject: =?UTF-8?Q?diffoscope_155_released_=F0=9F=92=A0?= Message-ID: <159679936386.3075470.1170305396666456722@tinycat.chris-lamb.co.uk> Hi, The diffoscope maintainers are pleased to announce the release of version 155 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 155 includes the following changes: [ Chris Lamb ] * Bump Python requirement from 3.6 to 3.7 - most distributions are either shipping3.5 or 3.7, so supporting 3.6 is not somewhat unnecessary and also more difficult to test locally. * Improvements to setup.py: - Apply the Black source code reformatter. - Add some URLs for the site of PyPI.org. - Update "author" and author email. * Explicitly support Python 3.8. [ Frazer Clews ] * Move away from the deprecated logger.warn method logger.warning. [ Mattia Rizzolo ] * Document ("classify") on PyPI that this project works with Python 3.8. ## Download Version 155 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From jathanblackred at gmail.com Fri Aug 7 21:10:25 2020 From: jathanblackred at gmail.com (jathan) Date: Fri, 7 Aug 2020 16:10:25 -0500 Subject: To do tasks in Reproducible Builds Message-ID: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> Hi, I was visiting the Reproducible Builds websitesite and the Debian Salsa repo looking for some list of "To do tasks" in the team. Do we have something to view which tasks need to be done with priorities or how do you organise in that way please? Regards Jathan -- Por favor evita enviarme adjuntos en formato de word o powerpoint, si quieres saber porque lee esto: http://www.gnu.org/philosophy/no-word-attachments.es.html ?C?mbiate a GNU/Linux! http://getgnulinux.org/es -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From chris at reproducible-builds.org Sat Aug 8 15:56:33 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Sat, 08 Aug 2020 16:56:33 +0100 Subject: Please review the draft for July's report In-Reply-To: <159672551720.2697252.14371979042246937759@tinycat.chris-lamb.co.uk> References: <159672551720.2697252.14371979042246937759@tinycat.chris-lamb.co.uk> Message-ID: <49160267-ae04-4dfa-9974-48c93666ee57@www.fastmail.com> Chris Lamb wrote: > Please review the draft for July's Reproducible Builds report: This has now been published; many thanks to all who contributed. Please share the following URL: https://reproducible-builds.org/reports/2020-07/ Alternatively, if you are into that kind of thing, please consider retweeting: https://twitter.com/ReproBuilds/status/1292127275064930304 Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From chris at reproducible-builds.org Tue Aug 11 12:02:03 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Tue, 11 Aug 2020 12:02:03 -0000 Subject: Reproducible builds for Debian iso images? In-Reply-To: <87lfim46u7.fsf@ponder> References: <87lfim46u7.fsf@ponder> Message-ID: <5ffbb4f3-c83d-4fa0-b9c8-240234228c82@www.fastmail.com> Vagrant Cascadian wrote: > > Are the Debian install and live iso images deterministically reproducible? > > Unfortunately no. There has been some work in that direction, and it > would be a good thing to improve further! On the installer images, I did a bunch of work on the Debian side and in the various upstream projects that it uses, and I believe they are actually reproducible. However, there are at least four issues until they can be generally advertised as such: First, we are not continually testing them. This is pending on (at least) [0] being merged, and there may be more issues or regressions that have come up since that was written. Second, the official images are not being built in "reproducible mode" and nobody has asked the Debian Installer team to do this yet. This is related to our third problem in that there is no build attestation document for a Debian installer image yet - a loose installer equivalent for a .buildinfo file. We would then need buy-in from the Installer team to add more steps to their release process to additionally validate and promise their builds are reproducible before publishing them, and to make sure the .buildinfo equivalent is signed and published, etc. etc. Live images are actually a significantly different problem space. This is due to what I call the "postinst problem". That is to say, "making a build reproducible" involves making the build system and build scripts deterministic. However, when you build a live image, you are actually running the installation scripts for these packages instead to construct that image -- they are being installed to your virtual .iso file, rather than being built from source. There are many of these scripts in Debian, but the main one is called the "postinst" script, hence my name. I make the distinction because outside of Tails, etc. there has been little to no sustained effort to make these installation scripts deterministic, and many of them are patently non-deterministic. I am therefore less optimistic about the timeline of this, especially given that (a) there has been little interest in my "vanilla" installation image work to begin with and (b) all of the policy work that I outlined above would also be required before Debian could say they had "reproducible live images". [0] https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13 Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From holger at layer-acht.org Wed Aug 12 15:54:28 2020 From: holger at layer-acht.org (Holger Levsen) Date: Wed, 12 Aug 2020 15:54:28 +0000 Subject: To do tasks in Reproducible Builds In-Reply-To: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> References: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> Message-ID: <20200812155428.GA16667@layer-acht.org> Hi Jathan, On Fri, Aug 07, 2020 at 04:10:25PM -0500, jathan wrote: > I was visiting the Reproducible Builds websitesite and the Debian Salsa > repo looking for some list of "To do tasks" in the team. Do we have > something to view which tasks need to be done with priorities or how do > you organise in that way please? did you see https://reproducible-builds.org/contribute/ ? -- cheers, Holger ------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C "... the premise [is] that privacy is about hiding a wrong. It's not. Privacy is an inherent human right, and a requirement for maintaining the human condition with dignity and respect." (Bruce Schneier) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From jathanblackred at gmail.com Wed Aug 12 17:44:58 2020 From: jathanblackred at gmail.com (jathan) Date: Wed, 12 Aug 2020 12:44:58 -0500 Subject: To do tasks in Reproducible Builds In-Reply-To: <20200812155428.GA16667@layer-acht.org> References: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> <20200812155428.GA16667@layer-acht.org> Message-ID: <6a936142-1e94-3e34-4cb1-f570b98eb4ec@gmail.com> On 12/08/2020 10:54, Holger Levsen wrote: > Hi Jathan, > > On Fri, Aug 07, 2020 at 04:10:25PM -0500, jathan wrote: >> I was visiting the Reproducible Builds websitesite and the Debian Salsa >> repo looking for some list of "To do tasks" in the team. Do we have >> something to view which tasks need to be done with priorities or how do >> you organise in that way please? > > did you see https://reproducible-builds.org/contribute/ ? > > Hi Holger! Thanks a lot for your response :) Yes I did. I read that page, but I found it more in the way about how someone can collaborate in RB. I was looking for some real time resource in which you can see tasks to be done, in progress and done, something similar like a Kanban board or some kind of tasks tracking. I will check more detailed anyway the resources available at the contribution page. Thank you for pointing this! Regards Jathan -- Por favor evita enviarme adjuntos en formato de word o powerpoint, si quieres saber porque lee esto: http://www.gnu.org/philosophy/no-word-attachments.es.html ?C?mbiate a GNU/Linux! http://getgnulinux.org/es -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From bernhardout at lsmod.de Thu Aug 13 03:52:16 2020 From: bernhardout at lsmod.de (Bernhard M. Wiedemann) Date: Thu, 13 Aug 2020 05:52:16 +0200 Subject: To do tasks in Reproducible Builds In-Reply-To: <6a936142-1e94-3e34-4cb1-f570b98eb4ec@gmail.com> References: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> <20200812155428.GA16667@layer-acht.org> <6a936142-1e94-3e34-4cb1-f570b98eb4ec@gmail.com> Message-ID: <423fbf98-d9e7-fc79-90da-312e84a34a1b@lsmod.de> On 12/08/2020 19.44, jathan wrote: > On 12/08/2020 10:54, Holger Levsen wrote: >> Hi Jathan, >> >> On Fri, Aug 07, 2020 at 04:10:25PM -0500, jathan wrote: >>> I was visiting the Reproducible Builds websitesite and the Debian Salsa >>> repo looking for some list of "To do tasks" in the team. Do we have >>> something to view which tasks need to be done with priorities or how do >>> you organise in that way please? >> >> did you see https://reproducible-builds.org/contribute/ ? >> >> > Hi Holger! > > Thanks a lot for your response :) Yes I did. I read that page, but I > found it more in the way about how someone can collaborate in RB. I was > looking for some real time resource in which you can see tasks to be > done, in progress and done, something similar like a Kanban board or > some kind of tasks tracking. I will check more detailed anyway the > resources available at the contribution page. Thank you for pointing this! I think, overall it would be good to get some toolchain issues fixed, that affect multiple packages: https://trello.com/c/pHLGpzDQ/39-mono https://trello.com/c/9aSypA7E/71-rust-libgit2 https://trello.com/c/kfKHyItI/70-go-buildid https://trello.com/c/I9voedvB/7-pyc-rb java has plenty. In openSUSE, xmvn is the worst https://bugzilla.opensuse.org/show_bug.cgi?id=1162112 When I last looked through the list of compilers and interpreters, those that built reproducibly were rather few. perl, bash, llvm, ocaml and ruby2.7 are the positive exceptions now. http://ismypackagereproducibleyet.org/?pkg=perl Maybe we can also find more/better datasources for this tool? There are also pretty hard challenges, e.g. when you look at how clisp and emacs create their binaries through memory dumps. Or when you want to make gcc build reproducibly even with profile-guided-optimizations and its huge profile run that is a whole gcc build. When I talks to our (SUSE) compiler guys, they said, there might be some gcc patches possible to make profiling react less on variations in ordering, so that running independent A,B would yield the same as B,A. Ciao Bernhard M. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: OpenPGP digital signature URL: From jathanblackred at gmail.com Thu Aug 13 22:57:39 2020 From: jathanblackred at gmail.com (jathan) Date: Thu, 13 Aug 2020 17:57:39 -0500 Subject: To do tasks in Reproducible Builds In-Reply-To: <423fbf98-d9e7-fc79-90da-312e84a34a1b@lsmod.de> References: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> <20200812155428.GA16667@layer-acht.org> <6a936142-1e94-3e34-4cb1-f570b98eb4ec@gmail.com> <423fbf98-d9e7-fc79-90da-312e84a34a1b@lsmod.de> Message-ID: <426d2f6c-8639-9d89-89d9-894f25757b6e@gmail.com> On 12/08/2020 22:52, Bernhard M. Wiedemann wrote: > > > On 12/08/2020 19.44, jathan wrote: >> On 12/08/2020 10:54, Holger Levsen wrote: >>> Hi Jathan, >>> >>> On Fri, Aug 07, 2020 at 04:10:25PM -0500, jathan wrote: >>>> I was visiting the Reproducible Builds websitesite and the Debian Salsa >>>> repo looking for some list of "To do tasks" in the team. Do we have >>>> something to view which tasks need to be done with priorities or how do >>>> you organise in that way please? >>> >>> did you see https://reproducible-builds.org/contribute/ ? >>> >>> >> Hi Holger! >> >> Thanks a lot for your response :) Yes I did. I read that page, but I >> found it more in the way about how someone can collaborate in RB. I was >> looking for some real time resource in which you can see tasks to be >> done, in progress and done, something similar like a Kanban board or >> some kind of tasks tracking. I will check more detailed anyway the >> resources available at the contribution page. Thank you for pointing this! > > I think, overall it would be good to get some toolchain issues fixed, > that affect multiple packages: > > https://trello.com/c/pHLGpzDQ/39-mono > https://trello.com/c/9aSypA7E/71-rust-libgit2 > https://trello.com/c/kfKHyItI/70-go-buildid > https://trello.com/c/I9voedvB/7-pyc-rb > > java has plenty. In openSUSE, xmvn is the worst > https://bugzilla.opensuse.org/show_bug.cgi?id=1162112 > > When I last looked through the list of compilers and interpreters, those > that built reproducibly were rather few. perl, bash, llvm, ocaml and > ruby2.7 are the positive exceptions now. > http://ismypackagereproducibleyet.org/?pkg=perl > Maybe we can also find more/better datasources for this tool? > > > There are also pretty hard challenges, e.g. when you look at how clisp > and emacs create their binaries through memory dumps. > > Or when you want to make gcc build reproducibly even with > profile-guided-optimizations and its huge profile run that is a whole > gcc build. > When I talks to our (SUSE) compiler guys, they said, there might be some > gcc patches possible to make profiling react less on variations in > ordering, so that running independent A,B would yield the same as B,A. > > > Ciao > Bernhard M. > Hi Bernhard, Thank you for sharing your personal Trello board. It is very useful! Can people help in those tasks being of yours? Maybe I can mention you Kanban Board besides the current collaboration and documentation pages of the Reproducible Builds website in a presentation I will share in Spanish. Tell me if is possible to do it or not to respect your decision. By the way. Is this tool mentioned somewhere in the RB website? http://ismypackagereproducibleyet.org/?pkg=lxde&query=query I was looking if it is listed at the Reproducible Builds tools page https://reproducible-builds.org/tools/ but I did not find it. I have visited the ismypackagereproducibleyet Git Repo http://ismypackagereproducibleyet.org/?pkg=lxde&query=query and the project looks nice, Regards Jathan -- Por favor evita enviarme adjuntos en formato de word o powerpoint, si quieres saber porque lee esto: http://www.gnu.org/philosophy/no-word-attachments.es.html ?C?mbiate a GNU/Linux! http://getgnulinux.org/es -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From chris at reproducible-builds.org Fri Aug 14 09:26:17 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 14 Aug 2020 10:26:17 +0100 Subject: =?UTF-8?Q?diffoscope_156_released_=F0=9F=92=A0?= Message-ID: <159739620646.1485653.17544171933324081357@tinycat.chris-lamb.co.uk> Hi, The diffoscope maintainers are pleased to announce the release of version 156 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 156 includes the following changes: [ Chris Lamb ] * Update PPU tests for compatibility with Free Pascal versions 3.2.0 or greater. (Closes: #968124) * Emit a debug-level logging message when our ppudump(1) version does not match file header. * Add and use an assert_diff helper that loads and compares a fixture output to avoid a bunch of test boilerplate. [ Frazer Clews ] * Apply some pylint suggestions to the codebase. ## Download Version 156 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From gnu at toad.com Sat Aug 15 10:57:13 2020 From: gnu at toad.com (John Gilmore) Date: Sat, 15 Aug 2020 03:57:13 -0700 Subject: To do tasks in Reproducible Builds: emacs In-Reply-To: <426d2f6c-8639-9d89-89d9-894f25757b6e@gmail.com> References: <12f81fbf-36fc-5e33-48b4-5fa9ae79d3c3@gmail.com> <20200812155428.GA16667@layer-acht.org> <6a936142-1e94-3e34-4cb1-f570b98eb4ec@gmail.com> <423fbf98-d9e7-fc79-90da-312e84a34a1b@lsmod.de> <426d2f6c-8639-9d89-89d9-894f25757b6e@gmail.com> Message-ID: <16669.1597489033@hop.toad.com> >> There are also pretty hard challenges, e.g. when you look at how clisp >> and emacs create their binaries through memory dumps. The recent Emacs release announced: Emacs 27.1 has a wide variety of new features, including: ... - Portable dumping used instead of unexec So, the job of making Emacs reproducible may have just gotten much easier! John From mattia at mapreri.org Sun Aug 16 13:03:31 2020 From: mattia at mapreri.org (Mattia Rizzolo) Date: Sun, 16 Aug 2020 15:03:31 +0200 Subject: Shall we partecipate in Outreachy Winter 2020? Message-ID: <20200816130331.GY1281559@mapreri.org> Hello everybody, the next Outreachy season is coming, and as so I'd like to hear your opinions on wether we should partecipate to it. In particular, I'm looking for good proposals that might not be the usual ones we put up some years ago when we last partecipated (see https://wiki.debian.org/Outreachy/Round15/Projects/ReproducibleBuildsOfDebian for what we did). Looking forward for your ideas! -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. More about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `- -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From chris at reproducible-builds.org Mon Aug 17 16:58:03 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Mon, 17 Aug 2020 16:58:03 -0000 Subject: Shall we partecipate in Outreachy Winter 2020? In-Reply-To: <20200816130331.GY1281559@mapreri.org> References: <20200816130331.GY1281559@mapreri.org> Message-ID: Mattia Rizzolo wrote; > the next Outreachy season is coming, and as so I'd like to hear your > opinions on whether we should participate in it. > > In particular, I'm looking for good proposals that might not be the > usual ones we put up some years ago when we last participated (see > https://wiki.debian.org/Outreachy/Round15/Projects/ReproducibleBuildsOfDebian > for what we did). If it helps, we are looking for (for example) workflow changes, large refactoring work, new features of our tools, specific reproducibility fixes, etc. etc. But crucially they should fit in that sweet spot of (a) requiring more time and energy than a weekend project, but (b) are also not too complicated that they would take forever and that (c) they are actually possible to 'complete' in a satisfactory manner. If they could (d) benefit a lot of people and have a big impact, that would naturally be beneficial over a potential 'personal' need? :) Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From lamby at debian.org Wed Aug 19 16:52:03 2020 From: lamby at debian.org (Chris Lamb) Date: Wed, 19 Aug 2020 16:52:03 -0000 Subject: =?UTF-8?Q?Re:_[rb-general]__Bug#926242:_jenkins.debian.org:_Please_test_?= =?UTF-8?Q?reproducibility_status_of_Debian_Installer_images?= In-Reply-To: References: <87zhnb5mb0.fsf@yucca> <20190525113041.GB3829@riva.ucam.org> <20190526132440.GD3829@riva.ucam.org> <4c8c2965-d710-4c56-b523-8558b86ec491@www.fastmail.com> <8736knack8.fsf@hands.com> <87zhmv8xdv.fsf@hands.com> <87cf3812-3145-457d-94d1-da03b09829cd@www.fastmail.com> <52099112-fbe8-4667-88fe-c292e8fbcbf8@www.fastmail.com> <20190709054020.2pyrylesv4k2z36j@mraw.org> <139712fc-7ddd-4bbf-8263-37317d909f8e@www.fastmail.com> Message-ID: Chris Lamb wrote: > > > My current plan is (1) breathing a little, (2) getting the needed > > > bugfixes into 10.1. > > > > Whoops, I'm afraid I totally neglected to followup on this so I > > apologise this got stalled. Anyway, anything I can do to help? > > I've made an initial step of taking my patch from: > > https://bugs.debian.org/926242#127 > > ? and submitting it as a MR on salsa here: > > https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/13 May I make a gentle request to get this MR merged? It's been open for about 5 months now, only affects the build system and is only to handle cases where we have the stranger [foo=bar] arguments in sources.list(5) entries, which is unlikely to be the case for any official builds. As I write in my latest comment on the MR, it is not *strictly* blocking testing whether d-i images are reproducible, but it is making it really rather difficult -- I'm using awful 140-line local shell script, rather using our far-superior testing framework, and we have likely been accumulating regressions since last time I was seriously working on this. Regards, -- ,''`. : :' : Chris Lamb `. `'` lamby at debian.org ? chris-lamb.co.uk `- From chris at reproducible-builds.org Fri Aug 21 11:43:51 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 21 Aug 2020 12:43:51 +0100 Subject: =?UTF-8?Q?diffoscope_157_released_=F0=9F=92=A0?= Message-ID: <159800974453.3742527.1755814644591601882@tinycat.chris-lamb.co.uk> Hi, The diffoscope maintainers are pleased to announce the release of version 157 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 157 includes the following changes: [ Chris Lamb ] * Try obsensibly "data" files named .pgp against pgpdump to determine whether they are PGP files. (Closes: reproducible-builds/diffoscope#211) * Don't raise an exception when we encounter XML files with "" declarations inside the DTD, or when a DTD or entity references an external resource. (Closes: reproducible-builds/diffoscope#212) * Temporarily drop gnumeric from Build-Depends as it has been removed from testing due to Python 2.x deprecation. (Closes: #968742) * Codebase changes: - Add support for multiple file extension matching; we previously supported only a single extension to match. - Move generation of debian/tests/control.tmp to an external script. - Move to our assert_diff helper entirely in the PGP tests. - Drop some unnecessary control flow, unnecessary dictionary comprehensions and some unused imports found via pylint. * Include the filename in the "... not identified by any comparator" logging message. ## Download Version 157 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From chris at reproducible-builds.org Fri Aug 28 11:30:57 2020 From: chris at reproducible-builds.org (Chris Lamb) Date: Fri, 28 Aug 2020 12:30:57 +0100 Subject: =?UTF-8?Q?diffoscope_158_released_=F0=9F=92=A0?= Message-ID: <159861309109.2354673.13517008570627759187@tinycat.chris-lamb.co.uk> Hi, The diffoscope maintainers are pleased to announce the release of version 158 of diffoscope. diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. Version 158 includes the following changes: * Improve PGP support: - Support extracting of files within PGP signed data. (Closes: reproducible-builds/diffoscope#214) - pgpdump(1) can successfully parse some unrelated, non-PGP binary files, so check that the parsed output contains something remotely sensible before identifying it as a PGP file. * Don't use Python's repr(...)-style output in "Calling external command" logging output. * Correct a typo of "output" in an internal comment. ## Download Version 158 is available from Debian unstable as well as PyPI, and will shortly be available on other platforms surely. More details can be found here: https://diffoscope.org/ ? but source tarballs may be located here: https://diffoscope.org/archive/ The corresponding Docker image may be run via (for example): $ docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \ registry.salsa.debian.org/reproducible-builds/diffoscope a b ## Contribute diffoscope is developed within the "Reproducible builds" effort. - Git repository https://salsa.debian.org/reproducible-builds/diffoscope - Docker image, eg. registry.salsa.debian.org/reproducible-builds/diffoscope https://salsa.debian.org/reproducible-builds/diffoscope - Issues and feature requests https://salsa.debian.org/reproducible-builds/diffoscope/issues - Contribution instructions (eg. to file an issue) https://reproducible-builds.org/contribute/salsa/ Regards, -- o ? ? Chris Lamb o o reproducible-builds.org ? ? ? o From lamby at debian.org Mon Aug 31 12:02:03 2020 From: lamby at debian.org (Chris Lamb) Date: Mon, 31 Aug 2020 12:02:03 -0000 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: <20200829013827.GA746225@thunder.hadrons.org> References: <20200829013827.GA746225@thunder.hadrons.org> Message-ID: [adding rb-general to CC] Hi Guillem, > Holger proposed to bundle the .buildinfo files into .deb archives > during the DebConf talk. I've mentioned to Holger that I'm not seeing > this as being feasible and mentioned various reasons why, but I'm also > still open to explore this possibility. So I've added these in a wiki > page: > > The majority Debian's documentation is either littered around the internet, in obscure mailing list posts, in IRC backlogs or simply in people's minds. This kind of document pushes back against this organisational antipattern, so thank you. With regards to your question, I do not believe you are missing anything here, except perhaps to clarify exactly which .debs you would put the .buildinfo into. I assume you mean all of the binary .debs (noting your later caveat regarding .udebs), but it might be worth being specific for clarity. In terms of my own opinion, you remark that: this would make a simple file comparison [..] require some kind of tool This does indeed go against one of the stated original design principles as well as the unstated ?sthetic ones that I hold personally. I have also empirically observed that the platforms that adopt a "oh, you just need this small tool" approach do not appear to gain as much traction too. Now, I cannot back this up scientifically, but I don't believe this is purely for technical reasons but also cognitive ones. As in, there is something deeply psychologically reassuring and satisfying to humans when a reproducible artefact can be seen to be identical using just our "eyes" and without any tools whatsoever. I might completely trust some tool technically and even trust it from a security perspective (!) yet it somehow does not feel nearly as "secure", right or intuitive. (As an obiter dictum, are we sure it was Holger who was proposing this idea in the talk, rather than mentioning it? I think he has previously echoed my view on the "no special tools" principle, hence this minor remark. Am willing to be corrected either way.) Regards, -- ,''`. : :' : Chris Lamb `. `'` lamby at debian.org ? chris-lamb.co.uk `- From holger at layer-acht.org Mon Aug 31 12:39:07 2020 From: holger at layer-acht.org (Holger Levsen) Date: Mon, 31 Aug 2020 12:39:07 +0000 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: References: <20200829013827.GA746225@thunder.hadrons.org> Message-ID: <20200831123907.GA26694@layer-acht.org> Dear Chris, On Mon, Aug 31, 2020 at 12:02:03PM -0000, Chris Lamb wrote: > (As an obiter dictum, are we sure it was Holger who was proposing this > idea in the talk, rather than mentioning it? I think he has previously > echoed my view on the "no special tools" principle, hence this minor > remark. Am willing to be corrected either way.) yes, it was me who proposed it (watch my dc20 talk! :) and who still thinks it's a good idea. sadly I didn't have the time to start the discussion in a bug (I only came to this conclusion the day before the talk, though I have thought about this for the last 5 years) and I probably still won't have the time until next week. (*) I'd appreciate we'd use a bug for discussing this, so whatever the outcome will be, we'll have a canonical and truely long living url to reference the discussion. thanks. (*) so the blame for not discussing this in a bug right now in the first place, goes to me. hooray. similarily, I don't think an internal Debian discussion immediatly belongs on the rb-general list... I do believe we have these two distict lists for a reason :) also, I will not share my thoughts about Guillem's and Chris' reply here (and *now*), before I had the opportunity to put the reasoning behind my thoughts in a bug report. And I'd hope my thoughts why are laid out clearly in my talk available at https://meetings-archive.debian.net/pub/debian-meetings/2020/DebConf20/49-reproducing-bullseye-in-practice.webm finally, I'm sorry if I come accross harsh. I feel pressured and misunder- stood and that I need to react now. I wish I felt different. -- cheers, Holger ------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C That morning, the young barista woman told me that a customer came in with a mask, but not wearing it. When she asked the customer to put on her mask please, the woman said: "Why? There's no-one in here." -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From guillem at debian.org Mon Aug 31 13:03:55 2020 From: guillem at debian.org (Guillem Jover) Date: Mon, 31 Aug 2020 15:03:55 +0200 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: <20200831123907.GA26694@layer-acht.org> References: <20200829013827.GA746225@thunder.hadrons.org> <20200831123907.GA26694@layer-acht.org> Message-ID: <20200831130355.GA864325@thunder.hadrons.org> Hi! On Mon, 2020-08-31 at 12:39:07 +0000, Holger Levsen wrote: > On Mon, Aug 31, 2020 at 12:02:03PM -0000, Chris Lamb wrote: > > (As an obiter dictum, are we sure it was Holger who was proposing this > > idea in the talk, rather than mentioning it? I think he has previously > > echoed my view on the "no special tools" principle, hence this minor > > remark. Am willing to be corrected either way.) > > yes, it was me who proposed it (watch my dc20 talk! :) and who still > thinks it's a good idea. sadly I didn't have the time to start the > discussion in a bug (I only came to this conclusion the day before the > talk, though I have thought about this for the last 5 years) and I > probably still won't have the time until next week. (*) > > I'd appreciate we'd use a bug for discussing this, so whatever the outcome > will be, we'll have a canonical and truely long living url to reference the > discussion. As I hinted (but should probably have been more clear, sorry about that) on our private mail exchange, a bug report seemed premature to me, given that it's really not clear (to me at least) this is the way to go. I tend to find bug reports not a very good medium for broad design work TBH, and they end up not being very visible once they are closed, so need to be referenced from other places, such as a wiki. :) As a summary of a concluded spec to be implemented sure, but otherwise (at least for dpkg) they feel more like clutter than anything else. If you insist on opening a bug, then I'll go along, as closing would seem inappropriate though, but meh. :D > also, I will not share my thoughts about Guillem's and Chris' reply > here (and *now*), before I had the opportunity to put the reasoning > behind my thoughts in a bug report. And I'd hope my thoughts why are laid > out clearly in my talk available at I think the reasoning is clear, but perhaps I didn't capture it correctly in the wiki page, but the problem I'm seeing is in the implications of the (current) proposal. As I mention in there perhaps there are other ways to accomplish a similar thing but I'm not seeing either how those alternatives could unstuck the current infra deployment problem TBH. > https://meetings-archive.debian.net/pub/debian-meetings/2020/DebConf20/49-reproducing-bullseye-in-practice.webm > > finally, I'm sorry if I come accross harsh. I feel pressured and misunder- > stood and that I need to react now. I wish I felt different. Oh! Hmm I didn't mean this as pressure, I thought you were actually eager to get this discussed publicly, so I went ahead and published what I understood the proposal was, which perhaps I've not captured correctly either. I'm happy to sit on this for whatever time you need, personally I see no hurry myself. :) Thanks, Guillem From kpcyrd at rxv.cc Mon Aug 31 18:05:41 2020 From: kpcyrd at rxv.cc (kpcyrd) Date: Mon, 31 Aug 2020 18:05:41 +0000 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: <20200831130355.GA864325@thunder.hadrons.org> References: <20200829013827.GA746225@thunder.hadrons.org> <20200831123907.GA26694@layer-acht.org> <20200831130355.GA864325@thunder.hadrons.org> Message-ID: <20200831175512.GQ3496@localhost> I'm a bit short on time, sorry in advance if the email is a little short/blunt: - What was the original motivation of putting the size and checksum of the package into the buildinfo file? We aren't tracking this info in Arch Linux and it turned out we didn't need those fields to implement a rebuilder. Please consider simply dropping those fields instead of trying to build a tool to work around this. In Arch Linux we consider the buildinfo file a build parameter to ensure the build environment is always identical, but strictly speaking it's not a build output (even though it's generated during the package build, but you can generate it without actually running the whole build). Having access to the build outputs is not necessary and out of scope of "recording the build environment". In my opinion everything in the buildinfo file that goes beyond "a collection of parameters for the build" is feature-creep at the cost of complexity. This also solves the .changes problem (if I understood it correctly). The buildinfo file is available very early (as long as you stop referencing build outputs) and you can simply include it when creating the deb in the first place instead of manipulating it afterwards. - The current debian reproducible builds effort is very focused on debian.org, but virtually none of that can be downstreamed by debian derivates. Having externally hosted buildinfo files is an effort that every downstream would need to repeat and every rebuilder need to know about. All Arch Linux downstreams I've checked ship buildinfo files, while zero debian downstreams do. This is an advantage that's currently not mentioned yet. - The "having the buildinfo file in the binary package is wasteful" argument is a micro optimization that pushes a non-trivial amount of extra complexity on the debian r-b developers. Considering that debian rebuilder tooling is still very sparse due to the lack of developer resources I'm not sure that's a smart trade-off. - I don't understand the concern about source-only uploads. The uploader can't know the build environment that buildd is going to setup, therefore the buildinfo file needs to be generated by buildd anyway. Sorry for being rather Arch centric in this email, but I think it's a good idea to ensure you're familiar with how other distros solved the problem that debian is facing since a few years. From vagrant at reproducible-builds.org Mon Aug 31 20:18:53 2020 From: vagrant at reproducible-builds.org (Vagrant Cascadian) Date: Mon, 31 Aug 2020 13:18:53 -0700 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: <20200831175512.GQ3496@localhost> References: <20200829013827.GA746225@thunder.hadrons.org> <20200831123907.GA26694@layer-acht.org> <20200831130355.GA864325@thunder.hadrons.org> <20200831175512.GQ3496@localhost> Message-ID: <87a6yash5e.fsf@ponder> On 2020-08-31, kpcyrd wrote: > I'm a bit short on time, sorry in advance if the email is a little short/blunt: > > - What was the original motivation of putting the size and checksum of the > package into the buildinfo file? We aren't tracking this info in Arch Linux > and it turned out we didn't need those fields to implement a rebuilder. > Please consider simply dropping those fields instead of trying to build a > tool to work around this. It's an attestation of of "with source X and build environment Y I produced artifacts Z". So I guess you're saying if the .buildinfo is embedded in the binary package, you can get the hashes of the artifacts from the binaries themselves as, obviously, you already have the binaries. > In Arch Linux we consider the buildinfo file a build parameter to ensure the > build environment is always identical, but strictly speaking it's not a build > output (even though it's generated during the package build, but you can > generate it without actually running the whole build). Having access to the build > outputs is not necessary and out of scope of "recording the build > environment". In my opinion everything in the buildinfo file that goes beyond > "a collection of parameters for the build" is feature-creep at the cost of > complexity. > > This also solves the .changes problem (if I understood it correctly). The > buildinfo file is available very early (as long as you stop referencing build > outputs) and you can simply include it when creating the deb in the first > place instead of manipulating it afterwards. I'm definitely coming around to this idea. > - The current debian reproducible builds effort is very focused on debian.org, > but virtually none of that can be downstreamed by debian derivates. Having > externally hosted buildinfo files is an effort that every downstream would > need to repeat and every rebuilder need to know about. All Arch Linux > downstreams I've checked ship buildinfo files, while zero debian downstreams > do. This is an advantage that's currently not mentioned yet. Nice point! > - The "having the buildinfo file in the binary package is wasteful" argument is > a micro optimization that pushes a non-trivial amount of extra complexity on > the debian r-b developers. Considering that debian rebuilder tooling is still > very sparse due to the lack of developer resources I'm not sure that's a > smart trade-off. It is largely very compressable, though with source packages that produce multiple binaries, it's a bit more resources; shipping in each binary instead of once per source package. If you then extract the .buildinfo when installing the package is another resource question; Arch Linux does not, it might make it easier to get that information out of the installed system than re-fetch it. Might not be worth it, though. That said, embedding the .buildinfo seems like a much more workable approach to a problem we've had for so long... > - I don't understand the concern about source-only uploads. The uploader can't > know the build environment that buildd is going to setup, therefore the > buildinfo file needs to be generated by buildd anyway. The .buildinfo is also generated by the buildd. This is part of the "multiple signers" property that .buildinfo files were attempting to address. For the initial upload, you typically have one .buildinfo from the uploader, and one from each of the architectures it was built on. The buildd on the relevent architecture in this case becomes a (somewhat sloppy) rebuilder. If in fact the .buildinfo is embedded in the binary package, and the .buildinfo itself is made effectively reproducible, it's been pointed out to me on IRC that you could then just sign the binary packge itself. Even if we do manage to start embedding .buildinfo information inside the .deb files, we could still produce external .buildinfo files with checksums just as is done now, though minor differences in build environment will always result in differing builds then even if the packages are otherwise reproducible... > Sorry for being rather Arch centric in this email, but I think it's a good idea > to ensure you're familiar with how other distros solved the problem that > debian is facing since a few years. Thank you very much! live well, vagrant -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 227 bytes Desc: not available URL: From lamby at debian.org Mon Aug 31 22:58:03 2020 From: lamby at debian.org (Chris Lamb) Date: Mon, 31 Aug 2020 22:58:03 -0000 Subject: Evaluation of bundling .buildinfo in .deb proposal In-Reply-To: <20200831123907.GA26694@layer-acht.org> References: <20200829013827.GA746225@thunder.hadrons.org> <20200831123907.GA26694@layer-acht.org> Message-ID: Hi Holger, > > (As an obiter dictum, are we sure it was Holger who was proposing this > > idea in the talk, rather than mentioning it? I think he has previously > > echoed my view on the "no special tools" principle, hence this minor > > remark. Am willing to be corrected either way.) > > yes, it was me who proposed it (watch my dc20 talk! :) and who still > thinks it's a good idea. Ah, I am clearly misremembering both parts. I must have incorrectly connected you mentioning the idea with you repeating it from someone else. (I was very much present at your talk although I admit I did not rewatch it in order to write my message earlier today.) I am also clearly misremembering a discussion on the actual merits of the idea from our summit meetings. But it is of no real consequence, and you would surely be 'allowed' to change your mind in any case. :) > finally, I'm sorry if I come across harsh. I feel pressured and misunder- > stood and that I need to react now. I wish I felt different. Oof, I am sorry to hear that. It will likely be of no real help to you, but like Guillem implied in his own email, this was not my intention. In fact, it was quite the opposite ? I was trying to ensure that your views were accurately represented. I suspect I share much of the frustration that underlies this. Regards, -- ,''`. : :' : Chris Lamb `. `'` lamby at debian.org ? chris-lamb.co.uk `-