<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<font size="4">Dear all,<br>
<br>
Thanks for your inputs. Many thanks Eli for your great
substantiated explanation.<br>
<br>
Since none of this is publicly documented at Github (AFAIK), the
idea is to use Eli's <a moz-do-not-send="true"
href="https://lists.reproducible-builds.org/pipermail/rb-general/2021-October/002422.html">email</a>
as a reference URL about the topic.<br>
<br>
Best regards,<br>
<br>
--Martin<br>
<br>
</font><br>
<div class="moz-cite-prefix">On 10/24/21 03:42, Eli Schwartz wrote:<br>
</div>
<blockquote type="cite"
cite="mid:51e49ac1-f081-f019-048c-2be576e5d058@archlinux.org">
<pre class="moz-quote-pre" wrap="">On 10/23/21 5:51 AM, Martin Monperrus wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Dear all,
FYI, Github's autogenerated release tarballs are not deterministic (see
discussion on keybase <a class="moz-txt-link-rfc2396E" href="https://github.com/keybase/client/issues/10800"><https://github.com/keybase/client/issues/10800></a>,
and Bitcoin-core release warning
<a class="moz-txt-link-rfc2396E" href="https://github.com/bitcoin/bitcoin/releases/tag/v22.0"><https://github.com/bitcoin/bitcoin/releases/tag/v22.0></a>).
Does anybody have good connections at Github to get this fixed?
Best regards,
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
I see a bunch of assertions that were counter-asserted. As several
people in this thread have stated, the output of the `git archive`
command is specifically designed to be reproducible these days.
Github specifically uses `git archive`, and that has meaningful
ramifications people depend on, such as respecting .gitattributes (e.g.
export-ignore), and means that
gzip -dc <archive-file>.tar.gz | git get-tar-commit-id
will print the tar commit id, per the git-archive documentation.
A long time ago, Github's archives failed to be deterministic because
git-archive was not deterministic. Since then, people periodically panic
that it still isn't, but no one has provided credible proof that I can
recall.
I do not count cases that were successfully proven to have been upstream
developers force pushing and overwriting a public tag because they
wanted to revise history.
...
Note also that GNU gzip's output algorithm is stable and deterministic.
Other compression algorithms are not necessarily so... zstd for example
documents that zstd compression is always deterministic *iff* you use
the same version of zstd, which does no good for code hosting sites
relying on git-archive... but then again, zstd is also a format under
active development.
Note that it does depend on which *implementation* of gzip you use. For
example, most people use GNU gzip (I believe github always has), but
busybox gzip ***used to*** produce different output.
This was actually fixed by Daniel Edgecumbe (after I encouraged him to
do so) and mentioned in the rb-general status update at:
<a class="moz-txt-link-freetext" href="https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001647.html">https://lists.reproducible-builds.org/pipermail/rb-general/2019-September/001647.html</a>
The patches were applied in
<a class="moz-txt-link-freetext" href="https://git.busybox.net/busybox/commit/?id=c660cc1b7714fffbac95c9378ff4b73de650a6de">https://git.busybox.net/busybox/commit/?id=c660cc1b7714fffbac95c9378ff4b73de650a6de</a>
and busybox v1.32.0 produces the same output as GNU gzip (and is
expected to remain stable). This is relevant to anyone who actually runs
a software forge using busybox and/or Alpine Linux, which I suspect lies
drastically out of the Github interest zone.
It specifically was relevant to sourcehut (<a class="moz-txt-link-freetext" href="https://git.sr.ht">https://git.sr.ht</a>) and was in
fact a cause of non-reproducibility on that software forge... which is
now fixed.
And, again, this literally just means you're dependent on the stability
of the third-party compressor as your single point of failure (other
than possibly using absolutely, anciently, decrepit versions of git from
before git-archive was specifically modeled to be reproducible).
If anyone can actually point to a real life case in the last 4 years
where github autogenerated tarballs have actually changed, without
changing the contents (that is to say, the union of unpacked files,
leading directory prefix, and git-get-tar-commit-id header) I would
actually love to hear about that. I've asked people before, and I
believe I've always gotten one of 4 responses:
- "I heard that it can happen"
- "it happened before famous-date which is specifically acknowledged as
the cutoff point for git-archive itself"
- "it happened right here" followed by verification that that specific
case was a force pushed tag change
- "it happened right here" followed by verification that that specific
case was a repository rename, which resulted in the extracted
directory itself changing (so tar -xf ... && cd foo-1.0 became
instead tar -xf ... && cd bar-1.0)
Given the relative popularity of people *saying* it can happen, surely
there must be some evidence... somewhere... that this is still a problem?
Meanwhile, it has remained true for years, that I can reproduce exact
checksums of a github release using the following command line:
$ git config --get alias.github-release
!f() { local repo=$(basename "$(pwd)") tag=$1; git archive
--prefix=${repo}-${tag#v}/ -o ${repo}-${tag#v}.tar.gz ${tag}; }; f
$ git github-release <tag>
$ git github-release <sha1>
Maybe someone can point at git-archive itself modifying the format of
its output stream? Since that really is what it all boils down to, I
would imagine.
</pre>
</blockquote>
<br>
</body>
</html>