reproducible .pyc files (& python-for-android)

[Frederik Rietdijk]

Recently I spent some time again as well on making the Python interpreters
in Nixpkgs build reproducibly. The following Nix expression results in
deterministic builds of the 3.x interpreters we have. Search for
`determinis` and you will see the changes we do to get there. Maybe it's of
help to you.

https://github.com/NixOS/nixpkgs/blob/f19ad43d674e5bbaa70a000f65031ab09a32a338/pkgs/development/interpreters/python/cpython/default.nix

On Mon, Jan 4, 2021 at 2:41 PM Felix C. Stegerman <flx at obfusk.net> wrote:

> Hi Bernhard (& Chris),
>
> * "Bernhard M. Wiedemann" <bernhardout at lsmod.de> [2021-01-04 12:48]:
> > Am 04.01.21 um 11:23 schrieb Chris Lamb:
> > >> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> > >> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> > >> armeabi-v7a -- which is thus definitely the same version on both
> > >> machines).
> > >
> > > As I understand it, recent versions Python can use SOURCE_DATE_EPOCH
> > > in its internal py_compile routine to ensure that .pyc files are
> > > reproducible.
> >
> > This is not a timestamp issue, though. If those are varying, they are in
> > the header (first 12 bytes) of the .pyc.
>
> Indeed.  I should perhaps have mentioned that I am indeed setting
> SOURCE_DATE_EPOCH (though that is noted in the PR I [1] linked).  For
> reference: the Python version is 3.9.1.
>
> > │  000000f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> > │ -00000100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> > │ +00000100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
> >
> > I have seen this before and remember something about python string
> > reference counters being dumped into these pickle files and that varied
> > from ordering, so that
> > py_compile py1.py py2.py
> > produced different results than
> > py_compile py2.py py1.py
>
> That's interesting.  But p4a is running "hostpython -OO -m compileall
> -b -f $DIRECTORY" -- and compileall walks the directory tree in sorted
> order -- so I don't think that's the problem here.
>
> I have currently sidestepped the issue by using a Debian buster docker
> container (with usrmerge installed) in GitHub Actions (instead of
> the default Ubuntu image).  I now get identical .apks there and on a
> local buster VM (with the same build path) \o/
>
> If anyone is interested in the modifications I had to make to p4a to
> get it to build reproducibly: see the PR [1].  It also has comments on
> what does and does not (yet) work.
>
> Thanks.
>
> - Felix
>
> [1] https://github.com/kivy/python-for-android/pull/2390
>
> P.S. @Chris as a longtime Debian sid user I would also consider that
> my 'home' distribution :)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20210104/664abdbe/attachment.htm>