reproducible .pyc files (& python-for-android)

[Felix C. Stegerman]

Hi Bernhard (& Chris),

* "Bernhard M. Wiedemann" <bernhardout at lsmod.de> [2021-01-04 12:48]:
> Am 04.01.21 um 11:23 schrieb Chris Lamb:
> >> p4a compiles those with "hostpython -OO -m compileall -b -f" (where
> >> hostpython is the cross-compiled Python for the target -- arm64-v8a or
> >> armeabi-v7a -- which is thus definitely the same version on both
> >> machines).
> >
> > As I understand it, recent versions Python can use SOURCE_DATE_EPOCH
> > in its internal py_compile routine to ensure that .pyc files are
> > reproducible.
>
> This is not a timestamp issue, though. If those are varying, they are in
> the header (first 12 bytes) of the .pyc.

Indeed.  I should perhaps have mentioned that I am indeed setting
SOURCE_DATE_EPOCH (though that is noted in the PR I [1] linked).  For
reference: the Python version is 3.9.1.

> │  000000f0: 6d5a 0d62 6469 7374 5f77 696e 696e 7374  mZ.bdist_wininst
> │ -00000100: 5a05 6368 6563 6b5a 0675 706c 6f61 644e  Z.checkZ.uploadN
> │ +00000100: da05 6368 6563 6b5a 0675 706c 6f61 644e  ..checkZ.uploadN
>
> I have seen this before and remember something about python string
> reference counters being dumped into these pickle files and that varied
> from ordering, so that
> py_compile py1.py py2.py
> produced different results than
> py_compile py2.py py1.py

That's interesting.  But p4a is running "hostpython -OO -m compileall
-b -f $DIRECTORY" -- and compileall walks the directory tree in sorted
order -- so I don't think that's the problem here.

I have currently sidestepped the issue by using a Debian buster docker
container (with usrmerge installed) in GitHub Actions (instead of
the default Ubuntu image).  I now get identical .apks there and on a
local buster VM (with the same build path) \o/

If anyone is interested in the modifications I had to make to p4a to
get it to build reproducibly: see the PR [1].  It also has comments on
what does and does not (yet) work.

Thanks.

- Felix

[1] https://github.com/kivy/python-for-android/pull/2390

P.S. @Chris as a longtime Debian sid user I would also consider that
my 'home' distribution :)