[rb-general] Bug#917407: python3.7: different pyc depending on how python3.7 is installed

[Johannes 'josch' Schauer]

Source: python3.7
Severity: minor
User: reproducible-builds at lists.alioth.debian.org
Usertags: toolchain

Hi,

I noticed that the pyc files generated by Python 3.7 (but not by earlier
Python versions) is different depending on how the package is installed.
I use __init__.cpython-37.pyc as an example but other .pyc files also
differ.

With debootstrap --include:

$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 debootstrap --variant=minbase --include=python3 unstable debian-unstable
$ md5sum debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc
df6fe61fe176e4858ce2062233d2280e  debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc

Installing with apt after a minbase debootstrap:

$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 debootstrap --variant=minbase unstable debian-unstable
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable apt-get install python3
$ md5sum debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc
2c6fdc51b035428a3881f3eef70e3a5b  debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc

Manually installing with dpkg (in an order that respects Pre-Depends):

$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 debootstrap --variant=minbase unstable debian-unstable
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable apt install bzip2 file libexpat1 libmagic-mgc libmagic1 libmpdec2 libreadline7 libsqlite3-0 libssl1.1 mime-support readline-common xz-utils
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable apt-get install --download-only python3
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable sh -c "cd /var/cache/apt/archives; dpkg -i libpython3.7-minimal_3.7.2~rc1-1_amd64.deb python3.7-minimal_3.7.2~rc1-1_amd64.deb"
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable sh -c "cd /var/cache/apt/archives; dpkg -i python3-minimal_3.7.1-3_amd64.deb"
$ sudo SOURCE_DATE_EPOCH=1545769394 LC_ALL=C.UTF-8 chroot debian-unstable sh -c "cd /var/cache/apt/archives; dpkg -i libpython3.7-stdlib_3.7.2~rc1-1_amd64.deb libpython3-stdlib_3.7.1-3_amd64.deb python3_3.7.1-3_amd64.deb python3.7_3.7.2~rc1-1_amd64.deb"
$ md5sum debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc
2c6fdc51b035428a3881f3eef70e3a5b  debian-unstable/usr/lib/python3.7/collections/__pycache__/__init__.cpython-37.pyc

So manually installing with dpkg in the right order yields the same hash
as with apt. So despite bug #917386 this might not be an apt bug.

I found that depending on the postinstall that is executed, the hash is
different. So I removed __init__.cpython-37.pyc and then ran
dpkg-reconfigure for several packages. Here are the hashes of
__init__.cpython-37.pyc depending on the postinst that got run:

python3.7:         c06080cb5a72be8e409814025236f097
python3.7-minimal: df6fe61fe176e4858ce2062233d2280e
python3-minimal:   b940cc549d39dd3e521a41b8eba3a295
python3:           b940cc549d39dd3e521a41b8eba3a295

Debian policy §6.2 requires idempotency for individual maintainer
scripts which is not violated here. Nevertheless I would argue, that it
would make sense for different Python3 maintainer scripts not to leave
the same files with different content depending on which maintainer
script is being run.

As by the analysis above, there exist at least five different ways that
pyc files can end up. For the sake of reproducibility, there should
probably only be one result independent on how Python3 is installed.

Thanks!

cheers, josch