Need help with getting a package to build reproducibly on arm*

[Vagrant Cascadian]

On 2021-01-08, Vagrant Cascadian wrote:
> On 2021-01-07, Vagrant Cascadian wrote:
>> On 2021-01-07, Michael Biebl wrote:
>>> Am 07.01.21 um 18:24 schrieb Michael Biebl:
>>>> as can be seen at [1], systemd does not build reproducibly on armhf and
>>>> arm64 (while there is no problem on amd64 and i386).
>>>> 
>>>> The problem is, I have no idea what the diffoscope diff [2] means and
>>>> how I can make the package build reproducibly everywhere or how I can
>>>> further investigate this.
>>>> 
>>>> Any help here is greatly appreciated as I think reproducible-builds are
>>>> a great effort and I'd like to support that as much as I can.
>>>> 
>>>> Regards,
>>>> Michael
>>>> 
>>>> 
>>>> [1]
>>>> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/systemd.html
>>>> [2]
>>>> https://tests.reproducible-builds.org/debian/rb-pkg/unstable/arm64/diffoscope-results/systemd.html
>>
>> My best wild guesses would be parallelism, filesystem ordering or locale
>> differences causing various sort ordering differences.
>>
>> I'm running a local build on arm64 with "reprotest --auto-build" to see
>> if it can help give us any better leads, will see if that shows anything
>> more useful... it could take some time on not particularly fast
>> hardware.
>
> First attempt was reproducible for me (two normalized builds and one
> varied build), though I couldn't vary the clock with reprotest
> (libfaketime appears to trigger issues with building systemd)... or
> fileordering, user, group or hostname due to some limitations my my
> typical test environment. The command I ran was:
>
>   reprotest  --verbose --min-cpus=1 --vary=-user_group,-domain_host,-fileordering,-time auto -- null
...

But the second attempt for some reason did produce some interesting
results... why it didn't happen the first time suggests it is not
deterministic.

│ │ │ ├── ./usr/bin/bootctl
...
│ │ │ │ ├── strings --all --bytes=8 {}
│ │ │ │ │ @@ -250,15 +250,15 @@
│ │ │ │ │  SystemdOptions
│ │ │ │ │  Failed to set SystemdOptions EFI variable: %m
│ │ │ │ │  supported
│ │ │ │ │  not supported
│ │ │ │ │  Failed to query reboot-to-firmware state: %m
│ │ │ │ │  Failed to parse argument: %s
│ │ │ │ │  Failed to set reboot-to-firmware option: %m
│ │ │ │ │ -/EFI/systemd/systemd-bootaa64.efi
│ │ │ │ │ +/EFI/systemd/systemd-bootarm.efi
│ │ │ │ │  Failed to access EFI variables. Is the "efivarfs" filesystem mounted?
│ │ │ │ │  Failed to determine current boot order: %m

This suggests to me that the running kernel is somehow used to determine
the userspace architecture, effectively similar to:

  https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_arch_issue.html


The armhf builders on tests.reproducible-builds.org for Debian do not
systematically test this. I'm not sure about the arm64 builders, but I
think they run the same kernel, so it seems unlikely to be the only
issue. Also surprised the i386 builder doesn't catch this. Then again,
it only happened on my second try, which suggests this is
non-deterministic in some way; maybe the slower armhf/aarch64
architectures trigger it more often?

I'll later post the results of the diffoscope output somewhere and give
a closer look.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20210108/2f97ecec/attachment.sig>