RBOS / reproducible-openSUSE

[Bernhard M. Wiedemann]

On 18/02/2025 14.32, Justin Cappos wrote:
> This is really awesome!   One thing I am curious about is what sort of 
> issues testing on 5 machines helped you to find?   Did most issues show 
> up when you tested on 2 machines?   What sort of things were missed?
Yes. Most issues were even found with 1 machine, rebuilding in 2 
different directories there to make comparing easier.

I did different kinds of tests. First just building twice on one machine 
found the worst offenders.
Then I did a pass with my 'rbk' (= rebuild with KVM)-script that adds 
variations for year, CPU, #cores, hostname that found an issue with 
copyright year in qt6-webengine.

I also did a compilation round with 14-core-VMs and found that one 
package only produced different binaries with 5 or more cores.

3 of the 5 machines were the same model of EPYC/Rome, but when 
rebuilding on a newer Ryzen, I found one or two race-conditions that did 
not cause variations before.

The 5th machine was a 2010 Intel machine - there I found that some 
packages required x86_64-v3 to build.

In the past I also had one issue that was hard to reproduce that only 
occurred on an older machine with HDD storage, so races / timing issues 
seem to be the most likely to slip through.

Some packages also have non-determinism with 1-bit entropy (e.g. 2 
elements in a hash with random iteration order) - those will give 2 
identical results in 50% of cases.

And if there is an auto-detection for the AVX512 CPU-flag, you cannot 
find that on older machines.


Ciao
Bernhard M.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: OpenPGP digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20250219/439390ea/attachment.sig>