[rb-general] auto-analyzing indeterminism

[Bernhard M. Wiedemann]

On 2017-07-18 15:00, Ximin Luo wrote:
> Very nice! Could you explain a bit how your "bisect" algorithm works? I'm not very good at reading perl..
> 
> Does "0 0 1 0 1 0" mean the package was unreproducible with either
> 
> - only the third variation, or
> - only the fifth variation,

Yes.
It was reproducible when omitting the third and fifth variation together.

In general 0 here means we apply more variation (=disabled tweak to
reduce indeterminism) and 1 means less variation (=enabled tweak).

So packages that are already fully reproducible get a result of
0 0 0 0 0 0
and packages that remain unreproducible even with all tweaks applied get
1 1 1 1 1 1

My 'stupid_bisect' algorithm implemented atm, starts with
$cur = $max = 1 1 1 1 1 1
goes through the list of tweaks (from right to left), disables one bit,
and if it finds that the package stops building reproducibly, it keeps
the tweak-bit on for the following loop iteration.


Since the tweaks are not fully orthogonal, ordering has a slight
influence. E.g. a man-page date could be normalized by using my
reproducible-faketools-date, by not varying the build VM's date or by
applying strip-nondeterminism (not sure here) so only 1 bit might remain
set (the leftmost one).

> So I was leaning towards just testing {1}, {2}, {3}, ... etc each individually. I'm not sure if that's what you implemented though.

it is similar to that.