[rb-general] Unreproducible test logs in output

[Eric Myhre]

I won't offer any opinions on how this interfaces with debian concepts,
but FWIW from some other perspectives...

When building things like this in Repeatr formulas, I list two different
output paths for capture.  The run record points to both, so they can be
reviewed together, but I only expect the binaries to be reproducible,
and the one containing the logs is of course unlikely to end up in any
release or passed on to other processes directly.

I think in-toto has a related concept as well: their paper describes a
step of a software pipeline has having both "products" and "byproducts",
where the latter category includes logs and such which are not expected
to be useful to feed forward.

So it seems the idea of handling logs separately, and expected
non-reproducibility there, has at least some other support.  Your
proposal sounds reasonable to me.

(It is sort of generically horrifying if the exit codes of tests cease
to be meaningful, though.)



On 05/10/2017 05:49 AM, Ximin Luo wrote:
> I meant to send this to rb-general as well:
>
> Ximin Luo:
>> I've been reviewing Debian's "required" and "build-essential" package sets. One new issue that's come up, is that the gcc-6 and binutils packages both put test logs in their binary output.
>>
>> Of course this has no chance of ever being reproducible, due to parallelism, build paths, and many many things. I don't think it would be reasonable to try to work towards making this reproducible.
>>
>> The justification for *including* them is that many very very large programs have test suites that aren't expected to completely pass all the time. Now, sometimes it's possible to ignore specific failing tests, and that is what I have been personally doing for the rust compiler in Debian. 
>>
>> However, in other cases there are just too many, and the maintainer(s) physically don't have enough time to ignore these tests, since nothing else useful will get done. I guess this is the case for gcc-6 and binutils, and I know this is the case for another very large program I help maintain in Debian, sagemath with ~300k test cases.
>>
>> In these cases, it is much easier for the package maintainer to run the tests, pass the build even if their are failures, and distribute the test logs along with the output so that they can be examined later. Indeed I've been very tempted to do this myself for sagemath.
>>
>> How shall we handle this in the context of reproducible builds?
>>
>> As a first proposal, perhaps we could allow package maintainers to define a set of "non-installable binary output" that does not have to be reproducible. This would only be used in very exceptional circumstances such as the ones described above, they can't be installed via the package manager, but they can be downloaded and manually extracted to analyse or distribute test results with.
>>
>> OTOH if we tell package maintainers "don't put test logs in binary output", we need to offer an alternative that caters to the needs of very very large packages, as described earlier. A simple interface to download the build logs of a given binary package, might be sufficient.
>>
>> X
>>