[rb-general] Unreproducible test logs in output

[Ximin Luo]

Bernhard M. Wiedemann:
> On 2017-05-10 19:22, Hanno Böck wrote:
>> I think the reasonable way to handle this is to provide the test output
>> at a stable URL and reference it.
> 
> one stable URL will not be good enough, because there can be different
> codestreams of the software (and maintenance-updates for each of them),
> you also want to have the associated test-results available for
> inspection for each of those versions.
> 
> Thus in openSUSE we have separate FOO-testsuite packages like
> https://build.opensuse.org/package/show/Base:System/coreutils-testsuite
> that always get updated and rebuilt when the main package does,
> but can be handled differently in other regards, e.g. are not included
> on the DVD, or do not even produce a package, but just the build log
> (so more of a test-log then).
> 

Thanks for the pointer. How does this work, in detail? Is it like a "hack" around the packaging system, i.e. "coreutils-testsuite" is just like any other package that depends on "coreutils", except it builds nothing and its build process is just to "run some tests on coreutils"?

This sort of thing is possible to achieve in Debian as well, in fact it's probably more graceful to do it via the "autopkgtest" mechanism that lets you define post-install tests in debian/tests/. I don't believe we have a standardised way to capture output from this nor distribute it, however. Do RPM-based systems have a standardised way to distribute build logs?

When you say "can be handled differently in other regards", do you mean there already exists a mechanism, or that one could make one in the future? This is the status of Debian as well, for the time being. I'm sure "it's possible", but I was wondering if other distros had already solved this.

Using in-toto's terminology as mentioned by Eric, I guess what I'm suggesting is that distributions could offer a way for packages to (a) define structured by-products and (b) distribute these alongside installable packages, but in a non-installable way.

Indeed, a buildinfo file is already one example of a structured by-product. They are not expected to be completely reproducible, yet we'd like them to be available and distributable, and we don't want to install them into standard system filepaths. Test logs also share these same characteristics. With some effort (which may or may not be worth it) we could probably tweak the design work we've already done for buildinfo files, and extend it to cover this type of byproduct as well.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git