Continuous integration: SOURCE_DATE_EPOCH and testing

[James Addison]

On Thu, 9 Jan 2025 at 16:42, David A. Wheeler <dwheeler at dwheeler.com> wrote:
>
> > On Thu, Jan 9, 2025, at 02:28, James Addison via rb-general wrote:
> >> are there any
> >> recommendations about whether SOURCE_DATE_EPOCH must-not / can /
> >> should / must be configured during (immediate and/or later) testing of
> >> software that has been built reproducibly?  And if so, for what
> >> reason(s)?
>
> At least some testing should be done WITH the SOURCE_DATE_EPOCH set.
> You generally want your testing before release to be as similar as possible to the
> production environment, because every difference can hide a bug.
> That ideal is typically not *fully* realized, but it's worth shooting for.

With the exception of build environments, I'd expect few production
systems to have have SOURCE_DATE_EPOCH configured - so perhaps that is
an argument _not_ to configure it during testing?

For the remaining situations -- build systems / toolchains, where
SOURCE_DATE_EPOCH is a runtime input -- tests to cover
SOURCE_DATE_EPOCH-related functionality definitely make sense; we want
to make sure it has the intended effect, and ideally not other
side-effects.

But I'd argue that most of that test coverage should, ideally, be
provided within the test suites of the software that uses
SOURCE_DATE_EPOCH as an input; for example by using test
setup/fixtures that temporarily set SOURCE_DATE_EPOCH in the
environment.  I suppose we could argue that those mechanisms could
somehow be unreliable.. but provided that they're implemented by FOSS
and reproducible software, they'd be no less reliable than configuring
the environment variable on a host system before running the test
suite within that environment.

Counter to that is one situation where I do think it's valuable to
vary SOURCE_DATE_EPOCH 'outside' of the test suite: it could help to
detect tests that have a different outcome based on the variable;
possibly/likely an indication of an implementation bug (this is, in
effect, what happened in the Sphinx failure mentioned).