Sphinx: localisation changes / reproducibility

[James Addison]

A follow-up: after doing more work to try to confirm the behaviour of
the fix -- something I should have done before even starting
development! -- I was confused that I couldn't replicate the original
problem when using a version of the codebase _before_ my proposed fix
pr#10949 was applied.

I now believe that the issue had, in fact, already been fixed between
versions v4.5.0 and v5.0.0 of Sphinx - so I've offered a revert of my
changes.

Details about tracing the location of the existing fix (using 'git
bisect') can be found here:
https://github.com/sphinx-doc/sphinx/issues/9778#issuecomment-1501172176

Attempting to learn from this: I find the experience interesting
because it seems that I deluded myself into believing that I'd
resolved a bug -- and unfortunately drew in a bunch of other people's
time doing that -- when, in fact, following good time-and-vesion-aware
engineering practices (not always easy when keen to contribute to a
project, but as demonstrated here, potentially very important) could
have avoided much of that work in the first place.

The original bugreport was detailed and quite clear, so I think that
much of the fault here was from me not carefully reading and
considering how to proceed.  There could be other learnings /
recommendations (I'm mulling over some ideas related to
continuous-bug-checking), but I'm unable to make any clear
recommendations there yet, partly because I think that
bug-checking-and-verification, while valuable, is not always
considered desirable or economically-beneficial work -- and partly
because writing test cases to reproduce bugs (which could make
continuous evaluation of stale bugs easier) often performs 80%+
(guestimate, in my opinion) of the work to find the cause of the bug
-- meaning that it's frequently worthwhile to combine with fixing the
bug (in other words: there's often a significant overlap between
developing a bug reproduction test case and fixing the bug).

Probably nothing new to many of the folks on this mailing list and/or
seasoned software engineers generally, but I figured I'd try to
document my findings :)

On Sat, 8 Apr 2023 at 11:10, James Addison <jay at jp-hosting.net> wrote:
>
> Hi folks,
>
> A set of reproducible-build-related changes[1] that I've developed for
> sphinx (a documentation project generator) have been accepted for
> inclusion in v6.2.0 of sphinx.
>
> I'm optimistic that those changes can address a sizable category[2] of
> reproducible build failures related to translation of documentation
> during software builds (reproducible build testing intentionally
> varies the host LANGUAGE setting to shake out unintended sources of
> build variation, and some sphinx projects fail rb-tests due to that).
>
> However.. with the changes merged (although not yet released) I'm
> beginning to have some doubts about them.
>
> The positive effect of the changes is that I expect they'll help to
> confirm and achieve reproducibility for a good chunk of remaining
> non-reproducible software.
>
> The downside is: disabling localization -- or perhaps more accurately:
> emitting all documentation for each project using 'null'[3]
> translation -- seems like a fairly blunt, and perhaps unwelcome (for
> consumers) way to achieve reproducibility.
>
>
> A longer, better path to achieve reproducibility would be to support
> building documentation in _all_ available translated locales during
> Sphinx project builds (something that is not yet supported -- and with
> at least one component that I'm aware of (objects.inv) that doesn't
> seem to support multi-language content).  Doing that should produce
> output artifacts that users of any supported locale can access in a
> relevant localised way, and that can be made bit-for-bit consistent.
>
> In summary: I'm writing partly optimistically because I think the
> merged changes could improve and help to confirm reproducibility of
> software during testing.  But I also feel a bit conflicted about the
> way I've approached the changes and their implications, so I'm also
> keen to gather feedback and thoughts.
>
> Thank you,
> James
>
> [1] - https://github.com/sphinx-doc/sphinx/pull/10949
>
> [2] - https://tests.reproducible-builds.org/debian/issues/unstable/sphinxdoc_translations_issue.html
>
> [3] - https://docs.python.org/3/library/gettext.html#the-nulltranslations-class