Need Help with duperemove in Debian

[Marc Haber]

Hi Chris,

sthanks for your analysis. It looks like you were correct.

On Thu, Sep 04, 2025 at 02:24:04PM -0700, Chris Lamb wrote:
>The TL;DR is that kpcyrd correctly identified the root cause as an
>issue due to the order of the object linking and Roland was pretty
>close in his remark about locales.
>
>The real causes, however, are that GNU Make has some interesting and
>nonintuitive sorting tripwires and you were caught out with them due
>to an issue due to some particular filenames chosen by the upstream
>duperemove developers.

... and them not using autotools, right? Or would that also happen with 
plain autotools?

>You perhaps wouldn't have thought to look for this specifically, but
>in the reprotest log that you linked to, the linking order was
>reversed between two files only: filerec.o and file_scan.o.

I wouldn't have KNOWN to look for this ;-)

As someone who is just passing by the science of reproducible builds, 
those huge logs and those powerful tools are scaried.

>Cutting out a huge amount of noise from the log, cc was being called
>like this between the two builds used for the comparison:
>
>  cc […] filerec.o file_scan.o […] -o duperemove
>  cc […] file_scan.o filerec.o […] -o duperemove

Finding those things is an art! I bow to you for seeing that. That needs 
a lot of experience.

>This was because, in the Estonian et_EE.UTF-8 locale that reprotest
>uses, underscores sort after the numbers and letters. In the C locale,
>by contrast, underscores sort *before* the letters: "file_scan.o"
>is therefore placed before "filerec.o".
>
>This is more of a slight weirdness in the C locale rather then
>something in ee_EE. (ee_EE is still a good choice for reprotest, however,
>because whilst en_US and friends sort the lower case before the upper
>case, et_EE.UTF-8 and the "C" locale sort capital letters before the
>lower case. Also, Estonian sorts the letter 'z' between 's' and 't'.
>Also, I'm fun at parties.)

Definetely.

>The third confusing thing about sorting files in Make is that
>attempting to naively pipe the wildcard function to sort isn't
>actually valid Make code, and then, rather unhelpfully, you don't get
>any message that you've made a mistake. Your first attempt to fix
>the package:
>
>   $(wildcard *.c | sort)
>
>… looks like it should do something useful, but it will still vary
>depending on the current collation. But this is because it's essentially
>bogus code:
>
>   $(wildcard *.c | you can actually put anything here and receive no error surprise hahahaha)

Wohoo. I'm excited to hear abot that.

>The only way to reliably get simple, byte-comparison, "C" sorting is
>via:
>
>   $(sort $(wildcard *.c))

How, eh, intuitive.

>Indeed, I can confirm that the following patch makes the proposed
>duperemove package (i.e. cloned from your debian/latest branch)
>reproducible:
>
>  -CFILES = $(filter-out tests.c,$(wildcard *.c))
>  +CFILES = $(filter-out tests.c,$(sort $(wildcard *.c)))

I can confirm that.

>No other changes are required. (This is better than changing the
>debian/rules file as mentioned above, as it can be sent upstream.)

Is that what you are suggesting? Or would there be a more "elegant" fix 
tht upstream could take?

>Anyway, hope this helps you and others in the future when they hit
>similar underscore-related or GNU Make sorting foo.

It does. My test branch in salsa is all green now.

The only reason that I have not uploaded yet is that duperemove is my 
test ground for a possible new Debian contributor, and this trivial 
upstream patch is the perfect example to teach them about gbp pq, Debian 
patches, Salsa Merge Requests and the upload process. I just need them 
to find time to listen to the lecture, and if they don't do that soon 
I'll upload myself and alone soon.

Thank you for helping, reading that was enlightening!

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421