What should be the proper practice to manage `.dsc` files on Reprepro?

[Vagrant Cascadian]

On 2022-05-26, Yaobin Wen wrote:
> In my company, we use *Ubuntu (18.04)* and are practicing reproducible
> builds. Our code is built into a lot of .*deb* packages using *debuild* (and
> related tools). We have made a lot of effort to make our builds
> reproducible by following the Achieve deterministic builds
> <https://reproducible-builds.org/docs/> and Known issues related to
> reproducible builds
> <https://tests.reproducible-builds.org/debian/index_issues.html>. We have
> made a lot of progress and are still working on it.

It is great to hear of your work on reproducible builds!


> We set up a company-wise *Reprepro server to serve the Debian packages
> that we build regularly*. We publish both *.deb* files and the *GPG-signed
> .dsc* files to the Reprepro server. BTW, our build system is designed to do
> an "*change-only build*": if a package is not changed since the last build,
> i.e., its *changelog* is not changed, the package is not built again this
> time. *Their .deb and .dsc files are still added to Reprepro but because
> these files remain unchanged, Reprepro can successfully "add" them (but in
> fact they are skipped)*. I figured this point may be important to
> understand why I have my questions below.

I don't follow what you mean by "the package is not built again this
time" and "Their .deb and .dsc files are still added to Reprepro". How
can a package be added if it is not built?

What's unclear to me is why you're uploading the same .deb and .dsc
files to the same reprepro repository multiple times... ?


> Although we have solved many reproducibility issues in the .*deb* files, *I
> found the .dsc files were changed when* I rebuilt the packages (by deleting
> the previously built *.deb* and *.dsc* files) so Reprepro refuses to
> include them and reports the following error:
>
> ERROR: '<path-to-dsc-on-build-machine>' cannot be included as
>> 'pool/<path-to-dsc-in-reprepro>'.
>> Already existing files can only be included again, if they are the same,
>> but:
>> md5 expected: <md5-1>, got: <md5-2>
>> sha1 expected: <sha1-1>, got: <sha1-2>
>> sha256 expected: <sha256-1>, got: <sha256-2>
>
>
> *diffoscope* told me the `.*dsc*` files *only differ in their GPG
> signatures* - the related source tarball (<filename>.orig.tar.gz) and
> debian tarball (<filename>.debian.tar.xz) *have not changed between
> builds.*

That's to be expected...


> I understand that, because as this SO answer says
> <https://security.stackexchange.com/a/78958/80050>, the GPG signature is
> generated using the creation time as an input. I found the issue
> cryptographic_signature
> <https://tests.reproducible-builds.org/debian/issues/unstable/cryptographic_signature_issue.html>
> that
> made me think we should not have signed our .*dsc* files, but the Debian
> Admin's Handbook
> <https://www.debian.org/doc/manuals/debian-handbook/sect.source-package-structure.en.html>
> shows that the .*dsc* files are supposed to be signed by the maintainers.
> In addition, in the Known Issues list
> <https://tests.reproducible-builds.org/debian/index_issues.html>, I didn't
> seem to find any issue that's related with the .*dsc* files.

If you want to know which party claims to have built a given .dsc file,
you need the signatures on them. If you track that information some
other way, you *could* use unsigned .dsc files...

Or you could re-use the original .dsc files, if all the contents they
reference are bit-for-bit identical. If you want to store the new ones
somewhere else as a "proof of having rebuilt it again" you could do
that, but obviously not in the exact same repository.


> *After reading around, I'm guessing my understanding about
> reproducible builds may not be totally correct, so I want to ask here:*
>
>    1. *Should the .dsc files be reproducible, too?* Because Reprepro can
>    manage .*dsc* files, I've been thinking that .*dsc* files should be
>    reproducible, but now it seems not?

If you build them in the same build environment, with the same source
code, they should be reproducible *minus the signatures*, as you've
noted...

Generally, from a reproducible builds perspective, the .dsc file is
considered an input to the build process rather than a result of a build
process. Though admittedly, .dsc files are themselves artifacts of a
source-only build, so it is a bit of a grey area.


>    2. In my case, since my company maintains both .*deb* files and .*dsc* files
>    in Reprepro, if one day we need to build the code of an earlier version, we
>    would inevitably generate different .*dsc* files because of the GPG
>    signatures. *Am I supposed to publish the .dsc files to the same
>    Reprepro server that we maintain our regular build?* Because I've been
>    thinking .*dsc* files should also be reproducible, I've been thinking we
>    should keep using the same Reprepro server. *But now it looks like we
>    need to prepare a second Reprepro server to hold the packages of the
>    earlier version.*

So you're looking to be able to recreate your whole repository from
scratch (maybe from git repositories or some other VCS?) at some future
date, reproducibly?


>    3. *How does everyone else maintain their Reprepro server?* Do they keep
>    publishing the build artifacts to the same server after a build? Or do they
>    delete the previously published artifacts before publishing the new build?
>    Or do they even recreate the Reprepro server every time they make a new
>    build?

Typically with a Debian-style repository, it's not expected that you
will ever upload the same version of any given object more than
once. This is partly because apt and related tools are designed with
that assumption in mind, and will behave poorly if you in fact feed a
repository packages with the same version but different content and then
try to use those packages in the real world.


So, still a bit unsure what you're actually trying to accomplish; if you
spell that out a little more clearly it might be easier to make good
suggestions.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20220526/3eb67a21/attachment.sig>