[rb-general] Uploading buildinfo files to buildinfo.debian.net

[Vagrant Cascadian]

On 2018-10-23, Vagrant Cascadian wrote:
> Main blocker that comes to mind is publishing of buildinfo files
> submitted to the archive in a way that people can actually download them
> who are not Debian developers:
>
>   https://bugs.debian.org/763822
>   https://bugs.debian.org/862073

Thanks for bringing this up, it was the final straw inciting me to
action on this particular point!

I started the process of uploading all the .buildinfo files available on
ftp.debian.org to buildinfo.debian.net.

Then I hope to set up a cron job to do uploads at least daily with a
little better error-handling.... Would be more ideal to have something
more formally integrated into infrastructure, but maybe I can work out a
proof-of-concept implementation as a basis for something that can be
integrated.

Still a lot of work to do to make those buildinfo files useable, but at
least it puts the data somewhere where anyone can work on solving the
remaining issues of validating what's actually in the archive.


Since I've got some numbers handy...

All the individual .buildinfo files for 2017 came to:

  4574MB    2017

Compressed into monthly tarballs of .buildinfo files:

  594MB

So that's about an 87% compression rate! Individual .buildinfo files
compressed to around 50%.

For the first 3 months or so of 2017, many of the .buildinfo files
weren't signed, so are probably much smaller. Reasonably
buildinfo.debian.net won't accept unsigned buildinfo files.

The Debian archive accepts some signatures types (elliptic curve) that
buildinfo.debian.net doesn't yet:

  https://github.com/lamby/buildinfo.debian.net/issues/51

So presumably any signed with such keys won't yet get accepted either.


For 2018 the numbers were similar, though a bit larger(more consistant
use of tools that produce .buildinfo files?). Sorry I don't have the
exact details on hand at the moment.


So I'm guessing the overall storage burden would be around 5GB per year
uncompressed and growing slightly as the archive grows and architectures
are added (minus the occasional removed architecture)...


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20181025/c41cb1b3/attachment.sig>