[rb-general] Uploading buildinfo files to buildinfo.debian.net

[Vagrant Cascadian]

On 2019-02-15, Holger Levsen wrote:
> I've just been re-reading this old and joyful thread... :)

I come back to it now and then myself... :)


> On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote:
>> I started the process of uploading all the .buildinfo files available on
>> ftp.debian.org to buildinfo.debian.net.
>> 
>> Then I hope to set up a cron job to do uploads at least daily with a
>> little better error-handling.... Would be more ideal to have something
>> more formally integrated into infrastructure, but maybe I can work out a
>> proof-of-concept implementation as a basis for something that can be
>> integrated.
>
> did you manage to setup this cron job?

I had thought I left more detail about the current status, but
apparently not! Thanks for the nudge.

I have a cron job running on coccia.debian.org since November, as my own
"vagrant" user:

  coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos

Logs for various upload passes in are the same directory, which should
probably be migrated to sqlite or some real database. The script is
checked into it's own git repository, but not properly pushed
anywhere.

The cron job runs several times per day, checking the queues for
buildinfos uploaded both the current day yesterday to make sure we don't
miss a .buildinfo file uploaded in the middle of a processing run. If
coccia were down for longer than 24 hours, it might need to manually be
run to check for missing ones.

The vast majority of buildinfo files uploaded to the archive should be
present in buildinfo.debian.net since November 2018. I also "manually"
uploaded all the available buildinfo files from 2017-2018 (most of the
very small number from 2016 failed for one reason or another).

There are a small number of buildinfo uploads that buildinfo.debian.net
rejects for some reason probably related to ed25519 signing keys:

  https://github.com/lamby/buildinfo.debian.net/issues/51

There are a few individual developers uploading unsigned .buildinfo
files, as well as a few buildds for non-release architectures
(e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
legitimate use for the technique described in:

  https://xkcd.com/1181/

Which basically means I don't even bother attempting to upload unsigned
buildinfo files.


So, it's working, but we probably would need a little more work on it to
integrate into debian's infrastructure.


live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20190215/91f7dabb/attachment.sig>