[rb-general] Uploading buildinfo files to buildinfo.debian.net

[Holger Levsen]

Hi,

On Fri, Feb 15, 2019 at 01:51:40PM -0800, Vagrant Cascadian wrote:
> On 2019-02-15, Holger Levsen wrote:
> > I've just been re-reading this old and joyful thread... :)
> I come back to it now and then myself... :)

and here we go again...

I'll also kind of top-post now, because I want to emphasize one
important bit of information:

we now have two similar implementations of a buildinfo server for Debian
.buildinfo files:

- https://buildinfo.debian.net - this service exists since Oct. 2016
- https://buildinfos.debian.net - this service exist since March 2019

I've set up the latter because the former doesnt provide an easy means
to only get .buildinfo files for .debs from ftp.debian.org and because
it was easy to set it up. (The code lives in jenkins.debian.net.git)

I'm not really happy about the duplication of services and the very
similar names, but I couldnt come up with a better name and this is all
prototyping land anyway, so - for now - I just went ahead.

I'm also not sure how to proceed with those two services in future. For
now I think it's ok to have both and let us experiment with both.


> > On Thu, Oct 25, 2018 at 12:56:20PM -0700, Vagrant Cascadian wrote:
> >> I started the process of uploading all the .buildinfo files available on
> >> ftp.debian.org to buildinfo.debian.net.
> >> 
> >> Then I hope to set up a cron job to do uploads at least daily with a
> >> little better error-handling.... Would be more ideal to have something
> >> more formally integrated into infrastructure, but maybe I can work out a
> >> proof-of-concept implementation as a basis for something that can be
> >> integrated.
> >
> > did you manage to setup this cron job?
> 
> I had thought I left more detail about the current status, but
> apparently not! Thanks for the nudge.
> 
> I have a cron job running on coccia.debian.org since November, as my own
> "vagrant" user:

I have a similar cronjob on coccia...

>   coccia.debian.org:~vagrant/rb-buildinfos/upload-buildinfos
> 
> Logs for various upload passes in are the same directory, which should
> probably be migrated to sqlite or some real database. The script is
> checked into it's own git repository, but not properly pushed
> anywhere.
> 
> The cron job runs several times per day, checking the queues for
> buildinfos uploaded both the current day yesterday to make sure we don't
> miss a .buildinfo file uploaded in the middle of a processing run. If
> coccia were down for longer than 24 hours, it might need to manually be
> run to check for missing ones.
> 
> The vast majority of buildinfo files uploaded to the archive should be
> present in buildinfo.debian.net since November 2018. I also "manually"
> uploaded all the available buildinfo files from 2017-2018 (most of the
> very small number from 2016 failed for one reason or another).

buildinfos.debian.net has all .buildinfo files since December 2016.

> There are a small number of buildinfo uploads that buildinfo.debian.net
> rejects for some reason probably related to ed25519 signing keys:
>   https://github.com/lamby/buildinfo.debian.net/issues/51

see above :) (=buildinfos.d.n has those.)

> There are a few individual developers uploading unsigned .buildinfo
> files, as well as a few buildds for non-release architectures
> (e.g. hurd-i386, kfreebsd-*). To hadle those, I actually had a
> legitimate use for the technique described in:
>   https://xkcd.com/1181/
> Which basically means I don't even bother attempting to upload unsigned
> buildinfo files.

see above :) (=buildinfos.d.n has those.)
 
> So, it's working, but we probably would need a little more work on it to
> integrate into debian's infrastructure.

this is also true for buildinfos.d.n.


-- 
tschau,
	Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

In Europe there are people prosecuted by courts because they saved other people
from drowning in the  Mediterranean Sea.  That is almost as absurd  as if there
were people being prosecuted because they save humans from drowning in the sea.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20190429/1dcc2ece/attachment.sig>