[rb-general] [FOSDEM16] Reproducible FreeBSD and variants

[Holger Levsen]

Hi Steven,

(I believe) somehow your mail did not make it to the rb-general list, are you 
subscribed? (leaving full quoted context for that reason…)

On Donnerstag, 11. Februar 2016, Steven Chamberlain wrote:
> Holger Levsen wrote:
> > > I've been trying to get started testing Debian GNU/kFreeBSD package
> > > reproducibility (and I'd like it being mass-rebuilt anyway to find
> > > FTBFS or other bugs sooner).  Just a few weeks ago I got Jenkins
> > > working on kfreebsd (after porting some dependency) so it can run as a
> > > master or slave.
> > 
> > cool. what are the specs, roughly?
> 
> Currently this is a Core i7-980X, 24 GiB RAM, 2x 1.4TB disks.

nice

> It is a server I rent out for business uses, but outside office hours
> it was quite idle so I schedule jobs to run overnight and on weekends:
> some rebootstrap jobs, and rebuilding debian-installer per Git commit.
> I'd like to add reproducible jobs once I figure out how they work.

I'm not sure I'll be comfortable building "anything+everything" on this box 
then. ("rented for business uses")

The jenkins set up is not designed with security in mind, in the sense that I 
don't trust the box on *anything*, it's ok to produce QA results, but… I'm not 
comfortable giving it access to trusted machines. (even in jails, etc & YMMV, 
obviously…)

> I've decided to open up the Jenkins web interace now (except HTTP POST
> because I really don't trust its security!)
> http://jenkins.kfreebsd.eu/jenkins/

nice!

> I've put Jenkins master in its own separate chroot jail.  The jobs run
> one at a time, in another separate sid chroot jail.  Jails prevent
> access to files or devices of the host, and I can firewall their network
> access if I want to.
> 
> > I dont see us moving to DSA
> > maintained host. patches for that (="less sudo usage…") welcome ;)
> 
> This is a major concern for me at the moment, as I allow Jenkins to do
> many things on the host (outside of any jail) via sudo to set up the sid
> chroot.  I'm experimenting with better ways to do this.

well, things should probably work nicely in a jail where jenkins thinks it can 
do what it wants :-)

> I'd started out with sbuild, but it is incompatible with jails.  It was
> also really slow, and that's a major concern for me with the limited
> resources I have.  Optimizing is fun also.

I dont think I want to build reproducible Debian packages on one arch in a 
different way than on the others. So pbuilder for now (until we maybe switch 
to sbuild with the patch for reproducible rebuilds once we're rebuilding 
against sid…)

> With ZFS I can set sync=disabled on a whole chroot which is similar to
> using Linux 'eatmydata', and makes APT/dpkg stages run really fast.
> 
> I'm also trying ccache, and preserving the cache directory between job
> runs.  Hasn't helped much though.
> 
> What might be nice is if the host's sshd could set up so that, upon
> login as user 'jenkins', it would give a root shell in a freshly-created
> jail.  ZFS snapshots and clones could make that really fast.  In that
> kind of setup, Jenkins need not run on build machines (saving hundreds
> of MiB RAM).
> 
> Mostly I'd like to reduce the setup/teardown time so that a small
> package like 'hello' takes only the smallest amount of time to build.
> I think this is where most time would be wasted given how many small
> packages there are in the archive.

I don't think that time is *that* relevant as long as you use tmpfs (or 
something similar) and have enough RAM. maybe then it even becomes irrelevant 
:)

Also we have packages in the archive which build twice in <60s (on our amd64 
"hw"), while the average time is 8min. 

So currently we're building 60*24/8*32=5760 packages a day (/8 because average 
build time is 8min and *32 because we have 32 amd64 builders).

If you bring down the setup time to 0s you would "only" build 6560 packages a 
day (60*24/7*32), or 13.8% faster.

I suspect the numbers will look similar for armhf, but I'll leave that as an 
excercize for the reader :)


cheers,
	Holger
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20160212/61a06ad4/attachment.sig>