[rb-general] [FOSDEM16] Reproducible FreeBSD and variants

[Steven Chamberlain]

Hi!

In the BSD devroom at FOSDEM '16, reproducible builds were discussed,
but there is remarkable tie-in between these, Debian's reproducible
builds effort and some of my own projects:


  * Reproducible builds in FreeBSD packages (Baptiste Daroussin)
    https://fosdem.org/2016/schedule/event/reproducible_freebsd_packages/

FreeBSD base is mostly reproducible, but the most work will need to be
done in FreeBSD Ports (packages).  The poudriere tool should help with
this as it normalises the build environment, like sbuild.  And
diffoscope was uploaded to FreeBSD Ports today!


  * ElectroBSD - Getting a reproducible BSD out of the door (Fabian Keil)
    https://fosdem.org/2016/schedule/event/electrobsd/

Applying very high standards to what makes a free and secure OS,
this project is something like a linux-libre equivalent based on FreeBSD
(de-blobbed), with privacy enhancements (having Tor installed and
disk encryption enabled by default).

It is self-hosting and builds reproducibly from source using a native
ElectroBSD build environment, as an intended feature.  But it seems only
variation in time, hostname and CPU has been tested yet and not other
things.  The situation with Ports will be the same as for FreeBSD, I
think.

The release images are not reproducible since they are a UFS filesystem
built by makefs.  But I think I already wrote a patch for this!
http://sources.debian.net/patches/patch/makefs/20100306-6/maxtime-option.diff/
I will soon rebase that on FreeBSD-CURRENT and send it upstream.


Also, I think ho1ger said that the r-b.org Jenkins slave for FreeBSD is
awkward to manage, but I have ideas for that which I'm slowly working
on...

I've been trying to get started testing Debian GNU/kFreeBSD package
reproducibility (and I'd like it being mass-rebuilt anyway to find FTBFS
or other bugs sooner).  Just a few weeks ago I got Jenkins working on
kfreebsd (after porting some dependency) so it can run as a master or
slave.

I think deployment tools would help, since r-b.org has so many slaves
now.  It may even help with outsourcing the systems administration, if
that could be done via Git.  DSA maintains a Puppet configuration
already for kfreebsd buildds.  We could maybe reuse it someday to deploy
a kfreebsd slave, or several?

The other missing part, something else I'm working on is easier
creation of jails.  Those allow to run a mix of FreeBSD or Debian
GNU/kFreeBSD chroots on one machine.  With that, a kfreebsd Jenkins
slave could spawn temporary jails (like sbuild) to run the existing
FreeBSD reproducibility tests, even for more than one version, and maybe
ElectroBSD?  Plus some new jobs to test kfreebsd packages, and maybe
even poudriere for testing FreeBSD Ports.

I have a test environment (on a not so fast machine, and may not have
access to it indefinitely) already running rebootstrap jobs (where I
rebuilt GNU/Linux i386 from GNU/kFreeBSD).  I'm using some ZFS features
to get better performance out of the hardware (LZ4 disk compression;
force disabling sync in the entire chroot), another interesting feature
of ZFS is it tends to randomise readdir() order.

A ZFS-based kfreebsd host could quite easily spawn ZFS *and* UFS-based
chroots (using ZVOLs allocated from ZFS, a bit like a loop mount, still
having all the performance benefits), where UFS has a regular readdir()
ordering.

With that, and by changing some of the jail parameters like the
hostname, and customising the setup of the chroot for each job;  it
could actually perform most of the variations listed at:
https://tests.reproducible-builds.org/reproducible.html#variation

So given some time, this sounds do-able, the only missing things I
was already working on anyway.  Some hardware would be eventually needed
to run all of this, but KVM guests are known to work (the kfreebsd
buildds are such).  For the moment I have enough to do development on.

Regards,
-- 
Steven Chamberlain
steven at pyro.eu.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20160131/970210a7/attachment.sig>