rebuilderd release and DiVOC Arch Linux Rebuilding workshop

[kpcyrd]

I usually don't post on this email list, but I have two updates that
might be relevant for others on this list:

We (Arch Linux) have held our first(?) rebuilder workshop. The session
was held on irc, we handed out a written document with instructions on
how to install and use repro and had multiple people with zero prior
experience in reproducible builds verify their first package.

[1]: https://github.com/archlinux/archlinux-repro

I've also spontaneously decided to release my rebuilderd repo[2]. You
are very welcome to have a look or even try to run it, but you should
keep in mind that it's considered experimental (and shouldn't be exposed
to the internet yet).

[2]: https://github.com/kpcyrd/rebuilderd

The project consists of 3 components:

- rebuilderd: the brain that keeps track of the pkg states, queue and
  hands out tasks to workers
- rebuilderd-worker: receives a task from rebuilderd, shell out to the
  actual rebuilder (eg. repro for archlinux) and report the result back
- rebuilderctl: the tool that's used to manage rebuilderd, query for
  results, inspect the queue, shows what the workers are currently
  doing, and syncs the package index

I took a few design decisions with this project:

- Distributable: I've tried to make sure the project is easy to package,
  I'm going to upload it to Arch Linux in the near future. Building from
  source is also supposed to be well supported obviously. Rebuilderd is
  trivial to run in docker (the workers may not).

- Easy to setup: In addition to packages in your distro it's also
  supposed to be easy to setup. Configuring databases is annoying every
  time, so I'm storing data in sqlite that's managed by rebuilderd.
  Getting a rebuilder up and running is supposed to be a matter of
  1) rebuilderd: Configure a shared secret so only legit workers can
     receive work and submit results.
  2) rebuilderd-worker: Configure the shared secret and address of
     rebuilderd.
  No annoying "create a postgres user and configure its permissions",
  everything is supposed to just work™ with the two exceptions listed
  above.

- Tracking the package databases: The sync command is behaving like a
  package manager, downloads the package index and tracks it. Every new
  package versionthat's observed is added to the queue for verification.
  This must not depend on the regular package manager like apt or
  pacman.

- Doesn't rely on a distro: it should be possible to track Arch Linux
  with a rebuilder system that runs on debian and vice versa. This
  includes the package syncing code. The only exception is the worker
  that may depend on a specific distro.

- Not a CI system: I'm not recording build artifacts or diffing them. If
  the rebuilt package matches I'm flagging it as GOOD, in all other
  cases I'm flagging it as BAD. Troubleshooting is supposed to be done
  locally.

- No signing (yet): This is going to change eventually, but right now I
  want to focus on independent verification itself. The results are
  considered experimental and https is supposed to be sufficient for
  now.

The first client for this is going to be a script that takes the list of
packages installed on your system and shows you how many % of your
system are verified already (using a list of urls for rebuilders to
query).

There are a few things I need to take care of, afterwards I'm going to
setup a public rebuilderd (for Arch Linux) and follow up with a url and
how to query it. Please let me know if you're interested in doing the
same.