How could we accelerate *deployment* of verified reproducible builds?

[Richard Purdie]

On Sat, 2021-01-30 at 12:22 +0000, Holger Levsen wrote:
> On Fri, Jan 29, 2021 at 05:39:01PM -0500, David A. Wheeler wrote:
> > What would be especially helpful for accelerating deployment of
> > verified reproducible builds in a few key places? E.g., what tools,
> > infrastructure, people paid to do XYZ?
> 
> first, having verified reproducible builds! then, we can deploy them.

I believe Yocto Project can do this today. Obviously its easy to say
that so I intend to prove it :).

To have a verified build you need to share some kind of configuration
and something to verify against.

The binaries I'm going to 'share' for verification are a linux kernel
bzImage and a tarball of a busybox based rootfs linked against musl as
a libc. The configuration you need for this is:

Poky repo: git://git.yoctoproject.org/poky
Poky revision: 36aef08dcd5e45c4138ccd72e8de01157f7213c4

and the configuration:

PACKAGE_CLASSES = "package_ipk"
TCLIBC = "musl"
INHERIT += "rm_work"
DEBUG_FLAGS = "${DEBUG_PREFIX_MAP}"
DISTRO_FEATURES_remove = "opengl"
EXTRA_IMAGEDEPENDS_qemux86-64 = ""
IMAGE_FSTYPES_qemux86-64 = "tar.bz2"
IMAGE_CMD_tar_qemux86-64 = "${IMAGE_CMD_TAR} --format=gnu --sort=name --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"

which gives the sha256sum of the output binaries:

99c6d9ba1162043f348cef1b8385a03e4246323182bdada0200ab69bfe61ecd4  core-image-minimal-qemux86-64.tar.bz2
e36841e544c8ffe628d56e08e0c3965fb350b05c6dc553390283b8330e2ebdcd  bzImage

I've put a small bit of shell script at the end of the mail which takes
this information and builds that result. To verify, run as script with
a parameter specifying which directory to use to build in. I will warn
this this will download source code for everything it needs and it will
build and use its own compiler to build the output. As such it needs
network bandwidth, disk space and will take a while.

You might wonder why I'm not being specific about which distro to use
or the path it should run in. It doesn't matter, it handles that. It
will run on most recent Linux systems and it should tell you if there
are any dependencies it needs which are missing (it needs python 3.6+
and to be able to compile things with gcc 6+).

You could extract the output tar.bz2 and "sudo chroot /bin/ash" into
the image if you wanted.

You may wonder why I'm specifying the tar command to use. I actually
found a bug when testing this as the output differed, using gnu tar
format on one machine and not on another. We'll get that fixed, we tend
to diff packages rather than full images which is why that hadn't been
spotted. I added some config to avoid that issue here for now.

If you want to rerun this with more speed, the standard "cache" options
for Yocto Project can be added:

DL_DIR = "/media/sources/"
SSTATE_DIR = "/media/sstate/"

where downloads are placed into DL_DIR and our cache artefacts (sstate)
are placed in the second location. If you place those outside the build
directory it will make subsequent executions much faster.

The other configuration options are mainly to try and cut down the
build time a bit, musl builds more quickly than glibc and its
faster/simpler without added qemu dependencies or debug information.

I'm taking a bit of a risk by making a bold claim quite publicly and
this could go wrong but I think its interesting to explore :)

Cheers,

Richard


#!/bin/bash
VBDIR=$1
if [ -z "$VBDIR" -o -e "$VBDIR" ]; then
    echo "Please specify an empty directory to run the build in"
    exit 1
fi
mkdir -p $VBDIR
cd $VBDIR
git clone git://git.yoctoproject.org/poky
cd $VBDIR/poky 
git checkout 36aef08dcd5e45c4138ccd72e8de01157f7213c4
. $VBDIR/poky/oe-init-build-env $VBDIR/poky/build
printf 'PACKAGE_CLASSES = "package_ipk"
TCLIBC = "musl"
INHERIT += "rm_work"
DL_DIR = "/media/sources/"
SSTATE_DIR = "/media/sstate/master"
DEBUG_FLAGS = "${DEBUG_PREFIX_MAP}"
DISTRO_FEATURES_remove = "opengl"
EXTRA_IMAGEDEPENDS_qemux86-64 = ""
IMAGE_FSTYPES_qemux86-64 = "tar.bz2"
IMAGE_CMD_tar_qemux86-64 = "${IMAGE_CMD_TAR} --format=gnu --sort=name --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
' > $VBDIR/poky/build/conf/auto.conf
bitbake core-image-minimal || exit 1
sha256sum $VBDIR/poky/build/tmp/deploy/images/qemux86-64/bzImage
sha256sum $VBDIR/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.tar.bz2
echo $VBDIR/poky/build/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.tar.bz2 ready!