[rb-general] rpm ecosystem update

Wed Dec 5 15:13:58 CET 2018

Hi,

while preparing for our r-b world summit in Paris,
it became obvious that people are interested to hear what happened in
the rpm ecosystem during the last year (and sometimes even years when it
was not covered before).

I think, it is mostly rather boring stuff and thus would want us to
waste as little precious Paris-time as possible, so I gathered as much
as I could in this email. This makes it possible to have a follow-up
session where we assume the below to be known and only cover remaining
open topics and questions.

I included 4 main parts:

1. rpm itself
2. our Open Build Service (OBS)
3. OBS-associated osc and obs-build tools
4. build-compare

1. rpm
====================

The "RPM Package Manager" is used in openSUSE, Fedora/RedHat, Mandriva,
Qubes OS and some smaller distributions - similar in spirit to dpkg. In
prior years we had already added most required patches that allowed to
produce bit-identical rpms. These are activated by 4 rpm macros,
documented in https://en.opensuse.org/openSUSE:Reproducible_Builds :

%source_date_epoch_from_changelog Y
%clamp_mtime_to_source_date_epoch Y
%use_source_date_epoch_as_buildtime Y
%_buildhost reproducible

Last year got two more patches merged to cover edge cases:

- https://github.com/rpm-software-management/rpm/pull/536
build: Make sure SOURCE_DATE_EPOCH is in the past
When a .changes file was updated in the morning, SOURCE_DATE_EPOCH was
set to 12:00 of the same day, which broke mtime clamping in rpm.

- https://github.com/rpm-software-management/rpm/pull/485
find-debuginfo.sh: sort output of find
My previous tests mostly did builds without debuginfo because that
increased build speed. But when wanting to verify official builds, the
'nachbau' replica-builds had to be done with debuginfo enabled and that
uncovered this one small ordering issue.

2. OBS
====================

OBS is the software we use to manage and build all our SUSE and openSUSE
distributions. There is a free public hosted version at
https://build.opensuse.org with significant build power and storage
space behind.
OBS also allows to produce packages for Debian, ArchLinux and
Fedora/RedHat based distributions from the same sources, making it very
attractive to projects that do not want to operate their own
infrastructure for building and hosting.
E.g. see Debian and Ubuntu download URLs in
https://www.openscad.org/downloads.html + https://lutris.net/downloads/

It is already possible to create verifiable, bit-identical binaries in OBS.
https://build.opensuse.org/project/prjconf/Application:ERP:GNUHealth:Factory
is already producing 23 such bit-reproducible verifiable binaries.

I have written about my verification efforts in
https://www.suse.com/c/?p=42014

Here, my verification efforts uncovered issues with artifacts that
undergo a 2nd signing step inside of OBS. This is related to
UEFI/secure-boot and affected grub2, xen and all kmp (out-of-tree kernel
module) packages.

https://github.com/openSUSE/pesign-obs-integration/pull/8 already fixed
some issues there, but there are more open:
https://github.com/openSUSE/pesign-obs-integration/issues/9

Without the private keys, the best result one could get there, would be
to strip off the signatures and produce bit-identical binaries to those.
TBD.

In https://github.com/openSUSE/open-build-service/issues/5784 I also
discovered issues with unreproducible noarch packages and implications
from how OBS handles them.
Internally, OBS uses arch-specific noarch packages for depending builds.
But OBS only ever publishes one noarch package from the first arch
listed in a project's metadata. So when a user wants to verify a build,
it might be impossible, because only a different build of the noarch
package is available than was originally used.
This could affect Debian (arch 'all') and others as well.

There have been previous discussions on this at
https://lists.reproducible-builds.org/pipermail/rb-general/2018-September/001165.html
continued in the following month
https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001191.html
Overall, this can involve issues that are hard to solve.
E.g. when floating-point or fixed point arithmetics do different
rounding in different architectures.
But that work did also make bugs visible, so is definitely useful.

3. osc and obs-build
====================

osc is the OBS CLI tool. In r-b I use it for "osc up" and "osc build".

obs-build is the component used in both OBS server-side builds and local
"osc build". It sets up a disposable build-root environment for use with
either kvm, lxc or chroot. It runs the build in there in a controlled
way. It thereby normalizes many sources of indeterminism (user, path,
umask, env). It also prevents internet access and ensures only the
packages listed as "BuildRequire" are there, in order to enforce
repeatable builds.

https://github.com/openSUSE/obs-build/pull/424/commits/f50a8c339470323f497809349155f0262ca09809
fixed ordering indeterminism for packages with multiple .spec files and
.changes files. Most effort went into the added test-suite to prevent
future regressions there.

https://github.com/openSUSE/obs-build/pull/469 disable ext4 dir_index
proposed, but not merged yet. Would decrease filesystem-related
indeterminism in official builds. This has trust-unrelated advantages in
saving build power for follow-up builds and re-publishs.

https://github.com/openSUSE/osc/pull/444 improve chroot
makes some r-b debugging easier

https://github.com/openSUSE/osc/pull/309 Speedup osc status
speeds up my test-cycles of update+build

https://github.com/openSUSE/obs-build/pull/328 / 326
https://github.com/openSUSE/osc/pull/290
allow to pass any option to the build command
Is used for passing the varying -cpu type option
and -rtc base option to kvm to build in the future without having to
mess with the system clock.

https://github.com/openSUSE/osc/issues/376 _multibuild bug
From my side, it is still unsolved and annoying, because my
stracebuild/autoprovenance and autoclassify tools rely on pulling in
packages from another repo into a build environment.

https://github.com/openSUSE/obs-build/pull/245
run kvm builds as non-root
was a security-related contribution there, only tangentially related

https://github.com/openSUSE/obs-build/pull/376
Support kvm builds on Debian
To make it easier to run my reproducibleopensuse tools on Debian servers.

4. build-compare
====================

build-compare is openSUSE's chimera written in bash, combining things
that strip-nondeterminism and diffoscope do.
The stripping is only temporary and not shipped in packages, though.
In OBS, its return value is used to decide if it is worth to publish a
new binary package or to trigger builds of depending packages.

I also use it in my reproducibleopensuse scripts when packages are not
bit-identical to see the diffs and to prioritize my fixing work on the
major diffs that cannot be auto-ignored.

https://github.com/openSUSE/build-compare/pull/17
Allow to disable many content filters

https://github.com/openSUSE/build-compare/pull/19
handle .egg as .zip

https://github.com/openSUSE/build-compare/pull/25
fix diff returning 0 by mistake

https://github.com/openSUSE/build-compare/pull/27
erroneously reported jar files as identical

https://github.com/openSUSE/build-compare/pull/28
bugfix, ignored diffs in rpm metadata

https://github.com/openSUSE/build-compare/pull/29
jar: always list files without date+size
helps to auto-ignore more ordering issues in .jar files

https://github.com/openSUSE/build-compare/pull/30
better quoting of strings to allow diffing files with spaces in its name

As can be seen, there have already been 3 bugs making it report results
as 'identical' even though there were significant differences. This is
why the final goal is to get rid of it and have proper bit-identical
build results.
Somewhere along the way, we could disable or drop filters to make the
remaining code more manageable.
The largest blocker to that goal are build hostnames and timestamps in
rpm metadata where people seem to still want the real thing for now.

This write-up only covers a small part of my r-b work - e.g. not the
200+ upstream r-b patches that help making various individual packages
reproducible for anyone, not just our rpm builds.

Ciao
Bernhard M.
-- 
openSUSE Developer and Cloud Software Developer and Sysadmin
SUSE LINUX GmbH, Nürnberg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://lists.reproducible-builds.org/pipermail/rb-general/attachments/20181205/639f0e2c/attachment.sig>