[rb-general] Yocto Project interest in reproducible builds

[Richard Purdie]

On Sat, 2016-12-10 at 21:59 +0000, Holger Levsen wrote:
> On Tue, Nov 15, 2016 at 11:30:36AM +0000, Richard Purdie wrote:
> > I talked with Holger at ELC-E. The Yocto Project has a strong
> > interest
> > in reproducible builds, its something we've been working on for a
> > while
> > but obviously there is much work remaining.
> yay!
> 
> > 
> > I'm not sure how to collaborate, our policy is where we have
> > patches,
> > to get them upstream if we can and that obviously helps everyone.
> yay!
>  
> > 
> > I would be happy to have the Yocto Project listed as supporting
> > reproducible builds and to see if there are ways we can collaborate
> > further.
> do you have a page explaining Yocto and reproducible builds or some
> such? 

We haven't spelt this out specifically (as yet) but we have a few
things going on:

a) Firstly, the project aims to have builds which are entirely
reproducible in that if you run a build today, then run that same
configuration X time in the future, the binaries you get out the build
should be binary identical, except for timestamps.

This implies that the host system you run the build on and the path you
run the build in should not affect the target system output.

We're not here today with this but where we identify problems, we aim
to fix them. We plan to do more testing to find issues.

b) We're aiming to have a recipe build specific sysroot in our next
release (April) instead of a shared sysroot. I've written from proof of
concept patches which are in review at the moment.

c) We base our builds on hashes of input metadata, reusing the outputs
if the inputs are the same (our sstate mechanism). We have some pretty
interesting technology for calculating those. 

This is relevant as our sstate files need to be reusable regardless of
build path and can be target or native binaries, each brings some
challenges. We have mechanisms for working around hardcoded paths but
would prefer to remove the need for them.

d) There is a related problem of knowing when the input has changed,
had the output changed? This is useful in the context of package feeds
amongst other things to know if we should update them or not. We have
binary build comparison tools at an early stage of development to allow
us to reduce unnecessary churn in package feeds but its not been looked
at for a while and I think your tools may have surpassed ours now, we
need to look at that.

e) We have SDKs which also need to be relocatable and run anywhere
they're installed to. We actually include our own C library to do this
in a "run anywhere" scenario and do some interesting relocation tricks.


So in many ways our overall project objectives are already aligned,
we've just been doing it without spelling it out. There are various
reasons this is becoming something we as a project need to focus on
though and make sure we get right.

One big advantage we have compared with something like debian is that
that we can rebuild everything from the ground up for our core test
images in around 20-60 mins depending on the build hardware available.
Its therefore very easy for us to compare two build configurations side
by side, spot differences and then also test that differences are
resolved.

I have personally been a bit distracted by b) above which needs to be
done early in our release cycle. I'm hoping once that is sorted out we
can start evaluating where we stand and seeing if/as/where we can
collaborate more in some of the above areas though.

Is there anything specific you want to know or any areas you think we
may be able to help specifically?


> Also, (again) we would love to have you or someone else from the
> yocto project
> at https://reproducible-builds.org/events/berlin2016/ (and still!
> even
> though it's now pretty short notice…)

Personally, I've ended up with a conflict this week and can't make it
which I'm disappointed about. I did try and find someone else but
travel budget is proving very hard to find at the moment too.

I am raising the profile of reproducibility within the project and you
should see some people starting to get involved in various ways over
the coming weeks/months though.

I haven't forgotten about adding the project to the list of supporters
on the website and am also planning to join the irc meeting. Are there
any other steps you think we could take?

Cheers,

Richard