[rb-general] Reproducible Java builds with Maven

[Arnout Engelen]

On Tue, Nov 27, 2018 at 4:02 PM Hervé Boutemy <hboutemy at apache.org> wrote:
> On the question "where to publish", I think we have no choice when artifacts
> go to Maven Central: there is one "official" build that goes into Central,
> then this build requires its build descriptor created during the "official"
> build. And this "official" build descriptor will be signed with the same pgp
> key as the "official" artifact.

Yes, makes sense.

I have updated sbt-reproducible-builds to produce the buildinfo in .properties
file format and publish+sign it along with the other artifacts. This particular
project is published to bintray (this is common for sbt plugins), I
will test with
a library that is published to maven central later.

The published project looks like this:
https://bintray.com/raboofje/sbt-plugins/sbt-reproducible-builds/0.19#files/net.bzzt%2Fsbt-reproducible-builds%2Fscala_2.12%2Fsbt_1.0%2F0.19
('Files' tab)

For third-party attestations, perhaps we should set up our own artifactory or
similar?


Arnout

> For checks done by users, that may reinforce the value for the info, this is
> another question: these won't go to Central, since adding new files is not
> something expected by Central. I think this reinforcement is something
> optional in our case: whoever wants to rebuild the artifact and check it
> against the official one, using info from the buildinfo to know how far the
> build environment was from the original "official" build info.

> Le mardi 27 novembre 2018, 12:26:33 CET Arnout Engelen a écrit :
> > On Tue, Nov 27, 2018 at 9:58 AM Hervé BOUTEMY <herve.boutemy at free.fr> wrote:
> > > Yes, the Buildinfo seems an interesting part to work together.
> > >
> > > I'm quite a noob on that, I'll need some pointers on basic info first.
> >
> > Buildinfo is more of a general concept rather than a well-defined
> > specifciation, what it looks like exactly will probably vary per
> > 'ecosystem'.
> >
> > Some words on its purpose and the conventions adopted by Debian, Arch and
> > Tails can be found at https://reproducible-builds.org/docs/recording/ . For
> > Scala for now
> > I generally followed the layout from the Debian approach.
> >
> > > Then it seems the way we look at this topic is quite different when you
> > > think as a Linux distribution manager or as a Java/Maven user publishing
> > > to Maven Central = what I'm looking first as Maven developer.
> > >
> > > Should we start by defining a convention for anybody to publish a
> > > Buildinfo
> >
> > That might make sense to me. For sbt-reproducible-builds, I kept it
> > simple so far:
> >
> > sbt-reproducible-builds_2.12_0.16_all_aengelen-1543317396000.buildinfo.asc:
> >
> > Architecture: all
> > JavaVersion: 1.8.0_171
> > Format: 1.8
> > ScalaVersion: 2.12.6
> > SbtVersion: 1.2.1
> > Version: 0.16
> > ScalaBinaryVersion: 2.12
> > Build-Architecture: all
> > Checksums-Sha256:
> >   cd577828ea6ca50c4b1256c57360a391a4904e7e2f4d189a20f735c619f468a3
> > 24627 sbt-reproducible-builds-0.16.jar
> > Source: sbt-reproducible-builds_2.12
> > Package: sbt-reproducible-builds_2.12
> > Binary: sbt-reproducible-builds_2.12
> >
> > While this format allows multiple checksums in one attestation, for
> > our ecosystem I think it would make sense to have an attestation
> > per jar, even for multi-project builds.
> >
> > We should probably include things like groupId, artifactId, and 'canonical'
> > upstream mvn/ivy repository?
> >
> > > alongside his binary artifacts while publishing to Central?
> >
> > Where to publish those is actually an interesting question: an important
> > part of reproducible builds, IMO, is that it's not just the package
> > maintainer that shares
> > an attestation, but third parties can share their independent
> > attestations as well,
> > adding to the confidence that the artifact indeed corresponds to the
> > sources. Those third parties of course cannot publish those to Central
> > alongside the 'official' package.
> >
> > For that reason I kept the attestations separate for now - also that makes
> > it easier to iterate on the format a bit more without feeling we're
> > 'setting things in stone' just yet.
> >
> >
> > Kind regards,
> >
> > Arnout
> >
> > > Le lundi 26 novembre 2018, 09:40:44 CET Arnout Engelen a écrit :
> > > > On Mon, Nov 26, 2018 at 9:08 AM Hervé Boutemy <hboutemy at apache.org>
> wrote:
> > > > > A few years ago, the work on this started and I created a Wiki page
> > > > > [1] at
> > > > > Maven to try to consolidate efforts from many isolated people I met
> > > > > who
> > > > > were interested in the topic: this Wiki page did not attract many
> > > > > contributions nor even discussions on Maven mailing lists, I hope this
> > > > > thread at reproducible- builds will help convergence between efforts.
> > > >
> > > > Thanks, I wasn't aware of this page.
> > > >
> > > > > And one thing that worries me is the variability introduced by the JDK
> > > > > version used: this one is quite generic to Java, I don't know if there
> > > > > is
> > > > > currently a global strategy that we could reuse.
> > > >
> > > > I don't think there is much to do except including the JDK version in
> > > > the Buildinfo.
> > > >
> > > > > Anybody interested in working together?
> > > >
> > > > Quite possibly! I do a lot of programming in Scala (another language
> > > > targeting the JVM),
> > > > and have been working on improving reproducibility there by
> > > > introducing a r-b plugin for
> > > > its sbt build system, sbt-reproducible-builds[1].
> > > >
> > > > That uses the maven plugin you mentioned[2] as a basis for
> > > > post-processing the artifact
> > > > (though I'm planning to extract the logic to a separate library). I
> > > > agree it would be good to
> > > > fix more things 'at the source', but (as you mentioned above) I
> > > > suspect some aspects such
> > > > as jar file generation will probably need post-processing for the
> > > > foreseeable future.
> > > >
> > > > It also has some (crude, very incomplete) features for uploading
> > > > signed Buildinfo attestations
> > > > and comparing them with Buildinfo's uploaded by others.
> > > >
> > > > This might be an area we could work together on: putting together the
> > > > conventions and
> > > > infrastructure to share Buildinfo attestations for JVM library
> > > > projects. In the JVM world
> > > > it is common to distribute libraries independently through
> > > > repositories such as Maven
> > > > Central, which might be a bit different from how Linux distributions
> > > > work. Starting on
> > > > that would be interesting. So far I've been using
> > > > sbt-reproducible-builds with a (very)
> > > > simple web service to collect Buildinfo's,
> > > > reproducible-builds-certification-repository[3].
> > > > Unfortunately my example server is currently not running so I can't
> > > > point to that
> > > > right now.
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Arnout
> > > >
> > > > [1]: https://github.com/raboof/sbt-reproducible-builds
> > > > [2]: https://github.com/Zlika/reproducible-build-maven-plugin
> > > > [3]:
> > > > http://github.com/raboof/reproducible-builds-certification-repository