[rb-general] Reproducible Java builds with Maven

Hervé Boutemy hboutemy at apache.org
Sat Dec 22 06:46:31 CET 2018 

Le jeudi 20 décembre 2018, 19:13:48 CET Arnout Engelen a écrit :
> On Wed, Dec 19, 2018 at 10:15 AM Arnout Engelen <arnout at bzzt.net> wrote:
> > On Tue, Nov 27, 2018 at 4:02 PM Hervé Boutemy <hboutemy at apache.org> wrote:
> > > On the question "where to publish", I think we have no choice when
> > > artifacts go to Maven Central: there is one "official" build that goes
> > > into Central, then this build requires its build descriptor created
> > > during the "official" build. And this "official" build descriptor will
> > > be signed with the same pgp key as the "official" artifact.
> > 
> > Yes, makes sense.
> > 
> > I have updated sbt-reproducible-builds to produce the buildinfo in
> > .properties file format and publish+sign it along with the other
> > artifacts.
> 
> Seems to work handsomely when publishing to sonatype (which releases
> to maven central) as well:
> 
>  
> https://oss.sonatype.org/content/repositories/snapshots/com/scalapenos/stam
> ina-core_2.12/0.1.5-SNAPSHOT/

yeah, great!

this stamina-core_2.12-0.1.5-SNAPSHOT.buildinfo is exactly what I intend to 
create with a Maven plugin: there are many details on the content that we'll 
need to discuss => that deserves another thread

There is only one key point that I'll note here: in the current example, I 
could not find any pointer to the (re)buildable  source tarball or source 
control management (I suppose Git, given pom content). When I read 
"source=com.scalapenos:stamina-core_2.12", I expected to find stamina-
core_2.12-0.1.5-SNAPSHOT-source-release.zip file in the repository...


such .buildinfo file should be created by any build tool for the JVM: is there 
a wiki in [r-b] where we can maintain a list of build tools with pointers to 
associated feature to generate this file?
We could also maintain in this wiki page a formal description of the file 
content.
And perhaps a list of repository formats also, since Maven repository format 
is for sure a first target given Maven Central repository usefulness in the 
JVM ecosystem, but there may be other repository formats that may be useful.

> > For third-party attestations, perhaps we should set up our own artifactory
> > or similar?
> 
> Any preferences?
I'm not convinced any classical classical Maven repository tool is adapted, 
since such tools expect to become read-only once the release is published: 
rebuilders will publish attestations after the official release, in append-
only mode I imagine = not a classical repository scenario (even when not a 
Maven repository)

IMHO, a first step is to have us be able to rebuild packages from each other 
and share our results on this mailing list
then we'll see how we can share our results on a more convenient location than 
a mailing-list 


Really a good step forward, let's continue!

Regards,

Hervé

> 
> 
> Arnout
> 
> > > For checks done by users, that may reinforce the value for the info,
> > > this is another question: these won't go to Central, since adding new
> > > files is not something expected by Central. I think this reinforcement
> > > is something optional in our case: whoever wants to rebuild the
> > > artifact and check it against the official one, using info from the
> > > buildinfo to know how far the build environment was from the original
> > > "official" build info.
> > > 
> > > Le mardi 27 novembre 2018, 12:26:33 CET Arnout Engelen a écrit :
> > > > On Tue, Nov 27, 2018 at 9:58 AM Hervé BOUTEMY <herve.boutemy at free.fr> 
wrote:
> > > > > Yes, the Buildinfo seems an interesting part to work together.
> > > > > 
> > > > > I'm quite a noob on that, I'll need some pointers on basic info
> > > > > first.
> > > > 
> > > > Buildinfo is more of a general concept rather than a well-defined
> > > > specifciation, what it looks like exactly will probably vary per
> > > > 'ecosystem'.
> > > > 
> > > > Some words on its purpose and the conventions adopted by Debian, Arch
> > > > and
> > > > Tails can be found at https://reproducible-builds.org/docs/recording/
> > > > . For
> > > > Scala for now
> > > > I generally followed the layout from the Debian approach.
> > > > 
> > > > > Then it seems the way we look at this topic is quite different when
> > > > > you
> > > > > think as a Linux distribution manager or as a Java/Maven user
> > > > > publishing
> > > > > to Maven Central = what I'm looking first as Maven developer.
> > > > > 
> > > > > Should we start by defining a convention for anybody to publish a
> > > > > Buildinfo
> > > > 
> > > > That might make sense to me. For sbt-reproducible-builds, I kept it
> > > > simple so far:
> > > > 
> > > > sbt-reproducible-builds_2.12_0.16_all_aengelen-1543317396000.buildinfo
> > > > .asc:
> > > > 
> > > > Architecture: all
> > > > JavaVersion: 1.8.0_171
> > > > Format: 1.8
> > > > ScalaVersion: 2.12.6
> > > > SbtVersion: 1.2.1
> > > > Version: 0.16
> > > > ScalaBinaryVersion: 2.12
> > > > Build-Architecture: all
> > > > 
> > > > Checksums-Sha256:
> > > >   cd577828ea6ca50c4b1256c57360a391a4904e7e2f4d189a20f735c619f468a3
> > > > 
> > > > 24627 sbt-reproducible-builds-0.16.jar
> > > > Source: sbt-reproducible-builds_2.12
> > > > Package: sbt-reproducible-builds_2.12
> > > > Binary: sbt-reproducible-builds_2.12
> > > > 
> > > > While this format allows multiple checksums in one attestation, for
> > > > our ecosystem I think it would make sense to have an attestation
> > > > per jar, even for multi-project builds.
> > > > 
> > > > We should probably include things like groupId, artifactId, and
> > > > 'canonical'
> > > > upstream mvn/ivy repository?
> > > > 
> > > > > alongside his binary artifacts while publishing to Central?
> > > > 
> > > > Where to publish those is actually an interesting question: an
> > > > important
> > > > part of reproducible builds, IMO, is that it's not just the package
> > > > maintainer that shares
> > > > an attestation, but third parties can share their independent
> > > > attestations as well,
> > > > adding to the confidence that the artifact indeed corresponds to the
> > > > sources. Those third parties of course cannot publish those to Central
> > > > alongside the 'official' package.
> > > > 
> > > > For that reason I kept the attestations separate for now - also that
> > > > makes
> > > > it easier to iterate on the format a bit more without feeling we're
> > > > 'setting things in stone' just yet.
> > > > 
> > > > 
> > > > Kind regards,
> > > > 
> > > > Arnout
> > > > 
> > > > > Le lundi 26 novembre 2018, 09:40:44 CET Arnout Engelen a écrit :
> > > > > > On Mon, Nov 26, 2018 at 9:08 AM Hervé Boutemy
> > > > > > <hboutemy at apache.org>
> > > 
> > > wrote:
> > > > > > > A few years ago, the work on this started and I created a Wiki
> > > > > > > page
> > > > > > > [1] at
> > > > > > > Maven to try to consolidate efforts from many isolated people I
> > > > > > > met
> > > > > > > who
> > > > > > > were interested in the topic: this Wiki page did not attract
> > > > > > > many
> > > > > > > contributions nor even discussions on Maven mailing lists, I
> > > > > > > hope this
> > > > > > > thread at reproducible- builds will help convergence between
> > > > > > > efforts.
> > > > > > 
> > > > > > Thanks, I wasn't aware of this page.
> > > > > > 
> > > > > > > And one thing that worries me is the variability introduced by
> > > > > > > the JDK
> > > > > > > version used: this one is quite generic to Java, I don't know if
> > > > > > > there
> > > > > > > is
> > > > > > > currently a global strategy that we could reuse.
> > > > > > 
> > > > > > I don't think there is much to do except including the JDK version
> > > > > > in
> > > > > > the Buildinfo.
> > > > > > 
> > > > > > > Anybody interested in working together?
> > > > > > 
> > > > > > Quite possibly! I do a lot of programming in Scala (another
> > > > > > language
> > > > > > targeting the JVM),
> > > > > > and have been working on improving reproducibility there by
> > > > > > introducing a r-b plugin for
> > > > > > its sbt build system, sbt-reproducible-builds[1].
> > > > > > 
> > > > > > That uses the maven plugin you mentioned[2] as a basis for
> > > > > > post-processing the artifact
> > > > > > (though I'm planning to extract the logic to a separate library).
> > > > > > I
> > > > > > agree it would be good to
> > > > > > fix more things 'at the source', but (as you mentioned above) I
> > > > > > suspect some aspects such
> > > > > > as jar file generation will probably need post-processing for the
> > > > > > foreseeable future.
> > > > > > 
> > > > > > It also has some (crude, very incomplete) features for uploading
> > > > > > signed Buildinfo attestations
> > > > > > and comparing them with Buildinfo's uploaded by others.
> > > > > > 
> > > > > > This might be an area we could work together on: putting together
> > > > > > the
> > > > > > conventions and
> > > > > > infrastructure to share Buildinfo attestations for JVM library
> > > > > > projects. In the JVM world
> > > > > > it is common to distribute libraries independently through
> > > > > > repositories such as Maven
> > > > > > Central, which might be a bit different from how Linux
> > > > > > distributions
> > > > > > work. Starting on
> > > > > > that would be interesting. So far I've been using
> > > > > > sbt-reproducible-builds with a (very)
> > > > > > simple web service to collect Buildinfo's,
> > > > > > reproducible-builds-certification-repository[3].
> > > > > > Unfortunately my example server is currently not running so I
> > > > > > can't
> > > > > > point to that
> > > > > > right now.
> > > > > > 
> > > > > > 
> > > > > > Kind regards,
> > > > > > 
> > > > > > Arnout
> > > > > > 
> > > > > > [1]: https://github.com/raboof/sbt-reproducible-builds
> > > > > > [2]: https://github.com/Zlika/reproducible-build-maven-plugin
> > > > > > [3]:
> > > > > > http://github.com/raboof/reproducible-builds-certification-reposit
> > > > > > ory
> 
> _______________________________________________
> rb-general at lists.reproducible-builds.org mailing list
> 
> To change your subscription options, visit
> https://lists.reproducible-builds.org/listinfo/rb-general.
> 
> To unsubscribe, send an email to
> rb-general-unsubscribe at lists.reproducible-builds.org.

More information about the rb-general mailing list