[rb-general] Reproducible Java builds with Maven

Hervé Boutemy hboutemy at apache.org
Sat Dec 22 18:37:32 CET 2018 

Le samedi 22 décembre 2018, 11:22:57 CET Arnout Engelen a écrit :
> On Sat, Dec 22, 2018 at 6:46 AM Hervé Boutemy <hboutemy at apache.org> wrote:
> > Le jeudi 20 décembre 2018, 19:13:48 CET Arnout Engelen a écrit :
> > > https://oss.sonatype.org/content/repositories/snapshots/com/scalapenos/s
> > > tam
> > > ina-core_2.12/0.1.5-SNAPSHOT/
> > 
> > There is only one key point that I'll note here: in the current example, I
> > could not find any pointer to the (re)buildable  source tarball or source
> > control management (I suppose Git, given pom content). When I read
> > "source=com.scalapenos:stamina-core_2.12", I expected to find stamina-
> > core_2.12-0.1.5-SNAPSHOT-source-release.zip file in the repository...
> 
> Perhaps we should follow the conventions of the scm section in the POM
> file format (https://maven.apache.org/pom.html#SCM)?
> 
> I think I'd rather point to VCS than to a source tarball, considering the
> VCS the 'source of truth' rather than any (also derived) source tarball.
Both options are useful: source tarballs are more stable over time than VCS.
That's why source tarballs are for example what Apache Software Foundation 
provides and archives, the VCS is just convenience and changes over time (from 
cvs to svn then git and perhpas something else in the future)

Then we just need to know that if you refer to Maven coordinates, it's 
expected to have -source-release.zip file in the repository

> 
> > such .buildinfo file should be created by any build tool for the JVM: is
> > there a wiki in [r-b] where we can maintain a list of build tools with
> > pointers to associated feature to generate this file?
> 
> I think we could create a 'JVM' section at
> https://reproducible-builds.org/docs/recording/ .
> It's not a wiki, but it's maintained in git
> (https://salsa.debian.org/reproducible-builds/reproducible-website)
> which would be convenient enough for me.
ok, I'll see how to contribute here

> 
> > > > For third-party attestations, perhaps we should set up our own
> > > > artifactory
> > > > or similar?
> > > 
> > > Any preferences?
> > 
> > I'm not convinced any classical classical Maven repository tool is
> > adapted,
> > since such tools expect to become read-only once the release is published:
> > rebuilders will publish attestations after the official release, in
> > append-
> > only mode I imagine = not a classical repository scenario (even when not a
> > Maven repository)
> 
> Actually I think sonatype already supports appending new artifacts to
> existing releases, as long as the filenames do not overlap
any repository tool can add new independant files (and I'm voluntarily 
avoiding any tool or company name): but do we really want to add one file per 
rebuilder?
And I'm not yet talking about the other key question: giving rights to add 
files for rebuild info, but not to other files.

> - but I'm
> not opposed to building something custom, either. Let's experiment
> and gather requirements.
yes, we really need to go slowly on this since it's clear that there is no 
easy solution and no common understanding on what is feasible with Maven 
repository format

> 
> > IMHO, a first step is to have us be able to rebuild packages from each
> > other and share our results on this mailing list
> 
> Let's! Could you check out tag v0.19 of
> https://github.com/raboof/sbt-reproducible-builds, run
> 'sbt ++2.12.6 clean reproducibleBuildsCertification' with openjdk8 and
> compare against
> the buildinfo on
> https://bintray.com/raboofje/sbt-plugins/sbt-reproducible-builds/0.19
> ?
ok, let's go: I'm not a Scala user, then I'll just try to rebuild as a pure 
Scala/sbt newbie 
I downloaded sbt-1.2.7, switched to my own JDK 8 install (which is an Oracle 
1.8.0_161) and launched the command you gave me

Here is my result:
$ cat target/scala-2.12/sbt-1.0/sbt-reproducible-builds_2.12-0.19+3-
c553fe74_herve-1545498708000.buildinfo
name=sbt-reproducible-builds
group_id=net.bzzt
artifact_id=sbt-reproducible-builds_2.12
version=0.19+3-c553fe74
build_architecture=all
source=net.bzzt:sbt-reproducible-builds_2.12
binary=net.bzzt:sbt-reproducible-builds_2.12
package=net.bzzt:sbt-reproducible-builds_2.12
java.version=1.8.0_161
sbt.version=1.2.3
scala.version=2.12.6
scala.binary-version=2.12
date=1545498708000
checksums_sha256.0.filename=sbt-reproducible-builds-0.19+3-c553fe74.jar
checksums_sha256.0.length=45876
checksums_sha256.0.checksum=86489f88b86ffdeb56e33b67d64c4b68aad8a5a67fcb4fe8190019c0d5c52948

here is my analysis:
- I did not get the same version than you: did you provide me the full exact 
command to launch?
- scala.version is useful for the command line to launch, but definitely 
scala.binary-version seems redundant
- I don't get the same date: is it useful to record?
- and at the end, I don't get the same checksum...

Looks like we need to change something somewhere 

Regards,

Hervé

> 
> > then we'll see how we can share our results on a more convenient location
> > than a mailing-list
> 
> Sounds good!
> 
> > Really a good step forward, let's continue!
> 
> Looking forward to it!
> 
> 
> Arnout
> 
> > > > > For checks done by users, that may reinforce the value for the info,
> > > > > this is another question: these won't go to Central, since adding
> > > > > new
> > > > > files is not something expected by Central. I think this
> > > > > reinforcement
> > > > > is something optional in our case: whoever wants to rebuild the
> > > > > artifact and check it against the official one, using info from the
> > > > > buildinfo to know how far the build environment was from the
> > > > > original
> > > > > "official" build info.
> > > > > 
> > > > > Le mardi 27 novembre 2018, 12:26:33 CET Arnout Engelen a écrit :
> > > > > > On Tue, Nov 27, 2018 at 9:58 AM Hervé BOUTEMY
> > > > > > <herve.boutemy at free.fr>
> > 
> > wrote:
> > > > > > > Yes, the Buildinfo seems an interesting part to work together.
> > > > > > > 
> > > > > > > I'm quite a noob on that, I'll need some pointers on basic info
> > > > > > > first.
> > > > > > 
> > > > > > Buildinfo is more of a general concept rather than a well-defined
> > > > > > specifciation, what it looks like exactly will probably vary per
> > > > > > 'ecosystem'.
> > > > > > 
> > > > > > Some words on its purpose and the conventions adopted by Debian,
> > > > > > Arch
> > > > > > and
> > > > > > Tails can be found at
> > > > > > https://reproducible-builds.org/docs/recording/
> > > > > > . For
> > > > > > Scala for now
> > > > > > I generally followed the layout from the Debian approach.
> > > > > > 
> > > > > > > Then it seems the way we look at this topic is quite different
> > > > > > > when
> > > > > > > you
> > > > > > > think as a Linux distribution manager or as a Java/Maven user
> > > > > > > publishing
> > > > > > > to Maven Central = what I'm looking first as Maven developer.
> > > > > > > 
> > > > > > > Should we start by defining a convention for anybody to publish
> > > > > > > a
> > > > > > > Buildinfo
> > > > > > 
> > > > > > That might make sense to me. For sbt-reproducible-builds, I kept
> > > > > > it
> > > > > > simple so far:
> > > > > > 
> > > > > > sbt-reproducible-builds_2.12_0.16_all_aengelen-1543317396000.build
> > > > > > info
> > > > > > .asc:
> > > > > > 
> > > > > > Architecture: all
> > > > > > JavaVersion: 1.8.0_171
> > > > > > Format: 1.8
> > > > > > ScalaVersion: 2.12.6
> > > > > > SbtVersion: 1.2.1
> > > > > > Version: 0.16
> > > > > > ScalaBinaryVersion: 2.12
> > > > > > Build-Architecture: all
> > > > > > 
> > > > > > Checksums-Sha256:
> > > > > >   cd577828ea6ca50c4b1256c57360a391a4904e7e2f4d189a20f735c619f468a3
> > > > > > 
> > > > > > 24627 sbt-reproducible-builds-0.16.jar
> > > > > > Source: sbt-reproducible-builds_2.12
> > > > > > Package: sbt-reproducible-builds_2.12
> > > > > > Binary: sbt-reproducible-builds_2.12
> > > > > > 
> > > > > > While this format allows multiple checksums in one attestation,
> > > > > > for
> > > > > > our ecosystem I think it would make sense to have an attestation
> > > > > > per jar, even for multi-project builds.
> > > > > > 
> > > > > > We should probably include things like groupId, artifactId, and
> > > > > > 'canonical'
> > > > > > upstream mvn/ivy repository?
> > > > > > 
> > > > > > > alongside his binary artifacts while publishing to Central?
> > > > > > 
> > > > > > Where to publish those is actually an interesting question: an
> > > > > > important
> > > > > > part of reproducible builds, IMO, is that it's not just the
> > > > > > package
> > > > > > maintainer that shares
> > > > > > an attestation, but third parties can share their independent
> > > > > > attestations as well,
> > > > > > adding to the confidence that the artifact indeed corresponds to
> > > > > > the
> > > > > > sources. Those third parties of course cannot publish those to
> > > > > > Central
> > > > > > alongside the 'official' package.
> > > > > > 
> > > > > > For that reason I kept the attestations separate for now - also
> > > > > > that
> > > > > > makes
> > > > > > it easier to iterate on the format a bit more without feeling
> > > > > > we're
> > > > > > 'setting things in stone' just yet.
> > > > > > 
> > > > > > 
> > > > > > Kind regards,
> > > > > > 
> > > > > > Arnout
> > > > > > 
> > > > > > > Le lundi 26 novembre 2018, 09:40:44 CET Arnout Engelen a écrit :
> > > > > > > > On Mon, Nov 26, 2018 at 9:08 AM Hervé Boutemy
> > > > > > > > <hboutemy at apache.org>
> > > > > 
> > > > > wrote:
> > > > > > > > > A few years ago, the work on this started and I created a
> > > > > > > > > Wiki
> > > > > > > > > page
> > > > > > > > > [1] at
> > > > > > > > > Maven to try to consolidate efforts from many isolated
> > > > > > > > > people I
> > > > > > > > > met
> > > > > > > > > who
> > > > > > > > > were interested in the topic: this Wiki page did not attract
> > > > > > > > > many
> > > > > > > > > contributions nor even discussions on Maven mailing lists, I
> > > > > > > > > hope this
> > > > > > > > > thread at reproducible- builds will help convergence between
> > > > > > > > > efforts.
> > > > > > > > 
> > > > > > > > Thanks, I wasn't aware of this page.
> > > > > > > > 
> > > > > > > > > And one thing that worries me is the variability introduced
> > > > > > > > > by
> > > > > > > > > the JDK
> > > > > > > > > version used: this one is quite generic to Java, I don't
> > > > > > > > > know if
> > > > > > > > > there
> > > > > > > > > is
> > > > > > > > > currently a global strategy that we could reuse.
> > > > > > > > 
> > > > > > > > I don't think there is much to do except including the JDK
> > > > > > > > version
> > > > > > > > in
> > > > > > > > the Buildinfo.
> > > > > > > > 
> > > > > > > > > Anybody interested in working together?
> > > > > > > > 
> > > > > > > > Quite possibly! I do a lot of programming in Scala (another
> > > > > > > > language
> > > > > > > > targeting the JVM),
> > > > > > > > and have been working on improving reproducibility there by
> > > > > > > > introducing a r-b plugin for
> > > > > > > > its sbt build system, sbt-reproducible-builds[1].
> > > > > > > > 
> > > > > > > > That uses the maven plugin you mentioned[2] as a basis for
> > > > > > > > post-processing the artifact
> > > > > > > > (though I'm planning to extract the logic to a separate
> > > > > > > > library).
> > > > > > > > I
> > > > > > > > agree it would be good to
> > > > > > > > fix more things 'at the source', but (as you mentioned above)
> > > > > > > > I
> > > > > > > > suspect some aspects such
> > > > > > > > as jar file generation will probably need post-processing for
> > > > > > > > the
> > > > > > > > foreseeable future.
> > > > > > > > 
> > > > > > > > It also has some (crude, very incomplete) features for
> > > > > > > > uploading
> > > > > > > > signed Buildinfo attestations
> > > > > > > > and comparing them with Buildinfo's uploaded by others.
> > > > > > > > 
> > > > > > > > This might be an area we could work together on: putting
> > > > > > > > together
> > > > > > > > the
> > > > > > > > conventions and
> > > > > > > > infrastructure to share Buildinfo attestations for JVM library
> > > > > > > > projects. In the JVM world
> > > > > > > > it is common to distribute libraries independently through
> > > > > > > > repositories such as Maven
> > > > > > > > Central, which might be a bit different from how Linux
> > > > > > > > distributions
> > > > > > > > work. Starting on
> > > > > > > > that would be interesting. So far I've been using
> > > > > > > > sbt-reproducible-builds with a (very)
> > > > > > > > simple web service to collect Buildinfo's,
> > > > > > > > reproducible-builds-certification-repository[3].
> > > > > > > > Unfortunately my example server is currently not running so I
> > > > > > > > can't
> > > > > > > > point to that
> > > > > > > > right now.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Kind regards,
> > > > > > > > 
> > > > > > > > Arnout
> > > > > > > > 
> > > > > > > > [1]: https://github.com/raboof/sbt-reproducible-builds
> > > > > > > > [2]: https://github.com/Zlika/reproducible-build-maven-plugin
> > > > > > > > [3]:
> > > > > > > > http://github.com/raboof/reproducible-builds-certification-rep
> > > > > > > > osit
> > > > > > > > ory
> > > 
> > > _______________________________________________
> > > rb-general at lists.reproducible-builds.org mailing list
> > > 
> > > To change your subscription options, visit
> > > https://lists.reproducible-builds.org/listinfo/rb-general.
> > > 
> > > To unsubscribe, send an email to
> > > rb-general-unsubscribe at lists.reproducible-builds.org.
> > 
> > _______________________________________________
> > rb-general at lists.reproducible-builds.org mailing list
> > 
> > To change your subscription options, visit
> > https://lists.reproducible-builds.org/listinfo/rb-general.
> > 
> > To unsubscribe, send an email to
> > rb-general-unsubscribe at lists.reproducible-builds.org.
> _______________________________________________
> rb-general at lists.reproducible-builds.org mailing list
> 
> To change your subscription options, visit
> https://lists.reproducible-builds.org/listinfo/rb-general.
> 
> To unsubscribe, send an email to
> rb-general-unsubscribe at lists.reproducible-builds.org.

More information about the rb-general mailing list