[rb-general] Reproducible Java builds with Maven

[Arnout Engelen]

On Sat, Dec 22, 2018 at 6:46 AM Hervé Boutemy <hboutemy at apache.org> wrote:
> Le jeudi 20 décembre 2018, 19:13:48 CET Arnout Engelen a écrit :
> > https://oss.sonatype.org/content/repositories/snapshots/com/scalapenos/stam
> > ina-core_2.12/0.1.5-SNAPSHOT/
>
> There is only one key point that I'll note here: in the current example, I
> could not find any pointer to the (re)buildable  source tarball or source
> control management (I suppose Git, given pom content). When I read
> "source=com.scalapenos:stamina-core_2.12", I expected to find stamina-
> core_2.12-0.1.5-SNAPSHOT-source-release.zip file in the repository...

Perhaps we should follow the conventions of the scm section in the POM
file format (https://maven.apache.org/pom.html#SCM)?

I think I'd rather point to VCS than to a source tarball, considering the
VCS the 'source of truth' rather than any (also derived) source tarball.

> such .buildinfo file should be created by any build tool for the JVM: is there
> a wiki in [r-b] where we can maintain a list of build tools with pointers to
> associated feature to generate this file?

I think we could create a 'JVM' section at
https://reproducible-builds.org/docs/recording/ .
It's not a wiki, but it's maintained in git
(https://salsa.debian.org/reproducible-builds/reproducible-website)
which would be convenient enough for me.

> > > For third-party attestations, perhaps we should set up our own artifactory
> > > or similar?
> >
> > Any preferences?
> I'm not convinced any classical classical Maven repository tool is adapted,
> since such tools expect to become read-only once the release is published:
> rebuilders will publish attestations after the official release, in append-
> only mode I imagine = not a classical repository scenario (even when not a
> Maven repository)

Actually I think sonatype already supports appending new artifacts to
existing releases, as long as the filenames do not overlap - but I'm
not opposed to building something custom, either. Let's experiment
and gather requirements.

> IMHO, a first step is to have us be able to rebuild packages from each other
> and share our results on this mailing list

Let's! Could you check out tag v0.19 of
https://github.com/raboof/sbt-reproducible-builds, run
'sbt ++2.12.6 clean reproducibleBuildsCertification' with openjdk8 and
compare against
the buildinfo on
https://bintray.com/raboofje/sbt-plugins/sbt-reproducible-builds/0.19
?

> then we'll see how we can share our results on a more convenient location than
> a mailing-list

Sounds good!

> Really a good step forward, let's continue!

Looking forward to it!


Arnout

> > > > For checks done by users, that may reinforce the value for the info,
> > > > this is another question: these won't go to Central, since adding new
> > > > files is not something expected by Central. I think this reinforcement
> > > > is something optional in our case: whoever wants to rebuild the
> > > > artifact and check it against the official one, using info from the
> > > > buildinfo to know how far the build environment was from the original
> > > > "official" build info.
> > > >
> > > > Le mardi 27 novembre 2018, 12:26:33 CET Arnout Engelen a écrit :
> > > > > On Tue, Nov 27, 2018 at 9:58 AM Hervé BOUTEMY <herve.boutemy at free.fr>
> wrote:
> > > > > > Yes, the Buildinfo seems an interesting part to work together.
> > > > > >
> > > > > > I'm quite a noob on that, I'll need some pointers on basic info
> > > > > > first.
> > > > >
> > > > > Buildinfo is more of a general concept rather than a well-defined
> > > > > specifciation, what it looks like exactly will probably vary per
> > > > > 'ecosystem'.
> > > > >
> > > > > Some words on its purpose and the conventions adopted by Debian, Arch
> > > > > and
> > > > > Tails can be found at https://reproducible-builds.org/docs/recording/
> > > > > . For
> > > > > Scala for now
> > > > > I generally followed the layout from the Debian approach.
> > > > >
> > > > > > Then it seems the way we look at this topic is quite different when
> > > > > > you
> > > > > > think as a Linux distribution manager or as a Java/Maven user
> > > > > > publishing
> > > > > > to Maven Central = what I'm looking first as Maven developer.
> > > > > >
> > > > > > Should we start by defining a convention for anybody to publish a
> > > > > > Buildinfo
> > > > >
> > > > > That might make sense to me. For sbt-reproducible-builds, I kept it
> > > > > simple so far:
> > > > >
> > > > > sbt-reproducible-builds_2.12_0.16_all_aengelen-1543317396000.buildinfo
> > > > > .asc:
> > > > >
> > > > > Architecture: all
> > > > > JavaVersion: 1.8.0_171
> > > > > Format: 1.8
> > > > > ScalaVersion: 2.12.6
> > > > > SbtVersion: 1.2.1
> > > > > Version: 0.16
> > > > > ScalaBinaryVersion: 2.12
> > > > > Build-Architecture: all
> > > > >
> > > > > Checksums-Sha256:
> > > > >   cd577828ea6ca50c4b1256c57360a391a4904e7e2f4d189a20f735c619f468a3
> > > > >
> > > > > 24627 sbt-reproducible-builds-0.16.jar
> > > > > Source: sbt-reproducible-builds_2.12
> > > > > Package: sbt-reproducible-builds_2.12
> > > > > Binary: sbt-reproducible-builds_2.12
> > > > >
> > > > > While this format allows multiple checksums in one attestation, for
> > > > > our ecosystem I think it would make sense to have an attestation
> > > > > per jar, even for multi-project builds.
> > > > >
> > > > > We should probably include things like groupId, artifactId, and
> > > > > 'canonical'
> > > > > upstream mvn/ivy repository?
> > > > >
> > > > > > alongside his binary artifacts while publishing to Central?
> > > > >
> > > > > Where to publish those is actually an interesting question: an
> > > > > important
> > > > > part of reproducible builds, IMO, is that it's not just the package
> > > > > maintainer that shares
> > > > > an attestation, but third parties can share their independent
> > > > > attestations as well,
> > > > > adding to the confidence that the artifact indeed corresponds to the
> > > > > sources. Those third parties of course cannot publish those to Central
> > > > > alongside the 'official' package.
> > > > >
> > > > > For that reason I kept the attestations separate for now - also that
> > > > > makes
> > > > > it easier to iterate on the format a bit more without feeling we're
> > > > > 'setting things in stone' just yet.
> > > > >
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Arnout
> > > > >
> > > > > > Le lundi 26 novembre 2018, 09:40:44 CET Arnout Engelen a écrit :
> > > > > > > On Mon, Nov 26, 2018 at 9:08 AM Hervé Boutemy
> > > > > > > <hboutemy at apache.org>
> > > >
> > > > wrote:
> > > > > > > > A few years ago, the work on this started and I created a Wiki
> > > > > > > > page
> > > > > > > > [1] at
> > > > > > > > Maven to try to consolidate efforts from many isolated people I
> > > > > > > > met
> > > > > > > > who
> > > > > > > > were interested in the topic: this Wiki page did not attract
> > > > > > > > many
> > > > > > > > contributions nor even discussions on Maven mailing lists, I
> > > > > > > > hope this
> > > > > > > > thread at reproducible- builds will help convergence between
> > > > > > > > efforts.
> > > > > > >
> > > > > > > Thanks, I wasn't aware of this page.
> > > > > > >
> > > > > > > > And one thing that worries me is the variability introduced by
> > > > > > > > the JDK
> > > > > > > > version used: this one is quite generic to Java, I don't know if
> > > > > > > > there
> > > > > > > > is
> > > > > > > > currently a global strategy that we could reuse.
> > > > > > >
> > > > > > > I don't think there is much to do except including the JDK version
> > > > > > > in
> > > > > > > the Buildinfo.
> > > > > > >
> > > > > > > > Anybody interested in working together?
> > > > > > >
> > > > > > > Quite possibly! I do a lot of programming in Scala (another
> > > > > > > language
> > > > > > > targeting the JVM),
> > > > > > > and have been working on improving reproducibility there by
> > > > > > > introducing a r-b plugin for
> > > > > > > its sbt build system, sbt-reproducible-builds[1].
> > > > > > >
> > > > > > > That uses the maven plugin you mentioned[2] as a basis for
> > > > > > > post-processing the artifact
> > > > > > > (though I'm planning to extract the logic to a separate library).
> > > > > > > I
> > > > > > > agree it would be good to
> > > > > > > fix more things 'at the source', but (as you mentioned above) I
> > > > > > > suspect some aspects such
> > > > > > > as jar file generation will probably need post-processing for the
> > > > > > > foreseeable future.
> > > > > > >
> > > > > > > It also has some (crude, very incomplete) features for uploading
> > > > > > > signed Buildinfo attestations
> > > > > > > and comparing them with Buildinfo's uploaded by others.
> > > > > > >
> > > > > > > This might be an area we could work together on: putting together
> > > > > > > the
> > > > > > > conventions and
> > > > > > > infrastructure to share Buildinfo attestations for JVM library
> > > > > > > projects. In the JVM world
> > > > > > > it is common to distribute libraries independently through
> > > > > > > repositories such as Maven
> > > > > > > Central, which might be a bit different from how Linux
> > > > > > > distributions
> > > > > > > work. Starting on
> > > > > > > that would be interesting. So far I've been using
> > > > > > > sbt-reproducible-builds with a (very)
> > > > > > > simple web service to collect Buildinfo's,
> > > > > > > reproducible-builds-certification-repository[3].
> > > > > > > Unfortunately my example server is currently not running so I
> > > > > > > can't
> > > > > > > point to that
> > > > > > > right now.
> > > > > > >
> > > > > > >
> > > > > > > Kind regards,
> > > > > > >
> > > > > > > Arnout
> > > > > > >
> > > > > > > [1]: https://github.com/raboof/sbt-reproducible-builds
> > > > > > > [2]: https://github.com/Zlika/reproducible-build-maven-plugin
> > > > > > > [3]:
> > > > > > > http://github.com/raboof/reproducible-builds-certification-reposit
> > > > > > > ory
> >
> > _______________________________________________
> > rb-general at lists.reproducible-builds.org mailing list
> >
> > To change your subscription options, visit
> > https://lists.reproducible-builds.org/listinfo/rb-general.
> >
> > To unsubscribe, send an email to
> > rb-general-unsubscribe at lists.reproducible-builds.org.
>
>
>
>
> _______________________________________________
> rb-general at lists.reproducible-builds.org mailing list
>
> To change your subscription options, visit https://lists.reproducible-builds.org/listinfo/rb-general.
>
> To unsubscribe, send an email to rb-general-unsubscribe at lists.reproducible-builds.org.