[rb-general] Reproducible Java builds with Maven
hboutemy at apache.org
Mon Nov 26 23:51:57 CET 2018
looks like we are many interested people, just missed a common place to
discuss and work together. This seems to be fixed :)
Le lundi 26 novembre 2018, 10:00:47 CET Bernhard M. Wiedemann a écrit :
> Hi Hervé,
> thanks for raising this topic.
> On 26/11/2018 09.08, Hervé Boutemy wrote:
> > Currently, there is a plugin that does the job as archive post-processing,
> > but one intent would be to have native reproducible builds when that does
> > not hurt other objectives, like performance or anything else: we did
> > already change many plugins that for sure improve the situation, but I
> > don't have a real view on how far we are from native reproducible builds.
> I think, one major consideration is how many code-places touch .jar
> files, because if we want to get rid of post-processing, we might need
> to patch all of them to have normalized mtime values in the .jar.
I just published a global Maven sources overview:
Jar files are created by packaging plugins, using shared archiver, that uses
Plexus archiver component: this gives the global view on what will be impacted
at Maven level. Of course, there may be some additional plugins in the
ecosystem, to update later.
> > I know that one key step will be to have reproducible archives creation
> > (with reproducible timestamps and order), which won't be easy without
> > sacrificing performance (ConcurrentJarCreator was created to improve
> > performance, but adds even more variability in order...)
> Just because you use parallelism does not mean, your output has to vary.
> E.g. you can play with 'pigz' (parallel gzip):
> dd if=/dev/urandom bs=1M count=100 of=input
> time pigz -p 2 -nc1 input | md5sum
> notice how it produces bit-identical results, independent of the number
> of threads (given with -p).
great, this proves this can be done: "just" need to be correctly
> > Anybody interested in working together?
> I'm interested, but so far kept my fingers off Java as much as I could,
> because I felt it is hard to work with.
yes, it's a matter of taste :)
> With Maven, there are also major issues in a related domain, that is
> called "bootstrappable builds".
> With openSUSE we are doing all builds offline to ensure that we can
> repeat builds later (without worry about offline or hacked servers), but
> for maven this often meant we had to download 300 MB of someone else's
> binaries to use in the build.
> is an ongoing effort to build more things from source.
> But bootstrapping would be much easier if maven and its plugins could be
> built with ant, make or some other pre-existing bootstrappable build system.
> Do you know if such is possible or who would be interested in making it
we had an Ant build.xml until Maven 3.3.9
that we stopped maintaining for Maven 3.5
But even that one was using prebuilt binaries for Modello, Plexus utils and
Fully bootstrapapable build never has really been a Maven primary objective...
> Bernhard M.
More information about the rb-general