[rb-general] Reproducible Java builds with Maven

[Hervé Boutemy]

Hi all,

looks like we are many interested people, just missed a common place to 
discuss and work together. This seems to be fixed :)

Le lundi 26 novembre 2018, 10:00:47 CET Bernhard M. Wiedemann a écrit :
> Hi Hervé,
> 
> thanks for raising this topic.
> 
> On 26/11/2018 09.08, Hervé Boutemy wrote:
> > Currently, there is a plugin that does the job as archive post-processing,
> > but one intent would be to have native reproducible builds when that does
> > not hurt other objectives, like performance or anything else: we did
> > already change many plugins that for sure improve the situation, but I
> > don't have a real view on how far we are from native reproducible builds.
> 
> I think, one major consideration is how many code-places touch .jar
> files, because if we want to get rid of post-processing, we might need
> to patch all of them to have normalized mtime values in the .jar.
I just published a global Maven sources overview:
https://maven.apache.org/scm.html
Jar files are created by packaging plugins, using shared archiver, that uses 
Plexus archiver component: this gives the global view on what will be impacted 
at Maven level. Of course, there may be some additional plugins in the 
ecosystem, to update later.

> 
> > I know that one key step will be to have reproducible archives creation
> > (with reproducible timestamps and order), which won't be easy without
> > sacrificing performance (ConcurrentJarCreator was created to improve
> > performance, but adds even more variability in order...)
> 
> Just because you use parallelism does not mean, your output has to vary.
> E.g. you can play with 'pigz' (parallel gzip):
> dd if=/dev/urandom bs=1M count=100 of=input
> time pigz -p 2 -nc1 input | md5sum
> 
> notice how it produces bit-identical results, independent of the number
> of threads (given with -p).
great, this proves this can be done: "just" need to be correctly 
implemented...

> 
> > Anybody interested in working together?
> 
> I'm interested, but so far kept my fingers off Java as much as I could,
> because I felt it is hard to work with.
yes, it's a matter of taste :)

> 
> With Maven, there are also major issues in a related domain, that is
> called "bootstrappable builds".
> With openSUSE we are doing all builds offline to ensure that we can
> repeat builds later (without worry about offline or hacked servers), but
> for maven this often meant we had to download 300 MB of someone else's
> binaries to use in the build.
> 
> https://build.opensuse.org/project/show/home:fstrba:maven
> is an ongoing effort to build more things from source.
> 
> But bootstrapping would be much easier if maven and its plugins could be
> built with ant, make or some other pre-existing bootstrappable build system.
> 
> Do you know if such is possible or who would be interested in making it
> possible?
we had an Ant build.xml until Maven 3.3.9
https://github.com/apache/maven/tree/maven-3.3.9
that we stopped maintaining for Maven 3.5
But even that one was using prebuilt binaries for Modello, Plexus utils and 
many dependencies...
Fully bootstrapapable build never has really been a Maven primary objective...

Regards,

Hervé

> 
> Ciao
> Bernhard M.