[rb-general] Reproducible Java builds with Maven

[Julien Lepiller]

Le 2018-11-26 10:00, Bernhard M. Wiedemann a écrit :
> Hi Hervé,
> 
> thanks for raising this topic.
> 
> On 26/11/2018 09.08, Hervé Boutemy wrote:
>> Currently, there is a plugin that does the job as archive 
>> post-processing, but
>> one intent would be to have native reproducible builds when that does 
>> not hurt
>> other objectives, like performance or anything else: we did already 
>> change
>> many plugins that for sure improve the situation, but I don't have a 
>> real view
>> on how far we are from native reproducible builds.
> 
> I think, one major consideration is how many code-places touch .jar
> files, because if we want to get rid of post-processing, we might need
> to patch all of them to have normalized mtime values in the .jar.
> 
>> I know that one key step will be to have reproducible archives 
>> creation (with
>> reproducible timestamps and order), which won't be easy without 
>> sacrificing
>> performance (ConcurrentJarCreator was created to improve performance, 
>> but adds
>> even more variability in order...)
> 
> Just because you use parallelism does not mean, your output has to 
> vary.
> E.g. you can play with 'pigz' (parallel gzip):
> dd if=/dev/urandom bs=1M count=100 of=input
> time pigz -p 2 -nc1 input | md5sum
> 
> notice how it produces bit-identical results, independent of the number
> of threads (given with -p).
> 
>> Anybody interested in working together?
> 
> I'm interested, but so far kept my fingers off Java as much as I could,
> because I felt it is hard to work with.
> 
> With Maven, there are also major issues in a related domain, that is
> called "bootstrappable builds".
> With openSUSE we are doing all builds offline to ensure that we can
> repeat builds later (without worry about offline or hacked servers), 
> but
> for maven this often meant we had to download 300 MB of someone else's
> binaries to use in the build.
> 
> https://build.opensuse.org/project/show/home:fstrba:maven
> is an ongoing effort to build more things from source.
> 
> But bootstrapping would be much easier if maven and its plugins could 
> be
> built with ant, make or some other pre-existing bootstrappable build 
> system.
> 
> Do you know if such is possible or who would be interested in making it
> possible?

Hi,

Answering about the bootstrap issue, in Guix we have a proper bootstrap, 
with no binary involved, except for gnu toolchain. I've written a small 
paragraph here: : 
https://bootstrapping.miraheze.org/wiki/Build_Systems#Apache_Maven. 
Basically, that's a few hundred dependencies that were built with a 
"manual" ant-build-system instead of maven. Even with maven though, we 
lack plugins to actually build things in an offline environment. I'm 
trying to build them, but there are again a lot of missing dependencies.

Scala and sbt are also affected by this issue: sbt and scala are written 
in scala (so they both need a scala compiler) and are both built using 
sbt (so they both need sbt). Some sbt dependencies require sbt too. Does 
openSUSE have a bootstrap path for these packages?

To go back to the original subject, let's imagine we have every plugin 
required to build a package with maven. As a distribution, how do we use 
the reproducible-build-plugin to ensure the build is reproducible? Do we 
have to modify pom.xml files (we'll modify them anyway so dependency 
versions correspond to what we have packaged), if so, can you give an 
example?

Thank you!

> 
> Ciao
> Bernhard M.
> 
> 
> _______________________________________________
> rb-general at lists.reproducible-builds.org mailing list
> 
> To change your subscription options, visit
> https://lists.reproducible-builds.org/listinfo/rb-general.
> 
> To unsubscribe, send an email to
> rb-general-unsubscribe at lists.reproducible-builds.org.