Clever Geek Handbook

Experience transferring a Maven project to Multi-Release Jar: already possible, but still difficult

I have a small StreamEx library that extends the capabilities of the Java 8 Stream API. I traditionally compile the library through Maven, and for the most part, everything suits me. However, I wanted to experiment.







Some things in the library should work differently in different versions of Java. The most striking example is the new Stream API methods like takeWhile



, which appeared only in Java 9. My library provides an implementation of these methods in Java 8 too, but when you expand the Stream API yourself, you come under some restrictions that I’ll not mention here. I wish Java 9+ users had access to a standard implementation.







In order for the project to continue compiling using Java 8, this is usually done using reflection tools: we find out if there is a corresponding method in the standard library and, if so, we call it, and if not, then we use our implementation. However, I decided to use the MethodHandle API , because it declares less call overhead. You can get MethodHandle



in advance and save it in a static field: 







 MethodHandles.Lookup lookup = MethodHandles.publicLookup(); MethodType type = MethodType.methodType(Stream.class, Predicate.class); MethodHandle method = null; try { method = lookup.findVirtual(Stream.class, "takeWhile", type); } catch (NoSuchMethodException | IllegalAccessException e) { // ignore }


And then use it: 







 if (method != null) { return (Stream<T>)method.invokeExact(stream, predicate); } else { // Java 8 polyfill }


This is all good, but it looks ugly. And most importantly, at every point where a variation of implementations is possible, you have to write such conditions. A slightly alternative approach is to separate the strategies of Java 8 and Java 9 as an implementation of the same interface. Or, to save the size of the library, simply implement everything for Java 8 in a separate non-final class, and substitute an inheritor for Java 9. It was done something like this: 







 //    Internals static final VersionSpecific VER_SPEC = System.getProperty("java.version", "").compareTo("1.9") > 0 ? new Java9Specific() : new VersionSpecific();


Then at points of use, you can simply write return Internals.VER_SPEC.takeWhile(stream, predicate)



. All the magic with method handles is now only in the Java9Specific



class. This approach, by the way, saved the library for Android users who had previously complained that it did not work in principle. The Android virtual machine is not Java, it does not even implement the Java 7 specification. In particular, there are no methods with a polymorphic signature like invokeExact



, and the very presence of this call in the bytecode breaks everything. Now these calls are placed in a class that is never initialized.







However, all this is still ugly. A beautiful solution (at least in theory) is to use the Multi Release Jar, which came with Java 9 ( JEP-238 ). To do this, part of the classes must be compiled under Java 9 and compiled class files placed in META-INF/versions/9



inside the Jar file. In addition, you need to add the line Multi-Release: true



to the manifest. Then Java 8 will successfully ignore all this, and Java 9 and newer will load new classes instead of classes with the same names that are located in the usual place.







The first time I tried to do this more than two years ago, shortly before the release of Java 9. It went very hard, and I quit. It was difficult even to make the project compile by the compiler from Java 9: ​​many Maven plugins simply broke due to changed internal APIs, changed java.version



string java.version



or something else.







A new attempt this year was more successful. Plugins have for the most part been updated and work in the new Java quite adequately. The first step I translated the entire assembly into Java 11. For this, in addition to updating the plugin versions, I had to do the following:









The next step was the adaptation of the tests. Since the library behavior is obviously different in Java 8 and Java 9, it would be logical to run tests for both versions. Now we are running everything under Java 11, so Java 8-specific code is not tested. This is a fairly large and non-trivial code. To fix this, I made an artificial pen: 







 static final VersionSpecific VER_SPEC = System.getProperty("java.version", "").compareTo("1.9") > 0 && !Boolean.getBoolean("one.util.streamex.emulateJava8") ? new Java9Specific() : new VersionSpecific();


Now just pass -Done.util.streamex.emulateJava8=true



when running the tests,

to test what usually works in Java 8. Now add a new <execution>



block to the maven-surefire-plugin



configuration with argLine = -Done.util.streamex.emulateJava8=true



, and the tests pass two times.







However, I would like to consider the total coverage tests. I use JaCoCo, and if you don't tell him anything, then the second run will simply overwrite the results of the first. How does JaCoCo work? It first runs the prepare-agent



target, which sets the argLine Maven property, signing something like -javaagent:blah-blah/.m2/org/jacoco/org.jacoco.agent/0.8.4/org.jacoco.agent-0.8.4-runtime.jar=destfile=blah-blah/myproject/target/jacoco.exec



. I want two different exec files to be formed. You can hack it this way. Add destFile=${project.build.directory}



prepare-agent



configuration. Rough but effective. Now argLine



will end in blah-blah/myproject/target



. Yes, this is not a file at all, but a directory. But we can substitute the file name already at the start of the tests. We return to the maven-surefire-plugin



and set argLine = @{argLine}/jacoco_java8.exec -Done.util.streamex.emulateJava8=true



for the Java 8 run and argLine = @{argLine}/jacoco_java11.exec



for the Java 11 run. Then it’s easy to combine these two files using the merge



target, which the JaCoCo plugin also provides, and we get a common coverage.







Update: godin in the comments said that it was unnecessary, you can write to one file, and it will automatically glue the result with the previous one. Now I’m not sure why this scenario didn’t work for me initially.







Well, we are well prepared to switch to the Multi-Release Jar. I found a number of recommendations on how to do this. The first suggested the use of a multi-modular Maven project. I don’t feel like it: this is a great complication of the project structure: there are five pom.xml, for example. To fool around for the sake of a couple of files that need to be compiled in Java 9 seems to be too much. Another suggested starting compilation through maven-antrun-plugin



. Here I decided to look only as a last resort. It is clear that any problem in Maven can be solved using Ant, but this is somehow quite clumsy. Finally, I saw a recommendation to use the third-party multi-release-jar-maven-plugin plugin . It already sounded tasty and right.







The plugin recommends placing source codes specific for new versions of Java in directories like src/main/java-mr/9



, which I did. I still decided to avoid collisions in the class names to the maximum, so the only class (even the interface) that is present in both Java 8 and Java 9, I have this: 







 // Java 8 package one.util.streamex; /* package */ interface VerSpec { VersionSpecific VER_SPEC = new VersionSpecific(); } // Java 9 package one.util.streamex; /* package */ interface VerSpec { VersionSpecific VER_SPEC = new Java9Specific(); }


The old constant moved to a new place, but nothing else really changed. Only now the Java9Specific



class Java9Specific



become much simpler: all squats with MethodHandle have been successfully replaced with direct method calls.







The plugin promises to do the following things:









For it to work, it took quite a few steps.







  1. Change packaging from jar



    to multi-release-jar



    .







  2. Add build-extension: 







     <build> <extensions> <extension> <groupId>pw.krejci</groupId> <artifactId>multi-release-jar-maven-plugin</artifactId> <version>0.1.5</version> </extension> </extensions> </build>


  3. Copy configuration from maven-compiler-plugin



    . I had only the default version in the spirit of <source>1.8</source>



    and <arg>-Xlint:all</arg>









  4. I thought that maven-compiler-plugin



    can now be removed, but it turned out that the new plugin does not replace compilation of tests, so for it the Java version was reset to default (1.5!) And the -Xlint:all



    argument disappeared. So I had to leave.







  5. In order not to duplicate the source and target for the two plugins, I found out that they both respect the properties of maven.compiler.source



    and maven.compiler.target



    . I installed them and removed the versions from the plugin settings. However, it suddenly turned out that maven-javadoc-plugin



    uses source



    from the maven-compiler-plugin



    settings to find out the URL of the standard JavaDoc, which must be linked when referencing standard methods. And now he does not respect maven.compiler.source



    . Therefore, I had to return <source>${maven.compiler.source}</source>



    to the maven-compiler-plugin



    settings. Fortunately, no other changes were needed to generate JavaDoc. It can very well be generated from Java 8 sources, because the whole version carousel does not affect the library API.







  6. The maven-bundle-plugin



    , which turned my library into an OSGi artifact. He simply refused to work with packaging = multi-release-jar



    . In principle, I never liked him. He writes a set of additional lines to the manifest, while spoiling the sort order and adding more garbage. Fortunately, it turned out that it is not difficult to get rid of it by writing everything you need by hand. Only, of course, not in maven-jar-plugin



    , but in the new one. The whole configuration of the multi-release-jar



    plugin eventually became like this (I defined some properties like project.package



    myself): 







     <plugin> <groupId>pw.krejci</groupId> <artifactId>multi-release-jar-maven-plugin</artifactId> <version>0.1.5</version> <configuration> <compilerArgs><arg>-Xlint:all</arg></compilerArgs> <archive> <manifestEntries> <Automatic-Module-Name>${project.package}</Automatic-Module-Name> <Bundle-Name>${project.name}</Bundle-Name> <Bundle-Description>${project.description}</Bundle-Description> <Bundle-License>${license.url}</Bundle-License> <Bundle-ManifestVersion>2</Bundle-ManifestVersion> <Bundle-SymbolicName>${project.package}</Bundle-SymbolicName> <Bundle-Version>${project.version}</Bundle-Version> <Export-Package>${project.package};version="${project.version}"</Export-Package> </manifestEntries> </archive> </configuration> </plugin>


  7. Tests. We no longer have one.util.streamex.emulateJava8



    , but you can achieve the same effect by modifying the class-path tests. Now the opposite is true: by default, the library works in Java 8 mode, and for Java 9 you need to write: 







     <classesDirectory>${basedir}/target/classes-9</classesDirectory> <additionalClasspathElements>${project.build.outputDirectory}</additionalClasspathElements> <argLine>@{argLine}/jacoco_java9.exec</argLine>


    An important point: classes-9



    should go ahead of ordinary class files, so I had to transfer the usual ones to additionalClasspathElements



    , which are added after.







  8. Sources. I’m going to have source-jar, and it would be nice to pack Java 9 sources into it so that, for example, the debugger in the IDE can correctly display them. I'm not really worried about duplicate VerSpec



    , because there is one line that runs only on initialization. It’s normal for me to leave only an option from Java 8. However, it would be nice to Java9Specific.java



    to attach. This can be done by manually adding an additional source directory: 







     <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>build-helper-maven-plugin</artifactId> <version>3.0.0</version> <executions> <execution> <phase>test</phase> <goals><goal>add-source</goal></goals> <configuration> <sou​rces> <sou​rce>src/main/java-mr/9</sou​rce> </sou​rces> </configuration> </execution> </executions> </plugin>


    Having collected the artifact, I connected it to a test project and checked it in the IntelliJ IDEA debugger. Everything works beautifully: depending on the version of the virtual machine used to run the test project, we end up in a different source when debugging.







    It would be cool to have this done by the multi-release-jar plugin itself, so I made such a suggestion.







  9. JaCoCo. It turned out to be the most difficult with him, and I could not do without outside help. The fact is that the plug-in perfectly generated exec-files for Java-8 and Java-9, normally glued them into one file, however, when generating reports in XML and HTML, they stubbornly ignored the sources from Java-9. Rummaging around in the source , I saw that it only generates a report for the class files found in project.getBuild().getOutputDirectory()



    . This directory, of course, can be replaced, but in fact I have two of them: classes



    and classes-9



    . Theoretically, you can copy all classes into one directory, change the outputDirectory



    and start JaCoCo, and then change the outputDirectory



    back so as not to break the JAR assembly. But that sounds completely ugly. In general, I decided to postpone the solution to this problem in my project for now, but I wrote to the guys from JaCoCo that it would be nice to be able to specify several directories with class files.







    To my surprise, just a few hours later one of the developers of JaCoCo godin came to my project and brought a pull-request , which solves the problem. How does it decide? Using Ant, of course! It turned out that the Ant-plugin for JaCoCo is more advanced and can generate a summary report for several source directories and class files. He didn’t even need a separate merge



    step, because he could feed several exec files at once. In general, Ant could not be avoided, so be it. The main thing that worked, and pom.xml grew by only six lines.













    I even tweeted in hearts:











So I got a very working project that builds a beautiful Multi-Release Jar. At the same time, the percentage of coverage even increased, because I removed all sorts of catch (NoSuchMethodException | IllegalAccessException e)



that were unattainable in Java 9. Unfortunately, this project structure is not supported by IntelliJ IDEA, so I had to abandon the POM import and configure the project in the IDE manually . I hope that in the future there will still be a standard solution that will be automatically supported by all plugins and tools.







More articles:


All Articles