How Java 8 is supported on Android

Hello, Habr! I bring to your attention a translation of a wonderful article from a series of articles by the notorious Jake Worton about how Android 8 is supported by Java.







The original article is here



I worked from home for several years, and I often heard my colleagues complain about Android supporting different versions of Java.



This is a rather complicated topic. First you need to decide what we mean by “Java support in Android”, because in one version of the language there can be a lot of things: features (lambdas, for example), bytecode, tools, APIs, JVM and so on.



When they talk about Java 8 support in Android, they usually mean support for language features. So, let's start with them.

Lambdas

One of the main innovations of Java 8 was lambdas.

The code has become more concise and simple, lambdas have saved us from writing cumbersome anonymous classes using an interface with a single method inside.

class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(s -> System.out.println(s)); } private static void sayHi(Logger logger) { logger.log("Hello!"); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

After compiling this using javac and legacy dx tool



, we get the following error:

 $ javac *.java $ ls Java8.java Java8.class Java8$Logger.class $ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class Uncaught translation error: com.android.dx.cf.code.SimException: ERROR in Java8.main:([Ljava/lang/String;)V: invalid opcode ba - invokedynamic requires --min-sdk-version >= 26 (currently 13) 1 error; aborting
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

This error occurs due to the fact that lambdas use the new instruction in the bytecode - invokedynamic



, which was added in Java 7. From the error text, you can see that Android only supports it starting with the 26 API (Android 8).



It doesn’t sound very good, because hardly anyone will release an application with 26 minApi. To get around this, the so-called desugaring process is used , which makes lambda support possible on all versions of the API.

History of Desaccharization

She's pretty colorful in the Android world. The goal of desaccharization is always the same - to allow new language features to work on all devices.



Initially, for example, to support lambdas in Android, developers connected the Retrolambda plugin. He used the same built-in mechanism as the JVM, converting lambdas to classes, but he did it in runtime, and not at compile time. The generated classes were very expensive in terms of the number of methods, but over time, after refinements and improvements, this indicator dropped to something more or less reasonable.



Then, the Android team announced a new compiler that supported all Java 8 features and was more productive. It was built on top of the Eclipse Java compiler, but instead of generating a Java bytecode, it generated a Dalvik bytecode. However, its performance is still poor.



When the new compiler (fortunately) was abandoned, the Java bytecode transformer in the Java bytecode, which did the juggling, was integrated into the Android Gradle Plugin from Bazel , Google’s build system. And its performance was still low, so in parallel the search continued for a better solution.



And now we were dexer



- D8 , which was supposed to replace the dx tool



. Desaccharization was now performed during the conversion of compiled JAR files to .dex



(dexing). The D8 is much better in performance compared to dx



, and since Android Gradle Plugin 3.1 it has become the default dexer.

D8

Now, using D8, we can compile the code above.

 $ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --output . \ *.class $ ls Java8.java Java8.class Java8$Logger.class classes.dex
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

To see how the D8 converted lambda, you can use the dexdump tool



, which is included in the Android SDK. It will display quite a lot of everything, but we will focus only on this:

 $ $ANDROID_HOME/build-tools/28.0.2/dexdump -d classes.dex [0002d8] Java8.main:([Ljava/lang/String;)V 0000: sget-object v0, LJava8$1;.INSTANCE:LJava8$1; 0002: invoke-static {v0}, LJava8;.sayHi:(LJava8$Logger;)V 0005: return-void [0002a8] Java8.sayHi:(LJava8$Logger;)V 0000: const-string v0, "Hello" 0002: invoke-interface {v1, v0}, LJava8$Logger;.log:(Ljava/lang/String;)V 0005: return-void …
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If you haven’t read the bytecode yet, don’t worry: a lot of what is written here can be understood intuitively.



In the first block, our main



method with index 0000



gets a reference from the INSTANCE



field to the INSTANCE



class Java8$1



. This class was generated during



. The main method bytecode also does not mention the body of our lambda anywhere, so most likely it is associated with the Java8$1



class. Index 0002



then calls the sayHi



static method using the link to INSTANCE



. The sayHi



requires Java8$Logger



, so it seems Java8$1



implements this interface. We can verify this here:

 Class #2 - Class descriptor : 'LJava8$1;' Access flags : 0x1011 (PUBLIC FINAL SYNTHETIC) Superclass : 'Ljava/lang/Object;' Interfaces - #0 : 'LJava8$Logger;'
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The SYNTHETIC



flag means that the Java8$1



class Java8$1



been generated and the list of interfaces that it includes contains the Java8$Logger



.

This class represents our lambda. If you look at the implementation of the log



method, you will not see the body of the lambda.

 … [00026c] Java8$1.log:(Ljava/lang/String;)V 0000: invoke-static {v1}, LJava8;.lambda$main$0:(Ljava/lang/String;)V 0003: return-void …
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Instead, the static



method of the Java8



class is Java8



- lambda$main$0



. Again, this method is presented only in bytecode.

 … #1 : (in LJava8;) name : 'lambda$main$0' type : '(Ljava/lang/String;)V' access : 0x1008 (STATIC SYNTHETIC) [0002a0] Java8.lambda$main$0:(Ljava/lang/String;)V 0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream; 0002: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V 0005: return-void
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The SYNTHETIC



flag again tells us that this method was generated, and its bytecode just contains the lambda body: a call to System.out.println



. The reason the lambda body is inside Java8.class is simple - it may need to access private



members of the class, which the generated class will not have access to.



Everything you need to understand how desaccharization works is described above. However, looking at it in the Dalvik bytecode, you can see that everything is much more complicated and frightening there.

Source Transformation

To better understand how desaccharization occurs, let's try step by step to convert our class into something that will work on all versions of the API.



Take the same class with lambda as a basis:

 class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(s -> System.out.println(s)); } private static void sayHi(Logger logger) { logger.log("Hello!"); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

First, the lambda body is moved to the package private



method.

  public static void main(String... args) { - sayHi(s -> System.out.println(s)); + sayHi(s -> lambda$main$0(s)); } + + static void lambda$main$0(String s) { + System.out.println(s); + }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Then a class is implemented that implements the Logger



interface, inside which a block of code from the lambda body is executed.

  public static void main(String... args) { - sayHi(s -> lambda$main$0(s)); + sayHi(new Java8$1()); } @@ } + +class Java8$1 implements Java8.Logger { + @Override public void log(String s) { + Java8.lambda$main$0(s); + } +}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Next, a singleton instance of Java8$1



, which is stored in the static



variable INSTANCE



.

  public static void main(String... args) { - sayHi(new Java8$1()); + sayHi(Java8$1.INSTANCE); } @@ class Java8$1 implements Java8.Logger { + static final Java8$1 INSTANCE = new Java8$1(); + @Override public void log(String s) {
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Here is the final dubbed class that can be used on all versions of the API:

 class Java8 { interface Logger { void log(String s); } public static void main(String... args) { sayHi(Java8$1.INSTANCE); } static void lambda$main$0(String s) { System.out.println(s); } private static void sayHi(Logger logger) { logger.log("Hello!"); } } class Java8$1 implements Java8.Logger { static final Java8$1 INSTANCE = new Java8$1(); @Override public void log(String s) { Java8.lambda$main$0(s); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If you look at the generated class in the Dalvik bytecode, you will not find names like Java8 $ 1 - there will be something like -$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY



. The reason why such naming is generated for the class, and what are its advantages, draws to a separate article.

Native lambda support

When we used the dx tool



to compile a class containing lambdas, an error message said that this would only work with 26 APIs.

 $ $ANDROID_HOME/build-tools/28.0.2/dx --dex --output . *.class Uncaught translation error: com.android.dx.cf.code.SimException: ERROR in Java8.main:([Ljava/lang/String;)V: invalid opcode ba - invokedynamic requires --min-sdk-version >= 26 (currently 13) 1 error; aborting
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Therefore, it seems logical that if we try to compile this with the —min-api 26



flag, then desaccharization will not occur.

 $ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --min-api 26 \ --output . \ *.class
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

However, if we dump the .dex



file, then it can still be found in it -$$Lambda$Java8$QkyWJ8jlAksLjYziID4cZLvHwoY



. Why is that? Is this a D8 bug?



To answer this question, and also why desaccharization always occurs , we need to look inside the Java bytecode of the Java8



class.

 $ javap -v Java8.class class Java8 { public static void main(java.lang.String...); Code: 0: invokedynamic #2, 0 // InvokeDynamic #0:log:()LJava8$Logger; 5: invokestatic #3 // Method sayHi:(LJava8$Logger;)V 8: return } …
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Inside the main



method, we again see invokedynamic at index 0



. The second argument in the call is 0



- the index of the bootstrap method associated with it.



Here is a list of bootstrap methods:

 … BootstrapMethods: 0: #27 invokestatic java/lang/invoke/LambdaMetafactory.metafactory:( Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String; Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType; Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;) Ljava/lang/invoke/CallSite; Method arguments: #28 (Ljava/lang/String;)V #29 invokestatic Java8.lambda$main$0:(Ljava/lang/String;)V #28 (Ljava/lang/String;)V
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Here the bootstrap method is called metafactory



in the java.lang.invoke.LambdaMetafactory



class. He lives in the JDK and creates anonymous on-the-fly classes in runtime for lambdas, just like D8 generates them in compute time.



If you look at the Android java.lang.invoke





or to the AOSP java.lang.invoke



, we see that this class is not in the runtime. That's why de-juggling always happens at compile time, no matter what minApi you have. The VM supports bytecode instructions similar to invokedynamic



, but the invokedynamic



built-in to the JDK LambdaMetafactory



not available for use.

Method references

Along with lambdas, Java 8 added method references - this is an effective way to create a lambda whose body references an existing method.



Our Logger



interface is just such an example. The lambda body referred to System.out.println



. Let's turn the lambda into a reference method:

  public static void main(String... args) { - sayHi(s -> System.out.println(s)); + sayHi(System.out::println); }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

When we compile it and take a look at the bytecode, we will see one difference with the previous version:

 [000268] -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM.log:(Ljava/lang/String;)V 0000: iget-object v0, v1, L-$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM;.f$0:Ljava/io/PrintStream; 0002: invoke-virtual {v0, v2}, Ljava/io/PrintStream;.println:(Ljava/lang/String;)V 0005: return-void
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Instead of calling the generated Java8.lambda$main$0



, which contains a call to System.out.println



, now System.out.println



is called directly.



A class with a lambda is no longer a static



singleton, but by the 0000



index in the bytecode, we see that we get a link to PrintStream



- System.out



, which is then used to call println



on it.



As a result, our class turned into this:

  public static void main(String... args) { - sayHi(System.out::println); + sayHi(new -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(System.out)); } @@ } + +class -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM implements Java8.Logger { + private final PrintStream ps; + + -$$Lambda$1Osqr2Z9OSwjseX_0FMQJcCG_uM(PrintStream ps) { + this.ps = ps; + } + + @Override public void log(String s) { + ps.println(s); + } +}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Default



and static



methods in interfaces

Another important and major change that Java 8 brought was the ability to declare default



and static



methods in interfaces.

 interface Logger { void log(String s); default void log(String tag, String s) { log(tag + ": " + s); } static Logger systemOut() { return System.out::println; } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

All this is also supported by D8. Using the same tools as before, it is easy to see a logged-in version of Logger with default



and static



methods. One of the differences with lambdas and method references



is that the default and static methods are implemented in the Android VM and, starting with the 24 API, D8 will not decouple them.

Maybe just use Kotlin?

Reading the article, most of you probably thought about Kotlin. Yes, it supports all Java 8 features, but they are implemented by kotlinc



in the same way as D8, with the exception of some details.



Therefore, Android support for new versions of Java is still very important, even if your project is 100% written in Kotlin.



It is possible that in the future Kotlin will cease to support Java 6 and Java 7 bytecode. IntelliJ IDEA , Gradle 5.0 switched to Java 8. The number of platforms running on older JVMs is decreasing.

Desugaring APIs

All this time I talked about Java 8 features, but did not say anything about the new APIs - streams, CompletableFuture



, date / time and so on.



Returning to the Logger example, we can use the new date / time API to find out when messages were sent.

 import java.time.*; class Java8 { interface Logger { void log(LocalDateTime time, String s); } public static void main(String... args) { sayHi((time, s) -> System.out.println(time + " " + s)); } private static void sayHi(Logger logger) { logger.log(LocalDateTime.now(), "Hello!"); } }
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Compile it again with javac



and convert it to the Dalvik bytecode with D8, which decouples it for support on all versions of the API.

 $ javac *.java $ java -jar d8.jar \ --lib $ANDROID_HOME/platforms/android-28/android.jar \ --release \ --output . \ *.class
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

You can even run this on your device to make sure that it works.

 $ adb push classes.dex /sdcard classes.dex: 1 file pushed. 0.5 MB/s (1620 bytes in 0.003s) $ adb shell dalvikvm -cp /sdcard/classes.dex Java8 2018-11-19T21:38:23.761 Hello
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If API 26 and above is on this device, the Hello message will appear. If not, we will see the following:

 java.lang.NoClassDefFoundError: Failed resolution of: Ljava/time/LocalDateTime; at Java8.sayHi(Java8.java:13) at Java8.main(Java8.java:9)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

D8 dealt with lambdas, the reference method, but did nothing to work with LocalDateTime



, and this is very sad.



Developers have to use their own implementations or wrappers on date / time api, or use libraries like ThreeTenBP



to work with time, but why can't you do D8 with your hands?

Epilogue

Lack of support for all the new Java 8 APIs remains a big problem in the Android ecosystem. Indeed, it is unlikely that each of us can allow us to specify the 26 min API in our project. Libraries supporting both Android and JVM cannot afford to use the API introduced to us 5 years ago!



And even though Java 8 support is now part of D8, every developer should still explicitly specify source and target compatibility in Java 8. If you write your own libraries, you can strengthen this trend by laying out libraries that use Java 8 bytecode (even if you are not using new language features).



A lot of work is being done on D8, so it seems that everything will be ok in the future with support for language features. Even if you write only on Kotlin, it is very important to force the Android development team to support all new versions of Java, improve bytecode and new APIs.



This post is a written version of my talk Digging into D8 and R8 .