Build an Android app. Task with an asterisk

Hello, Habr! In the summer, I spoke at the Summer Droid Meetup with a report on building an Android application. The video version can be found here: habr.com/ru/company/funcorp/blog/462825 . And for those who like to read more, I just wrote this article.

It's about what it is - an Android application. We will collect in different ways Hello, world !: start from the console and see what happens under the hood of build systems, then back to the past, remember Maven and learn about modern Bazel and Buck solutions. And finally, all this is comparable.

We thought about a possible change in the assembly system when we started a new project. It seemed to us that this is a good opportunity to look for some alternatives to Gradle. Moreover, it is easier to do this at the start than to translate an existing project. The following Gradle flaws pushed us to this step:

he definitely has problems with incremental assembly, although he can see progress in this direction;
he does poorly with very large monolithic projects;
it happens that a demon starts for a very long time;
demanding on the machine on which it is running.

APK

First of all, remember what the Android application consists of: compiled code, resources and AndroidManifest.xml.

Sources are in the classes.dex file (there may be several files, depending on the size of the application) in a special dex format that the Android virtual machine can work with. Today it is ART, on older devices - Dalvik. In addition, you can find the lib folder, where the native sources are arranged in subfolders. They will be named depending on the target processor architecture, for example x86, arm, etc. If you use exoplayer, then you probably have lib. And the aidl folder, which contains interprocess communication interfaces. They will come in handy if you need to access a service running in another process. Such interfaces are used both in Android itself and inside GooglePlayServices.

Various non-compiled resources like pictures are in the res folder. All compiled resources, such as styles, lines, etc., are merged into a resource.arsc file. In the assets folder, as a rule, they put everything that does not fit into resources, for example, custom fonts.

In addition to all this, the APK contains AndroidManifest.xml. In it, we describe the various components of the application, such as Activity, Service, different permissions, etc. It lies in binary form, and in order to look inside, it will first have to be converted into a human-readable file.

CONSOLE

Now that we know what the application consists of, we can try to build Hello, world! from the console using the tools that the Android SDK provides. This is a pretty important step in understanding how build systems work, because they all rely on these utilities to one degree or another. Since the project is written in Kotlin, we need its compiler for the command line. It is easy to download separately.

The assembly of the application can be divided into the following steps:

Download and unpack all libraries on which the project depends. In my case, this is the appcompat backward compatibility library, which, in turn, depends on appcompat-core, so we pump it out as well;
generate R.java. This wonderful class contains the identifiers of all resources in the application and is used to access them in code;
we compile the sources into bytecode and translate it into Dex, because the Android virtual machine does not know how to work with the usual bytecode;
we pack everything in the APK, but first align all incompressible resources, such as pictures, relative to the beginning of the file. This allows the price of a completely insignificant increase in the size of the APK to significantly accelerate its work. Thus, the system can directly map resources to RAM using the mmap () function.
sign the application. This procedure protects the integrity of the APK and confirms authorship. And thanks to this, for example, the Play Market can verify that the application was built by you.

build script

function preparedir() { rm -r -f $1 mkdir $1 } PROJ="src/main" LIBS="libs" LIBS_OUT_DIR="$LIBS/out" BUILD_TOOLS="$ANDROID_HOME/build-tools/28.0.3" ANDROID_JAR="$ANDROID_HOME/platforms/android-28/android.jar" DEBUG_KEYSTORE="$(echo ~)/.android/debug.keystore" GEN_DIR="build/generated" KOTLIN_OUT_DIR="$GEN_DIR/kotlin" DEX_OUT_DIR="$GEN_DIR/dex" OUT_DIR="out" libs_res="" libs_classes="" preparedir $LIBS_OUT_DIR aars=$(ls -p $LIBS | grep -v /) for filename in $aars; do DESTINATION=$LIBS_OUT_DIR/${filename%.*} echo "unpacking $filename into $DESTINATION" unzip -o -q $LIBS/$filename -d $DESTINATION libs_res="$libs_res -S $DESTINATION/res" libs_classes="$libs_classes:$DESTINATION/classes.jar" done preparedir $GEN_DIR $BUILD_TOOLS/aapt package -f -m \ -J $GEN_DIR \ -M $PROJ/AndroidManifest.xml \ -S $PROJ/res \ $libs_res \ -I $ANDROID_JAR --auto-add-overlay preparedir $KOTLIN_OUT_DIR compiledKotlin=$KOTLIN_OUT_DIR/compiled.jar kotlinc $PROJ/java $GEN_DIR -include-runtime \ -cp "$ANDROID_JAR$libs_classes"\ -d $compiledKotlin preparedir $DEX_OUT_DIR dex=$DEX_OUT_DIR/classes.dex $BUILD_TOOLS/dx --dex --output=$dex $compiledKotlin preparedir $OUT_DIR unaligned_apk=$OUT_DIR/unaligned.apk $BUILD_TOOLS/aapt package -f -m \ -F $unaligned_apk \ -M $PROJ/AndroidManifest.xml \ -S $PROJ/res \ $libs_res \ -I $ANDROID_JAR --auto-add-overlay cp $dex . $BUILD_TOOLS/aapt add $unaligned_apk classes.dex rm classes.dex aligned_apk=$OUT_DIR/aligned.apk $BUILD_TOOLS/zipalign -f 4 $unaligned_apk $aligned_apk $BUILD_TOOLS/apksigner sign --ks $DEBUG_KEYSTORE $aligned_apk

According to the figures, it turns out that a clean assembly takes 7 seconds, and the incremental assembly does not lag behind it, because we do not cache anything and rebuild everything every time.

Maven

It was developed by the guys at the Apache Software Foundation to build Java projects. Build configs for it are described in XML. The early revisions of Maven were collected by Ant, and now they have switched to the latest stable release.

Pros of Maven:

It supports caching assembly artifacts, i.e. incremental build should be faster than clean;
able to resolve third-party dependencies. Those. When you specify a dependency on a third-party library in the Maven or Gradle config, you do not need to worry about what it depends on;
There is a bunch of detailed documentation because it has been on the market for quite some time.
and it can be a familiar build mechanism if you recently came to the world of Android development from the backend.

Cons Maven:

Depends on the version of Java installed on the machine on which the assembly takes place;
The Android plugin is now supported by third-party developers: personally, I consider this a very significant drawback, because one day they may stop doing this;
XML is not very suitable for describing build configs due to its redundancy and cumbersomeness;
well, and as we will see later, it runs slower than Gradle, at least on a test project.

To build, we need to create pom.xml, which contains a description of our project. In the header, we indicate the basic information about the collected artifact, as well as the version of Kotlin.

build config pom.xml

 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>myapplication</artifactId> <version>1.0.0</version> <packaging>apk</packaging> <name>My Application</name> <properties> <kotlin.version>1.3.41</kotlin.version> </properties> <dependencies> <dependency> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-stdlib</artifactId> <version>${kotlin.version}</version> </dependency> <dependency> <groupId>com.google.android</groupId> <artifactId>android</artifactId> <version>4.1.1.4</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-maven-plugin</artifactId> <version>${kotlin.version}</version> <executions> <execution> <id>compile</id> <phase>process-sources</phase> <goals> <goal>compile</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>com.simpligility.maven.plugins</groupId> <artifactId>android-maven-plugin</artifactId> <extensions>true</extensions> <configuration> <sdk> <platform>28</platform> <buildTools>28.0.3</buildTools> </sdk> <failOnNonStandardStructure>false</failOnNonStandardStructure> </configuration> </plugin> </plugins> </build> </project>

In terms of numbers, everything is not very rosy. A clean assembly takes about 12 seconds, while an incremental one - 10. This means that Maven somehow reuses artifacts from previous assemblies, or, in my opinion, it is more likely that the plug-in for building an Android project prevents it from doing this

Now they are using all this, I think, first of all, the creators of the plugin are the guys from simpligility. More reliable information about this issue could not be found.

Bazel

Engineers in the bowels of Google invented Bazel to build their projects and relatively recently transferred it to open source. For the description of build-configs python-like Skylark or Starlark is used, both names have a place to be. It is assembled using its own latest stable release.

Pros of Bazel:

support for different programming languages. If you believe the documentation, then he knows how to collect projects for iOs, Android or even a backend;
Can cache previously collected artifacts
able to work with Maven dependencies;
Bazel has very cool, in my opinion, support for distributed projects. He can specify specific revisions of git repositories as dependencies, and he will unload them and cache them during the build process. To support scalability, Bazel can, for example, distribute various targets on cloud-based build servers, which allows you to quickly build bulky projects.

Cons of Bazel:

all this charm is very difficult to maintain, because the build configs are very detailed and describe the assembly at a low level;
among other things, it seems that Bazel is now actively developing. Because of this, some examples are not collected, and those that are collected can use the outdated functionality, which is marked as deprecated;
the documentation now also leaves much to be desired, especially when compared to Gradle;
on small projects, warming up and analyzing build-configs may take longer than the assembly itself, which is not good, in my opinion.

Conceptually, the basic Bazel config consists of WORKSPACE, where we describe all sorts of global things for a project, and BUILD, which contains directly targets for assembly.

Let's describe WORKSPACE. Since we have an Android project, the first thing we configure is the Android SDK. Also, a rule for unloading configs is imported here. Then, since the project is written in Kotlin, we must specify the rules for it. Here we do this, referring to a specific revision directly from the git repository.

WORKSPACE

 android_sdk_repository( name = "androidsdk", api_level = 28, build_tools_version = "28.0.3" ) load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive") # # KOTLIN RULES # RULES_KOTLIN_VERSION = "990fcc53689c8b58b3229c7f628f843a60cb9f5c" http_archive( name = "io_bazel_rules_kotlin", url = "https://github.com/bazelbuild/rules_kotlin/archive/%s.zip" % RULES_KOTLIN_VERSION, strip_prefix = "rules_kotlin-%s" % RULES_KOTLIN_VERSION ) load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kotlin_repositories", "kt_register_toolchains") kotlin_repositories() kt_register_toolchains()

Now let's get started on the BUILD.

First we import the rule for assembling Kotlin and describe what we want to collect. In our case, this is an Android application, so we use android_binary, where we set the manifest, minimum SDK, etc. Our application will depend on the source, so we mention them in deps and move on to what they are and where to find them. The code will also depend on the resources and the appcompat library. For resources, we use the usual target for assembling android sources, but we only assign resources to it without java classes. And we describe a couple of rules that import third-party libraries. It also mentions appcompat_core, which appcompat depends on.

BUILD

 load("@io_bazel_rules_kotlin//kotlin:kotlin.bzl", "kt_android_library") android_binary( name = "app", custom_package = "com.example.myapplication", manifest = "src/main/AndroidManifest.xml", manifest_values = { "minSdkVersion": "15", }, deps = [ ":lib", ], ) kt_android_library( name = "lib", srcs = glob(["src/main/java/**/*"]), deps = [ ":res", ":appcompat", ], ) android_library( name = "res", resource_files = glob(["src/main/res/**/*"]), manifest = "src/main/AndroidManifest.xml", custom_package = "com.example.myapplication", ) aar_import( name = "appcompat", aar = "libs/appcompat.aar", deps = [ ":appcompat_core", ] ) aar_import( name = "appcompat_core", aar = "libs/core.aar", )

In numbers for such a small project, everything looks sad. More than half a minute to a clean build Hello, world! - lots of. Incremental build time is also far from perfect.

Bazel is used by its creators (Google) for some of their projects, including server ones, as well as Dropbox and Huawei, which collect mobile applications for them. And the notorious Dagger 2 is also going to Bazel.

Buck

It was invented by defectors from Google to Facebook. He used Python to describe the configs, and then migrated to the Skylark mentioned today. He is going, all of a sudden, using the Ant system.

Buck Pros:

supports different programming languages and can build both Andriod and iOS;
Can cache previously collected artifacts
Buck made their own dex implementation, which works faster than the standard one and hangs with the system daemon. So they save time on dex initialization. Engineers have really optimized a lot. For example, Buck does not collect code that depends on the library if the interface has not changed when changing the library internals. Similarly for resources: if the identifiers have not changed, then when changing resources, the code is not reassembled.
there is a plugin that can hide Buck behind the Gredlovsky config. Those. you get something like a normal Gradle project, which is actually built through Buck.

Cons Buck:

it is as hard to maintain as Bazel. Those. here it is also necessary to describe low-level rules that clearly describe the assembly process;
among other things, Buck cannot resolve Maven dependencies on its own.

So, what does the assembly config for Hello, world! through buck? Here we describe one configuration file, where we indicate that we want to build an Android project that will be signed with a debug key. The application will likewise depend on the source - lib in the deps array. Next comes the target with signature settings. I am using a debit key that comes with the Android SDK. Immediately after it is a target that will collect the source of Kotlin for us. Like Bazel, it depends on resources and compatibility libraries.

We describe them. There is a separate target for resources in Buck, so bikes are not useful. Following are the rules for downloaded third-party libraries.

BUILD

 android_binary( name = 'app', manifest = 'src/main/AndroidManifest.xml', manifest_entries = { 'min_sdk_version': 15, }, keystore = ':debug_keystore', deps = [ ':lib', ], ) keystore( name = 'debug_keystore', store = 'debug.keystore', properties = 'debug.keystore.properties', ) android_library( name = 'lib', srcs = glob(['src/main/java/*.kt']), deps = [ ':res', ':compat', ':compat_core', ], language = 'kotlin', ) android_resource( name = 'res', res = "src/main/res", package = 'com.example.myapplication', ) android_prebuilt_aar( name = 'compat', aar = "libs/appcompat.aar", ) android_prebuilt_aar( name = 'compat_core', aar = "libs/core.aar", )

This whole thing is going very briskly. A clean assembly takes a little more than 7 seconds, while an incremental assembly takes completely invisible 200 milliseconds. I think this is a very good result.

This is what Facebook does. In addition to their flagship application, they collect Facebook Messenger for them. And Uber, who made the plugin for Gradle and Airbnb with Lyft.

findings

Now that we’ve talked about each build system, we can compare them with each other using the example Hello, world! The console assembly pleases with its stability. The execution time of the script from the terminal can be considered a reference for the assembly of clean builds, because third-party costs for parsing scripts are minimal here. In this case, I would call Maven an obvious outsider for an extremely insignificant increase in incremental assembly. Bazel parses configs for a very long time and initializes: there is an idea that it somehow caches the initialization results, because the incremental build it runs is much faster than clean. Buck is the undisputed leader of this collection. Very fast both clean and incremental assembly.

Now compare the pros and cons. I will not include Maven in the comparison, because it clearly loses to Gradle and is almost never used in the market. I unite Buck and Bazel, because they have approximately the same advantages and disadvantages.

So, about Gradle:

the first and, in my opinion, the most important thing is that it is simple. Very simple;
out of the box, unloads and unloads dependencies;
for him there are a lot of different trainings and documentation;
actively supported by both Google and the community. Great integration with Android Studio, the current flagship development tool. And all the new features for building an Android application first appear in Gradle.

About Buck / Bazel:

can definitely be very fast compared to Gradle. I believe that this is especially noticeable on very large projects
You can keep one project in which there will be source codes of both iOS and Android, and assemble them with one build system. This allows some parts of the application to fumble between platforms. For example, this is how Chromium is going;
forced to describe dependencies in detail and thus literally force the developer to be multi-modular.

Do not forget about the cons.

Gradle pays for its simplicity by being slow and inefficient.

Buck / Bazel, on the contrary, because of its speed, suffers from the need to describe the build process in configs in more detail. Well, since they appeared on the market relatively recently, there are not many documentation and various cheat sheets.

iFUNNY

Perhaps you have a question how we collect iFunny. Just like many - using Gradle. And there are reasons for this:

It is not yet clear what gain in assembly speed this will give us. A clean build of iFunny takes almost 3 minutes, and incremental - about a minute, which is actually not very long.
Buck or Bazel build configs are more difficult to maintain. In the case of Buck, you also need to monitor the relevance of the connected libraries and the libraries on which they depend.
It is banal expensive to transfer an existing project from Gradle to Buck / Bazel, especially in conditions of incomprehensible profit.

If your project is going to take more than 45 minutes and there are about 20 people in the Android development team, then it makes sense to think about changing the build system. If you and your friend are sawing a startup, then use Gradle and drop these thoughts.

I will be glad to discuss the prospects of Gradle alternatives in the comments!

Link to the project

All Articles