Dependency Resolution Tools Semantics

Dependency Resolution Tool

A dependency resolver (hereinafter referred to as resolver, approx. Transl.) Or a package manager is a program that defines a consistent set of modules taking into account the restrictions set by the user.

Constraints are usually specified by module names and version numbers. In the JVM ecosystem for Maven modules, the organization name (group id) will also be indicated. In addition, restrictions may include version ranges, excluded modules, version overrides, etc.

The three main categories of packages are represented by OS packages (Homebrew, Debian packages, etc.),

modules for specific programming languages ​​(CPAN, RubyGem, Maven, etc) and application-specific extensions (Eclipse plugins, IntelliJ plugins, VS Code extensions).

Resolver semantics

In a first approximation, we can represent the dependencies of the modules as a DAG (directed acyclic graph, directed acyclic graph).

This representation is called the dependency graph. Consider the dependencies of two modules:

• a:1.0
  
  
  
  depends on c:1.0
  
  
  
  
• b:1.0
  
  
  
  depends on c:1.0
  
  
  
  and d:1.0
  
  
  
  

 +-----+ +-----+ |a:1.0| |b:1.0| +--+--+ +--+--+ | | +<-------+ | | vv +--+--+ +--+--+ |c:1.0| |d:1.0| +-----+ +-----+
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

If the module depends on a:1.0



and b:1.0



, then a full list of dependencies will be presented a:1.0



, b:1.0



, c:1.0



and d:1.0



. And this is just a tree tour.

The situation will become more complicated if transitive dependencies are specified by a range of versions:

• a:1.0
  
  
  
  depends on c:1.0
  
  
  
  
• b:1.0
  
  
  
  depends on c:[1.0,2)
  
  
  
  and d:1.0
  
  
  
  

 +-----+ +-----+ |a:1.0| |b:1.0| +--+--+ +--+--+ | | | +-----------+ | | | vvv +--+--+ +--+------+ +--+--+ |c:1.0| |c:[1.0,2)| |d:1.0| +-----+ +---------+ +-----+
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Or, if different versions are specified for transitive dependencies:

• a:1.0
  
  
  
  depends on c:1.0
  
  
  
  
• b:1.0
  
  
  
  depends on c:1.2
  
  
  
  and d:1.2
  
  
  
  

Or, if exceptions are thrown for the dependency:

• dependence on a:1.0
  
  
  
  , which depends on c:1.0
  
  
  
  , excluding c:*
  
  
  
  
• b:1.0
  
  
  
  depends on c:1.2
  
  
  
  and d:1.2
  
  
  
  

Different resolvers interpret the restrictions set by users differently. I call such rules the semantics of resolvers.

You may need to know some of these semantics, for example:

• semantics of your own module (determined by the build tool you use);
• the semantics of the libraries you use (determined by the build tool that the author used);
• the semantics of the modules that your module will use as a dependency (defined by the end-user build tool).

Dependency Resolution Tools in the JVM Ecosystem

Since I support sbt



, I have to work primarily in the JVM ecosystem.

Semantics Maven: nearest-wins

In graphs where there is a conflict of dependencies (in the dependency graph a



there are many different versions of component d



, for example d:1.0



and d:2.0



), Maven uses the nearest-wins strategy to resolve the conflict.

Resolving dependency conflicts - a process that determines which version of an artifact will be selected if several different versions of the same artifact are found among the dependencies. Maven selects the closest definition. Those. uses the version that is closest to your project in the dependency tree.

You can always guaranteed to use the right version by explicitly declaring it in the POM of the project. Note that if two versions of the dependency have the same depth in the tree, the first one will be selected. The closest definition means that the version closest to the project will be used in the dependency tree. For example, if the dependencies for A



, B



and C



defined as A -> B -> C -> D 2.0



and A -> E -> D 1.0



, then, when building A



, D 1.0



will be used, because the path from A



to D



via E



shorter (than via B



and C



, approx. transl.).

This means that many Java modules published using Maven were compiled using the nearest-wins



semantics. To illustrate this, create a simple pom.xml



:

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>foo</artifactId> <version>1.0.0</version> <packaging>jar</packaging> <dependencyManagement> <dependencies> <dependency> <groupId>com.typesafe.play</groupId> <artifactId>play-ws-standalone_2.12</artifactId> <version>1.0.1</version> </dependency> </dependencies> </dependencyManagement> </project>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

mvn dependency:build-classpath



returns the resolved classpath



.

It is noteworthy that the resulting tree uses com.typesafe:config:1.2.0



even though Akka 2.5.3



transitively dependent on com.typesafe:config:1.3.1



.

mvn dependency:tree



gives that visual confirmation:

 [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ foo --- [INFO] com.example:foo:jar:1.0.0 [INFO] \- com.typesafe.play:play-ws-standalone_2.12:jar:1.0.1:compile [INFO] +- org.scala-lang:scala-library:jar:2.12.2:compile [INFO] +- javax.inject:javax.inject:jar:1:compile [INFO] +- com.typesafe:ssl-config-core_2.12:jar:0.2.2:compile [INFO] | +- com.typesafe:config:jar:1.2.0:compile [INFO] | \- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.0.4:compile [INFO] \- com.typesafe.akka:akka-stream_2.12:jar:2.5.3:compile [INFO] +- com.typesafe.akka:akka-actor_2.12:jar:2.5.3:compile [INFO] | \- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.8.0:compile [INFO] \- org.reactivestreams:reactive-streams:jar:1.0.0:compile
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Many libraries provide backward compatibility, but direct compatibility is not guaranteed with a few exceptions, which is alarming.

Semantics of Apache Ivy: latest-wins

By default, Apache Ivy uses the latest-wins strategy to resolve dependency conflicts.

If this container is not present, then the default conflict manager is used for all modules. The current default conflict manager is "latest-revision".

Note: The container conflicts



is one of the Ivy files.

Up to version SBT 1.3.x



internal dependency resolver is Apache Ivy. The pom.xml



used earlier is described in SBT a little more briefly:

 ThisBuild / scalaVersion := "2.12.8" ThisBuild / organization := "com.example" ThisBuild / version := "1.0.0-SNAPSHOT" lazy val root = (project in file(".")) .settings( name := "foo", libraryDependencies += "com.typesafe.play" %% "play-ws-standalone" % "1.0.1", )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In the sbt shell



enter show externalDependencyClasspath



to get the resolved classpath. It should indicate the version of com.typesafe:config:1.3.1



. In addition, the following warning will still be displayed:

 [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Calling the evicted



in the sbt shell



allows you to get a conflict resolution report:

 sbt:foo> evicted [info] Updating ... [info] Done updating. [info] Here are other dependency conflicts that were resolved: [info] * com.typesafe:config:1.3.1 is selected over 1.2.0 [info] +- com.typesafe.akka:akka-actor_2.12:2.5.3 (depends on 1.3.1) [info] +- com.typesafe:ssl-config-core_2.12:0.2.2 (depends on 1.2.0) [info] * com.typesafe:ssl-config-core_2.12:0.2.2 is selected over 0.2.1 [info] +- com.typesafe.play:play-ws-standalone_2.12:1.0.1 (depends on 0.2.2) [info] +- com.typesafe.akka:akka-stream_2.12:2.5.3 (depends on 0.2.1)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

In the latest-wins



semantics, specifying config:1.2.0



in practice means "provide me with version 1.2.0 or higher."

This behavior is slightly more preferable than the nearest-wins



strategy, because the versions of transitive libraries are not downgraded. However, the evicted



call should verify that the substitutions were made correctly.

Semantics Coursier: latest-wins

Before we come to the description of semantics, I will answer an important question - how is Coursier pronounced. According to Alex Arshambo's note, it is pronounced chick-sie .

Interestingly, the Coursier documentation has a page on versioning , which talks about dependency resolution semantics.

Consider the intersection of given intervals:

• If it is empty (intervals do not intersect), then there is a conflict.
• If no intervals are specified, it is assumed that the intersection is represented by (,) (the interval corresponding to all versions).
  
  Then, consider specific versions:
  
  
  • We discard specific versions below the boundaries of the interval.
  • If there are specific versions above the boundaries of the interval, then there is a conflict.
  • If specific versions are within the boundaries of the interval, the result should be the latest of them.
  • If there are no specific versions within or above the bounds of the interval, the result should be the interval.

Because it is said



, therefore - this is the semantics of latest-wins



.

You can verify this by taking sbt 1.3.0-RC3



, which uses Coursier.

 ThisBuild / scalaVersion := "2.12.8" ThisBuild / organization := "com.example" ThisBuild / version := "1.0.0-SNAPSHOT" lazy val root = (project in file(".")) .settings( name := "foo", libraryDependencies += "com.typesafe.play" %% "play-ws-standalone" % "1.0.1", )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Calling show externalDependencyClasspath



from the sbt 1.3.0-RC3



console will return com.typesafe:config:1.3.1



, as expected. The Conflict Resolution Report reports the same:

 sbt:foo> evicted [info] Here are other dependency conflicts that were resolved: [info] * com.typesafe:config:1.3.1 is selected over 1.2.0 [info] +- com.typesafe.akka:akka-actor_2.12:2.5.3 (depends on 1.3.1) [info] +- com.typesafe:ssl-config-core_2.12:0.2.2 (depends on 1.2.0) [info] * com.typesafe:ssl-config-core_2.12:0.2.2 is selected over 0.2.1 [info] +- com.typesafe.play:play-ws-standalone_2.12:1.0.1 (depends on 0.2.2) [info] +- com.typesafe.akka:akka-stream_2.12:2.5.3 (depends on 0.2.1)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Note: Apache Ivy emulates the semantics of nearest-wins



?

When resolving module dependencies from the Maven repository, Ivy converts the POM



file and puts the force="true"



attribute in ivy.xml



in the cache.

For example, cat ~/.ivy2/cache/com.typesafe.akka/akka-actor_2.12/ivy-2.5.3.xml



:

  <dependencies> <dependency org="org.scala-lang" name="scala-library" rev="2.12.2" force="true" conf="compile->compile(*),master(compile);runtime->runtime(*)"/> <dependency org="com.typesafe" name="config" rev="1.3.1" force="true" conf="compile->compile(*),master(compile);runtime->runtime(*)"/> <dependency org="org.scala-lang.modules" name="scala-java8-compat_2.12" rev="0.8.0" force="true" conf="compile->compile(*),master(compile);runtime->runtime(*)"/> </dependencies>
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Ivy documentation says:

These two latest



conflict managers take into account the force



dependency attribute.

Thus, direct dependencies can declare a force



attribute (see dependency), indicating that of direct dependency and indirect revisions, preference should be given to direct dependency revisions.

For me, this formulation means that force="true"



conceived in order to redefine the latest-wins



logic and emulate the nearest-wins



semantics. But, fortunately, this was not destined to happen, and we now have the latest-wins



: as we can see, sbt 1.2.8



picks up com.typesafe:config:1.3.1



.

However, one can observe the effect of force="true"



when using a strict conflict manager that seems to be broken.

 ThisBuild / conflictManager := ConflictManager.strict
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

The problem is that a strict conflict manager does not seem to prevent version substitution. show externalDependencyClasspath



cheerfully returns com.typesafe:config:1.3.1



.

A related problem is that adding a version of com.typesafe:config:1.3.1



, which a strict conflict manager put in the graph, leads to an error.

 ThisBuild / scalaVersion := "2.12.8" ThisBuild / organization := "com.example" ThisBuild / version := "1.0.0-SNAPSHOT" ThisBuild / conflictManager := ConflictManager.strict lazy val root = (project in file(".")) .settings( name := "foo", libraryDependencies ++= List( "com.typesafe.play" %% "play-ws-standalone" % "1.0.1", "com.typesafe" % "config" % "1.3.1", ) )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

It looks like this:

 sbt:foo> show externalDependencyClasspath [info] Updating ... [error] com.typesafe#config;1.2.0 (needed by [com.typesafe#ssl-config-core_2.12;0.2.2]) conflicts with com.typesafe#config;1.3.1 (needed by [com.example#foo_2.12;1.0.0-SNAPSHOT]) [error] org.apache.ivy.plugins.conflict.StrictConflictException: com.typesafe#config;1.2.0 (needed by [com.typesafe#ssl-config-core_2.12;0.2.2]) conflicts with com.typesafe#config;1.3.1 (needed by [com.example#foo_2.12;1.0.0-SNAPSHOT])
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

About versioning

We mentioned the latest-wins



semantics, suggesting that versions in a string representation may occur in some order.

Therefore, versioning is part of the semantics.

Versioning Procedure in Apache Ivy

This Javadoc comment says that when creating the comparator of versions, Ivy focused on the function of comparing versions from PHP :

This function first replaces _, - and + with a dot .



in string representations of versions and also adds .



before and after everything that is not a number. So, for example, '4.3.2RC1' becomes '4.3.2.RC.1'. She then compares the parts received from left to right.



For parts containing special elements ( dev



, alpha



or a



, beta



or b



, RC



or rc



, #



, pl



or p



) *, the elements are compared in the following order:



any string that is not a special element <dev <alpha = a <beta = b <RC = rc <# <pl = p.



Thus, not only different levels (for example, '4.1' and '4.1.2') can be compared, but PHP-specific versions containing information about the development status.

* approx. perev.

We can check how versions are ordered by writing a small function.

 scala> :paste // Entering paste mode (ctrl-D to finish) val strategy = new org.apache.ivy.plugins.latest.LatestRevisionStrategy case class MockArtifactInfo(version: String) extends org.apache.ivy.plugins.latest.ArtifactInfo { def getRevision: String = version def getLastModified: Long = -1 } def sortVersionsIvy(versions: String*): List[String] = { import scala.collection.JavaConverters._ strategy.sort(versions.toArray map MockArtifactInfo) .asScala.toList map { case MockArtifactInfo(v) => v } } // Exiting paste mode, now interpreting. scala> sortVersionsIvy("1.0", "2.0", "1.0-alpha", "1.0+alpha", "1.0-X1", "1.0a", "2.0.2") res7: List[String] = List(1.0-X1, 1.0a, 1.0-alpha, 1.0+alpha, 1.0, 2.0, 2.0.2)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Versioning Procedure in Coursier

The GitHub page on dependency resolution semantics has a section on versioning.

Coursier uses Maven's tailored versioning order. Before comparing, the string representations of the versions are broken down into separate elements ...

To obtain such elements, versions are separated by the characters., -, and _ (and the separators themselves are discarded), and by letter-to-number or number-to-letter replacements.

To write a test, create a subproject with libraryDependencies += "io.get-coursier" %% "coursier-core" % "2.0.0-RC2-6"



and run console



:

 sbt:foo> helper/console [info] Starting scala interpreter... Welcome to Scala 2.12.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_212). Type in expressions for evaluation. Or try :help. scala> import coursier.core.Version import coursier.core.Version scala> def sortVersionsCoursier(versions: String*): List[String] = | versions.toList.map(Version.apply).sorted.map(_.repr) sortVersionsCoursier: (versions: String*)List[String] scala> sortVersionsCoursier("1.0", "2.0", "1.0-alpha", "1.0+alpha", "1.0-X1", "1.0a", "2.0.2") res0: List[String] = List(1.0-alpha, 1.0, 1.0-X1, 1.0+alpha, 1.0a, 2.0, 2.0.2)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

As it turns out, Coursier orders version numbers in a completely different order than Ivy.

If you used permissive alphabetic tags, then this ordering can cause some confusion.

About version ranges

Usually, I avoid using version ranges, although they are widely used in webjars and npm modules republished on Maven Central. Something like "is-number": "^4.0.0"



may be written in the module "is-number": "^4.0.0"



which will correspond to [4.0.0,5)



.

Version range handling in Apache Ivy

In this assembly, angular-boostrap:0.14.2



depends on angular:[1.3.0,)



.

 ThisBuild / scalaVersion := "2.12.8" ThisBuild / organization := "com.example" ThisBuild / version := "1.0.0-SNAPSHOT" lazy val root = (project in file(".")) .settings( name := "foo", libraryDependencies ++= List( "org.webjars.bower" % "angular" % "1.4.7", "org.webjars.bower" % "angular-bootstrap" % "0.14.2", ) )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Calling show externalDependencyClasspath



in sbt 1.2.8



will return angular-bootstrap:0.14.2



and angular:1.7.8



. And where did 1.7.8



go? When Ivy encounters a range of versions, it essentially goes to the Internet and finds what it can manage to find, sometimes even using screenscraping.

Such processing of version ranges makes assemblies non-repeating (running the same assembly every few months gives you different results).

Handling version ranges in Coursier

Coursier dependency resolution section on github page

reads:

Specific versions at intervals are preferred

If your module has a dependency on [1.0,2.0) and 1.4, version approval will be performed in favor of 1.4.

If there is a dependency on 1.4, then this version will be preferred in the range [1.0,2.0).

It looks promising.

 sbt:foo> show externalDependencyClasspath [warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings. [info] * Attributed(/Users/eed3si9n/.sbt/boot/scala-2.12.8/lib/scala-library.jar) [info] * Attributed(/Users/eed3si9n/.coursier/cache/v1/https/repo1.maven.org/maven2/org/webjars/bower/angular/1.4.7/angular-1.4.7.jar) [info] * Attributed(/Users/eed3si9n/.coursier/cache/v1/https/repo1.maven.org/maven2/org/webjars/bower/angular-bootstrap/0.14.2/angular-bootstrap-0.14.2.jar)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

show externalDependencyClasspath



on the same assembly with angular-bootstrap:0.14.2



returns angular-bootstrap:0.14.2



and angular:1.4.7



as expected. This is an improvement over Ivy.

Consider the more complicated case when multiple disjoint version ranges are used. For example:

 ThisBuild / scalaVersion := "2.12.8" ThisBuild / organization := "com.example" ThisBuild / version := "1.0.0-SNAPSHOT" lazy val root = (project in file(".")) .settings( name := "foo", libraryDependencies ++= List( "org.webjars.npm" % "randomatic" % "1.1.7", "org.webjars.npm" % "is-odd" % "2.0.0", ) )
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Calling show externalDependencyClasspath



in sbt 1.3.0-RC3



returns the following error:

 sbt:foo> show externalDependencyClasspath [info] Updating https://repo1.maven.org/maven2/org/webjars/npm/kind-of/maven-metadata.xml No new update since 2018-03-10 06:32:27 https://repo1.maven.org/maven2/org/webjars/npm/is-number/maven-metadata.xml No new update since 2018-03-09 15:25:26 https://repo1.maven.org/maven2/org/webjars/npm/is-buffer/maven-metadata.xml No new update since 2018-08-17 14:21:46 [info] Resolved dependencies [error] lmcoursier.internal.shaded.coursier.error.ResolutionError$ConflictingDependencies: Conflicting dependencies: [error] org.webjars.npm:is-number:[3.0.0,4):default(compile) [error] org.webjars.npm:is-number:[4.0.0,5):default(compile) [error] at lmcoursier.internal.shaded.coursier.Resolve$.validate(Resolve.scala:394) [error] at lmcoursier.internal.shaded.coursier.Resolve.validate0$1(Resolve.scala:140) [error] at lmcoursier.internal.shaded.coursier.Resolve.$anonfun$ioWithConflicts0$4(Resolve.scala:184) [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14) [error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307) [error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41) [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) [error] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [error] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [error] at java.lang.Thread.run(Thread.java:748) [error] (update) lmcoursier.internal.shaded.coursier.error.ResolutionError$ConflictingDependencies: Conflicting dependencies: [error] org.webjars.npm:is-number:[3.0.0,4):default(compile) [error] org.webjars.npm:is-number:[4.0.0,5):default(compile)
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
     

Technically, that's right, because these ranges do not overlap. While sbt 1.2.8



allows this in is-number:4.0.0



.

Due to the fact that version ranges are common enough to be annoying, I submit a Pull Request to Coursier to implement additional latest-wins



semantics rules that allow you to select later versions from the lower bounds of the ranges.

See coursier / coursier # 1284 .

Conclusion

The resolver semantics define a specific classpath based on user-defined constraints.

Usually, differences in details are manifested in different ways of resolving version conflicts.

• Maven uses the nearest-wins
  
  
  
  strategy, which can downgrade transitive dependencies.
• Ivy applies the latest-wins
  
  
  
  strategy.
• Coursier primarily uses the latest-wins
  
  
  
  strategy, while trying to set versions more strictly.
• The version range handler in Ivy goes to the Internet, which makes the same build non-repeatable.
• Coursier and Ivy organize string representations of versions in very different ways.

Not even such subtleties of the Scala ecosystem will be discussed at ScalaConf on November 26 in Moscow. Artem Seleznev will introduce the practice of working with the database in functional programming without JDBC. Wojtek Pitula will talk about integration and tell how he created an application in which he placed all the working libraries. And 16 more reports full of technical hardcore will be presented at the conference.