[MNG-7906] Dependency Management import (BOM) does not work the "maven way"

[jira-importer]

Tamas Cservenak opened MNG-7906 and commented

This affects all released Maven versions so far (precisely since 2.0.9 MNG-3220).

Problem reproducer: https://github.com/cstamas/MNG-7852 (repo name is wrong, obviously).

In short: unlike with dependencies, where you CAN override some "deep transitive" dependency by re-declaring it directly as 1st level dependency in POM, for depMgt import this does not work, actually, it works quite the opposite ("first comes, wins"). Moreover, Maven remains silent about this, as reproducer shows, and all of this goes unnoticed.

Solution: at least depMgt import should make "the maven way", maybe not by default (to not break existing builds) but configurable. Problem is solved if in reproducer:

• with fix enabled, junit 5.9.3 is used, AND
• with fix disabled, Maven yells about ignored depMgt import

---

Issue Links:

• MNG-3220 Allow managed dependencies to be imported into other projects
  ("is caused by")

• MPH-183 Effective-pom + verbose should show import path to BOM dependencyManagement

• MNG-7344 Effective pom should contain more finegrained details regarding its content origin: track dependencyManagement import

• MNG-7854 Non directly managed imported and conflicting depMgt entries should warn

Remote Links:

• GitHub Pull Request #1396
  

0 votes, 7 watchers

[jira-importer]

Hervé Boutemy commented

I know it's suprising, particularly when we have no way to see the path to an imported dependencyManagement (see MPH-183 MNG-7344)

I'm not convinced having the latest import override first imports really makes sense: order in dependency declaration matters

but for sure, currently debugging nested dependencyManagement import is hard: I saw it on sigstore-maven-plugin 0.4.0 where the import tree is huge

[jira-importer]

Tamas Cservenak commented

Well, in maven deps it work like it: the "closer" it is, it "wins".
For example, if you have guice as dependency, whether you declare direct dependency on guava before or after guice, it will "win".

But in case of depMgr this is NOT the case, and is counter intuitive. It will work ONLY if you declare depMgt entry BEFORE (as it is "first wins" implemented).

[jira-importer]

Hervé Boutemy commented

MPH-183 is a first step: be able to know where the effective import comes from

then on the dedupe algorithm and clarification, we'll be able to discuss: aalmiray is doing a series of talks in many conferences for 1 or 2 years on Maven Dependency surprises, we should probably ask him to share his examples so we can improve explanations and feedback when building

on this notion of "closer", we need to define that it means depth first then order

once we explicit that, in the case of dependencyManagement, there is no depth, then there is only order: that's the same, just a notion that de-facto does not exist

When I saw Andres talk first, I found that most cases he reported were due to cases like that: no clarity on what a vague term like "closer" means, then surprises when trying to apply in a context that has different initial facts. But no rush on conclusions, let's go step by step on this dependencyManagement import clarification (and I have to fight against saying "dependency import", because dependencyManagement is fundamentally different from dependency: there is no transitivity, then no depth, then consequences)

[jira-importer]

Elliotte Rusty Harold commented

I went down this rabbit hole for 2+ years, and yes, it's confusing. However, that ship has sailed. I don't see any realistic possibility to change the rules now without breaking a large part of the existing Maven and Java ecosystem.

I've done a lot of work on improving the documentation of the admittedly weird Maven dependency resolution rules, e.g. using the word "closer" without defining that word in sufficient detail, but if there are places in the official docs that are still confusing, feel free to file docs bugs and assign them to me. Beyond that, I'm tempted to close this as Won't Fix.

[jira-importer]

Elliotte Rusty Harold commented

Also, I'm very hesitant to introduce more flags and switches into Maven, whether in pom.xml or the CLI. It's already overly complex and confusing.

[jira-importer]

Guillaume Nodet commented

While MPH-183 could be a way to investigate the problem, not reporting anything is plainly and simply wrong.
Let's take a simple use case:

• in your project, you see a wrong version of a dependency being imported
• you add a <dependency> section in your dependencyManagement POM section, usually at the end
• you rebuild, but the wrong version is still used
• you raise a JIRA after having spent hours to investigate the problem with no solution

So I think the first step would be to add a WARNING when the POM being built contains a managed dependency which is lost because not first.
This will explain to the user what's wrong and that the dependency should be moved up at the top of the managed dependency section in order to be useful. That should be easily done and back ported to 3.9.x.

The behaviour could then be changed in 4.x with or without a toggle flag for compatibility.

But definitely, the first step should be to WARN the user that there's a problem.

[jira-importer]

Henning Schmiedehausen commented

Tamas Cservenak pinged me on basepom/maven-plugins#10

FWIW, I am very much with Elliotte Rusty Harold. The current behavior exists for close to 20 years and changing it would break large parts of the ecosystem.

Frankly, the fact that there are multiple ways to do dep resolution over the years, the ever-changing nature of the maven resolver and the complicated and under-documented way to access repositories make it hard for maven users to get this right and near impossible for plugin writers to builds something that matches the core behavior.

"effective pom" is not a debugging tool for the faint of heart. running it on a major project makes it near impossible to understand its output.

If you want a tool that actually works for humans, try https://github.com/basepom/dependency-versions-check-maven-plugin

It produces output like this:

[INFO] All dependencies for 'compile' scope:
[INFO] com.fasterxml.jackson.core:jackson-annotations: compile 2.13.4 (2.10.3)
[INFO] com.github.docker-java:docker-java-api: compile 3.2.13
[INFO] com.github.docker-java:docker-java-transport: compile 3.2.13
[INFO] com.github.docker-java:docker-java-transport-zerodep: compile 3.2.13
[INFO] com.google.code.findbugs:jsr305: compile 1.3.9
[INFO] junit:junit: compile 4.13.2
[INFO] net.bytebuddy:byte-buddy: compile 1.10.14 (!1.12.10!)
[INFO] net.java.dev.jna:jna: compile 5.8.0
[INFO] org.apache.commons:commons-compress: compile 1.21
[INFO] org.apache.flink:flink-shaded-force-shading: compile 16.2
[INFO] org.apache.logging.log4j:log4j-api: compile 2.17.1
[INFO] org.apache.logging.log4j:log4j-core: compile 2.17.1
[INFO] org.apache.logging.log4j:log4j-slf4j-impl: compile 2.17.1
[INFO] org.apiguardian:apiguardian-api: compile 1.1.2
[INFO] org.assertj:assertj-core: compile 3.23.1
[INFO] org.jetbrains:annotations: compile 17.0.0
[INFO] org.junit.jupiter:junit-jupiter: compile 5.9.1
[INFO] org.junit.jupiter:junit-jupiter-api: compile 5.9.1
[INFO] org.junit.jupiter:junit-jupiter-params: compile 5.9.1
[INFO] org.junit.platform:junit-platform-commons: compile 1.9.1 (1.8.2)
[INFO] org.junit.platform:junit-platform-engine: compile 1.9.1
[INFO] org.junit.vintage:junit-vintage-engine: compile 5.9.1
[INFO] org.opentest4j:opentest4j: compile 1.2.0
[INFO] org.rnorth.duct-tape:duct-tape: compile 1.0.8
[INFO] org.slf4j:slf4j-api: compile 1.7.36 (1.7.25, 1.7.35)
[INFO] org.testcontainers:testcontainers: compile 1.17.2

which users can actually use to figure out what is wrong with their build. It does approximate what the resolver does (and there is no way for me to actually validate that it matches the core behavior).

[jira-importer]

Henning Schmiedehausen commented

the major problem of each of these resolver discussions is BTW summarized on this page: https://basepom.github.io/dependency-versions-check-maven-plugin/development/

Apache Maven operates under the assumption that every dependency is always backwards compatible and when two versions are compared, using the “higher” version will satisfy any dependency.

As long as this assumption is baked into maven (IAW, replacing "0.0.1" with "1.0.0" is fine) and maven does not fail as soon as it finds two different versions of the same dependency referenced in the dependency tree (with subsequent human input to decide whether two versions for a given artifact are compatible or not), I see no real point to discuss how dependency resolution should work and which version should "win".

Other ecosystems have e.g. baked Semantic versioning into their dependency model. Java/Maven has not (because it predates it by years). 

Unless Maven starts encouraging (enforcing?) artifacts to create a version policy (e.g. "this artifact follows semantic versioning") and maven can consume that and behave accordingly, I am not sure why we try to "fix" the resolve behavior. The best bet IMHO is to simply accept that we have settled on the way the resolver works today, the maven users rely on it and stop trying to improve it.  

e.g. semantic versioning would mean that you can NOT replace 1.0.0 with 2.0.0. Which maven happily does today. 

[jira-importer]

Tamas Cservenak commented

I have several issues with statements in previous post, but the major one is this: it references a plugin page that has following statement "Apache Maven operates under the assumption that every dependency is always backwards compatible and when two versions are compared, using the 'higher' version will satisfy any dependency". Am unsure from where Henning Schmiedehausen takes this to be granted, or even assumes this statement is true, as source code of resolver does NOT show this at all (or please point me to the code that implements this behaviour).

What I know, is that we most probably talk about "conflict resolution". The Javadoc of conflict resolver is here: https://github.com/apache/maven-resolver/blob/master/maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/ConflictResolver.java#L37-L54

It points us ("The exact rules by which a winning node and its effective scope are determined are controlled by user-supplied implementations of VersionSelector...") toward version selector: https://github.com/apache/maven-resolver/blob/master/maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/NearestVersionSelector.java#L41-L42

In short, the statement quoted does not stand at all ("operates under the assumption that every dependency is always backwards compatible and when two versions are compared, using the 'higher' version will satisfy any dependency") for Maven3. Someone very interested in this topic may do a homework, and dig into Maven2 source code and be surprised.

Maven will choose "nearest version" not "highest version". As graph root is level 0, is your POM or root dependency of graph being collected, each node has well defined distance from root: the length of the path you need to make to get from root to given node. And the most interesting thing: user DO have saying here, as direct dependencies on POM have distance 1, so they will always win. In other words, user CAN affect what is chosen for "winner".

[jira-importer]

Hervé Boutemy commented

So I think the first step would be to add a WARNING when the POM being built contains a managed dependency which is lost because not first.
This will explain to the user what's wrong and that the dependency should be moved up at the top of the managed dependency section in order to be useful. That should be easily done and back ported to 3.9.x.

this idea about warning only on the POM being built is the right approach: not too noisy, actionable, not breaking anything

(going back to the dependencyManagement import topic)

[jira-importer]

Henning Schmiedehausen commented

Tamas Cservenak you literally made my point. I was not about "highest version wins" (I wrote that but it is incorrect as you pointed out). My main point is that maven works under the assumption that every version is always backwards compatible to each other. 

If you have a dependency A that uses library "1.0.0" and then add that library to your pom with version "2.0.0", you end up with "2.0.0". And that is {}wrong{}. The build should fail. Because there is no guarantee that you can replace 1.0.0 with 2.0.0.

The other way around is worse: if you have a library that declares a dependency "1.1.0" and then you use the "1.0.0" dependency, you downgrade a transitive dependency with a non-compatible version. 

This behavior is the sole reason why the dvc plugin exists. 

 

I very much heard the point that this statement is incorrect, but not because "best wins" but "nearest wins".  So let me rephrase that (and I will update the plugin pages) : 

Apache Maven operates under the assumption that it can replace any dependency version with any other version based on a "closest to the root wins" algorithm, that makes it almost impossible to predict which version of a dependency will end up in the build unless all used dependencies are locked down (e.g. by adding entries and the resulting build tree is analyzed to ensure that the different packages end up with versions that are compatible. Maven has no builtin concept of backward or forward compatible versions, it assumes that every version can be replaced with any other version based on this "nearest is best" algorithm and the resulting tree still makes sense.

Not sure that this makes your life easier or better. Let me know if you can agree or disagree that this statement reflects reality better. 

My recommendation would be that we start adopting an actual compatibility policy (semantic versioning seems to have won) but allow projects to override that policy (maybe with a new element in the POM; 4.0.0 may be a good opportunity). But the current behavior is so poorly understood that people are bound to get it wrong. I got it wrong. :) 

 

 

 

[jira-importer]

Guillaume Nodet commented

I understand better, and I agree that the fact Maven does not enforce any compatibility policy can lead to wrong results.  But I would point out that it has been the case for 20 years and I had not heard much complaint about it.

Another point worth mentioning is that Maven also operates under the assumption that everything uses a flat class loader, which actually leads to the first point.  If we consider the OSGi world, the system allows multiple versions of the same artifact…

Another point about the compatibility policy: this would have to be defined in the POM, but I’m not sure that using semantic versioning is a good idea, as those are just guidelines and may not reflect the reality.  If we were to add something to the POM, I think each artifact should have a way to tell with which artifact it is backward compatible : I.e. when publishing version X, it’s mainly about indicating which version Y <= X to define a compatibility range.  If using server, it would be the previous major version.  But this would need a way to force a version anyway.  This is quite a tricky area…

[jira-importer]

Tamas Cservenak commented

Moreover, to extend Guillaumes point: "project version" G:A:1.x can change to G:A2:2.x as Maven GAV (you have two more coordinates to "express" the intent) is not "version only" as in npm, ruby-gems etc. You could even "encode" major version in A, so shift in x.y.z version into groupId:artifact-x:y.z or alike (this is just for example sake). Moreover, there is a long history of changes like these even in ASF history.

This is a reason more why "semantic versioning" is not one-to-one applicable to Maven universe, simply put, maven universe is "broader" than that laid down by SemVer.

[jira-importer]

Lenny Primak commented

I would think that 4.0 would be the place to break the compatibility.

First of all, I don't think it's as big of a deal, as depMgmt import isn't as nearly widely used, and the actual conflicting use cases are few and far between (IMHO)

I would put this behind a system property so this can be easily switched back within settings.xml or an individual pom file, as well as the command line.

 

[jira-importer]

Elliotte Rusty Harold commented

This is confusing and surprising. If we were working from a green field, I would agree with changing this. However, we're not. I don't think we should add a configurable option to change this. Adding more options only increases the complexity, and makes it all the more confusing. We should document this carefully and fully, but we should not change the behavior.

At this late date it's hard to see this as a bug. It's really de facto working as intended.