How to fix the dependency graph to resolve deduplicate errors? - sbt

Question

I'm facing some deduplicate errors like below, which indicate that some of my dependencies import other dependencies which contains files with the same path name. Since they have the same path, they cannot be included together in the jar file that I'm trying to create with sbt-assembly.
I understand that the clean way to fix it is by fixing my dependencies. The conflicting dependencies from the example below seem to be reactive-streams and reactive-streams-flow-adapters, but I'm not sure what they are and where they come from. How can I find which of my dependencies are importing them?
If I can figure that out, how can I fix it? Is there a way other than just removing one of them?
An example of deduplicate errors:
[error] (assembly) deduplicate: different file contents found in the following:
[error] /home/guillaume/.cache/coursier/v1/https/repo1.maven.org/maven2/org/reactivestreams/reactive-streams-flow-adapters/1.0.2/reactive-streams-flow-adapters-1.0.2.jar:org/reactivestreams/FlowAdapters$FlowPublisherFromReactive.class
[error] /home/guillaume/.cache/coursier/v1/https/repo1.maven.org/maven2/org/reactivestreams/reactive-streams/1.0.3/reactive-streams-1.0.3.jar:org/reactivestreams/FlowAdapters$FlowPublisherFromReactive.class
Here is a list of my dependencies, if it helps:
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.12.2",
libraryDependencies += "com.novocode" % "junit-interface" % "0.11" % "test",
libraryDependencies += "com.softwaremill.sttp.client3" %% "core" % "3.3.6",
libraryDependencies += "com.softwaremill.sttp.client3" %% "httpclient-backend-zio" % "3.3.6",
libraryDependencies += "dev.zio" %% "zio" % "1.0.9",
// libraryDependencies += "dev.zio" %% "zio-streams" % "1.0.9",
libraryDependencies += "org.apache.commons" % "commons-compress" % "1.20",
libraryDependencies += "org.scalactic" %% "scalactic" % "3.2.9",
libraryDependencies += "org.scalatest" %% "scalatest" % "3.2.9" % "test",
libraryDependencies += ("tech.sparse" %% "toml-scala" % "0.2.2").cross(CrossVersion.for3Use2_13),

Answer 1

I'm not familiar with Reactive Streams, but it looks like they made some changes during the patch version and moved classes around in 1.0.3. See Reactive Streams 1.0.3 is here!
When we released 1.0.2, we shipped a compatibility/conversion library to seamlessly convert between the java.util.concurrent.Flow and the org.reactivestreams namespaces—in 1.0.3 these adapters are instead included in the main 1.0.3 jar.
Investigate transitive dependencies using dependencyTree
How can I find which of my dependencies are importing them?
If you're using the latest sbt 1.5.4, it has Johannes's dependency graph plugin built-in by default, so you can run dependencyTree from sbt shell:
sbt:root> dependencyTree
[info] root:root_2.13:0.1.0-SNAPSHOT [S]
....
[info] +-com.softwaremill.sttp.client3:httpclient-backend-zio_2.13:3.3.6 [S]
[info] | +-com.softwaremill.sttp.client3:httpclient-backend_2.13:3.3.6 [S]
[info] | | +-com.softwaremill.sttp.client3:core_2.13:3.3.6 [S]
[info] | | | +-com.softwaremill.sttp.model:core_2.13:1.4.7 [S]
[info] | | | +-com.softwaremill.sttp.shared:core_2.13:1.2.5 [S]
[info] | | | +-com.softwaremill.sttp.shared:ws_2.13:1.2.5 [S]
[info] | | | +-com.softwaremill.sttp.model:core_2.13:1.4.7 [S]
[info] | | | +-com.softwaremill.sttp.shared:core_2.13:1.2.5 [S]
[info] | | |
[info] | | +-org.reactivestreams:reactive-streams-flow-adapters:1.0.2
[info] | | +-org.reactivestreams:reactive-streams:1.0.2 (evicted by: 1.0.3)
[info] | | +-org.reactivestreams:reactive-streams:1.0.3
....
This tells us that org.reactivestreams:reactive-streams-flow-adapters:1.0.2 was pulled in from com.softwaremill.sttp.client3:httpclient-backend_2.13:3.3.6.
Excluding reactive-streams-flow-adapters
If I can figure that out, how can I fix it? Is there a way other than just removing one of them?
In this case, reactive-streams-flow-adapters is subsumed by reactive-streams post-1.0.3, so I think we can exclude it from the dependency as follows:
libraryDependencies += ("com.softwaremill.sttp.client3" %% "httpclient-backend-zio" % "3.3.6")
.exclude("org.reactivestreams", "reactive-streams-flow-adapters"),
Ignoring module-info.class
To ignore module-info.class we can add this to build.sbt:
ThisBuild / assemblyMergeStrategy := {
case "module-info.class" => MergeStrategy.discard
case x =>
val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
oldStrategy(x)
}
Result
sbt:root> assembly
[info] Strategy 'discard' was applied to 10 files (Run the task at debug level to see details)
[success] Total time: 3 s, completed Jun 26, 2021 6:02:04 PM