I have the following build.sbt file.
import AssemblyKeys._
name := "approxstrmatch"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies+="org.apache.spark" %% "spark-core" % "1.0.0"
resolvers += "AkkaRepository" at "http://repo.akka.io/releases/"
// My merge strategy is specified here.
lazy val app = Project("approxstrmatch", file("approxstrmatch"),
settings = buildSettings ++ assemblySettings ++ Seq(
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "servlet", xs # _*) => MergeStrategy.first
case PathList("javax", "transaction", xs # _*) => MergeStrategy.first
case PathList("javax", "mail", xs # _*) => MergeStrategy.first
case PathList("javax", "activation", xs # _*) => MergeStrategy.first
case PathList(ps # _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x => old(x)
}
})
)
mainClass in assembly := Some("approxstrmatch.JaccardScore")
// jarName in assembly := "approstrmatch.jar"
When I execute the following command sbt assembly-merge-strategy there's an error I don't understand. Any help appreciated.
approxstrmatch]$ sbt assembly-merge-strategy
[info] Loading project definition from /apps/sameert/software/approxstrmatch/project
[info] Set current project to approxstrmatch (in buildfile:/apps/sameert/software/approxstrmatch/)
[error] Not a valid command: assembly-merge-strategy
[error] No such setting/task
My understanding tells me there's no assembly-merge-strategy task in sbt-assembly plugin (I can only suspect you use that plugin in your build).
Execute assembly as described in https://github.com/sbt/sbt-assembly#assembly-task as "an awesome new assembly task which will compile your project, run your tests, and then pack your class files and all your dependencies into a single JAR file".
There is a setting named assemblyMergeStrategy (aka assembly-merge-strategy). It's just that you won't directly use it. The way sbt-assembly uses it is scoped to assembly task:
mergeStrategy in assembly <<= ....
So here's what you have to do to call it from the shell:
$ sbt assembly::assemblyMergeStrategy
[info] blabla other things...
[info] <function1>
add assemblySettings in your build.sbt will help
Related
In my project there are 3 sub projects under root. build.sbt is as below.
proj_C depends on proj_A and proj_B.
If I created the assembly proj_C package with below command. It success and the assembly package could be imported in other projects.
sbt "project proj_C" assembly
If I publish with "sbt publish", as I defined addArtifact in proj_C settings, an assembly jar package is also generated and then published. But when I try to compile another project which imports this assembly jar, it will below error
[error] unresolved dependency: proj_A;1.0.0: not found
part of build.sbt is as below. Could anyone point me what I made wrong in my way?
Many thanks!
artifact in (Compile, assembly) := {
val art = (artifact in (Compile, assembly)).value
art.withClassifier(Some("assembly"))
}
lazy val assemblySettings = Seq(
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
}
)
lazy val root = Project(base = file("."))
.disablePlugins(sbtassembly.AssemblyPlugin)
.aggregate(proj_A, proj_B, proj_C)
.settings(
commonSettings,
skip in publish := true,
name := "proj_root"
)
lazy val proj_A= (project in file("proj_A"))
.disablePlugins(sbtassembly.AssemblyPlugin)
.settings(
commonSettings,
skip in publish := true,
name := "proj_A"
)
lazy val proj_B= (project in file("proj_B"))
.disablePlugins(sbtassembly.AssemblyPlugin)
.settings(
commonSettings,
skip in publish := true,
name := "proj_B"
)
lazy val proj_C= (project in file("proj_C"))
.settings(
commonSettings,
assemblySettings,
addArtifact(artifact in (Compile, assembly), assembly),
name := "proj_C"
) dependsOn(proj_A, proj_B)
First of all, I hope you know that the publishing of the fat jar is not recommended. And to be honest, in your case I really see no benefit in doing so.
If you simply publish A, B, C separately and then add the dependency in your other project it will all be automatically downloaded (along with dependencies of those projects). And the dependency management will be much easier...
But, since you want to add the A-assembly dependency, by the error I guess that you are actually adding the wrong jar. My guess would be that you publish both C.jar and C-assembly.jar, and you added the dependency like:
"your.organisation" %% "C" % "version"
but you should have:
"your.organisation" %% "C" % "version" classifier "assembly"
So with one dependency I've entered a level of entanglement that I can't escape. I hate to think what will happen when I bring in the commented jars:
libraryDependencies ++= Seq(
// "org.apache.avro" % "avro" % "1.8.1" excludeAll ExclusionRule(organization = "log4j"),
// "org.apache.kafka" %% "kafka" % "0.10.0.0",
"org.apache.hive" % "hive-jdbc" % "1.2.2"
excludeAll ExclusionRule(organization = "log4j")
exclude("org.apache.hadoop", "hadoop-yarn-api"),
"log4j" % "log4j" % "1.2.16"
)
Using sbt assembly, I am getting the following deduplicate problem:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-api-jdo\jars\datanucleus-api-jdo-3.2.6.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-core\jars\datanucleus-core-3.2.10.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-rdbms\jars\datanucleus-rdbms-3.2.9.jar:plugin.xml
Where I'm stuck is trying to find a merge strategy that allows these three jars to happily coexist in one fat jar. I've tried several variations of the strategy below, but am making no progress:
assemblyMergeStrategy in assembly := {
case PathList("javax", "transaction", xs # _*) => MergeStrategy.first
case PathList(xs # _*) if xs.last endsWith "plugin.xml" => MergeStrategy.discard
// case PathList("org", "datanucleus", "datanucleus-api-jdo", xs # _*) => MergeStrategy.last
// case PathList("org", "datanucleus", "datanucleus-rdbms", xs # _*) => MergeStrategy.last
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
I must have some misunderstanding of how this works because it seems to me that the second line should find every plugin.xml file in every jar and nuke it.
Has anyone successfully included hive-jdbc in a fat jar?
UPDATE:
case "plugin.xml" => MergeStrategy.discard //or .last should work, I would think, but that throws:
[error] (*:assembly) java.util.NoSuchElementException
So I load up the project this morning and give it one more try. This time
case "plugin.xml" => MergeStrategy.last
does not throw, and I have a fat jar.
Guh.
UPDATE:
I don't know where the problem lies (Intellij, SBT Console, sbt-assembly), but there is some caching going on that makes it really difficult to troubleshoot merge issues. I'm finding that the only reliable way to ensure that changes I make to the code in build.sbt are actually applied is to shut down the entire IDE and re-open it.
I am writing a akka application. While creating far jar of application , I dont want scala libraries to be packaged with the jar. My build.sbt looks as follows:
lazy val root = (project in file(".")).
settings(
name :="akka-app",
version :="1.0",
scalaVersion :="2.10.4",
mainClass in Compile := Some("sample.hello.HelloWorld")
)
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.3.4" % "provided",
"com.typesafe" % "config" % "1.2.1"
)
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
But this sbt packages scala with jar. I want only com.typesafe.config library to be present in the jar. Any solution how to achieve this?
You can exclude Scala by modifying the option in the assemblyOption setting:
assemblyOption in assembly :=
(assemblyOption in assembly).value.copy(includeScala = false)
I'm trying to run sbt assembly on my project but I get error saying:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
[error] ^
I have the following structure:
MyProject
- project
- assembly.sbt
- build.properties
- BuildSettings.scala
- MyProjectBuild.scala
- src
- main
- com
- mypkg
- MyMainClass.scala
I have the following in my assembly.sbt:
resolvers += Resolver.url("artifactory", url("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
// dont upgrade to 0.12.0 as there is assembly conflict
My build.properties is:
sbt.version=0.13.6
My BuildSettings.scala is:
import sbt._
import Keys._
object BuildSettings {
lazy val basicSettings = Seq[Setting[_]](
organization := "com.eon.vpp",
version := "0.1.0-SNAPAHOT",
description := "vpp metrics producer to a kafka instance",
scalaVersion := "2.11.7",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos
)
// sbt-assembly settings for building one fat jar
import sbtassembly.Plugin._
import AssemblyKeys._
lazy val sbtAssemblySettings = assemblySettings ++ Seq(
jarName in assembly := {
name.value + "-" + version.value + ".jar"
},
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
)
lazy val buildSettings = basicSettings ++ sbtAssemblySettings
}
Any suggestions as to why is this error?
Yes, I figured out what the problem was. I had to move the plugins.sbt file inside the project folder. It was that simple!
In build.scala I have the following:
mergeStrategy <<= (mergeStrategy in assembly) {(old) => {
case PathList("javax", "servlet", "resources", xs # _*) => MergeStrategy.first
case x => old(x)
}}
However when I run assembly I see:
[info] Merging 'javax/servlet/resources/web-app_2_2.dtd' with strategy 'deduplicate'
showing that it is using the "deduplicate" strategy, not the "first" strategy. This gives the following error:
[error] {file:/home/dan/tesla/}tesla-appengine/*:assembly: deduplicate: different file contents found in the following:
[error] /home/dan/.ivy2/cache/com.google.appengine/appengine-tools-sdk/jars/appengine-tools-sdk-1.7.3.jar:javax/servlet/resources/web-app_2_2.dtd
[error] /home/dan/.ivy2/cache/javax.servlet/servlet-api/jars/servlet-api-2.5.jar:javax/servlet/resources/web-app_2_2.dtd
I had not applied this setting in the proper scope. I needed to set
mergeStrategy in assembly <<= ...