In build.scala I have the following:
mergeStrategy <<= (mergeStrategy in assembly) {(old) => {
case PathList("javax", "servlet", "resources", xs # _*) => MergeStrategy.first
case x => old(x)
}}
However when I run assembly I see:
[info] Merging 'javax/servlet/resources/web-app_2_2.dtd' with strategy 'deduplicate'
showing that it is using the "deduplicate" strategy, not the "first" strategy. This gives the following error:
[error] {file:/home/dan/tesla/}tesla-appengine/*:assembly: deduplicate: different file contents found in the following:
[error] /home/dan/.ivy2/cache/com.google.appengine/appengine-tools-sdk/jars/appengine-tools-sdk-1.7.3.jar:javax/servlet/resources/web-app_2_2.dtd
[error] /home/dan/.ivy2/cache/javax.servlet/servlet-api/jars/servlet-api-2.5.jar:javax/servlet/resources/web-app_2_2.dtd
I had not applied this setting in the proper scope. I needed to set
mergeStrategy in assembly <<= ...
Related
So with one dependency I've entered a level of entanglement that I can't escape. I hate to think what will happen when I bring in the commented jars:
libraryDependencies ++= Seq(
// "org.apache.avro" % "avro" % "1.8.1" excludeAll ExclusionRule(organization = "log4j"),
// "org.apache.kafka" %% "kafka" % "0.10.0.0",
"org.apache.hive" % "hive-jdbc" % "1.2.2"
excludeAll ExclusionRule(organization = "log4j")
exclude("org.apache.hadoop", "hadoop-yarn-api"),
"log4j" % "log4j" % "1.2.16"
)
Using sbt assembly, I am getting the following deduplicate problem:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-api-jdo\jars\datanucleus-api-jdo-3.2.6.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-core\jars\datanucleus-core-3.2.10.jar:plugin.xml
[error] C:\Users\G517329\.ivy2\cache\org.datanucleus\datanucleus-rdbms\jars\datanucleus-rdbms-3.2.9.jar:plugin.xml
Where I'm stuck is trying to find a merge strategy that allows these three jars to happily coexist in one fat jar. I've tried several variations of the strategy below, but am making no progress:
assemblyMergeStrategy in assembly := {
case PathList("javax", "transaction", xs # _*) => MergeStrategy.first
case PathList(xs # _*) if xs.last endsWith "plugin.xml" => MergeStrategy.discard
// case PathList("org", "datanucleus", "datanucleus-api-jdo", xs # _*) => MergeStrategy.last
// case PathList("org", "datanucleus", "datanucleus-rdbms", xs # _*) => MergeStrategy.last
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
I must have some misunderstanding of how this works because it seems to me that the second line should find every plugin.xml file in every jar and nuke it.
Has anyone successfully included hive-jdbc in a fat jar?
UPDATE:
case "plugin.xml" => MergeStrategy.discard //or .last should work, I would think, but that throws:
[error] (*:assembly) java.util.NoSuchElementException
So I load up the project this morning and give it one more try. This time
case "plugin.xml" => MergeStrategy.last
does not throw, and I have a fat jar.
Guh.
UPDATE:
I don't know where the problem lies (Intellij, SBT Console, sbt-assembly), but there is some caching going on that makes it really difficult to troubleshoot merge issues. I'm finding that the only reliable way to ensure that changes I make to the code in build.sbt are actually applied is to shut down the entire IDE and re-open it.
I'm trying to run sbt assembly on my project but I get error saying:
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
[error] ^
I have the following structure:
MyProject
- project
- assembly.sbt
- build.properties
- BuildSettings.scala
- MyProjectBuild.scala
- src
- main
- com
- mypkg
- MyMainClass.scala
I have the following in my assembly.sbt:
resolvers += Resolver.url("artifactory", url("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
// dont upgrade to 0.12.0 as there is assembly conflict
My build.properties is:
sbt.version=0.13.6
My BuildSettings.scala is:
import sbt._
import Keys._
object BuildSettings {
lazy val basicSettings = Seq[Setting[_]](
organization := "com.eon.vpp",
version := "0.1.0-SNAPAHOT",
description := "vpp metrics producer to a kafka instance",
scalaVersion := "2.11.7",
scalacOptions := Seq("-deprecation", "-encoding", "utf8"),
resolvers ++= Dependencies.resolutionRepos
)
// sbt-assembly settings for building one fat jar
import sbtassembly.Plugin._
import AssemblyKeys._
lazy val sbtAssemblySettings = assemblySettings ++ Seq(
jarName in assembly := {
name.value + "-" + version.value + ".jar"
},
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
)
lazy val buildSettings = basicSettings ++ sbtAssemblySettings
}
Any suggestions as to why is this error?
Yes, I figured out what the problem was. I had to move the plugins.sbt file inside the project folder. It was that simple!
I have the following build.sbt file.
import AssemblyKeys._
name := "approxstrmatch"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies+="org.apache.spark" %% "spark-core" % "1.0.0"
resolvers += "AkkaRepository" at "http://repo.akka.io/releases/"
// My merge strategy is specified here.
lazy val app = Project("approxstrmatch", file("approxstrmatch"),
settings = buildSettings ++ assemblySettings ++ Seq(
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "servlet", xs # _*) => MergeStrategy.first
case PathList("javax", "transaction", xs # _*) => MergeStrategy.first
case PathList("javax", "mail", xs # _*) => MergeStrategy.first
case PathList("javax", "activation", xs # _*) => MergeStrategy.first
case PathList(ps # _*) if ps.last endsWith ".html" => MergeStrategy.first
case "application.conf" => MergeStrategy.concat
case "unwanted.txt" => MergeStrategy.discard
case x => old(x)
}
})
)
mainClass in assembly := Some("approxstrmatch.JaccardScore")
// jarName in assembly := "approstrmatch.jar"
When I execute the following command sbt assembly-merge-strategy there's an error I don't understand. Any help appreciated.
approxstrmatch]$ sbt assembly-merge-strategy
[info] Loading project definition from /apps/sameert/software/approxstrmatch/project
[info] Set current project to approxstrmatch (in buildfile:/apps/sameert/software/approxstrmatch/)
[error] Not a valid command: assembly-merge-strategy
[error] No such setting/task
My understanding tells me there's no assembly-merge-strategy task in sbt-assembly plugin (I can only suspect you use that plugin in your build).
Execute assembly as described in https://github.com/sbt/sbt-assembly#assembly-task as "an awesome new assembly task which will compile your project, run your tests, and then pack your class files and all your dependencies into a single JAR file".
There is a setting named assemblyMergeStrategy (aka assembly-merge-strategy). It's just that you won't directly use it. The way sbt-assembly uses it is scoped to assembly task:
mergeStrategy in assembly <<= ....
So here's what you have to do to call it from the shell:
$ sbt assembly::assemblyMergeStrategy
[info] blabla other things...
[info] <function1>
add assemblySettings in your build.sbt will help
I have a Scala project that is divided into several subprojects:
lazy val core: Seq[ProjectReference] = Seq(common, json_scalaz7, json_scalaz)
I'd like to make the core lazy val conditional on the Scala version I'm currently using, so I tried this:
lazy val core2: Seq[ProjectReference] = scalaVersion {
case "2.11.0" => Seq(common, json_scalaz7)
case _ => Seq(common, json_scalaz7, json_scalaz)
}
Simply speaking, I'd like to exclude json_scalaz for Scala 2.11.0 (when the value of the scalaVersion setting is "2.11.0").
This however gives me the following compilation error:
[error] /home/diego/work/lift/framework/project/Build.scala:39: type mismatch;
[error] found : sbt.Project.Initialize[Seq[sbt.Project]]
[error] required: Seq[sbt.ProjectReference]
[error] lazy val core2: Seq[ProjectReference] = scalaVersion {
[error] ^
[error] one error found
Any idea how to solve this?
Update
I'm using sbt version 0.12.4
This project is the Lift project, which compiles against "2.10.0", "2.9.2", "2.9.1-1", "2.9.1" and now we are working on getting it to compile with 2.11.0. So creating a compile all task would not be practical, as it would take a really long time.
Update 2
I'm hoping there is something like this:
lazy val scala_xml = "org.scala-lang.modules" %% "scala-xml" % "1.0.1"
lazy val scala_parser = "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.1"
...
lazy val common =
coreProject("common")
.settings(description := "Common Libraties and Utilities",
libraryDependencies ++= Seq(slf4j_api, logback, slf4j_log4j12),
libraryDependencies <++= scalaVersion {
case "2.11.0" => Seq(scala_xml, scala_parser)
case _ => Seq()
}
)
but for the projects list
Note how depending on the scala version, I add the scala_xml and scala_parser_combinator libraries
You can see the complete build file here
Cross building a project
Simply speaking, I'd like to exclude json_scalaz for Scala 2.11.0
The built-in support in sbt for this is called cross building, which is described in Cross-Building a Project. Here's from the section with a bit of correction:
Define the versions of Scala to build against in the crossScalaVersions setting. For example, in a .sbt build definition:
crossScalaVersions := Seq("2.10.4", "2.11.0")
To build against all versions listed crossScalaVersions, prefix the action to run with +. For example:
> +compile
Multiple-project builds
sbt also has built-in support to aggregate tasks across multiple projects, which is described Aggregation. If what you need eventually is normal built-in tasks like compile and test, you could set up a dummy aggregate without json_scalaz.
lazy val withoutJsonScalaz = (project in file("without-json-scalaz")).
.aggregate(liftProjects filterNot {_ == json_scalaz}: _*)
From the shell, you should be able to use this as:
> ++2.11.0
> project withoutJsonScalaz
> test
Getting values from multiple scopes
Another feature you might be interested in is ScopeFilter. This has the ability to traverse multiple projects beyond usual aggregation and cross building. You would need to create a setting whose type is ScopeFilter and set it based on scalaBinaryVersion.value. With scope filters, you can do:
val coreProjects = settingKey[ScopeFilter]("my core projects")
val compileAll = taskKey[Seq[sbt.inc.Analysis]]("compile all")
coreProjects := {
(scalaBinaryVersion.value) match {
case "2.10" => ScopeFilter(inProjects(common, json_scalaz7, json_scalaz))
}
}
compileAll := compileAllTask.value
lazy val compileAllTask = Def.taskDyn {
val f = coreProjects.value
(compile in Compile) all f
}
In this case compileAll would have the same effect as +compile, but you could aggregate the result and do something interesting like sbt-unidoc.
I want to combine the sbt-release plugin with the Play framework.
The plugins reads the current version number from a file version.sbt. Its content is
version in ThisBuild := "0.41.0-SNAPSHOT"
I would like to use this setting in my main build file but the variable version is of type sbt.SettingKey.
There is an evaluate method but for the life of me I can't figure out what to pass in to get the String I defined in version.sbt.
I tried the accepted answer's solution but it didn't compile. (Play 2.1.5)
[error] (ss: sbt.Project.Setting[_]*)sbt.Project <and>
[error] => Seq[sbt.Project.Setting[_]]
[error] cannot be applied to (Seq[sbt.ModuleID])
[error] val main = play.Project(appName).settings(appDependencies).settings(releaseSettings).settings(
[error] ^
[error] one error found
Instead I came up with this solution:
...
lazy val appSettings = Defaults.defaultSettings ++ ... ++ releaseSettings
val main = play.Project(appName, dependencies = appDependencies, settings = appSettings).settings(
version <<= version in ThisBuild,
...
)
This is a little shortcoming with the play.Project constructor, it excepts a static version number, not one from a setting key.
However, the only required parameter is the application name, so you can switch from something like:
val main = play.Project(appName, appVersion, appDependencies, settings =
Defaults.defaultSettings ++ releaseSettings ).settings(...)
to
val main = play.Project(appName).settings(appDependencies).
settings(releaseSettings).settings(...)
Normally, the version defined in version.sbt should be picked up here automagically. If it isn't, you can always add to the above:
.settings(applicationVersion <<= version in ThisBuild)