As part of my project build, I'd like to unzip a zip artifact of a managed dependency into a specific directory of the project. Before starting to use SBT I was doing this via an ANT script that would fetch the zip artifact from a maven dependency and unzip it.
My question(s) are:
how to specify that I want to depend on the zip dependency? I have defined it like so:
"eu.delving" % "sip-creator" % "0.4.6-SNAPSHOT"
but this doesn't fetch the zip artifact
where / how to hook into the build process to run the unzip (and how to refer to the artifact file in that context)?
If you want to extract a set of managed dependencies, the code below should work. I tested it in sbt 0.12.0, but it should also work with 0.11.x.
import sbt._
import Keys._
import Classpaths.managedJars
object TestBuild extends Build {
lazy val jarsToExtract = TaskKey[Seq[File]]("jars-to-extract", "JAR files to be extracted")
lazy val extractJarsTarget = SettingKey[File]("extract-jars-target", "Target directory for extracted JAR files")
lazy val extractJars = TaskKey[Unit]("extract-jars", "Extracts JAR files")
lazy val testSettings = Defaults.defaultSettings ++ Seq(
// define dependencies
libraryDependencies ++= Seq(
"com.newrelic" % "newrelic-api" % "2.2.1"
),
// collect jar files to be extracted from managed jar dependencies
jarsToExtract <<= (classpathTypes, update) map { (ct, up) =>
managedJars(Compile, ct, up) map { _.data } filter { _.getName.startsWith("newrelic-api") }
},
// define the target directory
extractJarsTarget <<= (baseDirectory)(_ / "extracted"),
// task to extract jar files
extractJars <<= (jarsToExtract, extractJarsTarget, streams) map { (jars, target, streams) =>
jars foreach { jar =>
streams.log.info("Extracting " + jar.getName + " to " + target)
IO.unzip(jar, target)
}
},
// make it run before compile
compile in Compile <<= extractJars map { _ => sbt.inc.Analysis.Empty }
)
lazy val test: Project = Project("test", file(".")) settings (testSettings: _*)
}
If you simply have jar files to extract, you can add them as unmanaged dependencies, ie. putting them into the /lib folder. See: https://github.com/harrah/xsbt/wiki/Getting-Started-Library-Dependencies
If you really have zip files (or want to extract the unmanaged dependencies), you can change the above code to list them:
// list jar files to be extracted
jarsToExtract <<= (baseDirectory) map { dir => Seq(dir / "lib" / "newrelic-api-2.2.1.zip") },
You should now be able to manually extract them from sbt and they should automatically be extracted before compile:
> clean
[success] Total time: 0 s, completed Oct 12, 2012 5:39:16 PM
> extract-jars
[info] Extracting newrelic-api-2.2.1.zip to /Users/balagez/Sites/test/extracted
[success] Total time: 0 s, completed Oct 12, 2012 5:39:22 PM
> compile
[info] Extracting newrelic-api-2.2.1.zip to /Users/balagez/Sites/test/extracted
[success] Total time: 0 s, completed Oct 12, 2012 5:39:24 PM
Now you can add a new task or extend the existing one which extracts the zip file from the extracted dependency. If you don't need the contents of the dependency, you can use the task-temporary-directory setting which gives you a temporary directory writable by sbt:
// define the target directory
extractJarsTarget <<= taskTemporaryDirectory,
Related
I have an SBT project which pulls in dependencies. I only want to pull in the direct dependencies - not any transitive dependencies. I'd like to find the filename of the dependency that's pulled in, so that I can copy it somewhere.
e.g. given a build.sbt file with the following contents:
libraryDependencies += "org.eclipse.jetty" % "jetty-server" % "9.4.28.v20200408"
I would like to know where is the jetty-server jar on the file system.
I have tried adding the following to my build.sbt file:
lazy val mytaskKey: TaskKey[Unit] = TaskKey[Unit]("mytask")
def mytask: Def.Setting[Task[Unit]] = mytaskKey := {
val updateReport = update.value
updateReport.allFiles foreach { f =>
println(f)
}
}
mytask
When I run this, I get a full list of dependencies:
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-library.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/9.4.28.v20200408/jetty-server-9.4.28.v20200408.jar
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-compiler.jar
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-reflect.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/jline/jline/2.14.6/jline-2.14.6.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.12/jansi-1.12.jar
I don't want that full list - I just want the jetty jar. i.e.
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/9.4.28.v20200408/jetty-server-9.4.28.v20200408.jar
How might I get this list?
Yes, there is with either intransitive() or notTransitive() classifiers. It's documented here.
I'm building a Docker image with a fat jar. I use the sbt-assembly plugin to build the jar, and the sbt-native-packager to build the Docker image. I'm not very familiar with SBT and am running into the following issues.
I'd like to declare a dependency on the assembly task from the docker:publish task, such that the fat jar is created before it's added to the image. I did as instructed in the doc, but it's not working. assembly doesn't run until I invoke it.
publish := (publish dependsOn assembly).value
One of the steps in building the image is copying the fat jar. Since assembly plugin creates the jar in target/scala_whatever/projectname-assembly-X.X.X.jar, I need to know the exact scala_whatever and the jar name. assembly seems to have a key assemblyJarName but I'm not sure how to access it. I tried the following which fails.
Cmd("COPY", "target/scala*/*.jar /app.jar")
Help!
Answering my own questions, the following works:
enablePlugins(JavaAppPackaging, DockerPlugin)
assemblyMergeStrategy in assembly := {
case x => {
val oldStrategy = (assemblyMergeStrategy in assembly).value
val strategy = oldStrategy(x)
if (strategy == MergeStrategy.deduplicate)
MergeStrategy.first
else strategy
}
}
// Remove all jar mappings in universal and append the fat jar
mappings in Universal := {
val universalMappings = (mappings in Universal).value
val fatJar = (assembly in Compile).value
val filtered = universalMappings.filter {
case (file, name) => !name.endsWith(".jar")
}
filtered :+ (fatJar -> ("lib/" + fatJar.getName))
}
dockerRepository := Some("username")
import com.typesafe.sbt.packager.docker.{Cmd, ExecCmd}
dockerCommands := Seq(
Cmd("FROM", "username/spark:2.1.0"),
Cmd("WORKDIR", "/"),
Cmd("COPY", "opt/docker/lib/*.jar", "/app.jar"),
ExecCmd("ENTRYPOINT", "/opt/spark/bin/spark-submit", "/app.jar")
)
I completely overwrite the docker commands because the defaults add couple of scripts that I don't need because I overwrite the entrypoint as well. Also, the default workdir is /opt/docker which is not where I want to put the fat jar.
Note that the default commands are shown by show dockerCommands in sbt console.
These are my project's dependencies:
libraryDependencies ++= Seq(
javaJdbc,
javaEbean,
cache,
javaWs,
"com.company" % "common_2.11" % "2.3.3"
)
In com.company.common_2.11-2.3.3 there is a jar file common_2.11-2.3.3-adtnl.jar.
How can I during compilation process in build.sbt tell SBT to extract its contents to specific folder in my project?
Use the following in build.sbt:
def unpackjar(jar: File, to: File): File = {
println(s"Processing $jar and saving to $to")
IO.unzip(jar, to)
jar
}
resourceGenerators in Compile += Def.task {
val jar = (update in Compile).value
.select(configurationFilter("compile"))
.filter(_.name.contains("common"))
.head
val to = (target in Compile).value / "unjar"
unpackjar(jar, to)
Seq.empty[File]
}.taskValue
It assumes that "common" is a unique part amongst all of your dependencies. You'd need to fix filter otherwise.
It also assumes that you don't really want the files at compile, but a bit later when package is called. You'd need to move the code between Def.task{...} to compile in Compile block like:
compile in Compile <<= (compile in Compile).dependsOn(Def.task {
val jar = (update in Compile).value
.select(configurationFilter("compile"))
.filter(_.name.contains("common"))
.head
val to = (target in Compile).value / "unjar"
unpackjar(jar, to)
Seq.empty[File]
})
Because of the large size of some resource files, I'd like sbt package to create 2 jar files at the same time, e.g. project-0.0.1.jar for the classes and project-0.0.1-res.jar for the resources.
Is this doable?
[SOLUTION] based on the answer below thanks to #gilad-hoch
1) unmanagedResources in Compile := Seq()
Now it's just classes in the default jar.
2)
val packageRes = taskKey[File]("Produces a jar containing only the resources folder")
packageRes := {
val jarFile = new File("target/scala-2.10/" + name.value + "_" + "2.10" + "-" + version.value + "-res.jar")
sbt.IO.jar(files2TupleRec("", file("src/main/resources")), jarFile, new java.util.jar.Manifest)
jarFile
}
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File, String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if (f.isFile) Seq((f, s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/", f)
}
}
}
(packageBin in Compile) <<= (packageBin in Compile) dependsOn (packageRes)
Now when I do "sbt package", both the default jar and a resource jar are produced at the same time.
to not include the resources in the main jar, you could simply add the following line:
unmanagedResources in Compile := Seq()
to add another jar, you could define a new task. it would generally be something like that:
use sbt.IO jar method to create the jar.
you could use something like:
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File,String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if(f.isFile) Seq((f,s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/",f)
}
}
}
files2TupleRec("",file("path/to/resources/dir")) //usually src/main/resources
or use the built-in methods from Path to create the sources: Traversable[(File, String)] required by the jar method.
that's basically the whole deal...
The document http://www.scala-sbt.org/0.13.0/docs/Detailed-Topics/Tasks.html explains how to add a task to build.sbt, but how do you add one to build.scala? Thanks
The part where you declare the TaskKey is the same in either format: val myTask = taskKey....
The part where you write out your Initialize[Task[T]] is the same: myTask := ....
The only difference is the context in which the latter thing appears.
In the .sbt format, it appears by itself, separated from other things by blank lines.
In the .scala format, you have to add the setting to the project. That's documented at http://www.scala-sbt.org/release/docs/Getting-Started/Full-Def.html and is the same regardless of whether we're talking about a task or a regular setting.
Here's a complete working example:
import sbt._
object MyBuild extends Build {
val myTask = taskKey[Unit]("...")
lazy val root =
Project(id = "MyProject", base = file("."))
.settings(
myTask := { println("hello") }
)
}
When defining a task in one project of a multi project build and using a "root" project to aggregate other projects, the aggregation means that any tasks in subprojects can be run from the root project as well, in fact this will run the tasks in all subprojects - see Multi-project builds. This is generally useful, for example to compile all subprojects when the compile task is run from the root project. However in this case it is a little confusing.
So the task is not accessible in all projects, but is accessible in both the subproject where you define the task, and the aggregating (root) project. The task is still running in the project where it is defined, it can just be called when in the root project.
To demonstrate this, we can have the same "hello" task defined in multiple sub projects, which are aggregated in a root project:
import sbt._
import Keys._
object Build extends Build {
val hwsettings = Defaults.defaultSettings ++ Seq(
organization := "organization",
version := "0.1",
scalaVersion := "2.10.4"
)
val hello = TaskKey[Unit]("hello", "Prints hello.")
lazy val projectA = Project(
"a",
file("a"),
settings = hwsettings ++ Seq(
hello := {println("Hello from project A!")}
))
lazy val projectB = Project(
"b",
file("b"),
settings = hwsettings ++ Seq(
hello := {println("Hello from project B!")}
))
lazy val projectC = Project(
"c",
file("c"),
settings = hwsettings
)
lazy val root = Project (
"root",
file ("."),
settings = hwsettings
) aggregate (projectA, projectB, projectC)
}
Note that projects a and b have "hello" tasks, and c does not. When we use "hello" from the root project, the aggregation causes the task to run in projects a AND b:
> project root
[info] Set current project to root (in build file:/Users/trepidacious/temp/multiProjectTasks/)
> hello
Hello from project A!
Hello from project B!
[success] Total time: 0 s, completed 24-Dec-2014 23:00:23
We can also switch to project a or b, and running hello will only run the task in the project we are in:
> project a
[info] Set current project to a (in build file:/Users/trepidacious/temp/multiProjectTasks/)
> hello
Hello from project A!
[success] Total time: 0 s, completed 24-Dec-2014 23:00:27
> project b
[info] Set current project to b (in build file:/Users/trepidacious/temp/multiProjectTasks/)
> hello
Hello from project B!
[success] Total time: 0 s, completed 24-Dec-2014 23:00:30
Finally, if we switch to project c, hello is not defined:
> project c
[info] Set current project to c (in build file:/Users/trepidacious/temp/multiProjectTasks/)
> hello
[error] Not a valid command: hello (similar: shell, help, reload)
[error] No such setting/task
[error] hello
[error] ^
>
This aggregation can be disabled, as described here.