SBT: How to Dockerize a fat jar? - sbt

I'm building a Docker image with a fat jar. I use the sbt-assembly plugin to build the jar, and the sbt-native-packager to build the Docker image. I'm not very familiar with SBT and am running into the following issues.
I'd like to declare a dependency on the assembly task from the docker:publish task, such that the fat jar is created before it's added to the image. I did as instructed in the doc, but it's not working. assembly doesn't run until I invoke it.
publish := (publish dependsOn assembly).value
One of the steps in building the image is copying the fat jar. Since assembly plugin creates the jar in target/scala_whatever/projectname-assembly-X.X.X.jar, I need to know the exact scala_whatever and the jar name. assembly seems to have a key assemblyJarName but I'm not sure how to access it. I tried the following which fails.
Cmd("COPY", "target/scala*/*.jar /app.jar")
Help!

Answering my own questions, the following works:
enablePlugins(JavaAppPackaging, DockerPlugin)
assemblyMergeStrategy in assembly := {
case x => {
val oldStrategy = (assemblyMergeStrategy in assembly).value
val strategy = oldStrategy(x)
if (strategy == MergeStrategy.deduplicate)
MergeStrategy.first
else strategy
}
}
// Remove all jar mappings in universal and append the fat jar
mappings in Universal := {
val universalMappings = (mappings in Universal).value
val fatJar = (assembly in Compile).value
val filtered = universalMappings.filter {
case (file, name) => !name.endsWith(".jar")
}
filtered :+ (fatJar -> ("lib/" + fatJar.getName))
}
dockerRepository := Some("username")
import com.typesafe.sbt.packager.docker.{Cmd, ExecCmd}
dockerCommands := Seq(
Cmd("FROM", "username/spark:2.1.0"),
Cmd("WORKDIR", "/"),
Cmd("COPY", "opt/docker/lib/*.jar", "/app.jar"),
ExecCmd("ENTRYPOINT", "/opt/spark/bin/spark-submit", "/app.jar")
)
I completely overwrite the docker commands because the defaults add couple of scripts that I don't need because I overwrite the entrypoint as well. Also, the default workdir is /opt/docker which is not where I want to put the fat jar.
Note that the default commands are shown by show dockerCommands in sbt console.

Related

In a sbt project, how to get full list of dependencies with scope?

In a sbt project, how to get full list of dependencies(including transitive dependencies) with scope?
For Example:
xxx.jar(Compile)
yyy.jar(Provided)
zzz.jar(Test)
...
From the sbt shell you can execute one or all of the following commands:
Test / fullClasspath
Runtime / fullClasspath
Compile / fullClasspath
Which will output the jars associated with the scope (Test/Runtime/Compile).
If you want to get a bit more fancy, sbt provides a number of ways of interacting with the outputs generated through the dependency management system. The documentation is here.
For example, you could add this to your build.sbt file:
lazy val printReport = taskKey[Unit]("Report which jars are in each scope.")
printReport := {
val updateReport: UpdateReport = update.value
val jarFilter: ArtifactFilter = artifactFilter(`type` = "jar")
val testFilter = configurationFilter(name = "test")
val compileFilter = configurationFilter(name = "compile")
val testJars = updateReport.matching(jarFilter && testFilter)
val compileJars = updateReport.matching(jarFilter && compileFilter)
println("Test jars:\n===")
for (jar <- testJars.sorted) yield {println(jar.getName)}
println("\n\n******\n\n")
println("compile jars:\n===")
for (jar <- compileJars.sorted) yield {println(jar.getName)}
}
It creates a new task printReport which can be executed like a normal sbt command with sbt printReport. It takes the value of the UpdateReport which is generated by the update task, and then filters for jar files in the respective test/compile scopes before printing the results.

In SBT, Is there a way of just downloading the top-level dependencies?

I have an SBT project which pulls in dependencies. I only want to pull in the direct dependencies - not any transitive dependencies. I'd like to find the filename of the dependency that's pulled in, so that I can copy it somewhere.
e.g. given a build.sbt file with the following contents:
libraryDependencies += "org.eclipse.jetty" % "jetty-server" % "9.4.28.v20200408"
I would like to know where is the jetty-server jar on the file system.
I have tried adding the following to my build.sbt file:
lazy val mytaskKey: TaskKey[Unit] = TaskKey[Unit]("mytask")
def mytask: Def.Setting[Task[Unit]] = mytaskKey := {
val updateReport = update.value
updateReport.allFiles foreach { f =>
println(f)
}
}
mytask
When I run this, I get a full list of dependencies:
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-library.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/9.4.28.v20200408/jetty-server-9.4.28.v20200408.jar
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-compiler.jar
/Users/dylan/.sbt/boot/scala-2.12.10/lib/scala-reflect.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.12/1.0.6/scala-xml_2.12-1.0.6.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/jline/jline/2.14.6/jline-2.14.6.jar
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/fusesource/jansi/jansi/1.12/jansi-1.12.jar
I don't want that full list - I just want the jetty jar. i.e.
/Users/dylan/.coursier/cache/v1/https/repo1.maven.org/maven2/org/eclipse/jetty/jetty-server/9.4.28.v20200408/jetty-server-9.4.28.v20200408.jar
How might I get this list?
Yes, there is with either intransitive() or notTransitive() classifiers. It's documented here.

SBT Subprojects do not recognize plugin commands

I'm having an issue with getting SBT Subprojects to recognize commands provided by plugins. I have the following plugin source:
object DemoPlugin extends AutoPlugin {
override lazy val projectSettings = Seq(commands += demoCommand)
lazy val demoCommand =
Command.command("demo") { (state: State) =>
println("Demo Plugin!")
state
}
}
Which is used by a project configured as follows:
lazy val root = project in file(".")
lazy val sub = (project in file("sub")).
enablePlugins(DemoPlugin).
settings(
//...
)
The plugin is, of course, listed in project/plugins.sbt. However, when I open up sbt in the project, I see the following:
> sub/commands
[info] List(sbt.SimpleCommand#413d2cd1)
> sub/demo
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: demo (similar: doc)
[error] sub/demo
Even stranger, using consoleProject, I can see that the command in the project is the one defined by DemoPlugin!
scala> (commands in sub).eval.map { c => c.getClass.getMethod("name").invoke(c) }
res0: Seq[Object] = List(demo)
I'm looking to be able to type sub/demo, and have it perform the demo command. Any help would be much appreciated!
Commands aren't per-project. They only work for the top-level project.
It's also recommended to try and use tasks, or if needed input tasks where you might want to use a command.
If you really need a command, there's a way to have a sort of "holder" task, see the answer to Can you access a SBT SettingKey inside a Command?

Using sbt-aether-deploy with sbt-native-packager

Has anyone published an sbt-native-packager produced artifact (tgz in my case) using sbt-aether-deploy to a nexus repo? (I need this for the timestamped snapshots, specifically the "correct" version tag in nexus' artifact-resolution REST resource).
I can do one or the other but can't figure out how to add the packagedArtifacts in Universal to the artifacts that sbt-aether-deploy deploys to do both.
I suspect the path to pursue would be to the addArtifact() the packagedArtifacts in Universal or creating another AetherArtifact and then to override/replace the deployTask to use that AetherArtifact?
Any help much appreciated.
I am the author of the sbt-aether-deploy plugin, and I just came over this post.
import aether.AetherKeys._
crossPaths := false //needed if you want to remove the scala version from the artifact name
enablePlugins(JavaAppPackaging)
aetherArtifact := {
val artifact = aetherArtifact.value
artifact.attach((packageBin in Universal).value, "dist", "zip")
}
This will also publish the other main artifact.
If you want to disable publishing of the main artifact, then you will need to rewrite the artifact coordinates. Maven requires a main artifact.
I have added a way to replace the main artifact for this purpose, but I can now see that way is kind of flawed. It will still assume that the artifact is published as a jar file. The main artifact type is locked down to that, since the POM packaging is set to jar by default by SBT.
If this is an app, then that limitation is probably OK, since Maven will never resolve that into an artifact.
The "proper" way in Maven terms is to add a classifier to the artifact and change the "packaging" in the POM file to "pom". We will see if I get around to changing that particular part.
Ok, I think I got it amazingly enough. If there's a better way to do it I'd love to hear. Not loving that blind Option.get there..
val tgzCoordinates = SettingKey[MavenCoordinates]("the maven coordinates for the tgz")
lazy val myPackagerSettings = packageArchetype.java_application ++ deploymentSettings ++ Seq(
publish <<= publish.dependsOn(publish in Universal),
publishLocal <<= publishLocal.dependsOn(publishLocal in Universal)
)
lazy val defaultSettings = buildSettings ++ Publish.settings ++ Seq(
scalacOptions in Compile ++= Seq("-encoding", "UTF-8", "-target:jvm-1.7", "-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls"),
testOptions in Test += Tests.Argument("-oDF")
)
lazy val myAetherSettings = aetherSettings ++ aetherPublishBothSettings
lazy val toastyphoenixProject = Project(
id = "toastyphoenix",
base = file("."),
settings = defaultSettings ++ myPackagerSettings ++ myAetherSettings ++ Seq(
name in Universal := name.value + "_" + scalaBinaryVersion.value,
packagedArtifacts in Universal ~= { _.filterNot { case (artifact, file) => artifact.`type`.contains("zip")}},
libraryDependencies ++= Dependencies.phoenix,
tgzCoordinates := MavenCoordinates(organization.value + ":" + (name in Universal).value + ":tgz:" + version.value).get,
aetherArtifact <<= (tgzCoordinates, packageZipTarball in Universal, makePom in Compile, packagedArtifacts in Universal) map {
(coords: MavenCoordinates, mainArtifact: File, pom: File, artifacts: Map[Artifact, File]) =>
createArtifact(artifacts, pom, coords, mainArtifact)
}
)
)
I took Peter's solution and reworked it slightly, avoiding the naked Option.get by creating the MavenCoordinates directly:
import aether.MavenCoordinates
import aether.Aether.createArtifact
name := "mrb-test"
organization := "me.mbarton"
version := "1.0"
crossPaths := false
packageArchetype.java_application
publish <<= (publish) dependsOn (publish in Universal)
publishLocal <<= (publishLocal) dependsOn (publishLocal in Universal)
aetherPublishBothSettings
aetherArtifact <<= (organization, name in Universal, version, packageBin in Universal, makePom in Compile, packagedArtifacts in Universal) map {
(organization, name, version, binary, pom, artifacts) =>
val nameWithoutVersion = name.replace(s"-$version", "")
createArtifact(artifacts, pom, MavenCoordinates(organization, nameWithoutVersion, version, None, "zip"), binary)
}
The nameWithoutVersion replace works around SBT native packager including the version in the artifact name:
Before: me/mbarton/mrb-test-1.0/1.0/mrb-test-1.0.zip
After: me/mbarton/mrb-test/1.0/mrb-test-1.0.zip
crossPaths avoids the Scala postfix on the version.

Producing two separate jars for sources and resources with package in SBT?

Because of the large size of some resource files, I'd like sbt package to create 2 jar files at the same time, e.g. project-0.0.1.jar for the classes and project-0.0.1-res.jar for the resources.
Is this doable?
[SOLUTION] based on the answer below thanks to #gilad-hoch
1) unmanagedResources in Compile := Seq()
Now it's just classes in the default jar.
2)
val packageRes = taskKey[File]("Produces a jar containing only the resources folder")
packageRes := {
val jarFile = new File("target/scala-2.10/" + name.value + "_" + "2.10" + "-" + version.value + "-res.jar")
sbt.IO.jar(files2TupleRec("", file("src/main/resources")), jarFile, new java.util.jar.Manifest)
jarFile
}
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File, String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if (f.isFile) Seq((f, s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/", f)
}
}
}
(packageBin in Compile) <<= (packageBin in Compile) dependsOn (packageRes)
Now when I do "sbt package", both the default jar and a resource jar are produced at the same time.
to not include the resources in the main jar, you could simply add the following line:
unmanagedResources in Compile := Seq()
to add another jar, you could define a new task. it would generally be something like that:
use sbt.IO jar method to create the jar.
you could use something like:
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File,String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if(f.isFile) Seq((f,s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/",f)
}
}
}
files2TupleRec("",file("path/to/resources/dir")) //usually src/main/resources
or use the built-in methods from Path to create the sources: Traversable[(File, String)] required by the jar method.
that's basically the whole deal...

Resources