I have sbt project with several modules that need to be packaged into jar archives. As I see in logs sbt packages each project into jar sequentially:
[info] Packaging C:path\target\scala-2.11\projectFirst ...
[info] Packaging C:path\target\scala-2.11\projectSecond ...
[info] Packaging C:path\target\scala-2.11\projectThrird ...
Projects do not depend of each other and can be packaged in parallel. Is there a possibility to setup sbt to package projects in parallel?
In sbt documentation we can find following sentence:
By default, sbt executes tasks in parallel.
I prepared experiment to check this. I created project with 4 subprojects. This is a code:
main build.sbt:
lazy val sub1 = project
lazy val sub2 = project
lazy val sub3 = project
lazy val sub4 = project
Keys.`package` in Compile <<= (Keys.`package` in Compile).dependsOn(Def.task {
})
sub1/build.sbt
Keys.`package` in Compile <<= (Keys.`package` in Compile).dependsOn(Def.task {
for( a <- 1 to 2){
Thread.sleep(1000)
println( "SubProject 1: " + a );
}
})
In sub2, sub3 and sub4 build.sbt is almost identical as in sub1. I changed only number of iterations and printed number.
I received following output:
> package
SubProject 4: 1
SubProject 1: 1
SubProject 4: 2
SubProject 1: 2
SubProject 4: 3
SubProject 3: 1
SubProject 4: 4
SubProject 3: 2
SubProject 4: 5
SubProject 3: 3
SubProject 4: 6
SubProject 3: 4
SubProject 4: 7
SubProject 2: 1
SubProject 4: 8
SubProject 2: 2
SubProject 4: 9
SubProject 2: 3
[info] Updating {file:/D:/sbt/abc/}sub1...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
SubProject 4: 10
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
[info] Resolving org.scala-lang#jline;2.10.4 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/D:/sbt/abc/}abc...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
[info] Resolving org.scala-lang#jline;2.10.4 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/D:/sbt/abc/}sub3...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
[info] Resolving org.scala-lang#jline;2.10.4 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/D:/sbt/abc/}sub2...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
[info] Resolving org.scala-lang#jline;2.10.4 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
SubProject 4: 11
[info] Done updating.
[info] Packaging D:\sbt\abc\sub1\target\scala-2.10\sub1_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[info] Packaging D:\sbt\abc\target\scala-2.10\abc_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[info] Packaging D:\sbt\abc\sub3\target\scala-2.10\sub3_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[info] Packaging D:\sbt\abc\sub2\target\scala-2.10\sub2_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
SubProject 4: 12
SubProject 4: 13
SubProject 4: 14
SubProject 4: 15
SubProject 4: 16
SubProject 4: 17
SubProject 4: 18
SubProject 4: 19
SubProject 4: 20
[info] Updating {file:/D:/sbt/abc/}sub4...
[info] Resolving org.scala-lang#scala-library;2.10.4 ...
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
[info] Resolving org.scala-lang#jline;2.10.4 ...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Packaging D:\sbt\abc\sub4\target\scala-2.10\sub4_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
So there are four subprojects. In every subproject 'package' task depends on created task (which prints and sleeps in a loop). When I execute 'package' command then then this task is executed for wach sub-project. But for each one the dependent task is called. As we can see in output these dependent tasks run pallalelly but only two at a time. This coresponds to following sentence from doc:
(Although never exposed as a setting, the maximum number of tasks running at a given time was internally configurable as well.)
Following three lines from output:
[info] Resolving org.scala-lang#scala-compiler;2.10.4 ...
SubProject 4: 10
[info] Resolving org.scala-lang#scala-reflect;2.10.4 ...
provide proof that packaging task from one subproject runs parallelly with dependent task from other subproject.
Related
I'm using sbt 0.13.8 and sbt-assembly 0.13.0 in a multi-module project. Everything works great until I try to turn off tests during assembly.
As instructed in the docs I add the line
test in assembly := {}
in the settings of one of my modules and get the following error
/Users/lanny/work/IdeaProjects/Search/build.sbt:61: error: type mismatch;
found : sbt.TaskKey[sbt.File]
required: java.io.File
test in assembly := {},
^
sbt.compiler.EvalException: Type error in expression
I'd be happy to show the results of 'inspect test' but alas, I cannot run sbt without removing the line causing the error. Here's what I see with the line removed.
[info] Set current project to search-hadoop (in build file:/Users/lanny/work/IdeaProjects/Search/)
> inspect assembly::test
[info] Task: Unit
[info] Description:
[info] Executes all tests.
[info] Provided by:
[info] {file:/Users/lanny/work/IdeaProjects/Search/}hadoop/*:assembly::test
[info] Defined at:
[info] (sbtassembly.AssemblyPlugin) AssemblyPlugin.scala:32
[info] Dependencies:
[info] hadoop/test:test
[info] Reverse dependencies:
[info] hadoop/*:assemblyPackageScala::test
[info] hadoop/*:assemblyPackageDependency::test
[info] hadoop/*:assembly
[info] Delegates:
[info] hadoop/*:assembly::test
[info] hadoop/*:test
[info] {.}/*:assembly::test
[info] {.}/*:test
[info] */*:assembly::test
[info] */*:test
[info] Related:
[info] test/*:assemblyPackageDependency::test
[info] third_party/test:test
[info] third_party/*:assemblyPackageDependency::test
[info] search/*:assembly::test
[info] search/*:assemblyPackageScala::test
[info] server/*:assemblyPackageDependency::test
[info] search/*:assemblyPackageDependency::test
[info] server/test:test
[info] hadoop/test:test
[info] hadoop/*:assemblyPackageScala::test
[info] ...
sbt.Keys.test in assembly := {}
does the trick.
As stated in the documentation, for sbt 0.13.6+, add this line to your project/assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")
So, I have a SBT project, with two custom tasks jooq:codegen and flyway:migrate from [ https://github.com/sean8223/jooq-sbt-plugin, https://github.com/sean8223/flyway-sbt-plugin ]
Not that it is relevant here, but the flyway:migrate task creates the schema in the database, and jooq:codegen generates code from the schema. Hence, it is imperative that flyway:migrate runs before jooq:codegen. So, I added the following line in my build.sbt
(codegen in JOOQ) <<= (codegen in JOOQ) dependsOn (migrate in Flyway)
Also, compile needs the generated code from jooq:codegen, but the plugin takes care of it by default.
Here is the weird part. When I run sbt compile, I get:
~/N/p/d/davion git:data-access ❯❯❯ sbt compile
[info] Loading project definition from /Users/rohan/Nomadly/projects/davion-projects/davion/project
[info] Set current project to davion (in build file:/Users/rohan/Nomadly/projects/davion-projects/davion/)
[info] Done updating.
[info] Initialising properties : /jooq-config4460313300896426081.xml
... OUTPUT TRUNCATED ...
[info] Table records generated : Total: 669.172ms, +44.536ms
[info] Routines fetched : 0 (0 included, 0 excluded)
[info] Packages fetched : 0 (0 included, 0 excluded)
[info] GENERATION FINISHED! : Total: 688.254ms, +19.082ms
[info] Compiling 7 Scala sources and 17 Java sources to /Users/rohan/Nomadly/projects/davion-projects/davion/target/classes...
[success] Total time: 24 s, completed 14 May, 2015 12:13:35 PM
So, the flyway:migrate task doesn't run. But when I run sbt jooq:codegen, this happens:
~/N/p/d/davion git:data-access ❯❯❯ sbt jooq:codegen
[info] Loading project definition from /Users/rohan/Nomadly/projects/davion-projects/davion/project
[info] Set current project to davion (in build file:/Users/rohan/Nomadly/projects/davion-projects/davion/)
[info] Flyway (Command-line Tool) v.2.0.3
[info]
[info] Current schema version: 6
[info] Schema is up to date. No migration necessary.
[info] Initialising properties : /jooq-config6431105742854017589.xml
[info] License parameters
... OUTPUT TRUNCATED ...
I have no idea why this is happening. If a task chain is setup where 'A' depends on 'B' which depends on 'C', then shouldn't running 'A' execute both 'C' and 'B' (and in that order)? Why does 'C' not run as a transitive dependency and how can I remedy this?
Since Play Framework 2.3 assets are packaged into one jar archive file. I would like to publish this jar automatically with the project, i.e. upon publish or publishLocal I want the assets jar to be published as well.
How to achieve that?
After inspect tree dist I managed to find the task playPackageAssets that generates the assets file:
[play-publish-webjar] $ inspect playPackageAssets
[info] Task: java.io.File
[info] Description:
[info]
[info] Provided by:
[info] {file:/Users/jacek/sandbox/play-publish-webjar/}root/*:playPackageAssets
[info] Defined at:
[info] (sbt.Defaults) Defaults.scala:641
[info] Dependencies:
[info] *:playPackageAssets::packageConfiguration
[info] *:playPackageAssets::streams
[info] Reverse dependencies:
[info] *:scriptClasspath
[info] universal:mappings
[info] Delegates:
[info] *:playPackageAssets
[info] {.}/*:playPackageAssets
[info] */*:playPackageAssets
A naive solution might be to attach the assets webjar as is generated by playPackageAssets to publishLocal task's artifacts. Add the following to build.sbt (the types are to show what you work with):
import play.PlayImport.PlayKeys._
packagedArtifacts in publishLocal := {
val artifacts: Map[sbt.Artifact, java.io.File] = (packagedArtifacts in publishLocal).value
val assets: java.io.File = (playPackageAssets in Compile).value
artifacts + (Artifact(moduleName.value, "asset", "jar", "assets") -> assets)
}
Repeat it for the other tasks you want to exhibit similar behaviour.
I'm however quite doubtful it's the best solution.
How to set up sbt-assembly to include jars with runtime scope? What's the rationale for not doing it by default?
This is now being discussed under sbt/sbt-assembly#120.
sbt-assembly is intended to wire full classpath from Runtime configuration. And it's coded as follows:
fullClasspath in assembly <<= fullClasspath or (fullClasspath in Runtime),
For Spark, for whatever reason fullClasspath is wired, so that's likely interfering. The reason sbt-assembly included fullClasspath I think is now historical, so I might get rid of "fullClasspath or" part. As noted in the issue, the current workaround is likely:
fullClasspath in assembly := (fullClasspath in Runtime).value
FYI for those who learn sbt the above sbt-0.13+ format with := corresponds to the older pre-sbt-0.13 format with := as follows:
fullClasspath in assembly <<= (fullClasspath in Runtime)
My tests show that the plugin does include dependencies with runtime scope as evidenced with the following example.
project/plugins.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
build.sbt
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.2" % "runtime"
assemblySettings
This is a sample session:
> about
[info] This is sbt 0.13.6-SNAPSHOT
[info] The current project is {file:/C:/dev/sandbox/runtime-assembly/}runtime-assembly 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.4
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin, org.sbtidea.SbtIdeaPlugin, de.johoop.jacoco4sbt.JacocoPlugin, com.timushev.sbt.updates.UpdatesPlugin, sbtassembly.Plugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4
> assembly
[info] Updating {file:/C:/dev/sandbox/runtime-assembly/}runtime-assembly...
[info] Resolving org.jacoco#org.jacoco.agent;0.6.4.201312101107 ...
[info] downloading http://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.1.2/logback-classic-1.1.2.jar ...
[info] [SUCCESSFUL ] ch.qos.logback#logback-classic;1.1.2!logback-classic.jar (1122ms)
[info] downloading http://repo1.maven.org/maven2/ch/qos/logback/logback-core/1.1.2/logback-core-1.1.2.jar ...
[info] [SUCCESSFUL ] ch.qos.logback#logback-core;1.1.2!logback-core.jar (1313ms)
[info] Done updating.
[info] Including: slf4j-api-1.7.6.jar
[info] Including: logback-classic-1.1.2.jar
[info] Including: logback-core-1.1.2.jar
[info] Including: scala-library.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF\MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] SHA-1: 0f520cdfc30de8ced8408a3c25b326b3ff5afda8
[info] Packaging C:\dev\sandbox\runtime-assembly\target\scala-2.10\runtime-assembly-assembly-0.1-SNAPSHOT.jar ...
[info] Done packaging.
Jar verification:
/cygdrive/c/dev/sandbox/runtime-assembly
$ jar -tf target/scala-2.10/runtime-assembly-assembly-0.1-SNAPSHOT.jar | grep logback | wc -l
592
I'm converting an sbt 0.7.x build script to sbt 0.11.2. I'm writing a task to collect various JARs together from subprojects. In the old build, part of the task does the following:
deployedProjects.foreach {
p: BasicScalaProject =>
p.managedClasspath(config("compile")) --- p.managedClasspath(config("provided"))
// etc
}
How can I do the equivalent in sbt 0.11?
Updated to add:
In particular:
How can I write a task that depends on a list of settings/tasks? For example, how would I write a task that depends on all the managedClasspaths from a List of subprojects (without bundling it all into a tuple).
Is there a particular scope for getting the managed jars that are or are not marked as "provided"?
In sbt 0.11.x there is the task managedClasspath:
> inspect managed-classpath
[info] Task: scala.collection.Seq[sbt.Attributed[java.io.File]]
[info] Description:
[info] The classpath consisting of external, managed library dependencies.
[info] Provided by:
[info] {file:/Users/heiko/tmp/test/}default-f3fb6c/compile:managed-classpath
[info] Dependencies:
[info] compile:classpath-configuration
[info] compile:classpath-types
[info] compile:update
[info] Reverse dependencies:
[info] compile:external-dependency-classpath
[info] Delegates:
[info] compile:managed-classpath
[info] *:managed-classpath
[info] {.}/compile:managed-classpath
[info] {.}/*:managed-classpath
[info] */compile:managed-classpath
[info] */*:managed-classpath
[info] Related:
[info] test:managed-classpath
[info] runtime:managed-classpath
Looking at the delegates you see that you can scope this task to various configurations, e.g. compile:
> show compile:managed-classpath
[info] Updating {file:/Users/heiko/tmp/test/}default-f3fb6c...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Done updating.
[info] ArraySeq(Attributed(/Users/heiko/.sbt/boot/scala-2.9.1/lib/scala-library.jar))