How to exclude files in a custom clean task? - sbt

I'm working on a custom TaskKey that makes a clean that keeps a folder in target directory - it's a database that I don't want to fill everytime.
So I tried something like that:
lazy val cleanOutput = taskKey[Unit]("Prints 'Hello World'")
cleanOutput := clean.value
cleanKeepFiles in cleanOutput <+= target { target => target / "database" }
It seems that the statement in cleanOutput is not taken in consideration.
Even when I do just the following it doesn't work:
cleanKeepFiles in clean <+= target { target => target / "database" }
But the following works:
cleanKeepFiles <+= target { target => target / "database" }
Why is the difference?

There are keys, which may have defined values in different scopes (project, configuration or task).
A key can be defined, even if it's not used in any scope, as well as a key can be given a value in any scope. The later doesn't mean that the value will be used by a specific task. It means you can reuse keys declared by sbt for your purposes.
What you do you declare a new taskKey. You define your task to call clean. You then define cleanKeepFiles in the scope of your new task to equal to whatever the value was before, plus your database directory in the target.
The value is set correctly, but the clean task will not look for it in the scope of your task.
You can verify it:
> show cleanOutput::cleanKeepFiles
[info] List(/home/lpiepiora/Desktop/sbt/stack-overflow/q-24020437/target/.history, /home/lpiepiora/Desktop/sbt/stack-overflow/q-24020437/target/database)
Additionally you can check:
> inspect *:cleanKeepFiles
[info] Setting: scala.collection.Seq[java.io.File] = List(/home/lpiepiora/Desktop/sbt/stack-overflow/q-24020437/target/.history)
[info] Description:
[info] Files to keep during a clean.
[info] Provided by:
[info] {file:/home/lpiepiora/Desktop/sbt/stack-overflow/q-24020437/}q-24020437/*:cleanKeepFiles
[info] Defined at:
[info] (sbt.Defaults) Defaults.scala:278
[info] Dependencies:
[info] *:history
[info] Reverse dependencies:
[info] *:clean
[info] Delegates:
[info] *:cleanKeepFiles
[info] {.}/*:cleanKeepFiles
[info] */*:cleanKeepFiles
[info] Related:
[info] *:cleanOutput::cleanKeepFiles
You can also see that sbt knows, that you have set it in the scope *:cleanOutput::cleanKeepFiles, it is just not using it.
Where it will look for it?. You can check it by inspecting the clean task.
> inspect clean
[info] Task: Unit
[info] Description:
[info] Deletes files produced by the build, such as generated sources, compiled classes, and task caches.
// next lines are important
[info] Dependencies:
[info] *:cleanKeepFiles
[info] *:cleanFiles
You can see that one of the dependencies is *:cleanKeepFiles, the * means Global configuration. This means that the clean task will look for the setting in that scope. You can change your setting to:
cleanKeepFiles += target.value / "database"
This will set it in the correct scope used by the clean task.
Edit
There is a doClean function which you can reuse. Given that, you can define your clean task like this:
val cleanKeepDb = taskKey[Unit]("Cleans folders keeping database")
cleanKeepDb := Defaults.doClean(cleanFiles.value, (cleanKeepFiles in cleanKeepDb).value)
cleanKeepFiles in cleanKeepDb += target.value / "database"

Related

sbt: Invoking tests after other tasks

I have a multi-project build, and want to run tests after starting two Docker containers. This is my custom task:
runTestsWithDocker := Def.taskDyn {
startDirectoryServer.value
val containerId = buildOrStartTestDatabase.value
Def.task {
(test in Test).value
sLog.value.info("running inside dynamic task")
containerId
}
}.value
As you can see from the output below, the Docker containers are started, and the log message is written from the dynamic task. However, there's no test output (and the build executes far too quickly for the tests to have run).
> runTestsWithDocker
[info] logging into ECR registry 123456789012.dkr.ecr.us-east-1.amazonaws.com
[info] checking repository for image container1:1.2.3-1200
[info] successfully logged-in to ECR registry 123456789012.dkr.ecr.us-east-1.amazonaws.com
[info] DockerSupport: pulling 123456789012.dkr.ecr.us-east-1.amazonaws.com/container2:latest
[info] DockerSupport: docker run -d -p 389:389 123456789012.dkr.ecr.us-east-1.amazonaws.com/container2:latest
[info] container ID: 80d16a268c6e13dd810f8c271ca8778fc8eaa6835f2d0640fa62d032ff052345
[info] image already exists; no need to build
[info] DockerSupport: pulling 123456789012.dkr.ecr.us-east-1.amazonaws.com/container1:1.2.3-1200
[info] DockerSupport: docker run -d -p 5432:5432 123456789012.dkr.ecr.us-east-1.amazonaws.com/container1:1.2.3-1200
[info] container ID: 2de559b0737e69d61b1234567890123bd123456789012d382ba8ffa40e0480cf
[info] Updating {file:/home/ubuntu/Workspace/mybuild/}mybuild...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] running inside dynamic task
[success] Total time: 2 s, completed Jun 5, 2019 9:05:20 PM
I'm assuming that my scope is incorrect, and that I need to refer to test in some other scope, but I have no idea what that might be (I've tried Compile and ThisBuild as random stabs in the dark).
I've also seen (test in Test).result.value from other questions in SO. Thinking that maybe the test task was doing something non-standard I tried it, but with the same (non) result.
Lastly, I'm running SBT 0.13.16, so any convincing argument (as in a bug report) that it's a problem with that version would make me upgrade sooner than planned (my current goal is to refactor the build then upgrade).
Update: here's the output from inspect. It doesn't show the dependency on test, but I'm assuming that's because it's invoked from a dynamic task.
> inspect runTestsWithDocker
[info] Task: java.lang.String
[info] Description:
[info] Runs the test suite, after starting the LDAP server and running/initializing the test database
[info] Provided by:
[info] {file:/home/ubuntu/Workspace/mybuild/}mybuild/*:runTestsWithDocker
[info] Defined at:
[info] /home/ubuntu/Workspace/mybuild/build.sbt:597
[info] Dependencies:
[info] mybuild/*:buildOrStartTestDatabase
[info] mybuild/*:startDirectoryServer
[info] mybuild/*:settingsData
[info] Reverse dependencies:
[info] mybuild/*:publishTestDatabase
[info] Delegates:
[info] mybuild/*:runTestsWithDocker
[info] {.}/*:runTestsWithDocker
[info] */*:runTestsWithDocker
Update: if I specify a single sub-project, it correctly runs the tasks in that sub-project.
runTestsWithDocker := Def.taskDyn {
startDirectoryServer.value
val containerId = buildOrStartTestDatabase.value
Def.task {
(test in (subproject,Test)).result.value
containerId
}
}.value
So it looks like maybe the root project isn't aggregating? We're relying on the "default root" project, so I think my next change will be to create an explicit root project.
It turned out that the default root project was not in fact "aggregat[ing] all other projects in the build." Once I created this project and explicitly aggregated the other sub-projects under it, I was able to specify my task like so:
runTestsWithDocker := Def.taskDyn {
startDirectoryServer.value
val containerId = buildOrStartTestDatabase.value
Def.task {
(test in (root,Test)).result.value
containerId
}
}.value
:shrug:

How to set system property for xsbt-web-plugin's jetty()?

I've migrated my project to 0.13.5 and started using the xsbt-web-plugin.
I'd like to configure logback to be using a configuration file outside the classpath which is set by a system property logback.configurationFile (so I can keep the logconfig outside the war file).
Previously I would simply set:
System.setProperty("logback.configurationFile", "/some/path/logback.xml")
inside the project/build.scala and logback would pick it up.
However, after upgrading sbt to 0.13.5 and migrating to the xsbt-web-plugin system properties set in sbt don't seem to be available at runtime(jetty).
I've tried setting system properties in different ways, also by passing it along using the -D flag when starting sbt.
On the sbt console I can see the property:
eval sys.props("logback.configurationFile")
[info] ans: String = /some/path/logback.xml
But it's not available inside the webapp.
Any ideas on how to set system properties to be available inside the webapp?
I've tried both jetty() and tomcat(). Same behaviour.
Update:
I ended up with:
jetty(options = new ForkOptions(runJVMOptions = Seq("-Dlogback.configurationFile=/some/path/logback.xml")))
that works.
Use javaOptions in container += "-Dlogback.configurationFile=/some/path/logback.xml" as described in Set forked JVM options.
javaOptions alone (without in container) should work, too, as could be seen in inspect actual under Dependencies and Delegates:
[play-new-app] $ inspect actual container:javaOptions
[info] Task: scala.collection.Seq[java.lang.String]
[info] Description:
[info] Options passed to a new JVM when forking.
[info] Provided by:
[info] {file:/Users/jacek/sandbox/play-new-app/}root/container:javaOptions
[info] Defined at:
[info] /Users/jacek/sandbox/play-new-app/build.sbt:26
[info] Dependencies:
[info] */*:javaOptions
[info] Delegates:
[info] container:javaOptions
[info] *:javaOptions
[info] {.}/container:javaOptions
[info] {.}/*:javaOptions
[info] */container:javaOptions
[info] */*:javaOptions
[info] Related:
[info] */*:javaOptions

How do scopes influence keys in a build? What is a scope axis?

I am trying to understand what SBT scope axes are and how they influence scoped keys in the map of the build.
There are three scope axes : project, configuration and task.
Is the following tuple the scoped key for the map of the build?
(value of project axis, value of configuration axis, value of task axis, value of unscoped key)
Or is a scoped key (conceptually) a pair having one of the following three forms ?
(value of project axis, value of unscoped key)
(value of configuration axis, value of unscoped key)
(value of task axis, value of unscoped key)
Or is the scoped key something else, if yes how does it look like (conceptually) ?
So the question could be what is (conceptually) a scoped key?
Is it a four tuple?
Is it a pair?
Is it something else, if yes, then how does it looks like (conceptually)?
From Custom settings and tasks:
Keys have one of three types: SettingKey, TaskKey and InputKey.
An example of a key can be:
val scalaVersion = settingKey[String]("The version of Scala used for building.")
It's available by default in sbt and to query for a value you execute show:
> show scalaVersion
[info] 2.10.4
Execute inspect to learn about the details of a key (it's not exactly about a key but a setting as you soon learn):
> inspect scalaVersion
[info] Setting: java.lang.String = 2.10.4
[info] Description:
[info] The version of Scala used for building.
[info] Provided by:
[info] */*:scalaVersion
[info] Defined at:
[info] (sbt.Defaults) Defaults.scala:236
[info] Reverse dependencies:
[info] *:libraryDependencies
[info] *:scalaInstance
[info] *:evicted
[info] *:dependencyUpdatesData
[info] *:update
[info] *:allDependencies
[info] Delegates:
[info] *:scalaVersion
[info] {.}/*:scalaVersion
[info] */*:scalaVersion
[info] Related:
[info] */*:scalaVersion
You can define your own key with the macro settingKey:
lazy val abc: sbt.SettingKey[String] = settingKey[String]("My own setting key")
According to the scaladoc of sbt.SettingKey:
Identifies a setting. It consists of three parts: the scope, the name, and the type of a value associated with this key. The scope is represented by a value of type Scope. The name and the type are represented by a value of type AttributeKey[T]. Instances are constructed using the companion object.
As you can see in the scaladoc, any SettingKey has the required def scope: Scope that describes the scope of the setting. Soon to be explained.
You can't see the key in the build until you assign it a value (via a computation). You assign an Initialize[T] or just T to a SettingKey[T] via :=, += and ++= (more often used and recommended) or <++=, <+= and <<= operators (that are Scala methods described in the scaladoc of sbt.SettingKey).
abc := "my value"
The above line is equivalent to the following line (they're interchangeable):
abc.:=("my value")
As you can see in the definition of :=:
final def :=(v: T): Def.Setting[T]
the result of it is Setting[T].
So, the pair (SettingKey[T], Initialize[T]) or simply (SettingKey[T], T) is a Setting[T].
A setting of type Setting[T] where T is the type of the value that is a result of the setting's evaluation describes a transformation to the map of all the settings in a build. It's a recipe for a value that's going to be available under a key after the build's loaded.
Now, the last part, the scope.
As you may've noticed, a setting is always in a scope. You can change the scope of a setting using final def in(scope: Scope): SettingKey[T].
Use inspect to learn about a setting.
> inspect abc
[info] Setting: java.lang.String = my value
[info] Description:
[info] My new setting key
[info] Provided by:
[info] {file:/Users/jacek/sandbox/sbt-theory/}sbt-theory/*:abc
[info] Defined at:
[info] /Users/jacek/sandbox/sbt-theory/build.sbt:3
[info] Delegates:
[info] *:abc
[info] {.}/*:abc
[info] */*:abc
You can see that the setting delegates (section Delegates) to *:abc, {.}/*:abc and */*:abc. The other places that I'm going to describe as A/B:K are A - the project axis, B - the configuration axis and K the key or the task scope (since sbt allows for attibutes that appear after ::).
Guess what the value of *:abc is. inspect it.
> inspect *:abc
[info] Setting: java.lang.String = my value
[info] Description:
[info] My new setting key
[info] Provided by:
[info] {file:/Users/jacek/sandbox/sbt-theory/}sbt-theory/*:abc
[info] Defined at:
[info] /Users/jacek/sandbox/sbt-theory/build.sbt:3
[info] Delegates:
[info] *:abc
[info] {.}/*:abc
[info] */*:abc
Guess what the value of {.}/*:abc that corresponds to this build denoted as {.} in the global configuration *. inspect it.
> inspect {.}/*:abc
[info] No entry for key.
[info] Description:
[info] My new setting key
[info] Delegates:
[info] {.}/*:abc
[info] */*:abc
[info] Related:
[info] *:abc
There's no value for abc setting in ThisBuild (aka thisBuild) scope. Set it in build.sbt using:
abc in ThisBuild := "abc in ThisBuild"
inspect it (after reload to reload the build's changes):
> inspect {.}/*:abc
[info] Setting: java.lang.String = abc in ThisBuild
[info] Description:
[info] My new setting key
[info] Provided by:
[info] {file:/Users/jacek/sandbox/sbt-theory/}/*:abc
[info] Defined at:
[info] /Users/jacek/sandbox/sbt-theory/build.sbt:5
[info] Delegates:
[info] {.}/*:abc
[info] */*:abc
[info] Related:
[info] *:abc
With the change build.sbt looks as follows:
lazy val abc: sbt.SettingKey[String] = settingKey[String]("My new setting key")
abc := "my value"
abc in ThisBuild := "abc in ThisBuild"
that says that there's abc key that has different values per scope - ThisBuild and Global. Clear? Ask away!

SBT Config extend vs DefaultSettings

If I define an SBT config with
val MyConfig = config("my") extend Test
is that basically the same as doing
val MyConfig = config("my")
val mySettings = inConfig(MyConfig)(Defaults.testSettings)
and then importing mySettings inside a build definition ?
No, calling extend method is not the same thing as calling inConfig. extend just returns a new configuration with passed in configurations prepended extendsConfigs, and it will not introduce any new settings.
When you add MyConfig into the project, it becomes part of the scoped key resolution path:
val MyConfig = config("my") extend Test
val root = (project in file(".")).
configs(MyConfig)
Suppose you type my:test in the sbt shell. Since test task is not found under my configuration, it will traverse extendsConfigs and check if the tasks are available under them. The first one it's going to hit is Test since we prepended it. You can check this by running inspect my:test:
root> inspect my:test
[info] Task: Unit
[info] Description:
[info] Executes all tests.
[info] Provided by:
[info] {file:/Users/eugene/work/quick-test/sbt-so/}root/test:test
[info] Defined at:
[info] (sbt.Defaults) Defaults.scala:365
[info] Delegates:
[info] my:test
[info] test:test
[info] runtime:test
[info] compile:test
[info] *:test
[info] {.}/my:test
[info] {.}/test:test
[info] {.}/runtime:test
[info] {.}/compile:test
[info] {.}/*:test
[info] */my:test
[info] */test:test
[info] */runtime:test
[info] */compile:test
[info] */*:test
[info] Related:
[info] test:test
"Provided by" says it delegated to root/test:test. This mechanism allows you to share some of the settings but override others, but you still have to know the inner wiring of the settings scoped to tasks etc, so it's tricky business. You probably already know, but I'll link to Additional test configurations, which specifically discusses configurations for testing.

Different compile options for tests and release in SBT?

I have a project where I need to disable assertions when creating the binaries. Now I could just do:
scalacOptions += "-Xdisable-assertions"
But then also the unit tests would be run without assertions. Is there a (hopefully) simple way to achieve what I need?
How do you create the binaries? What task/command do you use?
Use the task as the scope for scalacOptions to have different values for them. See Scoping by configuration axis:
By default, all the keys associated with compiling, packaging, and
running are scoped to a configuration and therefore may work
differently in each configuration. The most obvious examples are the
task keys compile, package, and run; but all the keys which affect
those keys (such as sourceDirectories or scalacOptions or
fullClasspath) are also scoped to the configuration.
Use inspect when in doubt.
> inspect scalacOptions
[info] Task: scala.collection.Seq[java.lang.String]
[info] Description:
[info] Options for the Scala compiler.
[info] Provided by:
[info] {file:/C:/dev/sandbox/task-dependsOn/}task-dependson/compile:scalacOptions
[info] Defined at:
[info] (sbt.Classpaths) Defaults.scala:1424
[info] Dependencies:
[info] task-dependson/compile:autoCompilerPlugins
[info] task-dependson/compile:settingsData
[info] task-dependson/compile:update
[info] task-dependson/compile:buildDependencies
[info] task-dependson/compile:thisProjectRef
[info] Delegates:
[info] task-dependson/compile:scalacOptions
[info] task-dependson/*:scalacOptions
[info] {.}/compile:scalacOptions
[info] {.}/*:scalacOptions
[info] */compile:scalacOptions
[info] */*:scalacOptions
[info] Related:
[info] b/compile:scalacOptions
[info] b/test:scalacOptions
[info] task-dependson/test:scalacOptions
[info] task-dependson/jacoco:scalacOptions
[info] a/jacoco:scalacOptions
[info] */*:scalacOptions
[info] a/test:scalacOptions
[info] a/compile:scalacOptions
[info] b/jacoco:scalacOptions
The Compile configuration scope is the default one (see show defaultConfiguration for a project) so scalacOptions += "-Xdisable-assertions" is in fact scalacOptions in Compile += "-Xdisable-assertions". Use different configuration, say Test, and you'll get different results.
There's however a hitch in SBT (I missed the very first time I responded) - settings are chained and when a setting is not defined in a scope, it gets its value from a more general scope. When I said, scalacOptions +=... is in fact scalacOptions in Compile I missed the important feature of settings - scalacOptions is global while scalacOptions in Compile is Compile-scoped.

Resources