SBT run task after all subprojects - sbt

I want to write a task that is run after all subproject tasks are complete.
Eg, if I do
sbt a b then after task a is complete on ALL subprojects I want to execute task b. I don't want to do (a b) on each project.
Is that possible?
In fact, I'll take it so that I modify the build.sbt directly. I don't necessarily have to specify it at the command line.

I wrote a blog post on the subject: sequencing tasks with sbt-sequential.
addCommandAlias
Here's an example. We'll define a custom task a in sub1 and sub2 and b in root. The simplest way to achieve sequential execution is using addCommandAlias, so we'll just do that.
lazy val a = taskKey[Unit]("a")
lazy val b = taskKey[Unit]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(addCommandAlias("ab", ";a;b"): _*).
settings(
b := {
println("b")
}
)
lazy val sub1 = (project in file("sub1")).
settings(a := println("a - sub1"))
lazy val sub2 = (project in file("sub2")).
settings(a := println("a - sub2"))
You can run this from shell as sbt ab.
$ sbt ab
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a - sub2
a - sub1
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
b
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
Def.taskDyn
Here's another example. This time using Def.taskDyn, which is also featured in the blog post.
I'm constructing a ScopeFilter from the aggregate and then I'm dispatching task a to them.
lazy val a = taskKey[File]("a")
lazy val b = taskKey[Seq[File]]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(
b := (Def.taskDyn {
val proj = thisProject.value
val filter = ScopeFilter(inProjects(proj.aggregate: _*))
Def.task {
val allValues = a.all(filter).value
println(allValues.mkString(","))
allValues
}
}).value
)
lazy val sub1 = (project in file("sub1")).
settings(a := new File("a"))
lazy val sub2 = (project in file("sub2")).
settings(a := new File("b"))
You can run this from shell as sbt b.
$ sbt b
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a,b
[success] Total time: 0 s, completed Nov 23, 2014 9:42:16 PM

Related

References to undefined settings at runtime on a recursive task

While trying to execute sequentially a task run.toTask("") sequentially in an aggregate project, sbt cannot get settings:
[error] (run) sbt.internal.util.Init$RuntimeUndefined: References to undefined settings at runtime.
[error] ScopedKey(Scope(Select(ProjectRef(,exampleUsingOlderSparkVersion)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleWordCount)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleSparkCodeMigration)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleSimpleApp)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
with this definition:
def example(project:Project):Project = project.dependsOn(core)
lazy val exampleSimpleApp = (project in file("examples/simple-app")).configure(example)
lazy val exampleSparkCodeMigration = (project in file("examples/spark-code-migration")).configure(example)
lazy val exampleUsingOlderSparkVersion = (project in file("examples/using-older-spark-version")).configure(example)
lazy val exampleWordCount = (project in file("examples/word-count")).configure(example)
lazy val examples = (project in file("examples"))
.aggregate(exampleSimpleApp, exampleSparkCodeMigration, exampleUsingOlderSparkVersion, exampleWordCount)
.settings(
run := (Def.taskDyn {
val proj = thisProject.value
val filter = ScopeFilter(inProjects(proj.aggregate: _*))
run.toTask("").all(filter) // <- here
}).value
)
Is there a way to avoid that?
trying with:
lazy val childAggFilter = ScopeFilter(inAggregates(ThisProject, includeRoot = false))
lazy val runAll = taskKey[Unit]("run all childs")
lazy val examples = (project in file("examples"))
.aggregate(exampleSimpleApp, exampleSparkCodeMigration, exampleUsingOlderSparkVersion, exampleWordCount)
.settings(
runAll := Def.sequential(run.toTask("") all childAggFilter).value
)
I don't get a better result

How to wait for just the first task to complete

If there are a set of concurrent tasks scheduled, how to wait for the first task to complete?.
Does exists something like wait([t1, t2], return_when=:FIRST_COMPLETED)?
function task1()
rt = rand()
#info "task1 time: $rt"
sleep(rt)
#info "task 1 done"
end
function task2()
rt = rand()
#info "task2 time: $rt"
sleep(rt)
#info "task 2 done"
end
t1 = #async task1()
t2 = #async task2()
# wait for just the first completed task
# ???
I think something like this would work:
julia> function waitfirst(ts)
c = Channel{Int}()
for (i, t) in enumerate(ts)
#async begin
wait(t)
put!(c, i)
end
end
r = take!(c)
#info "Task $r won"
end
waitfirst (generic function with 1 method)
julia> waitfirst([#async(task1()), #async(task2())])
[ Info: task1 time: 0.6723665203367726
[ Info: task2 time: 0.4815093245244271
[ Info: task 2 done
[ Info: Task 2 won
julia> [ Info: task 1 done
julia>
julia> waitfirst([#async(task1()), #async(task2())])
[ Info: task1 time: 0.3452659680724035
[ Info: task2 time: 0.8576370382519976
[ Info: task 1 done
[ Info: Task 1 won
julia> [ Info: task 2 done
julia>
You have to make sure the inputs are scheduled first, though, otherwise it deadlocks.
Similar to channel, you can also use Conditions like this:
c = Condition()
t1 = #async (task1(); notify(c))
t2 = #async (task2(); notify(c))
wait(c)
#info "wake"
Note this only schedule the main task, not directly switch to it. i.e. it is possible that when t1 finished, the main task is waken, but t2 get scheduled first and do some computation heavy work (blocking) before the main task actually continue. You can take a look at yield if you want to make sure the main task runs first.

Running R in Scala using rscala causing issues

I recently came across an R package called rscala. https://darrenjw.wordpress.com/tag/rscala/
I tried executing the example however the program never completed running. I am not sure what may be wrong. Whenever I try to instantiate RClient, it seems to run forever. Please help.
For me the following code runs:
import breeze.stats.distributions._
import breeze.linalg._
import org.ddahl.rscala.RClient
object ScalaToRTest {
def main(args: Array[String]): Unit = {
// first simulate some data consistent with a Poisson regression model
val x = Uniform(50,60).sample(1000)
val eta = x map { xi => (xi * 0.1) - 3 }
val mu = eta map { math.exp }
val y = mu map { Poisson(_).draw }
// call to R to fit the Poission regression model
val R = RClient() // initialise an R interpreter
R.x=x.toArray // send x to R
R.y=y.toArray // send y to R
R.eval("mod <- glm(y~x,family=poisson())") // fit the model in R
// pull the fitted coefficents back into scala
val beta = DenseVector[Double](R.evalD1("mod$coefficients"))
// print the fitted coefficents
println(beta)
}
}
Output:
DenseVector(-3.1683714618415855, 0.1031332817387318)
build.sbt
name := "scalaRdemo"
version := "0.1"
scalaVersion := "2.12.3"
scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature")
resolvers ++= Seq(
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
libraryDependencies += "org.scalanlp" %% "breeze-natives" % "0.13.2"
libraryDependencies += "org.scalanlp" %% "breeze" % "0.13.2"
libraryDependencies += "org.ddahl" %% "rscala" % "2.3.5"

Struct member selected from type, it is not visible and will not be selected

I have a function that uses the Unix lib for its time functions:
let rfc822 (t: Unix.tm) : string =
Printf.sprintf "%s, %s %s %d %s:%s:%s %s"
(List.nth short_days t.tm_wday)
(padInt t.tm_yday 2 "0")
(List.nth short_month t.tm_mon)
(t.tm_year + 1900)
(padInt t.tm_hour 2 "0")
(padInt t.tm_min 2 "0")
(padInt t.tm_sec 2 "0")
"GMT"
I'm getting this warning:
ocamlbuild -libs unix,str -Is recore/src,ostd/src,owebl/src app.native
+ /usr/bin/ocamlc -c -I recore/src -I ostd/src -I owebl/src -o recore/src/time.cmo recore/src/time.ml
File "recore/src/time.ml", line 45, characters 27-34:
Warning 40: tm_wday was selected from type Unix.tm.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
File "recore/src/time.ml", line 46, characters 14-21:
Warning 40: tm_yday was selected from type Unix.tm.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
File "recore/src/time.ml", line 46, characters 4-28:
Error: This expression has type 'a -> string
but an expression was expected of type string
Command exited with code 2.
Compilation unsuccessful after building 13 targets (12 cached) in 00:00:00.
Makefile:8: recipe for target 'old' failed
make: *** [old] Error 10
How do I deal with this warning? I would much rather avoid opening the Unix module if possible.
(Please ignore the actual compile error.)
You can write t.Unix.tm_yday
$ ocaml
OCaml version 4.02.1
# let f (t: Unix.tm) = t.tm_yday;;
Warning 40: tm_yday was selected from type Unix.tm.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
val f : Unix.tm -> int = <fun>
# let f (t: Unix.tm) = t.Unix.tm_yday;;
val f : Unix.tm -> int = <fun>
Update
To find this in the documents, you need to look for the definition of field:
field ::= [ module-path . ] field-name
A field name can include a module name (or a sequence of module names, for nested modules) before the field name itself.
Update 2
There are also two syntaxes for opening a module locally. They look like overkill for this tiny function, but might be tidier for more complex ones. The module's symbols are directly available throughout the subexpression.
$ ocaml
OCaml version 4.02.1
# let f t = Unix.(t.tm_yday);;
val f : Unix.tm -> int = <fun>
# let f t = let open Unix in t.tm_yday;;
val f : Unix.tm -> int = <fun>
These are documented as language extensions in Local opens.

How to specify the order of custom task execution between projects?

Problem
I want to use sbt to execute custom shell scripts (as a part of a bigger build) within multiple directories in an order that respects a defined dependency graph. For example, given directories A, B, C and D and dependencies B after A, C after B, D after B the following is correct:
A, B, C, D
A, B, D, C
and the following is not correct:
B, A, C, D
A, C, B, D etc.
I created a custom key and custom task.
val script = taskKey[String]("script")
script := println("script")
and defined 4 projects:
lazy val a = Project(id = "a", base = file("A"))
lazy val b = Project(id = "b", base = file("B")).dependsOn(a)
lazy val c = Project(id = "c", base = file("C")).dependsOn(b)
lazy val d = Project(id = "d", base = file("D")).dependsOn(b)
I'm trying to run "script" task in SBT Console and it produces a random execution order that does not respect dependencies above.
Question
Does dependsOn mean a generic dependency or some Java/Scope/Whatever specific dependency?
Is there a solution at all and if yes then what is it?

Resources