While trying to execute sequentially a task run.toTask("") sequentially in an aggregate project, sbt cannot get settings:
[error] (run) sbt.internal.util.Init$RuntimeUndefined: References to undefined settings at runtime.
[error] ScopedKey(Scope(Select(ProjectRef(,exampleUsingOlderSparkVersion)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleWordCount)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleSparkCodeMigration)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
[error] ScopedKey(Scope(Select(ProjectRef(,exampleSimpleApp)), Zero, Zero, Zero),run) referenced from setting(ScopedKey(Scope(Select(ProjectRef(,examples)), Zero, Zero, Zero),run)) at LinePosition(build.sbt,140)
with this definition:
def example(project:Project):Project = project.dependsOn(core)
lazy val exampleSimpleApp = (project in file("examples/simple-app")).configure(example)
lazy val exampleSparkCodeMigration = (project in file("examples/spark-code-migration")).configure(example)
lazy val exampleUsingOlderSparkVersion = (project in file("examples/using-older-spark-version")).configure(example)
lazy val exampleWordCount = (project in file("examples/word-count")).configure(example)
lazy val examples = (project in file("examples"))
.aggregate(exampleSimpleApp, exampleSparkCodeMigration, exampleUsingOlderSparkVersion, exampleWordCount)
.settings(
run := (Def.taskDyn {
val proj = thisProject.value
val filter = ScopeFilter(inProjects(proj.aggregate: _*))
run.toTask("").all(filter) // <- here
}).value
)
Is there a way to avoid that?
trying with:
lazy val childAggFilter = ScopeFilter(inAggregates(ThisProject, includeRoot = false))
lazy val runAll = taskKey[Unit]("run all childs")
lazy val examples = (project in file("examples"))
.aggregate(exampleSimpleApp, exampleSparkCodeMigration, exampleUsingOlderSparkVersion, exampleWordCount)
.settings(
runAll := Def.sequential(run.toTask("") all childAggFilter).value
)
I don't get a better result
Related
I have implemented a function for sampling from M different normal distributions N times in Rust because my R code was too slow. It is also parallelized. Here is the pure Rust code:
use rand_distr::{Normal, Distribution};
use rayon::prelude::*;
fn rust_rprednorm(n: i32, means: Vec<f64>, sds: Vec<f64>) -> Vec<Vec<f64>> {
let mut preds = vec![vec![0.0; n as usize]; means.len()];
preds.par_iter_mut().enumerate().for_each(|(i, e)| {
let mut rng = rand::thread_rng();
(0..n).into_iter().for_each(|j| {
let normal = Normal::new(means[i], sds[i]).unwrap();
e[j as usize] = normal.sample(&mut rng);
})
});
preds
}
Which I am trying to call from R using the rextendr library. The code is inside a utils.R file I import using source():
code <- r"(
use rand_distr::{Normal, Distribution};
use rayon::prelude::*;
#[extendr]
fn rust_rprednorm(n: i32, means: Vec<f64>, sds: Vec<f64>) -> Vec<Vec<f64>> {
let mut preds = vec![vec![0.0; n as usize]; means.len()];
preds.par_iter_mut().enumerate().for_each(|(i, e)| {
let mut rng = rand::thread_rng();
(0..n).into_iter().for_each(|j| {
let normal = Normal::new(means[i], sds[i]).unwrap();
e[j as usize] = normal.sample(&mut rng);
})
});
preds
}
)"
rust_source(code = code, dependencies = list(`rand` = "0.8.5", `rand_distr` ="0.4.3", `rayon` = "1.6.1"))
The error is:
Error in `invoke_cargo()`:
! Rust code could not be compiled successfully. Aborting.
✖ error[E0277]: the trait bound `Robj: From<Vec<Vec<f64>>>` is not satisfied
--> src\lib.rs:6:5
|
6 | #[extendr]
| ^^^^^^^^^^ the trait `From<Vec<Vec<f64>>>` is not implemented for `Robj`
|
= help: the following other types implement trait `From<T>`:
<Robj as From<&'a [T]>>
<Robj as From<&Altrep>>
<Robj as From<&Primitive>>
<Robj as From<&Robj>>
<Robj as From<&Vec<T>>>
<Robj as From<&extendr_api::Complexes>>
<Robj as From<&extendr_api::Doubles>>
<Robj as From<&extendr_api::Environment>>
and 71 others
= note: this error originates in the attribute macro `extendr` (in Nightly builds, run with -Z macro-backtrace for more info)
✖ error: aborting due to previous error
Traceback:
1. source("inla_predictive_distribution_utils.R")
2. withVisible(eval(ei, envir))
3. eval(ei, envir)
4. eval(ei, envir)
5. rust_source(code = code, dependencies = list(rand = "0.8.5",
. rand_distr = "0.4.3", rayon = "1.6.1"))
6. invoke_cargo(toolchain = toolchain, specific_target = specific_target,
. dir = dir, profile = profile, quiet = quiet, use_rtools = use_rtools)
7. check_cargo_output(compilation_result, message_buffer, tty_has_colors(),
. quiet)
8. ui_throw("Rust code could not be compiled successfully. Aborting.",
. error_messages, call = call, glue_open = "{<{", glue_close = "}>}")
9. withr::with_options(list(warning.length = message_limit_bytes),
. rlang::abort(message, class = "rextendr_error", call = call))
10. force(code)
11. rlang::abort(message, class = "rextendr_error", call = call)
12. signal_abort(cnd, .file)
I figured a nested vector would be supported since a regular vector is, but it appears not, do I need to implement this trait for Robj, or is there another way to go about it? I am also not sure if this is the recommended way to call Rust code from R.
I recently came across an R package called rscala. https://darrenjw.wordpress.com/tag/rscala/
I tried executing the example however the program never completed running. I am not sure what may be wrong. Whenever I try to instantiate RClient, it seems to run forever. Please help.
For me the following code runs:
import breeze.stats.distributions._
import breeze.linalg._
import org.ddahl.rscala.RClient
object ScalaToRTest {
def main(args: Array[String]): Unit = {
// first simulate some data consistent with a Poisson regression model
val x = Uniform(50,60).sample(1000)
val eta = x map { xi => (xi * 0.1) - 3 }
val mu = eta map { math.exp }
val y = mu map { Poisson(_).draw }
// call to R to fit the Poission regression model
val R = RClient() // initialise an R interpreter
R.x=x.toArray // send x to R
R.y=y.toArray // send y to R
R.eval("mod <- glm(y~x,family=poisson())") // fit the model in R
// pull the fitted coefficents back into scala
val beta = DenseVector[Double](R.evalD1("mod$coefficients"))
// print the fitted coefficents
println(beta)
}
}
Output:
DenseVector(-3.1683714618415855, 0.1031332817387318)
build.sbt
name := "scalaRdemo"
version := "0.1"
scalaVersion := "2.12.3"
scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature")
resolvers ++= Seq(
"Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Sonatype Releases" at "https://oss.sonatype.org/content/repositories/releases/"
)
libraryDependencies += "org.scalanlp" %% "breeze-natives" % "0.13.2"
libraryDependencies += "org.scalanlp" %% "breeze" % "0.13.2"
libraryDependencies += "org.ddahl" %% "rscala" % "2.3.5"
I'm constructing a simple type provider, but I seem to be running into problems when referencing types I created. For instance, given
namespace Adder
type Summation = Summation of int
module QuickAdd =
let add x y = x + y |> Summation
I want to make the following test case pass:
module Adder.Tests
open Adder
open NUnit.Framework
type Simple = QuickAddProvider<1, 2>
[<Test>]
let ``Simple sample is 3`` () =
let foo = Simple()
Assert.AreEqual(foo.Sample, Summation 3)
With the following type provider:
namespace Adder
open Microsoft.FSharp.Core.CompilerServices
open ProviderImplementation.ProvidedTypes
open System.Reflection
[<TypeProvider>]
type public QuickAddProvider (config : TypeProviderConfig) as this =
inherit TypeProviderForNamespaces ()
let ns = "Adder"
let asm = Assembly.GetExecutingAssembly()
let paraProvTy = ProvidedTypeDefinition(asm, ns, "QuickAddProvider", Some typeof<obj>)
let buildTypes (typeName:string) (args:obj[]) =
let num1 = args.[0] :?> int
let num2 = args.[1] :?> int
let tpType = ProvidedTypeDefinition(asm, ns, typeName, Some typeof<obj>)
let result = QuickAdd.add num1 num2
let orig = ProvidedProperty("Sample", typeof<Summation>, GetterCode = (fun args -> <## result ##>))
tpType.AddMember(orig)
tpType.AddMember(ProvidedConstructor([], InvokeCode = (fun args -> <## () ##>)))
tpType
let parameters =
[ProvidedStaticParameter("Num1", typeof<int>)
ProvidedStaticParameter("Num2", typeof<int>)]
do paraProvTy.DefineStaticParameters(parameters, buildTypes)
do this.AddNamespace(ns, [paraProvTy])
[<TypeProviderAssembly>]
do()
I run into unexpected errors in the test file:
The type provider 'Adder.QuickAddProvider' reported an error in the context of provided type 'Adder.QuickAddProvider,Num1="1",Num2="2"', member 'get_Sample'. The error: Unsupported constant type 'Adder.Summation'
With the following errors in the generated file:
The type "Summation" is not defined
The namespace or module "Adder" is not defined
The test case compiles and passes when replacing the Summation type with int, so I know my type provider isn't terribly wrong. Do I need to somehow "import" the Summation type somewhere?
This error usually means that you are creating a quotation that contains a value of custom type. The quotations in type providers can only contain values of primitive types - the compiler knows how to serialize these - but it cannot handle custom types.
In the snippet, this happens here:
let result = QuickAdd.add num1 num2
let orig = ProvidedProperty("Sample", typeof<Summation>, GetterCode = (fun args ->
<## result ##>))
Here, the GetterCode returns a quotation containing value of type Summation which is not supported. To make this work, you can do various things - generally, you'll need to come up with some other quoted expression that produces the value you want.
One option is to do the calculation inside the quotation rather than outside:
<## QuickAdd.add num1 num2 ##>
The other option would be to re-create the Summation value in the quotation:
let (Summation n) = result
<## Summation n ##>
This works, because it only needs to serialize a primitive int value and then generate a call to the Summation case constructor.
I want to write a task that is run after all subproject tasks are complete.
Eg, if I do
sbt a b then after task a is complete on ALL subprojects I want to execute task b. I don't want to do (a b) on each project.
Is that possible?
In fact, I'll take it so that I modify the build.sbt directly. I don't necessarily have to specify it at the command line.
I wrote a blog post on the subject: sequencing tasks with sbt-sequential.
addCommandAlias
Here's an example. We'll define a custom task a in sub1 and sub2 and b in root. The simplest way to achieve sequential execution is using addCommandAlias, so we'll just do that.
lazy val a = taskKey[Unit]("a")
lazy val b = taskKey[Unit]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(addCommandAlias("ab", ";a;b"): _*).
settings(
b := {
println("b")
}
)
lazy val sub1 = (project in file("sub1")).
settings(a := println("a - sub1"))
lazy val sub2 = (project in file("sub2")).
settings(a := println("a - sub2"))
You can run this from shell as sbt ab.
$ sbt ab
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a - sub2
a - sub1
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
b
[success] Total time: 0 s, completed Nov 22, 2014 8:36:18 PM
Def.taskDyn
Here's another example. This time using Def.taskDyn, which is also featured in the blog post.
I'm constructing a ScopeFilter from the aggregate and then I'm dispatching task a to them.
lazy val a = taskKey[File]("a")
lazy val b = taskKey[Seq[File]]("b")
lazy val root = (project in file(".")).
aggregate(sub1, sub2).
settings(
b := (Def.taskDyn {
val proj = thisProject.value
val filter = ScopeFilter(inProjects(proj.aggregate: _*))
Def.task {
val allValues = a.all(filter).value
println(allValues.mkString(","))
allValues
}
}).value
)
lazy val sub1 = (project in file("sub1")).
settings(a := new File("a"))
lazy val sub2 = (project in file("sub2")).
settings(a := new File("b"))
You can run this from shell as sbt b.
$ sbt b
[info] Loading global plugins from ...
[info] Loading project definition from ...
[info] Set current project to root (in build ...)
a,b
[success] Total time: 0 s, completed Nov 23, 2014 9:42:16 PM
I need the object (or "singleton object" or "companion object"... anything but the class) defined by a string name. In other words, if I have:
package myPackage
object myObject
...then is there anything like this:
GetSingletonObjectByName("myPackage.myObject") match {
case instance: myPackage.myObject => "instance is what I wanted"
}
In scala 2.10 we can do like this
import scala.reflect.runtime.universe
val runtimeMirror = universe.runtimeMirror(getClass.getClassLoader)
val module = runtimeMirror.staticModule("package.ObjectName")
val obj = runtimeMirror.reflectModule(module)
println(obj.instance)
Scala is still missing a reflection API. You can get the an instance of the companion object by loading the companion object class:
import scala.reflect._
def companion[T](implicit man: Manifest[T]) : T =
man.erasure.getField("MODULE$").get(man.erasure).asInstanceOf[T]
scala> companion[List$].make(3, "s")
res0: List[Any] = List(s, s, s)
To get the untyped companion object you can use the class directly:
import scala.reflect.Manifest
def companionObj[T](implicit man: Manifest[T]) = {
val c = Class.forName(man.erasure.getName + "$")
c.getField("MODULE$").get(c)
}
scala> companionObj[List[Int]].asInstanceOf[List$].make(3, "s")
res0: List[Any] = List(s, s, s)
This depends on the way scala is mapped to java classes.
Adjustment to Thomas Jung's answer above: you would do better to say companion[List.type] because a) this should be a stable way to refer to it, not dependant on the name mangling scheme and b) you get the unerased types.
def singleton[T](implicit man: reflect.Manifest[T]) = {
val name = man.erasure.getName()
assert(name endsWith "$", "Not an object: " + name)
val clazz = java.lang.Class.forName(name)
clazz.getField("MODULE$").get(clazz).asInstanceOf[T]
}
scala> singleton[List.type].make(3, "a")
res0: List[java.lang.String] = List(a, a, a)
Barring reflection tricks, you can't. Note, for instance, how the method companion is defined on Scala 2.8 collections -- it is there so an instance of a class can get the companion object, which is otherwise not possible.
Scala 2:
val runtimeMirror = universe.runtimeMirror(this.getClass.getClassLoader)
val objectClass = Class.forName("org.test.MyObject$")
val moduleSymbol = runtimeMirror.staticModule(objectClass.getName)
val moduleMirror = runtimeMirror.reflectModule(moduleSymbol)
val instance = moduleMirror.instance
Or just use Java API to get the value from the static field:
val instance = objectClass.getField("MODULE$").get(null)
Mind the $ in the object name.