How can I make a task depend on another task? - sbt

I'm new to sbt and I try to create a script for either deploy my application or to deploy and run the application.
What already works for me is
sbt deploy
which will successfully deploy the final .jar file to the remove location.
However, I don't know how to make deployAndRunTask dependent on deployTask. I've tried several things but none of them worked so far.
My last hope was
deployAndRunTask := {
val d = deployTask.value
}
However, this does not seem to work.
This is the script that I'm currently at but sbt deploy-run will only execute the deployAndRunTask task but not the deyployTask.
// DEPLOYMENT
val deployTask = TaskKey[Unit]("deploy", "Copies assembly jar to remote location")
deployTask <<= assembly map { (asm) =>
val account = "user#example.com"
val local = asm.getPath
val remote = account + ":" + "/home/user/" + asm.getName
println(s"Copying: $local -> $account:$remote")
Seq("scp", local, remote) !!
}
val deployAndRunTask = TaskKey[Unit]("deploy-run", "Deploy and run application.")
deployAndRunTask := {
val d = deployTask.value
}
deployAndRunTask <<= assembly map { (asm) =>
println(s"Running the script ..")
}
What is the problem here?

The problem is that you define your task and then redefine it. So only the latter definition is taken into account. You cannot separate task definition and its dependency on another task. Also you're using a couple of outdated things in sbt:
use taskKey macro and you don't need to think about task name, because it's the same as the key name:
val deploy = taskKey[Unit]("Copies assembly jar to remote location")
val deployAndRun = taskKey[Unit]("Deploy and run application.")
Then you can refer to them as deploy and deployAndRun both in build.sbt and in the sbt shell
replace <<= with := and keyname map { (keyvalue) => ... } with just keyname.value. Things are more concise and easier to write.
You can read more about Migrating from sbt 0.13.x.
So here's your deployAndRun task definition with these changes:
deployAndRun := {
val d = deploy.value
val asm = assembly.value
println(s"Running the script ..")
}
It's dependent both on deploy and assembly tasks and will run them both before doing anything else. You can also use dependsOn, but I think it's unnecessary here.
You may also be interested in looking into Defining a sequential task with Def.sequential and Defining a dynamic task with Def.taskDyn.

Related

Distributing APK splits with Firebase App Distribution

“Is it possible to use Firebase App Distribution with APK split? It doesn't declare a dependency on assemble task, are there any workarounds for this?"
The problem with the gradle plugin is that it
doesn't declare dependency on assemble task (in general, regardless of apk splits, by gradle convention, you shouldn't just "expect" the apks to be there)
doesn't generate tasks per apk splits -- but you do for flavors
Try the following work around:
// Generate firebase app distribution task variants for all abis
applicationVariants.all { variant ->
variant.outputs.all { output ->
def abi = output.getFilter(com.android.build.OutputFile.ABI)
if (abi == null) return
def abiName = abi.replace("_", "").replace("-", "")
task("appDistributionUpload${abiName.capitalize()}${variant.name.capitalize()}", type: com.google.firebase.appdistribution.gradle.UploadDistributionTask_Decorated) {
appDistributionProperties = new com.google.firebase.appdistribution.gradle.AppDistributionProperties(
new com.google.firebase.appdistribution.gradle.AppDistributionExtension(),
project,
variant
)
appDistributionProperties.apkPath = output.outputFile.absolutePath
appDistributionProperties.serviceCredentialsFile = project.file("secrets/ci-firebase-account.json")
appDistributionProperties.releaseNotes = abi
appDistributionProperties.groups = "ra-testers"
// Add dependsOn respective assemble task, so it actually
// builds apk it wants to upload, not just expect it to be there
dependsOn "assemble${variant.name.capitalize()}"
}
}
}

Find master project root directory in sbt multi-project

Is it possible to reference the master project from a subproject, in a multi-project sbt file?
I am writing a custom task, and I need to find two directories:
from the master project: the baseDirectory
from the subproject: the target
Of course, each of these are available inside their own projects. But I need to access them in the same code.
How can I do that?
The project layout is:
some/dir/build.sbt
val masterRoot = baseDirectory.value.getAbsolutePath // this works
lazy val root = (project in file(".")).aggregate(subproject)
some/dir/subproject/build.sbt
lazy val someTask = TaskKey[String]("someTask")
someTask := {
val subprojectTarget = target.value.getAbsolutePath // this works
val masterRootBroken = baseDirectory.in(root).value.getAbsolutePath // root is not found
// I need access to subprojectTarget AND masterRoot here
}
Alternatively, can I set a value into a SettingKey in the graph in the master project, and read it in the subproject?
I think there are two options available to you
First option is multi files project structure that you already have
build.sbt:
val sub = (project in file("sub"))
val root = (project in file("."))
Note: None of the above lines are mandatory. They are defined just to represent some possible additional logic like aggregate.
And sub/build.sbt with content
val root = (project in file("..")) //Note that ".." is used to refer to root project folder
val combinedPath = TaskKey[String]("combinedPath")
combinedPath := {
target.value.getAbsolutePath + baseDirectory.in(root).value.getAbsolutePath
}
Second one is to combine all build.sbt files into one build.sbt in root project with content
val combinedPath = TaskKey[String]("combinedPath")
val sub = (project in file("sub"))
.settings(
combinedPath := {
target.value.getAbsolutePath + baseDirectory.in(root).value.getAbsolutePath
}
)
lazy val root = (project in file("."))
Definition of task combinedPath is done in settings of sub project and it can refer to baseDirectory.in(root) of root.

How do you restore private NuGet packages from private VSTS feeds with Cake

I have a task which restores our NuGet package for our dotnet core application:
Task("Restore-Packages")
.Does(() =>
{
DotNetCoreRestore(sln, new DotNetCoreRestoreSettings {
Sources = new[] {"https://my-team.pkgs.visualstudio.com/_packaging/my-feed/nuget/v3/index.json"},
Verbosity = DotNetCoreVerbosity.Detailed
});
});
However when run on VSTS it errors with the following:
2018-06-14T15:10:53.3857512Z C:\Program Files\dotnet\sdk\2.1.300\NuGet.targets(114,5): error : Unable to load the service index for source https://my-team.pkgs.visualstudio.com/_packaging/my-feed/nuget/v3/index.json. [D:\a\1\s\BitCoinMiner.sln]
2018-06-14T15:10:53.3857956Z C:\Program Files\dotnet\sdk\2.1.300\NuGet.targets(114,5): error : Response status code does not indicate success: 401 (Unauthorized). [D:\a\1\s\BitCoinMiner.sln]
How do I authorize access for the build agent to our private VSTS?
I literally just had this same problem, apparently the build agents in VSTS can't get to your private VSTS feed without an access token so you are going to have to create a Personal Access Token in VSTS and provide that to the built in Cake method to add an authenticated VSTS Nuget feed as one of the sources. Here, I have wrapped it in my own convenience Cake method that checks to see if the package feed is already present, if not, then it adds it:
void SetUpNuget()
{
var feed = new
{
Name = "<feedname>",
Source = "https://<your-vsts-account>.pkgs.visualstudio.com/_packaging/<yournugetfeed>/nuget/v3/index.json"
};
if (!NuGetHasSource(source:feed.Source))
{
var nugetSourceSettings = new NuGetSourcesSettings
{
UserName = "<any-odd-string>",
Password = EnvironmentVariable("NUGET_PAT"),
Verbosity = NuGetVerbosity.Detailed
};
NuGetAddSource(
name:feed.Name,
source:feed.Source,
settings:nugetSourceSettings);
}
}
and then I call it from the "Restore" task:
Task("Restore")
.Does(() => {
SetUpNuget();
DotNetCoreRestore("./<solution-name>.sln");
});
Personally, I prefer to keep PATs away from the source control so here I am reading from env vars. In VSTS you can create an environment variable under the Variables tab of your CI build configuration.
Hope this helps! Here is a link to Cake's documentation.
As pointed out by both #KevinSmith and #NickTurner, a better approach to accessing the VSTS feed is by using the pre-defined system variable System.AccessToken as opposed to using limited validity, manually created and cumbersome PATs. This variable is available on the build agents for the current build to use. More info here.
One way of using this token in the Cake script is as follows:
First, expose the system variable as an environment variable for the Cake task in azure-pipelines.yml
steps:
- task: cake-build.cake.cake-build-task.Cake#0
displayName: 'Cake '
inputs:
target: Pack
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
Then in Cake you can access it like you would any environment variable, so in my case:
if (!NuGetHasSource(source:feed.Source))
{
Information($"Nuget feed {feed.Source} not found, adding...");
var nugetSourceSettings = new NuGetSourcesSettings
{
UserName = "whoosywhatsit",
Password = EnvironmentVariable("SYSTEM_ACCESSTOKEN"),
Verbosity = NuGetVerbosity.Detailed
};
NuGetAddSource(
name:feed.Name,
source:feed.Source,
settings:nugetSourceSettings);
}
This seems to work! If there are better approaches of accessing this variable in Cake please let me know. Please also note in my case, I am only using this to restore packages from my VSTS feed, not for pushing to it. That I do via a DotNetCoreCLI#2 task in the YML like so:
- task: DotNetCoreCLI#2
displayName: 'dotnet nuget push'
inputs:
command: push
packagesToPush: 'artifacts/package.nupkg'
publishVstsFeed: '<id of my VSTS feed>'
And Azure pipeline handles the rest.

Different scalac options for different scopes or tasks?

I am trying to use a compiler plugin with sbt (I'm on 0.13.5), passed along in my build.sbt as:
autoCompilerPlugins := true
scalacOptions += "-Xplugin:myCompilerPluginJar.jar"
This works, the plugin runs, however I would really like to only run the plugin on some explicit compiles (perhaps with a scoped compile task or a custom task).
If I try something like:
val PluginConfig = config("plugin-config") extend(Compile)
autoCompilerPlugins := true
scalacOptions in PluginConfig += "-Xplugin:myCompilerPluginJar.jar"
The plugin does not run on "plugin-config:compile". In fact if I have
scalacOptions in Compile += "-Xplugin:myCompilerPluginJar.jar"
The plugin still runs on "test:compile" or compile on any other scope. I would guess I am probably not understanding something correctly with the configs/scopes.
I also tried:
lazy val pluginCommand = Command.command("plugincompile") { state =>
runTask(compile in Compile,
append(Seq(scalacOptions in Compile += "Xplugin:myCompilerPluginJar.jar"), state)
)
state
}
commands += pluginCommand
But the plugin doesn't actually run on that command, so again I am probably not understanding something there.
Any and all help welcome.
So I have come to hacky solution; I thought I would share it here in case anyone else stumbles upon this question.
val safeCompile = TaskKey[Unit]("safeCompile", "Compiles, catching errors.")
safeCompile := (compile in Compile).result.value.toEither.fold(
l => {
println("Compilation failed.")
}, r => {
println("Compilation success. " + r)})
//Hack to allow "-deprecation" and "-unchecked" in scalacOptions by default
scalacOptions <<= scalacOptions map { current: Seq[String] =>
val default = "-deprecation" :: "-unchecked" :: Nil
if (current.contains("-Xplugin:myCompilerPluginJar.jar")) current else default
}
addCommandAlias("depcheck", "; set scalacOptions := Seq(\"-deprecation\", \"-unchecked\", \"-Xplugin:myCompilerPluginJar.jar\"); safeCompile; set scalacOptions := Seq(\"-deprecation\", \"-unchecked\")")
As a quick guide, this code:
Defines a custom task "safeCompile" that runs the "Compile:compile" task, but succeeds even on errors (this is needed so that the sequence of commands defined later on doesn't break on compilation failure).
Declares "scalacOptions" to be dependent on a function that checks if the plugin is turned on (leaving the options untouched if it is) and otherwise sets the options to the default I want for the project (Seq("-deprecation", "-unchecked")). This is a hack so that these settings are on by default and so that a bare "scalacOptions :=" definition doesn't override the settings done in the aliased command sequence. (Using Seq.append and Seq.distinct might be a nicer way to do this hacky part).
Defines an aliased command sequence that: turns the plugin on, safeCompiles, turns the plugin off.
Comments are welcome, and if you get something cleaner to work, please share!

Cleanest way in Gradle to get the path to a jar file in the gradle dependency cache

I'm using Gradle to help automate Hadoop tasks. When calling Hadoop, I need to be able to pass it the path to some jars that my code depends on so that Hadoop can send that dependency on during the map/reduce phase.
I've figured out something that works, but it feels messy and I'm wondering if there's a feature I'm missing somewhere.
This is a simplified version of my gradle script that has a dependency on the solr 3.5.0 jar, and a findSolrJar task that iterates through all of the jar files in the configuration to find the right one:
apply plugin: 'groovy'
repositories {
mavenCentral()
}
dependencies {
compile 'org.apache.solr:solr-solrj:3.5.0'
}
task findSolrJar() {
println project.configurations.compile*.toURI().find { URI uri -> new File(uri).name == 'solr-solrj-3.5.0.jar'}
}
running this gives me output like this:
gradle findSolrJar
file:/Users/tnaleid/.gradle/caches/artifacts-8/filestore/org.apache.solr/solr-solrj/3.5.0/jar/74cd28347239b64fcfc8c67c540d7a7179c926de/solr-solrj-3.5.0.jar
:findSolrJar UP-TO-DATE
BUILD SUCCESSFUL
Total time: 2.248 secs
Is there a better way to do this?
Your code can be simplified a bit, for example project.configurations.compile.find { it.name.startsWith("solr-solrj-") }.
You can also create a dedicated configuration for an artifact, to keep it clean; and use asPath if the fact that it can potentially return several locations works well for your use case (happens if it resolves same jar in several locations):
configurations {
solr
}
dependencies {
solr 'org.apache.solr:solr-solrj:3.5.0'
}
task findSolrJars() {
println configurations.solr.asPath
}
To avoid copy-paste, in case you as well need that jar in compile configuration, you may add this dedicated configuration into compile one, like:
dependencies {
solr 'org.apache.solr:solr-solrj:3.5.0'
compile configurations.solr.dependencies
}
I needed lombok.jar as a java build flag to gwt builds this worked great !
configurations {
lombok
}
dependencies {
lombok 'org.projectlombok:lombok+'
}
ext {
lombok = configurations.lombok.asPath
}
compileGwt {
jvmArgs "-javaagent:${lombok}=ECJ"
}
I was surprised that the resolution worked early enough in the configuraiton phase, but it does.
Here is how I did it:
project.buildscript.configurations.classpath.each {
String jarName = it.getName();
print jarName + ":"
}
I recently had this problem as well. If you are building a java app, the problem at hand is normally that want to get the group:module (groupId:artifactId) to path-to-jar mapping (i.e. the version is not a search criteria as in one app there is normally only one version of each specific jar).
In my gradle 5.1.1 (kotlin-based) gradle build I solved this problem with:
var spec2File: Map<String, File> = emptyMap()
configurations.compileClasspath {
val s2f: MutableMap<ResolvedModuleVersion, File> = mutableMapOf()
// https://discuss.gradle.org/t/map-dependency-instances-to-file-s-when-iterating-through-a-configuration/7158
resolvedConfiguration.resolvedArtifacts.forEach({ ra: ResolvedArtifact ->
s2f.put(ra.moduleVersion, ra.file)
})
spec2File = s2f.mapKeys({"${it.key.id.group}:${it.key.id.name}"})
spec2File.keys.sorted().forEach({ it -> println(it.toString() + " -> " + spec2File.get(it))})
}
The output would be some like:
:jing -> /home/tpasch/scm/db-toolchain/submodules/jing-trang/build/jing.jar
:prince -> /home/tpasch/scm/db-toolchain/lib/prince-java/lib/prince.jar
com.github.jnr:jffi -> /home/tpasch/.gradle/caches/modules-2/files-2.1/com.github.jnr/jffi/1.2.18/fb54851e631ff91651762587bc3c61a407d328df/jffi-1.2.18-native.jar
com.github.jnr:jnr-constants -> /home/tpasch/.gradle/caches/modules-2/files-2.1/com.github.jnr/jnr-constants/0.9.12/cb3bcb39040951bc78a540a019573eaedfc8fb81/jnr-constants-0.9.12.jar
com.github.jnr:jnr-enxio -> /home/tpasch/.gradle/caches/modules-2/files-2.1/com.github.jnr/jnr-enxio/0.19/c7664aa74f424748b513619d71141a249fb74e3e/jnr-enxio-0.19.jar
After that, it is up to you to do something useful with this Map. In my case I add some --path-module options to my Java 11 build like this:
val patchModule = listOf(
"--patch-module", "commons.logging=" +
spec2File["org.slf4j:jcl-over-slf4j"].toString(),
"--patch-module", "org.apache.commons.logging=" +
spec2File["org.slf4j:jcl-over-slf4j"].toString()
)
patchModule.forEach({it -> println(it)})
tasks {
withType<JavaCompile> {
doFirst {
options.compilerArgs.addAll(listOf(
"--release", "11",
"--module-path", classpath.asPath
) + patchModule)
// println("Args for for ${name} are ${options.allCompilerArgs}")
}
}
}

Resources