How to execute a bash script as sbt task? - sbt

I want to automatically build documentation for my Java Play 2.3 application.
At the moment, I use a Makefile to generate images from *.dotfiles and combine Markdown sources together into Html/PDF:
dot diagram1.dot -Tpdf -o diagram1.pdf
dot diagram2.dot -Tpdf -o diagram2.pdf
pandoc doc1.markdown -o doc1.pdf
# ...
Now I want to run these simple bash commands directly from SBT.
What's the best way to do this?
I found some SBT Documentation plugins in the SBT reference, but nothing to run a simple shell script.

You can find some answers in External Processes in the official documentation of sbt, e.g.
To run an external command, follow it with an exclamation mark !:
"find project -name *.jar" !
Don't forget to use import scala.sys.process._ so ! can be resolved as a method of String.
Do the following in activator console (aka sbt shell) to execute yourshell.sh - mind the eval command and the quotes around the name of the script:
eval "yourshell.sh" !
To have it available as a task add the following to build.sbt of your project:
lazy val execScript = taskKey[Unit]("Execute the shell script")
execScript := {
"yourshell.sh" !
}

We had the requirement to execute some npm scripts as sbt task and let the build fail if one of the npm scripts fails. Took me some time to find a way to create a Task that works on Windows and Unix. And so here is what I came up with.
lazy val buildFrontend = taskKey[Unit]("Execute frontend scripts")
buildFrontend := {
val s: TaskStreams = streams.value
val shell: Seq[String] = if (sys.props("os.name").contains("Windows")) Seq("cmd", "/c") else Seq("bash", "-c")
val npmInstall: Seq[String] = shell :+ "npm install"
val npmTest: Seq[String] = shell :+ "npm run test"
val npmLint: Seq[String] = shell :+ "npm run lint"
val npmBuild: Seq[String] = shell :+ "npm run build"
s.log.info("building frontend...")
if((npmInstall #&& npmTest #&& npmLint #&& npmBuild !) == 0) {
s.log.success("frontend build successful!")
} else {
throw new IllegalStateException("frontend build failed!")
}
},
(run in Compile) <<= (run in Compile).dependsOn(buildFrontend)

Related

Adding zsh ternary operator in npm script - bad subsitution

I am wanting to use a ternary operator in my npm script:
"scripts": {
"build" "node ${%(ENVVAR=nope.foo.bar)}.js"
}
So that I can use it like:
ENVVAR=nope yarn build
And obtain the resulting command of:
"node foo.js"
However my attempts above (tried parens and square brackets) always yield bad substitution
What is the proper syntax to include a ternary operator in my parameter expansion for this (zsh shell)
The npm docs state:
"The actual shell your script is run within is platform dependent. By default, on Unix-like systems it is the /bin/sh command, on Windows it is the cmd.exe. The actual shell referred to by /bin/sh also depends on the system. As of npm#5.1.0 you can customize the shell with the script-shell configuration."
Therefore given that the shell npm utilizes on *nix is typically sh consider the following npm script instead (Unless of course your npm's script-shell has actually been configured to zsh):
"scripts": {
"build": "[[ $ENVVAR = \"nope\" ]] && val=foo || val=bar; node \"${val}.js\""
}
Note: You can check which shell npm is utilizing via the npm config command: npm config get script-shell
Alternatively, for cross-platform regardless of which shell npm is configured to utilize, consider evaluating a node.js script inline which "shells-out" the node ... command. For example the following (albeit somewhat verbose) npm script utilizes execSync:
"scripts": {
"build": "node -e \"const val = process.env.ENVVAR === 'nope' ? 'foo' : 'bar'; require('child_process').execSync('node ' + val + '.js', { stdio: [0,1,2] })\""
}

How to run remote ssh command in sbt script?

I want to deploy and automatically run a script on a remote machine.
However, I don't know how I can issue a remote command.
I've tried several things and plugins but none of that stuff seems to work.
Is there a way to do it?
val deployAndRunTask = TaskKey[Unit]("deploy-run", "Deploy and run application.")
deployAndRunTask := {
// Deploy .jar file
val _ = deployTask.value
println("Running the script ..")
}
Here is a working example, which you can save in test.sbt in the top level directory of your SBT project. The shellRun method runs an arbitrary command. I use shellRun to run ssh, connect to the local machine, and list the files in my home directory.
import scala.sys.process.Process
val deployAndRunTask = TaskKey[Unit]("deploy-and-run-task", "Short example")
deployAndRunTask := {
def shellRun(command: String*) = Process(command.toSeq).!!.trim
val result = shellRun("/usr/bin/ssh", "localhost", "ls")
println(result)
}
Run it by typing:
sbt deployAndRunTask

Specifying rootProject in build.sbt

In my Build.scala, I have:
override def rootProject = Some(frontendProject)
I'm trying to convert to the newer build.sbt format, but don't know the equivalent of this line. How do I set the project for sbt to load by default when using build.sbt?
I'm still not sure that I understood you right, but you said about multi-project build, so I assume that you want to define a root project which aggregates subprojects. Here is how you can do that (in your root build.sbt):
lazy val root = project.in( file(".") ).aggregate(subProject1, subProject2)
lazy val subProject1 = project in file("subProject1")
lazy val subProject2 = project in file("subProject2")
See sbt documentation about multi-projects.
Then if you want to set the default project to load on sbt startup to a sub-project, in addition to your answer to this SO question, I can suggest
run sbt with sbt "project XXX" shell command
or adding this line to your build.sbt:
onLoad in Global := { Command.process("project XXX", _: State) } compose (onLoad in Global).value
In both cases sbt first loads the root project and then the subproject.
I've found the following script to be helpful:
#!/usr/bin/env bash
exec "sbt" "project mysubproject" "shell"
exit $?

Can sbt execute "compile test:compile it:compile" as a single command, say "*:compile"?

I'm running compile test:compile it:compile quite often and...would like to cut the number of keystrokes to something like *:compile. It doesn't seem to work, though.
$ sbt *:compile
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/oss/scalania/project
[info] Set current project to scalania (in build file:/Users/jacek/oss/scalania/)
[error] No such setting/task
[error] *:compile
[error] ^
Is it possible at all? I use SBT 0.13.
test:compile implies a compile so compile doesn't need to be explicitly run before test:compile. If your IntegrationTest configuration extends Test, it:compile implies test:compile.
One option is to define an alias that executes multiple commands:
sbt> alias compileAll = ; test:compile ; it:compile
See help alias and help ; for details. You can make this a part of your build with:
addCommandAlias("compileAll", "; test:compile ; it:compile")
The other option is to define a custom task that depends on the others and call that:
lazy val compileAll = taskKey[Unit]("Compiles sources in all configurations.")
compileAll := {
val a = (compile in Test).value
val b = (compile in IntegrationTest).value
()
}

Generate xcarchive into a specific folder from the command line

For the purposes of CI, I need to be able to generate an XCARCHIVE and an IPA file in our nightly build. The IPA is for our testers, to be signed with our ad-hoc keys, and the XCARCHIVE is to send to the client so that they can import it into Xcode and submit it to the app store when they're happy with it.
Generating the IPA is simple enough with a bit of googling, however how to generate the .XCARCHIVE file is what eludes me. The closest I've found is:
xcodebuild -scheme myscheme archive
However, this stores the .xcarchive in some hard-to-find folder, eg:
/Users/me/Library/Developer/Xcode/Archives/2011-12-14/MyApp 14-12-11 11.42 AM.xcarchive
Is there some way to control where the archive is put, what its name is, and how to avoid having to re-compile it? I guess the best possible outcome would be to generate the xcarchive from the DSYM and APP that are generated when you do an 'xcodebuild build' - is this possible?
Xcode 5 now supports an -archivePath option:
xcodebuild -scheme myscheme archive -archivePath /path/to/AppName.xcarchive
You can also now export a signed IPA from the archive you just built:
xcodebuild -exportArchive -exportFormat IPA -exportProvisioningProfile my_profile_name -archivePath /path/to/AppName.xcarchive -exportPath /path/to/AppName.ipa
Starting with Xcode 4 Preview 5 there are three environment variables that are accessible in the scheme archive's post-actions.
ARCHIVE_PATH: The path to the archive.
ARCHIVE_PRODUCTS_PATH: The installation location for the archived product.
ARCHIVE_DSYMS_PATH: The path to the product’s dSYM files.
You could move/copy the archive in here. I wanted to have a little more control over the process in a CI script, so I saved a temporary file that could easily be sourced in my CI script that contained these values.
BUILD_DIR=$PROJECT_DIR/build
echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $BUILD_DIR/archive_paths.sh
echo "ARCHIVE_PRODUCTS_PATH=\"$ARCHIVE_PRODUCTS_PATH\"" >> $BUILD_DIR/archive_paths.sh
echo "ARCHIVE_DSYMS_PATH=\"$ARCHIVE_DSYMS_PATH\"" >> $BUILD_DIR/archive_paths.sh
echo "INFOPLIST_PATH=\"$INFOPLIST_PATH\"" >> $BUILD_DIR/archive_paths.sh
Then in my CI script I can run the following:
xcodebuild -alltargets -scheme [Scheme Name] -configuration [Config Name] clean archive
source build/archive_paths.sh
ARCHIVE_NAME=AppName-$APP_VERSION-$APP_BUILD.xcarchive
cp -r "$ARCHIVE_PATH" "$BUILD_DIR/$ARCHIVE_NAME"
I have just solved this one - just add the argument -archivePath to your xcode build command line, given the initial question that would mean:
xcodebuild -scheme myscheme archive
becomes ...
xcodebuild -scheme myscheme archive -archivePath Build/Archive
(Note: paths are relative, I output my build to $PWD/Build)
This will then place your .app folder in:
Build/Archive.xarchive/Products/Application
If your build target already has your signing certificate and provisioning profile in it you can then create your IPA file without re-signing using the following command:
xcrun -v -sdk iphoneos PackageApplication -v `pwd`'/Build/Archive.xarchive/Products/Application/my.app' -o `pwd`'/myapp.ipa'
(Note: xcrun doesn't like relative paths hence the pwd)
The -v args dump lots of useful information - this command can fail to sign properly and still exit with code 0, sigh!
If you are finding that you can't run the built .ipa it's probably a signing issue that you can do a double check on using:
codesign --verify -vvvv myapp.app
If it's signed correctly and un-tampered with the output will have this in:
myapp.app: valid on disk
myapp.app: satisfies its Designated Requirement
If not you will see something similar to this:
Codesign check fails : /blahpath/myapp.app: a sealed resource is missing or invalid
file modified: /blahpath/ls-ios-develop.app/Assets.car
... which generally means you are trying to use an intermediate output directory rather than the proper archive.
My current solution is to rename the user's existing archives folder, run the build, and do a 'find' to copy the archives where i want, then delete the archives folder and rename the old folder back as it was, with code like this in my ruby build script:
# Move the existing archives out of the way
system('mv ~/Library/Developer/Xcode/Archives ~/Library/Developer/Xcode/OldArchivesTemp')
# Build the .app, the .DSYM, and the .xcarchive
system("xcodebuild -scheme \"#{scheme}\" clean build archive CONFIGURATION_BUILD_DIR=\"#{build_destination_folder}\"")
# Find the xcarchive wherever it was placed and copy it where i want it
system("find ~/Library/Developer/Xcode/Archives -name *.xcarchive -exec cp -r {} \"#{build_destination_folder}\" \";\"")
# Delete the new archives folder with this new xcarchive
system('rm -rf ~/Library/Developer/Xcode/Archives')
# Put the old archives back
system('mv ~/Library/Developer/Xcode/OldArchivesTemp ~/Library/Developer/Xcode/Archives')
Its a bit hacky but i don't see a better solution currently. At least it preserves the user's 'archives' folder and all their pre-existing archives.
--Important note!--
I since found out that the line of code where i find the archive and cp it to the folder i want doesn't copy the symlinks inside the archive correctly, thus breaking the code signing in the app. You'll want to replace that with a 'mv' or something that maintains symlinks. Cheers!
Here's a bit of bash that I've come up with for our Jenkins CI system. These commands should be run in a script immediately after the xcodebuild archive command finishes.
BUILD_DIR="${WORKSPACE}/build"
XCODE_SCHEME="myscheme"
# Common path and partial filename
ARCHIVE_BASEPATH="${HOME}/Library/Developer/Xcode/Archives/$(date +%Y-%m-%d)/${XCODE_SCHEME}"
# Find the latest .xcarchive for the given scheme
NEW_ARCHIVE=$(ls -td "${ARCHIVE_BASEPATH}"* | head -n 1)
# Zip it up so non-Apple systems won't treat it as a dir
pushd "${NEW_ARCHIVE%/*}"
zip -r "${BUILD_DIR}/${NEW_ARCHIVE##*/}.zip" "${NEW_ARCHIVE##*/}"
popd
# Optional, disk cleanup
rm -rf "${NEW_ARCHIVE}"
The BUILD_DIR is used to collect artifacts so that it's easy to archive them from Jenkins with a glob such as build/*.ipa,build/*.zip
Similar to the others, but perhaps a little simpler since I try to record the .xcarchive file's location. (I also don't move the archives folder, so this will work better if you're doing multiple builds at the same time.)
My caller build script generates a new tempfile and sets its path to an environment variable named XCARCHIVE_PATH_TMPFILE. This environment variable is available in my scheme's Archive post-action shell script, which then that writes the .xcarchive's path to that file. The build script that can then read that file after it calls xcodebuild archive.
post-action shell script
echo $ARCHIVE_PATH > "$XCARCHIVE_PATH_TMPFILE"
On Xcode 4.6 it is possible to specify a post-build action for the scheme to be compiled into an xcarchive:
echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $PROJECT_DIR/archive_paths.sh
A build script can be used to check if $ARCHIVE_PATH is defined after running xcodebuild and if this is the case, the output xcarchive can be moved into a designated folder.
This method is not very maintainable if the targets in the project are a large number, as for each one it is necessary to tag the corresponding scheme as 'shared' and add the post-build action.
To address this problem, I have created a build script that generates the archive path programmatically by extracting the last build that matches the target name on the current day. This method works reliably as long as there aren't multiple builds with the same target name running on the machine (this may be a problem in production environments where multiple concurrent builds are run).
#!/bin/bash
#
# Script to archive an existing xcode project to a target location.
# The script checks for a post-build action that defines the $ARCHIVE_PATH as follows:
# echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $PROJECT_DIR/archive_paths.sh
# If such post-build action does not exist or sourcing it doesn't define the $ARCHIVE_PATH
# variable, the script tries to generate it programmatically by finding the latest build
# in the expected archiving folder
#
post_build_script=archive_paths.sh
build_errors_file=build_errors.log
OUTPUT=output/
XCODEBUILD_CMD='/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild'
TARGET_SDK=iphoneos
function archive()
{
echo "Archiving target '$1'"
# Delete $post_build_script if it already exists as it should be generated by a
# post-build action
rm -f $post_build_script
# Use custom provisioning profile and code sign identity if specified, otherwise
# default to project settings
# Note: xcodebuild always returns 0 even if the build failed. We look for failure in
# the stderr output instead
if [[ ! -z "$2" ]] && [[ ! -z "$3" ]]; then
${XCODEBUILD_CMD} clean archive -scheme $1 -sdk "${TARGET_SDK}" \
"CODE_SIGN_IDENTITY=$3" "PROVISIONING_PROFILE=$2" 2>$build_errors_file
else
${XCODEBUILD_CMD} clean archive -scheme $1 -sdk "${TARGET_SDK}"
2>$build_errors_file
fi
errors=`grep -wc "The following build commands failed" $build_errors_file`
if [ "$errors" != "0" ]
then
echo "BUILD FAILED. Error Log:"
cat $build_errors_file
rm $build_errors_file
exit 1
fi
rm $build_errors_file
# Check if archive_paths.sh exists
if [ -f "$post_build_script" ]; then
source "$post_build_script"
if [ -z "$ARCHIVE_PATH" ]; then
echo "'$post_build_script' exists but ARCHIVE_PATH was not set.
Enabling auto-detection"
fi
fi
if [ -z "$ARCHIVE_PATH" ]; then
# This is the format of the xcarchive path:
# /Users/$USER/Library/Developer/Xcode/Archives/`date +%Y-%m-%d`/$1\
# `date +%d-%m-%Y\ %H.%M`.xcarchive
# In order to avoid mismatches with the hour/minute of creation of the archive and
# the current time, we list all archives with the correct target that have been
# built in the current day (this may fail if the build wraps around midnight) and
# fetch the correct file with a combination of ls and grep.
# This script can break only if there are multiple targets with exactly the same
# name running at the same time.
EXTRACTED_LINE=$(ls -lrt /Users/$USER/Library/Developer/Xcode/Archives/`date
+%Y-%m-%d`/ | grep $1\ `date +%d-%m-%Y` | tail -n 1)
if [ "$EXTRACTED_LINE" == "" ]; then
echo "Error: couldn't fetch archive path"
exit 1
fi
# ls -lrt prints lines with the following format
# drwxr-xr-x 5 mario 1306712193 170 25 Jul 17:17 ArchiveTest 25-07-2013
# 17.17.xcarchive
# We can split this line with the " " separator and take the latest bit:
# 17.17.xcarchive
FILE_NAME_SUFFIX=$(echo $EXTRACTED_LINE | awk '{split($0,a," "); print a[11]}')
if [ "$FILE_NAME_SUFFIX" == "" ]; then
echo "Error: couldn't fetch archive path"
exit 1
fi
# Finally, we can put everything together to generate the path to the xcarchive
ARCHIVE_PATH="/Users/$USER/Library/Developer/Xcode/Archives/`date
+%Y-%m-%d`/$1 `date +%d-%m-%Y` $FILE_NAME_SUFFIX/"
fi
# Create output folder if it doesn't already exist
mkdir -p "$OUTPUT"
# Move archived xcarchive build to designated output folder
mv -v "$ARCHIVE_PATH" "$OUTPUT"
}
# Check number of command line args
if [ $# -lt 1 ]; then
echo "Syntax: `basename $0` <target name> [/path/to/provisioning-profile]
[<code sign identity]"
exit 1
fi
if [ ! -z "$2" ]; then
PROVISIONING_PROFILE="$2"
fi
if [ ! -z "$3" ]; then
SIGN_PROVISIONING_PROFILE="$3"
else
if [ ! -z "$PROVISIONING_PROFILE" ]; then
SIGN_PROVISIONING_PROFILE=$(cat "$PROVISIONING_PROFILE" | egrep -a -o
'[A-Fa-f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}')
fi
fi
archive "$1" "$PROVISIONING_PROFILE" "$SIGN_PROVISIONING_PROFILE"
Full source code with an example Xcode project can be found here:
https://github.com/bizz84/Xcode-xcarchive-command

Resources