I am using SBT for building a java project and have a requirement of copying text files (that are not resources, but used by java classes to read instead).
I am inexperienced with either SBT or Scala (needed for build.scala file)
Any help would be really appreciated.
For example, if my directory structure is:
test
|- files
|- one.text
|- main
|- java
|- Test.java
I want the one.text file available as well in the target folder once I execute an sbt goal like
sbt test
Following lines in your build.sbt should do the trick:
unmanagedResourceDirectories in Test <+= (baseDirectory) {_ / "files"}
unmanagedSourceDirectories in Test <+= (baseDirectory) {_ / "main" / "java"}
You have a non standard project layout though. If you can change it to a standard "maven style":
project/src/main/java
project/src/main/resources
project/src/test/java/{Test.java, ...}
project/src/test/resources/{one.text, ...}
sbt will do resource copying automatically.
Related
I work on a Java project, whose tests I want to convert to scala. I saw that it might be more convenient to package the entire project jar with sbt, rather than with maven.
However, I currently have a single pom.xml file, that creates a jar with all dependencies inside ("fat jar") using maven shade plugin, and runs the tests. This is achieved via the "mvn package" command.
With sbt, I saw that 2-3 files are needed just for the fat jar - build.sbt, assembly.sbt, possibly plugins.sbt.
Is there some way by which I can have a single xxx.sbt file, and run one / several sbt commands, to get the same effect?
No, you need at least two files: project/plugins.sbt with the
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
line and build.sbt with the assembly settings. You can merge the *.sbt files in the root directory, sbt reads them all regardless of the name anyway. But the files in the project/ directory are different. You can read more about it in https://www.scala-sbt.org/1.x/docs/Organizing-Build.html
I use sbt 0.13.5 to build a project with a few config files meant to be human-edited after deployment.
Using sbt-native-packager's stage command builds my basic output directory structure for deployment. This results in a bin folder with a start script and all my jars in a lib folder. Great.
That just leaves the text config files that should be copied into a conf folder verbatim alonside the bin/ and lib/ folders (not included in any jar).
How can I copy these configuration files to the output directory on the filesystem (not into any jars) using sbt?
I'm unsure about an out-of-the-box support for src/main/conf directory or similar, and unless you find it, use the following as a workaround:
packageArchetype.java_application
mappings in Universal ++= {
((sourceDirectory in Compile).value / "conf" * "*").get.map { f =>
f -> s"conf/${f.name}"
}
}
It maps files under src/main/conf to conf directory in the package.
NB: I'm pretty sure I've seen somewhere in the code support for the conf directory.
Default Universal convention is to copy files/directories under src/univeral/. So, to include a conf/ directory in the distribution, just add src/universal/conf and copy configuration files there. Credit: this link
This seems to be a 'better' answer, as I am using sbt together with sbt-native-packager plugin:
mappings in Universal ++= contentOf( baseDirectory.value / "conf" )
This takes all the files under the specified "conf" folder and dumps them in the package root folder.
Which uses the MappingsHelper I stumbled upon in the very terse documentation for it in the sbt-native-packager.
I've just stumbled upon the sbt-pack plugin. The development stream seems steady. It's surprising to me as I believed that the only plugin for (quoting sbt-pack's headline) "creating distributable Scala packages." is sbt-assembly (among the other features).
What are the key differences between the plugins? When should I use one over the other?
(Disclaimer: I maintain sbt-assembly)
sbt-assembly
sbt-assembly creates a fat JAR - a single JAR file containing all class files from your code and libraries. By evolution, it also contains ways of resolving conflicts when multiple JARs provide the same file path (like config or README file). It involves unzipping of all library JARs, so it's a bit slow, but these are heavily cached.
sbt-pack
sbt-pack keeps all the library JARs intact, moves them into target/pack directory (as opposed to ivy cache where they would normally live), and makes a shell script for you to run them.
sbt-native-packager
sbt-native-packager is similar to sbt-pack but it was started by a sbt committer Josh Suereth, and now maintained by highly capable Nepomuk Seiler (also known as muuki88). The plugin supports a number of formats like Windows msi file and Debian deb file. The recent addition is a support for Docker images.
All are viable means of creating deployment images. In certain cases like deploying your application to a web framework etc., it might make things easier if you're dealing with one file as opposed to a dozen.
Honorable mention: sbt-progard and sbt-onejar.
Although Eugene Yokota's explanation is complete, I would like to explain the mentioned plugins with package command in the aspect of usages and how different results are generated.
Directory settings and build.sbt
lazy val commonSettings = Seq(
organization := "stackOverFlow",
scalaVersion := "2.11.12",
version := "1.0",
)
lazy val app = (project in file ("app")).
enablePlugins(PackPlugin).
settings(commonSettings)
Above build.sbt file declares project called app and includes all the source files in the app directory. To enable Pack plugins, enablePlugins(PackPlugin) should be included in the sbt file.
Also, I've put the below line in project/plugins.sbt file to use pack plugins in our project
addSbtPlugin("org.xerial.sbt" % "sbt-pack" % "0.9.3")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
The package is already integrated into the sbt by default, so you don't have to explicitly specify the plugins using addSbtPlugins. However, the sbt-pack and sbt-assembly plugins are not included in the sbt by default, so you have to specify that you want to use them. addSbtPlugin is a way to say that "I want to use xxx, yyy plugins in my project" to your sbt.
Also, I implemented two contrived scala files in the ./app/src/main/scala:
AppBar.scala
class AppBar {
def printDescription() = println(AppBar.getDescription)
}
object AppBar {
private val getDescription: String = "Hello World, I am AppBar"
def main (args: Array[String]): Unit = {
val appBar = new AppBar
appBar.printDescription()
}
}
AppFoo.scala
class AppFoo {
def printDescription() = println(AppFoo.getDescription)
}
object AppFoo {
private val getDescription: String = "Hello World, I am AppFoo"
def main (args: Array[String]): Unit = {
val appFoo = new AppFoo
appFoo.printDescription()
}
}
sbt package
This is very basic sbt command included in the sbt to help you distribute your project through the jar file. The jar file generated by the package command is located in the projectDirectoy/target/scala-2.11/app_2.11-1.0.jar (Here, the specified scalaVersion and version setting keys included in the build.sbt file are used to generate the jar file name).
When you look inside the jar, you can see the class files generated by the sbt tool, which is the result of compiling the sources in the app/src/main/scala. Also, it includes a MANIFEST file.
$vi projectDirectoy/target/scala-2.11/app_2.11-1.0.jar
META-INF/MANIFEST.MF
AppBar$.class
AppBar.class
AppFoo.class
AppFoo$.class
Note that it only includes the class files generated from the scala files located in the app/src/main/scala directory. The jar file generated by the package command does not include any scala related libraries such as collection in the scala library (e.g., collection.mutable.Map.class). Therefore, to execute the program you may require scala library because the generate jar file only contains the minimal classes generated from the scala sources that I implemented. That is the reason why the jar file contains AppBar.class, AppBar$.class for companion object, etc.
sbt-assembly
As mentioned by the Eugene Yokota, sbt-assembly also help you distribute your project through generating the jar file; however the generated jar file includes not only the class files generated by your source code, but also all the libraries that you need to execute the program. For example, to execute the main function defined in the AppFoo object, you may need scala libraries. Also, when you add external libraries in your project, which can be included by adding the dependencies to the libraryDependencies key.
libraryDependencies ++= Seq("org.json4s" %% "json4s-jackson" % "3.5.3")
For example, you can include json4s libraries in your project, and jar files related to supporting json4s in your project also will be added to the final jar file generated by the sbt-assembly. In other words, when you invoke assembly in your sbt, it generates one jar file containing all the requirements to execute your program, so that you don't need another dependency to execute yout program.
When you prompt assembly command in your sbt shell, then it will generate one jar file in your target directory. In this case, you may find the app-assembly-1.0.jar in the app/target/scala-2.11 directory. When you look inside the jar file, you can find that it contains lots of classes.
$vi projectDirectoy/target/scala-2.11/app_2.11-1.0.jar
ETA-INF/MANIFEST.MF
scala/
scala/annotation/
scala/annotation/meta/
scala/annotation/unchecked/
scala/beans/
scala/collection/
scala/collection/concurrent/
scala/collection/convert/
scala/collection/generic/
scala/collection/immutable/
scala/collection/mutable/
scala/collection/parallel/
scala/collection/parallel/immutable/
scala/collection/parallel/mutable/
scala/collection/script/
scala/compat/
scala/concurrent/
scala/concurrent/duration/
scala/concurrent/forkjoin/
scala/concurrent/impl/
scala/concurrent/util/
scala/io/
scala/math/
scala/ref/
scala/reflect/
scala/reflect/macros/
scala/reflect/macros/internal/
scala/runtime/
scala/sys/
scala/sys/process/
scala/text/
scala/util/
scala/util/control/
scala/util/hashing/
scala/util/matching/
AppBar$.class
AppBar.class
AppFoo$.class
AppFoo.class
......
As mentioned before, because the jar file generated by the assembly contains all the dependencies such as scala and external libraries to execute your program in the jar, you may think that you can invoke the main functions defined in the AppFoo object and AppBar object.
jaehyuk#ubuntu:~/work/sbt/app/target/scala-2.11$ java -cp './*' AppFoo
Hello World, I am AppFoo
jaehyuk#ubuntu:~/work/sbt/app/target/scala-2.11$ java -cp './*' AppBar
Hello World, I am AppBar
Yeah~ you can execute the main function using the generated jar file.
sbt-pack
sbt-pack is almost same as the sbt-assembly; it saves all the library on which your project depends as jar files required to execute your program. However, sbt-pack doesn't integrate all the dependencies into one jar files, instead, it generates multiple jar files which correspond to one library dependencies and your classes (e.g., AppFoo.class).
Also, interestingly it automatically generates scripts for invoking all the main functions defined in your scala source files and Makefiles to install the program. Let's take a look at the pack directory created after you prompt pack command on your sbt shell.
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls
bin lib Makefile VERSION
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls bin/
app-bar app-bar.bat app-foo app-foo.bat
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls lib/
app_2.11-1.0.jar sbt_2.12-0.1.0-SNAPSHOT.jar scala-library-2.11.12.jar
jaehyuk#ubuntu:~/work/sbt/app/target/pack$
As shown in the above, two directories and two files are created; bin contains all the script files to execute the functions defined in your sources (each file is a script that helps you execute the main method defined in your scala files); lib contains all the required jar files to execute your program; and lastly Makefile can be used to install your program and dependent libraries in your system.
For the details, please refer the github pages for each plugins.
I'm writing a library which depends on code (let's call it foo.jar) which is only available as a binary jar. As is standard, I'm putting this in the lib/ directory so SBT will treat is as an unmanaged dependency. This is fine so far.
However, since this is a library, I'd like to be able to publish it so that other projects which depend on it to also have access to the unmanaged code in foo.jar without having to manually locate it. I originally thought I could use a fat jar plugin such as SBT Assembly to create a jar with the dependencies, but that doesn't affect what is actually published using sbt publish-local – it only creates a fat jar when you run sbt assembly. Is there some standard simple way to handle this? It seems like a bad idea for every library which uses unmanaged dependencies to break when used by other projects downstream so I wonder if I'm missing something obvious.
I don't know if that's a good use of sbt-assembly, since other libraries could depend on a different version of foo.jar etc.
One way to work around it is to publish foo.jar in a Maven repository yourself. Some people in Scala and/or sbt community have been talking about bintray. It's still in beta, but looks promising if you want some jars published.
You might be able to get the result you want by manipulating the mappings in (Compile, packageBin) to include the files you want your packaged jar to have (publish uses the output from packageBin). This technique will allow you to include absolutely any file you want within the jar. The official sbt doc is here: http://www.scala-sbt.org/0.12.3/docs/Howto/package.html#contents
As an example, consider the common case of including a .properties file within your jar. Lets say you need to include "messages.properties" under the path "com/bigco/messages.properties" in your packaged jar. And lets say that this file is under src/main/scala/ ... You can add the following to your build.sbt:
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / "src" / "main" / "scala" / "com" / "bigco" / "messages.properties") -> "com/bigco/messages.properties"
}
To attempt to answer your original question, you could unzip foo.jar and add each one of the class files within to the packaged jar, according to their correct package paths. So something similar to
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / path / to / unzipped / file.class) -> "path.to.unzipped.file.class"
...
}
Or you might be able to get away with simply including foo.jar at the root of the packaged jar like so:
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / "lib" / "foo.jar") -> "foo.jar"
}
I have tried to install SBT on my macbook. When I run it, it doesn't ask me for any project definitions (e.g. title) and simply says
[info] Set current project to default (in build file:/Users/qui/Documents/Programming/test2/)
It then goes to what looks like the sbt interpreter.
When I look inside "test2", there is a project and target directory but I dont see a src directory to work with
Clearly I have gone wrong somewhere in my installation but I'm unsure where. Any ideas?
Update
So I just installed 0.10 on a fresh fedora install. And I am getting the exact same problem, same "info" message and it has only created a project and target directory
I must be doing something idiotic right? What am I doing wrong? :p
I work with SBT 0.13 so...your mileage may vary.
sbt's default behaviour
What you experience is the default behaviour of sbt. The tool expects that the project files are already in place or when there is no project files it doesn't bother to create them - the default values are just applied to the current directory that effectively becomes the project directory for a project called by the name of the directory it's in. SBT then opens sbt shell.
jacek:~/sandbox/stackoverflow/testaaa
$ tree
.
0 directories, 0 files
jacek:~/sandbox/stackoverflow/testaaa
$ sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Updating {file:/Users/jacek/.sbt/0.13/plugins/}global-plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to testaaa (in build file:/Users/jacek/sandbox/stackoverflow/testaaa/)
[testaaa]>
Quoting Running from the official documentation of SBT.
Running sbt with no command line arguments starts it in interactive
mode. Interactive mode has a command prompt (with tab completion and
history!).
Example
In your case, when you started sbt in /Users/qui/Documents/Programming/test2/ it silently assumed it's the project directory and applied the default settings.
The following sbt session is in test2 directory, too. I use help to display the help of a setting key and then use the key to display its value.
jacek:~/sandbox/stackoverflow/test2
$ tree
.
0 directories, 0 files
jacek:~/sandbox/stackoverflow/test2
$ sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Set current project to test2 (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> help name
Project name.
[test2]> name
[info] test2
[test2]> help organization
Organization/group ID.
[test2]> organization
[info] default
[test2]> help version
The version/revision of the current module.
[test2]> version
[info] 0.1-SNAPSHOT
[test2]> help scalaVersion
The version of Scala used for building.
[test2]> scalaVersion
[info] 2.10.2
(I've changed the prompt so the name of the project, i.e. the name of the directory sbt has been started in, is displayed before the >).
You can change the value of a key with the set command that Evaluates a Setting and applies it to the current project.
[test2]> help set
set [every] <setting-expression>
Applies the given setting to the current project:
1) Constructs the expression provided as an argument by compiling and loading it.
2) Appends the new setting to the current project's settings.
3) Re-evaluates the build's settings.
This command does not rebuild the build definitions, plugins, or configurations.
It does not automatically persist the setting(s) either.
To persist the setting(s), run 'session save' or 'session save-all'.
If 'every' is specified, the setting is evaluated in the current context
and the resulting value is used in every scope. This overrides the value
bound to the key everywhere.
[test2]> set scalaVersion := "2.10.3"
[info] Defining *:scalaVersion
[info] The new value will be used by *:allDependencies, *:dependencyUpdatesData and 11 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to test2 (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> scalaVersion
[info] 2.10.3
In the other question on StackOverflow #regis-jean-gilles has showed how to set the other settings using the set command.
[test2]> set name := "My test2 sbt project"
[info] Defining *:name
[info] The new value will be used by *:description, *:normalizedName and 8 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to My test2 sbt project (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> set version := "1.0"
[info] Defining *:version
[info] The new value will be used by *:isSnapshot, *:projectId and 5 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to My test2 sbt project (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> set scalaVersion := "2.10.3"
[info] Defining *:scalaVersion
[info] The new value will be used by *:allDependencies, *:dependencyUpdatesData and 11 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to My test2 sbt project (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> session save
[info] Reapplying settings...
[info] Set current project to My test2 sbt project (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
[test2]> exit
The build.sbt file will then contain all the settings as if there'd been set there in the first place.
$ cat build.sbt
name := "My test2 sbt project"
version := "1.0"
scalaVersion := "2.10.3"
By default, sbt creates various files in target directory. When you look inside the target directory, there are no files - just an empty directory. The same applies to project that also may or may not hold target directory. They're assumed to be available and if there's not, they're created by default.
When you change a setting in sbt's interactive shell (with set), you can save the session with session save.
[test2]> help session
session <command>
Manipulates session settings, which are temporary settings that do not persist past the current sbt execution (that is, the current session).
Valid commands are:
clear, clear-all
Removes temporary settings added using 'set' and re-evaluates all settings.
For 'clear', only the settings defined for the current project are cleared.
For 'clear-all', all settings in all projects are cleared.
list, list-all
Prints a numbered list of session settings defined.
The numbers may be used to remove individual settings or ranges of settings using 'remove'.
For 'list', only the settings for the current project are printed.
For 'list-all', all settings in all projets are printed.
remove <range-spec>
<range-spec> is a comma-separated list of individual numbers or ranges of numbers.
For example, 'remove 1,3,5-7'.
The temporary settings at the given indices for the current project are removed and all settings are re-evaluated.
Use the 'list' command to see a numbered list of settings for the current project.
save, save-all
Makes the session settings permanent by writing them to a '.sbt' configuration file.
For 'save', only the current project's settings are saved (the settings for other projects are left alone).
For 'save-all', the session settings are saved for all projects.
The session settings defined for a project are appended to the first '.sbt' configuration file in that project.
If no '.sbt' configuration file exists, the settings are written to 'build.sbt' in the project's base directory.
[test2]> session save
[info] Reapplying settings...
[info] Set current project to test2 (in build file:/Users/jacek/sandbox/stackoverflow/test2/)
Once you do that, a build.sbt with your own setting(s) is saved. That might be a good starting point for further configuration of a project.
jacek:~/sandbox/stackoverflow/test2
$ cat build.sbt
scalaVersion := "2.10.3"
Typesafe Activator
According to the home page of Typesafe Activator:
Typesafe Activator is a browser-based or command-line tool that helps
developers get started with the Typesafe Reactive Platform.
Under the covers, Activator is a UI built atop of sbt as demo'ed by Josh Suereth in the screencast Introducing sbt 0.13.2.
It appears that that's the only blessed solution for setting up sbt projects out of the many templates available in Activator.
giter8 - sbt project (layout) templates
If you however need some help to lay out the directory structure and have a ready-to-use project setup, you may want to use giter8 that's a command line tool to apply templates defined on github
Say, you want to create a project with scalaz dependency. You may want to use adinapoli/scalaz-revolver (see the list of available templates).
jacek:~/sandbox/stackoverflow
$ g8 adinapoli/scalaz-revolver
Simple scala project with sbt-revolver
organization [org.example]: pl.japila
name [Scala sbt-revolver project]:
scala_version [2.9.2]: 2.10.3
version [0.1-SNAPSHOT]:
Template applied in ./scala-sbt-revolver-project
jacek:~/sandbox/stackoverflow
$ cd scala-sbt-revolver-project/
jacek:~/sandbox/stackoverflow/scala-sbt-revolver-project
$ tree
.
├── README
├── build.sbt
├── project
│ ├── Build.scala
│ ├── build.properties
│ └── plugins.sbt
└── src
└── main
└── scala
└── pl
└── japila
└── ScalaSbtrevolverProject.scala
6 directories, 6 files
See Create a project directory with source code to find out more.
np - new sbt project generation made simple(r)
As pointed out in the comments by #0__ below, there's another project that aims at simplifying how new projects in sbt are created - np. That seems exactly what you needed.
In https://github.com/softprops/np#for-sbt-013 there's a complete description of what's needed to set it up and create new sbt projects using the utility that boils down to:
Registering the sbt plugin. Add the following to ~/.sbt/0.13/plugins/np.sbt.
addSbtPlugin("me.lessis" % "np" % "0.2.0")
Define a custom global overrides in ~/.sbt/0.13/np.sbt. Add the following to the file.
seq(npSettings:_*)
(NpKeys.defaults in (Compile, NpKeys.np)) ~= {
_.copy(org="me.lessis", version="0.1.0-SNAPSHOT")
}
Use the np plugin's command - np. Create an empty directory for the sbt project and run sbt np.
jacek:~/sandbox/stackoverflow
$ mkdir np-sandbox/
jacek:~/sandbox/stackoverflow
$ cd np-sandbox/
jacek:~/sandbox/stackoverflow/np-sandbox
$ sbt np
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Set current project to np-sandbox (in build file:/Users/jacek/sandbox/stackoverflow/np-sandbox/)
[info] Generated build file
[info] Generated source directories
[success] Total time: 0 s, completed Dec 7, 2013 12:51:42 PM
jacek:~/sandbox/stackoverflow/np-sandbox
$ tree
.
├── build.sbt
├── src
│ ├── main
│ │ ├── resources
│ │ └── scala
│ └── test
│ ├── resources
│ └── scala
└── target
└── streams
└── compile
└── np
└── $global
└── out
12 directories, 2 files
jacek:~/sandbox/stackoverflow/np-sandbox
$ cat build.sbt
organization := "me.lessis"
name := "default"
version := "0.1.0-SNAPSHOT"
No you're not doing something wrong, previous versions of sbt (0.7.x) did ask you if you wanted to create your project.
sbt version 0.10.x is a complete rewrite and does not act the same way (i.e. ask you to create a project on startup).
The old project was on googlecode but has since moved to github, you can find the documentation for 0.10.x at https://github.com/harrah/xsbt/wiki, in particular https://github.com/harrah/xsbt/wiki/Settings if you come from a 0.7.x background.
It's a bit hard to wrap your head around the new settings system at first, but trust me when I say you'll love it :)
As described in the np plugin readme, the required steps would be :
mkdir -p src/{main,test}/scala
touch build.sbt && vi build.sbt # fill in the basics (name, organization, version)
touch README.md && vi README.md
sbt
... start coding
One-liner to create the directory-structure and files (all empty):
mkdir -p ./src/{main,test}/{scala,java,resources}; mkdir project; touch build.sbt; touch project/build.properties
I found and updated a script by Alvin Alexander (author of the amazing book Scala Cookbook) that does just what you want.
Once again, I provide a bash script because the task of creating the directories and the files is pedantic, but simple. Making a full fledged plugin for SBT for this is an overkill.
This script creates an SBT project directory beneath the current directory.
Directory/Project Name (MyFirstProject): ProjectX
Create .gitignore File? (Y/n):
Create README.md File? (Y/n):
-----------------------------------------------
Directory/Project Name: ProjectX
Create .gitignore File?: y
Create README.md File?: y
-----------------------------------------------
Create Project? (Y/n):
Project created. See the following URL for build.sbt examples:
http://alvinalexander.com/scala/sbt-syntax-examples
$ cat build.sbt
name := "ProjectX"
version := "1.0"
scalaVersion := "2.11.7"
$
And finally, I believe SBT launched without any arguments should behave exactly like this script.