I work on a Java project, whose tests I want to convert to scala. I saw that it might be more convenient to package the entire project jar with sbt, rather than with maven.
However, I currently have a single pom.xml file, that creates a jar with all dependencies inside ("fat jar") using maven shade plugin, and runs the tests. This is achieved via the "mvn package" command.
With sbt, I saw that 2-3 files are needed just for the fat jar - build.sbt, assembly.sbt, possibly plugins.sbt.
Is there some way by which I can have a single xxx.sbt file, and run one / several sbt commands, to get the same effect?
No, you need at least two files: project/plugins.sbt with the
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.9")
line and build.sbt with the assembly settings. You can merge the *.sbt files in the root directory, sbt reads them all regardless of the name anyway. But the files in the project/ directory are different. You can read more about it in https://www.scala-sbt.org/1.x/docs/Organizing-Build.html
Related
I've just stumbled upon the sbt-pack plugin. The development stream seems steady. It's surprising to me as I believed that the only plugin for (quoting sbt-pack's headline) "creating distributable Scala packages." is sbt-assembly (among the other features).
What are the key differences between the plugins? When should I use one over the other?
(Disclaimer: I maintain sbt-assembly)
sbt-assembly
sbt-assembly creates a fat JAR - a single JAR file containing all class files from your code and libraries. By evolution, it also contains ways of resolving conflicts when multiple JARs provide the same file path (like config or README file). It involves unzipping of all library JARs, so it's a bit slow, but these are heavily cached.
sbt-pack
sbt-pack keeps all the library JARs intact, moves them into target/pack directory (as opposed to ivy cache where they would normally live), and makes a shell script for you to run them.
sbt-native-packager
sbt-native-packager is similar to sbt-pack but it was started by a sbt committer Josh Suereth, and now maintained by highly capable Nepomuk Seiler (also known as muuki88). The plugin supports a number of formats like Windows msi file and Debian deb file. The recent addition is a support for Docker images.
All are viable means of creating deployment images. In certain cases like deploying your application to a web framework etc., it might make things easier if you're dealing with one file as opposed to a dozen.
Honorable mention: sbt-progard and sbt-onejar.
Although Eugene Yokota's explanation is complete, I would like to explain the mentioned plugins with package command in the aspect of usages and how different results are generated.
Directory settings and build.sbt
lazy val commonSettings = Seq(
organization := "stackOverFlow",
scalaVersion := "2.11.12",
version := "1.0",
)
lazy val app = (project in file ("app")).
enablePlugins(PackPlugin).
settings(commonSettings)
Above build.sbt file declares project called app and includes all the source files in the app directory. To enable Pack plugins, enablePlugins(PackPlugin) should be included in the sbt file.
Also, I've put the below line in project/plugins.sbt file to use pack plugins in our project
addSbtPlugin("org.xerial.sbt" % "sbt-pack" % "0.9.3")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
The package is already integrated into the sbt by default, so you don't have to explicitly specify the plugins using addSbtPlugins. However, the sbt-pack and sbt-assembly plugins are not included in the sbt by default, so you have to specify that you want to use them. addSbtPlugin is a way to say that "I want to use xxx, yyy plugins in my project" to your sbt.
Also, I implemented two contrived scala files in the ./app/src/main/scala:
AppBar.scala
class AppBar {
def printDescription() = println(AppBar.getDescription)
}
object AppBar {
private val getDescription: String = "Hello World, I am AppBar"
def main (args: Array[String]): Unit = {
val appBar = new AppBar
appBar.printDescription()
}
}
AppFoo.scala
class AppFoo {
def printDescription() = println(AppFoo.getDescription)
}
object AppFoo {
private val getDescription: String = "Hello World, I am AppFoo"
def main (args: Array[String]): Unit = {
val appFoo = new AppFoo
appFoo.printDescription()
}
}
sbt package
This is very basic sbt command included in the sbt to help you distribute your project through the jar file. The jar file generated by the package command is located in the projectDirectoy/target/scala-2.11/app_2.11-1.0.jar (Here, the specified scalaVersion and version setting keys included in the build.sbt file are used to generate the jar file name).
When you look inside the jar, you can see the class files generated by the sbt tool, which is the result of compiling the sources in the app/src/main/scala. Also, it includes a MANIFEST file.
$vi projectDirectoy/target/scala-2.11/app_2.11-1.0.jar
META-INF/MANIFEST.MF
AppBar$.class
AppBar.class
AppFoo.class
AppFoo$.class
Note that it only includes the class files generated from the scala files located in the app/src/main/scala directory. The jar file generated by the package command does not include any scala related libraries such as collection in the scala library (e.g., collection.mutable.Map.class). Therefore, to execute the program you may require scala library because the generate jar file only contains the minimal classes generated from the scala sources that I implemented. That is the reason why the jar file contains AppBar.class, AppBar$.class for companion object, etc.
sbt-assembly
As mentioned by the Eugene Yokota, sbt-assembly also help you distribute your project through generating the jar file; however the generated jar file includes not only the class files generated by your source code, but also all the libraries that you need to execute the program. For example, to execute the main function defined in the AppFoo object, you may need scala libraries. Also, when you add external libraries in your project, which can be included by adding the dependencies to the libraryDependencies key.
libraryDependencies ++= Seq("org.json4s" %% "json4s-jackson" % "3.5.3")
For example, you can include json4s libraries in your project, and jar files related to supporting json4s in your project also will be added to the final jar file generated by the sbt-assembly. In other words, when you invoke assembly in your sbt, it generates one jar file containing all the requirements to execute your program, so that you don't need another dependency to execute yout program.
When you prompt assembly command in your sbt shell, then it will generate one jar file in your target directory. In this case, you may find the app-assembly-1.0.jar in the app/target/scala-2.11 directory. When you look inside the jar file, you can find that it contains lots of classes.
$vi projectDirectoy/target/scala-2.11/app_2.11-1.0.jar
ETA-INF/MANIFEST.MF
scala/
scala/annotation/
scala/annotation/meta/
scala/annotation/unchecked/
scala/beans/
scala/collection/
scala/collection/concurrent/
scala/collection/convert/
scala/collection/generic/
scala/collection/immutable/
scala/collection/mutable/
scala/collection/parallel/
scala/collection/parallel/immutable/
scala/collection/parallel/mutable/
scala/collection/script/
scala/compat/
scala/concurrent/
scala/concurrent/duration/
scala/concurrent/forkjoin/
scala/concurrent/impl/
scala/concurrent/util/
scala/io/
scala/math/
scala/ref/
scala/reflect/
scala/reflect/macros/
scala/reflect/macros/internal/
scala/runtime/
scala/sys/
scala/sys/process/
scala/text/
scala/util/
scala/util/control/
scala/util/hashing/
scala/util/matching/
AppBar$.class
AppBar.class
AppFoo$.class
AppFoo.class
......
As mentioned before, because the jar file generated by the assembly contains all the dependencies such as scala and external libraries to execute your program in the jar, you may think that you can invoke the main functions defined in the AppFoo object and AppBar object.
jaehyuk#ubuntu:~/work/sbt/app/target/scala-2.11$ java -cp './*' AppFoo
Hello World, I am AppFoo
jaehyuk#ubuntu:~/work/sbt/app/target/scala-2.11$ java -cp './*' AppBar
Hello World, I am AppBar
Yeah~ you can execute the main function using the generated jar file.
sbt-pack
sbt-pack is almost same as the sbt-assembly; it saves all the library on which your project depends as jar files required to execute your program. However, sbt-pack doesn't integrate all the dependencies into one jar files, instead, it generates multiple jar files which correspond to one library dependencies and your classes (e.g., AppFoo.class).
Also, interestingly it automatically generates scripts for invoking all the main functions defined in your scala source files and Makefiles to install the program. Let's take a look at the pack directory created after you prompt pack command on your sbt shell.
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls
bin lib Makefile VERSION
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls bin/
app-bar app-bar.bat app-foo app-foo.bat
jaehyuk#ubuntu:~/work/sbt/app/target/pack$ ls lib/
app_2.11-1.0.jar sbt_2.12-0.1.0-SNAPSHOT.jar scala-library-2.11.12.jar
jaehyuk#ubuntu:~/work/sbt/app/target/pack$
As shown in the above, two directories and two files are created; bin contains all the script files to execute the functions defined in your sources (each file is a script that helps you execute the main method defined in your scala files); lib contains all the required jar files to execute your program; and lastly Makefile can be used to install your program and dependent libraries in your system.
For the details, please refer the github pages for each plugins.
I was a heavy Maven user and now I'm gradually using SBT for some of my projects.
I'd like to know how could I use SBT to create a standalone Java project? This project should be packaged as a JAR file and this JAR file would be used as a dependency in another SBT project.
In Maven, I could tell in my pom.xml what type of artifact it should produce when I build it. Is there something similar that I can do in SBT?
There is a difference between standalone and making a project useable as a dependency or another project. In the first case, you would use a plugin such as sbt-assembly. What it will do is create one jar file containing the project class files along with all of its dependencies. If you write an application, what you get is a double-clickable jar that you can execute from anywhere.
If you want to use your project A as a dependency for another project B, you have different options. You could just package the class files of A, using sbt package (answer of #Channing Walton). Then you could drop the resulting .jar file in the lib directory of project B. However, if A also requires libraries, you must make sure that they also end up in project B's libraries.
A better approach is to publish your project. You can do that purely on your local machine, using sbt publish-local. That will store the jar as produced by package in a special local directory which can be accessed from sbt in another project, along with a POM file that contains the dependencies of A. It will use a group-ID (organization) and artifact-ID (name) and a version of your project A. For example, in build.sbt:
name := "projecta"
version := "0.1.0-SNAPSHOT"
organization := "com.github.myname"
scalaVersion := "2.10.3"
publishMavenStyle := true
After publishing with sbt publish-local, you can add the following dependency to your project B:
libraryDependencies += "com.github.myname" %% "projecta" % "0.1.0-SNAPSHOT"
If you have a pure Java project, you can omit the Scala version suffix, i.e. in Project A:
crossPaths := false
autoScalaLibrary := false
And then in Project B:
libraryDependencies += "com.github.myname" % "projecta" % "0.1.0-SNAPSHOT"
(using only one % character between group and artifact ID).
More on publishing in the sbt documentation.
'sbt package' will produce a jar file.
If you want it to be executable you need to add the following to your .sbt config:
mainClass in Compile := Some("your.main.Class")
Sure, you can use 'sbt package' command, it creates a jar file but this jar will be without any dependencies. To run it necessary to specify 'classpath' arg to the libraries.
In your case you wish a standalone runnable file. And you need to add the dependencies.
To do this you can use 'assembly' plugin for SBT, see https://github.com/sbt/sbt-assembly/
Afterward you can just run 'sbt assembly' command, it provides a fat jar file with all dependencies that you can deploy and run anywhere and at any time.
For details see this article
publishLocal
builds the artifact and publish in the local Ivy repository making it available for your local project dependencies.
publishM2
same as above, but the artifact is published in local Maven repo instead of Ivy repo.
I think the easiest way to produce a stand-alone jar with your project in it,
is sadly not lying inside sbt.
I personally use my IDE: Intellij to make the jar (through the 'build artifact' feature).
Thanks to Intellij I can easily choose which library I want to include in the jar or not, (for instance the scala stl).
IMHO, this is by far the simplest method to get an executable jar for your project.
If you put the scala stl you can run your jar with the "java -jar" command, if you don't you have to run it somewhere with the correct version of scala installed with "scala".
I'm writing a library which depends on code (let's call it foo.jar) which is only available as a binary jar. As is standard, I'm putting this in the lib/ directory so SBT will treat is as an unmanaged dependency. This is fine so far.
However, since this is a library, I'd like to be able to publish it so that other projects which depend on it to also have access to the unmanaged code in foo.jar without having to manually locate it. I originally thought I could use a fat jar plugin such as SBT Assembly to create a jar with the dependencies, but that doesn't affect what is actually published using sbt publish-local – it only creates a fat jar when you run sbt assembly. Is there some standard simple way to handle this? It seems like a bad idea for every library which uses unmanaged dependencies to break when used by other projects downstream so I wonder if I'm missing something obvious.
I don't know if that's a good use of sbt-assembly, since other libraries could depend on a different version of foo.jar etc.
One way to work around it is to publish foo.jar in a Maven repository yourself. Some people in Scala and/or sbt community have been talking about bintray. It's still in beta, but looks promising if you want some jars published.
You might be able to get the result you want by manipulating the mappings in (Compile, packageBin) to include the files you want your packaged jar to have (publish uses the output from packageBin). This technique will allow you to include absolutely any file you want within the jar. The official sbt doc is here: http://www.scala-sbt.org/0.12.3/docs/Howto/package.html#contents
As an example, consider the common case of including a .properties file within your jar. Lets say you need to include "messages.properties" under the path "com/bigco/messages.properties" in your packaged jar. And lets say that this file is under src/main/scala/ ... You can add the following to your build.sbt:
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / "src" / "main" / "scala" / "com" / "bigco" / "messages.properties") -> "com/bigco/messages.properties"
}
To attempt to answer your original question, you could unzip foo.jar and add each one of the class files within to the packaged jar, according to their correct package paths. So something similar to
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / path / to / unzipped / file.class) -> "path.to.unzipped.file.class"
...
}
Or you might be able to get away with simply including foo.jar at the root of the packaged jar like so:
mappings in (Compile, packageBin) <+= baseDirectory map { base =>
(base / "lib" / "foo.jar") -> "foo.jar"
}
I created simple Java Servlet: WelcomeServlet.java.
Than, I tried compile this file via:
javac WelcomeServlet.java
In result I see compile error:
package javax.servlet doesn't exit
I try find solution for this error with Google. And I find first part of answer: java compiler doesnt see servlet-api.jar file.
I know, that Apache Tomcat in it lib folder contains servlet-api.jar file.
So, I have this file, but where I must copy this file??
I try different folders:
echo %JAVA_HOME%
C:\Program Files\Java\jdk1.6.0_26
%PATH% contains this line: C:\Program Files\Java\jdk1.6.0_26\bin
So, I copy in:
%JAVA_HOME%\bin
%JAVA_HOME%\lib
%JAVA_HOME%\jre\lib
And in result same error.
And only after I copy servlet-api.jar in directory:
%JAVA_HOME%\jre\lib\ext
compilation complite sucessful.
My question: Why? Why I must copy in folder %JAVA_HOME%\jre\lib\ext ??
Where This moment describe in documentation?
And other question we have some official docs or specifications that describe folder structure for jdk folder??
You'll need to specify the directory or directories you want the compiler to search by using the -classpath command line option when running javac. The reason the compiler found your .jar in %JAVA_HOME%\jre\lib\ext is because it searches the extension directories by default.
This is for Java 1.5, but I believe it is more or less still correct:
http://docs.oracle.com/javase/1.5.0/docs/tooldocs/findingclasses.html
The link Shaun provides is a more complete answer. But in short, using the classpath is the best way to introduce 3rd party or external (to the JDK/JRE) libraries. The classpath is a concept much like the %PATH% or the $PATH variables, but specifies locations for java to use for lookup rather than the shell to use for lookup of executables.
The classpath provides the java compiler or java virtual machine a list of items to use when searching for resources. This "path" may include directories or files. It will typically include jar files and sometimes locations of configuration files. Many Java based lookup schemes for files configuration or otherwise use some variant of what is accomplished by [Class#getResourceAsStream()][1]'s use of walking the Classpath.
I have rarely seen an incident where putting a jar file in the lib/ext location was preferred to utilizing the Classpath.
The classpath is typically an environment variable (%CLASSPATH% or $CLASSPATH) or specified on the command line when running java or javac (e.g. -cp or -classpath see the help from the executable you are running).
Build tools such as Ant and Maven will also provide abstractions to defining the list of jars to be utilized by your applications and are highly recommended to be used for any length of repetitive change code, build, test, run cycles.
I have a bunch of JAR files (from a maven2 project) and maven reports some package could not be found (org.openanzo.client.jena to be exact). I want to dig into the JAR files downloaded as the result of maven dependency resolution and find what packages are thus available from these JAR files. Insights?
UPDATE: Apparently, the only good solution to inspect insides of a jar file is the "jar" utility or one can use the facilities of their IDE to do so.
jar tvf filename.jar will show you the contents of a jar file without requiring you to extract it.
But I think that maybe what you are really trying to do is find the right coordinates for the dependency that you are missing, since obviously none of the ones you have right now are supplying the package you are looking for (in other words, checking their contents is not likely to help you).
I confess that the first place I would suggest to check is Sonatype's public Nexus instance. A search for your example turns up nothing, though. Usually that means the project is not trying to get their stuff into Maven Central or other major repositories (which is okay), so you have to resort to a web search. Usually the first two sections of the package tell you where to look (openanzo.org in your case).
If you are on Linux or a Mac, you could go to the terminal at the root of the folder containing your JARs and type:
# grep -ri "org.openanzo.client.jena" *
It will return a recursive list of all JAR files that contain that package name. If it returns 0 results, then none of those JARS contain that package.
If you wanted to do a more exhaustive search, you could unJAR the JAR files. The directory structure and .class files will be organized by packages in folders.
# jar xvf filename.jar
If you are on Windows, you can unJAR a JAR file using a tool such as 7Zip.
#Carsten
you do not have to rename a .jar file to .zip. You can directly open the jar file in winzip/or other zip utility (assuming windows OS)
#ashy_32bit
try using "jar class finder" eclipse plugin from IBM. Simple plugin for finding classes (if you know the class name)
OR
as carsten suggested... set the jar files as lib files and manually look it up
OR
create a batch file called a.bat (where you have all your jar files directly under a single folder) and paste the following 4 lines
#ECHO OFF
dir /b *.jar > allJarFilesList.txt
FOR /F %%A IN (allJarFilesList.txt) DO jar -tf %%A > list_of_packages.txt
FOR %%B IN (list_of_packages.txt) DO FIND /I "com/sun" %%B
NOTE the "com/sun" in the last line.. it is hard coded, you can pass as argument as well...
I know this is very basic form and can be improved "a lot" like looking up in various sub directories.
hope this helps :-)
.jar files are just ZIP compressed archives, rename it to zip, open it with your favourite unzip programm, and traverse through the directory.
If you add the jar file to a eclipse project, you can traverse through the lib in th project explorer.
HTH
Assuming maven downloaded the jar files,the files will be loaded in to a local repository.
You could use maven browser that comes packaged with Eclipse to browse and search for artifacts in your repository.(usually in userdir/.m2/repository)
Note:You can explore your repository directly if you want. You will understand the packages that were downloaded. But I suggest using the plugin.
If you are using Intellij IDEA, each project contains a tree called External Library that allows you to search and explore your libraries.