Using sbt-assembly 0.14.6:
addSbtPlugin("com.eed3si9n" %% "sbt-assembly" % "0.14.6")
Creating an uber jar for a Spark app with sbt-assembly, including a couple dependencies to be shaded:
libraryDependencies += "org.json4s" % "json4s-native_2.11" % "3.5.3",
libraryDependencies += "com.typesafe" % "config" % "1.3.3",
That's actually a second attempt, I really want to
libraryDependencies += "org.json4s" %% "json4s-native" % "3.5.3",
libraryDependencies += "com.typesafe" % "config" % "1.3.3",
but thought maybe shading was causing my trouble when using %% in the moduleID. So, what works and what doesn't:
sbt-assembly can shade a couple libraries like so:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("org.json4s.**" -> "shaded_json4s.#1").inAll,
ShadeRule.rename("com.typesafe.config.**" -> "my_conf.#1")
.inLibrary("com.typesafe" % "config" % "1.3.3")
.inProject
),
Example shaded classes:
my_conf/parser/ConfigNode.class
shaded_json4s/FieldSerializer.class
Both libraries get shaded. I'd like to get more specific, avoiding inAll:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("org.json4s.**" -> "shaded_json4s.#1")
.inLibrary("org.json4s" % "json4s-native_2.11" % "3.5.3")
.inProject,
ShadeRule.rename("com.typesafe.config.**" -> "my_conf.#1")
.inLibrary("com.typesafe" % "config" % "1.3.3")
.inProject
),
But that doesn't shade json4s classes:
my_conf/parser/ConfigNode.class
org/json4s/FieldSerializer.class
But what I started with, and would really prefer is:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("org.json4s.**" -> "shaded_json4s.#1")
.inLibrary("org.json4s" %% "json4s-native" % "3.5.3")
.inProject,
ShadeRule.rename("com.typesafe.config.**" -> "my_conf.#1")
.inLibrary("com.typesafe" % "config" % "1.3.3")
.inProject
),
But that doesn't shade json4s classes, either:
my_conf/parser/ConfigNode.class
org/json4s/FieldSerializer.class
Is there something I need to do differently for shading to work on json4s when using inLibrary?
Related
Following this tutorial, I am asked to add enablePlugins(WindowsPlugin) to my SBT configuration.
I did this by stating exactly this line in my build.sbt but all I get is "Cannot resolve symbol". Do I need to add the dependency somewhere?
Is this an auto plugin and can anyone explain to me what an auto plugin actually is and how I use it?
UPDATE: My build.sbt looks like that:
name := "ApplicationName"
version := "0.3-SNAPSHOT"
scalaVersion := "2.13.1"
enablePlugins(WindowsPlugin)
mainClass in assembly := Some("application.ConfigEditorApplication")
assemblyJarName in assembly := s"application-$version.jar"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs#_*) => MergeStrategy.discard
case PathList("reference.conf") => MergeStrategy.concat
case x => MergeStrategy.first
}
libraryDependencies += "org.apache.commons" % "commons-lang3" % "3.9"
libraryDependencies += "commons-io" % "commons-io" % "2.6"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"
libraryDependencies += "com.typesafe.scala-logging" % "scala-logging_2.13" % "3.9.2"
libraryDependencies += "com.typesafe.akka" %% "akka-actor-typed" % "2.6.3"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.1.1" % "test"
libraryDependencies += "org.scalamock" %% "scalamock" % "4.4.0" % Test
libraryDependencies += "org.mockito" % "mockito-scala_2.13" % "1.11.3"
libraryDependencies += "org.mockito" % "mockito-scala-scalatest_2.13" % "1.11.3"
I found the solution to my problem: From the beginning, I suspected, that the plugin needs to be added before it can be enabled. Unfortunately, nothing of that sort was mentioned in the tutorial I was following.
The plugin which has to be added is the native-packager plugin: addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.0").
You should create your auto pluggin in your build.sbt. The build.sbt file must be at the root of your projet, at the same level with the file src.
You have information about it here and here.
In the page you mentioned they say you should set this in your build.sbt. Try this.
// general package information (can be scoped to Windows)
maintainer := "Josh Suereth <joshua.suereth#typesafe.com>"
packageSummary := "test-windows"
packageDescription := """Test Windows MSI."""
// wix build information
wixProductId := "ce07be71-510d-414a-92d4-dff47631848a"
wixProductUpgradeId := "4552fb0e-e257-4dbd-9ecb-dba9dbacf424"
UPDATE
Also, I found this question which is related to yours. It is true, it is an old one, but it might give you some hints. Some answers suggest performing updatings, others deleting and then reimporting the project.
I am new to sbt build.
I would like to add java files of a dependent project (say Proj A) to my compiling project (Proj B).
Running sbt compile in Proj B throws error that dependent project's java package/classes are not found.
I went through the link: https://www.scala-sbt.org/0.13/docs/Multi-Project.html but its not clear to me add this dependency to make it work.
I tried adding a below line in build.sbt, but it didnt work.
lazy val projB = project.dependsOn(/projA)
Updated
build.sbt of projB:
organization := "com.org"
name := "projB"
version := "1"
resolvers ++= Seq(
"Typesafe" at "http://repo.typesafe.com/typesafe/releases/",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/",
)
lazy val projB = project.dependsOn(projA)
// the library dependencies of springframework here
build.sbt of Proj A:
organization := "com.org"
name := "proj A"
version := "1"
resolvers ++= Seq(
"Typesafe" at "http://repo.typesafe.com/typesafe/releases/",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/",
)
// the library dependencies of springframework here
When i do sbt compile on proj B, it throws error the dependent classes are not found. Class Hbase is in Proj A.
[error] import com.org.config.Hbase;
[error] **\hbase\HbaseDAO.java:38:1:
cannot find symbol
[error] symbol: class Hbase
[error] location: class com.org.hbase.HbaseDAO
[error] private Hbase hbase;
[error] (Compile / compileIncremental) javac returned non-zero exit code
[error] Total time: 6 s, completed 29/08/2019 9:58:39 AM
Updated build.sbt after the suggestion:
inThisBuild(
Seq(
organization := "com.org",
version := "1",
resolvers ++= Seq(
"Typesafe" at "http://repo.typesafe.com/typesafe/releases/",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/",
)
)
)
lazy val root = project
.in(file("."))
.aggregate(projA,projB)
lazy val projA = project.settings(
// project A settings and library dependencies
libraryDependencies += "org.springframework.boot" % "spring-boot-starter-
parent" % "2.1.6.RELEASE" pomOnly()
libraryDependencies += "org.springframework.boot" % "spring-boot-starter-
web" % "2.1.6.RELEASE"
libraryDependencies += "org.springframework.data" % "spring-data-hadoop-
hbase" % "2.3.0.RELEASE"
libraryDependencies += "org.mortbay.jetty" % "jetty" % "7.0.0.pre5"
libraryDependencies += "io.netty" % "netty-all" % "5.0.0.Alpha2"
libraryDependencies += "commons-beanutils" % "commons-beanutils" % "1.9.4"
libraryDependencies += "commons-beanutils" % "commons-beanutils-core" %
"1.8.3"
libraryDependencies += "xerces" % "xercesImpl" % "2.12.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-server-
nodemanager" % "3.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "3.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.7.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "3.2.0"
libraryDependencies += "org.apache.hbase" % "hbase-client" % "2.1.1"
libraryDependencies += "org.apache.hbase" % "hbase" % "2.1.1" pomOnly()
libraryDependencies += "org.apache.hbase" % "hbase-common" % "2.1.1"
)
lazy val projB = project
.dependsOn(projA)
.settings(
// project B settings and library dependencies
libraryDependencies += "org.springframework.boot" % "spring-boot-starter-
parent" % "2.1.6.RELEASE" pomOnly()
libraryDependencies += "org.springframework.boot" % "spring-boot-starter-
web" % "2.1.6.RELEASE"
libraryDependencies += "org.springframework.data" % "spring-data-hadoop-
hbase" % "2.3.0.RELEASE"
libraryDependencies += "org.mortbay.jetty" % "jetty" % "7.0.0.pre5"
libraryDependencies += "io.netty" % "netty-all" % "5.0.0.Alpha2"
libraryDependencies += "commons-beanutils" % "commons-beanutils" % "1.9.4"
libraryDependencies += "commons-beanutils" % "commons-beanutils-core" %
"1.8.3"
libraryDependencies += "xerces" % "xercesImpl" % "2.12.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-server-
nodemanager" % "3.2.0"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" %
"2.10.0.pr2"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "3.2.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "3.2.0"
libraryDependencies += "org.apache.hbase" % "hbase-client" % "2.1.1"
libraryDependencies += "org.apache.hbase" % "hbase" % "2.1.1" pomOnly()
libraryDependencies += "org.apache.hbase" % "hbase-common" % "2.1.1"
)
An error is thrown while sbt compile after the below library dependency in both project settings projA and projB
libraryDependencies += "org.springframework.boot" % "spring-boot-starter-
web" % "2.1.6.RELEASE"
')' expected but string literal found is thrown for this line in projA settings and
';' expected but string literal found is thrown for this line in projB settings.
I couldnt get much clue with this err.
Looking at the two snippets you posted, I'm guessing that you have two separate build.sbt files, one for each subproject. This makes them independent and one project just doesn't see the other. While it may be possible to have multiple build.sbt files for the subprojects, it's recommended to define the whole multiproject build in a single build.sbt file in the root of the project.
For example, if you structure your project like this:
├── project
│ ├── build.properties
│ └── plugins.sbt
├── projA
│ └── src
├── projB
│ └── src
└── build.sbt
Then you can put all the build settings and subproject relations in the root build.sbt:
inThisBuild(
Seq(
organization := "com.org",
version := "1",
resolvers ++= Seq(
"Typesafe" at "http://repo.typesafe.com/typesafe/releases/",
"Java.net Maven2 Repository" at "http://download.java.net/maven/2/",
)
)
)
lazy val root = project
.in(file("."))
.aggregate(projA, projB)
lazy val projA = project
.settings(
// project A settings and library dependencies
)
lazy val projB = project
.dependsOn(projA)
.settings(
// project B settings and library dependencies
)
Then if you launch an sbt shell from the root of the project, you can call compile (or any other task) to compile both projA and projB, or you can call projA/compile to compile that subproject specifically.
You are already reading documentation, so you know where to find more information. Notice that the link you provided points to the old documentation, at the top there is a banner pointing to the new page: https://www.scala-sbt.org/1.x/docs/Multi-Project.html
How can I add process parameters using sbt-native-packager configuration? I want to add the options for redirect process stderr to file? To have the result like that:
sudo -u app bash -c "app >>/var/log/app/stderr.log 2>&1"
I use sbt-native-packager 1.2.0-M5 for build deb package with JavaServerAppPackaging, JDebPackaging, SystemdPlugin, UpstartPlugin the exception in logs, only in stderr. Also I must delete app pid manually after crash and if it exists, then I have error in stderr.
My plugins.sbt:
resolvers += Resolver.bintrayRepo("sbt", "sbt-plugin-releases")
// The Play plugin
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.5.8-netty-4.1")
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.2.0-M5")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.2")
addSbtPlugin("com.lightbend.sbt" % "sbt-javaagent" % "0.1.1")
libraryDependencies += "org.vafer" % "jdeb" % "1.3" artifacts (Artifact("jdeb", "jar", "jar"))
my build.sbt:
...
debianPackageDependencies in Debian ++= Seq("postgresql-9.5 (>= 9.5.1)")
lazy val root = (project in file(".")).enablePlugins(PlayScala, JavaAgent)
scalaVersion := "2.11.8"
val akkaVersion = "2.4.10"
libraryDependencies ++= Seq(
"org.postgresql" % "postgresql" % "9.4.1208",
"org.scalikejdbc" %% "scalikejdbc" % "2.4.0",
"org.scalikejdbc" %% "scalikejdbc-config" % "2.4.0",
"org.scalikejdbc" %% "scalikejdbc-play-initializer" % "2.5.1",
"org.flywaydb" %% "flyway-play" % "3.0.1",
"com.typesafe.akka" %% "akka-contrib" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
"io.dropwizard.metrics" % "metrics-core" % "3.1.2",
"io.dropwizard.metrics" % "metrics-jvm" % "3.1.2",
"org.coursera" % "dropwizard-metrics-datadog" % "1.1.4",
"com.typesafe.akka" %% "akka-testkit" % akkaVersion % Test,
"com.relayrides" % "pushy" % "0.8",
"com.relayrides" % "pushy-dropwizard-metrics-listener" % "0.8",
"org.eclipse.jetty.alpn" % "alpn-api" % "1.1.3.v20160715" % "runtime",
ws,
specs2 % Test
)
resolvers += "Typesafe Releases" at "http://repo.typesafe.com/typesafe/maven-releases/"
resolvers += Resolver.mavenLocal
routesGenerator := InjectedRoutesGenerator
javaOptions in Test ++= Seq("-Dlogger.resource=logback-test.xml")
scalacOptions in Universal ++= Seq("-unchecked", "-deprecation", "-notailcalls")
javaOptions in Universal ++= Seq(
"-J-server",
...
)
...
import com.typesafe.sbt.packager.archetypes.systemloader._
// UpstartPlugin for ubuntu 14.04, SystemdPlugin for ubuntu 16.04
enablePlugins(JavaServerAppPackaging, JDebPackaging, SystemdPlugin, UpstartPlugin)
requiredStartFacilities := Some("datadog-agent.service, systemd-journald.service, postgresql.service")
javaAgents += "org.mortbay.jetty.alpn" % "jetty-alpn-agent" % "2.0.4" % "dist"
ps I found a workaround, in ubuntu 16.04 I can use journald to collect all the logs in the system.
Thanks for updating the question with all relevant information. There are a couple of things here.
Only one Systemloader plugin
You enable SystemdPlugin and UpstartPlugin. If it works, it only works by accident. No version of native-packager was designed to support multiple systemloader for a single package type in a single build module.
The solution is to create sub modules with the relevant systemloader enabled.
Logging to stderr
You are right regarding systemd. It provides facilities to capture the log output of your process. If you like you can add your findings to the native-packager documentation ( there is a systemd plugin section ).
The upstart support in native-packager is rather simple. There weren't a lot of requeset as Ubuntu is switching to systemd and you can always fallback to systemv. Which brings me to the solution to your problem.
You can use the SystemVPlugin, which supports a daemon_log_file. The systemv documentation provides you with the necessary details.
cheers,
Muki
I would like to download a zip artifact and find the corresponding file in local repository.
Where I can declare the zip extension ?
libraryDependencies ++= Seq(
"com.acme" % "audit-agent" % "0.7" % "test" // ??? where I put zip ?
)
May be, I can just use some object to reference the artifact, download it, and file the filename ?
Any Idea ?
Use from method of sbt.ModuleID in libraryDependencies as described in Explicit URL:
libraryDependencies += "organization" % "myModuleName" % "1.0" from "https://myhost.pl/slinky.zip"
Then follow How to extract dependency jar to specific folder during compilation? and use update and .filter:
val jar = (update in Compile).value
.select(configurationFilter("compile"))
.filter(_.name.contains("myModuleName"))
.head
There is a more native way:
libraryDependencies += "org" % "name" % "rev" artifacts(Artifact("name", "type", "ext"))
or in your case
libraryDependencies ++= Seq(
"com.acme" % "audit-agent" % "0.7" % "test" artifacts(Artifact("audit-agent", "zip", "zip")))
I'm trying to create a plugin for sbt 0.12.1 that will generate java files from WSDL, compile them, and then publish the jar.
My project layout is like:
./build.sbt
./project/build.sbt
./project/WsdlBuild.scala
./src/main/wsdl/...many wsdl files...
I'm using axis to generate the java files, and build.sbt looks like:
name := "zxtm-api"
organization := "com.giltgroupe.zeus"
unmanagedBase <<= baseDirectory { base => base / "wsdl-lib" }
libraryDependencies ++= Seq(
"axis" % "axis-wsdl4j" % "1.2.1",
"commons-logging" % "commons-logging" % "1.0.4",
"commons-discovery" % "commons-discovery" % "0.2",
"log4j" % "log4j" % "1.2.8",
"org.apache.axis" % "axis" % "1.4",
"org.apache.axis" % "axis-ant" % "1.4",
"org.apache.axis" % "axis-jaxrpc" % "1.4",
"org.apache.axis" % "axis-saaj" % "1.4"
)
gilt.zxtm.WsdlBuild.wsdlSettings
(There was one jar we couldn't find in any maven repo that's in wsdl-lib)
project/build.sbt is very similar:
libraryDependencies ++= Seq(
"axis" % "axis-wsdl4j" % "1.2.1",
"commons-logging" % "commons-logging" % "1.0.4",
"commons-discovery" % "commons-discovery" % "0.2",
"log4j" % "log4j" % "1.2.8",
"org.apache.axis" % "axis" % "1.4",
"org.apache.axis" % "axis-ant" % "1.4",
"org.apache.axis" % "axis-jaxrpc" % "1.4",
"org.apache.axis" % "axis-saaj" % "1.4"
)
unmanagedBase <<= baseDirectory { base => base / "wsdl-lib" }
So I wrote the code in WsdlBuild.scala to generate the java files, and ended up with something like:
object WsdlBuild extends Plugin {
lazy val wsdlSourceDir = SettingKey[File]("wsdl-source-dir")
lazy val wsdlToJava = TaskKey[Unit]("wsdl-to-java")
lazy val managedSrcDir = file("target/src_managed/wsdl")
val wsdlSettings = inConfig(Compile)(Seq(
compile <<= compile dependsOn wsdlToJava,
javaSource := managedSrcDir,
managedSourceDirectories := Seq(managedSrcDir)
)) ++ Seq(
wsdlToJava <<= (wsdlSourceDir, managedSourceDirectories in Compile, state) map {
(wsdlDir, managedDirs, s) =>
// by convention use the first one. Not obvious why there is
// ever more than one
createJavaFromWsdl(wsdlDir, managedDirs.head, s.log)
},
wsdlSourceDir := file("src/main/wsdl")
)
def createJavaFromWsdl(wsdlDir: File, outputDir: File, log: Logger): File = { ... }
So this sort of works. If I run compile, it generates the wsdl correctly. But if I publish-local, it doesn't compile. So in order to publish or publish-local, and I have to manually compile first.
Any suggestions?
Generating sources and resources is described in this howto of the sbt docs.
In your case, wsdlSettings might look like:
val wsdlSettings = inConfig(Compile)(Seq(
sourceGenerators <+= wsdlToJava,
wsdlSourceDir <<= baseDirectory(_ / "src/main/wsdl"),
wsdlToJava <<= (wsdlSourceDir, sourceManaged, streams) map {
(wsdlDir, managedDir, s) =>
createJavaFromWsdl(wsdlDir, managedDir, s.log)
},
)
Some changes unrelated to your question:
Get the logger from streams. This sends output to a task-specific logger so that you can retrieve it individually. See this howto for more information on this.
Always use absolute paths, often by basing a file on baseDirectory. See Use absolute paths.
The question is quite old, although the problem might be still relevant to someone.
In my case, I approached a very similar problem by having an sh script that does all the dirty work of WSDL generation with wsimport (comes with Java out of the box). A dedicated sbt subproject wraps it as a task and executes on compilation. Such subproject can be easily inserted into any other, bigger sbt setup where you can just add a dependency on it.
Enough talking, here's a template on GitHub that demonstrates exactly that: https://github.com/sainnr/sbt-scala-wsdl-template. Hope it saves someone a good couple of hours messing around with WSDL and build tools. Feel free to fork or improve it if you find it any helpful!