I have an SBT project, with spark dependencies. These dependencies are provided at runtime, and hence I import them under provided scope.
val hadoop = Seq("org.apache.hadoop" % "hadoop-client" % "3.3.1" % provided)
val spark = Seq(
"org.apache.spark" %% "spark-core" % SparkVersion % provided,
"org.apache.spark" %% "spark-sql" % SparkVersion % provided,
"org.apache.spark" %% "spark-mllib" % SparkVersion % provided
)
lazy val coreDto = (project in file("xxxx"))
.enablePlugins(BuildInfoPlugin)
.enablePlugins(PackPlugin)
.settings(
name := "xxxx",
moduleName := "xxxx",
version := "1.0",
libraryDependencies ++= (hadoop ++ spark))
All is well till now. Now I have a new scenario, where I have to publish the jar to our maven repository. And successfully I am able to publish it. The issue now is: provided scope. As the compile time dependencies are not appropriately set.
Question: How do I configure, where the provided scope is ignored during publishing? But still considered when I package it?
Using this to publish in case if helpful
lazy val publishSettings = Seq(
publishMavenStyle := true,
publishTo := {
val url = "https://xxxxl/maven/v1/"
if (version.value.trim.toUpperCase.endsWith("SNAPSHOT"))
Some("snapshots".at(url))
else
Some("releases".at(url))
},
aetherDeploy / logLevel := Level.Info,
aetherOldVersionMethod := true,
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")
)
I have something like this in my build.sbt:
lazy val someDeps = Seq(
libraryDependencies += "com.example" %% "foo" % "1.3.37",
// more
)
lazy val some_library = project.in(file("libs/somelibrary")).
settings(commonSettings).
settings(
// project-specific settings
libraryDependencies ++= someDeps
)
lazy val something_with_deps_provided = project.in(file("swdp")).
settings(commonSettings).
settings(
// project-specific settings
libraryDependencies ++= someDeps.map(d => d % "provided")
).dependsOn(some_library)
When I now use the sbt-assembly-plugin to create the assembly of something_with_deps_provided, it still puts the dependencies into the resulting jar, ignoring the provided. Is it possible to set a transitive dependency to provided later and if yes, how is it done?
In cases like this, excludeDependencies can be used as described in
SBT manual here:
Exclude Transitive Dependencies.
With your example:
lazy val something_with_deps_provided = project.in(file("swdp"))
.settings(commonSettings)
.dependsOn(some_library)
.settings(
// project-specific settings
excludeDependencies ++= someDeps.map { d =>
ExclusionRule(
organization = d.organization,
name = d.name
)
}
)
The dependencies from someDeps will no longer be included in the
assembly JAR for something_with_deps_provided project.
I'm having a problem moving from sbt version 0.13.2-M3 to 0.13.5-RC3 where transitive dependencies that 13.2-M3 successfully resolves fail to be resolved by 0.13.5-RC3.
I get unresolved dependency errors where the version is "working#".
It's happening when I have a multi-project build with two sub-projects, one of which depends on the other. They both have dependencies whose maven poms specify a common parent (though I'm not sure if that's a red herring or not).
It only happens when the dependencies aren't already in the local ivy cache.
A minimal repro Build is:
import sbt._
import Keys._
object BarBuild extends Build {
val buildSettings = Seq(scalaVersion := "2.10.3")
lazy val root = Project(
id = "bar",
base = file(".")
) aggregate(withSolrCore, withSolrClient)
lazy val withSolrCore = Project(
id = "withSolrCore",
base = file("solrCore"),
settings = buildSettings ++ Seq(
libraryDependencies ++= Seq("org.apache.solr" % "solr-core" % "4.7.1")
)
) dependsOn (withSolrClient)
lazy val withSolrClient = Project(
id = "withSolrClient",
base = file("solrClient"),
settings = buildSettings ++ Seq(
libraryDependencies ++= Seq("org.apache.solr" % "solr-solrj" % "4.7.1")
)
)
}
With build.properties's sbt.version=0.13.5-RC3 I see lots of errors like
[warn] module not found: org.apache.lucene#lucene-analyzers-kuromoji;working#heraclitus.local
and
[error] unresolved dependency: org.apache.lucene#lucene-core;working#heraclitus.local: not found
but with sbt.version=0.13.2-M3 everything's peachy.
I'm not sure if I'm doing something wrong or something's up with sbt, but at this point I suspect the latter.
Thanks.
This is a known ivy issue. The workaround is to override the versions of all the dependencies in the full transitive closure that's breaking with the "real" versions to use. (I derived the real ones by running update on a stub project with only the problem dependency with an older version of sbt, 0.13.2, which is pre-ivy-bug), like,
dependencyOverrides ++= Set(
"com.google.guava" % "guava" % "14.0.1",
"com.google.protobuf" % "protobuf-java" % "2.5.0",
"com.googlecode.concurrentlinkedhashmap" % "concurrentlinkedhashmap-lru" % "1.2",
"com.spatial4j" % "spatial4j" % "0.4.1",
...
)
Has anyone published an sbt-native-packager produced artifact (tgz in my case) using sbt-aether-deploy to a nexus repo? (I need this for the timestamped snapshots, specifically the "correct" version tag in nexus' artifact-resolution REST resource).
I can do one or the other but can't figure out how to add the packagedArtifacts in Universal to the artifacts that sbt-aether-deploy deploys to do both.
I suspect the path to pursue would be to the addArtifact() the packagedArtifacts in Universal or creating another AetherArtifact and then to override/replace the deployTask to use that AetherArtifact?
Any help much appreciated.
I am the author of the sbt-aether-deploy plugin, and I just came over this post.
import aether.AetherKeys._
crossPaths := false //needed if you want to remove the scala version from the artifact name
enablePlugins(JavaAppPackaging)
aetherArtifact := {
val artifact = aetherArtifact.value
artifact.attach((packageBin in Universal).value, "dist", "zip")
}
This will also publish the other main artifact.
If you want to disable publishing of the main artifact, then you will need to rewrite the artifact coordinates. Maven requires a main artifact.
I have added a way to replace the main artifact for this purpose, but I can now see that way is kind of flawed. It will still assume that the artifact is published as a jar file. The main artifact type is locked down to that, since the POM packaging is set to jar by default by SBT.
If this is an app, then that limitation is probably OK, since Maven will never resolve that into an artifact.
The "proper" way in Maven terms is to add a classifier to the artifact and change the "packaging" in the POM file to "pom". We will see if I get around to changing that particular part.
Ok, I think I got it amazingly enough. If there's a better way to do it I'd love to hear. Not loving that blind Option.get there..
val tgzCoordinates = SettingKey[MavenCoordinates]("the maven coordinates for the tgz")
lazy val myPackagerSettings = packageArchetype.java_application ++ deploymentSettings ++ Seq(
publish <<= publish.dependsOn(publish in Universal),
publishLocal <<= publishLocal.dependsOn(publishLocal in Universal)
)
lazy val defaultSettings = buildSettings ++ Publish.settings ++ Seq(
scalacOptions in Compile ++= Seq("-encoding", "UTF-8", "-target:jvm-1.7", "-deprecation", "-feature", "-unchecked", "-Xlog-reflective-calls"),
testOptions in Test += Tests.Argument("-oDF")
)
lazy val myAetherSettings = aetherSettings ++ aetherPublishBothSettings
lazy val toastyphoenixProject = Project(
id = "toastyphoenix",
base = file("."),
settings = defaultSettings ++ myPackagerSettings ++ myAetherSettings ++ Seq(
name in Universal := name.value + "_" + scalaBinaryVersion.value,
packagedArtifacts in Universal ~= { _.filterNot { case (artifact, file) => artifact.`type`.contains("zip")}},
libraryDependencies ++= Dependencies.phoenix,
tgzCoordinates := MavenCoordinates(organization.value + ":" + (name in Universal).value + ":tgz:" + version.value).get,
aetherArtifact <<= (tgzCoordinates, packageZipTarball in Universal, makePom in Compile, packagedArtifacts in Universal) map {
(coords: MavenCoordinates, mainArtifact: File, pom: File, artifacts: Map[Artifact, File]) =>
createArtifact(artifacts, pom, coords, mainArtifact)
}
)
)
I took Peter's solution and reworked it slightly, avoiding the naked Option.get by creating the MavenCoordinates directly:
import aether.MavenCoordinates
import aether.Aether.createArtifact
name := "mrb-test"
organization := "me.mbarton"
version := "1.0"
crossPaths := false
packageArchetype.java_application
publish <<= (publish) dependsOn (publish in Universal)
publishLocal <<= (publishLocal) dependsOn (publishLocal in Universal)
aetherPublishBothSettings
aetherArtifact <<= (organization, name in Universal, version, packageBin in Universal, makePom in Compile, packagedArtifacts in Universal) map {
(organization, name, version, binary, pom, artifacts) =>
val nameWithoutVersion = name.replace(s"-$version", "")
createArtifact(artifacts, pom, MavenCoordinates(organization, nameWithoutVersion, version, None, "zip"), binary)
}
The nameWithoutVersion replace works around SBT native packager including the version in the artifact name:
Before: me/mbarton/mrb-test-1.0/1.0/mrb-test-1.0.zip
After: me/mbarton/mrb-test/1.0/mrb-test-1.0.zip
crossPaths avoids the Scala postfix on the version.
I am trying to include the breeze-natives dependency only when packaging the app (universal:packageBin and debian:packageBin) while always including the breeze dependency. Here is what I came up with :
val breezeDependencySettings = {
val breezeUniversalNativesDependency = libraryDependencies in Universal += D.breezeNatives
val breezeDebianNativesDependency = libraryDependencies in Debian += D.breezeNatives
val breezeDependency = libraryDependencies += D.breeze
Seq(breezeUniversalNativesDependency, breezeDebianNativesDependency, breezeDependency)
}
And in the project that I want to package, I use
settings = (mySettings) ++ SbtNativePackager.packageArchetype.java_server ++
Dependencies.breezeDependencySettings
However, the breeze-natives dependency is not included in the final package created by universal:packageBin. (breeze is included correctly though)
What am I doing wrong?
Not 100% clear on your requirement but have you tried ExportJars := true?
See the excerpt from my build in my question here: https://stackoverflow.com/questions/23035100/how-to-remove-version-from-artifactid-generated-by-sbt-native-packager