I seem to be unable finding the root cause of the first eviction seen below. As seen below, running evicted does not provide details relevant to it.
This all occurs after having added in project/plugins.sbt:
libraryDependencies += "io.spray" %% "spray-json" % "1.3.1"
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * io.spray:spray-json_2.10:1.2.6 -> 1.3.1
[warn] Run 'evicted' to see detailed eviction warnings
[projectName] $ evicted
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * com.googlecode.efficient-java-matrix-library:ejml:0.23 -> 0.19 (caller: edu.arizona.sista:processors:3.3, edu.stanford.nlp:stanford-corenlp:3.3.1)
[warn] * com.google.guava:guava:(15.0, 13.0.1) -> 16.0.1 (caller: org.sorm-framework:sorm:0.3.16, com.jolbox:bonecp:0.8.0.RELEASE, com.typesafe.play:play-ws_2.11:2.3.7)
[warn] * xml-apis:xml-apis:(2.0.2, 1.3.03) -> 1.4.01 (caller: xalan:xalan:2.7.0, xerces:xercesImpl:2.11.0, xom:xom:1.2.5, com.io7m.xom:xom:1.2.10)
[warn] * io.netty:netty:(3.9.2.Final, 3.6.3.Final) -> 3.9.3.Final (caller: com.ning:async-http-client:1.8.14, com.typesafe.netty:netty-http-pipelining:1.1.2, com.typesafe.play:play_2.11:2.3.7)
[success] Total time: 2 s, completed Apr 9, 2015 8:04:11 PM
At runtime, I get this crash without even trying to use spray-json, so something really isn't right about how this dependency is managed or evicted in my sbt build:
[error] (web-assets:less::jstaskJsOptions) java.lang.NoSuchMethodError: spray.json.JsArray.<init>(Lscala/collection/immutable/List;)V
There is no version 1.2.6 of this dependency explicitly defined in my build.sbt in the first place, maybe it is required by other plugins there, but how can that be traced without the eviction details? How can I fathom the details of this mysterious eviction?
You can use
show externalDependencyClasspath
to get the toString representation that includes all transitive dependencies but it looks quite cumbersome. I suggest to install the sbt-dependency-graph plugin which makes the dependencyTree task available, that shows you your full classpath in a readable format.
Related
I'm trying to run scoverage on a crossBuild sbt coverage +test coverageAggregate and i'm getting:
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/Users/module/"), "module_name"):
[error] org.scoverage:scalac-scoverage-runtime _2.12, _2.13
from what I can tell my dependencies are with %% , except one pure java dependecy
sbt version is 0.13.9 and scala 2.11.7/ I know previous versions of sbt relied on scala 2.10 - is that still the case? I have a Java project for which I added an assembly.sbt file in the project directory (as per the sbt assembly instructions for this version of sbt assembly):
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")
I ran sbt reload/clean as well as compile. However, when I try to run assembly, I get the following exception:
> assembly
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly
[error] assembly
[error] ^
what step(s) a I missing before I can create a runnable jar for my java project using sbt?
I can setup a merge strategy if one is needed but I received a different exception when I tried using that. Any helpful suggestions would be quite welcome.
Update
I had created the assemly.sbt file in the wrong location. When I placed the assembly.sbt file in the same directory as the build.sbt, and ran reload/compile, I got the following error:
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.14.2: not found
[error] Total time: 2 s, completed Mar 13, 2016 3:20:02 PM
**running with last *:update**
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.eed3si9n:sbt-assembly:0.14.2 (scalaVersion=2.11, sbtVersion=0.13)
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.eed3si9n:sbt-assembly:0.14.2 (scalaVersion=2.11, sbtVersion=0.13) (/home/me/test-server/assembly.sbt#L1-2)
[warn] +- com.test:test-server_2.11:1.0.0
sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.14.2: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:294)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:132)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
at sbt.IvySbt.withIvy(Ivy.scala:127)
at sbt.IvySbt.withIvy(Ivy.scala:124)
at sbt.IvySbt$Module.withModule(Ivy.scala:155)
at sbt.IvyActions$.updateEither(IvyActions.scala:168)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1392)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1388)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$90.apply(Defaults.scala:1422)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$90.apply(Defaults.scala:1420)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1425)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1419)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1442)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1371)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1325)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error] (*:update) sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.14.2: not found
Project structure
I will add screenshots later. For now, please bear with me.
I tried creating the assembly.sbt file inside the project subdirectory - it did not work. Then I added it to the plugins.sbt inside the same project subdirectory. When it did not work, I copied the assembly.sbt and plugin.sbt into the root directory of the project but it did not solve the issue.
Project directory: test-server
I have added assembly.sbt and build.sbt inside /test-server
I have also tried adding it ti /test-server/project
Neither worked. I am reasonably sure I am doing something wrong or missing something in my config. Kindly share your thoughts as to what needs to be done to resolve the issue.
assembly.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")
The issue was due to a typo in my assembly.sbt file. The assembly.sbt file needs to be inside the project subdirectory i.e. if your project root directory is test-server, one needs to place the assembly.sbt file inside the test-server/project subdirectory.
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.2")
So, I am running into the same issue that many are experiencing - see the error below.
WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0 (): java.io.IOException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
The instructions for fixing this seem to be missing, incomplete, or overly complex. I've resolved other reference issues simply by adding them to the packages list in spark-submit. I've tried doing the same with this htrace dependency too, but with no luck.
This is my current spark-submit command:
spark-submit --class com.biz.HbasePassthrough \
--packages \
org.apache.spark:spark-streaming-kafka_2.10:1.3.0,org.apache.hbase:hbase-common:1.0.0,org.apache.hbase:hbase-client:1.0.0,org.apache.hbase:hbase-server:1.0.0,org.json4s:json4s-jackson_2.10:3.2.11,org.apache.htrace:htrace:3.1.0-incubating,org.apache.htrace:htrace-core:3.1.0-incubating,org.apache.htrace:htrace-hbase:3.1.0-incubating,org.apache.hbase:hbase-annotations:1.0.0 \
./my-spark_2.10-1.0.8.jar \
>spark_log 2>&1
Output of grep -n1 "htrace" spark_log
8-org.json4s#json4s-jackson_2.10 added as a dependency
9:org.apache.htrace#htrace added as a dependency
10:org.apache.htrace#htrace-core added as a dependency
11:org.apache.htrace#htrace-hbase added as a dependency
12-org.apache.hbase#hbase-annotations added as a dependency
--
36- found org.mortbay.jetty#jetty-util;6.1.26 in central
37: found org.apache.htrace#htrace-core;3.1.0-incubating in central
38- found org.apache.hbase#hbase-client;1.0.0 in central
--
75- found com.fasterxml.jackson.core#jackson-core;2.3.1 in central
76: found org.apache.htrace#htrace;3.1.0-incubating in central
77: found org.apache.htrace#htrace-hbase;3.1.0-incubating in central
78-:: resolution report :: resolve 6423ms :: artifacts dl 45ms
--
111- org.apache.hbase#hbase-server;1.0.0 from central in [default]
112: org.apache.htrace#htrace;3.1.0-incubating from central in [default]
113: org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
114: org.apache.htrace#htrace-hbase;3.1.0-incubating from central in [default]
115- org.apache.kafka#kafka_2.10;0.8.1.1 from central in [default]
--
182-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/json4s-jackson_2.10.jar at http://192.168.240.209:37644/jars/json4s-jackson_2.10.jar with timestamp 1448926760838
183:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-core.jar at http://192.168.240.209:37644/jars/htrace-core.jar with timestamp 1448926760845
184:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-hbase.jar at http://192.168.240.209:37644/jars/htrace-hbase.jar with timestamp 1448926760846
185-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/hbase-annotations.jar at http://192.168.240.209:37644/jars/hbase-annotations.jar with timestamp 1448926760847
All of that output at the beginning of the job seems to suggest that it found the package and will use it successfully, but it doesn't work.
I did finally find something that I think got me past the issue - here. I adapted this by adding the following two lines to my spark-submit command above
--driver-class-path "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
Why didn't the packages reference work?
I'm trying to use jbpm2 6.0.0.Final in my sbt-based project.
Some problems occure with external dependencies. Artifact version is written using pom's variable reference.
I can add those missing dependencies by hand and it will fix the matter but I still need to guest what versions are needed.
Any idea what is the best solution to this problem?
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.eclipse.jdt.core.compiler#ecj;${version.org.eclipse.jdt.core.compiler}: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
In jBPM we use the jbpm-bom to define all the version, look at here, it might help: https://github.com/droolsjbpm/droolsjbpm-build-bootstrap
I want to use scalax.io._ to manipulate file operations with SBT.
When I ran it, I got the error message showing scalax is not found.
>sbt run
import scalax.io._
[error] ^
[error] iotest.scala:49: not found: object scalax
How to find the library dependency for this particular one?
A more general question, how to obtain the library dependency information for any library?
For example, if I need use actor in scala, I need specify a library dependency. How to find the library dependency?
"the library dependency information for any library" is part of the library's documentation and the author(s) is supposed to publish the info for different project management tools, sbt including. After all, what would be the purpose of developing a library that's hard to use?
Use http://search.maven.org/ to search for a library, and when you search for scala-io you'll get a list of available scala-io libraries.
Since I've never worked with the library I copied the ScalaIOExample example from Scala IO Documentation to have a working example. It needs the scalax.io and scalax.file packages that are distributed as scala-io-file artifact. Searching for the artifact leads to Artifact Details For com.github.scala-incubator.io:scala-io-file_2.10:0.4.2 with information on how to use it with Scala SBT in Dependency Information section.
With this, I created the following build.sbt in a sbt project:
scalaVersion := "2.10.3"
libraryDependencies += "com.github.scala-incubator.io" %% "scala-io-file" % "0.4.2"
It will add scala-io-file_2.10-0.4.2.jar to classpath and executing run in the project gives the following results:
$ sbt run
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/sandbox/stackoverflow/sbt-scala-io/project
[info] Set current project to sbt-scala-io (in build file:/Users/jacek/sandbox/stackoverflow/sbt-scala-io/)
[info] Running ScalaIOExample
Not interrupting system thread Thread[Keep-Alive-Timer,8,system]
Not interrupting system thread Thread[Keep-Alive-SocketCleaner,8,system]
[success] Total time: 5 s, completed Dec 31, 2013 11:16:42 PM