NoSuchMethodError when using Scala in R with rscala - r

I'm having trouble using rscala package in R to call precompiled Scala code.
I have an Scala project running on version 2.11.12. I build a fat jar with sbt-assembly and make it available to rscala through rscala::scala("path-to-jar.jar")
But whenever I try to run any function/method/whatever from the package, I get the following error:
java.lang.reflect.InvocationTargetException
java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
at myPackage.myClass$.myMethod(myClass.scala:116)
at $line6.$read$$iw$$iw$$iw$$iw$.$anonfun$res0$1(<console>:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.ddahl.rscala.server.Server.$anonfun$invoke$3(Server.scala:305)
at org.ddahl.rscala.server.Server.wrap(Server.scala:47)
at org.ddahl.rscala.server.Server.invoke(Server.scala:305)
at org.ddahl.rscala.server.Server.run(Server.scala:400)
at org.ddahl.rscala.server.Main$.delayedEndpoint$org$ddahl$rscala$server$Main$1(Main.scala:105)
at org.ddahl.rscala.server.Main$delayedInit$body.apply(Main.scala:11)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at org.ddahl.rscala.server.Main$.main(Main.scala:11)
at org.ddahl.rscala.server.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.reflect.internal.util.ScalaClassLoader.$anonfun$run$2(ScalaClassLoader.scala:106)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:132)
at scala.reflect.internal.util.ScalaClassLoader.run(ScalaClassLoader.scala:106)
at scala.reflect.internal.util.ScalaClassLoader.run$(ScalaClassLoader.scala:98)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:132)
at scala.tools.nsc.CommonRunner.run(ObjectRunner.scala:28)
at scala.tools.nsc.CommonRunner.run$(ObjectRunner.scala:27)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
at scala.tools.nsc.CommonRunner.runAndCatch(ObjectRunner.scala:35)
at scala.tools.nsc.CommonRunner.runAndCatch$(ObjectRunner.scala:34)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:73)
at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:92)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:108)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Error in pop(details, transcompileInfo, envir) : Invocation error.
It does allow me to import its classes and objects, so it does seem to be in the classpath, but that's it.
My build.sbt includes crossScalaVersions := Seq("2.11.12", "2.12.7", "2.13.0-M5") as required in the package.
Any thoughts? It seems like a scala version error, since running rscala::scalaConfig() shows me it is using scalaFullVersion 2.12.8, but I can't seem to choose a scala installation, so I can't really expect this to be the problem.
EDIT:
After Jasper-M's suggestion, I set environment variable RSCALA_SCALA_MAJORVERSION=2.11 and reexecuted the config with rscala::scalaConfig(download='scala'). Now it returns scalaMajorVersion as 2.11.
Now, when I create an instance of the scala bridge with rscala::scala(), it returns a message (not an error) saying:
cat: /release: No such file or directory
If I run println() it compiles with no issues. However, if I instantiate the scala bridge with a precompiled fat JAR of my own and do ANYTHING (like println), it returns the following error:
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/reflect/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/reflect/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading package, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/package.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
error: error while loading Predef, class file '$HOME/.rscala/scala/lib/scala-library.jar(scala/Predef.class)' is broken
(class java.lang.RuntimeException/Scala class file does not contain Scala annotation)
I'm really clueless here. Any ideas?

It's most definitely a Scala version error.
My best guess is that you have to set the environment variable RSCALA_SCALA_MAJORVERSION=2.11 . E.g. by calling
Sys.setenv(RSCALA_SCALA_MAJORVERSION = "2.11")
and then run
rscala::scalaConfig(download = "scala")

Related

Jar file created with JavaFX will not load because of error java.util.MissingResourceException: Can't find bundle for base name lng, locale en_US?

I have a resource bundle with language properties. If the local language is French, it loads french texts or else default English. My program works okay when using Intellij but when I create the jar file and try to run it by doing java -jar myapp.jar then it crashes and shows error.
Exception in Application start method
Exception in thread "main" java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:900)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(LauncherImpl.java:195)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javafx.fxml.LoadException:file:/C:/Users/rsoph/OneDrive/Desktop/C195new/out/artifacts/C195_Project_jar/C195_Project.jar!/view/login_welcome_screen.fxml
at javafx.fxml.FXMLLoader.constructLoadException(FXMLLoader.java:2625)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2595)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2466)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3237)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3194)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3163)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3136)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3113)
at javafx.fxml.FXMLLoader.load(FXMLLoader.java:3106)
at main.Main.start(Main.java:33)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
... 1 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.sun.javafx.reflect.Trampoline.invoke(MethodUtil.java:76)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.sun.javafx.reflect.MethodUtil.invoke(MethodUtil.java:273)
at com.sun.javafx.fxml.MethodHelper.invoke(MethodHelper.java:83)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2591)
... 17 more
Caused by: java.util.MissingResourceException: Can't find bundle for base name lng, locale en_US
at java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1556)
at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:932)
at controller.LoginWelcomeScreenController.initialize(LoginWelcomeScreenController.java:61)
... 29 more
This is how I use the resource bundle
ResourceBundle resBundle = ResourceBundle.getBundle("lng", Locale.getDefault());
if (Locale.getDefault().getLanguage().equals("fr")) {
usernameLabel.setText(resBundle.getString("username"));
passwordLabel.setText(resBundle.getString("password"));
loginButton.setText(resBundle.getString("login"));
exitButton.setText(resBundle.getString("exit"));
}
My program structure looks like this.
EDIT: This is how I created the jar executable file.
File -> Project Structure -> Artifacts -> click on the plus sign (+) -> Jar -> From modules with dependencies.
Select the main class and OK. Followed by Clicking on the + sign and adding all files from the javafx bin folder to the File ( all the dll files).
Go to Build -> Build artifacts -> Build.
The bundle name you are looking up is "lng", but your the files in your bundle have prefixes of "Lng".
The case of the resource file name and the bundle lookup string should match.
If you are using a Windows system, the lookup on the file name is case-insensitive when using the file protocol to access the unpackaged resources.
But, when packaged in a jar, the lookup of the resources using the jar protocol is case sensitive.
Which explains why you can find the resources when not packaged in a jar, and why the lookup fails after jar packaging.
In addition to the issue with the jar packaging, relying on case insensitive file names also means that the project would fail if used on an OS with a case sensitive file system, like a Mac or Linux system.

log4j Error in building Corda sample project

When I build the sample corda project with gradle on windows (gradlew.bat deployNodes)
I see the error about DELETE command in log4j:
2019-10-28 16:30:47,207 main ERROR Delete contains invalid attributes "IfFileName", "IfLastModified"
2019-10-28 16:30:47,213 main ERROR Missing Delete conditions: unconditional Delete not supported
2019-10-28 16:30:47,217 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.rolling.action.DeleteAction for element Delete: java.lang.IllegalArgume
ntException: Unconditional Delete not supported java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:76)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:121)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at net.corda.core.utilities.KotlinUtilsKt.contextLogger(KotlinUtils.kt:42)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:108)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:107)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at net.corda.cliutils.CliWrapperBase$Companion.getLogger(CordaCliWrapper.kt:65535)
at net.corda.cliutils.CliWrapperBase$Companion.access$getLogger$p(CordaCliWrapper.kt:107)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:142)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:106)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73)
at net.corda.node.Corda.main(Corda.kt:13)
Caused by: java.lang.IllegalArgumentException: Unconditional Delete not supported
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.(DeleteAction.java:71)
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.createDeleteAction(DeleteAction.java:212)
... 43 more
2019-10-28 16:30:47,606 main ERROR Null object returned for Delete in DefaultRolloverStrategy.
2019-10-28 16:30:47,736 main ERROR Delete contains invalid attributes "IfFileName", "IfLastModified"
2019-10-28 16:30:47,737 main ERROR Missing Delete conditions: unconditional Delete not supported
2019-10-28 16:30:47,739 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.rolling.action.DeleteAction for element Delete: java.lang.IllegalArgume
ntException: Unconditional Delete not supported java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:76)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:121)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at net.corda.core.utilities.KotlinUtilsKt.contextLogger(KotlinUtils.kt:42)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:108)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:107)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at net.corda.cliutils.CliWrapperBase$Companion.getLogger(CordaCliWrapper.kt:65535)
at net.corda.cliutils.CliWrapperBase$Companion.access$getLogger$p(CordaCliWrapper.kt:107)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:142)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:106)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73)
at net.corda.node.Corda.main(Corda.kt:13)
Caused by: java.lang.IllegalArgumentException: Unconditional Delete not supported
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.(DeleteAction.java:71)
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.createDeleteAction(DeleteAction.java:212)
... 43 more
I got latest verison of corda samples. (from master branch). I encounter the same problem with other branches.
My jdk version is 1.8.0_212.
I used cordapp-examples project
How can I solve this problem?
It looks to me like you are pulling in JAR files from somewhere other than those specified by the Corda build.
The default branch for the samples is release-V4 and that is the one I'm using. I run the following and it works fine:
git clone https://github.com/corda/samples
cd cordapp-example
gradlew deployNodes
I'd check that your classpath variable isn't set (as this will cause additional JAR files in those directories to be included when running Corda) by running :
echo %CLASSPATH%
Additionally, from the command-line I'd run:
java -version
where java
To make sure the version of java you think you're picking up is the one you are picking up.
I had a similar issue related to log4j:
the log4j2 error unable to invoke factory method in class class RollingFileApender
Error while generating node info file
To fix this (on windows), I took the following steps:
Right click on MyPC
Click advanced system settings
Click environment variables
Change JAVA_HOME to point to same jdk directory being used in IntelliJ for corDapp. (I changed mine from OpenJDK\jdk-13.0.1 to C:\Program Files\Java\jdk1.8.0_231)
Restart IntelliJ and rerun ./gradlew deployNodes
The problem was about the regional settings of my computer.
By defining an Environment variable
JAVA_TOOL_OPTIONS=-Duser.language=en -Duser.country=US -Duser.region=US -Dfile.encoding=UTF-8 -Duser.variant=US
I solved the problem.
Thanks all for your attention.

Cannot find class in same jar during SBT build

I am trying to build an sbt plugin like swagger2markup, but I face an issue when executing this in sbt:
org.apache.commons.configuration2.ex.ConfigurationRuntimeException: java.lang.ClassNotFoundException: org.apache.commons.configuration2.PropertiesConfiguration
at org.apache.commons.configuration2.beanutils.BeanHelper.fetchBeanClass(BeanHelper.java:549)
..snip..
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.PropertiesConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.commons.lang3.ClassUtils.getClass(ClassUtils.java:909)
So, trying to lookup the PropertiesConfiguration from a class within the same jar is failing. It seems that sbt runs tasks in a classloader that does not include dependencies. This happens when I run the task from the sbt console.
commons-configuration2 is using commons-lang3 to load the class file for ConfigurationBuilder. And commons-lang is trying to use ClassLoader from the current thread to perform the loading see ClassUtils, but it seems that sbt doesn't give you any guaranties about Thread.currentThread().getContextClassLoader, see discussion for more details.
The workaround is to current thread classLoader manually, like:
Thread.currentThread().setContextClassLoader(
PluignObject.getClass.getClassLoader
)
But to be honest, I'm not sure how safe is this solution.
The workaround I managed in the end, was to invoke the task in a forked jvm process:
val runner: ScalaRun = initScoped(myTask.scopedKey, Defaults.runnerInit).value
val log: Logger = streams.value.log
val arguments = Seq()
runner.run(mainClass, classpath.map(_.data), arguments, log)
This worked for me when there was a main class available in the tool I was using.

teradata export from hive to teradata class not found

I am trying to export from hive table to teradata using TDCH connector, I am getting below errror:-
15/05/07 08:01:03 INFO tool.ConnectorExportTool: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:81)
at com.teradata.connector.common.tool.ConnectorExportTool.run(ConnectorExportTool.java:61)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
I understand from error that hive-metastore jar is missing. But it is already there in hive/lib folder
hive-metastore.jar -> hive-metastore-0.9.0.jar
Is already present in the path: /usr/hdp/2.2.4.2-2/hive/lib
You have to define environment variables LIB_JARS and HADOOP_CLASSPATH.
You can use the environment variable LIB_JARS with the parameter -libjars
You can find an example under https://developer.teradata.com/sites/all/files/Teradata%20Connector%20for%20Hadoop%20Tutorial%20v1%200%20final.pdf.
According to the README, following jars are necessairy:
Hive Job(version 0.11.0 as example):
a) hive-metastore-0.11.0.jar
b) hive-exec-0.11.0.jar
c) hive-cli-0.11.0.jar
d) libthrift-0.9.0.jar
e) libfb303-0.9.0.jar
f) jdo2-api-2.3-ec.jar
g) slf4j-api-1.6.1.jar
h) datanucleus-core-3.0.9.jar
i) datanucleus-rdbms-3.0.8.jar
j) commons-dbcp-1.4.jar
k) commons-pool-1.5.4.jar
l) antlr-runtime-3.4.jar
m) datanucleus-api-jdo-3.0.7.jar
HCatalog Job:
a) above Hive required jar files
b) hcatalog-core-0.11.0.jar
The problem is that org/apache/hadoop/hive/metastore/api/MetaException class is not available to java runtime. can you ensure that the Jar is under valid runtime classpath.
few try outs can be try passing the classpath explicitly to jvm via -cp , alternatively you can try putting the jar in launch directory to ensure its present.
Sometimes you may have all the jar files in your "lib" folder, but for some reason Oozie may not be reading it simply because you have a typo in your job.properties or coordinator.properties. Double check your properties file.
The "job.properties" file should read like the following -
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/apps/myapp/workflow/
The "coordinator.properties" file should read like the following -
oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/apps/myapp/workflow/
and your "lib" folder with jars should be at -
/apps/myapp/workflow/lib

running mahout RecommenderJob on EMR

I'm trying to run a RecommenderJob on amazon EMR. I have a jar called SmartJukebox.jar (not runnable) and it contains a class main.TrackRecommander (and that's it).
I created a job flow with the jar:
s3n://smartjukebox/SmartJukebox.jar
and args:
main.TrackRecommander --input s3n://smartjukebox/ratings.csv --output s3n://smartjukebox/output --usersFile s3n://smartjukebox/user.txt.
The class TrackRecommander uses the class RecommenderJob.
I run the job flow and i get this in the error log -
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/cf/taste/hadoop/item/RecommenderJob
at main.TrackRecommander.main(TrackRecommander.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.cf.taste.hadoop.item.RecommenderJob
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 6 more
now i see that the JVM can't find RecommenderJob and i didn't put RecommenderJob in my jar. I thought EMR would have mahout jars built in, but i can't find anything about that.
what is the solution here?
Thanks.
You're problem is exactly what you say: "I didn't put RecommenderJob in my jar." Unless you put those classes in your JAR, of course it can't be found. Why would EMR have this built in? Add the Mahout ".job" file classes to your JAR first.
You will need to create a job jar which contains all the classes required by the code to run which includes the mahout classes too.
Take a look at
https://github.com/tdunning/MiA
Check how to create a job jar using maven assembly plugin in pom.xml and the job.xml in the src/main/resources directory.
IF you exclude the hadoop classes then you can run it on any hadoop instance.

Resources