I'm trying to run a RecommenderJob on amazon EMR. I have a jar called SmartJukebox.jar (not runnable) and it contains a class main.TrackRecommander (and that's it).
I created a job flow with the jar:
s3n://smartjukebox/SmartJukebox.jar
and args:
main.TrackRecommander --input s3n://smartjukebox/ratings.csv --output s3n://smartjukebox/output --usersFile s3n://smartjukebox/user.txt.
The class TrackRecommander uses the class RecommenderJob.
I run the job flow and i get this in the error log -
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/cf/taste/hadoop/item/RecommenderJob
at main.TrackRecommander.main(TrackRecommander.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.cf.taste.hadoop.item.RecommenderJob
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 6 more
now i see that the JVM can't find RecommenderJob and i didn't put RecommenderJob in my jar. I thought EMR would have mahout jars built in, but i can't find anything about that.
what is the solution here?
Thanks.
You're problem is exactly what you say: "I didn't put RecommenderJob in my jar." Unless you put those classes in your JAR, of course it can't be found. Why would EMR have this built in? Add the Mahout ".job" file classes to your JAR first.
You will need to create a job jar which contains all the classes required by the code to run which includes the mahout classes too.
Take a look at
https://github.com/tdunning/MiA
Check how to create a job jar using maven assembly plugin in pom.xml and the job.xml in the src/main/resources directory.
IF you exclude the hadoop classes then you can run it on any hadoop instance.
Related
I have a resource bundle with language properties. If the local language is French, it loads french texts or else default English. My program works okay when using Intellij but when I create the jar file and try to run it by doing java -jar myapp.jar then it crashes and shows error.
Exception in Application start method
Exception in thread "main" java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:900)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(LauncherImpl.java:195)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javafx.fxml.LoadException:file:/C:/Users/rsoph/OneDrive/Desktop/C195new/out/artifacts/C195_Project_jar/C195_Project.jar!/view/login_welcome_screen.fxml
at javafx.fxml.FXMLLoader.constructLoadException(FXMLLoader.java:2625)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2595)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2466)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3237)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3194)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3163)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3136)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:3113)
at javafx.fxml.FXMLLoader.load(FXMLLoader.java:3106)
at main.Main.start(Main.java:33)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
... 1 more
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.sun.javafx.reflect.Trampoline.invoke(MethodUtil.java:76)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.sun.javafx.reflect.MethodUtil.invoke(MethodUtil.java:273)
at com.sun.javafx.fxml.MethodHelper.invoke(MethodHelper.java:83)
at javafx.fxml.FXMLLoader.loadImpl(FXMLLoader.java:2591)
... 17 more
Caused by: java.util.MissingResourceException: Can't find bundle for base name lng, locale en_US
at java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
at java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1556)
at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:932)
at controller.LoginWelcomeScreenController.initialize(LoginWelcomeScreenController.java:61)
... 29 more
This is how I use the resource bundle
ResourceBundle resBundle = ResourceBundle.getBundle("lng", Locale.getDefault());
if (Locale.getDefault().getLanguage().equals("fr")) {
usernameLabel.setText(resBundle.getString("username"));
passwordLabel.setText(resBundle.getString("password"));
loginButton.setText(resBundle.getString("login"));
exitButton.setText(resBundle.getString("exit"));
}
My program structure looks like this.
EDIT: This is how I created the jar executable file.
File -> Project Structure -> Artifacts -> click on the plus sign (+) -> Jar -> From modules with dependencies.
Select the main class and OK. Followed by Clicking on the + sign and adding all files from the javafx bin folder to the File ( all the dll files).
Go to Build -> Build artifacts -> Build.
The bundle name you are looking up is "lng", but your the files in your bundle have prefixes of "Lng".
The case of the resource file name and the bundle lookup string should match.
If you are using a Windows system, the lookup on the file name is case-insensitive when using the file protocol to access the unpackaged resources.
But, when packaged in a jar, the lookup of the resources using the jar protocol is case sensitive.
Which explains why you can find the resources when not packaged in a jar, and why the lookup fails after jar packaging.
In addition to the issue with the jar packaging, relying on case insensitive file names also means that the project would fail if used on an OS with a case sensitive file system, like a Mac or Linux system.
When I build the sample corda project with gradle on windows (gradlew.bat deployNodes)
I see the error about DELETE command in log4j:
2019-10-28 16:30:47,207 main ERROR Delete contains invalid attributes "IfFileName", "IfLastModified"
2019-10-28 16:30:47,213 main ERROR Missing Delete conditions: unconditional Delete not supported
2019-10-28 16:30:47,217 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.rolling.action.DeleteAction for element Delete: java.lang.IllegalArgume
ntException: Unconditional Delete not supported java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:76)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:121)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at net.corda.core.utilities.KotlinUtilsKt.contextLogger(KotlinUtils.kt:42)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:108)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:107)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at net.corda.cliutils.CliWrapperBase$Companion.getLogger(CordaCliWrapper.kt:65535)
at net.corda.cliutils.CliWrapperBase$Companion.access$getLogger$p(CordaCliWrapper.kt:107)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:142)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:106)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73)
at net.corda.node.Corda.main(Corda.kt:13)
Caused by: java.lang.IllegalArgumentException: Unconditional Delete not supported
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.(DeleteAction.java:71)
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.createDeleteAction(DeleteAction.java:212)
... 43 more
2019-10-28 16:30:47,606 main ERROR Null object returned for Delete in DefaultRolloverStrategy.
2019-10-28 16:30:47,736 main ERROR Delete contains invalid attributes "IfFileName", "IfLastModified"
2019-10-28 16:30:47,737 main ERROR Missing Delete conditions: unconditional Delete not supported
2019-10-28 16:30:47,739 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.rolling.action.DeleteAction for element Delete: java.lang.IllegalArgume
ntException: Unconditional Delete not supported java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:964)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:904)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:896)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:548)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:620)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:637)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231)
at org.apache.logging.log4j.core.async.AsyncLoggerContext.start(AsyncLoggerContext.java:76)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:121)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at net.corda.core.utilities.KotlinUtilsKt.contextLogger(KotlinUtils.kt:42)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:108)
at net.corda.cliutils.CliWrapperBase$Companion$logger$2.invoke(CordaCliWrapper.kt:107)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at net.corda.cliutils.CliWrapperBase$Companion.getLogger(CordaCliWrapper.kt:65535)
at net.corda.cliutils.CliWrapperBase$Companion.access$getLogger$p(CordaCliWrapper.kt:107)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:142)
at net.corda.cliutils.CliWrapperBase.call(CordaCliWrapper.kt:106)
at picocli.CommandLine.execute(CommandLine.java:1173)
at picocli.CommandLine.access$800(CommandLine.java:141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
at net.corda.cliutils.CordaCliWrapperKt.start(CordaCliWrapper.kt:73)
at net.corda.node.Corda.main(Corda.kt:13)
Caused by: java.lang.IllegalArgumentException: Unconditional Delete not supported
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.(DeleteAction.java:71)
at org.apache.logging.log4j.core.appender.rolling.action.DeleteAction.createDeleteAction(DeleteAction.java:212)
... 43 more
I got latest verison of corda samples. (from master branch). I encounter the same problem with other branches.
My jdk version is 1.8.0_212.
I used cordapp-examples project
How can I solve this problem?
It looks to me like you are pulling in JAR files from somewhere other than those specified by the Corda build.
The default branch for the samples is release-V4 and that is the one I'm using. I run the following and it works fine:
git clone https://github.com/corda/samples
cd cordapp-example
gradlew deployNodes
I'd check that your classpath variable isn't set (as this will cause additional JAR files in those directories to be included when running Corda) by running :
echo %CLASSPATH%
Additionally, from the command-line I'd run:
java -version
where java
To make sure the version of java you think you're picking up is the one you are picking up.
I had a similar issue related to log4j:
the log4j2 error unable to invoke factory method in class class RollingFileApender
Error while generating node info file
To fix this (on windows), I took the following steps:
Right click on MyPC
Click advanced system settings
Click environment variables
Change JAVA_HOME to point to same jdk directory being used in IntelliJ for corDapp. (I changed mine from OpenJDK\jdk-13.0.1 to C:\Program Files\Java\jdk1.8.0_231)
Restart IntelliJ and rerun ./gradlew deployNodes
The problem was about the regional settings of my computer.
By defining an Environment variable
JAVA_TOOL_OPTIONS=-Duser.language=en -Duser.country=US -Duser.region=US -Dfile.encoding=UTF-8 -Duser.variant=US
I solved the problem.
Thanks all for your attention.
I've been using jxbrpowser 6.2 with a trial license. Trying to load it in OSGi enviroment generates the following error:
Exception in thread "JavaFX Application Thread" java.lang.ExceptionInInitializerError
at com.teamdev.jxbrowser.chromium.internal.ipc.IPC.getDefault(Unknown Source)
at com.teamdev.jxbrowser.chromium.BrowserContext.<init>(Unknown Source)
(...)
at com.sun.glass.ui.win.WinApplication$$Lambda$39/30523883.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.teamdev.jxbrowser.chromium.internal.ChromiumExtractorException: Failed to extract chromium binaries into C:\Users\Vladimir\AppData\Local\Temp\jxbrowser-chromium-43.0.2357.52.6.2
at com.teamdev.jxbrowser.chromium.internal.ChromiumExtractor.extract(Unknown Source)
at com.teamdev.jxbrowser.chromium.internal.ipc.IPC.<init>(Unknown Source)
at com.teamdev.jxbrowser.chromium.internal.ipc.IPC.create(Unknown Source)
at com.teamdev.jxbrowser.chromium.internal.ipc.IPC$a.<clinit>(Unknown Source)
... 15 more
Caused by: java.lang.IllegalArgumentException: The /chromium-windows.zip resource cannot be found in JAR files.
at com.teamdev.jxbrowser.chromium.internal.ZIPExtractor.extract(Unknown Source)
at com.teamdev.jxbrowser.chromium.internal.e.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
... 19 more
I can successfully load jxbrowser in the non-OSGi environment, as well as in OSGi environment when resources are already extracted.
Is there any way to make jxbrowser successfully extracts the resources from the jar?
It is looking like you don't have a correct OSGI bundle for JxBrowser. As far as I know jxbrowser supports OSGI bundle distribution, you can simply install it into your OSGI container.
Alternatively you can create your own bundle with which is only exports JxBrowser public API, but contains JxBrowser libraries you need. You can automate it with Felix bundle plugin for example http://felix.apache.org/documentation/subprojects/apache-felix-maven-bundle-plugin-bnd.html
In order to make such a bundle working you will need to initialize an temporary browser instance (just create a browser component) inside bundle activator. So that chromium binaries will be extracted when bundle is installed in your OSGi container.
I am trying to export from hive table to teradata using TDCH connector, I am getting below errror:-
15/05/07 08:01:03 INFO tool.ConnectorExportTool: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at com.teradata.connector.common.tool.ConnectorJobRunner.runJob(ConnectorJobRunner.java:81)
at com.teradata.connector.common.tool.ConnectorExportTool.run(ConnectorExportTool.java:61)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.teradata.hadoop.tool.TeradataExportTool.main(TeradataExportTool.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
I understand from error that hive-metastore jar is missing. But it is already there in hive/lib folder
hive-metastore.jar -> hive-metastore-0.9.0.jar
Is already present in the path: /usr/hdp/2.2.4.2-2/hive/lib
You have to define environment variables LIB_JARS and HADOOP_CLASSPATH.
You can use the environment variable LIB_JARS with the parameter -libjars
You can find an example under https://developer.teradata.com/sites/all/files/Teradata%20Connector%20for%20Hadoop%20Tutorial%20v1%200%20final.pdf.
According to the README, following jars are necessairy:
Hive Job(version 0.11.0 as example):
a) hive-metastore-0.11.0.jar
b) hive-exec-0.11.0.jar
c) hive-cli-0.11.0.jar
d) libthrift-0.9.0.jar
e) libfb303-0.9.0.jar
f) jdo2-api-2.3-ec.jar
g) slf4j-api-1.6.1.jar
h) datanucleus-core-3.0.9.jar
i) datanucleus-rdbms-3.0.8.jar
j) commons-dbcp-1.4.jar
k) commons-pool-1.5.4.jar
l) antlr-runtime-3.4.jar
m) datanucleus-api-jdo-3.0.7.jar
HCatalog Job:
a) above Hive required jar files
b) hcatalog-core-0.11.0.jar
The problem is that org/apache/hadoop/hive/metastore/api/MetaException class is not available to java runtime. can you ensure that the Jar is under valid runtime classpath.
few try outs can be try passing the classpath explicitly to jvm via -cp , alternatively you can try putting the jar in launch directory to ensure its present.
Sometimes you may have all the jar files in your "lib" folder, but for some reason Oozie may not be reading it simply because you have a typo in your job.properties or coordinator.properties. Double check your properties file.
The "job.properties" file should read like the following -
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/apps/myapp/workflow/
The "coordinator.properties" file should read like the following -
oozie.use.system.libpath=true
oozie.coord.application.path=${nameNode}/apps/myapp/workflow/
and your "lib" folder with jars should be at -
/apps/myapp/workflow/lib
I'm having trouble running a custom jar on Elastic Map-Reduce
I'm using jdk1.6.0_26, Hadoop 0.20.205, and compiling with Eclipse on my computer and everything works perfectly fine
for example if I ran the following on my computer it would be successful
hadoop jar MaxTemperature.jar input/temperature.txt output
I specified the jar as the following on AWS
s3n://chrishadoop/MaxTemperature.jar
and I specified the arguments as
s3n://chrishadoop/input/temperature.txt s3n://chrishadoop/output
I did not specify the main class because I pointed to it in the manifest
Here is the jar I'm using, I will make it public for a little while
https://s3.amazonaws.com/chrishadoop/MaxTemperature.jar
Here is the error I'm getting
2012-07-08 19:31:39,824 INFO com.amazonaws.elasticmapreduce.statepusher.StatePusher (main): Pusher awoke, starting to push data into simpledb...
2012-07-08 19:31:40,552 FATAL com.amazonaws.elasticmapreduce.statepusher.StatePusher (main): Fatal Exception raised while extracting data from hadoop and pushing to simpledb
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
at com.amazonaws.elasticmapreduce.statepusher.StatePusher.run(StatePusher.java:65)
at com.amazonaws.elasticmapreduce.statepusher.StatePusher.main(StatePusher.java:205)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 2 more
There is a version of Jackson which is installed as part of the AMI, and I'm guessing you're bundling a different version of Jackson? The error seems to be happening in the support code which makes "enable debugging" work.