I have installed Cloudera CDH QuickStart VM 5.5, and I'm running a Sqoop action in my Oozie workflow. I encountered an error that says MySQL JDBC driver is missing and I came across to a SO answer here that says the mysql-connector-java.jar should be placed in Oozie's HDFS shared lib path, under sqoop path.
When I browse the Oozie's HDFS shared lib path, however, I've noticed two sqoop subdirectories to copy the jar.
/user/oozie/share/lib/sqoop
and
/user/oozie/share/lib/lib_20151118030154/sqoop
Aside from sqoop, hive, pig, distcp, and mapreduce-streaming paths also exist on both lib and lib/lib_20151118030154.
So the question is: where do I place my connector jar: on the first or the second one?
What's the difference (or difference of purpose) of these two paths in relation to jars of sqoop, hive, pig, distcp, and mapreduce-streaming for Oozie?
The lib_20151118030154 sub-dir would be the current version of the ShareLibs, as of 18-NOV-2015. The versioning allows you to make updates without stopping the Oozie service -- check the documentation here.
In other words: the Oozie service keeps in memory a list of the JARs in each ShareLib (based on what was present for the latest version at boot time), so that adding a JAR will not make a difference until (a) you stop/restart the service or (b) you resync the service as explained in the doc above.
Related
I am have this Exception please help me!
"Error occurred during initialization of boot layer
java.lang.module.Find Exception: Module test not found"
But i write VM option "--module-path "D:\UT java\javafx-sdk-17.0.1\lib" --add-modules javafx.controls,javafx.fxml"
and i have module-info.java "
requires javafx.fxml;
requires javafx.controls;
requires javafx.graphics;
requires java.sql;
requires java.desktop;
requires jdk.jfr;"
i add my sdk. And if i create javafx demo project and execute him it work. and if i start change fxml file and change controller i have this exception.
I have IntellIJIdea 2021, javafx-sdk-17.0.1, jdbc jr 8,11,16
Steps to fix:
Delete the JavaFX sdk (you don’t need it).
Delete old Java versions (they are obsolete).
Update your IntelliJ IDE and IDE plugins to the most recent release, 2021.3.2+.
Create a new JavaFX project using JDK and JavaFX 17.0.2+.
Select Maven for the build system unless you know and prefer Gradle.
Do not set VM arguments, you don’t need them.
Adding modules via the --add-modules VM arguments is unnecessary when you have a valid module-info.java file.
The --module-path is still required so that the modules can be found, but Idea will provide the correct path for your modules automatically when it recognizes the modules through your Maven dependencies.
So you don't need to explicitly define the --module-path VM argument yourself for a Maven based build (that would be difficult to do anyway because the modules are all downloaded to different directories in your local maven repository).
Test it works following the Idea create new JavaFX project execution instructions.
Add additional modules one at a time by adding their maven dependency to pom.xml and the requires clause to module-info.java.
Ensure you synchronize the Maven and Idea projects between each
addition.
See, for example, this question on correctly adding the javafx.media module.
Adding other modules such as javafx.web, javafx.fxml or javafx.swing follows a similar pattern.
Test between each addition by building and running the project, to ensure you haven’t broken anything.
Copy your original source code into the appropriate package directories under the new project source directory:
src/main/java
Place resources in:
src/main/resources
following the Eden resource location guide.
Fix any errors, ensure everything compiles and runs, then test it.
We are using Corda Version 4 for our application.
We understand that the command gradlew.bat deployNodes creates following jars -
CorDapp (contracts, states, flows)
Corda platform
Dependencies
When any change is made in the contract/states/flows code, we had to run the command gradlew.bat deployNodes each time. Due to this the "Corda platform" and "Dependencies" jars always get recreated and consequently increases development time.
Does Corda platform provides alternative way to Only create "CorDapp" jar file and not the remaining ones?
You can use following cmd to generate only jar files
./gradlew build
this will generate jar files in you build/lib folder
I'm trying to run spark streaming job on DC/OS platform and I've got issue with kafka packages. When I'm trying to include Kafka library and its dependencies (jar file downloaded from Maven, added to artifactory and read from there) with the use of --jars mode as follows:
dcos spark run --submit-args"--jars https://../../../spark-streaming 2.11-2.2.1.jar --conf spark.executor.memory=2g --py-files=https://../../../libs.zip,https://../../../test.py etc"
it seems that file libs.zip, test.py are correctly read but .jar file is omitted.
Any idea why? Is there any workaround for this kind of issue?
Thanks in advance for any help!
I'm not sure why the dcos spark submit command doesn't support --jar option, but you can use the spark.mesos.uris property to download artifacts to the working directory of a Spark driver and executor.
I'm not sure how your Python-based Spark job is going to use JARs, but you may need setting the spark.executor.extraClassPath and spark.driver.extraClassPath configuration property as well.
I checked https://docs.corda.net/deploying-a-node.html for deploying in windows server. I can see deploying nodes using NSSM Manager.
When I deploy nodes, how it will access my application which is placed as a jar under /opt/corda /CordaApp.jar
Also, When I run nssm.bat file under each nodes, my cmd is going on running with the first cmd and not stopping. Nothing proceed after that.
There is a typo in the docs. Where it says:
Create a directory called plugins in /opt/corda and save your CorDapp jar file to it. Alternatively, download one of our sample CorDapps to the plugins directory
It should read instead:
Create a directory called plugins in C:\Corda\ and save your CorDapp jar file to it. Alternatively, download one of our sample CorDapps to the plugins directory
This was fixed by the following PR: https://github.com/corda/corda/pull/2607.
I have a project where I need to run javaee7 microservice with payara micro.
For this reason, I build a payara uber jar with maven, so I can just run it as
java -jar MicroService.jar
(http://blog.payara.fish/creating-uber-jar-with-payara-micro )
How can I set the logging level for payara, so that only warnings and errors are displayed, without INFO?
You can specify an alternative logging.properties file using --logProperties option followed by path to the file. This is not documented in the official documentation, but you can get a brief info about it when you execute java -jar payara-micro.jar --help.
You can find a template logging.properties file when you create a domain directory by running a plain Payara Micro with the --rootDir argument - it will copy all documentation from the JAR file into the specified folder (the folder must exist - might be empty). You can find logging.properties in config/logging.properties, take it, edit it, and pass it to Payara Micro using the --logProperties argument.
If you want to specify log levels within the uber JAR, without adding a separate logging.properties file, you may configure the logging either by running appropriate asadmin commands from within your application or directly setting log level on JUL loggers using LogManager API.