I'm trying to run spark streaming job on DC/OS platform and I've got issue with kafka packages. When I'm trying to include Kafka library and its dependencies (jar file downloaded from Maven, added to artifactory and read from there) with the use of --jars mode as follows:
dcos spark run --submit-args"--jars https://../../../spark-streaming 2.11-2.2.1.jar --conf spark.executor.memory=2g --py-files=https://../../../libs.zip,https://../../../test.py etc"
it seems that file libs.zip, test.py are correctly read but .jar file is omitted.
Any idea why? Is there any workaround for this kind of issue?
Thanks in advance for any help!
I'm not sure why the dcos spark submit command doesn't support --jar option, but you can use the spark.mesos.uris property to download artifacts to the working directory of a Spark driver and executor.
I'm not sure how your Python-based Spark job is going to use JARs, but you may need setting the spark.executor.extraClassPath and spark.driver.extraClassPath configuration property as well.
Related
I want to run edumips.jar. Whenever I try to open, Java control panel is opened.
When I execute:
strong text
It is first time to download java and excecute .jar file.
What I want:
Please tell me what I'm doing wrong...
UPDATE: there is now an MSI installer for EduMIPS64 which allows you to install and run the simulator without worrying about installing a JRE and configuring it: https://github.com/EduMIPS64/edumips64/releases/download/v1.2.8/EduMIPS64-1.2.8.msi (version 1.2.8 was released at the time of updating this answer, there might be more recent ones).
You need to make sure the Java runtime is associated with the .jar extension to open it with double-click.
To open it once, try right-clicking on the JAR file select Open With... and find the Java runtime binary (it's "OpenJDK Platform Binary" on my Windows machine).
You may also be able to fix this by re-installing the Java Runtime (JRE): usually installers associate .jar files to the Java binary automatically.
Worst case, if Java is installed and the PATH is set up correctly, you can run it from the command line by using the command java -jar edumips64-1.2.6-standalone.jar.
To open the command line, press Win+x and choose Windows PowerShell. Your JAR might be in the Downloads directory so try java -jar Downloads\edumips64-1.2.6-standalone.jar.
I am following documentation in here :https://docs.corda.net/network-bootstrapper.html to bootstrap test network. In the section: "Providing CorDapps to the Network Bootstrapper", we are asked to place cordapp jar along with the conf files.
i run the command below:
java -jar corda-network-bootstrapper-3.2-corda-executable.jar "."
After, I see message Bootstrapping complete!
But, when i go into the node folders, none of them have the cordapp in their directories ? How do i know that cordapp is installed ?
Also, another issue with the command is I cant provide the directory using --dir parameter.
Can you please suggest any work around for these issues ?
Thanks.
Update
Below are the screenshots for commands in Mac OSX:
This is my folder structure:
Error I am getting after I execute the command:
Your jar should be present inside your node's root/cordapps directory. If it's not, than bootstrapper is unable to identify your cordapp. Try adding --verbose and it should print something like "Found the following CorDapps: "
Also when you start the corda nodes, CLI will show all installed cordApps.
For your dir issue, please add the error message here and the type of your operating system. It's working fine here on Linux.
I have installed Cloudera CDH QuickStart VM 5.5, and I'm running a Sqoop action in my Oozie workflow. I encountered an error that says MySQL JDBC driver is missing and I came across to a SO answer here that says the mysql-connector-java.jar should be placed in Oozie's HDFS shared lib path, under sqoop path.
When I browse the Oozie's HDFS shared lib path, however, I've noticed two sqoop subdirectories to copy the jar.
/user/oozie/share/lib/sqoop
and
/user/oozie/share/lib/lib_20151118030154/sqoop
Aside from sqoop, hive, pig, distcp, and mapreduce-streaming paths also exist on both lib and lib/lib_20151118030154.
So the question is: where do I place my connector jar: on the first or the second one?
What's the difference (or difference of purpose) of these two paths in relation to jars of sqoop, hive, pig, distcp, and mapreduce-streaming for Oozie?
The lib_20151118030154 sub-dir would be the current version of the ShareLibs, as of 18-NOV-2015. The versioning allows you to make updates without stopping the Oozie service -- check the documentation here.
In other words: the Oozie service keeps in memory a list of the JARs in each ShareLib (based on what was present for the latest version at boot time), so that adding a JAR will not make a difference until (a) you stop/restart the service or (b) you resync the service as explained in the doc above.
I have developed a TideSDK application and am now ready to package it, but I'm having problems with the network type installer.
It always gives me code 404 on the Application first run:
Could not query info: Invalid HTTP Status Code (404)
I presume the installer is having difficulty with reaching the correct servers and downloading the needed runtime, but I have run through most solutions on this forum, and none have worked.
So I tried a bundle packaging, as it should include such runtime, but I must be doing something wrong, since it does not bundle within the MSI.
The code I'm executing is as follows:
C:\TideSDK\sdk\win32\1.2.0.RC6d\tibuild.py -p --type=BUNDLE --os=win32 "C:\path_to_app\app_dir"
I also tried:
C:\TideSDK\sdk\win32\1.2.0.RC6d\tibuild.py -p -t bundle --os=win32 "C:\path_to_app\app_dir"
And all the uppercase/lowercase combinations. Also tried version 1.2.0.4, without sucess. Am I doing something wrong?
the network type installer is not available anymore, since appcelerator has canceled their services for titanium desktop.
So you can only do bundle packaging. Try the following command:
python tibuild.py --dest=. --type=bundle --package=. "c:\path\to\your\app\dir"
This should build and package your app and create a installer for it.
Change "dest" and "package" to the directories where you want to have the built app and installation package.
You can omit the OS parameter, since the builder can only generate builds for the current OS.
I am having the same issue described in this post on the py2app mailing list.
I have a python application that uses a sqlite database. On my machine, which has all the dependencies installed, there are no issues. However, when I bundle the application with py2app, clicking a menu that causes the database to be accessed results in this error:
Database error: Driver not loaded Driver not loaded
For the Windows installer, the files in \Qt\version\plugins\sqldrivers\*.* can be copied to \myApp\sqldrivers\*
The same files on the Mac can be found in　/opt/local/share/qt4/plugins/sqldrivers (installed via Macports).
However, copying the sqldrivers directory to my application's Resources or Frameworks directories still results in the same error.
How can I add sqlite support into my application that is built using py2app?
Turns out the pyside recipe does have a way to specify which qt-plugins you need...
options=dict(py2app={
'argv_emulation': True,
'qt_plugins' : "sqldrivers",
}
),
This puts all the sqldrivers into the right directory and setups qt.conf correctly.
have you tried what he said
in this post ?
py2app setup.py usage question
it mentioned
you need to include the sqlalchemy.dialects.sqlite as a package
I managed to get this to work as follows:
After building with py2app, inside the application's Contents directory, make a new plugins directory.
Then copy sqldrivers/libqsqlite.dylib into this plugins directory.
Afterwards, install_name_tool has to be used to change the library links in libqsqlite.dylib to point to the Qt libraries in the application's Frameworks directory rather than the system Qt libraries.