Livy-Upload JAR in Driver classpath - livy

Question

I am unable to create livy interactive session with a dependent JAR required in driver-classpath, with the following command:
curl -H "Content-Type: application/json" -X POST -d '{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}' -i http://<LIVY_SERVER_IP:PORT>/sessions
The JAR file here is present on local driver Path. Also tried using HDFS path in the following manner hdfs://<NM_IP>:<NM_Port>/data/XXX-0.0.1-SNAPSHOT.jar
Following are the livy server logs when trying to create interactive session
19/08/12 17:26:56 INFO sessions.InteractiveSessionManager: Registering new session 0
19/08/12 17:26:58 INFO utils.LineBufferedStream: Exception in thread "main" java.lang.NullPointerException
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.livy.rsc.driver.JobWrapper.cancel(JobWrapper.java:90)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriver.shutdown(RSCDriver.java:127)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriver.run(RSCDriver.java:356)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.livy.rsc.driver.RSCDriverBootstrapper.main(RSCDriverBootstrapper.java:93)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at java.lang.reflect.Method.invoke(Method.java:498)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
19/08/12 17:26:58 INFO utils.LineBufferedStream: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
However, when the same JAR is being used in running a python job with batch livy session, it gets successfully completed. Below mentioned is the command:
curl -H "Content-Type: application/json" -X POST --data '{"file": "/data/test.py", "conf": {"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}' http://<LIVY_SERVER_IP:PORT>/batches
The python file /data/test.py is present on HDFS path.
I have tried whitelisting the directory path via modifying livy.conf file with the following changes livy.file.local-dir-whitelist = /data/

Answer 1

Livy Interactive session is 2 Step process. The files need to be there is HDFS while creating the session.
Create an Interactive session
Use the session ID to execute the statements.
curl -X POST --data '{"kind": "pyspark", "pyFiles" : ["/some hdfs location/splitter.py"]}' -H "Content-Type: application/json" -H "X-Requested-By: root"http://localhost:8999/sessions
Please refer:: https://livy.apache.org/examples/
https://community.hortonworks.com/questions/191738/invoke-livy-with-pyfiles-attribute.html