Including external jars after launching Pyspark in Jupyter notebook - jar

I have been trying in vain to include external jars into pyspark/Jupyter notebook env after the notebook has been launched. (Alternatively, the jars are included successfully either with:
1. Use --jar ojdbc6.jar
2. include the jar in spark-defaults.conf )
I have tried below all failed:
Idea 1:
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession \
.builder \
.appName("testjars") \
.config("spark.jars", "C:\\Users\\U620311\\ojdbc6.jar") \
.config("spark.driver.extraClassPath", "C:\\Users\\U620311\\ojdbc6.jar") \
.config("spark.executor.extraClassPath", "C:\\Users\\U620311\\ojdbc6.jar") \
.getOrCreate()
spark.sparkContext._conf.getAll()
Invoke jar file as below:
DB_USER = ""
DB_PASS = ""
DB_DRIVER = "oracle.jdbc.driver.OracleDriver"
JDBC_URL = "<connection string>"
query = '''(select * from <tablename>) t1'''
ojdbcDF = (sqlContext.read.format("jdbc").option("driver", DB_DRIVER).option("url", JDBC_URL).option("dbtable", query).option("user", DB_USER).option("password", DB_PASS).load())
Idea 2:
I have played with all permutations. Include extraclasspath but not spark.jars and vice versa. no success.
Idea 3:
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars ojdbc6.jar pyspark-shell'
import pyspark
sc.stop()
sc = pyspark.SparkContext()
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
DB_USER = "<uid>"
DB_PASS = "<pwd>"
DB_DRIVER = "oracle.jdbc.driver.OracleDriver"
JDBC_URL = "<connection string>"
query = '''(select * from <tablename>) t1'''
ojdbcDF = (sqlContext.read.format("jdbc").option("driver", DB_DRIVER).option("url", JDBC_URL).option("dbtable", query).option("user", DB_USER).option("password", DB_PASS).load())
All of the above lead to the same error message. Would you please help in identifying if there is missing configuration or missing code. Please suggest an alternatives to try.
Py4JJavaError: An error occurred while calling o373.load.
: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:38)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:78)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$$anonfun$6.apply(JDBCOptions.scala:78)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:78)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:34)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:330)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)

Related

Time Failure Issue - Katalon

I found on the internet a solution for Time failure issue(Without adding any random delay), but anytime I add it to my code got an error.
Here is the error:
Test Cases/Logging into account/Logging in - Right details FAILED because (of) (Stack trace: groovy.lang.MissingPropertyException: No such property: G_Timeout_Small for class: internal.GlobalVariable
at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1004)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1859)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1835)
at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3735)
at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
at Script1543314158625.run(Script1543314158625.groovy:39)
at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:183)
at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:108)
at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:294)
at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:285)
at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:264)
at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:256)
at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:200)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:99)
at com.kms.katalon.core.main.TestCaseMain.runTestCase(TestCaseMain.java:90)
at com.kms.katalon.core.main.TestCaseMain$runTestCase$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:149)
at TempTestCase1543410680790.run(TempTestCase1543410680790.groovy:22)
at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:263)
at groovy.lang.GroovyShell.run(GroovyShell.java:518)
at groovy.lang.GroovyShell.run(GroovyShell.java:507)
at groovy.ui.GroovyMain.processOnce(GroovyMain.java:653)
at groovy.ui.GroovyMain.run(GroovyMain.java:384)
at groovy.ui.GroovyMain.process(GroovyMain.java:370)
at groovy.ui.GroovyMain.processArgs(GroovyMain.java:129)
at groovy.ui.GroovyMain.main(GroovyMain.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
)
Test Cases/Logging into account/Logging in - Right details.run:39
Here is my code:
import static com.kms.katalon.core.checkpoint.CheckpointFactory.findCheckpoint
import static com.kms.katalon.core.testcase.TestCaseFactory.findTestCase
import static com.kms.katalon.core.testdata.TestDataFactory.findTestData
import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject
import com.kms.katalon.core.checkpoint.Checkpoint as Checkpoint
import com.kms.katalon.core.cucumber.keyword.CucumberBuiltinKeywords as CucumberKW
import com.kms.katalon.core.mobile.keyword.MobileBuiltInKeywords as Mobile
import com.kms.katalon.core.model.FailureHandling as FailureHandling
import com.kms.katalon.core.testcase.TestCase as TestCase
import com.kms.katalon.core.testdata.TestData as TestData
import com.kms.katalon.core.testobject.TestObject as TestObject
import com.kms.katalon.core.webservice.keyword.WSBuiltInKeywords as WS
import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI
import internal.GlobalVariable as GlobalVariable
'Opening the browser.'
WebUI.openBrowser('')
'Importing account details from file.'
def plik = new File('Config.txt')
'Importing account details from file.'
def lines = plik.readLines()
WebUI.waitForPageLoad(0)
'Loading the page.'
WebUI.navigateToUrl(lines[0])
'Typing the account name.\n'
WebUI.setText(findTestObject('Logging into the account/account area'), lines[1])
'Typing the password'
WebUI.setText(findTestObject('Logging into the account/password area'), lines[2])
'Logging into the website with the logging button'
WebUI.click(findTestObject('Logging into the account/log button'), FailureHandling.STOP_ON_FAILURE)
WebUI.waitForPageLoad(GlobalVariable.G_Timeout_Small)
'Verifying wether logged or not.'
WebUI.verifyElementPresent(findTestObject('Logging into the account/main div after being logged'), 0)
'Closing the browser.'
WebUI.closeBrowser()
and place where I found it:
https://docs.katalon.com/katalon-studio/tutorials/solving_common_issue_wait_time.html
Anyone has any idea why there is that error ? In advance "Yes there is importe library for Global Variable.
You need to set initial value for GlobalVariable.G_Timeout_Small.
Go to execution profiles ("Profiles" at the top of the "Tests Explorer" on the right side of Katalon Sudio window) and set G_Timeout_Small to initial (number) value.

How to write to JDBC source with SparkR 1.6.0?

With SparkR 1.6.0 I can read from a JDBC source with the following code,
jdbc_url <- "jdbc:mysql://localhost:3306/dashboard?user=<username>&password=<password>"
df <- sqlContext %>%
loadDF(source = "jdbc",
url = jdbc_url,
driver = "com.mysql.jdbc.Driver",
dbtable = "db.table_name")
But after performing a calculation, when I try to write the data back to the database I've hit a roadblock as attempting...
write.df(df = df,
path = "NULL",
source = "jdbc",
url = jdbc_url,
driver = "com.mysql.jdbc.Driver",
dbtable = "db.table_name",
mode = "append")
...returns...
ERROR RBackendHandler: save on 55 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
java.lang.RuntimeException: org.apache.spark.sql.execution.datasources.jdbc.DefaultSource does not allow create table as select.
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:259)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2066)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141)
at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86)
at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38)
at io.netty.channel.SimpleChannelIn
Looking around the web I found this which tells me that a patch for this error was included as of version 2.0.0; and we also get the functions read.jdbc and write.jdbc.
For this question, though, assume I'm stuck with SparkR v1.6.0. Is there a way to write to JDBC sources (i.e. is there a workaround that would allow me to use DataFrameWriter.jdbc() from SparkR)?
The short answer is, no, the JDBC write method was not supported by SparkR until version 2.0.0.

Load .RData from an R script in Scala using Renjin

I have an R script load.R that attempts to do
load("test.RData")
When I run:
engine.eval(new FileReader("load.R"))
An exception occurs:
Exception in thread "main" java.lang.NoClassDefFoundError: org/tukaani/xz/XZInputStream
at org.renjin.primitives.io.connections.Connections.gzfile(Connections.java:79)
at org.renjin.primitives.R$primitive$gzfile.doApply(R$primitive$gzfile.java:75)
at org.renjin.primitives.R$primitive$gzfile.apply(R$primitive$gzfile.java:36)
at org.renjin.primitives.special.InternalFunction.apply(InternalFunction.java:45)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.eval.Context.evaluate(Context.java:163)
at org.renjin.sexp.Closure.doApply(Closure.java:83)
at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:73)
at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
at org.renjin.sexp.Closure.apply(Closure.java:78)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.primitives.special.AssignLeftFunction.assignLeft(AssignLeftFunction.java:57)
at org.renjin.primitives.special.AssignLeftFunction.apply(AssignLeftFunction.java:41)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.primitives.special.BeginFunction.apply(BeginFunction.java:38)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.primitives.special.IfFunction.apply(IfFunction.java:40)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.primitives.special.BeginFunction.apply(BeginFunction.java:38)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.eval.Context.evaluate(Context.java:163)
at org.renjin.sexp.Closure.doApply(Closure.java:83)
at org.renjin.eval.ClosureDispatcher.apply(ClosureDispatcher.java:73)
at org.renjin.eval.ClosureDispatcher.applyClosure(ClosureDispatcher.java:58)
at org.renjin.sexp.Closure.apply(Closure.java:78)
at org.renjin.eval.Context.evaluateCall(Context.java:302)
at org.renjin.eval.Context.evaluate(Context.java:222)
at org.renjin.eval.Context.evaluateExpressionVector(Context.java:285)
at org.renjin.eval.Context.evaluate(Context.java:220)
at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:145)
at org.renjin.script.RenjinScriptEngine.eval(RenjinScriptEngine.java:104)
at com.infosistema.imp.restservice.data_treatment.DataTreatment.<init>(DataTreatment.scala:34)
at com.infosistema.imp.restservice.data_treatment.Main$.<init>(DataTreatment.scala:45)
at com.infosistema.imp.restservice.data_treatment.Main$.<clinit>(DataTreatment.scala)
at com.infosistema.imp.restservice.data_treatment.Main.main(DataTreatment.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.ClassNotFoundException: org.tukaani.xz.XZInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 46 more
I don't know R. I'm merely attempting to use R code to process some data in a Java/Scala application using Renjin.
Reading the data section on the documention nothing is said about loading data from .RData files.
UPDATED:
In my build.sbt file I have:
resolvers += "BeDataDriven" at https://nexus.bedatadriven.com/content/groups/public",
libraryDependencies += "org.renjin" % "renjin-script-engine" % "0.8.1886"
It is not enough to just have renjin-script-engine.jar on your classpath, you need its dependencies as well.
If you are using SBT to build your Scala project, then you need to add the following to your build file:
resolvers +=
"BeDataDriven" at "https://nexus.bedatadriven.com/content/groups/public"
lazy val root = (project in file(".")).
settings(
name := "renjin-test",
version := "1.0",
scalaVersion := "2.10.6",
libraryDependencies += "org.renjin" % "renjin-script-engine" % "0.8.1886"
)
For other build tools, see:
http://docs.renjin.org/en/latest/library/project-setup.html
The problem was coursier not being able to resolve all dependencies (without any notice).
Removing addSbtPlugin("io.get-coursier" % "sbt-coursier" % "1.0.0-M15") from plugins.sbt and using sbt's default resolver fixed the issue.

Invoking R scripts from Spark Java on Windows fails with Cannot run program

Application fails with java.io.IOException: Cannot run program
"C:\Users..\Temp\spark-70bf05bd-6b8a-4c9f-b256-edbfb5f687b7\userFiles-cadcc324-4e7f-4952-8ec2-f442c7bd1cf8\finddistance.R":
CreateProcess error=193, %1 is not a valid Win32 application
Note: This is an example from Orielly's "Learning Spark" chapter 6 from https://github.com/databricks/learning-spark/blob/master/src/main/java/com/oreilly/learningsparkexamples/java/ChapterSixExample.java
Spark Java Code invoking R script
Example 6-18. Driver program using pipe() to call finddistance.R in Java
//JavaPairRDD<String, CallLog[]> contactsContactLists;
// Compute the distance of each call using an external R program
// adds our script to a list of files for each node to download with this job
String distScript = "./src/R/finddistance.R";
String distScriptName = "finddistance.R";
sc.addFile(distScript);
JavaRDD<String> pipeInputs = contactsContactLists.values().map(new VerifyCallLogs()).flatMap(
new FlatMapFunction<CallLog[], String>() { public Iterable<String> call(CallLog[] calls) {
ArrayList<String> latLons = new ArrayList<String>();
for (CallLog call: calls) {
latLons.add(call.mylat + "," + call.mylong +
"," + call.contactlat + "," + call.contactlong);
}
return latLons;
}
});
JavaRDD<String> distances = pipeInputs.pipe(SparkFiles.get(distScriptName));
System.out.println(StringUtils.join(distances.collect(), ","));
R script
#!/usr/bin/env Rscript
library("Imap")
f <- file("stdin")
open(f)
while(length(line <- readLines(f,n=1)) > 0) {
# process line
contents <- Map(as.numeric, strsplit(line, ","))
mydist <- gdist(contents[[1]][1], contents[[1]][2], contents[[1]][3], contents[[1]][4],
units="m", a=6378137.0, b=6356752.3142, verbose = FALSE)
write(mydist, stdout())
}
Stack Trace
java.io.IOException: Cannot run program"C:\Users\..\AppData\Local\Temp\spark-70bf05bd-6b8a-4c9f-b256-edbfb5f687b7\userFiles-cadcc324-4e7f-4952-8ec2-f442c7bd1cf8\finddistance.R": CreateProcess error=193, %1 is not a valid Win32 application
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.spark.rdd.PipedRDD.compute(PipedRDD.scala:119)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
at java.lang.ProcessImpl.start(ProcessImpl.java:137)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)

How to use Spark 1.2.0 in Play 2.2.3 project as it fails with NoSuchMethodError: akka.util.Helpers?

Have you ever had a problem with Play framework? In my case, first of all I have build all in one jar: spark-assebmly-1.2.0-hadoop2.4.0.jar, and Spark works perfectly from a shell. But there are two questions:
Should I use this assebmled Spark_jar in Play_project and how?? Because I try to move it into the lib_directiry and it did n`t help to provide any Spark_imports.
If I'm defining Spark library like: "org.apache.spark" %% "spark-core" % "1.2.0"
PLAY FRAMEWORK CODE:
Build.scala :
val appDependencies = Seq(
jdbc
,"org.apache.spark" %% "spark-streaming" % "1.2.0"
,"org.apache.spark" %% "spark-core" % "1.2.0"
,"org.apache.spark" %% "spark-sql" % "1.2.0"
TestEntity.scala :
package models
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.hadoop.conf.Configuration
import models.SparkMain
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext._
object TestEntity {
val TestEntityPath = "/home/t/PROD/dict/TestEntity .txt"
val TestEntitySpark= SparkMain.sc.textFile(TestEntityPath, 4).cache
val TestEntityData = TestEntitySpark.flatMap(_.split(","))
def getFive() : Seq[String] = {
println("TestEntity.getFive")
TestEntityData.take(5)
}
}
SparkMain.scala :
package models
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.hadoop.conf.Configuration
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext._
import org.apache.spark.streaming.{ Seconds, StreamingContext }
import StreamingContext._
import org.apache.spark.sql.SQLContext
import org.apache.spark.SparkConf
object SparkMain {
val driverPort = 8080
val driverHost = "localhost"
val conf = new SparkConf(false) // skip loading external settings
.setMaster("local[4]") // run locally with enough threads
.setAppName("firstSparkApp")
.set("spark.logConf", "true")
.set("spark.driver.port", s"$driverPort")
.set("spark.driver.host", s"$driverHost")
.set("spark.akka.logLifecycleEvents", "true")
val sc = new SparkContext(conf)
}
and controller code, which use Spark stuff :
def test = Action {
implicit req => {
val chk = TestEntity.getFive
Ok("it works")
}
}
..in runtime a have this errors:
[info] o.a.s.SparkContext - Spark configuration:
spark.akka.logLifecycleEvents=true
spark.app.name=firstSparkApp
spark.driver.host=localhost
spark.driver.port=8080
spark.logConf=true
spark.master=local[4]
[warn] o.a.s.u.Utils - Your hostname, uisprk resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
[warn] o.a.s.u.Utils - Set SPARK_LOCAL_IP if you need to bind to another address
[info] o.a.s.SecurityManager - Changing view acls to: t
[info] o.a.s.SecurityManager - Changing modify acls to: t
[info] o.a.s.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(t); users with modify permissions: Set(t)
[error] application -
! #6l039e8d5 - Internal server error, for (GET) [/ui] ->
play.api.Application$$anon$1: Execution exception[[RuntimeException: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;]]
at play.api.Application$class.handleError(Application.scala:293) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.DefaultApplication.handleError(Application.scala:399) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:166) [play_2.10-2.2.3.jar:2.2.3]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$13$$anonfun$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:163) [play_2.10-2.2.3.jar:2.2.3]
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) [scala-library-2.10.4.jar:na]
at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185) [scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;
at play.api.mvc.ActionBuilder$$anon$1.apply(Action.scala:314) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:109) ~[play_2.10-2.2.3.jar:2.2.3]
at play.utils.Threads$.withContextClassLoader(Threads.scala:18) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:108) ~[play_2.10-2.2.3.jar:2.2.3]
at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:107) ~[play_2.10-2.2.3.jar:2.2.3]
Caused by: java.lang.NoSuchMethodError: akka.util.Helpers$.ConfigOps(Lcom/typesafe/config/Config;)Lcom/typesafe/config/Config;
at akka.remote.RemoteSettings.<init>(RemoteSettings.scala:48) ~[akka-remote_2.10-2.3.4-spark.jar:na]
at akka.remote.RemoteActorRefProvider.<init>(RemoteActorRefProvider.scala:114) ~[akka-remote_2.10-2.3.4-spark.jar:na]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_72]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_72]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_72]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_72]
How to tie the library? through dependency or assembled_jar?
Any advice, please.
nosuchmethodeersrror exception is 100 % due to mismatch of jars version at compile time and runtime.
check the versions of jar. Also I have some questions about architecture of your app
Instead of calling spark code from play framework you can also call spark submit from shell scripts which looks better in your case. Even you can do it from your play application. no need to include jar in play app classpath.
The problem with the configuration is Akka dependency of Apache Spark and Play Framework -- they both depend on Akka and, as you've faced it, different and incompatible versions should be resolved at build time with evicted command in sbt.
You may want to use update command and find the reports in target/resolution-cache/reports quite useful.

Resources