I need to use sparknlp to do lemmatization in python, i want to use the pretrained pipeline, however need to do it offline. what is the correct way to do this? i am not able to find any python example.
I am passing token as the inputcol for lemmatization and lemma as the outputcol. Following is my code:
documentAssembler = DocumentAssembler().setInputCol("Transcript").setOutputCol("document")
tokenizer = Tokenizer().setInputCols(['document']).setOutputCol('token')
lemmatizer = LemmatizerModel().load("xx").setInputCols(["token"]).setOutputCol("lemma")
error message:
Py4JJavaError: An error occurred while calling None.com.johnsnowlabs.nlp.annotators.LemmatizerModel.
: java.lang.NoClassDefFoundError: Could not initialize class com.johnsnowlabs.util.ConfigHelper$
at com.johnsnowlabs.nlp.serialization.Feature.<init>(Feature.scala:22)
at com.johnsnowlabs.nlp.serialization.MapFeature.<init>(Feature.scala:145)
at com.johnsnowlabs.nlp.annotators.LemmatizerModel.<init>(LemmatizerModel.scala:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
Not sure why you're getting this specific error (how did you initialize Spark?), but this is another issue.
To use the lemmatizer offline you first need to download a pretrained one, for example from here, and unzip it. Then change your code to
lemmatizer = LemmatizerModel().load("/path/to/unzipped/model").setInputCols...
Related
I am able to run fix with UseDataDictionary=N but when I turn this into UseDataDictionary=Y
FIX44.xml: Could not parse data dictionary file
Someone please help me what is wrong.
Caused by: quickfix.ConfigError: Could not parse data dictionary file
at quickfix.DataDictionary.load(DataDictionary.java:857)
at quickfix.DataDictionary.read(DataDictionary.java:838)
... 14 more
Caused by: java.lang.NoClassDefFoundError: org/w3c/dom/ls/DocumentLS
[DEFAULT]
ConnectionType=initiator
HeartBtInt=60
ReconnectInterval=1
FileStorePath=.\fixfiles\initiator
FileLogPath=.\log
StartTime=00:00:00
EndTime=00:00:00
UseDataDictionary=Y
DataDictionary=FIX44.xml
SocketReuseAddress=Y
SocketKeepAlive=Y
SocketTcpNoDelay=Y
ResetOnLogon=Y
Well the QF library is saying that it can load the DD but not parse it, therefore your DD is corrupt. I would recommend looking at the default DD that comes with the install pack.
one of my apps is throwing the following exception on an Samsung S7:
java.lang.RuntimeException:
at android.app.NotificationManager.notifyAsUser
(NotificationManager.java:342)
at android.app.NotificationManager.notify (NotificationManager.java:292)
at android.app.NotificationManager.notify (NotificationManager.java:276)
at
org.mypapp.test.service.NotificationReceiverRemoteViews.updateNotification
(NotificationReceiverRemoteViews.java:202)
at org.mypapp.test.service.NotificationReceiverRemoteViews.onReceive
(NotificationReceiverRemoteViews.java:63)
at android.support.v4.content.LocalBroadcastManager.executePendingBroadcasts
(LocalBroadcastManager.java:308)
at android.support.v4.content.LocalBroadcastManager.access$000
(LocalBroadcastManager.java:46)
at android.support.v4.content.LocalBroadcastManager$1.handleMessage
(LocalBroadcastManager.java:118)
at android.os.Handler.dispatchMessage (Handler.java:102)
at android.os.Looper.loop (Looper.java:154)
at android.app.ActivityThread.main (ActivityThread.java:6642)
at java.lang.reflect.Method.invoke (Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run
(ZygoteInit.java:1468)
at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1358)
Caused by: android.os.TransactionTooLargeException:
at android.os.BinderProxy.transactNative (Native Method)
at android.os.BinderProxy.transact (Binder.java:628)
at android.app.INotificationManager$Stub$Proxy.enqueueNotificationWithTag
(INotificationManager.java:899)
at android.app.NotificationManager.notifyAsUser
(NotificationManager.java:336)
Here the code at line 202:
notificationManager.notify(NOTIFICATION_ID, mBuilder.build());
I also use an Samsung S7 and I have never gotten that error. I can't reproduce the error with the ansdroid studio emulator either. So I don't know how to fix an error I can't reproduce.
Any idea?
Possible reason is that you update RemoteViews instance before NotificationManager#notify instead of recreating a new one thus producing memory leak. I described this case in another answer
Apologies as this might be very basic question on the topic but I am new to Gremlin/DSE Graph and i tried many ways to extract data i am inserting to my graph but somehow i am unable to make it work.
Here is what i have:
1. Graph with allow_scans set to true
2. Schema with propertyKey and vertexes defined and materialized index on NodeID of all Vertexes.
There are no relationships right now, just vertexes with data points.
I wrote a program to insert all my nodes to DSE Graph which is working successfully as i get response like below after program created every Vertex:
Result({u'id': {u'out_vertex': {u'community_id': 853347840, u'~label': u'vertex', u'member_id': 14}, u'~type': u'Name', u'local_id': u'00000000-0000-8012-0000-000000000000'}, u'value': u'amount', u'label': u'Name'})]
Ok So now the nodes are inserted, i want to extract them and print their names:
So i did:
g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').values('Name');
Above failed successfully with blank result. I mean it runs successfully with null output like there is no error but then there is no output. (null in gremlin-console and 'Success - No Results' in DataStax Studio)
Then i came across documentation that graph will not know if the 'has' will only return one node or more, so i used next for iterating as per documentation and tutorials:
g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').next().values('Name');
Even this failed with
org.apache.tinkerpop.gremlin.driver.exception.ResponseException
(Datastax studio doesnt show more information) - How can i debug this further?
I even came across lambda approach in which i use map :
g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').map{it.get().value('Name')};
which responded with 'it' not being defined.
(i even tried valueMap - not sure if it was even required)
What am i doing wrong to find and print properties values of a node?
Any directions or query which can help me extract the names and other properties? Even a multi-step query? however i don't think this should be that complicated.
UPDATE:
As per answer i get the following traceback:
gremlin> :> g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').valueMap(true).next();
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException
Type ':help' or ':h' for help.
Display stack trace? [yN]y
org.apache.tinkerpop.gremlin.groovy.plugin.RemoteException
at org.apache.tinkerpop.gremlin.console.groovy.plugin.DriverRemoteAcceptor.submit(DriverRemoteAcceptor.java:170)
at org.apache.tinkerpop.gremlin.console.commands.SubmitCommand.execute(SubmitCommand.groovy:41)
at org.codehaus.groovy.tools.shell.Shell.execute(Shell.groovy:104)
at org.codehaus.groovy.tools.shell.Groovysh.super$2$execute(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.tools.shell.Groovysh.executeCommand(Groovysh.groovy:259)
at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:84)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:152)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:401)
I am able to do some similar operations in another graph. Is something wrong with the graph?
UPDATE 2
My Graph vertices were wrongly defined.
The key to drill down to this solution is in the ~label in the result. It points to vertex instead it should be Field
While defining the data insert team had put label in quotes when they had to put mention label without quotes. Hence i was not able to traverse the nodes.
You need to make sure to iterate the traversal. Most commonly you would use either:
iterate() get zero result
next() get one result
toList() get many results
I'd guess that NodeID is unique, so try something like this:
g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').
values('Name').next();
If you're interested in all of the properties on that vertex, try:
g.V().hasLabel('FIELD').has('NodeID','2559b635f077e86c7370ab1c4c798a06').
valueMap(true).next();
I'm using the package: IBrokers. It works well for me when I request historical data. Also the call to reqAccountUpdates() works well.
I am having problems with this script:
# myscript.r
.libPaths("rpackages")
library(IBrokers)
tws2 = twsConnect(2)
print('Attempting BUY')
mytkr = twsFuture("ES","GLOBEX","201412")
myorderid = sample(1001:3001, 1)
IBrokers:::.placeOrder(tws2, mytkr, twsOrder(myorderid, "BUY", "1", "MKT"))
twsDisconnect(tws2)
Sometimes the above script works okay. Usually though it fails. When it fails, it seems to connect okay.
Then I see this in my TWS console:
03:47:45:581 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Message type -1. Socket I/O error -
03:47:45:581 JTS-EServerSocket-290: Anticipated error
jextend.d: Socket I/O error -
at jextend.sc.b(sc.java:364)
at jextend.ch.sb(ch.java:1534)
at jextend.ch.run(ch.java:1390)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at jextend.xh.d(xh.java:45)
at jextend.sc.c(sc.java:579)
at jextend.sc.r(sc.java:227)
at jextend.af.a(af.java:232)
at jextend.sc.f(sc.java:650)
at jextend.pd.a(pd.java:822)
at jextend.sc.b(sc.java:358)
... 3 more
03:47:45:583 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Socket connection for client{2} has closed.
03:47:45:583 JTS-EWriter14-291: [2:47:71:1:0:0:0:ERR] Unable write to socket client{2} -
03:47:45:584 JTS-EServerSocketNotifier-288: Terminating
Can you offer any ideas on how you would wrestle with this issue?
One other piece of info:
I think the call to reqIds() may be necessary. Sometimes reqIds() would return an id not high enough. Then, I'd use it and placeOrder() would fail. So, I call reqIds() but then use Sys.time() to give me an id which is larger than the last ID I used.
Another problem may have been some code-text I copied out of a PowerPoint. Some of the code-characters may have been corrupt.
The main problem was orderid.
I need to be careful how I generate orderid.
Also I may have moused in bad characters from a powerpoint preso which had an example.
I posted some code which works in a comment in this thread.
Dan
Hey everyone I am trying to parse out a DateTime from a string that looks like "20110406080000.000[-4:EDT]" and am running into problems with the [-4:EDT]
DateTimeFormat.forPattern("yyyyMMddHHmmss.SSS[ZZ]").parseDateTime("20110406080000.000[-4:EDT]") results in the following error
java.lang.IllegalArgumentException: Invalid format: "20110406080000.000[-4:EDT]" is malformed at "-4:EDT]"
at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:673)
at .<init>(<console>:8)
at .<clinit>(<console>)
at RequestResult$.<init>(<console>:9)
at RequestResult$.<clinit>(<console>)
at RequestResult$scala_repl_result(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at scala.tools.nsc.Interpreter$Request$$anonfun$loadAndRun$1$$anonfun$apply$17.apply(Interpreter.scala:988)
at scala.tools.nsc.Interpreter$...
Any suggestions would be greatly appreciated.
You need to either
strip the suffix off before parsing
write your own DateTimeParser for the end part, using DateTimeFormatterBuilder to combine your parser with a standard parser for the first part