Production environment-> 2 app server domains, 2 web server domains, 3 process schedulers [AIX, 1 NT], NON RAC Oracle DB
PT 8.55.15, HCM 9.2 DB 12.2.0.1
We are running a custom AE program in our production environment. This AE that calls delivered EOEN application packages. No customization's have been made in this package. Custom component interface is also involved. This AE has been running in our environment since 5 years ago. No change has been done to the code in near past.
This AE has been behaving very weirdly since some 2-3 weeks ago. It errors out for the first time but goes to success when re-run in second/third run. We are not changing any parameters nor bouncing any services in second/third run. This application package [EOEN_MVC.EOEN_MODEL] makes calls to PeopleCode function - GetNextNumberWithGapsCommit.
We had set trace on this AE and found that the program goes to error every time the "GetNextNumberWithGapsCommit" function is called. These are the lines from the trace:
3905919 23:51:18.263 0.005472 Cur#8.15073404.HCMPRO RC=0
Dur=0.000094 COM Stmt=SELECT DESCR,DESCRLONG FROM PS_EOEN_REGE_LNG
WHERE EOEN_EVENT_NAME=:1 AND LANGUAGE_CD = :2
3905920 23:51:18.264 0.000974 Cur#8.15073404.HCMPRO RC=0
Dur=0.000001 Bind-1 type=2 length=16 value=CreateTriggerESP
3905921 23:51:18.265 0.000966 Cur#8.15073404.HCMPRO RC=0
Dur=0.000000 Bind-2 type=2 length=3 value=ESP
3905922 23:51:18.270 0.004369 258: If
All(&RS_RegEvnt(1).EOEN_REG_EVNT.EOEN_EVENT_NAME.Value) Then
3905923 23:51:18.271 0.001026 Fetch Field:
EOEN_REG_EVNT.EOEN_EVENT_NAME Value=CreateTriggerESP
3905924 23:51:18.272 0.001004 259:
&NextEventID = GetNextNumberWithGapsCommit(EOEN_CONFIG.EOEN_LAST_ID,
2147483647, 1);
3905925 23:51:18.273 0.001065 Cur#7.15073404.HCMPRO RC=0
Dur=0.000031 COM Stmt=UPDATE PS_EOEN_CONFIG SET EOEN_LAST_ID =
EOEN_LAST_ID + 1
3905926 23:51:18.274 0.001453 Cur#2.15073404.HCMPRO RC=0
Dur=0.000042 COM Stmt=SELECT
TO_CHAR(SYSTIMESTAMP,'YYYY-MM-DD-HH24.MI.SS.FF') FROM PSCLOCK
3905927 23:51:18.278 0.003895 Caught Exception: Error
grave de SQL. (2,125) EOEN_MVC.EOEN_MODEL.EOENInterface.OnExecute
Name:RaiseEvent PCPC:14959 Statement:259
Called from:FUNCLIB_HR_ESP.TRGR_FUNCTIONS_ESP.FieldFormula
Name:Create_Triggers_ESP Statement:60
Called from:JOB.REPORTS_TO.SavePostChange Statement:3
Error message in the process monitor is same as above:-
Error -> Error grave de SQL. (2,125)
EOEN_MVC.EOEN_MODEL.EOENInterface.OnExecute Name:RaiseEvent PCPC:14959
Statement:259 Called
from:FUNCLIB_HR_ESP.TRGR_FUNCTIONS_ESP.FieldFormula
Name:Create_Triggers_ESP Statement:60 Called
from:JOB.REPORTS_TO.SavePostChange Statement:3
I have tried keeping the DBFlags parameter as "8" and "0" in both application server and process schedulers.
I also checked the EOEN_MSG_CHNL queue, its associated service operation EOEN_MSG, handlers, routings etc. Everything is active and running. No failed IB messages. Domain status active.
Nothing works. And the error remains.
Any suggestions may be of great help.
Related
Greenplum_Connector_0,0: The following SQL statement failed: INSERT INTO GPCC_TT_20211121154035261_15420_0_XXXXX_TABLE_NAME (COLUMN1,COLUMN2,...) SELECT COLUMN1,COLUMN2,... FROM GPCC_ET_20211121154035417_15420_0. The statement reported the following reason: [SQLCODE=HY000][Native=3,484,948] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: missing data for column "xyz_id" (seg2 slice1 192.168.0.0:00 pid=30826)(Where External table gpcc_et_20211121154035417_15420_0, line 91 of gpfdist://ABCD:123/DDCETLMIG_15420_gpw_3_3_20211121154035261: "AG?199645?ABCD EFGH. - HELLOU - JSF RT ADF?MMM?+1?A?DAD. SDA?0082323209?N?N..."; File copy.c; Line 5211; Routine NextCopyFromX; )
The trick here is to read the error message carefully. Somehow your job has managed not to provide a value for column xyz_id. Check your job design thoroughly.
I'm creating a gnome shell extension and implementing the search provider. In the activateResult method I want to run some code like
GLib.spawn_command_line_sync('xdotool windowactivate ' + window_id);
GLib.spawn_command_line_sync('xdotool key "ctrl+r"');
GLib.spawn_command_line_sync('xdotool type ' + some_text);
The problem is that only the first command works, and I get some errors like:
Jul 27 20:05:09 comp org.gnome.Shell.desktop[3334]: Window manager warning: Received a NET_CURRENT_DESKTOP message from a broken (outdated) client who sent a 0 timestamp
Jul 27 20:05:09 comp org.gnome.Shell.desktop[3334]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x2e00001 (somestuff)
Jul 27 20:05:09 comp org.gnome.Shell.desktop[3334]: Window manager warning: last_focus_time (93207838) is greater than comparison timestamp (93207584). This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW. Trying to work around...
Jul 27 20:05:09 comp org.gnome.Shell.desktop[3334]: Window manager warning: last_user_time (93207838) is greater than comparison timestamp (93207584). This most likely represents a buggy client sending inaccurate timestamps in messages such as _NET_ACTIVE_WINDOW. Trying to work around...
Jul 27 20:05:09 comp org.gnome.Shell.desktop[3334]: Window manager warning: 0x2e00001 (somestuff) appears to be one of the offending windows with a timestamp of 93207838. Working around..
One thing I tried was to combine all the xdotool commands with bash -c "... ... ..." with no luck.
After selecting a search result, how can I switch to a window and simulate key presses?
(I'm brand new to gnome stuff, gjs stuff, and even JS, but, do write python daily)
edit: Just tried spawn_command_line_async and it works. Feels sloppy, someone with more experienced might have a better answer.
If you are writing a GNOME Shell extension, it's worth noticing that there are a number of GNOME platform libraries that you can rely on being available.
For simulating mouse and keyboard event, Atspi will work in most situations. As a simple example of simulating pressing the a key:
const Atspi = imports.gi.Atspi;
Atspi.generate_keyboard_event(0, 'a', Atspi.KeySynthType.STRING);
Activating a window, on the other hand, is probably something best done with the built-in window manager functions, however these aren't documented and you'd have to inspect the GNOME Shell JavaScript source or the source of another extension that does this.
I have created a report in AX 2012 and Visual Studio properly but once I try to open it the following list of erors appears:
Thank you
Error executing code: FormRun (data source) has no valid runable code in method 'new'.
Stack trace
(C)\Classes\FormRun\new
(C)\Classes\SysSetupFormRun\new - line 4
(C)\Classes\ClassFactory\formRunClassOnClient - line 10
(C)\Classes\ClassFactory\formRunClass - line 24
(C)\Classes\DialogForm\doInit - line 12
(C)\Classes\Dialog\doInit - line 12
(C)\Classes\Dialog\doRun - line 24
(C)\Classes\SysOperationDialog\run - line 12
(C)\Classes\SysOperationController\dialogShow - line 6
(C)\Classes\SysOperationController\dialogRun - line 40
(C)\Classes\SysOperationController\prompt - line 9
(C)\Classes\SrsReportRunController\runToScreen - line 26
(C)\Classes\SrsReportRunController\runReport - line 68
(C)\Classes\SrsReportRunController\run - line 27
(C)\Classes\SysOperationController\startOperation - line 10
(C)\Classes\SrsReportRunController\startOperation - line 12
To me, this is mostly base code, so it's a compile or CIL-compile issue most likely.
Have you compiled your CIL or performed an incremental CIL compile?
Refresh your report caches (Tools>Caches>Refresh Report Server).
Are the WCF settings in the AXC used for the report server refreshed?
Is the report server using a specific AXC that may not be refreshed? (https://technet.microsoft.com/en-us/library/hh389774.aspx)
You can find that the issue is with form (as the infolog suggest). The name of the form can be found out if you debug the code in classes.
Try putting debugger on DialogForm.doInit(). In this method, you can find the name of the form in the object variable, "form", used inside this method in a call to args.object(form).
You have to compile this form; Restore it and report does get generated.
Remember, DialogForm.doInit() gets called 2 times, Once, launching input UI form and another when trying to launch report viewer. So if the input/contract form is getting opened and the error comes before showing the report output, its the second time when you are interested to find the name of the form.
In my case, culprit forms were, SysOperationTemplateForm and SRSReportViewer form.
With SparkR, I'm trying for a PoC to collect an RDD that I created from text files which contains around 4M lines.
My Spark cluster is running in Google Cloud, is bdutil deployed and is composed with 1 master and 2 workers with 15gb of RAM and 4 cores each. My HDFS repository is based on Google Storage with gcs-connector 1.4.0.
SparkR is intalled on each machine, and basic tests are working on small files.
Here is the script I use :
Sys.setenv("SPARK_MEM" = "1g")
sc <- sparkR.init("spark://xxxx:7077", sparkEnvir=list(spark.executor.memory="1g"))
lines <- textFile(sc, "gs://xxxx/dir/")
test <- collect(lines)
First time I run this, it seems to be working fine, all the tasks are run successfully, spark's ui says that the job completed, but I never get the R prompt back :
15/06/04 13:36:59 WARN SparkConf: Setting 'spark.executor.extraClassPath' to ':/home/hadoop/hadoop-install/lib/gcs-connector-1.4.0-hadoop1.jar' as a work-around.
15/06/04 13:36:59 WARN SparkConf: Setting 'spark.driver.extraClassPath' to ':/home/hadoop/hadoop-install/lib/gcs-connector-1.4.0-hadoop1.jar' as a work-around.
15/06/04 13:36:59 INFO Slf4jLogger: Slf4jLogger started
15/06/04 13:37:00 INFO Server: jetty-8.y.z-SNAPSHOT
15/06/04 13:37:00 INFO AbstractConnector: Started SocketConnector#0.0.0.0:52439
15/06/04 13:37:00 INFO Server: jetty-8.y.z-SNAPSHOT
15/06/04 13:37:00 INFO AbstractConnector: Started SelectChannelConnector#0.0.0.0:4040
15/06/04 13:37:54 INFO GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop1
15/06/04 13:37:55 WARN LoadSnappy: Snappy native library is available
15/06/04 13:37:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/04 13:37:55 WARN LoadSnappy: Snappy native library not loaded
15/06/04 13:37:55 INFO FileInputFormat: Total input paths to process : 68
[Stage 0:=======================================================> (27 + 10) / 68]
Then after a CTRL-C to get the R prompt back, I try to run the collect method again, here is the result :
[Stage 1:==========================================================> (28 + 9) / 68]15/06/04 13:42:08 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver]
java.lang.OutOfMemoryError: Java heap space
at org.spark_project.protobuf.ByteString.toByteArray(ByteString.java:515)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:64)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
I understand the exception message, but I don't understand why I am getting this the second time.
Also, why the collect never returns after completing in Spark?
I Googled every piece of information I have, but I had no luck finding a solution. Any help or hint would be greatly appreciated!
Thanks
This does appear to be a simple combination of Java in-memory object representations being inefficient combined with some apparent long-lived object references which cause some collections to fail to be garbage-collected in time for the new collect() call to overwrite the old one in-place.
I experimented with some options, and for my sample 256MB file that contains ~4M lines, I indeed reproduce your behavior where collect is fine the first time, but OOMs the second time, when using SPARK_MEM=1g. I then set SPARK_MEM=4g instead, and then I'm able to ctrl+c and re-run test <- collect(lines) as many times as I want.
For one thing, even if references didn't leak, note that after the first time you ran test <- collect(lines), the variable test is holding that gigantic array of lines, and the second time you call it, the collect(lines) executes before finally being assigned to the test variable and thus in any straightforward instruction-ordering, there's no way to garbage-collect the old contents of test. This means the second run will make the SparkRBackend process hold two copies of the entire collection at the same time, leading to the OOM you saw.
To diagnose, on the master I started SparkR and first ran
dhuo#dhuo-sparkr-m:~$ jps | grep SparkRBackend
8709 SparkRBackend
I also checked top and it was using around 22MB of memory. I fetched a heap profile with jmap:
jmap -heap:format=b 8709
mv heap.bin heap0.bin
Then I ran the first round of test <- collect(lines) at which point running top showed it using ~1.7g of RES memory. I grabbed another heap dump. Finally, I also tried test <- {} to get rid of references to allow garbage-collection. After doing this, and printing out test and showing it to be empty, I grabbed another heap dump and noticed RES still showed 1.7g. I used jhat heap0.bin to analyze the original heap dump, and got:
Heap Histogram
All Classes (excluding platform)
Class Instance Count Total Size
class [B 25126 14174163
class [C 19183 1576884
class [<other> 11841 1067424
class [Lscala.concurrent.forkjoin.ForkJoinTask; 16 1048832
class [I 1524 769384
...
After running collect, I had:
Heap Histogram
All Classes (excluding platform)
Class Instance Count Total Size
class [C 2784858 579458804
class [B 27768 70519801
class java.lang.String 2782732 44523712
class [Ljava.lang.Object; 2567 22380840
class [I 1538 8460152
class [Lscala.concurrent.forkjoin.ForkJoinTask; 27 1769904
Even after I nulled out test, it remained about the same. This shows us 2784858 instances of char[], for a total size of 579MB, and also 2782732 instances of String, presumably holding those char[]'s above it. I followed the reference graph all the way up, and got something like
char[] -> String -> String[] -> ... -> class scala.collection.mutable.DefaultEntry -> class [Lscala.collection.mutable.HashEntry; -> class scala.collection.mutable.HashMap -> class edu.berkeley.cs.amplab.sparkr.JVMObjectTracker$ -> java.util.Vector#0x785b48cd8 (36 bytes) -> sun.misc.Launcher$AppClassLoader#0x7855c31a8 (138 bytes)
And then AppClassLoader had something like thousands of inbound references. So somewhere along that chain something should've been removing their reference but failing to do so, causing the entire collected array to sit in memory while we try to fetch a second copy of it.
Finally, to answer your question about hanging after the collect, it appears it has to do with the data not fitting in the R process's memory; here's a thread related to that issue: https://www.mail-archive.com/user#spark.apache.org/msg29155.html
I confirmed that using a smaller file with only a handful of lines, and then running collect indeed does not hang.
I'm using the package: IBrokers. It works well for me when I request historical data. Also the call to reqAccountUpdates() works well.
I am having problems with this script:
# myscript.r
.libPaths("rpackages")
library(IBrokers)
tws2 = twsConnect(2)
print('Attempting BUY')
mytkr = twsFuture("ES","GLOBEX","201412")
myorderid = sample(1001:3001, 1)
IBrokers:::.placeOrder(tws2, mytkr, twsOrder(myorderid, "BUY", "1", "MKT"))
twsDisconnect(tws2)
Sometimes the above script works okay. Usually though it fails. When it fails, it seems to connect okay.
Then I see this in my TWS console:
03:47:45:581 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Message type -1. Socket I/O error -
03:47:45:581 JTS-EServerSocket-290: Anticipated error
jextend.d: Socket I/O error -
at jextend.sc.b(sc.java:364)
at jextend.ch.sb(ch.java:1534)
at jextend.ch.run(ch.java:1390)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at jextend.xh.d(xh.java:45)
at jextend.sc.c(sc.java:579)
at jextend.sc.r(sc.java:227)
at jextend.af.a(af.java:232)
at jextend.sc.f(sc.java:650)
at jextend.pd.a(pd.java:822)
at jextend.sc.b(sc.java:358)
... 3 more
03:47:45:583 JTS-EServerSocket-290: [2:47:71:1:0:0:0:ERR] Socket connection for client{2} has closed.
03:47:45:583 JTS-EWriter14-291: [2:47:71:1:0:0:0:ERR] Unable write to socket client{2} -
03:47:45:584 JTS-EServerSocketNotifier-288: Terminating
Can you offer any ideas on how you would wrestle with this issue?
One other piece of info:
I think the call to reqIds() may be necessary. Sometimes reqIds() would return an id not high enough. Then, I'd use it and placeOrder() would fail. So, I call reqIds() but then use Sys.time() to give me an id which is larger than the last ID I used.
Another problem may have been some code-text I copied out of a PowerPoint. Some of the code-characters may have been corrupt.
The main problem was orderid.
I need to be careful how I generate orderid.
Also I may have moused in bad characters from a powerpoint preso which had an example.
I posted some code which works in a comment in this thread.
Dan