sorry for English. I am having a problem when running a Java program made to interact with USB modems, it uses threads and through the rxtx library establishes communication with up to 24 COM ports, the error happens when I have exactly 24 modems connected through USB hub to machine. It closes itself and generates the following .log file (part of)
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000001800087dd, pid=6564, tid=0x00000000000017a8
#
# JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.0_112-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [rxtxSerial.dll+0x87dd]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
--------------- T H R E A D ---------------
Current thread (0x0000000017667000): JavaThread "Thread-32" [_thread_in_native, id=6056, stack(0x000000001e640000,0x000000001e740000)]
siginfo: ExceptionCode=0xc0000005, writing address 0x0000000000000100
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x00000000162e4ba0 is an unknown value
RCX=0x0000000000000002 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x000000001e73eab0 is pointing into the stack for thread: 0x0000000017667000
RBP=0x000000001e73ed28 is pointing into the stack for thread: 0x0000000017667000
RSI=0x0000000016ae623a is an unknown value
RDI=0x00000000162e4d20 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x0000000000000006 is an unknown value
R11=0x0000000000000001 is an unknown value
R12=0x000000001e73ed30 is pointing into the stack for thread: 0x0000000017667000
R13=0x0000000019553998 is pointing into metadata
R14=0x000000001e73ed30 is pointing into the stack for thread: 0x0000000017667000
R15=0x0000000017667000 is a thread
Stack: [0x000000001e640000,0x000000001e740000], sp=0x000000001e73eab0, free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [rxtxSerial.dll+0x87dd]
C [rxtxSerial.dll+0xaab4]
C [rxtxSerial.dll+0x213d]
C 0x0000000002738a4b
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j gnu.io.RXTXPort.open(Ljava/lang/String;)I+0
j gnu.io.RXTXPort.<init>(Ljava/lang/String;)V+125
j gnu.io.RXTXCommDriver.getCommPort(Ljava/lang/String;I)Lgnu/io/CommPort;+78
j gnu.io.CommPortIdentifier.open(Ljava/lang/String;I)Lgnu/io/CommPort;+171
j tw.modem.Porta.abrePorta()V+11
j tw.modem.Porta.run()V+50
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x0000000016a33000 JavaThread "Thread-41" [_thread_in_native, id=6436, stack(0x000000001ecf0000,0x000000001edf0000)]
0x0000000016a32000 JavaThread "Thread-40" [_thread_in_native, id=6008, stack(0x000000001ebc0000,0x000000001ecc0000)]
0x0000000016a31800 JavaThread "Thread-39" [_thread_in_native, id=676, stack(0x000000001e950000,0x000000001ea50000)]
0x0000000016a30800 JavaThread "Thread-38" [_thread_in_native, id=6368, stack(0x0000000019190000,0x0000000019290000)]
0x0000000016a30000 JavaThread "Thread-37" [_thread_in_native, id=7160, stack(0x000000001e750000,0x000000001e850000)]
0x0000000016a2f000 JavaThread "Thread-36" [_thread_in_native, id=5372, stack(0x000000001e210000,0x000000001e310000)]
0x0000000017665800 JavaThread "Thread-35" [_thread_in_native, id=3228, stack(0x00000000188b0000,0x00000000189b0000)]
0x0000000016a2d800 JavaThread "Thread-33" [_thread_blocked, id=1060, stack(0x000000001e850000,0x000000001e950000)]
=>0x0000000017667000 JavaThread "Thread-32" [_thread_in_native, id=6056, stack(0x000000001e640000,0x000000001e740000)]
0x0000000017666000 JavaThread "Thread-31" [_thread_blocked, id=3368, stack(0x000000001e430000,0x000000001e530000)]
0x0000000017661800 JavaThread "Thread-30" [_thread_blocked, id=6312, stack(0x000000001e310000,0x000000001e410000)]
0x0000000017662800 JavaThread "Thread-29" [_thread_blocked, id=2556, stack(0x000000001e0b0000,0x000000001e1b0000)]
0x0000000017660000 JavaThread "Thread-28" [_thread_blocked, id=6584, stack(0x000000001dc60000,0x000000001dd60000)]
0x0000000017663000 JavaThread "Thread-27" [_thread_blocked, id=748, stack(0x0000000019da0000,0x0000000019ea0000)]
0x0000000017661000 JavaThread "Thread-26" [_thread_blocked, id=4080, stack(0x000000001c990000,0x000000001ca90000)]
0x0000000017664000 JavaThread "Thread-25" [_thread_blocked, id=6452, stack(0x000000001dec0000,0x000000001dfc0000)]
0x0000000017664800 JavaThread "Thread-24" [_thread_blocked, id=4544, stack(0x000000001db50000,0x000000001dc50000)]
0x0000000016a21800 JavaThread "Image Animator 3" daemon [_thread_blocked, id=6328, stack(0x0000000019850000,0x0000000019950000)]
0x0000000017626800 JavaThread "Timer-0" [_thread_blocked, id=6772, stack(0x00000000199b0000,0x0000000019ab0000)]
0x00000000176c7800 JavaThread "TimerQueue" daemon [_thread_blocked, id=5760, stack(0x0000000019700000,0x0000000019800000)]
0x00000000003be800 JavaThread "DestroyJavaVM" [_thread_blocked, id=3488, stack(0x0000000002350000,0x0000000002450000)]
0x00000000177a9800 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=424, stack(0x0000000019300000,0x0000000019400000)]
0x000000001753b000 JavaThread "AWT-Windows" daemon [_thread_in_native, id=2300, stack(0x0000000018610000,0x0000000018710000)]
0x0000000016b65800 JavaThread "AWT-Shutdown" [_thread_blocked, id=7152, stack(0x0000000018490000,0x0000000018590000)]
0x0000000016b64800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=3384, stack(0x00000000180c0000,0x00000000181c0000)]
0x000000001697a800 JavaThread "Abandoned connection cleanup thread" daemon [_thread_blocked, id=5112, stack(0x0000000017380000,0x0000000017480000)]
0x000000001683e000 JavaThread "Service Thread" daemon [_thread_blocked, id=6648, stack(0x0000000017040000,0x0000000017140000)]
0x00000000167c5000 JavaThread "C1 CompilerThread2" daemon [_thread_blocked, id=7044, stack(0x0000000016e50000,0x0000000016f50000)]
0x00000000167c4800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=5560, stack(0x0000000016c00000,0x0000000016d00000)]
0x0000000014f7a800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3980, stack(0x0000000016300000,0x0000000016400000)]
0x0000000014f78000 JavaThread "Attach Listener" daemon [_thread_blocked, id=4684, stack(0x00000000166b0000,0x00000000167b0000)]
0x0000000014f74800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5064, stack(0x0000000016530000,0x0000000016630000)]
0x0000000014f5a800 JavaThread "Finalizer" daemon [_thread_blocked, id=4300, stack(0x0000000016430000,0x0000000016530000)]
0x0000000014f13800 JavaThread "Reference Handler" daemon [_thread_blocked, id=1556, stack(0x0000000016170000,0x0000000016270000)]
Other Threads:
0x0000000014f0b800 VMThread [stack: 0x0000000015f80000,0x0000000016080000] [id=3788]
0x000000001686c000 WatcherThread [stack: 0x0000000017160000,0x0000000017260000] [id=6028]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
--------------- S Y S T E M ---------------
OS: Windows 7 , 64 bit Build 7601 (6.1.7601.23569)
CPU:total 4 (2 cores per cpu, 2 threads per core) family 6 model 42 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, clmul, ht, tsc, tscinvbit
Memory: 4k page, physical 4104836k(1314976k free), swap 8207812k(4569232k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.112-b15) for windows-amd64 JRE (1.8.0_112-b15), built on Sep 22 2016 21:31:56 by "java_re" with MS VC++ 10.0 (VS2010)
The function abrePorta, responsible for opening the COM port that is in the report and is here.
private synchronized void abrePorta() {
try {
serialPort = (SerialPort) serialPortId.open("Porta", 6000);
serialPort.setSerialPortParams(115200, SerialPort.DATABITS_8, SerialPort.STOPBITS_1,
SerialPort.PARITY_NONE);
serialPort.setFlowControlMode(SerialPort.FLOWCONTROL_RTSCTS_IN | SerialPort.FLOWCONTROL_RTSCTS_OUT);
at = new AtCommandPort(serialPort);
} catch (Exception e) {
System.err.println(e.getMessage());
e.printStackTrace();
}
}
Anyone have any idea what it could be? Would it be a mistake to establish communication with the ports? Thanks
I fixed this issue. By replace rxtxSerial64.dll which is use for 64bit OS.
I faced the same issue here. I found that memory management in C lib cause this problem.
Related
I'm try to run vault instance on aws and when i want to run command: vault operator init -key-shares=5 -key-threshold=3 -format json on Ansible role and i have error code :
fatal: [vault]: FAILED! => {"changed": true, "cmd": "vault operator init -key-shares=5 -key-threshold=3 -format json", "delta": "0:00:00.054870", "end": "2021-12-12 14:30:50.956504", "msg": "non-zero return code", "rc": 2, "start": "2021-12-12 14:30:50.901634", "stderr": "Error initializing: Put \"http://127.0.0.1:8200/v1/sys/init\": dial tcp 127.0.0.1:8200: connect: connection refused", "stderr_lines": ["Error initializing: Put \"http://127.0.0.1:8200/v1/sys/init\": dial tcp 127.0.0.1:8200: connect: connection refused"], "stdout": "", "stdout_lines": []}
When i'm on my vault server and when i do service vault status, i have this result :
vault.service - a tool for managing secrets
Loaded: loaded (/etc/systemd/system/vault.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-12-12 14:19:47 UTC; 6min ago
Docs: https://vaultproject.io/docs/
Process: 5152 ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl (code=exited, status=213/SECUREBITS)
Main PID: 5152 (code=exited, status=213/SECUREBITS)
Dec 12 14:19:47 ip-172-31-37-194 systemd[1]: Started a tool for managing secrets.
Dec 12 14:19:47 ip-172-31-37-194 systemd[5152]: vault.service: Failed to set process secure bits: Operation not perm
Dec 12 14:19:47 ip-172-31-37-194 systemd[5152]: vault.service: Failed at step SECUREBITS spawning /usr/local/bin/vau
Dec 12 14:19:47 ip-172-31-37-194 systemd[1]: vault.service: Main process exited, code=exited, status=213/SECUREBITS
Dec 12 14:19:47 ip-172-31-37-194 systemd[1]: vault.service: Failed with result 'exit-code'.
There'is my 2 config files :
vault.hcl :
disable_mlock = true
listener "tcp" {
address = "http://{{ listener_address }}"
tls_disable = 1
}
backend "file" {
path = "/var/lib/vault"
}
my vault.service :
[Unit]
Description=a tool for managing secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault.hcl
[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT
[Install]
WantedBy=multi-user.target
I didn't find anything yet who could unlock this situation, if someone have an idea.
**Apache Airflow version:**1.10.9-composer
Kubernetes Version : Client Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.12-gke.6002", GitCommit:"035184604aff4de66f7db7fddadb8e7be76b6717", GitTreeState:"clean", BuildDate:"2020-12-01T23:13:35Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}
Environment: Airflow, running on top of Kubernetes - Linux version 4.19.112
OS : Linux version 4.19.112+ (builder#7fc5cdead624) (Chromium OS 9.0_pre361749_p20190714-r4 clang version 9.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm-project c11de5eada2decd0a495ea02676b6f4838cd54fb) (based on LLVM 9.0.0svn)) #1 SMP Fri Sep 4 12:00:04 PDT 2020
Kernel : Linux gke-europe-west2-asset-c-default-pool-dc35e2f2-0vgz
4.19.112+ #1 SMP Fri Sep 4 12:00:04 PDT 2020 x86_64 Intel(R) Xeon(R) CPU # 2.20GHz GenuineIntel GNU/Linux
What happened ?
A running task is marked as Zombie after the execution time crossed the latest heartbeat + 5 minutes.
The task is running in background in another application server, triggered using SSHOperator.
[2021-01-18 11:53:37,491] {taskinstance.py:888} INFO - Executing <Task(SSHOperator): load_trds_option_composite_file> on 2021-01-17T11:40:00+00:00
[2021-01-18 11:53:37,495] {base_task_runner.py:131} INFO - Running on host: airflow-worker-6f6fd78665-lm98m
[2021-01-18 11:53:37,495] {base_task_runner.py:132} INFO - Running: ['airflow', 'run', 'dsp_etrade_process_trds_option_composite_0530', 'load_trds_option_composite_file', '2021-01-17T11:40:00+00:00', '--job_id', '282759', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/dsp_etrade_trds_option_composite_0530.py', '--cfg_path', '/tmp/tmpge4_nva0']
Task Executing time:
dag_id dsp_etrade_process_trds_option_composite_0530
duration 7270.47
start_date 2021-01-18 11:53:37,491
end_date 2021-01-18 13:54:47.799728+00:00
Scheduler Logs during that time:
[2021-01-18 13:54:54,432] {taskinstance.py:1135} ERROR - <TaskInstance: dsp_etrade_process_etrd.push_run_date 2021-01-18 13:30:00+00:00 [running]> detected as zombie
{
textPayload: "[2021-01-18 13:54:54,432] {taskinstance.py:1135} ERROR - <TaskInstance: dsp_etrade_process_etrd.push_run_date 2021-01-18 13:30:00+00:00 [running]> detected as zombie"
insertId: "1ca8zyfg3zvma66"
resource: {
type: "cloud_composer_environment"
labels: {3}
}
timestamp: "2021-01-18T13:54:54.432862699Z"
severity: "ERROR"
logName: "projects/asset-control-composer-prod/logs/airflow-scheduler"
receiveTimestamp: "2021-01-18T13:54:55.714437665Z"
}
Airflow-webserver log :
X.X.X.X - - [18/Jan/2021:13:54:39 +0000] "GET /_ah/health HTTP/1.1" 200 187 "-" "GoogleHC/1.0"
{
textPayload: "172.17.0.5 - - [18/Jan/2021:13:54:39 +0000] "GET /_ah/health HTTP/1.1" 200 187 "-" "GoogleHC/1.0"
"
insertId: "1sne0gqg43o95n3"
resource: {2}
timestamp: "2021-01-18T13:54:45.401670481Z"
logName: "projects/asset-control-composer-prod/logs/airflow-webserver"
receiveTimestamp: "2021-01-18T13:54:50.598807514Z"
}
Airflow Info logs :
2021-01-18 08:54:47.799 EST
{
textPayload: "NoneType: None
"
insertId: "1ne3hqgg47yzrpf"
resource: {2}
timestamp: "2021-01-18T13:54:47.799661030Z"
severity: "INFO"
logName: "projects/asset-control-composer-prod/logs/airflow-scheduler"
receiveTimestamp: "2021-01-18T13:54:50.914461159Z"
}
[2021-01-18 13:54:47,800] {taskinstance.py:1192} INFO - Marking task as FAILED.dag_id=dsp_etrade_process_trds_option_composite_0530, task_id=load_trds_option_composite_file, execution_date=20210117T114000, start_date=20210118T115337, end_date=20210118T135447
Copy link
{
textPayload: "[2021-01-18 13:54:47,800] {taskinstance.py:1192} INFO - Marking task as FAILED.dag_id=dsp_etrade_process_trds_option_composite_0530, task_id=load_trds_option_composite_file, execution_date=20210117T114000, start_date=20210118T115337, end_date=20210118T135447"
insertId: "1ne3hqgg47yzrpg"
resource: {2}
timestamp: "2021-01-18T13:54:47.800605248Z"
severity: "INFO"
logName: "projects/asset-control-composer-prod/logs/airflow-scheduler"
receiveTimestamp: "2021-01-18T13:54:50.914461159Z"
}
Airflow Database shows the latest heartbeat as:
select state, latest_heartbeat from job where id=282759
--------------------------------------
state | latest_heartbeat
running | 2021-01-18 13:48:41.891934
Airflow Configurations:
celery
worker_concurrency=6
scheduler
scheduler_health_check_threshold=60
scheduler_zombie_task_threshold=300
max_threads=2
core
dag_concurrency=6
Kubernetes Cluster :
Worker nodes : 6
What was expected to happen ?
The backend process takes around 2hrs 30 minutes to finish. During
such long running jobs the task is detected as zombie. Eventhough the
worker node is still processing the task. The state of the job is
still marked as 'running'. State if the task is not known during the
run time.
Example of nginx.conf
server {
set $abc_variable "abcabc";
........
}
How can I access abc_variable with the help of module api defined here https://www.nginx.com/resources/wiki/extending/api/
I'm using following code
ngx_str_t var = ngx_string("abc_variable");
ngx_uint_t key = ngx_hash_strlow(var.data, var.data, var.len);
ngx_http_variable_value_t *val = NULL;
val = ngx_http_get_variable(r, &var, key);
But I'm getting follow error
019/12/04 01:24:02 [notice] 12442#0: signal 17 (SIGCHLD) received from 12444
2019/12/04 01:24:02 [alert] 12442#0: worker process 12444 exited on signal 11 (core dumped)
2019/12/04 01:24:02 [notice] 12442#0: start worker process 12561
2019/12/04 01:24:02 [notice] 12442#0: signal 29 (SIGIO) received
2019/12/04 01:24:02 [debug] 12561#0: setting SA_RESTART for signal 1
You can use ngx-http-get-variable.
I'm running Yarn on Oracle BDA X7-2, specs:
Cloudera Enterprise 5.14.3
Java 1.8.0_171
PGX 2.7.1
I'm trying to run PGX on Yarn following this manual:
https://docs.oracle.com/cd/E56133_01/2.5.0/tutorials/yarn.html
Managed to run the installation script, completed the config file provided by it with the following:
{
"pgx_yarn_jar_hdfs_path": "hdfs:/user/pgx/pgx-yarn-2.7.1.jar",
"pgx_war_hdfs_path": "hdfs:/user/pgx/pgx-webapp-2.7.1.war",
"pgx_conf_hdfs_path": "hdfs:/user/pgx/pgx.conf",
"pgx_log4j_conf_hdfs_path": "hdfs:/user/pgx/log4j2.xml",
"pgx_dist_log4j_conf_hdfs_path": "hdfs:/user/pgx/dist_log4j.xml",
"pgx_cluster_host_hdfs_path": "hdfs:/user/pgx/cluster-host.tgz",
"zookeeper_connect_string": "bda1node05,bda1node06,bda1node07",
"standard_library_path": "/usr/lib64/gcc/4.8.2",
"min_heap_size": "512m",
"max_heap_size": "12g",
"container_cores": 9,
"container_memory": 0,
"container_priority": 0,
"num_machines": 1
}
Yarn has a pgx-service application in RUNNING state, no errors in stderr, the log shows me the service is running in the address:
http://bda1node06:7007
And the linux Java process is running with the following command:
/usr/java/default/bin/java -Xms512m -Xmx12g oracle.pgx.yarn.PgxService bda1node06 /u11/hadoop/yarn/nm/usercache/root/appcache/application_1539869144089_2070/container_e22_1539869144089_2070_01_000002/pgx-server.war 7007 bda1node05,bda1node06,bda1node07 /pgx-8eef44e2-1657-403a-8193-0102f5266680
And after the execution of the PGX client for testing purposes:
$PGX_HOME/bin/pgx --base_url http://bda1node06:7007
I get:
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: cannot connect to server; requested http://bda1node06:7007/version?extendedInfo=true and expected status 200, got 404 instead; response body = ""
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at oracle.pgx.api.PgxFuture.get(PgxFuture.java:99)
at oracle.pgx.api.ServerInstance.createSession(ServerInstance.java:559)
at oracle.pgx.shell.Console.initSession(Console.java:280)
at oracle.pgx.shell.Console.(Console.java:153)
at oracle.pgx.shell.Console.main(Console.java:296)
Caused by: java.lang.IllegalStateException: cannot connect to server; requested http://bda1node06:7007/version?extendedInfo=true and expected status 200, got 404 instead; response body = ""
at oracle.pgx.api.ClientApiProvider.lambda$versionCheck$2(ClientApiProvider.java:189)
at oracle.pgx.client.RemoteUtils.lambda$asyncRequest$5(RemoteUtils.java:278)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I have no idea of how to debug and check if there's any extra path needed in the connection URL.
How may I proceed to debug?
Thanks in advance!
By default, PGX has a base path of /pgx, which means you should connect as follows:
$PGX_HOME/bin/pgx --base_url http://bda1node06:7007/pgx
I'll do a little follow up here.
We've managed to start a pgx server and manipulate hbase graph! :D
PGX "Hello World"
We wrote a small code to insert vertices, edgex, instantiate pgx and run a simple example, this is it:
cfg = GraphConfigBuilder.forPropertyGraphHbase().setName('sinapse').setZkQuorum('bda1node05').build()
opg = OraclePropertyGraph.getInstance(cfg)
a = opg.addVertex()
a.setProperty('nome', 'Felipe')
b = opg.addVertex()
b.setProperty('nome', 'Rhenan')
c = opg.addVertex()
c.setProperty('nome', 'Hugo')
opg.addEdge(a, b, 'Pai de')
opg.addEdge(b, c, 'Pai de')
opg.addEdge(a, c, 'Avo de')
opg.commit()
session = Pgx.createSession('sinapsepgx')
analyst = session.createAnalyst()
pgxGraph = session.readGraphWithProperties(opg.getConfig(), true)
analyst.countTriangles(pgxGraph, true)
And that worked just fine!
Client - Server architecture
The next step, we moved to a client/server mode, starting the start-server script.
We managed to do that just fine too!
This is our config files:
server.conf
{
"port": 7007,
"enable_tls": false,
"enable_client_authentication": false
}
pgx.conf
{
"allow_idle_timeout_overwrite": true,
"allow_local_filesystem": false,
"allow_task_timeout_overwrite": true,
"enable_gm_compiler": true,
"enterprise_scheduler_config": {
"analysis_task_config": {
"priority": "MEDIUM",
"weight": 12,
"max_threads": 12
},
"fast_analysis_task_config": {
"priority": "HIGH",
"weight": 1,
"max_threads": 12
},
"num_io_threads_per_task": 12
},
"preload_graphs": [
{"path": "graphs/sinapse_conf.json",
"name": "sinapse"}
],
"max_active_sessions": 1024,
"max_queue_size_per_session": -1,
"max_snapshot_count": 0,
"memory_cleanup_interval": 600,
"path_to_gm_compiler": null,
"release_memory_threshold": 0.85,
"session_idle_timeout_secs": 0,
"session_task_timeout_secs": 0,
"strict_mode": true,
"tmp_dir": "/tmp"
}
sinapse_conf.json
{
"edge_props": [
{
"name": "relacao",
"type": "string"
}
],
"db_engine": "HBASE",
"vertex_props": [
{
"name": "nome",
"type": "string"
},
{
"name": "cpf",
"type": "string"
}
],
"format": "pg",
"name": "sinapse",
"error_handling": {},
"vertex_id_type": "long",
"attributes": {},
"loading": {},
"zk_quorum": "bda1node05,bda1node06,bda1node07"
}
start-script ran just fine with that, preloaded our hbase graph, works like a charm.
Connected to the server using the pgx client:
./bin/pgx -b http://localhost:7007
And managed to do the same we did in the groovy shell.
That's awesome.
PGX on Yarn
Well, now we are back in our challenge: run and manage PGX on Yarn.
We've copied our pgx.conf file to the hdfs, like this:
hdfs://user/pgx/pgx.conf
{
"allow_idle_timeout_overwrite": true,
"allow_local_filesystem": false,
"allow_task_timeout_overwrite": true,
"enable_gm_compiler": true,
"enterprise_scheduler_config": {
"analysis_task_config": {
"priority": "MEDIUM",
"weight": 12,
"max_threads": 12
},
"fast_analysis_task_config": {
"priority": "HIGH",
"weight": 1,
"max_threads": 12
},
"num_io_threads_per_task": 12
},
"preload_graphs": [
{"path": "graphs/sinapse_conf.json",
"name": "sinapse"}
],
"max_active_sessions": 1024,
"max_queue_size_per_session": -1,
"max_snapshot_count": 0,
"memory_cleanup_interval": 600,
"path_to_gm_compiler": null,
"release_memory_threshold": 0.85,
"session_idle_timeout_secs": 0,
"session_task_timeout_secs": 0,
"strict_mode": true,
"tmp_dir": "/tmp"
}
/opt/oracle/oracle-spatial-graph/property_graph/pgx/yarn/conf/yarn.conf
{
"pgx_yarn_jar_hdfs_path": "hdfs://mpmapas-ns/user/pgx/pgx-yarn-2.7.1.jar",
"pgx_war_hdfs_path": "hdfs://mpmapas-ns/user/pgx/pgx-webapp-2.7.1.war",
"pgx_conf_hdfs_path": "hdfs://mpmapas-ns/user/pgx/pgx.conf",
"pgx_log4j_conf_hdfs_path": "hdfs://mpmapas-ns/user/pgx/log4j2.xml",
"pgx_dist_log4j_conf_hdfs_path": "hdfs://mpmapas-ns/user/pgx/dist_log4j.xml",
"pgx_cluster_host_hdfs_path": "hdfs://mpmapas-ns/user/pgx/cluster-host.tgz",
"zookeeper_connect_string": "bda1node05.pgj.rj.gov.br,bda1node06.pgj.rj.gov.br,bda1node07.pgj.rj.gov.br",
"standard_library_path": "/usr/lib64/gcc/4.8.2",
"min_heap_size": "512m",
"max_heap_size": "12g",
"container_cores": 9,
"container_memory": 0,
"container_priority": 0,
"num_machines": 1
}
Also, #albert recomended us to remove the log4j2.xml from the server/shared-mem/pgx-webapp-2.7.1.war file so we may handle log4j logging using only the file placed on our hdfs folder.
So we've unpacked, removed, repacked the war file, edited the log4j2.xml file on hdfs like this:
hdfs://user/pgx/log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss,SSS} %p %C{1} - %m%n"/>
</Console>
<File name="LogFile" fileName="file:/tmp/pg_trace.log">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
</File>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="LogFile"/>
</Root>
<Logger name="oracle.pgx.engine.admin.Ctrl" level="debug">
<AppenderRef ref="LogFile"/>
</Logger>
<Logger name="pgx.dist.cluster_host" level="debug">
<AppenderRef ref="LogFile"/>
</Logger>
</Loggers>
</Configuration>
And finally ran the yarn start server command, just like this:
yarn jar yarn/pgx-yarn-2.7.1.jar yarn/conf/yarn.conf
And we get the bottom of the logfile that seems realy nice!:
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:os.version=4.1.12-124.14.1.el7uek.x86_64
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:user.name=root
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/oracle/oracle-spatial-graph/property_graph/pgx
18/12/11 16:25:03 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=bda1node05.pgj.rj.gov.br,bda1node06.pgj.rj.gov.br,bda1node07.pgj.rj.gov.br sessionTimeout=10000 watcher=oracle.pgx.yarn.ClientZkClient#32da97fd
18/12/11 16:25:03 INFO zookeeper.ClientCnxn: Opening socket connection to server bda1node07.pgj.rj.gov.br/192.168.8.7:2181. Will not attempt to authenticate using SASL (unknown error)
18/12/11 16:25:03 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /192.168.8.5:33299, server: bda1node07.pgj.rj.gov.br/192.168.8.7:2181
18/12/11 16:25:03 INFO zookeeper.ClientCnxn: Session establishment complete on server bda1node07.pgj.rj.gov.br/192.168.8.7:2181, sessionid = 0x3668759ae4553df, negotiated timeout = 10000
18/12/11 16:25:05 INFO yarn.StartService: waiting for PGX service (yarn appId == 'application_1539869144089_2555') to come up ...
18/12/11 16:25:10 INFO yarn.StartService: retrieved PGX host: http://bda1node07.pgj.rj.gov.br:7007
18/12/11 16:25:10 INFO yarn.StartService: to connect a remote shell to this host, run '$PGX_HOME/bin/pgx --base_url http://bda1node07.pgj.rj.gov.br:7007'
18/12/11 16:25:10 INFO yarn.StartService: to shut the PGX service down, run 'yarn application -kill application_1539869144089_2555'
18/12/11 16:25:10 INFO zookeeper.ZooKeeper: Session: 0x3668759ae4553df closed
18/12/11 16:25:10 INFO zookeeper.ClientCnxn: EventThread shut down
But connecting to it still returns 404 ;(
The last intel I may give you is the yarn stderr log, wich also informs that we are not using log4j correctly:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/u09/hadoop/yarn/nm/filecache/890/pgx-yarn-2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
18/12/11 16:25:06 INFO yarn.AppMaster: register app
18/12/11 16:25:06 INFO yarn.AppMaster: RM response = [queue=root.users.root,maxCap=<memory:65536, vCores:9>]
18/12/11 16:25:06 INFO yarn.AppMaster: max capability of cluster: <memory:65536, vCores:9>
18/12/11 16:25:06 INFO yarn.AppMaster: attempting to allocate 1 containers
18/12/11 16:25:06 INFO yarn.AppMaster: attempt 1: got 0 containers. Available: <memory:194560, vCores:180>
18/12/11 16:25:06 INFO yarn.AppMaster: attempt 2: got 0 containers. Available: <memory:194560, vCores:180>
18/12/11 16:25:06 INFO yarn.AppMaster: attempt 3: got 1 containers. Available: <memory:129024, vCores:171>
18/12/11 16:25:06 INFO yarn.AppMaster: copy hdfs://mpmapas-ns/user/pgx/pgx-yarn-2.7.1.jar into pgx-yarn.jar
18/12/11 16:25:06 INFO yarn.AppMaster: copy hdfs://mpmapas-ns/user/pgx/pgx-webapp-2.7.1.war into pgx-server.war
18/12/11 16:25:06 INFO yarn.AppMaster: copy hdfs://mpmapas-ns/user/pgx/pgx.conf into conf/pgx.conf
18/12/11 16:25:06 INFO yarn.AppMaster: copy hdfs://mpmapas-ns/user/pgx/log4j2.xml into conf/log4j2.xml
18/12/11 16:25:07 INFO yarn.AppMaster: server env = {CLASSPATH=conf:pgx-server/WEB-INF/lib/*:pgx-yarn.jar:$HADOOP_CONF_DIR}
18/12/11 16:25:07 INFO yarn.AppMaster: server command = $JAVA_HOME/bin/java -Xms512m -Xmx12g oracle.pgx.yarn.PgxService bda1node07.pgj.rj.gov.br $PWD/pgx-server.war 7007 bda1node05.pgj.rj.gov.br,bda1node06.pgj.rj.gov.br,bda1node07.pgj.rj.gov.br /pgx-37a121ce-e028-432c-8761-104027126c3b 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr;
18/12/11 16:25:07 INFO yarn.AppMaster: check for completion
18/12/11 16:25:08 INFO yarn.AppMaster: check for completion
18/12/11 16:25:08 INFO yarn.AppMaster: check for completion
18/12/11 16:25:09 INFO yarn.AppMaster: check for completion
18/12/11 16:25:09 INFO yarn.AppMaster: check for completion
18/12/11 16:25:10 INFO yarn.AppMaster: check for completion
18/12/11 16:25:10 INFO yarn.AppMaster: check for completion
18/12/11 16:25:11 INFO yarn.AppMaster: check for completion
18/12/11 16:25:11 INFO yarn.AppMaster: check for completion
18/12/11 16:25:12 INFO yarn.AppMaster: check for completion
.
.
.
This is the farthest we've managed to go.
We can start our work now! That's realy exciting.
Now I know how to properly start a service, preload, insert, manage data, and we will import our existing graph database to it and do some experimentation.
Would be lovely to have this running on Yarn at the production level.
Thank you all for the extreme dedication and attention.
In my java application multiple threads periodically getting system´s "process list" through a dll(named ProcessDLL.dll). When there is only one thread in my application then it works fine. But with more than one threads i get a EXCEPTION_ACCESS_VIOLATION.
Is this a problem with DLL implementation or in java application where multiple threads trying to access dll? Any ideas...
Following is my Error log:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x7c9108d3, pid=2880, tid=1584
#
# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) Client VM (14.2-b01 mixed mode windows-x86 )
# Problematic frame:
# C [ntdll.dll+0x108d3]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
--------------- T H R E A D ---------------
Current thread (0x0a9c9800): JavaThread "taskmgr" [_thread_in_native, id=1584, stack(0x0be70000,0x0bec0000)]
siginfo: ExceptionCode=0xc0000005, reading address 0x656c6441
Registers:
EAX=0x656c6441, EBX=0x0b2f0178, ECX=0x656c6449, EDX=0x000001aa
ESP=0x0bebf778, EBP=0x0bebf834, ESI=0x0b3a31b0, EDI=0x0b2f0000
EIP=0x7c9108d3, EFLAGS=0x00010293
Top of Stack: (sp=0x0bebf778)
0x0bebf778: 0b2f21d0 0b3a3860 00000000 00000000
0x0bebf788: 0bebf76c 0b3a31b0 0b2f0178 7403c6dc
0x0bebf798: 0b3b3c50 ffffffff 74005087 74005bec
0x0bebf7a8: 0b3b0ff8 0b6e30b8 656c6441 00000000
0x0bebf7b8: 00000000 0bebf7f0 00000000 0b6e2fa8
0x0bebf7c8: 0b2a1ef8 6a957160 656c6449 000006a8
0x0bebf7d8: 0b6e7740 00000001 0b712c98 00000035
0x0bebf7e8: 0bebf80c 7400605b 0b3b0ff8 74004866
Instructions: (pc=0x7c9108d3)
0x7c9108c3: d9 74 16 8d 41 f8 89 85 7c ff ff ff 66 8b 55 e4
0x7c9108d3: 66 3b 10 0f 87 93 fe ff ff 8d 46 08 89 85 64 ff
Stack: [0x0be70000,0x0bec0000], sp=0x0bebf778, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [ntdll.dll+0x108d3]
C [ProcessDLL.dll+0x127a2]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j process.info.NTProcessWin32.jniGetProcessInfosN([Ljava/lang/String;J)[Lprocess/info/NTProcess;+0
j process.info.NTProcessWin32.getProcessInfos([Ljava/lang/String;J)[Lprocess/info/NTProcess;+9
j process.info.InstanceController.getAndSetProcesses()V+17
j process.info.ControllerImpl$UpdateGraphTimer.run()V+7
j java.util.TimerThread.mainLoop()V+221
j java.util.TimerThread.run()V+1
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x003a7000 JavaThread "DestroyJavaVM" [_thread_blocked, id=1288, stack(0x00840000,0x00890000)]
=>0x0a9c9800 JavaThread "taskmgr" [_thread_in_native, id=1584, stack(0x0be70000,0x0bec0000)]
0x0afb8400 JavaThread "notepad" [_thread_blocked, id=3408, stack(0x0be20000,0x0be70000)]
0x0aee8800 JavaThread "mspaint" [_thread_blocked, id=3188, stack(0x0bdd0000,0x0be20000)]
0x0aee7400 JavaThread "mmc" [_thread_blocked, id=3196, stack(0x0bd80000,0x0bdd0000)]
0x0aa5dc00 JavaThread "firefox" [_thread_blocked, id=1224, stack(0x0bd30000,0x0bd80000)]
0x0b0b7c00 JavaThread "eclipse" [_thread_in_native, id=2628, stack(0x0bc80000,0x0bcd0000)]
0x0b7e4400 JavaThread "cmd" [_thread_blocked, id=3804, stack(0x0bc30000,0x0bc80000)]
0x0b7e0800 JavaThread "7zFM" [_thread_blocked, id=296, stack(0x0bbe0000,0x0bc30000)]
0x0b0b5000 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=3724, stack(0x0b790000,0x0b7e0000)]
0x0af9cc00 JavaThread "Thread-1" daemon [_thread_blocked, id=244, stack(0x0b0c0000,0x0b110000)]
0x0aa5a800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1480, stack(0x0ae20000,0x0ae70000)]
0x0aa57400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3656, stack(0x0add0000,0x0ae20000)]
0x0aa52800 JavaThread "JDWP Command Reader" daemon [_thread_in_native, id=3756, stack(0x0ad80000,0x0add0000)]
0x0aa47400 JavaThread "JDWP Event Helper Thread" daemon [_thread_blocked, id=3672, stack(0x0ad30000,0x0ad80000)]
0x0aa44c00 JavaThread "JDWP Transport Listener: dt_socket" daemon [_thread_blocked, id=3500, stack(0x0ace0000,0x0ad30000)]
0x0aa39c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=3660, stack(0x0abf0000,0x0ac40000)]
0x0aa38800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2604, stack(0x0aba0000,0x0abf0000)]
0x0aa29400 JavaThread "Finalizer" daemon [_thread_blocked, id=1628, stack(0x0ab50000,0x0aba0000)]
0x0aa24c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=3940, stack(0x0ab00000,0x0ab50000)]
Other Threads:
0x0aa20800 VMThread [stack: 0x0aab0000,0x0ab00000] [id=3048]
0x0aa6e000 WatcherThread [stack: 0x0ae70000,0x0aec0000] [id=3232]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
def new generation total 960K, used 733K [0x02910000, 0x02a10000, 0x02df0000)
eden space 896K, 74% used [0x02910000, 0x029b76a8, 0x029f0000)
from space 64K, 99% used [0x02a00000, 0x02a0fff8, 0x02a10000)
to space 64K, 0% used [0x029f0000, 0x029f0000, 0x02a00000)
tenured generation total 4096K, used 3164K [0x02df0000, 0x031f0000, 0x06910000)
the space 4096K, 77% used [0x02df0000, 0x031073c0, 0x03107400, 0x031f0000)
compacting perm gen total 12288K, used 8431K [0x06910000, 0x07510000, 0x0a910000)
the space 12288K, 68% used [0x06910000, 0x0714bc90, 0x0714be00, 0x07510000)
No shared spaces configured.
Dynamic libraries:
0x00400000 - 0x00424000 C:\Program Files\Java\jdk1.6.0_16\bin\javaw.exe
0x7c900000 - 0x7c9b2000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f6000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f03000 C:\WINDOWS\system32\RPCRT4.dll
0x77fe0000 - 0x77ff1000 C:\WINDOWS\system32\Secur32.dll
0x7e410000 - 0x7e4a1000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f59000 C:\WINDOWS\system32\GDI32.dll
0x76390000 - 0x763ad000 C:\WINDOWS\system32\IMM32.DLL
0x7c340000 - 0x7c396000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\msvcr71.dll
0x6d8b0000 - 0x6db3b000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\client\jvm.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x6d860000 - 0x6d86c000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\verify.dll
0x6d3e0000 - 0x6d3ff000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\java.dll
0x6d340000 - 0x6d348000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\hpi.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d430000 - 0x6d459000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\jdwp.dll
0x6d750000 - 0x6d756000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\npt.dll
0x6d8a0000 - 0x6d8af000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\zip.dll
0x6d2b0000 - 0x6d2b7000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\dt_socket.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\msvcrt.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76d60000 - 0x76d79000 C:\WINDOWS\system32\iphlpapi.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x10000000 - 0x1003b000 C:\DATA\ProcessDLL.dll
0x74000000 - 0x74056000 C:\WINDOWS\system32\pdh.dll
0x763b0000 - 0x763f9000 C:\WINDOWS\system32\comdlg32.dll
0x773d0000 - 0x774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.6028_x-ww_61e65202\COMCTL32.dll
0x77f60000 - 0x77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
0x7c9c0000 - 0x7d1d7000 C:\WINDOWS\system32\SHELL32.dll
0x77a80000 - 0x77b15000 C:\WINDOWS\system32\CRYPT32.dll
0x77b20000 - 0x77b32000 C:\WINDOWS\system32\MSASN1.dll
0x74320000 - 0x7435d000 C:\WINDOWS\system32\ODBC32.dll
0x711a0000 - 0x711a6000 C:\WINDOWS\system32\odbcbcp.dll
0x77c00000 - 0x77c08000 C:\WINDOWS\system32\VERSION.dll
0x774e0000 - 0x7761e000 C:\WINDOWS\system32\ole32.dll
0x77120000 - 0x771ab000 C:\WINDOWS\system32\OLEAUT32.dll
0x73000000 - 0x73026000 C:\WINDOWS\system32\WINSPOOL.DRV
0x0b280000 - 0x0b297000 C:\WINDOWS\system32\odbcint.dll
0x5e750000 - 0x5e75d000 C:\WINDOWS\system32\perfproc.dll
0x6d6c0000 - 0x6d6d3000 C:\Program Files\Java\jdk1.6.0_16\jre\bin\net.dll
VM Arguments:
jvm_args: -agentlib:jdwp=transport=dt_socket,suspend=y,address=localhost:1349 -Dfile.encoding=UTF-8
java_command: process.info.MainClass
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=C:\Program Files\Java\jdk1.6.0_16
PATH=C:\Program Files\Java\jdk1.6.0_16\jre\bin;C:/Program Files/Java/jre6/bin/client;C:/Program Files/Java/jre6/bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Java\jdk1.6.0_16\bin;C:\Program Files\Ant\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\NSIS;;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\WINDOWS\system32\WindowsPowerShell\v1.0
USERNAME=Administrator
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 42 Stepping 7, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows XP Build 2600 Service Pack 3
CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 42 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2
Memory: 4k page, physical 2096620k(1088228k free), swap 4036496k(3130120k free)
vm_info: Java HotSpot(TM) Client VM (14.2-b01) for windows-x86 JRE (1.6.0_16-b01), built on Jul 31 2009 11:26:58 by "java_re" with MS VC++ 7.1
time: Wed Apr 04 11:15:02 2012
elapsed time: 30 seconds
I think you PrecessDLL.dll is not Thread-Safe. You'll have to synchronize your code, to access it only once per time. Or you must fix your ProcessDLL.dll to be multithread-safe.
You may also update your JVM (1.6.16 is quite old), but I don't think, that this version leads to the error.
update
# The crash happened outside the Java Virtual Machine in native code.
The better approach (from my point of view) is to fix the DLL.