We are trying to execute 1 workflow for the first time on the uploaded blueprints(simple.yaml - blueprints downloaded from git hub) and getting an error:
" Internal error occurred in manager REST server - RuntimeError: Failed to find "create_deployment_environment" execution for deployment deploy2 "
Logs are as below :
ubuntu#cloudifyprecise:/opt/celery/cloudify.management__worker/work$ clearat cloudify.management_worker.log# ps aux | grep celery
ubuntu 13217 0.1 1.9 194856 39664 ? Ssl Jul30 10:38 /opt/celery/cloudify.management__worker/env/bin/python -m celery.bin.celeryd --include=cloudify_system_workflows.deployment_environment,plugin_installer.tasks,worker_installer.tasks,riemann_controller.tasks,cloudify.plugins.workflows --broker=amqp:// -n celery.cloudify.management --events --app=cloudify --loglevel=debug -Q cloudify.management --logfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.log --pidfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.pid --autoscale=5,2 --maxtasksperchild=10
ubuntu 13245 0.0 1.6 119084 34520 ? S Jul30 0:22 /opt/celery/cloudify.management__worker/env/bin/python -m celery.bin.celeryd --include=cloudify_system_workflows.deployment_environment,plugin_installer.tasks,worker_installer.tasks,riemann_controller.tasks,cloudify.plugins.workflows --broker=amqp:// -n celery.cloudify.management --events --app=cloudify --loglevel=debug -Q cloudify.management --logfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.log --pidfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.pid --autoscale=5,2 --maxtasksperchild=10
ubuntu 13246 0.0 1.6 119084 34588 ? S Jul30 0:21 /opt/celery/cloudify.management__worker/env/bin/python -m celery.bin.celeryd --include=cloudify_system_workflows.deployment_environment,plugin_installer.tasks,worker_installer.tasks,riemann_controller.tasks,cloudify.plugins.workflows --broker=amqp:// -n celery.cloudify.management --events --app=cloudify --loglevel=debug -Q cloudify.management --logfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.log --pidfile=/opt/celery/cloudify.management__worker/work/cloudify.management_worker.pid --autoscale=5,2 --maxtasksperchild=10
ubuntu 19572 0.0 0.0 8104 940 pts/8 S+ 07:04 0:00 grep --color=auto celery
ubuntu#cloudifyprecise:/opt/celery/cloudify.management__worker/work$ cat cloudify.management_worker.log
[2015-07-30 10:14:19,534: DEBUG/MainProcess] [Worker] Loading modules.
[2015-07-30 10:14:19,536: DEBUG/MainProcess] [Worker] Claiming components.
[2015-07-30 10:14:19,536: DEBUG/MainProcess] [Worker] Building boot step graph.
[2015-07-30 10:14:19,541: DEBUG/MainProcess] [Worker] New boot order: {ev, queues, beat, pool, mediator, autoreloader, timers, state-db, autoscaler, consumer}
[2015-07-30 10:14:19,551: DEBUG/MainProcess] Starting celery.worker.hub.Hub...
[2015-07-30 10:14:19,551: DEBUG/MainProcess] celery.worker.hub.Hub OK!
[2015-07-30 10:14:19,552: DEBUG/MainProcess] Starting celery.concurrency.processes.TaskPool...
[2015-07-30 10:14:19,555: DEBUG/MainProcess] celery.concurrency.processes.TaskPool OK!
[2015-07-30 10:14:19,556: DEBUG/MainProcess] Starting celery.worker.mediator.Mediator...
[2015-07-30 10:14:19,569: DEBUG/MainProcess] celery.worker.mediator.Mediator OK!
[2015-07-30 10:14:19,569: DEBUG/MainProcess] Starting __builtin__.NoneType...
[2015-07-30 10:14:19,570: DEBUG/MainProcess] __builtin__.NoneType OK!
[2015-07-30 10:14:19,570: DEBUG/MainProcess] Starting celery.worker.consumer.Consumer...
[2015-07-30 10:14:19,570: WARNING/MainProcess] celery#celery.cloudify.management ready.
[2015-07-30 10:14:19,570: DEBUG/MainProcess] consumer: Re-establishing connection to the broker...
[2015-07-30 10:14:19,587: DEBUG/MainProcess] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2013 GoPivotal, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'basic.nack': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'platform': u'Erlang/OTP', u'version': u'3.2.4'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
[2015-07-30 10:14:19,590: DEBUG/MainProcess] Open OK!
[2015-07-30 10:14:19,590: INFO/MainProcess] consumer: Connected to amqp://guest#127.0.0.1:5672//.
[2015-07-30 10:14:19,591: DEBUG/MainProcess] using channel_id: 1
[2015-07-30 10:14:19,591: DEBUG/MainProcess] Channel open
[2015-07-30 10:14:19,599: DEBUG/MainProcess] consumer: basic.qos: prefetch_count->4
[2015-07-30 10:14:19,599: DEBUG/MainProcess] using channel_id: 2
[2015-07-30 10:14:19,600: DEBUG/MainProcess] Channel open
[2015-07-30 10:14:19,610: DEBUG/MainProcess] consumer: Ready to accept tasks!
ubuntu#cloudifyprecise:/opt/celery/cloudify.management__worker/work$
Looking at the error code that you can find here: https://github.com/cloudify-cosmo/cloudify-manager/blob/3.1-build/rest-service/manager_rest/blueprints_manager.py
There could be several scenarios that you would get this error.
If for example you have deleted the deployment before running this workflow,or if you have stopped some of the services of the manager.
I would try to recreate this deployment and execute the install workflow to see if it does reproduce.
Related
When I start the data nodeid 1 (10.1.1.103) of MySQL Cluster 8.0 on Ubuntu 22.04 LTS I am getting the following error:
# ndbd
Failed to open /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list: No such file or directory
2023-01-02 17:16:55 [ndbd] INFO -- Angel connected to '10.1.1.102:1186'
2023-01-02 17:16:55 [ndbd] INFO -- Angel allocated nodeid: 2
When I start data nodeid 2 (10.1.1.105) I get the following error:
# ndbd
Failed to open /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list: No such file or directory
2023-01-02 11:10:04 [ndbd] INFO -- Angel connected to '10.1.1.102:1186'
2023-01-02 11:10:04 [ndbd] ERROR -- Failed to allocate nodeid, error: 'Error: Could not alloc node id at 10.1.1.102:1186: Connection done from wrong host ip 10.1.1.105.'
The management node log file reports (on /var/lib/mysql-cluster/ndb_1_cluster.log):
2023-01-02 11:28:47 [MgmtSrvr] INFO -- Node 2: Initial start, waiting for 3 to connect, nodes [ all: 2 and 3 connected: 2 no-wait: ]
What is the relevance of failing to open: /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list: No such file or directory?
Why is data node on 10.1.1.105 unable to allocate a nodeid?
I initially installed a single Management Node on 10.1.1.102:
wget https://dev.mysql.com/get/Downloads/MySQL-Cluster-8.0/mysql-cluster_8.0.31-1ubuntu22.04_amd64.deb-bundle.tar
tar -xf mysql-cluster_8.0.31-1ubuntu22.04_amd64.deb-bundle.tar
dpkg -i mysql-cluster-community-management-server_8.0.31-1ubuntu22.04_amd64.deb
mkdir /var/lib/mysql-cluster
vi /var/lib/mysql-cluster/config.ini
The configuration set up on config.ini:
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2 # Number of replicas
[ndb_mgmd]
# Management process options:
hostname=10.1.1.102 # Hostname of the manager
datadir=/var/lib/mysql-cluster # Directory for the log files
[ndbd]
hostname=10.1.1.103 # Hostname/IP of the first data node
NodeId=2 # Node ID for this data node
datadir=/usr/local/mysql/data # Remote directory for the data files
[ndbd]
hostname=10.1.1.105 # Hostname/IP of the second data node
NodeId=3 # Node ID for this data node
datadir=/usr/local/mysql/data # Remote directory for the data files
[mysqld]
# SQL node options:
hostname=10.1.1.102 # In our case the MySQL server/client is on the same Droplet as the cluster manager
I then started and killed the running server and created a systemd file for Cluster manager:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini
pkill -f ndb_mgmd
vi /etc/systemd/system/ndb_mgmd.service
Adding the following configuration:
[Unit]
Description=MySQL NDB Cluster Management Server
After=network.target auditd.service
[Service]
Type=forking
ExecStart=/usr/sbin/ndb_mgmd -f /var/lib/mysql-cluster/config.ini
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
[Install]
WantedBy=multi-user.target
I then reloaded the systemd daemon to apply the changes, started and enabled the Cluster Manager and checked its active status:
systemctl daemon-reload
systemctl start ndb_mgmd
systemctl enable ndb_mgmd
Here is the status of the Cluster Manager:
# systemctl status ndb_mgmd
● ndb_mgmd.service - MySQL NDB Cluster Management Server
Loaded: loaded (/etc/systemd/system/ndb_mgmd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-01-01 08:25:07 CST; 27min ago
Main PID: 320972 (ndb_mgmd)
Tasks: 12 (limit: 9273)
Memory: 2.5M
CPU: 35.467s
CGroup: /system.slice/ndb_mgmd.service
└─320972 /usr/sbin/ndb_mgmd -f /var/lib/mysql-cluster/config.ini
Jan 01 08:25:07 nuc systemd[1]: Starting MySQL NDB Cluster Management Server...
Jan 01 08:25:07 nuc ndb_mgmd[320971]: MySQL Cluster Management Server mysql-8.0.31 ndb-8.0.31
Jan 01 08:25:07 nuc systemd[1]: Started MySQL NDB Cluster Management Server.
I then set up a data node on 10.1.1.103, installing dependencies, downloading the data node and setting up its config:
apt update && apt -y install libclass-methodmaker-perl
wget https://dev.mysql.com/get/Downloads/MySQL-Cluster-8.0/mysql-cluster_8.0.31-1ubuntu22.04_amd64.deb-bundle.tar
tar -xf mysql-cluster_8.0.31-1ubuntu22.04_amd64.deb-bundle.tar
dpkg -i mysql-cluster-community-data-node_8.0.31-1ubuntu22.04_amd64.deb
vi /etc/my.cnf
I entered the address of the Cluster Management Node in the configuration:
[mysql_cluster]
# Options for NDB Cluster processes:
ndb-connectstring=10.1.1.102 # location of cluster manager
I then created a data directory and started the node:
mkdir -p /usr/local/mysql/data
ndbd
This is when I got the "Failed to open" error result on data nodeid 1 (102.1.1.103):
# ndbd
Failed to open /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list: No such file or directory
2023-01-02 17:16:55 [ndbd] INFO -- Angel connected to '10.1.1.102:1186'
2023-01-02 17:16:55 [ndbd] INFO -- Angel allocated nodeid: 2
UPDATED (2023-01-02)
Thank you #MauritzSundell. I corrected the (private) IP addresses above and no longer got:
# ndbd
Failed to open /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list: No such file or directory
ERROR: Unable to connect with connect string: nodeid=0,10.1.1.2:1186
Retrying every 5 seconds. Attempts left: 12 11 10 9 8 7 6 5 4 3 2 1, failed.
2023-01-01 14:41:57 [ndbd] ERROR -- Could not connect to management server, error: ''
Also #MauritzSundell, in order to use the ndbmtd process rather than the ndbd process, does any alteration need to be made to any of the configuration files (e.g. /etc/systemd/system/ndb_mgmd.service)?
What is the appropriate reference/tutorial documentation for MySQL Cluster 8.0? Is it MySQL Cluster "MySQL NDB Cluster 8.0" on:
https://downloads.mysql.com/docs/mysql-cluster-excerpt-8.0-en.pdf
Or is it "MySQL InnoDB Cluster" on:
https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-introduction.html
Not sure I understand the difference.
I am trying to run a dedicated coreNLP server using a CentOS7 server hosted on Linode.com. I have followed all the instructions from https://stanfordnlp.github.io/CoreNLP/corenlp-server.html#dedicated-server and then read and consequently updated my scripts and user permissions using this post: Dedicated CoreNLP Server Control Issues but unfortunately I still can't connect to the service through any means. Every time it says 'connection refused' if I try and connect through localhost, or load the ip in my browser, or try and connect through he android app I am attempting to develop.
Prior to starting the coreNLP service I disabled all firewalls using iptables scripts and disabling firewalld. I am running as user nlp on port 80. I tried to icmp ping the server from another machine which worked fine.
I changed access permissions in the /opt/corenlp directory although I'm not sure that made a difference. Here is my terminal output. I used systemctl although the sudo service command also produced the same thing.
sudo systemctl start corenlp
[nlp#aqghost /]$ sudo systemctl status corenlp -l
● corenlp.service - (null)
Loaded: loaded (/etc/rc.d/init.d/corenlp; bad; vendor preset: disabled)
Active: active (exited) since Thu 2021-06-17 13:53:59 BST; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 9413 ExecStart=/etc/rc.d/init.d/corenlp start (code=exited, status=0/SUCCESS)
Jun 17 13:53:59 aqghost systemd[1]: Starting (null)...
Jun 17 13:53:59 aqghost corenlp[9413]: CoreNLP server is already running!
Jun 17 13:53:59 aqghost corenlp[9413]: If you are sure this is a mistake, delete the file:
Jun 17 13:53:59 aqghost corenlp[9413]: /opt/corenlp/corenlp.shutdown
Jun 17 13:53:59 aqghost systemd[1]: Started (null).
[nlp#aqghost /]$ sudo systemctl start corenlp
[nlp#aqghost /]$ sudo systemctl status
● aqghost
State: running
Jobs: 0 queued
Failed: 0 units
Since: Thu 2021-06-17 10:35:33 BST; 3h 18min ago
CGroup: /
├─1 /usr/lib/systemd/systemd --switched-root --system --deserializ
├─user.slice
│ ├─user-0.slice
│ │ └─session-25.scope
│ │ ├─9263 sshd: root#pts/1
│ │ └─9267 -bash
│ └─user-1000.slice
│ └─session-21.scope
│ ├─8745 sshd: nlp [priv]
│ ├─8751 sshd: nlp#pts/0
│ ├─8752 -bash
│ ├─9426 sudo systemctl status
│ ├─9428 systemctl status
│ └─9429 less
└─system.slice
├─tuned.service
│ └─799 /usr/bin/python2 -Es /usr/sbin/tuned -l -P
├─rsyslog.service
│ └─797 /usr/sbin/rsyslogd -n
├─sshd.service
│ └─796 /usr/sbin/sshd -D
├─crond.service
│ └─577 /usr/sbin/crond -n
├─systemd-logind.service
│ └─573 /usr/lib/systemd/systemd-logind
├─NetworkManager.service
│ └─572 /usr/sbin/NetworkManager --no-daemon
├─dbus.service
│ └─564 /usr/bin/dbus-daemon --system --address=systemd: --nofor
├─polkit.service
│ └─563 /usr/lib/polkit-1/polkitd --no-debug
├─irqbalance.service
│ └─559 /usr/sbin/irqbalance --foreground
├─auditd.service
│ └─512 /sbin/auditd
├─systemd-udevd.service
│ └─430 /usr/lib/systemd/systemd-udevd
├─system-serial\x2dgetty.slice
│ └─serial-getty#ttyS0.service
│ └─583 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220
├─system-getty.slice
│ └─getty#tty1.service
│ └─584 /sbin/agetty --noclear tty1 linux
└─systemd-journald.service
└─399 /usr/lib/systemd/systemd-journald
[nlp#aqghost /]$ sudo ps aux | grep java
nlp 9431 0.0 0.0 112812 964 pts/0 S+ 13:54 0:00 grep --color=auto java
(Note. I seemed to make the server work ONCE, for about 20 seconds. I'm not sure how and why but I could connect via my app that i'm dveloping and via localhost on the same server. This is also what showed in my stderr.log and stdout.log: (ignore the fact that theres four outputs here, i'm still leanring JSON formatting and i think i sent the same thing))
sudo cat /opt/corenlp/stdout.log
the quick brown fox jumped over the lazy dog
{"inputFormat":"text","data":"I am going to the shops.","annotators":"tokenize, ssplit, pos, lemma, ner, parse, dcoref","outputFormat":"json"}
{"inputFormat":"text","data":"I am going to the shops.","annotators":"tokenize, ssplit, pos, lemma, ner, parse, dcoref","outputFormat":"json"}
{"inputFormat":"text","data":"I am going to the shops.","annotators":"tokenize, ssplit, pos, lemma, ner, parse, dcoref","outputFormat":"json"}
{"inputFormat":"text","data":"I am going to the shops.","annotators":"tokenize, ssplit, pos, lemma, ner, parse, dcoref","outputFormat":"json"}
$ sudo cat /opt/corenlp/stderr.log
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - Server default properties:
(Note: unspecified annotator properties are English defaults)
inputFormat = text
outputFormat = json
prettyPrint = false
[main] INFO CoreNLP - Threads: 4
[main] INFO CoreNLP - Starting server...
[main] INFO CoreNLP - StanfordCoreNLPServer listening at /0.0.0.0:80
[pool-1-thread-1] INFO CoreNLP - [/127.0.0.1:60614] API call w/annotators tokenize,ssplit,pos
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[pool-1-thread-1] INFO edu.stanford.nlp.tagger.maxent.MaxentTagger - Loading POS tagger from edu/stanford/nlp/models/pos-tagger/english-left3words-distsim.tagger ... done [1.0 sec].
[pool-1-thread-1] INFO CoreNLP - [/128.243.2.12:48192] API call w/annotators <unknown>
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Searching for resource: StanfordCoreNLP.properties ... found.
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator lemma
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
[pool-1-thread-1] INFO edu.stanford.nlp.ie.AbstractSequenceClassifier - Loading classifier from edu/stanford/nlp/models/ner/english.all.3class.distsim.crf.ser.gz ... done [1.2 sec].
[pool-1-thread-1] INFO edu.stanford.nlp.ie.AbstractSequenceClassifier - Loading classifier from edu/stanford/nlp/models/ner/english.muc.7class.distsim.crf.ser.gz ... done [0.8 sec].
[pool-1-thread-2] INFO CoreNLP - [/128.243.2.12:48196] API call w/annotators <unknown>
[pool-1-thread-1] INFO edu.stanford.nlp.ie.AbstractSequenceClassifier - Loading classifier from edu/stanford/nlp/models/ner/english.conll.4class.distsim.crf.ser.gz ... done [1.1 sec].
[pool-1-thread-1] INFO edu.stanford.nlp.time.JollyDayHolidays - Initializing JollyDayHoliday for SUTime from classpath edu/stanford/nlp/models/sutime/jollyday/Holidays_sutime.xml as sutime.binder.1.
[pool-1-thread-1] INFO edu.stanford.nlp.time.TimeExpressionExtractorImpl - Using following SUTime rules: edu/stanford/nlp/models/sutime/defs.sutime.txt,edu/stanford/nlp/models/sutime/english.sutime.txt,edu/stanford/nlp/models/sutime/english.holidays.sutime.txt
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokensRegexNERAnnotator - ner.fine.regexner: Read 580705 unique entries out of 581864 from edu/stanford/nlp/models/kbp/english/gazetteers/regexner_caseless.tab, 0 TokensRegex patterns.
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokensRegexNERAnnotator - ner.fine.regexner: Read 4867 unique entries out of 4867 from edu/stanford/nlp/models/kbp/english/gazetteers/regexner_cased.tab, 0 TokensRegex patterns.
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.TokensRegexNERAnnotator - ner.fine.regexner: Read 585572 unique entries from 2 files
[pool-1-thread-3] INFO CoreNLP - [/128.243.2.12:48228] API call w/annotators <unknown>
[pool-1-thread-4] INFO CoreNLP - [/128.243.2.12:48230] API call w/annotators <unknown>
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.NERCombinerAnnotator - numeric classifiers: true; SUTime: true [no docDate]; fine grained: true
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator depparse
[pool-1-thread-1] INFO edu.stanford.nlp.parser.nndep.DependencyParser - Loading depparse model: edu/stanford/nlp/models/parser/nndep/english_UD.gz ... Time elapsed: 67.3 sec
[pool-1-thread-1] INFO edu.stanford.nlp.parser.nndep.Classifier - PreComputed 20000 vectors, elapsed Time: 1.244 sec
[pool-1-thread-1] INFO edu.stanford.nlp.parser.nndep.DependencyParser - Initializing dependency parser ... done [68.6 sec].
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator coref
[pool-1-thread-1] INFO edu.stanford.nlp.coref.statistical.SimpleLinearClassifier - Loading coref model edu/stanford/nlp/models/coref/statistical/ranking_model.ser.gz ... done [1.5 sec].
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.CorefMentionAnnotator - Using mention detector type: dependency
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator kbp
[pool-1-thread-1] INFO edu.stanford.nlp.pipeline.KBPAnnotator - Loading KBP classifier from: edu/stanford/nlp/models/kbp/english/tac-re-lr.ser.gz
Killed
Here is my service script modified according to the linked post:
#!/bin/bash
#
# A script to start/stop the CoreNLP server on port 80, made
# in particular for the configuration running at corenlp.run.
# This script should be placed into:
#
# /etc/init.d/corenlp
#
# To run it at startup, link to the script using:
#
# ln -s /etc/init.d/conenlp /etc/rc.2/S75corenlp
#
# Usage:
#
# service corenlp [start|stop]
# ./corenlp [start|stop]
#
#
# Set this to the username you would like to use to run the server.
# Make sure that this user can authbind to port 80!
#
SERVER_USER="nlp"
CORENLP_DIR="/opt/corenlp"
do_start()
{
if [ -e "$CORENLP_DIR/corenlp.shutdown" ]; then
echo "CoreNLP server is already running!"
echo "If you are sure this is a mistake, delete the file:"
echo "$CORENLP_DIR/corenlp.shutdown"
else
export CLASSPATH=""
for JAR in `find "$CORENLP_DIR" -name "*.jar"`; do
CLASSPATH="$CLASSPATH:$JAR"
done
nohup su nlp -c "sudo /usr/bin/authbind --deep java -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=/opt/corenlp -cp "$CLASSPATH" -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+UseStringDeduplication -XX:SoftRefLRUPolicyMSPerMB=100 -mx20g edu.stanford.nlp.pipeline.StanfordCoreNLPServer -port 80" \
> "$CORENLP_DIR/stdout.log" 2> "$CORENLP_DIR/stderr.log" &
echo "CoreNLP server started."
fi
}
do_stop()
{
if [ ! -e "$CORENLP_DIR/corenlp.shutdown" ]; then
echo "CoreNLP server is not running"
else
KEY=`cat "$CORENLP_DIR/corenlp.shutdown"`
curl "localhost/shutdown?key=$KEY"
echo "CoreNLP server stopped"
fi
}
do_restart()
{
do_stop
sleep 3
do_start
}
case $1 in
start) do_start
;;
stop) do_stop
;;
restart) do_restart
;;
esac
Sorry for all the info. I am new to linux and running a server and initialising scripts so please be easy on me.
I have built an image based on mariadb:10.1 which basically adds a new cluster.conf but facing the following error on the second node after the first node started working successfully. Can somebody help me debug here?
Error log tail
2016-09-28 10:12:55 139799503415232 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/pc.cpp:connect():162
2016-09-28 10:12:55 139799503415232 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
2016-09-28 10:12:55 139799503415232 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'test_cluster' at 'gcomm://172.17.0.2,172.17.0.3,172.17.0.4': -110 (Connection timed out)
2016-09-28 10:12:55 139799503415232 [ERROR] WSREP: gcs connect failed: Connection timed out
2016-09-28 10:12:55 139799503415232 [ERROR] WSREP: wsrep::connect(gcomm://172.17.0.2,172.17.0.3,172.17.0.4) failed: 7
2016-09-28 10:12:55 139799503415232 [ERROR] Aborting
MySQL init process failed.
Debugging steps taken
NOTE: Container IP addresses were ensured to be the same as shown.
To ensure networking between containers is working, tried creating another container which could login to the first container's mysql instance.
This is definitely not related to MYSQL_HOST
To see if the container was running out of memory, I used docker stats and saw that the failed container was using only a meagre 142MB all through its lifecycle until it failed, which is way lesser than the total memory it was allowed (~4GB).
I am using Docker for Mac, but tried running the same on a CentOS VirtualBox and gives the same results. Doesn't look like Docker on Mac has a problem.
Config
[mysqld]
user=mysql
binlog_format=ROW
bind-address=0.0.0.0
default_storage_engine=innodb
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=122M
innodb_file_per_table=1
innodb_doublewrite=1
query_cache_size=0
query_cache_type=0
wsrep_on=ON
wsrep_provider=/usr/lib/libgalera_smm.so
wsrep_sst_method=rsync
Steps to start containers
# bootstrap node
docker run --rm -e MYSQL_ROOT_PASSWORD=123 \
activatedgeek/mariadb:devel \
--wsrep-cluster-name=test_cluster \
--wsrep-cluster-address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4 \
--wsrep-new-cluster
# add node into cluster
docker run --rm -e MYSQL_ROOT_PASSWORD=123 \
activatedgeek/mariadb:devel \
--wsrep-cluster-name=test_cluster \
--wsrep-cluster-address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4
# add node into cluster
docker run --rm -e MYSQL_ROOT_PASSWORD=123 \
activatedgeek/mariadb:devel \
--wsrep-cluster-name=test_cluster \
--wsrep-cluster-address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4
This problem is caused due to the hanging init process. The configurations and CLI arguments above are correct. The only thing to be done before the init process starts is to create and empty mysql directory in the data directory (/var/lib/mysql by default). The must only be created on all nodes except the bootstrap node.
mkdir -p /var/lib/mysql/mysql
See sample MariaDB Cluster for usage which uses a custom MariaDB image and is a proof of concept for creating clusters.
I guess your containers should either expose the required ports:
-p 3306:3306 -p 4444:4444 -p 4567:4567 -p 4568:4568
or should be --link (ed) together.
I have installed riak (apt-get) on an EC2 instance, lucid, amd64 with libssl.
When running riak start I get:
Attempting to restart script through sudo -H -u riak
Riak failed to start within 15 seconds,
see the output of 'riak console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.
Running riak console:
Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot /usr/lib/riak/releases/1.3.1/riak
-embedded -config /etc/riak/app.config
-pa /usr/lib/riak/lib/basho-patches
-args_file /etc/riak/vm.args -- console
Root: /usr/lib/riak
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:2:2] [async-threads:64] [kernel-poll:true]
/usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,riak_core, {shutdown,{riak_core_app,start,[normal,[]]}}}"}
Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_core, {shutdown,{riak_core_app,start,[normal,[]]}}})
The error logs:
2013-04-24 11:36:20.897 [error] <0.146.0> CRASH REPORT Process riak_core_handoff_listener with 1 neighbours exited with reason: bad return value: {error,eaddrinuse} in gen_server:init_it/6 line 332
2013-04-24 11:36:20.899 [error] <0.145.0> Supervisor riak_core_handoff_listener_sup had child riak_core_handoff_listener started with riak_core_handoff_listener:start_link() at undefined exit with reason bad return value: {error,eaddrinuse} in context start_error
2013-04-24 11:36:20.902 [error] <0.142.0> Supervisor riak_core_handoff_sup had child riak_core_handoff_listener_sup started with riak_core_handoff_listener_sup:start_link() at undefined exit with reason shutdown in context start_error
2013-04-24 11:36:20.903 [error] <0.130.0> Supervisor riak_core_sup had child riak_core_handoff_sup started with riak_core_handoff_sup:start_link() at undefined exit with reason shutdown in context start_error
I'm new to Riak and basically tried to run through the "Fast Track" docs.
None of the default core IP settings in the configs have been changed. They are still set to {http, [ {"127.0.0.1", 8098 } ]}, {handoff_port, 8099 }
Any help would be greatly appreciated.
I know this is old but there is some solid documentation about the errors in the crash.dump file on the Riak site.
I'm preparing to use nginx/uwsgi with flask for a website i'm developing, but i'm running into problems. NB the website itself runs great using flask's debug :5000 port, but i want to go into production now. To explain what i did.
It's a linode ubuntu 12.04LTS server, I installed it like this:
# install nginx
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:nginx/stable
sudo apt-get update
sudo apt-get upgrade --show-upgraded
sudo apt-get install nginx-full
# installing uwsgi
sudo apt-get install build-essential python-dev libxml2-dev
sudo apt-get install libc6 libexpat1 libgd2-xpm libgeoip1 libpam0g libpcre3 libssl1.0.0 libxml2 libxslt1.1 zlib1g
sudo pip install uwsgi
# python basics
sudo apt-get install python-pip build-essential python-dev
sudo pip install virtualenv
sudo pip install virtualenvwrapper
sudo mkdir -p /srv/www/li/
cd /srv/www/li/
virtualenv venv
source /srv/www/li/venv/bin/activate
pip install flask
Then i set out to configure everything, but I already run into trouble with uwsgi (never mind NGINX, which will be the next step.
sudo nano /etc/uwsgi/apps-available/li.xml
<uwsgi>
<plugin>python</plugin>
<socket>/run/uwsgi/app/li.socket</socket>
<chmod-socket>666</chmod-socket>
<chdir>/srv/www/li</chdir>
<pythonpath>/srv/www/li</pythonpath>
<virtualenv>/srv/www/li/venv</virtualenv>
<module>li</module>
<wsgi-file>/srv/www/li/li.py</wsgi-file>
<callable>app</callable>
<master/>
<processes>4</processes>
<harakiri>60</harakiri>
<reload-mercy>8</reload-mercy>
<cpu-affinity>1</cpu-affinity>
<stats>/tmp/stats.socket</stats>
<max-requests>2000</max-requests>
<limit-as>512</limit-as>
<reload-on-as>256</reload-on-as>
<reload-on-rss>192</reload-on-rss>
<no-orphans/>
<vacuum/>
</uwsgi>
sudo ln -s /etc/uwsgi/apps-available/li.xml /etc/uwsgi/apps-enabled/li.xml
However if i run it, i get:
uwsgi --xml /etc/uwsgi/apps-enabled/li.xml
[uWSGI] parsing config file /etc/uwsgi/apps-enabled/li.xml
open("./python_plugin.so"): No such file or directory [core/utils.c line 4755]
!!! UNABLE to load uWSGI plugin: ./python_plugin.so: cannot open shared object file: No such file or directory !!!
*** Starting uWSGI 1.4.6 (64bit) on [Thu Feb 28 16:30:53 2013] ***
compiled with version: 4.6.3 on 28 February 2013 12:38:22
os: Linux-3.7.10-x86_64-linode30 #1 SMP Wed Feb 27 14:29:31 EST 2013
nodename: demo
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /run/uwsgi/app
detected binary path: /usr/local/bin/uwsgi
your processes number limit is 63594
limiting address space of processes...
your process address space limit is 536870912 bytes (512 MB)
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 bound to UNIX address /run/uwsgi/app/li.socket fd 3
Python version: 2.7.3 (default, Aug 1 2012, 05:25:23) [GCC 4.6.3]
Set PythonHome to /srv/www/li/venv
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xa86e20
your server socket listen backlog is limited to 100 connections
mapped 362120 bytes (353 KB) for 4 cores
*** Operational MODE: preforking ***
added /srv/www/li/ to pythonpath.
/srv/www/li/venv/local/lib/python2.7/site-packages/mongoengine/fields.py:744: FutureWarning: ReferenceFields will default to using ObjectId strings in 0.8, set DBRef=True if this isn't desired
warnings.warn(msg, FutureWarning)
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xa86e20 pid: 14934 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 14934)
spawned uWSGI worker 1 (pid: 14940, cores: 1)
mapping worker 1 to CPUs: 0
spawned uWSGI worker 2 (pid: 14941, cores: 1)
mapping worker 2 to CPUs: 1
spawned uWSGI worker 3 (pid: 14942, cores: 1)
mapping worker 3 to CPUs: 2
spawned uWSGI worker 4 (pid: 14943, cores: 1)
unlink(): Operation not permitted [core/socket.c line 109]
bind(): Address already in use [core/socket.c line 141]
...brutally killing workers...
mapping worker 4 to CPUs: 3
VACUUM: unix socket /run/uwsgi/app/li.socket removed.
So i get the unlink operation not permitted and the bind address already in use errors (next to the python_plugin error of which i also have no clue how to solve that!). If i run as sudo, it seems to work fine ->
sudo uwsgi --xml /etc/uwsgi/apps-enabled/li.xml
[uWSGI] parsing config file /etc/uwsgi/apps-enabled/li.xml
open("./python_plugin.so"): No such file or directory [core/utils.c line 4755]
!!! UNABLE to load uWSGI plugin: ./python_plugin.so: cannot open shared object file: No such file or directory !!!
*** Starting uWSGI 1.4.6 (64bit) on [Thu Feb 28 15:47:41 2013] ***
compiled with version: 4.6.3 on 28 February 2013 12:38:22
os: Linux-3.7.10-x86_64-linode30 #1 SMP Wed Feb 27 14:29:31 EST 2013
nodename: demo
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /run/uwsgi
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 63594
limiting address space of processes...
your process address space limit is 536870912 bytes (512 MB)
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
uwsgi socket 0 bound to UNIX address /run/uwsgi/app/li.socket fd 3
Python version: 2.7.3 (default, Aug 1 2012, 05:25:23) [GCC 4.6.3]
Set PythonHome to /srv/www/li/venv
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1fc9d00
your server socket listen backlog is limited to 100 connections
mapped 362120 bytes (353 KB) for 4 cores
*** Operational MODE: preforking ***
added /srv/www/li/ to pythonpath.
/srv/www/li/venv/local/lib/python2.7/site-packages/mongoengine/fields.py:744: FutureWarning: ReferenceFields will default to using ObjectId strings in 0.8, set DBRef=True if this isn't desired
warnings.warn(msg, FutureWarning)
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1fc9d00 pid: 14755 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 14755)
spawned uWSGI worker 1 (pid: 14761, cores: 1)
mapping worker 1 to CPUs: 0
spawned uWSGI worker 2 (pid: 14762, cores: 1)
mapping worker 2 to CPUs: 1
spawned uWSGI worker 3 (pid: 14763, cores: 1)
mapping worker 3 to CPUs: 2
spawned uWSGI worker 4 (pid: 14764, cores: 1)
*** Stats server enabled on /tmp/stats.socket fd: 16 ***
mapping worker 4 to CPUs: 3
Can anyone please help me? As www-data is in the www-data group and he runs it, I tried some stuff:
sudo usermod -a -G www-data $USER
sudo chown -R $USER:www-data /srv/www/li
sudo chmod -R g+r+w+x /srv/www/li
sudo chown -R $USER:www-data /etc/uwsgi/apps-enabled
sudo chmod -R g+r+w+x /etc/uwsgi/apps-enabled
sudo chown -R $USER:www-data /run/uwsgi/app
sudo chmod -R g+r+w+x /run/uwsgi/app
But that really didn't help either. I also tried a tcp port instead of the unix /run/uwsgi/app/ port that didn't make any difference either...
This is driving me crazy :( I hope someone has a clue on what's happening here.
Kind regards,
Carst
edit: after a server restart it still gives an erro but a different one:
geoadmin#demo:~$ uwsgi --xml /etc/uwsgi/apps-enabled/li.xml
[uWSGI] parsing config file /etc/uwsgi/apps-enabled/li.xml
*** Starting uWSGI 1.4.6 (64bit) on [Thu Feb 28 18:47:36 2013] ***
compiled with version: 4.6.3 on 28 February 2013 12:38:22
os: Linux-3.7.10-x86_64-linode30 #1 SMP Wed Feb 27 14:29:31 EST 2013
nodename: demo
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /home/geoadmin
detected binary path: /usr/local/bin/uwsgi
your processes number limit is 63594
limiting address space of processes...
your process address space limit is 536870912 bytes (512 MB)
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
bind(): No such file or directory [core/socket.c line 141]
This was consistently my #1 result on google, and this page was relatively unhelpful to me, so I'm going to add my answer, even though it's fairly obvious in retrospect.
My problem was a permissions problem with my stats socket. If you change your uWSGI config's uid or gid parameters, make sure you either chmod or rm all of your old sockets/ pids, and their parent folders.
In my case I was trying to place the .sock file in the /vagrant directory, which is a machine mounted folder of virtual box and is not good for much more than reads and writes.
Place the .sock file outside of a virtualbox mount-point preferably in /tmp The FHS says: /var/run
Ref:
https://stackoverflow.com/a/7580524/1695680
Ok, after the later edit i checked the directories and the socket directory didnt exist (anymore); i think it had to do with the original apt-get install versus my later pip install... still have the issue with the python plugin but will check if it's necessary for nginx or if it will work without it... 8 hours of work over a reset, d'oh ;)
#bearrito:
In the end I put the socket in the tmp directory to avoid rights issues:
<uwsgi>
<uid>www-data</uid>
<gid>www-data</gid>
<plugin>python</plugin>
<socket>/tmp/li.socket</socket>
<chmod-socket>666</chmod-socket>
<chdir>/srv/www/li</chdir>
<pythonpath>/srv/www/li</pythonpath>
<virtualenv>/srv/www/li/venv</virtualenv>
<module>li</module>
<wsgi-file>/srv/www/li/li.py</wsgi-file>
<callable>app</callable>
<master/>
<processes>2</processes>
<pidfile>/tmp/li.pid</pidfile>
<harakiri>120</harakiri>
<reload-mercy>8</reload-mercy>
<cpu-affinity>1</cpu-affinity>
<stats>/tmp/stats.socket</stats>
<max-requests>2000</max-requests>
<limit-as>2048</limit-as>
<reload-on-as>2048</reload-on-as>
<reload-on-rss>1024</reload-on-rss>
<no-orphans/>
<vacuum/>
</uwsgi>
I hope this helps!
For me the solution was to remove /var/run/uwsgi/.sock and
chmod 775 /var/run/uwsgi
chmod 777 /var/log/uwsgi
or wherever your uwsgi files are.