eucalyptus-cloud.service 4.4.4 constantly crashing with JVM memory errors - eucalyptus

I'm trying to work through the manual installation guide. When I get as far as registering an admin account for the console (euare-accountcreate) or registering services (euserv-register-service) the eucalyptus-cloud service inconsistently crashes with the same JVM memory error. As an example a few times I've been able to register the UFS or create the admin account but when I move on to perform another step in the install it will fail and checking the service status shows that it has crashed. I don't have experience with Java memory errors and could really use some help understanding whats going on and how to investigate this type of error in general:
[root#cloud ~]# euserv-describe-services ufs-10.0.0.2
euserv-describe-services: error: connection error (('Connection
aborted.', BadStatusLine("''",)))
[root#cloud ~]# systemctl status eucalyptus-cloud -l
● eucalyptus-cloud.service - Eucalyptus cloud web services
Loaded: loaded (/usr/lib/systemd/system/eucalyptus-cloud.service;
enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2018-10-10 08:45:09 EDT;
4s ago
Process: 45951 ExecStart=/usr/sbin/eucalyptus-cloud $CLOUD_OPTS
(code=exited, status=1/FAILURE)
Main PID: 45951 (code=exited, status=1/FAILURE)
CGroup: /system.slice/eucalyptus-cloud.service
├─46206 /usr/bin/postgres -D /var/lib/eucalyptus/db/data -
h0.0.0.0 -p8777
├─46207 postgres: logger process
├─46209 postgres: checkpointer process
├─46210 postgres: writer process
├─46211 postgres: wal writer process
├─46212 postgres: autovacuum launcher process
└─46213 postgres: stats collector process
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: OpenJDK 64-Bit Server VM
warning: INFO: os::commit_memory(0x00007f496534b000, 12288, 0) failed;
error='Cannot allocate memory' (errno=12)
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: #
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # There is insufficient
memory for the Java Runtime Environment to continue.
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # Native memory
allocation (mmap) failed to map 12288 bytes for committing reserved
memory.
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: # An error report file
with more information is saved as:
Oct 10 08:45:08 cloud eucalyptus-cloud[45951]: #
/tmp/hs_err_pid45954.log
Oct 10 08:45:09 cloud eucalyptus-cloud[45951]: 2018-10-10 08:45:09
45951 ERROR 0574 Service exit with a return value of 1.
Oct 10 08:45:09 cloud systemd[1]: eucalyptus-cloud.service: main
process exited, code=exited, status=1/FAILURE
Oct 10 08:45:09 cloud systemd[1]: Unit eucalyptus-cloud.service
entered failed state.
Oct 10 08:45:09 cloud systemd[1]: eucalyptus-cloud.service failed.
Edit to include some version info:
[root#cloud ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core)
[root#cloud ~]# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
EDIT to note my point of confusion: This system only has 25G of RAM in use and 223G free. So I need help understanding how java is running out of memory.

try this fix:
echo 1999999 > /proc/sys/vm/max_map_count

Related

installing asterisk on ubuntu 18.04

Assuming you are Database Root
Checking if SELinux is enabled...Its not (good)!
Reading /etc/asterisk/asterisk.conf...Done
Checking if Asterisk is running and we can talk to it as the 'asterisk' user...Error!
Error communicating with Asterisk. Ensure that Asterisk is properly installed and running as the asterisk user
Asterisk appears to be running as asterisk
Try starting Asterisk with the './start_asterisk start' command in this directory
tried ./start_asterisk start ./install -n
help help please, what's the problem, 3rd day I'm trying to solve the problem.
● asterisk.service - Asterisk PBX
Loaded: loaded (/lib/systemd/system/asterisk.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Sat 2020-07-25 01:12:16 UTC; 32min ago
Docs: man:asterisk(8)
Process: 84496 ExecStart=/usr/sbin/asterisk -g -f -p -U asterisk (code=dumped, signal=SEGV)
Main PID: 84496 (code=dumped, signal=SEGV)
Jul 25 01:12:16 webserver systemd[1]: asterisk.service: Scheduled restart job, restart counter is at 91.
Jul 25 01:12:16 webserver systemd[1]: Stopped Asterisk PBX.
Jul 25 01:12:16 webserver systemd[1]: asterisk.service: Start request repeated too quickly.
Jul 25 01:12:16 webserver systemd[1]: asterisk.service: Failed with result 'core-dump'.
Jul 25 01:12:16 webserver systemd[1]: Failed to start Asterisk PBX.```
Selinux enabled - can be issue here.
For check what is really gooing try start asterisk manually and see verbose log
asterisk -vvvgc

What is the command to invoke RStudio from Command Line in linux environment?

I am looking for the command to invoke RStudio from Linux. I VNC into the linux system and don't see any other option to launch RStudio from the gnome-terminal. I know R files can be directly read in the terminal, but I searched and couldn't find any command to invoke RStudio.
Are you referring to the RStudio desktop version, or the server instance? The former is started by just calling rstudio:
edd#rob:~$ type -p rstudio
/usr/bin/rstudio
edd#rob:~$
The latter is always running and started as system process:
edd#rob:~$ systemctl status rstudio-server
● rstudio-server.service - RStudio Server
Loaded: loaded (/etc/systemd/system/rstudio-server.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2018-07-15 20:12:01 CDT; 1 weeks 1 days ago
Main PID: 1665 (rserver)
Tasks: 3 (limit: 4915)
CGroup: /system.slice/rstudio-server.service
└─1665 /usr/lib/rstudio-server/bin/rserver
Jul 15 20:12:00 rob systemd[1]: Starting RStudio Server...
Jul 15 20:12:01 rob systemd[1]: Started RStudio Server.
edd#rob:~$
If you want to access the latter you do not need VNC. Just point your browser at port 8787 on that machine.
Have you tried to launch it from here ?
/usr/bin/rstudio
PS : You need R to be installed on the system too.
On Arch the command is
rstudio-bin

mount encrypted loopback in suse/systemd: unknown filesystem type 'crypt'

This might be specific to openSUSE / systemd.
I'm having trouble mounting an encrypted loopback file using the procedure described on the SDB:Encrypted filesystems knowledge base. I get this behaviour:
[mjl#tesla:~]
[11:12] $ sudo systemctl start /home/mjl/key
Job for home-mjl-key.mount failed. See "systemctl status home-mjl-key.mount" and "journalctl -xe" for details.
[mjl#tesla:~]
[11:12] 1 $ sudo systemctl status home-mjl-key.mount
● home-mjl-key.mount - /home/mjl/key
Loaded: loaded (/etc/fstab; bad; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2018-03-11 11:12:41 AEDT; 3s ago
Where: /home/mjl/key
What: /home/mjl/.tomb
Docs: man:fstab(5)
man:systemd-fstab-generator(8)
Process: 12949 ExecMount=/usr/bin/mount /home/mjl/.tomb /home/mjl/key -t crypt -o loop,user,acl,user_xattr (code=exited, status=32)
Mar 11 11:12:41 tesla systemd[1]: Mounting /home/mjl/key...
Mar 11 11:12:41 tesla mount[12949]: mount: unknown filesystem type 'crypt'
Mar 11 11:12:41 tesla systemd[1]: home-mjl-key.mount: Mount process exited, code=exited status=32
Mar 11 11:12:41 tesla systemd[1]: Failed to mount /home/mjl/key.
Mar 11 11:12:41 tesla systemd[1]: home-mjl-key.mount: Unit entered failed state.
[mjl#tesla:~]
[11:12] 3 $
The /home/mjl/.tomb loopback file was created using YaST Partitioner; I specified that I did not want it mounted at system start time, but that users should be allowed to mount it.
So it created the file, added an entry to /etc/cryptab and also this entry to /etc/fstab:
[mjl#tesla:~]
[11:12] 3 $ tail -n1 /etc/fstab
/home/mjl/.tomb /home/mjl/key crypt loop,user,noauto,acl,user_xattr,nofail 0 0
[mjl#tesla:~]
[11:15]$
There is the 'crypt' filesystem type.
My question is: how should I be mounting this as a user? Is systemd failing because of the filesystem type, or because I haven't told it the encryption key?
I've also tried mounting directly:
[mjl#tesla:~]
[11:16]$ sudo mount /home/mjl/key
mount: unknown filesystem type 'crypt'
[mjl#tesla:~]
The same error. So I guess I'm not mounting it correctly. Do I need to do something with cryptsetup

uWSGI, Nginx, Flask app service keeps failing

Going to my app produces a 502 gateway error. Found out that it was because my how_lit.service is failing. But I am having trouble finding out why.
Tried editing the application and the ini document. Cannot figure out whats wrong.
The Nginx and uWSGI services are up and running fine.
Service Status:
lit#digitalocean:~/howlit$ sudo service how_lit status
[sudo] password for lit:
● how_lit.service - uWSGI instance to serve how lit rest api
Loaded: loaded (/etc/systemd/system/how_lit.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2016-08-04 00:30:44 EDT; 5 days ago
Process: 14294 ExecStart=/home/lit/howlit/env/bin/uwsgi --ini /home/lit/howlit/howlit.ini (code=exited, status=1/FAILURE)
Main PID: 14294 (code=exited, status=1/FAILURE)
Aug 04 00:30:44 digitalocean systemd[1]: Started uWSGI instance to serve how lit rest api.
Aug 04 00:30:44 digitalocean uwsgi[14294]: [uWSGI] getting INI configuration from /home/lit/howlit/howlit.ini
Aug 04 00:30:44 digitalocean systemd[1]: how_lit.service: Main process exited, code=exited, status=1/FAILURE
Aug 04 00:30:44 digitalocean systemd[1]: how_lit.service: Unit entered failed state.
Aug 04 00:30:44 digitalocean systemd[1]: how_lit.service: Failed with result 'exit-code'.
Directory and Permissions:
lit#digitalocean:~/howlit$ ls -l .
total 16
drwx---r-x 6 lit www-data 4096 Jul 29 11:47 env
-rwx---r-x 1 lit www-data 202 Aug 3 23:29 howlit.ini
-rwx---r-x 1 lit www-data 1203 Aug 3 23:01 how_lit_restapi.py
-rwxr-xr-x 1 lit www-data 72 Aug 3 23:27 wsgi.py
/etc/systemd/system/how_lit.service:
lit#digitalocean:~/howlit$ cat /etc/systemd/system/how_lit.service
[Unit]
Description=uWSGI instance to serve how lit rest api
After=network.target
[Service]
User=lit
Group=www-data
WorkingDirectory=/home/lit/howlit/
Environment="PATH=/home/lit/howlit/env/bin"
ExecStart=/home/lit/howlit/env/bin/uwsgi --ini /home/lit/howlit/howlit.ini
[Install]
WantedBy=multi-user.target
howlit.ini file:
lit#digitalocean:~/howlit$ cat howlit.ini
[uwsgi]
module = wsgi:app
uid = lit
gid = www-data
master = true
processes = 5
socket = how_lit_restapi.sock
chmod-sock = 666
vacum = true
die-on-term = true
gto = /var/log/uwsgi/%n.log
Tried running it by hand:
lit#digitalocean:~/howlit$ /home/lit/howlit/env/bin/uwsgi --ini /home/lit/howlit/howlit.ini
[uWSGI] getting INI configuration from /home/lit/howlit/howlit.ini
*** Starting uWSGI 2.0.13.1 (64bit) on [Tue Aug 9 18:28:25 2016] ***
compiled with version: 5.4.0 20160609 on 29 July 2016 11:48:08
os: Linux-4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016
nodename: digitalocean
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/lit/howlit
detected binary path: /home/lit/howlit/env/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 1896
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
bind(): Permission denied [core/socket.c line 230]
permission error again?
SOLVED IT: By sending my socket into tmp, but still getting bad gateway error when I navigate to my site :(
Solved my own problem.
First I checked my services.
sudo service nginx status
sudo service uwsgi status
sudo service how_lit status
then I saw them all running and up but was still getting the bad gateway error. Well after checking the logs had no errors. I had to assume my configs.
Then I realized my mistake....I never restarted all of it, just certain parts at certain times. So I restarted every single one as such:
sudo service nginx restart
sudo service uwsgi restart
sudo service how_lit restart
now it works.
About the permission issue I tried it by putting the socket into the /tmp directory that way www-data group users can access it as well as root. I learned that you need to be able to create the socket and allow access to the system for it.
I moved it out of tmp btw later for production as I was told that was not best practice.

Maria DB not starting. Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details

I am trying to install maria db and getting the following issue.
[root#localhost ~]# service mysqld start
Redirecting to /bin/systemctl start mysqld.service
Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details.
I tried 'systemctl status mariadb.service' and 'journalctl -xn' and follows the details.
[root#localhost ~]# systemctl status mariadb.service
mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)
Active: failed (Result: exit-code) since Sun 2014-09-21 17:19:44 IST; 23s ago
Process: 2712 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 2711 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 2683 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 2711 (code=exited, status=0/SUCCESS)
Sep 21 17:19:42 localhost.localdomain mysqld_safe[2711]: 140921 17:19:42 mysqld_safe Logging to '/var/lib/mysql/localhost.localdomain.err'.
Sep 21 17:19:42 localhost.localdomain mysqld_safe[2711]: 140921 17:19:42 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Sep 21 17:19:43 localhost.localdomain mysqld_safe[2711]: 140921 17:19:43 mysqld_safe mysqld from pid file /var/lib/mysql/localhost.localdoma...d ended
Sep 21 17:19:44 localhost.localdomain systemd[1]: mariadb.service: control process exited, code=exited status=1
Sep 21 17:19:44 localhost.localdomain systemd[1]: Failed to start MariaDB database server.
Sep 21 17:19:44 localhost.localdomain systemd[1]: Unit mariadb.service entered failed state.
[root#localhost ~]# journalctl -xn
-- Logs begin at Sun 2014-09-21 02:33:29 IST, end at Sun 2014-09-21 17:20:11 IST. --
Sep 21 17:16:26 localhost.localdomain systemd[1]: Started dnf makecache.
-- Subject: Unit dnf-makecache.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit dnf-makecache.service has finished starting up.
--
-- The start-up result is done.
Sep 21 17:18:11 localhost.localdomain NetworkManager[683]: <warn> nl_recvmsgs() error: (-33) Dump inconsistency detected, interrupted
Sep 21 17:19:42 localhost.localdomain systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has begun starting up.
Sep 21 17:19:42 localhost.localdomain mysqld_safe[2711]: 140921 17:19:42 mysqld_safe Logging to '/var/lib/mysql/localhost.localdomain.err'.
Sep 21 17:19:42 localhost.localdomain mysqld_safe[2711]: 140921 17:19:42 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Sep 21 17:19:43 localhost.localdomain mysqld_safe[2711]: 140921 17:19:43 mysqld_safe mysqld from pid file /var/lib/mysql/localhost.localdomain.pid end
Sep 21 17:19:44 localhost.localdomain systemd[1]: mariadb.service: control process exited, code=exited status=1
Sep 21 17:19:44 localhost.localdomain systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mariadb.service has failed.
--
-- The result is failed.
Sep 21 17:19:44 localhost.localdomain systemd[1]: Unit mariadb.service entered failed state.
Sep 21 17:20:11 localhost.localdomain NetworkManager[683]: <warn> nl_recvmsgs() error: (-33) Dump inconsistency detected, interrupted
Can any one please help?
I have tried uninstalling and installing many a times but received the same error.
Thanks in advance.
Most of the time if the system journal (journalctl) doesn't show what the problem was, the MariaDB error log (located in /var/lib/mysql/localhost.localdomain.err) does. Looking into that file you usually see what the problem is.
Most commonly errors that do not disappear after a reinstallation mean that your data directory (by default /var/lib/mysql/) is corrupted and the database needs to be reinstalled with mysql_install_db. To make sure you do a clean installation, remove all files located in the data directory and then run sudo mysql_install_db --user=mysql.
I solved as follows:
After installing
Run: > mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/
Then: > mysql_secure_installation
And then: systemctl start mariadb
with this this, I can resolved.
A quick update for anyone coming here through a Web search.
I had a "failed to start" message following a Debian 9 -> Debian 10 (Buster) in-place server upgrade, and after a bit of digging I found that the following line in /etc/mysql/my.cnf needed updating:
From:
[mysqld]
: (other stuff)
:
innodb_large_prefix
To:
[mysqld]
: (other stuff)
:
innodb_large_prefix = "ON"
The clue was the following lines in /var/log/mysql/error.log
2020-06-06 16:41:24 0 [ERROR] /usr/sbin/mysqld: option '--innodb-large-prefix' requires an argument
2020-06-06 16:41:24 0 [ERROR] Parsing options for plugin 'InnoDB' failed.
Not sure about your case, But u can check if mariadb/mysql client is deleted accidentally,
Since in my case, I had deleted the mariadb client repo from shared file,So reinstalled the client as,
sudo apt-get install libmariadb-dev
Note: But before installing client do one thing, for rails app just change the mysql version in gemfile and try to install as bundle install mysql2, If its mariadb client issue then it will throw error mentioning that need to install mariadb client or mysql2 client as,
sudo apt-get install libmariadb-dev
OR
sudo apt-get install libmysqlclient-dev
Please refer other answers in case its not mariadb client issue 😊 😜
This is the optimal solution for this problem:
netstat -tulpn | grep LISTEN
Now we need to look for mysqld service.
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 32029/mysqld
now kill the process in my case it is (32029).
kill 32029
Now use systemctl start mariadb.service

Resources