Today I stopped/started my GlassfishV3 instance and now I cannot access the addmin console located at http://servername:4848/. The screen says: "The admin console is loading..." This is going on forever now.
I have tried as follows:
I have tried adding the following entry to my domain.xml located at /glassfishv3/glassfish/domains/domain1/config as suggested in another Stack Overflow Q&A but after restarting the server still no luck.
<java-options>-Dcom.sun.enterprise.tools.admingui.NO_NETWORK=true</java-options>
I have also installed glassfishv3 on my local machine and cannot recreate the problem.I can go to http://localhost:4848 without any problem.
I have also looked at the server.log and jvm.log files located under the /glassfishv3/glassfish/domains/domain1/logs and nothing there that shed some light.
Any help would be very much appreciated
I had similar symptoms, and I tried some of what Dario had suggested as well, but it didn't work. It could be that I had a unique configuration for my dev env: I'm running Glassfish 3.1 on a VirtualBox Ubuntu 11.04 64-bit guest on a Windows 7 64-bit host. Quite by accident, I discovered an additional symptom: if I turned off the network on the Ubuntu guest, the console would load successfully on a localhost browser instance. That is, on the Ubuntu guest with the network off, I could successfully navigate to http://localhost:4848 and show the Glassfish admin console as expected. However, if the Ubuntu guest's network was on, I had the exact behavior suggested by the original poster: http://localhost:4848 would just sit forever on the inial loading page.
To make a long story short, I found that adding the following argument to the JVM options for server-config fixed the problem:
-Djava.net.preferIPv4Stack=true
When I made that change and restarted the Glassfish server, everything worked.
(Note that I also had in place some of the other settings recommended above, i.e., NO_NETWORK=true, and I'd adjusted the JVM memory footprint and set it to -server instead of -client. It could be that these settings are required as well, though they weren't sufficient on their own in my case.)
I was having this exact same problem. I could deploy in run mode, but it would hang forever in Debug mode. IntelliJ was hanging on the breakpoints. I muted the breakpoints, and glassfish3 worked good as new. I didn't have to change any domain.xml settings. Check your breakpoints!
I found a solution to my problem. Setting the java-option to NO_NETWORK to true did not work so I upgraded from 3.0.1 to 3.1 and it got fixed. Not immediately though, I had to stop/start the Glassfish server a couple of times before I got into the admin console without any really long delays.
Solution
The solution was to upgrade from the command line using the pkg utility.
You can find the steps in this link:
http://download.oracle.com/docs/cd/E18930_01/html/821-2437/gkthu.html#gktjf
Or do as follows:
Go to as-install-parent/bin
./pkg image-update
as-install-parent/glassfish/bin/asadmin start-domain --upgrade domain-name
as-install-parent/glassfish/bin/asadmin start-domain domain-name
UPDATE
I had peformance issues again and I found this other solution in Joshi's tech blog:
http://joshitech.blogspot.com/2009/09/glassfish-application-server.html
Basically add the following jvm options in the domain.xml. It should increase Glassfish boot up and deployment performance:
<jvm-options>-server</jvm-options>
<jvm-options>-Xms3000m</jvm-options>
<jvm-options>-Xmx3000m</jvm-options>
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
<jvm-options>-XX:+AggressiveHeap</jvm-options>
<jvm-options>-XX:+AggressiveOpts</jvm-options>
<jvm-options>-XX:+UseParallelGC</jvm-options>
<jvm-options>-XX:+UseParallelOldGC</jvm-options>
<jvm-options>-XX:ParallelGCThreads=5</jvm-options>
I don't know if you are referencing this answer, but there is a second step described (disabling update module).
Two more ideas:
Check if the NO_NETWORK=true option really works (there should be no ads in GF admin console)
Watch the server.log (glassfish-install-dir/glassfis/domains/domain1/logs) during startup and look for the last log entry before the delay occurs. This could be a hint for the source of the delay.
Beware of blindly following Dario's example unless you've lots more RAM than most do.
-Xms3000m gives 3gb to Glassfish. Do YOU have that much spare RAM?
I tried this on my 4gb Mac with 1gb for Glassfish. Made no discernable difference at all...performance still sux.
Related
I am having an issue with a relatively small openstack cluster deployed with kolla-ansible. The issue is that after a few days the controllers stop working. When I go into the docker container logs, I see in all of them that there are Too Many Open Files. I have tried changing limits.conf sysctl max files for processes and user. After all of that, the issue still shows up.
One interesting thing is that this was not happening until I had to reboot all of the controllers. I rebooted them because I needed to increase the amount of ram that they have after they died swapping. My first thought was that kolla-ansible is setting a configuration after running deploy, but I can't seem to find any point in the repo when kolla-ansible is changing ulimits or other.
Any theories what could cause this? Would it be related to increasing ram? Should I run reconfigure/deploy on each controller? I've tried looking in kolla-ansible's docs and forums and couldn't see where anyone else was having this issue.
Update this hasn't been fixed yet:
I submitted a bug report, https://bugs.launchpad.net/kolla-ansible/+bug/1901898
I don't know your used versions of Kolla-Ansible and your Linux, but your problem seems really related to this one:
On Ubuntu 16.04, please uninstall lxd and lxc packages. (An issue exists with cgroup mounts, mounts exponentially increasing when restarting container) (source: docs.openstack.org/kolla-ansible/4.0.0/quickstart.html)
I had this problem with the exponentially growing number of mount-pointers after the restart of my docker-containers too. My single-node test-deployment had become very slow based on this problem, but I can't remember at the moment, that I would had the same error with too many open files.
You can delete the packages with apt-get remove lxc-common lxcfs lxd lxd-client. I had done this fix together with a complete reinstallation of the kolla-ansible installation, so I don't know, if this also helps with an already existing installation. You should also use docker-ce instead of the docker from the apt-repos.
This was fixed with a workaround in bug https://bugs.launchpad.net/keystonemiddleware/+bug/1883659 problem was neutron server was keeping memcached connections open and not closing them until the memcached container reached too many files open. There is a work around mentioned in the bug link.
TCP connections work fine as I am able to converse with someone over zoom and teamviewer. However, whenever I attempt to access another webpage, I get a network error. Google seems to work fine for some reason but any webpage I go to listed by Google fails to connect. The only way I can open up my http connections is by ending a task called "init" inside of task manager. This shuts down my vscode as well as my ubuntu terminals I have running. If someone knows the solution please do tell. It's really annoying having to close out my vscode and terminals as well as my local servers to look up information and debug.
I found a fix to this issue.
So I was running a Windows Subsystem for Linux and my Windows Build was outdated as well as WSL. When I updated Windows and upgraded to WSL2, my issue was resolved and I don't seem to get any more network errors.
My goal is to follow "Deploying a Meteor app with Nginx from scratch" tutorial available here
After installing Meteor, Node, Forever and Git, do the npm install, I try to "run meteor" to see if it works.
After downloading meteor-tools, the process begins to extract meteor-tools... looks like it is hanging for a couple minutes and then stops without any warning.
So my guess is that something causes the extraction to quit, but i don't know what exactly.
Yes, Meteor likes plenty of RAM. I would recommend using Phusion Passenger with nginx for Meteor, it's very easy to set up, and their tutorials/getting started is very good:
https://www.phusionpassenger.com/library/install/nginx/install/oss/
I haven't found the exact reason. However, I got it working. I was using a DigitalOcean server (512Mb/20G). I tried with a big server (16G/160G) and it works.
So I guess my server Ram OR Disk capacity was too small.
Edit:
setting up smaller configuration, i noticed that the minimum for Meteor to work is: 2Gb of RAM and 40Gb Disk.
After a server restart I sometimes get random CompilationException errors like this:
CompilationException: CS0006: Metadata file `/tmp/apache-temp-aspnet-0/ca373c84/assembly/shadow/10cad1cb/02cb7ade_1201ab15_00000001/ZedGraph.dll' could not be found -> HttpException: Single file build failed
and you can't access the application. If I restart the server again then the problem disappears.
The file that triggers the error can be one of my own dll's or a dependency, like mysql.dll for instance.
I am using mono 2.10.2, CentOS release 5.7 (Final), Apache and mod_mono on a Linode VPS.
Any idea of what can be the cause?
Edit: Since my troubles started when I moved to Linode and they use XEN, I have found the option --with-xen_opt=yes that looks promissing:
http://mono-project.com/Advanced_Mono_Compile_Options
Sounds like a bug (race condition?) with the shadow copying and compilation. There have been others found and fixed before... see http://grendello.blogspot.com/2007/08/new-modmono-and-xsp-developments.html
And this is a report of a similar problem in 2006 http://lists.ximian.com/pipermail/mono-list/2006-March/030970.html
Rather than rebooting the system, you can try restarting apache. Note, I have found it can help to stop apache and then start, rather than a restart.
You can also enable the mod_mono control panel and restart the application from there. Add this in apache config.
<Location /monocp>
SetHandler mono-ctrl
Order deny,allow
Deny from all
Allow from 127.0.0.1 <YOURIP>
</Location>
To automate this process take a look at Monit.
FWIW, I had the exact same issue on a dedicated server with no virtualization (8 core i7, x86 Ubuntu Server Edition), so I do not believe this to be a Xen-induced issue. For either the OP (SCL) or anyone else with this issue, I have documented my work around in another SO post here, as well as having filed a bug report with Mono/Xamarin.
It does seem to be a race condition, but not related to virtualization.
It seems that the problems where caused by XEN virtualization and mono precompiled binaries. I compiled mono with the option --with-xen_opt=yes but I wasn't able to compile XSP so I couldn't test if that solved it for sure but I moved exactly the same application to a dedicated server and after 2 million requests I didn't experience any error.
Update: Check Mahmoud answer
Today I have deployed an app to our production application server GlassfishV3 through Jenkins CI to the autodeploy folder. The app server went down, and I cannot bring it back up.
My goal is to have the server up and running the same as prior to deploying the application. This is what I have done:
First find the PID of the process running at port 4848: nestat -nlept
Then kill the PID by doing kill -9 PID
Remove the war file that Jenkinks just put in the autodeploy directory just in case if that is the problem.
Start the server again by doing ./asadmin start-domain domain1
The server takes FOREVER to start !!! In fact it never starts successfully as I cannot access the admin console at 4848 or any of the other apps that were already running. However, it leaves a process running at 4848.
I looked at the jvm.log and server.log and I found a java.net.BindException:No free port within range.........
So my questions are as follows:
Do you know what is going on?
Do you know how to fix it?
Do you know of a way to speed up the ./asadmin start-domain domain1 process?
Note: In our QA app server (Same version, same OS, Same Java, Same Grails) it does not happen. Really frustrated with this issue.
Thanks a lot for your help. Any help would be very much appreciated as this is a production issue that has several applications down for a few hours already.
Dario
I can deploy my application now, basically it boiled down to increasing the MaxPermSize jvm option
Under the config folder, edit domain.xml and change the default size to this:
-XX:MaxPermSize=256m
You can always increase it as necessary.
Also, if that is not enough you can also change the max heap size in that same file
-Xmx512m . I have left it as is but if required you can change that to 6g or more on a 64 bit OS. On a 32 bit OS it will only recognize up to 3.5g.
Hope this helps somebody else in the future, as this issue kept me at work until 9:00PM
UPDATE:
I had peformance issues again and I found this other solution in Joshi's tech blog:
http://joshitech.blogspot.com/2009/09/glassfish-application-server.html
Basically add the following jvm options in the domain.xml. It should increase Glassfish boot up and deployment performance:
<jvm-options>-server</jvm-options>
<jvm-options>-Xms3000m</jvm-options>
<jvm-options>-Xmx3000m</jvm-options>
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
<jvm-options>-XX:+AggressiveHeap</jvm-options>
<jvm-options>-XX:+AggressiveOpts</jvm-options>
<jvm-options>-XX:+UseParallelGC</jvm-options>
<jvm-options>-XX:+UseParallelOldGC</jvm-options>
<jvm-options>-XX:ParallelGCThreads=5</jvm-options>