We have a process (written in c++ /managed), which receives network data via tcpip.
After running the process for a while while tracking network load, it seems that network get into freeze state and the process does not getting data, there are other processes in the system that using networking (same nic) which operates normally.
the process gets out of this frozen situation by itself after several minutes.
Any idea what is happening?
Any counter i can track to see if my process reach some limitations ?
It is going to be very difficult to answer specifically,
-- without knowing what exactly is your process/application about,
-- whether it is a network chat application, or a file server/client, or ......
-- without other details about your process how it is implemented, what libraries it uses, if relevant to problem.
Also you haven't mentioned what OS and environment you are running this process under,
there is very little anyone can help . It could be anything, a busy wait loopl in your code, locking problems if its a multi-threaded code,....
Nonetheless , here are some options to check:
If its linux try below commands to debug and monitor the behaviour of the process and see what could be problem-
top
Check top to see ow much resources(CPU, memory) your process is using and if there is anything abnormally high values in CPU usage for it.
pstack
This should stack frames of the process executing at time of the problem.
netstat
Run this with necessary options (tcp/udp) to check what is the stae of the network sockets opened by your process
gcore -s -c
This forces your process to core when the mentioned problem happens, and then analyze that core file using gdb
gdb
and then use command where at gdb prompt to get full back trace of the process (which functions it was executing last and previous function calls.
Related
I have a multithreaded process running on RHEL6.x (64bit).
I find that the process hangs and some threads (of the same process) crash most of the time when I try to bring up the process. Some threads wait for shared memory between the threads to get created (I can see that all of it does not get created).
But when I use strace , the process does not hang and it works just fine (all of the memory that is supposed to be created, gets created). Even interrupting strace after the memory gets created, keeps the process running fine for good.
I have read this:
strace fixes hung process
which did give me an idea. But I am still unclear on this as the version of RHEL that they have used is not mentioned.
Also, another point is that, changing the kernel to a fedora (compatible) kernel did not produce the issue.
So, I would just like to know how exactly does strace affect a process ? (or is it just the stack that moves back to the kernel as pointed out in the link) ?
I am using makeCluster function from R package snow from Linux machine to start a SOCK cluster on a remote Linux machine. All seems settled for the two machines to communicate succesfully (I am able to estabilish ssh connections between the two). But:
makeCluster("192.168.128.24",type="SOCK")
does not throw any result, just hangs indefinitely.
What am I doing wrong?
Thanks a lot
Unfortunately, there are a lot of things that can go wrong when creating a snow (or parallel) cluster object, and the most common failure mode is to hang indefinitely. The problem is that makeSOCKcluster launches the cluster workers one by one, and each worker (if successfully started) must make a socket connection back to the master before the master proceeds to launch the next worker. If any of the workers fail to connect back to the master, makeSOCKcluster will hang without any error message. The worker may issue an error message, but by default any error message is redirected to /dev/null.
In addition to ssh problems, makeSOCKcluster could hang because:
R not installed on a worker machine
snow not installed on a the worker machine
R or snow not installed in the same location as the local machine
current user doesn't exist on a worker machine
networking problem
firewall problem
and there are many more possibilities.
In other words, no one can diagnose this problem without further information, so you have to do some troubleshooting in order to get that information.
In my experience, the single most useful troubleshooting technique is manual mode which you enable by specifying manual=TRUE when creating the cluster object. It's also a good idea to set outfile="" so that error messages from the workers aren't redirected to /dev/null:
cl <- makeSOCKcluster("192.168.128.24", manual=TRUE, outfile="")
makeSOCKcluster will display an Rscript command to execute in a terminal on the specified machine, and then it will wait for you to execute that command. In other words, makeSOCKcluster will hang until you manually start the worker on host 192.168.128.24, in your case. Remember that this is a troubleshooting technique, not a solution to the problem, and the hope is to get more information about why the workers aren't starting by trying to start them manually.
Obviously, the use of manual mode bypasses any ssh issues (since you're not using ssh), so if you can create a SOCK cluster successfully in manual mode, then probably ssh is your problem. If the Rscript command isn't found, then either R isn't installed, or it's installed in a different location. But hopefully you'll get some error message that will lead you to the solution.
If makeSOCKcluster still just hangs after you've executed the specified Rscript command on the specified machine, then you probably have a networking or firewall issue.
For more troubleshooting advice, see my answer for making cluster in doParallel / snowfall hangs.
When I start a process in background in a terminal and some how if terminal gets closed then we can not interact that process any more. I am not sure but I think process also get killed. Can any one please tell me how can I detach that process from my terminal. So even if I close terminal then I can interact with same process in new terminal ?
I am new to unix so your extra information will help me.
The command you're looking for is disown.
disown <processid>
This is as close as you can get to a nohup. It detaches the process from the current login and allows it to continue running. Thanks David Korn!
http://www2.research.att.com/~gsf/man/man1/disown.html
and I just found reptyr which lets you reparent a disowned process.
https://github.com/nelhage/reptyr
It's already in the packages for ubuntu.
BUT if you haven't started the process yet and you're planning on doing this in the future then the way to go is screen and tmux. I prefer screen.
You might also consider the screen command. It has the "restore my session" functionality. Admittedly I have never used it, and forgot about it.
Starting the process as a daemon, or with nohup might not do everything you want, in terms of re-capturing stdout/stdin.
There's a bunch of examples on the web. On google try, "unix screen command" and "unix screen tutorial":
http://www.thegeekstuff.com/2010/07/screen-command-examples/
GNU Screen: an introduction and beginner's tutorial
First google result for "UNIX demonizing a process":
See the daemon(3) manpage for a short overview. The main thing of daemonizing
is going into the background without quiting or holding anything up. A list of
things a process can do to achieve this:
fork()
setsid()
close/redirect stdin/stdout/stderr to /dev/null, and/or ignore SIGHUP/SIGPIPE.
chdir() to /.
If started as a root process, you also want to do the things you need to be root
for first, and then drop privileges. That is, change effective user to the "daemon"
user or "nobody" with setuid()/setgid(). If you can't drop all privileges and need
root access sometimes, use seteuid() to temporary drop it when not needed.
If you're forking a daemon then also setup child handlers and, if calling exec,
set the close on exec flags on all file descriptors your children won't need.
And here's a HOWTO on creating Unix daemons: http://www.netzmafia.de/skripten/unix/linux-daemon-howto.html
'Interact with' can mean a couple of things.
The reason why a program, started at the command-line, exits when the terminal ends, is because the shell, when it exits, sends that process a HUP signal (see documentation for kill(1) for some introduction; HUP, by the way, is short for 'hang up', and originally indicated that the user had hung up the modem/telephone). The default response to a HUP signal is that a process is terminated – that is, the invoked program exits.
The details are slightly more fiddly, but this is the general intuition.
The nohup command tells the shell to start the program, and to do so in a way that this HUP signal is ignored. That is, the program keeps going after the invoking terminal exits.
You can still interact with this program by sending it signals (see kill(1) again), but this is a very limited sort of interaction, and depends on your program being written to do sensible things when it receives those signals (signals USR1 and USR2 are useful things to trap, if you're into that sort of thing). Alternatively, you can interact via named pipes, or semaphores, or other bits of inter-process communication (IPC). That gets fiddly pretty quickly.
I suspect what you're after, though, is being able to reattach a terminal to the process. That's a rather more complicated process, and applications like screen do suitably complicated things behind the scenes to make that happen.
The nohup thing is a sort of quick-and-dirty daemonisation. The daemon(3) function does the daemonisation 'properly', doing various bits of tidy-up as described in YePhIcK's answer, to comprehensively break the link with the process/terminal that invoked it. You can interact with that daemonised process with the same IPC tools as above, but not straightforwardly with a terminal.
We have an application which is deployed to a WebSphere server running on UNIX, and we are experiencing two issues:
a system hang which recovers after a few minutes - to investigate, we will need the thread dump (javacore).
a system hang which does not recover and requires WebSphere to be restarted - to investigate, we will need the thread dump and heap dump.
The problem is: when a system hang occurs, we do not know whether it is issue 1 or 2.
Ideally we would like to manually generate the thread dump first, and wait to see if the system recovers. If it does not, then we generate the thread dump and the heap dump, before restarting WebSphere.
I know about the kill -3 (or kill -QUIT) command. The command would generate thread dump only (if the parameter IBM_HEAPDUMP=false), or thread dump and heap dump (if IBM_HEAPDUMP=true). However, IBM_HEAPDUMP has to be set before WebSphere is started and cannot be changed while WebSphere is running.
Is my understanding correct, regarding the IBM_HEAPDUMP parameter and the kill -3 command?
Also, is it possible get the logs in the way I described? (i.e. when generating JVM diagnostics, choose whether to generate heap dump or not on the fly)
Your understanding is consistent with everything I've read.
However, I believe you can accomplish what you want by using wsadmin scripting. This article describes how to force javacores and heapdumps on a Windows platform where kill -3 is not available, but the same commands can be run on any WebSphere system.
From within wsadmin or a wsadmin script, execute:
set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
$AdminControl invoke $jvm generateHeapDump
$AdminControl invoke $jvm dumpThreads
I would like to know if there is a way that an MPI process send a kill signal to another MPI process?
Or differently, is there a way to exit from an MPI environment graciously, when one of the process is still active? (i.e. mpi_abort() prints an error message).
Thanks
No, this is not possible within an MPI application using the MPI library.
Individual processes would not be aware of the location of the other processes, nor of the process IDs of the other processes - and there is nothing in the MPI spec to make the kill you are wanting.
If you were to do this manually, then you'd need to MPI_Alltoall to exchange process IDs and hostnames across the system, and then you would need to spawn ssh/rsh to visit the required node when you wanted to kill something. All in all, it's not portable, not clean.
MPI_Abort is the right way to do what you are trying to achieve. From the Open MPI manual:
"This routine makes a "best attempt" to abort all tasks in the group of comm." (ie. MPI_Abort(MPI_COMM_WORLD, -1) is what you need.
Any output during MPI_Abort would be machine specific - so you may, or may not, receive the error message you mention.