On page 145 of Operating System Concepts (Avi Silberschatz, 9th edition), it states that
Note that ordinary pipes require a parent–child relationship between the communicating processes on both UNIX and Windows systems. This
means that these pipes can be used only for communication between
processes on the same machine.
However, when I execute the command ps -el | grep 0 on a linux machine, both processes ps and grep have bash as their parent process. Other resources confirms that the shell pipe is an ordinary pipe that is unidirectional. If a parent-child relationship is required, how come the ps and grep process are siblings under bash? Just to confirm my understanding about pipes, is the statement in the textbook simply wrong or have I misunderstood something here?
The statement is true, but must be interpreted mixing with descriptor inheritance. What it means is that to setup a piping you need a parent-child relationship, not that communication with pipe is only possible between parent and children. So in your case, bash creates the pipe and transmit it to both children ps and grep (one writing in it while the other reads). Thus, both children communicates with the a pipe created by their parent.
Related
zcat big.txt.gz | split -l 1000000 - prefix
where big.txt.gz is 150 GB, say it has ~1 billion lines.
In this case, does the unix pipe "stream" the data into split, or is the zcat operation completed, and then split is performed afterwards?
It was not clear to me from other pages if the above command would crash because it couldn't hold all of the gunzipped data in the pipe buffer before executing split, or if the gunzipped data would be "streamed" into split.
In general, the streaming behavior of unix pipe is unclear - when does pipe wait until all previous operations are finished before feeding input into the next commands stdin?
For example, if I were to link several more commands, would it crash due to lack of memory? e.g.
zcat big.txt.gz | tr 'a' 'b' | sed 's/foo/bar/g' | grep 'hello'
A pipe has a limited capacity. [...] Applications should not rely on a particular capacity: an application should be designed so that a reading process consumes data as soon as it is available, so that a writing process does not remain blocked.
I'm not sure why there's any doubt here: the processes are running simultaneously, and the upstream process is writing while the downstream process is reading. Or at least that's the ideal specifically requested by this man page.
Now, it's possible that a given command may try to suck in all of its input before doing anything, and that too large an input may crash that command. But that's very different from the pipe buffer getting overfilled.
I'm using Zabbix 3.2. I want to calculate the traffic statistics on network interface based on the program name.
Like for getting total incoming traffic, we use net.if.in[if,] , by same way is it possible to retreive traffic utilized by each running process like in Nethogs. If so, pls share the Item key. Or, if there is any sh script to do the same.
Thanks in advance.
You haven't specified the operating system, but the question is tagged 'unix' and you mention nethogs and shell scripts - I'll assume Linux.
It might be a bit too much to monitor traffic for all of the processes - there could be hundreds of them, and even though many would not use the network, on a server system many would.
It is also important how you want to structure the data. For example, do you want to split it up per process name, or per individual process? Or maybe even process name and its parameters - in case of running several Java JVMs on the same box. You would have to decide on all this, as it will affect the data collection.
As sending data to Zabbix, the simplest way on the Zabbix side would be monitoring by process name only, and creating items in advance, if you know all the process names you will be interested in. If you do not know them, you will have to use Zabbix low level discovery to automatically create items as new processes appear.
And we finally get to the data collection part. Here, it indeed might be the easiest to use nethogs (keeping in mind that UDP is not supported). You can run nethogs in "trace" mode, which is pretty much the same as the "batch" mode for top. In this mode, output is simply printed to stdout.
nethogs -c 1 -d 60 -t
Here, the parameters mean:
-c - how many times to print output
-d - for how long to sleep between iterations, including the time before the first output
-t - tracing or batch mode
Nethogs also supports setting traffic output type with the -v flag. You'd have to decide how you want to visualise this:
0 - KB/s
1 - total KB
2 - total B
3 - total MB
With Zabbix, you probably will not want to use modes 1 or 3 - it is better to store data in bytes and allow Zabbix to add the multiplier as needed. In case of the KB/s mode (0), it is probably worth adding an item multiplier of 1024 to store data in bytes and again benefiting from the automatic unit application at Zabbix. Note that in any case you will want to run nethog instances back-to-back, to avoid windows where you are not collecting data. One way to minimise possibility of any windows would be running nethogs constantly (without supplying -c option) and redirecting output to a file. A script would then parse the file and send the data to Zabbix with zabbix_sender.
You wouldn't run this as a normal Zabbix user parameter, neither as an active, nor passive check - it would block for too long. Consider using atd (see this howto) or nohup to launch a script that sends data to Zabbix with zabbix_sender instead.
Note that you must run nethogs as root - use sudo for that.
I'm not aware of any existing scripts for this, but the following might get you started:
nethogs -c 1 -d 1 -t | awk 'BEGIN {FS="[[:space:]/]+"}; /Refreshing/,0 \
{if ($1 != "Refreshing:" && $1 != "unknown") {print $(NF-4), $(NF-1), $NF}}'
Here, awk grabs only program lines and prints out program name and sent/received traffic.
So I have a moderately complex set of requirements for my worker processes.
I want to use a the master slave topology, and a nondefault working directory.
I also want to mix both local and remote workers.
As far as I can tell from readying the --machine-file section of the documentation.
It will not let me do that.
So I am looking at the -L <file parameter
>julia -h
...
-L, --load Load immediately on all processors
...
So if I do not use the -p or --machine-file` flags, then there is initially only one processer so the all processors just mean on the only processor.
So I tried this out
start_workers.jl
addprocs([
("cluster_c4_1",:auto),
("cluster_c4_2",:auto)
],
dir="/mnt/",
topology=:master_slave
)
addprocs(
dir="/mnt/",
topology=:master_slave
)
test.jl
println("*************")
println(workers())
println("-------------")
Running it:
>julia -L start_workers.jl pl.jl
*************
[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21]
-------------
So it looks all good, got my 20 workers.
Have I done anything unreasonable? Is this the best way?
That's exactly how I'm deploying it on a HPC cluster under Torque scheduler. In fact I'm in the process of re-writing the the cluster manager to support more options when adding processes through the Torque scheduling systems in particular, so I've spent quite a bit of time looking into this.
You might also want to be aware there are various ClusterManagers, Pkg.add("ClusterManagers") that extend the ability of addprocs under a variety of environments, such as when you need to request the resources from a scheduler. It looks like passwordless ssh is possible for you, so the default cluster manager is sufficient in your case.
I don't believe there is any way of defining the extra topology and directory parameters on the command line, so your approach is correct.
Stdin and stdout are single files that are shared by multiple processes to take in input from the users. So how does the OS make sure that only the input give to a particular program is visible in the stdin for than program?
Your assumption that stdin/stdout (while having the same logical name) are shared among all processes is wrong at best.
stdin/stdout are logical names for open files that are forwarded (or initialized) by the process that has started a given process. Actually, with the standard fork-and-exec pattern the setup of those may occur already in the new process (after fork) before exec is being called.
stdin/stdout usually are just inherited from parent. So, yes there exist groups of processes that share stdin and/or stdout for a given filenode.
Also, as a filedescriptor may be a side of a pipe, you need not have file from a filesystem (or a device node) linked to any of the well known standard channels (you also should include stderr into your considerations).
The normal way of setup is:
the parent (e.g. your shell) is calling fork
the forked process (child) is setting up environment, standard I/O channels and anything else.
the child then executes exec to overlay the process with the target image to be executed.
When setting up: it either will keep the existing channels or replace them with new ones e.g. creating a pipe and linking the endpoints appropriately. (To be honest, creating the pipe need to happen before the fork in that simplified description)
This way, most of the process have their own I/O channels.
Nevertheless, multiple processes may write into a channel they are connected to (have a valid filedescriptor to). When reading each junk of data (usually lines with terminals or blocks with files) is being read by a single reader only. So if you have several (running) processes reading from a terminal as stdin, only one will read your typing, while the other(s) will not see this typing at all.
I have an MPI program that uses a master process and multiple worker processes. I want to have the master process running on a single compute node alone, while the worker processes run on another node. The worker processes should be assigned by socket (for example as it is done with the --map-by-socket option). Is there any option to assign the master process and the working processes to different nodes or to assign it manually, by consulting the rank maybe?
Thanks
Assignment of ranks to hosts simultaneously with binding is possible via the use of rankfiles. In your case, assuming that each node has two 4-core CPUs, something like this should do it (for Open MPI 1.7 and newer):
rank 0=host1 slots=0-7
rank 1=host2 slots=0:0-3
rank 2=host2 slots=1:0-3
For older versions, instead of slots=0:0-3 and slots=1:0-3 one should use slots=0-3 and slots=4-7 respectively (assuming that cores are numbered linearly which might not be the case). Then the rankfile is supplied to mpiexec via the --rankfile option. It supersedes the hostfile.
Another option would be to do an MIMD launch. In that case one could split the MPI job into several parts and provide different distribution and binding arguments for each part:
mpiexec -H host1 -n 1 --bind-to none ./program : \
-H host2 -n 2 --bind-to socket --map-by socket ./program
The easiest way I am aware of doing this is by using the --hostfile option of OpenMPI.
If you are using any decent batch system you should have a list of your hosts and slots in some simple file or environment variable and you can parse that into a hostfile.
If you run your application "by hand" you can generate such a list on your own.