First off, I have keypairs, this is not a passphrase question though ssh is involved.
I also have MPICH, Hydra, SLURM and lamd ... this is a cluster computing question.
Node0 will boot but node1 gets hung. I have had this problem for days now. My nfs mirror works just fine and I can run Game Of Life on 8 cores on node2 ... that is really cool too, just ask me about it...
BUT, when I want to run on all three nodes together I hit a password request from each node as node0 uses ssh to send the processes. Again, not a passphrase problem, HYDRA (slurm and lamd as well) wants my user password from node1. Basically my login credential. I can change that to an MPICHuser account; however the dilemma will remain.
Unless I create MPICHusers on all three nodes without passwords at all ... can that be done? It seems like the epitome of security risk.
So the question is, can I automate the password credential whenever # pops up in a way that won't hang lamboot?
It is late, looking at what I have makes me wonder if slurm is the new culprit.
Here is more or less what I am looking at:
me#wherever:/mirror/GameOfLife$ mpiexec.hydra -f /mirror/machinefile -n 10 ./life 10 10 30
[mpiexec#wherever] HYDU_process_mfile_token (utils/args/args.c:296): token node0 not supported at this time
[mpiexec#wherever] HYDU_parse_hostfile (utils/args/args.c:343): unable to process token
[mpiexec#wherever] mfile_fn (ui/mpich/utils.c:336): error parsing hostfile
[mpiexec#wherever] match_arg (utils/args/args.c:152): match handler returned error
[mpiexec#wherever] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec#wherever] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec#wherever] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec#wherever] main (ui/mpich/mpiexec.c:153): error parsing parameters
me#wherever:/mirror/GameOfLife$
That is not the problem.
I am looking toward Slurm comparability. Several things happen at nearly the same time in a specific order. The handler has to have terminal control in an instant so the master node can begin sending.
Before I added Slurm the hydra machinefile was working but node0 could not "grab" the keyboard.
Where should Slurm look for an equivalent file?
I am wondering if I should remove hydra.
Related
Is there anything out there to monitor SaltStack installations besides halite? I have it installed but it's not really what we are looking for.
It would be nice if we could have a web gui or even a daily email that showed the status of all the minions. I'm pretty handy with scripting but I don't know what to script.
Anybody have any ideas?
In case by monitoring you mean operating salt, you can try one of the following:
SaltStack Enterprise GUI
Foreman
SaltPad
Molten
Halite (DEPRECATED by SaltStack)
These GUI will allow you more than just knowing whether or not minions are alive. They will allow you to operate on them in the same manner you could with the salt client.
And in case by monitoring you mean just whether the salt master and salt minions are up and running, you can use a general-purpose monitoring solutions like:
Icinga
Naemon
Nagios
Shinken
Sensu
In fact, these tools can monitor different services on the hosts they know about. The host can be any machine that has an ip address and the service can be any resource that can be queried via the underlying OS. Example of host can be a server, router, printer... And example of service can be memory, disk, a process, ...
Not an absolute answer, but we're developing saltpad, which is a replacement and improvement of halite. One of its feature is to display the status of all your minions. You can give it a try: Saltpad Project page on Github
You might look into consul while it isn't specifically for SaltStack, I use it to monitor that salt-master and salt-minion are running on the hosts they should be.
Another simple test would be to run something like:
salt --output=json '*' test.ping
And compare between different runs. It's not amazing monitoring, but at least shows your minions are up and communicating with your master.
Another option might be to use the salt.runners.manage functions, which comes with a status function.
In order to print the status of all known salt minions you can run this on your salt master:
salt-run manage.status
salt-run manage.status tgt="webservers" expr_form="nodegroup"
I had to write my own. To my knowledge, there is nothing out there which will do this, and halite didn't work for what I needed.
If you know Python, it's fairly easy to write an application to monitor salt. For example, my app had a thread which refreshed the list of hosts from the salt keys from time to time, and a few threads that ran various commands against that list to verify they were up. The monitor threads updated a dictionary with a timestamp and success/fail for each host after they ran. It had a hacked together HTML display color coded to reflect the status of each node. Took me a about half a day to write it.
If you don't want to use Python, you could, painfully, do something similar to this inefficient, quick, untested hack using command line tools in bash.
minion_list=$(salt-key --out=txt|grep '^minions_pre:.*'|tr ',' ' ') # You'
for minion in ${minion_list}; do
salt "${minion}" test.ping
if [ $? -ne 0 ]; then
echo "${minion} is down."
fi
done
It would be easy enough to modify to write file or send an alert.
halite was depreciated in favour of paid ui version, sad, but true - still saltstack does the job. I'd just guess your best monitoring will be the one you can write yourself, happily there's a salt-api project (which I believe was part of halite, not sure about this), I'd recommend you to use this one with tornado as it's better than cherry version.
So if you want nice interface you might want to work with api once you set it up... when setting up tornado make sure you're ok with authentication (i had some trouble in here), here's how you can check it:
Using Postman/Curl/whatever:
check if api is alive:
- no post data (just see if api is alive)
- get request http://masterip:8000/
login (you'll need to take token returned from here to do most operations):
- post to http://masterip:8000/login
- (x-www-form-urlencoded data in postman), raw:
username:yourUsername
password:yourPassword
eauth:pam
im using pam so I have a user with yourUsername and yourPassword added on my master server (as a regular user, that's how pam's working)
get minions, http://masterip:8000/minions (you'll need to post token from login operation),
get all jobs, http://masterip:8000/jobs (you'll n need to post token from login operation),
So basically if you want to do anything with saltstack monitoring just play with that salt-api & get what you want, saltstack has output formatters so you could get all data even as a json (if your frontend is javascript like) - it lets you run cmd's or whatever you want and the monitoring is left to you (unless you switch from the community to pro versions) or unless you want to use mentioned saltpad (which, sorry guys, have been last updated a year ago according to repo).
btw. you might need to change that 8000 port to something else depending on version of saltstack/tornado/config.
Basically if you want to have an output where you can check the status of all the minions then you can run a command like
salt '*' test.ping
salt --output=json '*' test.ping #To get output in Json Format
salt manage.up # Returns all minions status
Or else if you want to visualize the same with a Dashboard then you can see some of the available options like Foreman, SaltPad etc.
I have a few work flows where I would like R to halt the Linux machine it's running on after completion of a script. I can think of two similar ways to do this:
run R as root and then call system("halt")
run R from a root shell script (could run the R script as any user) then have the shell script run halt after the R bit completes.
Are there other easy ways of doing this?
The use case here is for scripts running on AWS where I would like the instance to stop after script completion so that I don't get charged for machine time post job run. My instance I use for data analysis is an EBS backed instance so I don't want to terminate it, simply suspend. Issuing a halt command from inside the instance is the same effect as a stop/suspend from AWS console.
I'm impressed that works. (For anyone else surprised that an instance can stop itself, see notes 1 & 2.)
You can also try "sudo halt", as you wouldn't need to run as a root user, as long as the user account running R is capable of running sudo. This is pretty common on a lot of AMIs on EC2.
Be careful about what constitutes an assumption of R quitting - believe it or not, one can crash R. It may be better to have a separate script that watches the R pid and, once that PID is no longer active, terminates the instance. Doing this command inside of R means that if R crashes, it never reaches the call to halt. If you call it from within another script, that can be dangerous, too. If you know Linux well, what you're looking for is the PID from starting R, which you can pass to another script that checks ps, say every 1 second, and then terminates the instance once the PID is no longer running.
I think a better solution is to use the EC2 API tools (see: http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ for documentation) to terminate OR stop instances. There's a difference between the two of these, and it matters if your instance is EBS backed or S3 backed. You needn't run as root in order to terminate the instance - the fact that you have the private key and certificate shows Amazon that you're the BOSS, way above the hoi polloi who merely have root access on your instance.
Because these credentials can be used for mischief, be careful about running API tools from a given server, you'll need your certificate and private key on the server. That's a bad idea in the event that you have a security problem. It would be better to message to a master server and have it shut down the instance. If you have messaging set up in any way between instances, this can do all the work for you.
Note 1: Eric Hammond reports that the halt will only suspend an EBS instance, so you still have storage fees. If you happen to start a lot of such instances, this can clutter things up. Your original question seems unclear about whether you mean to terminate or stop an instance. He has other good advice on this page
Note 2: A short thread on the EC2 developers forum gives advice for Linux & Windows users.
Note 3: EBS instances are billed for partial hours, even when restarted. (See this thread from the developer forum.) Having an auto-suspend close to the hour mark can be useful, assuming the R process isn't working, in case one might re-task that instance (i.e. to save on not restarting). Other useful tools to consider: setTimeLimit and setSessionTimeLimit, and various checkpointing tools (I have a Q that mentions a couple). Using an auto-kill is useful if one has potentially badly behaved code.
Note 4: I recently learned of the shutdown command in package fun. This is multi-platform. See this blog post for commentary, and code is here. Dangerous stuff, but it could be useful if you want to adapt to Windows. I haven't tried it, though.
Update 1. Three more ideas:
You could use .Last() and runLast = TRUE for q() and quit(), which could shut down the instance.
If using littler or a script that invokes the script via Rscript, the same command line functions could be used.
My favorite package of today, tcltk2 has a neat timer mechanism, called tclTaskSchedule() that can be used to schedule the execution of an expression. You could then go crazy with the execution of stuff just before a hourly interval has elapsed.
system("echo 'rootpassword' | sudo halt")
However, the downside is having your root password in plain text in the script.
AFAIK those ways you mentioned are the only ones. In any case the script will have to run as root to be able to shut down the machine (if you find a way to do it without root that's possibly an exploit). You ask for an easier way but system("halt") is just an additional line at the end of your script.
sudo is an option -- it allows you to run certain commands without prompting for any password. Just put something like this in /etc/sudoers
<username> ALL=(ALL) PASSWD: ALL, NOPASSWD: /sbin/halt
(of course replacing with the name of user running R) and system('sudo halt') should just work.
We're having a strange issue here. Unfortunately I'm not at liberty to post the code here because it belongs to the vendor, but hopefully can provide enough info so that isn't necessary.
Much could be explained if someone here could definitively say whether or not it is possible for 'bob' to execute "some_script.sh", and have it exit out of his su'd session MID-WAY and have the remainder of the script continue as user 'pete' (who originally su'd to impersonate 'bob')
I'm fairly certain that this is not possible, but is there anyone out there that knows otherwise?
Things to keep in mind:
scriptA.sh executes multiple other processes (serially) which make changes to a database
lets assume that these are 01.sql through to 10.sql.
It does this by executing 'sqlplus' with appropriate arguments.
The 'sqlplus' binary is available to user 'bob', but NOT user 'pete'.
User pete is currently su'd into user 'bob', because pete doesn't have sqlplus on his PATH.
The output we saw in the script/database suggested that the '01.sql' was executed successfully, but not the others - they failed with a 'sqlplus command not found' error.
Cheers,
From the information you've given, "No. It is not possible."
If user 'pete' su's to 'bob', he creates a new shell as 'bob'. Any processes started from that new shell are run as 'bob'. Any script, if run as 'bob', cannot change it's user/UID to 'pete'. Any processes started as 'bob' cannot change their user/UID (absent setuid bit on the executable).
It's far more likely that part of the script is modifying the shell environment and changing its PATH or some other error within the script.
Using a script to attempt to switch UIDs is doomed to failure. A permissions system that requires shared logins is – by definition – insecure.
Use the AIX permissions system the way it was intended. If you need bob's credentials to perform a task then have a program that Sandipan can execute which will elevate to bob permissions for only that which needs bob's credentials.
This is the Principle of Least Privilege and is how to handle the job. Shared passwords are a bad hack that encourages worse hacks.
I have a bit of a problem with Xen. Each time I try to run xm create I get the following error:
dom0:~# xm create -c staros.xm
Using config file "./staros.xm". Started domain StarOS-3 xenconsole: Could not read tty from store: No such file or directory
Is this familiar to anyone?
I believe my config is in order. At first I suspected the path to qemu-dm wasn't set correctly.
The error you are describing could mean two things:
It is documenting a well known race in xenstore
The psuedo TTY needed to attach to a domain's console is stored in xenstore in several places. The Xen console client establishes an inotify style watch on that value, so that it can reconnect to the console if the backing file descriptor happens to change. However, takes a few seconds for that information to be populated in xenstore from the time that the domain is initially created.
If you post the output of xm info, it would be easy to see if you are dealing with a well known race.
The backing psuedo terminal can't be created
Common reasons for this would be /dev/pts not being mounted. If you run xenstore-ls /local/domain/{domain_id} after starting the domain without the -c option, you will see the contents of the store for that domain. Look for the line (near the bottom) that says
tty="/dev/pts/{pty}"
Verify that the pty does, in fact, exist.
The xen console daemon uses two actual file descriptors to make it happen. The first is a psuedo file descriptor (obtained via xs_fileno()) on that specific piece of information in the node, so it can poll() to see if that information changes. The second is a real FD returned from open() (yes, O_NONBLOCK is passed) which actually reads/writes to the psuedo tty.
It looks like it's not even finding the psuedo FD from xenstore, which means the backing pty is likely existentially challenged.
What I have: the login/password for a UNIX user (alice).
Who I am: some other UNIX user (bob).
What I need to do: start a process programmatically (foo) as the other user (alice).
What the end result should be: the process is running and displays alice as the owner if a "ps" is done. For purposes of privileges, acts as if alice started it.
Basically I need to write some code that does the equivalent of "su -c 'foo' - alice"
Ideally I don't want to have to set any special bits or permissions on the executable in question (foo).
I see only two possibilites to start a process as alice on UNIX/Linux from a process owned by bob that cannot setuid itself.
call a setuid program
communicate with an already running process that can start processes as alice
Never say never, but I think this is probably impossible in any unix-portable way. The setuid(2) call (and friends) succeeds only if the current uid is either the same as the target one (modulo some subtleties about effective and real uids) or if the current uid is 0 (ie, root). That is, you can't change from one non-root uid to another.
Having the password doesn't help. The password is used for the initial authentication to the system, whether it be via login, ssh, or some GUI login dialogue, but the password is the concern of those programs alone, and not of the system as such. Put another way, the kernel doesn't care about your password, and it's the kernel that you have to talk to if you want to change your uid.
That is, you're probably therefore obliged to consider indirect routes, such as the ones Peter G mentioned.
(Yes, some unixes may have a way of doing this, but that's platform-specific).
I know I'm not adding any positive advice here, only the possibly time-saving negative advice of 'nothing to see here; move right along...'