Related
I had a question where I asked about dumping a tar'ed folder to an SSH target. This is useful when backing up servers that have no enough disk space. The solution I got and it works, is this:
tar czv <stuff to backup> | ssh user#server.com 'cat > /home/user/backupfolder/backup.tar.gz'
But the problem now is that I usually pipe tar to openssl to encrypt my packages with
tar zcpf / | openssl des3 -salt | dd of=backup.des3
My question: Now I wanna do the same, but with dumping the resulting archive directly into the SSH target, as given in that answered question but with OpenSSL encryption.
I tried the following:
tar zcpf / | openssl des3 -salt | ssh user#backupserver.com 'cat > /some/dir/backup.tar.gz'
This causes a race between the password asked by OpenSSL and the password asked by ssh, which results in a problem making OpenSSL refuse the input password when repeating it.
Note: Using a public key for logging in without a password isn't considered a useful solution for my case. (EDIT: Also passing the password to OpenSSL is not a solution).
Is there a way to do this without problems? I see the solution as to have OpenSSL ask me about the password, and then ssh ask me about the password, with no racing between them.
Thanks in advance. If you require any additional details, please ask.
You can pass the password with pass argument to encryption.
This should prevent seeking of encryption password later in time and then clash with the password prompt for SSH.
tar zcpf / | openssl des3 -salt -pass pass:1234 | ssh user#backupserver.com 'cat > /some/dir/backup.tar.gz'
EDIT:
This answer does not suit the OP's requirement. The edit made later mentions that passing password as argument is not acceptable.
So I finally managed to create my own bash script that will do the job and fix the problem. Remember that the script has to be run as root if you want to backup the root folder. You can make it fancier by adding root checker at the beginning of the script.
#!/bin/bash
dirToBackup=/
targetDir=/path/to/encrypted/backup/file.des3
sshuser=user
sshserver=myserver.com
read -s -p "Enter Encryption Password: " mypassword1
printf "\n"
read -s -p "Repeat Encryption Password: " mypassword2
printf "\n"
#make sure the passwords match
if [ "$mypassword1" != "$mypassword2" ]
then
echo "Passwords don't match! Exiting."
exit
else
#if they match, make sure they're not empty
if [ -z "$mypassword1" ]
then
echo "Password cannot be empty... Exiting."
else
tar zcp $dirToBackup | openssl des3 -salt -pass pass:$mypassword1 | ssh ${sshuser}#${sshserver} "dd of=${targetDir}"
fi
fi
I am trying to add my GPG public key as a part of our appliance installation process. The purpose of it to encrypt any important files like logs before admin pulling them into his local using admin portal and then decrypt them using private key.
The plan is to export public key into a file and make appliance installation process to import it using gpg --import command. But I realized, the key is needed to be trusted/signed before do any encryption.
How to make this key is trusted without any human intervention at the time of installation?
Btw, our appliance os is ubuntu vm and we use kickstart to automate.
Advance thanks for all help.
Your question is really "How do I encrypt to a key without gpg balking at the fact that the key is untrusted?"
One answer is you could sign the key.
gpg --edit-key YOUR_RECIPIENT
sign
yes
save
The other is you could tell gpg to go ahead and trust.
gpg --encrypt --recipient YOUR_RECIPIENT --trust-model always YOUR_FILE
Coincidentally I have a similar situation to the OP - I'm trying to use public/private keys to sign and encrypt firmware for different embedded devices. Since no answer yet shows how to add trust to a key you already have imported, here is my answer.
After creating and testing the keys on a test machine, I exported them as ascii:
$ gpg --export -a <hex_key_id> > public_key.asc
$ gpg --export-secret-keys -a <hex_key_id> > private_key.asc
Then secure-copied and imported them to the build server:
$ gpg --import public_key.asc
$ gpg --import private_key.asc
Important: add trust
Now edit the key to add ultimate trust:
$ gpg --edit-key <user#here.com>
At the gpg> prompt, type trust, then type 5 for ultimate trust, then y to confirm, then quit.
Now test it with a test file:
$ gpg --sign --encrypt --yes --batch --status-fd 1 --recipient "recipient" --output testfile.gpg testfile.txt
which reports
...
[GNUPG:] END_ENCRYPTION
without adding trust, I get various errors (not limited to the following):
gpg: There is no assurance this key belongs to the named user
gpg: testfile.bin: sign+encrypt failed: Unusable public key
There's an easier way to tell GPG to trust all of its keys by using the --trust-model option:
gpg -a --encrypt -r <recipient key name> --trust-model always
From the man page:
--trust-model pgp|classic|direct|always|auto
Set what trust model GnuPG should follow. The models are:
always Skip key validation and assume that used
keys are always fully trusted. You generally
won't use this unless you are using some
external validation scheme. This option also
suppresses the "[uncertain]" tag printed
with signature checks when there is no evidence
that the user ID is bound to the key. Note that
this trust model still does not allow the use
of expired, revoked, or disabled keys.
Add trusted-key 0x0123456789ABCDEF to your ~/.gnupg/gpg.conf replacing the keyid. This is equivalent to ultimately trusting this key which means that certifications done by it will be accepted as valid. Just marking this key as valid without trusting it is harder and either requires a signature or switching the trust-model to direct. If you are sure to only import valid keys you can simply mark all keys as valid by adding trust-model always. In the latter case ensure that you disable automatic key retrieval (not enabled by default).
This worked for me:
Trying to encrypt a file responds with this:
gpg -e --yes -r <uid> <filename>
It is NOT certain that the key belongs to the person named
in the user ID. If you *really* know what you are doing,
you may answer the next question with yes.
Use this key anyway? (y/N)
That causes my shell script to fail.
So I:
$gpg --edit-key <uid>
gpg> trust
Please decide how far you trust this user to correctly verify other
users' keys (by looking at passports, checking fingerprints from
different sources, etc.)
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu
Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
Please note that the shown key validity is not necessarily correct
unless you restart the program.
gpg> quit
Now the encrypt works properly.
Based on #tersmitten's article and a bit of trial and error, I ended up with the following command line to trust all keys in a given keyring without user interaction. I use it for keys used with both StackEschange Blackbox and hiera-eyaml-gpg:
# The "-E" makes this work with both GNU sed and OS X sed
gpg --list-keys --fingerprint --with-colons |
sed -E -n -e 's/^fpr:::::::::([0-9A-F]+):$/\1:6:/p' |
gpg --import-ownertrust
Personally, I prefer a solution which stores the results in the trustdb file itself rather than depends on user environment outside the shared Git repo.
Here's a trick I've figured out for automation of GnuPG key management, hint heredoc + --command-fd 0 is like magic. Below is an abridged version of one of the scripts that's been written to aid in automation with GnuPG.
#!/usr/bin/env bash
## First argument should be a file path or key id
Var_gnupg_import_key="${1}"
## Second argument should be an integer
Var_gnupg_import_key_trust="${2:-1}"
## Point to preferred default key server
Var_gnupg_key_server="${3:-hkp://keys.gnupg.net}"
Func_import_gnupg_key_edit_trust(){
_gnupg_import_key="${1:-${Var_gnupg_import_key}}"
gpg --no-tty --command-fd 0 --edit-key ${_gnupg_import_key} <<EOF
trust
${Var_gnupg_import_key_trust}
quit
EOF
}
Func_import_gnupg_key(){
_gnupg_import_key="${1:-${Var_gnupg_import_key}}"
if [ -f "${_gnupg_import_key}" ]; then
echo "# ${0##*/} reports: importing key file [${_gnupg_import_key}]"
gpg --no-tty --command-fd 0 --import ${_gnupg_import_key} <<EOF
trust
${Var_gnupg_import_key_trust}
quit
EOF
else
_grep_string='not found on keyserver'
gpg --dry-run --batch --search-keys ${_gnupg_import_key} --keyserver ${Var_gnupg_key_server} | grep -qE "${_grep_string}"
_exit_status=$?
if [ "${_exit_status}" != "0" ]; then
_key_fingerprint="$(gpg --no-tty --batch --dry-run --search-keys ${_gnupg_import_key} | awk '/key /{print $5}' | tail -n1)"
_key_fingerprint="${_key_fingerprint//,/}"
if [ "${#_key_fingerprint}" != "0" ]; then
echo "# ${0##*/} reports: importing key [${_key_fingerprint}] from keyserver [${Var_gnupg_key_server}]"
gpg --keyserver ${Var_gnupg_key_server} --recv-keys ${_key_fingerprint}
Func_import_gnupg_key_edit_trust "${_gnupg_import_key}"
else
echo "# ${0##*/} reports: error no public key [${_gnupg_import_key}] as file or on key server [${Var_gnupg_key_server}]"
fi
else
echo "# ${0##*/} reports: error no public key [${_gnupg_import_key}] as file or on key server [${Var_gnupg_key_server}]"
fi
fi
}
if [ "${#Var_gnupg_import_key}" != "0" ]; then
Func_import_gnupg_key "${Var_gnupg_import_key}"
else
echo "# ${0##*/} needs a key to import."
exit 1
fi
Run with script_name.sh 'path/to/key' '1' or script_name.sh 'key-id' '1' to import a key and assign a trust value of 1 or edit all values with script_name.sh 'path/to/key' '1' 'hkp://preferred.key.server'
Encryption should now be without complaint but even if it does the following --always-trust option should allow encryption even with complaint.
gpg --no-tty --batch --always-trust -e some_file -r some_recipient -o some_file.gpg
If you wish to see this in action, then check the Travis-CI build logs and how the helper script GnuPG_Gen_Key.sh is used for both generating and importing keys in the same operation... version two of this helper script will be much cleaner and modifiable but it's a good starting point.
This oneliner updates the trustdb with the ownertrust values from STDIN -- by extracting the fingerprint to the format required by --import-ownertrust flag.
This flag, as detailed on gpg man page, should be used In case of a severely damaged trustdb and/or if you have a recent backup of the ownertrust values, you may re-create the trustdb.
gpg --list-keys --fingerprint \
| grep ^pub -A 1 \
| tail -1 \
| tr -d ' ' \
| awk 'BEGIN { FS = "\n" } ; { print $1":6:" }' \
| gpg --import-ownertrust
One way to trust imported gpg keys:
gpg --import <user-id.keyfile>
fpr=`gpg --with-colons --fingerprint <user-id> |awk -F: '$1 == "fpr" {print$10; exit}'`
gpg --export-ownertrust && echo $fpr:6: |gpg --import-ownertrust
here, I assume that you import a key with the <user-id> from <user-id.keyfile>. The second line only extracts fingerprint, you can drop it if you know the fingerprint beforehand.
I think, I figured way to do this.
I used 'gpg --import-ownertrust' to export my trust db into a text file then removed all of my keys from it except public key I needed to push. And then imported my public key and edited owner-trust file on to server. This seems like working.
Now I am having trouble implementing these steps in Kickstart file:-(
With powershell, here is how to trust john.doe#foo.bar (adapted from #tersmitten blog post):
(gpg --fingerprint john.doe#foo.bar | out-string) -match 'fingerprint = (.+)'
$fingerprint = $Matches[1] -replace '\s'
"${fingerprint}:6:" | gpg --import-ownertrust
Note: using cinst gpg4win-vanilla
There is a way to autotrust key using --edit-key, but without getting into interactive shell (so can be automated in script). Below is a sample for windows:
(echo trust &echo 5 &echo y &echo quit) | gpg --command-fd 0 --edit-key your#email.com
Unix based:
echo -e "5\ny\n" | gpg --homedir . --command-fd 0 --expert --edit-key user#exaple.com trust;
For more info read this post. It details if you are creating more than one key.
I used following script for import key:
#!/bin/bash
function usage() {
cat >&2 <<EOF
Usage: $0 path_of_private_key
Example: gpg_import.sh ~/.ssh/my_gpg_private.key
Import gpg key with trust.
EOF
exit 1
}
[[ $# -lt 1 ]] && usage
KEY_PATH=$1
KEY_ID=$(gpg --list-packets ${KEY_PATH}/${GPG_PRIVATE_KEY} | awk '/keyid:/{ print $2 }' | head -1)
gpg --import ${KEY_PATH}/${GPG_PRIVATE_KEY}
(echo trust &echo 5 &echo y &echo quit) | gpg --command-fd 0 --edit-key $KEY_ID
I am using windows with gpgwin4.0.2 installed.
Open the Kleopatra (the GUI) -> Certificates -> Right Click -> Certify. Once the has been certify, this message will not show any.
Try this :
(echo trust &echo 5 &echo y &echo quit &echo save) | gpg --homedir 'gpgDirectory/' --batch --command-fd 0 --edit-key 'youKey'
--homedir : not required
Usually, stdout is line-buffered. In other words, as long as your printf argument ends with a newline, you can expect the line to be printed instantly. This does not appear to hold when using a pipe to redirect to tee.
I have a C++ program, a, that outputs strings, always \n-terminated, to stdout.
When it is run by itself (./a), everything prints correctly and at the right time, as expected. However, if I pipe it to tee (./a | tee output.txt), it doesn't print anything until it quits, which defeats the purpose of using tee.
I know that I could fix it by adding a fflush(stdout) after each printing operation in the C++ program. But is there a cleaner, easier way? Is there a command I can run, for example, that would force stdout to be line-buffered, even when using a pipe?
you can try stdbuf
$ stdbuf --output=L ./a | tee output.txt
(big) part of the man page:
-i, --input=MODE adjust standard input stream buffering
-o, --output=MODE adjust standard output stream buffering
-e, --error=MODE adjust standard error stream buffering
If MODE is 'L' the corresponding stream will be line buffered.
This option is invalid with standard input.
If MODE is '0' the corresponding stream will be unbuffered.
Otherwise MODE is a number which may be followed by one of the following:
KB 1000, K 1024, MB 1000*1000, M 1024*1024, and so on for G, T, P, E, Z, Y.
In this case the corresponding stream will be fully buffered with the buffer
size set to MODE bytes.
keep this in mind, though:
NOTE: If COMMAND adjusts the buffering of its standard streams ('tee' does
for e.g.) then that will override corresponding settings changed by 'stdbuf'.
Also some filters (like 'dd' and 'cat' etc.) dont use streams for I/O,
and are thus unaffected by 'stdbuf' settings.
you are not running stdbuf on tee, you're running it on a, so this shouldn't affect you, unless you set the buffering of a's streams in a's source.
Also, stdbuf is not POSIX, but part of GNU-coreutils.
Try unbuffer (man page) which is part of the expect package. You may already have it on your system.
In your case you would use it like this:
unbuffer ./a | tee output.txt
The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.
You can use setlinebuf from stdio.h.
setlinebuf(stdout);
This should change the buffering to "line buffered".
If you need more flexibility you can use setvbuf.
You may also try to execute your command in a pseudo-terminal using the script command (which should enforce line-buffered output to the pipe)!
script -q /dev/null ./a | tee output.txt # Mac OS X, FreeBSD
script -c "./a" /dev/null | tee output.txt # Linux
Be aware the script command does not propagate back the exit status of the wrapped command.
The unbuffer command from the expect package at the #Paused until further notice answer did not worked for me the way it was presented.
Instead of using:
./a | unbuffer -p tee output.txt
I had to use:
unbuffer -p ./a | tee output.txt
(-p is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments)
The expect package can be installed on:
MSYS2 with pacman -S expect
Mac OS with brew install expect
Update
I recently had buffering problems with python inside a shell script (when trying to append timestamp to its output). The fix was to pass -u flag to python this way:
run.sh with python -u script.py
unbuffer -p /bin/bash run.sh 2>&1 | tee /dev/tty | ts '[%Y-%m-%d %H:%M:%S]' >> somefile.txt
This command will put a timestamp on the output and send it to a file and stdout at the same time.
The ts program (timestamp) can be installed with the moreutils package.
Update 2
Recently, also had problems with grep buffering the output, when I used the argument grep --line-buffered on grep to it stop buffering the output.
If you use the C++ stream classes instead, every std::endl is an implicit flush. Using C-style printing, I think the method you suggested (fflush()) is the only way.
The best answer IMO is grep's --line-buffer option as stated here:
https://unix.stackexchange.com/a/53445/40003
This is my first question on stackoverflow!
I want to have a unix script that will run grep on the console output. Here is what my script does:
1. Telnet into a remote server (I have done this part successfully)
2. On successful login, the remote server displays outputs information on the console. I need to run grep on that console output (need help with this)
So, I need a script to run grep on the output appearing on the console.
Any thoughts??
Thanks,
Puneet
Use SSH instead. It's more secure and far easier to script.
ssh remoteusername#remotehost:/path/to/remote/script | grep 'something'
with appropriate key setup, it won't even prompt you for a password.
Have you tried I/O redirection? You could either do
$ your-command > output.txt
and then run grep on that file, or just directly pipe the output through grep like so
$ your-command | grep ...
See this article or google around for similar. There are probably thousands of good articles about this around the web.
Instead of telnet, I would suggest using netcat (nc). You could then pass your login credentials via standard input and grep the standard output (nc prints anything sent by the server on standard output).
nc <host> <port> <auth.txt | grep 'string'
What you want to do is probably using a pipe. You can probably see it in the above answers it's the | sign you see in the command. It may be difficult to locate on your keyboard, depending on the layout. (I have to admit it is not very often used).
Pipes will redirect the output of one command. Instead of sending it to the console, they will send it as an input of another command.
cmd1 | grep foo is equivalent to running grep foo on the output of cmd1 (you can replace cmd1 by your netstat command).
One last thing is that you can have as many pipes as you want. For instance on my machine I can run ls -ltr | tail -1 | awk '{print $9}' | grep foo to look for the word foo in the last modified file.
There are many things that all programmers should know, but I am particularly interested in the Unix/Linux commands that we should all know. For accomplishing tasks that we may come up against at some point such as refactoring, reporting, network updates etc.
The reason I am curious is because having previously worked as a software tester at a software company while I am studying my degree, I noticed that all of developers (who were developing Windows software) had 2 computers.
To their left was their Windows XP development machine, and to the right was a Linux box. I think it was Ubuntu. Anyway they told me that they used it because it provided powerful unix operations that Windows couldn't do in their development process.
This makes me curious to know, as a software engineer what do you believe are some of the most powerful scripts/commands/uses that you can perform on a Unix/Linux operating system that every programmer should know for solving real world tasks that may not necessarily relate to writing code?
We all know what sed, awk and grep do. I am interested in some actual Unix/Linux scripting pieces that have solved a difficult problem for you, so that other programmers may benefit. Please provide your story and source.
I am sure there are numerous examples like this that people keep in their 'Scripts' folder.
Update: People seem to be misinterpreting the question. I am not asking for the names of individual unix commands, rather UNIX code snippets that have solved a problem for you.
Best answers from the Community
Traverse a directory tree and print out paths to any files that match a regular expression:
find . -exec grep -l -e 'myregex' {} \; >> outfile.txt
Invoke the default editor(Nano/ViM)
(works on most Unix systems including Mac OS X)
Default editor is whatever your
"EDITOR" environment variable is
set to. ie: export
EDITOR=/usr/bin/pico which is
located at ~/.profile under Mac OS
X.
Ctrl+x Ctrl+e
List all running network connections (including which app they belong to)
lsof -i -nP
Clear the Terminal's search history (Another of my favourites)
history -c
I find commandlinefu.com to be an excellent resource for various shell scripting recipes.
Examples
Common
# Run the last command as root
sudo !!
# Rapidly invoke an editor to write a long, complex, or tricky command
ctrl-x ctrl-e
# Execute a command at a given time
echo "ls -l" | at midnight
Esoteric
# output your microphone to a remote computer's speaker
dd if=/dev/dsp | ssh -c arcfour -C username#host dd of=/dev/dsp
How to exit VI
:wq
Saves the file and ends the misery.
Alternative of ":wq" is ":x" to save and close the vi editor.
grep
awk
sed
perl
find
A lot of Unix power comes from its ability to manipulate text files and filter data. Of course, you can get all of these commands for Windows. They are just not native in the OS, like they are in Unix.
and the ability to chain commands together with pipes etc. This can create extremely powerful single lines of commands from simple functions.
Your shell is the most powerful tool you have available
being able to write simple loops etc
understanding file globbing (e.g. *.java etc.)
being able to put together commands via pipes, subshells. redirection etc.
Having that level of shell knowledge allows you to do enormous amounts on the command line, without having to record info via temporary text files, copy/paste etc., and to leverage off the huge number of utility programs that permit slicing/dicing of data.
Unix Power Tools will show you so much of this. Every time I open my copy I find something new.
I use this so much I am actually ashamed of myself. Remove spaces from all filenames and replace them with an underscore:
[removespaces.sh]
#!/bin/bash
find . -type f -name "* *" | while read file
do
mv "$file" "${file// /_}"
done
My personal favorite is the lsof command.
"lsof" can be used to list opened file descriptors, sockets, and pipes.
I find it extremely useful when trying to figure out which processes have used which ports/files on my machine.
Example: List all internet connections without hostname resolution and without port to port name conversion.
lsof -i -nP
http://www.manpagez.com/man/8/lsof/
If you make a typo in a long command, you can rerun the command with a substitution (in bash):
mkdir ~/aewseomeDirectory
you can see that "awesome" is mispelled, you can type the following to re run the command with the typo corrected
^aew^awe
it then outputs what it substituted (mkdir ~/aweseomeDirectory) and runs the command. (don't forget to undo the damage you did with the incorrect command!)
The tr command is the most under-appreciated command in Unix:
#Convert all input to upper case
ls | tr a-z A-Z
#take the output and put into a single line
ls | tr "\n" " "
#get rid of all numbers
ls -lt | tr -d 0-9
When solving problems on faulty linux boxes, by far the most common key sequence I type end up typing is alt+sysrq R E I S U B
The power of this tools (grep find, awk, sed) comes from their versatility, so giving a particular case seems quite useless.
man is the most powerful comand, because then you can understand what you type instead of just blindly copy pasting from stack overflow.
Example are welcome, but there are already topics for tis.
My most used :
grep something_to_find * -R
which can be replaced by ack and
find | xargs
find with results piped into xargs can be very powerful
some of you might disagree with me, but nevertheless, here's something to talk about. If one learns gawk ( other variants as well) throughly, one can skip learning and using grep/sed/wc/cut/paste and a few other *nix tools. all you need is one good tool to do the job of many combined.
Some way to search (multiple) badly formatted log files, in which the search string may be found on an "orphaned" next line. For example, to display both the 1st, and a concatenated 3rd and 4th line when searching for id = 110375:
[2008-11-08 07:07:01] [INFO] ...; id = 110375; ...
[2008-11-08 07:07:02] [INFO] ...; id = 238998; ...
[2008-11-08 07:07:03] [ERROR] ... caught exception
...; id = 110375; ...
[2008-11-08 07:07:05] [INFO] ...; id = 800612; ...
I guess there must be better solutions (yes, add them...!) than the following concatenation of the two lines using sed prior to actually running grep:
#!/bin/bash
if [ $# -ne 1 ]
then
echo "Usage: `basename $0` id"
echo "Searches all myproject's logs for the given id"
exit -1
fi
# When finding "caught exception" then append the next line into the pattern
# space bij using "N", and next replace the newline with a colon and a space
# to ensure a single line starting with a timestamp, to allow for sorting
# the output of multiple files:
ls -rt /var/www/rails/myproject/shared/log/production.* \
| xargs cat | sed '/caught exception$/N;s/\n/: /g' \
| grep "id = $1" | sort
...to yield:
[2008-11-08 07:07:01] [INFO] ...; id = 110375; ...
[2008-11-08 07:07:03] [ERROR] ... caught exception: ...; id = 110375; ...
Actually, a more generic solution would append all (possibly multiple) lines that do not start with some [timestamp] to its previous line. Anyone? Not necessarily using sed, of course.
for card in `seq 1 8` ;do
for ts in `seq 1 31` ; do
echo $card $ts >>/etc/tuni.cfg;
done
done
was better than writing the silly 248 lines of config by hand.
Neded to drop some leftover tables that all were prefixed with 'tmp'
for table in `echo show tables | mysql quotiadb |grep ^tmp` ; do
echo drop table $table
done
Review the output, rerun the loop and pipe it to mysql
Finding PIDs without the grep itself showing up
export CUPSPID=`ps -ef | grep cups | grep -v grep | awk '{print $2;}'`
Best answers from the Community
Traverse a directory tree and print out paths to any files that match a regular expression:
find . -exec grep -l -e 'myregex' {} \; >> outfile.txt
Invoke the default editor(Nano/ViM)
(works on most Unix systems including Mac OS X)
Default editor is whatever your
"EDITOR" environment variable is
set to. ie: export
EDITOR=/usr/bin/pico which is
located at ~/.profile under Mac OS
X.
Ctrl+x Ctrl+e
List all running network connections (including which app they belong to)
lsof -i -nP
Clear the Terminal's search history (Another of my favourites)
history -c
Repeat your previous command in bash using !!. I oftentimes run chown otheruser: -R /home/otheruser and forget to use sudo. If you forget sudo, using !! is a little easier than arrow-up and then home.
sudo !!
I'm also not a fan of automatically resolved hostnames and names for ports, so I keep an alias for iptables mapped to iptables -nL --line-numbers. I'm not even sure why the line numbers are hidden by default.
Finally, if you want to check if a process is listening on a port as it should, bound to the right address you can run
netstat -nlp
Then you can grep the process name or port number (-n gives you numeric).
I also love to have the aid of colors in the terminal. I like to add this to my bashrc to remind me whether I'm root without even having to read it. This actually helped me a lot, I never forget sudo anymore.
red='\033[1;31m'
green='\033[1;32m'
none='\033[0m'
if [ $(id -u) -eq 0 ];
then
PS1="[\[$red\]\u\[$none\]#\H \w]$ "
else
PS1="[\[$green\]\u\[$none\]#\H \w]$ "
fi
Those are all very simple commands, but I use them a lot. Most of them even deserved an alias on my machines.
Grep (try Windows Grep)
sed (try Sed for Windows)
In fact, there's a great set of ports of really useful *nix commands available at http://gnuwin32.sourceforge.net/. If you have a *nix background and now use windows, you should probably check them out.
You would be better of if you keep a cheatsheet with you... there is no single command that can be termed most useful. If a perticular command does your job its useful and powerful
Edit you want powerful shell scripts? shell scripts are programs. Get the basics right, build on individual commands and youll get what is called a powerful script. The one that serves your need is powerful otherwise its useless. It would have been better had you mentioned a problem and asked how to solve it.
Sort of an aside, but you can get powershell on windows. Its really powerful and can do a lot of the *nix type stuff. One cool difference is that you work with .net objects instead of text which can be useful if you're using the pipeline for filtering etc.
Alternatively, if you don't need the .NET integration, install Cygwin on the Windows box. (And add its directory to the Windows PATH.)
The fact you can use -name and -iname multiple times in a find command was an eye opener to me.
[findplaysong.sh]
#!/bin/bash
cd ~
echo Matched...
find /home/musicuser/Music/ -type f -iname "*$1*" -iname "*$2*" -exec echo {} \;
echo Sleeping 5 seconds
sleep 5
find /home/musicuser/Music/ -type f -iname "*$1*" -iname "*$2*" -exec mplayer {} \;
exit
When things work on one server but are broken on another the following lets you compare all the related libraries:
export MYLIST=`ldd amule | awk ' { print $3; }'`; for a in $MYLIST; do cksum $a; done
Compare this list with the one between the machines and you can isolate differences quickly.
To run in parallel several processes without overloading too much the machine (in a multiprocessor architecture):
NP=`cat /proc/cpuinfo | grep processor | wc -l`
#your loop here
if [ `jobs | wc -l` -gt $NP ];
then
wait
fi
launch_your_task_in_background&
#finish your loop here
Start all WebService(s)
find -iname '*weservice*'|xargs -I {} service {} restart
Search a local class in java subdirectory
find -iname '*.java'|xargs grep 'class Pool'
Find all items from file recursivly in subdirectories of current path:
cat searches.txt| xargs -I {} -d, -n 1 grep -r {}
P.S searches.txt: first,second,third, ... ,million
:() { :|: &} ;:
Fork Bomb without root access.
Try it on your own risk.
You can do anything with this...
gcc