I was looking up the meaning of flags like -a in zsh if statements, eg.
if [[ -a file.txt ]]; do
# do something
fi
and I found this
-a file
true if file exists.
-e file
true if file exists.
What is the difference between -a and -e? And if there is none, why do they both exist?
POSIX sheds some light on this.
tl;dr: Ksh traditionally used -a and several other shells followed suit. POSIX instead borrowed -e from Csh to avoid confusion. Now many shells support both.
The -e primary, possessing similar functionality to that provided by the C shell, was added because it provides the only way for a shell script to find out if a file exists without trying to open the file. Since implementations are allowed to add additional file types, a portable script cannot use:
test -b foo -o -c foo -o -d foo -o -f foo -o -p foo
to find out if foo is an existing file. On historical BSD systems, the existence of a file could be determined by:
test -f foo -o -d foo
but there was no easy way to determine that an existing file was a regular file. An early proposal used the KornShell -a primary (with the same meaning), but this was changed to -e because there were concerns about the high probability of humans confusing the -a primary with the -a binary operator.
Related
I want to remove all files in a directory except some in UNIX. Part I desired files have a known name and for the other part, I'm using ls|grep command. But the ls | grep is working when there is only one occurrence and not when there is more than one. it is the same with find|grep. here are my commands:
rm -v !("R1.r"|"R2.r"|"r2.par"|$(ls|grep nario)|"sh.sh")
rm -v !("R1.r"|"R2.r"|"r2.par"|$(find|grep nario)|"sh.sh")
Is there any problem with my commands???
It looks like you're trying to use BASH specific extglob syntax. ls|grep won't work in the middle of the glob, because each pattern needs to be separated by a |, which won't happen with ls|grep. The easier way to do what you want is to use the shell globbing to find the files under the directory you're looking for instead. Make sure you do the following:
Are using BASH
Have extglob enabled: shopt -s extglob
Have globstar enabled: shopt -s globstar
Use file globbing rather than ls | grep
Then try again:
rm -v !("R1.r"|"R2.r"|"r2.par"|**/*nario*|"sh.sh")
Note: globstar requires BASH version 4 or higher.
Is it possible to apply the same search and replace in multiple files in vim? I'll give an example below.
I have multiple .txt files — sad1.txt until sad5.txt. To open them, I'll use vim sad* and it opened already. Now inside the 5 txt files they have similar word like happy999; I would like to change it to happy111. I am currently using this code:
argdo %s/happy999/happy111/gc | wq!
Eventually only the sad1.txt is changed. What should I do to run one script in the 5 txt files?
Use:
:set aw
:argdo %s/happy999/happy111/g
The first line sets auto-write mode, so when you switch between files, vim will write the file if it has changed.
The second line does your global search and replace.
Note that it doesn't use wq! since that exits. If you don't want to use auto-write, then you could use:
:argdo %s/happy999/happy111/g | w
This avoids terminating vim at the end of editing the first file.
Also consider looking on vi and vim for answers to questions about vi and vim.
That is a task for sed -i (-i for "in place", works only with GNU sed). Yet, if you really want to use vim or you do need the /c to confirm the replace, you can do it in two ways:
With some help from the shell:
for i in sad*.txt; do
vim -c ':%s/happy999/happy111/gc' -c ':wq' "$i"
done
(the /c will still work, and vim will ask for each confirmation)
Or with pure VIM
vim -c ':%s/happy999/happy111/gc' -c ':w' -c ':n' \
-c ':%s/happy999/happy111/gc' -c ':w' -c ':n' \
-c ':%s/happy999/happy111/gc' -c ':w' -c ':n' \
-c ':%s/happy999/happy111/gc' -c ':w' -c ':n' \
-c ':%s/happy999/happy111/gc' -c ':wq' sad*.txt
(In my humble opinion this last one looks horrible and repetitive and has no real advantages over the shell for, but it shows that pure vim can do it)
No doubt, argdo is great, but to type that much boilerplate becomes quite annoying over the time.
Give a try to far.vim. It's such a tool that provide many IDEs.
If you don't need/want to be prompted for confirmation on each search and replace, use the following command, after opening your files with vim sad*:
:argdo %s/happy999/happy111/g | update
You can find more info by looking at the documentation for argdo in vim (:h argdo) or here:
http://vim.wikia.com/wiki/Search_and_replace_in_multiple_buffers
I'm writing a utility program in Common Lisp and building it with Clozure CL; I would like to be able to use the command-line option -d with the program, but for some reason this particular option won't make it through to (ccl::command-line-arguments). Here is a minimal example:
(defun main ()
(format t "~s~%" (ccl::command-line-arguments))
(quit))
I compiled with
(save-application "opts"
:toplevel-function 'main
:prepend-kernel t)
and here's some sample output:
~/dev/scratch$ ./opts -c -a -e
("./opts" "-c" "-a" "-e")
~/dev/scratch$ ./opts -c -d -e
("./opts" "-c" "-e")
~/dev/scratch$ ./opts -b --frogs -c -d -e -f -g -h --eye --jay -k -l
("./opts" "--frogs" "-c" "-e" "-f" "-g" "-h" "--eye" "--jay" "-k" "-l")
The -b and -d options appear to be getting lost. The documentation on command line arguments for ccl isn't very helpful. I thought maybe because ccl itself takes -b as an argument, that option might have gotten eaten for some reason, but it doesn't take -d (which is eaten), and it does take -e and -l which aren't. Nothing on saving applications seemed helpful.
I'm pretty sure it's Clozure-specific (and not, say, the shell eating them), because other stuff seems to be getting all the arguments:
#!/usr/bin/python
import sys
print sys.argv
yields
~/dev/scratch$ ./opts.py -a -b -c -d -e
['./opts.py', '-a', '-b', '-c', '-d', '-e']
and
#!/bin/bash
echo "$#"
gives
~/dev/scratch$ ./opts.sh -a -b -c -d -e
-a -b -c -d -e
This is all taking place on lubuntu 15.10 with bash as the shell.
If anyone could shed some light on why this is happening or how I can end up with all my command-line switches, I'd be appreciative.
Thanks.
According to the source code of the 1.11 release, -b and -d are options used by the lisp kernel.
Since I'm unsure about licence issues, I just provide the link to the relevant file: http://svn.clozure.com/publicsvn/openmcl/release/1.11/source/lisp-kernel/pmcl-kernel.c
Command line arguments are processed in the function process_options, where for options -b (--batch) and -d (--debug) - among others - a variable num_elide is set to 1. A bit further down, this leads to overwriting the option with the following argument (argv[k] = argv[j];).
The code also shows a possible fix: Supply -- (two dashes) once as argument before -b or -d. When above function encounters a -- it stops processing the rest of the arguments, thus leaving them unchanged to be possibly taken up into "lisp world" shortly after.
Turns out this has already been solved at SO before:
https://stackoverflow.com/a/5522169/1116364
I want to execute following command in shell script
cp /somedire/*.(txt|xml|xsd) /destination/dir/
But this does not run inside shell script. Any quick help?
createjob.sh: line 11: syntax error near unexpected token `('
My shell is zsh.
Thanks
Nayn
Your use of parentheses and alternation is a zsh-specific construct. It doesn't work in other shells, including zsh in sh compatibility mode.
If you want to keep using this construct, you'll have to invoke zsh as zsh (presumably by replacing #!/bin/sh by #!/bin/zsh or something like that).
If you need your script to run on ksh, use #!/bin/ksh or #!/usr/bin/env ksh and
cp /somedire/*.#(txt|xml|xsd) /destination/dir/
If you also need to support bash, that same command with the # will work provided you run the following commands first:
shopt -s extglob 2>/dev/null ## tell bash to parse ksh globbing extensions
setopt ksh_glob 2>/dev/null ## tell zsh to parse ksh globbing extensions
If you need POSIX sh compatibility, you'll have to use three separate commands, and prepare for an error message if any of the three extensions has no match. A more robust solution would use find:
find /somedire -name /somedire -o -type d -prune -o \
\( -name '*.txt' -o -name '*.xml' -o '*.xsd' \) \
-exec sh -c 'cp "$#" "$0"' /destination/dir {} +
No idea about zsh but Bash doesn’t know about regular expressions in paths, only wildcards.
You can try using find:
find -E . -regex '.*\.(txt|xml|xsd)' -exec cp {} /destination/dir \;
Have a look at the manpage for an explanation of the syntax of find.
This would work in bash, and probably zsh as well: cp /somedire/*.{txt,xml,xsd} /destination/dir/
It's not in POSIX, though, so it won't work with most /bin/sh's.
I have list of files which contain particular patterns, but those files have been tarred. Now I want to search for the pattern in the tar file, and to know which files contain the pattern without extracting the files.
Any idea...?
the tar command has a -O switch to extract your files to standard output. So you can pipe those output to grep/awk
tar xvf test.tar -O | awk '/pattern/{print}'
tar xvf test.tar -O | grep "pattern"
eg to return file name one pattern found
tar tf myarchive.tar | while read -r FILE
do
if tar xf test.tar $FILE -O | grep "pattern" ;then
echo "found pattern in : $FILE"
fi
done
The command zgrep should do exactly what you want, directly.
for example
zgrep "mypattern" *.gz
http://linux.about.com/library/cmd/blcmdl1_zgrep.htm
GNU tar has --to-command. With it you can have tar pipe each file from the archive into the given command. For the case where you just want the lines that match, that command can be a simple grep. To know the filenames you need to take advantage of tar setting certain variables in the command's environment; for example,
tar xaf thing.tar.xz --to-command="awk -e '/thing.to.match/ {print ENVIRON[\"TAR_FILENAME\"] \":\", \$0}'"
Because I find myself using this often, I have this:
#!/bin/sh
set -eu
if [ $# -lt 2 ]; then
echo "Usage: $(basename "$0") <pattern> <tarfile>"
exit 1
fi
if [ -t 1 ]; then
h="$(tput setf 4)"
m="$(tput setf 5)"
f="$(tput sgr0)"
else
h=""
m=""
f=""
fi
tar xaf "$2" --to-command="awk -e '/$1/{gsub(\"$1\", \"$m&$f\"); print \"$h\" ENVIRON[\"TAR_FILENAME\"] \"$f:\", \$0}'"
This can be done with tar --to-command and grep --label:
tar xaf archive.tar.gz --to-command 'egrep -Hn --label="$TAR_FILENAME" your_pattern_here || true'
--label gives grep the filename
-H tells grep to display the filename, and -n the line number
|| true because otherwise grep will exit with an error if the pattern is not found, and tar will complain about that.
xaf means to extract, and automagically decompress based off the file extension
--to-command has tar pass each file in the tarfile to a separate invocation of grep, and sets various environment variables with info about the file. See the manpage for more info.
Pretty heavily based off of Chipaca's answer (and Daniel H's comment), but this should be a bit easier to use and just uses tar and grep.
Python's tarfile module along with Tarfile.extractfile() will allow you to inspect the tarball's contents without extracting it to disk.
The easiest way is probably to use avfs. I've used this before for such tasks.
Basically, the syntax is:
avfsd ~/.avfs # Sets up a avfs virtual filesystem
rgrep pattern ~/.avfs/path/to/file.tar#/
/path/to/file.tar is the path to the actual tar file.
Pre-pending ~/.avfs/ (the mount point) and appending # lets avfs expose the tar file as a directory.
That's actually very easy with ugrep option -z:
-z, --decompress
Decompress files to search, when compressed. Archives (.cpio,
.pax, .tar, and .zip) and compressed archives (e.g. .taz, .tgz,
.tpz, .tbz, .tbz2, .tb2, .tz2, .tlz, and .txz) are searched and
matching pathnames of files in archives are output in braces. If
-g, -O, -M, or -t is specified, searches files within archives
whose name matches globs, matches file name extensions, matches
file signature magic bytes, or matches file types, respectively.
Supported compression formats: gzip (.gz), compress (.Z), zip,
bzip2 (requires suffix .bz, .bz2, .bzip2, .tbz, .tbz2, .tb2, .tz2),
lzma and xz (requires suffix .lzma, .tlz, .xz, .txz).
For example:
ugrep -z PATTERN archive.tgz
This greps each of the archived files to display PATTERN matches with the archived filenames. Archived filenames are shown in braces to distinguish them from ordinary filenames. Everything else is the same as grep (ugrep has the same options and produces the same output). For example:
$ ugrep -z "Hello" archive.tgz
{Hello.bat}:echo "Hello World!"
Binary file archive.tgz{Hello.class} matches
{Hello.java}:public class Hello // prints a Hello World! greeting
{Hello.java}: { System.out.println("Hello World!");
{Hello.pdf}:(Hello)
{Hello.sh}:echo "Hello World!"
{Hello.txt}:Hello
If you just want the file names, use option -l (--files-with-matches) and customize the filename output with option --format="%z%~" to get rid of the braces:
$ ugrep -z Hello -l --format="%z%~" archive.tgz
Hello.bat
Hello.class
Hello.java
Hello.pdf
Hello.sh
Hello.txt
Tarballs (.tar.gz/.tgz, .tar.bz2/.tbz, .tar.xz/.txz, .tar.lzma/.tlz) are searched as well as .zip archives.
You can mount the TAR archive with ratarmount and then simply search for the pattern in the mounted view:
pip install --user ratarmount
ratarmount large-archive.tar mountpoint
grep -r '<pattern>' mountpoint/
This should be much faster than iterating over each file and printing it to stdout, especially for compressed TARs.
Here is a simple comparison benchmark:
function checkFilesWithRatarmount()
{
local pattern=$1
local archive=$2
ratarmount "$archive" "$archive.mountpoint"
'grep' -r -l "$pattern" "$archive.mountpoint/"
}
function checkEachFileViaStdOut()
{
local pattern=$1
local archive=$2
tar --list --file "$archive" | while read -r file; do
if tar -x --file "$archive" -O -- "$file" | grep -q "$pattern"; then
echo "Found pattern in: $file"
fi
done
}
function createSampleTar()
{
for i in $( seq 40 ); do
head -c $(( 1024 * 1024 )) /dev/urandom | base64 > $i.dat
done
tar -czf "$1" [0-9]*.dat
}
createSampleTar myarchive.tar.gz
time checkEachFileViaStdOut ABCD myarchive.tar.gz
time checkFilesWithRatarmount ABCD myarchive.tar.gz
sleep 0.5s
fusermount -u myarchive.tar.gz.mountpoint
Results in seconds for a 55 MiB uncompressed and 42 MiB compressed TAR archive containing 40 files:
Compression
Ratarmount
Bash Loop over tar -O
none
0.31 +- 0.01
0.55 +- 0.02
gzip
1.1 +- 0.1
13.5 +- 0.1
bzip2
1.2 +- 0.1
97.8 +- 0.2
Of course, these results are highly dependent on the archive size and how many files the archive contains. These test examples are pretty small because I didn't want to wait too long but they already show the problem. The more files there are, the longer it takes for tar -O to jump to the correct file. And for compressed archives, it will be quadratically slower the larger the archive size is because everything before the requested file has to be decompressed and each file is requested separately. Both of these problems are solved by ratarmount.