I recently found out this solution to less through compressed gz files parellelly based on the cores available.
find . -name "*.gz" | xargs -n 1 -P 3 zgrep -H '{pattern to search}'
P.S. 3 is the number of cores
I was wondering if there was a way to do it for bz2 files as well.
Currently I am using this command:
find -type f -name '*.bz2' -execdir bzgrep "{text to find}" {} /dev/null \;
Change *.gz to *.bz2; change zgrep to bzgrep, and there you are.
For a bit of extra safety around unusual filenames, use -print0 on the find end and -0 on the xargs:
find . -name "*.bz2" -print0 | xargs -0 -n 1 -P 3 bzgrep -H '{pattern to search}'
Related
I am currently running the below jpegoptim command via cron to have jpegoptim look for jpg files to compress.
find /home/public_html/public/uploads -name '*.jpg' -type f -print0 | xargs -0 jpegoptim -o -p --size=300k --strip-all
This works, but is there a way to include .jpeg, .JPG, and .JPEG without having to run the same command multiple times? Also, is it okay to run every hour, or would this keep compressing the same files eventually reducing the quality?
You can match all of *.jpg, *.JPG, *.jpeg, *.JPEG by using two case-insensitive name matches.
find /home/public_html/public/uploads \
'(' -iname '*.jpg' -o -iname '*.jpeg' ')' -type f -print0 \
| xargs -0 jpegoptim -o -p --size=300k --strip-all
You can keep a separate timestamp to avoid reprocessing the same files, for example
find /home/public_html/public/uploads \
-newer .STAMP '(' -iname '*.jpg' -o -iname '*.jpeg' ')' -type f -print0 \
| xargs -0 jpegoptim -o -p --size=300k --strip-all
touch .STAMP
I have directory: D:/Temp, where there are a lot of subfolders with text files. Each folder has "file.txt". In some file.txt files is a word - "pattern". I would like check how many pattern words there are, and also get the filepath to that file.txt:
find D:/Temp -type f -name "file.txt" -exec basename {} cat {} \; | sed -n '/pattern/p' | wc -l
Output should be:
4
D:/Temp/abc1/file.txt
D:/Temp/abc2/file.txt
D:/Temp/abc3/file.txt
D:/Temp/abc4/file.txt
Or similar.
You could use GNU grep :
grep -lr --include file.txt "pattern" "D:/Temp/"
This will return the file paths.
grep -cr --include file.txt "pattern" "D:/Temp/"
This will return the count (counting the pattern occurences rather than the number of files)
Explanation of the flags :
-r makes grep recursively browse its target, that can then be a directory
--include <glob> makes grep restrict its recursive browsing to files matching the <glob>.
-l makes grep only return the files path. Additionnaly, it will stop parsing a file as soon as it has encountered the pattern.
-c makes grep only return the number of matches
If your file names don't contain spaces then all you need is:
awk '/pattern/{print FILENAME; cnt++; nextfile} END{print cnt+0}' $(find D:/Temp -type f -name "file.txt")
The above used GNU awk for nextfile.
I'd propose you to use two commands : one for find all the files:
find ./ -name "file.txt" -exec fgrep -l "-pattern" {} \;
Another for counting them:
find ./ -name "file.txt" -exec fgrep -l "-pattern" {} \; | wc -l
Previously I've used:
grep -Hc "pattern" $(find D:/temp -type f -name "file.txt")
This will only work if file.txt is found. Otherwise you could use the following which will account for when both files are found or not found:
searchFiles=$(find D:/temp -type f -name "file.txt"); [[ ! -z "$searchFiles" ]] && grep -Hc "pattern" $searchFiles
The output for this would look more like:
D:/Temp/abc1/file.txt 2
D:/Temp/abc2/file.txt 1
D:/Temp/abc3/file.txt 1
D:/Temp/abc4/file.txt 1
I would use
find D:/Temp -type f -name "file.txt" -exec dirname {} \; > tmpfile
wc -l tmpfile
cat tmpfile
rm tmpfile
Give a try to this safe and standard version:
find D:/Temp -type f -name file.txt -printf "%p\0" | xargs -0 bash -c 'printf "%s" "${#}"; grep -c "pattern" "${#}"' | grep ":[1-9][0-9]*$"
For each file.txt file found in D:/Temp directory and sub-directories, the xargs command prints the filename and the number of lines which contain pattern (grep -c).
A final grep ":[1-9][0-9]*$" selects only filenames with a count greater than 0.
The way I'm reading your question, I'm going to answer as if:
some but not all file.txt files contain pattern,
you want a list of the paths leading to file.txt with pattern, and
you want a count of pattern in each of those files.
There are a few options. (Always multiple ways to do anything.)
If your bash is version 4 or higher, you can use globstar to recurse through directories:
shopt -s globstar
for file in **/file.txt; do
if count=$(grep -c 'pattern' "$file"); then
printf "%d %s\n" "$count" "${file%/*}"
fi
done
This works because the if evaluation considers a failed grep (i.e. zero occurrences) to be FALSE, and thus does not print results.
Note that this may be high impact because it launches a separate grep on each file that is found. A lighter weight alternative might be to run a single grep on the fileglob, and parse the results:
shopt -s globstar
grep -c 'pattern' **/file.txt | grep -v ':0$'
This also depends on bash 4, and of course if you have millions of files you may overwhelm bash's command line maximum length. The output of this will be obvious, but you'll need to parse it with care if your filenames contain colons. I.e. cut -d: -f2 may not cut it.
One more option that leverages grep instead of bash might be:
grep -r --include 'file.txt' -c 'pattern' ./ | grep -v ':0$'
This uses GNU grep's --include option which modified the behaviour of -r (recursive). It should work in Linux, FreeBSD, NetBSD, OSX, but not with the default grep on OpenBSD or most SVR4 (Solaris, HP/UX, etc).
Note that I have tested none of these. No liability assumed. May contain nuts.
This should do it:
find . -name "file.txt" -type f -printf '%p\n' | awk '{print} END { print NR }'
I'm trying to purge all thumbnails created by Wordpress because of a CMS switchover that I'm planning.
find -name \*-*x*.* | xargs rm -f
But I dont know bash or regex well enough to figure out how to add a bit more specifity such as only the following will be removed
All generated files have the syntax of
<img-name>-<width:integer>x<height:integer>.<file-ext> syntax
You didn't quote or escape all your wildcards, so the shell will try to expand them before find executes.
Quoting it should work
find -name '*-*x*.*'| xargs echo rm -f
Remove the echo when you're satisfied it works. You could also check that two of the fields are numbers by switching to -regex, but not sure if you need/want that here.
regex soultion
find -regex '^.*/[A-Za-z]+-[0-9]+x[0-9]+\.[A-Za-z]+$' | xargs echo rm -f
Note: I'm assuming img-name and file-ext can only contain letters
You can try this:
find -type f | grep -P '\w+-\d+x\d+\.\w+$' | xargs rm
If you have spaces in the path:
find -type f | grep -P '\w+-\d+x\d+\.\w+$' | sed -re 's/(\s)/\\\1/g' | xargs rm
Example:
find -type f | grep -P '\w+-\d+x\d+\.\w+$' | sed -re 's/(\s)/\\\1/g' | xargs ls -l
-rw-rw-r-- 1 tiago tiago 0 Jun 22 15:14 ./image-800x600.png
-rw-rw-r-- 1 tiago tiago 0 Jun 22 15:17 ./test 2/test 3/image-800x600.png
The below GNU find command will remove all the files which contain this <img-name>-<width:integer>x<height:integer>.<file-ext> syntax string. And also i assumed that the corresponding files has . in their file-names.
find . -name "*.*" -type f -exec grep -l '<img-name>-<width:integer>x<height:integer>.<file-ext> syntax' {} \; | xargs rm -f
Explanation:
. Directory in which find operation is going to takeplace.(. represnts your current directory)
-name "*.*" File must have dot in their file-names.
-type f Only files.
-exec grep -l '<img-name>-<width:integer>x<height:integer>.<file-ext> syntax' {} print the file names which contain the above mentioned pattern.
xargs rm -f For each founded files, the filename was fed into xargs and it got removed.
I have a list of certain files that I see using the command below, but how can I copy those files listed into another folder, say ~/test?
find . -mtime 1 -exec du -hc {} +
Adding to Eric Jablow's answer, here is a possible solution (it worked for me - linux mint 14 /nadia)
find /path/to/search/ -type f -name "glob-to-find-files" | xargs cp -t /target/path/
You can refer to "How can I use xargs to copy files that have spaces and quotes in their names?" as well.
Actually, you can process the find command output in a copy command in two ways:
If the find command's output doesn't contain any space, i.e if the filename doesn't contain a space in it, then you can use:
Syntax:
find <Path> <Conditions> | xargs cp -t <copy file path>
Example:
find -mtime -1 -type f | xargs cp -t inner/
But our production data files might contain spaces, so most of time this command is effective:
Syntax:
find <path> <condition> -exec cp '{}' <copy path> \;
Example
find -mtime -1 -type f -exec cp '{}' inner/ \;
In the second example, the last part, the semi-colon is also considered as part of the find command, and should be escaped before pressing Enter. Otherwise you will get an error something like:
find: missing argument to `-exec'
find /PATH/TO/YOUR/FILES -name NAME.EXT -exec cp -rfp {} /DST_DIR \;
If you're using GNU find,
find . -mtime 1 -exec cp -t ~/test/ {} +
This works as well as piping the output into xargs while avoiding the pitfalls of doing so (it handles embedded spaces and newlines without having to use find ... -print0 | xargs -0 ...).
This is the best way for me:
cat filename.tsv |
while read FILENAME
do
sudo find /PATH_FROM/ -name "$FILENAME" -maxdepth 4 -exec cp '{}' /PATH_TO/ \; ;
done
If I issue the find command as follows:
find . -name *.ear
It prints out:
./dir1/dir2/earFile1.ear
./dir1/dir2/earFile2.ear
./dir1/dir3/earFile1.ear
I want to 'print' the name and the size to the command line:
./dir1/dir2/earFile1.ear 5000 KB
./dir1/dir2/earFile2.ear 5400 KB
./dir1/dir3/earFile1.ear 5400 KB
find . -name '*.ear' -exec ls -lh {} \;
just the h extra from jer.drab.org's reply. saves time converting to MB mentally ;)
You need to use -exec or -printf. Printf works like this:
find . -name *.ear -printf "%p %k KB\n"
-exec is more powerful and lets you execute arbitrary commands - so you could use a version of 'ls' or 'wc' to print out the filename along with other information. 'man find' will show you the available arguments to printf, which can do a lot more than just filesize.
[edit] -printf is not in the official POSIX standard, so check if it is supported on your version. However, most modern systems will use GNU find or a similarly extended version, so there is a good chance it will be implemented.
A simple solution is to use the -ls option in find:
find . -name \*.ear -ls
That gives you each entry in the normal "ls -l" format. Or, to get the specific output you seem to be looking for, this:
find . -name \*.ear -printf "%p\t%k KB\n"
Which will give you the filename followed by the size in KB.
Using GNU find, I think this is what you want. It finds all real files and not directories (-type f), and for each one prints the filename (%p), a tab (\t), the size in kilobytes (%k), the suffix " KB", and then a newline (\n).
find . -type f -printf '%p\t%k KB\n'
If the printf command doesn't format things the way you want, you can use exec, followed by the command you want to execute on each file. Use {} for the filename, and terminate the command with a semicolon (;). On most shells, all three of those characters should be escaped with a backslash.
Here's a simple solution that finds and prints them out using "ls -lh", which will show you the size in human-readable form (k for kilobytes and M for megabytes):
find . -type f -exec ls -lh \{\} \;
As yet another alternative, "wc -c" will print the number of characters (bytes) in the file:
find . -type f -exec wc -c \{\} \;
find . -name '*.ear' -exec du -h {} \;
This gives you the filesize only, instead of all the unnecessary stuff.
Awk can fix up the output to give just what the questioner asked for. On my Solaris 10 system, find -ls prints size in KB as the second field, so:
% find . -name '*.ear' -ls | awk '{print $2, $11}'
5400 ./dir1/dir2/earFile2.ear
5400 ./dir1/dir2/earFile3.ear
5400 ./dir1/dir2/earFile1.ear
Otherwise, use -exec ls -lh and pick out the size field from the output.
Again on Solaris 10:
% find . -name '*.ear' -exec ls -lh {} \; | awk '{print $5, $9}'
5.3M ./dir1/dir2/earFile2.ear
5.3M ./dir1/dir2/earFile3.ear
5.3M ./dir1/dir2/earFile1.ear
Try the following commands:
GNU stat:
find . -type f -name *.ear -exec stat -c "%n %s" {} ';'
BSD stat:
find . -type f -name *.ear -exec stat -f "%N %z" {} ';'
however stat isn't standard, so du or wc could be a better approach:
find . -type f -name *.ear -exec sh -c 'echo "{} $(wc -c < {})"' ';'
Just list the files (-type f) that match the pattern (-name '*.ear) using the disk-usage command (du -h) and sort the files by the human-readable file size (sort -h):
find . -type f -name '*.ear' -exec du -h {} \; | sort -h
Output
5.0k ./dir1/dir2/earFile1.ear
5.4k ./dir1/dir2/earFile2.ear
5.4k ./dir1/dir3/earFile1.ear
I struggled with this on Mac OS X where the find command doesn't support -printf.
A solution that I found, that admittedly relies on the 'group' for all files being 'staff' was...
ls -l -R | sed 's/\(.*\)staff *\([0-9]*\)..............\(.*\)/\2 \3/'
This splits the ls long output into three tokens
the stuff before the text 'staff'
the file size
the file name
And then outputs tokens 2 and 3, i.e. output is number of bytes and then filename
8071 sections.php
54681 services.php
37961 style.css
13260 thumb.php
70951 workshops.php
Why not use du -a ? E.g.
find . -name "*.ear" -exec du -a {} \;
Works on a Mac
This should get you what you're looking for, formatting included (i.e. file name first and size afterward):
find . -type f -iname "*.ear" -exec du -ah {} \; | awk '{print $2"\t", $1}'
sample output (where I used -iname "*.php" to get some result):
./plugins/bat/class.bat.inc.php 20K
./plugins/quotas/class.quotas.inc.php 8.0K
./plugins/dmraid/class.dmraid.inc.php 8.0K
./plugins/updatenotifier/class.updatenotifier.inc.php 4.0K
./index.php 4.0K
./config.php 12K
./includes/mb/class.hwsensors.inc.php 8.0K
You could try this:
find. -name *.ear -exec du {} \;
This will give you the size in bytes. But the du command also accepts the parameters -k for KB and -m for MB. It will give you an output like
5000 ./dir1/dir2/earFile1.ear
5400 ./dir1/dir2/earFile2.ear
5400 ./dir1/dir3/earFile1.ear
find . -name "*.ear" | xargs ls -sh
$ find . -name "test*" -exec du -sh {} \;
4.0K ./test1
0 ./test2
0 ./test3
0 ./test4
$
Scripter World reference
find . -name "*.ear" -exec ls -l {} \;
If you need to get total size, here is a script proposal
#!/bin/bash
totalSize=0
allSizes=`find . -type f -name *.ear -exec stat -c "%s" {} \;`
for fileSize in $allSizes; do
totalSize=`echo "$(($totalSize+$fileSize))"`
done
echo "Total size is $totalSize bytes"
You could try for loop:
for i in `find . -iname "*.ear"`; do ls -lh $i; done