I have a bunch of file in one directory, what I wanted to do is:
cat a-12-08.json b-12-08_others.json b-12-08-mian.json >> new.json
But there are too many files, is there any command I can use to cat all files with "12-08" in their filename?
I found the solution below.
Here is the answer:
cat *12-08* >> new.json
you can use find to do what you want to archive:
find . -type f -name '*12-08*' -exec sh -c 'grep "one" {} && cat {} >> /tmp/output.txt' \;
In this way you can cat the files with contain the word that you looking for
Use a wildcard name:
cat *12-08* >>new.json
This will work as long as there aren't so many files that you exceed the maximum length of a command line, ARG_MAX (2MB on the Linux systems I checked).
Related
I need to concatenate a large number of text files from a series of directories. All the text files have the same name but some folders do not contain the file and just need to be skipped.
When I use cat ./**/File.txt > newFile.txt I get the following error /bin/cat: Argument list too long.
I tried using the ulimit command a few different ways but that did not work.
I have tried:
find . -name File.txt -exec cat {} \; > newFile.txt
find . -name File.txt -exec cat {} \+ > newFile.txt
find . -type f -name File.txt | xargs cat
and this results in the files being concatenated twice. For example, I have 3 text files named File.txt, each in a different directory, each with a different line of text:
test1
test2
test3
When I do the above commands my newFile.txt looks like:
test1
test2
test3
test1
test2
test3
I can't figure out why this is happening twice. When I use the command cat ./**/File.txt > newFile.txt on my small test set, it works fine and I end up with one file that has:
test1
test2
test3
I also tried
for a in File.txt ; do cat $a >> newFile.txt ; done
but get the message
cat: File.txt: No such file or directory
because some of the directories do not contain this text file, is my guess.
Is there another way to do this, or is there a reason my files are being concatenated twice?
Here's how I would do it
find . -name File.txt -exec cat {} >> output.txt \;
This searches for all occurrences of the file File.txt and appends the cat'ed output of that file to the file output.txt
However, I have tried your find command and it too also works.
find . -name File.txt -exec cat {} \; > newFile.txt
I would suggest that you clear down the output file newFile.txt before you try either your find or my find as follows:
>newFile.txt
This is a handy way to empty a file's contents. (Although this should not matter to you right now emptying a file by redirecting nothing to it can be done even if another process is writing to the file)
Hope this helps.
How to I list only directories that contain certain files. I am running on a Solaris box. Example, I want to list sub-directories of directory ABC that contain files that end with .out, .dat and .log .
Thanks
Something along these lines might work out for you:
find ABC/ \( -name "*.out" -o -name "*.log" \) -print | while read f
do
echo "${f%/*}"
done | sort -u
The sort -u bit could be just uniq instead, but either should work.
Should work on bash or ksh. Probably not so much on /bin/sh - you'd have to replace the variable expansion with something like echo "${f}" | sed -e 's;/[^/]*$;;' or something else that would strip off the last component of the path. dirname "${f}" would be good for that, but I don't recall if Solaris includes that utility...
I want to concatenate the files whose name does not include "_BASE_". I thought it would be somewhere along the lines of ...
ls | grep -v _BASE_ | cat > all.txt
the cat part is what I am not getting right. Can anybody give me some idea about this?
Try this
ls | grep -v _BASE_ | xargs cat > all.txt
You can ignore some files with ls using --ignore option and then cat them into a file.
ls --ignore="*_BASE_*" | xargs cat > all.txt
Also you can do that without xargs:
cat $( ls --ignore="*_BASE_*" ) > all.txt
UPD:
Dale Hagglund noticed, that filename like "Some File" will appear as two filenames, "Some" and "File". To avoid that you can use --quoting-style=WORD option, when WORD can be shell or escape.
For example, if --quoting-style=shell Some File will print as 'Some File' and will be interpreted as one file.
Another problem is output file could the same of one of lsed files. We need to ignore it too.
So answer is:
outputFile=a.txt; ls --ignore="*sh*" --ignore="${outputFile}" --quoting-style=shell | xargs cat > ${outputFile}
If you want to get also files from subdirectories, `find' is your friend:
find . -type f ! -name '*_BASE_*' ! -path ./all.txt -exec cat {} >> all.txt \+
It searches files in the current directory and its subdirectories, it finds only files (-type f), ignores files matching to wildcard pattern *_BASE_*, ignores all.txt, and executes cat in the same manner as xargs would.
Basically we need to change the end of line characters for a group of files.
Is there a way to accomplish this with a batch file? Is there a freeware utility?
dos2unix
It could be done with somewhat shorter command.
find ./ -type f | xargs -I {} dos2unix {}
You should be able to use tr in combination with xargs to do this.
On the Unix side at least, this should be the simplest way. However, I tried doing it that way once on a Windows box over a decade ago, but discovered that the Windows version of tr was translating my terminators right back to Windows format for me. :-( However, I think in the interveneing decade the tools have gotten smarter.
Combine find with dos2unix/fromdos to convert a directory of files (excluding binary files).
Just add this to your .bashrc:
DOS2UNIX=$(which fromdos || which dos2unix) \
|| echo "*** Please install fromdos or dos2unix"
function finddos2unix {
# Usage: finddos2unix Directory
find $1 -type f -exec file {} \; | grep " text" | cut -d ':' -f1 | xargs $DOS2UNIX
}
First, DOS2UNIX finds whether you have the utility installed, and picks one to use
Find makes a list of all files, then file appends the ": ASCII text" after each text file.
Finally, grep picks the text files, Cut removes all text after ':', and xargs makes this one big command line for DOS2UNIX.
How to rename all the files in one directory to new name using the command mv. Directory have 1000s of files and requirement is to change the last character of each file name to some specific char. Example: files are
abc.txt
asdf.txt
zxc.txt
...
ab_.txt
asd.txt
it should change to
ab_.txt
asd_.txt
zx_.txt
...
ab_.txt
as_.txt
You have to watch out for name collisions but this should work okay:
for i in *.txt ; do
j=$(echo "$i" | sed 's/..txt$/_.txt/')
echo mv \"$i\" \"$j\"
#mv "$i" "$j"
done
after you uncomment the mv (I left it commented so you could see what it does safely). The quotes are for handling files with spaces (evil, vile things in my opinion :-).
If all files end in ".txt", you can use mmv (Multiple Move) for that:
mmv "*[a-z].txt" "#1_.txt"
Plus: mmv will tell you when this generates a collision (in your example: abc.txt becomes ab_.txt which already exists) before any file is renamed.
Note that you must quote the file names, else the shell will expand the list before mmv sees it (but mmv will usually catch this mistake, too).
If your files all have a .txt suffix, I suggest the following script:
for i in *.txt
do
r=`basename $i .txt | sed 's/.$//'`
mv $i ${r}_.txt
done
Is it a definite requirement that you use the mv command?
The perl rename utility was written for this sort of thing. It's standard for debian-based linux distributions, but according to this page it can be added really easily to any other.
If it's already there (or if you install it) you can do:
rename -v 's/.\.txt$/_\.txt/' *.txt
The page included above has some basic info on regex and things if it's needed.
Find should be more efficient than for file in *.txt, which expands all of your 1000 files into a long list of command line parameters. Example (updated to use bash replacement approach):
find . \( -type d ! -name . -prune \) -o \( -name "*.txt" \) | while read file
do
mv $file ${file%%?.txt}_.txt
done
I'm not sure if this will work with thousands of files, but in bash:
for i in *.txt; do
j=`echo $i |sed 's/.\.txt/_.txt/'`
mv $i $j
done
You can use bash's ${parameter%%word} operator thusly:
for FILE in *.txt; do
mv $FILE ${FILE%%?.txt}_.txt
done