I want to copy my directory structure excluding the files. Is there any option in the tar to ignore all files and copy only the Directories recursively.
You can use find to get the directories and then tar them:
find .. -type d -print0 | xargs -0 tar cf dirstructure.tar --no-recursion
If you have more than about 10000 directories use the following to work around xargs limits:
find . -type d -print0 | tar cf dirstructure.tar --no-recursion --null --files-from -
Directory names that contain spaces or other special characters may require extra attention. For example:
$ mkdir -p "backup/My Documents/stuff"
$ find backup/ -type d | xargs tar cf directory-structure.tar --no-recursion
tar: backup/My: Cannot stat: No such file or directory
tar: Documents: Cannot stat: No such file or directory
tar: backup/My: Cannot stat: No such file or directory
tar: Documents/stuff: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
Here are some variations to handle these cases of "unusual" directory names:
$ find backup/ -type d -print0 | xargs -0 tar cf directory-structure.tar --no-recursion
Using -print0 with find will emit filenames as null-terminated strings; with -0 xargs will interpret arguments that same way. Using null as a terminator helps ensure that even filenames with spaces and newlines will be interpreted correctly.
It's also possible to pipe results straight from find to tar:
$ find backup/ -type d | tar cf directory-structure.tar -T - --no-recursion
Invoking tar with -T - (or --files-from -) will cause it to read filenames from stdin, expecting each filename to be separated by a line break.
For maximum effect this can be combined with options for null-terminated strings:
$ find . -type d -print0 | tar cf directory-structure.tar --null --files-from - --no-recursion
Of these I consider this last version to be the most robust, because it supports both unusual filenames and (unlike xargs) is not inherently limited by system command-line sizes. (see xargs --show-limits)
for i in `find . -type d`; do mkdir -p /tmp/tar_root/`echo $i|sed 's/\.\///'`; done
pushd /tmp/tar_root
tar cf tarfile.tar *
popd
# rm -fr /tmp/tar_root
go into the folder you want to start at (that's why we use find dot)
save tar file somewhere else. I think I got an error leaving it right there.
tar with r not c. I think with cf you keep creating new files and you only
get the last set of file subdirectories. tar r appends to the tar file.
--no-recursion because the find is giving you your whole list of files already
so you don't want to recurse.
find . -type d |xargs tar rf /somewhereelse/whatever-dirsonly.tar --no-recursion
tar tvf /somewhereelse/whatever-dirsonly.tar |more to check what you got.
For AIX:
tar cvfD some-tarball.tar `find /dir_to_start_from -type d -print`
Related
I have a directory structure as below
output/a/1/multipleFiles
output/a/2/multipleFiles
output/a/3/multipleFiles
output/b/1/multipleFiles
output/b/2/multipleFiles
output/b/3/multipleFiles
I want to know number of lines each directory has. So basically, number of lines at each inner most directory level instead of file level. The innermost directories 1, 2, 3 are different kinds of output we generate for our analytics which contains multiple hadoop part-xxxx files.
I moved to output directory and tried the below command.
find . -maxdepth 2 -type d -name '*' | awk -F "/" 'NF==3' | awk '{print $0"/*"}' | xargs wc -l
But I am getting an error as
wc: ./a/1/*: No such file or directory
wc: ./a/2/*: No such file or directory
wc: ./a/3/*: No such file or directory
but if I try
wc -l ./a/1/*
I am getting correct output for that specific folder.
What am I missing here.
EDIT:
I updated my command as below to remove unnecessary awk commands.
find . -mindepth 2 -maxdepth 2 -type d -name '*' | xargs wc -l
This again results in error as
wc: ./a/1: Is a directory
wc: ./a/2: Is a directory
wc: ./a/2: Is a directory
Give a try to execdir, for example:
find . -maxdepth 2 -type f -execdir wc -l {} \;
This will run the command wc -l {} only within the directory that the file has been found, from the man:
-execdir The -execdir primary is identical to the -exec primary with
the exception that utility will be executed from the
directory that holds the current file.
I am attempting to come up with a method to remotely find a list of files on our AIX UNIX machine that meet, what seems in windows, like simple criteria. It needs to be case insensitive (sigh), use wildcards (*) and possibly contain spaces in the path.
For my tests below I was using the ksh shell. However it will need to work in an ssh shell as well.
I am attempting to implement secure FTP in Visual Basic 6 (I know) using plink, command line and a batch file.
Basically find a file like the one below but with case insensitivity:
ls -1 -d -p "/test/rick/01012017fosterYYY - Copy.txt" | grep -v '.*/$'
Thanks for any help.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] do
es not exist.
ls: 0653-341 The file - does not exist.
ls: 0653-341 The file [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
success - as long as there are no spaces.
ls -1 -d -p "/test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[Tt][Xx][Tt]" | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy].[T
t][Xx][Tt] does not exist.
-- Assumption: We cannot use quotes with wildcard characters
ls -1 -d -p "/test/rick/01012017fosterYYY - Copy.txt" | grep -v '.*\/$'**
success. not case insensitive.
ls -1 -d -p /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr
][Yy][Yy][Yy][ does not exist.
ls: 0653-341 The file ][-][ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ ][-][ ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/[0][1][0][1][2][0][1][7][Ff][Oo][Ss][Tt][Ee][Rr
][Yy][Yy][Yy][ does not exist.
ls: 0653-341 The file ][-][ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]?-?[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
success. not very helpful though.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ ]-[ ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy][ d
oes not exist.
ls: 0653-341 The file ]-[ does not exist.
ls: 0653-341 The file ][Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]{ }-{ }[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]{ d
oes not exist.
ls: 0653-341 The file }-{ does not exist.
ls: 0653-341 The file }[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p /test/rick/*01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt]* | grep -v '.*\/$'**
fails with
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] d
oes not exist.
ls: 0653-341 The file - does not exist.
ls: 0653-341 The file [Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls -1 -d -p "/test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] - [Cc][Oo][Pp][Yy].[Tt][Xx][Tt]" | grep -v '.*\/$'**
fails with:
ls: 0653-341 The file /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy] -
[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] does not exist.
ls doesn't do pattern matching, any wildcard expansion (globbing) is done by the shell. The glob pattern language is different from regular expressions. Read the ksh documentation for information about globbing ("File Name Generation" in the manpage).
So when you do:
$ touch foo flo fum
$ ls -1 f[ol]o
flo
foo
... the shell notices the globbing characters [], reads the directory contents, replaces it with the matching files, and passes those as parameters to ls. You can show this by using echo instead:
$ echo f[ol]o
flo foo
ksh has globbing options available with the ~() construct, option i is "Treat the match as case insensitive" :
ksh$ touch foo FoO FOO
ksh$ echo ~(i)foo
foo FoO FOO
bash has a nocaseglob shopt option:
bash$ shopt -s nocaseglob
bash$ touch fOo
bash$ echo FO*
foo
Although note that some globbing character needs to be present to make the magic happen:
bash$ echo FOO
FOO
bash$ echo [F]OO
foo
(to keep this option change local, see https://unix.stackexchange.com/questions/310957/how-to-undo-a-set-x/310963)
It looks as if you're using grep -v '.*/$' to remove lines that are directories. The .* is superfluous here -- grep -v '/$' is equivalent.
But find is a better tool for this kind of searching and filtering, implementing -type f (match regular files) by actually looking at the file attributes, rather than by parsing a bit of ASCII in a listing.
$ touch foo FOO FoO
$ mkdir fOo
$ find . -maxdepth 1 -type f -iname "foo"
./FOO
./foo
./FoO
You could use find's -iname option to allow for case-insensitive searching, so for the example you've provided any of the following should find your file:
find /test/rick -maxdepth 1 -iname '01012017fosterYYY - copy.txt'
# or
find /test/rick -maxdepth 1 -iname '01012017fosteryyy - copy.txt'
# or
find /test/rick -maxdepth 1 -iname '01012017FOSTERyyy - cOpY.txt'
-maxdepth 1 : don't search in sub-directories
-iname : allow for case-insensitive searching
For case insensitive wildcard searches when -maxdepth and -iname flags are not available for AIX Find , you can pass the Find results to Grep:
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt"
find [InThisFolder] [ExcludeSubfolders] [FileTypes] | grep [InsensitiveWildcardName]
Though, this can still be problematic if you have a folder structure like "/test/rick/rick/".
The following code gives results with the current directory signifier ".":
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt"
But you can pass the results to sed and find "/./" and replace with "/".
find /test/rick/. \( ! -name . -prune \) -type f -print | grep -i ".*foster.*\.txt" | sed 's/\/\.\//\//g'
* UPDATE *
Based on this page: http://mywiki.wooledge.org/ParsingLs
I’ve come up with the following command (for loop on file expansion or globbing) which avoids the problematic "/test/rick/rick/" folder structure from the find | grep solution above. It searches a folder from any folder, handles spaces, and handles case insensitivity without having to specify escape characters or upper/lower matching ([Aa]).
Just modify the searchfolder and searchpattern:
searchfolder="/test/rick"; searchpattern="*foster*.txt"; for file in "$searchfolder"/*.*; do [[ -e "$file" ]] || continue; if [[ "$(basename "$file" | tr '[:upper:]' '[:lower:]')" = $searchpattern ]]; then echo "$file"; fi; done
It does this:
Set the folder path to search (searchfolder="/test/rick";)
Set the search pattern (searchpattern="*foster*.txt")
Loop for every file on the search folder (for file in "$searchfolder"/*.*;)
Make sure the file exists ( [[ -e "$file" ]] || continue;)
Transform any base file name uppercase characters to lowercase (basename "$file" | tr '[:upper:]' '[:lower:]')
Test if the lowered base file name matches the search pattern and if so
then print the full path and filename (if [[ $(basename "$file" | tr
'[:upper:]' '[:lower:]') = $searchpattern ]]; then echo "$file"; fi;)
Tested on AIX (Version 6.1.0.0) in ksh (Version M-11/16/88f) and ksh93 (Version M-12/28/93e).
What I finally used (because I don't have access to -maxdepth or -iname) was just to use case insensitive wildcards together with quotes around spaces.
ls -1 -d -p /test/rick/01012017[Ff][Oo][Ss][Tt][Ee][Rr][Yy][Yy][Yy]' '-' '[Cc][Oo][Pp][Yy].[Tt][Xx][Tt] | grep -v '.*\/$'
That way I don't have to install or upgrade anything and probably cause more problems just so I can get a simple list of files.
NOTE: AIX UNIX will still throw in some garbage errors if you have any sub directories under the path. I tapped out on this and just parsed these useless messages out on the client side.
Thanks everyone who responded.
I have a directory (dir) (with files and subdirectories):
ls -1 dir
plot.pdf
subdir.1
subdir.2
obj.RDS
And then ls -1 for either subdir.1 or subdir.2:
plot.pdf
PC.pdf
results.csv
de.pdf
de.csv
de.RDS
I would like to tar and gzip dir (in unix) and I'd like to exclude all RDS files (the the level right below dir and the ones in its subdirectories).
What's the easiest way to achieve that? Perhaps in a one liner
Something like:
find dir -type f -not -name '*.RDS' -print0 |
tar --null -T- -czf TARGET.tgz
should do it.
First, find finds the files, and then tar accepts the list via -T- (= --files-from /dev/stdin).
-print0 on find combined wth --null on tar protect from weird filenames.
-czf == Create gZipped File
You can add v to get verbose output.
To later inspect the contents, you can do:
tar tf TARGET.tgz
tar --exclude=*.RDS -Jcf outputball.tar dir_to_compress
this will ignore *.RDS across any dir or subdirs
decompress using
tar -xvf outputball.tar
I'd like to create a tar file of all the files in a directory minus sub-directory's in that directory and place that tar file in one of the sub-directory's. For example, I have several .txt files in /test and also another directory in /test called ArchivedFiles. I'd like to tell the tar command to archive all of the .txt files and place it in /test/ArchivedFiles.
How do I go about doing this?
tar cf test/foo/test.tar -- `find test -maxdepth 1 -name '*.txt' -type f`
I think that should do what you want.
An option which will not work due to the age of your tar command is:
find test -maxdepth 1 -type f -name '*.txt' -print0 | tar -cf test/foo/test.tar --null --files-from -
You are having problems, so you can try the following commands:
tar cf test/foo/test.tar `find test -maxdepth 1 -name '*.txt' -type f`
echo tar cf test/foo/test.tar `find test -maxdepth 1 -name '*.txt' -type f`
tar c f test/foo/test.tar `find test -maxdepth 1 -name '*.txt' -type f`
find test -maxdepth 1 -name '*.txt' -type f
And pastebin the output so that we can see what is happening.
Given that find is very legacy as well, let us try the following:
tar cf test/foo/test.tar test/*.txt
The following command will work. It will place any subdirectories into the tar but it doesn't put the contents of those subdirectories into the tar. This means that if you put the tar into a subdir you won't have to worry about it putting itself inside itself inside itself...
For example, if you want to put all of the files which are in the current directory into a tar file called mytar.tar which is in a subdir called d1:
tar --no-recursion -cvf d1/mytar.tar *
I am using AIX.
When I try to copy all the file in a folder to another folder with the following command:
cp ./00012524/*.PDF ./dummy01
The shell complains:
ksh: /usr/bin/cp: 0403-027 The parameter list is too long.
How to deal with it? My folder contain 8xxxx files, how can I copy them very fast? each file have size of 4x kb to 1xx kb.
Use find command in *nix:
find ./00012524 -type f -name "*.PDF" -exec cp {} ./dummy01/ \; -print
The cp command has a limitation of files which you can copy simultaneous.
One possibility you can copy them using multiple times the cp command bases on your file patterns, like:
cp ./00012524/A*.PDF ./dummy01
cp ./00012524/B*.PDF ./dummy01
cp ./00012524/C*.PDF ./dummy01
...
cp ./00012524/*.PDF ./dummy01
You can also copy trough find command:
find ./00012524 -name "*.PDF" -exec cp {} ./dummy01/ \;
$ ( cd 00012524; ls | grep '\.PDF$' | xargs -I{} cp {} ../dummy01/ )
The -t flag to cp is useful here:
find ./00012524 -name \*.PDF -print | xargs cp -t ./dummy01
The best command to copy large number of files from one directory to another.
find /path/to/source/ -name "*" -exec cp -ruf "{}" /path/to/destination/ \;
This helped me a lot.
You should be able use a for loop, e.g.
for f in $(ls ./00012524/*.pdf)
do
cp $f ./dummy01
done
I have no way of testing this, but it should work in theory.
you can do something like this and grab each line of the directory
# you can use the -rv to check the status of the command verbose
for i in /from_dir/*; do cp -rv "$i" /to_dir/; done