I work in an X11 window on a MAC OS X machine. Now I have hundreds of files in one directory, each file name containing a substring such as "1970", "1971",..., "2014", etc. indicating that the file is for that year. Now I have just created subdirectories named "1970", "1971", ..., "2014".
What is the one-line UNIX command that would move all the files into the subdirectories corresponding to their years?
If the year-name sub-directories are in the single directory that currently contains all the files, then you should be able to use something like this, assuming that the current directory is that single directory:
shopt -s nullglob
for year in {1970..2014}
do
mv *?${year}* $year
mv *${year}?* $year
done
The globbing insists on at least one more character in the name to be moved than just the year, either before or after the year, to prevent an attempt to move 1970 into itself (which would fail). You need two mv commands to prevent a-1970-b from matching both glob expressions (which would cause the second to fail as the file would have already been removed). Using globbing like this preserves spaces etc in file names correctly. (Using command substitution, etc, does not.)
The shopt command means that if there are no files for a given glob, there'll be nothing in the output. That will generate a usage error from mv (a nuisance), but is otherwise harmless. You could decide to filter such error messages if you really want to; you probably don't want to send all error messages to /dev/null, though.
Since you're on a Mac, you don't have GNU mv with the very useful -t target option.
You said you need a single line command; replace each newline except the one after do with a semicolon; replace the newline after do with a space.
If you know that the year is never at the beginning or end of the file name, you can use a single mv *?${year}?* $year command.
Related
I have two folders,
/home/epi/folder1/
home/epi/folder2/
And
home/epi/destination/
I want to move *.txt files from the first two folders into destination, but there are sometimes 5 files, sometimes 100k.
I want to move a max of 40k files (combined) (no matter how many files are in there). These could be in either folder1 or folder2, so in the first step I generate a list.txt using find and head that contains 40k filenames.
I've tried a few things but am struggling to get both the specific files moved (in the list), due to the fact they could be in either folder. If I use --files-from, it doesn't seem to allow me to use multiple source directories. It takes one, then expects the destination location. If I leave that out, I can specify two source folders and move all the files, but then, I'm moving everything.
I've tried using `cat list.txt`, but 40k is too large a list of arguments to pass into the rsync. It works fine for smaller numbers, but I don't want the overhead of calling it multiple times for the same list cut into chunks.
a)
rsync -O -av --stats --files-from=list.txt --remove-source-files --log-file=test.log /home/epi/destination
b)
rsync -O -av `cat list.txt` --stats --remove-source-files --log-file=log.txt --progress --temp-dir=/temp /home/epi/destination
With a) I get an
`rsync error: syntax or usage error (code 1) at options.c(1652) [client=3.0.6]`
With b) due to the size of the arguments I get
`rsync: /usr/bin/rsync: cannot execute [Argument list too long]`
Is there a way in a unix shell (specifically Ubuntu) to change directory into the xth directory that was printed from the ls command?
I know you can sort a directory in multiple ways, but using the output from ls to get the xth directory?
An example shell:
$ ls
$ first_dir second_dir third_really_long_and_complex_dir
where I want to move into the third_really_long_and_complex_dir by passing 3 (or 2 in proper array format).
I know I could simply copy and paste, but if I'm already using the keyboard, it would be easier to type something like "cdls 2" or something like that if I knew the index.
The main problem with cd in an interactive session is that you generally want to change the current directory of the shell that is processing the command prompt. That means that launching a sub-shell (e.g. a script) would not help, since any cd calls would not affect the parent shell.
Depending on which shell you are using, however, you might be able to define a function to do this. For example in bash:
function cdls() {
# Save the current state of the nullglob option
SHOPT=`shopt -p nullglob`
# Make sure that */ expands to nothing when no directories are present
shopt -s nullglob
# Get a list of directories
DIRS=(*/)
# Restore the nullblob option state
$SHOPT
# cd using a zero-based index
cd "${DIRS[$1]}"
}
Note that in this example I absolutely refuse to parse the output of ls, for a number of reasons. Instead I let the shell itself retrieve a list of directories (or links to directories)...
That said, I suspect that using this function (or anything to this effect) is a very good way to set yourself up for an enormous mess - like using rm after changing to the wrong directory. File-name auto-completion is dangerous enough already, without forcing yourself to count...
In a Linux or Mac environment, Vim’s glob() function doesn’t match dot files such as .vimrc or .hiddenfile. Is there a way to get it to match all files including hidden ones?
The command I’m using:
let s:BackupFiles = glob("~/.vimbackup/*")
I’ve even tried setting the mysterious {flag} parameter to 1, and yet it still doesn’t return the hidden files.
Update: Thanks ib! Here’s the result of what I’ve been working on: delete-old-backups.vim.
That is due to how the glob() function works: A single-star pattern
does not match hidden files by design. In most shells, the default
globbing style can be changed to do so (e.g., via shopt -s dotglob
in Bash), but it is not possible in Vim, unfortunately.
However, one has several possibilities to solve the problem still.
First and most obvious is to glob hidden and not hidden files
separately and then concatenate the results:
:let backupfiles = glob(&backupdir..'/*').."\n"..glob(&backupdir..'/.[^.]*')
(Be careful not to fetch the . and .. entries along with hidden files.)
Another, perhaps more convenient but less portable way is to use
the backtick expansion within the glob() call:
:let backupfiles = glob('`find '..&backupdir..' -maxdepth 1 -type f`')
This forces Vim to execute the command inside backticks to obtain
the list of files. The find shell command lists all files (-type f)
including the hidden ones, in the specified directory (-maxdepth 1
forbids recursion).
Have two folders with approx. 150 java property files.
In a shell script, how to compare both folders to see if there is any new property file in either of them and what are the differences between the property files.
The output should be in a report format.
To get summary of new/missing files, and which files differ:
diff -arq folder1 folder2
a treats all files as text, r recursively searched subdirectories, q reports 'briefly', only when files differ
diff -r will do this, telling you both if any files have been added or deleted, and what's changed in the files that have been modified.
I used
diff -rqyl folder1 folder2 --exclude=node_modules
in my nodejs apps.
Could you use dircmp ?
Diff command in Unix is used to find the differences between files(all types). Since directory is also a type of file, the differences between two directories can easily be figure out by using diff commands. For more option use man diff on your unix box.
-b Ignores trailing blanks (spaces and tabs)
and treats other strings of blanks as
equivalent.
-i Ignores the case of letters. For example,
`A' will compare equal to `a'.
-t Expands <TAB> characters in output lines.
Normal or -c output adds character(s) to the
front of each line that may adversely affect
the indentation of the original source lines
and make the output lines difficult to
interpret. This option will preserve the
original source's indentation.
-w Ignores all blanks (<SPACE> and <TAB> char-
acters) and treats all other strings of
blanks as equivalent. For example,
`if ( a == b )' will compare equal to
`if(a==b)'.
and there are many more.
I did some havoc on my computer, when I played with the commands suggested by vezult [1]. I expected the one-liner to ask file-names to be removed. However, it immediately removed my files in a folder:
> find ./ -type f | while read x; do rm "$x"; done
I expected it to wait for my typing of stdin:s [2]. I cannot understand its action. How does the read command work, and where do you use it?
What happened there is that read reads from stdin. When you put it at the end of a pipe, it read from that pipe.
So your find becomes
file1
file2
and so on; read reads that and replaces x successively with file1 then file2, and so your loop becomes
rm "file1"
rm "file2"
and sure enough, that rm's every file starting at the current directory ".".
A couple hints.
You didn't need the "/".
It's better and safer to say
find . -type f
because should you happen to type ". /" (ie, dot SPACE slash) find will start at the current directory and then go look starting at the root directory. That trick, given the right privileges, would delete every file in the computer. "." is already the name of a directory; you don't need to add the slash.
The find or rm commands will do this
It sounds like what you wanted to do was go through all the files in all the directories starting at the current directory ".", and have it ASK if you want to delete it. You could do that with
find . -type f -exec rm -i {} \;
or
find . -type f -ok rm {} \;
and not need a loop at all. You can also do
rm -r -i *
and get nearly the same effect, except that it will try to delete directories too. If the directory is empty, that'll even work.
Another thought
Come to think of it, unless you have a LOT of files, you could also do
rm -i `find . -type f`
Now the find in backquotes will become a bunch of file names on the command line, and the '-i' interactive flag on rm will ask the yes or no question.
Charlie Martin gives you a good dissection and explanation of what went wrong with your specific example, but doesn't address the general question of:
When should you use the read command?
The answer to that is - when you want to read successive lines from some file (quite possibly the standard output of some previous sequence of commands in a pipeline), possibly splitting the lines into several separate variables. The splitting is done using the current value of '$IFS', which normally means on blanks and tabs (newlines don't count in this context; they separate lines). If there are multiple variables in the read command, then the first word goes into the first variable, the second into the second, ..., and the residue of the line into the last variable. If there's only one variable, the whole line goes into that variable.
There are many uses. This is one of the simpler scripts I have that uses the split option:
#!/bin/ksh
#
# #(#)$Id: mkdbs.sh,v 1.4 2008/10/12 02:41:42 jleffler Exp $
#
# Create basic set of databases
MKDUAL=$HOME/bin/mkdual.sql
ELEMENTS=$HOME/src/sqltools/SQL/elements.sql
cat <<! |
mode_ansi with log mode ansi
logged with buffered log
unlogged
stores with buffered log
!
while read dbs logging
do
if [ "$dbs" = "unlogged" ]
then bw=""; cw=""
else bw="-ebegin"; cw="-ecommit"
fi
sqlcmd -xe "create database $dbs $logging" \
$bw -e "grant resource to public" -f $MKDUAL -f $ELEMENTS $cw
done
The cat command with a here-document has its output sent to a pipe, so the output goes into the while read dbs logging loop. The first word goes into $dbs and is the name of the (Informix) database I want to create. The remainder of the line is placed into $logging. The body of the loop deals with unlogged databases (where begin and commit do not work), then run a program sqlcmd (completely separate from the Microsoft new-comer of the same name; it's been around since about 1990) to create a database and populate it with some standard tables and data - a simulation of the Oracle 'dual' table, and a set of tables related to the 'table of elements'.
Other scripts that use the read command are bigger (by far), but generally read lines containing one or more file names and some other attributes of relevance, and then apply an appropriate transform to the files using the attributes.
Osiris JL: file * | grep 'sh.*script' | sed 's/:.*//' | xargs wgrep read
esqlcver:read version letter
jlss: while read directory
jlss: read x || exit
jlss: read x || exit
jlss: while read file type link owner group perms
jlss: read x || exit
jlss: while read file type link owner group perms
kb: while read size name
mkbod: while read directory
mkbod:while read dist comp
mkdbs:while read dbs logging
mkmsd:while read msdfile master
mknmd:while read gfile sfile version notes
publictimestamp:while read name type title
publictimestamp:while read name type title
Osiris JL:
'Osiris JL: ' is my command line prompt; I ran this in my 'bin' directory. 'wgrep' is a variant of grep that only matches entire words (to avoid words like 'already'). This gives some indication of how I've used it.
The 'read x || exit' lines are for an interactive script that reads a response from standard input, but exits if the command gets EOF (for example, if standard input comes from /dev/null).