I realise it is likely the each of the requirements listed below is available individually within the forum here but I am struggling to bring it all together (if at all possible!).
Hoping someone has the patience and time to point me in right direction to make this happen.
What I need to do is the following:
Scan a directory (and all sub-directories) for a particular filename
NOTE: Whilst there are many files within the sub-directories with the filename in question, we only wish to target those in a sub-directory with a suffix of JERRY
ie. In the below example the files indicated by the arrow would be targeted
ONE\NEW1-JERRY\FILENAME.TXT <----
ONE\NEW1-TOM\FILENAME.TXT
ONE\NEW1-SYLVESTER\FILENAME.TXT
TWO\NEW2-JERRY\FILENAME.TXT <----
TWO\NEW2-TOM\FILENAME.TXT
TWO\NEW2-SYLVESTER\FILENAME.TXT
THREE\NEW3-JERRY\FILENAME.TXT <----
THREE\NEW3-TOM\FILENAME.TXT
THREE\NEW3-SYLVESTER\FILENAME.TXT
FOUR\NEW4-JERRY\FILENAME.TXT <----
FOUR\NEW4-TOM\FILENAME.TXT
FOUR\NEW4-SYLVESTER\FILENAME.TXT
When file is found matching the filename and is within the sub-directory listed above take a copy of the file (to remain in same directory) & rename based on the following criteria:
a) Created date/time
b) Certain content within the file
The content in the file is always located on ROW 8 and it is the first 9 characters
Original filename: FILENAME.txt
Finished Product: FILENAME-20121129#1300-123456789.txt
Thanks in advance!
you should tell us what have you tried so far and what are your mains problems...
try this :
(remove the -whatif flag to actually copy files)
#list dir & subdirs
ls c:\ -recurse |
Foreach {
#find subdirs named JERRY
if($_.PSIsContainer -and $_.name -match "JERRY"){
ls $_.fullname -filter 'FILENAME*' |
Foreach{
$fcontent=get-Content $_.FullName
$content=$fcontent[7].Substring(0,9)
$newName=Get-Date -UFormat "%Y%m%d"
$destination="$($_.Directory)\$newName#$content.txt"
write-verbose $destination
Copy-Item $_.FullName -Destination $destination -WhatIf
}
}
}
Related
I have a folder that contains a lot of files. In this case images.
I need to organise these images into a directory structure.
I have a spreadsheet that contains the filenames and the corresponding path where the file should be copied to. I've saved this file as a text document named files.txt
+--------------+-----------------------+
| image01.jpg | path/to/destination |
+--------------+-----------------------+
| image02.jpg | path/to/destination |
+--------------+-----------------------+
I'm trying to use rsync with the --files-from flag but can't get it to work.
According to man rsync:
--include-from=FILE
This option is related to the --include option, but it specifies a FILE that contains include patterns (one per line). Blank lines in the file and lines starting with ';' or '#' are ignored. If FILE is -, the list will be read from standard input
Here's the command i'm using: rsync -a --files-from=/path/to/files.txt path/to/destinationFolder
And here's the rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.1/rsync/options.c(1436) [client=2.6.9]
It's still pretty unclear to me how the files.txt document should be formatted/structured and why my command is failing.
Any help is appreciated.
I am trying to copy over the contents from a file to a directory.
I wrote the command find . -name "file" | while read f; do cp "$$f" directory; done
This however moves the actual filename with the contents. I only want the contents of "file" to get placed into the directory appreciate any assistance.
My zip file has my folders inside. After unzipping my zip file, I want to iterate a loop for available folders inside the zip.
Inside loop condition is like below:
If my folder has index file (This is a file contains some data), then only I want to run some process (I know what this process is..). Otherwise we can ignore that folder.
Then loop will continue with other folder if there are anything
Thanks advance..
something like this?
(note: I assume $destdir will only contain the zipfile and its extraction!)
zipfile="/path/to/the/zipfile.zip"
destdir="/path/to/where/you/want/to/unzip"
indexfile="index.txt" #name of the index files
mkdir -p "$destdir" 2>/dev/null #make "sure" it exists.. but ignore errors in case it already exists
cd "$destdir" || { echo "Can not go into destdir=$destdir" ; Exit 1 ; }
#at that point, we are inside $destdir : we can start to work:
unzip "$zipfile"
for i in ./*/ ; do # you could change ./*/ to ./*/*/ if the zip contains a master directory too
cd "$i" && { #the && is important: you want to be sure you could enter that subdir!
if [ -e ./"$indexfile" ]; then
dosomething # you can define the function dosomething and use it here..
# or just place commands here
fi
cd - #we can safely this works, as we started there...
}
done
note: I iterate on ./*/ instead of */ as the dirname could contain a leding -, and therefore make cd -something not work (it would say it can't recognise some options!) ! this goes away with ./, cd ./-something will work !
What does . (dot) mean as the destination of the cp command?
For example:
cp ~dir1/dir2/dir3/executableFile.x .
When this executes it copies the file successfully with the correct file name, but I'm wondering is this what a destination of '.' will always do or is there another purpose?
Within the reference material I've seen, dots are used in front of files to indicate 'hidden', but in that has no relation to the command above.
dot represents the current directory
while dotdot is the parent directory.
As EvilTeach's answer says, . is the current directory, and .. is the parent directory.
There are basically two ways to use the cp command:
cp file1 file2
will copy file1 to file2, creating file2 if it doesn't exist or (depending on permissions) possibly clobbering it if it does.
The other way is:
cp file1 file2 ... dir
where dir is an existing directory. With this form, you can specify one or more files, and they'll all be copied into the specified directory dir with their existing names.
(This can be a pitfall sometimes; cp foo bar behaves very differently depending on whether there's an existing directory named bar.)
As you mention, files (including directories) whose names start with . are hidden. What this means is that (a) the ls command won't list them (unless you use the -a or -A option), and (b) a shell wildcard such as * or *.txt will omit them. (GUI directory managers such as Nautilus may also omit them, depending on your settings.)
This applies to the current directory . and the parent directory ... ls won't include the . and .. entries in its output; ls -a will.
I have a highly structured hierarchical directory containing multiple files that need to be moved into a flat structure and renamed at the same time. The original path and name must be logged along with the new path and name and eventually loaded into a database. Finally, each renamed file must get a unique, unguessable (IE: encrypted or hashed) file name. When the renamed file is moved into the new directory structure, I also want to limit the # of files in each directory, so each directory would be created with a sequential number for its name and then the files would be loaded into it until a maximum number of files was reached (eg: 255) before rolling into a new directory with the next sequential number for its name.
Is there a tool / software that does this? I did some initial research and nothing came up with the following criteria:
batch rename & copy into alternative (flatter) structure
hash / encrypt filename and ensure uniqueness
sequentially name folders and limit file count
log each file's original name and path, and new (encrypted) name and path
I have several Bash scripts I have used in the past to migrate hand-made file repositories to hashed repositories to be accessed and managed from a web application (mostly PHP apps). In these repositories filenames are hashed (to avoid collisions with files with the same content/name) and files are distributed evenly (in a deterministic fashion or randomly) to keep files-per-dir count low for performance reasons. The following is one fully-working example:
#!/bin/bash
MAXFILESPERDIR=500
TARGETROOTDIR="./newrepository"
RANDOMDISTRIBUTION=1
if [ -d "$1" ]; then
LOGFILE=$(basename $0).$(date +"_%Y%m%d_%H%M").${$}.log
SQLFILE=$(basename $0).$(date +"_%Y%m%d_%H%M").${$}.sql
SOURCEDIR="$1"
TOTALSOURCEFILES=$(find "$1" -type f | wc -l)
let "TOTALTARGETDIRS=$TOTALSOURCEFILES / $MAXFILESPERDIR"
PADLENTARGETDIRS=${#TOTALTARGETDIRS}
PADLENTARGETFILE=${#TOTALSOURCEFILES}
echo "We will create $TOTALTARGETDIRS directories to hold $MAXFILESPERDIR files per directory."
if [ "$RANDOMDISTRIBUTION" == "1" ] ; then
echo "We will rename and distribute each file randomly."
else
echo "We will rename and distribute each file uniformly."
fi
echo "Do you want to continue?"
select choice in yes no ; do
if [ "$choice" == "yes" ] ; then
COUNTER=1
find "$1" -type f | while read SOURCEFILE ; do {
CHECKSUMFILE=$(sha1sum "$SOURCEFILE" | cut -d " " -f 1)
CHECKSUMNAME=$(echo "$SOURCEFILE" | sha1sum | cut -d " " -f 1)
DETERMINISTICNONCE=$(printf "%0${PADLENTARGETFILE}d\n" $COUNTER)
if [ "$RANDOMDISTRIBUTION" == "1" ] ; then
PROBABILISTICNONCE=$(let "XX=$RANDOM % $TOTALTARGETDIRS + 1" ; printf "%0${PADLENTARGETDIRS}d\n" $XX;)
else
PROBABILISTICNONCE=$(let "XX=$COUNTER % $TOTALTARGETDIRS + 1" ; printf "%0${PADLENTARGETDIRS}d\n" $XX;)
fi
FILEDATE=$(stat -c %z "$SOURCEFILE" | cut -d "." -f 1)
FILESIZE=$(stat -c %s "$SOURCEFILE")
echo "Source file $SOURCEFILE" >> $LOGFILE
echo "Target file $TARGETROOTDIR/$PROBABILISTICNONCE/$PROBABILISTICNONCE$CHECKSUMFILE$DETERMINISTICNONCE" >> $LOGFILE
echo "INSERT INTO files (Filename, Location, Checksum, CDate, Size) VALUES ('$PROBABILISTICNONCE$CHECKSUMFILE$DETERMINISTICNONCE', '$PROBABILISTICNONCE', '$CHECKSUMFILE', '$FILEDATE', $FILESIZE);" >> $SQLFILE
mkdir -p $TARGETROOTDIR/$PROBABILISTICNONCE
cp -v "$SOURCEFILE" $TARGETROOTDIR/$PROBABILISTICNONCE/$PROBABILISTICNONCE$CHECKSUMFILE$DETERMINISTICNONCE
let "COUNTER+=1"
} ; done
echo "Done."
echo
break
fi
if [ "$choice" == "no" ] ; then
echo
echo "Operation cancelled"
echo
break
fi
done
else
echo
echo "Missing source directory"
echo
fi
Just run it from the root of your new repository. You can configure it modifying the first variables: MAXFILESPERDIR defines how many files to store per-directory, TARGETROOTDIR is the name of the first-level directory to create the first level directory (it uses only two levels, the first one is really a single root), and RANDOMDISTRIBUTION defines if the files will be distributed randomly (it may look uneven, specially for small runs) or deterministically (just counting).
How it works (FYI, just in case this is not what you are looking for but maybe you can get some ideas):
Count the source files.
Calculate how many target directories will create.
Ask for confirmation.
For each file:
Calculate the SHA1 hash for the file content.
Create a deterministic nonce.
Create a probabilistic nonce (if RANDOMDISTRIBUTION is 1, otherwise just a counter).
Get the size and modification date.
Combine the values of the random value with the hash and the counter to get the new file name (the path will be the random value).
Log the source and target full paths.
Create and log a SQL insert query.
Create the target directory (if it does not exist).
Copy the file. (You can move it if you want but I'm playing safe).
Finish
If you set RANDOMDISTRIBUTION to 1 and run the script several times, you'll get duplicates of your source files, as each file will get different target filename/path each time you run it. If RANDOMDISTRIBUTION is set to something else, everytime you run the script the files will be renamed the same way (for the same file set, if you add or remove files, they will get different names/paths).
The objective of using a random value + hash + counter is to be sure we can handle duplicates (won't collide thanks to the counter) while still distributing the files randomly (for long enough runs, this will distribute the files evenly).
Also, the preffix of the generated file name is the name of the directory too, so that if you have the file name and the directory name length, you can calculate the directory name (just in case you don't store that in your database table).
Finally, this is a one-time migration script, it was not really written to be executed regularly over the same set of files.