I want to download a large number of ftp files. I made a list file that contain the links of the thousands of ftp. FTP download results in 'gbff.gz' files. Now say for some reason I want to restrict the number of downloaded files (with .gz extension) in current directory to 5.
To test that I made a while loop in R that use bash system command:
setwd("~/Desktop/test")
a<-system('find . | grep -i ".gz$" |wc -l')
while (a<5) {
system('wget -nc -tries=1 -i list.txt')
}
But seems like the while loop is not working. I mean, ideally it should break the loop when the number of .gz files in current directory is more than 5, but the download continues for all links in list file.
N.B.- My apologies for making such a hybrid script. As I already working in R it seems easy to me. I would also appreciate any alternate bash/awk/sed script if that is more suitable for this.
N.B2- FYI, I use -nc tag in wget, so a re-download of an already existing file should not occur.
Related
Is there an app or process to download full directories and files in batch? Either using a txt file containing URLs or simply copying and pasting the URLs?
I've tried jdownloader2, IDM, free download manager, etc with no luck
wget seems a possibility from my research, I have never used it before...
Example:
I wish to download to C:\Files\
http:\\www.files.com\dir1\dir2\*.*
http:\\www.files.com\dira\*.*
http:\\www.files.com\dirb\dird\*.*
http:\\www.files.com\a.jpg
http:\\www.files.com\b.txt
http:\\www.files.com\c.nfo
http:\\www.files.com\d.png
And when I open C:\Files\ I have the following:
dir2\*.*
dira\*.*
dird\*.*
a.jpg
b.txt
c.nfo
d.png
I hope that I explained this clear enough.
Thank you for reading this!
I recently attempted to move some files by running an -exec mv command with find (command linked below). When I did this, I mistyped the destination directory path (so the directory did not yet exist) and mv created what appears to be an executable instead of a directory?
When I run "Get Info" one image renders and the file size is about the correct size for an image, but hundreds of files were supposed to be copied. Have I lost this data for good? Is there any way to get macOS to recognize this "executable" as a directory?
This is the command I used:
find . -type f -name "*.JPG" -exec mv {} ../../DestinationFolderName \;
Here's an image showing a successful mv into an existing directory, and what happened when I put a path to a directory that did not yet exist.
Unfortunately "mv" to a name that doesn't exist is interpreted as a filename rather than a directory. So the OS has, one-by-one, copied your JPG file on top of each other. The resulting file is most likely whatever JPG happened to be the one it moved last (if you rename it to JPG extension you can check which one).
So, very unfortunately, you probably need to investigate a data recovery tool for MacOS quickly (and do so before you've done things that make more files on your disk, as much a possible). The "ghosts" of the files are for now at least mostly still present on your hard drive (as deallocated segments), but are back in the pool to be overwritten as you create new files (even when your browser creates temporary cache files, and things like that). It's a conundrum.
If you don't have a backup/time-machine of the files, the best thing to do is get a MacOS data recovery program QUICKLY.
VERY sorry not to have a happier answer.
I am trying to move about 1000 files that all begin with "simulation." into one directory entitled "simulations." I am using a remote server, and the files are currently in my home directory on the server. I need to move them to a separate directory because I need to, ultimately, append all the "simulation." files into one file. Is there a way to either append only the files in my home directory that begin with "simulation." or move only these files into a new directory?
Thank you.
Assuming you can change directories to the desired path on the remote server... and the simulations are located in /currentPath ... then....
cd desiredPath
mkdir simulations
mv /currentPath/simulation* simulations
(to futher answer your question... if you wanted to append all the files together, you could type cat simulation* > allSimulations.txt
I have a number of zip files located in a single folder eg:
file1.gz
file2.gz
file3.gz
file4.gz
I'm looking for a way of automatically unzipping these using a batch job to a similarly named folder structure so for example the contents of file1.gz will drop into a folder named file1.
I have been told that 7zip would address my issue but can't figure out how to go about it.
Any help is greatly appreciated.
Which OS are you using? This is something you'd do using the shell's capabilities, you could write
for A in *.gz ; do gunzip $A ; done
I'm using gunzip here, because .gz is actually gzip, But you can use the 7zip CLI tool as well, of course. If you're on Windows, then I recommend installing a real shell (the standard cmd.exe can not really be considered a shell IMHO).
I need to regularly send a collection of log files that can grow quite large, so I would like to only send the last n lines of the each of the files.
for example:
/usr/local/data_store1/file.txt (500 lines)
/usr/local/data_store2/file.txt (800 lines)
Given a file with a list of needed files named files.txt, I would like to create an archive (tar or zip) with the last 100 lines of each of those files.
I can do this by creating a separate directory structure with the tail-ed files, but that seems like a waste of resources when there's probably some piping magic that can happen to accomplish it. Full directory structure also must be preserved since files can have the same names in different directories.
I would like the solution to be a shell script if possible, but perl (without added modules) is also acceptable (this is for Solaris machines that don't have ruby/python/etc.. installed on them.)
You could try
tail -n 10 your_file.txt | while read line; do zip /tmp/a.zip $line; done
where a.zip is the zip file and 10 is n or
tail -n 10 your_file.txt | xargs tar -czvf test.tar.gz --
for tar.gz
You are focusing in an specific implementation instead of looking at the bigger picture.
If the final goal is to have an exact copy of the files on the target machine while minimizing the amount of data transfered, what you should use is rsync, which automatically sends only the parts of the files that have changed and also can automatically compress while sending and decompress while receiving.
Running rsync doesn't need any more daemons on the target machine that the standard sshd one, and to setup automatic transfers without passwords you just need to use public key authentication.
There is no piping magic for that, you will have to create the folder structure you want and zip that.
mkdir tmp
for i in /usr/local/*/file.txt; do
mkdir -p "`dirname tmp/${i:1}`"
tail -n 100 "$i" > "tmp/${i:1}"
done
zip -r zipfile tmp/*
Use logrotate.
Have a look inside /etc/logrotate.d for examples.
Why not put your log files in SCM?
Your receiver creates a repository on his machine from where he retrieves the files by checking them out.
You send the files just by commiting them. Only the diff will be transmitted.