Requirement
Several files in remote machines ought to be deleted via sh. Name of the files to be deleted are know
Approach
1) script was written with ftp (requires credential) and delete command. File names were passed as array(iterated via for loop-with ftp+delete commands enclosed within for loop). files were not getting deleted by this approach
2) another approach attempted was to pass temp.ftp(which contains delete command) to ftp command and rm the temp.ftp file eg.ftp <
Request
require pointers to delete muliple files in remote machine via shell script
I recommend using ssh instead of ftp for interfacing with the remote unix machine.
SSH allows you to run remote commands easily and securely.
Read this article for more info.
Related
I want to move files from local folder to remote URL in a scheduled way using WebDAV.
I found this URL useful but this shows the script to transfer only single file, instead I want to transfer all files from a local folder to remote URL through winscp using WebDAV protocol:
http://winscp.net/eng/docs/guide_automation
Any pointers for this would be helpful.
Use the following WinSCP script:
open http://user:password#example.com/
put d:\path\* /home/user/
close
Read about file masks.
If you really need to move files (as opposite to copy), add the -delete switch to the put command:
put -delete d:\path\* /home/user/
For scheduling, see the WinSCP guide to scheduling file transfers.
I use Excel + R on Windows on a rather slow desktop. I have a full admin access to very fast Ubuntu-based server. I am wondering: how to remotely execute commands on the server?
What I can do is to save the needed variables with saveRDS, and load them on server with loadRDS, execute the commands on server, and then save the results and load them on Windows.
But it is all very interactive and manual, and can hardly be done on regular basis.
Is there any way to do the stuff directly from R, like
Connect with the server via e.g. ssh,
Transfer the needed objects (which can be specified manually)
Execute given code on the server and wait for the result
Get the result.
I could run the whole R remotely, but then it would spawn a network-related problems. Most R commands I do from within Excel are very fast and data-hungry. I just need to remotely execute some specific commands, not all of them.
Here is my setup.
Copy your code and data over using scp. (I used github, so I clone my code from github. This has the benefit of making sure that my work is reproducible)
(optional) Use sshfs to mount the remote folder on your local machine. This allows you to edit the remote files using your local text editor instead of ssh command line.
Put all things you want to run in an R script (on the remote server), then run it via ssh in R batch mode.
There are a few options, the simplest is to exchange secure keys to avoid entering SSH/SCP passwords manually all the time. After this, you can write a simple R script that will:
Save necessary variables into a data file,
Use scp to upload the data file to ubuntu server
Use ssh to run remote script that will process the data (which you have just uploaded) and store the result in another data file
Again, use scp command to transfer the results back to your workstation.
You can use R's system command to run scp and ssh with necessary options.
Another option is to set up cluster worker at the remote machine, then you can export the data using clusterExport and evaluate expressions using clusterEvalQ and clusterApply.
There are a few more options:
1) You can do the stuff directly from R by using Rserve. See: https://rforge.net/
Keep in mind that Rserve can accept connections from R clients, see for example how to connect to Rserve with an R client.
2) You can set up cluster on your linux machine and then use these cluster facilities from your windows client. The simplest is to use Snow, https://cran.r-project.org/package=snow, also see foreach and many other cluster libraries.
I'm searching for a solution to easily delete files on a SFTP server automatically after a specific period of days.
I read something about automation via PSFTP from PuTTY and WinSCP. But there are no examples for deleting files...
With WinSCP, use the rm command with a filemask with a time constraint (to select only files with certain age, if I understand correctly, what you want to do).
E.g. to delete all files older than 5 days:
rm *<5D
For all the other instructions to assemble a script and schedule its run, see:
Automate file transfers (or synchronization) to FTP server or SFTP server
Schedule file transfers (or synchronization) to FTP/SFTP server
See also:
Documentation for the rm command;
Documentation for the file masks.
SFTP server is also a SSH server. Write a simple code in any language that you're comfortable with and schedule it with crontab.
I am trying to transfer a file to a remote Unix server using scp. On that server, there is a service which polls the target directory to detect incoming files for processing. I would like to ensure that the polling service does not pick up new files before the copy is complete. Is there a way of doing that?
My file transfer process is a simple scp command embedded in a larger Java program. Ideally, a solution which did not involve changing the Jana would be best (for reasons involving change control processes).
You can scp the file to a different (/tmp) directory and move the
file via ssh after transfer is complete. The different directory needs to be on the same partition as the final destination directory otherwise there will be a copy operation and you'll face a similar problem. Another service on the destination machine can do this move operation.
You can copy the file as hidden (prefix the filename with .) and copy, then move
If you can modify the polling service, you can check active scp processes and ignore files matching scp arguments.
You can check for open files with lsof +d $directory and ignore them in the polling server
I suggest copying the file using rsync instead of scp. rsync already copies new files to temporary filenames, and has many other useful features for file synchronization as well.
$ rsync -a source/path/ remotehost:/target/path/
Of course, you can also copy file-by-file if that's your preference.
If rsync's temporary filenames are sufficient to avoid being picked up by your polling service, then you could simply replace your scp command with a shell script that acts as a wrapper for rsync, eliminating the need to change your Java program.
You would need to know the precise format that your Java program uses to call the scp command, to make sure that the options you feed to rsync do what you expect.
You would also need to figure out how your Java program calls scp. If it does so by full pathname (i.e. /usr/bin/scp), then this solution might put other things at risk on your system that depend on scp (like you, for example, expecting scp to behave as it usually does instead of as a wrapper). Changing a package-installed binary like /usr/bin/scp may also "break" your package registration, making it difficult to install future security updates because a binary has changed to a shell script. And of course, there might be security implications to any change you make.
All in all, I suspect you're better off changing your Java program to make it do precisely what you want, even if that is to launch a shell script to handle aspects of automation that you want to be able to change in the future without modifying your Java.
Good luck!
Suppose I have a file called helloworld.txt on an AS400 and I want to write a script to automate the daily upload of the source file helloworld.txt on an AS400 server to upload to a Unix SFTP server, say sftp://exampleunixsftp.com?
Does someone have such a script? Is OpenSSH the only tool that can be used on the AS400 to get this accomplished or are there any other methods? If LFTP could be installed on the AS400, that would be an easy solution but since it's only for Unix/Linux/Win/Mac then I don't have this option. I read Scott Klement's article at:
http://systeminetwork.com/article/ssh-scp-and-sftp-tools-openssh
I just need an example basic script to get this done. I'd appreciate this.
The MidrangeWiki topic on SSH has some information and examples that may be helpful.
sftp is intended primarily as an interactive utility. scp performs the same function non-interactively.
The command, once authentication is set up, would simply be:
scp helloworld.txt username#exampleunixsftp.com:<destination path>
LFTP can be installed with some preparation in the PASE environment. See Open Source Binaries. Currently offline. Use the Google cache.