We have a client SFTP using which we receive our files. The problem is that their SFTP behaves in a wierd way. All the folder and files in it has the same timestamp, even if the file is posted today it shows timestamp as 01/01/2013
How is this possible?
If i download the same files to my local it shows different timestamp (which i believe is the original timestamp for the files)
The Left side is my local and the right side is SFTP.
Is there any way to identify what are the files that are posted today directly from SFTP, during this conditions(Timestamp not updating).
We are looking for a way to have it automated but not sure how to do it in this scenario
Any help will be much appreciated
If you're using a command-line sftp client you can just use the -p configuration flag to preserve timestamps either when starting the sftp client or on download.
For example, doing this downloads all files in a directory and sets their timestamp to now:
sftp> mget *
Using the -p flag preserves the timestamps of the source, however:
sftp> mget -p *
I assume the graphical client you're using has something similar.
Typically for automated file transfers such as you described here I would recommend using rsync. rsync has the ability to only transfer what has changed and can preserve the timestamps. It has many options. It comes with Linux and the Mac. There likely is rsync for Windows if needed.
Related
I'm searching for a solution to easily delete files on a SFTP server automatically after a specific period of days.
I read something about automation via PSFTP from PuTTY and WinSCP. But there are no examples for deleting files...
With WinSCP, use the rm command with a filemask with a time constraint (to select only files with certain age, if I understand correctly, what you want to do).
E.g. to delete all files older than 5 days:
rm *<5D
For all the other instructions to assemble a script and schedule its run, see:
Automate file transfers (or synchronization) to FTP server or SFTP server
Schedule file transfers (or synchronization) to FTP/SFTP server
See also:
Documentation for the rm command;
Documentation for the file masks.
SFTP server is also a SSH server. Write a simple code in any language that you're comfortable with and schedule it with crontab.
I am trying to transfer a file to a remote Unix server using scp. On that server, there is a service which polls the target directory to detect incoming files for processing. I would like to ensure that the polling service does not pick up new files before the copy is complete. Is there a way of doing that?
My file transfer process is a simple scp command embedded in a larger Java program. Ideally, a solution which did not involve changing the Jana would be best (for reasons involving change control processes).
You can scp the file to a different (/tmp) directory and move the
file via ssh after transfer is complete. The different directory needs to be on the same partition as the final destination directory otherwise there will be a copy operation and you'll face a similar problem. Another service on the destination machine can do this move operation.
You can copy the file as hidden (prefix the filename with .) and copy, then move
If you can modify the polling service, you can check active scp processes and ignore files matching scp arguments.
You can check for open files with lsof +d $directory and ignore them in the polling server
I suggest copying the file using rsync instead of scp. rsync already copies new files to temporary filenames, and has many other useful features for file synchronization as well.
$ rsync -a source/path/ remotehost:/target/path/
Of course, you can also copy file-by-file if that's your preference.
If rsync's temporary filenames are sufficient to avoid being picked up by your polling service, then you could simply replace your scp command with a shell script that acts as a wrapper for rsync, eliminating the need to change your Java program.
You would need to know the precise format that your Java program uses to call the scp command, to make sure that the options you feed to rsync do what you expect.
You would also need to figure out how your Java program calls scp. If it does so by full pathname (i.e. /usr/bin/scp), then this solution might put other things at risk on your system that depend on scp (like you, for example, expecting scp to behave as it usually does instead of as a wrapper). Changing a package-installed binary like /usr/bin/scp may also "break" your package registration, making it difficult to install future security updates because a binary has changed to a shell script. And of course, there might be security implications to any change you make.
All in all, I suspect you're better off changing your Java program to make it do precisely what you want, even if that is to launch a shell script to handle aspects of automation that you want to be able to change in the future without modifying your Java.
Good luck!
Suppose I have a file called helloworld.txt on an AS400 and I want to write a script to automate the daily upload of the source file helloworld.txt on an AS400 server to upload to a Unix SFTP server, say sftp://exampleunixsftp.com?
Does someone have such a script? Is OpenSSH the only tool that can be used on the AS400 to get this accomplished or are there any other methods? If LFTP could be installed on the AS400, that would be an easy solution but since it's only for Unix/Linux/Win/Mac then I don't have this option. I read Scott Klement's article at:
http://systeminetwork.com/article/ssh-scp-and-sftp-tools-openssh
I just need an example basic script to get this done. I'd appreciate this.
The MidrangeWiki topic on SSH has some information and examples that may be helpful.
sftp is intended primarily as an interactive utility. scp performs the same function non-interactively.
The command, once authentication is set up, would simply be:
scp helloworld.txt username#exampleunixsftp.com:<destination path>
LFTP can be installed with some preparation in the PASE environment. See Open Source Binaries. Currently offline. Use the Google cache.
I have a weird problem. I'm using pscp.exe from within a C# program (with process.start) to upload files to an SFTP server. Now I have set up a new server with the same program, where I upload to the same SFTP server as before, but It runs incredibly slow in the new server.
The weird thing is that when I try uploading the files manually via FileZilla, the upload goes as fast as expected, but not when using the program.
Can anyone explain this? Am I missing something obvious like a windows setting or something?
SSH supports what we call pipelining - sending multiple SSH packets without waiting response to each packet. OpenSSH supports this functionality, while Putty doesn't (at least didn't until recently). That's what you observe. Another reason is choice of algorithms. If AES is negotiated, it's faster than DES and 3DES used by default by older applications.
I ended up rewriting the SFTP transfer to use the .Net wrapper for WinSCP in stead. The solution was fast, and the file transfer also. Here's a link to the documentation.
Uploading files using WinSCP is like 10 times faster.
To do that from command line, first you got to add the winscp.com file to your %PATH%. It's not a top-level domain, but an executable .com file, which is located in your WinSCP installation directory.
Then just issue a simple command and your file will be uploaded much faster putty ever could:
WinSCP.com /command "open sftp://username:password#example.com:22" "put your_large_file.zip /var/www/somedirectory/" "exit"
And make sure your check the synchronize folders feature, which is basically what rsync does, so you won't ever want to use pscp.exe again.
WinSCP.com /command "help synchronize"
Filezilla can use multiple concurrent connections and reuse open connections. I believe PSCP is a relatively simple application.
A library like SFTP.NET will probably yield better results than running a child pscp process.
It would also help to use the ZipPackage to compress the files when sending them.
Requirement
Several files in remote machines ought to be deleted via sh. Name of the files to be deleted are know
Approach
1) script was written with ftp (requires credential) and delete command. File names were passed as array(iterated via for loop-with ftp+delete commands enclosed within for loop). files were not getting deleted by this approach
2) another approach attempted was to pass temp.ftp(which contains delete command) to ftp command and rm the temp.ftp file eg.ftp <
Request
require pointers to delete muliple files in remote machine via shell script
I recommend using ssh instead of ftp for interfacing with the remote unix machine.
SSH allows you to run remote commands easily and securely.
Read this article for more info.