I need to download files from sftp location using "Get a file with SFTP" in Pentaho. My connections goes through but i can't download any file from this location.
Can anyone please suggest, how can I download files using SFTP step in Pentaho?
Some more information is required: Are you getting an error? If so, which one? If not, then you're probably not specifying the folder and/or the file names correctly. It's often an issue of incorrect regular expressions.
the following example works, assuming it's not a connection issue:
General tab:
SFTP server name/ IP: ${your_sftp_host_here}
Port: 22 (or else, if set differently on the source server)
Username: ${your_user_name}
Password: ${your_password}
Files tab:
Remote directory: ${your_remote_directory} (in relation to the root folder you land in)
Wildcard (regular expression): ${your_expression} (for example: .*2015-02-09.csv would look for any csv file that starts with any prefix and ends with this particular date)
Target directory: ${your_local_target_dir}
Should that not work for you, simply use the command line to check whether you can even connect to via SFTP, regardless of PDI.
I too receive an error, my error is:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
Algorithm negotiation fail
Might be because my personal laptop, where this instance of Kettle resides, cannot SFTP to the device due to firewall rules.
Related
I am interacting with a SFTP server that deletes files once they are downloaded. To prevent data loss, I need to read the file, land it on persistent storage and then close the connection to indicate I have received it. What's not obvious to me is what happens if I can't safely store the file. Is there a way to close the connection in a way that semantically indicates 'I'm closing the connection and it failed'? All I am finding in the RFC is a SSH_FXP_CLOSE message, which seems to only signal successful transfer. All the other error message types appear to only be used when a server returns a response to a client, not the other way around.
You cannot signal an error to the SFTP server, that's not what SFTP is intended for.
For your particular case, simply closing an SFTP connection without closing the file explicitly (not sending the SSH_FXP_CLOSE) could indicate to the server that something went wrong.
Though it really depends on your server implementation, what it considers an error. The documentation of the SFTP server should describe what it is that triggers the delete.
In SFTP there's nothing like a "download" operation (contrary to FTP RETR command). There are only trivial file operations, like opening file (for reading or writing), reading piece of a file, closing a file. So it is not as simple as "server deletes the file after it is downloaded". The rule can say for example "server deletes the file after it is closed after being previously opened for reading" or something like that.
We have a java utility which is used to extract mail attachments in Unix. Its running fine in Solaris environment. It takes input as mailbox file location which contains the mail stored as a queue. Based on the subject which is passed as arguement it finds the mail in mailbox file and extract the attachement and copy it some defined remote location in unix.
The problem I am facing is that when I try to run the same utility in AIX environment. There is a field called content-length which was coming in the mail messages in Solaris environment but its absent in AIX environment. Can someone tell me how can we set the content-length in AIX environment ?
The Content-Length header is a UNIX-specific extension originally introduced in System V UNIX, so I'm surprised that AIX is no longer using it.
Are you reading the file in UNIX "mbox" format? Are you using the JavaMail mbox provider? It should work both with and without the Content-Length header.
I have an IN folder and an OUT folder. I have Receive Location with XMLReceive pipeline set. and Send Port with Pass through Transmit Pipeline set. Between these two (receive and send) ports I am calling a web service with custom piplines (both send and receive).
Problem is when I drop a file in IN folder I am receiving two files in OUT folder,
1st one is the same which I dropped in IN folder.
2nd one is the required resultant file.
My Question is why I am getting original file again. Where I am doing wrong? Please help
Regards,
Check your send port filters and subscriptions, then adjust your filters.
So, I used the example on http://www.jcraft.com/jsch/examples/Sftp.java and I was trying to connect to ftp.secureftp-test.com.
That destination is a valid testing SFTP server (as mentioned in secureftp-test dot com/). I confirmed it by connecting to the server through nautilus. I also made sure that the password on my program was correct as well.
But, when I run the program and it hits session.connect(), it just says "INFO: Connection established" then it becomes quiet for a minute then it return:
"com.jcraft.jsch.JSchException: connection is closed by foreign host"
I am pretty sure that it got stuck in the while loop of the library but I do not know why. I tried it against my local ftp server and it also had the same problem.
During the quiet moment, I can type. But pressing enter does not send anything to the server.
Has anyone heard or seen the same problem?
Okay, it seems that secureftp-test.com is NOT an sftp server. That is ftps server.
What I did instead was creating my own sftp:
http://wiki.vpslink.com/Configuring_vsftpd_for_secure_connections_(TLS/SSL/SFTP)
Then run the example code but making sure that I have these lines before doing session.connect:
String knownHostsFilename = "/home/yourname/.ssh/known_hosts";
jsch.setKnownHosts(knownHostsFilename);
Suppose I want to transfer just a portion of a file over FTP - is it possible using a standard FTP protocol?
In HTTP I could use a Range header in the request to specify the data range of the remote resource. If it's a 1mb file, I could ask for the bytes from 600k to 700k.
Is there anything like that in FTP? I am reading the FTP RFC, don't see anything, but want to make sure I'm not missing anything.
There's a Restart command in FTP - would that work?
Addendum
After getting Brian Bondy's answer below, I wrote a read-only Stream class that wraps FTP. It supports Seek() and Read() operations on a resource that is read via FTP, based on the REST verb.
Find it at http://cheeso.members.winisp.net/srcview.aspx?dir=streams&file=FtpReadStream.cs
It's pretty slow to Seek(), because setting up the data socket takes a long time. Best results come when you wrap that stream in a BufferedStream.
Yes you can use the REST command.
REST sets the point at which a subsequent file transfer should start. It is used usually for restarting interrupted transfers. The command must come right before a RETR or STOR and so come after a PORT or PASV.
From FTP's RFC 959:
RESTART (REST) The argument field
represents the server marker at which
file transfer is to be restarted. This
command does not cause file transfer
but skips over the file to the
specified data checkpoint. This
command shall be immediately followed
by the appropriate FTP service command
which shall cause file transfer to
resume.
Read more:
http://www.faqs.org/rfcs/rfc959.html#ixzz0jZp8azux
You should check out how GridFTP does parallel transfers. That's using the sort of techniques that you want (and might actually be code that it is better to borrow rather than implementing from scratch yourself).