I have an IN folder and an OUT folder. I have Receive Location with XMLReceive pipeline set. and Send Port with Pass through Transmit Pipeline set. Between these two (receive and send) ports I am calling a web service with custom piplines (both send and receive).
Problem is when I drop a file in IN folder I am receiving two files in OUT folder,
1st one is the same which I dropped in IN folder.
2nd one is the required resultant file.
My Question is why I am getting original file again. Where I am doing wrong? Please help
Regards,
Check your send port filters and subscriptions, then adjust your filters.
Related
I've got a weird issue which totally confused me. I have an orchestration that receives a file through Receive port and puts it to Send port with delivery notification. After that orchestration performs some other actions but my issue is with the part mentioned above. It works: I put a file to Receive location and it goes to appropriate destination folder.
But, if I assign to the Send port a filter that uses fields promoted in message and filter's expression is False it still works - I receive file in destination folder (however filter is False and Send port should not process this file). And once filter becomes True I receive...THREE equal files in destination folder!
Are there any logs that I can go through to try to figure out where these two extra files come from?
Even when you add a filter to a send port, if the Orchestration is bound to the send port, it will still send through that send port, regardless what other filters you put on it.
In the Admin Console you do a query for Subscriptions and find the send port, you will notice that it will have an OR condition with http://schemas.microsoft.com/BizTalk/2003/system-properties.SPTransportID == {GUID} and that is the filter that is used when a an Orchestration sends to a bound send port.
So check what the filter looks like for your send port. Also check to see if you have a send port group set up that includes that send port, as that would also cause a message to be sent to that send port if the filter on that matches.
What is probably happening is that the Orchestration is sending one, and the filter is matching the message received from the Receive Location. You will have to do some more digging to see why the third message.
Is there any reason to be worried about the possibility that upon attempting to upload a file via SFTP, the server successfully writes out the file but then the connection with the client breaks before any confirmation is sent back, hence the client is left not knowing whether the file was uploaded successfully?
The contents of the file are such that it's vital the client knows for sure that it was written out in full. To be certain, is it best to re-attempt connecting to the server and confirm if the file was written in full? Or should I be able to trust that the client-side library would do this for me?
It's not just possible, this is observed from time to time (mostly with misconfigured servers). So your client should be ready to track its latest operation, and if it was eg. file upload, then the client would need to reconnect and check if the file is on the server, and has the expected size and date. On the other hand, such scenario can fail in cases when the server moves away the file right after upload completion (usually for processing).
I need to download files from sftp location using "Get a file with SFTP" in Pentaho. My connections goes through but i can't download any file from this location.
Can anyone please suggest, how can I download files using SFTP step in Pentaho?
Some more information is required: Are you getting an error? If so, which one? If not, then you're probably not specifying the folder and/or the file names correctly. It's often an issue of incorrect regular expressions.
the following example works, assuming it's not a connection issue:
General tab:
SFTP server name/ IP: ${your_sftp_host_here}
Port: 22 (or else, if set differently on the source server)
Username: ${your_user_name}
Password: ${your_password}
Files tab:
Remote directory: ${your_remote_directory} (in relation to the root folder you land in)
Wildcard (regular expression): ${your_expression} (for example: .*2015-02-09.csv would look for any csv file that starts with any prefix and ends with this particular date)
Target directory: ${your_local_target_dir}
Should that not work for you, simply use the command line to check whether you can even connect to via SFTP, regardless of PDI.
I too receive an error, my error is:
com.jcraft.jsch.JSchException: Algorithm negotiation fail
Algorithm negotiation fail
Might be because my personal laptop, where this instance of Kettle resides, cannot SFTP to the device due to firewall rules.
I have requirement where I need to send Flat file data to multiple customers.
One way is to create Send port for each of the customer and put a filter on the port to send the data specific to the Customer ID.
Now problem that I see is that customer will increase in coming days for sure so creating a new port for each customer is not a big deal but still I looking for some dynamic approach.
Can this be done with some kind of dynamic configuration for a single port?
You could use a Dynamic Send Port. This will give you the possibility to either use information in the message to configure the destination of the port run time. Another way is to have a database where you use some sort of customer id in the message to look up the destination and use that to configure the send port run time.
Suppose I want to transfer just a portion of a file over FTP - is it possible using a standard FTP protocol?
In HTTP I could use a Range header in the request to specify the data range of the remote resource. If it's a 1mb file, I could ask for the bytes from 600k to 700k.
Is there anything like that in FTP? I am reading the FTP RFC, don't see anything, but want to make sure I'm not missing anything.
There's a Restart command in FTP - would that work?
Addendum
After getting Brian Bondy's answer below, I wrote a read-only Stream class that wraps FTP. It supports Seek() and Read() operations on a resource that is read via FTP, based on the REST verb.
Find it at http://cheeso.members.winisp.net/srcview.aspx?dir=streams&file=FtpReadStream.cs
It's pretty slow to Seek(), because setting up the data socket takes a long time. Best results come when you wrap that stream in a BufferedStream.
Yes you can use the REST command.
REST sets the point at which a subsequent file transfer should start. It is used usually for restarting interrupted transfers. The command must come right before a RETR or STOR and so come after a PORT or PASV.
From FTP's RFC 959:
RESTART (REST) The argument field
represents the server marker at which
file transfer is to be restarted. This
command does not cause file transfer
but skips over the file to the
specified data checkpoint. This
command shall be immediately followed
by the appropriate FTP service command
which shall cause file transfer to
resume.
Read more:
http://www.faqs.org/rfcs/rfc959.html#ixzz0jZp8azux
You should check out how GridFTP does parallel transfers. That's using the sort of techniques that you want (and might actually be code that it is better to borrow rather than implementing from scratch yourself).