I am interacting with a SFTP server that deletes files once they are downloaded. To prevent data loss, I need to read the file, land it on persistent storage and then close the connection to indicate I have received it. What's not obvious to me is what happens if I can't safely store the file. Is there a way to close the connection in a way that semantically indicates 'I'm closing the connection and it failed'? All I am finding in the RFC is a SSH_FXP_CLOSE message, which seems to only signal successful transfer. All the other error message types appear to only be used when a server returns a response to a client, not the other way around.
You cannot signal an error to the SFTP server, that's not what SFTP is intended for.
For your particular case, simply closing an SFTP connection without closing the file explicitly (not sending the SSH_FXP_CLOSE) could indicate to the server that something went wrong.
Though it really depends on your server implementation, what it considers an error. The documentation of the SFTP server should describe what it is that triggers the delete.
In SFTP there's nothing like a "download" operation (contrary to FTP RETR command). There are only trivial file operations, like opening file (for reading or writing), reading piece of a file, closing a file. So it is not as simple as "server deletes the file after it is downloaded". The rule can say for example "server deletes the file after it is closed after being previously opened for reading" or something like that.
Related
Is there any reason to be worried about the possibility that upon attempting to upload a file via SFTP, the server successfully writes out the file but then the connection with the client breaks before any confirmation is sent back, hence the client is left not knowing whether the file was uploaded successfully?
The contents of the file are such that it's vital the client knows for sure that it was written out in full. To be certain, is it best to re-attempt connecting to the server and confirm if the file was written in full? Or should I be able to trust that the client-side library would do this for me?
It's not just possible, this is observed from time to time (mostly with misconfigured servers). So your client should be ready to track its latest operation, and if it was eg. file upload, then the client would need to reconnect and check if the file is on the server, and has the expected size and date. On the other hand, such scenario can fail in cases when the server moves away the file right after upload completion (usually for processing).
Short summary: How to accept content that may be endless and not uploaded at once (the connection needs to be kept alive), in a scenario where I'm the server and I'd like the clients to make those uploads in a RESTful (or something close) way ?
In the same way that I can make an http server that keeps the connection alive with a client and may continue sending content that the client reads and parses intantly (probably using a browser), I need to keep a connection opened with a client that will send me data that may not end or be continuously uploaded.
One (simple) way to do this would be simply to have a TCP server and then clients would write data to a socket.
But how do I do this with an HTTP PUT request ? This answers half of the question: "How will I parse a file upload continuously, without the upload finishing ?" But how will clients proceed to upload something that is not even a file and are separate blocks of data, like if they would be writing those blocks of data to a socket ? Is it even possible ?
If your data isn't going to have a discrete end, then you're not really performing an upload; you're doing a streaming scenario. For a streaming scenario, socket handling is much more appropriate.
First I think Sonier is right. But I found this solution by Felix called "Streaming file uploads with node.js" which might be useful.
Furthermore I think node.js might not be best fit for this, because everything has to be kept in memory and with big file-size you can hit a very hard wall. Some other popular node.js file upload solutions are:
https://github.com/felixge/node-formidable
https://github.com/rootslab/formaline
https://github.com/FooBarWidget/multipart-parser
A simple problem. I have an ASHX handler which generates a report. Unfortunately, this process can take 2 or more minutes to finish and Azure will close the connection before this handler can respond. Why? Because the connection is idle for too long, thus it is killed off.
So, I need to keep this connection alive in some way. To make it a bit more complex, the handler is called from a Silverlight application which will call the handler from a frame on the current webpage or (when not running from a browser) create a new browser instance to call the handler.
My challenge is to get around this timeout with a minimum amount of code. But also, the code needs to work exactly as it does now!
Opening the handler in a separate frame or browser window allows the report to be saved anywhere on the system of the user. If I would download it from within the Silverlight code, I will not have proper write access. There will be no permission given to any Silverlight application that needs to write to the local disk, thus the work-around with the browser/frame.
Not too sure about HTTP transport, but you can certainly use TCP keep-alives at the socket level. However, then you need to create socket listener to download HTTP content (way overkill).
Perhaps there is a much simpler solution? Why don't you have the client make the request to generate the report and have the handler return a SAS signature (time limited, read-only signature) to where the report will eventually be put in blob storage. This is very quick and requires no open TCP connection. The report generator should simply create the report in a file to be downloaded at the blob location it sent to the client (any GUID would work here) instead of streaming it back over the response. Finally, the client just needs to poll the location until it gets a file. Now you are nice and asynchronous with short open connections and don't have to worry about this TCP timeout issue. The code to do this is far, far less complex than anything to work around a TCP timeout.
Suppose I want to transfer just a portion of a file over FTP - is it possible using a standard FTP protocol?
In HTTP I could use a Range header in the request to specify the data range of the remote resource. If it's a 1mb file, I could ask for the bytes from 600k to 700k.
Is there anything like that in FTP? I am reading the FTP RFC, don't see anything, but want to make sure I'm not missing anything.
There's a Restart command in FTP - would that work?
Addendum
After getting Brian Bondy's answer below, I wrote a read-only Stream class that wraps FTP. It supports Seek() and Read() operations on a resource that is read via FTP, based on the REST verb.
Find it at http://cheeso.members.winisp.net/srcview.aspx?dir=streams&file=FtpReadStream.cs
It's pretty slow to Seek(), because setting up the data socket takes a long time. Best results come when you wrap that stream in a BufferedStream.
Yes you can use the REST command.
REST sets the point at which a subsequent file transfer should start. It is used usually for restarting interrupted transfers. The command must come right before a RETR or STOR and so come after a PORT or PASV.
From FTP's RFC 959:
RESTART (REST) The argument field
represents the server marker at which
file transfer is to be restarted. This
command does not cause file transfer
but skips over the file to the
specified data checkpoint. This
command shall be immediately followed
by the appropriate FTP service command
which shall cause file transfer to
resume.
Read more:
http://www.faqs.org/rfcs/rfc959.html#ixzz0jZp8azux
You should check out how GridFTP does parallel transfers. That's using the sort of techniques that you want (and might actually be code that it is better to borrow rather than implementing from scratch yourself).
I have a Flex client using a Flash binary (TCP) socket for communication with a Java server. I have a localhost (Apache) server providing a crossdomain.xml file which is wide open just while I am testing.
My code successfully loads the policy file on startup.
I then connect the socket to the server without any difficulty and send a message and get a response. All good so far.
However, when I send a second message through the same socket I get a pause of about 12 seconds then a sandbox violation error:
Security Error: Error #2048: Security sandbox violation: file:///C:/apache_root/ttt1/ttt1.swf cannot load data from localhost:45455.
This is the same port and socket through which the first message succeeded.
I tried re-loading the policy file before every send, but I get the same result.
Any idea why this might be happening? I clearly have an open socket at one point. I am flushing the socket after each send and I tried doing that after each read as well, but the same result.
Thanks in advance
EDIT:
If I recreate the socket prior to every call my code works. I am struggling to believe that this is correct, but maybe there is a Socket setting I am missing.
As far as I know if you're doing binary sockets the crossdomain.xml is not loaded via http.
Have you checked your apache's access logs if the crossdomain is even queried?
You might get connection from flash via tcp from flash asking for the file on your java server (not using http. It just sends the string "" or similar). Look out for them. If you don't answer them within 3 seconds (or so) flash throws an sandbox violation.
The first thing you have to do when you want to make a socket connection is to load the policy file. This only has to be done once per load of the SWF.
Security.allowDomain(host);
Security.loadPolicyFile("xmlsocket://"+host+":"+port);
The request will be made on the assigned port(45455 in your case) your server will have to listen on that port for a request "<policy-file-request/>" without the quotes.
When that request is found then you need to return to the client the crossdomain.xml
with node <allow-access-from domain="*" to-ports="*" />
After the cross domain is sent you need to close the socket on the server side
On the client side you need to ignore the domain response as Flex will handle that however at that time you can reconnect to the socket server.
At this time you can do your data send/receive.
I have a feeling the reason it actually worked for you is because you were using the connection for the policy file to transmit your data before it timed out.
I would suggest reading up on the new style of crossdomain policies and also reading up on the protocol you are using for your socket server
I think it depends on the sandbox-policy you used in the compilation process of your swf not on your crossdomain.xml... maybe this docu helps you:Security sandboxes
But I'm not 100% sure
This sort of sounds like a cache problem. Perhaps you're pulling the first socket connection out of cache and the second one gets rejected because it's getting a 200 from the server.
You might want to add localhost to your flash security exceptions list for debugging. that will quiet the sandbox errors until you get your piece to it's production environment.