When is PostedFile.InputStream available when uploading a large file?
I'd like to pass a Stream to another process and I'm hoping that if a large file was being uploaded that I can pass the Stream straight to that new process w/o writing to the file system. Since the process and/or upload could take a while, I'm wondering if I can start reading the InputStream immediately or whether I have to wait for the whole file to be transferred to the server before it can be processed.
I guess a more general question is - what's the lifecycle of a POST request when file upload is involved?
The PostedFile.InputStream isn't available until the entire file has been uploaded. IIS6 caches the file in memory while IIS7 now caches the file to disk before handing off the input stream to your method.
You can get a HttpModule such as NeatUpload which allows you access to the bits while they're uploading.
Related
I have an ASP.NET web application and I want my users to be able to upload large files. However, some files are very large and uses too much memory.
In principle it should be possible to receive the request stream and write it directly to a FileWriter stream, removing any need to load the entire file into memory first.
I've tried accessing Request.InputStream and writing it directly to a file. It works, but a test using larger files reveal that Request.InputStream is only available after the entire request is already loaded into memory.
Can someone tell me an approach I can use to receive a normal Request.InputStream in ASP.NET and directly write it to a file without first loading it into memory?
Note, the file is sent through a normal request in a browser by posting a form with a file field.
(I actually use BlueImp JQuery File Upload but I don't think it's relevant to this question)
The process is called byte serving.
Byte Serving:
Byte serving is the process of sending only a portion of an HTTP/1.1 message from a server to a client. Byte serving begins when an HTTP server advertises its willingness to serve partial requests using the Accept-Ranges response header. A client then requests a specific part of a file from the server using the Range request header.
Is seems that IIS and ASP.NET are capable of handling Accept-Range headers. There is a Range Controller on Microsoft git repositories.
Here is an article that may be useful in configuring IIS to handle these requests.
My ASP.NET Application has an FileUpload control. My server doesn't have any antivirus program. If I add a byte to binary content of the file before saving file, does my server affect from virus? When displaying file, I will remove extra byte from the content.
Thanks for replies.
A virus will only cause you problems if it is run on the server (i.e. the file is opened). You can get around this by renaming all uploaded files with a .resources extension. All requests for this type of file are sent by IIS to ASP.NET, which rejects them. So effectively, the files store the data but can't be opened/run at all. Then you can still serve them back by reading their content in an ASP.NET page/module, and returning the data as a file with the correct extension.
Transforming the data as you suggest will also provide a level of protection, though I'd probably do more than add a byte to the end. Perhaps run the whole stream through a reversible algorithm (e.g. a fast encryption or something).
Of course, this doesn't protect the client from any virus.
I'd like to understand what happen under the hood when you do an web upload.
I guess one of these:
The file is loaded in memory by the browser, sent to the web server buffer memory, and then the app is notified to collect it.
The file is being readed by the browser and at the same time sent to the web server, that can start to save the bytes progresively.
I've tried to upload a very large file, and put a breakpoint on the frist line of the method receiving the upload. I've seen how the browser toke a lot of time loading... but the breakpoint was still not hit, and after a while the breakpoint is hit.
I want to understand this, because in the worst scenario, if I allow big uploads, they could blow up the server memory at some point.
What does happen if I upload a 2Gb file? (considering that the web server/app accepts that length) would it take 2Gb of server memory?
Cheers.
The documentation for the HttpPostedFile class (which represents a file uploaded to the server in ASP.NET) specifies:
Files are uploaded in MIME
multipart/form-data format. By
default, all requests, including form
fields and uploaded files, larger than
256 KB are buffered to disk, rather
than held in server memory.
I need to export a large amount of data (~100mb) from a sql table to a user via the web. What would be the best solution for doing so? One thought was to export the data to a folder on the db server, compress it (by some means) and then provide a download link for the user. Any other methods for doing so? Also, can we compress data from within sql server?
Any approaches are welcome.
I wouldn't tie up the database waiting for the user to download 100Mb, even for a high speed user. When the user requests the file have them specify an email address. Then call an asynch process to pull the data, write it to a temp file (don't want > 100mb in memory after all), then zip the temp file to a storage location, then send the user an email with a link to download the file.
You can respond to a page request with a file:
Response.AddHeader("Content-Disposition",
"attachment; filename=yourfile.csv");
Response.ContentType = "text/plain";
Be sure to turn buffering off, so IIS can start sending the first part of the file while you are building the second:
Response.BufferOutput = false;
After that, you can start writing the file like:
Response.Write("field1,field2,field3\r\n");
When the file is completely written, end the response, so ASP.NET doesn't append a web page to your file:
Response.End();
This way, you don't have to write files on your web servers, you just create the files in memory and send them to your users.
If compression is required, you can write a ZIP file in the same way. This is a nice free library to create ZIP files.
Your approach works fine. SSIS + 7zip might be useful for automating the process if you need to do it more than a couple times.
If XML is OK, one approach would be to select the data "FOR XML" like this:
http://www.sqljunkies.ddj.com/Article/296D1B56-8BDD-4236-808F-E62CC1908C4E.scuk
And then spit out the raw XML directly to the browser as content-type: text/xml. Also be sure to set up Gzip compression on your web server for files with XML extensions. http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/502ef631-3695-4616-b268-cbe7cf1351ce.mspx?mfr=true
This will shrink the XML file down to 1/3 or maybe 1/4 the size as it's transferred. This wouldn't be the highest performance option because of the inherent wasted space in XML files, but a lot depends on what format you're looking for in the end.
Another option would be to use the free CSharpZipLib to compress the XML (or whatever format you want) into a zip file that the user would download. Along those lines, if this is something that will be used frequently you might want to look into caching and storing the zip file on the web server with some sort of expiration so it's not regenerated for every single request.
The download link is a perfectly valid and reasonable solution. Another would be to automatically redirect the user to that file so they didn't need to click a link. It really depends on your workflow and UI experience.
I would suggest against implementing compression in the SQL Server engine. Instead look at the DotNetZip library (Or System.IO.Conpression if you think your users have the capability of uncompressing gzip archives) and implement the compression within the web application.
What should I use for writing a file to the response? There are two different options as I see it. Option one is to read the file into a stream and then write the bytes to the browser with
Response.BinaryWrite(new bytes[5])
Next option is to just write the file from the file system directly with Response.WriteFile. Any advantages/disadvantages with either approach?
Edit: Corrected typos
Response.TransmitFile is preferred if you have the file on disk and are using at least asp.net 2.0.
Response.WriteFile reads the whole file into memory then writes the file to response. TransmitFile "Writes the specified file directly to an HTTP response output stream without buffering it in memory."
The other consideration is whether this is a file which is written one time or frequently. If you are frequently writing this file, then you might want to cache it, thus Response.BinaryWrite makes the most sense.
If you have it in memory, I would not write it to the file system and use Response.WriteFile.