I'd like to understand what happen under the hood when you do an web upload.
I guess one of these:
The file is loaded in memory by the browser, sent to the web server buffer memory, and then the app is notified to collect it.
The file is being readed by the browser and at the same time sent to the web server, that can start to save the bytes progresively.
I've tried to upload a very large file, and put a breakpoint on the frist line of the method receiving the upload. I've seen how the browser toke a lot of time loading... but the breakpoint was still not hit, and after a while the breakpoint is hit.
I want to understand this, because in the worst scenario, if I allow big uploads, they could blow up the server memory at some point.
What does happen if I upload a 2Gb file? (considering that the web server/app accepts that length) would it take 2Gb of server memory?
Cheers.
The documentation for the HttpPostedFile class (which represents a file uploaded to the server in ASP.NET) specifies:
Files are uploaded in MIME
multipart/form-data format. By
default, all requests, including form
fields and uploaded files, larger than
256 KB are buffered to disk, rather
than held in server memory.
Related
I have an ASP.NET web application and I want my users to be able to upload large files. However, some files are very large and uses too much memory.
In principle it should be possible to receive the request stream and write it directly to a FileWriter stream, removing any need to load the entire file into memory first.
I've tried accessing Request.InputStream and writing it directly to a file. It works, but a test using larger files reveal that Request.InputStream is only available after the entire request is already loaded into memory.
Can someone tell me an approach I can use to receive a normal Request.InputStream in ASP.NET and directly write it to a file without first loading it into memory?
Note, the file is sent through a normal request in a browser by posting a form with a file field.
(I actually use BlueImp JQuery File Upload but I don't think it's relevant to this question)
The process is called byte serving.
Byte Serving:
Byte serving is the process of sending only a portion of an HTTP/1.1 message from a server to a client. Byte serving begins when an HTTP server advertises its willingness to serve partial requests using the Accept-Ranges response header. A client then requests a specific part of a file from the server using the Range request header.
Is seems that IIS and ASP.NET are capable of handling Accept-Range headers. There is a Range Controller on Microsoft git repositories.
Here is an article that may be useful in configuring IIS to handle these requests.
Hi from long time i have doubt. when we use "http" protocol to download something the download starts from the first byte of the file. I mean if there is a file of 2MB on the site and when we click it, it starts downloading from the first byte. But when we give the link of the file to the download managers they work differently. I mean after downloading few bytes if we pause they stop downloading and when we resume they start from where they have stopped(not from the beginning). how is this possible?
The answer is the server setting. If a server allows the client to read the file from somewhere after the first byte, the client can specify the number of bytes to skip and the server will start sending the file from that position in the file. If the server doesn't allow then the client is forced to start reading the file from the beginning, whether any download manager is used or not.
For example 4shared.com always allows to start from beginning.
Note: In such cases using any download manager provides no gains.
It really depends on the server where file is hosted if it allows the byte-seeking. In other words, if a file hosting service has "streaming" feature than just "download" feature, applications like download managers will be able to pull a file in pieces & combine them after all the pieces have been downloaded.
My ASP.NET Application has an FileUpload control. My server doesn't have any antivirus program. If I add a byte to binary content of the file before saving file, does my server affect from virus? When displaying file, I will remove extra byte from the content.
Thanks for replies.
A virus will only cause you problems if it is run on the server (i.e. the file is opened). You can get around this by renaming all uploaded files with a .resources extension. All requests for this type of file are sent by IIS to ASP.NET, which rejects them. So effectively, the files store the data but can't be opened/run at all. Then you can still serve them back by reading their content in an ASP.NET page/module, and returning the data as a file with the correct extension.
Transforming the data as you suggest will also provide a level of protection, though I'd probably do more than add a byte to the end. Perhaps run the whole stream through a reversible algorithm (e.g. a fast encryption or something).
Of course, this doesn't protect the client from any virus.
Ok, so here's the problem: I'm reading the stream from a FileUpload control, reading in chunks of n bytes and writing the array in a loop until I reach the stream's end.
Now the reason I do this is because I need to check several things while the upload is still going on (rather than doing a Save(); which does the whole thing in one go). Here's the problem: when doing this from the local machine, I can see the file just fine as it's uploading and its size increases (had to add a Sleep(); clause in the loop to actually get to see the file being written).
However, when I upload the file from a remote machine, I don't get to see it until the the file has completed uploading. Also, I've added another call to write the progress to a text file as the progress is going on, and I get the same thing. Local: the file updates as the upload goes on, remote: the token file only appears after the upload's done (which is somewhat useless since I need it while the upload's still happening).
Is there some sort of security setting in (or ASP.net) that maybe saves files in a temporary location for remote machines as opposed to the local machine and then moves them to the specified destination? I would liken this with ASP.net displaying error messages when browsing from the local machine (even on the public hostname) as opposed to the generic compilation error page/generic exception page that is shown when browsing from a remote machine (and customErrors are not off)
Any clues on this?
Thanks in advance.
FileUpload control renders as an <input type="file"> HTML element; this way, your browser will open that file, read ALL content, encode and send it.
Your ASP.NET request just starts after IIS receives all browser data.
This way, you'll need to code a client component (Flash, Java applet, Silverlight) to send a file in small chunks and rebuild that at server-side.
EDIT: Some information on MSDN:
To control whether the file to upload is temporarily stored in memory or on the server while the request is being processed, set the requestLengthDiskThreshold attribute of the httpRuntime element. This attribute enables you to manage the size of the input stream buffer. The default is 256 bytes. The value that you specify should not exceed the value that you specify for the maxRequestLength attribute.
I understand that you want to check the file which is being uploaded for it's content.
If this is your requirement then why not add a textbox and populate it while you are reading the file from HttpPostedFile.
When is PostedFile.InputStream available when uploading a large file?
I'd like to pass a Stream to another process and I'm hoping that if a large file was being uploaded that I can pass the Stream straight to that new process w/o writing to the file system. Since the process and/or upload could take a while, I'm wondering if I can start reading the InputStream immediately or whether I have to wait for the whole file to be transferred to the server before it can be processed.
I guess a more general question is - what's the lifecycle of a POST request when file upload is involved?
The PostedFile.InputStream isn't available until the entire file has been uploaded. IIS6 caches the file in memory while IIS7 now caches the file to disk before handing off the input stream to your method.
You can get a HttpModule such as NeatUpload which allows you access to the bits while they're uploading.