Getting ERR_CONNECTION_RESET after more or less 2 minutes of uploading a rather big file (90 MB) through asp.net fileupload control.
Testing this on a shared host environment so I don't know exactly what policies are enforced on me.
web.config settings are sufficient I think:
<httpRuntime requestValidationMode="2.0" maxRequestLength="1000000" executionTimeout="45000" />
Should I be looking at other async file-upload controls? Am I missing a web.config setting? Is the upload control simply not sufficient for large files on slow connections?
There can be a number of reasons why the connection is reset and upping the max request length works to a point but you are right about looking into async file uploaders. The most important part is using one which "chunks" the files into smaller pieces and so avoids request limits etc. I have had the best experience with plupload:
http://www.plupload.com/
Here is some code for receiving the files (this is MVC but you can refactor to use a handler in .NET classic):
[HttpPost]
public ActionResult UploadImage(int? chunk, int? chunks, string name)
{
var fileData = Request.Files[0];
if (fileData != null && fileData.ContentLength > 0)
{
var path = GetTempImagePath(name);
fileSystem.EnsureDirectoryExistsForFile(path);
// Create or append the current chunk of file.
using (var fs = new FileStream(path, chunk == 0 ? FileMode.Create : FileMode.Append))
{
var buffer = new byte[fileData.InputStream.Length];
fileData.InputStream.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, buffer.Length);
}
}
return Content("Chunk uploaded", "text/plain");
}
Related
I am trying to stream on the fly zipped files but memory consumption is high. For example, to zip total file size of 2.8 GB is taking nearly 5 GB of processor memory.
[Route("zip")]
public class ZipController : ControllerBase
{
private readonly HttpClient _httpClient;
public ZipController()
{
_httpClient = new HttpClient();
}
[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{
Response.ContentType = "application/octet-stream";
Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");
using var zipArchive =
new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
foreach (var (key, value) in input.FilePathsToUrls)
{
var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
await using var zipStream = zipEntry.Open();
await using var stream = await _httpClient.GetStreamAsync(value);
await stream.CopyToAsync(zipStream);
}
}
}
I believe you should be able to call Response.StartAsync:
[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{
Response.ContentType = "application/octet-stream";
Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");
await Response.StartAsync();
using var zipArchive = new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
foreach (var (key, value) in input.FilePathsToUrls)
{
var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
await using var zipStream = zipEntry.Open();
await using var stream = await _httpClient.GetStreamAsync(value);
await stream.CopyToAsync(zipStream);
}
}
StartAsync should start the response being sent. Note that neither the response headers nor the status code can be modified once StartAsync is called.
In particular, this means that your exception handling will be different. Previously, an exception (e.g., from a bad URL in the request) would cause an exceptional status code (i.e., 500). With a streaming response, any exceptions after StartAsync cannot change the status code; it's already been sent. Instead, it will appear to the client as though the connection was terminated without a clean close. Complicating this a bit further, this behavior is not uncommon for web servers to do in the successful case, so clients may not complain - they would just end up with truncated (invalid) zip files. (In the case of streaming zips, the "file table" in the zip is sent last instead of first).
So, this should work, but I also recommend:
Ensure your exception logging works for exceptions after StartAsync. There is no way to return error details to the client, so you must rely on logging.
If you control the client, test out this new error situation, and see if you can detect it. If it's not detectable using that client, then ensure your code validates the zip.
Nothing about the zip file format should require a large amount of memory for this use case. It's essential all the files in order, with a table at the end describing the zip structure, and file offsets. This makes it possible to stream very efficiently without using much memory at all.
You may not need to write this yourself, ZipStreamer is a micro service you host that does exactly this (disclosure, I'm the author). It's designed to solve the exact problems you are hitting by streaming the bytes out as soon as they come in, with a fixed buffer size to prevent blowing up memory. It can stream hundreds of zips files in parallel using only a few MB of memory.
If you need this to be part of your application, here are some suggestions.
Disable compression will save CPU, and a bit of memory. Depending on your files, compression might not be a major benefit (jpegs actually get bigger after zip compression). If you're zipping just to combine many files into one, this will really help. But this doesn't explain using GB of memory.
Ensure you're not holding the stream content any longer than you need to, it looks like you are. Start streaming back asap as #Stephen suggested with StartAsync.
I have a web site running in its own Application Pool (IIS 8). Settings for the pool are default i.e. recycle every 29 hours.
Our web server only has 8gb RAM and I have noticed that the worker process for this web site regularly climbs to 6gb RAM and slows the server to a crawl. This is the only site currently on the web server.
I also have SQL Express 2016 installed as well. The site is using EF version 6.1.3.
The MVC site is very straightforward. It has a GETPDF controller which finds a row in a table, gets PDF info stored in a field then serves it back to the browser as follows :-
using (eBillingEntities db = new eBillingEntities())
{
try
{
string id = model.id;
string emailaddress = Server.HtmlEncode(model.EmailAddress).ToLower().Trim();
eBillData ebill = db.eBillDatas.ToList<eBillData>().Where(e => e.PURL == id && e.EmailAddress.ToLower().Trim() == emailaddress).FirstOrDefault<eBillData>();
if (ebill != null)
{
// update the 'Lastdownloaded' field.
ebill.LastDownloaded = DateTime.Now;
db.eBillDatas.Attach(ebill);
var entry = db.Entry(ebill);
entry.Property(en => en.LastDownloaded).IsModified = true;
db.SaveChanges();
// Find out from the config record whether the bill is stored in the table or in the local pdf folder.
//
Config cfg = db.Configs.ToList<Config>().Where(c => c.Account == ebill.Account).FirstOrDefault<Config>();
bool storePDFDataInEBillTable = true;
if (cfg != null)
{
storePDFDataInEBillTable = cfg.StorePDFDataInEBillDataTable;
}
// End of Modification
byte[] file;
if (storePDFDataInEBillTable)
{
file = ebill.PDFData;
}
else
{
string pathToFile = "";
if (string.IsNullOrEmpty(cfg.LocalPDFDataFolder))
pathToFile = cfg.LocalBackupFolder;
else
pathToFile = cfg.LocalPDFDataFolder;
if (!pathToFile.EndsWith(#"\"))
pathToFile += #"\";
pathToFile += ebill.PDFFileName;
file = System.IO.File.ReadAllBytes(pathToFile);
}
MemoryStream output = new MemoryStream();
output.Write(file, 0, file.Length);
output.Position = 0;
HttpContext.Response.AddHeader("content-disposition", "attachment; filename=ebill.pdf");
return new FileStreamResult(output, "application/pdf");
}
else
return View("PDFNotFound");
}
catch
{
return View("PDFNotFound");
}
Are there any memory leaks here?
Will the file byte array and the memory stream get freed up?
Also, is there anything else I need to do concerning clearing up the entity framework references?
If the code looks OK, where would be a good place to start looking?
Regards
Are there any memory leaks here?
No.
Will the file byte array and the memory stream get freed up?
Eventually, yes. But that may be the cause of your excessive memory use.
Also, is there anything else I need to do concerning clearing up the entity framework references?
No.
If the code looks OK, where would be a good place to start looking?
If this code is the cause of your high memory use, it's because you are loading files into memory. And you're loading two copies of each file in memory, once in a byte[] and copying to a MemoryStream.
There's no need to do that.
To eliminate the second copy of the file use the MemoryStream(byte[]) constructor instead of copying the bytes from the byte[] to an empty MemoryStream.
To eliminate the first copy in memory, you can stream the data into a temporary file that will be the target of your FileStreamResult, or initialize the FileStreamResult using a ADO.NET stream.
See https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sqlclient-streaming-support
If you go to ADO.NET streaming your DbContext, will need to be scoped to your Controller, instead of a local variable, which is a good practice in any case.
In addition to David's advice. I noticed that I was doing the following
**db.eBillDatas.ToList<eBillData>()**
therefore I was getting all the data from the database then fetching it again with the where clause.
I didn't notice the problem until the database started to fill up.
I removed that part and now the IIS worker processing is about 100mb.
I'm trying to upload a large file to a document library, but it fails after just a few seconds. The upload single document fails silently, upload multiple just shows a failed message. I've turned up the file size limit on the web application to 500MB, and the IIS request length to the same (from this blog), and increased the IIS timeout for good measure. Are there any other size caps that I've missed?
Update I've tried a few files of various sizes, anything 50MB or over fails, so I assume something somewhere is still set to the webapp default.
Update 2 Just tried uploading using the following powershell:
$web = Get-SPWeb http://{site address}
$folder = $web.GetFolder("Site Documents")
$file = Get-Item "C:\mydoc.txt" // ~ 150MB
$folder.Files.Add("SiteDocuments/mydoc.txt", $file.OpenRead(), $false)
and get this exception:
Exception calling "Add" with "3" argument(s): "<nativehr>0x80070003</nativehr><nativestack></nativestack>There is no file with URL 'http://{site address}/SiteDocuments/mydoc.txt' in this Web."
which strikes me as odd as of course the file wouldn't exist until it's been uploaded? N.B. while the document library has the name Site Documents, it has the URL SiteDocuments. Not sure why...
Are you sure you updated the right webapp? Is the filetype blocked by the server? Is there adequate space in your content database? I would check ULS logs after that and see if there is another error since it seems you hit the 3 spots you would need too update.
for uploading a large file, you can use the PUT method instead of using the other ways to upload a document.
by using a put method you will save the file into content database directly. see the example below
Note: the disadvantage of the code below is you cannot catch the object that is responsible for uploading directly, on other word, you cannot update the additional custom properties of the uploaded document directly.
public static bool UploadFileToDocumentLibrary(string sourceFilePath, string targetDocumentLibraryPath)
{
//Flag to indicate whether file was uploaded successfuly or not
bool isUploaded = true;
try
{
// Create a PUT Web request to upload the file.
WebRequest request = WebRequest.Create(targetDocumentLibraryPath);
//Set credentials of the current security context
request.Credentials = CredentialCache.DefaultCredentials;
request.Method = “PUT”;
// Create buffer to transfer file
byte[] fileBuffer = new byte[1024];
// Write the contents of the local file to the request stream.
using (Stream stream = request.GetRequestStream())
{
//Load the content from local file to stream
using (FileStream fsWorkbook = File.Open(sourceFilePath, FileMode.Open, FileAccess.Read))
{
//Get the start point
int startBuffer = fsWorkbook.Read(fileBuffer, 0, fileBuffer.Length);
for (int i = startBuffer; i > 0; i = fsWorkbook.Read(fileBuffer, 0, fileBuffer.Length))
{
stream.Write(fileBuffer, 0, i);
}
}
}
// Perform the PUT request
WebResponse response = request.GetResponse();
//Close response
response.Close();
}
catch (Exception ex)
{
//Set the flag to indiacte failure in uploading
isUploaded = false;
}
//Return the final upload status
return isUploaded;
}
and here are an example of calling this method
UploadFileToDocumentLibrary(#”C:\test.txt”, #”http://home-vs/Shared Documents/textfile.pdf”);
I'm working on a ASP.NET (3.5) website that contains a treeview; the nodes' values are set to a filepath (on the server) of a PDF file. When the user clicks a treenode, the server-side code gets the node value (file path), creates a FileInfo object, and (after setting all the header Content-type, Cache-Control, etc. correctly) calls Response.TransmitFile(xxxxpath) to send to the user's browser.
This works fine on the major browsers, on major devices (PCs, Macs, iOS devices). The file downloads correctly and opens on the user's machine. But on certain devices and certain browsers, the PDF file does not open. On Android devices, it appears that Firefox downloads and opens the PDFs correctly, but the stock Android browser does not. On a Kindle Fire, it appears the Silk browser downloads the file successfully, but when trying to open it, I see an error: "PDF trailer not found"....or it says the PDF is DRM-protected (which it is not). I haven't tried another browser (if there is one) on the Fire.
I've experimented using anchor links in static HTML markup, and the problem browsers appear to download and display the PDFs correctly when accessed this way. There seems to be an issue (inconsistency?) with the way ASP.NET sends the response to the browser when done via code. I've used Response.Flush, Response.WriteFile, Response.BinaryWrite, Response.Close, Response.End, etc., and they all produce the same result: MOST browsers handle the file, but SOME cannot handle the PDF.
So, is there some issue with the way ASP.NET constructs the Response object (especially when sending back PDFs) that some browsers don't like? TIA.
Quite simply, the answer to your question is "No." You may want to post your code if you have doubts about whether or not you're doing it correctly; otherwise: 'no'. :)
I would think the browsers you mentioned are much more suspect than something as simple and established as writing to the Response stream.
For reference here is a tried and true way I do it using iHttpHandler:
public void ProcessRequest(System.Web.HttpContext context)
{
string sFilePath = context.Server.MapPath(String.Concat("~/App_LocalResources", context.Request.QueryString["path"]));
try
{
context.Response.Clear();
context.Response.ContentType = "application/octet-stream";
int iReadLength = 16384;
int iLastReadLength = iReadLength;
byte[] buffer = new byte[iReadLength];
if (System.IO.File.Exists(sFilePath))
{
System.IO.FileInfo fInfo = new System.IO.FileInfo(sFilePath);
context.Response.AddHeader("Content-Length", fInfo.Length.ToString());
context.Response.AddHeader("Content-Disposition", String.Format("attachment; filename=\"{0}\"", fInfo.Name));
using (System.IO.FileStream input = new System.IO.FileStream(sFilePath, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{
try
{
while (iLastReadLength > 0 && context.Response.IsClientConnected)
{
iLastReadLength = input.Read(buffer, 0, iLastReadLength);
if (iLastReadLength > 0)
{
context.Response.OutputStream.Write(buffer, 0, iLastReadLength);
context.Response.OutputStream.Flush();
}
}
context.Response.Flush();
}
catch
{
}
finally
{
input.Close();
}
}
}
}
catch
{
}
finally
{
}
}
Since you've indicated you are pulling the file from another place, here is how to write that to a memory stream. Just pull from your Response Stream from the remote server (or File Stream, Sql Binary Reader, w/e really), then reset your MemoryStream position and then use the functionality above to write to your Response stream to the client.
int iReadLength = 16384;
long lLastReadLength = iReadLength;
long lDataIndex = 0;
byte[] buffer = new byte[iReadLength];
using (System.IO.MemoryStream msTemp = new System.IO.MemoryStream())
{
while (lLastReadLength > 0)
{
lLastReadLength = reader.GetBytes(0, lDataIndex, buffer, 0, iReadLength);
lDataIndex += lLastReadLength;
if (lLastReadLength > 0)
{
msTemp.Write(buffer, 0, Convert.ToInt32(lLastReadLength));
msTemp.Flush();
}
}
// Reset Memory Position
msTemp.Position = 0;
// Now write to the Response Stream here
}
What are the latest and greatest ways to compress the ASP.NET ViewState content?
What about the performance of this? Is it worth it to keep the pages quick and minimize data-traffic?
How can I make:
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE"
value="/wEPDwUKMTM4Mjc3NDEyOWQYAQUeX19Db250cm9sc1JlcXVpcmVQb3N0QmFja0tleV9fFgkFLGN0b
DAwJENvbnRlbnRQbGFjZUhvbGRlcl9NYWluQ29udGVudCRSYWRCdXQxBSxjdGwwMCRDb250ZW50UGxhY2VIb
2xkZXJfTWFpbkNvbnRlbnQkUmFkQnV0MQUsY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyX01haW5Db250ZW50J
FJhZEJ1dDIFLGN0bDAwJENvbnRlbnRQbGFjZUhvbGRlcl9NYWluQ29udGVudCRSYWRCdXQyBSxjdGwwMCRDb
250ZW50UGxhY2VIb2xkZXJfTWFpbkNvbnRlbnQkUmFkQnV0MwUsY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyX
01haW5Db250ZW50JFJhZEJ1dDQFLGN0bDAwJENvbnRlbnRQbGFjZUhvbGRlcl9NYWluQ29udGVudCRSYWRCd
XQ0BSxjdGwwMCRDb250ZW50UGxhY2VIb2xkZXJfTWFpbkNvbnRlbnQkUmFkQnV0NQUsY3RsMDAkQ29udGVud
FBsYWNlSG9sZGVyX01haW5Db250ZW50JFJhZEJ1dDXz21BS0eJ7991pzjjj4VXbs2fGBw==" />
Into sometning like this:
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE"
value="/wEPDwUKMTM4Mjc3N==" />
The simple answer might not be what you want to hear. Too often, controls on the page have viewstate by default when they really don't need it. It's a good idea to switch off viewstate until you know you're going to need it, and only switch it on for the (hopefully) few cases where you actually want to keep the view state.
Avoid using ViewState
Use compression on the IIS server.
You can wireup something that will compress the viewstate into and out of a page by doing something like:
public abstract class PageBase : System.Web.UI.Page
{
private ObjectStateFormatter _formatter = new ObjectStateFormatter();
private static byte[] Compress( byte[] data )
{
var compressedData = new MemoryStream();
var compressStream = new GZipStream(output, CompressionMode.Compress, true);
compressStream.Write(data, 0, data.Length);
compressStream.Close();
return compressedData.ToArray();
}
private static byte[] Uncompress( byte[] data )
{
var compressedData = new MemoryStream();
input.Write(compressedData, 0, compressedData.Length);
input.Position = 0;
var compressStream = new GZipStream(compressedData, CompressionMode.Decompress, true);
var uncompressedData = new MemoryStream();
var buffer = new byte[64];
var read = compressStream.Read(buffer, 0, buffer.Length);
while (read > 0)
{
uncompressedData.Write(buffer, 0, read);
read = compressStream.Read(buffer, 0, buffer.Length);
}
compressStream.Close();
return uncompressedData.ToArray();
}
protected override void SavePageStateToPersistenceMedium(object viewState)
{
var ms = new MemoryStream();
_formatter.Serialize(ms, viewState);
var viewStateBytes = ms.ToArray();
ClientScript.RegisterHiddenField("__COMPRESSED_VIEWSTATE"
, Convert.ToBase64String( Compress(viewStateArray)) );
}
protected override object LoadPageStateFromPersistenceMedium()
{
var compressedViewState = Request.Form["__COMPRESSED_VIEWSTATE"];
var bytes = Uncompress( Convert.FromBase64String( compressedViewState ) );
return _formatter.Deserialize( Convert.ToBase64String( bytes ) );
}
}
I realize this is an old thread, but we have been using Telerik's RadCompression HttpModule for a while now and it works incredibly well at compressing ViewState, AJAX and Web Service responses. You can also cheat and save ViewState in session - good for low traffic sites.
http://www.telerik.com/help/aspnet-ajax/radcompression.html
Again, after some research into this I summarized my findings in a blog-post about Compressing View State.
To save a compressed View State, this is what I did:
protected override void SavePageStateToPersistenceMedium(object state) {
SaveCompressedPageState(state);
}
private void SaveCompressedPageState(object state) {
byte[] viewStateBytes;
using(MemoryStream stream = new MemoryStream()) {
ObjectStateFormatter formatter = new ObjectStateFormatter();
formatter.Serialize(stream, state);
viewStateBytes = stream.ToArray();
}
byte[] compressed = CompressionHelper.Compress(viewStateBytes);
string compressedBase64 = Convert.ToBase64String(compressed);
ClientScript.RegisterHiddenField(ViewStateFieldName, compressedBase64);
}
And for the loading-part, this code made it work for me:
protected override object LoadPageStateFromPersistenceMedium() {
return LoadCompressedPageState();
}
private object LoadCompressedPageState() {
string viewState = Request.Form[ViewStateFieldName];
if(string.IsNullOrEmpty(viewState)) {
return string.Empty;
}
byte[] decompressed = CompressionHelper.Decompress(viewState);
string decompressedBase64 = Convert.ToBase64String(decompressed);
ObjectStateFormatter formatter = new ObjectStateFormatter();
return formatter.Deserialize(decompressedBase64);
}
Seb, ViewState is already compressed... that is what you are seeing... a compressed version of your controls. If you want less overhead, then don't use viewstate :)
Viewstate use should be kept to a minimum!
This is an XML-lized visualization of your posted viewstate:
<viewstate>
<Pair>
<Pair>
<String>1382774129</String>
</Pair>
</Pair>
</viewstate>
<controlstate>
<HybridDictionary>
<DictionaryEntry>
<String>__ControlsRequirePostBackKey__</String>
<ArrayList>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut1</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut1</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut2</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut2</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut3</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut4</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut4</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut5</String>
<String>ctl00$ContentPlaceHolder_MainContent$RadBut5</String>
</ArrayList>
</DictionaryEntry>
</HybridDictionary>
</controlstate>
Basically just a few radiobuttons which like to know of their existance. (browsers don't send an <input type="radio"> field with the postdata if it is not checked). This is pretty minimal already.
It can likely be compressed by hooking in the load/save methods or HTTP modules, but this may not be really practical nor really needed.
In case the viewstate is much bigger in your real app, avoid getting objects in the viewstate at all. This can be achieved by initializing the controls in the OnInit() or Page_Init() methods instead of the default Page_Load().
The rationale behind this can be found at
http://weblogs.asp.net/infinitiesloop/archive/2006/08/03/Truly-Understanding-Viewstate.aspx
and http://msdn.microsoft.com/en-us/library/ms972976.aspx
A quick summary:
ViewState is just the backing store for almost all control properties, including defaults.
After the defaults are set by OnInit(), the TrackViewState() method will is called.
Any subsequent changes (e.g. by Page_Load()) or an eventhandler, will be tracked and submitted to the client. This way those controls can restore their state at the next request.
Instead of relying at the framework to restore objects, restore objects in OnInit() when needed. (e.g. repopulating the options of a DropDownList from the database).
One exception:
If a control is dynamically added to the control tree, it plays a catch-up. Their OnInit() method may run at a later moment, causing those properties to end up in the viewstate after all. If the initialization of the control can't happen in OnInit(), setting EnableViewState="false" can be used as workaround.
Each time my viewstate grows unexpectedly, I'm using the "ViewState Decoder 2.2" app to find out what ended up in the viewstate. Often, it's not needed for the data to be there.
And a final word:
The viewstate is not used for repopulating forms!!
Those values are already submitted with the postdata.
Compressing view state fails in certain cases:
- If you are using update panel on page don’t use compression mode.
- If somehow you are changing the view state in result of ICallBack code don’t use compression mode, as this will don’t reflect the correct view state on post back.
The best way to minimize the view state is just to not use it. It will cause you to do some extra work programming (repopulating control values etc on post back, but it will save you on the amount of information you send to the browser). You can't tamper with it.
Here is a link to the view state on MSDN:
http://msdn.microsoft.com/en-us/library/ms972976.aspx
Here is a link describing some best practices:
http://mnairooz.blogspot.com/2007/01/aspnet-20-viewstate-and-good-practices.html
And One on disabling the ViewState:
http://www.codeproject.com/KB/aspnet/ASPNET_Best_Practices.aspx