I have implemented following async blob upload method to upload multiple blocks.
var container = GetContainer(containerName);
var blob = container.GetBlockBlobReference(blobName);
String[] base64EncodedBlockIds = new String[10];// 10- number of Blocks
//To upload the blocks in parallel - 10 parallel blocks
ParallelLoopResult parallelLoopResult = Parallel.For(0,10, i =>
{
String base64EncodedBlockId = Convert.ToBase64String(System.BitConverter.GetBytes(i));
byte[] bytesMemoryStream = GetBytesFromStream(stream);
using (MemoryStream memoryStream = new MemoryStream(bytesMemoryStream))
{
blob.PutBlock(base64EncodedBlockId, memoryStream, null);// throws an exception "The value for one of the HTTP headers is not in the correct format"
}
base64EncodedBlockIds[i] = base64EncodedBlockId;
});
blob.PutBlockList(base64EncodedBlockIds);
It throws an exception "The value for one of the HTTP headers is not in the correct format".
Need your inputs
Regards,
Vivek
BlockIDs within a blob must all be the same length (number of characters). BlockID "10" is longer than the others, which is probably the source of your problem.
One solution would be to zero-pad the BlockIDs to the same length.
In my case "The value for one of the HTTP headers is not in the correct format" error occurred because I was trying to write an empty block (memoryStream had 0 bytes). PutBlock failed beause the content length was 0 in the header.
Related
I am trying to stream on the fly zipped files but memory consumption is high. For example, to zip total file size of 2.8 GB is taking nearly 5 GB of processor memory.
[Route("zip")]
public class ZipController : ControllerBase
{
private readonly HttpClient _httpClient;
public ZipController()
{
_httpClient = new HttpClient();
}
[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{
Response.ContentType = "application/octet-stream";
Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");
using var zipArchive =
new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
foreach (var (key, value) in input.FilePathsToUrls)
{
var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
await using var zipStream = zipEntry.Open();
await using var stream = await _httpClient.GetStreamAsync(value);
await stream.CopyToAsync(zipStream);
}
}
}
I believe you should be able to call Response.StartAsync:
[HttpPost]
public async Task Zip([FromBody] JsonToZipInput input)
{
Response.ContentType = "application/octet-stream";
Response.Headers.Add($"Content-Disposition", $"attachment; filename=\"{input.FileName}\"");
await Response.StartAsync();
using var zipArchive = new ZipArchive(Response.BodyWriter.AsStream(), ZipArchiveMode.Create);
foreach (var (key, value) in input.FilePathsToUrls)
{
var zipEntry = zipArchive.CreateEntry(key, CompressionLevel.Optimal);
await using var zipStream = zipEntry.Open();
await using var stream = await _httpClient.GetStreamAsync(value);
await stream.CopyToAsync(zipStream);
}
}
StartAsync should start the response being sent. Note that neither the response headers nor the status code can be modified once StartAsync is called.
In particular, this means that your exception handling will be different. Previously, an exception (e.g., from a bad URL in the request) would cause an exceptional status code (i.e., 500). With a streaming response, any exceptions after StartAsync cannot change the status code; it's already been sent. Instead, it will appear to the client as though the connection was terminated without a clean close. Complicating this a bit further, this behavior is not uncommon for web servers to do in the successful case, so clients may not complain - they would just end up with truncated (invalid) zip files. (In the case of streaming zips, the "file table" in the zip is sent last instead of first).
So, this should work, but I also recommend:
Ensure your exception logging works for exceptions after StartAsync. There is no way to return error details to the client, so you must rely on logging.
If you control the client, test out this new error situation, and see if you can detect it. If it's not detectable using that client, then ensure your code validates the zip.
Nothing about the zip file format should require a large amount of memory for this use case. It's essential all the files in order, with a table at the end describing the zip structure, and file offsets. This makes it possible to stream very efficiently without using much memory at all.
You may not need to write this yourself, ZipStreamer is a micro service you host that does exactly this (disclosure, I'm the author). It's designed to solve the exact problems you are hitting by streaming the bytes out as soon as they come in, with a fixed buffer size to prevent blowing up memory. It can stream hundreds of zips files in parallel using only a few MB of memory.
If you need this to be part of your application, here are some suggestions.
Disable compression will save CPU, and a bit of memory. Depending on your files, compression might not be a major benefit (jpegs actually get bigger after zip compression). If you're zipping just to combine many files into one, this will really help. But this doesn't explain using GB of memory.
Ensure you're not holding the stream content any longer than you need to, it looks like you are. Start streaming back asap as #Stephen suggested with StartAsync.
I need to extract uploads from http-trafic. How could do that? First of all, the request-method will be POST. Secondly, there will be a Content-Type header-field. I do not want to extract formular-data, but uploads like mail-attachements.
The content type is per specification multipart/form-data.
This is a special content type which can be visualized as multiple sub-requests in one big request. Each of those sub-requests (one form-data element) has their own set of headers. The content type of the actual data is in there.
Here's an example how it look like with 1 normal field and 1 file field (in HTML terms, when using <input name="textfield"><input type="file" name="filefield">):
Content-Type: multipart/form-data;boundary=SOME_BOUNDARY
--SOME_BOUNDARY
content-disposition: form-data;name="textfield"
content-type: text/plain;charset=UTF-8
value of textfield here
--SOME_BOUNDARY
content-disposition: form-data;name="filefield";filename="some.ext"
content-type: application/octet-stream
binary file content here
--SOME_BOUNDARY--
As to parsing and extracting this data, practically every programming language has builtin/3rd party APIs for this. As you didn't tell anything about which one you're using, it's impossible to give a targeted answer. In case of for example Java, that would be either the 3rd party library Apache Commons FileUpload or when you're using Servlet 3.0, the API-provided request.getPart() method.
If (and I by no means am saying this is the correct way) you just want to save data from a byte array, you should look at how to read the POST body at:
Reading POST body with bottle.py
Reading the data and then creating a new file should do the trick.
Based on #BalusC s solution I made a little extension method for .NET's build in WebClient class which does not support Multipart upload out of the box.
Usage
Just mix string values and files (enclosed in #)
public void UploadMultipart()
{
var fileName = "/some/existing/file.ext";
using (var client = new WebClient())
{
var values = new NameValueCollection();
values.Add("id", Guid.NewGuid().ToString());
values.Add("name", Path.GetFileNameWithoutExtension(fileName));
values.Add("file", $"#{fileName}#");
var result = client.UploadMultipart(address, method, values);
var content = client.Encoding.GetString(result);
}
}
Extension method
public static byte[] UploadMultipart(this WebClient client,
string address, string method, NameValueCollection values)
{
string boundary = DateTime.Now.Ticks.ToString("x");
client.Headers.Add("Content-Type", "multipart/form-data; boundary=" + boundary);
var sb = new StringBuilder()
.AppendLine();
foreach (var key in values.AllKeys)
{
var contentDispositon = $"form-data;name=\"{key}\"";
var contentType = $"text/plain;charset={client.Encoding.WebName}";
var value = values[key];
if (value.StartsWith("#") && value.EndsWith("#"))
{
// if a value is enclosed in hashes we expect this to be a path to a file
// file=#/path/to/file.ext#
var fileName = value.Trim('#');
var file = File.ReadAllBytes(fileName);
value = client.Encoding.GetString(file);
contentType = "application/octet-stream";
contentDispositon = $"form-data;name=\"{key}\"filename=\"{Path.GetFileName(fileName)}\"";
}
sb.AppendLine($"--{boundary}")
.AppendLine($"Content-Disposition: {contentDispositon}")
.AppendLine($"Content-Type: {contentType}")
.AppendLine()
.AppendLine(value);
}
sb.AppendLine($"--{boundary}--");
var data = client.Encoding.GetBytes(sb.ToString());
return client.UploadData(address, method, data);
}
I generate request to a certain API which is mentioned here for sending an sms http://help.voxeo.com/go/help/evolution.sms.postapi
The request generates an XML response as follow
<rsp stat="ok">
<success msg="accepted" transactionid="2e47fe224d25559a696a7bdddec1828b" messageid="cf0d21f067e5b386a2e042134687eb5c"/>
</rsp>
I want to read if rsp stat in response is ok or fail how can i do it .
These are the first two lines how can i get particular xml tag out of response stream
HttpWebResponse response = (HttpWebResponse)myReq.GetResponse();
Stream content = response.GetResponseStream();
Why not use XmlDocument to parse the XML. For example,
using(var reader = new StreamReader(content))
{
var doc = new XmlDocument();
doc.LoadXml(reader.ReadToEnd());
// you may want to compare case in-sensitive
if (doc.DocumentElement.Attributes["stat"].Value == "ok")
{
// success
}
}
(There is also Load method that would load from stream directly but I am not sure if it expects xml declaration at the beginning or not)
Yet another alternative is using XmlReader in case response could be lengthy and you wants to parse the initial bits as soon as its available.
try to read with XmlTextReader (http://msdn.microsoft.com/en-us/library/system.xml.xmltextreader%28v=vs.71%29.aspx)
I have a controller action declared as follows:
[Authorize(Order = 0, Roles = "Requester,Controller,Installer")]
public FileStreamResult ExportJobCards()
The body of this method builds a collection of CSV lines, and attempts to return them as a file as follows:
using (var sw = new StreamWriter(new MemoryStream()))
{
foreach (var line in lines)
{
sw.WriteLine(line);
}
return new FileStreamResult(sw.BaseStream, "text/csv");
}
When I request this action using the following action link...
Html.ActionLink("Export to Excel", "ExportJobCards")
...the export method executes properly, i.e. all the required CSV data is present in the lines collection in the above code, but I get a File Not Found error rendered as the end result.
EDIT:
In agreement with Tommy's observation, I moved the return out of the using, and I now get a file, but the file is empty. The new code that actually produces a file, ableit empty, is:
var sw = new StreamWriter(new MemoryStream());
foreach (var line in lines)
{
sw.WriteLine(line);
}
sw.Flush();
return new FileStreamResult(sw.BaseStream, "text/csv");
With your current setup, the Using statement is disposing of the StringWriter before the return can complete, which is resulting in the null reference/file not found error. Remove the using statement or set the StringWriter to another variable before you exit out and you should be good to go on getting rid of the File Not Found error.
A thought on your second issue now, looking into memorystreams as filestream results, you may need to change your return to this
sw.BaseStream.seek(0, SeekOrigin.Begin)
return new FileStreamResult(sw.BaseStream, "text/csv");
as the pointer is still at the end of the stream when you return.
It throws that error because you're not giving it a file stream. What you want is the FileContentResult into which you can pass arbitrary content. This content needs to be a byte array of your content, probably easiest to:
use a stringbuilder rather than a streamwriter
get your string from the builder
use the static method System.Text.UnicodeEncoding.Unicode.GetBytes(string) to get the byte array
Give the byte array to FileContentResult
As you have to write this code anyway the easiest thing to do would be to create a new FileStringResult that inherits from the base FileResult that can take in a string or stringbuilder. Override WriteFile(HttpResponseBase response) to do the string to byte[] conversion and push that into the response. Take a look at the FileStreamResult class from the MVC sources, it's very small and easy to do.
I need to load an external web (not local) page into my site (some link), but only a part of it.
What are the options for doing so?
That depends on whether or not the external page is local, or on a different domain. If it's local, you can use $.load() in the jQuery library. This has an optional parameter to specify which element in the remote-dom to load it:
$("#links").load("/Main_Page #jq-p-Getting-Started li");
If the page is on another domain, you'll need a proxy script. You can do this with PHP and the phpQuery (php port of jQuery) library. You'll just use file_get_contents() to get the actual remote-dom, and then pull out the elements you want based on jQuery-like selectors.
$f = fopen('http://www.quran.az/2/255', 'r');
and so on...
Once you get the whole page as Michael Todd outlined, you will likely need to either use substring methods for a static means to slice up the content or you can use regex's for a more dynamic way to grab the content. An intro article on Regex's in ASP.Net can be found here. Good luck!
To load a web page in .Net, use the HttpWebRequest class.
Example taken from MSDN, here:
private string StringGetWebPage(String uri)
{
const int bufSizeMax = 65536; // max read buffer size conserves memory
const int bufSizeMin = 8192; // min size prevents numerous small reads
StringBuilder sb;
// A WebException is thrown if HTTP request fails
try
{
// Create an HttpWebRequest using WebRequest.Create (see .NET docs)!
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
// Execute the request and obtain the response stream
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
// Content-Length header is not trustable, but makes a good hint.
// Responses longer than int size will throw an exception here!
int length = (int)response.ContentLength;
// Use Content-Length if between bufSizeMax and bufSizeMin
int bufSize = bufSizeMin;
if (length > bufSize)
bufSize = length > bufSizeMax ? bufSizeMax : length;
// Allocate buffer and StringBuilder for reading response
byte[] buf = new byte[bufSize];
sb = new StringBuilder(bufSize);
// Read response stream until end
while ((length = responseStream.Read(buf, 0, buf.Length)) != 0)
sb.Append(Encoding.UTF8.GetString(buf, 0, length));
}
catch (Exception ex)
{
sb = new StringBuilder(ex.Message);
}
return sb.ToString();
}
Note that this will return the entire page and not just a portion of it. You'll then need to sift through the page to find the information you're looking for.