I would like to use HttpClient to read the chunked (in the sense of HTTP 1.1 chunked transfer encoding) content asynchronously.
I am looking at HttpContent async methods at:
MSDN link
However, in the case of returned Task for byte array, for example:
The returned Task object will complete after all of the content has been written as a byte array
I am getting chunked content precisely because server doesn't know ahead of time when will all of the data be available, thus I don't know when will all of the content arrive. Rather than waiting, possibly for hours, for the task to complete, I would like to be able to read the chunks as they arrive.
Can I somehow read part of the response content, like have some task that would complete when every 4K bytes of content are received in response?
Is using HttpClient advantageous at all in this case?
Using HttpClient.SendAsync you can pass a HttpCompletionOption parameter to tell HttpClient not to buffer the response for you and return as soon as it gets the headers. Then you can use ReadAsStreamAsync to get a stream that will allow you to pull the data as it arrives.
Here is a complete example of how to download a file without reading its content to memory with an explanation. Works beautifully.
static async Task HttpGetForLargeFileInRightWay()
{
using (HttpClient client = new HttpClient())
{
const string url =
"https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
using (HttpResponseMessage response = await client.GetAsync(url,
HttpCompletionOption.ResponseHeadersRead))
using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
{
string fileToWriteTo = Path.GetTempFileName();
using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
{
await streamToReadFrom.CopyToAsync(streamToWriteTo);
}
}
}
}
Or instead of using CopyToAsync() you can read stream by using StreamReader
using (var stream = await response.Content.ReadAsStreamAsync())
using (var reader = new StreamReader(stream))
{
int bytesCount = 100;
var buffer = new char[bytesCount];
reader.ReadBlock(buffer, 0, bytesCount);
}
Related
Here is the client :
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("http://localhost/MP.Business.Implementation.FaceAPI/");
client.DefaultRequestHeaders
.Accept
.Add(new MediaTypeWithQualityHeaderValue("application/octet-stream"));
using (var request = new HttpRequestMessage(HttpMethod.Post, client.BaseAddress + "api/Recognition/Recognize"))
{
request.Content = new ByteArrayContent(pic);
request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
await client.PostAsync(request.RequestUri, request.Content);
}
}
and the server :
[System.Web.Http.HttpPost]
public string Recognize(byte[] img)
{
//do someth with the byte []
}
I am getting error:
415 Unsupported Media Type
all the time - The request entity's media type 'application/octet-stream' is not supported for this resource. What can i do about it? I've found some answered threads here , but it didnt help.
While byte[] would be a great way to represent application/octet-stream data, this is not the case by default in Web API.
My workaround is in ASP.NET Core 1.1 - the details may be different in other variants.
In your controller method, remove the img parameter. Instead, refer to the Request.Body, which is a Stream. e.g. to save to a file:
using (var stream = new FileStream(someLocalPath, FileMode.Create))
{
Request.Body.CopyTo(stream);
}
The situation is similar for returning binary data from a GET controller method. If you make the return type byte[] then it is formatted with base64! This makes it significantly larger. Modern browsers are perfectly capable of handling raw binary data so this is no longer a sensible default.
Fortunately there is a Response.Body https://github.com/danielearwicker/ByteArrayFormatters:
Response.ContentType = "application/octet-stream";
Response.Body.Write(myArray, 0, myArray.Length);
Make the return type of your controller method void.
UPDATE
I've created a nuget package that enables direct use of byte[] in controller methods. See: https://github.com/danielearwicker/ByteArrayFormatters
I have been trying to find a way to make this task more efficient. I am consuming a REST based web service and need to update information for over 2500 clients.
I am using fiddler to watch the requests, and I'm also updating a table with an update time when its complete. I'm getting about 1 response per second. Are my expectations to high? I'm not even sure what I would define as 'fast' in this context.
I am handling everything in my controller and have tried running multiple web requests in parallel based on examples around the place but it doesn't seem to make a difference. To be honest I don't understand it well enough and was just trying to get it to build. I suspect it is still waiting for each request to complete before firing again.
I have also increased connections in my web config file as per another suggestion with no success:
<system.net>
<connectionManagement>
<add address="*" maxconnection="20" />
</connectionManagement>
</system.net>
My Controllers action method looks like this:
public async Task<ActionResult> UpdateMattersAsync()
{
//Only get matters we haven't synced yet
List<MatterClientRepair> repairList = Data.Get.AllUnsyncedMatterClientRepairs(true);
//Take the next 500
List<MatterClientRepair> subRepairList = repairList.Take(500).ToList();
FinalisedMatterViewModel vm = new FinalisedMatterViewModel();
using (ApplicationDbContext db = new ApplicationDbContext())
{
int jobCount = 0;
foreach (var job in subRepairList)
{
// If not yet synced - it shouldn't ever be!!
if (!job.Synced)
{
jobCount++;
// set up some Authentication fields
var oauth = new OAuth.Manager();
oauth["access_token"] = Session["AccessToken"].ToString();
string uri = "https://app.com/api/v2/matters/" + job.Matter;
// prepare the json object for the body
MatterClientJob jsonBody = new MatterClientJob();
jsonBody.matter = new MatterForUpload();
jsonBody.matter.client_id = job.NewClient;
string jsonString = jsonBody.ToJSON();
// Send it off. It returns the whole object we updated - we don't actually do anything with it
Matter result = await oauth.Update<Matter>(uri, oauth["access_token"], "PUT", jsonString);
// update our entities
var updateJob = db.MatterClientRepairs.Find(job.ID);
updateJob.Synced = true;
updateJob.Update_Time = DateTime.Now;
db.Entry(updateJob).State = System.Data.Entity.EntityState.Modified;
if (jobCount % 50 == 0)
{
// save every 50 changes
db.SaveChanges();
}
}
}
// if there are remaining files to save
if (jobCount % 50 != 0)
{
db.SaveChanges();
}
return View("FinalisedMatters", Data.Get.AllMatterClientRepairs());
}
}
And of course the Update method itself which handles the Web requesting:
public async Task<T> Update<T>(string uri, string token, string method, string json)
{
var authzHeader = GenerateAuthzHeader(uri, method);
// prepare the token request
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Headers.Add("Authorization", authzHeader);
request.Method = method;
request.ContentType = "application/json";
request.Accept = "application/json, text/javascript";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(json);
request.ContentLength = bytes.Length;
System.IO.Stream os = request.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
os.Close();
WebResponse response = await request.GetResponseAsync();
using (var reader = new System.IO.StreamReader(response.GetResponseStream()))
{
return JsonConvert.DeserializeObject<T>(reader.ReadToEnd());
}
}
If it's not possible to do more than 1 request per second then I'm interested in looking at an Ajax solution so I can give the user some feedback while it is processing. In my current solution I cannot give the user feedback while the action method hasn't reached 'return' yet can I?
Okay it's taken me a few days (and a LOT of trial and error) but I've worked this out. Hopefully it can help others. I finally found my silver bullet. And it was probably the place I should have started:
MSDN: Consuming the Task-based Asynchronous Pattern
In the end this following line of code is what brought it all to light.
string [] pages = await Task.WhenAll(from url in urls select DownloadStringAsync(url));
I substituted a few things to make it work for a Put request as follows:
HttpResponseMessage[] results = await Task.WhenAll(from p in toUpload select client.PutAsync(p.uri, p.jsonContent));
'toUpload' is a List of MyClass:
public class MyClass
{
// the URI should be relative to the base pase
// (ie: /api/v2/matters/101)
public string uri { get; set; }
// a string in JSON format, being the body of the PUT request
public StringContent jsonContent { get; set; }
}
The key was to stop trying to put my PutAsync method inside a loop. My new line of code IS still blocking until ALL responses have come back, but that is what I wanted. Also, learning that I could use this LINQ style expression to create a Task List on the fly was immeasurably helpful. I won't post all the code (unless someone wants it) because it's not as nicely refactored as the original and I still need to check whether the response of each item was 200 OK before I record it as successfully saved in my database. So how much faster is it?
Results
I tested a sample of 50 web service calls from my local machine. (There is some saving of records to a SQL Database in Azure at the end).
Original Synchronous Code: 70.73 seconds
Asynchronous Code: 8.89 seconds
That's gone from 1.4146 requests per second down to a mind melting 0.1778 requests per second! (if you average it out)
Conclusion
My journey isn't over. I've just scratched the surface of asynchronous programming and am loving it. I need to now work out how to save only the results that have returned 200 OK. I can deserialize the HttpResponse which returns a JSON object (which has a unique ID I can look up etc.) OR I could use the Task.WhenAny method, and experiment with Interleaving.
I have the following method that writes a stream in a HttpResponse object.
public HttpResponse ShowPDF(Stream stream)
{
MemoryStream memoryStream = (MemoryStream) stream;
httpResponse.Clear();
httpResponse.Buffer = true;
httpResponse.ContentType = "application/pdf";
httpResponse.BinaryWrite(memoryStream.ToArray());
httpResponse.End();
return httpResponse;
}
In order to test it, I need to recover the processed stream.
Is there someway to read the stream from the httpResponse object?
I have two ideas... one to mock the HttpResponse, and the other is to simulate a web server.
1. Mocking HttpResponse
I wrote this before I knew which mocking framework you used. Here's how you could test your method using TypeMock.
This assumes that you pass your httpResponse variable to the method, changing the method as follows:
public void ShowPDF(Stream stream, HttpResponse httpResponse)
Of course you would change this to passing it to a property on your Page object instead, if it is a member of your Page class.
And here's an example of how you could test using a fake HttpResponse:
internal void TestPDF()
{
FileStream fileStream = new FileStream("C:\\deleteme\\The Mischievous Nerd's Guide to World Domination.pdf", FileMode.Open);
MemoryStream memoryStream = new MemoryStream();
try
{
memoryStream.SetLength(fileStream.Length);
fileStream.Read(memoryStream.GetBuffer(), 0, (int)fileStream.Length);
memoryStream.Flush();
fileStream.Close();
byte[] buffer = null;
var fakeHttpResponse = Isolate.Fake.Instance<HttpResponse>(Members.ReturnRecursiveFakes);
Isolate.WhenCalled(() => fakeHttpResponse.BinaryWrite(null)).DoInstead((context) => { buffer = (byte[])context.Parameters[0]; });
ShowPDF(memoryStream, fakeHttpResponse);
if (buffer == null)
throw new Exception("It didn't write!");
}
finally
{
memoryStream.Close();
}
}
2. Simulate a Web Server
Perhaps you can do this by simulating a web server. It might sound crazy, but it doesn't look like it's that much code. Here are a couple of links about running Web Forms outside of IIS.
Can I run a ASPX and grep the result without making HTTP request?
http://msdn.microsoft.com/en-us/magazine/cc163879.aspx
I'm attempting to write a Java Servlet to receive binary data requests and reply to them, using HttpServletRequest.getOutputStream() and HttpServletResponse.getInputStream(). This is for a project which involves having a request sent by a Silverlight client to which this servlet responds to through an HTTP POST connection. For the time being, to test the Servlet I'm implementing a client in Java which I'm more familiar with than Silverlight.
The problem is that in my test project I send the data from a Servlet client as a byte array and expect to receive a byte array with the same length -- only it doesn't, and instead I'm getting a single byte. Therefore I'm posting here the relevant code snippets in the hopes that you might point me where I'm doing wrong and hopefully provide relevant bibliography to help me further.
So here goes.
The client servlet handles POST requests from a very simple HTML page with a form which I use as front-end. I'm not too worried about using JSP etc, instead I'm focused on making the inter-Servlet communication work.
// client HttpServlet invokes this method from doPost(request,response)
private void process(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String firstName = (String) request.getParameter("firstname");
String lastName = (String) request.getParameter("lastname");
String xmlRequest = "<MyRequest><Person><Name Firstname=\""+firstName+"\" Lastname=\""+lastName+"\" /></Person></MyRequest>";
OutputStream writer = null;
InputStream reader = null;
try {
URL url = new URL("http://localhost:8080/project/Server");
URLConnection conn = url.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
writer = conn.getOutputStream();
byte[] baXml = xmlRequest.getBytes("UTF-8");
writer.write(baXml, 0,baXml.length);
writer.flush();
// perhaps I should be waiting here? how?
reader = conn.getInputStream();
int available = reader.available();
byte[] data = new byte[available];
reader.read(data,0,available);
String xmlResponse = new String(data,"UTF-8");
PrintWriter print = response.getWriter();
print.write("<html><body>Response:<br/><pre>");
print.write(xmlResponse);
print.write("</pre></body></html>");
print.close();
} finally {
if(writer!=null)
writer.close();
if(reader!=null)
reader.close();
}
}
The server servlet handles HTTP POST requests. This is done by receiving requests the requests from a client Servlet for testing purposes above, but in the future I intend to use it for clients in other languages (specifically, Silverlight).
// server HttpServlet invokes this method from doPost(request,response)
private void process(HttpServletRequest request, HttpServetResponse response)
throws ServletException, IOException {
ServletInputStream sis = null;
try {
sis = request.getInputStream();
// maybe I should be using a BufferedInputStream
// instead of the InputStream directly?
int available = sis.available();
byte[] input = new byte[available];
int readBytes = sis.read(input,0,available);
if(readBytes!=available) {
throw new ServletException("Oops! readBytes!=availableBytes");
}
// I ONLY GET 1 BYTE OF DATA !!!
// It's the first byte of the client message, a '<'.
String msg = "Read "+readBytes+" bytes of "
+available+" available from request InputStream.";
System.err.println("Server.process(HttpServletRequest,HttpServletResponse): "+msg);
String xmlReply = "<Reply><Message>"+msg+"</Message></Reply>";
byte[] data = xmlReply.getBytes("UTF-8");
ServletOutputStream sos = response.getOutputStream();
sos.write(data, 0,data.length);
sos.flush();
sos.close();
} finally {
if(sis!=null)
sis.close();
}
}
I have been sticking to byte arrays instead of using BufferInputStreams so far because I've not decided yet if I'll be using e.g. Base64-encoded strings to transmit data or if I'll be sending binary data as-is.
Thank you in advance.
To copy input stream to output stream use the standard way:
InputStream is=request.getInputStream();
OutputStream os=response.getOutputStream();
byte[] buf = new byte[1000];
for (int nChunk = is.read(buf); nChunk!=-1; nChunk = is.read(buf))
{
os.write(buf, 0, nChunk);
}
The one thing I can think of is that you are reading only request.getInputStream().available() bytes, then deciding that you have had everything. According to the documentation, available() will return the number of bytes that can be read without blocking, but I don't see any mention of whether this is actually guaranteed to be the entire content of the input stream, so am inclined to assume that no such guarantees are made.
I'm not sure how to best find out when there is no more data (maybe Content-Length in the request can help?) without risking blocking indefinitely at EOF, but I would try looping until having read all the data from the input stream. To test that theory, you could always scan the input for a known pattern that occurs further into the stream, maybe a > matching the initial < that you are getting.
I don't want to do anything fancy on Twitter except post to it via my site once a day. I have searched around a bit and there are all sorts of super-complex ways to do every little thing that Twitter does, but there seems to be little documentation on how to do the simplest thing, which is make a post!
Does anyone know how to do this? Or can you at least point me in the right direction? I don't need full wrappers or anything (http://apiwiki.twitter.com/Libraries#C/NET), just one simple function that will post to Twitter.
Thanks!
This is the easiest implementation ever. Up and running in under 2 minutes: Twitterizer
Its fairly simple; you just need to post an xml file to a web page using webrequest.create. This example is close (assumes you have the xml for the message in another place and just pass it into twitterxml variable as a string. The url might not be the right one; found it on this [page][1] which defines the interface
WebRequest req = null;
WebResponse rsp = null;
try
{
string twitterXML = "xml as string";
string uri = "http://twitter.com/statuses/update.format";
req = WebRequest.Create(uri);
//req.Proxy = WebProxy.GetDefaultProxy(); // Enable if using proxy
req.Method = "POST"; // Post method
req.ContentType = "text/xml"; // content type
// Wrap the request stream with a text-based writer
StreamWriter writer = new StreamWriter(req.GetRequestStream());
// Write the XML text into the stream
writer.WriteLine(twitterXML);
writer.Close();
// Send the data to the webserver
rsp = req.GetResponse();
}
[1]: http://apiwiki.twitter.com/Twitter-REST-API-Method%3A-statuses update
There are a couple different ways of doing this, they vary depending on the tools you want to use and have access to. Option 1 will work right out of the box, but the coding can be complicated. Option 3 you will have to download tools for, but once there installed and loaded you should be able to consume the twitter api very quickly.
Use WebRequest.Create to create/send messages to remote endpoints
Use WCF, create a mirror endpoint and access the twitter api using client only endpoint.
Use the WCF REST Starter Kit Preview 2, which has a new class called the HttpClient. I would have to recommend this technique if you can. Here is a great video Consuming a REST Twitter Feed in under 3 minutes.
Here is a sample of using the WCF REST Starter Kit's HttpClient:
public void CreateFriendship(string friend)
{
using (var client = new HttpClient())
{
var url = string.Format("http://www.twitter.com/friendships/create/{0}.xml?follow=true", friend);
client.Post(url)
.CheckForTwitterError()
.EnsureStatusIs(HttpStatusCode.OK);
}
}
Add a comment if you'd like more info about a particular method.
Update:
For Option #1 see this question: Remote HTTP Post with C#
There are a few ways of doing this, you can check out http://restfor.me/twitter and it will give you the code from RESTful documentation.
Essentially making any authenticated call you can follow this logic:
///
/// Executes an HTTP POST command and retrives the information.
/// This function will automatically include a "source" parameter if the "Source" property is set.
///
/// The URL to perform the POST operation
/// The username to use with the request
/// The password to use with the request
/// The data to post
/// The response of the request, or null if we got 404 or nothing.
protected string ExecutePostCommand(string url, string userName, string password, string data) {
WebRequest request = WebRequest.Create(url);
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
if (!string.IsNullOrEmpty(userName) && !string.IsNullOrEmpty(password)) {
request.Credentials = new NetworkCredential(userName, password);
byte[] bytes = Encoding.UTF8.GetBytes(data);
request.ContentLength = bytes.Length;
using (Stream requestStream = request.GetRequestStream()) {
requestStream.Write(bytes, 0, bytes.Length);
using (WebResponse response = request.GetResponse()) {
using (StreamReader reader = new StreamReader(response.GetResponseStream())) {
return reader.ReadToEnd();
}
}
}
}
return null;
}