I am trying to call a webservice that return too much data just to extract a small piece of data.
So, I decided not to use the standard Client which is generated by Java.
I use the following code to do the connection:
HttpURLConnection connection;
byte[] requestData = .....
URL url = new URL(wsUrl);
connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "text/xml");
connection.setRequestProperty("Content-Length", String.valueOf(requestData.length));
connection.connect();
OutputStream connOs = connection.getOutputStream();
connOs.write(requestData);
connOs.close();
InputStream is = connection.getInputStream(); // <<< THIS IS THE MOST TIME CONSUMING, it takes about 70 ms
byte[] rply = stream2Bytes(is);
is.close();
connection.disconnect();
The most time is consumed in the call to connection.getInputStream(); which it takes about 70ms.
I am trying setting many request headers to reduce this time but cannot reach.
My understanding it that the HttpUrlConnection uses HTTP1.1 protocol that uses Connection=KEEP-ALIVE header by default so that the underlying TCP connection is reused.
connection.getInputStream(); - function which wait for server response... you can't speed up this proccess.
Related
I'm downloading a file from an HTTP server that has to be requested using POST data. The download completes, and when I target .NET CORE 2.1, it downloads a 1 MB file in around 50 msec. However, I'm developing for an application targeting .NET Framework 4.7.1, and when I run the EXACT same code targeting that framework (in a brand new empty project even), instead of 50 msec, it takes roughly 1200 times longer to download the EXACT same file from the EXACT same server. The file does eventually download successfully, but it takes far too long.
Looking at Wireshark data, I can see the when targeting either framework, the server sends the data mostly as packets with 1300 bytes of payload per packet. When targeting .NET CORE 2.1, it sends all of the packets in rapid succession. When targeting .NET Framework 4.7.1, it rapidly sends exactly 12 packets containing 1300 bytes and one packet containing 784 bytes (totaling exactly 16384 bytes, or 2^14), then it sends nothing for about 1 second, then it sends another burst of 16384 bytes of data, then it pauses for about 1 second, and so forth until the entire file has been sent.
What am I doing wrong? Since I know the server is capable of sending all of the packets in rapid succession, what do I need to change about my request to make it happen?
This is my code:
Uri address = new Uri("http://servername/file.cgi");
HttpWebRequest request = WebRequest.CreateHttp(address);
string postData = "filename=/folder/filename.xml&Download=Download";
var data = Encoding.ASCII.GetBytes(postData);
request.Credentials = new NetworkCredential("myusername", "mypassword");
request.Method = "POST";
request.KeepAlive = true;
request.UserAgent = "ThisApp";
request.Accept = "text/xml";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using (Stream output = File.OpenWrite("ReceivedFile.xml"))
using (Stream input = response.GetResponseStream())
{
input.CopyTo(output);
}
Thanks!!
Also, I've already tried several things I've found on other posts (this one was particularly relevant: HttpWebRequest is extremely slow!), but none of them have helped:
ServicePointManager.DefaultConnectionLimit = 200;
ServicePointManager.Expect100Continue = false;
request.Proxy = null;
request.SendChunked = true;
request.ServicePoint.ReceiveBufferSize = 999999;
I've also tried adding this to app.config:
<connectionManagement>
<add address="*" maxconnection="200"/>
</connectionManagement>
Found it! I needed to add BOTH of these lines:
ServicePointManager.Expect100Continue = false;
ServicePointManager.UseNagleAlgorithm = false;
In the application I am currently working on there is a backend java app that is caching a bunch of data. The asp.net part is allowing users to update database tables. Each time the DB is updated the cache in the java application should be cleared. So basically I have a list of 4 URLs that each need to be hit in order to clear the cache. My basic solution was to loop through each url and create a HttpWebRequest and get then get the response. So basically I have this for each request:
var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.ContentLength = 0;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
string responseString = readStream.ReadToEnd();
returnList.Add(string.Format("Refresh response from {0}.<br />{1}", url, responseString));
readStream.Close();
receiveStream.Close();
}
On my local machine everything works great. But when I deploy to our development server it just hangs and does nothing. If I remove request.ContentLength = 0; then the remote server throws a 411: Length expected error.
I am really stuck here and any help would be greatly appreciated.
Either a solution to the HttpWebRequest problem I am having or a different solution to calling each URL would work, I'm not picky.
Thanks in advance.
Why are using request.method as "POST"? Are you posting any data, if not try removing both content length and request method.
Pretty sure this was a network issue. I tried hitting a different url (the load balancer) and had no problems so the java guys are making a changes so I can just hit the load balancer and whatever server the request ends up on will make sure all servers caches are cleared.
The code that is working:
var request = (HttpWebRequest)WebRequest.Create(url);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
string responseString = readStream.ReadToEnd();
returnString = string.Format(#"Refresh response from<br />{0}{1}", url, responseString);
readStream.Close();
receiveStream.Close();
}
I've been working on a BlackBerry post request, and the request is getting sent, but the parameters don't seem to be. Here is my code:
HttpConnection httpConnection = (HttpConnection) Connector.open(url);
httpConnection.setRequestMethod(HttpConnection.POST);
httpConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
URLEncodedPostData encPostData = new URLEncodedPostData("UTF-8", false);
encPostData.append("time", "1314144000");
System.out.println("url: " + httpConnection.getURL());
byte[] postData = encPostData.toString().getBytes("UTF-8");
System.out.println("post data: " + encPostData.toString());
httpConnection.setRequestProperty("Content-length", String.valueOf(postData.length));
System.out.println("url: " + httpConnection.getURL());
System.out.println("message:" + httpConnection.getResponseMessage());
OutputStream os = httpConnection.openOutputStream();
os.write(postData);
os.flush();
os.close();
The response I get from the server(which we set up) is that we didn't send a time stamp. Is there something wrong with my
encPostData.append("time", "1314144000");
code?
Your call to getResponseMessage() before writing the post data is forcing a response before anything has been written on the connection.
System.out.println("message:" + httpConnection.getResponseMessage());
Move that to the end, after the output stream data has been written, and I think it will work better for you.
Make the http connection in read write mode.Might be that is the problem making http connection without mode link
HttpConnection connection = (HttpConnection) Connector.open("url", Connector.READ_WRITE);
See below link for making http connection.
blackberry server connection problem
I am truing to integrate fusemail in asp.net 2.0. I am using HttpWebRequest for requesting the API pages. It has recently come to my notice that HttpWebRequest fails the first time and then continues and subsequent requests succeed.
say ( i know if i use goto it is a bad programming approach) if i use this code
retry:
try
{
Uri uri = new Uri("http://www.fusemail.com/api/request.html");
if (uri.Scheme == Uri.UriSchemeHttp)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
request.PreAuthenticate = true;
request.Method =
WebRequestMethods.Http.Post;
//request.ReadWriteTimeout = System.Threading.Timeout.Infinite;
//request.Timeout = System.Threading.Timeout.Infinite;
request.ContentLength = data.Length;
request.ContentType =
"application/x-www-form-urlencoded";
//request.UserAgent = Request.UserAgent;
request.UserAgent = "Mozilla/4.0";
request.KeepAlive = false;
request.ServicePoint.Expect100Continue = true;
//request.Accept = "Accept: text/html,application/xhtml+xml,application/xml";
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(data);
writer.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string tmp = reader.ReadToEnd();
response.Close();
//Response.Write(tmp);
if (!String.IsNullOrEmpty(tmp))
{
return tmp;
}
}
return String.Empty;
}
catch (WebException ex)
{
goto retry;
}
it works after failing once. i am writing to a text file in case of an error and after i failed request it works the second time. I am using ASP.Net 2.0 and the website is hosted on IIS 7 with Windows Server 2008 Standard. Also pinging the API address it fails the first time and then responds
C:\>ping 67.207.202.118
Pinging 67.207.202.118 with 32 bytes of data:
Reply from 192.168.0.253: **Destination host unreachable**.
Reply from 67.207.202.118: bytes=32 time=218ms TTL=49
Reply from 67.207.202.118: bytes=32 time=218ms TTL=49
Reply from 67.207.202.118: bytes=32 time=217ms TTL=49
Ping statistics for 67.207.202.118:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 217ms, Maximum = 218ms, Average = 217ms
The first time it fails in HttpWebRequest it fails with this error
System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 67.207.202.118:80
Is there an authentication issue the first time?. i read on some forums it first sends a 401 Unauthorized and then it can connect. I am unable to verify this using Fiddler.
Is there anything wrong with IIS configuration?
This is not a programming issue at all, I have faced a similar problem later and it was a network configuration problem due to ISA server / Firewall settings.
You have to contact your network administrator to check this issue.
I wish this helped you.
Yours,
Mohamed Kamal Elbeah
Senior .Net Developer
I recently came by this same issue. The solution in my case involved my testing environment, since I had multiple Ethernet adapters connected to my computer. While you may have a different IP for each of your Ethernet adapters, if they are all assigned to the same subnet this may cause a problem. The TCP connection is only attempted using one NIC at a time. So in my case, on the first try it would attempt the connection on one adapter that was not connected to my remote host, then on the second try it would connect using the second adapter which was connected. - Hope this helps someone.
I'am using the WebRequest class in .net and POST data to a server which is responding with a Response.
The wierd thing is that its working when I started fiddler to analyze my network traffic, but without fiddler it isn't.
So i started to analyze the package which is sent to and from my computer with WireShark. With in this program its simple to follow the TCP-stream. So when I had fiddler on, I can see the correct Request-header/body is sent, and gets the Response-header/body. The strange part is when i dont use fiddler the Request-header is sent, then i´ve got the Response-header/body, and finally the request-body in the end of the TCP-stream.
Here is my code i've been elaborating:
string lcUrl = "http://XX.XX.XXX.XX";
// *** Establish the request
HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl);
string lcPostData = testdata;
loHttp.Method = "POST";
byte [] lbPostBuffer = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData);
loHttp.ContentLength = lbPostBuffer.Length;
loHttp.Credentials = CredentialCache.DefaultCredentials;
//loHttp.SendChunked = true;
loHttp.ServicePoint.Expect100Continue = false;
Stream loPostData = loHttp.GetRequestStream();
loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length);
loPostData.Close();
HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse();
Encoding enc = System.Text.Encoding.GetEncoding(1252);
StreamReader loResponseStream = new StreamReader(loWebResponse.GetResponseStream(), enc);
string lcHtml = loResponseStream.ReadToEnd();
loWebResponse.Close();
loResponseStream.Close();
Please use following code. Seems that you have problems with time when underlying stream are send to remote server.
string lcUrl = "http://XX.XX.XXX.XX";
// *** Establish the request
HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl);
string lcPostData = testdata;
loHttp.Method = "POST";
byte[] lbPostBuffer = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData);
loHttp.ContentLength = lbPostBuffer.Length;
loHttp.Credentials = CredentialCache.DefaultCredentials;
//loHttp.SendChunked = true;
loHttp.ServicePoint.Expect100Continue = false;
using (Stream loPostData = loHttp.GetRequestStream())
{
loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length);
}
string lcHtml;
using (HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse())
{
Encoding enc = System.Text.Encoding.GetEncoding(1252);
using (StreamReader loResponseStream = new StreamReader(loWebResponse.GetResponseStream(), enc))
{
lcHtml = loResponseStream.ReadToEnd();
}
}
// Perform processing of data here....
Also I could suggest you add following code in the app.config file for your application. This is helps when server returns response that not conforms with way how .NET handle HTTP request.
<configuration>
<system.net>
<settings>
<httpWebRequest
useUnsafeHeaderParsing="true"
/>
</settings>
</system.net>
</configuration>
I have a suspicion that the client is waiting for the "HTTP/1.1 100 continue" response from the server. This is how it works. When you are posting data to the server, sometimes the server might not be ready to accept the data just yet. For eg, it wants to authenticate the client first.
So, when you send a POST request, the client just sends the request headers, with an "Expect: 100-continue" appended.
POST /url HTTP/1.1
Server: Server-name/fqdn
Content-Length: 100
Expect: 100-continue
If the server is ready to receive the data it responds with:
HTTP/1.1 100 continue
Server: server-name/fqdn
Now, the client can send the data.
However if the server is not ready to receive the data, and wants to authenticate the client, it will respond with a different status code.
If you post your wireshark trace to pastebin.com I can verify, but I suspect this is what is happening.
The reason you dont see this in fiddler might be that fiddler is using HttpListener to listen to HTTP request, and HTTP listener hides the intermediate response like 100-continue from the app (in this case fiddler).