I have a low-traffic ASP.NET MVC 4 application that run on Windows Server 2008 / IIS 7.5.
My problem is that:
The first request is slow around 15 sec. (Subsequent requests are fine)
Another request after about 2 minutes without any requests always gets a slow response (around 15 sec)
After first request if I make another request from new browser, again it takes 15 sec.
A scenario to address the problem (the document size is 24 KB):
Time: 16:26 - Using Chrome - First Request takes 15 sec. Subsequent requests are fine.
Time: 16:27 - Using Firefox - First Request takes 15 sec. Subsequent requests are fine.
Time: 16:30 - Using IE 11 - First Request takes 15 sec. Subsequent requests are fine.
Here is all screenshots of Developer Tools/Network Tab
And also Fiddler time output:
Request Count: 1
Bytes Sent: 380 (headers:380; body:0)
Bytes Received: 7,217 (headers:409; body:6,808)
ACTUAL PERFORMANCE
--------------
ClientConnected: 22:41:26.377
ClientBeginRequest: 22:41:26.378
GotRequestHeaders: 22:41:26.378
ClientDoneRequest: 22:41:26.378
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 28ms
HTTPS Handshake: 0ms
ServerConnected: 22:41:26.407
FiddlerBeginRequest: 22:41:26.407
ServerGotRequest: 22:41:26.407
ServerBeginResponse: 22:41:41.496
GotResponseHeaders: 22:41:41.496
ServerDoneResponse: 22:41:41.503
ClientBeginResponse: 22:41:41.503
ClientDoneResponse: 22:41:41.504
Overall Elapsed: 00:00:15.1258651
It shows 15 sec delay between ServerGotRequest and ServerBeginResponse
This issue just occurs at my home. There is no problem at work or my friends computers (means another ISP). Also I have no problem in internet speed, then I tested my website connection:
output for ping neshoonak.ir
Reply from 94.232.172.248: bytes=32 time=67ms TTL=122
Reply from 94.232.172.248: bytes=32 time=56ms TTL=122
Reply from 94.232.172.248: bytes=32 time=63ms TTL=122
output for ping 8.8.8.8
Reply from 8.8.8.8: bytes=32 time=134ms TTL=47
Reply from 8.8.8.8: bytes=32 time=171ms TTL=47
Reply from 8.8.8.8: bytes=32 time=132ms TTL=47
I tested some sites placed in same data-center that my site located
and found that all of them have the same problem (there is no problem at work at all).
My reseller hosting says there is 3 data-center and I have problem with 2 of them (just at home). He proposed to move my website to the third data-center. But it may occurs for my site visitors and I don't want to solve it just for me.
Please help!
There are two possibilies that come to mind:
Caching: there are some resources that are not cached and thus they need to be loaded and that takes a while. How do you identify if this is the case? Install Fiddler and open the page. You will see the http response codes. How to fix? Cache :)
App pool: does not seem to be your issue but want to mention for other readers. This happened to me in an app for a Microsoft portal. They wanted near instant loading and it worked fine... Sometimes. I debugged and profiled the code a few times until it hit me. The problem was that the app pool was recycled and it needed to start and load everything. How did I fix? I set up a cron job every 5 minutes to open one page and this kept the app loaded in memory this getting near instant responses.
Hope this helps!
One good test is to set up a monitoring service like www.site24x7.com and have it ping your site from multiple locations and you can see response time there.
Related
In an effort to reduce 'Initial Server Response Time' and so have a better Google PageSpeed Insights, I've been trying to optimize that 4.5Kb request's response time which takes around 270ms TTFB and 0.71ms content download (measured using dev tools).
The app is hosted on a Linode in India which is physically near. I turned on logs on Nginx as I was suspecting something was wrong with it but it shows a total response time of 25ms.
Given that Nginx defines the total response time as 'Full request time, starting when NGINX reads the first byte from the client and ending when NGINX sends the last byte of the response body', I expected that ultimately the user would get the response in a little more than 25ms but never 10x that.
Any ideas what I could be missing here? What else can I look at?
UPDATE: I have made the decision to migrate my Linode to Singapore from Mumbai and the results are far better now, I moved from 270ms TTFB to ~100ms. Lesson learned, even though India is close, Singapore's fast internet speed makes it a more suitable place to host my app in.
From nginx logging docs
$request_time – Full request time, starting when NGINX reads the first
byte from the client and ending when NGINX sends the last byte of the
response body
...NGINX sends the last byte...
Meaning it has sent the last byte to the underlying OS. So TCP socket buffers might have stored the bytes and are trying to send them to the client.
Here is an analysis of this scenario.
Nginx does not care about the RTT (Round Trip Time) between the client and the server. That's an OS/client problem.
Pinging the server from the client could give you an idea of the order of response time. If ping time is greater than nginx's $response_time, performance can't be expected to be close to $request_time.
ping -c3 -s 1450 www.kernel.org
PING ord.git.kernel.org (147.75.58.133) 1450(1478) bytes of data.
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=1 ttl=48 time=191 ms
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=2 ttl=48 time=192 ms
1458 bytes from ord1.git.kernel.org (147.75.58.133): icmp_seq=3 ttl=48 time=198 ms
--- ord.git.kernel.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 191.155/194.026/198.468/3.205 ms
As a ball park approach, if your response size is 4.5kB and max TCP packet size is ~ 1.5kB, you could expect total time to be at best, 3 times the ping time.
On a Linux box the maximum transmission unit (MTU) is 1500:
ip addr | grep 'eth0: .*mtu'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
DNS resolution might have an influence.
I am looking at fiddler statistics below is my result:
ACTUAL PERFORMANCE
--------------
ClientConnected: 13:37:48.551
ClientBeginRequest: 13:37:49.281
GotRequestHeaders: 13:37:49.281
ClientDoneRequest: 13:37:49.283
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 0ms
HTTPS Handshake: 0ms
ServerConnected: 13:37:48.708
FiddlerBeginRequest: 13:37:49.283
ServerGotRequest: 13:37:49.284
ServerBeginResponse: 13:37:49.627
GotResponseHeaders: 13:37:49.627
ServerDoneResponse: 13:38:25.833
ClientBeginResponse: 13:38:25.835
ClientDoneResponse: 13:38:25.872
Overall Elapsed: 0:00:36.590
As you see, ServerDoneResponse minus GotResponseHeaders, seems to be time required for client to get response from server.
I have checked ServerDoneResponse means "Exact time that Fiddler got the first bytes of the server's HTTP response. "
ServerDoneResponse - Exact time that Fiddler got the last bytes of the server's HTTP response.
From URL = http://fiddler.wikidot.com/timers
But it does not mentions me about GotResponseHeaders ?
So my understanding is, for getting response from server to client it has taken most of the time i.e. ServerBeginResponse: 13:37:49.627 minus ServerDoneResponse: 13:38:25.833 ? Is it correct ?
I did not get any documentation for this but I think 'GotResponseHeaders' is the time when Fiddler starts getting the response back from the server. Not necessarily that the response has been completed at that time. To confirm this, I did a few tests with scenarios mentioned below:
Added a delay at server side code so that it starts the response a little later.
Downloaded a 10MB file so that it takes some time for the response to complete.
In the first case when I added a delay before responding, 'ServerBeginResponse' was delayed too and 'GotResponseHeaders' was same as 'ServerBeginResponse'. This suggests that Fiddler got the response at the same time server started responding(again, it's not necessary that the response has been completed).
for the second case too, 'GotResponseHeaders' was same as 'ServerBeginResponse' suggesting 'GotResponseHeaders' is the time when Fiddler started receiving the response. Note that since the file download took some time, 'ServerDoneResponse' was delayed by few seconds.
In above case, if ServerDoneResponse minus GotResponseHeaders is taking long then you should check the size of the response and also the network latency.
I use jmeter to test a tomcat web application running behind f5's load balancer.
From the pcap file I captured in jmeter node I see that there is a duplicate ack and then
16 seconds later the client retransmit the packet and do nothing within 20 seconds.
As tomcat's default socket timeout is set as 20 seconds the client received Http error.
My question is:
What could be the reason that make client's TCP stack retransmit after 16 seconds and later do nothing within 20 seconds? Is it too busy? Is there a way to find out the reason?
In an embedded device, What is a practical amount time to allow idle HTTP connections to stay open?
I know back in the olden days of the net circa 1999 internet chat rooms would sometimes just hold the connection open and send replies as they came in. Idle timeouts and session length of HTTP connections needed to be longer in those days...
How about today with ajax and such?
REASONING: I am writing a transparent proxy for an embedded system with low memory. I am looking for ways to prevent DoS attacks.
My guess would be 3 minutes, or 1 minute. The system has extremely limited RAM and it's okay if it breaks rare and unpopular sites.
In the old days (about 2000), an idle timeout was up to 5 minutes standard. These days it tends to be 5 seconds to 50 seconds. Apache's default is 5 seconds. With some special apps defaulting to 120 seconds.
So my assumption is, that with AJAX, long held-open HTTP connections are no longer needed.
How about allowing idle HTTP connections to remain open unless another communication request comes in? If a connection is open and no one else is trying to communicate, the open connection won't hurt anything. If someone else does try to communicate, send a FIN+ACK to the first connection and open the second. Many http clients will attempt to receive multiple files using the same connection if possible, but can reconnect between files if necessary.
Any thoughts on why I might be getting tons of "hangs" when trying to download a file via HTTP, based on the following?
Server is IIS 6
File being downloaded is a binary file, rather than a web page
Several clients hang, including TrueUpdate and FlexNet web updating packages, as well as custom .NET app that just does basic HttpWebRequest/HttpWebResponse logic and downloads using a response stream
IIS log file signature when success is 200 0 0 (sc-status sc-substatus sc-win32-status)
For failure, error signature is 200 0 64
sc-win32-status of 64 is "the specified network name is no longer available"
I can point firefox at the URL and download successfully every time (perhaps some retry logic is happening under the hood)
At this point, it seems like either there's something funky with my server that it's throwing these errors, or that this is just normal network behavior and I need to use (or write) a client that is more resilient to the failures.
Any thoughts?
Perhaps your issue was a low level networking issue with the ISP as you speculated in your reply comment. I am experiencing a similar problem with IIS and some mysterious 200 0 64 lines appearing in the log file, which is how I found this post. For the record, this is my understanding of sc-win32-status=64; I hope someone can correct me if I'm wrong.
sc-win32-status 64 means “The specified network name is no longer available.”
After IIS has sent the final response to the client, it waits for an ACK message from the client.
Sometimes clients will reset the connection instead of sending the final ACK back to server. This is not a graceful connection close, so IIS logs the “64” code to indicate an interruption.
Many clients will reset the connection when they are done with it, to free up the socket instead of leaving it in TIME_WAIT/CLOSE_WAIT.
Proxies may have a tendancy to do this more often than individual clients.
I've spent two weeks investigating this issue. For me I had the scenario in which intermittent random requests were being prematurely terminated. This was resulting in IIS logs with status code 200, but with a win32-status of 64.
Our infrastructure includes two Windows IIS servers behind two NetScaler load balancers in HA mode.
In my particular case, the problem was that the NetScaler had a feature called "Intergrated Caching" turned on (http://support.citrix.com/proddocs/topic/ns-optimization-10-5-map/ns-IC-gen-wrapper-10-con.html).
After disabling this feature, the request interruptions ceased. And the site operated normally. I'm not sure how or why this was causing a problem, but there it is.
If you use a proxy or a load balancer, do some investigation of what features they have turned on. For me the cause was something between the client and the server interrupting the requests.
I hope that this explanation will at least save someone else's time.
Check the headers from the server, especially content-type, and content-length, it's possible that your clients don't recognize the format of the binary file and hang while waiting for bytes that never come, or maybe they close the underlying TCP connection, which may cause IIS to log the win32 status 64.
Spent three days on this.
It was the timeout that was set to 4 seconds (curl php request).
Solution was to increase the timeout setting:
//curl_setopt($ch, CURLOPT_TIMEOUT, 4); // times out after 4s
curl_setopt($ch, CURLOPT_TIMEOUT, 60); // times out after 60s
You will have to use wireshare or network monitor to gather more data on this problem. Me think.
I suggest you put Fiddler in between your server and your download client. This should reveal the differences between Firefox and other cients.
Description of all sc-win32-status codes for reference
https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-
ERROR_NETNAME_DELETED
64 (0x40)
The specified network name is no longer available.