I have a song download portal, which is run under IIS7 web server configure with PHP v 5.4.24.Windows version 2012. When we goes to download a song file(mp3) like 4-6 mb then connection release after 20 sec and start downloading again after some time and then again release connection after 20 sec. But small file like 1 mb download properly. To resolve the issue, i have done below changes into php.ini & IIS configuration.But problem still exists :
1. memory_limit = 512M [increase memory limit in php.ini].
2.default_socket_timeout = 600 [increase default socket time out value php.ini]
3.Set Idle Time-out=0 into IIS default application pool.
please suggest to resolve the issue.
i have resolve the issue. Just need to unmarked the "Enable HTTP keep Alive" option in IIS.Go to IIS->HTTP Response Headers->Set Common Headers-> Enable HTTP keep Alive Section.(Unmarked the option)
Related
We have a recurring problem wherein, our remote repositories that use either certificates or passwords to access their upstream, become inaccessable after a server restart. If, however, we simply view one of them in the admin UI, and change anything in it, something trivial even and save, all of the remotes simultaneously regain their access.
For example for we have a remote to https://registry.redhat.io, which uses a password and token
There is no proxy
socket timeout (MS) 60000
Advanced config options:
Cache:
Unused Artifacts Cleanup Period (Hr) 168
Metadata Retrieval Cache Period (Sec) 0
Metadata Retrieval Cache Timeout (Sec) 60
Assumed Offline Period (Sec) 0
Missed Retrieval Cache Period (Sec) 0
Others:
nothing checked
It's running on Ubuntu 20.04.5.
The Server typically reboots about every 2 weeks after a kernel update comes through. Is there anything I can do to ensure those connections dont get severed each time?
Problem
In our webapp (Angular + JAX-RS REST backend running on WebLogic + IIS proxy) we have 1 REST endpoint which returns an XLSX download (octet-stream). These XLSX files can be huge (up to the XLSX limit of 1M rows).
After some time, on slow connections, the download fails (ERR_CONNECTION_RESET in Chrome devtools).
The exact time when this happens varies:
Some days after 4-6 minutes, other days after 10-12 minutes. No clear pattern.
Fast(er) downloads work fine, are always succesful. I have seen downloads of hundreds of MBs finish succesfully in (eg.) 8 minutes, but others fail at (eg.) 11 minutes.
The problem is that I do not understand why the download fails and why the connection is reset. Any pointers or tips on how to test and debug this problem are welcome.
As far as I understand ERR_CONNECTION_RESET just means that something reset the connection. Just looking at the response headers gives no indication on who reset it.
Question
How can I understand why the download fails and who resets the connection?
The logfiles do not state which component resets the connection.
Setup
The webapp is deployed on WebLogic 12.2 on the internal network.
IIS 8.5 acts as a reverse proxy making the webapp accessible on the internet.
Details
When I download without IIS as reverse proxy (from our internal network), but with a speed limit in Chrome devtools), the download is always succesful. I've had downloads with 50kb/s which finished fine in 2 hours.
We cannot find any settings in IIS which influence this behaviour, so I am hesitant to definitively conclude that IIS causes the connection reset, since the precise time varies.
The WebLogic (exception) logs state that writing to the OutputStream fails because of a closed connection. No exceptions or log entries which indicate that WebLogic closed the connection.
Using other download speeds makes no difference. There is no direct relation to speed and connection reset time.
The download is never stalled.
VPN connection does not seem a factor, people with and without VPN experience the same problem.
Changing proxy is unfortunately not an immediate solution. Large corporate. Without understanding and knowing precisely that (if) IIS is the problem - not going to happen.
WebLogic exception
Caused by: java.net.SocketException: Socket closed
at weblogic.socket.NIOOutputStream.convertToSocketException(NIOOutputStream.java:250) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.NIOOutputStream.access$600(NIOOutputStream.java:33) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.NIOOutputStream$BlockingWriter.flush(NIOOutputStream.java:482) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.NIOOutputStream$BlockingWriter.write(NIOOutputStream.java:334) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.NIOOutputStream.write(NIOOutputStream.java:220) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.JSSEFilterImpl.writeToNetwork(JSSEFilterImpl.java:829) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.JSSEFilterImpl.wrapAndWrite(JSSEFilterImpl.java:789) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.JSSEFilterImpl.write(JSSEFilterImpl.java:503) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.socket.JSSESocket$JSSEOutputStream.write(JSSESocket.java:154) ~[com.oracle.weblogic.server.muxers.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:628) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:590) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:474) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutput$3.checkForFlush(ChunkOutput.java:760) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutput.write(ChunkOutput.java:373) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ChunkOutputWrapper.write(ChunkOutputWrapper.java:165) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at weblogic.servlet.internal.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:186) ~[com.oracle.weblogic.servlet.jar:12.2.1.4]
at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:325) ~[org.glassfish.jersey.containers.jersey-container-servlet-core.jar:?]
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229) ~[org.glassfish.jersey.core.jersey-common.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299) ~[org.glassfish.jersey.core.jersey-common.jar:?]
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) ~[?:1.8.0_261]
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) ~[?:1.8.0_261]
at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:331) ~[?:1.8.0_261]
at org.apache.poi.util.IOUtils.copy(IOUtils.java:317) ~[org.apache.poi-poi-3.17.jar:3.17]
at org.apache.poi.xssf.streaming.SXSSFWorkbook.copyStreamAndInjectWorksheet(SXSSFWorkbook.java:501) ~[org.apache.poi-poi-ooxml-3.17.jar:3.17]
at org.apache.poi.xssf.streaming.SXSSFWorkbook.injectData(SXSSFWorkbook.java:391) ~[org.apache.poi-poi-ooxml-3.17.jar:3.17]
at org.apache.poi.xssf.streaming.SXSSFWorkbook.write(SXSSFWorkbook.java:936) ~[org.apache.poi-poi-ooxml-3.17.jar:3.17]
...
IIS logs
The only line I could find relevant to this problem is:
1.2.3.4, -, 9/15/2020, 9:20:14, W3SVC3, HSTWEB, 2.3.4.5, 561236, 1813, 9658662, 500, 0, GET, /api/resources/export/FOO, sorting=1,
In short, I am trying to do a load test. But I cannot create the desired throughput on the IIS server (Windows Server 2016 Datacenter) even though there seems to be no bottleneck in terms of cpu, memory, disk or network.
Here is my configuration:
IIS Server: 16 vCPU, 32GB memory
SQL Server: 4 vCPU, 8GB memory
Test Server (sending the requests): 8 vCPU, 16GB memory
In order to remove concurrency limits on the IIS server, I did the following changes:
<serverRuntime appConcurrentRequestLimit="1000000" />
<applicationPool
maxConcurrentRequestsPerCPU="1000000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="1000000" />
Default Application Pool Queue Length: 65000
<processModel minWorkerThreads="5000">
I have created a WPF application that creates the desired number of concurrent requests towards the IIS server using HttpClient and deployed it on the test server. (I changed the service point default connection limit to 1000000 as well.) And I tested with 5000 requests which all returned 200 OK.
Normally, one request returns in 20ms. And here are the results of the test I obtained in the WPF application:
Total time starting from sending the first request through getting the last response: 9380ms
Average response time : 3919ms
Max. response time: 7243ms
Min. response time: 77ms
When I look at the performance counters on the test server, I see that 5000 requests completed in about 3 seconds. Here is the graph I obtained from perfmon:
But when I look at the performance counters on the IIS server, I see that requests are continually received and executed during the course of 9 seconds. So, the average throughput observed is about 400 requests per second. I also tried the test with 10000 requests but the average throughput is always around 400 req/sec.
Why doesn't ASP.NET complete receiving all the requests at the end of the first 3 seconds? How can I increase throughput to any desired value so that I can conduct a proper load test?
After a lot of experimenting, I found out that any value over 2000 for minWorkerThreads seem to be ignored. I checked it using the ThreadPool.GetMinThreads method. And I also added the maxWorkerThreads value of 2100 as #StephenCleary suggested. With these values, the problem disappeared. But the strange thing is that, I have not seen such a limitation on the minWorkerThreads value in any of the MS documentations.
When you set the httpRuntime executionTimeout, does that cut off a response that has already been partially sent to the client?
I've noticed in the IIS request queue that occasionally there are some requests that run for a lot longer than our executionTimeout setting and the state is SendResponse in the IIS Web Core module. Does the executionTimeout include the time that it takes to send the response to the client?
The only documentation that I could find is on the following page in this snip, but it is from 2003 for older versions of IIS:
https://msdn.microsoft.com/en-us/library/ms972959.aspx
Request Execution Time. The number of milliseconds taken to execute the last request. In version 1.0 of the Framework, the execution time begins when the worker process receives the request, and stops when the ASP.NET ISAPI sends HSE_REQ_DONE_WITH_SESSION to IIS. For IIS version 5, this includes the time taken to write the response to the client, but for IIS version 6, the response buffers are sent asynchronously, and so the time taken to write the response to the client is not included.
The IIS version is 7.5 on Windows Server 2008 R2 running an ASP.NET 4.5 web application and debug is set to false.
I apologize for not addressing your question directly, but there is something that might help with your problem:
There's a minBytesPerSecond setting for setting a lower bound on transmission speed in the <webLimits> section. I suspect it was originally added as a defense against SlowLoris, but you can bump the value up to kick out users whose pipe can't support your site. The default value is 240, which is pretty low.
Any thoughts on why I might be getting tons of "hangs" when trying to download a file via HTTP, based on the following?
Server is IIS 6
File being downloaded is a binary file, rather than a web page
Several clients hang, including TrueUpdate and FlexNet web updating packages, as well as custom .NET app that just does basic HttpWebRequest/HttpWebResponse logic and downloads using a response stream
IIS log file signature when success is 200 0 0 (sc-status sc-substatus sc-win32-status)
For failure, error signature is 200 0 64
sc-win32-status of 64 is "the specified network name is no longer available"
I can point firefox at the URL and download successfully every time (perhaps some retry logic is happening under the hood)
At this point, it seems like either there's something funky with my server that it's throwing these errors, or that this is just normal network behavior and I need to use (or write) a client that is more resilient to the failures.
Any thoughts?
Perhaps your issue was a low level networking issue with the ISP as you speculated in your reply comment. I am experiencing a similar problem with IIS and some mysterious 200 0 64 lines appearing in the log file, which is how I found this post. For the record, this is my understanding of sc-win32-status=64; I hope someone can correct me if I'm wrong.
sc-win32-status 64 means “The specified network name is no longer available.”
After IIS has sent the final response to the client, it waits for an ACK message from the client.
Sometimes clients will reset the connection instead of sending the final ACK back to server. This is not a graceful connection close, so IIS logs the “64” code to indicate an interruption.
Many clients will reset the connection when they are done with it, to free up the socket instead of leaving it in TIME_WAIT/CLOSE_WAIT.
Proxies may have a tendancy to do this more often than individual clients.
I've spent two weeks investigating this issue. For me I had the scenario in which intermittent random requests were being prematurely terminated. This was resulting in IIS logs with status code 200, but with a win32-status of 64.
Our infrastructure includes two Windows IIS servers behind two NetScaler load balancers in HA mode.
In my particular case, the problem was that the NetScaler had a feature called "Intergrated Caching" turned on (http://support.citrix.com/proddocs/topic/ns-optimization-10-5-map/ns-IC-gen-wrapper-10-con.html).
After disabling this feature, the request interruptions ceased. And the site operated normally. I'm not sure how or why this was causing a problem, but there it is.
If you use a proxy or a load balancer, do some investigation of what features they have turned on. For me the cause was something between the client and the server interrupting the requests.
I hope that this explanation will at least save someone else's time.
Check the headers from the server, especially content-type, and content-length, it's possible that your clients don't recognize the format of the binary file and hang while waiting for bytes that never come, or maybe they close the underlying TCP connection, which may cause IIS to log the win32 status 64.
Spent three days on this.
It was the timeout that was set to 4 seconds (curl php request).
Solution was to increase the timeout setting:
//curl_setopt($ch, CURLOPT_TIMEOUT, 4); // times out after 4s
curl_setopt($ch, CURLOPT_TIMEOUT, 60); // times out after 60s
You will have to use wireshare or network monitor to gather more data on this problem. Me think.
I suggest you put Fiddler in between your server and your download client. This should reveal the differences between Firefox and other cients.
Description of all sc-win32-status codes for reference
https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-
ERROR_NETNAME_DELETED
64 (0x40)
The specified network name is no longer available.