ASP.NET HttpWebRequest stop sending requests, high CPU usage - asp.net

Question

I have an old ASP.NET application. We send out httpWebRequest to a remote REST server and retrieve XML back, most of the time the application works fine. Recently, we got some high CPU usage issue several times a day.
During the high CPU usage time, we monitored those httpWebRequest connections (by checking netstat for the w3wp process). At the very beginning, the connections change to "CLOSE_WAIT" status from "ESTABLISHED", then after those connections timeout, those connections disappeared one by one, and then there is no connection any more.
After resetting the IIS, when the w3wp.exe process start again, we still could not find any connections to httpWebRequest target server. So the CPU usage keep staying at high level. Even after several round of reset, it won't solve the issue, until we saw some connections start to connect to httpWebRequest target server, the CPU usage went down.
I actually thought it could be the issue of my code not handling the httpWebRequest properly, I posted another question here: How to close underlying connections after catch httpwebrequest timeout?.
As mentioned in that question, I also found lots of timeout exceptions for System.Net.HttpWebRequest.GetResponse(). We found 3500 of the same exception within 5 minutes when CPU usage is really high.
What could cuase this type of issue and what could be the medicine? Why won't the application send out request any more (since there is no connection in netstat)?
Here is the source code just in case:
System.Net.HttpWebResponse httpWebResponse = null;
System.IO.Stream stream = null;
XmlTextReader xmlTextReader = null;
try
{
System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(request);
httpWebRequest.ReadWriteTimeout = 10000;
httpWebRequest.Timeout = 10000;
httpWebRequest.KeepAlive = false;
httpWebRequest.Method = "GET";
httpWebResponse = (System.Net.HttpWebResponse)httpWebRequest.GetResponse();
stream = httpWebResponse.GetResponseStream();
xmlTextReader = new XmlTextReader(stream);
xmlTextReader.Read();
xmlDocument.Load(xmlTextReader);
//Document processing code.
//...
}
catch
{
//Catch blcok with error handle
}
finally
{
if (xmlTextReader != null)
xmlTextReader.Close();
if (httpWebResponse != null)
httpWebResponse.Close();
if (stream != null)
stream.Close();
}

Answer 1

From your description, it is not clear to me that your high CPU utilization is related to your outgoing HTTP requests. High CPU utilization could be due to a bug in your code, a bug in CLR, IIS, or something else. Without knowing which component is consuming the CPU, you wont be able to do anything further.
If I were you, I would first try to attach a sampling profiler to the W3WP process, and see which component is consuming the CPU. That should point you to the next steps in resolving this issue.

Answer 2

I would suggest you to try sending requests asynchronously to avoid blocking the main thread:
using (var client = new WebClient())
{
client.OpenReadCompleted += (sender, e) =>
{
using (var reader = XmlReader.Create(e.Result))
{
// Process the XML document here
}
};
client.OpenReadAsync(new Uri("http://www.example.com"));
}

Answer 3

Finding the reason for the high CPU utilization can take some time since you will have to locate the code that is causing the problem. I am working through this right now on a vb.net app that I recently developed. In the meantime, I have developed a page that has a button which an Administrative user can click to stop the W3WP.exe process. It is a great stop gap measure until the problem code can be identified and updated. Here is the code I used. Just create a .aspx page with a button that call the following code on the corresponding .aspx.vb page. The code uses the command prompt to get the Tasklist and writes this to a file. I then parse the text file for the PID of the W3WP.exe worker process. Then, I access the command prompt programmatically again to terminate the W3WP.exe process using the PID.
Imports System.Web
Partial Class TAP
Inherits System.Web.UI.Page
Protected Sub btnStop_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnStop.Click
Call thStop_IIS()
End Sub
Function thStop_IIS()
Dim varRetval As String
Dim varPID As String
dim x as long
Dim savePath As String = Request.PhysicalApplicationPath + "exports\"
If Dir(savePath + "filename.txt") = "filename.txt" Then
Kill(savePath + "filename.txt")
End If
varRetval = Shell("cmd /c tasklist > " + savePath + "filename.txt")
For x = 1 To 90000000
Next x
varPID = thParse_File_Return_PID(savePath + "filename.txt")
varRetval = Shell("cmd /c taskkill /pid " & varPID & " /F")
Return True
End Function
Function thParse_File_Return_PID(ByVal varFileToParse As String) As Integer
On Error GoTo handler_err
'Dim FILE_NAME As String = "C:\Users\Owner\Documents\test.txt"
Dim FILE_NAME As String = varFileToParse
Dim TextLine As String
Dim varPID As Integer
Dim x As Long
If System.IO.File.Exists(FILE_NAME) = True Then
Dim objReader As New System.IO.StreamReader(FILE_NAME)
Do While objReader.Peek() <> -1
'TextLine = TextLine & objReader.ReadLine() & vbNewLine
TextLine = objReader.ReadLine() & vbNewLine
If InStr(TextLine, "w3wp.exe") Then
varPID = Mid(TextLine, 31, 4)
End If
Loop
thParse_File_Return_PID = varPID
Else
thParse_File_Return_PID = 0
End If
handler_exit:
Exit Function
handler_err:
'MsgBox(Err.Number & " " & Err.Description & ":" & "thParse_File_Return_Pages")
Resume
End Function
End Class