Load page then process rows in Asp.net - asp.net

I have a webpage that is a site monitoring tool for my company. Basically, it pulls a list of 150 IP addresses from a database, and checks if the webpage loads for them. This takes about 15 minutes to load, I'd like it to load the list and go 1 by 1 and update the status with text or an icon.
Here is my Function block, any way to thread this or help me get to what I need to get to?
Function SiteMonitorResults(ByVal WebAddress As String)
Try
'Code Example
Dim httpReq As HttpWebRequest = DirectCast(WebRequest.Create(WebAddress), HttpWebRequest)
httpReq.AllowAutoRedirect = False
Dim httpRes As HttpWebResponse = DirectCast(httpReq.GetResponse(), HttpWebResponse)
' Close the response.
httpRes.Close()
' Code for NotFound resources goes here.
If httpRes.StatusCode = HttpStatusCode.OK Then
Return "Online"
Else
Return "Offline"
End If
Catch ex As Exception
Return "Unknown"
End Try
End Function

Basically, I would go for something like this, using System.Threading.Tasks and System.Net.Http
( sorry for C# code )
I left out try catch for readability, but they are required, or the code will crash on the first DNS problem (for example)
public string CheckAddresses(List<string> addresses)
{
List<string> result = new List<string>();
List<Task> tasks = new List<Task>();
addresses.ForEach(address =>
{
var task = new HttpClient().GetAsync(address).ContinueWith(
res => result.Add(String.Format("{0} : {1}", address, res.Result.IsSuccessStatusCode)));
tasks.Add(task);
});
Task.WaitAll(tasks.ToArray());
return string.Join(", ", result.ToArray());
}
Hope this will help

Related

How to return a file from a ASP.Net API call?

I have read a number of answers, and been searching the web for the last few weeks to try and find an answer to this.
I have read: How to return a file (FileContentResult) in ASP.NET WebAPI
and: Returning binary file from controller in ASP.NET Web API
When I am debugging, it will just continuously callback, running the method over and over.
When running on a server (IIS), depending on the browser, there will be a message saying This site can’t be reached, or it will download the file, but indicate that the download has faield.
The code I am using is
<HttpGet, Route("api/Reports/Create")>
Public Function CreateReport() As HttpResponseMessage
Dim data = GetReportBuffer()
'This is just for testing
'To check that the report is created correctly
'I can open up temp.pdf, it is fine
Using fs As New FileStream("c:\temp\temp.pdf", FileMode.Create)
data.CopyTo(fs)
fs.Flush()
fs.Close()
End Using
Dim result = Request.CreateResponse(HttpStatusCode.OK)
'This should be StreamContent(data... but using this for testing
result.Content = New StreamContent(New FileStream("c:\temp\temp.pdf", FileMode.Open))
result.Content.Headers.ContentType = New MediaTypeHeaderValue("application/octet-stream")
result.Content.Headers.ContentDisposition = New ContentDispositionHeaderValue("attachment") With {.FileName = $"{fllId}.{ext}"}
Return result
End Function
I have also tries ByteArrayContent, but the result is exactly the same.
Is there anything I am not doing correctly?

vb.net multiple webproxy in httpwebrequest

I am currently working on a VB.net project where I need to get http responses from a certain URI but the requests needs to go through http proxy which I am perfectly fine with. The problem occurred when I realised sometimes our proxy servers are not working and then the application throws an error. I want my app to check whether the proxy is working or not, if not then I want it to take another proxy from the proxy list/array. And also, please feel free to share if you have any alternative ideas.
Currently I am using this (which is static and when it throws an error I need to manually change the proxy):
Dim proxyObject As WebProxy = New WebProxy("192.168.0.10:80")
request.Proxy = proxyObject
What I want is something like this:
If WebProxy("192.168.0.10:80") is working fine Then
Execute the response
Else
Take the next proxy address from the list/array and go back to the starting
of "If"
End If
FYI: my proxies doesn't require authentication.
I apologise if I couldn't explain it properly and to be honest I'm fairly new in VB.net.
Thanks a lot for your time and patience. Appreciate your help.
Borrowing from this question
Dim urlList As New List(Of String) 'Urls stored here
For each urlString as string in urlList
If CheckProxy(urlString) Then
'Execute the response
else
Continue For 'or something else here, mark it as bad?
End If
next
Public Shared Function CheckProxy(ByVal Proxy As String) As Boolean
Dim prx As Uri = Nothing
If Uri.TryCreate(Proxy, UriKind.Absolute, prx) Then
Return CheckProxy(prx)
ElseIf Uri.TryCreate("http://" & Proxy, UriKind.Absolute, prx) Then
Return CheckProxy(prx)
Else
Return False
End If
End Function
Public Shared Function CheckProxy(ByVal Proxy As Uri) As Boolean
Dim iProxy As Socket = New Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
iProxy.ReceiveTimeout = 500 : iProxy.SendTimeout = 500
Try
'' Connect using a timeout (1/2 second)
Dim result As IAsyncResult = iProxy.BeginConnect(Proxy.Host, Proxy.Port, Nothing, Nothing)
Dim success As Boolean = result.AsyncWaitHandle.WaitOne(500, True)
If (Not success) Then
iProxy.Close() : Return False
End If
Catch ex As Exception
Return False
End Try
Dim bytData() As Byte, strData As String
Dim iDataLen As Integer = 1024
strData = String.Format("CONNECT {0}:{1} HTTP/1.0{2}{2}", "www.google.com", 80, vbNewLine)
bytData = System.Text.ASCIIEncoding.ASCII.GetBytes(strData)
If iProxy.Connected Then
iProxy.Send(bytData, bytData.Length, SocketFlags.None)
ReDim bytData(1024)
Do
Try
iDataLen = iProxy.Receive(bytData, bytData.Length, SocketFlags.None)
Catch ex As Exception
iProxy.Close() : Return False
End Try
If iDataLen > 0 Then
strData = System.Text.ASCIIEncoding.ASCII.GetString(bytData)
Exit Do
End If
Loop
Else
Return False
End If
iProxy.Close()
Dim strAttribs() As String
strAttribs = strData.Split(" "c)
If strAttribs(1).Equals("200") Then
Return True
Else
Return False
End If
End Function

HttpException Request timed out on Ajax Calls

I have a web page that has ajax calls on different checkbox selections. At back end web method is called that fills with dataset and returns JSON string response on call. HttpException Request timed out issue is tracked in our error log. This issue is repeating on live server not locally. Almost 100000 users are accessing this page and this is the main page of the site.
I have repeated this issue locally with executionTimeout to set 5 seconds. But on live I can't understand which call is making Httprequest Time Out.
Please suggest if some one has some work around.
Sample code is below:
GetASJSON() -- called by jquery
LoadData() -- called in GetASJSON()
/****
Public Shared Function GetAsJSON() As String -- called by Ajax call from jquery
Dim ds As DataSet = LoadData()
If TotalRecords = 0 Then
ds = LoadData()
Message = "NA"
End If
Dim _serializer = New JavaScriptSerializer()
Dim jSonRes As String = String.Empty
Try
Dim objects As List(Of Object) = New List(Of Object)
Dim i As Integer = 0
For Each row As DataRow In ds.Tables(0).Rows
objects.Add(New With {})
Next
If (objects.Count.Equals(0)) Then
jSonRes = _serializer.Serialize(New With {._Status = "Error", ._Message = "No Arrangement Found"})
Else
jSonRes = _serializer.Serialize(New With {._Status = "OK", ._Data = objects, ._Message = Message})
End If
Return jSonRes
End Function
******/
Private Shared Function LoadData() As DataSet
Dim ds As DataSet
ds = ds = ManagerClass.GetByFilter(param1, param2, param3)
Return ds
End Function
Without having much details about the request timeout value in the live server or knowing what the underlying issue is, you could try to add this to the web.config in your live application which will set the request time out in seconds:
<configuration>
<system.web>
<httpRuntime executionTimeout="300" />
</system.web>
</configuration>
Also, if you are running IIS you can use the Failed Request Tracing. You can set it up to dump traces of any request taking more than X amount of time.

How to write response into Buffer instead of rendering on Screen?

I am developing app where I need to capture an Information from Webpage after giving credentials automatically. Some how I managed to do Automatic login and redirection of page. Here is my code :
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://abcd.com.au/categories/A_dfn/sdf");
HttpWebResponse res = req.GetResponse() as HttpWebResponse;
StringBuilder sb = new StringBuilder();
byte[] buf = new byte[10000];
Stream resStream = res.GetResponseStream();
string s = null;
int c = 0;
do
{
c = resStream.Read(buf, 0, buf.Length);
if (c != 0) {
s = ASCIIEncoding.ASCII.GetString(buf, 0, c);
sb.Append(s);
}
} while (c > 0);
string oldhead = "class=\"login_button\">";
string newhead = "class=\"login_button\"> <script type=\"text/javascript\">document.getElementById('btn').click()</script>";
sb.Replace(oldhead, newhead);
string oldbtn = "value=\"Submit\"";
string newbtn = "value=\"Submit\" id=\"btn\" ";
sb.Replace(oldbtn, newbtn);
string oldAction = "<form action=\"/login\" method=\"post\">";
string newAction = "<form action=\"https://abcd.com.au/login?orig_req_url=%2Fcategories/A_dfn/sdf\" method=\"post\">";
sb.Replace(oldAction, newAction);
string oldUsername = "<input id=\"login_email\" type=\"text\" name=\"user[email_address]\" class=\"textBox\" value=\"\">";
string newUserName = "<input id=\"login_email\" type=\"text\" name=\"user[email_address]\" class=\"textBox\" value=\"abc#xyz.com.au\">";
sb.Replace(oldUsername, newUserName);
string oldPass = "<input id=\"login_password\" type=\"password\" name=\"user[password]\" class=\"textBox\" value=\"\">";
string newPass = "<input id=\"login_password\" type=\"password\" name=\"user[password]\" class=\"textBox\" value=\"abc\">";
sb.Replace(oldPass,newPass);
Response.Write(sb);
This is show me expected output as I want by rendering page(Response.write(sb)). But, now I want to do same thing without redirecting to "https://abcd.com.au/login?orig_req_url=%2Fcategories/A_dfn/sdf" and want to do more stuff on this. I expect to get output of Response.Write(sb) in some buffer. Is it possible to do?
Here is example, that explains exactly what I want to do.
I am looking for an product's qty say name : Screw 15mm, this resides in page https://abcd.com.au/%2Fcategories/A_dfn/sdf.
So, I am requesting this url first, but as need login to access that page, its redirecting me to login page, filling username and password, pressing login button by using javascript,and then redirected to Originally Requested page. And on this page I want to find for that product, and return information to my web app.
All this I want to do without showing to user.
I just want to show retrieved information.
Thanks.
What you are looking for is a persisted session. Your approach towards this problem is incorrect. You are triggering the submit on the client-side. What you are trying to achieve should be done on the server-side.
The key to your scenario is to persist (store) the session & cookies set by the login page; then before your next request for the product info, inject the credential into the requesting webRequest.
Use the WebRequest object to load the login page.
Store any info (cookies) sent by the login page Response header.
create a new WebRequest object with the provided Response header, inject in userid/password.
Store any credentials returned by the Response.
Proceed to request for the quote info.
There is no generic way to do this without knowing the website you are trying to screen-scrap from. But the general step is as above. Basically, you need to create a custom class for this.
Also, you need HTMLAgilityPack to parse the HTML nodes. It is the correct method.
EDIT: Added my codes. Just so happen that I've created this class before sometime ago. So, you're in luck. However, you will need HTMLAgilityPack installed & referenced to use it. You can download HAP at: http://htmlagilitypack.codeplex.com/ If you want to do any serious screen-scraping, HAP is the de-facto standard.
Public Class clsBrowserSession
'=================================================================================================================================
'This is a special Browser Post class
' Instead of just POST to a URL as per the clsWeb.fnsPostResponse()
' clsBrowserSession allows us to LOAD a page first, persist all the cookies and variables, and then only POST to the target URL.
' The reason is that some program will drop (lets say) a SessionID as an input when you first load the page.
' and when you post, without the SessionID (variable), it will reject the POST. Thus clsBrowserSession can solve this problem.
'=================================================================================================================================
' USAGE:
' Dim voBrowserSession As New clsBrowserSession
' voBrowserSession.sbLoadPage("https://xxx.yyy.net.my/publicncdenq/index.htm")
' voBrowserSession.proFormElements("UserID") = "myID"
' voBrowserSession.proFormElements("Password") = "myPassword"
' Dim vsResponseHTML As String = voBrowserSession.Post("https://xxx.yyy.net.my/publicncdenq/index.htm")
Private vbIsPostingInProgress As Boolean
Public voCookies As System.Net.CookieCollection
Public proHTMLDoc As HtmlAgilityPack.HtmlDocument
Public proFormElements As clsFormElementCollection
Public Sub sbLoadPage(pvsURL As String)
vbIsPostingInProgress = False
fnoCreateWebRequestObject().Load(pvsURL)
End Sub
Public Function Post(pvsURL As String) As String
vbIsPostingInProgress = True
fnoCreateWebRequestObject().Load(pvsURL, "POST")
Return proHTMLDoc.DocumentNode.InnerHtml
End Function
Private Function fnoCreateWebRequestObject() As HtmlAgilityPack.HtmlWeb
Dim voWeb As New HtmlAgilityPack.HtmlWeb
voWeb.UseCookies = True
voWeb.PreRequest = New HtmlAgilityPack.HtmlWeb.PreRequestHandler(AddressOf event_OnPreRequest)
voWeb.PostResponse = New HtmlAgilityPack.HtmlWeb.PostResponseHandler(AddressOf event_OnAfterResponse)
voWeb.PreHandleDocument = New HtmlAgilityPack.HtmlWeb.PreHandleDocumentHandler(AddressOf event_OnPreHandleDocument)
Return voWeb
End Function
Private Sub sbAddPostDataTo(pvoRequest As Net.HttpWebRequest)
Dim vsPayload As String = proFormElements.fnsAssemblePostPayload()
Dim vabyteBuffer As Byte() = Text.Encoding.UTF8.GetBytes(vsPayload.ToCharArray())
pvoRequest.ContentLength = vabyteBuffer.Length
pvoRequest.ContentType = "application/x-www-form-urlencoded"
pvoRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11"
pvoRequest.GetRequestStream().Write(vabyteBuffer, 0, vabyteBuffer.Length)
End Sub
Private Sub sbAddvoCookiesTo(pvoRequest As Net.HttpWebRequest)
If (Not IsNothing(voCookies)) Then
If voCookies.Count > 0 Then pvoRequest.CookieContainer.Add(voCookies)
End If
End Sub
Private Sub sbSaveCookiesFrom(pvoResponse As Net.HttpWebResponse)
If pvoResponse.Cookies.Count > 0 Then
If IsNothing(voCookies) Then voCookies = New Net.CookieCollection
voCookies.Add(pvoResponse.Cookies)
End If
End Sub
Private Sub sbSaveHtmlDocument(pvoHTMLDocument As HtmlAgilityPack.HtmlDocument)
proHTMLDoc = pvoHTMLDocument
proFormElements = New clsFormElementCollection(proHTMLDoc)
End Sub
Protected Function event_OnPreRequest(pvoRequest As Net.HttpWebRequest) As Boolean
sbAddvoCookiesTo(pvoRequest)
If vbIsPostingInProgress Then sbAddPostDataTo(pvoRequest)
Return True
End Function
Protected Sub event_OnAfterResponse(pvoRequest As System.Net.HttpWebRequest, pvoResponse As Net.HttpWebResponse)
sbSaveCookiesFrom(pvoResponse)
End Sub
Protected Sub event_OnPreHandleDocument(pvoHTMLDocument As HtmlAgilityPack.HtmlDocument)
sbSaveHtmlDocument(pvoHTMLDocument)
End Sub
'-----------------------------------------------------------------------------------------------------
'Form Elements class
' Note: This element class will only capture (any) INPUT elements only, which should be enough
' for most cases. It can be easily modified to add other SELECT, TEXTAREA, etc voInputs
'-----------------------------------------------------------------------------------------------------
Public Class clsFormElementCollection
Inherits Dictionary(Of String, String)
Public Sub New(htmlDoc As HtmlAgilityPack.HtmlDocument)
Dim voInputs As Collections.Generic.IEnumerable(Of HtmlAgilityPack.HtmlNode) = htmlDoc.DocumentNode.Descendants("input")
For Each voInput As HtmlAgilityPack.HtmlNode In voInputs
Dim vsName = voInput.GetAttributeValue("name", "undefined")
Dim vsValue = voInput.GetAttributeValue("value", "")
If vsName <> "undefined" Then Add(vsName, vsValue)
Next
End Sub
Public Function fnsAssemblePostPayload() As String
Dim sb As New Text.StringBuilder
For Each voKeyValuePair In Me
Dim vsValue = System.Web.HttpUtility.UrlEncode(voKeyValuePair.Value)
sb.Append("&" & voKeyValuePair.Key & "=" & vsValue)
Next
Return sb.ToString.Substring(1)
End Function
End Class
End Class
Just make the above into a class object and instantiate it. The usage example is in the comment. You want the vsResponseHTML string.

ASP.NET HttpWebRequest stop sending requests, high CPU usage

I have an old ASP.NET application. We send out httpWebRequest to a remote REST server and retrieve XML back, most of the time the application works fine. Recently, we got some high CPU usage issue several times a day.
During the high CPU usage time, we monitored those httpWebRequest connections (by checking netstat for the w3wp process). At the very beginning, the connections change to "CLOSE_WAIT" status from "ESTABLISHED", then after those connections timeout, those connections disappeared one by one, and then there is no connection any more.
After resetting the IIS, when the w3wp.exe process start again, we still could not find any connections to httpWebRequest target server. So the CPU usage keep staying at high level. Even after several round of reset, it won't solve the issue, until we saw some connections start to connect to httpWebRequest target server, the CPU usage went down.
I actually thought it could be the issue of my code not handling the httpWebRequest properly, I posted another question here: How to close underlying connections after catch httpwebrequest timeout?.
As mentioned in that question, I also found lots of timeout exceptions for System.Net.HttpWebRequest.GetResponse(). We found 3500 of the same exception within 5 minutes when CPU usage is really high.
What could cuase this type of issue and what could be the medicine? Why won't the application send out request any more (since there is no connection in netstat)?
Here is the source code just in case:
System.Net.HttpWebResponse httpWebResponse = null;
System.IO.Stream stream = null;
XmlTextReader xmlTextReader = null;
try
{
System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(request);
httpWebRequest.ReadWriteTimeout = 10000;
httpWebRequest.Timeout = 10000;
httpWebRequest.KeepAlive = false;
httpWebRequest.Method = "GET";
httpWebResponse = (System.Net.HttpWebResponse)httpWebRequest.GetResponse();
stream = httpWebResponse.GetResponseStream();
xmlTextReader = new XmlTextReader(stream);
xmlTextReader.Read();
xmlDocument.Load(xmlTextReader);
//Document processing code.
//...
}
catch
{
//Catch blcok with error handle
}
finally
{
if (xmlTextReader != null)
xmlTextReader.Close();
if (httpWebResponse != null)
httpWebResponse.Close();
if (stream != null)
stream.Close();
}
From your description, it is not clear to me that your high CPU utilization is related to your outgoing HTTP requests. High CPU utilization could be due to a bug in your code, a bug in CLR, IIS, or something else. Without knowing which component is consuming the CPU, you wont be able to do anything further.
If I were you, I would first try to attach a sampling profiler to the W3WP process, and see which component is consuming the CPU. That should point you to the next steps in resolving this issue.
I would suggest you to try sending requests asynchronously to avoid blocking the main thread:
using (var client = new WebClient())
{
client.OpenReadCompleted += (sender, e) =>
{
using (var reader = XmlReader.Create(e.Result))
{
// Process the XML document here
}
};
client.OpenReadAsync(new Uri("http://www.example.com"));
}
Finding the reason for the high CPU utilization can take some time since you will have to locate the code that is causing the problem. I am working through this right now on a vb.net app that I recently developed. In the meantime, I have developed a page that has a button which an Administrative user can click to stop the W3WP.exe process. It is a great stop gap measure until the problem code can be identified and updated. Here is the code I used. Just create a .aspx page with a button that call the following code on the corresponding .aspx.vb page. The code uses the command prompt to get the Tasklist and writes this to a file. I then parse the text file for the PID of the W3WP.exe worker process. Then, I access the command prompt programmatically again to terminate the W3WP.exe process using the PID.
Imports System.Web
Partial Class TAP
Inherits System.Web.UI.Page
Protected Sub btnStop_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnStop.Click
Call thStop_IIS()
End Sub
Function thStop_IIS()
Dim varRetval As String
Dim varPID As String
dim x as long
Dim savePath As String = Request.PhysicalApplicationPath + "exports\"
If Dir(savePath + "filename.txt") = "filename.txt" Then
Kill(savePath + "filename.txt")
End If
varRetval = Shell("cmd /c tasklist > " + savePath + "filename.txt")
For x = 1 To 90000000
Next x
varPID = thParse_File_Return_PID(savePath + "filename.txt")
varRetval = Shell("cmd /c taskkill /pid " & varPID & " /F")
Return True
End Function
Function thParse_File_Return_PID(ByVal varFileToParse As String) As Integer
On Error GoTo handler_err
'Dim FILE_NAME As String = "C:\Users\Owner\Documents\test.txt"
Dim FILE_NAME As String = varFileToParse
Dim TextLine As String
Dim varPID As Integer
Dim x As Long
If System.IO.File.Exists(FILE_NAME) = True Then
Dim objReader As New System.IO.StreamReader(FILE_NAME)
Do While objReader.Peek() <> -1
'TextLine = TextLine & objReader.ReadLine() & vbNewLine
TextLine = objReader.ReadLine() & vbNewLine
If InStr(TextLine, "w3wp.exe") Then
varPID = Mid(TextLine, 31, 4)
End If
Loop
thParse_File_Return_PID = varPID
Else
thParse_File_Return_PID = 0
End If
handler_exit:
Exit Function
handler_err:
'MsgBox(Err.Number & " " & Err.Description & ":" & "thParse_File_Return_Pages")
Resume
End Function
End Class

Resources