Im trying to create a HttpRequest using asp.net with this simple code:
Dim request As HttpWebRequest = HttpWebRequest.Create(url)
request.Credentials = CredentialCache.DefaultCredentials
Dim responsee As HttpWebResponse = request.GetResponse() ''Fell
Dim dataStream As Stream = responsee.GetResponseStream()
Dim reader As New StreamReader(dataStream)
Dim html As String = reader.ReadToEnd()
reader.Close()
responsee.Close()
the code works fine, but the web sites im trying to get with this code doesnt get the full answer.
For example: Travelocity (this link will take you seaching flighs between New York and Los angeles.
what actually happens, that the request return as finish quite fast, but in the meanwhile the search is working, and just after few second later the page is finished his load.
When im using the code above, im getting the first response which is incomplete.
How can i solve this? can i set the request object with waiting time to assure the full response? or is there any other solution for sites like this?
Thank you.
Related
I have a DNN 9.2 website that I try to keep responsive to new visitors by using a KeepAlive.aspx web page. This helps when it has not been accessed in a while. My problem is I'm using a third party module that uses Razor Templates that takes 10-15 seconds to compile the first time a visitors goes to the page. It is blazing fast after that. I've written a small vb.net application that I keep running to try an hit the pages that have the Razor Templates.
Dim url As String = "Module Page address"
Dim wReq As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
Dim oWebResponse As Net.WebResponse
oWebResponse = wReq.GetResponse()
Dim respStream As Stream = oWebResponse.GetResponseStream()
Dim reader As StreamReader = New StreamReader(respStream, Encoding.UTF8)
oWebResponse.Dispose()
respStream.Dispose()
reader.Dispose()
I loop this every 60 seconds. It "seems" to help. Is this a good way to keep it active? I have NO control over the code for the module. End users get very irritated in this day and age of instant response when they have to wait 10+seconds for a simple page to load. Is there a better way?
I've been struggling with this for about a day. Basically, I want to write an excel macro to loop through a list in excel, query a webpage and retrieve some data. Ideally I just want to retrieve the data I need so I can place it in an adjacent cell, but I'd do whatever it takes at this point.
The page is ASP.net, which I have no experience in; if it was .php I could probably manage, but I’m not even sure how to post to .aspx through javascript.
I can loop through my data just fine, and once I get the data I can write it to excel, so there are two parts I’m struggling with:
Part 1 – querying the webpage
This is the page I want to query. I need to search in Property Address and retrieve data from the results. The address I'll use for the example is 400 W Church St. I thought it may be simple to submit a form like ".../ParcelSearch.aspx?name=...&value=...", but no dice.
Part 2 – grabbing the data
On the results, there is a table DetailsSummary_Master up top, with fieldsets that are defined with <legend> tags. I need the data in <legend>Municipality</legend>:
I can’t figure out what to do, loop through the <td>s? I thought maybe I could GetElementByID or maybe by tag, but I can’t seem to figure it out.
VBA
I used a few SO threads to try to figure it out so far. First, Second and Third, but I can't even seem to get it to POST properly. I'm keeping the subs separate for now.
This is what I have (stolen from the other thread) in regards to my problem:
Sub SubmitForm()
Dim objIE As Object
Dim xmlhttp As Object
Dim ieButton As Object
Dim strResponse As String
Dim strUrl As String
Dim strID As String
Dim strValue As String
Dim strSubmit As String
strID = "?name=ctl00_ctl00_ctl00_ctl00_ContentMain_ContentMain_ContentMain_ContentMain_TabContainer1_Searches_SubTabContainer1_QuickSearches_CompositAddressSearch1_AddressSearch1_ctl00_Address&value="
strValue = "400 W Church St"
strSubmit = strID & strValue
strUrl = "http://www.ocpafl.org/searches/ParcelSearch.aspx"
Set objIE = CreateObject("InternetExplorer.Application")
objIE.navigate "about:blank"
Set xmlhttp = CreateObject("MSXML2.ServerXMLHTTP")
'~~> Indicates that page that will receive the request and the type of request being submitted
xmlhttp.Open "POST", "http://www.ocpafl.org/searches/ParcelSearch.aspx", False
'~~> Indicate that the body of the request contains form data
xmlhttp.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
'~~> Send the data as name/value pairs
xmlhttp.Send "strSubmit"
strResponse = xmlhttp.responseText
objIE.navigate strUrl
objIE.Visible = True
Do While objIE.readystate <> 4
DoEvents
Loop
objIE.document.Write strResponse
Set xmlhttp = Nothing
End Sub
I don't actually need to run it through IE, I'd like to run it all hidden. I'm running this on Excel 2007 at work, but I have 2010 at home. We also have ridiculous IE8, so the less of that, the better. And I can loop or use an array, but I just can't seem to interface with the query. Any help would be greatly appreciated.
For making the query, given the complexity of the form fields that the ASPX page is expecting on postback, you might find it easier to control the browser when making this call. It will be rather slow, but it should work.
A fairly reliable tool for this is Selenium, and there are plugins to control Selenium from Excel VBA.
Edit: This Excel VBA code snippet should read out "Municipality Orlando". You need to parameterize the below code and add cases for error conditions for your final version to query by any street address to get its municipality. This should get you started though. I used Selenium IDE with Firefox to generate the VBA code based on recording user actions, and then came up with an XPath query to grab the text.
Dim selenium As New SeleniumWrapper.WebDriver
selenium.Start "firefox", "http://www.ocpafl.org/searches/ParcelSearch.aspx"
selenium.setImplicitWait 5000
selenium.setImplicitWait 5000
selenium.Open "/searches/ParcelSearch.aspx"
selenium.Click "id=popup_ok"
selenium.Type "id=ctl00_ctl00_ctl00_ctl00_ContentMain_ContentMain_ContentMain_ContentMain_TabContainer1_Searches_SubTabContainer1_QuickSearches_CompositAddressSearch1_AddressSearch1_ctl00_Address", "400 W Church St"
selenium.Click "id=ctl00_ctl00_ctl00_ctl00_ContentMain_ContentMain_ContentMain_ContentMain_TabContainer1_Searches_SubTabContainer1_QuickSearches_PropertyNameSearch1_ctl00"
selenium.Click "id=ctl00_ctl00_ctl00_ctl00_ContentMain_ContentMain_ContentMain_ContentMain_TabContainer1_Searches_SubTabContainer1_QuickSearches_CompositAddressSearch1_AddressSearch1_ctl00_ActionButton1"
Dim municipalityResult As String
municipalityResult = selenium.getText("//fieldset[contains(legend,'Municipality')]")
selenium.stop
I am Trying to display a website portion on my web page.and for this i dnt want to use iframe.But any other idea which can get data from a website and then display it on my web page like.
this is website
http://www.sugaronline.com/
and want to display this portion on m y web page
http://i.stack.imgur.com/CVsrh.jpg
Please any one tell me is there any way to do this except using iframe?
Update
Shared Function GetHtmlPage(ByVal strURL As String) As String
Dim strResult As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest = HttpWebRequest.Create(strURL)
objResponse = objRequest.GetResponse()
Using sr As New StreamReader(objResponse.GetResponseStream())
strResult = sr.ReadToEnd()
sr.Close()
End Using
Return strResult
End Function
Dim responses As String = GetHtmlPage(theurl)
Refer existing posts
Using HTTPWebRequest in ASP.NET VB
or simply Google this
If you are using PHP, try using CURL, it is a server side request which fetches the data from the target site and then you can embed it in your page.
On the other hand, if you are using .Net, you can use httpwebrequest to do the same.
In short, you will have to make a server side request from your server to the target website to fetch the data and you can embed that data on your page as if it has come from your website.
Please check out this post
How to use httpwebrequest to pull image from website to local file
Here all you need to do is
when you have the image bytes ready, you need to write those bytes to response but before that, you will have to send proper headers to let the browser understand that you are now going to send the image.
When using HttpWebRequest to submit xml post data to a remote server, it inserts exclamation marks throughout the xml data when it sends it, causing the SOAP server to reject it and abort the connection. I have a HTML form submitter and it submits the xml in a textarea just fine without errors.
Here is my basic page logic for submitting the xml data, I have tried using the byte array as well but got the same errors.
Dim submitPage As HttpWebRequest = WebRequest.Create(requestUrl)
submitPage.Method = "POST"
Dim postData As StringBuilder = New StringBuilder()
postData.Append("Submit+XML=Submit&xmldata=" & System.Web.HttpUtility.UrlEncode(CompiledXML))
Dim writer As StreamWriter = Nothing
submitPage.ContentLength = postData.ToString().Length
writer = New StreamWriter(submitPage.GetRequestStream())
writer.Write(postData.ToString())
writer.Close()
So what happens here is is takes my XML string and puts it in the post but it also puts in exclamation marks about every 10 lines.
I have been working on this problem a couple days with no luck.
This was a bug in PHP on the remote server, where if you had a string over a certain length and you stuff it into the email function without line breaks it puts in exclamation points for you. Very odd behaviour but that was what was happening here.
I need to emulate an Excel Web Query in .net Below is the sample code. I get an Error500 when I attempt to do this in .net, however in Excel it works fine. Any ideas on what I am doing wrong? When I change the URI to a normal website it works fine, and returns the html from the page, which i what i am after. I wonder if the problem lies from the fact that I am trying to return a datatable
Dim oHttpWebRequest As System.Net.HttpWebRequest
Dim oStream As System.IO.Stream
Dim sChunk As String
oHttpWebRequest = (System.Net.HttpWebRequest.Create("http://somesite/foo.jsp"))
Dim oHttpWebResponse As System.Net.WebResponse = oHttpWebRequest.GetResponse()
oStream = oHttpWebResponse.GetResponseStream
sChunk = New System.IO.StreamReader(oStream).ReadToEnd()
oStream.Close()
oHttpWebResponse.Close()
Here is the Query from Excel
WEB
1
http:/somesite/foo.jsp
Selection=DataTable
Formatting=None
PreFormattedTextToColumns=True
ConsecutiveDelimitersAsOne=True
SingleBlockTextImport=False
DisableDateRecognition=False
DisableRedirections=False
Edit
I am getting the error when I getReponse from the server
I found the problem I was having.
I used fiddler to figure out the headers that were being sent via excel and compared those with the headers .net was sending
http://www.fiddler2.com/Fiddler2/version.asp
I had to add the following lines of code to add these two headers in order for it to work
oHttpWebRequest.Headers.Add(HttpRequestHeader.Pragma, "no-cache")
oHttpWebRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us")