Using ASP.NET, how could I get the page source into a string?
I'm getting the error:
BC30002: Type 'WebRequest' not defined
Thanks for any help.
My code
<html>
<head><title>Get HTML Page Source</title></head>
<body>
<center>Page Source Display<br><br></center>
<%
DIM strUrl, websitesource, strHtml
Response.Write("Page Source Display<br>")
strUrl = "http://www.jb.com.br"
Response.Write(strUrl)
Dim myReq As WebRequest = WebRequest.Create(strUrl)
Dim myWebResponse As WebResponse = myReq.GetResponse()
Dim dataStream As Stream = myWebResponse.GetResponseStream()
Dim reader As New StreamReader(dataStream, System.Text.Encoding.UTF8)
Dim responseFromServer As String = reader.ReadToEnd()
%>
</body>
</html>
I am attempting to get the new Google reCaptcha working in my ASP.NET project and I am having problems getting it to be the new one "I'm not a robot".
I had the old one in there and after doing much research on the developers.google.com web site, everything looks the same (they even point me to a download of the same dll - 1.0.5). So, I got the new keys and put them in and it works but it looks just like the old reCaptcha.
Has anyone gotten the new one to work with their ASP.Net? What am I missing?
EDIT:
So playing around in a test app and searching some other web sites I found that if I create a page like this:
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
<title>reCAPTCHA demo: Simple page</title>
<script src="https://www.google.com/recaptcha/api.js" async defer></script>
</head>
<body>
<form id="form1" runat="server" action="?" method="POST">
<div>
<div class="g-recaptcha" data-sitekey="My Public Key"></div>
<br/>
<asp:Button ID="Button1" runat="server" Text="Submit" />
</div>
</form>
</body>
</html>
And then in my code-behind (Button1_Click), I do this:
Dim Success As Boolean
Dim recaptchaResponse As String = request.Form("g-recaptcha-response")
If Not String.IsNullOrEmpty(recaptchaResponse) Then
Success = True
Else
Success = False
End If
The recaptchaResponse will either be empty or filled in depending on if they are a bot or not. The issue is, I now need to take this response and send it to google with my private key so I can verify that the response was not provided by a bot, in my code-behind, but I cannot figure out how. I tried this (in place of Success = True):
Dim client As New System.Net.Http.HttpClient()
client.BaseAddress = New Uri("https://www.google.com/recaptcha/")
client.DefaultRequestHeaders.Accept.Clear()
client.DefaultRequestHeaders.Accept.Add(New Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"))
Dim response As Net.Http.HttpResponseMessage = Await client.GetAsync("api/siteverify?secret=My Private key&response=" + recaptchaResponse)
If (response.IsSuccessStatusCode) Then
Dim CaptchResponse As ReCaptchaModel = Await response.Content.ReadAsAsync(Of ReCaptchaModel)()
Success = CaptchResponse.success
Else
Success = False
End If
But, I could not figure out how to get the async stuff working and I cannot find anything on what ReCaptchaModel is, so I found another way to call a web service and get a json response and tried this instead:
Dim request As Net.WebRequest = Net.WebRequest.Create("https://www.google.com/recaptcha/")
Dim Data As String = "api/siteverify?secret=My Private Key&response=" + recaptchaResponse
request.Method = "POST"
request.ContentType = "application/json; charset=utf-8"
Dim postData As String = "{""data"":""" + Data + """}"
'get a reference to the request-stream, and write the postData to it
Using s As IO.Stream = request.GetRequestStream()
Using sw As New IO.StreamWriter(s)
sw.Write(postData)
End Using
End Using
'get response-stream, and use a streamReader to read the content
Using s As IO.Stream = request.GetResponse().GetResponseStream()
Using sr As New IO.StreamReader(s)
'decode jsonData with javascript serializer
Dim jsonData = sr.ReadToEnd()
Stop
End Using
End Using
But, this just gives me the content of the web page at https://www.google.com/recaptcha. Not what I want. The Google page isn't very useful and I am stuck on where to go. I need some help either calling the Google verify service or if anyone has found another way to do this from ASP.NET.
I had just about given up when I ran across something unrelated that made me think about it again and in a different way. In my last attempt above, I was attempting to pass the private key and recaptcha response as the data, so I tried it in the create of the WebRequest and it worked. Here is the final solution:
Using the same HTML posted above, I created a function that I can call in the button click event where I check the Page.IsValid and call this function:
Private Function IsGoogleCaptchaValid() As Boolean
Try
Dim recaptchaResponse As String = Request.Form("g-recaptcha-response")
If Not String.IsNullOrEmpty(recaptchaResponse) Then
Dim request As Net.WebRequest = Net.WebRequest.Create("https://www.google.com/recaptcha/api/siteverify?secret=My Private Key&response=" + recaptchaResponse)
request.Method = "POST"
request.ContentType = "application/json; charset=utf-8"
Dim postData As String = ""
'get a reference to the request-stream, and write the postData to it
Using s As IO.Stream = request.GetRequestStream()
Using sw As New IO.StreamWriter(s)
sw.Write(postData)
End Using
End Using
''get response-stream, and use a streamReader to read the content
Using s As IO.Stream = request.GetResponse().GetResponseStream()
Using sr As New IO.StreamReader(s)
'decode jsonData with javascript serializer
Dim jsonData = sr.ReadToEnd()
If jsonData = "{" & vbLf & " ""success"": true" & vbLf & "}" Then
Return True
End If
End Using
End Using
End If
Catch ex As Exception
'Dont show the error
End Try
Return False
End Function
I'm sure there are improvements to be made to the code, but it works. I couldn't see adding references to some JSON libraries for reading one thing I just check the string.
Thank you for sharing this. It worked for me. I went ahead and converted it to C# (since that's what I was using) and added a few things.
I changed the validation step. I split the JSON string and evaluated if success was found where it should be.
I used the ConfigurationManager to store the ReCaptcha Keys.
Finally, I changed it from using a WebRequest to using and HttpClient. This cut the code in half because I don't need to read the stream now.
Feel free to use this code as well.
private static bool IsReCaptchaValid(string response)
{
if (string.IsNullOrWhiteSpace(response))
{
return false;
}
var client = new HttpClient();
string result =
client.GetStringAsync(string.Format("{0}?secret={1}&response={2}", ConfigurationManager.AppSettings["ReCaptchaValidationLink"],
ConfigurationManager.AppSettings["ReCaptchaSecretKey"], response)).Result;
string[] split = result.Split('\"');
return split[1] == "success";
}
I took a slightly different approach, using the data-callback option and a Session parameter. The following sits within the MainContent block of the .aspx file:
<asp:ScriptManager ID="scrEnablePage" EnablePageMethods="true" runat="server" />
<asp:Panel ID="pnlCaptcha" runat="server" Visible="true">
<div class="g-recaptcha"
data-sitekey='<asp:Literal ID="litKey" runat="server" Text="<%$ AppSettings:recaptchaPublicKey%>" />'
data-callback="handleCaptcha"></div>
</asp:Panel>
<script src="https://www.google.com/recaptcha/api.js" async defer></script>
<script type="text/javascript">
function handleCaptcha(e) {
PageMethods.RecaptchaValid(e);
location.reload(true);
}
</script>
Then in the code-behind:
Private Const GoogleUrl As String = "https://www.google.com/recaptcha/api/siteverify?secret={0}&response={1}"
Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
pnlCaptcha.Visible = Not (Session("VerifiedHuman") = "True")
...
End Sub
<System.Web.Services.WebMethod(EnableSession:=True)> _
Public Shared Sub RecaptchaValid(response As String)
Dim client As New System.Net.WebClient()
Dim outcome As Dictionary(Of String, String)
Dim result As String = String.Join(vbCrLf,
{"{", """success"": true", "}"})
Dim serializer As New System.Web.Script.Serialization.JavaScriptSerializer()
Dim url As String = String.Format(GoogleUrl,
ConfigurationManager.AppSettings.Get("recaptchaPrivateKey"),
response)
Try
result = client.DownloadString(url)
Catch ex As System.Net.WebException
Exit Sub ' Comment out to default to passing
End Try
outcome = serializer.Deserialize(Of Dictionary(Of String, String))(result)
HttpContext.Current.Session("VerifiedHuman") = outcome("success")
End Sub
Now in Page_Load you can check Session("VerifiedHuman") = "True" and update your page controls accordingly, hiding the panel with the Captcha control and showing the other appropriate items.
Note that this takes the keys from Web.config, i.e.
<configuration>
<appSettings>
<add key="recaptchaPublicKey" value="..." />
<add key="recaptchaPrivateKey" value="..." />
...
</appSettings>
...
</configuration>
This adds a few things. It converts the response from Google into a Json object, it adds a timeout on the verification request, and it adds a verification of the hostname (required by Google if sending requests from multiple domains and the domains aren't listed in the Google Admin area).
Imports Newtonsoft.Json
Public Class Google
Public Class ReCaptcha
Private Const secret_key = "YOUR_SECRET_KEY"
Public Shared Function Validate(Request As HttpRequest, hostname As String) As Boolean
Dim g_captcha_response = Request.Form("g-recaptcha-response")
If Not String.IsNullOrEmpty(g_captcha_response) Then
Dim response = ExecuteVerification(g_captcha_response)
If Not response.StartsWith("ERROR:") Then
Dim json_obj = JsonConvert.DeserializeObject(Of ValidateResponse)(response)
If json_obj.success Then
If json_obj.hostname.ToLower = hostname.ToLower Then Return True
End If
End If
End If
Return False
End Function
Private Shared Function ExecuteVerification(g_captcha_response As String) As String
Dim request As Net.WebRequest = Net.WebRequest.Create("https://www.google.com/recaptcha/api/siteverify?secret=" & secret_key & "&response=" & g_captcha_response)
request.Timeout = 5 * 1000 ' 5 Seconds to avoid getting locked up
request.Method = "POST"
request.ContentType = "application/json"
Try
Dim byteArray As Byte() = Encoding.UTF8.GetBytes("")
request.ContentLength = byteArray.Length
Dim dataStream As Stream = request.GetRequestStream()
dataStream.Write(byteArray, 0, byteArray.Length)
dataStream.Close()
Dim response As Net.WebResponse = request.GetResponse()
dataStream = response.GetResponseStream()
Dim reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
reader.Close()
response.Close()
Return responseFromServer
Catch ex As Exception
Return "ERROR: " & ex.Message
End Try
End Function
Public Class ValidateResponse
Public Property success As Boolean
Public Property challenge_ts As DateTime
Public Property hostname As String
<JsonProperty("error-codes")>
Public Property error_codes As List(Of String)
End Class
End Class
End Class
So in the button's Click event, just call:
If Google.ReCaptcha.Validate(Request, Request.Url.Host) Then
' good to go
Else
' validation failed
End If
Dim url As New Uri("http://www.testpage.com")
If url.Scheme = Uri.UriSchemeHttp Then
'Create Request Object
Dim objRequest As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
'Set Request Method
objRequest.Method = WebRequestMethods.Http.[Get]
'Get response from requested url
Dim objResponse As HttpWebResponse = DirectCast(objRequest.GetResponse(), HttpWebResponse)
'Read response in stream reader
Dim reader As New StreamReader(objResponse.GetResponseStream())
Dim tmp As String = reader.ReadToEnd()
objResponse.Close()
'Set response data to container
Label1.Text = tmp
End If
How Would I only scrape part of a webpage..The code succesfulyl get the full html content.
For Example..I want to scrape eveyrthing between <div id="content"> </div>
Once you have the page's full html content in a string variable, you can use Regular Expressions over this string to return the parts you want to extract.
Since you have not provided details on what you want to extract, I will provide you with a link on how to use Regular Expressions.
A short tutorial on Regular Expressions can be found here
What i am trying to do is get my user to enter in a phone number and message and then post it to text marketer which send the message.
at the moment if i use a response.redirect the message sense..
response.redirect("http://www.textmarketer.biz/gateway/?username=*****&password=*****&message=test+message&orig=test&number=447712345678")
However, i do not want to send the user there. all i want to do it post the data to the url and that's all for now and the user stay on the current page.
Any help?
actually, you don't have to do this server side (vb), just plain html will do the trick:
<html>
<body>
<form action="http://google.com" method="post">
<input type="hidden" value="somevalue"/>
<input Type="submit" value="Submit"/>
</form>
</body>
</html>
this will post the data (and in effect, redirect) to google.com.
Maybe you could use client script (jQuery) - $.ajax() or $.post(). but I think you will face cross domain restrictions (there is a workaround but its not that clean and straightforward).
Another is using the HttpWebRequest class. This is server side and the post will originate from your server instead of the client (as what the 1st approach will do). upon calling request.GetResponse(), you can retrieve the output from the remote server and render it on your page. But if you want to post and redirect to the remote url then I guess you should use the first approach.
EDIT:
try this in VB:
Option Infer On
Imports System.Net
Imports System.Text
Public Class Test
Private Sub TESTRUN()
Dim s As HttpWebRequest
Dim enc As UTF8Encoding
Dim postdata As String
Dim postdatabytes As Byte()
s = HttpWebRequest.Create("http://www.textmarketer.biz/gateway/")
enc = New System.Text.UTF8Encoding()
postdata = "username=*****&password=*****&message=test+message&orig=test&number=447712345678"
postdatabytes = enc.GetBytes(postdata)
s.Method = "POST"
s.ContentType = "application/x-www-form-urlencoded"
s.ContentLength = postdatabytes.Length
Using stream = s.GetRequestStream()
stream.Write(postdatabytes, 0, postdatabytes.Length)
End Using
Dim result = s.GetResponse()
End Sub
End Class
update2:
a GET request using HttpWebRequest in VB.net.
Dim s As HttpWebRequest
Dim username = "username=" + HttpUtility.UrlEncode("yourusername")
Dim password = "password=" + HttpUtility.UrlEncode("yourp#assword)!==&#(*#)!##(_")
Dim message = "message=" + HttpUtility.UrlEncode("yourmessage")
Dim orig = "orig=" + HttpUtility.UrlEncode("dunno what this is")
Dim num = "number=" + HttpUtility.UrlEncode("123456")
Dim sep = "&"
Dim sb As New StringBuilder()
sb.Append(username).Append(sep).Append(password).Append(sep)
sb.Append(message).Append(sep).Append(orig).Append(sep).Append(num)
s = HttpWebRequest.Create("http://www.textmarketer.biz/gateway/?" + sb.ToString())
s.Method = "GET"
Dim result = s.GetResponse()
you have to use the webrequest class. refer http://msdn.microsoft.com/en-us/library/debx8sh9.aspx
Don't do this server side, but client side using AJAX.
The jQuery ajax library is quite good.
What are valid values for the MediaType Property on a HttpWebRequest?
I want to do something like this:
Dim url As String = HttpContext.Current.Request.Url.AbsoluteUri
Dim req As System.Net.HttpWebRequest = DirectCast(System.Net.WebRequest.Create(New Uri(url)), System.Net.HttpWebRequest)
' Add the current authentication cookie to the request
Dim cookie As HttpCookie = HttpContext.Current.Request.Cookies(FormsAuthentication.FormsCookieName)
Dim authenticationCookie As New System.Net.Cookie(FormsAuthentication.FormsCookieName, cookie.Value, cookie.Path, HttpContext.Current.Request.Url.Authority)
req.CookieContainer = New System.Net.CookieContainer()
req.CookieContainer.Add(authenticationCookie)
req.MediaType = "PRINT"
Dim res As System.Net.WebResponse = req.GetResponse()
'Read data
Dim ResponseStream As Stream = res.GetResponseStream()
'Write content into the MemoryStream
Dim resReader As New BinaryReader(ResponseStream)
Dim docStream As New MemoryStream(resReader.ReadBytes(CInt(res.ContentLength)))
Thanks.
I think this wikipedia page should give you a fairly comprehensive list of media types:
Media Types