Using server integration and .net, I post the original request to SagePay and get the NextURL fine, and so go to the payment pages... step through them OK, but then I get the error:
Server error 5006: Unable to redirect to Vendor's web site. The Vendor failed to provide a RedirectionURL.
HTTP error 500: The request was unsuccessful due to an unexpected condition encountered by the server.
But I am sending a RedirectionURL (though the docs call is RedirectURL, which is somewhat confusing - anyway, I've tried using both. This si what I'm sending back from my NotificatioURL - what's wrong?
Dim sb As New StringBuilder
sb.Append("Status=OK")
sb.Append("&StatusDetail=Fine")
sb.Append("&RedirectURL=https://mydomain.co.uk/sagepay.aspx")
Dim urlTEST As String = "https://test.sagepay.com/gateway/service/vspserver-register.vsp"
Dim urlLIVE As String = "https://live.sagepay.com/gateway/service/vspserver-register.vsp"
Try
Dim data As Byte() = Encoding.UTF8.GetBytes(sb.ToString)
Dim request As WebRequest = WebRequest.Create(urlTEST)
request.Method = "POST"
request.ContentType = "application/x-www-form-urlencoded"
request.ContentLength = data.Length
ServicePointManager.ServerCertificateValidationCallback = AddressOf ValidateRemoteSSLCertificate
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Dim stream = request.GetRequestStream()
stream.Write(data, 0, data.Length)
stream.Close()
Dim response As WebResponse = request.GetResponse()
response.Close()
Catch ex As Exception
'log error
End Try
Update: I gather this POST (unlike the initial one with the basket info) requires that the data be sent plain text key-value pairs separated by CrLf's, so I amended this to
Dim sb As New StringBuilder
sb.AppendLine("Status=OK")
sb.AppendLine("StatusDetail=Fine")
sb.Append("RedirectURL=https://mydomain.co.uk/sagepay.aspx")
but it still fails with the same errors. I also tried using, instead of the WebRequst class, the simpler:
Dim client As WebClient = New WebClient()
Dim reply As String = client.UploadString(urlTEST, sb.ToString)
But still no joy. Also tried changing the request.ContentType to "text/plain", but nope.
Come on, someone, please - this si basic to the their operations, someone must have done it....
Blimey - it never pays to be clever. This script doesn't require WebRequest or anything, just write to the simple Response object. Sigh.. as Linus said ....
I came across a website which seems simple enough that I was pretty confident that I will be able to read its data using HttpWebRequest and will be able to do the GET and POST requests. The GET requests are working fine. POST request also not generating any error but still the posted form data has no effect on the results which are returned. The form data posted have fields to filter the data as per dates but regardless the fact that every required data is posted the data returned is not filtered. I have added every header, form data and also added cookies with the request.
The url for the webpage is http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0
This seems like a very ordinary website but as it is an aspx page and involves ViewState and Event Validation hence this was expected not to be very easy.
My first step was to analyze the site's GET and POST using Fiddler and this amazes me because Fiddler is not capturing any traffic for this url. I have tried Charles but that itself is not capturing this url. Other then this Url Fiddler and Charles both are capturing everything else. I also like to mention that when I called the Url from a console application using HttpWebRequest then both Fiddler and Charles captured it but they are not capturing it from Chrome, FireFox and Internet Explorer 11.
So I analyzed the Network activity using Developer tool in FireFox and everything was visible which includes (Headers, Parameters and Cookies). In Chrome no cookies were present. When I inspect the cookies by creating HttpWebRequest and got the response there were no cookies present. So something is really strange going o with this website.
I have somehow managed to create a simple function to create the request and get the response. What I am doing is that I am creating a GET request first and get the Website string and extract Viewstate, EventValidation etc from it. I use this information to be used in second HttpWebRequest which is a post. Now everything works fine and I get the response but not as expected. I want the records between two give dates and I have specified these dates in the form data but still the POST request does not return the filtered data. I have mentioned the function that I have created below and I will really appreciate any suggestions that why is this happening and how to handle this. To understand this has become a challenge to me as I cannot understand why this simple website is not showing up in Fiddler. (This uses Javascript Postback)
The code may look long and scary but rather it is very simple and straight forward.
Try
' First GET Request to obtain Viewstate, Eventvalidation etc
Dim objRequest2 As Net.HttpWebRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)
objRequest2.Method = "GET"
objRequest2.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
objRequest2.Headers.Add("Accept-Encoding", "gzip, deflate")
objRequest2.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
objRequest2.KeepAlive = True
objRequest2.ContentType = "application/x-www-form-urlencoded"
objRequest2.Host = "www.bseindia.com"
objRequest2.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
objRequest2.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip
Dim LoginRes2 As Net.HttpWebResponse
Dim sr2 As IO.StreamReader
LoginRes2 = objRequest2.GetResponse()
sr2 = New IO.StreamReader(LoginRes2.GetResponseStream)
Dim getString As String = sr2.ReadToEnd()
Dim getCookieCollection = objRequest2.CookieContainer
' get the page ViewState
Dim viewStateFlag As String = "id=""__VIEWSTATE"" value="""
Dim i As Integer = getString.IndexOf(viewStateFlag) + viewStateFlag.Length
Dim j As Integer = getString.IndexOf("""", i)
Dim viewState As String = getString.Substring(i, j - i)
' get page EventValidation
Dim eventValidationFlag As String = "id=""__EVENTVALIDATION"" value="""
i = getString.IndexOf(eventValidationFlag) + eventValidationFlag.Length
j = getString.IndexOf("""", i)
Dim eventValidation As String = getString.Substring(i, j - i)
' get page EventValidation
Dim viewstateGeneratorFlag As String = "id=""__VIEWSTATEGENERATOR"" value="""
i = getString.IndexOf(viewstateGeneratorFlag) + viewstateGeneratorFlag.Length
j = getString.IndexOf("""", i)
Dim viewStateGenerator As String = getString.Substring(i, j - i)
viewState = System.Web.HttpUtility.UrlEncode(viewState)
eventValidation = System.Web.HttpUtility.UrlEncode(eventValidation)
Dim LoginRes As Net.HttpWebResponse
Dim sr As IO.StreamReader
Dim objRequest As Net.HttpWebRequest
' Second POST request to post the form data along with cookies
objRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)
Dim formDataCollection As New NameValueCollection
formDataCollection.Add("__EVENTTARGET", "")
formDataCollection.Add("__EVENTARGUMENT", "")
formDataCollection.Add("__VIEWSTATE", viewState)
formDataCollection.Add("__VIEWSTATEGENERATOR", viewStateGenerator)
formDataCollection.Add("__EVENTVALIDATION", eventValidation)
formDataCollection.Add("fmdate", "20160104")
formDataCollection.Add("eddate", "20160204")
formDataCollection.Add("hidCurrentDate", "2016/02/04")
formDataCollection.Add("ctl00_ContentPlaceHolder1_hdnCode", "")
formDataCollection.Add("txtDate", "04/01/2016")
formDataCollection.Add("ddlCalMonthDiv3", "1")
formDataCollection.Add("ddlCalYearDiv3", "2016")
formDataCollection.Add("txtTodate", "04/02/2016")
formDataCollection.Add("ddlCalMonthDiv4", "2")
formDataCollection.Add("ddlCalYearDiv4", "2016")
formDataCollection.Add("Hidden1", "")
formDataCollection.Add("ctl00_ContentPlaceHolder1_GetQuote1_smartSearch", "Enter Security Name / Code / ID")
formDataCollection.Add("btnSubmit.x", "44")
formDataCollection.Add("btnSubmit.y", "2")
Dim strFormdata As String = formDataCollection.ToString()
Dim encoding As New ASCIIEncoding
Dim postBytes As Byte() = encoding.GetBytes(strFormdata)
objRequest.Method = "POST"
objRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
objRequest.Headers.Add("Accept-Encoding", "gzip, deflate")
objRequest.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
objRequest.Headers.Add("Cache-Control", "private, max-age=60")
objRequest.KeepAlive = True
objRequest.ContentType = "application/x-www-form-urlencoded"
objRequest.Host = "www.bseindia.com"
objRequest.Headers.Add("Origin", "http://www.bseindia.com")
objRequest.Referer = "http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"
objRequest.Headers.Add("Upgrade-Insecure-Requests", "1")
objRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
objRequest.ContentType = "text/html; charset=utf-8"
objRequest.Date = "Thu, 04 Feb 2016 13:42:04 GMT"
objRequest.Headers.Add("Server", "Microsoft-IIS/8.0")
objRequest.Headers.Add("Vary", "Accept-Encoding")
objRequest.Headers.Add("X-AspNet-Version", "2.0.50727")
objRequest.Headers.Add("ASP.NET", "ASP.NET")
objRequest.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip
Dim gaCookies As New CookieContainer()
Dim cookie1 As New Cookie("__asc", "f673f0d5152a823bc335f575d34")
cookie1.Domain = ".bseindia.com"
cookie1.Path = "/"
gaCookies.Add(cookie1)
Dim cookie2 As New Cookie("__auc", "f673f0d5152a823bc335f575d34")
cookie2.Domain = ".bseindia.com"
cookie2.Path = "/"
gaCookies.Add(cookie2)
Dim cookie3 As New Cookie("__utma", "253454874.280640365.1454519857.1454519865.1454519865.1")
cookie3.Domain = ".bseindia.com"
cookie3.Path = "/"
gaCookies.Add(cookie3)
Dim cookie4 As New Cookie("__utmb", "253454874.1.10.1454519865")
cookie4.Domain = ".bseindia.com"
cookie4.Path = "/"
gaCookies.Add(cookie4)
Dim cookie5 As New Cookie("__utmc", "253454874")
cookie5.Domain = ".bseindia.com"
cookie5.Path = "/"
gaCookies.Add(cookie5)
Dim cookie6 As New Cookie("__utmt", "1")
cookie6.Domain = ".bseindia.com"
cookie6.Path = "/"
gaCookies.Add(cookie6)
Dim cookie7 As New Cookie("__utmz", "253454874.1454519865.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)")
cookie7.Domain = ".bseindia.com"
cookie7.Path = "/"
gaCookies.Add(cookie7)
Dim cookie8 As New Cookie("_ga", "GA1.2.280640365.1454519857")
cookie8.Domain = ".bseindia.com"
cookie8.Path = "/"
gaCookies.Add(cookie8)
Dim cookie9 As New Cookie("_gat", "1")
cookie9.Domain = ".bseindia.com"
cookie9.Path = "/"
gaCookies.Add(cookie9)
Dim postStream As Stream = objRequest.GetRequestStream()
postStream.Write(postBytes, 0, postBytes.Length)
postStream.Flush()
postStream.Close()
LoginRes = objRequest.GetResponse()
sr = New IO.StreamReader(LoginRes.GetResponseStream)
ReadWebsite = sr.ReadToEnd()
sr.Close()
sr = Nothing
LoginRes.Close()
LoginRes = Nothing
objRequest = Nothing
Exit Function
Catch ex As Exception
ReadWebsite = Nothing
End Try
Note: (Raw form data for dates without viewstate and eventvalidation)
fmdate:20160130
eddate:20160205
hidCurrentDate:2016/02/05
ctl00_ContentPlaceHolder1_hdnCode:
txtDate:04/01/2016
ddlCalMonthDiv3:1
ddlCalYearDiv3:2016
txtTodate:04/02/2016
ddlCalMonthDiv4:2
ddlCalYearDiv4:2016
Hidden1:
ctl00_ContentPlaceHolder1_GetQuote1_smartSearch:Enter Security Name / Code / ID
btnSubmit.x:55
btnSubmit.y:13
You could consider running the site in a browser and using a tool to control the browser instead directly issuing GET/POST requests. This may be easier and slightly more robust than your current approach.
E.g. Selenium Web Driver http://www.seleniumhq.org/projects/webdriver/
You would load the page, set the values of the form fields (using css style selectors to find the appropriate fields) and then click the button. You can automate all of this and get the page source (unfortunately I don't think you can get the full html in it's current state, after javascript has run, but potentially you can use the api to get the elements you need).
Api documentation: http://seleniumhq.github.io/selenium/docs/api/dotnet/
You indeed should include ALL fields from the form, including hidden ones and ASP session identifier that is stored in cookies. That way you fully emulate browser' request and achieve your goal. To show what you have to submit - http://pastebin.com/AsSABgU6
I'm having troubles authenticated my first scraper against the login page for my hockey pool (just looking to grab the stats once a night for dislaying in a nicer format).
//my variables
string userName = "My Pool";
string password = "hockey";
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "ctl00$MainContent$txtPoolName=" + userName + "&ctl00$MainContent$txtPassword=" + password;
byte[] postDataBytes = encoding.GetBytes(postData);
string url = "http://www.pickuphockey.com/hp/hp_login.aspx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.Headers.Add("Cache-Control: max-age=0");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding: gzip,deflate");
request.Headers.Add("Accept-Language: en-GB,en-US;q=0.8,en;q=0.6");
request.Headers.Add("Upgrade-Insecure-Requests: 1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36";
request.ContentLength = postDataBytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Headers.Add("Origin: http://www.pickuphockey.com");
request.Referer = "http://www.pickuphockey.com/hp/hp_login.aspx";
request.Host = "www.pickuphockey.com";
request.AllowAutoRedirect = false;
request.CookieContainer = new CookieContainer();
using (var stream = request.GetRequestStream())
{
stream.Write(postDataBytes, 0, postDataBytes.Length);
stream.Close();
}
//cookie container to save for the next request
var cookieContainer = new CookieContainer();
//get the cookies from the login page
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
foreach (Cookie cookie in response.Cookies)
{
cookieContainer.Add(cookie);
}
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
var r = readStream.ReadToEnd();
}
I've been working with various examples throughout Stack Overflow and am using fiddler to analyse the web calls made; trying to match all the headers in the browser login with the headers that I'm adding to my code.
Right now when I code break and look at the variable r, I see the html for a page saying that I need to enter a value for the username/password. I assume this means that I'm not setting the parameters correctly in this call but nothing I read or tweak seems to get past this error. I've masked my actual user/password but I assume that even incorrect values (if someone tries the code) should result in a different error message saying something about a bad password. Any web call gurus out there that can spot what I'm doing wrong?
Once the code works I'll wrap it up into some nicer helper class, I know it's a tad messy at the moment.
Appreciate the time.
I finally found the problem with my parameter string which needs to be formatted depending on the exact page login page I'm trying to access.
string postData = "txtPoolName=" + userName + "&txtPasswordTemp=" + password + "&txtPassword=" + password + "&Submit2=Go";
I figured this part out by using Chrome's inspector on the network traffic when I performed a successful login manually.
The page keeps getting back to login. I verified that the credentials are passed to the request. No luck. Please help. I need this done in the next day or two. Anything I am missing here? I have gone this forum the past two days but just could not figure it. Sorry if this is duplicate.
Dim response As Net.HttpWebResponse
Dim myURL1 As String
Dim response As Net.HttpWebResponse
Dim myURL1 As String
Dim myURL2 As String
Dim myURL3 As String
Dim cookieJar As New CookieContainer
Dim myReq As Net.HttpWebRequest
Dim strngData as String
' Network Credentials
Dim cred As New System.Net.NetworkCredential ("myLogin", "myPassword", ".genscape.com")
' Login Page URL
myURL1 = https://apps.genscape.com/NAPowerRT/login.jsp
'URL to scrape data
myURL2 = https://apps.genscape.com/NAPowerRT/planthistory.do?
plantid=9976&numDays=1&format=0/genscape-03_26_12-03_27_12-9708.csv
' Action URL in Form with method = POST
myURL3 = "https://apps.genscape.com/NAPowerRT/j_spring_security_check"
myReq = Net.HttpWebRequest.Create(myURL1)
myCache.GetCredential(myURL1, 0, "http")
myReq.Credentials = cred
'Get Cookies
myURL1 = "https://apps.genscape.com/NAPowerRT/login.jsp"
myReq = Net.HttpWebRequest.Create(myURL1)
myReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
myReq.Method = "GET"
myReq.CookieContainer = cookieJar
myReq.KeepAlive = True
response = myReq.GetResponse
For Each tempCookie As Net.Cookie In response.Cookies
cookieJar.Add(tempCookie)
Next
myReq.GetResponse.Close()
'Sent POST data
myReq = Net.HttpWebRequest.Create(myURL3)
myReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
myReq.Method = "Post"
myReq.AllowAutoRedirect = True
myReq.ContentType = "text/html"
myReq.Credentials = cred
myReq.CookieContainer = cookieJar
'Send request to URL that has data with cookies from previous GETs
' ----- The below results in login page instead of page where the csv is located
myReq = Net.HttpWebRequest.Create(myURL2)
myReq.Credentials = cred
myReq.Method = "get"
myReq.ContentType = "text/csv"
myReq.Accept = "text/html"
myReq.AllowAutoRedirect = True
myReq.KeepAlive = True
myReq.CookieContainer = cookieJar
'Get the data from the page
response = myReq.GetResponse
Dim streamreponse As Stream = response.GetResponseStream
Dim stream As StreamReader = New StreamReader(response.GetResponseStream())
strngData = stream.ReadToEnd()
response.Close()
The NetworkCredential class is for HTTP authentication. The app you are trying to log into uses form-based authentication.
Instead of using HTTP auth, simply build a urlencoded POST request with the login and password. There is also a hidden field in the login form that may be significant.
It also looks like the company you are trying to get data from offers a proper data feed product. It is likely a better option.
i am using mvaayoo api for sending the messages from my website.
i have read the documentation and do the same even then i am not able to send the messages.
i am using this sample code
string strUrl = "http://api.mVaayoo.com/mvaayooapi/MessageCompose?user=
Username:Password&senderID=mVaayoo&receipientno=919849558211&msgtxt=This is a test from mVaayoo API&state=4";
WebRequest request = HttpWebRequest.Create(strUrl);
HttpWebResponse response = (HttpWebResponse)request.EndGetResponse();
Stream s = (Stream)response.GetResponseStream();
StreamReader readStream = new StreamReader( s );
string dataString = readStream.ReadToEnd();
response.Close();
s.Close();
readStream.Close();
Help me please
Thanks,
Rajbir
I think the problem is because of the space in URL.
Use this for URL ;
string strUrl = "http://api.mVaayoo.com/mvaayooapi/MessageCompose?user=
Username:Password&senderID=mVaayoo&receipientno=919849558211&msgtxt=This%20is%20a%20test%20from%20mVaayoo%20API&state=4";
I replaced all space with %20 in the URL. I suffered with the same problem in java.. so it should work..