asp.net cross server httpwebrequest cookie login - asp.net

I am building a sister site. I want my logged in user to be able to login to the sister site.
The user is entered into both databases and has a token (guid) that matches.
I am posting the token in a token-auth page via httpwebrequest to the sister site.
The sister sites locates the user from the database with the matching token.
(so far so good)
The token-auth page (via httpwebrequest) is supposed to set a cookie that my forms authentication checks. (Then the page does a redirect to the sister site and user should be logged in.)
The problem is the last part. The cookie is not being set by the token-auth page via httpwebrequest.
Thus, forms authentication fails and the user login appears.
I see the cookie from the httpwebrequest via the CookieContainer; however it's not being saved to the cookies on the computer... and then the authentication on the redirect fails.
Anyone know how to get the cookies to save via httpwebrequest? This should be possible right?
Here's some code:
The HttpWebRequest page (on load)
Dim baseURL As String = "http://localhost:5894"
Dim poststring As String = String.Format("token={0}", u.toolkit_token)
Dim url As String = baseURL & "/GetAuthToken.aspx"
Dim cookies As CookieContainer = New CookieContainer()
Dim req As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
req.Accept = "*/*"
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"
req.Method = "POST"
req.ContentType = "application/x-www-form-urlencoded"
req.AllowAutoRedirect = False
req.CookieContainer = cookies
Dim bytedata() As Byte = Encoding.UTF8.GetBytes(poststring)
req.ContentLength = bytedata.Length
Dim rs As Stream = req.GetRequestStream()
rs.Write(bytedata, 0, bytedata.Length)
rs.Close()
Dim res As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
Dim sr As Stream = res.GetResponseStream()
Dim result As String = String.Empty
Dim reader As New StreamReader(sr)
result = reader.ReadToEnd
If result = "200" Then
Response.Redirect(baseURL)
Else
Response.Write("Error: Token Not Authorized.")
End If
The Auth-Token page
If Not Request.Form("token") Is Nothing Then
Dim u As BusinessLayer.DataContainer.oUser = Nothing
u = BusinessLayer.BusinessObject.GetUserByToken(Request.Form("token"))
If u IsNot Nothing Then
'-----Set Cookie
Dim cookie As HttpCookie = Nothing
Dim _CookieId As String = Guid.NewGuid().ToString() & "-" & Guid.NewGuid().ToString()
Call BusinessLayer.BusinessObject.UpdateUsersCookieId(_CookieId, u.id)
cookie = New HttpCookie("KeepSignedIn")
cookie.Values.Add("KeepSignedIn", "True")
cookie.Values.Remove("CookieId")
cookie.Values.Add("CookieId", _CookieId)
cookie.Expires = Now.AddYears(1)
Response.Cookies.Add(cookie)
'---------------
Response.Write("200")
End If
End If
Please advise on how to get the Auth-Token page to save it's cookies to the file system.
Is it a cross-domain issue? How else would you go about this?
I should also note that if I login from the site directly, not using the token page, the forms authentication works using the cookie. I've used this code for years. I'm certain it is not an issue with that. The cookie is just not there to authentication against when using the token-auth page.

Related

Authenticating using OATH2 on Azure app registration works fine on IIS express, but fails in IIS

I have an application that I am trying to add a layer of SSO, authenticating against Azure AD app registration. The code works fine running in IIS Express, but fails with a 400 Bad Request whenever I attempt to run it from any IIS environment, including localhost.
I have a working function that requests an authorisation code, which works with no issues and the code is returned in the querystring. The issue happens in the next stage, where I use that code to retrieve the user's Sub ID from Microsoft. This is the code I have:
'Get the base azure values
Dim AzureClient As String = ClientInfo
Dim AzureSecret As String = AzureSecret
Dim AuthRedirectUri As String = "The address of the page"
Dim TenantID As String = AzureTenantID
Dim codeVerifier = Verifier string passed in earlier function
Dim httpWReq As HttpWebRequest = DirectCast(WebRequest.Create("https://login.microsoftonline.com/" & TenantID & "/oauth2/v2.0/token"), HttpWebRequest)
httpWReq.Method = "POST"
httpWReq.Host = "login.microsoftonline.com"
httpWReq.ContentType = "application/x-www-form-urlencoded"
Dim postData As String = "client_id=" & AzureClient
postData += "&scope=openid&20email&20profile"
postData += "&code=" & Code
postData += "&redirect_uri=" & AuthRedirectUri
postData += "&grant_type=authorization_code"
postData += "&code_verifier=" & codeVerifier
postData += "&client_secret=" & AzureSecret
Dim encoding As New ASCIIEncoding()
Dim byteArray As Byte() = encoding.GetBytes(postData)
' Set the ContentLength property of the WebRequest.
httpWReq.ContentLength = byteArray.Length
Using streamWriter = New StreamWriter(httpWReq.GetRequestStream())
streamWriter.Write(postData)
End Using
' Get the response.
Dim response As WebResponse = httpWReq.GetResponse() <--- This is where the 400 Bad Request is thrown in IIS, but not IIS Express
Dim responseString As String = New StreamReader(response.GetResponseStream()).ReadToEnd()
Dim o As OAuth2AccessTokenReponse = DirectCast(JsonConvert.DeserializeObject(responseString, GetType(OAuth2AccessTokenReponse)), OAuth2AccessTokenReponse)
Dim IDToken As String = o.id_token
Dim stream = IDToken
Dim handler = New JwtSecurityTokenHandler()
Dim jsonToken = handler.ReadToken(stream)
Dim tokenS = TryCast(jsonToken, JwtSecurityToken)
Dim subID = tokenS.Claims.First(Function(claim) claim.Type = "sub").Value
Return subID
I've compared the calls coming from both environments and they are identical. I have both localhost addresses (localhost IIS address and IIS Express port) so the only differences are the port numbers used in the redirect URI field.
Does anyone have any idea what could be throwing this?
Problem sorted. After much, much digging about I found that IIS was still supporting TLS1.0 - removed that and everything works fine now.

WebClient.DownloadFile returns login page even when credentials are provided

I am trying to download a file that is behind a login page. I have looked into many ways to add credentials but the response is always the log in page's html. I have tried 2 passes with client.uploadValues or client.uploadstring to get a cookie and a single pass just providing credentials.
Public Sub downloadImage()
Dim username As String = "xxxxx"
Dim password As String = "xxxxx"
Using client As New WebClient()
client.Credentials = New NetworkCredential(username, password)
client.DownloadFile("https://retailerservices.domain.com/Image/ItemHighRes/26634/1", AppDomain.CurrentDomain.BaseDirectory & "test.jpg")
End Using
End Sub
Any recommendations?

Posted Form data using HtppWebRequest has not effect

I came across a website which seems simple enough that I was pretty confident that I will be able to read its data using HttpWebRequest and will be able to do the GET and POST requests. The GET requests are working fine. POST request also not generating any error but still the posted form data has no effect on the results which are returned. The form data posted have fields to filter the data as per dates but regardless the fact that every required data is posted the data returned is not filtered. I have added every header, form data and also added cookies with the request.
The url for the webpage is http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0
This seems like a very ordinary website but as it is an aspx page and involves ViewState and Event Validation hence this was expected not to be very easy.
My first step was to analyze the site's GET and POST using Fiddler and this amazes me because Fiddler is not capturing any traffic for this url. I have tried Charles but that itself is not capturing this url. Other then this Url Fiddler and Charles both are capturing everything else. I also like to mention that when I called the Url from a console application using HttpWebRequest then both Fiddler and Charles captured it but they are not capturing it from Chrome, FireFox and Internet Explorer 11.
So I analyzed the Network activity using Developer tool in FireFox and everything was visible which includes (Headers, Parameters and Cookies). In Chrome no cookies were present. When I inspect the cookies by creating HttpWebRequest and got the response there were no cookies present. So something is really strange going o with this website.
I have somehow managed to create a simple function to create the request and get the response. What I am doing is that I am creating a GET request first and get the Website string and extract Viewstate, EventValidation etc from it. I use this information to be used in second HttpWebRequest which is a post. Now everything works fine and I get the response but not as expected. I want the records between two give dates and I have specified these dates in the form data but still the POST request does not return the filtered data. I have mentioned the function that I have created below and I will really appreciate any suggestions that why is this happening and how to handle this. To understand this has become a challenge to me as I cannot understand why this simple website is not showing up in Fiddler. (This uses Javascript Postback)
The code may look long and scary but rather it is very simple and straight forward.
Try
' First GET Request to obtain Viewstate, Eventvalidation etc
Dim objRequest2 As Net.HttpWebRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)
objRequest2.Method = "GET"
objRequest2.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
objRequest2.Headers.Add("Accept-Encoding", "gzip, deflate")
objRequest2.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
objRequest2.KeepAlive = True
objRequest2.ContentType = "application/x-www-form-urlencoded"
objRequest2.Host = "www.bseindia.com"
objRequest2.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
objRequest2.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip
Dim LoginRes2 As Net.HttpWebResponse
Dim sr2 As IO.StreamReader
LoginRes2 = objRequest2.GetResponse()
sr2 = New IO.StreamReader(LoginRes2.GetResponseStream)
Dim getString As String = sr2.ReadToEnd()
Dim getCookieCollection = objRequest2.CookieContainer
' get the page ViewState
Dim viewStateFlag As String = "id=""__VIEWSTATE"" value="""
Dim i As Integer = getString.IndexOf(viewStateFlag) + viewStateFlag.Length
Dim j As Integer = getString.IndexOf("""", i)
Dim viewState As String = getString.Substring(i, j - i)
' get page EventValidation
Dim eventValidationFlag As String = "id=""__EVENTVALIDATION"" value="""
i = getString.IndexOf(eventValidationFlag) + eventValidationFlag.Length
j = getString.IndexOf("""", i)
Dim eventValidation As String = getString.Substring(i, j - i)
' get page EventValidation
Dim viewstateGeneratorFlag As String = "id=""__VIEWSTATEGENERATOR"" value="""
i = getString.IndexOf(viewstateGeneratorFlag) + viewstateGeneratorFlag.Length
j = getString.IndexOf("""", i)
Dim viewStateGenerator As String = getString.Substring(i, j - i)
viewState = System.Web.HttpUtility.UrlEncode(viewState)
eventValidation = System.Web.HttpUtility.UrlEncode(eventValidation)
Dim LoginRes As Net.HttpWebResponse
Dim sr As IO.StreamReader
Dim objRequest As Net.HttpWebRequest
' Second POST request to post the form data along with cookies
objRequest = DirectCast(HttpWebRequest.Create("http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"), HttpWebRequest)
Dim formDataCollection As New NameValueCollection
formDataCollection.Add("__EVENTTARGET", "")
formDataCollection.Add("__EVENTARGUMENT", "")
formDataCollection.Add("__VIEWSTATE", viewState)
formDataCollection.Add("__VIEWSTATEGENERATOR", viewStateGenerator)
formDataCollection.Add("__EVENTVALIDATION", eventValidation)
formDataCollection.Add("fmdate", "20160104")
formDataCollection.Add("eddate", "20160204")
formDataCollection.Add("hidCurrentDate", "2016/02/04")
formDataCollection.Add("ctl00_ContentPlaceHolder1_hdnCode", "")
formDataCollection.Add("txtDate", "04/01/2016")
formDataCollection.Add("ddlCalMonthDiv3", "1")
formDataCollection.Add("ddlCalYearDiv3", "2016")
formDataCollection.Add("txtTodate", "04/02/2016")
formDataCollection.Add("ddlCalMonthDiv4", "2")
formDataCollection.Add("ddlCalYearDiv4", "2016")
formDataCollection.Add("Hidden1", "")
formDataCollection.Add("ctl00_ContentPlaceHolder1_GetQuote1_smartSearch", "Enter Security Name / Code / ID")
formDataCollection.Add("btnSubmit.x", "44")
formDataCollection.Add("btnSubmit.y", "2")
Dim strFormdata As String = formDataCollection.ToString()
Dim encoding As New ASCIIEncoding
Dim postBytes As Byte() = encoding.GetBytes(strFormdata)
objRequest.Method = "POST"
objRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
objRequest.Headers.Add("Accept-Encoding", "gzip, deflate")
objRequest.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6,ur;q=0.4")
objRequest.Headers.Add("Cache-Control", "private, max-age=60")
objRequest.KeepAlive = True
objRequest.ContentType = "application/x-www-form-urlencoded"
objRequest.Host = "www.bseindia.com"
objRequest.Headers.Add("Origin", "http://www.bseindia.com")
objRequest.Referer = "http://www.bseindia.com/corporates/Insider_Trading_new.aspx?expandable=0"
objRequest.Headers.Add("Upgrade-Insecure-Requests", "1")
objRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
objRequest.ContentType = "text/html; charset=utf-8"
objRequest.Date = "Thu, 04 Feb 2016 13:42:04 GMT"
objRequest.Headers.Add("Server", "Microsoft-IIS/8.0")
objRequest.Headers.Add("Vary", "Accept-Encoding")
objRequest.Headers.Add("X-AspNet-Version", "2.0.50727")
objRequest.Headers.Add("ASP.NET", "ASP.NET")
objRequest.AutomaticDecompression = DecompressionMethods.Deflate Or DecompressionMethods.GZip
Dim gaCookies As New CookieContainer()
Dim cookie1 As New Cookie("__asc", "f673f0d5152a823bc335f575d34")
cookie1.Domain = ".bseindia.com"
cookie1.Path = "/"
gaCookies.Add(cookie1)
Dim cookie2 As New Cookie("__auc", "f673f0d5152a823bc335f575d34")
cookie2.Domain = ".bseindia.com"
cookie2.Path = "/"
gaCookies.Add(cookie2)
Dim cookie3 As New Cookie("__utma", "253454874.280640365.1454519857.1454519865.1454519865.1")
cookie3.Domain = ".bseindia.com"
cookie3.Path = "/"
gaCookies.Add(cookie3)
Dim cookie4 As New Cookie("__utmb", "253454874.1.10.1454519865")
cookie4.Domain = ".bseindia.com"
cookie4.Path = "/"
gaCookies.Add(cookie4)
Dim cookie5 As New Cookie("__utmc", "253454874")
cookie5.Domain = ".bseindia.com"
cookie5.Path = "/"
gaCookies.Add(cookie5)
Dim cookie6 As New Cookie("__utmt", "1")
cookie6.Domain = ".bseindia.com"
cookie6.Path = "/"
gaCookies.Add(cookie6)
Dim cookie7 As New Cookie("__utmz", "253454874.1454519865.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)")
cookie7.Domain = ".bseindia.com"
cookie7.Path = "/"
gaCookies.Add(cookie7)
Dim cookie8 As New Cookie("_ga", "GA1.2.280640365.1454519857")
cookie8.Domain = ".bseindia.com"
cookie8.Path = "/"
gaCookies.Add(cookie8)
Dim cookie9 As New Cookie("_gat", "1")
cookie9.Domain = ".bseindia.com"
cookie9.Path = "/"
gaCookies.Add(cookie9)
Dim postStream As Stream = objRequest.GetRequestStream()
postStream.Write(postBytes, 0, postBytes.Length)
postStream.Flush()
postStream.Close()
LoginRes = objRequest.GetResponse()
sr = New IO.StreamReader(LoginRes.GetResponseStream)
ReadWebsite = sr.ReadToEnd()
sr.Close()
sr = Nothing
LoginRes.Close()
LoginRes = Nothing
objRequest = Nothing
Exit Function
Catch ex As Exception
ReadWebsite = Nothing
End Try
Note: (Raw form data for dates without viewstate and eventvalidation)
fmdate:20160130
eddate:20160205
hidCurrentDate:2016/02/05
ctl00_ContentPlaceHolder1_hdnCode:
txtDate:04/01/2016
ddlCalMonthDiv3:1
ddlCalYearDiv3:2016
txtTodate:04/02/2016
ddlCalMonthDiv4:2
ddlCalYearDiv4:2016
Hidden1:
ctl00_ContentPlaceHolder1_GetQuote1_smartSearch:Enter Security Name / Code / ID
btnSubmit.x:55
btnSubmit.y:13
You could consider running the site in a browser and using a tool to control the browser instead directly issuing GET/POST requests. This may be easier and slightly more robust than your current approach.
E.g. Selenium Web Driver http://www.seleniumhq.org/projects/webdriver/
You would load the page, set the values of the form fields (using css style selectors to find the appropriate fields) and then click the button. You can automate all of this and get the page source (unfortunately I don't think you can get the full html in it's current state, after javascript has run, but potentially you can use the api to get the elements you need).
Api documentation: http://seleniumhq.github.io/selenium/docs/api/dotnet/
You indeed should include ALL fields from the form, including hidden ones and ASP session identifier that is stored in cookies. That way you fully emulate browser' request and achieve your goal. To show what you have to submit - http://pastebin.com/AsSABgU6

HttpWebRequest against PickupHockey.com

I'm having troubles authenticated my first scraper against the login page for my hockey pool (just looking to grab the stats once a night for dislaying in a nicer format).
//my variables
string userName = "My Pool";
string password = "hockey";
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "ctl00$MainContent$txtPoolName=" + userName + "&ctl00$MainContent$txtPassword=" + password;
byte[] postDataBytes = encoding.GetBytes(postData);
string url = "http://www.pickuphockey.com/hp/hp_login.aspx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "POST";
request.Headers.Add("Cache-Control: max-age=0");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding: gzip,deflate");
request.Headers.Add("Accept-Language: en-GB,en-US;q=0.8,en;q=0.6");
request.Headers.Add("Upgrade-Insecure-Requests: 1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36";
request.ContentLength = postDataBytes.Length;
request.ContentType = "application/x-www-form-urlencoded";
request.Headers.Add("Origin: http://www.pickuphockey.com");
request.Referer = "http://www.pickuphockey.com/hp/hp_login.aspx";
request.Host = "www.pickuphockey.com";
request.AllowAutoRedirect = false;
request.CookieContainer = new CookieContainer();
using (var stream = request.GetRequestStream())
{
stream.Write(postDataBytes, 0, postDataBytes.Length);
stream.Close();
}
//cookie container to save for the next request
var cookieContainer = new CookieContainer();
//get the cookies from the login page
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
foreach (Cookie cookie in response.Cookies)
{
cookieContainer.Add(cookie);
}
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
var r = readStream.ReadToEnd();
}
I've been working with various examples throughout Stack Overflow and am using fiddler to analyse the web calls made; trying to match all the headers in the browser login with the headers that I'm adding to my code.
Right now when I code break and look at the variable r, I see the html for a page saying that I need to enter a value for the username/password. I assume this means that I'm not setting the parameters correctly in this call but nothing I read or tweak seems to get past this error. I've masked my actual user/password but I assume that even incorrect values (if someone tries the code) should result in a different error message saying something about a bad password. Any web call gurus out there that can spot what I'm doing wrong?
Once the code works I'll wrap it up into some nicer helper class, I know it's a tad messy at the moment.
Appreciate the time.
I finally found the problem with my parameter string which needs to be formatted depending on the exact page login page I'm trying to access.
string postData = "txtPoolName=" + userName + "&txtPasswordTemp=" + password + "&txtPassword=" + password + "&Submit2=Go";
I figured this part out by using Chrome's inspector on the network traffic when I performed a successful login manually.

HTTP Post and Get to download CSV file - Error

The page keeps getting back to login. I verified that the credentials are passed to the request. No luck. Please help. I need this done in the next day or two. Anything I am missing here? I have gone this forum the past two days but just could not figure it. Sorry if this is duplicate.
Dim response As Net.HttpWebResponse
Dim myURL1 As String
Dim response As Net.HttpWebResponse
Dim myURL1 As String
Dim myURL2 As String
Dim myURL3 As String
Dim cookieJar As New CookieContainer
Dim myReq As Net.HttpWebRequest
Dim strngData as String
' Network Credentials
Dim cred As New System.Net.NetworkCredential ("myLogin", "myPassword", ".genscape.com")
' Login Page URL
myURL1 = https://apps.genscape.com/NAPowerRT/login.jsp
'URL to scrape data
myURL2 = https://apps.genscape.com/NAPowerRT/planthistory.do?
plantid=9976&numDays=1&format=0/genscape-03_26_12-03_27_12-9708.csv
' Action URL in Form with method = POST
myURL3 = "https://apps.genscape.com/NAPowerRT/j_spring_security_check"
myReq = Net.HttpWebRequest.Create(myURL1)
myCache.GetCredential(myURL1, 0, "http")
myReq.Credentials = cred
'Get Cookies
myURL1 = "https://apps.genscape.com/NAPowerRT/login.jsp"
myReq = Net.HttpWebRequest.Create(myURL1)
myReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
myReq.Method = "GET"
myReq.CookieContainer = cookieJar
myReq.KeepAlive = True
response = myReq.GetResponse
For Each tempCookie As Net.Cookie In response.Cookies
cookieJar.Add(tempCookie)
Next
myReq.GetResponse.Close()
'Sent POST data
myReq = Net.HttpWebRequest.Create(myURL3)
myReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
myReq.Method = "Post"
myReq.AllowAutoRedirect = True
myReq.ContentType = "text/html"
myReq.Credentials = cred
myReq.CookieContainer = cookieJar
'Send request to URL that has data with cookies from previous GETs
' ----- The below results in login page instead of page where the csv is located
myReq = Net.HttpWebRequest.Create(myURL2)
myReq.Credentials = cred
myReq.Method = "get"
myReq.ContentType = "text/csv"
myReq.Accept = "text/html"
myReq.AllowAutoRedirect = True
myReq.KeepAlive = True
myReq.CookieContainer = cookieJar
'Get the data from the page
response = myReq.GetResponse
Dim streamreponse As Stream = response.GetResponseStream
Dim stream As StreamReader = New StreamReader(response.GetResponseStream())
strngData = stream.ReadToEnd()
response.Close()
The NetworkCredential class is for HTTP authentication. The app you are trying to log into uses form-based authentication.
Instead of using HTTP auth, simply build a urlencoded POST request with the login and password. There is also a hidden field in the login form that may be significant.
It also looks like the company you are trying to get data from offers a proper data feed product. It is likely a better option.

Resources