IIS 7.5 response does not contain setCookie when User-Agent = CasperJS - asp.net

HTTP header sent by CasperJS contains:
...
- User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) CasperJS/1.0.2+Phantomjs/1.8.2 Safari/534.34
...
Response does not contain SetCookie value!
When I change user agent manually:
PageSettings: {
userAgent: "User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.97 Safari/537.22"
}
All works fine. I'm using ASP.Net MVC.
How fix It on server side?

This is probably because the runtime determines your browser capabilities based on the user agent header. Try to turn it off like this (or similarily) to try if it helps:
// Global.asax.cs
void Application_OnBeginRequest( object sender, EventArgs e )
{
HttpApplication app = ( HttpApplication )sender;
HttpContext ctx = app.Context;
...
if (
ctx.Request != null &&
!string.IsNullOrEmpty( ctx.Request.UserAgent ) &&
ctx.Request.UserAgent.Contains( "CasperJS" )
)
ctx.Request.Browser.Adapters.Clear();
..
}
The snippet above is not necessarily a good idea - it totally removes the rendering adapter (which may possibly not be good!) for some browsers but it should be a good starting point for something more fancy.
I hope this helps, we were able to resolve some rendering issues with the snippet and I hope it also helps with the cookie issue.

I found a solve here: https://stackoverflow.com/a/4816391/1010404
I put generic.browser into App_Browsers folder.
File contains:
<browsers>
<browser refID="Default">
<capabilities>
<capability name="cookies" value="true" />
</capabilities>
</browser>
</browsers>
And all work fine.

Related

Python post requests, special character in data

I hope you can help with my issue:
I have this problematic script:
import requests
url='https://erdoterkep.nebih.gov.hu/geoserver/nebih/wfs'
r_headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0',
'Content-type' : 'text/plain'
}
search_str='*ÉGER*'
r_data = '<wfs:GetFeature xmlns:wfs="http://www.opengis.net/wfs" service="WFS" version="1.1.0" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><wfs:Query typeName="feature:KUL_RESZLET_VW" srsName="EPSG:900913"><ogc:Filter xmlns:ogc="http://www.opengis.net/ogc"><ogc:PropertyIsLike matchCase="false" wildCard="*" singleChar="." escapeChar="!"><ogc:PropertyName>KERES</ogc:PropertyName><ogc:Literal>'+search_str+'</ogc:Literal></ogc:PropertyIsLike></ogc:Filter></wfs:Query></wfs:GetFeature>'
print(requests.post(url, headers=r_headers, data=r_data, timeout=120).text)
I receive this as a response (604 characters):
<?xml version="1.0" encoding="UTF-8"?><wfs:FeatureCollection xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:wfs="http://www.opengis.net/wfs" xmlns:nebih="http://www.nebih.gov.hu/" xmlns:gml="http://www.opengis.net/gml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" numberOfFeatures="0" timeStamp="2022-08-22T16:46:45.508Z" xsi:schemaLocation="http://www.opengis.net/wfs https://erdoterkep.nebih.gov.hu:443/geoserver/schemas/wfs/1.1.0/wfs.xsd http://www.nebih.gov.hu/ https://erdoterkep.nebih.gov.hu:443/geoserver/nebih/wfs?service=WFS&version=1.1.0&request=DescribeFeatureType&typeName=nebih%3AKUL_RESZLET_VW"/>
If I do the same with postman, I get the correct result (200,956 characters):
click here to check postman screenshot
I separate the "Éger" string because ( I think ) it's cause the issue.
For example, when I use a different city name, like "Heves" it works well.
I generated a script with postman but it has the same issue:
import requests
url = "https://erdoterkep.nebih.gov.hu/geoserver/nebih/wfs"
payload = "<wfs:GetFeature xmlns:wfs=\"http://www.opengis.net/wfs\" service=\"WFS\" version=\"1.1.0\" xsi:schemaLocation=\"http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><wfs:Query typeName=\"feature:KUL_RESZLET_VW\" srsName=\"EPSG:900913\" xmlns:feature=\"http://www.nebih.gov.hu/\"><ogc:Filter xmlns:ogc=\"http://www.opengis.net/ogc\"><ogc:PropertyIsLike matchCase=\"false\" wildCard=\"*\" singleChar=\".\" escapeChar=\"!\"><ogc:PropertyName>KERES</ogc:PropertyName><ogc:Literal>*Éger*</ogc:Literal></ogc:PropertyIsLike></ogc:Filter></wfs:Query></wfs:GetFeature>"
headers = {
'Content-Type': 'text/plain'
}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)
I tried the following things:
changed the header content-type value to several things but nothing worked ( I think the server ignore that header )
Please help me, I'm stuck :-)
try :
response.text.decode("UTF-8")
it should work

I am loging into the following website via splash lua script but can't logg in

I want to log in into the https://login.starcitygames.com/ website by using splash integration with lua script. i first check in locallhost for testing them.
when detecting all the form css tags and entering log in credentials i failed to logged in.
The code are here:
function main(splash)
splash:set_custom_headers({
["user-agent"] = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36",
})
local url = splash.args.url
assert(splash:go(url))
assert(splash:wait(10))
splash:set_viewport_full()
local search_input = splash:select('input[type=text]')
search_input:send_text("(censored)#gmail.com")
local search_input = splash:select('input[name=password]')
search_input:send_text("(censored)")
assert(splash:wait(5))
local submit_button = splash:select('button[type=submit]')
submit_button:click()
assert(splash:wait(15))
return {
html = splash:html(),
png = splash.png(),
}
end
After when i run it on localhost 'http://0.0.0.0:8050/' then the following results is come and can't logged in .
May be the css tags i use is incorrect or anything .
I am new to splash lua so don't understand it.
the output is:
Try to replace local search_input = splash:select('input[type=text]') with local search_input = splash:select('input[name=username]').

How to get a response cookie with flurl

How to get a response cookie with flurl? I've searched for some references and studied on flurl.dev but still confused how to apply them. sorry I am not a programmer, I still have a lot to learn.
Simple code that i use :
var strUrl = await url
.WithHeaders(new
{
user_agent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
content_type = "application/json",
cookie = "cookie"
})
.PostJsonAsync(new
{
user = "user",
password = "password"
})
.ReceiveString();
Result : 200 OK
The problem here is that when you call ReceiveString(), you're getting the response body and effectively discarding the other aspects of the response message returned by PostJsonAsync. You can get the response, cookies, and body in separate steps like this:
var resp = await url
.WithHeaders(...)
.PostJsonAsync(...);
var cookies = resp.Cookies; // list of FlurlCookie objects
var body = await resp.GetStringAsync();

POST data into ASPX with requests. How to fix "remote mashine error" message

I'm trying to scrap data from aspx page using request with POST data.
On parsed html I'm getting an error "An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine."
I was searching for solutions a while but frankly I'm new in Python and can't really figure out what's wrong.
The ASPX has javaonclick function which opens a new window with data in html.
The code I've created is below.
Any help or suggestions would be greatly welcomed. Thank you!
import requests
from bs4 import BeautifulSoup
session = requests.Session()
url = 'http://ws1.osfi-bsif.gc.ca/WebApps/FINDAT/Insurance.aspx?T=0&LANG=E'
r=session.get(url)
soup = BeautifulSoup(r.content,'lxml')
viewstate = soup.select("#__VIEWSTATE")[0]['value']
eventvalidation = soup.select("#__EVENTVALIDATION")[0]['value']
payload = {
r'__EVENTTARGET': r'',
r'__EVENTARGUMENT': r'',
r'__LASTFOCUS': r'',
r'__VIEWSTATE': viewstate,
r'__VIEWSTATEGENERATOR': r'B2E4460D',
r'__EVENTVALIDATION': eventvalidation,
r'InsuranceWebPartManager$gwpinsuranceControl$insuranceControl$institutionType': r'radioButton1',
r'InsuranceWebPartManager$gwpinsuranceControl$insuranceControl$institutionDropDownList': r'F018',
r'InsuranceWebPartManager$gwpinsuranceControl$insuranceControl$reportTemplateDropDownList': r'C_LIFE-1',
r'InsuranceWebPartManager$gwpinsuranceControl$insuranceControl$reportDateDropDownList': r'3+-+2015',
r'InsuranceWebPartManager$gwpinsuranceControl$insuranceControl$submitButton': r'Submit'
}
HEADER = {
"Content-Type":"application/x-www-form-urlencoded",
"Content-Length":"11759",
"Host":"ws1.osfi-bsif.gc.ca",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36",
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language":"en-US,en;q=0.5",
"Cache-Control": "max-age=0",
"Accept-Encoding":"gzip, deflate",
"Connection":"keep-alive",
}
df = session.post(url, data=payload, headers=HEADER)
print df.text

Faking browser request in ASP.net C#

I'm using the code below to pull one of our 3rd party developed pages in so I can parse it as XML for my random bits of work.
Irritatingly we stil have a browser detection level set on the server that only allows certain browsers on to the site; so the question is how would I fake it so that the server thinks its a browser request?
static string GetHtmlPage(string strURL)
{
String strResult;
System.Net.WebResponse objResponse;
System.Net.WebRequest objRequest = System.Net.HttpWebRequest.Create(strURL);
objResponse = objRequest.GetResponse();
using (System.IO.StreamReader sr = new System.IO.StreamReader(objResponse.GetResponseStream()))
{
strResult = sr.ReadToEnd();
sr.Close();
}
return strResult;
}
Browser detection is done based on a header in the request to the server. All you need to do is set that header. However, with HttpWebRequest you don't set that through the headers collection but rather with the .UserAgent property.
...
System.Net.WebRequest objRequest =
System.Net.HttpWebRequest.Create(strURL);
//Pretend to be IE7
((System.Net.HttpWebRequest)objRequest).UserAgent =
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";
objResponse = objRequest.GetResponse();
...
You can use the ClientTarget attribute in the Page. E.g.
http://msdn.microsoft.com/en-us/library/system.web.ui.page.clienttarget.aspx
http://msdn.microsoft.com/en-sg/library/6379d90d(v=vs.85).aspx
Set the configuration as you wish...
E.g.
<configuration>
<system.web>
<clientTarget>
<add alias="ie5" userAgent="Mozilla/4.0 (compatible;MSIE 5.5;Windows NT 4.0)"/>
<add alias="ie4" userAgent="Mozilla/4.0 (compatible;MSIE 4.0;Windows NT 4.0)"/>
<add alias="uplevel" userAgent="Mozilla/4.0 (compatible;MSIE 4.0;Windows NT 4.0)"/>
<add alias="downlevel" userAgent="Unknown"/>
</clientTarget>
</system.web>
</configuration>
Then you can use it as follows.
<asp:Page ClientTarget="downlevel" />
This will fake the request!
I think most (if not all) browser detection is based on the User-Agent header, set by the HttpRequest.UserAgent property. I see there is a website for user-agent strings of various browsers: http://www.user-agents.org/
as with Waldens above but had to replace
objRequest.UserAgent =
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)";
with
((System.Net.HttpWebRequest)objRequest).UserAgent = "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)";
Otherwise it fell over. (I changed the browser to googlebot to evade our cookie server)

Resources