I've been trying to create an app which does some requests on Wizzair api, and found that there is this endpoint as /Api/search/search. While searching for flights in the browser this endpoint returns a list of flights based on the parameters provided as a json response. While accessing the same endpoint from postman and copying the same headers and body as the request I get a 428 response. That seems kinda odd, since the headers and body are exactly the same as the one in the Newtork tab in the Developer tools.
Here's a reference URL: https://wizzair.com/#/booking/select-flight/LTN/VIE/2022-07-23/2022-08-05/1/0/0/null
The added headers are:
Host: be.wizzair.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://wizzair.com/
Content-Type: application/json;charset=utf-8
X-RequestVerificationToken: <token>
Content-Length: 254
Origin: https://wizzair.com
Connection: keep-alive
Cookie: <some_cookies>
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
TE: trailers
And the body is added as raw json:
{"isFlightChange":false,"flightList":[{"departureStation":"LTN","arrivalStation":"VIE","departureDate":"2022-07-24"},{"departureStation":"VIE","arrivalStation":"LTN","departureDate":"2022-08-05"}],"adultCount":1,"childCount":0,"infantCount":0,"wdc":true}
The response from postman is:
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
Could anyone explain to me why there is a different behavior on the browser vs postman on the exact same request and if possible replicate the proper response in postman?
Don't know if it is still relevant.
But this one
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
enter code here
is a fingerprint of akamai bot protection. AFAIK it uses JS to tell real browser from scripted requests. It stores result in cookies, obfuscating it an every possible way. Good thing is that you can copy cookies from your browser session, and that way have several requests with meaningful results. After that akamai starts to try to change cookies again, and you'll have to start all over.
Related
I really want some validation after reading many websites on CORS to see if I have this
OPTIONS /frog/LOTS/upload/php.php HTTP/1.1
Host: staff.curriculum.local
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:14.0) Gecko/20100101 Firefox/14.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Origin: http://frogserver.curriculum.local
Access-Control-Request-Method: POST
Access-Control-Request-Headers: cache-control,x-requested-with
Pragma: no-cache
Cache-Control: no-cache
Here is when I think I respond with 403 ->
If my origin set on the server is not * and not that domain, I response with 403.
If I do not support POST (I may support GET), I respond with 403
If I do not support any ONE of the request headers, I respond with 403
For #1, if the domain is not supported, I will NOT send any Access Control headers in the response. I think that is ok.
for #2 and #3, I would send these headers assuming Origin request header was a match
Access-Control-Allow-Origin: http://frogserver.curriculum.local
Access-Control-Allow-Credentials: {exists and is true IF supported on server}
Access-Control-Allow-Headers: {all request headers we support and not related to incoming 'Access-Control-Request-Headers' in the request}
Access-Control-Allow-Methods: {all methods supported NOT related to incoming method POST that came in?}
Access-Control-Expose-Headers: {all headers we allow browser origin website to read}
Is my assumption correct here? or are some of the response headers related to the request headers in some way I am not seeing? (The names are similar but I don't think the behavior is tied to each other).
I would find it really odd that the request even needs to contain Access-Control-Request-Method & Access-Control-Request-Headers if we did not send back a 403 in the cases where we don't support all requested information on that endpoint, right? This is why I 'suspect' we are supposed to return a 403 along with what we do support?
thanks,
Dean
This is an example of HTTP request message transmitted to the web server. Inside headers there is an Accept header. I am confused about the meaning of it and how it is created. I thought it solely specifies my browsers capabilities to handle files. But that doesn't explain why does it differ when I visit amazon.com or joes-hardware.
There is also Content-Type header, which is a MIME for a file it requested. Same question. How does my browser know what is the type of file it requested? Is it based on the URI extension I requested or is this a generic header? This header seems to only be send in response headers. My mistake.
GET /tools.html HTTP/1.0
User-agent: Mozilla/4.75 [en] (Win98; U) Host: www.joes-hardware.com
Accept: text/html, image/gif, image/jpeg
Accept-language: en
First things first: Acceptand Accept-Language are headers defined in RFC 7231, section 5.3.2 and section 5.3.5, respectively. Together with Accept-* headers, they enable content negotiation through the client. There is an excellent article regarding content engotiation on the Mozilla Development Network. (On a side-note: The MDN is an excellent starting point for research. A lot of the articles are outdated, but the concepts are still largely valid)
The content of the Accept-Language is largely controlled by the language settings of a client's UI. Mozilla's Firefox (and - IIRC - Opera and Safari) allows to tweak these through its settings while MSIE seems to deduct them from the keyboard layouts installed in the system. There is nothing in the type of requested media that should influence this header.
The content of the Accept header on the other hand is very much depending on the context in which a resource is being requested. E.g. if you request a resource through your browser's address bar, the Accept header will pretty much read like "give me anything I can digest." If the browser is requesting a resource through an <img/>-tag, the header is going to differ in that the browser is trying to get a presentation of the requested resource that is fit for being displayed inside that tag. Same for <video/>, <audio/>, and <script/>.
Beyond that, I am not aware of any mechanisms effecting the Accept header. <a/>-tags have - unknownst to most - a type attribute which is carrying a MIME mediatype. This is, however, a fallback mechanism and should not alter Accept in any way.
As for your example, I took the liberty of requesting both sites and copying the relevant request headers:
amazon.com
GET / HTTP/1.1
Host: www.amazon.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
joes-hardware.com
GET / HTTP/1.1
Host: www.joes-hardware.com
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
The headers are no different when requesting /tools.html in the last example.
We have a custom webserver, written in C.
When the browser visits the page http://mydomain.com:30001/index.html,
our webserver will redirect the browser to mydomain.com:30001/login.html, by sending a http 307 response to the browser, then the browser will visit the login url.
This worked well in IE 8, and Chrome.
But in firefox(18+), when visiting the page http://mydomain.com:30001/index.html,
the browser cannot load the page(/index.html nor /login.html), and seems to be in the loading process forever. (And firebug > network panel shows nothing.)
I also tried firefox setting
Tools > Options > Advanced > General : Accessibility : [ ] "Warn me when web sites try to redirect or reload the page",
but has no effect and nothing changed.
So I wonder why firefox behaves different or there's other reason.
Update: here's firefox HTTP part captured in wireshark
1.REQUEST(when visiting http://mydomain.com:30001/index.html in the browser addressbar)
GET /index.html HTTP/1.1
Host: mydomain.com:30001
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
2.RESPONSE
HTTP/1.1 307 Temporary Redirect
Connection: keep-alive
Location: /login.html
that's all, and firefox does not fetch /login.html with another request.
By comparing responses from other servers, it looks like by adding
Content-Length: 0
in the response header solved the problem. Thanks.
According to the protocol, Content-Length can be determined by connection close if there's no Content-Length given.
My original response provides no Content-Length, means the browser is waiting the end of transfer of this response to know the right length, but setting Connection: keep-alive does not end this connection.
I guess IE or Chrome starts redirect processing right after
it knows it's a 307 redirect, while firefox does not do so until it
completes reading this response.
Here's a test case for 307 that works with Firefox: http://greenbytes.de/tech/tc/httpredirects/#t307loc. You'll have to find out what's different in your server.
I sent the following http request to google, and it returned a bad request page. Was there any wrong in my request? I was implementing a proxy server with C++. I redirected clients' requests to servers they want to connect to. Before redirecting, I inserted "\r\nConnection: close" to the request. Was the position I inserted to wrong? Thanks. (I use "###" to surround the request)
###GET http://www.google.com.tw/ HTTP/1.1
Host: www.google.com.tw
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-tw,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Proxy-Connection: keep-alive
Connection: close
Cookie:***
###
What you have in there is not correct per spec, although I wouldn't be surprised if some servers actually responded to it (but not Google's).
Proxy-Connection is a misnomer and not needed at all.
The GET request should provide a relative path, not an absolute one. To be clear: the client does need to send a full address in the GET header, but the proxy needs to extract it and rewrite it such that GET carries the path, and Host header carries the hostname.
To try a couple of simple experiments, simply telnet google.com 80 and copy paste your request followed by few CRLF's.
In the example at http://alx3apps.appspot.com/jsonrpc_example/ when I click the submit button, I notice (by using Firebug) that my browser submits the source:
{"params":["Hello ","Python!"],"method":"concat","id":1}
It's not posting a parameter (eg. json=[encoded string from above]), but rather just posting a raw string with the above value.
Is there an widely accepted way to replicated this via a GET request, or do I need to just urlencode the same string and include it as http://www.example.com/?json=%7b%22params%22%3a%5b%22Hello+%22%2c%22Python!%22%5d%2c%22method%22%3a%22concat%22%2c%22id%22%3a1%7d? I understand that some older browsers cannot handle a URI of more than 250 characters, but I'm OK with that.
A GET request doesn't usually transmit data in any other way besides headers, so you should pass the string encoded in the URL if you wish to use GET.
POST http://alx3apps.appspot.com/jsonrpc_example/json_service/ HTTP/1.1
Host: alx3apps.appspot.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Content-Type: application/json-rpc; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://alx3apps.appspot.com/jsonrpc_example/
Content-Length: 55
Pragma: no-cache
Cache-Control: no-cache
{"params":["Howdy","Python!"],"method":"concat","id":1}
In a normal form post the header Content-Type: application/x-www-form-urlencoded lets the server know to expect the format in key=val format whereas the page you linked sends Content-Type: application/json-rpc; charset=UTF-8. After the headers (which are terminated with the blank line) the data follows in the specified format.
You are correct that only POST submits data separately from the URI. So urlencoding it into the querystring is the only way to go, if you must use GET. (Well, I suppose you could try setting custom request headers or using cookies, but the only "widely accepted" way is to use the querystring.)