I have an IPad Safari sending two HTTP requests (two different PNG) files within 30 ms.
I thought that even on a Keep Alive HTTP 1.1 connection there should be a clear sequence of request/response.
What I saw is that the Safari browser sends two GET requests within 30 ms without waiting for an answer. This causes problems in some web servers.
Situation:
I have an HTML5 loading an SVG with further references to other images (like PNG and GIF). The problem does not occur on IPhone 5 but on the IPad.
Please seee this wireshark dump:
http://tinyurl.com/c7m37b9
(Frame 116/117)
IPad (1) Infos:
Version 5.1.1 (9B206)
Model MB2292FD
Safari 5.1
[GET /Licht_3.gif HTTP/1.1
...
User-Agent: Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3
Accept: */*
...
Accept-Language: de-de
Accept-Encoding: gzip, deflate
Connection: keep-alive]
[GET /Licht_3.gif HTTP/1.1
Host: 192.9.225.251:8081
...
Accept-Language: de-de
Accept-Encoding: gzip, deflate
Connection: keep-alive]
[HTTP/1.1 200 OK
...
Content-Type: image/png
Content-Length: 3921]
It looks like HTTP pipelining:
HTTP pipelining is a technique in which multiple HTTP requests are
sent on a single TCP connection without waiting for the corresponding
responses.
I think that this may be due to browser adhering to HTTP 1.1 Spec 8.2.4. which does allow client to retry the request if client is connected to server thru a middle layer and doesn't get a response from server before the connection is closed.
I have seen this error and have found following links to be helpful.
https://www.ravellosystems.com/blog/beware-http-requests-automatic-retries/
http://geek.starbean.net/?p=393
I am seeing this too from a mobile safari client. The only workaround I was able to find is to disable keepalive for the whole server in Nginx. We don't get duplicate requests and/or pipelining might not work from the Safari client when keepalive is disabled.
In Nginx, there is a keepalive_disable option, but it only works for POST requests on certain browsers, and doesn't have many options. We ended up disabling keepalive on that server using keepalive_timeout 0; I hope there is a better solution in the future.
I would think that a correct use of pipelining would submit multiple differing requests, instead of duplicate requests.
Related
I've been trying to create an app which does some requests on Wizzair api, and found that there is this endpoint as /Api/search/search. While searching for flights in the browser this endpoint returns a list of flights based on the parameters provided as a json response. While accessing the same endpoint from postman and copying the same headers and body as the request I get a 428 response. That seems kinda odd, since the headers and body are exactly the same as the one in the Newtork tab in the Developer tools.
Here's a reference URL: https://wizzair.com/#/booking/select-flight/LTN/VIE/2022-07-23/2022-08-05/1/0/0/null
The added headers are:
Host: be.wizzair.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://wizzair.com/
Content-Type: application/json;charset=utf-8
X-RequestVerificationToken: <token>
Content-Length: 254
Origin: https://wizzair.com
Connection: keep-alive
Cookie: <some_cookies>
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
TE: trailers
And the body is added as raw json:
{"isFlightChange":false,"flightList":[{"departureStation":"LTN","arrivalStation":"VIE","departureDate":"2022-07-24"},{"departureStation":"VIE","arrivalStation":"LTN","departureDate":"2022-08-05"}],"adultCount":1,"childCount":0,"infantCount":0,"wdc":true}
The response from postman is:
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
Could anyone explain to me why there is a different behavior on the browser vs postman on the exact same request and if possible replicate the proper response in postman?
Don't know if it is still relevant.
But this one
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
enter code here
is a fingerprint of akamai bot protection. AFAIK it uses JS to tell real browser from scripted requests. It stores result in cookies, obfuscating it an every possible way. Good thing is that you can copy cookies from your browser session, and that way have several requests with meaningful results. After that akamai starts to try to change cookies again, and you'll have to start all over.
The software I'm using is completely blind when it comes to HTTP. I'd like to add a really basic way to send GET requests over an existing TCP or TLS connection.
I've had a look at http.cap from the Wireshark wiki and it seems to me that an HTTP packet (when no Websocket upgrades are involved, that is) is simply some TCP payload. Therefore, could I simply get away by sending the following text as a hex stream over TCP? (1)
GET /download.html HTTP/1.1\r\n
Host: www.ethereal.com\r\n
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113\r\n
Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1\r\n
Accept-Language: en-us,en;q=0.5\r\n
Accept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
Keep-Alive: 300\r\n
Connection: keep-alive\r\n
Referer: http://www.ethereal.com/development.html\r\n
\r\n
If so, would TLS cause an issue? (2) I can't see how, but I'd rather ask now than face the problem later.
Also (3) Is the TCP header any different? Again, same reasoning as above. As far as I know it's a normal TCP packet.
Thanks in advance for your input. I'm a complete HTTP beginner, if my questions are odd that's why.
Let's say a client makes a request like the following (pulled from iOS):
GET /test.mp4 HTTP/1.1
Host: example.com:80
Range: bytes=0-1
X-Playback-Session-Id: 3DFA3BE3-CB22-4EC5-808F-B59A735DCECE
Accept-Encoding: identity
Accept: */*
Accept-Language: en-us
Connection: keep-alive
User-Agent: AppleCoreMedia/1.0.0.11B554a (iPad; U; CPU OS 7_0_4 like Mac OS X; en_us)
There other such requests out there, I believe Chrome might test the waters by asking for blank Range.
How can the server respond to any such request so that it does not need to honor Range , but rather treat it as a standard HTTP delivery, and the client will play the file?
Sending a regular header response and the data as though the client were not asking for Range does not seem to work.
EDIT: Conversely, if the client does not request a Range, is it okay to respond with HTTP 206 with full filesize in Content-Length and also Content-Range header (which client will ignore)?
If the server does not support the Range header, it would send a normal 200 reply to send the entire file. If the server supports the Range header, it would send a 206 or 416 reply, depending on whether the requested range can be satisfied or not. This is covered in RFC 2616 Section 14.35.
It is not OK to respond with 206 if the client did not request a Range.
Try responding with HTTP 1.0 - it doesn't support range requests at all.
Maybe the client will treat such a reply more gracefully.
We have a custom webserver, written in C.
When the browser visits the page http://mydomain.com:30001/index.html,
our webserver will redirect the browser to mydomain.com:30001/login.html, by sending a http 307 response to the browser, then the browser will visit the login url.
This worked well in IE 8, and Chrome.
But in firefox(18+), when visiting the page http://mydomain.com:30001/index.html,
the browser cannot load the page(/index.html nor /login.html), and seems to be in the loading process forever. (And firebug > network panel shows nothing.)
I also tried firefox setting
Tools > Options > Advanced > General : Accessibility : [ ] "Warn me when web sites try to redirect or reload the page",
but has no effect and nothing changed.
So I wonder why firefox behaves different or there's other reason.
Update: here's firefox HTTP part captured in wireshark
1.REQUEST(when visiting http://mydomain.com:30001/index.html in the browser addressbar)
GET /index.html HTTP/1.1
Host: mydomain.com:30001
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
2.RESPONSE
HTTP/1.1 307 Temporary Redirect
Connection: keep-alive
Location: /login.html
that's all, and firefox does not fetch /login.html with another request.
By comparing responses from other servers, it looks like by adding
Content-Length: 0
in the response header solved the problem. Thanks.
According to the protocol, Content-Length can be determined by connection close if there's no Content-Length given.
My original response provides no Content-Length, means the browser is waiting the end of transfer of this response to know the right length, but setting Connection: keep-alive does not end this connection.
I guess IE or Chrome starts redirect processing right after
it knows it's a 307 redirect, while firefox does not do so until it
completes reading this response.
Here's a test case for 307 that works with Firefox: http://greenbytes.de/tech/tc/httpredirects/#t307loc. You'll have to find out what's different in your server.
I sent the following http request to google, and it returned a bad request page. Was there any wrong in my request? I was implementing a proxy server with C++. I redirected clients' requests to servers they want to connect to. Before redirecting, I inserted "\r\nConnection: close" to the request. Was the position I inserted to wrong? Thanks. (I use "###" to surround the request)
###GET http://www.google.com.tw/ HTTP/1.1
Host: www.google.com.tw
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-tw,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Proxy-Connection: keep-alive
Connection: close
Cookie:***
###
What you have in there is not correct per spec, although I wouldn't be surprised if some servers actually responded to it (but not Google's).
Proxy-Connection is a misnomer and not needed at all.
The GET request should provide a relative path, not an absolute one. To be clear: the client does need to send a full address in the GET header, but the proxy needs to extract it and rewrite it such that GET carries the path, and Host header carries the hostname.
To try a couple of simple experiments, simply telnet google.com 80 and copy paste your request followed by few CRLF's.