The software I'm using is completely blind when it comes to HTTP. I'd like to add a really basic way to send GET requests over an existing TCP or TLS connection.
I've had a look at http.cap from the Wireshark wiki and it seems to me that an HTTP packet (when no Websocket upgrades are involved, that is) is simply some TCP payload. Therefore, could I simply get away by sending the following text as a hex stream over TCP? (1)
GET /download.html HTTP/1.1\r\n
Host: www.ethereal.com\r\n
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113\r\n
Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1\r\n
Accept-Language: en-us,en;q=0.5\r\n
Accept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n
Keep-Alive: 300\r\n
Connection: keep-alive\r\n
Referer: http://www.ethereal.com/development.html\r\n
\r\n
If so, would TLS cause an issue? (2) I can't see how, but I'd rather ask now than face the problem later.
Also (3) Is the TCP header any different? Again, same reasoning as above. As far as I know it's a normal TCP packet.
Thanks in advance for your input. I'm a complete HTTP beginner, if my questions are odd that's why.
Related
I've been trying to create an app which does some requests on Wizzair api, and found that there is this endpoint as /Api/search/search. While searching for flights in the browser this endpoint returns a list of flights based on the parameters provided as a json response. While accessing the same endpoint from postman and copying the same headers and body as the request I get a 428 response. That seems kinda odd, since the headers and body are exactly the same as the one in the Newtork tab in the Developer tools.
Here's a reference URL: https://wizzair.com/#/booking/select-flight/LTN/VIE/2022-07-23/2022-08-05/1/0/0/null
The added headers are:
Host: be.wizzair.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://wizzair.com/
Content-Type: application/json;charset=utf-8
X-RequestVerificationToken: <token>
Content-Length: 254
Origin: https://wizzair.com
Connection: keep-alive
Cookie: <some_cookies>
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
TE: trailers
And the body is added as raw json:
{"isFlightChange":false,"flightList":[{"departureStation":"LTN","arrivalStation":"VIE","departureDate":"2022-07-24"},{"departureStation":"VIE","arrivalStation":"LTN","departureDate":"2022-08-05"}],"adultCount":1,"childCount":0,"infantCount":0,"wdc":true}
The response from postman is:
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
Could anyone explain to me why there is a different behavior on the browser vs postman on the exact same request and if possible replicate the proper response in postman?
Don't know if it is still relevant.
But this one
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-3-7.htm","chlg_duration":30}
enter code here
is a fingerprint of akamai bot protection. AFAIK it uses JS to tell real browser from scripted requests. It stores result in cookies, obfuscating it an every possible way. Good thing is that you can copy cookies from your browser session, and that way have several requests with meaningful results. After that akamai starts to try to change cookies again, and you'll have to start all over.
Let's say a client makes a request like the following (pulled from iOS):
GET /test.mp4 HTTP/1.1
Host: example.com:80
Range: bytes=0-1
X-Playback-Session-Id: 3DFA3BE3-CB22-4EC5-808F-B59A735DCECE
Accept-Encoding: identity
Accept: */*
Accept-Language: en-us
Connection: keep-alive
User-Agent: AppleCoreMedia/1.0.0.11B554a (iPad; U; CPU OS 7_0_4 like Mac OS X; en_us)
There other such requests out there, I believe Chrome might test the waters by asking for blank Range.
How can the server respond to any such request so that it does not need to honor Range , but rather treat it as a standard HTTP delivery, and the client will play the file?
Sending a regular header response and the data as though the client were not asking for Range does not seem to work.
EDIT: Conversely, if the client does not request a Range, is it okay to respond with HTTP 206 with full filesize in Content-Length and also Content-Range header (which client will ignore)?
If the server does not support the Range header, it would send a normal 200 reply to send the entire file. If the server supports the Range header, it would send a 206 or 416 reply, depending on whether the requested range can be satisfied or not. This is covered in RFC 2616 Section 14.35.
It is not OK to respond with 206 if the client did not request a Range.
Try responding with HTTP 1.0 - it doesn't support range requests at all.
Maybe the client will treat such a reply more gracefully.
I have an IPad Safari sending two HTTP requests (two different PNG) files within 30 ms.
I thought that even on a Keep Alive HTTP 1.1 connection there should be a clear sequence of request/response.
What I saw is that the Safari browser sends two GET requests within 30 ms without waiting for an answer. This causes problems in some web servers.
Situation:
I have an HTML5 loading an SVG with further references to other images (like PNG and GIF). The problem does not occur on IPhone 5 but on the IPad.
Please seee this wireshark dump:
http://tinyurl.com/c7m37b9
(Frame 116/117)
IPad (1) Infos:
Version 5.1.1 (9B206)
Model MB2292FD
Safari 5.1
[GET /Licht_3.gif HTTP/1.1
...
User-Agent: Mozilla/5.0 (iPad; CPU OS 5_1_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B206 Safari/7534.48.3
Accept: */*
...
Accept-Language: de-de
Accept-Encoding: gzip, deflate
Connection: keep-alive]
[GET /Licht_3.gif HTTP/1.1
Host: 192.9.225.251:8081
...
Accept-Language: de-de
Accept-Encoding: gzip, deflate
Connection: keep-alive]
[HTTP/1.1 200 OK
...
Content-Type: image/png
Content-Length: 3921]
It looks like HTTP pipelining:
HTTP pipelining is a technique in which multiple HTTP requests are
sent on a single TCP connection without waiting for the corresponding
responses.
I think that this may be due to browser adhering to HTTP 1.1 Spec 8.2.4. which does allow client to retry the request if client is connected to server thru a middle layer and doesn't get a response from server before the connection is closed.
I have seen this error and have found following links to be helpful.
https://www.ravellosystems.com/blog/beware-http-requests-automatic-retries/
http://geek.starbean.net/?p=393
I am seeing this too from a mobile safari client. The only workaround I was able to find is to disable keepalive for the whole server in Nginx. We don't get duplicate requests and/or pipelining might not work from the Safari client when keepalive is disabled.
In Nginx, there is a keepalive_disable option, but it only works for POST requests on certain browsers, and doesn't have many options. We ended up disabling keepalive on that server using keepalive_timeout 0; I hope there is a better solution in the future.
I would think that a correct use of pipelining would submit multiple differing requests, instead of duplicate requests.
I sent the following http request to google, and it returned a bad request page. Was there any wrong in my request? I was implementing a proxy server with C++. I redirected clients' requests to servers they want to connect to. Before redirecting, I inserted "\r\nConnection: close" to the request. Was the position I inserted to wrong? Thanks. (I use "###" to surround the request)
###GET http://www.google.com.tw/ HTTP/1.1
Host: www.google.com.tw
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-tw,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Proxy-Connection: keep-alive
Connection: close
Cookie:***
###
What you have in there is not correct per spec, although I wouldn't be surprised if some servers actually responded to it (but not Google's).
Proxy-Connection is a misnomer and not needed at all.
The GET request should provide a relative path, not an absolute one. To be clear: the client does need to send a full address in the GET header, but the proxy needs to extract it and rewrite it such that GET carries the path, and Host header carries the hostname.
To try a couple of simple experiments, simply telnet google.com 80 and copy paste your request followed by few CRLF's.
I experience strange behavior when doing HTTP requests through sockets, here the request:
POST https://example.com:443/service/XMLSelect HTTP/1.1
Content-Length: 10926
Host: example.com
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Authorization: Basic XXX
SOAPAction: http://example.com/SubmitXml
Later on there goes body of my request with given content length.
After that I receive something like:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
Date: Tue, 30 Mar 2010 06:13:52 GMT
So everything seem to be fine here. I read all contents from network stream and successfully receive response. But my socket which I'm doing polling on switches it's modes like that:
write ( i write headers and request here )
read ( after headers sent i begin to receive response )
write ( STRANGE BEHAVIOUR HERE. WHY? here i send nothing really )
read ( here it switches to read back again )
last two steps can repeat several times. So I want to ask what leads for socket's mode change? And in this case it's not a big problem, but when I use gzip compression in my request ( no idea how it's related ) and ask server to send gzipped response to me like this:
POST https://example.com:443/service/XMLSelect HTTP/1.1
Content-Length: 1076
Accept-Encoding: gzip
Content-Encoding: gzip
Host: example.com
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Authorization: Basic XXX
SOAPAction: http://example.com/SubmitXml
I receive response like that:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Encoding: gzip
Content-Type: text/xml;charset=utf-8
Transfer-Encoding: chunked
Date: Tue, 30 Mar 2010 07:26:33 GMT
2000
�
I receive a chunk size and GZIP header, it's all okay. And here's what is happening with my poor little socket meanwhile:
write ( i write headers and request here )
read ( after headers sent i begin to receive response )
write ( STRANGE BEHAVIOUR HERE. And it finally sits here forever waiting for me to send something! But if i refer to HTTP I don't have to send anything more! )
What can it be related to? What it wants me to send? Is it remote web server's problem or do I miss something?
PS All actual service references and login/passwords replaced with fake ones :)
A socket becomes writable whenever there's space in the socket send buffer. The OS can't really know if your application has more data to send, but knows about its internal structures, like socket buffers. You have to explicitly add/remove the socket to the write fd_set for select(2) (enable/disable EPOLLOUT event for epoll(4)). This is usually handled with a state machine, like in libevent. Also polling works best with non-blocking sockets.
Hope this helps.