I have an application that is working as a proxy using QTcpSocket to intercept HTTP requests. I'm trying to extend that to work for HTTPS as well, but it seems that it's not working how I anticipated. Using QHttp is not possible for me due to the uniqueness of the application.
Currently, I have something like the following:
serverConnection = new QTcpSocket();
serverConnection->setProxy(proxy);
serverConnection->connectToHost(url_hostname, url_port);
serverConnection->write(request.toAscii());
connect(serverConnection, SIGNAL(readyRead()), this, SLOT(readServerData()), Qt::DirectConnection);
I've attempted to do something quite similar using QSslSocket, but unfortunately the outcome isn't what I expected.
serverSConnection = new QSslSocket();
serverSConnection->connectToHostEncrypted(url_hostname, url_port);
if (!serverSConnection->waitForEncrypted()) {
qDebug() << "waitForEncrypted failed";
}
serverSConnection->write(request.toAscii());
connect(serverSConnection, SIGNAL(readyRead()), this, SLOT(readSServerData()), Qt::DirectConnection);
Using the QSslSocket to do the request appears to trigger:
HTTP/1.1 406 Not Acceptable
The request I'm sending through looks something along the lines of the following:
Received Request: "CONNECT www.somesslhost.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.13) Gecko/2009080315 Ubuntu/9.04 (jaunty) Firefox/3.0.13
Proxy-Connection: keep-alive
Host: www.somesslhost.com
Is there something special about sending an HTTPS request through my application using QSslSocket that I'm missing?
With current information I can only get the idea that you're doing it wrong.
Your method is fine if your client is supposed to send plain HTTP requests to your application and it is going to forward them to HTTPS server.
If your client is instead capable of using HTTPS and uses your application as a standard proxy server, you aren't supposed to connect to the server like that. You get 406 because you're trying to convince the destination server to act as a proxy between you and itself.
You can take a look at Tunneling SSL Through a WWW Proxy, I think. Basically, you should parse the incoming request, establish a ‘raw’ connection to server, respond the client with establishment confirmation and then just forward the packets forth and back.
The whole SSL negotiation is to be done between your client and destination server, proxy server is only supposed to forward encrypted packets without being able to decrypt them.
I'm confused - you say:
I have an application that is working
as a proxy using QTcpSocket to
intercept HTTP requests.
What do you mean by "intercept"? Are you trying to write an HTTP proxy application? Please be more clear & specific, and we can probably help you out.
Related
I'm working on an application which takes HTTP message to and from the routers web server.
The problem i'm facing is in the HTTP basic authentication.
RFC 7617 states:
"the server can reply with a challenge using the 401 (Unauthorized) status code"
What I've seen from the browser HTTP captures that it isn't the case for every router. For example, TPLINK TLWR840N doesn't sends me 401 and i can get the resource by simply transferring http request along with the correct credentials in the form of base64{username:pass} in the http message as shown below.
GET //main/ddos.htm?_=1572950350469 HTTP/1.1
Host: 192.168.0.1
Accept: */*
Connection: keep-alive
Referer: http://192.168.0.1
Cookie: Authorization=Basic YeRtaW46YWRtaW5AMTIz
It gives me the requested content if the password is correctly given otherwise it redirects me to the login page (why this router doesn't follow the 401 protocol?).
I have another TPLINK TL-WR841N router which doesn't take credentials (in http message) in the form of base64{username:pass} as the previous router, but instead it takes credentials in the form of base64(user):md5(password). I have two question about this router (and all routers in general)
I want to know how the router communicates the protocol for credentials to the browser so that i can embed that thing in my application. I have inspected the http messages (in the Chrome/Firefox) but couldn't found the message where the protocol is being communicated.
When i login to TPLINK TL-WR841N router, unlike the previous model, the web browser contains some SessionID in the URL, e.g. the URL shows www.192.168.0.1/SessionID/path/to/resource. I would like to know how this SessionID is communicated to the browser?
People who write router maintenance applications, as well as people who design graphics cards driver installer screens (looking at you, AMD), do not adhere to any guidelines, best practices or protocols whatsoever.
But they don't need to, either. They've written an application that happens to use HTTP, but you're not obliged to use all of HTTP. They write the frond-end as well as the back-end, so they can closely control their server as well as their client.
The client most likely is a dumb couple of HTML pages that does some requests using JavaScript.
If they were to decide that the web interface authenticates to the server with a request header that literally states LetMeIn: true, then that would work as well.
HTTP does not mandate that the server should return a 401 when that header is missing or bears false, so they don't have to.
I'm building a HTTP client(for embedded devices) and I was wondering,
If I receive a HTTP 3xx response, and in the location header I get a hostname different from the one I had in the request. Should I disconnect the TCP connection and reconnect to the new host, or I just need to send a new request with a new host header and keep the old TCP connection alive.
Thank you
It doesn't make sense to reuse the original TCP connection if you're being redirected elsewhere. If my webserver only hosts example.com and I redirect you to elsewhere.net, my webserver will probably not respond to a request for elsewhere.net.
Worse, this also potentially sets you up for a great man-in-the-middle attack if my server redirects you to http://bank.com and you reuse the same TCP connection when sending a request to bank.com. My server can maliciously respond to requests with Host: bank.com, which isn't something you want to happen.
You can't assume the original connection can be reused unless the redirect is to the same same host with the same protocol.
Persistent HTTP connections are a little tricky with the number of client/server combinations. You can avoid the complexity by just wasting time closing and re-establishing each connection:
If you're implementing a HTTP/1.0 client, Connection: keep-alive isn't something you have to implement. Compliant servers should just close the connection after every request if you don't negotiate that you support persistent connections.
If you're implementing a HTTP/1.1 client and don't want to persist the connection, just send Connection: close with your request and a HTTP/1.1 server should close the connection.
We are trying to make a secure communication between our embedded system and web server.Firstly we implement HTTP connection to in our microcontroller. I am just connecting to 80 port of my web server and send simple GET request to this port as example below :
GET /foo.php?msg=test HTTP/1.1
HOST: foo.com
My questions is,How we will turn this to HTTPS ? Which port i should connect ?
Will be any difference on structure of GET request above ? Will i have to do some encryption manually or connect to "https" link instead "http" is enuogh for secure communication.
Thanks for any information
The only difference between a HTTP request and a HTTPS request is that the first is send over a plain TCP connection while the other is send over a TLS connection, i.e.:
with HTTP you establish a TCP connection and send the request over this connection
with HTTPS you establish a TCP connection, upgrade this connection to TLS (including proper certificate validation etc!) and then send the same request as you did with HTTP over this connection.
Apart from that I recommend to either use an established library for HTTP or carefully read the standard. Although HTTP looks simply it is actually not and there are many questions here where users try to do a simply HTTP request and trip over behavior they did not expect.
For example in your case the server might send the response with chunked encoding, with content-length or simply end it with connection close. And it might wait for further requests on the same connection since HTTP/1.1 implicitly enables HTTP keep-alive. Does your code really account for all these cases?
As I understand it, in 1.1 normally the browser issues a request and the server makes a single message response. The browser will not issue a new request until it has received the response to the previous message. So what ever response it receives is always interpreted as a response to the last message. Is my understanding correct?
When I open a page in Firefox, the server application parses the following request:
HGet / http/1.1
HeaderField(Host, localhost:8080)
HeaderField(User-Agent, Mozilla/5.0 (X11; Ubuntu; Linux x86_64;rv:41.0) Gecko/20100101 Firefox/41.0)
HeaderField(Accept, text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8)
HeaderField(Accept-Language, en-GB,en;q=0.5)
HeaderField(Accept-Encoding, gzip, deflate)
HeaderField(Connection, keep-alive)
After responding with the page this would normally be followed by a /favicon.ico request and things precede as expected. But now I have inserted the following line into my javascript to enable server sent events:
var evtSource = new EventSource("/");
which produces a second request:
HGet / http/1.1
HeaderField(Host, localhost:8080)
HeaderField(User-Agent, Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0)
HeaderField(Accept, text/event-stream)
HeaderField(Accept-Language, en-GB,en;q=0.5)
HeaderField(Accept-Encoding, gzip, deflate)
HeaderField(Referer, http://localhost:8080/)
HeaderField(Connection, keep-alive)
HeaderField(Pragma, no-cache)
HeaderField(Cache-Control, no-cache
So can I now send multiple messages when ever I want from the server? (Leaving aside timeout issues) If so how does the browser know to which request, the message (coming from the server) is a response? Does it rely on the contentType header field? Should I use a different uri in the event source? As I'm learning to keep things simple I'm not using encryption, which stops me using HTTP 2. but later I intend to use https. My preference for using the same uri for the normal get and post requests as for the Server Sent Events is that I don't want to put unnecessary information in the unencrypted response line.
Edit: my confusion came from forgetting that http 1.1 browsers will open multiple connections if they need them. So in my simple setup the browser only has one connection open, it converts that one into a Server Sent Event connection and then makes future requests on a new connection with a different client side port number.
So can I now send multiple messages whenever I want from the server?
Yes.
how does the browser know to which request, the message (coming from the server) is a response?
It's not a request. It is an open HTTP connection. So the browser is only receiving events on the open connection that it is holding open.
Make sense?
I am trying to fetch an RTSP stream over HTTP using a proxy. The behavior of the Real client seems to be a bit hectic: it tries all the possible ports, methods and protocols at once. The only thing that should work is HTTP GET over port 80. Such a request is indeed issued, and is received on the server. Here's how the request looks when it is sent by the proxy to the server:
GET /SmpDsBhgRl83c52ef2-d0f4-41ac-bada-93e5350f67d1?1="1" HTTP/1.0\r\n
Connection: Keep-Alive\r\n
Host: 10.194.5.162:80\r\n
Pragma: no-cache\r\n
User-Agent: RealPlayer G2\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Accept: application/x-rtsp-tunnelled, */*\r\n
ClientID: WinNT_5.1_6.0.14.806_RealPlayer_R41UKD_en-GB_686\r\n
X-Actual-URL: rtsp://10.194.5.162:554/01.mp3\r\n
\r\n
Here's the server's response:
HTTP/1.0 200 OK\r\n
Server: RMServer 1.0\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Pragma: no-cache\r\n
x-server-ipaddress: 10.194.5.162\r\n
Content-type: audio/x-pn-realaudio\r\n
\r\n
At this point 4 more bytes arrive from the server (their values are 48 02 02 00) - and that's it, nothing more. Does the server expect anything from the client at this point, and if so - what? Does this mode of operation work at all?
Some more info on this problem: apparently, the intended mechanism of working with RTSP over HTTP built into RealPlayer is as follows:
Try to connect to the following ports: 80, 8080, 554, 7070.
(Try also to download the file directly, just for the heck of it, by issuing GET http://hostname:port/mediafilename on port 80)
For each of the above ports, create 2 connections.
Send a GET request to one of the connections to the url http://hostname:port/SmpDsBhgRl<guid>?1="1", where <guid> is, yes, a freshly created GUID. Add a header to this request called X-Actual-URL containing the original RTSP URL.
Send a POST request on the other connection, to the URL http://hostname:port/SmpDsBhgRl with the GUID above as part of the body of the request. Send a Content-Length header of 32767 bytes, to prevent the proxy from closing the connection prematurely.
Start issuing commands to the server through the POST request, and get the corresponding RTSP stream as part of the GET response.
The strange stuff (if the above isn't strange enough) is that, for example, it works with Squid, but not if you use either of the ports 3128 or 8080! Somehow, the client uses the port it connects to to decide on the order of the requests or on when a request should be canceled, but anyway, as hard to believe as it is, it works with proxy port 9090, 3129, 8081, but not with 3128 or 8080.
Update #2: Here's the source of the RealPlayer with the explanation of the above behavior. Still no solution though.
Update #3: OK, in the light of the above, the magic value of 48 02 02 00 is clear: 48 == 'h' is for HTTP_RESPONSE, the next 02 is the length of the following data, the next 02 is called POST_NOT_RECEIVED (meaning that the POST request did not reach the server within a second from the corresponding GET request).
Update #4: This behavior (i.e. POST requests with huge Content-Length) is also characteristic of an ActiveX used by WebEx (and, possibly, many other web apps that need an open channel to the server).
First, you might want to read this:
http://developer.apple.com/quicktime/icefloe/dispatch028.html
Second, the HTTP requests (both GET and POST) need to be formatted so that they get proxied properly. I've seen proxies that insist on caching too much of the POST request, preventing it from reaching the server. Those proxies are buggy, but there's nothing you can do about that, and I was not able to work around that issue. Mostly I've seen this with anti-virus software that attempts to do transparent proxying of POST requests coming from the browser to scan them for private information like social security numbers. You might be running into the same problem.
Are you using McAfee's anti virus by any chance?
Also, it appears that Real invented its own way of doing the same thing, but the basic design is very similar - GET for the downstream link, POST for the upstream, with some magic cookie (in this case, the GUID) to tie the two together on the server. Either way, the POST should get to the server, and in your case it seems like it doesn't.
By the way, since the problem seems to be with the POST request not going through the proxy, how about posting that request, in addition to the GET?
See whether issuing the same request but bypassing the proxy (e.g., replay the request you posted above using Netcat) results in more than four bytes streamed in the response body.
See what TCP packets the proxy is receiving, for example, by eavesdropping on the TCP
traffic on the machine that's running the proxy, say, using Wireshark.