format of a https get request
I was trying to implement a HTTPS get request.
This is my sample HTTP get request.
GET example.com HTTP/1.1
Date: Mon, 22 Feb 1857 12:27:07 GMT
Host: xyz.com:5901
User-Agent:
Content-Length: 0
I have used SSL library to encrypt the message, but is there anyway to differentiate a HTTP message and https message with the request?
Right now I have put a condition to call the SSL library, but I havent found any example where http request varies from https request.
Can anyone guide me here.
HTTPS is HTTP over SSL/TLS. Essentially, the SSL/TLS connection is initiated by the client. Once the SSL/TLS connection is established, it's just like normal HTTP. As the RFC says:
Conceptually, HTTP/TLS is very simple. Simply use HTTP over TLS
precisely as you would use HTTP over TCP.
Related
I am an absolute beginner at internet programming. I tried to do some http request manually. I connected to youtube with telnet and tried to get html file on the main page:
ivan#LAPTOP-JSSQ9B0M:/mnt/d/PROJECTS$ telnet www.youtube.com 80
Trying 173.194.222.198...
Connected to wide-youtube.l.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.youtube.com
But for some reason the response is:
HTTP/1.1 301 Moved Permanently
Content-Type: application/binary
X-Content-Type-Options: nosniff
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Thu, 01 Dec 2022 18:13:19 GMT
Location: https://www.youtube.com/
Server: ESF
Content-Length: 0
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
As far as i know 301 error means that the resource i am trying to get has been moved to other place and the new address is placed in response in "location" header. But as you can see, address i am trying to connect and in location header are the same. When i try to do it with other sites (for example github.com, www.youtube.com, www.twitch.tv), i get the same thing - 301 Moved Permanently but the address in location header and mine are the same.
Question: Why does it happen and how to fix it?
Good on you for being curious and trying stuff on your own!
What you are doing with "telnet" isn't different at all than what your browser is doing when connecting to youtube.com. (well, there is HTTP 2/3 and QUIC, but that's a topic for another day).
It's establishing a raw TCP Connection to port 80 on youtube, and sending an HTTP request.
Now, you can "type it in" since HTTP is a plain-text protocol.
(Those are pretty rare and are getting even more scarce)
Anyway,
Youtube runs an HTTP Server on port 80, the default port for HTTP, (i.e. - when browsers go to youtube.com they set up a connection to port 80 and then try sending an HTTP GET Request) serving plain HTTP.
However!
HTTP is unencrypted, (note! it has nothing to do with it being a textual protocol),
so, regardless of your request parameters (uri, user-agent, etc),
that HTTP Server chooses (well, "was configured") to always redirects you to https://www.youtube.com, via the location header. (or in twitch's case to https://www.twitch.tv)
but the address in location header and mine are the same
notice the https?
Location: https://www.youtube.com/
that tells your browser to go to that website instead.
Via HTTPS - which is HTTP over TLS.
And browsers know that the default port for HTTPS is 443.
You "telnet" client isn't actually an http-client, and it doesn't automatically react to that "instruction".
But let's say you, the "client" in this case, do honor that redirection.
That "over TLS" part? that's tricky. you can't "type it in" yourself with plain-text letters. it's a pretty complex process, and "binary" (i.e. not in English) almost in it's entirety, depending on the version of TLS.
So you can't "fix it", it's a security feature -
their (youtube's and twitch's) policy is (rightfully so) - Never serve content over HTTP,
so that bad-guys who may be "snooping" in the middle can't observe / modify the requests / responses.
You can try with other websites that don't behave like that
for example, example.com
if you want to "programmatically" connect to HTTPS servers, you can do that with any cli-http client, like wget, curl, or invoke-webrequest with windows powershell,
or, with almost any programming language -
like with the requests module in python,
the "fetch" api in JS,
the Java HttpClient,
and so on.
or, if you're feeling particularly cool-
use a TLS wrapper and send the HTTP request over that.
Does that make sense?
feel free to drop any further questions below!
I came across a HTTP HELP method (https://portswigger.net/research/cracking-the-lens-targeting-https-hidden-attack-surface chapter "Invalid Host") and asked myself:
Are there any more systems that offer something like that?
I was wondering how did the pentester come up with this method.
Google couldn't help me here.
In the specific case, it was about an Apache Traffic Server, whose help could be queried as follows:
HELP / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:33:59 GMT
Transfer encoding: chunked
Connection: keep-alive
OK
Traffic Server Overseer Port
commands:
get <variable-list>
set <variable-name> = "<value>"
help
exit
example:
OK
get proxy.node.cache.contents.bytes_free
proxy.node.cache.contents.bytes_free = "56616048"
OK
Variable lists are conf / yts / stats records, separated by commas
And then applied specifically as follows:
GET / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
Content-Length: 34
GET proxy.config.alarm_email
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:57:02 GMT
Transfer encoding: chunked
Connection: keep alive
...
proxy.config.alarm_email = "nobody#yahoo-inc.com"
I figured out the answer:
This is a protocol specially customized for an Apache Traffic Server by Yahoo.
Apache Traffic Server allows you to create your own protocols using the "New Protocols Plugin": https://docs.trafficserver.apache.org/en/latest/developer-guide/plugins/new-protocol-plugins.en.html.
The protocol created here appears to be line-based.
The scenario was as follows:
An initial load balancer evaluated the host header in the incoming HTTP request in such a way that it forwarded the incoming request to the location entered there. This means that the attacker could determine to which internal location the request should be routed, in this case to an Apache traffic server sitting at IP:Port XX.X.XXX.XX: 8082. The underlying attack was a host header injection (https://portswigger.net/web-security/host-header).
The line-based self-made protocol now evaluated the individual lines of the HTTP request. This is how the information shown was achieved (like explained here https://www.youtube.com/watch?v=zP4b3pw94s0&feature=youtu.be&t=12m40s)
.
This means that the attacker was able to address the internal Apache traffic server via an HTTP request and the individual lines of the request were each understood as individual commands.
A HELP command has now been implemented by Yahoo here.
I have a SIM900 GSM module that I use to send GET and POST requests to servers.
Recently I rented a host for this purpose. I wrote a simple page using asp.net webforms to parse incoming data from the GSM module, everything was working until a few days ago I noticed that I no longer can receive data from my gsm module.
After investigating further I found out that the host I rented keeps returning HTTP 400 errors to my GSM module. These responses are not from IIS but from Microsoft-HTTPAPI/2.0. The request header is this:
GET /test/data?meow HTTP/1.1
Host : www.whatever.com
Connection : keep-alive
And this is the server response(body omitted):
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Sun, 11 Oct 2020 12:08:28 GMT
Connection: close
Content-Length: 339
I used Postman (application) to simulate the same request and everything worked just fine.
I also made an exact copy of a chrome request header and gave it to the module, but that didn't work either.
Note: I am not using sim900's HTTP commands I am connecting to a certain port(80 in this case) and making a get request manually.
Note 2:I have been given a Plesk panel to manage my website and do not have access to certain server settings.
The request will pass through the http.sys module before entering iis, which will intercept requests that do not comply with the rules, so your response comes from Microsoft-HTTPAPI / 2.0 with a status code of 400. The solution to this can be to modify the registry, but the setting in the registry is based on your application and request, and there is no universal modification method.
How to troubleshoot HTTP 400 errors
Http.sys registry settings for Windows
Another method is to suggest that you use a tool similar to Fiddler to capture the request sent by sim900 and the request sent by postman respectively. After the capture, compare them in detail to find out the differences, and modify the sim900 request to be the same as the postman request, conforming to http .sys rules.
We are trying to make a secure communication between our embedded system and web server.Firstly we implement HTTP connection to in our microcontroller. I am just connecting to 80 port of my web server and send simple GET request to this port as example below :
GET /foo.php?msg=test HTTP/1.1
HOST: foo.com
My questions is,How we will turn this to HTTPS ? Which port i should connect ?
Will be any difference on structure of GET request above ? Will i have to do some encryption manually or connect to "https" link instead "http" is enuogh for secure communication.
Thanks for any information
The only difference between a HTTP request and a HTTPS request is that the first is send over a plain TCP connection while the other is send over a TLS connection, i.e.:
with HTTP you establish a TCP connection and send the request over this connection
with HTTPS you establish a TCP connection, upgrade this connection to TLS (including proper certificate validation etc!) and then send the same request as you did with HTTP over this connection.
Apart from that I recommend to either use an established library for HTTP or carefully read the standard. Although HTTP looks simply it is actually not and there are many questions here where users try to do a simply HTTP request and trip over behavior they did not expect.
For example in your case the server might send the response with chunked encoding, with content-length or simply end it with connection close. And it might wait for further requests on the same connection since HTTP/1.1 implicitly enables HTTP keep-alive. Does your code really account for all these cases?
I am using a 3G UMTS connection. I am trying to implement HTTP tunneling to a server of mine
which listens on port 80 (this is done in order to bypass client's firewall). The problem is that the ISP's proxy server supports HTTP/1.0 which doesn't support persistent HTTP connection.
As a result, after one http request/response iteration between my client/server the ISP's proxy tears down the underlying TCP connection.
my client receives the following HTTP response:
HTTP/1.0 200 OK
Content-Type: application/octet-stream
Content-Length: yyy
X-Cache: MISS from ipmr5
Proxy-Connection: close
Content data
while my server actually sends:
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Type: application/octet-stream
Content-Length: yyy
Content data
Is there any workaround?
You could always use HTTPS. You will lose any benefits offered by the proxies (such as caching), but all of your HTTP headers will arrive at the server exactly as you sent them.
HTTP 1.0 proxies (which it seems your ISP uses) shouldn't be used in connection with Connection: Keep-Alive for persistent connections. The reasons for this are outlined in RFC-2068 (section 19.7.1). The short version, basically, is that your server is sending an invalid header for the kind of proxy you are using.