I am an absolute beginner at internet programming. I tried to do some http request manually. I connected to youtube with telnet and tried to get html file on the main page:
ivan#LAPTOP-JSSQ9B0M:/mnt/d/PROJECTS$ telnet www.youtube.com 80
Trying 173.194.222.198...
Connected to wide-youtube.l.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.youtube.com
But for some reason the response is:
HTTP/1.1 301 Moved Permanently
Content-Type: application/binary
X-Content-Type-Options: nosniff
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Thu, 01 Dec 2022 18:13:19 GMT
Location: https://www.youtube.com/
Server: ESF
Content-Length: 0
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
As far as i know 301 error means that the resource i am trying to get has been moved to other place and the new address is placed in response in "location" header. But as you can see, address i am trying to connect and in location header are the same. When i try to do it with other sites (for example github.com, www.youtube.com, www.twitch.tv), i get the same thing - 301 Moved Permanently but the address in location header and mine are the same.
Question: Why does it happen and how to fix it?
Good on you for being curious and trying stuff on your own!
What you are doing with "telnet" isn't different at all than what your browser is doing when connecting to youtube.com. (well, there is HTTP 2/3 and QUIC, but that's a topic for another day).
It's establishing a raw TCP Connection to port 80 on youtube, and sending an HTTP request.
Now, you can "type it in" since HTTP is a plain-text protocol.
(Those are pretty rare and are getting even more scarce)
Anyway,
Youtube runs an HTTP Server on port 80, the default port for HTTP, (i.e. - when browsers go to youtube.com they set up a connection to port 80 and then try sending an HTTP GET Request) serving plain HTTP.
However!
HTTP is unencrypted, (note! it has nothing to do with it being a textual protocol),
so, regardless of your request parameters (uri, user-agent, etc),
that HTTP Server chooses (well, "was configured") to always redirects you to https://www.youtube.com, via the location header. (or in twitch's case to https://www.twitch.tv)
but the address in location header and mine are the same
notice the https?
Location: https://www.youtube.com/
that tells your browser to go to that website instead.
Via HTTPS - which is HTTP over TLS.
And browsers know that the default port for HTTPS is 443.
You "telnet" client isn't actually an http-client, and it doesn't automatically react to that "instruction".
But let's say you, the "client" in this case, do honor that redirection.
That "over TLS" part? that's tricky. you can't "type it in" yourself with plain-text letters. it's a pretty complex process, and "binary" (i.e. not in English) almost in it's entirety, depending on the version of TLS.
So you can't "fix it", it's a security feature -
their (youtube's and twitch's) policy is (rightfully so) - Never serve content over HTTP,
so that bad-guys who may be "snooping" in the middle can't observe / modify the requests / responses.
You can try with other websites that don't behave like that
for example, example.com
if you want to "programmatically" connect to HTTPS servers, you can do that with any cli-http client, like wget, curl, or invoke-webrequest with windows powershell,
or, with almost any programming language -
like with the requests module in python,
the "fetch" api in JS,
the Java HttpClient,
and so on.
or, if you're feeling particularly cool-
use a TLS wrapper and send the HTTP request over that.
Does that make sense?
feel free to drop any further questions below!
Related
I came across a HTTP HELP method (https://portswigger.net/research/cracking-the-lens-targeting-https-hidden-attack-surface chapter "Invalid Host") and asked myself:
Are there any more systems that offer something like that?
I was wondering how did the pentester come up with this method.
Google couldn't help me here.
In the specific case, it was about an Apache Traffic Server, whose help could be queried as follows:
HELP / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:33:59 GMT
Transfer encoding: chunked
Connection: keep-alive
OK
Traffic Server Overseer Port
commands:
get <variable-list>
set <variable-name> = "<value>"
help
exit
example:
OK
get proxy.node.cache.contents.bytes_free
proxy.node.cache.contents.bytes_free = "56616048"
OK
Variable lists are conf / yts / stats records, separated by commas
And then applied specifically as follows:
GET / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
Content-Length: 34
GET proxy.config.alarm_email
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:57:02 GMT
Transfer encoding: chunked
Connection: keep alive
...
proxy.config.alarm_email = "nobody#yahoo-inc.com"
I figured out the answer:
This is a protocol specially customized for an Apache Traffic Server by Yahoo.
Apache Traffic Server allows you to create your own protocols using the "New Protocols Plugin": https://docs.trafficserver.apache.org/en/latest/developer-guide/plugins/new-protocol-plugins.en.html.
The protocol created here appears to be line-based.
The scenario was as follows:
An initial load balancer evaluated the host header in the incoming HTTP request in such a way that it forwarded the incoming request to the location entered there. This means that the attacker could determine to which internal location the request should be routed, in this case to an Apache traffic server sitting at IP:Port XX.X.XXX.XX: 8082. The underlying attack was a host header injection (https://portswigger.net/web-security/host-header).
The line-based self-made protocol now evaluated the individual lines of the HTTP request. This is how the information shown was achieved (like explained here https://www.youtube.com/watch?v=zP4b3pw94s0&feature=youtu.be&t=12m40s)
.
This means that the attacker was able to address the internal Apache traffic server via an HTTP request and the individual lines of the request were each understood as individual commands.
A HELP command has now been implemented by Yahoo here.
My nginx website is protected by IP range and I need to be on the premises or log in via vpn to see it:
geo $external_ip {
192.168.0.0/16 0;
}
…
if ($external_ip) {
return 302 https://www.different-site.com/info_explaining_to_log_in_via_vpn ;
}
…
# Normal location config for website
Now I access my website and I have forgotten to log in via the VPN. So I am being redirected to the info page that tells me nicely why I can't access the site and that I have to fire up my VPN client.
I do so. And now I reenter the URL to my website again, but the browser seems to remember the redirect and keeps returning the info page.
The same happens if I use a 307 status code.
After a while, the browser seems to forget that redirect and behaves as I expect again and shows me my site.
The same happens if I simply return 403, effectively discarding my info page. I did that because I thought it might be some cache statements. But also the default nginx 403 page seems to get remembered by the browser.
Is there a way how I can make the browser try to reload the orginal site with each try? I feel I am missing a vital brick of the puzzle.
Output of $ curl -I https://different-site.com/:
HTTP/2 302
server: nginx/1.10.3 (Ubuntu)
date: Mon, 11 May 2020 20:07:54 GMT
content-type: text/html
content-length: 170
location: different-site.com/info_explaining_to_log_in_via_vpn
pragma: no-cache
cache-control: no-cache
More information:
The problem only exists in the browser that initially opened the connection without the vpn in place. Any other browser or a curl run after initiating the vpn succeeds.
More information (2):
I am currently assuming that my browser is keeping a connection alive which is reused even once the VPN is started. That way, nginx still sees the old public IP and refuses access until that original connection is terminated.
According to https://serverfault.com/questions/775617/force-nginx-to-close-connection, I cannot terminate the connection server side unless I disable keepalive completely, which I don't want for performance reasons. So it looks like there is no way forward.
My next thought is to put something in front of nginx (haproxy?) which could determine a connection close depending on url.
Does it help to disable cache response headers?
add_header cache-control no-cache always;
add_header Last-Modified "" always;
etag off;
if_modified_since off;
I've had this issue before when my server originally sent a 301 before I changed it to 302. You can also clear Chrome's DNS cache with chrome://net-internals/#dns, or try a private/incognito window in your browser. Firefox/Chrome dev tools let you disable the cache and should pick up the new Location header if you reload the URL:
Try Cache-Control: no-store instead.
From the documentation for no-cache:
The response may be stored by any cache, even if the response is
normally non-cacheable. However, the stored response MUST always go
through validation with the origin server first before using it,
therefore, you cannot use no-cache in-conjunction with immutable. If
you mean to not store the response in any cache,
use no-store instead. This directive is not effective in preventing
caches from storing your response.
format of a https get request
I was trying to implement a HTTPS get request.
This is my sample HTTP get request.
GET example.com HTTP/1.1
Date: Mon, 22 Feb 1857 12:27:07 GMT
Host: xyz.com:5901
User-Agent:
Content-Length: 0
I have used SSL library to encrypt the message, but is there anyway to differentiate a HTTP message and https message with the request?
Right now I have put a condition to call the SSL library, but I havent found any example where http request varies from https request.
Can anyone guide me here.
HTTPS is HTTP over SSL/TLS. Essentially, the SSL/TLS connection is initiated by the client. Once the SSL/TLS connection is established, it's just like normal HTTP. As the RFC says:
Conceptually, HTTP/TLS is very simple. Simply use HTTP over TLS
precisely as you would use HTTP over TCP.
(This question is more of a reality check than anything else - I'm pretty sure this is possible, but want to be sure.)
I’m writing a proxy server that receives HTTP requests, passes them on to a remote server, and returns the remote server’s response.
I’ve run into a problem where one of the remote server responses is a 302 redirect that sets a cookie. Here’s an example:
server: nginx/0.7.65
date: Wed, 26 Jan 2011 04:42:18 GMT
connection: keep-alive
set-cookie: JSESSIONID=FFFFFFFUUUUUUUUUUU; Domain=.50.16.34.61; Path=/
location: http://50.16.34.61/client/whatever.html
content-length: "0"
When the client follows that redirect to 50.16.34.61 (a different domain than the proxy server) I notice that the cookies haven’t been set.
Am I trying to do something crazy here with the Domain part of the set-cookie directive?
Edit
Basically, I’m doing it wrong and according to http://www.ietf.org/rfc/rfc2109.txt the request-host really needs to be a superset of the Domain in the set-cookie directive.
/sad-trombone
The Domain= cookie stuff will only work if the Host: header that the browser asked for is a superset of the value used.
In your case, it seems like your proxy server is pretending to be another server, so it might make sense to intercept those Location: and Set-Cookie: header lines, among many other things, and replace the relevant bits to keep pointing at your server.
The client follows the redirect to 50.16.34.61, but was the page requested on that domain? The proxy isn't the problem here—if resource requested (which generated the 302) was on a different domain, you can't set a cookie for a different one.
The proxy certainly shouldn't be getting in the way.
Edit: hold on; is this not actually a HTTP proxy server, but rather a HTTP server that proxies?
I am trying to fetch an RTSP stream over HTTP using a proxy. The behavior of the Real client seems to be a bit hectic: it tries all the possible ports, methods and protocols at once. The only thing that should work is HTTP GET over port 80. Such a request is indeed issued, and is received on the server. Here's how the request looks when it is sent by the proxy to the server:
GET /SmpDsBhgRl83c52ef2-d0f4-41ac-bada-93e5350f67d1?1="1" HTTP/1.0\r\n
Connection: Keep-Alive\r\n
Host: 10.194.5.162:80\r\n
Pragma: no-cache\r\n
User-Agent: RealPlayer G2\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Accept: application/x-rtsp-tunnelled, */*\r\n
ClientID: WinNT_5.1_6.0.14.806_RealPlayer_R41UKD_en-GB_686\r\n
X-Actual-URL: rtsp://10.194.5.162:554/01.mp3\r\n
\r\n
Here's the server's response:
HTTP/1.0 200 OK\r\n
Server: RMServer 1.0\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Pragma: no-cache\r\n
x-server-ipaddress: 10.194.5.162\r\n
Content-type: audio/x-pn-realaudio\r\n
\r\n
At this point 4 more bytes arrive from the server (their values are 48 02 02 00) - and that's it, nothing more. Does the server expect anything from the client at this point, and if so - what? Does this mode of operation work at all?
Some more info on this problem: apparently, the intended mechanism of working with RTSP over HTTP built into RealPlayer is as follows:
Try to connect to the following ports: 80, 8080, 554, 7070.
(Try also to download the file directly, just for the heck of it, by issuing GET http://hostname:port/mediafilename on port 80)
For each of the above ports, create 2 connections.
Send a GET request to one of the connections to the url http://hostname:port/SmpDsBhgRl<guid>?1="1", where <guid> is, yes, a freshly created GUID. Add a header to this request called X-Actual-URL containing the original RTSP URL.
Send a POST request on the other connection, to the URL http://hostname:port/SmpDsBhgRl with the GUID above as part of the body of the request. Send a Content-Length header of 32767 bytes, to prevent the proxy from closing the connection prematurely.
Start issuing commands to the server through the POST request, and get the corresponding RTSP stream as part of the GET response.
The strange stuff (if the above isn't strange enough) is that, for example, it works with Squid, but not if you use either of the ports 3128 or 8080! Somehow, the client uses the port it connects to to decide on the order of the requests or on when a request should be canceled, but anyway, as hard to believe as it is, it works with proxy port 9090, 3129, 8081, but not with 3128 or 8080.
Update #2: Here's the source of the RealPlayer with the explanation of the above behavior. Still no solution though.
Update #3: OK, in the light of the above, the magic value of 48 02 02 00 is clear: 48 == 'h' is for HTTP_RESPONSE, the next 02 is the length of the following data, the next 02 is called POST_NOT_RECEIVED (meaning that the POST request did not reach the server within a second from the corresponding GET request).
Update #4: This behavior (i.e. POST requests with huge Content-Length) is also characteristic of an ActiveX used by WebEx (and, possibly, many other web apps that need an open channel to the server).
First, you might want to read this:
http://developer.apple.com/quicktime/icefloe/dispatch028.html
Second, the HTTP requests (both GET and POST) need to be formatted so that they get proxied properly. I've seen proxies that insist on caching too much of the POST request, preventing it from reaching the server. Those proxies are buggy, but there's nothing you can do about that, and I was not able to work around that issue. Mostly I've seen this with anti-virus software that attempts to do transparent proxying of POST requests coming from the browser to scan them for private information like social security numbers. You might be running into the same problem.
Are you using McAfee's anti virus by any chance?
Also, it appears that Real invented its own way of doing the same thing, but the basic design is very similar - GET for the downstream link, POST for the upstream, with some magic cookie (in this case, the GUID) to tie the two together on the server. Either way, the POST should get to the server, and in your case it seems like it doesn't.
By the way, since the problem seems to be with the POST request not going through the proxy, how about posting that request, in addition to the GET?
See whether issuing the same request but bypassing the proxy (e.g., replay the request you posted above using Netcat) results in more than four bytes streamed in the response body.
See what TCP packets the proxy is receiving, for example, by eavesdropping on the TCP
traffic on the machine that's running the proxy, say, using Wireshark.