I came across a HTTP HELP method (https://portswigger.net/research/cracking-the-lens-targeting-https-hidden-attack-surface chapter "Invalid Host") and asked myself:
Are there any more systems that offer something like that?
I was wondering how did the pentester come up with this method.
Google couldn't help me here.
In the specific case, it was about an Apache Traffic Server, whose help could be queried as follows:
HELP / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:33:59 GMT
Transfer encoding: chunked
Connection: keep-alive
OK
Traffic Server Overseer Port
commands:
get <variable-list>
set <variable-name> = "<value>"
help
exit
example:
OK
get proxy.node.cache.contents.bytes_free
proxy.node.cache.contents.bytes_free = "56616048"
OK
Variable lists are conf / yts / stats records, separated by commas
And then applied specifically as follows:
GET / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
Content-Length: 34
GET proxy.config.alarm_email
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:57:02 GMT
Transfer encoding: chunked
Connection: keep alive
...
proxy.config.alarm_email = "nobody#yahoo-inc.com"
I figured out the answer:
This is a protocol specially customized for an Apache Traffic Server by Yahoo.
Apache Traffic Server allows you to create your own protocols using the "New Protocols Plugin": https://docs.trafficserver.apache.org/en/latest/developer-guide/plugins/new-protocol-plugins.en.html.
The protocol created here appears to be line-based.
The scenario was as follows:
An initial load balancer evaluated the host header in the incoming HTTP request in such a way that it forwarded the incoming request to the location entered there. This means that the attacker could determine to which internal location the request should be routed, in this case to an Apache traffic server sitting at IP:Port XX.X.XXX.XX: 8082. The underlying attack was a host header injection (https://portswigger.net/web-security/host-header).
The line-based self-made protocol now evaluated the individual lines of the HTTP request. This is how the information shown was achieved (like explained here https://www.youtube.com/watch?v=zP4b3pw94s0&feature=youtu.be&t=12m40s)
.
This means that the attacker was able to address the internal Apache traffic server via an HTTP request and the individual lines of the request were each understood as individual commands.
A HELP command has now been implemented by Yahoo here.
Related
I am an absolute beginner at internet programming. I tried to do some http request manually. I connected to youtube with telnet and tried to get html file on the main page:
ivan#LAPTOP-JSSQ9B0M:/mnt/d/PROJECTS$ telnet www.youtube.com 80
Trying 173.194.222.198...
Connected to wide-youtube.l.google.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: www.youtube.com
But for some reason the response is:
HTTP/1.1 301 Moved Permanently
Content-Type: application/binary
X-Content-Type-Options: nosniff
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Thu, 01 Dec 2022 18:13:19 GMT
Location: https://www.youtube.com/
Server: ESF
Content-Length: 0
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
As far as i know 301 error means that the resource i am trying to get has been moved to other place and the new address is placed in response in "location" header. But as you can see, address i am trying to connect and in location header are the same. When i try to do it with other sites (for example github.com, www.youtube.com, www.twitch.tv), i get the same thing - 301 Moved Permanently but the address in location header and mine are the same.
Question: Why does it happen and how to fix it?
Good on you for being curious and trying stuff on your own!
What you are doing with "telnet" isn't different at all than what your browser is doing when connecting to youtube.com. (well, there is HTTP 2/3 and QUIC, but that's a topic for another day).
It's establishing a raw TCP Connection to port 80 on youtube, and sending an HTTP request.
Now, you can "type it in" since HTTP is a plain-text protocol.
(Those are pretty rare and are getting even more scarce)
Anyway,
Youtube runs an HTTP Server on port 80, the default port for HTTP, (i.e. - when browsers go to youtube.com they set up a connection to port 80 and then try sending an HTTP GET Request) serving plain HTTP.
However!
HTTP is unencrypted, (note! it has nothing to do with it being a textual protocol),
so, regardless of your request parameters (uri, user-agent, etc),
that HTTP Server chooses (well, "was configured") to always redirects you to https://www.youtube.com, via the location header. (or in twitch's case to https://www.twitch.tv)
but the address in location header and mine are the same
notice the https?
Location: https://www.youtube.com/
that tells your browser to go to that website instead.
Via HTTPS - which is HTTP over TLS.
And browsers know that the default port for HTTPS is 443.
You "telnet" client isn't actually an http-client, and it doesn't automatically react to that "instruction".
But let's say you, the "client" in this case, do honor that redirection.
That "over TLS" part? that's tricky. you can't "type it in" yourself with plain-text letters. it's a pretty complex process, and "binary" (i.e. not in English) almost in it's entirety, depending on the version of TLS.
So you can't "fix it", it's a security feature -
their (youtube's and twitch's) policy is (rightfully so) - Never serve content over HTTP,
so that bad-guys who may be "snooping" in the middle can't observe / modify the requests / responses.
You can try with other websites that don't behave like that
for example, example.com
if you want to "programmatically" connect to HTTPS servers, you can do that with any cli-http client, like wget, curl, or invoke-webrequest with windows powershell,
or, with almost any programming language -
like with the requests module in python,
the "fetch" api in JS,
the Java HttpClient,
and so on.
or, if you're feeling particularly cool-
use a TLS wrapper and send the HTTP request over that.
Does that make sense?
feel free to drop any further questions below!
I have been working on performing the load test on jmeter for 500 users per second. I am using JMeter for the same. While running the load test I am continuously getting the error on login API.
Below is the request and response which I am sending and receiving timeout.
Sample Request
POST https://example.com//9000/v1/api/user/login
POST data:
{
"email":"xyz#yopmail.com",
"password":"abcdef"
}
[no cookies]
Request Headers:
Connection: keep-alive
Content-Type: application/json
:
Content-Length: 79
Host: botstest.smartbothub.com
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)
sample response
Thread Name: Thread Group 1-310
Sample Start: 2017-12-27 11:30:06 IST
Load time: 61422
Connect Time: 1148
Latency: 61422
Size in bytes: 363
Sent bytes:286
Headers size in bytes: 171
Body size in bytes: 192
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""): text
Response code: 504
Response message: Gateway Time-out
Response headers:
HTTP/1.1 504 Gateway Time-out
Server: nginx/1.10.3 (Ubuntu)
Date: Wed, 27 Dec 2017 06:01:07 GMT
Content-Type: text/html
Content-Length: 192
Connection: keep-alive
HTTPSampleResult fields:
ContentType: text/html
DataEncoding: null
The nginx config file for the server is as shown below.
https://drive.google.com/file/d/1_XWYeqSAWZz6dtnTTCeLOEMzDwvEWwY1/view
Most likely your Nginx server is overloaded therefore request cannot be processed in the timely fashion causing the error.
It might be caused by several issues:
Nginx server simply lacks hardware resources (CPU, RAM, Network or Disk). Make sure it has enough headroom to operate by monitoring the aforementioned resources on Nginx side using i.e. JMeter PerfMon Plugin
Nginx server configuration is not suitable for high loads. Default configuration might not be suitable for handling 500 concurrent users so check out How to Configure nginx for Optimized Performance article to see whether your setup matches recommendations. Basically you need to combine points 1 and 2 to get most of your Nginx setup, if the machine is relatively idle and you're getting high response times and/or low throughput - something is wrong either with your web application or with infrastructure setup.
It might be a problem with your application which cannot generate response faster than Nginx times out. Check out its logs and inspect what's going on using profiling tools - it will allow you to detect where application spends the most of time
It might be an network infrastructure problem, i.e. you're using WiFi instead of LAN or there is a faulty router or you reached the limits of corporate proxy. It might be also a good idea to capture network traffic using a sniffer tool like Wireshark
But first of all read logs: JMeter logs, Nginx logs, OS logs, whatever exists and relevant. Most probably you will figure out the cause from them.
format of a https get request
I was trying to implement a HTTPS get request.
This is my sample HTTP get request.
GET example.com HTTP/1.1
Date: Mon, 22 Feb 1857 12:27:07 GMT
Host: xyz.com:5901
User-Agent:
Content-Length: 0
I have used SSL library to encrypt the message, but is there anyway to differentiate a HTTP message and https message with the request?
Right now I have put a condition to call the SSL library, but I havent found any example where http request varies from https request.
Can anyone guide me here.
HTTPS is HTTP over SSL/TLS. Essentially, the SSL/TLS connection is initiated by the client. Once the SSL/TLS connection is established, it's just like normal HTTP. As the RFC says:
Conceptually, HTTP/TLS is very simple. Simply use HTTP over TLS
precisely as you would use HTTP over TCP.
I am using a 3G UMTS connection. I am trying to implement HTTP tunneling to a server of mine
which listens on port 80 (this is done in order to bypass client's firewall). The problem is that the ISP's proxy server supports HTTP/1.0 which doesn't support persistent HTTP connection.
As a result, after one http request/response iteration between my client/server the ISP's proxy tears down the underlying TCP connection.
my client receives the following HTTP response:
HTTP/1.0 200 OK
Content-Type: application/octet-stream
Content-Length: yyy
X-Cache: MISS from ipmr5
Proxy-Connection: close
Content data
while my server actually sends:
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Type: application/octet-stream
Content-Length: yyy
Content data
Is there any workaround?
You could always use HTTPS. You will lose any benefits offered by the proxies (such as caching), but all of your HTTP headers will arrive at the server exactly as you sent them.
HTTP 1.0 proxies (which it seems your ISP uses) shouldn't be used in connection with Connection: Keep-Alive for persistent connections. The reasons for this are outlined in RFC-2068 (section 19.7.1). The short version, basically, is that your server is sending an invalid header for the kind of proxy you are using.
I am trying to fetch an RTSP stream over HTTP using a proxy. The behavior of the Real client seems to be a bit hectic: it tries all the possible ports, methods and protocols at once. The only thing that should work is HTTP GET over port 80. Such a request is indeed issued, and is received on the server. Here's how the request looks when it is sent by the proxy to the server:
GET /SmpDsBhgRl83c52ef2-d0f4-41ac-bada-93e5350f67d1?1="1" HTTP/1.0\r\n
Connection: Keep-Alive\r\n
Host: 10.194.5.162:80\r\n
Pragma: no-cache\r\n
User-Agent: RealPlayer G2\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Accept: application/x-rtsp-tunnelled, */*\r\n
ClientID: WinNT_5.1_6.0.14.806_RealPlayer_R41UKD_en-GB_686\r\n
X-Actual-URL: rtsp://10.194.5.162:554/01.mp3\r\n
\r\n
Here's the server's response:
HTTP/1.0 200 OK\r\n
Server: RMServer 1.0\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Pragma: no-cache\r\n
x-server-ipaddress: 10.194.5.162\r\n
Content-type: audio/x-pn-realaudio\r\n
\r\n
At this point 4 more bytes arrive from the server (their values are 48 02 02 00) - and that's it, nothing more. Does the server expect anything from the client at this point, and if so - what? Does this mode of operation work at all?
Some more info on this problem: apparently, the intended mechanism of working with RTSP over HTTP built into RealPlayer is as follows:
Try to connect to the following ports: 80, 8080, 554, 7070.
(Try also to download the file directly, just for the heck of it, by issuing GET http://hostname:port/mediafilename on port 80)
For each of the above ports, create 2 connections.
Send a GET request to one of the connections to the url http://hostname:port/SmpDsBhgRl<guid>?1="1", where <guid> is, yes, a freshly created GUID. Add a header to this request called X-Actual-URL containing the original RTSP URL.
Send a POST request on the other connection, to the URL http://hostname:port/SmpDsBhgRl with the GUID above as part of the body of the request. Send a Content-Length header of 32767 bytes, to prevent the proxy from closing the connection prematurely.
Start issuing commands to the server through the POST request, and get the corresponding RTSP stream as part of the GET response.
The strange stuff (if the above isn't strange enough) is that, for example, it works with Squid, but not if you use either of the ports 3128 or 8080! Somehow, the client uses the port it connects to to decide on the order of the requests or on when a request should be canceled, but anyway, as hard to believe as it is, it works with proxy port 9090, 3129, 8081, but not with 3128 or 8080.
Update #2: Here's the source of the RealPlayer with the explanation of the above behavior. Still no solution though.
Update #3: OK, in the light of the above, the magic value of 48 02 02 00 is clear: 48 == 'h' is for HTTP_RESPONSE, the next 02 is the length of the following data, the next 02 is called POST_NOT_RECEIVED (meaning that the POST request did not reach the server within a second from the corresponding GET request).
Update #4: This behavior (i.e. POST requests with huge Content-Length) is also characteristic of an ActiveX used by WebEx (and, possibly, many other web apps that need an open channel to the server).
First, you might want to read this:
http://developer.apple.com/quicktime/icefloe/dispatch028.html
Second, the HTTP requests (both GET and POST) need to be formatted so that they get proxied properly. I've seen proxies that insist on caching too much of the POST request, preventing it from reaching the server. Those proxies are buggy, but there's nothing you can do about that, and I was not able to work around that issue. Mostly I've seen this with anti-virus software that attempts to do transparent proxying of POST requests coming from the browser to scan them for private information like social security numbers. You might be running into the same problem.
Are you using McAfee's anti virus by any chance?
Also, it appears that Real invented its own way of doing the same thing, but the basic design is very similar - GET for the downstream link, POST for the upstream, with some magic cookie (in this case, the GUID) to tie the two together on the server. Either way, the POST should get to the server, and in your case it seems like it doesn't.
By the way, since the problem seems to be with the POST request not going through the proxy, how about posting that request, in addition to the GET?
See whether issuing the same request but bypassing the proxy (e.g., replay the request you posted above using Netcat) results in more than four bytes streamed in the response body.
See what TCP packets the proxy is receiving, for example, by eavesdropping on the TCP
traffic on the machine that's running the proxy, say, using Wireshark.