I have been working on performing the load test on jmeter for 500 users per second. I am using JMeter for the same. While running the load test I am continuously getting the error on login API.
Below is the request and response which I am sending and receiving timeout.
Sample Request
POST https://example.com//9000/v1/api/user/login
POST data:
{
"email":"xyz#yopmail.com",
"password":"abcdef"
}
[no cookies]
Request Headers:
Connection: keep-alive
Content-Type: application/json
:
Content-Length: 79
Host: botstest.smartbothub.com
User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_151)
sample response
Thread Name: Thread Group 1-310
Sample Start: 2017-12-27 11:30:06 IST
Load time: 61422
Connect Time: 1148
Latency: 61422
Size in bytes: 363
Sent bytes:286
Headers size in bytes: 171
Body size in bytes: 192
Sample Count: 1
Error Count: 1
Data type ("text"|"bin"|""): text
Response code: 504
Response message: Gateway Time-out
Response headers:
HTTP/1.1 504 Gateway Time-out
Server: nginx/1.10.3 (Ubuntu)
Date: Wed, 27 Dec 2017 06:01:07 GMT
Content-Type: text/html
Content-Length: 192
Connection: keep-alive
HTTPSampleResult fields:
ContentType: text/html
DataEncoding: null
The nginx config file for the server is as shown below.
https://drive.google.com/file/d/1_XWYeqSAWZz6dtnTTCeLOEMzDwvEWwY1/view
Most likely your Nginx server is overloaded therefore request cannot be processed in the timely fashion causing the error.
It might be caused by several issues:
Nginx server simply lacks hardware resources (CPU, RAM, Network or Disk). Make sure it has enough headroom to operate by monitoring the aforementioned resources on Nginx side using i.e. JMeter PerfMon Plugin
Nginx server configuration is not suitable for high loads. Default configuration might not be suitable for handling 500 concurrent users so check out How to Configure nginx for Optimized Performance article to see whether your setup matches recommendations. Basically you need to combine points 1 and 2 to get most of your Nginx setup, if the machine is relatively idle and you're getting high response times and/or low throughput - something is wrong either with your web application or with infrastructure setup.
It might be a problem with your application which cannot generate response faster than Nginx times out. Check out its logs and inspect what's going on using profiling tools - it will allow you to detect where application spends the most of time
It might be an network infrastructure problem, i.e. you're using WiFi instead of LAN or there is a faulty router or you reached the limits of corporate proxy. It might be also a good idea to capture network traffic using a sniffer tool like Wireshark
But first of all read logs: JMeter logs, Nginx logs, OS logs, whatever exists and relevant. Most probably you will figure out the cause from them.
Related
I came across a HTTP HELP method (https://portswigger.net/research/cracking-the-lens-targeting-https-hidden-attack-surface chapter "Invalid Host") and asked myself:
Are there any more systems that offer something like that?
I was wondering how did the pentester come up with this method.
Google couldn't help me here.
In the specific case, it was about an Apache Traffic Server, whose help could be queried as follows:
HELP / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:33:59 GMT
Transfer encoding: chunked
Connection: keep-alive
OK
Traffic Server Overseer Port
commands:
get <variable-list>
set <variable-name> = "<value>"
help
exit
example:
OK
get proxy.node.cache.contents.bytes_free
proxy.node.cache.contents.bytes_free = "56616048"
OK
Variable lists are conf / yts / stats records, separated by commas
And then applied specifically as follows:
GET / HTTP / 1.1
Host: XX.X.XXX.XX: 8082
Content-Length: 34
GET proxy.config.alarm_email
HTTP / 1.1 200 Connection Established
Date: Tue, 07 Feb 2017 16:57:02 GMT
Transfer encoding: chunked
Connection: keep alive
...
proxy.config.alarm_email = "nobody#yahoo-inc.com"
I figured out the answer:
This is a protocol specially customized for an Apache Traffic Server by Yahoo.
Apache Traffic Server allows you to create your own protocols using the "New Protocols Plugin": https://docs.trafficserver.apache.org/en/latest/developer-guide/plugins/new-protocol-plugins.en.html.
The protocol created here appears to be line-based.
The scenario was as follows:
An initial load balancer evaluated the host header in the incoming HTTP request in such a way that it forwarded the incoming request to the location entered there. This means that the attacker could determine to which internal location the request should be routed, in this case to an Apache traffic server sitting at IP:Port XX.X.XXX.XX: 8082. The underlying attack was a host header injection (https://portswigger.net/web-security/host-header).
The line-based self-made protocol now evaluated the individual lines of the HTTP request. This is how the information shown was achieved (like explained here https://www.youtube.com/watch?v=zP4b3pw94s0&feature=youtu.be&t=12m40s)
.
This means that the attacker was able to address the internal Apache traffic server via an HTTP request and the individual lines of the request were each understood as individual commands.
A HELP command has now been implemented by Yahoo here.
I have a SIM900 GSM module that I use to send GET and POST requests to servers.
Recently I rented a host for this purpose. I wrote a simple page using asp.net webforms to parse incoming data from the GSM module, everything was working until a few days ago I noticed that I no longer can receive data from my gsm module.
After investigating further I found out that the host I rented keeps returning HTTP 400 errors to my GSM module. These responses are not from IIS but from Microsoft-HTTPAPI/2.0. The request header is this:
GET /test/data?meow HTTP/1.1
Host : www.whatever.com
Connection : keep-alive
And this is the server response(body omitted):
HTTP/1.1 400 Bad Request
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Sun, 11 Oct 2020 12:08:28 GMT
Connection: close
Content-Length: 339
I used Postman (application) to simulate the same request and everything worked just fine.
I also made an exact copy of a chrome request header and gave it to the module, but that didn't work either.
Note: I am not using sim900's HTTP commands I am connecting to a certain port(80 in this case) and making a get request manually.
Note 2:I have been given a Plesk panel to manage my website and do not have access to certain server settings.
The request will pass through the http.sys module before entering iis, which will intercept requests that do not comply with the rules, so your response comes from Microsoft-HTTPAPI / 2.0 with a status code of 400. The solution to this can be to modify the registry, but the setting in the registry is based on your application and request, and there is no universal modification method.
How to troubleshoot HTTP 400 errors
Http.sys registry settings for Windows
Another method is to suggest that you use a tool similar to Fiddler to capture the request sent by sim900 and the request sent by postman respectively. After the capture, compare them in detail to find out the differences, and modify the sim900 request to be the same as the postman request, conforming to http .sys rules.
I'm seeing a weird situation where either Nginx or uwsgi seems to be building up a long queue of incoming requests, and attempting to process them long after the client connection timed out. I'd like to understand and stop that behavior. Here's more info:
My Setup
My server uses Nginx to pass HTTPS POST requests to uWSGI and Flask via a Unix file socket. I have basically the default configurations on everything.
I have a Python client sending 3 requests per second to that server.
The Problem
After running the client for about 4 hours, the client machine started reporting that all the connections were timing out. (It uses the Python requests library with a 7-second timeout.) About 10 minutes later, the behavior changed: the connections began failing with 502 Bad Gateway.
I powered off the client. But for about 10 minutes AFTER powering off the client, the server-side uWSGI logs showed uWSGI attempting to answer requests from that client! And top showed uWSGI using 100% CPU (25% per worker).
During those 10 minutes, each uwsgi.log entry looked like this:
Thu May 25 07:36:37 2017 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/polldata (ip 98.210.18.212) !!!
Thu May 25 07:36:37 2017 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 296] during POST /api/polldata (98.210.18.212)
IOError: write error
[pid: 34|app: 0|req: 645/12472] 98.210.18.212 () {42 vars in 588 bytes} [Thu May 25 07:36:08 2017] POST /api/polldata => generated 0 bytes in 28345 msecs (HTTP/1.1 200) 2 headers in 0 bytes (0 switches on core 0)
And the Nginx error.log shows a lot of this:
2017/05/25 08:10:29 [error] 36#36: *35037 connect() to unix:/srv/my_server/myproject.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 98.210.18.212, server: example.com, request: "POST /api/polldata HTTP/1.1", upstream: "uwsgi://unix:/srv/my_server/myproject.sock:", host: "example.com:5000"
After about 10 minutes the uWSGI activity stops. When I turn the client back on, Nginx happily accepts the POST requests, but uWSGI gives the same "writing to a closed pipe" error on every request, as if it's permanently broken somehow. Restarting the webserver's docker container does not fix the problem, but rebooting the host machine fixes it.
Theories
In the default Nginx -> socket -> uWSGI configuration, is there a long queue of requests with no timeout? I looked in the uWSGI docs and I saw a bunch of configurable timeouts, but all default to around 60 seconds, so I can't understand how I'm seeing 10-minute-old requests being handled. I haven't changed any default timeout settings.
The application uses almost all the 1GB RAM in my small dev server, so I think resource limits may be triggering the behavior.
Either way, I'd like to change my configuration so that requests > 30 seconds old get dropped with a 500 error, rather than getting processed by uWSGI. I'd appreciate any advice on how to do that, and theories on what's happening.
This appears to be an issue downstream on the uWSGI side.
It sounds like your backend code may be faulty in that it takes too long to process the requests, does not implement any sort of rate limiting for the requests, and does not properly catch if any of the underlying connections have been terminated (hence, you're receiving the errors that your code tries to write to closed pipelines, and possibly even start processing new requests long after the underlying connections have been terminated).
As per http://lists.unbit.it/pipermail/uwsgi/2013-February/005362.html, you might want to abort processing within your backend if not uwsgi.is_connected(uwsgi.connection_fd()).
You might want to explore https://uwsgi-docs.readthedocs.io/en/latest/Options.html#harakiri.
As last resort, as per Re: Understanding "proxy_ignore_client_abort" functionality (2014), you might want to change uwsgi_ignore_client_abort from off to on in order to not drop the ongoing uWSGI connections that have already been passed to the upstream (even if the client does subsequently disconnect) in order to not receive the closed pipe errors from uWSGI, as well as to enforce any possible concurrent connection limits within nginx itself (otherwise, the connections to uWSGI will get dropped by nginx should the client disconnect, and nginx would have no clue how many requests are being queued up within uWSGI for subsequent processing).
Seems like DoS attack on Nginx uWSGI returning 100% CPU usage with Nginx 502, 504, 500. IP spoofing is common in DoS attack. Exclude by checking the logs.
After receiving http packets from a website I see a request packet which its http header is like this,what does it mean "OpenNMS HttpMonitor\r\n" ?Its source address is not from that web page which I open!
GET / HTTP/1.1\r\n
[Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n]
Request Method: GET
Request URI: /
Request Version: HTTP/1.1
Connection: CLOSE \r\n
User-Agent: OpenNMS HttpMonitor\r\n
\r\n
I believe this may well be Rackspace's monitoring solution for cloud servers. Might be wrong though. Might be worth contacting your hosting provider to see if it's them. You can sort of check this by seeing if your server IP is in the same subnet.
Um, not sure why it is appearing in your context, but OpenNMS is a network monitoring suite that we used to use at work to monitor our network nodes.
http://www.opennms.org/
Your IP may be erroneously being monitored by some corporation? ^^
I am trying to fetch an RTSP stream over HTTP using a proxy. The behavior of the Real client seems to be a bit hectic: it tries all the possible ports, methods and protocols at once. The only thing that should work is HTTP GET over port 80. Such a request is indeed issued, and is received on the server. Here's how the request looks when it is sent by the proxy to the server:
GET /SmpDsBhgRl83c52ef2-d0f4-41ac-bada-93e5350f67d1?1="1" HTTP/1.0\r\n
Connection: Keep-Alive\r\n
Host: 10.194.5.162:80\r\n
Pragma: no-cache\r\n
User-Agent: RealPlayer G2\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Accept: application/x-rtsp-tunnelled, */*\r\n
ClientID: WinNT_5.1_6.0.14.806_RealPlayer_R41UKD_en-GB_686\r\n
X-Actual-URL: rtsp://10.194.5.162:554/01.mp3\r\n
\r\n
Here's the server's response:
HTTP/1.0 200 OK\r\n
Server: RMServer 1.0\r\n
Expires: Mon, 18 May 1974 00:00:00 GMT\r\n
Pragma: no-cache\r\n
x-server-ipaddress: 10.194.5.162\r\n
Content-type: audio/x-pn-realaudio\r\n
\r\n
At this point 4 more bytes arrive from the server (their values are 48 02 02 00) - and that's it, nothing more. Does the server expect anything from the client at this point, and if so - what? Does this mode of operation work at all?
Some more info on this problem: apparently, the intended mechanism of working with RTSP over HTTP built into RealPlayer is as follows:
Try to connect to the following ports: 80, 8080, 554, 7070.
(Try also to download the file directly, just for the heck of it, by issuing GET http://hostname:port/mediafilename on port 80)
For each of the above ports, create 2 connections.
Send a GET request to one of the connections to the url http://hostname:port/SmpDsBhgRl<guid>?1="1", where <guid> is, yes, a freshly created GUID. Add a header to this request called X-Actual-URL containing the original RTSP URL.
Send a POST request on the other connection, to the URL http://hostname:port/SmpDsBhgRl with the GUID above as part of the body of the request. Send a Content-Length header of 32767 bytes, to prevent the proxy from closing the connection prematurely.
Start issuing commands to the server through the POST request, and get the corresponding RTSP stream as part of the GET response.
The strange stuff (if the above isn't strange enough) is that, for example, it works with Squid, but not if you use either of the ports 3128 or 8080! Somehow, the client uses the port it connects to to decide on the order of the requests or on when a request should be canceled, but anyway, as hard to believe as it is, it works with proxy port 9090, 3129, 8081, but not with 3128 or 8080.
Update #2: Here's the source of the RealPlayer with the explanation of the above behavior. Still no solution though.
Update #3: OK, in the light of the above, the magic value of 48 02 02 00 is clear: 48 == 'h' is for HTTP_RESPONSE, the next 02 is the length of the following data, the next 02 is called POST_NOT_RECEIVED (meaning that the POST request did not reach the server within a second from the corresponding GET request).
Update #4: This behavior (i.e. POST requests with huge Content-Length) is also characteristic of an ActiveX used by WebEx (and, possibly, many other web apps that need an open channel to the server).
First, you might want to read this:
http://developer.apple.com/quicktime/icefloe/dispatch028.html
Second, the HTTP requests (both GET and POST) need to be formatted so that they get proxied properly. I've seen proxies that insist on caching too much of the POST request, preventing it from reaching the server. Those proxies are buggy, but there's nothing you can do about that, and I was not able to work around that issue. Mostly I've seen this with anti-virus software that attempts to do transparent proxying of POST requests coming from the browser to scan them for private information like social security numbers. You might be running into the same problem.
Are you using McAfee's anti virus by any chance?
Also, it appears that Real invented its own way of doing the same thing, but the basic design is very similar - GET for the downstream link, POST for the upstream, with some magic cookie (in this case, the GUID) to tie the two together on the server. Either way, the POST should get to the server, and in your case it seems like it doesn't.
By the way, since the problem seems to be with the POST request not going through the proxy, how about posting that request, in addition to the GET?
See whether issuing the same request but bypassing the proxy (e.g., replay the request you posted above using Netcat) results in more than four bytes streamed in the response body.
See what TCP packets the proxy is receiving, for example, by eavesdropping on the TCP
traffic on the machine that's running the proxy, say, using Wireshark.