notice that docker stop responding under load. Here is the steps to reproduce the issue:
docker-machine create -d virtualbox web
docker-machine ip web
192.168.99.253
eval $(docker-machine env web)
docker run --name app -d -p 3000:3000 ragesh/hello-express
ab -n 100000 -c 100 http://192.168.99.253:3000/
Server Hostname: 192.168.99.253
Server Port: 3000
Document Path: /
Document Length: 207 bytes
Concurrency Level: 100
Time taken for tests: 145.726 seconds
Complete requests: 100000
Failed requests: 0
Total transferred: 38900000 bytes
HTML transferred: 20700000 bytes
Requests per second: 686.22 [#/sec] (mean)
Time per request: 145.726 [ms] (mean)
Time per request: 1.457 [ms] (mean, across all concurrent requests)
Transfer rate: 260.68 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 15
Processing: 32 145 14.1 140 237
Waiting: 32 145 14.1 140 237
Total: 36 146 14.1 140 237
Percentage of the requests served within a certain time (ms)
50% 140
66% 146
75% 151
80% 155
90% 165
95% 173
98% 185
99% 196
100% 237 (longest request)
Everything looks good even -n 100000.
docker-machine create -d virtualbox router
eval $(docker-machine env router)
vi nginx.conf
upstream web {
server 192.168.99.253:3000;
}
server {
listen 80;
location / {
proxy_pass http://web;
}
}
docker run --name router -p 80:80 -v $(pwd)/nginx.conf:/etc/nginx/conf.d/default.conf:ro -d nginx
docker-machine ip router
192.168.99.252
ab -n 10000 -c 100 http://192.168.99.252:80/
Server Software: nginx/1.9.14
Server Hostname: 192.168.99.252
Server Port: 80
Document Path: /
Document Length: 207 bytes
Concurrency Level: 100
Time taken for tests: 32.631 seconds
Complete requests: 5957
Failed requests: 0
Total transferred: 2448327 bytes
HTML transferred: 1233099 bytes
Requests per second: 182.56 [#/sec] (mean)
Time per request: 547.773 [ms] (mean)
Time per request: 5.478 [ms] (mean, across all concurrent requests)
Transfer rate: 73.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2 52.6 0 3048
Processing: 2 168 56.2 164 3002
Waiting: 2 168 56.2 163 3002
Total: 2 169 97.7 164 5032
Percentage of the requests served within a certain time (ms)
50% 164
66% 171
75% 176
80% 178
90% 185
95% 192
98% 199
99% 209
100% 5032 (longest request)
Expect the Requests per second will drop, but not except it drop so dramatically. And run ab several times, got connection error very often.
Did I do something wrong?
Related
Given the following most basic of ASP.NET Core applications (note the Thread.Sleep):
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.Configure(appBuilder =>
appBuilder.Run(async context =>
{
var stopwatch = Stopwatch.StartNew();
Thread.Sleep(1000);
await context.Response.WriteAsync($"Finished in {stopwatch.ElapsedMilliseconds} milliseconds.");
}));
});
}
And the following appsettings.json
{
"Logging": {
"LogLevel": {
"Default": "None",
"Microsoft.AspNetCore.Hosting.Diagnostics" : "Information",
}
},
"AllowedHosts": "*"
}
If I run an even moderate load test (100 requests, using bombardier in my case) I see latency of around 5 seconds.
~/go/bin/bombardier http://localhost:5000 -l -n 100 -t 60s
Bombarding http://localhost:51568 with 100 request(s) using 125 connection(s)
100 / 100 [=================================================================================================================================] 100.00% 16/s 6s
Done!
Statistics Avg Stdev Max
Reqs/sec 19.46 250.28 4086.58
Latency 5.21s 366.21ms 6.05s
Latency Distribution
50% 5.05s
75% 5.05s
90% 6.04s
95% 6.05s
99% 6.05s
HTTP codes:
1xx - 0, 2xx - 100, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 3.31KB/s
However, all I see in the logs are
info: Microsoft.AspNetCore.Hosting.Diagnostics[2] Request finished in
1003.3658ms 200
Clearly the requests are taking more than 1 second. I believe the unaccounted 4 seconds are when the request is queued on the ThreadPool.
So my question is how can I measure this latency from inside my application?
I ran your application in my environment, and the ASP.NET logs very similar to yours:
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1022.4689ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/favicon.ico
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1004.1694ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1003.4582ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/favicon.ico
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1004.3703ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1003.3915ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/favicon.ico
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1004.3106ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1003.122ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1017.028ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1004.2742ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1006.5832ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1004.9214ms 200
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 GET http://localhost:5000/
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 1012.4532ms 200
As for bombardier, I got below output:
bombardier-windows-amd64.exe http://localhost:5000 -l -n 100 -t 60s
Bombarding http://localhost:5000 with 100 request(s) using 125 connection(s)
100 / 100 [==========================================================================================] 100.00% 11/s 8s
Done!
Statistics Avg Stdev Max
Reqs/sec 11.29 99.10 1303.09
Latency 5.78s 1.42s 8.78s
Latency Distribution
50% 5.17s
75% 7.79s
90% 7.88s
95% 8.24s
99% 8.34s
HTTP codes:
1xx - 0, 2xx - 100, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 2.27KB/s
And below is Chrome Dev tools network output:
Also I test it with cURL (note that I had to apply echo [%date%, %time%] second time after the curl command manually, better accuracy can be done by doing it in .bat file
but overall the output confirms that the request took ~ 1100 ms:
C:\curl\bin>echo [%date%, %time%] && curl http://localhost:5000/
[Sun 07/12/2020, 13:12:35.61]
Finished in 1001 milliseconds.
C:\curl\bin\>echo [%date%, %time%]
[Sun 07/12/2020, 13:12:37.45]
So based on all above, it seems that bombardier output differ than what other tools reports, hence we might misunderstood the meaning of its output latency! I made little change to the command by letting it handle the 100 requests using 10 connections only instead of default 125 connections, and the output was:
bombardier-windows-amd64.exe -c10 http://localhost:5000 -l -n 100 -t 60s
Bombarding http://localhost:5000 with 100 request(s) using 10 connection(s)
100 / 100 [==========================================================================================] 100.00% 8/s 11s
Done!
Statistics Avg Stdev Max
Reqs/sec 9.07 26.73 211.08
Latency 1.06s 179.57ms 2.04s
Latency Distribution
50% 1.01s
75% 1.02s
90% 1.07s
95% 1.19s
99% 1.86s
HTTP codes:
1xx - 0, 2xx - 100, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 1.79KB/s
Based on all above, I confirm that the single request taking ~ 1 second. as for bulk requests benchmarks, please try Postman otherwise we need to dig deeper to understand whatbombardier Latency means exactly and how its calculated.
Update
I made small console tool that fire HttpClient in Bulk and it confirms the ~ 1 second response time, also I tried two benchmarking tools from awesome-http-benchmark:
baton.exe -u http://localhost:5000 -c 10 -r 100
Configuring to send GET requests to: http://localhost:5000
Generating the requests...
Finished generating the requests
Sending the requests to the server...
Finished sending the requests
Processing the results...
====================== Results ======================
Total requests: 100
Time taken to complete requests: 10.2670832s
Requests per second: 10
===================== Breakdown =====================
Number of connection errors: 0
Number of 1xx responses: 0
Number of 2xx responses: 100
Number of 3xx responses: 0
Number of 4xx responses: 0
Number of 5xx responses: 0
=====================================================
cassowary run -u http://localhost:5000 -c 10 -n 100
Starting Load Test with 100 requests using 10 concurrent users
100% |████████████████████████████████████████| [10s:0s] 10.2299727s
TCP Connect.....................: Avg/mean=1.90ms Median=2.00ms p(95)=2.00ms
Server Processing...............: Avg/mean=1014.94ms Median=1008.00ms p(95)=1093.00ms
Content Transfer................: Avg/mean=0.17ms Median=0.00ms p(95)=1.00ms
Summary:
Total Req.......................: 100
Failed Req......................: 0
DNS Lookup......................: 5.00ms
Req/s...........................: 9.78
Finally I used nginx on port 2020 as rev proxy in front of kestrel to see bombardier output:
bombardier-windows-amd64.exe http://localhost:2020 -l -n 100 -t 60s
Bombarding http://localhost:2020 with 100 request(s) using 125 connection(s)
100 / 100 [==========================================================================================] 100.00% 9/s 10s
Done!
Statistics Avg Stdev Max
Reqs/sec 11.76 128.07 2002.66
Latency 9.08s 761.43ms 10.04s
Latency Distribution
50% 9.06s
75% 9.07s
90% 9.07s
95% 10.02s
99% 10.04s
HTTP codes:
1xx - 0, 2xx - 95, 3xx - 0, 4xx - 0, 5xx - 5
others - 0
Throughput: 2.51KB/s
As you can see, even with nginx it shows 9 seconds latency! that's should conclude an issue with Bombarding latency definition/calcs.
Below is nginx config:
server {
listen 2020;
location / {
proxy_pass http://localhost:5000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_redirect off;
}
Bonus
If you want to hack the performance of thread.sleep to be similar as await Task.Delay then change Main to:
public static void Main(string[] args)
{
ThreadPool.SetMinThreads(130, 130);//Don't Use in Production.
CreateHostBuilder(args).Build().Run();
}
I am hosting an HTTP page behind CloudFront, Amazon AWS's CDN. From my location, a typical request to this page competes in less than 20 milliseconds:
$ ab http://d2szlqeo43tgz6.cloudfront.net/jquery/jquery-2.1.4.min.js
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking d2szlqeo43tgz6.cloudfront.net (be patient).....done
Server Software: AmazonS3
Server Hostname: d2szlqeo43tgz6.cloudfront.net
Server Port: 80
Document Path: /jquery/jquery-2.1.4.min.js
Document Length: 84345 bytes
Concurrency Level: 1
Time taken for tests: 0.014 seconds
Complete requests: 1
Failed requests: 0
Write errors: 0
Total transferred: 84887 bytes
HTML transferred: 84345 bytes
Requests per second: 70.14 [#/sec] (mean)
Time per request: 14.258 [ms] (mean)
Time per request: 14.258 [ms] (mean, across all concurrent requests)
Transfer rate: 5814.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 1 0.0 1 1
Processing: 13 13 0.0 13 13
Waiting: 4 4 0.0 4 4
Total: 14 14 0.0 14 14
Whereas if I try to measure RTT via ping, the times are much greater than 20 ms:
$ ping d2szlqeo43tgz6.cloudfront.net
PING d2szlqeo43tgz6.cloudfront.net (54.230.149.139): 56 data bytes
64 bytes from 54.230.149.139: icmp_seq=0 ttl=53 time=77.230 ms
64 bytes from 54.230.149.139: icmp_seq=1 ttl=53 time=79.888 ms
64 bytes from 54.230.149.139: icmp_seq=2 ttl=53 time=96.338 ms
64 bytes from 54.230.149.139: icmp_seq=3 ttl=53 time=76.166 ms
So how can CloudFront serve HTTP content so much faster than it can respond to an ICMP echo request?
Just to be sure, I checked the response in Wireshark, and all HTTP content is coming from the network; no local cache is involved.
I am testing siege against a simple server which outputs the n'th Fibonacci number. The server works great when using curl:
[mto#localhost ~]$ curl http://localhost:8000?q=8
21
Doing the same with siege, yields the following:
[mto#localhost ~]$ siege 'http://localhost:8000?q=8' -r 4 -c 1
** SIEGE 3.0.9
** Preparing 1 concurrent users for battle.
The server is now under siege...
HTTP/1.1 400 0.00 secs: 73 bytes ==> GET /
HTTP/1.1 400 0.00 secs: 73 bytes ==> GET /
HTTP/1.1 400 0.00 secs: 73 bytes ==> GET /
HTTP/1.1 400 0.00 secs: 73 bytes ==> GET /
done.
Transactions: 4 hits
Availability: 100.00 %
Elapsed time: 1.01 secs
Data transferred: 0.00 MB
Response time: 0.00 secs
Transaction rate: 3.96 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 0.00
Successful transactions: 0
Failed transactions: 0
Longest transaction: 0.00
Shortest transaction: 0.00
FILE: /home/mto/siege.log
You can disable this annoying message by editing
the .siegerc file in your home directory; change
the directive 'show-logfile' to false.
As you can see the server is giving 400. My webserver, written with tornado, outputs the following:
[W 150311 16:58:20 web:1404] 400 GET / (127.0.0.1): Missing argument q
[W 150311 16:58:20 web:1811] 400 GET / (127.0.0.1) 0.85ms
[W 150311 16:58:20 web:1404] 400 GET / (127.0.0.1): Missing argument q
[W 150311 16:58:20 web:1811] 400 GET / (127.0.0.1) 0.71ms
[W 150311 16:58:20 web:1404] 400 GET / (127.0.0.1): Missing argument q
[W 150311 16:58:20 web:1811] 400 GET / (127.0.0.1) 0.72ms
[W 150311 16:58:20 web:1404] 400 GET / (127.0.0.1): Missing argument q
[W 150311 16:58:20 web:1811] 400 GET / (127.0.0.1) 0.79ms
How do I pass the query parameters to siege? The Siege man page says the following:
...
You can pass parameters using GET much like you would in a web browser:
www.haha.com/form.jsp?first=homer&last=simpson
If you invoke the URL as a command line argument, you should probably place it in
quotes.
...
I have tried to put the url in single, double and no quotes. I have also written the urls in a file and passed it to siege using -f, but no luck. I am using:
My environment:
SIEGE 3.0.9
GNOME Terminal 3.10.2
Fedora release 20 (Heisenbug)
Any ideas?
I am using SIEGE 4.0.4 and found that double quotes works from the answer of below question:
https://stackoverflow.com/a/9311812/5824101
siege does not like the url to be at the following form:
http://localhost:8000?q=8
To use query parameters I have to have a url with a path:
http://localhost:8000/fib?q=8
Then it works fine. I have not been able to find a work around
I am running multiple instances of siege, so siege is reusing the ports as a results some of the requests are not going through. Is there a way where the different siege instances could use different port ranges?
HTTP/1.1 200 0.00 secs: 146 bytes ==>
HTTP/1.1 200 0.00 secs: 146 bytes ==>
HTTP/1.1 200 0.00 secs: 146 bytes ==>
HTTP/1.1 200 0.00 secs: 146 bytes ==>
HTTP/1.1 200 0.01 secs: 146 bytes ==>
HTTP/1.1 200 0.00 secs: 146 bytes ==>
HTTP/1.1 200 0.01 secs: 146 bytes ==>
[alert] socket: 671299328 select timed out: Connection timed out
[alert] socket: 788797184 select timed out: Connection timed out
[alert] socket: 721655552 select timed out: Connection timed out
[alert] socket: 738440960 select timed out: Connection timed out
HTTP/1.1 200 0.01 secs: 146 bytes ==> /
HTTP/1.1 200 0.01 secs: 146 bytes ==> /
[alert] socket: 822368000 select timed out: Connection timed out
HTTP/1.1 200 0.01 secs: 146 bytes ==> /
HTTP/1.1 200 0.01 secs: 146 bytes ==> /
HTTP/1.1 200 0.01 secs: 146 bytes ==> /
I see you have a lot of requests one after another, do you consider that you can have problems with KeepAlive.
On server sockets are opened for a little bit longer than connection alone. You can run out of ports quite qucik if KeepAlive is set to high value.
You can set
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
in /etc/sysctl file and run #sysctl -p to active it.
Please have a try. Hope this can help you.
I'm trying to test my server on highload resistance with siege utility:
siege http://my.server.ru/ -d1 -r10 -c100
Siege outputs a lot of messages like this:
HTTP/1.1 200 0.46 secs: 10298 bytes ==> /
but sometimes there are error messages like this:
Error: socket: unable to connect sock.c:220: Connection timed out
or this:
warning: socket: -598608128 select timed out: Connection timed out
There is siege report after testing:
Transactions: 949 hits
Availability: 94.90 %
...
Successful transactions: 949
Failed transactions: 51
Longest transaction: 9.87
Shortest transaction: 0.37
In nginx logs on my server, only 950 messages with code 200 and response that all right.
"GET / HTTP/1.1" 200 10311 "-" "JoeDog/1.00 [en] (X11; I; Siege 2.68)"
Can anyone tell me what this means
Error: socket: unable to connect sock.c:220: Connection timed out
warning: socket: -598608128 select timed out: Connection timed out
and why in my nginx logs I only see responses with code 200?
It probably means your pipe is full and can't handle more connections. You can't make nginx or nginx backends accept more connections if if your pipe is full. Try testing against localhost. You will then be testing the stack rather than the stack and the pipe. It will resemble real load less, but give you an idea what you can handle with the bigger pipe.