I have 2 similar servers: 16 vCPUs, 2.4 GHz, Intel Xeon E5-2676v3, 64 GiB memory.
First of them generates load,second process requests.
Config load.ini:
[phantom]
address=0.0.0.0 ;target's address(chanched, of course)
port=443 ;target's port
rps_schedule=step(1000,10000,1000,15s) ;load scheme
ssl=1
header_http = 1.1
headers = [Host: api.somehost.io]
[Content-Type: application/json]
[Connection: close]
uris = /api/test
Expected:
Load will be generated step by step, start from 1 000 RPS, every 15 add 1 000 RPS, up to 10 000 RPS.
We have:
Expected 1000, have ~1000 (avg response time 7 ms).
Expected 2000, have ~2000 (avg response time 30 ms).
Expected 3000, have ~2700 (avg response time 250 ms).
Expected 4000, have ~2700 (avg response time 250 ms).
Further, no matter how much the planned increased RPS, actual remains within ~ 2700.
Have some suggestions:
1. Yandex Tank "understands", that server can not process such load and do not increase it.
2. Server can not establish more connections
Testing url - /api/test is processed by rails application + nginx as a proxy.
I carried out testing using static files to check second suggestion. Results: https://overload.yandex.net/8175
Number of connections more than 2700 = ~200 000.
But this number less than required in load.ini file - const(500000,15s).
Question: why Yandex Tand do not generate required load? or may be I understand results incorrectly?
With an average server's response time 250ms, for one second each phantom instance can send about 4 requests per second.
So with a default amount of phantom instances (1000) tank physically cannot send > ~4000rps - it has no available instances, all of them are busy sending and waiting data.
You could try to use more instances, like defining in [phantom] section instances=10000 It's mentioned in https://yandextank.readthedocs.io/en/latest/core_and_modules.html#basic-options
Related
Is there a way to automatically restart nginx with monit when it doesnt respond with status 200
The server seems to handle most of the requests but tends to lock up for a minute or two when it gets overwhelmed with requests. So I would like to have it check if the request responds to 504/502 to restart nginx
A simple sample to handle nginx is available from https://mmonit.com/wiki/Monit/ConfigurationExamples#NginX
and from https://www.mmonit.com/monit/documentation/monit.html#CONFIGURATION-EXAMPLES
some other samples are available.
And there are general information about handling the http status code from https://www.mmonit.com/monit/documentation/monit.html#HTTP
You can setup the Monit to test in shorter intervals, but one or two minutes is a good interval size (in general, from my point of view). In the Monit configuration monitrc file you can use a shorter interval.
set daemon 60 # check services at 60 seconds intervals
But you should not use a interval below 30 seconds (I prefer 60 seconds).
We have a web portal and we are using Nginx for rate-limiting by IP in this portal. So our current settings are something like
rate: 100 requests per second,
burst: 50
As per the documentation, Nginx uses milliseconds to calculate the number of requests. So, 100 requests per second translate to 1 request every 10 milliseconds. My question/confusion is how the will "burst" parameter behave if our Nginx server receives 10 requests in 10 milliseconds.
And is rate: 100 requests per second, burst: 50 equal to rate: 140 requests per second
burst: 10 .
I am using Fiddler to test my computer's http request performance, and
I want to send 200 parallel request exactly at same time (within variance 20ms).
My computer has 8 Core CPU.
And I found that I only can send 8x10 =80 requests at same time in maximum.
An example while sending 85 requests: https://upload.cc/i1/2021/03/17/VbUSmf.jpg
I'm quite sure that it's all about the limitation of Fiddler or the number of CPU core.
As my friend's computer has 10 core, and he can send 10x10 =100 requests exactly at same time.
How can I increase the maximum number of parallel request at same time?
~Greatly Appreciate for any help~
In short, I am trying to do a load test. But I cannot create the desired throughput on the IIS server (Windows Server 2016 Datacenter) even though there seems to be no bottleneck in terms of cpu, memory, disk or network.
Here is my configuration:
IIS Server: 16 vCPU, 32GB memory
SQL Server: 4 vCPU, 8GB memory
Test Server (sending the requests): 8 vCPU, 16GB memory
In order to remove concurrency limits on the IIS server, I did the following changes:
<serverRuntime appConcurrentRequestLimit="1000000" />
<applicationPool
maxConcurrentRequestsPerCPU="1000000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="1000000" />
Default Application Pool Queue Length: 65000
<processModel minWorkerThreads="5000">
I have created a WPF application that creates the desired number of concurrent requests towards the IIS server using HttpClient and deployed it on the test server. (I changed the service point default connection limit to 1000000 as well.) And I tested with 5000 requests which all returned 200 OK.
Normally, one request returns in 20ms. And here are the results of the test I obtained in the WPF application:
Total time starting from sending the first request through getting the last response: 9380ms
Average response time : 3919ms
Max. response time: 7243ms
Min. response time: 77ms
When I look at the performance counters on the test server, I see that 5000 requests completed in about 3 seconds. Here is the graph I obtained from perfmon:
But when I look at the performance counters on the IIS server, I see that requests are continually received and executed during the course of 9 seconds. So, the average throughput observed is about 400 requests per second. I also tried the test with 10000 requests but the average throughput is always around 400 req/sec.
Why doesn't ASP.NET complete receiving all the requests at the end of the first 3 seconds? How can I increase throughput to any desired value so that I can conduct a proper load test?
After a lot of experimenting, I found out that any value over 2000 for minWorkerThreads seem to be ignored. I checked it using the ThreadPool.GetMinThreads method. And I also added the maxWorkerThreads value of 2100 as #StephenCleary suggested. With these values, the problem disappeared. But the strange thing is that, I have not seen such a limitation on the minWorkerThreads value in any of the MS documentations.
I have about 120 000 records in the database.
The record example is: categories:example.com = ['category_1', 'category_2', 'category_3']
I have about 1k requests per second and for each request I call in nginx+lua
local categories, err = red:smembers("categories:"..clear_url)
Mostly requests has request time in access.log about 0-3ms but 2-4 requests per second has about 100-105ms.
Nginx+lua and redis db are on the same server and I use unix sock for connecting.
Load average is about 0,05
Please help me to find bottleneck.