I have about 120 000 records in the database.
The record example is: categories:example.com = ['category_1', 'category_2', 'category_3']
I have about 1k requests per second and for each request I call in nginx+lua
local categories, err = red:smembers("categories:"..clear_url)
Mostly requests has request time in access.log about 0-3ms but 2-4 requests per second has about 100-105ms.
Nginx+lua and redis db are on the same server and I use unix sock for connecting.
Load average is about 0,05
Please help me to find bottleneck.
Related
In short, I am trying to do a load test. But I cannot create the desired throughput on the IIS server (Windows Server 2016 Datacenter) even though there seems to be no bottleneck in terms of cpu, memory, disk or network.
Here is my configuration:
IIS Server: 16 vCPU, 32GB memory
SQL Server: 4 vCPU, 8GB memory
Test Server (sending the requests): 8 vCPU, 16GB memory
In order to remove concurrency limits on the IIS server, I did the following changes:
<serverRuntime appConcurrentRequestLimit="1000000" />
<applicationPool
maxConcurrentRequestsPerCPU="1000000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="1000000" />
Default Application Pool Queue Length: 65000
<processModel minWorkerThreads="5000">
I have created a WPF application that creates the desired number of concurrent requests towards the IIS server using HttpClient and deployed it on the test server. (I changed the service point default connection limit to 1000000 as well.) And I tested with 5000 requests which all returned 200 OK.
Normally, one request returns in 20ms. And here are the results of the test I obtained in the WPF application:
Total time starting from sending the first request through getting the last response: 9380ms
Average response time : 3919ms
Max. response time: 7243ms
Min. response time: 77ms
When I look at the performance counters on the test server, I see that 5000 requests completed in about 3 seconds. Here is the graph I obtained from perfmon:
But when I look at the performance counters on the IIS server, I see that requests are continually received and executed during the course of 9 seconds. So, the average throughput observed is about 400 requests per second. I also tried the test with 10000 requests but the average throughput is always around 400 req/sec.
Why doesn't ASP.NET complete receiving all the requests at the end of the first 3 seconds? How can I increase throughput to any desired value so that I can conduct a proper load test?
After a lot of experimenting, I found out that any value over 2000 for minWorkerThreads seem to be ignored. I checked it using the ThreadPool.GetMinThreads method. And I also added the maxWorkerThreads value of 2100 as #StephenCleary suggested. With these values, the problem disappeared. But the strange thing is that, I have not seen such a limitation on the minWorkerThreads value in any of the MS documentations.
Nginx's log has the ability to log the $bytes_sent and I am trying to figure out if you could simply collect the requests from the last second, sum them up and get the bandwidth per second.
My question is, when nginx logs the bytes sent for a http 200 request, is this really the amount of data that the customer received, or in other words, does this really represent the current bits per second that I would observe on this TCP port (say, 8080) if I used something like iftop?
My goal is to find a way to log the bits per second going out of the Nginx vhost(server block).
I have 2 similar servers: 16 vCPUs, 2.4 GHz, Intel Xeon E5-2676v3, 64 GiB memory.
First of them generates load,second process requests.
Config load.ini:
[phantom]
address=0.0.0.0 ;target's address(chanched, of course)
port=443 ;target's port
rps_schedule=step(1000,10000,1000,15s) ;load scheme
ssl=1
header_http = 1.1
headers = [Host: api.somehost.io]
[Content-Type: application/json]
[Connection: close]
uris = /api/test
Expected:
Load will be generated step by step, start from 1 000 RPS, every 15 add 1 000 RPS, up to 10 000 RPS.
We have:
Expected 1000, have ~1000 (avg response time 7 ms).
Expected 2000, have ~2000 (avg response time 30 ms).
Expected 3000, have ~2700 (avg response time 250 ms).
Expected 4000, have ~2700 (avg response time 250 ms).
Further, no matter how much the planned increased RPS, actual remains within ~ 2700.
Have some suggestions:
1. Yandex Tank "understands", that server can not process such load and do not increase it.
2. Server can not establish more connections
Testing url - /api/test is processed by rails application + nginx as a proxy.
I carried out testing using static files to check second suggestion. Results: https://overload.yandex.net/8175
Number of connections more than 2700 = ~200 000.
But this number less than required in load.ini file - const(500000,15s).
Question: why Yandex Tand do not generate required load? or may be I understand results incorrectly?
With an average server's response time 250ms, for one second each phantom instance can send about 4 requests per second.
So with a default amount of phantom instances (1000) tank physically cannot send > ~4000rps - it has no available instances, all of them are busy sending and waiting data.
You could try to use more instances, like defining in [phantom] section instances=10000 It's mentioned in https://yandextank.readthedocs.io/en/latest/core_and_modules.html#basic-options
I am designing a temperature and humidity logging system using microcontrollers and embedded C. The system sends the data to the server using GET method (could be POST,too) whenever it gathers a new data from the sensors. Whenever the power and/or the internet is gone, it logs to an SD card.
I want to send these logged data to the server whenever the internet comes back. When sent separately, each of my request is as follows and it takes about 5 seconds to complete even on my local network.
GET /add.php?userID=0000mac=000:000:000:000:000:000&id=0000000000sensor=000&temp=%2B000.000&hum=000.000 HTTP/1.1\r\nHost: 192.168.10.25\r\n\r\n
However, since my available RAM is very limited to only about 400 bytes in this microcontroller, I cannot buffer and send all the logged data in one request.
For an electricity/internet loss of 2 days, about 3000 of data set is logged. How do I send these to the server in a very quick way?
Well I think a simple for loop will do it. Here is the pseudocode:
foreach(loggedRequest){
loadTheRequestFromTheLogFile();
sendTheDataForThisRequest(); // Hang here until the server returns a response
clearMemoryFromPreviousRequest();
}
By waiting for the server's response, you will be sure that the data got to the server as well as that you will not have the ram full.
You can also go with an inner for loop with a fixed number of requests and then wait for all their responses. This will be faster but you have to do multiple tests to be sure you're not reaching to the ram limit.
What is the maximum size of HTTP packets? I am interested in the size of the response to an HTTP GET request (Not this! This question is about the request size). Is there a size at all? If I download a 1GiB file, does that end up as just 1 HTTP GET request? (Intuitively, I do not think this happens - Also, partial downloads / multi threaded downloaders would not work).
I know that there is a maximum length of an IP packet and TCP packets that are longer than that are fragmented across multiple IP packets. Does such a thing happen with HTTP as well? The reason I am looking for an answer to this question is to figure out the AWS S3 billing scheme which charges 1c / 10K get requests. So how many GET requests are begin served for 1GiB.
I got an answer on AWS S3 Forum by Seungyoung Kim
If you download the 5GiB file using GET request. S3's billing system
will count it as 1 GET request and 5GiB data transfer. FYI, if you
stop downloading file at 2GiB out of 5GiB, then they will count only
2GiB data transfer.
GET request is not fragmented normally unless you split up the call
into several consecutive GET range calls.
So, if you issue 5 individual GET requests using range header for each
1GiB data block, they will count it as 5 GET request but still same
5GiB(1GiB x 5 times) data transfer. Technically little more than 5GiB
due to additional HTTP protocol headers.
If this is right, there is no packet fragmentation of HTTP GET Response (although they come across several TCP packets and a lost TCP packet is handled by TCP and does not illicit another GET request to resume)