Here is the session i am using:
<sessions>
<session type="ts_http" name="Test" probability="100">
<for var="i" to="1" from="1">
<request subst="true">
<http version="1.1" contents="%%autoupload:readdata%%" method="POST" url="/UploadFile">
<http_header name="key" value="testkey"/>
<http_header name="Filename" value="test.zip"/>
</http>
</request>
</for>
</session>
The session has got only one post request. so the mean page response time and mean request response time are same as expected, in the tsung report.
but i was expecting the mean for user session also to be nearly same with deviation of only connection time.
below is snap of tsung report:
Name highest-10sec-mean lowest-10sec-mean Highest-Rate Mean-Rate Mean Count
connect 1.55 sec 4.11 msec 0.5 / sec 0.24 / sec 0.50 sec 47
page 26.35 sec 2.50 sec 0.9 / sec 0.24 / sec 12.83 sec 43
request 26.35 sec 2.50 sec 0.9 / sec 0.24 / sec 12.83 sec 43
session 30.83 sec 6.91 sec 0.9 / sec 0.25 / sec 17.73 sec 44
Wanted to understand what is it that getting added in the session mean time, such that the session time is higher than page/request time.
IIRC page means a consecutive sequence of requests within a session, without thinktimes/waits. Depending on the load you are configuring, session also includes work required to get the session started. As starting new sessions is not free, you could try to launch 1/10th users and let each user to 10 requests. page and session should be almost identical then.
It's a bit strange though, that you see almost 5 sec difference on the mean values. Could you provide more details on your environment? (os/tsung/erlang versions, entire configuration, ...)
Related
I have a recent updated MariaDB installation that suddenly hangs on a every-minute-cronjob after my computer resumes from sleep. Configuration worked fine before latest apt upgrade.
Process list shows an insert query from a ~100 insert batch operation.
I tried cancelling all running scripts before sending the computer to sleep, but at the next wakeup it will still fail. I have to kill -9 the database to even get it to restart.
Id? User? Host? db? Command? Time? State? Info? Progress?
52 xxxxxx localhost xxxxxx Execute 4 Update INSERT INTO xxxxxxx
and it will never finish. This also happens if i wait ~30 seconds after wakeup and start the script manually, so i kinda guess MariaDB handles sleep/resume state not too well.
Is this a known bug? Any way around having my PC run 24/7 to get around this?
Server variables too much for this post, only changed
innodb_buffer_pool_size = 64G
innodb_log_file_size = 16G
As suggested, i checked SHOW ENGINE INNODB STATUS while the lock is happening, cannot post this as a comment due to length
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 4664 srv_idle
srv_master_thread log flush and writes: 4663
----------
SEMAPHORES
----------
------------
TRANSACTIONS
------------
Trx id counter 314781977
Purge done for trx's n:o < 314781976 undo n:o < 0 state: running
History list length 142
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 314781976, ACTIVE 106 sec inserting
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1128, 1 row lock(s), undo log entries 1
MariaDB thread id 57, OS thread handle 140131630528064, query id 273212 localhost xxx Update
INSERT INTO xyzxyz (`xyz`, `xyz`, `time`, `xyz`, `xyz`) VALUES (?,?,?,?,?)
--------
FILE I/O
--------
Pending flushes (fsync) log: 0; buffer pool: 0
19307 OS file reads, 24298 OS file writes, 24302 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 932, free list len 2162, seg size 3095, 1 merges
merged operations:
insert 11, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 44207022268
Log flushed up to 44207022268
Pages flushed up to 44197540796
Last checkpoint at 44197540784
0 pending log flushes, 0 pending chkp writes
24306 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 68753031168
Dictionary memory allocated 424873192
Buffer pool size 4153344
Free buffers 4133333
Database pages 20011
Old database pages 7367
Modified db pages 3024
Percent of dirty pages(LRU & free pages): 0.073
Max dirty pages percent: 90.000
Pending reads 1
Pending writes: LRU 0, flush list 0
Pages made young 32, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 19243, created 741, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 20011, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 read views open inside InnoDB
Process ID=0, Main thread ID=0, state: sleeping
Number of rows inserted 24161, updated 113, deleted 0, read 623966
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
I checked whether it's a deadlock due to a bad update-query, but it's and INSERT-query with State update which just means mariadb is about to update the table (but never does). I use mariadb 10.6.7 on debian testing - i may need to use a stable release to sort this out. Sleeping happens from seconds (still happening) to hours.
I tried:
Removed PRIMARY and UNIQUE-key with the same indizes. Same problem
Change innodb_flush_log_at_trx_commit to zero to not have it update log after every insert - still stuck on first insert.
I have a data.frame of moderate size, and I tested storing and retrieving it to a network drive using both rds (uncompressed) and feather format. But the result shows that while write_feather is much faster than saveRDS, read_feather is much slower than readRDS.
Question(s). Does this have something to do with the particular network configuration of my workplace (i.e. is it just me)? or does it have something to do with the innate ability to handle remote files of read_feather and readRDS? Shall I stick to rds for now?
> print(object.size(impdata),unit="auto")
364.4 Mb
## SAVING
> system.time(feather::write_feather(impData,path="M:/waangData/test.feather"))
user system elapsed
0.52 0.16 4.80
> system.time(saveRDS(impData,file="M:/waangData/Data4predictImp.rds",compress=F))
user system elapsed
4.23 2.35 28.61
## READING
> system.time({t2=feather::read_feather("M:/waangData/test.feather")})
user system elapsed
0.59 1.54 134.39
> system.time({t=readRDS("M:/waangData/Data4predictImp.rds")})
user system elapsed
2.36 0.61 19.59
I am developing a couple of websites, but I only have paid for an EC2 nano instance on AWS. How many websites could I possible host there, assuming the websites will only have minimum traffic? Most of the websites are for personal use only.
Only one way to find out ;)
No definite answer possible because it depends on a lot of factors.
But if traffic is really low you will only be limited by the amount of disk space and as t2.nano runs on EBS storage this can be as big as you want. So you could fit a lot of websites!
t2.nano has only 512Mb memory so best to pick a not-so-memory-hungry webserver such as ngnix.
I run five very low traffic websites on my t2 nano - four of them Wordpress, one custom PHP. I run Nginx, PHP5.6, and MySQl 5.6 on the same instance. Traffic is extremely light, in the region of 2000 pages a day, which is about a page every 30 seconds. If you include static resources it'll be higher. CloudFlare runs as the CDN, which reduces static resource consumption significantly, but doesn't cache pages.
I have MySQL on the instance configured to use very little memory, currently 141MB physical RAM. Nginx takes around 10MB RAM. I have four PHP workers, each taking 150MB RAM, but of that 130MB is shared, so it's really 20MB per worker after the first.
Here's the output of a quick performance test on the t2.nano. Note that the Nginx page cache will be serving all of the pages.
siege -c 50 -t10s https://www.example.com -i -q -b
Lifting the server siege... done.
Transactions: 2399 hits
Availability: 100.00 %
Elapsed time: 9.60 secs
Data transferred: 14.82 MB
Response time: 0.20 secs
Transaction rate: 249.90 trans/sec ***
Throughput: 1.54 MB/sec
Concurrency: 49.42
Successful transactions: 2399
Failed transactions: 0
Longest transaction: 0.36
Shortest transaction: 0.14
Here it is with nginx page caching turned off
siege -c 5 -t10s https://www.example.com -i -q -b
Lifting the server siege... done.
Transactions: 113 hits
Availability: 100.00 %
Elapsed time: 9.99 secs
Data transferred: 0.70 MB
Response time: 0.44 secs
Transaction rate: 11.31 trans/sec ***
Throughput: 0.07 MB/sec
Concurrency: 4.95
Successful transactions: 113
Failed transactions: 0
Longest transaction: 0.70
Shortest transaction: 0.33
Nginx supports proxy response buffering, according to the mailing list (Nov 2013):
The proxy_buffering directive disables response buffering, not request
buffering.
As of now, there is no way to prevent request body buffering in nginx.
It's always fully read by nginx before a request is passed to an
upstream server. It's basically a part of nginx being a web
accelerator - it handles slow communication with clients by itself and
only asks a backend to process a request when everything is ready.
Since version 1.7.11 Nginx (Mar 2015) directive proxy_request_buffering see details:
Update: in nginx 1.7.11 the proxy_request_buffering directive is
available, which allows to disable buffering of a request body. It
should be used with care though, see docs.
See docs for more details:
Syntax: proxy_request_buffering on | off;
Default: proxy_request_buffering on;
Context: http, server, location
When buffering is enabled, the entire request body is read from the
client before sending the request to a proxied server.
When buffering is disabled, the request body is sent to the proxied
server immediately as it is received. In this case, the request cannot
be passed to the next server if nginx already started sending the
request body.
When HTTP/1.1 chunked transfer encoding is used to send the original
request body, the request body will be buffered regardless of the
directive value unless HTTP/1.1 is enabled for proxying.
The question is about buffering for Nginx (Load Balancer). For instance, we have the following scheme:
Nginx (LB) -> Nginx (APP) -> Backend
Nginx (APP) buffers the request from the load balancer and also buffers response from Backend. But does it make sense to buffer request and response on the Nginx (LB) side if both Nginx nodes are physically located close to each other (less than 2ms ping) with pretty fast network connection in between?
I measured the following benchmarks - Please note that this is for illustration only - Production loads would be significantly higher :
siege --concurrent=50 --reps=50 https://domain.com
proxy_request_buffering on;
Transactions: 2500 hits
Availability: 100.00 %
Elapsed time: 58.57 secs
Data transferred: 577.31 MB
Response time: 0.55 secs
Transaction rate: 42.68 trans/sec
Throughput: 9.86 MB/sec
Concurrency: 23.47
Successful transactions: 2500
Failed transactions: 0
Longest transaction: 2.12
Shortest transaction: 0.10
proxy_request_buffering off;
Transactions: 2500 hits
Availability: 100.00 %
Elapsed time: 57.80 secs
Data transferred: 577.31 MB
Response time: 0.53 secs
Transaction rate: 43.25 trans/sec
Throughput: 9.99 MB/sec
Concurrency: 22.75
Successful transactions: 2500
Failed transactions: 0
Longest transaction: 2.01
Shortest transaction: 0.09
Working on a project where we need to server a small static xml file ~40k / s.
All incoming requests are sent to the server from HAProxy. However, none of the requests will be persistent.
The issue is that when benchmarking with non-Persistent requests, the nginx instance caps out at 19 114 req/s. When persistent connections are enabled, performance increases by nearly an order of magnitude, to 168 867 req/s. The results are similar with G-wan.
When benchmarking non-persistent requests, CPU usage is minimal.
What can I do to increase performance with non-persistent connections and nginx?
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
finished in 52 sec, 315 millisec and 603 microsec, 19114 req/s, 5413 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 290000000 bytes total, 231000000 bytes http, 59000000 bytes data
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
finished in 5 sec, 921 millisec and 791 microsec, 168867 req/s, 48640 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 294950245 bytes total, 235950245 bytes http, 59000000 bytes data
Your 2 tests are similar (except HTTP Keep-Alives):
./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
And the one with HTTP Keep-Alives is 10x faster:
finished in 52 sec, 19114 req/s, 5413 kbyte/s
finished in 5 sec, 168867 req/s, 48640 kbyte/s
First, HTTP Keep-Alives (persistant connections) make HTTP requests run faster because:
Without HTTP Keep-Alives, the client must establish a new CONNECTION for EACH request (this is slow because of the TCP handshake).
With HTTP Keep-Alives, the client can send all requests at once (using the SAME CONNECTION). This is faster because there are less things to do.
Second, you say that the static file XML size is "small".
Is "small" nearer to 1 KB or 1 MB? We don't know. But that makes a huge difference in terms of available options to speedup things.
Huge files are usually served through sendfile() because it works in the kernel, freeing the usermode server from the burden of reading from disk and buffering.
Small files can use more flexible options available for application developers in usermode, but here also, file size matters (bytes and kilobytes are different animals).
Third, you are using 16 threads with your test. Are you really enjoying 16 PHYSICAL CPU Cores on BOTH the client and the server machines?
If that's not the case, then you are simply slowing-down the test to the point that you are no longer testing the web servers.
As you see, many factors have an influence on performance. And there are more with OS tuning (the TCP stack options, available file handles, system buffers, etc.).
To get the most of a system, you need to examinate all those parameters, and pick the best for your particular exercise.