I am trying AWS r3.4xlarge instance with RStudio Server Amazon Machine Image (AMI).
ubuntu#ip:~$ free -m
total used free shared buffers cached
Mem: 122953 8394 114558 0 13 232
-/+ buffers/cache: 8148 114804
Swap: 1023 0 1023
ubuntu#ip:~$
With 122GB RAM and 16vCPU I thought that R would be really fast with medium-sized datasetes. However, R uses only 8.3 GB when I run rpart() on dataset with 10M rows and 21 columns (german data replicated 1,000 times)
This is reasult of ulimit -a:
ubuntu#ip:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 983543
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 983543
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
rpart runs now for more than 30 minutes and I wonder why. Is this more CPU intensive task? It does not seem so from htop:
rpart() finished now! Is there a way how to make it faster or this slowness is cannot be avoided when using rpart?
Related
My.cnf
[mysql]
port = 3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
# === Required Settings ===
basedir = /usr
bind_address = 127.0.0.1
datadir = /var/lib/mysql
max_allowed_packet = 1024M
max_connect_errors = 1000000
pid-file=/var/lib/mysql/mysql.pid
port = 3306
skip_name_resolve
socket=/var/lib/mysql/mysql.sock
tmpdir = /tmp
sql_mode = ""
thread_handling = pool-of-threads
# === InnoDB Settings ===
default_storage_engine = InnoDB
innodb_buffer_pool_size = 36G # Use up to 70-80% of RAM
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_log_buffer_size = 16M
innodb_log_file_size = 1G
innodb_sort_buffer_size = 4M # UPD - Defines how much data is read into memory for sorting operations before writing to disk (default is 1M / max is 64M)
innodb_stats_on_metadata = 0
innodb_read_io_threads = 64
innodb_write_io_threads = 64
innodb_io_capacity = 2000 # Depends on the storage tech - use 2000 for SSD, more for NVMe
innodb_io_capacity_max = 4000 # Usually double the value of innodb_io_capacity
# === Connection Settings ===
max_connections = 100 # UPD - Important: high no. of connections = high RAM consumption
back_log = 512
thread_cache_size = 100
thread_stack = 192K
interactive_timeout = 180
wait_timeout = 180
# === Buffer Settings ===
join_buffer_size = 4M # UPD
read_buffer_size = 3M # UPD
read_rnd_buffer_size = 4M # UPD
sort_buffer_size = 4M # UPD
# === Table Settings ===
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
open_files_limit = 60000 # UPD
max_heap_table_size = 128M # Increase to 256M or 512M if you have lots of temporary tables because of missing indices in JOINs
tmp_table_size = 128M # Use same value as max_heap_table_size
# === Search Settings ===
ft_min_word_len = 3 # Minimum length of words to be indexed for search results
# === Binary Logging ===
disable_log_bin = 1
# === Error & Slow Query Logging ===
log_error = /var/lib/mysql/mysql_error.log
log_queries_not_using_indexes = 0 # Disabled on production
long_query_time = 5
slow_query_log = 0 # Disabled on production
slow_query_log_file = /var/lib/mysql/mysql_slow.log
[mysqldump]
quick
quote_names
max_allowed_packet = 1024M
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
My ram & cpu
[root#localhost ~]# cat /proc/cpuinfo | grep processor | wc -l
24
[root#localhost ~]# free -m
total used free shared buff/cache available
Mem: 48125 6136 12446 50 29541 41490
Swap: 0 0 0
mysqltuner
[root#localhost ~]# ./mysqltuner.pl
>> MySQLTuner 1.9.8
* Jean-Marie Renouard <jmrenouard#gmail.com>
* Major Hayden <major#mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.pl/
>> Run with '--help' for additional options and output filtering
[--] Skipped version check for MySQLTuner script
[!!] Your MySQL version 10.8.3-MariaDB is EOL software! Upgrade soon!
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/mysql_error.log exists
[--] Log file: /var/lib/mysql/mysql_error.log(70K)
[OK] Log file /var/lib/mysql/mysql_error.log is not empty
[OK] Log file /var/lib/mysql/mysql_error.log is smaller than 32 Mb
[OK] Log file /var/lib/mysql/mysql_error.log is readable.
[!!] /var/lib/mysql/mysql_error.log contains 339 warning(s).
[!!] /var/lib/mysql/mysql_error.log contains 51 error(s).
[--] 2 start(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:05 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 2) 2022-06-17 21:30:12 0 [Note] /usr/sbin/mariadbd: ready for connections.
[--] 1 shutdown(s) detected in /var/lib/mysql/mysql_error.log
[--] 1) 2022-06-19 16:42:04 0 [Note] /usr/sbin/mariadbd: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in Aria tables: 32.0K (Tables: 1)
[--] Data in InnoDB tables: 1.7G (Tables: 845)
[OK] Total fragmented tables: 0
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Views Metrics -----------------------------------------------------------------------------
-------- Triggers Metrics --------------------------------------------------------------------------
-------- Routines Metrics --------------------------------------------------------------------------
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 4d 6h 58m 28s (166M q [448.474 qps], 937K conn, TX: 203G, RX: 12G)
[--] Reads / Writes: 98% / 2%
[--] Binary logging is disabled
[--] Physical Memory : 47.0G
[--] Max MySQL memory : 137.9G
[--] Other process memory: 0B
[--] Total buffers: 36.4G global + 1.0G per thread (100 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[!!] Maximum reached memory usage: 93.2G (198.36% of installed RAM)
[!!] Maximum possible memory usage: 137.9G (293.37% of installed RAM)
[!!] Overall possible memory usage with other process exceeded memory
[OK] Slow queries: 0% (53/166M)
[OK] Highest usage of available connections: 56% (56/100)
[OK] Aborted connections: 0.01% (107/937871)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 874K sorts)
[!!] Joins performed without indexes: 46708
[OK] Temporary tables created on disk: 2% (6K on disk / 305K total)
[--] Thread cache not used with thread pool enabled
[OK] Table cache hit rate: 99% (77M hits / 77M requests)
[OK] table_definition_cache(40000) is upper than number of tables(1137)
[OK] Open file limit used: 0% (29/32K)
[OK] Table locks acquired immediately: 100% (2K immediate / 2K locks)
-------- Performance schema ------------------------------------------------------------------------
[!!] Performance_schema should be activated.
[--] Sys schema is installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 24 thread(s).
[--] Using default value is good enough for your version (10.8.3-MariaDB)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] No MyISAM table(s) detected ....
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 36.0G/1.7G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (2.77777777777778 %): 1.0G * 1/36.0G should be equal to 25%
[--] Number of InnoDB Buffer Pool Chunk : 64 for 1 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (36551426226 hits/ 36551507601 total)
[OK] InnoDB Write log efficiency: 90.84% (7736631 hits/ 8516998 total)
[OK] InnoDB log waits: 0.00% (0 waits / 780367 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/344.0K
[!!] Aria pagecache hit rate: 82.5% (38K cached / 6K reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: OFF
[--] Semi synchronous replication Slave: OFF
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
You are using n unsupported version for production environments
Upgrade as soon as possible to a supported version !
Check warning line(s) in /var/lib/mysql/mysql_error.log file
Check error line(s) in /var/lib/mysql/mysql_error.log file
Reduce your overall MySQL memory footprint for system stability
Dedicate this server to your database for highest performance.
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
Performance schema should be activated for better diagnostics
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this:
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
join_buffer_size (> 4.0M, or always use indexes with JOINs)
performance_schema=ON
innodb_log_file_size should be (=9G) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
[root#localhost ~]#
Tuning Primer
[root#localhost ~]# ./tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 10.8.3-MariaDB x86_64
Uptime = 4 days 9 hrs 29 min 40 sec
Avg. qps = 454
Total Questions = 172718132
Threads Connected = 33
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/10.8/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 5.000000 sec.
You have 53 out of 172718244 that take longer than 5.000000 sec. to complete
Your long_query_time seems to be fine
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/10.8/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 100
Current threads_cached = 0
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 100
Current threads_connected = 33
Historic max_used_connections = 56
The number of used connections is 56% of the configured maximum.
Your max_connections variable seems to be fine.
No InnoDB Support Enabled!
MEMORY USAGE
Max Memory Ever Allocated : 36.97 G
Configured Max Per-thread Buffers : 1.48 G
Configured Max Global Buffers : 36.14 G
Configured Max Memory Limit : 37.62 G
Physical Memory : 46.99 G
Max memory limit seem to be within acceptable norms
KEY BUFFER
No key reads?!
Seriously look into using some indexes
Current MyISAM index space = 0 bytes
Current key_buffer_size = 128 M
Key cache miss rate is 1 : 0
Key buffer free ratio = 81 %
Your key_buffer_size seems to be fine
QUERY CACHE
Query cache is enabled
Current query_cache_size = 1 M
Current query_cache_used = 16 K
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 1.65 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size
SORT OPERATIONS
Current sort_buffer_size = 4 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine
JOINS
./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
Current join_buffer_size = 4.00 M
You have had 48458 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 16319 tables
Current table_definition_cache = 40000 tables
You have a total of 957 tables
You have 1124 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 128 M
Current tmp_table_size = 128 M
Of 315112 temp tables, 2% were created on disk
Created disk tmp tables ratio seems fine
TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 155 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 0 : 172720142
Your table locking seems to be fine
[root#localhost ~]#
I can allocate 40gb ram to mariadb server.
I need 8gb ram. We can use all 24 processors.
Please help me how to do the settings.
The site will be used for a web-based game.
The tables were all using MyISAM.
I converted all of them to innodb and it works fine.
If you need additional information, let me know in the comments.
These seem excessive:
table_definition_cache = 40000 # UPD
table_open_cache = 40000 # UPD
Will you have thousands of tables? Why? Even if you do have that many, these are caches, hence they don't have to be big enough to handle everything.
For better performance turn on the slowlog with a low value such as long_query_time = 1. After a few hours, use pt-query-digest. More on the SlowLog
For a deeper analysis of the settings: http://mysql.rjweb.org/doc.php/mysql_analysis#tuning
Rate Per Second = RPS
Suggestions to consider for your my.cnf [mysqld] section to improve performance
read_rnd_buffer_size=32K # from 4M to reduce handler_read_rnd_next RPS of 20,873
read_buffer_size=1M # from 3M to reduce handler_read_next RPS of 20,918
innodb_buffer_pool_size=4G # from 36G because you only have 1.7G of data and indexes
net_buffer_length=96K # from 16K to accommodate sending 700MB data per hr
Let us know how your system is performing in a few days, please. View profile for contact information and we do have FREE Utility Scripts to assist with performance tuning.
This is only the beginning of improving performance for your installation. As you get deeper into the environment you will find many opportunities to improve response time and reduce system overhead. We are available to assist.
I have just done a clean install of R on Ubuntu 18.04, and it does not really work.
I can do stuff like make vectors and dataframes, and get summaries. I can also make a histogram with the hist command just fine. However, the plot command does not work.
The following code, which is very basic and should work just fine:
data(faithful)
plot(faithful$eruptions)
runs for about 30 seconds, before giving the following error:
Error: C stack usage 7970244 is too close to the limit
I have seen lots of posts of other people having the same error, but it seems to be because they are dealing with large datasets/lots of recursion or something like that, but I have this problem even with a dataset of just 3 values. R should definitely be able to handle this without me increasing the limit, and it should not take 30 seconds to run.
Does anybody know what the problem could be?
Edit:
Output of ulimit -a:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 28697
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 28697
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Version info:
R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
This is plain R in the terminal, not through any IDE.
Edit 2:
I have discovered that it works just fine on other user accounts on the computer. I've also started getting other weird issues (after a reboot I have to login to a Gnome session, then log out before I can log into plasma, but other users don't have that problem, sometimes the terminal can't launch, lots of things are crashing, etc) so I think this has nothing to do with R and is much bigger. Unfortunately, it might be a hardware issue (this computer has had wacky issues before on other operating systems).
I have a minimal Go HTTP server (code below). When I start the server with go run server.go and then fire off 5000 concurrent requests using
ab -c 5000 -n 5000 http://localhost:8080/
everything works as expected. However, if I start my server with the race detector flag:
go run -race server.go
then I get an issue running ApacheBench even with only 1000 concurrent requests:
apr_socket_recv: Connection reset by peer (54)
Interestingly, my Go server doesn't crash or print any error messages, and is able to continue receiving new requests. This suggests that the problem is not the Go process running out of memory because of the "-race" burden.
Additional details:
I'm running Go 1.10 on a Mac
ab -V tells me I'm using Version 2.3 of ab (default shipment with Macbook, and it looks like ab has been dropped from brew).
If I run ab with the -r flag so that it doesn't exit right away, I get the output: Test aborted after 10 failures. So it seems like my Go server must be dropping connections rather than queueing them up...
Go server code:
package main
import (
"log"
"net/http"
)
func handler(w http.ResponseWriter, r *http.Request) {
log.Printf("got one\n")
}
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
ulimit settings:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 4864
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
I created new vm instance using "Ubuntu Server 10.04 LTS (Lucid Lynx) - 32 bits" image and m1.small falvour which has 20 GB Disk (OpenStack Icehouse). When i logging to the vm and run df -h , I found that the VM does not use the entire assigned HD. The command results are shown as the following:
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 1.4G 595M 721M 46% /
none 1005M 144K 1005M 1% /dev
none 1007M 0 1007M 0% /dev/shm
none 1007M 36K 1007M 1% /var/run
none 1007M 0 1007M 0% /var/lock
none 1007M 0 1007M 0% /lib/init/rw
The "fdisk -l" shows the DH size is 20 GB:
Disk /dev/vda: 21.5 GB, 21474836480 bytes
4 heads, 32 sectors/track, 327680 cylinders
Units = cylinders of 128 * 512 = 65536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb9da
Device Boot Start End Blocks Id System
/dev/vda1 * 17 32768 2096128 83 Linux
I need the vm to take the full space assigned to it. Any idea how could I fix it? I want the solution to be applied on each vm I create, so I do not want to manually update the VM after instantiation. I also must use 10.04 image ( can not upgrdate to 14.04)
The problem here is the image. I grabbed that one and ran it up, it's pretty simple to run a
sudo resize2fs /dev/vda1
which will resize the filesystem to the size of the partition, which seems to be 2GB. Beyond that, you have to increase the partition size. For that I think you're probably best off using virt-resize, there are some good howto's out there e.g. askubuntu, in essence:
SSH into your openstack controller node
source keystonerc_admin (or whatever yours may be called)
nova list --all-tenants | grep <instance_name> or just grab the server guid from horizon
nova show <server_guid> and note which nova host your machine is running on. Also note the instance name (e.g. instance-00000adb)
SSH into that nova node
virsh dumpxml instance-00000adb and look for the image file. On mine, this is /var/lib/nova/instances/<server_guid>/disk but that may not always be the case?
yum install libguestfs-tools
truncate -r /var/lib/nova/instances/d887249a-0d95-473e-b4f2-41f71df4dbb5/disk /var/lib/nova/instances/d887249a-0d95-473e-b4f2-41f71df4dbb5/disk.new
truncate -s +2G /var/lib/nova/instances/d887249a-0d95-473e-b4f2-41f71df4dbb5/disk.new
virt-resize --expand /dev/sda1 /var/lib/nova/instances/d887249a-0d95-473e-b4f2-41f71df4dbb5/disk /var/lib/nova/instances/d887249a-0d95-473e-b4f2-41f71df4dbb5/disk.new
mv disk disk.old ; mv disk.new disk
NB - mine didn't quite work when I booted that up again, not got time to investigate yet but it can't be far off that, and hopefully this helps.
Once you've managed to boot that up again, then you can shut it down and create a snapshot from horizon. You can then use that snapshot just like any other image, and launch all subsequent VMs directly from there.
HTH.
Working on a project where we need to server a small static xml file ~40k / s.
All incoming requests are sent to the server from HAProxy. However, none of the requests will be persistent.
The issue is that when benchmarking with non-Persistent requests, the nginx instance caps out at 19 114 req/s. When persistent connections are enabled, performance increases by nearly an order of magnitude, to 168 867 req/s. The results are similar with G-wan.
When benchmarking non-persistent requests, CPU usage is minimal.
What can I do to increase performance with non-persistent connections and nginx?
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
finished in 52 sec, 315 millisec and 603 microsec, 19114 req/s, 5413 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 290000000 bytes total, 231000000 bytes http, 59000000 bytes data
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
finished in 5 sec, 921 millisec and 791 microsec, 168867 req/s, 48640 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 294950245 bytes total, 235950245 bytes http, 59000000 bytes data
Your 2 tests are similar (except HTTP Keep-Alives):
./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
And the one with HTTP Keep-Alives is 10x faster:
finished in 52 sec, 19114 req/s, 5413 kbyte/s
finished in 5 sec, 168867 req/s, 48640 kbyte/s
First, HTTP Keep-Alives (persistant connections) make HTTP requests run faster because:
Without HTTP Keep-Alives, the client must establish a new CONNECTION for EACH request (this is slow because of the TCP handshake).
With HTTP Keep-Alives, the client can send all requests at once (using the SAME CONNECTION). This is faster because there are less things to do.
Second, you say that the static file XML size is "small".
Is "small" nearer to 1 KB or 1 MB? We don't know. But that makes a huge difference in terms of available options to speedup things.
Huge files are usually served through sendfile() because it works in the kernel, freeing the usermode server from the burden of reading from disk and buffering.
Small files can use more flexible options available for application developers in usermode, but here also, file size matters (bytes and kilobytes are different animals).
Third, you are using 16 threads with your test. Are you really enjoying 16 PHYSICAL CPU Cores on BOTH the client and the server machines?
If that's not the case, then you are simply slowing-down the test to the point that you are no longer testing the web servers.
As you see, many factors have an influence on performance. And there are more with OS tuning (the TCP stack options, available file handles, system buffers, etc.).
To get the most of a system, you need to examinate all those parameters, and pick the best for your particular exercise.