I am trying to connect to a postgreSQL database (hosted on AWS RDS) via an SSH tunnel in R. So far, I have been able to connect using the following methods:
1.---------------------------
Opening the ssh tunnel in my terminal (MacOs) using
ssh -i {key file path} -f -N -L 5432:{db host}:5432 {ssh user}#{ssh host} -v
and then connecting to the database using
psql -hlocalhost -U{db user} -p5432 -dpostgres
2.---------------------------
Opening the ssh tunnel in my terminal and then running the following code in R to connect
conn <- dbConnect(
RPostgres::Postgres(),
dbname = db_name,
user = db_user,
password = db_password,
host = "127.0.0.1",
port = db_port
)
3.---------------------------
This is where the issue occurs. I'm able to connect by opening the ssh tunnel in R (in a background environment) with
tunnel_process <- callr::r_bg(
function(ssh_host, ssh_user, ssh_key, db_host, db_port) {
session <- ssh::ssh_connect(host = glue::glue("{ ssh_user }#{ ssh_host }"),
keyfile = ssh_key,
verbose = 3)
ssh::ssh_tunnel(session = session,
port = db_port,
target = glue::glue("{ db_host }:{ db_port }"))
},
args = list(ssh_host, ssh_user, ssh_key, db_host, db_port),
stdout = nullfile(),
stderr = nullfile()
)
But then I'm unable to use the same "dbConnect" code as above to connect. It only gives me the following error message
Error: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
I am however able to connect directly from the terminal using the connection code in part 1. but only if I run psql -hlocalhost -U{db user} -p5432 -dpostgres, then re-run the ssh tunnel code in R, and only THEN enter my password in the terminal. I would appear that every time I try to connect, it closes the SSH tunnel, so I then have to re-launch it before submitting my password.
Question---------------------------
From what I just detailed, it would appear that:
a. My database is reachable since I can easily connect through the terminal
b. My R code works since I'm able to use it to both successfully open the SSH tunnel AND connect to the database. I'm just unable to use both together for some reason.
c. The tunnel I open through R breaks any time I try to connect to the database. This is not the case for the tunnel I open directly in the terminal.
Since I want to be able to do everything directly from R, does anybody here have any ideas on what may be causing the issue?
EDIT---------------------------
Here's the log I get in R when I try to connect to the database, just before it closes the tunnel:
> ssh::ssh_tunnel(session = session,
+ port = db_port,
+ target = glue::glue("{ db_host }:{ db_port }"))
\ Waiting for connetion on port 5432... client connected!
channel_open: Creating a channel 43 with 64000 window and 32768 max packet
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=124,padding=18,comp=105,payload=105]
channel_open: Sent a SSH_MSG_CHANNEL_OPEN type direct-tcpip for channel 43
ssh_packet_socket_callback: packet: read type 80 [len=492,padding=16,comp=475,payload=475]
ssh_packet_process: Dispatching handler for packet type 80
ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00#openssh.com 0
ssh_packet_process: Couldn't do anything with packet type 80
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 91 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 91
ssh_packet_channel_open_conf: Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
ssh_packet_channel_open_conf: Remote window : 2097152, maxpacket : 32768
| Tunneled -1 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
channel_write_common: channel_write wrote 8 bytes
| Tunneled 7 bytes...ssh_packet_socket_callback: packet: read type 94 [len=28,padding=17,comp=10,payload=10]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 1 bytes data in 0 (local win=64000 remote win=2097144)
channel_default_bufferize: placing 1 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=63999 remote win=2097144)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=18,comp=9,payload=9]
grow_window: growing window (channel 43:0) to 1280000 bytes
ssh_channel_read_timeout: Read (1) buffered : 1 bytes. Window: 1280000
- Tunneled 8 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=316,padding=17,comp=298,payload=298]
channel_write_common: channel_write wrote 289 bytes
/ Tunneled 297 bytes...ssh_packet_socket_callback: packet: read type 94 [len=3964,padding=12,comp=3951,payload=3951]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 3942 bytes data in 0 (local win=1280000 remote win=2096855)
channel_default_bufferize: placing 3942 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276058 remote win=2096855)
ssh_channel_read_timeout: Read (3942) buffered : 3942 bytes. Window: 1276058
\ Tunneled 4239 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=156,padding=8,comp=147,payload=147]
channel_write_common: channel_write wrote 138 bytes
- Tunneled 4377 bytes...ssh_packet_socket_callback: packet: read type 94 [len=76,padding=15,comp=60,payload=60]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 51 bytes data in 0 (local win=1276058 remote win=2096717)
channel_default_bufferize: placing 51 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276007 remote win=2096717)
ssh_channel_read_timeout: Read (51) buffered : 51 bytes. Window: 1276007
| Tunneled 4428 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=140,padding=14,comp=125,payload=125]
channel_write_common: channel_write wrote 116 bytes
\ Tunneled 4544 bytes...ssh_packet_socket_callback: packet: read type 94 [len=60,padding=8,comp=51,payload=51]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 42 bytes data in 0 (local win=1276007 remote win=2096601)
channel_default_bufferize: placing 42 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275965 remote win=2096601)
ssh_channel_read_timeout: Read (42) buffered : 42 bytes. Window: 1275965
/ Tunneled 4586 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=60,padding=19,comp=40,payload=40]
channel_write_common: channel_write wrote 31 bytes
- Tunneled 4617 bytes...packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_channel_send_eof: Sent a EOF on client channel (43:0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_channel_close: Sent a close on client channel (43:0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
tunnel closed!
For reference, this is what the same log looks like when using the workaround detailed in 3. (re-running the ssh_tunnel right before submitting my password in the terminal):
> ssh::ssh_tunnel(session = session,
+ port = db_port,
+ target = glue::glue("{ db_host }:{ db_port }"))
\ Waiting for connetion on port 5432... client connected!
channel_open: Creating a channel 43 with 64000 window and 32768 max packet
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=124,padding=18,comp=105,payload=105]
channel_open: Sent a SSH_MSG_CHANNEL_OPEN type direct-tcpip for channel 43
ssh_packet_socket_callback: packet: read type 80 [len=492,padding=16,comp=475,payload=475]
ssh_packet_process: Dispatching handler for packet type 80
ssh_packet_global_request: Received SSH_MSG_GLOBAL_REQUEST packet
ssh_packet_global_request: UNKNOWN SSH_MSG_GLOBAL_REQUEST hostkeys-00#openssh.com 0
ssh_packet_process: Couldn't do anything with packet type 80
packet_send2: packet: wrote [len=12,padding=6,comp=5,payload=5]
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 91 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 91
ssh_packet_channel_open_conf: Received SSH2_MSG_CHANNEL_OPEN_CONFIRMATION
ssh_packet_channel_open_conf: Received a CHANNEL_OPEN_CONFIRMATION for channel 43:0
ssh_packet_channel_open_conf: Remote window : 2097152, maxpacket : 32768
| Tunneled -1 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
channel_write_common: channel_write wrote 8 bytes
| Tunneled 7 bytes...ssh_packet_socket_callback: packet: read type 94 [len=28,padding=17,comp=10,payload=10]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 1 bytes data in 0 (local win=64000 remote win=2097144)
channel_default_bufferize: placing 1 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=63999 remote win=2097144)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=28,padding=18,comp=9,payload=9]
grow_window: growing window (channel 43:0) to 1280000 bytes
ssh_channel_read_timeout: Read (1) buffered : 1 bytes. Window: 1280000
- Tunneled 8 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=316,padding=17,comp=298,payload=298]
channel_write_common: channel_write wrote 289 bytes
\ Tunneled 297 bytes...ssh_packet_socket_callback: packet: read type 94 [len=3964,padding=12,comp=3951,payload=3951]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 3942 bytes data in 0 (local win=1280000 remote win=2096855)
channel_default_bufferize: placing 3942 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276058 remote win=2096855)
ssh_channel_read_timeout: Read (3942) buffered : 3942 bytes. Window: 1276058
/ Tunneled 4239 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=156,padding=8,comp=147,payload=147]
channel_write_common: channel_write wrote 138 bytes
| Tunneled 4377 bytes...ssh_packet_socket_callback: packet: read type 94 [len=76,padding=15,comp=60,payload=60]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 51 bytes data in 0 (local win=1276058 remote win=2096717)
channel_default_bufferize: placing 51 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1276007 remote win=2096717)
ssh_channel_read_timeout: Read (51) buffered : 51 bytes. Window: 1276007
- Tunneled 4428 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=140,padding=14,comp=125,payload=125]
channel_write_common: channel_write wrote 116 bytes
/ Tunneled 4544 bytes...ssh_packet_socket_callback: packet: read type 94 [len=60,padding=8,comp=51,payload=51]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 42 bytes data in 0 (local win=1276007 remote win=2096601)
channel_default_bufferize: placing 42 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275965 remote win=2096601)
ssh_channel_read_timeout: Read (42) buffered : 42 bytes. Window: 1275965
\ Tunneled 4586 bytes...ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=92,padding=12,comp=79,payload=79]
channel_write_common: channel_write wrote 70 bytes
- Tunneled 4656 bytes...ssh_packet_socket_callback: packet: read type 94 [len=380,padding=15,comp=364,payload=364]
ssh_packet_process: Dispatching handler for packet type 94
channel_rcv_data: Channel receiving 355 bytes data in 0 (local win=1275965 remote win=2096531)
channel_default_bufferize: placing 355 bytes into channel buffer (stderr=0)
channel_rcv_data: Channel windows are now (local win=1275610 remote win=2096531)
ssh_channel_read_timeout: Read (355) buffered : 355 bytes. Window: 1275610
| Tunneled 5011 bytes...
Finally, here's the log when running ssh_connect:
> session <- ssh::ssh_connect(host = glue::glue("{ ssh_user }#{ ssh_host }"),
+ keyfile = ssh_key,
+ verbose = 3)
ssh_pki_import_privkey_base64: Trying to decode privkey passphrase=false
ssh_connect: libssh 0.8.6 (c) 2003-2018 Aris Adamantiadis, Andreas Schneider and libssh contributors. Distributed under the LGPL, please refer to COPYING file for information about your rights, using threading threads_pthread
ssh_socket_connect: Nonblocking connection socket: 50
ssh_connect: Socket connecting, now waiting for the callbacks to work
ssh_connect: Actual timeout : 10000
ssh_socket_pollcallback: Received POLLOUT in connecting state
socket_callback_connected: Socket connection callback: 1 (0)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
callback_receive_banner: Received banner: SSH-2.0-OpenSSH_7.4
ssh_client_connection_callback: SSH server banner: SSH-2.0-OpenSSH_7.4
ssh_analyze_banner: Analyzing banner: SSH-2.0-OpenSSH_7.4
ssh_analyze_banner: We are talking to an OpenSSH client version: 7.4 (70400)
ssh_known_hosts_read_entries: Failed to open the known_hosts file '/etc/ssh/ssh_known_hosts': No such file or directory
ssh_client_select_hostkeys: Changing host key method to "ecdsa-sha2-nistp256,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ssh-rsa,ssh-dss"
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=644,padding=9,comp=634,payload=634]
ssh_packet_socket_callback: packet: read type 20 [len=1276,padding=10,comp=1265,payload=1265]
ssh_packet_process: Dispatching handler for packet type 20
ssh_kex_select_methods: Negotiated curve25519-sha256,ecdsa-sha2-nistp256,aes256-ctr,aes256-ctr,hmac-sha2-256,hmac-sha2-256,none,none,,
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=44,padding=6,comp=37,payload=37]
ssh_packet_socket_callback: packet: read type 31 [len=260,padding=11,comp=248,payload=248]
ssh_packet_process: Dispatching handler for packet type 31
ssh_packet_dh_reply: Received SSH_KEXDH_REPLY
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=12,padding=10,comp=1,payload=1]
ssh_client_curve25519_reply: SSH_MSG_NEWKEYS sent
ssh_packet_socket_callback: Processing 112 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 21 [len=12,padding=10,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 21
ssh_packet_newkeys: Received SSH_MSG_NEWKEYS
crypt_set_algorithms2: Set output algorithm to aes256-ctr
crypt_set_algorithms2: Set HMAC output algorithm to hmac-sha2-256
crypt_set_algorithms2: Set input algorithm to aes256-ctr
crypt_set_algorithms2: Set HMAC input algorithm to hmac-sha2-256
ssh_packet_newkeys: Signature verified and valid
ssh_packet_socket_callback: Processing 96 bytes left in socket buffer
ssh_packet_socket_callback: packet: read type 7 [len=60,padding=6,comp=53,payload=53]
ssh_packet_process: Dispatching handler for packet type 7
ssh_packet_ext_info: Received SSH_MSG_EXT_INFO
ssh_packet_ext_info: Follows 1 extensions
ssh_packet_ext_info: Extension: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
ssh_connect: current state : 7
packet_send2: packet: wrote [len=28,padding=10,comp=17,payload=17]
ssh_service_request: Sent SSH_MSG_SERVICE_REQUEST (service ssh-userauth)
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
ssh_packet_socket_callback: packet: read type 6 [len=28,padding=10,comp=17,payload=17]
ssh_packet_process: Dispatching handler for packet type 6
ssh_packet_service_accept: Received SSH_MSG_SERVICE_ACCEPT
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=44,padding=4,comp=39,payload=39]
ssh_packet_socket_callback: packet: read type 51 [len=60,padding=15,comp=44,payload=44]
ssh_packet_process: Dispatching handler for packet type 51
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic
ssh_packet_userauth_failure: Access denied for 'none'. Authentication that can continue: publickey,gssapi-keyex,gssapi-with-mic
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_key_algorithm_allowed: Checking rsa-sha2-512 with list <ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss>
ssh_socket_unbuffered_write: Enabling POLLOUT for socket
packet_send2: packet: wrote [len=636,padding=11,comp=624,payload=624]
ssh_packet_socket_callback: packet: read type 52 [len=12,padding=10,comp=1,payload=1]
ssh_packet_process: Dispatching handler for packet type 52
ssh_packet_userauth_success: Authentication successful
I am analyzing some events against dns servers running unbound. In the course of this investigation I am running into traffic involving queries to the dns servers that are reported as having in some cases a source port between 1 and 1024. As far as my knowledge goes these are reserved for services so there should never be traffic originating / initiated from those to a server.
Since I also know this is a practice, not a law, that evolved over time, I know there is no technical limitation to put any number in the source port field of a packet. So my conclusion would be that these queries were generated by some tool in which the source port is filled with a random value (the frequency is about evenly divided over 0-65535, except for a peak around 32768) and that this is a deliberate attack.
Can someone confirm/deny the source port theory and vindicate my conclusion or declare me a total idiot and explain why?
Thanks in advance.
Edit 1: adding more precise info to settle some disputes below that arose due to my incomplete reporting.
It's definitely not a port scan. It was traffic arriving on port 53 UDP and unbound accepted it apparently as an (almost) valid dns query, while generating the following error messages for each packet:
notice: remote address is <ipaddress> port <sourceport>
notice: sendmsg failed: Invalid argument
$ cat raw_daemonlog.txt | egrep -c 'notice: remote address is'
256497
$ cat raw_daemonlog.txt | egrep 'notice: remote address is' | awk '{printf("%s\n",$NF)}' | sort -n | uniq -c > sourceportswithfrequency.txt
$ cat sourceportswithfrequency.txt | wc -l
56438
So 256497 messages, 56438 unique source ports used
$ cat sourceportswithfrequency.txt | head
5 4
3 5
5 6
So the lowest source port seen was 4 which was used 5 times
$ cat sourceportswithfrequency.txt | tail
8 65524
2 65525
14 65526
1 65527
2 65528
4 65529
3 65530
3 65531
3 65532
4 65534
So the highest source port seen was 65534 and it was used 4 times.
$ cat sourceportswithfrequency.txt | sort -n | tail -n 25
55 32786
58 35850
60 32781
61 32785
66 32788
68 32793
71 32784
73 32783
88 32780
90 32791
91 32778
116 2050
123 32779
125 37637
129 7077
138 32774
160 32777
160 57349
162 32776
169 32775
349 32772
361 32773
465 32769
798 32771
1833 32768
So the peak around 32768 is real.
My original question still stands: does this traffic pattern suggest an attack or is there an logical explanation for, for instance, the traffic with source ports < 1024?
As far as my knowledge goes these are reserved for services so there should never be traffic originating / initiated from those to a server.
It doesn't matter what the source port number is, as long as it's between 1 and 65,535. It's not like a source port of 53 means that there is a DNS server listening on the source machine.
The source port is just there to allow multiple connections / in-flight datagrams from one machine to another machine on the same destination port.
See also Wiki: Ephemeral port:
The Internet Assigned Numbers Authority (IANA) suggests the range 49152 to 65535 [...] for dynamic or private ports.[1]
That sounds like a port scan.
There are 65536 distinct and usable port numbers. (ibid.)
FYI: The TCP and UDP port 32768 is registered and used by IBM FileNet TMS.
Working on a project where we need to server a small static xml file ~40k / s.
All incoming requests are sent to the server from HAProxy. However, none of the requests will be persistent.
The issue is that when benchmarking with non-Persistent requests, the nginx instance caps out at 19 114 req/s. When persistent connections are enabled, performance increases by nearly an order of magnitude, to 168 867 req/s. The results are similar with G-wan.
When benchmarking non-persistent requests, CPU usage is minimal.
What can I do to increase performance with non-persistent connections and nginx?
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
finished in 52 sec, 315 millisec and 603 microsec, 19114 req/s, 5413 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 290000000 bytes total, 231000000 bytes http, 59000000 bytes data
[root#spare01 lighttpd-weighttp-c24b505]# ./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
finished in 5 sec, 921 millisec and 791 microsec, 168867 req/s, 48640 kbyte/s
requests: 1000000 total, 1000000 started, 1000000 done, 1000000 succeeded, 0 failed, 0 errored
status codes: 1000000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 294950245 bytes total, 235950245 bytes http, 59000000 bytes data
Your 2 tests are similar (except HTTP Keep-Alives):
./weighttp -n 1000000 -c 100 -t 16 "http://192.168.1.40/feed.txt"
./weighttp -n 1000000 -c 100 -t 16 -k "http://192.168.1.40/feed.txt"
And the one with HTTP Keep-Alives is 10x faster:
finished in 52 sec, 19114 req/s, 5413 kbyte/s
finished in 5 sec, 168867 req/s, 48640 kbyte/s
First, HTTP Keep-Alives (persistant connections) make HTTP requests run faster because:
Without HTTP Keep-Alives, the client must establish a new CONNECTION for EACH request (this is slow because of the TCP handshake).
With HTTP Keep-Alives, the client can send all requests at once (using the SAME CONNECTION). This is faster because there are less things to do.
Second, you say that the static file XML size is "small".
Is "small" nearer to 1 KB or 1 MB? We don't know. But that makes a huge difference in terms of available options to speedup things.
Huge files are usually served through sendfile() because it works in the kernel, freeing the usermode server from the burden of reading from disk and buffering.
Small files can use more flexible options available for application developers in usermode, but here also, file size matters (bytes and kilobytes are different animals).
Third, you are using 16 threads with your test. Are you really enjoying 16 PHYSICAL CPU Cores on BOTH the client and the server machines?
If that's not the case, then you are simply slowing-down the test to the point that you are no longer testing the web servers.
As you see, many factors have an influence on performance. And there are more with OS tuning (the TCP stack options, available file handles, system buffers, etc.).
To get the most of a system, you need to examinate all those parameters, and pick the best for your particular exercise.
I've kind of ran into an ugly snag. I developed a website for a client a few years back, and since then they've transferred their site to a different domain name provider and host. Now they want some updates, but when I try to access their site I get a Network Timeout (the page just tries to load for a few minutes, then firefox shows a Network Timeout error). I can access the site via a proxy, but proxies kinda suck and don't support everything, plus I'm a little paranoid about sending sensitive data through a proxy, not to mention I don't see how that would help me with FTP access and what not. I'm not exactly sure where along the line this problem is occurring... is my ISP blocking it, is the webserver blocking me, is it my router, or what's going on? I know of two sites that do this, and I think they're hosted by the same people.
The sites are http://fvringette.com/ and http://damngoodtimes.com/
#MarkusQ: traceroute for fvringette.com (which turns out to be the same as with damngoodtimes.com)
traceroute: Warning: Multiple interfaces found; using 134.117.14.35 # hme0
traceroute to 76.74.225.90 (76.74.225.90), 30 hops max, 40 byte packets
1 unix-gate.physics.carleton.ca (134.117.14.1) 0.973 ms 0.513 ms 0.514 ms
2 10.50.254.3 (10.50.254.3) 0.437 ms 0.385 ms 0.351 ms
3 10.30.33.1 (10.30.33.1) 0.488 ms 0.394 ms 0.370 ms
4 10.30.55.1 (10.30.55.1) 0.396 ms 0.416 ms 0.391 ms
5 134.117.254.242 (134.117.254.242) 0.708 ms 0.720 ms 0.704 ms
6 10.30.57.1 (10.30.57.1) 1.338 ms 1.221 ms 1.237 ms
7 kolker.fcican.com (207.34.252.249) 1.464 ms 1.544 ms 1.459 ms
8 * * *
9 154.11.3.17 (154.11.3.17) 7.355 ms 7.393 ms 7.426 ms
10 oc48.so-2-0-3.van-hc21e-cor-1.peer1.net (216.187.114.137) 62.762 ms 62.838 ms 62.625 ms
11 oc48.pos4-0.van-hc21e-dis-1.peer1.net (216.187.89.253) 62.795 ms 63.238 ms 62.893 ms
12 64.69.91.245 (64.69.91.245) 64.103 ms 62.908 ms 63.266 ms
13 64.69.91.245 (64.69.91.245) 63.094 ms !H 63.072 ms !H 63.173 ms !H
64.69.91.245 - Geo Information
IP Address 64.69.91.245
Host 64.69.91.245
Location CA CA, Canada
City Vancouver, BC v6b4n5
Organization Peer1 Internet Bandwidth & Server Co-Location Faci
ISP Peer 1 Network
AS Number AS13768
Latitude 49°25'00" North
Longitude 123°13'33" West
Distance 9281.29 km (5767.13 miles)
64.69.91.245 - Whois Information
OrgName: Peer 1 Network Inc.
OrgID: PER1
Address: 75 Broad Street
Address: 2nd Floor
City: New York
StateProv: NY
PostalCode: 10004
Country: US
NetRange: 64.69.64.0 - 64.69.95.255
CIDR: 64.69.64.0/19
NetName: PEER1-BLK-01
NetHandle: NET-64-69-64-0-1
Parent: NET-64-0-0-0-0
NetType: Direct Allocation
NameServer: NS1.PEER1.NET
NameServer: NS2.PEER1.NET
Comment:
RegDate: 2000-04-12
Updated: 2007-08-29
RTechHandle: ZP55-ARIN
RTechName: Peer1 Network Inc.
RTechPhone: +1-604-683-7747
RTechEmail: net-admin#peer1.net
OrgAbuseHandle: NSA-ARIN
OrgAbuseName: Peer 1 Network AUP Enforcement
OrgAbusePhone: +1-604-484-2588
OrgAbuseEmail: abuse#peer1.net
OrgTechHandle: ZP55-ARIN
OrgTechName: Peer1 Network Inc.
OrgTechPhone: +1-604-683-7747
OrgTechEmail: net-admin#peer1.net
OrgName: Peer1 Internet Bandwidth & Server Co-Location Facilities
OrgID: PIBSCF
Address: 2100-555 W. hastings St.
City: Vancouver
StateProv: BC
PostalCode: V6B 4N5
Country: CA
NetRange: 64.69.91.240 - 64.69.91.255
CIDR: 64.69.91.240/28
NetName: PEER1-GVLANPRI-01
NetHandle: NET-64-69-91-240-1
Parent: NET-64-69-64-0-1
NetType: Reassigned
Comment:
RegDate: 2002-03-14
Updated: 2002-03-14
RTechHandle: MT1763-ARIN
RTechName: Teolis, Mark
RTechPhone: +1-604-683-7747
RTechEmail: net-admin#peer1.net
# ARIN WHOIS database, last updated 2009-04-06 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.
It occured to me that it may be more useful if I do a trace from my home computer, to fvringette.com rather than from some random computer which may actually be able to connect. Output of tracert:
1 <1 ms <1 ms <1 ms <snip>
2 24 ms 24 ms 23 ms 70.71.106.1
3 24 ms 24 ms 24 ms rd1bb-ge5-0-0-1.vc.shawcable.net [64.59.149.2]
4 25 ms 29 ms 29 ms rc2bb-tge0-15-0-0.vc.shawcable.net [66.163.69.137]
5 28 ms 30 ms 29 ms rc2wh-tge0-15-1-0.vc.shawcable.net [66.163.69.121]
6 25 ms 24 ms 24 ms 204.239.129.213
7 26 ms 29 ms 29 ms oc48.pos3-0.van-spenc-dis-1.peer1.net [216.187.89.250]
8 27 ms 29 ms 30 ms 64.69.91.245
9 * * * Request timed out.
10 * * * Request timed out.
After this all requests just keep timing out. It's the same IP, 64.69.91.245 that seems to be causing the problem... does this mean that I'm just unlucky and got a dead server that won't forward my request? I have no idea how these things work.
I can ping 64.69.91.245, but not telnet 64.69.91.245. It says 'connect failed', not 'connection refused'... I can't think of why it would fail for me, but no one else?
Try doing a traceroute for starters.
The "!H"s on line 13 mean that the host (64.69.91.245) is unreachable--but not all the time (see line 12). This looks at a glance to be further than your ISP, maybe at theirs.
Next step would be to figure out who that is...
My guess is either their hardware is faulty, or that they are blocking you inadvertently.
Some networks have the policies to block telnet connections/packets on the premise that they are unencrypted and only allow SSH.
Of course they may be blocking you for another reason.
Perhaps you should discuss it with them. It may be the case that they have blocked your IP address because you've been trying to telnet into their servers all day. :)