We are using RabbitMQ/with MQTT adapter as a broker to connect nodes for monitoring.
Theoretically, being TCP based we can have a max of N connections/interface, as having N port, where N ~ [5000, 65000]. I also realize that by having multiple network interfaces, I can theoretically service far more number of connections.
Is there a sensible limit to the max number of connections and hence the number of nodes that can be serviced having a sane CPU/MEM usage pattern.
But, it would also mean maintaining & processing N connections concurrently on commodity HW (QuadCore 2GHz/8GB RAM). Is there any:
Formal process to derive how many connections can be serviced.
What would be the system/rabbitmq parameters that I should be look at to optimize.
It would indicate that it is also goverened by the daemon servicing the connections. Any formal architecture / design constraints (though at the moment we are sticking to RabbitMQ).
Currently, 30000 connections are possible to be serviced with below config, but with %cpu ~ 100% and memory almost getting exhausted > 60% with a throughput of ~ 80 (x300bytes) messages/sec.
HW: i7-3630QM CPU # 2.40GHz/4 cores/2 threads & 8GB RAM.
SW: Ubuntu/RabbitMQ with MQTT adapter.
**rabbitmq.config**
[
{rabbit,
[
{tcp_listeners, [5672]},
{loopback_users, []},
{vm_memory_high_watermark, 0.8},
{vm_memory_high_watermark_paging_ratio, 0.75}
]
},
{rabbitmq_mqtt, [{default_user, <<"guest">>},
{default_pass, <<"guest">>},
{allow_anonymous, true},
{vhost, <<"/">>},
{exchange, <<"amq.topic">>},
{subscription_ttl, 1800000},
{prefetch, 10},
{ssl_listeners, []},
{tcp_listeners, [1883]},
{tcp_listen_options, [binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true}]}]}].
Also, is it feasible to retire/evict connections while handing new connections based on a policy like LRU / LFU of cache-eviction. Can this be done at the system level (OS / TCP layer).
Thanks.JB.
Related
I am trying to establish 10k client connections(potentially 100k) with my 2 MQTT brokers using HAProxy as a load balancer.
I have a working simulator(using Java Paho library) that can simulate 10k clients. On the same machine I run 2 MQTT brokers in docker. For LB im using another machine with virtual image of Ubuntu 16.04.
When I connect directly to a MQTT Broker those connections are established without a problem, however when I use HAProxy I only get around 8.8k connections, while the rest throw: Error at client{insert number here}: Connection lost (32109) - java.net.SocketException: Connection reset. When I connect simulator directly to a broker (Same machine) about 20k TCP connections open, however when I use load balancer only 17k do. This leaves me thinking that LB is causing the problem.
It is important to add that whenever I run the simulator I'm unable to use the browser (Cannot connect to the internet). I havent tested if this is browser only, but could that mean that I actually run out of ports or something similar and the real issue here is not in the LB?
Here is my HAProxy configuration:
global
log /dev/log local0
log /dev/log local1 notice
maxconn 500000
ulimit-n 500000
maxpipes 500000
defaults
log global
mode http
timeout connect 3h
timeout client 3h
timeout server 3h
listen mqtt
bind *:8080
mode tcp
option tcplog
option clitcpka
balance leastconn
server broker_1 address:1883 check
server broker_2 address:1884 check
listen stats
bind 0.0.0.0:1936
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
This is what MQTT broker shows for every successful/unsuccessful connection
...
//Successful connection
1613382861: New connection from xxx:32850 on port 1883.
1613382861: New client connected from xxx:60974 as 356 (p2, c1, k1200, u'admin').
...
//Unsuccessful connection
1613382699: New connection from xxx:42861 on port 1883.
1613382699: Client <unknown> closed its connection.
...
And this is what ulimit -a shows on LB machine.
core file size (blocks) (-c) 0
data seg size (kb) (-d) unlimited
scheduling priority (-e) 0
file size (blocks) (-f) unlimited
pending signals (-i) 102355
max locked memory (kb) (-l) 82000
max memory size (kb) (-m) unlimited
open files (-n) 500000
POSIX message queues (bytes) (-q) 819200
real-time priority (-r) 0
stack size (kb) (-s) 8192
cpu time (seconds) (-t) unlimited
max user processes (-u) 500000
virtual memory (kb) (-v) unlimited
file locks (-x) unlimited
Note: The LB process has the same limits.
I followed various tutorials and increased open file limit as well as port limit and TCP header size, etc. The number of connected users increased from 2.8k to about 8.5-9k (Which is still way lower than the 300k author of the tutorial had). ss -s command shows about 17000ish TCP and inet connections.
Any pointers would greatly help!
Thanks!
You can't do a normal LB of MQTT traffic, as you can't "pin" the connection based on the MQTT Topic. If you send in a SUBSCRIBE to Broker1 for Topic "test/blatt/#", but the next client PUBLISHes to Broker2 "test/blatt/foo", then if the two brokers are not bridged, your first subscriber will never get that message.
If your clients are terminating the TCP connection sometime after the CONNECT, or the HAproxy is round-robin'ing the packets between the two brokers, you will get errors like this. You need to somehow persist the connections, and I don't know how you do that with HAproxy. Non-free LB's like A10 Thunder or F5 LTM can persist TCP connections...but you still need the MQTT brokers bridged for it all to work.
Turns out I was running out of resources on my computer.
I moved simulator to another machine and managed to get 15k connections running. Due to resource limits I cant get more than that. Computer thats running the serverside uses 20/32GB of RAM and the computer running simulator used 32/32GB for approx 15k devices. Now I see why running both on the same computer is not an option.
In the book "Computer Networking - A Top Down Approach", I just studied that, TCP sockets are uniquely identified by four parameters:
Sender's IP address
Sender's port number
Receiver's IP address
Receiver's port number
In the case of server, we know that the server's IP address and the port number is fixed. Clients' IP addresses and the port numbers will obviously vary. In case of new request, the server will open a new thread (to open a new TCP socket) to handle that request.
My question is that, is there any calculation we can perform to calculate maximum number of clients a single server machine can handle?
Thanks
Hi，it's a good question.
First, if one thread handles one connection, then the limitation of x86 computer will be really low. It will be something like 1000 for every process since the virtual memory is limited as 4GB. For x64, it will be relatively higher. But the switch between threads is not we want to see.
So, we use a thread to process lots of connections, that is I/O multiplexing.
What's more, this still has a limitation, we call it C10K and C10M, and we solve these probolems by non-blocking I/O multiplexing like epoll or IOCP, and the DPDK is useful to solve C10M.
If you want to reach a higher connection number, then we use distributed systems instead only one server.
It's a long story of the limitation of server, and I think you can learn much from them.
Its said that one of the advantages of HTTP 2 over HTTP 1 is that HTTP2 has streams of data. It can be up to 256 different streams in ONE TCP/IP connection. However, in HTTP 1 there can be up to 6 parallel connections. It's an improvement that HTTP 2 enables reading data from 256 resources, but still I think that 6 connections (in HTTP 1) have a better throughput that one TCP/IP connection (in HTTP 2). Still, HTTP2 is considered faster than HTTP 1. So... what don't I understand correctly ?
6 physical connections would have more throughput than one physical connection, all else being equal.
However the same doesn't apply to 6 different TCP/IP connections between the same computers, as these are virtual connections (assuming you don't have two network cards). The limiting factor is usually the latency and bandwidth of your internet connection rather than TCP/IP protocol itself.
In fact, due to the way TCP connections are created and handled, its actually much more efficient to have one TCP/IP connection. This is because of the cost of the initial connection (a three way TCP handshake, the HTTPS handshake and the fact that TCP connections use a process called Slow Start to slowly build up its capacity to the maximum speed that the network can handle) but also in the ongoing upkeep of the connection (as the Slow Start process happens again periodically unless the connection is fully utilised all the time - which is much more likely to happen with one connection that is used for everything, than it is to happen when your requests are split across 6 connections).
Also HTTP/1.1 only allows one request in flight at a time, so the connection is not able to be used until the response is returned (ignoring pipelining which is not really supported at all in HTTP/1.1). This not only limits the usefulness of the 6 connections, but also means that it's more likely that the connections will be underused which, given the issues with underused connections in TCP mentioned above, means they are likely to be slower as they throttle back and have to go through the Slow Start process again to build up to maximum capacity. HTTP/2 however allows those 256 streams to allow requests to be in flight at the same time. Which is both better than just 6 connections, and allows true multiplexing.
If you want to know more, then Ilya Grigorik had written an excellent book on the subject called High Performance Browser Networking which is even available online for free.
Based on a discussion with a colleague, I need some help:
Is it true, that without anything fancy (NAT, virtual-IP, etc.) a sw load-balancer can sustain a max of. ~64K backend (to a server) connections concurrently (at the same time), based on the port limit on the network interface it's bind on?
So for example, if there's a sw load-balancer connecting to 2 backend servers using long-lived TCP connections (not for HTTP) each of the servers can have up to ~64K connections at a given time?
(let's forget that the port limit is usually less than ~64K per connection)
Thanks.
The limitation comes from the fact that every connection through the LB requires 1 unique port on the internal interface of the LB.
If you only have 1 internal and 1 external interface pair then you will only be able to sustain approximately 64,000 connections due to port exhaustion. All the servers must share that limitation. This means that if you have 2 servers then they will have, if evenly distributed, around 32k connections each.
You can however easily lift this bottleneck by adding more than 1 internal interface on the LB.
In the case of a half open connection where the server crashes (no FIN or RESET sent to client), and the client attempts to send some data on this broken connection, each TCP segment will go un-ACKED. TCP will attempt to retransmit packets after some timeout. How many times will TCP attempt to retransmit before giving up and what happens in this case? How does it inform the operating system that the host is unreachable? Where is this specified in the TCP RFC?
If the server program crashes, the kernel will clean up all open sockets appropriately. (Well, appropriate from a TCP point of view; it might violate the application layer protocol, but applications should be prepared for this event.)
If the server kernel crashes and does not come back up, the number and timing of retries depends if the socket were connected yet or not:
tcp_retries1 (integer; default: 3; since Linux 2.2)
The number of times TCP will attempt to
retransmit a packet on an established connection
normally, without the extra effort of getting
the network layers involved. Once we exceed
this number of retransmits, we first have the
network layer update the route if possible
before each new retransmit. The default is the
RFC specified minimum of 3.
tcp_retries2 (integer; default: 15; since Linux 2.2)
The maximum number of times a TCP packet is
retransmitted in established state before giving
up. The default value is 15, which corresponds
to a duration of approximately between 13 to 30
minutes, depending on the retransmission
timeout. The RFC 1122 specified minimum limit
of 100 seconds is typically deemed too short.
(From tcp(7).)
If the server kernel crashes and does come back up, it won't know about any of the sockets, and will RST those follow-on packets, enabling failure much faster.
If any single-point-of-failure routers along the way crash, if they come back up quickly enough, the connection may continue working. This would require that firewalls and routers be stateless, or if they are stateful, have rulesets that allow preexisting connections to continue running. (Potentially unsafe, different firewall admins have different policies about this.)
The failures are returned to the program with errno set to ECONNRESET (at least for send(2)).