I have 2 systems: system 1 is running akka and HAProxy, system 2 is running REST components that make a request to akka.
Akka runs on port 4241 on system 1. System 2 is able to connect to System 1 when there is no HAProxy. After I installed HAProxy on system 1, the request from system 2 to system 1 errors out with the logs below:
ERROR[deal-akka.actor.default-dispatcher-18] EndpointWriter - dropping
message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://akkaSystemName#Server1IP:42431/]] arriving at
[akka.tcp://akkaSystemName#Server1IP:42431] inbound addresses are
[akka.tcp://akkaSystemName#Server1IP:4241]
HAProxy runs on 42431.
The HAProxy configuration is the following:
listen akka_tcp :42431
mode tcp
option tcplog
balance leastconn
server test1 Server1IP:4241 check
server test2 Server1IP:4241 check
The akka configuration is this:
actor {
provider = "akka.remote.RemoteActorRefProvider"
}
remote {
netty.tcp {
hostname = "Server1IP"
port = 4241
transport-protocol = tcp
# Sets the send buffer size of the Sockets,
# set to 0b for platform default
send-buffer-size = 52428800b
# Sets the receive buffer size of the Sockets,
# set to 0b for platform default
receive-buffer-size = 52428800b
maximum-frame-size = 52428800b
}
}
Any suggestion would be appreciated.
Updated answer:
Probably the Akka Remoting is not supposed to work with a Load Balancer. Look at this part of its documentation:
Akka Remoting is a communication module for connecting actor systems
in a peer-to-peer fashion, and it is the foundation for Akka
Clustering. The design of remoting is driven by two (related) design
decisions:
1.Communication between involved systems is symmetric: if a system A can connect to a system B then system B must also be able to connect
to system A independently.
2.The role of the communicating systems are symmetric in regards to connection patterns: there is no system that only accepts connections,
and there is no system that only initiates connections.
The consequence of these decisions is that it is not possible to
safely create pure client-server setups with predefined roles
(violates assumption 2) and using setups involving Network Address
Translation or Load Balancers (violates assumption 1).
For client-server setups it is better to use HTTP or Akka I/O.
For your case it seems reasonable to use Akka HTTP or Akka I/O on system 1 to accept and answer requests from system 2.
Old answer:
You have to set the bind-port property in the Akka configuration. Here is the quote from the Akka documentation:
# Use this setting to bind a network interface to a different port
# than remoting protocol expects messages at. This may be used
# when running akka nodes in a separated networks (under NATs or docker containers).
# Use 0 if you want a random available port. Examples:
#
# akka.remote.netty.tcp.port = 2552
# akka.remote.netty.tcp.bind-port = 2553
# Network interface will be bound to the 2553 port, but remoting protocol will
# expect messages sent to port 2552.
For your ports it should be like that:
port = 42431
bind-port = 4241
Related
At the moment, my RWS (RPi-WebRTC-Streamer) application works on my local network. I am now trying to connect it to my hosted coTURN server.
My main_rws_orig.js is pointing at my coTurn server:
var localTestingUrl = "ws://10.0.0.11:8889/rws/ws";
//var pcConfig = {"iceServers": [{"urls": "stun:stun.l.google.com:19302"}]};
var pcConfig = {"iceServers": [{"urls": "stun:172.104.xxx.xxx:3478"}]};
In using https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
I get the following when testing TURN and STUN:
0.009 rtp host 3376904655 udp c7f50fee-cbd0-4332-ae51-a013c4d35c5e.local 41771 126 | 30 | 255
0.091 rtp srflx 842163049 udp 42.116.95.19 41771 100 | 30 | 255
0.158 rtp relay 3617893847 udp 172.104.xxx.xxx 17857 2 | 30 | 255
39.809 Done
39.811
My coTurn web configuration tool is working also.
I have read about a signalling server, but have not found much documentation regarding it. I am just trying to figure out how to finish. How do connect my RWS application to the outside world using coturn.
Any tips or information will be greatly appreciated.
A signalling server is basically a service that sends the ICE candidates betweens the peers of your conversation. Usually it uses Websockets for this communication. The ICE candidates may include the CoTurn server credentials you provide to the WebRTC Object in JavaScript. But you need to share all candidates between the participants and for THIS you need the signalling server. You can use any language that supports full websockets communication like NodeJS or Java (not PHP!).
Take a look at this article, it describes this very well: https://www.html5rocks.com/en/tutorials/webrtc/infrastructure/#how-can-i-build-a-signaling-service
Except you explicitly want to use Peer-To-Peer WebRTC, I recommend you also to take a look at the tutorials of Kurento Media Server to get a better understanding of this principle, the NodeJS/Java is signalling between Kurento and your browser. Please note by using a media server it will be always in the middle between the participants, what has advantages like reducing the network usage of each participant + recording the whole conversation on the media server, but also disadvantages like no end-to-end encryption.
NodeJS example: https://doc-kurento.readthedocs.io/en/6.14.0/tutorials/node/tutorial-one2one.html
Java example: https://doc-kurento.readthedocs.io/en/6.14.0/tutorials/java/tutorial-one2one.html
Should a server application that listens on a port, able to detect and logs down any connection attempt done by Syn Scanning?
Test Scenario
I had written a windows program which i simply called it "simpleServer.exe".
This program is just a simulation of a very basic server application.
It listens on a port, and wait for incoming messages.
The listening Socket was defined to be a TCP Stream Socket.
that's all that this program is doing.
I had been deploying this exact same program on 2 different machines, both running on windows 7 professional 64bit.
This machine will act as a host.
and they are stationed in the same network area.
then, using the program "nmap",
i used another machine on the same network, to act as a client.
using the "-sS" parameter on "nmap", i do a Syn Scan, to the IP and Port of the listening simpleServer on both machine (one attempt at a time).
(note that the 2 hosts already had "wireshark" started, and is monitoring on tcp packets from the client's IP and to the listening port.)
In the "wireshark" entry, on both machine, i saw the expected tcp packet for Syn Scan:
client ----(SYN)----> host
client <--(SYN/ACK)-- host
client ----(RST)----> host
the above packet exchange suggests that the connection was not established.
But on the "simpleServer.exe", only one of it had "new incoming connection" printed in the logs, while the other instance was not alerted of any new incoming connection, hence no logs at all.
Code Snippets
// socket bind and listen was done above this loop
while(TRUE)
{
sClient=accept(sListen,(SOCKADDR*)&remoteAddr,&nAddrLen);
if(sClient == INVALID_SOCKET)
{
printf("Failed accept()");
continue;
}
dwSockOpt (sListen);
printf ("recv a connection: %s\n", inet_ntoa(remoteAddr.sin_addr));
closesocket(sClient);
}
side note:
yes, since it is just a simple program, the flow might be a little funny, such as no break in the while loop. so please don't mind this simple and flawed design.
Further Investigation
i had also put a getsockopt() in the "simpleServer" right after it went into listening state, to check the differences of both the listening socket's SOL_SOCKET option.
one notable difference i found between the two hosts, is the SO_MAX_MSG_SIZE.
the host that detects the incoming connection has a Hex value of 0x3FFFFFFF (1073741823), while the other one that has no logs is 0xFFFFFFFF (-1). not sure if this is related or not, but i just spammed whatever differences that i may found in my test environment. the other value of the SOL_SOCKET are more or less the same.
side note: i tested on some other machine, which covers another windows 7 professional, windows server 2008 r2, windows server 2003. i am not sure if it is coincidence or not, but machine that have SO_MAX_MSG_SIZE == -1, they all did not detect the connection of the Syn Scanning. but maybe it is just a coincidence. i have nothing to prove tho.
Help That I Needed
why is the different behavior from the 2 same of the same application on a different machine with the same OS?
what determines the value of the SO_MAX_MSG_SIZE? considering two same OS but having 2 different values.
If a connection is never established, accept() will never return. That disposes of 90% of your question.
The only explanation for the 'new incoming connection' (or 'recv a connection' or whatever it is) message is that something else connected.
SO_MAX_MSG_SIZE has no meaning for a TCP socket, let alone a listening TCP socket. So whatever variation you experienced is meaningless.
Based on a discussion with a colleague, I need some help:
Is it true, that without anything fancy (NAT, virtual-IP, etc.) a sw load-balancer can sustain a max of. ~64K backend (to a server) connections concurrently (at the same time), based on the port limit on the network interface it's bind on?
So for example, if there's a sw load-balancer connecting to 2 backend servers using long-lived TCP connections (not for HTTP) each of the servers can have up to ~64K connections at a given time?
(let's forget that the port limit is usually less than ~64K per connection)
Thanks.
The limitation comes from the fact that every connection through the LB requires 1 unique port on the internal interface of the LB.
If you only have 1 internal and 1 external interface pair then you will only be able to sustain approximately 64,000 connections due to port exhaustion. All the servers must share that limitation. This means that if you have 2 servers then they will have, if evenly distributed, around 32k connections each.
You can however easily lift this bottleneck by adding more than 1 internal interface on the LB.
What is the format of network block device protocol? It is stated that it is simple, but I can't find any RFC or similar things that describes what client and server should send.
Found myself. Looks like here is the document: https://github.com/yoe/nbd/blob/master/doc/proto.md . Not so simple...
Additionally there is simple Python-based server: http://lists.canonical.org/pipermail/kragen-hacks/2004-May/000397.html
Further adding to it, the exact communication format of NBD client and NBD server can be briefly described as following:
Client:
The client component of NBD configures a local block device as /dev/nbdX. Requests submitted to this device are sent through a socket to the server side implemented in userspace. The client can be configured with a userspace utility called nbd-client [1].
Server:
Server implements userspace handlers for requests sent by the client.
NBD can use Unix-domain sockets instead of network sockets to eliminate the
overhead of connection management. Furthermore, a multi-connect connection
can be used to increase performance due to introduced parallelism in the
server part [1]. High-performance server implementations with plugin support
exist, such as nbdkit or nbd-server.
In addition to the earlier useful answer that mentions proto.md below are some more useful resources that can help to understand in more detail the functions of client and server.
References and Resources:
[1] BUSE: Block Device in Userspace
[2] The Network Block Device-Linux Journal
[3] BDUS: implementing block devices in user space
Since Socket is used in hadoop src, guess it's TCP connections to send/recv messages and files, right?
How JVM translate these Socket instances to Linux system calls, is that socket/send, or select/poll?
If it's all about select/poll, I still can get ip/port through relative socket system calls, right?
When I collect all the sys_calls during a terasort (1 master, 3 slaves), I got rare connect/accept/socket sys_calls, and they are without any LAN ip in the sockaddr struct (either 0 or strange ones, IPv4). There are bunches of select/poll sys_calls, is that reasonable?
you could use "netstat --tcp -n" commands to check current tcp connection. i guess Hadoop should use TCP.
you may need use strace to start your Hadoop JVM. The strace will print system calls used by the running application. Usual application use sys_poll to check connection FD's status and use read/write or sendto/recvfrom syscall to receive transmit packet.
?
Right, these system call is only called once during connection setup through sys_socket system call, then application does many polls, transmit or receives operation on that socket.