I am trying to send data from a router to the kafka brokers. Since a router can only be configured in a way that it knows the IP and port number of the Kafka server.
I do not want to introduce a layer of java to consume messages from the router because it will cause latency.
Related
We have a Kafka consumer which is consuming the messages perfectly.
But the thing is , When the Kafka consumer started, the consumer is sending lot of TCP packets to Kafka Broker. I can sniff those packets using Wireshark.
Our Team understanding is like, It should send the TCP packet whenever the message is getting produced by the Producer and the Consumer should be able to consume those messages by sending a request to Broker. (Like Event Driven)
We cannot really understand why the TCP packets are getting transferred between the Broker and Consumer continuously when the consumer is getting started.
Kafka works as a pull based system communicating over TCP. That's why consumers continuously send TCP requests (poll() request and heartbeats, actually). You can check this link for more information about this design decision of Kafka.
I have a Kafka cluster on a private LAN, I want to have a consumer access the data on a different LAN, due to network restrictions, I can only access the main IP address (no DNS) of the cluster, let's call it master-node.
My consumer connects to the cluster without a problem, but the cluster instructs the consumer to fetch data from node1, node2 and node3, which I do NOT have network access to.
Is there a way to ask the master-node to gather the data on behalf of my consumer?
Consumers connect directly to the individual brokers which are leaders for individual partitions. This is to provide high scalability. By funnelling all traffic through a single endpoint, you are introducing a single point of failure.
If you need such a "proxy", then only option I am aware of would be the Kafka REST Proxy, and then you would have to consume and produce over HTTP rather than native Kafka clients.
Hi I am slightly confused on how UDP Hole Punching works and how I would implement it. According to this wikipedia article:
https://en.m.wikipedia.org/wiki/UDP_hole_punching#Flow
Both the clients that want to establish a p2p connection must set up a UDP conversation with the server in order to exchange ip's and punch holes. What I am confused on is lets say client a wants to initiate a p2p conversation with client b. How would client b know to connect to the sever in order for the clients to swap ip's? This is required else they would not know the other clients ip. Am I misunderstanding this concept somehow?
In the regular case the peers do not have static IP addresses and also public ports are allocated dynamically with transient routing rules, valid for typically 1 to 3 minutes.
There is no way to guess the dynamic port of the dual peer and even there is no way to establish immediate routing to it without predefined forwarding rules.
In contrast to often transcribed documentation, the hole punching through router + internet service provider is actually done with sending UDP packets to a public mediator server.
The peers contact each other by re-using exactly the public ip/port currently seen by the mediator server.
The mediator server's router necessarily has a forwarding rule to the server, so the server is publicly reachable and a public communication can be initated.
If the mediator server does not have a static address, a public DNS server is required to resolve the server's dynamic address.
For trapping local port mapping and returning packets to the right target,
all three nodes will maintain a mapping of a unique static node identifier to the current ip/port of incoming packets; each of the clients needs to send periodically alive message with client identifier to the mediator server and counterpart; the mediator server responds with an alive-acknowledge message carrying the mediator server's current public address/port. Here, optional port mapping rules of the routers are to be considered to get the actual public ports.
The dynamic adjustment of remote ports makes it difficult to have several independent communication channels, at least I'm not aware of a fork mechanism for UDP servers.
If there is a requirement for independent communication channels, e.g. like in FTP you have a command port and a streaming port, the packet protocol may be extended by a logical port and incoming packets may be dispatched according to the logical port.
Finally there are security risks:
1.) the communication could be hi-jacked, by anyone sniffing on any node of the routing path; he could send and alive message from a different address to one of the peers and so would inherit the peer's communication stream.
The minimum solution here is to add authentication to the alive messages.
2.) Certainly, encryption is mandatory for user data in any public network!
Due to uncertainty of delivery of UDP packets, encryption is just possible on packet base, as e.g. AES/ECB does, and so should be chosen strong.
I am able to access to my IOT device if I assign it a static IP with a service like dydns. But I would like to be able to reach it without relying on something.
I was thinking that my IOT device could ,on every start, write in a database its IP adress so my script on the server knows its ip.
The problem is that the IP will correspond to my hotspot IP. I am missing the connection part "hotspot -->IOT device".
For example I would like to be able to connect my IOT device to my mobile phone in hotspot mode.
One way to do this would be to re-design your system: if when it comes online the IoT device always connects (and stays connected) to a server component (which has a well-known/unchanging hostname/IP address), then the server can always send the IoT device a command over that TCP connection without knowing the hostname/IP address of the IoT device, and without it having to be contactable from the internet, i.e. this approach is firewall-friendly at the device end.
This is how the IoT is architected when using e.g. MQTT: devices connect inwards to the MQTT broker (i.e. server). MQTT also removes the need for the server application itself to need to connect to the clients. MQTT uses a concept called publish/subscribe with pre-agreed 'topics' - the client will subscribe to a 'command' topic, the server app publishes commands to that topic and the broker handles forwarding the command to the device. Incoming data from the device is published to another topic and the server application subscribes to that topic, the broker forwards the published data. You can try pub/sub example (using a browser, but real devices can also connect to the same server) using e.g. http://m2m.demos.ibm.com/utilities.html
I have a legacy setup that utilizes a Zigbee Controller within a LAN. We have an application that communicates with that controller over UDP to send commands to Zigbee devices.
We'd like to be able to send commands from a cloud-based server to a controller running within a LAN without forwarding a port on the firewall.
So ultimately: Our app in cloud => (something) => Local Firewall => LAN => (something) => Zigbee Controller
I'm wondering what the best (somethings) are to accomplish this. In the end, I want to take our current UDP messages that are being sent on (some) port, translate them to packets that can be sent to a device inside a LAN on port 80 which then re-sends them over the proper UDP port. I'm assuming this is somehow possible, but I'm not sure of the right approach / device.
Any suggestions?
You may try "hole punching" over UDP. The basic idea is to let the client (your ZigBee controller) send a periodic message to the cloud over UDP to maintain an active connection. Your cloud server will be able to send messages to the controller, without port forwarding and, because of UDP, without having to maintain an active session like it does over TCP.