Supposed I have two services that need to share and / or exchange data. Both instances are separate from each other, and they shall not know anything about where the other part is located.
Now in order for them to be able to share and / or exchange data, they need to connect to each other.
How do they find each other without the need to configure the IP addresses explicitly? In other words: How could they detect each other automatically?
Basically, I have two ideas:
Pull: You need to have a central service where they register. Then you can ask that service for the address of a service, and that service then returns those data. While this works, it has the drawback that it only shifts the problem to the next level: What if I have multiple instances of that service, and I don't want them to know each other in advance?
Push: Each service broadcasts its own address, so that other services get it to know. Each service repeats this from time to time. Drawback: This does hardly work in the internet.
Any idea of how I could solve this in an intelligent way?
PS: If you want to say so, I'm looking for a way to handle dynamic IPs without the need for a central DNS server.
The usual way is to have some fault-tolerant server where services register and can then look for other services - Curator framework implements that over zookeeper.
If you want autodiscovery then you should probably implement some sort of gossip protocol so that the servers would know which other servers are out there in a reliable way. You should keep in mind that getting gossip protocols right is tricky (e.g. some of past Amazon cloud failures where due to problems in their implementation)
"broadcast packets are not forwarded everywhere on a network, but only to devices within a broadcast domain."
If your devices are on different broadcast domains then broadcasting is not going to work.
You are probably going to have to implement your own central service, unless you can use one of the free dynamic dns servers, for example: Free
Related
I am looking for a basic thing yet I have not found not even a single good documentation on getting it done.
I want to allocate a floating IP, then associate it to a network interface of a droplet other than eth0.
The reason is I want to have the ability to very easily switch from one IP to the other with a programming language.
In a few words, I want to be able to do these two commands and both should provide a different response.
curl --interface eth0 https://icanhazip.com
curl --interface eth1 https://icanhazip.com
Also, I want to know what to do once I release the Floating IP, how do I roll back to the starting point.
All documentation I read, rely heavily on "ip route" and "route", most did not even work, some worked but replaced completely the old IP by the floating and that's not what I want, and also they did not show how to rollback the introduced configuration changes.
Please help, I spent 1 whole day now trying to get this to work for a project, and no results so far.
I guess there is no need to know DigitalOcean, how to make this work on other Cloud Providers would apply here too I think.
Update
After asking this on DigitalOcean community forum (https://www.digitalocean.com/community/questions/clear-guide-on-outbound-network-through-floating-ip), they claim that is not supported, although there may be some solutions to this if somebody can provide such a "hacky" solution I would take it too. Thanks
In the cloud (AWS. GCP etc.) ARP is emulated by the virtual network layer, meaning that only IPs assigned to VMs by the cloud platform can be resolved. Most of the L2 failover protocols do break for that reason. Even if ARP worked,the IP allocation process for these IPs (often called “floating IPs”) would not integrate with the virtual network in a standard way, so your OS can't just "grab" the IP using ARP and route the packets to itself.
I have not personally done this on Digital Ocean, but I assume that you can call the cloud's proprietary API to do this functionality if you would like to go this route.
See this link on GCP about floating IPs and their implementation. Hope this is helpful.
Here's an idea that needs to be tested:
Let's say you have Node1(10.1.1.1/24) and Node2(10.1.1.2/24)
Create a loopback interface on both VMs and set the same IP address for both like (10.2.1.1/32)
Start a heartbeat send/receive between them
When NodeA starts it automatically makes an API call to create a route for 10.2.1.1/32 and points to itself with preference 2
When NodeB starts it automatically makes an API call to create a route for 10.2.1.1/32 and points to itself with preference 1
The nodes could monitor each other to withdraw the static routes if the other fails. Ideally you would need a 3rd node to reach quorum and prevent split brain scenarios, but you get the idea right?
Here's the issue...
I work in a remote area of Alaska where cell service can be... questionable. We do have decent WiFi, however, is not openly available to staff because it has a low enough data limit that we don't want to deal with people streaming Netflix and running out the company data.
The big issue is that we want to use an app like Slack or Discord to communicate more effectively across the business. Because the cell service is spotty and the WiFi is currently off-limits, I was wondering if there was a way for me to create a WiFi network that was whitelisted to only allow Slack, for example. Then we get the benefits of using the WiFi without risking running out of data.
Thoughts? I was thinking about setting up a network proxy, but I wanted to get the internet's take on it before I dive down the rabbit hole.
The best way I can think of to handle something like this is to use a router that you can configure the dns server settings and block all dns entries that aren’t on your allowlist. This doesn’t strictly block traffic to everywhere but it will do a pretty good job.
You could also block all dns traffic that’s not going to the local dns server which would help not allow people around it. To have a hard block on things you would need to block specific options addresses which with services such as slack or discord could change randomly which would be hard to keep up on.
Another option that would work well is use your own self hosted version of mattermost, rocket.chat, or riot/matrix that you would have control over and knowledge of the IP address so that you can allowlist only those up addresses. The other advantage with this is if the business is just localized communication and you don’t need to chat across long distances then you could set this to work completely on a network with no internet access so you wouldn’t have to do any blocking because the wifi is completely separated from the internet.
Things are heavily based on your situation but I hope this gives you a good place to start
I have a web app and I would like to prevent DOS attacks by blocking an IP address if it make many request in a short period of time.
For example, if the same IP address makes 100 request in a second, I can assume that it's some kind of attack and I would like to block this IP.
However, making this check in the application layer seems too expensive - what is the optimal way to make this check?
Should I make this kind of check at my:
firewall
router
apache config
someplace else entirely ...
If you want to block IP addresses when they make a certain number of requests, this is best done at the Network layer. This would suggest that you do this either in your host machine's network stack or using a router (which operates at the network layer).
Some things you might want to consider though are:
- Are you really wanting to block access to the entire host based on an IP address, or do you want to block access to a specific application running on a specific port.
- Sometimes, by using NATs, one IP address may be making requests on behalf of many real hosts.
With any security application you need to have many layers of defence, so it would be a good idea to invest in a good firewall as well.
Some apps for generating APIs in django implement some methods for limiting the amount of request per second.
For example django-piston use throttling method to do that.
django-piston throttling
Thats an easy way to solve the problem.
We're implementing a SIP-based solution and have configured the setup to work with RTPProxy. Right now, we're routing everything through RTPProxy as we were having some issues with media transport relying on ICE. If we're not mistaken, a central relay server is necessary for relaying streaming data between two clients if they're behind symmetric NATs. In practice, is this a large percentage of all consumer users? How much bandwidth woudl we save if we implemented proper routing to skip the relay server when not necessary. Are there better solutions we're missing?
In falling order of usefulness:
There is a direct connection between the two endpoints in both directions. You just connect and you are essentially done.
There is a direct connection between the two endpoints in one direction. In that case you just connect via the right direction by trying both.
Both parties are behind NATs of some kind.
Luckily, UPnP works in one end, you can then upgrade the connection to the above scheme
UPnP doesn't work, but STUN does. Use it to punch a hole in the NAT. There are a couple of different protocols but the general trick is to negotiate via a middle man that coordinates the NAT-piercing.
You fall back to let another node on the network act as a relaying proxy.
If you implement the full list above, then you have to give up very few connections and don't have to spend much time on bandwidth utilization at proxies. The BitTorrent protocol, of which I am somewhat familiar, usually stops at UPnP, but provides a built-in test to test for connectivity through the NAT.
One really wonders why IPv6 did not get implemented earlier - this is a waste of programmers time.
Real world NAT types survey (not a huge dataset, though):
http://nattest.net.in.tum.de/results.php
According to Google, about 8% of the traffic has to be relayed: http://code.google.com/apis/talk/libjingle/important_concepts.html
A large percentage (if not the majority) of home users uses NAT, as that is what those xDSL/cable routers use to provide network access to the local network.
You can theoretically use UPnP to open ports and set-up forwarding rules on the router to go through the NAT transparently. Unfortunately (or fortunately, depending on who you are) many users disable UPnP as a matter of course on their router and may not appreciate having to add forwarding rules manually.
What you might be able to do (and what Skype does AFAIK) is to have some of the users that have clear network paths and enough bandwidth act as relay nodes. Apart from the routing and QoS issues, you would at least have to find some way to ensure the privacy of any relayed data from anyone, including the owner of the relay node. In addition, there might be legal issues to settle with this approach, apart from the technical ones.
I'm making a network game (1v1) where in-game its p2p - no need for a game server.
However, for players to be able to "find each other", without the need to coordinate in another medium and enter IP addresses (similar to the modem days of network games), I need to have a coordination/matching server.
I can't use regular web hosting because:
The clients will communicate in UDP.
Therefore I'll need to do UDP Hole Punching to be able to go through the NAT
That would require the server to talk in UDP and know the client's IP and port
afaik with regular web hosting (php/etc) I can only get the client's IP address and can only communicate in TCP (HTTP).
Options I am currently considering:
Use a hosting solution where my program can accept UDP connection. (any recommendations?)
UDPonNAT seems to do this but uses GTalk and requires each client to have a GTalk account for this (which probably makes it an unsuitable solution)
Any ideas? Thanks :)
First, let me say that this is well out of my realm of expertise, but I found myself very interested, so I've been doing some searching and reading.
It seems that the most commonly prescribed solution for UDP NAT traversal is to use a STUN server. I did some quick searches to see if there are any companies that will just straight-up provide you with a STUN hosting solution, but if there even were any, they were buried in piles of ads for simple web hosting.
Fortunately, it seems there are several STUN servers that are already up and running and free for public use. There is a list of public STUN servers at voip-info.org.
In addition, there is plenty more information to be had if you explore SO questions tagged "nat".
I don't see any other choice than to have a dedicated server running your code. The other solutions you propose are, shall we say, less than optimal.
If you start small, virtual hosting will be fine. Costs are pretty minimal.
Rather than a full-blown dedicated server, you could just get a cheap shared hosting service and have the application interface with a PHP page, which in turn interfaces with a MySQL database backend.
For example, Lunarpages has a $3/month starter package that includes 5gb of space and 50gb of bandwidth. For something this simple, that's all you should need.
Then you just have your application poll the web page for the list of games, and submit a POST request in order to add their own game to the list.
Of course, this method requires learning PHP and MySQL if you don't already know them. And if you do it right, you can have the PHP page enter a sort of infinite loop to keep the connection open and just feed updates to the client, rather than polling the page every few seconds and wasting a lot of bandwidth. That's way outside the scope of this answer though.
Oh, and if you're looking for something absolutely free, search for a free PHP host. Those exist too! Even with an ad-supported host, your app could just grab the page and ignore the ads when you parse the list of games. I know that T35 used to be one of my favorites because their free plan doesn't track space or bandwidth (it limits the per-file size, to eliminate their service being used as a media share, but it shouldn't be a problem for PHP files). But of course, I think in the long run you'll be better off going with a paid host.
Edit: T35 also says "Free hosting allows 1 domain to be hosted, while paid offers unlimited domain hosting." So you can even just pay for a domain name and link it to them! I think in the short term, that's your best (cheapest) bet. Of course, this is all assuming you either know or are willing to learn PHP in order to make this happen. :)
There's nothing that every net connection will support. STUN is probably good, UPnP can work for this.
However, it's rumored that most firewalls can be enticed to pass almost anything through UDP port 53 (DNS). You might have to argue with the OS about your access to that port though.
Also, check out SIP, it's another protocol designed for this sort of thing. With the popularity of VOIP, there may be decent built-in support for this in more firewalls.
If you're really committed to UDP, you might also consider tunneling it over HTTP.
how about you break the problem into two parts - make a game matcher client (that is distinct from the game), which can communicate via http to your cheap/shared webhost. All gamers who wants to use the game matching function use this. THe game matcher client then launches the actual game with the correct parameters (IP, etc etc) after obtaining the info from your server.
The game will then use the standard way to UDP punch thru NAT, etc etc, as per your network code. The game dont actually need to know anything about the matcher client or matcher server - in the true sense of p2p (like torrents, once you can obtain your peer's IPs, you can even disconnect from the tracker).
That way, your problems become smaller.
An intermediate solution between hosting your own dedicated server and a strictly P2P networking environment is the gnutella model. In that model, there are superpeers that act like local servers, having known IP addresses and being connected to (and thus having knowledge of) more clients than a typical peer. This still requires you to run at least one superpeer yourself, but it gives you the option to let other people run their own superpeers.