Followed this tutorial to setup two ec2 instances: 12 . Creation of two EC2 instances and how to establish ping communication - YouTube
The only difference is I used a linux image.
I setup a simple python http server on a machine (on port 8000). But I cannot access this from my other machine; whenever I curl, the program kind of waits. (It might eventually timeout but I wasn't patient enough to witness that).
However, the workaround, I figured, was that you have to add a port rule via the security group. I do not like this option since it means that that port (for the machine that hosts the web server) can be accessed via the internet.
I was looking for an experience similar to what people usually have at home with their routers; machines connected to the same home router can reach out to other machines on any port (provided the destination machine has some service hosted on that port).
What is the solution to achieve something like this when working with ec2?
The instance is open to the internet because you are allowing access from '0.0.0.0/0' (anywhere) in the inbound rule of the security group.
If you want to the communication to be allowed only between the instances and not from the public internet. You can achieve that by assigning the same security group to both the instances and modifying the inbound rule in the security group to allow all traffic or ICMP traffic sourced from security group itself.
You can read more about it here:
AWS Reference
I am trying to get the big picture although my primary domain is not networking.
Some question's narrowed down for which I'm not getting enough/proper answers online
Is the IP that is resolved by the DNS server when I hit www.google.com is same as any of the Google router's Gateway IP?
Do bigger companies like Amazon do port forwarding?
If point 2 is true, I suppose they must be port forwarding with only 443 (https) port which means, to use multiple static IP across different data centers, they need to have that many routers. So, if they have N static IP address which resolves to a website, then they must be having N routers right? Is this a fair assumption?
A gateway IP refers to a device on a network which sends local
network traffic to other networks. it sits between you and internet,or other network . its like a watchman.
Question 1 : google.com has multiple ip addresses lets say then , Yes, that is possible, and will need to be two A records. This is called Round-Robin DNS. Clients will semi-randomly use one of the two addresses.
question2: yes port forwarding happens more often than we think. ALL VPC's (virtual private clouds like AWS , GCP , Azure etc) use this as they dont want to expose servers/internal resources to the internet.
depending on the port number , particular service is exposed to requesting client. lets say we want to make a website public , then we explicitly expose port 80(http) 443(https) so that web crawlers and users can see them.
Port forwarding, sometimes called port mapping, allows computers or
services in private networks to connect over the internet with other
public or private computers or services.
google https://www.google.com:444/ wont work because they did not expose port 444 on their cloud router
but https://www.google.com:443/ will work because the server corresponding to google.com has explicitly left it open.
How IP is resolved:
Step 1 - Send a Request to Resolve a Domain Name
When you type www.google.com into a browser, in order to load the webpage, your computer asks for the IP address. Computers do not know in advance where they can find the necessary information, so they try searching through the DNS cache and for available external source. proceed from lower level caches to root/main servers.
Step 2+3 - Try to resolve an IP Locally
Before going externally, your computer loads the local DNS cache database to see if you already requested the IP for that domain name. Every computer has a temporary cache with the most recent DNS requests and attempts to connect to online sources. if required record is present locally its called "CACHE HIT" and query stops.
However A computer’s local DNS cache database does not always contain the necessary data to resolve a domain name this is called a "CACHE MISS" . In that case, the request goes further to your Internet Service Provider (ISP) and its DNS server.
Step 4 - ISPs Ask Outside DNS Servers to Provide an IP Address iff Cache miss
ISP DNS resolvers are configured to ask other DNS servers for correct IP address mapping until they can provide data back to the requester. These are iterative DNS queries.
When a DNS client sends such a request, the first responding server does not provide the needed IP address. Instead, it directs the request to another server that is lower in the DNS hierarchy, and that one to another until the IP address is fully resolved. There are a few stops in this process.
hierarchy looks like this (just for reference):
Root domain nameservers. Root servers themselves do not map IP addresses to domain names. Instead, they hold the information about all top-level domain (TLD) nameservers and point to their location. TLD is the rightmost section of a domain name... Root servers are critical since they are the first stop for all DNS lookup requests.
TLD nameservers. These servers contain the data for second-level domains, such as ‘phoenixnap’ in phoenixnap.com. Previously, the root server pointed to the location of the TLD server. Then, the TLD server needs to direct the request toward the server that contains the necessary data for the website we are trying to reach.
Authoritative nameserver. Authoritative servers are the final destination for DNS lookup requests. They provide the website’s IP address back to the recursive DNS servers. If the site has subdomains, the local DNS server will keep sending requests to the authoritative server until it finally resolves the IP address.
Step 5 - Receive the IP Address
Once the ISP’s recursive DNS server obtains the IP address by sending multiple iterative DNS queries, it finally returns it to your computer. The record for this request now stays cached on the hard drive. The browser can then fetch this IP from the cache and connect it to the website’s server.
ALL this happens in less than 1 second, most of the times. if you just registered a new domain it might take few hours to propagate this DNS cache globally hence newly registered websites do not show up sometimes.
About companies owning multiple IPs
Big companies have pool of IPs reserved for example 123.234.xxx.xxx which means a company has reserved 255*255 ips. they are mapped on a VPC(virtual private cloud)
and accessible vis a subnet masking and CIDR feature, like your EC2 instances on AWS
Is the IP that is resolved by the DNS server when I hit www.google.com is same as any of the Google router's Gateway IP?
For sure it should, but it is mostly a Google management question that only they will be able to answer right. The thing is that we must understand how DNS query's work for this.
Let's take a look of it:
Device A requests the IP address through a DNS query of the device B.
To do this, it uses the network port 53 (Domain) on which it will ask, depending on which DNS server is being used at the time, which is usually the home router. Then the router will ask the ISP's DNS server, which will respond with a cached response, or the query with another server on top of it if it does not have one; All this process is followed until a reliable cache response is reached or until the authoritative response server is reached, that is, the name server that manages the domain in question.
Only the authoritative response server contains the reliable information of which IP of the domain which is going to be reached.
I suppose that within Google's servers and its network they use Google's own DNS servers, which are 8.8.8.8 and 8.8.4.4 where the DNS records are obtained and consulted by caching from many sites.
In general terms Google's IP will change depending on where you are, I made a DIG query to Google's authoritative servers, however, I received a result based on location to improve the route and loading time of the site which was 142.250.73.238.
Do bigger companies like Amazon do port forwarding?
Yes, they do. To handle queries with load balancers or similar and even for caching dns requests.
If point 2 is true, I suppose they must be port forwarding with only 443 (https) port which means, to use multiple static IP across different data centers, they need to have that many routers. So, if they have N static IP address which resolves to a website, then they must be having N routers right? Is this a fair assumption?
This has multiple answers. By the way, they actually can do a secure DNS query.
if they have N static IP address which resolves to a website, then they must be having N routers right?
They don't have to, but if they want to they can.
"Is this a fair assumption?"
No, the IP's doesn't depend on a router, the router only routes to a computer/server which can have multiple IP's. By the other hand, each thing (computer, server, etc... must have an IP which can be also a WAN IP).
i am trying out Kubernetes on bare-metal, as a example I have docker containers exposing port 2002 (this is not HTTP).
I do not need to load balance traffic among my pods since each of new pod is doing its own jobs not for the same network clients.
Is there a software that will allow to access each new created service with new IP from internal DHCP so I can preserve my original container port?
I can create service with NodePort and access this pod by some randomly generated port that is forwarded to my 2002 port.
But i need to preserve that 2002 port while accessing my containers.
Each new service would need to be accessible by new LAN IP but with the same port as containers.
Is there some network plugin (LoadBalancer?) that will allow to forward from IP assigned by DHCP back to this randomly generated service port so I can access containers by original ports?
Starting service in Kubernetes, and then accessing this service with IP:2002, then starting another service but the same container image as previous, and then accessing it with another_new_IP:2002
Ah, that happens automatically within the cluster -- each Pod has its own IP address. I know you said bare metal, but this post by Lyft may give you some insight into how you can skip or augment the SDN and surface the Pod's IPs into routable address space, doing exactly what you want.
In more real terms: I haven't ever had the need to attempt such a thing, but CNI is likely flexible enough to interact with a DHCP server and pull a Pod's IP from a predetermined pool, so long as the pool is big enough to accommodate the frequency of Pod creation and termination.
Either way, I would absolutely read a blog post describing your attempt -- successful or not -- to pull this off!
On a separate note, be careful because the word Service means something specific within kubernetes, even though it is regrettably a word often used in a more generic term (as I suspect you did). Thankfully, a Service is designed to do the exact opposite of what you want to happen, so there was little chance of confusion -- just be aware.
I'm currently configuring DNS in my network with Bind9.
This network is splitted in VLANs and it should have 2 DNS zones : an internal zone (internal servers, users VLAN...) and an external zone (DMZ).
Of course the DNS mustn't give internal records to an external request.
I have just configured my DNS master (storing external and internal records) in "internal servers" VLAN and I'm asking myself how to deal with this problem :
-> My DNS master will not reply to an external request since it is in the internal zone (though 802.11Q is enabled), even with external records. Is it right?
-> My DNS mustn't be in the DMZ.
-> If I configure a slave on the DMZ to manage the external zone, who will just store external records, then I have to configure another slave to replicate the master (so 3 servers...).
Initially I just planned to configure 2 servers, a master and a slave which would just be a full replicate.
Am I missing something? Is there any better solution?
The architecture that you will follow depends on what you want to serve and where (internal clients, external both)?
In general, yes you need internal DNS servers these will have at least all the necessary entries for you internal network.
As for the servers in DMZ who will have access to them and from which path? If you need access to them from the internal network directly through firewall, it makes sense to have entries for them also in your internal dns servers.
If you serve also from servers in DMZ to external users you have two solutions either you declare them in public DNS if the external users are from internet or you should have another DNS server on your DMZ.
The key point here is to think who will visit your server and from which path, then it will become kind of obvious how you will configure DNS for them.
I have an ELB (in EC2 classic) running and one of my client want to hardcode an IP to his firewall rule to access our site.
I know that ELB doesn't provide static IP but is there a way to set up an instance (just for them) that they could hit and be used as gateway to our API?
(I was thinking of using HA Proxy on OpsWorks but it points directly to my instances and I need something that points to my ELB because SSL resolution happens at this level)
Any recommendation would be very helpful.
I assume you are running one or more instances behind your ELB.
You should be able to assign an elastic IP to one of those instances. In EC2 classic, EIPs will need to get reattached when you restart your instance.