GKE uses the kubenet network plugin for setting up container interfaces and configures routes in the VPC so that containers can reach eachother on different hosts.
Wikipedia defines an overlay as a computer network that is built on top of another network.
Should GKE's network model be considered an overlay network? It is built on top of another network in the sense that it relies on the connectivity between the nodes in the cluster to function properly, but the Pod IPs are natively routable within the VPC as the routes inform the network which node to go to to find a particular Pod.
VPC-native and non VPC native GKE clusters uses GCP virtual networking. It is not strictly an overlay network by definition. An overlay network would be one that's isolated to just the GKE cluster.
VPC-native clusters work like this:
Each node VM is given a primary internal address and two alias IP ranges. One alias IP range is for pods and the other is for services.
The GCP subnet used by the cluster must have at least two secondary IP ranges (one for the pod alias IP range on the node VMs and the other for the services alias IP range on the node VMs).
Non-VPC-native clusters:
GCP creates custom static routes whose destinations match pod IP space and services IP space. The next hops of these routes are node VMs by name, so there is instance based routing that happens as a "next step" within each VM.
I could see where some might consider this to be an overlay network. I don’t believe this is the best definition because the pod and service IPs are addressable from other VMs, outside of GKE cluster, in the network.
For a deeper dive on GCP’s network infrastructure, GCP’s network virtualization whitepaper can be found here.
Related
Long story short: Connecting to mongo atlas, and trying to whitelist the smallest ip range possible.
VPC peering won't work as the Mongo cluster is hosted in AWS, and it's just a Mongo Atlas limitation. Also, some of our mongo clusters are M5 (or lower), and they don't support VPC Peering.
That being said, I'm not sure what the public/external ip of my pods will be, when they attempt to connect to Mongo. If not narrowing the outbound ip range as much as possible, what other options exist.
The GKE cluster is not private, and it's autopilot
Found these two articles that shows how you can route egress traffic to a single IP that can be used for whitelisting your GKE from Mongo Atlas:
Route the GKE cluster's egress traffic via Cloud NAT
or
Route your Public GKE cluster’s egress traffic via NAT instances
Unfortunately, both of these options only work for Non Autopilot GKEs. For routing the GKE cluster's egress traffic via Cloud NAT, the desired networking behavior is currently not supported by public Autopilot GKE clusters. The cluster's IP masquerade configuration is not configured to perform SNAT within the cluster for packets sent from Pods to the internet. Currently, there isn't a way to configure the IP masquerade agent to not masquerade the pod range when reaching out to the internet in Autopilot clusters. As a result, pod egress traffic in public Autopilot GKE clusters will be using the node's external IP.
So to move forward with Cloud NAT it's either:
use a private GKE cluster, which can be in Autopilot mode.
use a public GKE cluster, but not in Autopilot mode.
Using Autopilot, you cannot use an inexpansive solution that is using kube-ip which will allow you to associate public GCP static ips to your GKE nodes. It is a daemon that runs in your kubernetes cluster and keeps checking and associating if necessary GCP ips that you bought earlier to your GKE nodes.
With Autopilot, what you can do is to use the Network Endpoint Group (NEG) abstraction layer that enables container-native load balancing. It allows to configure path-based or host-based routing to their your backend pods. The ingress is free, but you will pay for GCP loadbalancing which is much more expansive that reserving a few ips with kube-ip in most cases.
Here are more informations about GCP's NEG. Plus another related stackexchange question.
I have a compute engine instance with persistent file storage that I need outside of my GKE cluster.
I would like to open a specific TCP port on the Compute Engine instance so that only nodes within the GKE cluster can access it.
The Compute Engine instance and GKE cluster are in the same GCP project, network, and subnet.
The GKE cluster is not private and I have an ingress exposing the only service I want exposed to the internet.
I've tried creating firewall rules of three different types that do not work:
By shared service account on both Compute Engine instance and K8s nodes.
By network tags - (yes I am using the network tags as explicitly specified on the VM instance page).
By IP address, where I use network tag for target and private IANA IP ranges 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 for source.
The only thing that works is the last option but using 0.0.0.0/0 for source IP range.
I've looked at a few related questions such as:
Google App Engine communicate with Compute Engine over internal network
Can I launch Google Container Engine (GKE) in Private GCP network Subnet?
But I'm not looking to make my GKE cluster private and I have tried to create the firewall rules using network tags to no avail.
What am I missing or is this not possible?
Not sure how I missed this, fairly certain I tried something similar a couple months back but must have had something else misconfigured.
On the GKE cluster Details page, there is a pod address range. Setting the firewall source range to GKE pod address range gave me the the desired outcome.
In GCloud we have one Kubernetes cluster with two nodes, it is possible to setup all nodes to get the same external IP? Now we are getting two external IP's.
Thank you in advance.
The short answer is no, you cannot assign the very same external IP to two nodes or two instances, but you can use the same IP to access them, for example through a LoadBalancer.
The long answer
Depending on your scenario and the infrastructure you want to set up, several ways are available to expose different resources through the very same IP.
I do not know why you want to assign the same IP to the nodes, but since each node it is a Google Compute Engine instance you can set up a Load Balancer (TCP, SSL, HTTP(s), internal, ecc). In this way you reach the nodes as if they were not part of a Kubernetes cluster, basically you are treating them as Compute Engine instances and you will able to connect to any port they are listening on (for example an HTTP server or an external health check).
Notice that you will be not able to connect to the PODs in this way: the services and the containers are running in a separate software bases network and they will be not reachable if not properly set, for example with a NodePort.
On the other hand if you are interested in making your PODs running in two different kubernetes nodes reachable through a unique entry point you have to set up Kubernetes related ingress and load balancing to expose your services. This resources are based as well on the Google Cloud Platform Load Balancer components, but when created they trigger as well the required change to the Kubernetes Network.
I have a clustered legacy application that I am trying to deploy on kubernetes. The nodes in the cluster find each other using UDP broadcast. I cannot change this behaviour for various reasons.
When deployed on docker, this would be done by creating a shared network (i.e. docker network create --internal mynet, leading to a subnet e.g. 172.18.0.0/16), and connecting the containers containing the clustered nodes to the same network (docker network connect mynet instance1 and docker network connect mynet instance2). Then every instance starting would broadcast it's IP address periodically on this network using 172.18.255.255 until they have formed a cluster. Multiple such clusters could reside in the same kubernetes namespace, so preferrably I would like to create my own "private network" just for these pods to avoid port collisions.
Is there a way of creating such a network on kubernetes, or otherwise trick the application into believing it is connected to such a network (assuming the IP addresses of the other nodes are known)? The kubernetes cluster I am running on uses Calico.
Maybe you can Set label for your app pod, and try NetworkPolicy on Calico.
When using Docker swarm mode and exposing ports outside, you have at least three networks, the ingress network, the bridge network and the overlay network (used for internal cluster communications). The container joins these networks using one of eth0-2 (randomically each time) interfaces and from an application point of view is not easy to understand which of these is the cluster network (the correct one to use for service discovery client publish - e.g Spring Eureka).
Is there a way to customize network interface names in some way?
Not a direct answer to your question, but one of the key selling points of swarm mode is the built-in service discovery mechanism, which in my opinion works really nicely.
More related, I don't think it's possible to specify the desired interface for an overlay network. However, when creating a network, it is possible to define the subnet or the IP range of the network (https://docs.docker.com/engine/reference/commandline/network_create/). You could use that to identify the interface belonging to your overlay network, by checking if the bound IP address is part of the network you want to publish on.