I am trying to set up a HTTP load balancer for my Meteor app on google cloud. I have the application set up correctly, and I know this because I can visit the IP given in the Network Load Balancer.
However, when I try and set up a HTTP load balancer, the health checks always say that the instances are unhealthy (even though I know they are not). I tried including a route in my application that returns a status 200, and pointing the health check towards that route.
Here is exactly what I did, step by step:
Create new instance template/group for the app.
Upload image to google cloud.
Create replication controller and service for the app.
The network load balancer was created automatically. Additionally, there were two firewall rules allowing HTTP/HTTPS traffic on all IPs.
Then I try and create the HTTP load balancer. I create a backend service in the load balancer with all the VMs corresponding to the meteor app. Then I create a new global forwarding rule. No matter what, the instances are labelled "unhealthy" and the IP from the global forwarding rule returns a "Server Error".
In order to use HTTP load balancing on Google Cloud with Kubernetes, you have to take a slightly different approach than for network load balancing, due to the current lack of built-in support for HTTP balancing.
I suspect you created your service in step 3 with type: LoadBalancer. This won't work properly because of how the LoadBalancer type is implemented, which causes the service to be available only on the network forwarding rule's IP address, rather than on each host's IP address.
What will work, however, is using type: NodePort, which will cause the service to be reachable on the automatically-chosen node port on each host's external IP address. This plays more nicely with the HTTP load balancer. You can then pass this node port to the HTTP load balancer that you create. Once you open up a firewall on the node port, you should be good to go!
If you want more concrete steps, a walkthrough of how to use HTTP load balancers with Container Engine was actually recently added to GKE's documentation. The same steps should work with normal Kubernetes.
As a final note, now that version 1.0 is out the door, the team is getting back to adding some missing features, including native support for L7 load balancing. We hope to make it much easier for you soon!
Related
I'm currently working on a project where we are using Google Cloud. Within the Cloud we are using CloudRun to provide our services. One of these services is rather complex and has many different configuration options. To validate how these configurations affect the quality of the results and also to evaluate the quality of changes to the service, I would like to proceed as follows:
in addition to the existing service I deploy another instance of the service which contains the changes
I mirror all incoming requests and let both services process them, only the responses from the initial service are returned, but the responses from both services are stored
This allows me to create a detailed evaluation of the differences between the two services without having to provide the user with potentially worse responses.
For the implementation I have setup a NGINX which mirrors the requests. This is also deployed as a CloudRun service. This now accepts all requests and takes care of the authentication. The original service and the mirrored version have been configured in such a way that they can only be accessed internally and should therefore be accessed via a VPC network.
I have tried all possible combinations for the configuration of these parts but I always get 403 or 502 errors.
I have tried setting the NGINX service to the HTTP and HTTPS routes from the service, and I have tried all the VPC Connector settings. When I set the ingress from the service to ALL it works perfectly if I configure the service with HTTPS and port 443 in NGINX. As soon as I set the ingress to Internal I get errors with HTTPS -> 403 and with HTTP -> 502.
Does anyone have experience in this regard and can give me tips on how to solve this problem? Would be very grateful for any help.
If your Cloud Run service are internally accessible (ingress control set to internal only), you need to perform your request from your VPC.
Therefore, as you perfectly did, you plugged a serverless VPC connector on your NGINX service.
The set up is correct. Now, why it works when you route ALL the egress traffic and not only the private traffic to your VPC connector?
In fact, Cloud Run is a public resource, with a public URL, and even if you set the ingress to internal. This param say "the traffic must come to the VPC" and not say "I'm plugged to the VPC with a private IP".
So, to go to your VPC and access a public ressource (Your cloud run services), you need to route ALL the traffic to your VPC, even the public one.
I followed this guide: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes on how to setup an Nginx Ingress with Cert Manager with Kubernetes having DigitalOcean as a cloud provider.
The tutorial worked fine, I was able to setup everything according to what it was written. Though, (as it is stated) following the tutorial one ends up with three pods of which only one is in "Running 1/1", while the other two are "Down". Also when checking the comments section, it seems that it is quite a problem. Since if all the traffic gets routed to only 1 pods, it is not really scalable. Or am I missing something? Quoting from their tutorial:
Note: By default the Nginx Ingress LoadBalancer Service has
service.spec.externalTrafficPolicy set to the value Local, which
routes all load balancer traffic to nodes running Nginx Ingress Pods.
The other nodes will deliberately fail load balancer health checks so
that Ingress traffic does not get routed to them.
Mainly my question is: Is there a best practice that I am missing in order to have Kubernetes hosting my website? It seems I have to choose either scalability (having all the pods healthy and running) or getting IP of the client visitor.
And for whoever will ever find himself/herself in my situation, this is the reply I got from the DigitaOcean Support:
Unfortunately with that Kubernetes setup it would show those other
nodes as down without additional traffic configuration. It is possible
to skip the nginx ingress part and just use a DigitalOcean load
balancer but this again does require a good deal of setup and can be
more difficult then easy.
The suggestion to have a website with analytics (IP) and scalable was to setup a droplet with Nginx and setup a LoadBalancer to it. More specifically:
As for using a droplet this would be a normal website configuration
with Nginx as your webserver configured to serve content to your app.
You would have full access to your application and the Nginx logs on
the droplet itself. Putting a load balancer in front of this would
require additional configuration as load balancers do not pass the
x-forward header so the IP addresses of clients would not show up in
the logs by default. You would need to configured proxy protocol on
the load balancer and in your nginx configuration to be able to obtain
those IPs.
https://www.digitalocean.com/blog/load-balancers-now-support-proxy-protocol/
This is also a bit more complex unfortunately.
Hope it might save some time to someone
While exploring google cloud platform's Load balancer options
Advanced Menu shows multiple options which are a bit confusing.
there are multiple backends
backend service -> HTTP(S) LB
backend bucket -> HTTP(S) LB
regional backend service -> internal LB
target pools -> TCP LB
Just going through documentations for target pools and backend-service Looks to me they have similar parameters to configure and in the basic menu both are listed as backends.
I understand that target pools are used by TCP forwarding rules where as backend-service used by url map ( http/s Load balancer).
But Are there any other difference between these or is it just names?
A Backend Bucket allow you to use Google Cloud Storage bucket with HTTP(S) load balancing. It can handle request for static content. This option would be useful for a webpage that with static content and it would avoid the costs of resources than a instance would need.
The Backend Service is a centralized service that manages backends, which in turn manage an indeterminate number of instances that handle user requests.
The Target Pools resource defines a group of instances that should receive incoming traffic from forwarding rules. When a forwarding rule directs traffic to a target pool, Google Compute Engine picks an instance from these target pools based on a hash of the source IP and port and the destination IP and port.
This is why they both are listed as backend-services, because at the end they both do the same, but they specify for two different kind of load balancer. The backend service works for HTTP(S) load balancer and target pools are used for forwarding rules.
"A Network load balancer (unlike HTTP(s) load balancer) is a pass-through load balancer. It does not proxy connections from clients." On same note, TargetPools use forwarding rules, backend services use target proxies. Request is sent to instance in target pool "based on a hash of the source IP and port, destination IP and port, and protocol". Backend service has different mechanism to choose an instance group for e.g URL maps.
I've started to use kube-lego to manage load-balancing through a nginx controller, instead of GCP Load Balancer controller.
I am being charged for: Network Load Balancing: Forwarding Rule Minimum Service Charge in EMEA right now, at about 21$ per month.
Isn't the load balancing supposed to be done by nginx?
I have TCP Load Balancer in my GCP network resources.
Isn't possible to send the traffic to a node port and do the load balancing on the nginx controller instead ?
I thought the advantage of having a Nginx load balancer would avoid creating a load balancer on GCP and thus avoiding paying crazy expansive network resources.
What is the purpose of the nginx controller then? Beside maybe automatic certificate renewal with LE.
Docker Swarm mode achieves Inner Load Balancing, and as far as I know, nginx is called hard load balancing, zookeeper is kinda soft load balancing.
So what's the mechanism of the Inner Load Balancing coming along with Docker v1.12?
Does it embed a nginx inside or similar methods like zookeeper?
"Inner" load balancing? Not exactly.
Commit ea4fef2 documents it (docs/swarm/key-concepts.md) as
Swarm uses ingress load balancing to expose the services you want to make available externally to the Swarm.
Swarm can automatically assign the service a PublishedPort or you can configure a PublishedPort for the service in the 30000-32767 range.
External components, such as cloud load balancers, can access the service on the PublishedPort of any node in the cluster, even if the node is not currently running the service.
Swarm has an internal DNS component that automatically assigns each service in the Swarm DNS entry.
Swarm uses internal load balancing to distribute requests among services within the cluster based upon the services' DNS name.
Right now (docker 1.12 August 2016), that inner load balancing does not work consistently: issue 25325
➜ ~ time curl http://10.218.3.5:30000
I'm 272dd0310a95
curl http://10.218.3.5:30000 0.01s user 0.01s system 6% cpu 0.217 total
➜ ~ time curl http://10.218.3.5:30000
curl: (7) Failed to connect to 10.218.3.5 port 30000: Operation timed out
And swarmkit issue 1077 illustrates there is no plan yet to
provide capabilities for session stickiness (cookie-based etc.) in this router mesh.
As awesome as it would be, not all apps are stateless, and we need to route users to the proper container in certain cases
Because:
since we do load balancing at L3/L4 it cannot be bases on things like session cookie.
The best that can be done is to have Source IP based stickiness.
And source IP is not always good enough:
That wouldn't work for our case.
We would have an upstream load balancer (F5) which would make traffic appear to come from a single IP, the "SNAT pool" IP on the F5 since it is a full proxy.
Effectively, Source IP based stickiness would cause all requests to go to one container since all the source IPs would come from the same address.
So the internal load balancer remains quite "basic":
The main issue with adding "session stickyness" is that there are a hundred ways to do it.
It is also an L7 feature, whereas our load balancing operates at L3/4.
There are two high-level paths here:
Monitor events coming from the docker API to modify F5 state to route directly task slots.
Integrate with libnetwork and have the loadbalancer operate as an L7 LB would if it were running directly in the swarm.
The conclusion for now is:
If you want to handle all aspects of load balancing and not use IPVS, you can disable it by running services in DNSRR mode. You can run any load balancer inside of swarm to do load balancing, bypassing the service VIP and populate backends with the DNSRR entries.
That is why the latest release 1.12 has, with PR 827, added support for DNSRR mode and disabling ingress.