I am learning kubernetes and got into a point where I am very confused. I have installed metallb and ingress-nginx so it is possible to access content from outside. I saw several examples that are using nginx image running in pod despite they are also using ingress-nginx.
Isn't ingress-nginx capable to do all work as nginx image? Otherwise what roles those two play?
I need to deploy express server where I would like to utilize some nginx features like gzip so it is where reverse proxy comes.
So do I need to make it work on ingress-nginx level or from nginx image? And if I need nginx image does it mean that I need to run separately nginx image and my built node image with my express app?
Short answer: No
But it's complicated.
nginx image you mentioned is one of the most popular images (5th on the Docker Hub, by the time of this writing), is relatively small (133MB), and is easy to remember. That's why it is widely used as an example in many tutorials.
Isn't ingress-nginx capable to do all work as nginx image?
To some extent.
Pod and Ingress are different Kubernetes resources, and they act differently. nginx image is usually deployed as a container inside a pod.
In case of nginx ingress controller, similiar image is used for both Pod and Ingress (mentioned below).
Whenever you deploy (for example) a rewrite rule in ingress controller
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
corresponding entry is being created in nginx.conf file in nginx-ingress image (again, mentioned below).
You can read more here.
That being said, there is nginx/nginx-ingress image that is being deployed when you create Ingress Controller resource.
You can even pull it from Docker Hub
$ docker image pull nginx; docker image pull nginx/nginx-ingress
Using default tag: latest
latest: Pulling from library/nginx
69692152171a: Pull complete
30afc0b18f67: Pull complete
596b1d696923: Pull complete
febe5bd23e98: Pull complete
8283eee92e2f: Pull complete
351ad75a6cfa: Pull complete
Digest: sha256:6d75c99af15565a301e48297fa2d121e15d80ad526f8369c526324f0f7ccb750
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
Using default tag: latest
latest: Pulling from nginx/nginx-ingress
69692152171a: Already exists
30afc0b18f67: Already exists
596b1d696923: Already exists
febe5bd23e98: Already exists
8283eee92e2f: Already exists
351ad75a6cfa: Already exists
ff0027f23312: Pull complete
18361787d4a4: Pull complete
e1aba8353cbb: Pull complete
f9a4438768c4: Pull complete
Digest: sha256:a57fc7835c14b7294180f5c11ae6b81f2745189ef2adb57a5188952bf8aaa17a
Status: Downloaded newer image for nginx/nginx-ingress:latest
docker.io/nginx/nginx-ingress:latest
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx/nginx-ingress latest 1bc8d3e90493 2 weeks ago 177MB
nginx latest d1a364dc548d 2 weeks ago 133MB
As you can see nginx and nginx-ingress are two different images.
In fact, you can even build the image yourself.
Related
I have a Kubernetes cluster v1.22.1 set up in bare metal CentOS. I am facing a problem when setting up Nginx Ingress controller following this link.
I followed exactly the same in step 1-3 but got a CrashLoopBackOff error in nginx ingress controller pod. I checked the logs of the pod and found below:
[root#dev1 deployments]# kubectl logs -n nginx-ingress nginx-ingress-5cd5c7549d-hw6l7
I0910 23:15:20.729196 1 main.go:271] Starting NGINX Ingress controller Version=1.12.1 GitCommit=6f72db6030daa9afd567fd7faf9d5fffac9c7c8f Date=2021-09-08T13:39:53Z PlusFlag=false
W0910 23:15:20.770569 1 main.go:310] The '-use-ingress-class-only' flag will be deprecated and has no effect on versions of kubernetes >= 1.18.0. Processing ONLY resources that have the 'ingressClassName' field in Ingress equal to the class.
F0910 23:15:20.774788 1 main.go:314] Error when getting IngressClass nginx: the server could not find the requested resource
I believe I have the IngressClass setup properly as shown in below:
[root#dev1 deployments]# kubectl get IngressClass
NAME CONTROLLER PARAMETERS AGE
nginx nginx.org/ingress-controller <none> 2m12s
So I have no idea why it said Error when getting IngressClass nginx. Can anyone shed me some lights please?
Reproduction and what happens
I created a one node cluster using kubeadm on CentOS 7. And got the same error.
You and I were able to proceed further only because we missed this command at the beginning:
git checkout v1.12.1
The main difference is ingress-class.yaml has networking.k8s.io/v1beta1 in v1.12.1 and networking.k8s.io/v1 in master branch.
After I went here for the second time and switched the branch, I immediately saw this error:
$ kubectl apply -f common/ingress-class.yaml
error: unable to recognize "common/ingress-class.yaml": no matches for kind "IngressClass" in version "networking.k8s.io/v1beta1"
That looks like other resources are not updated to be used on kubernetes v1.22+ yet.
Please see deprecated migration guide - v1.22 - ingress
How to proceed further
I tested exactly the same approach on a cluster with v1.21.4 and it worked like a charm. So you may consider downgrading the cluster.
If you're not tight to using NGINX ingress controller (supported by Nginx inc, you can try ingress nginx which is developed by kubernetes community. I tested it on v1.22, it works fine. Please find
Installation on bare metal cluster.
P.S. It may be confusing, but there are two free nginx ingress controllers which are developed by different teams. Also there's a third option - NGINX Plus which is paid and has more option. Please see here the difference
I'm new to Kubernetes and wanted to use the NGINX Ingress Controller for the project I'm currently working on. I read some of the docs and watched some tutorials but I haven't really understood the:
installation process (should I use Helm, the git repo???)
how to properly configure the Ingress. For example, the Kubernetes docs say to use a nginx.conf file (https://kubernetes.io/docs/tasks/access-application-cluster/connecting-frontend-backend/#creating-the-frontend) which is never mentioned in the actual NGINX docs. They say to use ConfigMaps or annotations
Does anybody know of a blog post or tutorial that makes these things clear. Out of everything I've learned so far (both frontend and backend) developing and deploying to a cloud environment has got me lost. I've been stuck on a problem for a week and want to figure out it Ingress can help me.
Thanks!
Answering:
How should I install nginx-ingress
There is no one correct way to install nginx-ingress. Each way has its own advantages/disadvantages, each Kubernetes cluster could require different treatment (for example: cloud managed Kubernetes and minikube) and you will need to determine which option is best suited for you.
You can choose from running:
$ kubectl apply -f ...,
$ helm install ...,
terraform apply ... (helm provider),
etc.
How should I properly configure Ingress?
Citing the official documentation:
An API object that manages external access to the services in a cluster, typically HTTP.
-- Kubernetes.io: Docs: Concepts: Services networking: Ingress
Basically Ingress is a resource that tells your Ingress controller how it should handle specific HTTP/HTTPS traffic.
Speaking specifically about the nginx-ingress, it's entrypoint that your HTTP/HTTPS traffic should be sent to is a Service of type LoadBalancer named: ingress-nginx-controller (in a ingress-nginx namespace). In Docker with Kubernetes implementation it will bind to the localhost of your machine.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
spec:
ingressClassName: "nginx"
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
The modified example from the documentation will tell your Ingress controller to pass the traffic with any Host and with path: / (every path) to a service named nginx on port 80.
The above configuration after applying will be reflected by ingress-nginx in the /etc/nginx/nginx.conf file.
A side note!
Take a look on how the part of nginx.conf looks like when you apply above definition:
location / {
set $namespace "default";
set $ingress_name "minimal-ingress";
set $service_name "nginx";
set $service_port "80";
set $location_path "/";
set $global_rate_limit_exceeding n;
On how your specific Ingress manifest should look like you'll need to consult the documentation of the software that you are trying to send your traffic to and ingress-nginx docs.
Addressing the part:
how to properly configure the Ingress. For example, the Kubernetes docs say to use a nginx.conf file (https://kubernetes.io/docs/tasks/access-application-cluster/connecting-frontend-backend/#creating-the-frontend) which is never mentioned in the actual NGINX docs. They say to use ConfigMaps or annotations.
You typically don't modify nginx.conf that the Ingress controller is using by yourself. You write an Ingress manifest and rest is taken by Ingress controller and Kubernetes. nginx.conf in the Pod responsible for routing (your Ingress controller) will reflect your Ingress manifests.
Configmaps and Annotations can be used to modify/alter the configuration of your Ingress controller. With the Configmap you can say to enable gzip2 compression and with annotation you can say to use a specific rewrite.
To make things clearer. The guide that is referenced here is a frontend Pod with nginx installed that passes the request to a backend. This example apart from using nginx and forwarding traffic is not connected with the actual Ingress. It will not acknowledge the Ingress resource and will not act accordingly to the manifest you've passed.
A side note!
Your traffic would be directed in a following manner (simplified):
Ingress controller -> frontend -> backend
This example speaking from personal perspective is more of a guide how to connect frontend and backend and not about Ingress.
Additional resources:
Stackoverflow.com: Questions: Ingress nginx how sto serve assets to application
Stackoverflow.com: Questions: How nginx ingress controller backend protocol annotation works
The guide that I wrote some time ago should help you with the idea on how you can configure basic Ingress (it could be little outdated):
Stackoverflow.com: Questions: How can I access nginx ingress on my local
The most straightforward process of installing nginx ingress controller (or any other for that matter) would be using helm. This would need basic understanding of helm and how to work with helm charts.
Here's the repo: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
Follow the instructions in there - quite straightforward if you use the default values. For the configuration, you can customize the chart too before installing. Look at the Readme to see how to get all the configurable options.
Hope this helps as a starting point.
I want to deploy a .NET Core app with NGINX reverse proxy on Azure Kubernetes Service. What is the best way to organize the pods and containers?
Two single-container pods, one pod for nginx and one pod for the app (.net-core/kestrel), so each one can scale independently of the other
One multi-container pod, this single pod with two containers (one for nginx and one for the app)
One single-container pod, a single container running both the nginx and the .net app
I would choose the 1st option, but I don't know if it is the right choice, would be great to know the the pros and cons of each option.
If I choose the 1st option, is it best to set affinity to put nginx pod in the same node that the app pod? Or anti-affinity so they deploy on different nodes? Or no affinity/anti-affinity at all?
The best practice for inbound traffic in Kubernetes is to use the Ingress resource. This requires a bit of extra setup in AKS because there's no built-in ingress controller. You definitely don't want to do #2 because it's not flexible, and #3 is not possible to my knowledge.
The Kubernetes Ingress resource is a configuration file that manages reverse proxy rules for inbound cluster traffic. This allows you to surface multiple services as if they were a combined API.
To set up ingress, start by creating a public IP address in your auto-generated MC resource group:
az network public-ip create `
-g MC_rg-name_cluster-name_centralus `
-n cluster-name-ingress-ip `
-l centralus `
--allocation-method static `
--dns-name cluster-name-ingress
Now create an ingress controller. This is required to actually handle the inbound traffic from your public IP. It sits and listens to the Kubernetes API Ingress updates, and auto-generates an nginx.conf file.
# Note: you'll have to install Helm and its service account prior to running this. See my GitHub link below for more information
helm install stable/nginx-ingress `
--name nginx-ingress `
--namespace default `
--set controller.service.loadBalancerIP=ip.from.above.result `
--set controller.scope.enabled=true `
--set controller.scope.namespace="default" `
--set controller.replicaCount=3
kubectl get service nginx-ingress-controller -n default -w
Once that's provisioned, make sure to use this annotation on your Ingress resource: kubernetes.io/ingress.class: nginx
If you'd like more information on how to set this up, please see this GitHub readme I put together this week. I've also included TLS termination with cert-manager, also installed with Helm.
I try to setup nginx ingress (nodeport) on google container with proxy protocol so that the real ip can be forwarded to backend service, but ended up with broken header.
2017/02/05 13:48:52 [error] 18#18: *2 broken header: "�����~��]H�k��m[|����I��iv.�{y��Z �嵦v�Ȭq���2Iu4P�z;� o$�s����"���+�/�,�0̨̩����/" while reading PROXY protocol, client: 10.50.0.1, server: 0.0.0.0:443
If without the proxy protocol, thing works well. According to the https://blog.mythic-beasts.com/2016/05/09/proxy-protocol-nginx-broken-header/ this is due to the protocol v2 is used (binary), but nginx only can speak v1. Any suggestion?
GKE: With kubernetes v1.6+ source ip is preserved by default and can be found in headers under x-real-ip without setting any extra nginx config.
AWS: Source ip can be preserved by adding this to the annotations
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
namespace: nginx-ingress
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
labels:
app: nginx-ingress
Checkout this link
https://github.com/kubernetes/ingress/tree/master/examples/aws/nginx
Just ran into this problem myself. For me, I wasn't behind a load balancer (other than my nginx ingress), so I did not actually need proxy-protocol set.
However, I was getting 127.0.0.1 as the client ip still. The trick is that there was a bug in the version of the nginx ingress I was using (0.9.0-beta.5). Updating my container image to gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8 fixed the issue and I received the proper X-Forwarded-For header.
Note that the higher versions (up to beta.11 at the time of writing this) had the issue remaining, so I've stayed on beta.8 for the time being.
You can see the versions available at https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/nginx-ingress-controller.
If you are wanting to look at the configuration options available, check out https://github.com/kubernetes/ingress/tree/master/controllers/nginx.
I had this problem myself and this was the thing that finally made it work. Updating to version beta.8 of the nginx controller.
In case some people using AWS want to learn from my mistakes, don't go through manual configuration of the load balancer through the aws cli. The above mentioned service annotation does it all for you. I could have saved myself a lot of headache if I had realized that.
Description:
I've set up the nginx-proxy container which works really great with one of my two docker containers. Which is just a mini go web server on dev.MY_IP_ADDRESS.com.
I've set it up for my gitlab docker container as well which runs on MY_IP_ADDRESS.com:10080 but doesn't seem to work with gitlab.MY_IP_ADDRESS.com
I've done the same configurations as with my web server, by setting by adding an environment variable:
gitlab:
#other configs here
environment:
- VIRTUAL_HOST=gitlab.MY_IP_ADDERSS.com
#more configs here
The only difference is that I set up my go server and nginx-proxy server in the same docker-compose.yml and the gitlab one uses a different docker-compose.yml file. Unsure if this has anything to do with it.
I've attempted to docker-compose up each file in a different orders to see if that was an issue.
Error:
This is what I get when I go on gitlab.MY_IP_ADDRESS.com:
503 Service Temporarily Unavailable
nginx/1.11.8
Question:
Why isn't the reverse proxy for gitlab.MY_IP_ADDERSS.com working for gitlab? Is there a conflict somewhere? It works fine on MY_IP_ADDRESS.com:10080
If any logs are needed or any more information let me know. Thanks.
I completely forgot about this question, I actually found a solution which worked for me:
The problem is that your docker-gen is not able to find your GitLab and therefore does not generate the Nginx configuration for gitlab.MY_IP_ADDERSS.com.
To solve this you have three options:
1.) If you are using the solution with separate containers and launch the docker-gen container with the -only-exposed flag this might prevent it from finding GitLab. This was the issue in my case which is why I am mentioning it.
2.) In your case it will probably be because your GitLab container and your Nginx container do not share a common Docker network. Create one like docker create network nginx-proxy and add all your containers to it.
3.) Another solution proposed in this issue is to add a line network_mode: bridge to your GitLab container. I did not test this myself.