I am trying to use filebeat with nginx module to collect logs from nginx-ingress-controller and send directly to elasti but I keep getting an error:
Provided Grok expressions do not match field value: [172.17.0.1 - - [03/Dec/2022:00:05:01 +0000] \"GET /healthz HTTP/1.1\" 200 0 \"-\" \"kube-probe/1.24\" \"-\"]
This appear on Kibana under message error.
Note that I am running the latest helm for filebeat (8.5) and the nginx controller is nginx-ingress-controller-9.2.15 1.2.1.
My filebeat setting:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: false
templates:
- condition:
contains:
kubernetes.pod.name: redis
config:
- module: redis
log:
input:
type: container
containers.ids:
- "${data.kubernetes.container.id}"
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
- condition:
contains:
kubernetes.pod.name: nginx
config:
- module: nginx
access:
enabled: true
input:
type: container
containers.ids:
- "${data.kubernetes.container.id}"
paths:
- /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '["https://${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}"]'
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]
setup.ilm:
enabled: true
overwrite: true
policy_file: /usr/share/filebeat/ilm.json
setup.dashboards.enabled: true
setup.kibana.host: "http://kibana:5601"
ilm.json: |
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "1d"
}
}
},
"delete": {
"min_age": "7d",
"actions": {
"delete": {}
}
}
}
}
}
And the logs from the controller are:
172.17.0.1 - - [02/Dec/2022:23:43:49 +0000] "GET /healthz HTTP/1.1" 200 0 "-" "kube-probe/1.24" "-"
Can someone help me understand what am I doing wrong?
I'm using nginx controller with Minikube. I can access rabbitmq management but when i access the queues i got this error:
Not found
The object you clicked on was not found; it may have been deleted on the server.
if i use port-forward it's working correctly
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rabbitmq-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2
labels:
name: rabbitmq-ingress
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /rabbit(/|$)(.*)
backend:
service:
name: rabbitmq-management
port:
number: 15672
Looks like you forget to mention host into ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-wildcard-host
spec:
rules:
- host: "foo.bar.com"
http:
paths:
- pathType: Prefix
path: "/bar"
backend:
service:
name: service1
port:
number: 80
host: "foo.bar.com"
Read more : https://kubernetes.io/docs/concepts/services-networking/ingress/#hostname-wildcards
I am adding the external authentication using auth-url annotation. How to set conditional request headers for the auth-url api which depends on incoming calls? Can I set the request headers in nginx controller according to incoming calls?
Edited:
Hi,
This is about adding a custom header(Id) which is expected into auth-url. I am setting the Id header which is required in authorize api of auth-url but not receiving in the api. Is this the right method to set? My next question is If it is set how can I set it conditionally depending on from which host server the request is coming through?
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-url: http://ca6dd3adc439.ngrok.io/authorize
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-snippet: |
proxy_set_header Id "queryApps";
spec:
rules:
- host: "hw1.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-netcore-k8s
servicePort: 80
- host: "hw2.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-kubernetes-second
servicePort: 80
My next question is If it is set how can I set it conditionally depending on from which host server the request is coming through?
The best way would be to create two ingress objects with one where the external auth enabled for host hw1.yourdoman. For some reason while testing this the auth-snippet was not passing the header but with it works fine with configuration-snippet:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress-auth-on
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-url: http://ca6dd3adc439.ngrok.io/authorize
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Id "queryApps";
spec:
rules:
- host: "hw1.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-netcore-k8s
servicePort: 80
As you can see here it passes the desired header:
"path": "/",
"headers": {
"host": "hw1.yourdomain",
"x-request-id": "5e91333bed960802a67958d71e787b75",
"x-real-ip": "192.168.49.1",
"x-forwarded-for": "192.168.49.1",
"x-forwarded-host": "hw1.yourdomain",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"id": "queryApps",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
"method": "GET",
"body": "",
"fresh": false,
Moving on, the second ingress object has to be configured the auth disabled for host hw2.yourdomain:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress-auth-off
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: "hw2.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-kubernetes-second
servicePort: 80
You can then have a look at the nginx.conf to check how the those two ingresss objects are configure at controller level. This is the first ingress:
## start server hw1.yourdomain
server {
server_name hw1.yourdomain ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
location = /_external-auth-Lw {
internal;
set $proxy_upstream_name "default-hello-netcore-k8s-80";
hello-netcore-k8s.default.svc.cluster.local;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
--------
--------
# Pass the extracted client certificate to the auth provider
set $target http://hello-netcore-k8s.default.svc.cluster.local;
proxy_pass $target;
location / {
set $namespace "default";
set $ingress_name "hello-kubernetes-ingress-auth-on";
set $service_name "hello-netcore-k8s";
set $service_port "80";
set $location_path "/";
set $balancer_ewma_score -1;
set $proxy_upstream_name "default-hello-netcore-k8s-80";
# this location requires authentication
auth_request /_external-auth-Lw;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
--------
proxy_set_header Id "queryApps";
----
And this is the second one:
## start server hw2.yourdomain
server {
server_name hw2.yourdomain ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location / {
set $namespace "default";
set $ingress_name "hello-kubernetes-ingress-auth-off";
set $service_name "hello-kubernetes-second";
set $service_port "80";
set $location_path "/";
Your question is not pretty clear so I assume that it was something related to authentication and header injection. For NGINX ingress, there are a couple ways for you to setup the authentication. The second ways in the following will talk about the header injection.
The first method will be the easiest one. You simply setup the secret and the annotation on the ingress.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: auth-ingress
annotations:
nginx.ingress.kubernetes.io/auth-secret: my-secret
nginx.ingress.kubernetes.io/auth-type: basic
spec:
rules:
- http:
paths:
- path: /auth-url
backend:
service:
name: test
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: normal-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: test
port:
number: 80
The second one will be more complicated but it will be useful if you do your authentication with a particular header. You can inject the snippet of NGINX configuration to the ingress. Of course, if you want to do more manipulation like header adding, you can do it in this way as well.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: auth-ingress
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
if ( $some_condtion ) {
return 403;
}
spec:
rules:
- http:
paths:
- path: /auth-url
backend:
service:
name: test
port:80
I have configured nginx-ingress-controller in kubernetes and I am trying to achieve method based routing from kubernetes.
This is my ingress.yaml file below:
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
kubernetes.io/ingress.class: "nginx"
#nginx.ingress.kubernetes.io/use-regex: "true"
#nginx.ingress.kubernetes.io/app-root: /
#nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
location /tea {
limit_except GET {
deny all;
}
}
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
according to the annotation defined it is supposed to block all other methods like POST/DELETE...etc except GET method. But it is not denying any of the methods. Please help me how can I achieve method based routing using limit_except in nginx.
Thanks in advance.
As you can read in here, the configuration-snippet annotation is used for adding an additional configuration to the NGINX location.
If you want to add custom locations in the server block, you need to use server-snippet annotation. As you can read here:
Using the annotation nginx.ingress.kubernetes.io/server-snippet it is
possible to add custom configuration in the server configuration
block.
The following Ingress manifest should work:
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/server-snippet: |
location /tea {
limit_except GET {
deny all;
}
}
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
I followed the official document to install the NGINX ingress and tried to test with the Kubenretes /foo/bar/ example for regular expression.
Example:
kind: Ingress
metadata:
name: test-ingress-3
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: test.com
http:
paths:
- path: /foo/bar/bar
backend:
serviceName: test
servicePort: 80
- path: /foo/bar/[A-Z0-9]{3}
backend:
serviceName: test
servicePort: 80
NGINX config file expected content:
location ~* "^/foo/bar/[A-Z0-9]{3}" {
...
}
location ~* "^/foo/bar/bar" {
...
}
Actual content
location /foo/bar/[A-Z0-9]{3} {
...
}
location /foo/bar/bar {
...
}
Issue:
I am not able to get the same NGINX file content in ingress-controller.
How to resolve this?