I am adding the external authentication using auth-url annotation. How to set conditional request headers for the auth-url api which depends on incoming calls? Can I set the request headers in nginx controller according to incoming calls?
Edited:
Hi,
This is about adding a custom header(Id) which is expected into auth-url. I am setting the Id header which is required in authorize api of auth-url but not receiving in the api. Is this the right method to set? My next question is If it is set how can I set it conditionally depending on from which host server the request is coming through?
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-url: http://ca6dd3adc439.ngrok.io/authorize
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/auth-snippet: |
proxy_set_header Id "queryApps";
spec:
rules:
- host: "hw1.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-netcore-k8s
servicePort: 80
- host: "hw2.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-kubernetes-second
servicePort: 80
My next question is If it is set how can I set it conditionally depending on from which host server the request is coming through?
The best way would be to create two ingress objects with one where the external auth enabled for host hw1.yourdoman. For some reason while testing this the auth-snippet was not passing the header but with it works fine with configuration-snippet:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress-auth-on
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-url: http://ca6dd3adc439.ngrok.io/authorize
nginx.ingress.kubernetes.io/auth-method: POST
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Id "queryApps";
spec:
rules:
- host: "hw1.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-netcore-k8s
servicePort: 80
As you can see here it passes the desired header:
"path": "/",
"headers": {
"host": "hw1.yourdomain",
"x-request-id": "5e91333bed960802a67958d71e787b75",
"x-real-ip": "192.168.49.1",
"x-forwarded-for": "192.168.49.1",
"x-forwarded-host": "hw1.yourdomain",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"id": "queryApps",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
"method": "GET",
"body": "",
"fresh": false,
Moving on, the second ingress object has to be configured the auth disabled for host hw2.yourdomain:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-kubernetes-ingress-auth-off
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: "hw2.yourdomain"
http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: hello-kubernetes-second
servicePort: 80
You can then have a look at the nginx.conf to check how the those two ingresss objects are configure at controller level. This is the first ingress:
## start server hw1.yourdomain
server {
server_name hw1.yourdomain ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
location = /_external-auth-Lw {
internal;
set $proxy_upstream_name "default-hello-netcore-k8s-80";
hello-netcore-k8s.default.svc.cluster.local;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
--------
--------
# Pass the extracted client certificate to the auth provider
set $target http://hello-netcore-k8s.default.svc.cluster.local;
proxy_pass $target;
location / {
set $namespace "default";
set $ingress_name "hello-kubernetes-ingress-auth-on";
set $service_name "hello-netcore-k8s";
set $service_port "80";
set $location_path "/";
set $balancer_ewma_score -1;
set $proxy_upstream_name "default-hello-netcore-k8s-80";
# this location requires authentication
auth_request /_external-auth-Lw;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
--------
proxy_set_header Id "queryApps";
----
And this is the second one:
## start server hw2.yourdomain
server {
server_name hw2.yourdomain ;
listen 80 ;
listen 443 ssl http2 ;
set $proxy_upstream_name "-";
ssl_certificate_by_lua_block {
certificate.call()
}
location / {
set $namespace "default";
set $ingress_name "hello-kubernetes-ingress-auth-off";
set $service_name "hello-kubernetes-second";
set $service_port "80";
set $location_path "/";
Your question is not pretty clear so I assume that it was something related to authentication and header injection. For NGINX ingress, there are a couple ways for you to setup the authentication. The second ways in the following will talk about the header injection.
The first method will be the easiest one. You simply setup the secret and the annotation on the ingress.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: auth-ingress
annotations:
nginx.ingress.kubernetes.io/auth-secret: my-secret
nginx.ingress.kubernetes.io/auth-type: basic
spec:
rules:
- http:
paths:
- path: /auth-url
backend:
service:
name: test
port:
number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: normal-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
service:
name: test
port:
number: 80
The second one will be more complicated but it will be useful if you do your authentication with a particular header. You can inject the snippet of NGINX configuration to the ingress. Of course, if you want to do more manipulation like header adding, you can do it in this way as well.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: auth-ingress
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
if ( $some_condtion ) {
return 403;
}
spec:
rules:
- http:
paths:
- path: /auth-url
backend:
service:
name: test
port:80
I have configured nginx-ingress-controller in kubernetes and I am trying to achieve method based routing from kubernetes.
This is my ingress.yaml file below:
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
kubernetes.io/ingress.class: "nginx"
#nginx.ingress.kubernetes.io/use-regex: "true"
#nginx.ingress.kubernetes.io/app-root: /
#nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
location /tea {
limit_except GET {
deny all;
}
}
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
according to the annotation defined it is supposed to block all other methods like POST/DELETE...etc except GET method. But it is not denying any of the methods. Please help me how can I achieve method based routing using limit_except in nginx.
Thanks in advance.
As you can read in here, the configuration-snippet annotation is used for adding an additional configuration to the NGINX location.
If you want to add custom locations in the server block, you need to use server-snippet annotation. As you can read here:
Using the annotation nginx.ingress.kubernetes.io/server-snippet it is
possible to add custom configuration in the server configuration
block.
The following Ingress manifest should work:
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/server-snippet: |
location /tea {
limit_except GET {
deny all;
}
}
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
I have an application running inside EKS. Istio is used as a ServiceMesh. I am having some problem with https redirect to http and then to https. It looks problem is at istio virtual service, it momentarily switches to http which I want to prevent.
This is how we installed istio [Installed version is 1.5.1]
istioctl -n infrastructure manifest apply \
--set profile=default --set values.kiali.enabled=true \
--set values.gateways.istio-ingressgateway.enabled=true \
--set values.gateways.enabled=true \
--set values.gateways.istio-ingressgateway.type=NodePort \
--set values.global.k8sIngress.enabled=false \
--set values.global.k8sIngress.gatewayName=ingressgateway \
--set values.global.proxy.accessLogFile="/dev/stdout"
This is our virtual service. Cluster contains two deployments:
myapps-front
myapps-api
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: dev-sanojapps-virtual-service
namespace: istio-system
spec:
hosts:
- "dev-mydomain.com"
gateways:
- ingressgateway
http:
- match:
- uri:
prefix: /myapp/
- uri:
prefix: /myapp
rewrite:
uri: /
route:
- destination:
host: myapp-front.sanojapps-dev.svc.cluster.local
headers:
request:
set:
"X-Forwarded-Proto": "https"
"X-Forwarded-Port": "443"
response:
set:
Strict-Transport-Security: max-age=31536000; includeSubDomains
- match:
- uri:
prefix: /v1/myapp-api/
- uri:
prefix: /v1/myapp-api
rewrite:
uri: /
route:
- destination:
host: myapp-api.sanojapps-dev.svc.cluster.local
port:
number: 8080
- match:
- uri:
prefix: /
redirect:
uri: /myapp/
https_redirect: true
headers:
request:
set:
"X-Forwarded-Proto": "https"
"X-Forwarded-Port": "443"
response:
set:
Strict-Transport-Security: max-age=31536000; includeSubDomains
Below is front-end apps yaml deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-front
namespace: sanojapps-dev
labels:
app: myapp-front
spec:
selector:
matchLabels:
app: myapp-front
template:
metadata:
labels:
app: myapp-front
spec:
containers:
- name: myapp-front
image: <ECR_REPO:TAG>
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 50m
memory: 256Mi
Our gateway is configured like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: istio-system
name: sanojapps-ingress
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/subnets: ""
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:region:account:certificate/<ACM_ID>
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-2019-08
alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:region:account:regional/webacl/sanojapps-acl/<ACM_ID>
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: ssl-redirect
servicePort: use-annotation
- path: /*
backend:
serviceName: istio-ingressgateway
servicePort: 80
I followed the official document to install the NGINX ingress and tried to test with the Kubenretes /foo/bar/ example for regular expression.
Example:
kind: Ingress
metadata:
name: test-ingress-3
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: test.com
http:
paths:
- path: /foo/bar/bar
backend:
serviceName: test
servicePort: 80
- path: /foo/bar/[A-Z0-9]{3}
backend:
serviceName: test
servicePort: 80
NGINX config file expected content:
location ~* "^/foo/bar/[A-Z0-9]{3}" {
...
}
location ~* "^/foo/bar/bar" {
...
}
Actual content
location /foo/bar/[A-Z0-9]{3} {
...
}
location /foo/bar/bar {
...
}
Issue:
I am not able to get the same NGINX file content in ingress-controller.
How to resolve this?
I have installed the Nginx Ingress Controller through helm in the ingress namespace.
helm ls --namespace ingress
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nginx-ingress ingress 1 2020-03-15 10:47:51.143159 +0530 IST deployed nginx-ingress-1.34.2 0.30.0
The Service and Deployment is as follows
apiVersion: v1
kind: Service
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
type: LoadBalancer
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
template:
metadata:
labels:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
spec:
containers:
- name: test-service
image: "<acr-url>/test-service:c93c58c0bd4918de06d46381a89b293087262cf9"
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /devops/health/liveness
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /devops/health/readiness
port: 8080
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
env:
- name: test-serviceClientId
valueFrom:
secretKeyRef:
key: test-serviceClientId
name: test-service-133
- name: test-serviceClientSecret
valueFrom:
secretKeyRef:
key: test-serviceClientSecret
name: test-service-133
- name: test-serviceTenantClientId
valueFrom:
secretKeyRef:
key: test-serviceTenantClientId
name: test-service-133
- name: test-serviceTenantClientSecret
valueFrom:
secretKeyRef:
key: test-serviceTenantClientSecret
name: test-service-133
resources:
limits:
cpu: 800m
requests:
cpu: 300m
Configured the ingress on a service with rewrite as follows
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
tls:
- hosts:
- apiexample.centralus.cloudapp.azure.com
secretName: tls-secret
rules:
- host: "apiexample.centralus.cloudapp.azure.com"
http:
paths:
- path: /testservice(/|$)(.*)
backend:
serviceName: test-service
servicePort: 8080
The tls-secret has been generated using
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
Before applying the tls configuration in the ingress, I was able to get response from the api endpoint. The api endpoint is secured with oauth.
API Endpoint:
http://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint
After applying the TLS config on the ingress, and hitting
https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint
I am getting default backend 404.
I have tested the TLS with ingress using another sample service (which is not secured with oauth) and it seems to be working for that service.
Here's the configuration for the other services
apiVersion: apps/v1
kind: Deployment
metadata:
name: tea
spec:
replicas: 3
selector:
matchLabels:
app: tea
template:
metadata:
labels:
app: tea
spec:
containers:
- name: tea
image: nginxdemos/nginx-hello:plain-text
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: tea-svc
labels:
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: tea
The ingress for the service is configured as follows
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
tls:
- hosts:
- apiexample.centralus.cloudapp.azure.com
secretName: tls-secret
rules:
- host: apiexample.centralus.cloudapp.azure.com
http:
paths:
- path: /teaprefix(/|$)(.*)
backend:
serviceName: tea-svc
servicePort: 80
The endpoint
https://apiexample.centralus.cloudapp.azure.com/teaprefix/someurl
works fine.
Please let me know if there's anything missing in my configuration and any potential issues that i may have ignored.
Note: The Service and the Ingress are deployed in the default namespace and the Ingress Controller is running in the ingress namespace
The Nginx Ingress Controller is running in 2 Pods
Logs from Ingress Controller with TLS configuration
Pod1
10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 302 0 "-" "PostmanRuntime/7.23.0" 1495 0.004 [default-test-service-8080] [] 10.244.0.7:8080 0 0.004 302 f4671ede2f95148220c21fe44de6fdad
10.244.0.1 - - [22/Mar/2020:06:57:13 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 404 21 "http://apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1563 0.001 [upstream-default-backend] [] 10.244.0.225:8080 21 0.004 404 ed41b36bc6b89b60bc3f208539a0d44c
Pod2
10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 308 171 "https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1580 0.000 [upstream-default-backend] [] - - - - ce955b7bb5118169e99dd4051060c897
Logs from Ingress Controller without TLS configuration
10.244.0.1 - - [22/Mar/2020:07:04:34 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 200 276 "-" "PostmanRuntime/7.23.0" 1495 2.165 [default-test-service-8080] [] 10.244.0.4:8080 548 2.168 200 e866f277def90c398df4e509e45718b2
UPDATE
Disabling the authentication on the backend service (test-service) also results in the same behavior.
Without applying TLS, able to hit the endpoint using http without any Bearer Token.
After applying TLS, get a default backend - 404 when i hit the endpoint with https/http
UPDATE
Exposing the Service via ClusterIP without the
service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
annotation instead of LoadBalancer also does not seem to be helping. The endpoint works without TLS and with TLS applied, get a default backend - 404
UPDATE
The test-service is a Spring Boot Application with the following WebSecurityConfiguration
#Component
#EnableResourceServer
public class WebSecurityConfiguration extends ResourceServerConfigurerAdapter {
private static final Logger LOGGER = LoggerFactory.getLogger(WebSecurityConfiguration.class);
private final HealthCheckWebSecurity healthCheckWebSecurity = new HealthCheckWebSecurity();
private final Oauth2Settings oauth2Settings;
private final JwtTokenStore jwtTokenStore;
private final TenantService tenantService;
private final TransportGuaranteeWebSecurity transportGuaranteeWebSecurity;
#Autowired
public WebSecurityConfiguration(
Oauth2Settings oauth2Settings,
JwtTokenStore jwtTokenStore,
TenantService tenantService,
TransportGuaranteeWebSecurity transportGuaranteeWebSecurity) {
this.oauth2Settings = oauth2Settings;
this.jwtTokenStore = jwtTokenStore;
this.tenantService = tenantService;
this.transportGuaranteeWebSecurity = transportGuaranteeWebSecurity;
}
#Override
public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
String resourceId = oauth2Settings.getResource("default").getResourceId();
LOGGER.info("Resource service id: {}", resourceId);
resources.resourceId(resourceId).tokenStore(jwtTokenStore);
}
#Override
public void configure(HttpSecurity http) throws Exception {
http.requestMatchers().anyRequest();
http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
http.csrf().disable();
healthCheckWebSecurity.configure(http);
transportGuaranteeWebSecurity.configure(http);
http.authorizeRequests().anyRequest().permitAll();
http.addFilterAfter(buildTenancyContextFilter(), ChannelProcessingFilter.class);
http.addFilterAfter(buildLongUrlFilter(), ChannelProcessingFilter.class);
}
private TenancyContextFilter buildTenancyContextFilter() {
return new TenancyContextFilter(tenantService,
new PathVariableTenantExtractor(Arrays.asList("/{tenantAlias}/api/**")));
}
private LongRequestHttpFilter buildLongUrlFilter() {
return new LongRequestHttpFilter();
}
}
public final class TransportGuaranteeWebSecurity {
private TransportGuaranteeSettings transportGuaranteeSettings;
TransportGuaranteeWebSecurity(TransportGuaranteeSettings transportGuaranteeSettings) {
this.transportGuaranteeSettings = transportGuaranteeSettings;
}
public void configure(HttpSecurity httpSecurity) throws Exception {
if (httpsRequired()) {
httpSecurity.requiresChannel().anyRequest().requiresSecure();
} else {
httpSecurity.requiresChannel().anyRequest().requiresInsecure();
}
}
private boolean httpsRequired() {
final String transportGuarantee = transportGuaranteeSettings.getTransportGuarantee();
return !TransportGuaranteeSettings.TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee);
}
}
#ConfigurationProperties(prefix = "web.security")
public class TransportGuaranteeSettings {
static final String TRANSPORT_GUARANTEE_NONE = "NONE";
static final String TRANSPORT_GUARANTEE_CONFIDENTIAL = "CONFIDENTIAL";
private static final Logger LOGGER = LoggerFactory.getLogger(TransportGuaranteeSettings.class);
private static final String TRANSPORT_GUARANTEE_PROPERTY = "web.security.transportGuarantee";
private String transportGuarantee;
public String getTransportGuarantee() {
return transportGuarantee;
}
public void setTransportGuarantee(String transportGuarantee) {
this.transportGuarantee = transportGuarantee.trim();
logUnexpectedValue();
}
private void logUnexpectedValue() {
if (!TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee)
&& !TRANSPORT_GUARANTEE_CONFIDENTIAL.equalsIgnoreCase(transportGuarantee)) {
LOGGER.debug(
"Unknown value '{}' for property '{}' (expected '{}' or '{}'). Defaulted to '{}'.",
transportGuarantee, TRANSPORT_GUARANTEE_PROPERTY, TRANSPORT_GUARANTEE_NONE, TRANSPORT_GUARANTEE_CONFIDENTIAL,
TRANSPORT_GUARANTEE_CONFIDENTIAL);
}
}
}
In my application.yaml,
web.security.transportGuarantee: NONE
The tenancy context filter extracts the Tenant information from the URL and sets a ThreadLocal. There should not be any issue with that since I am able to hit the endpoint without the TLS configuration. I also do not see any issue with the TransportGuaranteeWebSecurity for the same reason.
Some more logs for Debug
kubectl get pods -owide --namespace ingress
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-5fcbccd545-bdh25 1/1 Running 1 15d 10.244.0.22 aks-agentpool-44086776-vmss000000 <none> <none>
nginx-ingress-controller-5fcbccd545-ptx6j 1/1 Running 0 15d 10.244.0.21 aks-agentpool-44086776-vmss000000 <none> <none>
nginx-ingress-default-backend-554d7bd77c-zxzlf 1/1 Running 0 15d 10.244.0.225 aks-agentpool-44086776-vmss000000 <none> <none>
kubectl get svc
test-service LoadBalancer 10.0.231.35 13.89.111.39 8080:31534/TCP 14d
tea-svc ClusterIP 10.0.12.216 <none> 80/TCP 17d
kubectl get ing
test-service apiexample.centralus.cloudapp.azure.com 10.240.0.4 80, 443 15d
I've reproduced your scenario in my GCP account, and didn't get the same result, so I'm posting my steps to make the troubleshoot of each components in order to make sure all of them is working properly. In resume, seem's the main problem is how the application is handle the paths or host.
Kubernetes: 1.15.3 (GKE)
Nginx Ingress: Installed following the offical docs
Based on your yaml, I removed the readiness and liveness probes, and environment envs to test, and changed the image to a nginx image (on port 80):
apiVersion: v1
kind: Service
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
template:
metadata:
labels:
app.kubernetes.io/name: test-service
app.kubernetes.io/instance: test-service
spec:
containers:
- name: test-service
image: nginx
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
After applied we can check if both (deployment and service) are running as expected, before apply ingress spec.
To check this, we can use a curl image to curl destination or dnsutil container by oficial kubernetes docs.
In this case I've used curlimages/curl do test:
apiVersion: v1
kind: Pod
metadata:
name: curl
namespace: default
spec:
containers:
- name: curl
image: curlimages/curl
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always
With curl container running we can check first if the container of our nginx image is correctly running and answers the requests 'curling' directly their IP.
The command below will create a variable named $pod from the pod with label app.kubernetes.io/name=test-service with the ip of the pod.
$ pod=$(kubectl get pods -ojsonpath='{.items[*].status.podIP}' -l app.kubernetes.io/name=test-service)
$ echo $pod
192.168.109.12
Using curl pod create earlier, we can check if the pod is processing the requests:
$ kubectl exec curl -- curl -Is $pod
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:08:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes
See the response HTTP/1.1 200 OK, let's move forward to test the service:
$ kubectl exec curl -- curl -Is test-service
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:11:13 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes
Same here, HTTP/1.1 200 OK for service.
Let's go further and deploy the nginx ingress without TLS yet, to make a test before and after:
Generating and applying the certificate:
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"
...
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created
Ingress without TLS (I've changed to port 80 to match with my nginx image):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: "apiexample.centralus.cloudapp.azure.com"
http:
paths:
- path: /testservice(/|$)(.*)
backend:
serviceName: test-service
servicePort: 80
Testing from my desktop to internet using the IP (omitted) provided by GCP:
$ curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" http://34.77.xxx.xx/testservice
HTTP/1.1 200 OK
Server: nginx/1.17.8
Date: Tue, 24 Mar 2020 10:41:21 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes
Until here everything is working fine. We can now add the TLS to ingress spec and try again:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-service
labels:
app.kubernetes.io/name: test-service
helm.sh/chart: test-service-0.1.0
app.kubernetes.io/instance: test-service
app.kubernetes.io/managed-by: Helm
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
tls:
- hosts:
- apiexample.centralus.cloudapp.azure.com
secretName: tls-secret
rules:
- host: "apiexample.centralus.cloudapp.azure.com"
http:
paths:
- path: /testservice(/|$)(.*)
backend:
serviceName: test-service
servicePort: 80
Testing using curl:
curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" https://34.77.147.74/testservice -k
HTTP/2 200
server: nginx/1.17.8
date: Tue, 24 Mar 2020 10:45:25 GMT
content-type: text/html
content-length: 612
vary: Accept-Encoding
last-modified: Tue, 03 Mar 2020 14:32:47 GMT
etag: "5e5e6a8f-264"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains
Ok, so that is working with TLS, so based on this we can conclude that your yaml spec is working, and maybe your are facing some issue with path in ingress definitions and your application:
You are using the annotation nginx.ingress.kubernetes.io/rewrite-target: /$2 and in the path /testservice(/|$)(.*)
It means that, any characters captured by (.*) will be assigned to the placeholder $2, which is then used as a parameter in the rewrite-target annotation.
Based on your ingress path regex:
apiexample.centralus.cloudapp.azure.com/testservice rewrites to apiexample.centralus.cloudapp.azure.com/
apiexample.centralus.cloudapp.azure.com/testservice/ rewrites to apiexample.centralus.cloudapp.azure.com/
apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint rewrites to apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint
When checked in nginx pod logs, you could see the requested url after rewrite:
2020/03/24 10:59:33 [error] 7#7: *186 open() "/usr/share/nginx/html/tenant/api/v1/endpoint" failed (2: No such file or directory), client: 10.20.1.61, server: localhost, request: "HEAD /tenant/api/v1/endpoint HTTP/1.1", host: "apiexample.centralus.cloudapp.azure.com"
So, by this test I could concluded that your deployment, service and ingress is working and doesn't has any typo or formatting problems. So my advice is to double check the application
Make sure your application is handle correctly the path;
If your applications is doing some URL validation, make sure it can handle http and https;
In case you have CORS enabled, adjust the ingress as mentioned here.
Since you didn't post any app as example to reproduce, my tests was limited a generic app as backend. If you could provide more details about backend application, or some generic app which produces the same behavior please let me know and will happy to improve my answer with more details.
References:
https://kubernetes.github.io/ingress-nginx/deploy/
https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/
https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/