load balance UDP syslog with custom health check - nginx

Question

I setup Nginx Plus to load-balance UDP syslog traffic. Here's a snippet from nginx.conf:
stream {
upstream syslog_standard {
zone syslog_zone 64k;
server cp01.woolford.io:1514 max_fails=1 fail_timeout=10s;
server cp02.woolford.io:1514 max_fails=1 fail_timeout=10s;
server cp03.woolford.io:1514 max_fails=1 fail_timeout=10s;
}
server {
listen 514 udp;
proxy_pass syslog_standard;
proxy_bind $remote_addr transparent;
health_check udp;
}
}
I was a little surprised to hear that NGINX Plus could perform health checks on UDP since UDP is, by design, unreliable. Since there is no acknowledgment in UDP, the messages effectively go into a black hole.
I'm trying to set up a somewhat fault-tolerant and scalable syslog ingestion pipeline. The loss of a node should be detected, by a health check, and be temporarily removed from the list of available servers.
This didn't work, despite the UDP health check. I think the UDP health check only works for services that respond to the caller (e.g. DNS). Since syslog doesn't respond, there's no way to check for errors, e.g. using match.
The process that's ingesting the syslog messages listens on port 1514 and has a REST interface on port 8073:
If the ingest process is healthy a GET request to /connectors/syslog/status on port 8073 returns:
{
"name": "syslog",
"connector": {
"state": "RUNNING",
"worker_id": "10.0.1.41:8073"
},
"tasks": [
{
"id": 0,
"state": "RUNNING",
"worker_id": "10.0.1.41:8073"
}
],
"type": "source"
}
I'd like to create a custom check to see that ingest is running. Is that possible with NGINX Plus? Can we check the health on a completely different port?

Answer 1

This is what I did:
stream {
upstream syslog_standard {
zone syslog_zone 64k;
server cp01.woolford.io:1514 max_fails=1 fail_timeout=10s;
server cp02.woolford.io:1514 max_fails=1 fail_timeout=10s;
server cp03.woolford.io:1514 max_fails=1 fail_timeout=10s;
}
match syslog_ingest_test {
send "GET /connectors/syslog/status HTTP/1.0\r\nHost: localhost\r\n\r\n";
expect ~* "RUNNING";
}
server {
listen 514 udp;
proxy_pass syslog_standard;
proxy_bind $remote_addr transparent;
health_check match=syslog_ingest_test port=8073;
}
}
The match=syslog_ingest_test health check performs a GET request to the URL at port 8073 (i.e. the port that contains the healthcheck endpoint of the ingest process) and confirms that it's running.
I can toggle the service off/on and NGINX detects it and reacts accordingly.