NGINX Proxy not working with HTTP/2 and gRPC - nginx

Question

I am trying to use NGINX as an "API Gateway" into my gRPC services - all within a Kubernetes Cluster. A Typescript React App is just making calls via the grpc-web module to an Envoy proxy, then to the API NGINX Proxy. (I have tested that end of the stack - and I'm 100% sure that envoy works fine).
NOTE: I may be making a mistake NOT using TLS with the Envoy Proxy (Which is the 'client' to NGINX) - so please comment if that's the mistake I'm making
For this to work with my gRPC endpoints, I need to enable HTTP/2 proxying (this is required for gRPC to work - it must be over HTTP/2). And so, following the official NGINX Documentation which is here: https://www.nginx.com/blog/nginx-1-13-10-grpc/ , my nginx.conf file looks like:
worker_processes auto;
events {}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent"';
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 1449 ssl http2;
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
ssl_certificate ./server.crt;
ssl_certificate_key ./server.key;
location /com.example.grpcService {
grpc_pass grpcs://api-grpc-server:9090;
proxy_buffer_size 512k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 512k;
grpc_set_header Upgrade $http_upgrade;
grpc_set_header Connection "Upgrade";
grpc_set_header Connection keep-alive;
grpc_set_header Host $host:$server_port;
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
grpc_set_header X-Forwarded-Proto $scheme;
}
}
}
I also heard from another forum that you MUST use TLS/SSL with HTTP/2 or it won't work, so I first tried it without - it didn't work. Then I tried it with the generated SSL certificates and it looks like I'm still getting a 400 error from the proxied service. The log looks like:
172.17.0.17 - - [05/Jan/2021:18:16:23 +0000] "PRI * HTTP/2.0" 400 157 "-" "-"
I have used OpenSSL for the certificates which resulted in .crt and .key files being generated - which I then used for BOTH my Spring Boot gRPC Server & NGINX Proxy. My OpenSSL version is OpenSSL 1.1.1c 28 May 2019.
I am using those same certificates on the actual gRPC Server itself, this looks like:
#Component
public class GrpcServerRunner implements CommandLineRunner, DisposableBean {
private final ConfigurableApplicationContext applicationContext;
private Server server;
public GrpcServerRunner(#Autowired ConfigurableApplicationContext applicationContext) {
this.applicationContext = applicationContext;
}
#Override
public void run(String... args) throws Exception {
File cert = new File("~/etc/ssl/server.crt");
File key = new File("~/etc/ssl/server.key");
BindableService service = applicationContext.getBean("grpcService", BindableService.class);
server = ServerBuilder.forPort(9090).useTransportSecurity(cert, key).addService(service).build();
runSever();
}
private void runSever() {
Thread thread = new Thread(() -> {
try {
server.awaitTermination();
} catch (InterruptedException e) {
e.printStackTrace();
}
});
thread.setDaemon(false);
thread.start();
}
#Override
public void destroy() {
server.shutdown();
}
}
I'd really appreciate any help, questions, feedback or solutions to this problem - so thanks in advance.

Answer 1

It actually had nothing to do with the gRPC Server or the Java Project.
Here's the root NGINX config file:
worker_processes auto;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr [$time_local] [$time_local] [$cookie_X-AUTH-TOKEN] '
'"$scheme $host $request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'($request_time)'
'(($sent_http_set_cookie))';
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# Upstream servers here
upstream api-server-address {
server api-server-address:9090;
keepalive 20;
}
# gRPC Client requirements set
client_max_body_size 0;
proxy_request_buffering off;
server {
listen 1449 http2;
include ./config/grpc-header-config.conf.conf;
# gRPC service proxied here
location /com.yourpackage {
auth_request_set $upstream_http_set_cookie;
auth_request_set $upstream_http_status;
grpc_pass grpc://api-service-address;
include config/grpc-header-config.conf;
}
default_type application/grpc;
}
}
The key file that made this work was this one (this is the one referenced by the root one in config/grpc-header-config.conf):
error_page 400 = #grpc_internal;
error_page 401 = #grpc_unauthenticated;
error_page 403 = #grpc_permission_denied;
error_page 404 = #grpc_unimplemented;
error_page 429 = #grpc_unavailable;
error_page 502 = #grpc_unavailable;
error_page 503 = #grpc_unavailable;
error_page 504 = #grpc_unavailable;
error_page 405 = #grpc_internal;
error_page 408 = #grpc_deadline_exceeded;
error_page 413 = #grpc_resource_exhausted;
error_page 414 = #grpc_resource_exhausted;
error_page 415 = #grpc_internal;
error_page 426 = #grpc_internal;
error_page 495 = #grpc_unauthenticated;
error_page 496 = #grpc_unauthenticated;
error_page 497 = #grpc_internal;
error_page 500 = #grpc_internal;
error_page 501 = #grpc_internal;
location #grpc_deadline_exceeded {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' *';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 4;
add_header 'grpc-message' 'deadline exceeded';
return 204;
}
location #grpc_permission_denied {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 7;
add_header 'grpc-message' 'permission denied';
return 204;
}
location #grpc_resource_exhausted {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 8;
add_header 'grpc-message' 'resource exhausted';
return 204;
}
location #grpc_unimplemented {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 12;
add_header 'grpc-message' unimplemented;
return 204;
}
location #grpc_internal {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 13;
add_header 'grpc-message' 'internal error';
return 204;
}
location #grpc_unavailable {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 14;
add_header 'grpc-message' 'unavailable';
return 204;
}
location #grpc_unauthenticated {
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Set-Cookie' $auth_cookie;
add_header 'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'grpc-status' 16;
add_header 'grpc-message' '401. Unauthorized.';
return 200;
}
I realize this looks super sketchy/hacky, but that's the only way I could do it. Feel free to improve this answer!
Your essentially setting the default protocol to gRPC and HTTP/2, then on any error page you just reset the statuses to match the gRPC conventions + spec so that your client will be able to parse the binaries. If you are using SSL with this, you just need to put the certificates on each side as normal, then change the grpc_pass to grpcs://api-server-address instead of what I have.
Feel free to add any constructive feedback or any questions! Cheers, Ben