Istio Ingress Gateway for gRPC with SIMPLE TLS : Remote Reset Error - grpc

We have been trying to Secure Gateways with SIMPLE TLS for our gRPC Backend which is deployed in Minikube (minikube version: v1.25.2) for now by following this link.
We were able to successfully access the gRPC service (gRPC server with .NET 6) over plaintext through Istio Ingress Gateway using grpcurl client.
But while we tried to use SIMPLE TLS, we have been experiencing -
ERROR:
Code: Unavailable
Message: upstream connect error or disconnect/reset before headers. reset reason: remote reset
Please find the steps -
Created a certificate and a private key for sc-imcps-bootstrap-lb.example.com (Sample Domain for gRPC Server for Minikube)
$ openssl req -out sc-imcps-bootstrap-lb.example.com.csr -newkey rsa:2048 -nodes -keyout sc-imcps-bootstrap-lb.example.com.key -config sc-imcps-bootstrap-lb.cnf
sc-imcps-bootstrap-lb.cnf
[req]
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
O = sc-imcps organization
OU = R&D
CN = sc-imcps-bootstrap-lb.example.com
$ openssl x509 -req -sha256 -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in sc-imcps-bootstrap-lb.example.com.csr -out sc-imcps-bootstrap-lb.example.com.crt -extfile v3.ext
v3.ext:
subjectAltName = #alt_names
[alt_names]
IP.1 = 10.97.36.53
DNS.1 = sc-imcps-bootstrap-lb.example.com
Create kubernetes secrets by following this command -
$ kubectl create -n istio-system secret tls sc-imcps-bootstrap-lb-credential --key=sc-imcps-bootstrap-lb.example.com.key --cert=sc-imcps-bootstrap-lb.example.com.crt
Created Gateway manifest. (kubectl apply -n foo -f gateway.yaml) [gateway.yaml is attached]
Configure the gateway's traffic routes. by creating VirtualService definition [virtualservice.yaml is attached]
Added Host Entry to C:\Windows\System32\drivers\etc\hosts file -
10.97.36.53 sc-imcps-bootstrap-lb.example.com
Client execution from host -
$ grpcurl -v -H Host:sc-imcps-bootstrap-lb.example.com -d '{"AppName": "SC", "AppVersion": 1, "PID": 8132, "ContainerID": "asd-2", "CloudInternal": true}' -cacert example.com.crt -proto imcps.proto sc-imcps-bootstrap-lb.example.com:443 imcps.IMCPS/Init
RESULT:
Resolved method descriptor:
// Sends a greeting
rpc Init ( .imcps.ClientInfo ) returns ( .imcps.InitOutput );
Request metadata to send:
(empty)
Response headers received:
(empty)
Response trailers received:
content-type: application/grpc
date: Tue, 18 Oct 2022 10:32:07 GMT
server: istio-envoy
x-envoy-upstream-service-time: 46
Sent 1 request and received 0 responses
ERROR:
Code: Unavailable
Message: upstream connect error or disconnect/reset before headers. reset reason: remote reset
NOTE:
$ istioctl version
client version: 1.15.0
control plane version: 1.15.0
data plane version: 1.15.0 (5 proxies)
Gateway :
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: sc-imcps-gateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: sc-imcps-bootstrap-lb-credential
hosts:
- sc-imcps-bootstrap-lb.example.com
Virtual Service:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: sc-imcps-bootstrap-route
spec:
hosts:
- sc-imcps-bootstrap-lb.example.com
gateways:
- sc-imcps-gateway
http:
- match:
- uri:
prefix: /imcps.IMCPS/Init
route:
- destination:
host: sc-imcps-bootstrap-svc
port:
number: 17080
Please find the logs from istio-proxy container from gRPC backend server pod -
2022-10-18T10:04:29.412448Z debug envoy http [C190] new stream
2022-10-18T10:04:29.412530Z debug envoy http [C190][S8764333332205046325] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-decorator-operation', 'sc-imcps-bootstrap-svc.foo.svc.cluster.local:17080/imcps.IMCPS/Init*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHAoMSU5TVEFOQ0VfSVBTEgwaCjEwLjg4LjAuNTMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKvwMKBkxBQkVMUxK0AyqxAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1ODVkNjQ1ODU1ChIKB3JlbGVhc2USBxoFaXN0aW8KOQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIWGhRpc3Rpby1pbmdyZXNzZ2F0ZXdheQovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKIgoXc2lkZWNhci5pc3Rpby5pby9pbmplY3QSBxoFZmFsc2UKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCi8KBE5BTUUSJxolaXN0aW8taW5ncmVzc2dhdGV3YXktNTg1ZDY0NTg1NS1icmt4NAobCglOQU1FU1BBQ0USDhoMaXN0aW8tc3lzdGVtCl0KBU9XTkVSElQaUmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9pc3Rpby1zeXN0ZW0vZGVwbG95bWVudHMvaXN0aW8taW5ncmVzc2dhdGV3YXkKFwoRUExBVEZPUk1fTUVUQURBVEESAioACicKDVdPUktMT0FEX05BTUUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXk='
'x-envoy-peer-metadata-id', 'router~10.88.0.53~istio-ingressgateway-585d645855-brkx4.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
2022-10-18T10:04:29.412567Z debug envoy connection [C190] current connecting state: false
2022-10-18T10:04:29.412674Z debug envoy router [C190][S8764333332205046325] cluster 'inbound|17080||' match for URL '/imcps.IMCPS/Init'
2022-10-18T10:04:29.412692Z debug envoy upstream transport socket match, socket default selected for host with address 10.244.120.108:17080
2022-10-18T10:04:29.412696Z debug envoy upstream Created host 10.244.120.108:17080.
2022-10-18T10:04:29.412729Z debug envoy upstream addHost() adding 10.244.120.108:17080
2022-10-18T10:04:29.412784Z debug envoy upstream membership update for TLS cluster inbound|17080|| added 1 removed 0
2022-10-18T10:04:29.412789Z debug envoy upstream re-creating local LB for TLS cluster inbound|17080||
2022-10-18T10:04:29.412742Z debug envoy router [C190][S8764333332205046325] router decoding headers:
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/foo/sa/default;Hash=dda6034f03e05bbb9d0183b80583ee9b5842670599dd86827c8f8b6a74060fa0;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
2022-10-18T10:04:29.412802Z debug envoy upstream membership update for TLS cluster inbound|17080|| added 1 removed 0
2022-10-18T10:04:29.412804Z debug envoy upstream re-creating local LB for TLS cluster inbound|17080||
2022-10-18T10:04:29.412809Z debug envoy pool queueing stream due to no available connections (ready=0 busy=0 connecting=0)
2022-10-18T10:04:29.412813Z debug envoy pool trying to create new connection
2022-10-18T10:04:29.412816Z debug envoy pool creating a new connection (connecting=0)
2022-10-18T10:04:29.412869Z debug envoy http2 [C320] updating connection-level initial window size to 268435456
2022-10-18T10:04:29.412873Z debug envoy connection [C320] current connecting state: true
2022-10-18T10:04:29.412875Z debug envoy client [C320] connecting
2022-10-18T10:04:29.412877Z debug envoy connection [C320] connecting to 10.244.120.108:17080
2022-10-18T10:04:29.412928Z debug envoy connection [C320] connection in progress
2022-10-18T10:04:29.412939Z debug envoy http [C190][S8764333332205046325] request end stream
2022-10-18T10:04:29.412960Z debug envoy upstream membership update for TLS cluster inbound|17080|| added 1 removed 0
2022-10-18T10:04:29.412965Z debug envoy upstream re-creating local LB for TLS cluster inbound|17080||
2022-10-18T10:04:29.412972Z debug envoy connection [C320] connected
2022-10-18T10:04:29.412975Z debug envoy client [C320] connected
2022-10-18T10:04:29.412979Z debug envoy pool [C320] attaching to next stream
2022-10-18T10:04:29.412981Z debug envoy pool [C320] creating stream
2022-10-18T10:04:29.412988Z debug envoy router [C190][S8764333332205046325] pool ready
2022-10-18T10:04:29.517255Z debug envoy http2 [C320] stream 1 closed: 1
2022-10-18T10:04:29.517291Z debug envoy client [C320] request reset
2022-10-18T10:04:29.517301Z debug envoy pool [C320] destroying stream: 0 remaining
2022-10-18T10:04:29.517318Z debug envoy router [C190][S8764333332205046325] upstream reset: reset reason: remote reset, transport failure reason:
2022-10-18T10:04:29.517366Z debug envoy http [C190][S8764333332205046325] Sending local reply with details upstream_reset_before_response_started{remote_reset}
2022-10-18T10:04:29.517607Z debug envoy http [C190][S8764333332205046325] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: remote reset'
'x-envoy-peer-metadata', 'ChwKDkFQUF9DT05UQUlORVJTEgoaCHNjLWltY3BzChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwogCgxJTlNUQU5DRV9JUFMSEBoOMTAuMjQ0LjEyMC4xMDgKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKjgIKBkxBQkVMUxKDAiqAAgoRCgNhcHASChoIc2MtaW1jcHMKMQoYY29udHJvbGxlci1yZXZpc2lvbi1oYXNoEhUaE3NjLWltY3BzLTU5Njg0YzY3ODgKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVpc3RpbwotCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgoaCHNjLWltY3BzCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoyCiJzdGF0ZWZ1bHNldC5rdWJlcm5ldGVzLmlvL3BvZC1uYW1lEgwaCnNjLWltY3BzLTAKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsChQKBE5BTUUSDBoKc2MtaW1jcHMtMAoSCglOQU1FU1BBQ0USBRoDZm9vCkkKBU9XTkVSEkAaPmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9mb28vc3RhdGVmdWxzZXRzL3NjLWltY3BzChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAobCg1XT1JLTE9BRF9OQU1FEgoaCHNjLWltY3Bz'
'x-envoy-peer-metadata-id', 'sidecar~10.244.120.108~sc-imcps-0.foo~foo.svc.cluster.local'
'date', 'Tue, 18 Oct 2022 10:04:29 GMT'
'server', 'istio-envoy'
2022-10-18T10:04:29.517689Z debug envoy http2 [C190] stream 3 closed: 0
2022-10-18T10:04:29.517832Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_response_messages_total , stat=12, recurrent=1
2022-10-18T10:04:29.517843Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_request_messages_total , stat=16, recurrent=1
2022-10-18T10:04:29.520398Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_requests_total , stat=24, recurrent=0
2022-10-18T10:04:29.522737Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_response_bytes , stat=18, recurrent=0
2022-10-18T10:04:29.526875Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_request_duration_milliseconds , stat=22, recurrent=0
2022-10-18T10:04:29.530799Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:664]::report() metricKey cache miss istio_request_bytes , stat=26, recurrent=0
2022-10-18T10:04:29.553171Z debug envoy http [C190] new stream
2022-10-18T10:04:29.553272Z debug envoy http [C190][S417038132095363947] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-decorator-operation', 'sc-imcps-bootstrap-svc.foo.svc.cluster.local:17080/imcps.IMCPS/Init*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHAoMSU5TVEFOQ0VfSVBTEgwaCjEwLjg4LjAuNTMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKvwMKBkxBQkVMUxK0AyqxAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1ODVkNjQ1ODU1ChIKB3JlbGVhc2USBxoFaXN0aW8KOQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIWGhRpc3Rpby1pbmdyZXNzZ2F0ZXdheQovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKIgoXc2lkZWNhci5pc3Rpby5pby9pbmplY3QSBxoFZmFsc2UKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCi8KBE5BTUUSJxolaXN0aW8taW5ncmVzc2dhdGV3YXktNTg1ZDY0NTg1NS1icmt4NAobCglOQU1FU1BBQ0USDhoMaXN0aW8tc3lzdGVtCl0KBU9XTkVSElQaUmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9pc3Rpby1zeXN0ZW0vZGVwbG95bWVudHMvaXN0aW8taW5ncmVzc2dhdGV3YXkKFwoRUExBVEZPUk1fTUVUQURBVEESAioACicKDVdPUktMT0FEX05BTUUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXk='
'x-envoy-peer-metadata-id', 'router~10.88.0.53~istio-ingressgateway-585d645855-brkx4.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '2'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
2022-10-18T10:04:29.553290Z debug envoy connection [C190] current connecting state: false
2022-10-18T10:04:29.553412Z debug envoy router [C190][S417038132095363947] cluster 'inbound|17080||' match for URL '/imcps.IMCPS/Init'
2022-10-18T10:04:29.553445Z debug envoy upstream Using existing host 10.244.120.108:17080.
2022-10-18T10:04:29.553462Z debug envoy router [C190][S417038132095363947] router decoding headers:
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-attempt-count', '2'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/foo/sa/default;Hash=dda6034f03e05bbb9d0183b80583ee9b5842670599dd86827c8f8b6a74060fa0;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
2022-10-18T10:04:29.553473Z debug envoy pool [C320] using existing fully connected connection
2022-10-18T10:04:29.553477Z debug envoy pool [C320] creating stream
2022-10-18T10:04:29.553487Z debug envoy router [C190][S417038132095363947] pool ready
2022-10-18T10:04:29.553519Z debug envoy http [C190][S417038132095363947] request end stream
2022-10-18T10:04:29.554585Z debug envoy http2 [C320] stream 3 closed: 1
2022-10-18T10:04:29.554607Z debug envoy client [C320] request reset
2022-10-18T10:04:29.554616Z debug envoy pool [C320] destroying stream: 0 remaining
2022-10-18T10:04:29.554631Z debug envoy router [C190][S417038132095363947] upstream reset: reset reason: remote reset, transport failure reason:
2022-10-18T10:04:29.554671Z debug envoy http [C190][S417038132095363947] Sending local reply with details upstream_reset_before_response_started{remote_reset}
2022-10-18T10:04:29.554756Z debug envoy http [C190][S417038132095363947] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: remote reset'
'x-envoy-peer-metadata', 'ChwKDkFQUF9DT05UQUlORVJTEgoaCHNjLWltY3BzChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwogCgxJTlNUQU5DRV9JUFMSEBoOMTAuMjQ0LjEyMC4xMDgKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKjgIKBkxBQkVMUxKDAiqAAgoRCgNhcHASChoIc2MtaW1jcHMKMQoYY29udHJvbGxlci1yZXZpc2lvbi1oYXNoEhUaE3NjLWltY3BzLTU5Njg0YzY3ODgKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVpc3RpbwotCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgoaCHNjLWltY3BzCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoyCiJzdGF0ZWZ1bHNldC5rdWJlcm5ldGVzLmlvL3BvZC1uYW1lEgwaCnNjLWltY3BzLTAKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsChQKBE5BTUUSDBoKc2MtaW1jcHMtMAoSCglOQU1FU1BBQ0USBRoDZm9vCkkKBU9XTkVSEkAaPmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9mb28vc3RhdGVmdWxzZXRzL3NjLWltY3BzChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAobCg1XT1JLTE9BRF9OQU1FEgoaCHNjLWltY3Bz'
'x-envoy-peer-metadata-id', 'sidecar~10.244.120.108~sc-imcps-0.foo~foo.svc.cluster.local'
'date', 'Tue, 18 Oct 2022 10:04:29 GMT'
'server', 'istio-envoy'
2022-10-18T10:04:29.554788Z debug envoy http2 [C190] stream 5 closed: 0
2022-10-18T10:04:29.554893Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=12
2022-10-18T10:04:29.554903Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=16
2022-10-18T10:04:29.554905Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=24
2022-10-18T10:04:29.554914Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=18
2022-10-18T10:04:29.554917Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=22
2022-10-18T10:04:29.554919Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=26
2022-10-18T10:04:29.561521Z debug envoy http [C190] new stream
2022-10-18T10:04:29.561614Z debug envoy http [C190][S7465002415732961759] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-decorator-operation', 'sc-imcps-bootstrap-svc.foo.svc.cluster.local:17080/imcps.IMCPS/Init*'
'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKHAoMSU5TVEFOQ0VfSVBTEgwaCjEwLjg4LjAuNTMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKvwMKBkxBQkVMUxK0AyqxAwodCgNhcHASFhoUaXN0aW8taW5ncmVzc2dhdGV3YXkKEwoFY2hhcnQSChoIZ2F0ZXdheXMKFAoIaGVyaXRhZ2USCBoGVGlsbGVyCjYKKWluc3RhbGwub3BlcmF0b3IuaXN0aW8uaW8vb3duaW5nLXJlc291cmNlEgkaB3Vua25vd24KGQoFaXN0aW8SEBoOaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1ODVkNjQ1ODU1ChIKB3JlbGVhc2USBxoFaXN0aW8KOQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIWGhRpc3Rpby1pbmdyZXNzZ2F0ZXdheQovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKIgoXc2lkZWNhci5pc3Rpby5pby9pbmplY3QSBxoFZmFsc2UKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCi8KBE5BTUUSJxolaXN0aW8taW5ncmVzc2dhdGV3YXktNTg1ZDY0NTg1NS1icmt4NAobCglOQU1FU1BBQ0USDhoMaXN0aW8tc3lzdGVtCl0KBU9XTkVSElQaUmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9pc3Rpby1zeXN0ZW0vZGVwbG95bWVudHMvaXN0aW8taW5ncmVzc2dhdGV3YXkKFwoRUExBVEZPUk1fTUVUQURBVEESAioACicKDVdPUktMT0FEX05BTUUSFhoUaXN0aW8taW5ncmVzc2dhdGV3YXk='
'x-envoy-peer-metadata-id', 'router~10.88.0.53~istio-ingressgateway-585d645855-brkx4.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '3'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
2022-10-18T10:04:29.561647Z debug envoy connection [C190] current connecting state: false
2022-10-18T10:04:29.561750Z debug envoy router [C190][S7465002415732961759] cluster 'inbound|17080||' match for URL '/imcps.IMCPS/Init'
2022-10-18T10:04:29.561796Z debug envoy upstream Using existing host 10.244.120.108:17080.
2022-10-18T10:04:29.561825Z debug envoy router [C190][S7465002415732961759] router decoding headers:
':method', 'POST'
':scheme', 'https'
':path', '/imcps.IMCPS/Init'
':authority', 'sc-imcps-bootstrap-lb.example.com:443'
'content-type', 'application/grpc'
'user-agent', 'grpcurl/v1.8.6 grpc-go/1.44.1-dev'
'te', 'trailers'
'x-forwarded-for', '10.88.0.1'
'x-forwarded-proto', 'https'
'x-request-id', '0d9b8e43-da2e-4f99-bbd8-a5c0c56f799f'
'x-envoy-attempt-count', '3'
'x-b3-traceid', '17b50b6247fe2fcbbc2b2057ef4db96d'
'x-b3-spanid', 'bc2b2057ef4db96d'
'x-b3-sampled', '0'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/foo/sa/default;Hash=dda6034f03e05bbb9d0183b80583ee9b5842670599dd86827c8f8b6a74060fa0;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'
2022-10-18T10:04:29.561841Z debug envoy pool [C320] using existing fully connected connection
2022-10-18T10:04:29.561844Z debug envoy pool [C320] creating stream
2022-10-18T10:04:29.561850Z debug envoy router [C190][S7465002415732961759] pool ready
2022-10-18T10:04:29.561877Z debug envoy http [C190][S7465002415732961759] request end stream
2022-10-18T10:04:29.616003Z debug envoy http2 [C320] stream 5 closed: 1
2022-10-18T10:04:29.616037Z debug envoy client [C320] request reset
2022-10-18T10:04:29.616045Z debug envoy pool [C320] destroying stream: 0 remaining
2022-10-18T10:04:29.616057Z debug envoy router [C190][S7465002415732961759] upstream reset: reset reason: remote reset, transport failure reason:
2022-10-18T10:04:29.616083Z debug envoy http [C190][S7465002415732961759] Sending local reply with details upstream_reset_before_response_started{remote_reset}
2022-10-18T10:04:29.616133Z debug envoy http [C190][S7465002415732961759] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: remote reset'
'x-envoy-peer-metadata', 'ChwKDkFQUF9DT05UQUlORVJTEgoaCHNjLWltY3BzChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwogCgxJTlNUQU5DRV9JUFMSEBoOMTAuMjQ0LjEyMC4xMDgKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjE1LjAKjgIKBkxBQkVMUxKDAiqAAgoRCgNhcHASChoIc2MtaW1jcHMKMQoYY29udHJvbGxlci1yZXZpc2lvbi1oYXNoEhUaE3NjLWltY3BzLTU5Njg0YzY3ODgKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVpc3RpbwotCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgoaCHNjLWltY3BzCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoyCiJzdGF0ZWZ1bHNldC5rdWJlcm5ldGVzLmlvL3BvZC1uYW1lEgwaCnNjLWltY3BzLTAKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsChQKBE5BTUUSDBoKc2MtaW1jcHMtMAoSCglOQU1FU1BBQ0USBRoDZm9vCkkKBU9XTkVSEkAaPmt1YmVybmV0ZXM6Ly9hcGlzL2FwcHMvdjEvbmFtZXNwYWNlcy9mb28vc3RhdGVmdWxzZXRzL3NjLWltY3BzChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAobCg1XT1JLTE9BRF9OQU1FEgoaCHNjLWltY3Bz'
'x-envoy-peer-metadata-id', 'sidecar~10.244.120.108~sc-imcps-0.foo~foo.svc.cluster.local'
'date', 'Tue, 18 Oct 2022 10:04:29 GMT'
'server', 'istio-envoy'
2022-10-18T10:04:29.616158Z debug envoy http2 [C190] stream 7 closed: 0
2022-10-18T10:04:29.616256Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=12
2022-10-18T10:04:29.616265Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=16
2022-10-18T10:04:29.616267Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=24
2022-10-18T10:04:29.616270Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=18
2022-10-18T10:04:29.616272Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=22
2022-10-18T10:04:29.616274Z debug envoy wasm wasm log stats_inbound stats_inbound: [extensions/stats/plugin.cc:645]::report() metricKey cache hit , stat=26
2022-10-18T10:04:29.664070Z debug envoy conn_handler [C321] new connection from 192.168.1.13:40686
PS : We have successfully implemented SIMPLE and MUTUAL TLS for REST Services.
Any help will be very much appreciated? I am stuck here! Eventually, after this, we will need to setup mTLS.
Thanks in advance.


We have been using gRPC server with .NET 6. And gRPC kestrel .NET 6 gRPC server is running in k8s under http transport, a minikube load balancer terminates SSL and sends request to the app with :scheme pseudo-header set to "https", but the actual transport is "http" results in this error. Here is the issue. Also find the discussions here thread-1 and thread-2,
For my case, the solution is to add following Kestrel Configuration -
webBuilder.UseKestrel(opts =>
{
opts.AllowAlternateSchemes = true;
});

Related

MailKit.Security.SslHandshakeException: An error occurred while attempting to establish an SSL or TLS connection

I am using MailKit to send emails. While testing the .Net code on my localhost machine it sends successfully, however when I run the code on my Ubuntu machine running on Nginx it shows an SSLHandshakeException. Also on the nginx error.log file it shows the errors:
SSL_do_handshake() failed (SSL: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol)
SSL_do_handshake() failed (SSL: error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low)
The supported protocols on the server are TLSv1.3 TLSv1.2

Istio causing frequent disconnection in grpc stream

I am trying to setup a grpc stream from outside world into istio cluster through istio ingress. I am able to establish the connection, but I am seeing a connection reset every 60sec.
Container logs tell "rpc error: code = Unavailable desc" before breaking
Looked into ingress and envoy logs. nothing much helpful. attached them below.
INGRESS LOGS
[2020-03-06T12:14:10.221Z] "- - -" 0 - "-" "-" 2679 2552 9993 - "-" "-" "-" "-" "10.244.0.93:5448" outbound|5448||grpc-broker.x-infra.svc.cluster.local 10.244.0.116:58094 10.244.0.116:443 10.222.2.9:37864 <xxxxxx DNS NAME xxxxxxxx> -
ENVOY LOGS
[2020-03-06T12:16:28.331Z] "- - -" 0 - "-" "-" 12021 2733 50282 - "-" "-" "-" "-" "127.0.0.1:5448" inbound|5448|tcp-broker|grpc-broker.x-infra.svc.cluster.local 127.0.0.1:56816 10.244.0.93:5448 10.244.0.116:34782 outbound_.5448_._.grpc-broker.x-infra.svc.cluster.local -
Should we add anything extra to grpc stream to work?
Cluster is default mTLS enabled, Source and destination pods are deployment and not stateful set.

From the istio side to make sure that istio is not shutting down the connection.:
This could be prevented by idleTimeout setting in DestinationRule.
According to istio documentation about idleTimeout:
The idle timeout for upstream connection pool connections. The idle timeout is defined as the period in which there are no active requests. If not set, there is no idle timeout. When the idle timeout is reached the connection will be closed. Note that request based timeouts mean that HTTP/2 PINGs will not keep the connection alive. Applies to both HTTP1.1 and HTTP2 connections.
So If You make DestinationRule like this:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: grpc-iddletimeout-policy
spec:
host: grpcservice.servicenamespace.svc.cluster.local
trafficPolicy:
connectionPool:
http:
idleTimeout: 2m
This should close the any HTTP/2 connection from Istio envoy proxy side after being idle for 2 minutes for grpcservice in servicenamespace namespace.
Hope it helps.

JMeter giving response message as "Non HTTP response code: java.net.SocketException/Non HTTP response message: Connection reset"

I am new to JMeter and trying to send HTTP request and an email using JMeter and SMTP.
I am getting JMeter error as Non HTTP response code:
java.net.SocketException/Non HTTP response message: Connection reset
and SMTP error as 500/
Could not connect to SMTP host: smtp.1and1.com, port: 587
I have done the following settings:
user.properties:
httpclient4.retrycount=1
hc.parameters.file=hc.parameters
hc.parameters :
http.connection.stalecheck$Boolean=true
These changes still didn't solve my problem. Can you please help that where should I have to change the setting to fix this.

Check your connection settings, if you are testing from a proxy, you need to specify your proxy server host and port to JMeter.
Run the jmeter[.bat] file from a command line with the following parameters:
-H [proxy server hostname or ip address]
-P [proxy server port]
-N [nonproxy hosts] (e.g. *.apache.org|localhost)
-u [username for proxy authentication - if required]
-a [password for proxy authentication - if required]
Example:
jmeter -H my.proxy.server -P 8000 -u username -a password -N localhost
Check this for more information:
JMeter proxy server

No logs about disconnecting HAProxy(TCP IP)

Problem: in the cat / var/log/haproxy.log redirected log, only the connections are displayed, and the records are duplicated, how to make a more complete log? Configuration /etc/haproxy/haproxy.cfg:
global
daemon
maxconn 8192
log 127.0.0.1 local1
log 127.0.0.1 local1 info
log 127.0.0.1 local1 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats group haproxy mode 660 level admin
defaults
log global
mode tcp
balance leastconn
timeout client 14400s
timeout server 14400s
timeout connect 14400s
retries 3
option dontlognull
listen sborka_30001
log global
mode tcp
option tcplog
# option tcpka
option logasap
# default_backend bck

Symfony2 - GeniusesOfSymfony/WebSocketBundle ssl error connection

I need to configure my project with ssl.
So, i have Mamp Pro, and i have created a new Generated SSL certificate.
My server is running via Https,
Right now, i cant connect to server via Javascripts client :
After 10 sec, this is the javascripts error that ive got :
WebSocket connection to 'wss://127.0.0.1:4000/' failed: Error in connection establishment: net::ERR_TIMED_OUT
(index):111 Disconnected for Connection could not be established. with code 3
and this is my configuration :
gos_web_socket:
pushers:
wamp:
host: 127.0.0.1
port: 4000
server:
port: 4000 #The port the socket server will listen on
host: 127.0.0.1 #The host ip to bind to
router:
resources:
- #AppBundle/Resources/config/pubsub/routing.yml
This is the javascripts code :
var websocket = WS.connect("wss://127.0.0.1:4000");
websocket.on("socket/connect", function(session)
Any help please ?

Resources