We are currently having a requirement of passing the data to topics from 3rd party API. First, we have tried with HTTP bridge but looks like it is not supporting the HTTPS protocol. Now, we got some suggestions for having or creating the Kafka connector to solve this. Now, we got confused about whether Kafka connect can solve this or not. How it will be solved? I am looking for some suggestions for solving this problem.
Kafka Connect can send records to an HTTP(S) endpoint, or read from one and into Kafka. It will not allow external users to read through it, to get Kafka records.
The Strimzi Bridge or Confluent REST Proxy should both support HTTPS, even if that means you need to place either behind an SSL terminated reserve proxy such as Nginx, Caddy, etc.
Related
I have a data vendor for real-time data that has a strict limit on the number of websocket connections I am allowed to make to their API. I have multiple microservices that need to consume this data, with some overlap in subscriptions. The clients do not need to communicate back anything beyond subscriptions.
I would like to design a system using a proxy-server that maintains a single websocket connection to the data vendor and then relays the appropriate messages to the clients via websocket. Optimally, the clients would be able to interact with the proxy server as if it were the data vendor's API.
I have looked at various (reverse?) proxy server solutions here, but have not found specific language about reducing the number of connections to the upstream data source. For example, I have looked into NGINX, but I can't tell if the proxy will combine client connections into a single upstream connection. The other solution I have researched is just putting all messages into a Kafka Pub/Sub via a connector, and have each client subscribe there.
I am curious if there are any existing, out of the box solutions to this problem before I implement my own solution.
I have a gRPC client, not dockerised, and server application, which I want to dockerise.
What I don't understand is that gRPC first creates a connection with a server, which involves a handshake. So, if I want to deploy the dockerised server on ECS with multiple instances, then how will the client switch from one to the other (e.g., if one gRPC server falls over).
I know AWS loadbalancer now works with HTTP 2, but I can't find information on how to handle the fact that the server might change after the client has already opened a connection to another one.
What is involved?
You don't necessarily need an in-line load balancer for this. By using a Round Robin client-side load balancing policy along with a DNS record that points to multiple backend instances, you should be able to get some level of redundancy.
I'd like to create a reverse proxy to expose several grpc backend services on one host. But I'd also like to whitelist certain grpc status exception categories, and drop all others. I think I've read somewhere that grpc exceptions go into http/2 trailers, so that might be an option.
I'm trying to find info on the grpc wire protocol for passing exceptions, but can't find anything amid all the info on protobuf itself.
Any hints/links ?
I'm looking for software that supports custom UDP proxying/load-balancing based on an external routing table. The general idea is that some external service maintains a routing table (maybe using redis/memcached etc) that maps an downstream IP/Port combo to the address of an upstream internal service. Downstream clients would send all UDP traffic to the load balancer, which should then forward (preserving downstream IP/Port to support Direct Server Return) to the appropriate internal service based on the routing table.
I've looked at a range of existing load balancers (Envoy, Traefik, nginx etc) but can't seem to find any that would do what I need. Is there something out there that satisfies (or can be twisted into satisfying) these requirements?
I found this question which seems quite similar, but still doesn't satisfy my goals fully: UDP load balancing using customised balancing method with session id inside UDP body.
According to this blog, NGINX seems like a good option for providing the transparent proxying & DRS, but I'm not sure how to achieve the custom routing using an external table.
Thanks!
I have a kubernetes cluster with running pods. In order to monitor and troubleshoot the infrastructure, I want to implement a centralized logging solution so all incoming and outgoing HTTP requests will be logged within one place.
For the incoming requests this is not a problem at all, I can use nginx log from ingress controller and present it.
I also understand that I can log outgoing requests inside the application I run in pod, but the problem is that applications from outside developers are also used and it may not contain logging implementation.
As for the outgoing requests, there is no any solution provided by default if I understand it right. I have explored k8s logging and k8s audit, but it does not provide such feature.
Probably, I need some network sniffer, but it is quite a low-level solution for such problem as I can see. So, the question is: is there any out-of-the-box implementation for such demand?
Thanks!
Take a look at a service mesh solution like Istio or Linkerd as well as tracing solutions like Jaeger or Zipkin. With these you can build to have full observability on how information flows in/out and through your kube cluster