The problem I'm trying to solve is latency between Microservice communication on the backend. Scenario. Client makes a request to service A, which then calls service B that calls service C before returning a response to B which goes to A and back to the client.
Request: Client -> A -> B -> C
Response: C -> B -> A -> Client
The microservices expose a REST interface that is accessed using HTTP. Where each new HTTP connection between services to submit requests is an additional overhead. I'm looking for ways to reduce this overhead without bringing in another transport mechanism into the mix (i.e. stick to HTTP and REST as much as possible). Some answers suggest using Apache Thrift but I'd like to avoid that. Other possible solutions are using Messaging Queues which I'd also like to avoid. (To keep operational complexity down).
Has anyone experience in microservices communication using HTTP Connection pooling or HTTP/2? The system is deployed on AWS where service groups are fronted by a ELB.
HTTP/1.0 working mode was to open a connection for each request, and close the connection after each response.
Using HTTP/1.0 from remote clients and clients inside microservices (e.g. those in A that call B, and those in B that call C) should be avoided because the cost of opening a connection for each request can contribute for most of the latency.
HTTP/1.1 working mode is to open a connection and then leave it open until either peer explicitly requests to close it. This allow for the connection to be reused for multiple requests, and it's a big win because it reduces the latency, it uses less resources, and in general it is more efficient.
Fortunately nowadays both remote clients (e.g. browsers) and clients inside microservices support HTTP/1.1 well, or even HTTP/2.
Surely browsers have connection pooling, and any decent HTTP client that you may use inside your microservices does also have connection pooling.
Remote clients and microservices clients should be using at least HTTP/1.1 with connection pooling.
Regarding HTTP/2, while I am a big promoter of HTTP/2 for browser-to-server usage, for REST microservices calls inside data centers I would benchmark the parameters you are interested in for both HTTP/1.1 and HTTP/2, and then see how they fare. I expect HTTP/2 to be on par with HTTP/1.1 for most cases, if not slightly better.
The way I would do it using HTTP/2 (disclaimer, I'm a Jetty committer) would be to offload TLS from remote clients using HAProxy, and then use clear-text HTTP/2 between microservices A, B and C using Jetty's HttpClient with HTTP/2 transport.
I'm not sure AWS ELB already supports HTTP/2 at the time of this writing, but if it does not please be sure to drop a message to Amazon asking to support it (many others already did that). As I said, alternatively you can use HAProxy.
For communication between microservices, you can use HTTP/2 no matter what is the protocol used by remote clients.
By using Jetty's HttpClient, you can very easily switch between the HTTP/1.1 and the HTTP/2 transports, so this gives you the maximum of flexibility.
If latency is really an issue to you then you should probably not be using service calls between your components. Rather you should minimize the number of times control passes to an out-of-band resource and be making the calls in-process, which is much faster.
However, in most cases, the overheads incurred by the service "wrappers" (channel construction, serialisation, marshalling, etc), are negligible enough and still well within adequate latency tolerances for the business process being supported.
So you should ask yourself:
Is latency really an issue for you, in respect to the business process? In my experience only engineers care about latency. Your business customers do not.
If latency is an issue, then can the latency definitively be attributed to the cost of making the service calls? Could there be another reason the calls are taking a long time?
If it is the services, then you should look at consuming the service code as an assembly, rather than out-of-band.
For the benefit of others running into this problem, apart from using HTTP/2, SSL/TLS Offloading, Co-location, consider using Caching where you can. This not only improves performance, but reduces dependency on downstream services. Also, consider data formats that are perform well.
latency between Micro-service communications is an issue for low latency applicaitons however number of call can be minimize by hybrid between micro-services and monolith
Emerging C++ microservices framework is best for low latency applications
https://github.com/CppMicroServices/CppMicroServices
Related
Imagine I'm building an ordinary old website. Not a game, not a chat program, an ordinary website. Let's say it's a stack overflow clone.
The client side would simply make service calls to the server side. The server is essentially a dumb data store and never sends down HTML. The client handles all templating via javascript.
If I established a single websocket connection and did all requests through that, would I see a significant speedup over doing ajax requests?
The obvious advantage to using a single connection is that it only has to be established once. But how much time does that actually save? I know establishing a TCP connection can be costly, but in the grand scheme of things, does it matter?
I would not recommend websockets for webpages. HTTP 1.1 can reuse a TCP-connection for multiple requests, it's only HTTP 1.0 that had to use a new TCP connection for each request.
SPDY is probably a protocol that do what you are looking for. See SPDY: An experimental protocol for a faster web, but it's only supported by Chrome.
If you use websockets, the requests will not be cached.
One HTTP connection can only by used for one HTTP request at the same time. Say that a page requested a 100Kb document, nothing else will be send from the client to the server until that 100Kb document has been transferred. This is called head-of-line blocking. The client can establish an additional connection with the server, but there is also a limit on the amount of concurrent connections with the same server.
One of the primary reasons for developing SPDY and later HTTP/2 was solving this exact problem. However, support for SPDY and HTTP/2 is not yet as widespread as for WebSocket. WebSocket can get you there earlier because it supports multiple streams in full-duplex mode.
Once HTTP/2 is better supported it will be the preferred solution for this problem, but WebSocket will still be better for real-time web applications, where server needs to push data to the client.
Have a look at the N2O framework, it was created to address the problems I described above. In N2O WebSocket is used to send all assets associated with a page.
How much speed you could gain from using WebSocket instead of standard HTTP requests pretty much depends on your specific website: how often it requests data from the server, how big is a typical response, etc.
I've been working with websockets lately in detail. Created my own server and there's a public demo. I don't have such detailed experience or knowledge re: http. (Although since websocket requests are upgraded http requests, I have some.)
On my end, the server reports details of each hit. Among them are a bunch of http keep-alive requests. My server doesn't handle them because they're not websocket requests. But it got my curiosity up.
The whole big thing about websockets is that the connection stays alive. Then you can pass messages in both directions (simultaneously even). I've read that the Keep-Alive HTTP connection is a relatively new development (I don't know how many years in people time, just that it's only included in the latest standard - 1.1 - is that actually old now?)
I guess I can assume that there's a behavioral difference between the two or there would have been no reason for a websocket standard? What's the difference?
A Keep Alive HTTP header since HTTP 1.0, which is used to indicate a HTTP client would like to maintain a persistent connection with HTTP server. The main objects is to eliminate the needs for opening TCP connection for each HTTP request. However, while there is a persistent connection open, the protocol for communication between client and server is still following the basic HTTP request/response pattern. In other word, server side can't push data to client.
WebSocket is completely different mechanism, which is used to setup a persistent, full-duplex connection. With this full-duplex connection, server side can push data to client and client should be expected to process data from server side at any time.
Quoting corresponding entries on Wikipedia for reference:
1) http://en.wikipedia.org/wiki/HTTP_persistent_connection
2) http://en.wikipedia.org/wiki/WebSocket
You should read up on COMET, a design pattern which shows the limits of HTTP Keep-Alive. Keep-Alive is over 12 years old now, so it's not a new feature of HTTP. The problem is that it's not sufficient; the client and server cannot communicate in a truly asynchronous manner. The client must always use a "hanging" request in order to get a message back from the server; the server may not just send a message to the client at any time it wants.
HTTP vs Websockets
REST (HTTP)
Resources benefit from caching when the representation of a resource changes rarely or multiple clients are expected to retrieve the resource.
HTTP methods have well-known idempotency and safety properties. A request is “idempotent” if it can be issued multiple times without resulting in unique outcomes.
The HTTP design allows for responses to describe errors with the request, with the resource, or to provide nuanced status information to differentiate between success scenarios.
Have request and response functionality.
HTTP v1.1 may allow multiple requests to reuse a single connection, there will generally be small timeout periods intended to control resource consumption.
You might be using HTTP incorrectly if…
Your design relies on a client polling the service often, without the user taking action.
Your design requires frequent service calls to send small messages.
The client needs to quickly react to a change to a resource, and it cannot predict when the change will occur.
The resulting design is cost-prohibitive. Ask yourself: Is a WebSocket solution substantially less effort to design, implement, test, and operate?
WebSockets
WebSocket design does not allow explicit or transparent proxies to cache messages, which can degrade client performance.
WebSocket protocol offers support only for error scenarios affecting the establishment of the connection. Once the connection is established and messages are exchanged, any additional error scenarios must be addressed in the messaging layer design, but WebSockets allow for a higher amount of efficiency compared to REST because they do not require the HTTP request/response overhead for each message sent and received.
When a client needs to react quickly to a change (especially one it cannot predict), a WebSocket may be best.
This makes the protocol well suited to “fire and forget” messaging scenarios and poorly suited for transactional requirements.
WebSockets were designed specifically for long-lived connection scenarios, they avoid the overhead of establishing connections and sending HTTP request/response headers, resulting in a significant performance boost
You might be using WebSockets incorrectly if..
The connection is used only for a very small number of events, or a very small amount of time, and the client does not - need to quickly react to the events.
Your feature requires multiple WebSockets to be open to the same service at once.
Your feature opens a WebSocket, sends messages, then closes it—then repeats the process later.
You’re re-implementing a request/response pattern within the messaging layer.
The resulting design is cost-prohibitive. Ask yourself: Is a HTTP solution substantially less effort to design, implement, test, and operate?
Ref: https://blogs.windows.com/buildingapps/2016/03/14/when-to-use-a-http-call-instead-of-a-websocket-or-http-2-0/
I was considering doing a chat server using node.js/socket.io. Should I make it a tcp server or a http server? I'd imagine tcp server would be more efficient, but can you send other stuff to it like file attachments etc? If tcp is more efficient, how much more so? Also, just wondering how many concurrent connections can one node.js server handle? Is it more work to do TCP or HTTP?
You are talking about 2 totally different approaches here - TCP is a transport layer protocol and HTTP is an application layer protocol. HTTP (usually) operates over TCP, so whichever option you choose, it will still be operating over TCP.
The efficiency question is sort of a moot point, because you are talking about different OSI layers. If you went for raw TCP sockets, your solution would probably be more efficient - in bandwidth at least - since HTTP contains a whole bunch of extra data (the headers) that would likely be irrelevant to your purposes (depending on the scale of the chat program). What you are talking about developing there is your own application layer protocol.
You can send anything you like over TCP - after all HTTP can send attachments, and that operates over TCP. FTP also operates over TCP, and that is designed purely for transferring "attachments". In order to do this, you would need to write your protocol so that it was able to tell the remote party that the following data was a file, then send the file data, then tell the remote party that the transfer is complete. Implementations of this are many and varied (the HTTP approach is completely different from the FTP approach) and your options are pretty much infinite.
I don't know for sure about the node.js connection limit, but I can say with a fair amount of confidence that it is limited by the operating system. This might help you get to grips with the answer to that question.
It is debatable whether it is more work to do it with TCP or HTTP - it's a lot of work to do it in both. I would probably lean more toward the TCP option being your best bet. While TCP would require you to design a protocol rather than/as well as an application, HTTP is not particularly suited to live, 2-way applications like chat servers. There are many implementations of chat over HTTP that use AJAX, but I can tell you from painful experience that they are a complete pain in the rear-end.
I would say that you should only be looking at HTTP if you are intending the endpoint (i.e. the client) to be a browser. If you are going to write a desktop app for the endpoint, a direct TCP link would definitely be the way to go. The main reason for this is that HTTP works in a request-response manner, where the client sends a request to the server, and the server responds. Over TCP you can open a single TCP stream, that can be used for bi-directional communication. This means that the server can push an event to the client instantly, while over HTTP you have to wait for the client to send a request, so you can respond with an event. If you were intending to use a browser as the client, it will make the whole file transfer thing much more tricky (the sending at least).
There are ways to implement this over HTTP using long-polling and server push (read this) but it can be a real pain to implement.
If you are going to implement this on a LAN (or possibly even over the internet) it is worth considering UDP over TCP - in a chat application it is not usually absolutely mission critical that messages arrive in the right order, and even if it was, users would probably not be able to type faster than the variations in network latency (probably <100ms). Then for file transfers you could either negotiate a seperate TCP socket for the data exchange (like FTP), or implement some kind of UDP ACK system (like TFTP).
I feel there is a lot more to say on this subject but right now I can't put it into words - I may extend this answer at some point.
Chat servers are the Hello World program in node. Use http.
As far as the question of how many concurrent connections can it handle, that all depends on your system. Set up a simple chat server and then try benchmarking it.
Also, check out http://search.npmjs.org/ and search for chat for a few pointers.
I have two servers running in iis on different computers and they need to communicate with each other. Because of firewalls etc http is the only real option. It's bidirectional communication, not just request/response. Web sockets would be good, but the spec for that isn't finished so I'm wondering if there are any other options I should look at?
I would spend a few minutes thinking about wether my communication could actually be done using request response, a lot of times you can convert your communication into request response. HTTP is really very request response based so you kind of have to hack around it using various comet techniques like long polling. If you have control over both ends of the communication then there is no reason not to use web sockets even if the standard isn't settled upon. So long as you agree with yourself how it should be implemented complying with the final standard isn't important.
I want to use a client-server protocol to push data to clients which will always remain connected, 24/7.
HTTP is a good general-purpose client-server protocol. I don't think the semantics possibly could be very different for any other protocol, and many good HTTP servers exist.
The critical factor is the number of connections: the application will gradually scale up to a very large number of clients, say 100,000. They cannot be servers because they have dynamic IP addresses and may be behind firewalls. So, a socket link must be established and preserved, which leads us to HTTP push. Only rarely will data actually be pushed to a given client, so we want to minimize the connection overhead too.
The server should handle this by accepting the connection, inserting the remote IP and port into a table, and leaving it idle. We don't want 100,000 threads running, just so many table entries and file descriptors.
Is there any way to achieve this using an off-the-shelf HTTP server, without writing at the socket layer?
Use Push Framework : http://www.pushframework.com.
It was designed for that goal of managing a large number of long-lived asynchronous full-duplex connections.
LightStreamer (http://www.lightstreamer.com/) is the tool that is made specifically for PUSH operations of HTTP.
It should solve this problem.
You could also look at Jetty + Continuations.