How can I write Http server in TornadoWeb that will support persistent Connections.
I mean will be able to receive many requests and answer to them without closing connection.
How does it actually work in async?
I just want to know how to write handler to handle persistent connection.
How actually would it work?
I have handler like that:
class MainHandler(RequestHandler):
count = 0
#asynchronous
def post(self):
#get header content type
content_type = self.request.headers.get('Content-Type')
if not content_type in ACCEPTED_CONTENT:
raise HTTPError(403, 'Incorrect content type')
text = self.request.body
self.count += 1
command = CommandObject(text, self.count, callback = self.async_callback(self.on_response))
command.execute()
def on_response(self, response):
if response.error: raise HTTPError(500)
body = response.body
self.write(body)
self.flush()
execute calls callback when finishes.
is my asumption right that with things that way post will be called many times
and for one connection count will increase with each httprequest from client?
but for each connection I will have separate count value?
I don't think that your assumption is correct. My understanding of the way the Tornado server works is that each request from the client will produce a new RequestHandler. The purpose of the #tornado.web.asynchronous decorator is to prevent the server from automatically closing the connection when your handler function (post, get, etc.) returns. But at the end of the day, I think there is just one response for each request.
I don't believe additional requests from the client will go to the same instance of the RequestHandler class. Instead, my understanding is that Tornado is set up to allow for the long-polling paradigm. Here is an example of the flow of communications:
Client makes a POST request to the Tornado server
Tornado server checks to see if a response is ready, if not you could add the RequestHandler to some kind of stack or queue (depending on your application architecture)
Server comes up with a response (maybe another user added a message to the queue that needs to be distributed to open connections, etc.) and distributes the response back to the RequestHandler and then calls the finish() function to close the connection
Client makes another POST request to repeat the process
I think if you want to implement true persistent connections you'll want to look into tornado.websocket (http://www.tornadoweb.org/documentation/websocket.html). I haven't experimented with that module yet so I'm afraid I can't give any input there.
Best of luck!
The Tornado web framework actually does come with it's own server implementation which supports persistent connections, so there should be no need to write your own server. There is a section in the documentation on how to use it in production (behind nginx).
From the source for tornado.web module, you can see that a new handler is always instantiated, I don't think there is anyway you can have handlers reused.
Related
In an old-school server environment, you initialize an SDK (like the Twitter SDK) when the server starts up, using dotenv to read secrets and tokens from your .env file like so:
import dotenv from 'dotenv';
import {Client} from 'twitter-api-sdk';
dotenv.config();
const twitterClient = new Client (TWITTER_SECRET_INFO);
And then you would use the twitterClient object to get data in one of the route handlers.
What's the best practice for initializing something like the twitter client in Hono with Cloudflare?
In the old service worker framework, I could have treated the secret info as a global environment variable much like in Node/Express, but in the new module worker code you have to access the environment variables as a parameter passed to a function call. It looks like Hono manages this by passing contexts to methods like .use/.get/.post.
Ideally, though, I wouldn't reinitialize the twitter connection on every request, especially since I'm just getting public info with a token, not dealing with any user login/password info.
Is there any way to do this in Hono/Cloudflare, or do I have to initialize the Twitter client middle ware each request? I looked at the Hono class constructer, but from what I can tell, all it does is take a router config object.
And from what I can tell of the cloudflare docs, module workers have the same issue. Whereas constants in a service worker were declared outside the route handler, it looks like everything in a module worker is declared inside a fetch handler. Is there anyway to initialize once during the life of the worker and not for each request?
In principle you could initialize the client on the first request:
let twitterClient = null;
export default {
async fetch(req, env, ctx) {
if (!twitterClient) {
twitterClient = new Client(env.TWITTER_SECRET_INFO);
}
// ... normal code ...
}
}
That said, though, is creating a new client actually expensive?
Constructing the client does not "initialize a connection". The client presumably makes requests by calling fetch(). The fetch() API doesn't expose any way to control the underlying connections used; each fetch() operates effectively independently. But, the Workers Runtime will automatically reuse connections behind the scenes, when possible. It could even reuse the same connection for two completely unrelated Workers, if they are contacting the same destination host. So it may be that even creating a new client with every request, you're already getting good connection reuse.
That said, perhaps the client has to do some sort of key exchange upfront, e.g. exchanging a long-lived refresh token for an access token. That is annoying to have to repeat on every request. So in that sense, maybe caching it in a global helps.
However, note that Workers creates LOTS of instances of your Worker around the world. You may find if you curl your Worker several times in a row, each request lands on a different instance. You may find that caching in global state does not actually have much impact unless you have a large amount of traffic.
Caching may be more effective if you use the Cache API to store cached values into the colo-wide cache. Unfortunately, client libraries designed for Node environments may not provide the right hooks to do this.
One final note: Note that putting live resources (things that are not just plain data structures) into the global scope can be dangerous on Workers, because in general a Promise created on behalf of one incoming request cannot be awaited in the context of some other request. So if that twitter client does do some sort of upfront key exchange and tries to have all requests wait for that to complete, you may find that if you receive multiple requests at once before the initial key exchange finishes, all except the first request end up failing. To be honest, I would recommend creating a new client for every request unless you see a measurable performance problem from this.
I'm writing a wrapper around gRPC unary calls, but I'm having an issue: let's say I have a ClientAsyncResponseReader object which is created and starts a request like so
response_reader_ = std::unique_ptr<grpc::ClientAsyncResponseReader<ResponseType>>(
grpc::internal::ClientAsyncResponseReaderFactory<ResponseType>::Create(
channel.get(), completion_queue, rpc_method, &client_context_, request, true
)
);
response_reader_->Finish(
response_sharedptr_.get(), status_sharedptr_.get(), static_cast<void*>(some_tag)
);
// Set a breakpoint here
where all of the arguments are valid.
I was under the impression that when the Finish call returned, the request object was guaranteed to have been sent out over the wire. However by setting a breakpoint after that Finish() call (in the client program, to be clear) and inspecting my server's logs, I've discovered that the server does not log the request until after I resume from the breakpoint.
This would seem to indicate that there's something else I need to wait on in order to ensure that the request is really sent out: and moreover, that the thread executing the code above still has some sort of role in sending out the request which appears post-breakpoint.
Of course, perhaps my assumptions are wrong and the server isn't logging the request as soon as it comes in. If not though, then clearly I don't understand gRPC's semantics as well as I should, so I was hoping for some more experienced insight.
You can see the code for my unary call abstraction here. It should be sufficient, but if anything else is required I'm happy to provide it.
EDIT: The plot thickens. After setting a breakpoint on the server's handler for the incoming requests, it looks like the call to Finish generally does "ensure" that the request has been sent out: except for the first request sent by the process. I guess that there is some state maintained either in grpc::channel or maybe even in grpc::completion_queue which is delaying the initial request
From the documentation
response_reader_ = std::unique_ptr<grpc::ClientAsyncResponseReader<ResponseType>>(
grpc::internal::ClientAsyncResponseReaderFactory<ResponseType>::Create(
channel.get(), completion_queue, rpc_method, &client_context_, request, true
)
);
This will start a call and write the request out (start=true). This function does not have a tag parameter. So there is no way the completion queue can notify when the call start is finished. Calling an RPC method is a bit complicated, it basically involves creating the network packet and putting it in the wire. It can fail if there is a transient failure of the transport or the channel completely gone or the user did something stupid. Another thing, why we need the tag notification is that the completion queue is really a contention point. All RPC objects talk to this, it can happen completion queue is not free and the request is still pending.
response_reader_->Finish(
response_sharedptr_.get(), status_sharedptr_.get(), static_cast<void*>(some_tag)
This one will request the RPC runtime to receive the server's response. The output is when the server response arrives, then the completion queue will notify the client. At this point. we assume that there is no error on the client side, everything okay and the request is already in flight. So the status of Finish call will never be false for unary rpc.
This would seem to indicate that there's something else I need to wait on in order to ensure that the request is really sent out: and moreover, that the thread executing the code above still has some sort of role in sending out the request which appears post-breakpoint.
Perhaps, you want to reuse the request object(I did some experiments on that). For me, I keep the request object in memory till the response arrives. There is no way to guarantee that the request object won't be required after the create call.
Let's imagine there is a server, that when receives a request with a car model queries all known car dealers looking for the cheapest one and responds back with the price (using whatever protocol). This actions takes a while.
In a casual blocking request/response server model, I do
request = "audi a8" // prepare a request and one line after have the response
response = server.findCheapestCar(request) // takes 20 seconds
I don't want to block my client main thread for 20 seconds, so would rather want it to be executed asynchronously. My understanding for something being asynchronous is that I can pass some sort of an object to it and carry one with my work. Once the server is ready with the response it will notify the object I passed -> Casual callback pattern.
This approach would require library match - both client ad the server need to know the object. But I want my asynchronous server built on Netty to be able to handle requests from various clients (C++/Python and others).
Where is the asynchronousness of netty coming from? What do I need on the client side to benefit from the asynchronousness?
Where is the asynchronousness of netty coming from?
Netty adopted the principle of eventloops which you may known from a language like JavaScript. This allows netty to work fully asynchronous. (For more information about eventloops and the basic underlying principle I would recommend this video about the evenloop in JavaScript)
What do I need on the client side to benefit from the asynchronousness?
Client sends request (containing payload and request id = clientside incrementing integer)
Server process the request for 50sec
Server sends response (containing the payload and the same request id the client send in his request)
Client receives the response and looks up the request id (If the client is able to find the request id and its underlying callback it will invoke it)
Hope that helped
I am working on prototyping a new web service for my company and we are considering Apache Camel as our integration framework. Here is a quick run-down of the high-level architecture:
-IBM Websphere MQ as the queuing solution
1) we receive http request
2) asynchronously persist this request
3a) do some processing on the request
3b) send to another tier for further processing
4) asynchronously update the request record in DB
5) respond to caller
What I want to do is:
When a http request comes in, put it on a queue to be processed and wait n seconds. If the web handler doesn't get a response in n seconds, reply to the caller with a custom message
Once the request is on the processing queue, a camel route is listening to this queue to process. When it pulls a message from queue, put a copy of the request on a different queue to be persisted asynchronously. Do some processing on the request. Then send it to another queue to be further processed and wait for a response. Then put it back on the persist queue to be asynchronously updated.
Then respond to web listener. Then web listener responds to web caller.
I am reading everything I can about Apache Camel and there is a lot of information about there. I might be on a little bit of information overload, and any help on the following concerns would be greatly appreciated:
1)
If the web listeners use an InOut exchange (with the first processing tier) without a replyTo queue defined, it will create a temporary queue for the response. What happens if this request times out? I understand I can set a requestTimeout on the exchange and, if it times out, catch that exception and set a custom message. But, will that temporary queue be killed? Or will they build up over time as requests time out?
2)
When it comes to scaling the processing tiers (adding more instances of those same routes on different machines), is it customary that if the instance that picks up the response (using a fixed reply to queue) is different than the instance that picked up the request, all the information about the original request is inside the message, so there is no need to share data across instances (unless of course there is data that is shared, like aggregrates and such)?
Any other tips and tricks when building a system like this would be very helpful.
Thanks!
I would say this solution is too complicated and there are too many areas which are hard both in terms of maintenance and also complexity. There is too much many steps mixing async and sync communication.
Why not simply the solution to the following steps:
Synchronously http request
Put message on MQ with reply to header
Message is picked up and sent to backend
If reply is not received within a given time transaction is terminated.
The reply to queue is removed
Requestor is notified.
This is the issue I encounter, which is design and implementation related :
I have a REST web service that accepts POST requests. Nothing special about it. It currently responds synchronously.
However, this web service is going to initiate a background process that may take some long time.
I do not want this service to respond 30 minutes later.
Instead, it should immediately return an ack response to the client, and nothing more (even after 30 minutes, there will be no more information to send).
How do I implement such behavior with Jersey ?
I read the page https://jersey.java.net/nonav/documentation/2.0/async.html#d0e6914.
Though it was an interesting reading, I did not find the way to only send an ACK typed response (something like an HTTP 200 code).
Maybe i am confused with asynchronous and the behavior I want to implement.
I just understood that I could create a new Thread within my #POST method to handle the background process, and just returns immediately the ACK response.
But does this newly thread live after the response has been sent back to the client ?
How would you implement this WS ?
I hope you will help me clarifying this point.
I think the Jersey 2 Asynchronous Server API you linked would still hold the client connection until the processing completes. The asynchronous processing is really internal to Jersey and does not affect the client experience.
If you want to return an ACK, you can use a regular Jersey method, delegate the work to another thread and then return immediately. I'd recommend HTTP 202 for this use case.
You may create a Thread to do so just like in the Jersey 2 example and it would survive the execution of the Jersey resource method invocation:
#POST
public Response asyncPost(String data) {
new Thread(...).start();
return Response.status(Response.Status.ACCEPTED).build();
}
This being said, creating threads is generally not recommended within app servers.
If you're using EE7, I'd recommend you look at JSR-236 http://docs.oracle.com/javaee/7/api/javax/enterprise/concurrent/package-summary.html
If you're using EE6, you can consider sending a message to a queue to be processed by a Message-Driven Beans (MDB) in the background.