What are the different approaches in developing a web-server?
So I guess there are (1) multi-thread (2) event-loop, is there anything else? What would be the pros/cons of each approach? when would you use each? can you list specific impl' for each approache
Different approach can be:
Single threaded: All connections are handled by a single thread that
"listens" for and awaits for connections and processes requests.It
is simple to implement but it is the most useless server as it can
only serve request at a time
Multithreaded:The server listens for requests and each incoming
request is allocated to a new thread to handle it.So each client
connection is handled by its dedicated thread. This approach(unlike
1) supports concurrent processing of client requests but does not
scale well since each new request creates a new thread at the server
and this takes a lot of resources.Eventually the server will hit a
limit
Multithreaded-Pools:Same idea as (2) but instead of creating a new
thread to handle each incoming request a thread from a thread-pool
is used.I.e. threads are created and placed on a pool for later
reuse.This scales very well supporting multiple client requests and
it is the standard approach.E.g. Tomcat works like this.
Event-Queue:Each incoming request is placed into a queue and is
processed by a background thread taking requests of the queue. It is
non-blocking and this type of asynchronous processing also scales
well.To be honest I am not sure if it is better than (3) in
performance.I think that tomcat can be configured for this using the
NIO architecture
You should add non-blocking I/O. Have a look at Netty.
Some servers like G-WAN mix Multithreaded-Pools and Event-Queues, letting the server saturate CPU Cores with each thread processing many connections.
Disclamer: I am involved in the development of this project.
Related
So I came across the idea of blocking and non-blocking I/O. But what I understood from the concept and some of the sample implementations is that we implement code on the server side to achieve this nature of the code.
But now my question is, if (for example postman sending HTTP request to the server) the request has to wait for the server to respond, then what's the point of non-blocking I/O? (Please correct me if I am wrong) Or the whole concept is just for the increase of throughput of the server instead of actual async nature w.r.t. to client.
For example, in one of my project what I did was created a post request to create a request in the system for processing which will return the transaction ID, now using this transaction id, I can query the server to know the outcome.
I may sound too naive, but the concept has confused me a lot. I do not understand this concept clearly. Please help.
Thanks
the request has to wait for the server to respond, then what's the point of non-blocking I/O?
There's a confusion. Waiting for a response and (non)blocking i/o are very loosely related. You always have to wait for response. That's why youve made the request to begin with. But the question is: how?
Non-blocking HTTP: "Dear server, here's my request, please process it and send me a response, I'm going to do something else in the meantime, like calculating n-th digit of Pi (I'm a weirdo)".
Blocking HTTP: "Dear server, here's my request, please process it and send me a response, I'm going to patiently wait for it doing nothing".
Or the whole concept is just for the increase of throughput of the server instead of actual async nature w.r.t. to client.
The whole concept is to be able to do other things while waiting for i/o at the same time. And to do that while minimizing the usage of threads which don't scale well.
Asynchronous systems, i.e. systems without "I'm going to wait idly" part tend to perform better at the cost of complexity.
Side note: nonblocking i/o can be used both on the server side and client side. For example almost all JS engines in browsers are built on top of some asynchronous engine. JS is often single-threaded, meaning nonblocking i/o is necessary to achieve any concurrency.
But what I understood from the concept and some of the sample implementations is that we implement code on the server side to achieve this nature of the code.
You implement code in whereever you are doing the non-blocking UI. What a server does has no bearing on whether a client uses blocking or non-blocking UI, and what a client does has no bearing on whether a server uses blocking or non-blocking UI.
if (for example postman sending HTTP request to the server) the request has to wait for the server to respond, then what's the point of non-blocking I/O?
So that you're not wasting resources.
Let's consider first a simple console application that hits the web and then does something with the results. In this case there's very little to gain with non-blocking I/O as the application is just going to be sitting around waiting for something to do anyway.
Now let's consider a simple console application that hits 50 different web resources and collates the responses. Now non-blocking I/O is more useful, because with blocking I/O it would have to either get one resource after the other, or spin up 50 threads. With non-blocking I/O one, a small number of threads is all that is needed to hit 50 resources and respond promptly to each returning a response.
Now let's consider a GUI version of this application that wants to remain responsive to user input, while also running on low-power low-memory devices in which blocked threads are all the more expensive. The advantages of the above are increased.
Finally, consider a web application that is doing I/O both with the client and also as a client to a database, file system and maybe other web applications. It may have multiple requests at the same time, and blocking on either the I/O it does with the client or any of the I/O it does with db, file or other applications would cost a thread, which would put a scalability limit on how many requests it can handle simultaneously. Not blocking on I/O allows threads to be used for other requests while the I/O is pending.
I'm new to GRPC. I want to know that if the server start a new thread to process when a GRPC client start one request.
There may be up to one Runnable enqueued to the Server's executor for application processing. Each request may generate more than one Runnable over time, but only one at a given time. The default executor is an unbounded cached thread pool, so worst-case each request gets its own thread initially, but later requests will generally reuse previous threads.
It is good practice for high QPS services to specify a fixed-sized executor, to avoid excessive number of threads and reduced context switch thrashing.
First I'll describe my case.
I have to do HTTPS requests to several APIs from my application and they should be ran concurrently.
I want to know if I should use a separate HTTP client per goroutine or I can share one client across all goroutines. Of course I'd like to enjoy connection reusing/pooling offered by the HTTP client, but I am concerned about it being thread(aka goroutine)-safe and if the client will run requests concurrently or they'll in fact be sequenced?
Http clients are thread safe according to the docs (https://golang.org/src/net/http/client.go):
Clients are safe for concurrent use by multiple goroutines.
The other question was should you use one client or one per request. You should use one client as per
https://pkg.go.dev/net/http#pkg-overview
"Clients and Transports are safe for concurrent use by multiple goroutines and for efficiency should only be created once and re-used"
We have a requirement to to support 10k+ users, where every user initiate a request and waits for a response from the server (the response can take as long as 20-30 seconds to arrive). it is only one request from the client, and after a long processing by the server, a response will be transmitted and then the connection will disconnect.
in the background, the server will do some DB search and wait for other background processes to notify on completion before responding to the client.
after doing some research i figured out we will need to use something like the atmosphere framework to support websockets/sse event/long polling along with an asynchronous server like netty (=> nettosphere) or jetty.
As for my experience - mostly Java EE world and Tomcat server.
my questions are:
what will be easier to implement in regard to my experience and our requirement: atmosphere + netty or atmoshphere+jetty? which one can scale better, has an easier learning curve and easier to implement other java technologies?
how do u implement in atmosphere a response that is sent only to the originating client and not broadcast to the rest of the clients? (all the examples i found are broadcast).
how can i implement in netty (or jetty) when using the atmosphere framework our response? i.e., the client send a request, after it is received in the server some background processes are run, and when they finish i need to locate the connection and transmit the response. is that achievable?
Some thoughts:
At 10k+ users, with 20-30 second response latency, you likely hit file descriptor limits if using just 1 network interface. Consider a solution that uses multiple network interfaces.
Your description of your request/response can be handled entirely with standard Servlet 3.0, standard HTTP/1.1, Async request handling, and large timeouts.
If your clients are web browsers, and you don't start sending a response from the server until the 20-30 second window, you might hit browser idle timeouts.
Atmosphere and Cometd do the same things, supporting long duration connections, with connection technique fallbacks, and with logical channel APIs.
I believe the AKKA framework will handle this sort of need. I am looking at using it to handle scaling issues possibly with a RabbitMQ to help off load work to potentially other servers that may be added later to scale as needed.
I'm building an ASP.NET service (a simple aspx) that requires a REQ call to a ZeroMQ REP node.
So I've to use the REQ/REP pattern, but I can't figure out the proper way to initialize the ZeroMQ context in the ASP.NET pipeline.
Moreover, can I share a single connection among the different ASP.NET threads and if so how?
edit: After some study it looks to me that an inproc router in a dedicated thread should be the way to go, since it would handle sincronization.
But more questions arise:
The other end of such an inproc node should be a DEALER? If so, should it connect to the REQ node? Or it should bind to a tcp port and I should code the REP server node to connect to it (the latter would be a bit cumbersome, since I could have different servers exposing the service)?
As an alternative, is it correct to build an inproc node bound to a ROUTER socket at one end, and connecting with REQ on the other? If so, should I code the node so that it handles a manual envelop of each message just to be able to send responses back to the correct requesting thread?
Is Application_Start the correct pipeline point to initialize the thread handling such router?
At the moment a ROUTER/DEALER inproc node that connect to the REQ server looks like the best option, but I'm not sure that it's possibile to connect from a DEALER socket. But this is still just a speculation and could be entirely wrong.
The zmq_socket manual states:
ØMQ sockets are not thread safe. Applications MUST NOT use a socket
from multiple threads except after migrating a socket from one thread
to another with a "full fence" memory barrier.