I read the postings from this entry. I'm wondering if there is a way to run multiple instances in different processes? Because when I understand it right, running all instances in the same configuration file will create a common master and several worker processes. If the master process dies then all instances die at the same time.
From my perspective it would be better to run them separately, so that each backend system gets its own master. Do you agree? How could I achieve that?
Thankyou.
I think the better solution is to use a HA Cluster. You'll use two master server configured in exactly the same way, and a balancer in front of them.
This configuration will hide a possible fault and will work in "reduced" mode.
Related
My stack is uWSGI, flask and nginx currently. I have a need to store data between requests (basically I receive push notifications from another service about events to the server and I want to store those events in the server memory, so client can just query server every n milliseconds, to receive latest update).
Normally this would not work, because of many reasons. One is a good deployment requires you to have several processes in uwsgi in production (and even maybe several machines to scale this out). But my case is very specific: I'm building a web app for a piece of hardware (You can think of your home router configuration page as a good example). This means no need to scale. I also do not have a database (at least not a traditional one) and probably normally 1-2 clients simultaneously.
if I specify --processes 1 --threads 4 in uwsgi, is this enough to ensure the data is kept in the memory as a single instance? Or do I also need to use --threads 1?
I'm also aware that some web servers clear memory randomly from time to time and restart the hosted app. Does nginx/uwsgi do that and where can I read about the rules?
I'd also welcome advises on how to design all of this, if there are better ways to handle this. Please note that I do not consider using any persistant storage for this - this does not worth the effort and may be even impossible due to hardware limitations.
Just to clarify: When I'm talking about one instance of data, I'm thinking of my app.py executing exactly one time and keeping the instances defined there for as long as the server lives.
If you don't need data to persist past a server restart, why not just build a cache object into you application that can do push and pop operations?
A simple array of objects should suffice, one flask route pushes new data to the array and another can pop the data off the array.
I have a machine with only nginx installed without passenger that acts as load balancer with ips of some machines in its upstream list. All the app machines have nginx with phusion passenger that serve the main application. Now some of the application machines are of medium type while others are large type. As far as I know the default nginx load balancing scheme is round robin. As the load is distributed among the large and medium machines equally, if the traffic is large the medium machines get overloaded and when its less the large machines resources are wasted. Now I use newrelic to monitor the cpu and memory on these machines and a script to get the data from newrelic, so is there any way to use this data to decide the traffic route on load balancer.
One way I know is to monitor and the mark machines in upstream good or bad and then replace the upstream with the good ones and reload the nginx.conf each time without complete restart. So my second question, is the way correct. In other words does it have any drawbacks or will it cause any issues?
Third and more general question is there a better way tackle this issue of load balancing?
You can use another load balancing algorithm that will distribute load more fair: http://nginx.org/r/least_conn or/and configure weights.
Making decision based on current cpu/memory usage isn't a good idea if your goal is faster request processing instead of meaningless numbers
I have a nginx server that has a redis master and two salves of the master. The slaves are read and the master is read and write. Nginx server is fastcgi using spawed python apps and using pyredis.
When is comes for a read from my nginx app, what is best practice for determining which server gets the read among the three? Is it determined in realtime? Do I just do simple random selection using round robin in real time?
Again, I just have on master. Soon I will have two and will use consistent hashing in python using http://pypi.python.org/pypi/hash_ring so select which server gets the keys.
For the interim, is it wise to select which server will get the read using the hash ring even though they should be exact copies?
Thanks,
what you should do is abstract the code that does that so it doesn't change your app logic later when you split the data.
and as for reading - I'd use just the slaves for that. you can use the hashing if you want, provided it doesn't affect your code and is abstracted.
I have a content management system running on a web server, that among others allows the user to upload assets like images, files, etc to the server.
The problem i have is that there will be 2 servers running behind a load balancer and i am trying to find an efficient way to handle the assets management.
The question i have is:
Will the assets be uploaded to one server every time? Or is there a chance that the images/files will end up into server1 or server2 depending on the load?
How to i serve the images if i don't know on which server they end up in? Will i have to keep the directories of these assets (images/files) synchronized between the two servers?
Thanks,
Synchronization is a tough problem to crack. You can do ad-hoc synchronization using Couchdb but that requires good knowledge of the low-level issues. Therefore you need to choose a write master.
DRDB
You could look at DRDB :D Use one server as the write-master and the other as the slave. Then you can server content from both. This approach is amazing for database-pairs.
Note: seperating your code and URL's for write-master and serve-only will be anoying
Couchdb
You could use couchdb but I think that might be overkill. This is for the LARGE amounts of data and high-levels of fault tolerance.
NFS
You could export the asset directory on the write-master as an nfs drive and import it from the other computer. But in this case it wouldn't be load-balanced in all cases -- i.e., only if the files are cached by the slave. You could use a third computer as an NFS server -- this would allow you to scale to more web-servers.
A central NFS server might just be your best solution as you can do without a write-master as every front-end server can perform writes. This is the approach I would use unless I am thinking of going past the peta-byte range :P
I would like to know how people dealing with logging across multiple web servers. E.g. Assume there are 2 webservers and some events during the users session are serviced from one, some from the other. How would you go about logging events from the session coherently in one place (without e.g.creating single points of failure)? Assuming we are using: ASP.Net MVC, log4net.
Or am I looking at this the wrong way - should I log seperately and then merge later?
Thanks,
S
UPDATE
Please also assume that the load balancers will not guarantee that a session is stuck to one server.
You definitely want your web servers to log locally rather than over a network. You don't want potential network outages to prevent logging operations and you don't want the overhead of a network operation for logging. You should have log rotation set up and all your web servers clock's synced. When log rotation rolls your log files over to a new file, have the completed log files from each web server shipped to a common destination where they can be merged. I'm not a .net guy but you should be able to find software out there to merge IIS logs (or whatever web server you're using). Then you analyze the merged logs. This strategy is optimal except in the case that you need real-time log analysis. Do you? Probably not. It's fairly resilient to failures (assuming you have redundant disks) because if a server goes down, you can just reboot it and reprocess any log ship, log merge or log analysis operations that were interrupted.
An interesting solution alternative:
Have 2 log files appenders
First one in the local machine
In case of network failure you'll keep this log.
Second log to a unix syslog service remotely (of course
a very consistent network connection)
I used a similar approach long time ago, and it work really well, there are
a lot of nice tools for analyzing unix logs.
Normally your load balancing would lock the user to one server after the session is started. Then you wouldn't have to deal with logs for a specific user being spread across multiple servers.
One thing you could try is to have the log file in a location that is accessible by all web servers and have log4net configured to write to it. This may be problematic, however, with multiple processes trying to write to the same file. I have read about NLog which may work better in this scenario.
Also, the log4net FAQ has a question and possible solution to this exact problem