Is it possible to limit replication on OpenStack Swift to one or possibly 2? 3 data replicas are currently being created.
The default replication in Swift is 3 replicas, but you can change that to any number greater or equal to 1. The swift-ring-builder command has a set-replicas verb for adjusting the number of replicas; see
https://docs.openstack.org/swift/xena/admin/objectstorage-ringbuilder.html#replica-counts
Of course, there is a tradeoff between the number of replicas and your cluster's ability to cope with loss of disk drives and servers.
You also asked this:
If I don't want replication on Openstack Swift(replication equal to 1), can I achieve this by disabling replicators? sudo swift-init object-replicator stop If I may, is this a better option?
I guess it would probably work ... until someone or something restarts the replicators.
But it is not a good idea:
Monitoring may complain that the replicators are not working(!)
Other swift tools may complain that the cluster is out of spec; e.g. swift-recon --replication.
If you have been previously running with (say) a 3 replica policy, then nothing will delete the (now) redundant replicas created before you turned off the replicators.
Related
Let us take some kind of distributed highly available database like Redis, Cassandra, Mongo etc. Replication factor = 5. Read quorum = 3. Write quorum = 3. Now a write occurs, but only 3 out of 5 get updated. So it is a success. But it so happens that one of the 3 successful updates crashes, and the other two are not yet updated. Disk, CPU is fffuttt. Now we have 2 out of 4 successfully updated. How will this be handled? We cannot achieve read quorum of 3/5 anymore. Should this be considered a case of lost data?
Now I understand that in usual case, it is considered that there is a software crash, and the dying instance will restart and everything will be good. This is a case of permanent hardware failure.
I think you are assuming that you have 5 machines in the cluster with the replication factor of 5. That means that every machine holds the same data. That's a very unusual setup.
I'm not sure about Redis or Mongo but in the Cassandra the read quorum will be achieved because more then 3 replicas are alive (4/5 actually).
In Cassandra there is a mechanism that will try to make the data consistent during the read. It's called read repair.
https://cassandra.apache.org/doc/latest/cassandra/operating/read_repair.html
We have a big application that uses 40 microservices (Spring boot) that each have about 5 database connections to a mariadb server. That causes too many connection errors on our mariadb server. Default is 151 however I was thinking of just setting the max connections to 1000 to be on the safe side. I cant find anywhere on the Internet if this is possible or even wise. Our MariaDB is running standalone on a VPS with 8GB memory. It is not running in a docker container or something like that. It is run directly on the VPS.
What is the maximum connections advisable taking into consideration that we might scale up with our microservices?
You can scale up your max_connections just fine. Put a line like
max_connections=250
in your MariaDB my.cnf file. But don't just set it to a very high number; each potential connection consumes RAM, and with only 8GiB you need to be a bit careful about that.
If you give this command you'll get a bunch of data about your connections.
SHOW STATUS LIKE '%connect%';
The important ones to watch:
Connection_errors_max_connections The number of connection attempts that failed because you ran out of connection slots.
Connections The total number of connections ever handled
Max_used_connections The largest number of simultaneous connections used.
Max_used_connections_time The date and time when the server had its largest number of connections.
The numbers shown are cumulative since the last server boot or the most recent FLUSH STATUS; statement.
Keep an eye on these. If you run short you can always add more. If you have to add many more connections as you scale up, you probably will need to provision your VPS with more RAM. The last two are cool because you can figure out whether you're getting hammered at a particular time of day.
And, in your various microservices be very careful to use connection pools of reasonable maximum size. Don't let your microservices grab more than ten connections unless you run into throughput trouble. You didn't say what client tech you use (nodejs? dotnet? php? Java?) so it's hard to give you specific advice how to do that.
I am having problems trying to build a database solution for the experiment to ensure HA and performance(sharding).
Now, I have a spider node and two galera clusters (3 nodes in each cluster), as shown in the figure below, and this configuration works well in general cases.:
However, as far as I know, when the spider engine performs sharding, it must assign primary IP to distribute SQL statements to two nodes in different Galera clusters.
So my first question here is:
Q1): When the machine .12 shuts down due to destruction, how can I make .13 or .14(one of them) automatically replace .12?
The servers that spider engine know
Q2): Are there any open source tools (or technologies) that can help me deal with this situation? If so, please explain how it works. (Maybe MaxScale? But I never knew what it is and what it can do.)
Q3): The motivation for this experiment is as follows. An automated factory has many machines, and each machine generates some data that must be recorded during the production process (maybe hundreds or thousands of data per second) to observe the operation of the machine and make the quality of each batch of products the best.
So my question is: how about this architecture (Figure 1)? or please provides your suggestions.
You could use MaxScale in front of the Galera cluster to make the individual nodes appear like a combined cluster. This way Spider will be able to seamlessly access the shard even if one of the nodes fails. You can take a look at the MaxScale tutorial for instructions on how to configure it for a Galera cluster.
Something like this should work:
This of course has the same limitation that a single database node has: if the MaxScale server goes down, you'll have to switch to a different MaxScale for that cluster. The benefit of using MaxScale is that it is in some sense stateless which means it can be started and stopped almost instantly. A network load balancer (e.g. ELB) can already provide some form of protection from this problem.
I am running nginx on a kubernetes cluster having 3 nodes.
I am wondering if there is any benefit of having for example 4 pods and limit their cpu/mem to approx. 1/4 of the nodes capacity vs running 1 pod per node limiting cpu/mem so that pod can use resources of the whole node (for the sake of simplicity, we leave cubernet services out of the equation).
My feeling is that the fewer pods the less overhead and going for 1 pod per node should be the best in performance?
Thanks in advance
With more then 1 Pod, you have a certain high availability. Your pod will die at one point, and if it is behind a controller (which is how is must be), it will be re-created, but you will have a small downtime.
Now, take into consideration that if you deploy more then one replica of your app, even though you give it 1/n resources, there is a base image and dependencies that are going to be replicated.
As an example, let's imagine an app that runs on Ubuntu, and has 5 dependencies:
If you run 1 replica of this app, you are deploying 1 Ubuntu + 5 dependencies + the app itself.
If you are run 4 replicas of this app, you are running 4 Ubuntus + 4*5 dependencies + 4 times the app.
My point is, if your base image would be big, and you would need heavy dependencies, it would be not a linear increase of resources.
Performance-wise, I don't think there is much difference. One of your nodes will be heavily bombed as all your requests will end up there, but if your nodes can handle it, there should be no problem.
What you are referring to is the difference between horizontal and vertical scaling. Regarding vertical scaling, you would increase the resources of your application as you see fit. Otherwise, you would scale horizontally by increasing the amount of replicas of your application.
Doing one or the other depends on features that you application may or may not have. In the case of nginx scaling horizontally would split traffic per pod and also per node which would result in a better throughput for your most likely reverse proxy.
I have a Mariadb Galera cluster with 2 nodes and it is up and running.
Before moving to production, I want to make sure that if a node crashes abruptly, It should come up on its own.
I tried using systemd "restart", but after killing the mysql process the mariadb service does not come up, so, is there any tool or method, that I can use to automate bringing up the nodes after crashes?
Galera clusters needs to have quorum (3 nodes).
In order to avoid a split-brain condition, the minimum recommended number of nodes in a cluster is 3. Blocking state transfer is yet another reason to require a minimum of 3 nodes in order to enjoy service availability in case one of the members fails and needs to be restarted. While two of the members will be engaged in state transfer, the remaining member(s) will be able to keep on serving client requests.
You can read more here.