I'm just getting into the concept of a Distributed System and its advantages and disadvantages. In the book I'm reading it discusses the complexity of a Distributed System and that they are inherently complex, it lists the following as potential reasons for complexity;
Heterogeneity
Asynchronous communication
Partial failures
What I am struggling to understand is what these concepts actually encompass (i.e what is a partial failure and what are the causes of a partial failure?), and how they are dealt with in modern systems? Does middleware successfully solve all three of these complexity issues within a system?
This question can be answered in many words, but I'll try to boil it down to essentials:
Heterogeneity is one of the main problems integration tries to solve. It is an inherent characteristic of most distributed systems and it refers to the fact that most often than not, when you have to integrate multiple systems, they will:
Be on different platforms, in different networks;
Differ in their capabilities in terms of integration;
Have disparities in data, even data referring to the same business domain;
Use and support different (sometimes even forgotten or unsupported) technologies and standards;
Have different owners (are controlled by different departments, companies).
All of the above add more and more complexity.
Asynchronous communication solves some problems of stateless communication but introduces whole other set of complexities, that can easily lead to problems when not implementation is not proper. This is mainly due to the fact that you only have guarantee that the message will be successfully received on the other end, but have no guarantee whatsoever when the operation will be processed, if ever. So it is much harder to carry out orchestration of interdependent asynchronous tasks, as opposed to synchronous tasks.
Partial failures - When you have processes that involve multiple interdependent write operations you need to ensure ACID transactions. Having to do so in scenarios when multiple systems are involved is even harder because you cannot achieve common transactional context as easily in heterogeneous distributed environment as you would if you were within the boundaries of a single system. Often you will need to implement opposite operations in services (or worse, implement two-phase commit), just to be able to compensate all prior writes in the process in case something goes wrong with one of the tasks.
Hope this clears things a bit!
The reason distributed systems are so complex is simple: time!
Perfect synchronization of state becomes impossible in distributed systems for the simple fact that some amount of time must pass between the point that a message leaves one server and that point it arrives at its intended destination. In addition to this, networks are a far more unreliable communication medium, meaning that message may never make it at all.
The lack of perfect time synchronization means that it's impossible to make absolute assumptions about the order of events. For instance, in a highly available distributed database, if two requests writing to the same resource arrive on two different servers nearly simultaneously, there's no way to determine the absolute order of those events. So, distributed systems must use approximations of logical time and conflict resolution to resolve these types of event order issues.
Partial Failure - In case of a transactionS involving many clients (#2or more),the scheduling technique being used many involve conflicting operations of a write and maybe a write, in the process of issuing a lock complexities arise like for in case of a deadlock. When the lock manager tries either to detect, avoid or prevent the system may partially fail leading to rollback of the whole process.
Related
The latest version of MPI includes these types of point to point (p-p) communication:
blocking
ordinary non-blocking
persistent non-blocking
partitioned non-blocking
As far as I know, historically blocking p-p communication was the first type that existed. Then different types of non-blocking p-p communication were introduced one after the other to increase performance. For example, they allow overlap of computation and communication. But are there cases where blocking p-p communication is actually faster than the non-blocking alternatives? If no, what does justify their existence? Simply backward compatibility and their simplicity of use?
It is a misconception that non-blocking communication was motivated by performance: it was mostly to be able to be able to express deadlock/serialization-free communication patterns. (Indeed, actually getting overlap of communication/computation was only possible through the "Iprobe trick". Only recently with "progress threads" has it become a more systematic possibility.)
Partitioned communication is intended for multi-threaded contexts. You may call that a performance argument, or a completely new use case.
Persistent sends have indeed the potential for performance improvement, since various setups and buffer allocations can be amortized. However, I don't see much evidence that MPI implementations actually do this. https://arxiv.org/abs/1809.10778
Finally, you're missing buffered, synchronized, and ready sends. These indeed have the potential to improve performance, though again I don't see much evidence that they do.
I'm doing some cursory research on distributed systems, relating to blockchain and consensus and I was wondering what the advantages of using an async system vs sync.
Is it faster? More secure? What reasons go into designing one system over another?
A synchronous distributed system isn't practical - you are assuming the network is perfect, and delivers messages within some time bound. If you can make that assumption, perhaps in a datacenter network, then designing a synchronous distributed algorithm will be much cheaper than an asynchronous equivalent. However, as soon as the assumption breaks, you must use an asynchronous algorithm, lest you lose the safety properties you want.
I find it difficult to convince myself the advantage of using complex design like DynamoDB over simple duplication strategy.
Let's say we want to build a distributed key/value data store over 5 servers. (each server has exactly the same duplica).
Eventual consistency system, like DynamoDB, typically uses complicated conflicts reconcile, vector timestamp, etc. to achieve eventually consistency.
But instead, why couldn't we simply do the following:
For write, client will issue the write command to all the servers. So all servers will execute the clients' write command in the same order. It will reply to clients before servers commit the write.
For read, client will just do a round robin, only one server at a time will take care of read command. (Other servers won't see the read command)
Yes, client may experience temporary stale data, but eventually all replica will have the same dataset, which is the same semantic as DynamoDB.
What's the disadvantage of this simple design vs Complicated DynamoDB?
Your strategy has a few disadvantages, but their exact nature depends on details you haven't covered.
One obvious example is dealing with network segmentation. That is, when one part of your network becomes segmented (disconnected) from another part.
In this case, you have a couple of choices about how to react when you try to write some data to the server, and that fails. You might just assume that it worked, and continue as if everything was fine. If you do that, and the server later comes back on line, a read may return stale data.
To prevent that, you might treat a failed write as a true failure, and refuse to accept the write until/unless all servers confirm the write. This, unfortunately, makes the system as a whole quite fragile--in fact, much more fragile (at least with respect to writing) than if you didn't replicate at all (because if any of the servers go off-line, you can't write any more). It also has one more problem: it limits write throughput to the (current) speed of the slowest server, so even if they're all working, unless they're perfectly balanced (unlikely to happen) you're wasting capacity.
To prevent those problems, many systems (including Paxos, if memory serves) use some sort of "voting" based system. That is, you attempt to write to all the servers. You consider the write complete if and only if the majority of servers confirm that they've received the write. Likewise on a read, you attempt to read from all the servers, and you consider a value properly read if and only if the majority of servers agree on the value.
This way, up to one fewer than half the servers can be off-line at any given time, and you can still read and write data. Likewise, if you have a few servers that react a little more slowly than the rest, that doesn't slow down operations overall.
Of course, you need to fill in quite a few details to create a working system--but the fact remains that the basic concept is pretty simple, as outlined above.
In a Rebus service bus, there is a single message transport queue per endpoint. It is possible for an endpoint to handle more than one message, and it is possible to have only a single endpoint in a system.
Other than the throughput of messages, what reasons are there to use more than a single endpoint in a Rebus service bus system?
Excellent question! :) There can be many reasons why you might want to have several Rebus endpoints active at the same time.
An obvious reason is that you might want to host the endpoints in separate processes so you can update them independently of each other. But since this reason is pretty obvious, I assume you are thinking about reasons one might want to host multiple Rebus endpoints in the same process.
Let me just mention a few(*):
Concurrency requirements
One endpoint might be hosting data that experiences contention and therefore does not benefit from being able to process messages concurrently - this endpoint will probably have only a few threads and low parallelism, possibly 1/1.
Another endpoint might be doing stream-based data processing (e.g. loading blobs from one place into another, downloading data from web services, etc.), which can be done with very high throughput and low resource requirements with one single thread and a high level of parallelism - e.g. 1/20.
Yet another endpoint might be doing a lot of serialization/deserialization, which is usually CPU-bound, and therefore might benefit from running on a many-core box with many worker threads and matching parallelism - e.g. 10/10.
As you can see, the type of tasks performed by an endpoint can call for a configuration that matches the nature of the tasks.
SLAs
One endpoint might be designated for processing low-priority background stuff, like e.g. moving data to cold storage, optimizing storage of historic data, etc.
Another endpoint might be processing messages where low latency is the most important quality attribute.
If these two were using the same queue, the low-priority background stuff could sometimes clog up the queue, hindering low-latency processing of the other messages.
Logical separation
I have many times started out by hosting several Rebus endpoints in the same process because it was easy to deal with during development, while keeping the endpoints separate because they were implementing different business functions.
This way it is easy to physically break them apart some time later on, allowing for a higher degree of separation and independence.
(*) Udi Dahan works with the concepts "business components" and "autonomous components" where the first one is an implementation of a business capability and the second one is what business components are decomposed into, mostly for technical reasons.
I guess you could say that the first two reasons I mentioned are separate endpoints for "autonomous component" reasons, whereas the third is separation because things belong to different business components.
Udi keeps a pretty strict view of these concepts that is completely orthogonal to how the system is physically composed, but I almost always end up with pretty high convergence between logical separation and physical separation.
I have been researching asynchronous messaging, and I like the way it elegantly deals with some problems within certain domains and how it makes domain concepts more explicit. But is it a viable pattern for general domain-driven development (at least in the service/application/controller layer), or is the design overhead such that it should be restricted to SOA-based scenarios, like remote services and distributed processing?
Great question :). The main problem with asynchronous messaging is that when folks use procedural or object oriented languages, working in an asynchronous or event based manner is often quite tricky and complex and hard for the programmer to read & understand. Business logic is often way simpler if its built in a kinda synchronous manner - invoking methods and getting results immediately etc :).
My rule of thumb is generally to try use simpler synchronous programming models at the micro level for business logic; then use asynchrony and SEDA at the macro level.
For example submitting a purchase order might just write a message to a message queue; but the processing of the purchase order might require 10 different steps all being asynchronous and parallel in a high performance distributed system with many concurrent processes & threads processing individual steps in parallel. So the macro level wiring is based on a SEDA kind of approach - but at the micro level the code for the individual 10 steps could be written mostly in a synchronous programming style.
Like so many architecture and design questions, the answer is "it depends".
In my experience, the strength of asynchronous messaging has been in the loose coupling it brings to a design. The coupling can be in:
Time - Requests can be handled asynchronously, helping overall scalability.
Space - As you point out, allowing for distributed processing in a more robust way than many synchronous designs.
Technology - Messages and queues are one way to bridge technology differences.
Remember that messages and queues are an abstraction that can have a variety of implementations. You don't necessarily need to use a JMS-compliant, transactional, high-performance messaging framework. Implemented correctly, a table in a relational database can act as a queue with the rows as messages. I've seen both approaches used to great effect.
I agree with #BradS too BTW
BTW here's a way of hiding the middleware from your business logic while still getting the benefits of loose coupling & SEDA - while being able to easily switch between a variety of different middleware technology - from in memory SEDA to JMS to AMQP to JavaSpaces to database, files or FTP etc