I know that the leader's response is required, but are other nodes required to respond in order for the operation to succeed?
Depends on what do you mean with "leader's response". Leader accepts a request from a client and sends a response to the client when a new log entry is committed. It is totally possible that the request is committed, but the communication has failed to notify the customer. In that case the customer does not actually know the outcome, the they should act accordingly.
To under the question in the title: raft original paper (https://raft.github.io/raft.pdf) is pretty clear on that: "A log entry is committed once the leader
that created the entry has replicated it on a majority of
the servers " [5.3]. In other words, majority has to accept the new entry for it to be accepted, which is equivalent to "operation succeeded".
Actually the leader response is not required for a record to be committed to the log. Leader
Related
I've been working on some ideas to determine the initiator of a transaction in Corda. It's quite hard to prove since transactions can be signed by one (trivial to prove), or more (hard to prove) parties.
However I'm now wondering if there's something inherent to the transaction itself that can be used to trivially determine who initiated a transaction.
When a transaction that requires multiple signatures is created, we call the CollectSignaturesFlow to obtain counter-party signatures, however if we do this before signing the transaction ourselves, we get the following exception:
The Initiator of CollectSignaturesFlow must have signed the transaction.
Therefore I could assume that the first signature in the transaction must have some from the initiator. Is it safe to assume that SignedTransaction.sigs[0] is the transaction initiator?
Footnote 1: The code that throws the above exception is implemented as such:
require(partiallySignedTx.sigs.any { it.by in myKeys }) {
"The Initiator of CollectSignaturesFlow must have signed the transaction."
}
This would suggest that one (or specifically more) signatures applied to the transaction will belong to the initiator.
Footnote 2: Would transaction storage (serialization/deserialization) have any impact on the ordering of the sigs list? - If it's order isn't deterministic, then we cannot rely on sigs[0] being the initiator.
Phoned a friend from r3, from Alex Koller
I looked at the question and it's not very clear to me what is Matthew trying to achieve. AFAIK corda doesn't record who initiated a transaction and trying to devise that from the order of the signatures on the tx is probably prone to error now or in the future. A possibility would be to look at the notary and see who requested the notarisation. But that may be suitable in minority of use cases. If the holder of the tx should need to know who initiated it then it may have to be recorded in one of the states. But you obviously cannot prove the correctness of the value in the state from the tx contracts, so it would have to be atested to by the other signers on the tx (in the collect signature responding flow, before the counterparties sign). I think we would need to know more about what he's doing to be able to advise. He should get in touch with Professional Services.
We are applying the request/reply semantics using spring-kafka with the ReplyingKafkaTemplate.
However, we noticed that sometimes the reply does not end up where it should.
Here is a rough description of our setup:
Service A
2 instances
consuming messages from topic-a which has 2 partitions. (each instance gets 1 partitions assigned).
Service A is the initiator.
Service B:
2 instances, consumes messages from topic-b, which also has 2 partitions.
Reacts to incoming messages from A and returns a reply message using the #SendTo annotation.
Observed behaviour:
When an instance of service A, e.g. A1, is sending a message to service B the send fails with a reply timeout. The request is consumed by B successfully and a reply is being returned, however it was consumed by the other instance, e.g. A2. From the logging I can see that A1 get topic-a-0 assigned, whereas A2 gets topic-a-1 assigned.
Suggestions from the docs:
Our scenario is described in this section of the docs: https://docs.spring.io/spring-kafka/reference/html/#replying-template
It gives a couple suggestions:
Give each instance a dedicated reply topic
Use reply partition header and use dedicated partitions for each instance
Our setup is based on a single topic for the whole service. So all incoming events and reply events are send to this and consumed from this topic. So option #1 is not desirable in our situation.
The downside of option #2 is that you cannot use the group management feature, which is a pitty because our services run on Kubernetes so we'd like to use the group management feature for maximum flexibility.
A third option?
So I was wondering if there was a third option:
Why not use group management and determine the assigned topic partitions of the reply container at runtime on the fly when sending a message and set the reply partition header.
It looks like the ReplyingKafkaTemplate#getAssignedReplyTopicPartitions method provides exactly this information.
This way, the partitions are not fixed and we can still use the group management feature.
The only downside I can foresee is that when the partitions are rebalanced after the request was sent but before the reply was received, the request could fail.
I already have tested something to see if it works and it looks like it does. The main reason for me to post this question is to check if my idea makes sense, are there any caveats to take into account. I'm wondering why this is not supported by spring-kafka out of the box.
If my solution makes sense, I am willing to raise an enhancement issue and provide a PR on the spring-kafka project.
The issue, as you describe, is there is no guarantee we'll get the same partition(s) after a rebalance.
The "third option" is to use a different group.id for each instance and set sharedReplyTopic=true. In this case all instances will get the reply and it will be discarded by the instance(s) that did not send the request.
The best solution, however, is to use a unique reply topic for each instance.
I am using DCMTK storescp.exe to receive images from a CR modality and then process/save them in my DB.
Is it possible to use other DCMTK binary to manually send the PatientName and PatientId to CR modality before the patient goes there?
I have read somewhere that the modality makes a query to get the Modality Worklist. I would like to reverse that flow. I want to directly send the Modality Worklist to the modality, whenever I like, without receiving the query from Modality.
Is that possible? If yes; how can I do that with DCMTK?
Please note that this is not an off-site tool request. I just want to know the DCMTK binary that implements required DICOM service/command.
You are looking for Modality Worklist or MWL service which implements C-FIND command.
SOP Class: 1.2.840.10008.5.1.4.31 [Modality Worklist Information Model – FIND].
But it does not work the way you are expecting; and it should not - for good.
MWL SCU (in your case - CR) initiates the query with the (optional) filters it suits. As usual, association happens and MWL SCP receives the MWL Request. It then fetch the data from its database matching the filters if any. It then sends one MWL Response for each row fetched from database, status for each response is PENDING. When all the rows are transferred, final SUCCESS response is sent. If no rows were found matching the filter, only final response is sent. If something goes wrong, proper failure response is sent. SCU then, sends the Release Request and on receiving Release Response, it closes the association.
Now, why your expected workflow is not possible?
Generally MWL SCP is implemented by RIS systems. These systems have tools/features to register the patient demographic data while/before admission of the patient in hospitals. They also have features to schedule the orders to be executed by Modalities. There might be multiple modalities in given DICOM Network (hospital). Though, RIS have a way to decide which order should go to which modality (based on AE Title if configured and used properly), they cannot push it because they are acting as SCP i.e. Server. As any other server in any network protocol, they have to wait for request from client i.e. SCU.
Further, though SCP may know which order should be sent to which modality, modality may not expecting that order for many reasons. So, the general flow in MWL is the way I explained above. You cannot implement your reverse workflow with any DICOM service/command.
Just for the sake of clarity:
All this has nothing to do with the data you received and stored in DB using storescp.exe. I mean, you do not generally send that data to modality as Modality Worklist.
MWL happens first. When modality get the MWL Worklist item, it conducts the study and acquires images with the demographic data received in MWL Worklist item. This way, errors are avoided, redundant inputs are avoided and flow is bit automated. When done, modality push (C-STORE) the instances (CR images in your case) to C-STORE SCP which is storescp.exe in your case.
Continuous stream of objects is saved in the Firebase.
They are HTTP traffic representation so they are ordered and paired from business perspective.
Each object has unique timestamp, should it be used as a key?
How to query/monitor such path for newly created documents to be process?
How to efficiently process and verify data integrity:
That the responses and requests are one after another
Verify that responses saved after requests really belongs to that request (there are business rules that predict response content)
How processed documents should(?) be marked to does not slow down process that waits for documents to process.
When request/response pair is verified is should be put together in a new object. How to save it in a non blocking way?
When response does not arrive until timeout a new half-filled object should be created and saved. How to monitor if a matching request arrives to update it?
How Firebase should be configured: indexes, paths (other?) to make processing efficient?
Here is my scenario. BizTalk needs to transfer a file from a shared/central document library. First BizTalk receives an incoming message with a reference/path to this document in the library. Then it simply needs to read it out from this library and send it (potentially through different adapters). This is in essence, a scenario not so remote from the ClaimCheck EAI pattern.
Some ways to implement a claim check have been documented, noticeably BizTalk ESB Toolkit Claim Check, and BizTalk 2009: Dealing with Extremely Large Messages, Part I & Part II. These implementations do however take the assumption that the send pipeline can immediately read the stream that has been “checked in.”
That is not my case: the document will take some time before it is available in the shared library, and I cannot delay the initial received message. That leaves me with 2 options: either introduce some delay via an orchestration or ensure the send port will later on retry if the document is not there yet.
(A delay can only be introduced via an orchestration, there is no time-based subscriptions in BizTalk. Right?)
Since this a message-only flow I’d figure I could skip the orchestration. I have seen ways on how to have "Custom Retry Logic in Message Only Solution Using Pipeline" but what I need is not only a way to control the retry behavior (as performed by the adapter) but also to enforce it right from within the pipeline…
Every attempt I made so far just ended up with a suspended message that won’t be automatically retried even though the send adapter had retry configured… If this is indeed possible, then where/what should I do?
Oh right… and there is queuing… but unfortunately neither on premises nor in the cloud ;)
OK I may be pushing the limits… but just out of curiosity…
Many thanks for your help and suggestions!
I'm puzzled as to how this could be done without an Orch. The only way I can think of would be along the lines of:
The receive port for the initial messages just 'eats' the messages,
e.g. subscribing these messages to a dummy Send port with the Null Adapter,
ignoring them totally.
You monitor the Shared document library with a receive port, looking for any ? any new? document there.
Any located documents are subscribed by a send port and sent downstream.
An orchestration based approach would be along the lines of:
Orch is triggered by a receive of the Initial notification of an 'upcoming' new file to the library. If your initial notification is request response (e.g. exposed web service, you can immediately and synchronously issue the response)
Another receive port is used to do the monitoring of availability and retrieval of the file from shared library, correlating to the original notification message (e.g. by filename, or other key)
A mechanism to handle the retry if the document isn't available, and potentially an eventual timeout, e.g. if the document never makes it to the shared library.
And on success, a send port to then send the document downstream
Placing the delay shape in the Orch will offer more scalability than e.g. using Thread.Sleep() or similar in custom adapter or pipeline code, since BTS just calculates ad stamps the 'awaken' timestamp on the SQL record and can then dehydrate the orch, freeing up the thread.
The 'is the file there yet?' check can be done with a retry loop, delaying after each failed check, with a parallel branch with a timeout e.g. after an hour or so.
The polling interval can be controlled in the receive location, so I do not understand what you mean by there is no time based subscriptions in Biztalk. You also have a schedule window.
One way to introduce delay is to send that initial message to an internal webservice, which will simply post back the message to Biztalk after a specified time interval.
There are also loopback adapters, which simply post the message back into the messagebox. This can be ammended to add a delay.