I've developed a channel in Mirth that sends an ORU message. The ACK will be then sent back asynchronously to a different channel on a specific port.
In order to be able to resend the ORU message in case an AR or AE is received back in the ACK I need to store this ORU somewhere to get access to it later when the ACK is received (remember it is asynchronous).
I am figuring out how to achieve this. My idea looks like this:
send ORU message and store it in a database
in the other channel wait for incomming ACKs
for an incomming ACK look for the related ORU in the database and depending if the ACK was positive or not, remove the ORU or resend it again
It would be nice if someone of you has some experience with it and can tell me if this is a proper way to do it and if not, how.
Case the idea is good, how should I implement the third step? I have already tried with a single channel but I cannot manage to resend the ORU.
I think your approach is reasonable. We tend to shy away from hard deletes and instead mark the message as "acknowledged" with the time the ACK (or NACK) message arrived. This gives you the ability to query for ACK/NACK/NULL and gives you a history of responses.
We keep track of our ORU by sampleId/timestamp and make note of the messageID. The messageID comes back in the ACK/NACK so it's easy to match up.
You probably don't want to resend (automatically) NACK'ed messages because what-ever caused the NACK in the first place (like a format issue) will not likely be resolved with a simple resend. Instead you may want a NACK to trigger an alert (like an email) to some group that can make the fix and resend manually.
You'll have to decide if you can safely resend missing "ACKs" and how long you should wait. Those are more business decisions than technical ones. For instance, you didn't receive an ACK/NACK, but was that because of network hiccup, an issue with Mirth, or did your original message actually not go through? If the receiving system is counting on 1 and only 1 message, then you'll need some way to reconcile before resending those messages that received no ACK.
Related
The description of the questions goes like this:
Someone recorded all the IP packets of a TCP connection between a client and a server for 30 minutes. In the record, he didn't find any packet that was ACK-only. How is this possible?
What is claimed to be a possible solution: For all the record time, the server sent data to the client, which the client processed, but he didn't send any data back to the server.
I am having trouble understanding how can it be possible.
From what I see, since the client didn't send any data to the server, and there weren't any ACK-only packets in the record, then the server didn't get any ACK from the client. Logically, I would think that since no ACK is received by the server, it will always do re-transmit. But also, since the server doesn't get anything from the client for 30 minutes, which seems like a long time for me, it will conclude that the connection is broken and stop it. (maybe even send an ACK only, but I am not sure about it).
Moreover, from what I know, when using keepalive, the sender gets and ACK-only packet from his peer.
Can anyone help me understand this?
Help would be appreciated
Perhaps more details would be helpful here. What kind of server/client? What protocol is being used and for what purpose?
Is the connection running as expected and this is just viewed as strange traffic you are trying to understand or is the connection timing out?
Some devices or softwares can be set to a "No ACK" state which means that no ACKs are sent nor are they expected.
One reason for this is usually bandwidth. ACKs do consume bandwidth and there are cases where bandwidth is such a great premium that packets being lost is preferable to bandwidth being consumed by ACKs. That type of traffic would probably do better with a UDP protocol but that is a completely different topic.
Another reason is that you don't care if packets are lost. Again, this would be better off as UDP instead of TCP, but someone may be trying to work within strange parameters is bending the rules about what type of traffic to advertise as in order to get around some issue.
Hopefully this is helpful, but if it does not apply, then please put in more details about the connection so that we can better understand what may be happening.
I would like to ask how does Rebus HTTP Gateway acknowledge message delivery so when OutboundService sends the message how it knows it can commit or rollback the transaction.
Intuitive answer would be that HTTP response acknowledges it however looking at the code
https://github.com/rebus-org/Rebus/blob/5fef6b400feaf569e0d6517ad9ee3f6da2f31820/src/Rebus.HttpGateway/Outbound/OutboundService.cs#L139
it seems no action is taken after reading the response.
Thanks in advance :)
It does a very simple "acknowledge" in the way that if no error occurs, then the message is assumed to have been delivered safely to the destination queue.
This means that the ubiquitous at least once-delivery guarantee holds across gateways as well, although the risk of receiving the same message twice will of course be greatly increased.
If it's important for you to process each message only once, you need to make your receiver idempotent - but that's generally the rule when you're doing messaging without distributed transactions, so it's no different from scenarios where there's no HTTP gateway involved.
I had to implement an application which in very short, sent packets every few seconds to a server, when the server received them it send a response to the client which only then proceeded to send another packet. This sounds all good but we were using TCP and the responses came as soon as the server got the packet and not post-processing or anything like that. So that makes me wonder, why would you do something like this? The client had a queue where I kept all the packets and did something like this:
try {
send packet // exception is thrown if connection is lost
remove packet from queue
} catch exception {
try to reconnect
}
so in this case the packet gets removed from the queue only if the send was successful.
Any idea about this? Is this best practice? I would appreciate if someone could clear this for me.
Thanks
One option would be to put the packets into a queue and send them. After sending move them into a "pending" queue. Once the other end has processed them, you mark them as completed. Then you are up against other problems. What if the other end processes them but the ack never gets to your end? This is a relatively researched problem and I suggest you research distributed transactions and two phase commit, if you need to be sure.
Sending isn't enough in some cases. If it's absolutely critical that the data you're shoving out the door MUST be received, then you should wait for an acknowledgement that the packet was received/processed by the remote end.
Even if the network-level stuff works perfectly and the packets arrive at the destination, that destination machine might still crash or otherwise lose the data. If you remove-on-send, then that data's gone. Waiting for acknowledgements from the remote end would at least give you the ability to resend packets that were corrupted/lost.
UDP package are like once gone you will never know that it is received or not. So it is not gurenteed weather package will be received. I have learned that mostly DNS use UDP package to send the request. How does the loss of DNS requested using UDP are handled?
If no response packet is received within a certain amount of time, the request is re-sent. Dan Bernstein suggests that most clients will retry up to four times.
There are two ways to decide that a UDP datagram has been lost. Neither is completely reliable.
The most common is a timeout. You send a message and wait for a response. If you don't get a response after some amount of time, you assume that either the message or the response were lost. At that point you can either try again, or give up. It is also possible that the message or response is just taking a really long time to get through the network, so you must account for duplicates. Note that all packet-switched communication, including TCP, works this way. TCP just hides the details for you.
Another method is to look for ICMP messages telling you that a packet has been dropped. For example, ICMP_UNREACH_PORT, ICMP_UNREACH_HOST, or ICMP_UNREACH_HOST_PROHIB. But not only are these messages rarely sent and subject to loss themselves, but you can sometimes receive them even when a message did get through successfully. At best, if you get an ICMP message you can think of it as a suggestion of what might have happened.
Most DNS implementations use a short timeout because duplication is no big deal. After a few repeats to one DNS server, it will try another one (assuming multiple servers are available). Most implementations will also cache information about which servers are responding and which are not.
Say my network connection drops for a few seconds and I miss some SignalR server-pushed messages.
When I regain network connectivity are the messages I missed lost? or does signalR handle them and push them out when I reconnect?
If it can't handle missed messages, then what is the recommended approach for ensuring consistency?
Periodically (2-3 mins) poll to check server-data?
Somehow detect loss of network on the client side and do an ajax call to get the data on network restoration?
something else?
Here are a couple of thoughts:
If you aren't sending a lot of messages per second, consider sending no data in the messages themselves. Instead, the message is just a "ping" to the clients telling them to go get the server data when they can. Combine that with a periodic poll, as you said, and you can be assured that you won't miss messages. They just might be delayed.
If you are sending a lot of messages quickly, how about adding a sequential ID to each one? Think of a SQL Identity column. Your clients would need to keep track of the most recent ID received. After a network reconnect, the client could ask for all messages since [Last ID]. If a message is received whose ID is not contiguous with the most recently received, you know that there was a disconnect and can ask the server for the missing information.