I was working though a problem and noticed some code where a previous programmer was passing messages using the standard convention of PID ! Message. I have been using gen_server:cast/2. I was wondering if somebody could explain to me the critical differences and considerations when choosing between the two?
There are a few minor differences:
Obviously, the gen_server handles casts in handle_cast and "normal" messages in handle_info.
A cast never fails; it always returns ok. Sending a message with ! fails with badarg if you are sending a message to an atom that is currently not registered by a process. (Sending a message to a pid never causes an error, even if the process is dead.)
If the gen_server is running on a remote node that is not currently connected to the local node, then gen_server:cast will spawn a background process to establish the connection and send the message, and return immediately, while ! only returns once the connection is established. (See the code for gen_server:do_send.)
As for when to choose one or the other, it's mostly a matter of taste. I'd say that if the message could be thought of as an asynchronous API function for the gen_server, then it should use cast, and have a specific API function in the gen_server callback module. That is, instead of calling gen_server:cast directly, like this:
gen_server:cast(foo_proc, {some_message, 42})
make a function call:
foo_proc:some_message(42)
and implement that function like the direct cast above. That encapsulates the specific protocol of the gen_server inside its own module.
In my mind, "plain" messages would be used for events, as opposed to API calls. An example would be monitor messages, {'DOWN', Ref, process, Id, Reason}, and events of a similar kind that might happen in your system.
In addition to legoscia post I would say that it is easier to trace dedicated function API than messages. Especially in prod environment.
Related
I want to pull messages off a MQS queue in a C client, and would love to do so asynchronously so I don't have to start (explicitly) multithreading. The messages will be forwarded to another system that acts "transactionally" but is completely incompatible with XA. So I'd like to have a way to explicitly commit (and thereby remove) a message that's been successfully handed off to the other system, and not commit if this failed, so that the last message is retained for a more successful later attempt.
I've read about the SYNCPOINT option and understand how I'd use that around a regular GET, but I haven's seen any hints on how to make asynchronous message retrieval have transactional behavior like this. Any hints, please?
I think you are describing using the asynchronous callback capability, ie you register a routine to be called when a message arrives, and ask for any get to be under syncpoint... An explanation of how some of it works is in here, https://share.confex.com/share/117/webprogram/Handout/Session9513/share_advanced_mqi.pdf page 4+
Effectively you get called with the MQ message under syncpoint, do your processing with another system, then commit or rollback the message before returning.
Be aware without the use of e.g. XA 2 phase commit, there is always going to be the windows of e.g. committing to the external system and a power outage means the message under the unit of work gets rolled back inside MQ as you didnt have time to perform the commit.
Edit: my misunderstanding, didn't realise that the application was using a callback to retrieve messages, which is indeed fully asynchronous behavior. Disregard the answer below.
Do MQGET with MQGMO_SYNCPOINT, then issue either MQCMIT or MQBACK.
"Asynchronous" and "synchronous" may be misnomers - these are your patterns of using MQ - whether you wait for a reply message or not, these patterns do not affect how MQ processes your calls. Transaction management (unit of work management) works across any MQI calls that use SYNCPOINT, no matter if they are part of a request/reply pattern or not.
I'm not sure if I'm doing this right.
Our orchestration looks like this:
ReceiveOrder
TryScope (Long Running)
AcknowledgementScope (Atomic)
ConstructOrderAckMessage
TransformOrderToAck (using a map)
SendOrderAckToMessageQueue
AtomicWebServiceScope
ImportOrderToDBExpression
Construct and send message to another process
CatchException
ConstructErrorExpression
HandleExceptionStartOrchestration
When we tested this with about 6000 orders, we noticed that all of them resulted in an acknowledgment message (SendOrderAckToMessageQueue). The acknowledgment is a simple XML based on a schema provided by the crew that sends the order to this orchestration.
However, not all of them got imported into the database (ImportOrderToDBExpression) (about 45). In fact, there are no errors or failures or suspended instances of any kind. There's nothing unusual about the orders that did not get imported. If it failed, it did so silently.
Please note, that the AcknowledgementScope portion is something added recently; prior to that all the orders got imported successfully.
Is this because I have the Scope set incorrectly in this orchestration? Where else could the problem be? Is there a better way to send acknowledgment in a fool proof way? Thanks for any advice.
You don't mention any Catch Blocks. Do you have Catch Blocks on all your Scopes?
If there is an Exception without a Catch Block or a Catch Block that does not log the Exception, it will appear to silently fail.
Yes, the main thing you are doing wrong is calling an external DLL to insert records into a database.
Unless that DLL is very well written to be multi-threading capable including limiting the number of concurrent connections and has good retry and error handling capabilities then it can encounter an error and silently fail.
Even if you do have errors being logged in the DLL to the Event Log, you then have to give permissions for the Application name that the DLL uses to write to the event logs, otherwise the DLL will fail in it's catch blocks trying to write to the event log.
What you should be doing is using a Send Port with the appropriate Adapter to send records to the database.
Also there are very few situations in which you need an atomic scope. With an atomic scope it is up to the developer to implement any rollback. Also you probably do not need a long running scope unless you expect your Orchestration to take a long while and that is should dehydrate while waiting for a response.
Sending the Acknowledgement after the BizTalk Orchestration has received the message is fine, as long as you can then somehow resume a failed message in BizTalk, so you need to have some sort of retry mechanism.
I'm using Rebus SQLTransport with XML serialized messages for integration with SQL Server. Messages represent changes done in SQL Server. Because of that the order of message delivery is essential.
It is because for example message1 may contain object that is referenced (by id) in message2. Another example is that message1 may contain remove request of some object that is required to accept new object from message2.
Aggregating messages into one message would be quite complicated because messages are generated by triggers.
Having message idempotence and one worker I guess that would work except the fact that won't work if error happens and message will be moved to error queue. The error is quite possible to happen because of validation or business logic exception. Because of that I believe only human can fix the problem with message and until that time other messages should not be delivered. So I wanted to ask for advice what would be best to do in that situation. As far as I saw retry number cannot be set to infinity so should I stop the service inside of handler until problem is solved by human?
Thanks in advance
If it's important that the messages are processed in order without any "holes", I suggest you assign a sequence number to each message.
This way, if the endpoint gets a message whose sequence number is greater than the expected sequence number it can throw an exception, thus preventing out-of-order messages to be processed.
I would only do this if errors are uncommon though, and only if the message volume is fairly small.
If in-order processing is required, a much better design would be to use another message processing library that supports a pull model, which I think would fit your scenario much better than Rebus' push model.
I see a lot of code do this kind of thing:
log.Fatal(http.ListenAndServe(":8080",nil))
I have never seen the http server return an error, but I would like to understand more about the possibility what types of failure scenarios I can encounter.
Will it stop if my handler code panics? Will it only return in the case where it can't bind initially?
What is the best practice for making sure the server stays active to the highest degree possible?
Will it stop if my handler code panics?
No. The built-in http server recovers from panics and logs them.
Will it only return in the case where it can't bind initially?
That's the usual reason, though it's possible that some other error could force the listener to close.
What is the best practice for making sure the server stays active to the highest degree possible?
If the http server's main loop exits, it's likely a fatal error that you don't want to try and recover from. If that call returns when trying to bind the address, you could check for the address in use error, then wait and try again.
// Example to show the error types and fields on a *nix system.
// You should check the validity of these assertions if you don't want to
// panic. Due to the fact that this reaches into syscall, it's probably
// not much better than checking the string for "address already in use"
if err.(*net.OpError).Err.(*os.SyscallError).Err == syscall.EADDRINUSE {
fmt.Println("Address in use")
}
Remember that ListenAndServe is actually 2 calls, and you can separate them yourself. It creates a net.Listener, then gives it to an http.Server's Serve(l net.Listener) method.
It may be worth noting that the go package documentation for log.Fatal says
Fatal is equivalent to Print() followed by a call to os.Exit(1).
Also, the documentation for http.ListenAndServe explains that
This function will block until the program is terminated.
So the log.Fatal call will only be made if the http listener is killed for some reason.
A panic will be handled, as the answer above says.
The http.ListenAndServe function should continue indefinitely until the underlying socket connection is broken. At that point log.Fatal will print any message returned by the listener, before calling the program to exit completely with an error code (1).
So the "best practice" to keep the listener alive would be to do anything that keeps the socket connection alive. At that stage, it's more likely to be a network consideration.
Good day guys!. I'm currently working on a system using JMS queues that send message over SMPP (using Logica SMPP library).
My problem is that I need to attach an internal id (that we manage within our system) to the message sequence id so that when in async mode I receive a response, the proper action can be taken for that particular message.
The first option I tried to implement was the use of optional parameters, as established for SMPP 3.4. I do not receive the optional parameters in the response (I've read that the response attaches the optional parameters depending on the provider).
A second approach was to keep a mapping in memory for those messages until their response is received (it saturates the memory, so it is a no go).
Can anyone else think on a viable solution for correlating an internal system ID of a message to its sequence number within an asynchronous SMPP environment?
Thank you for your time.
You need to keep a map of seq_nr - internal message id and delete from this map as soon you get an async response back from SMSC.
It should not saturate the memory as it will keep only inflight messages but you need to periodicaly iterate over the map and delete orphaned entries (as sometimes you will not get an reponse back from smsc).