I just did a quick test and seem that the published messages with bus.Publish does not get persisted in msmq until it has a subscriber.
Did I do something wrong in the configuration?
Is this by design? And Why?
Thanks
That is how publish/subscribe works with MSMQ - when a publisher publishes a message, it will look for the queue names of the subscribers in its subscription storage, and send a copy of the message to each subscriber.
It follows from this that if there's no subscribers, then no messages are actually sent.
Logically, it works the same way when using multicast-capable transports like e.g. RabbitMQ - with RabbitMQ, then message will be published to the broker, which will then distribute a copy of the message to each subscriber - and again, if there are none, the message will no be delivered to anyone and thus cannot be seen anywhere.
I hope that makes sense :)
Related
I have a lot of emails that were never sent in an app and queued.
I fixed a bug and they are all being sent but it's way too late right now.
Is there a way to clear the queue and delete all messages stored there ?
I'm currently working on an IRC-Chat and we want to add the option to chat with other people privately (User-To-User) which works fine, but the messages aren't stored, meaning that a user loses all private messages after disconnecting. They also can't message a person once they have disconnected.
All of this isn't the case with channels, where messages are stored for X time, allowing a asynchronous communication.
Is there a way of allowing asynchronous messaging for private User-To-User messages without storing the messages in an extra system? Or is this simply a limitation of IRC?
IRC isn't designed to store messages - it's a feature, not a bug. One way to get around this is to configure the server to 'simulate' past messages (essentially pretending to be the other user to send past messages). To do this, you would need to store messages - however, it would be on the server itself, not on a separate database or a third party.
We have a simple application, who upon every update of an entity sends out a notification to SNS(it could very well have been any other queuing system). Clients are listening to these notifications and they do a get of updated entity based on these notifications.
The problem we are facing is, when clients do a get, sometimes data is not replicated and we return 404 or sometimes stale data(even worse).
How can we mitigate this while sending notifications?
Here are Few strategies to mitigate this with pros and cons
Instead of sending notification from application send notification using database streams
For example dynamodb streams ans aws lambda. This pattern can be useful in the case of multiregion deployment as well. where all the subscriber, publisher will subscribe to their regional database streams. And also atomicity of sending message and writing to database is preserved. And we wont loose events in the case of regional failure.
Send delayed messages to your broker
Some borkers like activemq and sqs support this functionality, but SNS does not. A workaround for that could be writing to sqs queue which then writes to sns. This might be a good option when your database does not support streams.
Send special error code for retry-able gets
Since we know that eventual consistency is there we can return special error code to clients, so that they can retry based on this error code. The retry strategy should be exponential backoff. but this may mean giving away your problems to clients. Also we should have some sort of versioning in place.
Fetch from another region
If entity is not found in the same region application can go to another region or master database to fetch it. NOTE Don't do this. as it is an anti pattern. I am mentioning it here just for the sake of completion.
Send the full entity in message
If entities to be fetched by rest service is small and there are no security constrain around who can access what, we can send the full entity in message. This is ensure that client don't have to do explicit fetch of it every time a new message is arrived.
Scenario
I am building courier service system using Microservices. I am not sure of few things and here is my Scenario
Booking API - This is where customer Place order
Payment API - This is where we process the payment against booking
Notification API - There service is responsible for sending the notification after everything is completed.
The system is using event-driven Architecture. When customer places booking order , i commit local transaction in booking API and publish event. Payment API and notification API are subscribed to their respective event . Once Done Payment and notification API need to acknowledge back to Booking API.
My Questions is
After publishing the event my booking service can't block the call and goes back to the client (front end). How does my client app will have to check the status of transaction or it would know that transaction is completed? Does it poll every couple of seconds ? Since this is distributed transaction and any service can go down and won't be able to acknowledge back . In that case how do my client (front end) would know since it will keep on waiting. I am considering saga for distributed transactions.
What's the best way to achieve all of this ?
Event Sourcing
I want to implement Event sourcing to track the complete track of the booking order. Does i have to implement this in my booking API with event store ? Or event store are shared between services since i am supposed to catch all the events from different services . What's the best way to implement this ?
Many Thanks,
The way I visualize this is as follows (influenced by Martin Kleppmann's talk here and here).
The end user places an order. The order is written to a Kafka topic. Since Kafka has a log structured storage, the order details will be saved in the least possible time. It's an atomic operation ('A' in 'ACID') - all or nothing
Now as soon as the user places the order, the user would like to read it back (read-your-write). To acheive this we can write the order data in a distributed cache as well. Although dual write is not usually a good idea as it may cause partial failure (e.g. writing to Kafka is successful, but writing to cache fails), we can mitigate this risk by ensuring that one of the Kafka consumer writes the data in a database. So, even in a rare scenario of cache failure, the user can read the data back from DB eventually.
The status of the order in the cache as written at the time of order creation is "in progress"
One or more kafka consumer groups are then used to handle the events as follows: the payment and notification are handled properly and the final status will be written back to the cache and database
A separate Kafka consumer will then receive the response from the payment and notification apis and write the updates to cache, DB and a web socket
The websocket will then update the UI model and the changes would be then reflected in the UI through event sourcing.
Further clarifications based on comment
The basic idea here is that we build a cache using streaming for every service with data they need. For e.g. the account service needs feedback from the payment and notification services. Therefore, we have these services write their response to some Kafka topic which has some consumers that write the response back to order service's cache
Based on the ACID properties of Kafka (or any similar technology), the message will never be lost. Eventually we will get all or nothing. That's atomicity. If the order service fails to write the order, an error response is sent back to the client in a synchronous way and the user probably retries after some time. If the order service is successful, the response to the other services must flow back to its cache eventually. If one of the services is down for some time, the response will be delayed, but it will be sent eventually when the service resumes
The clients need not poll. The result will be propagated to it through streaming using websocket. The UI page will listen to the websocket As the consumer writes the feedback in the cache, it can also write to the websocket. This will notify the UI. Then if you use something like Angular or ReactJS, the appropriate section of the UI can be refreshed with the value received at the websocket. Until that happens user keeps seeing the status "in progress" as was written to the cache at the time of order creation Even if the user refreshes the page, the same status is retrieved from the cache. If the cache value expires and follows a LRU mechanism, the same value will be fetched from the DB and wriitten back to the cache to serve future requests. Once the feedback from the other services are available, the new result will be streamed using websocket. On page refresh, new status would be available from the cache or DB
You can pass an Identifier back to client once the booking is completed and client can use this identifier to query the status of the subsequent actions if you can connect them on the back end. You can also send a notification back to the Client when other events are completed. You can do long polling or you can do notification.
thanks skjagini. part of my question is to handle a case where other
microservices don't get back in time or never. lets say payment api is
done working and charged the client but didn't notify my order service
in time or after very long time. how my client waits ? if we timeout
the client the backend may have processed it after timeout
In CQRS, you would separate the Commands and Querying. i.e, considering your scenario you can implement all interactions with Queues for interaction. (There are multiple implementations for CQRS with event sourcing, but in simplest form):
Client Sends a request --> Payment API receives the request --> Validates the request (if validation fails throws error back to the user) --> On successful validation --> generates a GUID and writes the message request to Queue --> passes the GUID to the user
Payment API subscribes the payment queue --> After processing the request --> writes to Order queue or any other queues
Order APi subscribes to Order Queue and processes the request.
User has a GUID which can get him data for all the interactions.
If use a pub/sub as in Kafka instead of Kafka (all other subsequent systems can read from the same topic, you don't need to write for each queue)
If any of the services fail to process, once the services are restarted they should be able to pick where they left off, if the services are down in the middle of a transaction as long as they roll back their resp changes you system should be stable condition
I'm not 100% sure what you are asking. But it sounds like you should be using a messaging service. As #Saptarshi Basu mentioned kafka is good. I would really recommend NATS - although I'm biased because that's the one I work with
With NATS you can create request-reply messages to interface between client and booking service. That's a 1-1 communication
If you have multiple instances of each of your services running, you can use the Queuing service to automatically load balance. NATS will just randomly select a server for you
And then you can use pub-sub feeds for communication between all of your services.
This will give you a very resilient and scalable architecture, and NATS makes it all incredibly easy
I have been recently looking into NServiceBus, as I thought messaging would be a good way to reduce dependencies between systems. However, one of the first things that struck me is that the message publisher and all subscribers have to share the message definition DLL. What would happen in this scenario?:
Say there is one central system that handles client data. Whenever a client record is changed, it publishes a message, containing name and address. This has ten subscribers, which update their local copy of the data on receipt of the message.
One day, requirements change and one of the subscribers need the clients phone number as well. The message, the publisher, and the affected subscriber are all updated to handle the phone number, and they are all recompiled and released.
Will all nine other subscribers continue unaffected? Will they carry on as normal with the old Message DLL, or will they all need to be updated with the new DLL, recompiled and released as well?
The NServiceBus architecture is designed to be resilient to message structure changes (especially where the changes involve adding information like in your scenario). See the Versioning Sample page on the NServiceBus site.
It is not the case that you can handle versioning in NSB like they outline in the Versioning Sample.
You can do this if you are implementing NSB in a send/receive scenario. In this instance even though the contract is a messages DLL, the same DLL version does not need to be shared between senders and receivers. This is because providing the XML on the wire will de-serialize cleanly on the receiver end all will be well.
However, this completely breaks down in a pub-sub scenario. In this case, there is a dependency on the exact same version of the messaging assembly being shared between publisher and subscribers. This means the version, public key token etc all need to be identical. The reason for this is the subscription mechanism.
When your subscriber starts up it will send a subscription message to the publisher, who will then enter the subscription in the subscription data store. This subscription is for messages originating in a specific assembly version.
If the publisher then updates it's version of the messages DLL and receives a message which it needs to publish, it will do a lookup against the subscriptions it holds and evaluates each one in turn. Because the subscription exists for a previous version of the messages assembly the evaluation process will ignore that subscription entry, and therefore no message will be sent to the subscriber.
You need to be aware of this hard-dependency in the pub-sub scenario.
Hope this helps.
Edit
As of version 3.x of NServiceBus as long as your messages assembly major version is shared between publisher and subscriber then pub-sub will work as normal.