We need to trigger an another event when two independent async events are completed. We are publishing individual rabbitmq messages on the completion of each async event. I went through RabbitMQ documentation but did not find a way to handle this elegantly.
Scenario:
- Task A completed
- Task B completed
Start task C only if A & B are completed.
Is there any design pattern that can help me here ? Anything outside RabbitMQ is also fine. We would like to achieve this without polling. Tasks are totally independent happening in 3 different systems.
Can celery help in this?
We implemented this using Aggregator pattern in RabbitMQ itself.
http://www.enterpriseintegrationpatterns.com/patterns/messaging/Aggregator.html
We created a topic (aggregator). Task A and Task B are subscribed to it. Whenever any of the task gets completed, message is being pushed to the topic. Topic is maintaining the state of taskA and taskB and publishes another message once both tasks are completed.
Related
Background
Planning to set a up data pipeline using Flink.
The flow looks like this
Kafka --> Flink Job --> gRPC endpoint
Story so far
Blocking implementation is up and running. But that will not scale for high QPS
Tried simulating async behavior here
Problem
For Async Behavior not sure how the behavior would be
if CompletableFuture is used, per message it will be processed in Async manner, but will the next message be fetched for processing before processing of first is complete ? In other words, there is a way to achieve async processing within a task manager. But what is the behavior of Task manager in fetching next message / tuple ? Will is wait till Async process is complete or will it submit to CompletableFuture / Thread and fetch next message ? Not clear about that
Will using a custom threadpool cause any issues if not shutdown as the pipeline will be running over a long period ?
Any other solution to achieve async behavior in Flink sink ?
I would leverage Flink's support for async operators, and have a DiscardingSink, versus trying to implement a custom async sink.
And no, I don't see any reason why having a persistent thread pool would cause problems.
In below scenario, what would be the bahavior of Axon -
Command Bus recieved the command
It creates an event
However messaging infra is down (say kafka)
Does Axon has re-queing capability for event or any other alternative to handle this scenario.
If you're using Axon, you know it differentiates between Command, Event and Query messages. I'd suggest to be specific in your question which message type you want to retry.
However, I am going to make the assumption it's about events, as your stating Kafka.
If this is the case, I'd highly recommend reading the reference guide on the matter, as it states how you can uncouple Kafka publication from actual event storage in Axon.
Simply put, use a TrackingEventProcessor as the means to publish events on Kafka, as this will ensure a dedicate thread is used for publication instead of the same thread storing the event. Added, the TrackingEventProcessor can be replayed, thus "re-process" events.
Team:
I need to invoke a WF activity (XAML) from a WF service (XAMLX) asynchronously. I am already referencing the Microsoft.Activities.Extensions framework and I'm running on the Platform Update 1 for the state machine -- so if the solution is already in one of those libraries I'm ready!
Now, I need to invoke that activity (XAML) asynchronously -- but it has an output parameter that needs to set a variable in the service (XAMLX). Can somebody please provide me a solution to this?
Thanks!
* UPDATE *
Now I can post pictures, * I think *, because I have enough reputation! Let me put a couple out here and try to better explain my problem. The first picture is the WF Service that has the two entry points for the workflow -- the second is the workflow itself.
This workflow is an orchestration mechanism that constantly restarts itself, and has some failover mechanisms (e.g. exit on error threshold and soft exit) so that we can manage our queue of durable transactions using WF!
Now, we had this workflow working great when it was all one WF Service because we could call the service, get a response back and send the value of that response back into another entry point in a trigger to issue a soft exit. However, a new requirement has arrisen asking us to make the workflow itself a WF activity in another project and have the Receive/Send-Reply sequences in the WF Service Application project.
However, we need to be able to startup this workflow and forget about it -- then let it know somehow that a soft exit is necessary later on down the road -- but since WF executes on a single thread this has become a bit challenging at best.
Strictly speaking in XAML activities Parallel and ParallelForEach are how you perform asynchrony.
The workflow scheduler only uses a single thread (much like UI) so any activity that is running will typically be running on the same thread, unless it implements AsyncCodeActivity, in which case you are simply handing back the scheduler thread to the runtime while waiting for a callback from whichever async code your AsyncCodeActivity implementation is calling.
Therefore are you sure this is what you want to achieve? Do you mean you want to run it after you have sent your initial response? In this case place your activity after the Send Reply.
Please provide more info if these suggestions don't answer your question./
Update:
The original requirement posed (separating implementation from the service Receive/Send activities) may actually be solved by hosting the target activity as a service. See the following link
http://blog.petegoo.com/index.php/2011/09/02/building-an-enterprise-workflow-system-with-wf4/
I have a WF 4 service. I have the following sequence:
Receive activity
Some Other activities
SendReply
Other Activities
I have an asp.net calling this service.
Why does this client has to wait for all steps to complete?
I want that when step 3 is completed, the reply is sent to the client and the client can continue on its own.
any assistance would be greatly appreciated.
The response is ready to send but due to the async nature of WF4 it isn't actually send directly. You can either persist the workflow, using the PersistBeforeSend, or add a small sub second delay using a Delay activity.
When the Workflow hits the SendReply block it should send the reply at that point and allow the client to continue on. You could test that by putting a delay in of a couple of minutes at Step 4.
Is it possible that the steps after the SendReply are completing so quickly that it just looks like the Workflow is running those steps before sending the reply?
I'm building a simple task manager that will at this moment execute tasks in a serial manner. I have been reading about threads in flex and it seems it is not quite clear/prepared for real threads.
What I'm looking at this moment is a way to execute a method at the beginning or end of a flash builder update. This method will be the one that will take the responsibility to start tasks added in the previous update. The removing of finished tasks will be done through event notification (the task will notify it finished) then the scheduler will remove it and dispatch the message again to let the outside world know the task was over.
A rough workflow of the system woudl be:
1) Add Tasks to the scheduler. And listen to events of the task (finished, etc...)
2) At the beginning/ End of a flex update (don't know if this really happen) Start tasks waiting. And run tasks that have a runnable method per update.
3) When a task finishes it notifies the scheduler and it is removed from the scheduler queue and redispatches the event to let the outside world the task finsihed.
Could anybody suggest the correct place to have a method like this? Any suggestion to the scheduler?.
Thanks in advance,
Aaron.
Based on your description you don't seem to be doing anything new and that unique. I'd start first with researching existing task and concurrency solutions. If they won't do what you want, extending the code will probably still be easier than starting from scratch.
Get familiar first with Cairngorm 3 Tasks and/or Parsley Tasks.
Also take a look at the callLater() method.
Finally there is the GreenThreads project.