I have a microservice based project with kafka, which I used for event bus. I have a business process, which contains multiple microservices. Microservices asynchronous communicate with each other with help kafka. Each instance of business process has unique process_id.
Let's consider a example of some process:
User creates a process with request_id == 556bb813-bf77-4f5f-8bb0-1a59d6ba16b4 in API gateway service.
API gateway service produce some event to service1_in. Of course, event contains 556bb813-bf77-4f5f-8bb0-1a59d6ba16b4.
Microservice_1 consumes event from service1_in and produces event to service2_in.
Microservice_2 consumes event from service2_in and produces event to service1_in.
Microservice_1 consumes event from service1_in and produces event to service1_out.
API gateway service consumes event from service1_out and eventually returns result to Client.
So, I need to measure execution time bwtween differ steps of this process. For example, I want to know, duration between steps 3 and 5, or 2 and 6.
So, I don't know, how to measure it. I have only one workaround solution. I can have share memory (e.g., redis), where I can store points of time for each stem of process. In the end of process, I can calculate all metrics and push it datadog.
Related
I'm developing an HttpModule to capture logging and performance information for all requests to our web application. I'm tapping into the PreRequestHandlerExecute and PostRequestHandlerExecute events.
When my code executes within PostRequestHandlerExecute, is that code guaranteed to be executing within the same request context as the code which ran earlier on in PreRequestHandlerExecute? In other words, can I preserve state when handling the first event to be used during the execution of the second event?
Or is it possible for an HttpModule instance to service multiple concurrent requests, in which case any state necessary to connect the processing of these two events would need to be stored (and then fetched from) within the current HttpContext?
(I understand of course, that the HttpModule will be used to service multiple requests sequentially and that request-specific state from an earlier request will be irrelevant to a subsequent request. My question is specifically whether a single HttpModule can be used to handle multiple concurrent requests or whether I can rely on the fact that the HttpModule will be dedicated to a single request for the entire lifecycle of that request.)
Thanks for your advice!
As per the literature, and borne out by my own implementation, an HttpModule instance is indeed dedicated to a single Http transaction for the complete Request/Response transaction lifecycle.
https://learn.microsoft.com/en-us/dotnet/api/system.web.httpapplication?source=recommendations&view=netframework-4.8#remarks
https://learn.microsoft.com/en-us/previous-versions/bb470252(v=vs.140)?redirectedfrom=MSDN
State can be safely accumulated in one event to be used in subsequent events because all events the HttpModule instance will receive will belong to the same Request/Response transaction.
(Naturally, any accumulated state which is transaction-specific will need to be reinitialized for the subsequent transaction since an instance of an HttpModule is subject to be reused for a subsequent transaction once the current transaction has completed.)
I have furthermore seen that an instance of an HttpModule may be executing on a different thread than the thread on which it was originally created or previously executed.
I'm designing a system with multiple bounded contexts (microservices). I will have 2 kind of events.
Domain Events, which happens "in memory" within single transaction (sync)
Integration Events, which are used between bounded contexts (async)
My problem is, how to make sure that once transaction is committed (at this point I'm sure all Domain Events were processed successfully) that Integration Events are successful as well.
When my Transaction is committed, normally I will dispatch Integration Events (e.g. to the queue), but there is possibility that this queue is down as well, so previously just-committed transaction has to be "reverted". How?
The only solution that comes to my mind is to store Integration Events to the same DB, within the same Transaction, and then process the Integration Events records and push them to the queue - this would be something like "using current DB, as a pre-queue, before pushing it to The Real Queue (however I read that using DB for this is an anti-pattern).
Is there any pattern (reliable approach) to make sure both: Transaction commit and Message pushed to the queue is an atomic operation?
EDIT
After reading https://devblogs.microsoft.com/cesardelatorre/domain-events-vs-integration-events-in-domain-driven-design-and-microservices-architectures/ , the author actually suggests the approach of "pre-queue" in same DB (he calls it “ready to publish the event”).
Checkout transactional outbox pattern.
This pattern does create a pre-queue. But the nice part is that pushing messages from pre-queue to real queue is fully decoupled. Instead you have a middleman called, a message relay that reads your transaction logs and pushes your event from to the real queue. Now since sending message and your domain events are fully decoupled, you can do all your domain events in a single transaction.
And make sure you that all your services are idempontent(same result despite duplicate calls). This transactional outbox patter does guarantee that messages are published, but in case when the message relay fails just after publishing(before acknowledging) it would publish the same event again.
Idempotent services is also necessary in other scenarios. As the event bus(the real queue) could have the same issue. Event bus propagates events, services acknowledge, then network error, then since the event bus is not acknowledged, the same event would be sent again.
Hmm actually idempotence alone could solve the whole issue. After the domain events computation completes(single transaction), if publishing message fails the service can simply throw an error without roll back. Since the event is not acknowledged the event bus will send the same event again. Now since the service is idempotent, the same database transaction will not happen twice, it will basically overwrite or better(should) skip and directly move to message publishing and acknowledging.
There is a bank which creates a contract which is then accepted by the lender and the borrower. After signing the contract the lender provides fund to the borrower. The bank then creates an obligation state based on the data received by calling an external service automatically.
And Now
1) In API Layer, I am calling first flow which creates one state.
2) In API layer itself, On success of first flow , I am calling the http request to external service and get the data.
3) Now I pass the http response to the the second flow for creating the other state.
Can you please let me know if there is any issue with this approach.
Requirment is I want to trigger the first flow manually, but calling external service and initiating the second flow should happen automatically
I had referred the link given below.
Making asynchronous HTTP calls from flows
You'll make calls to an external service during the running of flows.
The best place to get started would be looking at the CorDapp samples here. In particular, take a look at the Accessing External Data section
I'm designing a part of a Java EE 6 application, consisting of EJB3 beans. Part of the requirements are multiple parallel (say a few hundred) long running (over days) database hunts. Individual hunts have different search parameters (start time, end time, query filter). Parameters may get changed over time.
Currently I'm thinking of the following:
SearchController (Stateless Session Bean) formulates a set of search parameters, sends it off to a SearchListener via JMS
SearchListener (Message Driven Bean) receives search parameters, instantiates a SearchWorker with the parameters
SearchWorker (SLSB) hunts repeatedly through the database; when it finds something, the result is sent off via JMS, and the search continues; when the given 'end-time' has reached, it ends
What I'm wondering now:
Is there a problem, with EJB3 instances running for days? (Other than that I need to be able to deal with container restarts...)
How do I know how many and which EJB instances of SearchWorker are currently running?
Is it possible to communicate with them individually (similar to sending a System V signal to a unix process), e.g. to send new parameters, to end an instance, etc..
If you're holding a huge ResultSet open for an extended period of time, you're likely to encounter either transaction timeouts or database locking issues.
There is no builtin mechanism for determining which bean instances are running in a method, so you would need to add your own mechanism. Your product might have some kind of performance monitoring that lets you know how many of each type of bean is currently running a method.
As for cross-thread communication, you would need to implement your own synchronization and periodically check in the bean method. You'll be outside the scope of standard EJB since each parallel call to a business method will allocate a new SLSB from the pool.
Problem: Some 300 candidates make a test using Flex. A test consist of some 100 exercises. After each exercise a .NET service is called to store the result. If a candidate finishes a test, all the data of his/her test is denormalized by Asp.NET. This denormalization can take some cpu and can take 5 to 10 seconds. Now, most of the times, some of the candidates have finished their test earlier than the rest, but still some 200 of them wait until their time is up. At that moment, 200 candidates finish their test and 200 sessions are denormalized at the same time. At this point, server load (cpu) is too high and cause calls to the webserver to go wrong. Now, instead of all these sessions being normalized concurrently, I would like to add them to a queue using MSMQ.
Question:
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax that listens to the queue? If there are messages, they are dealt one at the time.
Is it necessary to do this in a separate thread? What if in the global.asax you just call a singleton for instance that starts listening to the queue? In what thread will this singleton run? (what's the thread that calls global.asax)
What are best practices to implement this? Links? Resources? Tutorials? Examples?
I don't like the idea, but could you put an exe on the root of your website, an exe that starts a process listening to the queue...
If you get a message out of the queue, do you remove it when you pull it out or do you remove it if denormalization for this session was successful? If you remove it when you pull it out and something goes wrong...
I could also create my own queue in memory, but restarting the webserver would empty the queue and a lot of sessions would end up not being normalized, so I guess this is really a bad idea.
Is MSMQ a good choice or are there better alternatives?
You could consider using a WCF-Service with MSMQ transport. I used this approach in an application that calculates commissions:
User completes asp.net wizard configuring calculation parameters
Calculation Job is sent to WCF-Service using MSMQ transport
Service transaction is completed as soon as Job entered MSMQ
New transaction scope is created for processing Job instances
One drawback is that the transaction will require MSDTC which will add some overhead when targeting MS SQL Server and even more when dealing with Oracle.
IDesign provides a lot of useful samples and best practices on WCF queueing.
Personally, I use a servicebus for scenario's like that. I know this sounds like an overkill, but I think the .net servicebusses are so good that they require the least amount of code written by you, because it's not easy to create a good scheduler for background processes without disturbing the threads of the application pool the webapp is running in. NServicebus and MassTransit are both good an well enough documented servicebuses for your scenario. With a servicebus, you have a framework that writes to msmq and listens to msmq in several apps connected by the messagequeue. The bus makes it easy for you to create a separate app that runs as a background service and is connected with your web-app by the message queue. When you use topself (included in nservicebus and masstransit), an installer/uninstaller for the seperate apps is automatically generated by the service bus.
Question: Why don't you like the idea of having a separate exe?
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax
that listens to the queue? If there are messages, they are dealt one at
the time.
Is it necessary to do this in a separate thread? What if in the
global.asax you just call a singleton for instance that starts listening to
the queue? In what thread will this singleton run? (what's the thread that
calls global.asax)
[skip]
I don't like the idea, but could you put an exe on the root of your website, an exe that > starts a process listening to the queue...
Normally another program processes the queue - not ASP.NET. Either a windows service or an executable that you run under a scheduler (and there's no reason to put it in the root of your website).
If you get a message out of the queue, do you remove it when you pull
it out or do you remove it if denormalization for this session was
successful? If you remove it when you pull it out and something goes
wrong...
For critical work, you perform a transactional read. Items aren't removed from the queue until you commit your read operation, but while the transaction is open, no other process can get the item.
What are best practices to implement this? Links? Resources? Tutorials? Examples?
This tutorial is a good introduction and John Breakwell's blog is excellent and offers a lot of good links (including the ones in his easy-to-find sidebar "MSMQ Documentation").