I have a queue at NiFi that contains the items that will be processed through an API query (invokeHTTP). These items can be processed and return the answer with the data correctly (status 200), they can not be found (status 404) and also a failure (status 500).
However, in the case of status 404 and 500, false negatives can happen, so if I consult the same data that gave an error again, it returns with status 200. But there are cases that there really is a failure and it is not a false negative.
So I created a queue for retry and failure for them to enter involeHTTP again and consult the API. I put an expiration time of 5 minutes so that the data that is really at fault is not forever consulting the API.
However, I wanted to prioritize this Failure and Retry queue, so that by the time a data reaches it, it will be consulted in the API again, in front of the standard processing queue, so as not to lose the data that gave false negatives.
Is it possible to do this treatment with this self relationship or do you need a new flowfile?
Each queue can have a prioritizer configured on the queue's settings. Currently you have two separate queues for InvokeHttp, the failure/retry queue and the incoming queue from the matched relationship of EvaluateJsonPath. You need to put a funnel in front of InvokeHttp and send both of these queues to the funnel, then the funnel to InvokeHttp. This way you can create a single incoming queue to InvokeHttp and configure the prioritizer there.
In order to prioritize it correctly, you may want to use Flow File Attribute prioritizer. You would use UpdateAttribute to add a "priority" attribute to each flow file, the ones for failure/retry get priority "A" and the others get priority "B" (or anything that sorts after A).
Related
Does anyone know what happens when two nodes try to write data at the same time and both initiate the 2PC protocol? Does a request from one node get aborted and another one succeed? The failed node would retry with some exponential backoff?
If not, what happens?
Usually resource managers does not allow both nodes to participate in the same transaction in the same transaction branch at the same time. Probably second node/binary/thread which tries to join to the transaction will get timeout or some other error on xa_start(..., TMJOIN) call.
I have an Openstack Aodh alarm and it will start the action when the memory usage is greater than 85% for one minute. Now I would like to trigger the action immediately, manually, which means force the alarm action start even though the condition doesn't reach the limits, but how?
According to the docs, I've tried to set the state of Aodh alarm to alarm, but it didn't work, it evaluated the memory usage and do nothing(cause its less than 85%), then set the state back to ok again.
Are there any ways to force trigger Aodh alarm action? I would appreciate any help.
Here are the parts of my Aodh alarm:
aggregation_method: mean
alarm_actions: [u'trust+http://192.168.0.100:8004/v1/284e047522bd4adfa3aa5109f1c7513b/stacks/corey_test/d9915fd3-5086-4d38-971b-2694c41e8099/resources/rdgw_scaleup_policy/signal']
alarm_id: e6402673-9a8e-4745-a8df-699edd6ab57a
comparison_operator: gt
enabled: True
evaluation_periods: 1
granularity: 60
metric: memory_util
ok_actions: []
repeat_actions: True
resource_type: instance
severity: low
state: ok
state_reason: Transition to ok due to 1 samples inside threshold, most recent: 11.0
threshold: 85.0
type: gnocchi_aggregation_by_resources_threshold
Update 2020/11/04
The only thing that comes into my mind is to reduce the threshold and evalution_periods temporarily (ex: threshold:1, periods:1), that will force the alarm start scaling, after the new instance is created, recover the threshold and evalution_periods values back. It works but I don't think that is the best method.
The alarm actions are AFAIU just HTTP POSTs to the URLs listed in 'alarm_actions', so you can do it yourself (provided you have access to that URL).
In your particular case it is clearly a Heat stack scaling action. You should be able to make a HTTP POST to appropriately similar URL - replace trust+https://<host>:<port> part with public Heat endpoint (openstack catalog show orchestration) and add a valid Keystone token to the request header.
Alternatively, for Heat stack scaling you can use use the openstack stack resource signal command (that does effectively the same REST call, just helps you with auth and endpoint discovery) - the stack ID and the resource name are visible in the URL, so in your case it will be openstack stack resource signal d9915fd3-5086-4d38-971b-2694c41e8099 rdgw_scaleup_policy
My code performs the following:
1)Create grpc channel
2)start monitoring completion queue in a different thread
3)Issue shutdown on completion queue
After executing step 3, I expect "(cq.Next(&tag, &ok)" to return false as there are no pending events with above 3 steps. But it is observed that "(cq.Next(&tag, &ok)" never returns false. Please let me know if I am missing something.
Thanks,
Ikshu
In order to get channel state notification, a tag was being added to the queue and that use to always post some events. so the cq->next() never returned false. I fixed this issue by achieving same functionality by using already existing standard API for channel state. So closing the bug.
I need to get the number of messages that are ready. A queue has three types of messages: 1. Total 2. Unack'd 3. Ready
Ready is the ones' that are in the queue but haven't been consumed yet.
Currently I use requests
url = "http://<RABBITHOST>:15672/api/queues/%2f/{}".format(q)
res = requests.get(url, auth=("<user>","<password>")).json()
messages_in_queue = res.get("messages_ready")
The problem here is that I have to pass in the username and password. Using Pika I believe you can get the "total" messages. Is there any way to get the other two types (unack'd and ready) using Pika?
No, the AMQP protocol doesn't support getting unacknowledged messages. You will still have to use the HTTP API for that. If you do a passive queue declaration, the message count returned is the number of ready messages.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Within my BizTalk 2010 solution I have the following orchestration that’s is started by the receipt of a courier update message. It them makes a couple of call to AX's WCF AIF via two solicit-response ports, a Find request and an Update request.
For this application we are meeting audit requirements through use of the tracking database to store the message body. We are able to link to this from references provided in BAM when we use the TPE. The result for the customer is nice, they get a web portal from which they can view BAM data of message timings etc. but they can also click a link to pull up a copy of the message payloads from the tracking db. Although this works well and makes use of out-of-box functionality for payload storage it has led to relatively complex jobs for the archiving of the tracking db (but that's another story!).
My problem relates to continuation. I have created the following Tracking Profile:
I have associated the first of the orchestration's two solicit response ports with the continuation RcvToOdx based on the interchange Id and this works, I get the following single record written to the Completed activity table:
So, in this case we can assume that an entry was first written on receipt in the inbound message, with the TimeReceivedIntoBts column populated by the physical file receive port. The FindRequestToAx column was then populated by the physical WCF send port. Because this was bound to the RcvToOdx continuation Id and used the same interchange Id and the previously mentioned file receive message, the update was made to the same activity. Notification of the resulting response was also updated to the same activity record - into the FindResponseFromAx column.
My problem is that I would also like BAM to record a timestamp for the subsequent UpdateRequestToAx. Because this request will have the same interchange Id as the previous messages I would expect to be able to solve this problem by simply binding the AxUpdate send port (both send and receive parts of it) to the same continuation id, as seen in the following screen grab:
I also map the UpdateRequestToAx milestone to the physical Ax_TrackAndTraceUpdate_SendPort (Send) and the OrchestrationCompleted milestone to Ax_TrackAndTraceUpdate_SendPort (Receive)
Unfortunately, when I try this I get the following result:
Two problems can be seen from the above db screen grab:
1. Date for the update send port was inserted into a new activity record
2. The record was never completed
I was surprised by this because I'd thought since they update port was enlisted to use the same continuation, and the single InterchangeId was being used by all ports for the continuation Id then all the data milestones would be applied to a single activity.
In looking for a solution to this problem I came across the following post on Stack Overflow suggesting that the continuation must be closed from the BAM API: BAM Continuation issue with TPE. So, I tried this by calling the following method from an expression shape in my orchestration:
public static void EndBAMContinuation(string continuationId)
{
OrchestrationEventStream.EndActivity(CARRIER_ORDER_ACTIVITY_NAME, continuationId);
}
I can be sure the method was called ok because I wrapped the call with a log entry from the CAT framework which I could see in debug view:
I checked the RcvToOdx{867… continuation Id against the non-closed activity and confirmed they matched:
This would suggest that perhaps the request to end the continuation is being processed before the milestone of the received message from the UpdateAx call?
When I query the Relationsips tables I get the following results:
Could anyone please advise why the UpdateToAx activity is not being completed?
I realise that I may be able to solve the problem using only the BAM API but I really want to exhaust any possibility of the TPE being fit for purpose first since the TPE is widely used in other BizTalk solutions of the organisation.
To solve this problem I created a 2nd continuation in the TPE.
"RcvToOdx" continuation for the Find and "OdxToUpdate" continuation for the update - source is InterchangeId on the initial receive port - UPS_TrackAndTrace (same as for other "RcvToOdx" continuation), dest Id is the InterchangeId mapped to update send port.