According to https://docs.corda.net/releases/release-V3.0/key-concepts-node.html?highlight=drain#draining-mode
When I set the drainingMode to true. How do I monitor the number of in-flight flows according to the following :
As their number - which can be monitored through RPC - reaches zero, it is safe to shut the node down.
Having confirmed DrainingMode is activated using this
You can then query the state machine either using stateMachinesFeed or stateMachinesSnapshot, details here and here respectively.
The stateMachinesSnapshot will return a list of StateMachineInfo, each representing a live flow. You can interrogate the StateMachineInfo for details on the flow including why it was started etc.
The stateMachinesFeed in addition also provides an observable of future state machine adds/removed. You could use this for additional monitoring.
Related
I'm trying to create a trigger in zabbix which will show me a problem and alert me on my email whenever an interface in a cisco switch (with snmpv2) crosses 80% of it's bandwidth (100 mbps or 1000 mbps) without hardcoding anything, I tried using this trigger expression:
{/switch name:net.if.out[ifHCOutOctets./switch interface].min(10)}>80000000
I would like to know how can I write this trigger expression which works fine without applying it to every single interface item in every switch. I think that maybe macros could help in these situations but found no explanation or any guide about how to use them or how to use low level discovery which maybe have a part at the solution for my need.
Thanks in advance.
You are correct, you want to use Low Level Discovery to do this, as well as to discover all your interfaces. Low level discovery at a high level consists of two things. 1) you have to tell zabbix how to go discovery a bunch of dynamic things and asssign a LLD macro to them, that is done a the high level of the Discovery rule. 2) you have to tell zabbix what item protototypes, trigger protorypes, etc to dynamically create as actual items and triggers, every time the discovery rule runs.
Take a look at the Arista SNMPv2 template included with zabbix as an example. There are a number of Discovery Rules included in that template, one of which is the Network Interfaces discovery rule. Within the network interfaces discovery rule, zabbix basically does an snmp walk and gets a list of all the interfaces and assigns LLD(Low level discovery macros) for each interfaces such as #IFINDEX, #IFSTATUS, etc. The discovery Rule, like all LLD rules, takes the output of the "Network Interfaces" discovery rule, and uses them to dynamically create actual items on each host the template applied to.
The next part of this to understand is the prototypes. Once zabbbix finds all the network interfaces, your question should be, how do i get it to create new Items in my host for each interface it finds, and how do i get it to create triggers for each interface it finds dynamically, automatically and without user intervention. The answer is protoyptes. Prototypes are child elements of a Low Level Discovery. They are what actually creates the new items and triggers for every thing it discovered.
Take a look here for some examples and docs on low level discovery rules.
https://www.zabbix.com/documentation/4.2/manual/discovery/low_level_discovery#trigger_prototypes
Zabbix can create LLD rules via numerous discovery methods including SNMPv#, which is all capable of being configured in the UI or api, and other customer discovery rules not included through the use of user parmaters, externel checks, etc.
If your make and model of switch is already known to zabbix, a template in "Templates/Network Devices", at least i think thats the path, will exist just like the arista and juniper ones.
You can create custom low level discovery rules as well for non snmp stuff. basically you write a script that will go find the things you want to dynamically add to zabbix and your script needs to return a valid json output with #macronames and values you want added. For example a custom file system discovery rules, which shouldn't be needed because there already included if your using the agent, would produce lines like the ones shown in this example in the official docs.
https://www.zabbix.com/documentation/4.2/manual/discovery/low_level_discovery#creating_custom_lld_rules
In short, check to see if a template exists for your switch already and a discovery rule with the item prototypes to discover things the way you want them already. LLD basically allows zabbix too walk a dynamic data structure of any source, as long as that data structure has a definition known to zabbix, and you tell it what keys and values in the JSON you want to create as items, triggers, etc.
You should leverage the low level discovery feature of Zabbix.
After a standard setup you should have a "Template Module Interfaces SNMPv2", which is used by some other templates as a standard for interface discovery.
Within the template you will find a "Network Interfaces Discovery" discovery rule that runs every hour to:
query the target device for an interface list
create N items for each interface (bits in, bits out, speed, type etc), defined as Item Prototypes
create some triggers, defined as Trigger Prototypes
The speed item is the negotiated interface speed (ie: 1000 for a gigabit switch port), which is your 100% limit.
You can add some definitions to this template to get the alert you need:
Create a calculated item prototype and set its formula to
100*(currentOutputBits/speed)
Create a template macro to define your alert threshold, ie {$INTERFACE_OUTPUT_THRESHOLD}, and set it to 80
Create a trigger prototype that matches when the calculated item is
greater than {$INTERFACE_OUTPUT_THRESHOLD} for N minutes
Optionally, do the same for the currentInputBits item
This setup will create 1 additional item and 1 additional trigger for each physical or logical interface found on the target device: it makes sense to unflag the "create enable" option on the trigger prototype and enable it only on specific ports of specific devices.
The threshold macro can be changed at template level, affecting any device linked to it, or at host level for a custom threshold.
Thanks for your replies, I actually checked again the Template Module Interfaces SNMPv2 and saw that there is a prototype trigger that solves my question.
for anyone who wants to do the same with a switch:
Add to your switch (host) the "Template Module Interfaces SNMPv2" template.
Change the IF_UTIL_MAX macro to whatever value you want it to be, the default is 90 (this is the macro which is responsible for the percentage of the bandwidth which will trigger a problem, for example, if you change it to 60, when a host bandwidth utilization averages more than 60% in any interface for 15 minutes, a problem will be added to the problems tab or to the dashboard).
If the time of 15 minutes isn't right for you, you can change it by going to: configuration -> templates -> Template Module Interfaces SNMPv2 -> Discovery rules -> Network Interfaces Discovery -> Trigger prototypes -> search for a trigger name which contains high bandwidth usage -> in the problem expression and the recovery expression find the .avg() function and change the value inside it to whatever value is right for you, for an example: 15m = 15 minutes, 1s = 1 second etc...
I actually recommend to clone the trigger prototype, change it and then disable the built in one instead of just changing the time inside it for easily debugging errors in the long run. So you can change the name of the trigger prototype and then clone it by pressing clone in the bottom left corner of the screen, then change it's name and settings for whatever suits you the best.
Press add in the bottom left corner of the screen and if you took my advice you also need to click on the green "Yes" link of the built in trigger in the trigger prototypes table for it to disable.
You can also go ahead and try the other answers on this thread, I'm seriously thankful for them but I don't have enough time to check if those work since I already figured it out after reading Simone Zabberoni's and helllordkb's answers and checking the built in low level discovery in the "Template Module Interfaces SNMPv2" template.
Currently I use the AcknoledgingMessageListener to implement a Kafka consumer using spring-Kafka. This implementation helps me listen on a specific topic and process messages with a manual ack.
I now need to build the following capability:
Let us assume that for an some environmental exception or some entry of bad data via this topic, I need to replay data on a topic from and to a specific offset. This would be a manual trigger (mostly via the execution of a Java class).
It would be ideal if I can retrieve the messages between those offsets and feed it is a replay topic so that a new consumer can process those messages thus keeping the offsets intact on the original topic.
CosumerSeekAware interface - if this is the answer how can I trigger this externally? Via let say a mvn -Dexec. I am not sure if this is even possible
Also let say that I have an crash time stamp with me, is it possible to introspect the topic to find the offset corresponding to the crash so that I can replay from that offset?
Can I find offsets corresponding to some specific data so that I can replay those specific offsets?
All of these requirements are towards building a resilience layer around our Kafka capabilities. I need all of these to be managed by a separate executable class that can be triggered manually providing the relevant data (like time stamps etc). This class should determine offsets and then seek to that offset, retrieve the messages corresponding to those offsets and post them to a separate topic. Can someone please point me in the right direction? I’m afraid I’m going around in circles.
so that a new consumer can process those messages thus keeping the offsets intact on the original topic.
Just create a new listener container with a different group id (new consumer) and use a ConsumerAwareRebalanceListener (or ConsumerSeekAware) to perform the seeks when the partitions are assigned.
Here is a sample CARL that seeks all assigned topics based on a timestamp.
You will need some mechanism to know when the new consumer should stop consuming (at which time you can stop() the new container). Maybe set max.poll.records=1 on the new consumer so he doesn't prefetch past the failure point.
I am not sure what you mean by #3.
I'm writing an app using RxAndroidBle library (which is great) but due to a requirement of the Device vendor we must be able to cancel all operations after the device has reached a certain state. Is there a way of cancelling any read/write op that has been sent to the RxBleConnection?
Any ideas?
Thanks!
The library uses an internal ConnectionOperationQueue which tries to remove operations that have not yet reached execution state. A concrete implementation would vary case-to-case but using RxJava it should be fairly easy to achieve:
// First create a completable that will fulfil appropriately
Completable deviceReachedACertainState = ...;
// Then use it for every operation that you want to cancel (unsubscribe) when a certain state happens
Observable<byte[]> cancellableRead = rxBleConnection.readCharacteristic(UUID).takeUntil(deviceReachedACertainState);
Keep in mind that operations that were already taken off the queue for execution will not be cancelled.
I was going through the rpc semantics, at-least-once and at-most-once semantics, how does they work?
Couldn't understand the concept of their implementation.
In both cases, the goal is to invoke the function once. However, the difference is in their failure modes. In "at-least-once", the system will retry on failure until it knows that the function was successfully invoked, while "at-most-once" will not attempt a retry (or will ensure that there is a negative acknowledgement of the invocation before retrying).
As to how these are implemented, this can vary, but the pseudo-code might look like this:
At least once:
request_received = false
while not request_received:
send RPC
wait for acknowledgement with timeout
if acknowledgment received and acknowledgement.is_successful:
request_received = true
At most once:
request_sent = false
while not request_sent:
send RPC
request_sent = true
wait for acknowledgement with timeout
if acknowledgment received and not acknowledgement.is_successful:
request_sent = false
An example case where you want to do "at-most-once" would be something like payments (you wouldn't want to accidentally bill someone's credit card twice), where an example case of "at-least-once" would be something like updating a database with a particular value (if you happen to write the same value to the database twice in a row, that really isn't going to have any effect on anything). You almost always want to use "at-least-once" for non-mutating (a.k.a. idempotent) operations; by contrast, most mutating operations (or at least ones that incrementally mutate the state and are thus dependent on the current/prior state when applying the mutation) would need "at-most-once".
I should add that it is fairly common to implement "at most once" semantics on top of an "at least once" system by including an identifier in the body of the RPC that uniquely identifies it and by ensuring on the server that each ID seen by the system is processed only once. You can think of the sequence numbers in TCP packets (ensuring the packets are delivered once and in order) as a special case of this pattern. This approach, however, can be somewhat challenging to implement correctly on distributed systems where retries of the same RPC could arrive at two separate computers running the same server software. (One technique for dealing with this is to record the transaction where the RPC is received, but then to aggregate and deduplicate these records using a centralized system before redistributing the requests inside the system for further processing; another technique is to opportunistically process the RPC, but to reconcile/restore/rollback state when synchronization between the servers eventually detects this duplication... this approach would probably not fly for payments, but it can be useful in other situations like forum posts).
We have a BizTalk application where the order of messages being inputted is very important and has to be kept, meaning they have to be outputted in the same order. Normally ordered delivery would do the trick here.
However I read that ordered delivery is only guaranteed when you connect a receive location directly to a send port. The moment you use orchestrations the order delivery isn't guaranteed anymore. Is there a way to work around or fix this? Because this kind of ruins our whole application and we've been working on this for months.
I read a work around from Microsoft where they use an extra field which has a counter and where they use an end orchestration which checks the counters. But this is way too much work for us to do now. So this work around is a no go. Plus not all messages are translated which creates holes in our flow and not all messages are coming from the same source either which makes this work around useless anyway.
Any other ideas?
Check out this page.
It explains that if you have an orchestration that follows the singleton pattern to ensure only one instance of the orchestration exists, and you make sure you set the orchestration's receive port to ordered delivery, than you should get a valid end-to-end ordered delivery scenario
To provide end-to-end ordered delivery the following conditions must be met:
Messages must be received with an adapter that preserves the order of the messages when submitting them to BizTalk Server. In BizTalk Server 2006, examples of such adapters are MSMQ, MQSeries, and MSMQT. In addition, HTTP or SOAP adapters can be used to submit messages in order, but in that case the HTTP or SOAP client needs to enforce the order by submitting messages one at a time.
You must subscribe to these messages with a send port that has the Ordered Delivery option set to True.
If an orchestration is used to process the messages, only a single instance of the orchestration should be used, the orchestration should be configured to use a sequential convoy, and the Ordered Delivery property of the orchestration's receive port should be set to True.
Resequencing strategies for ordered delivery in BizTalk:
I recently responded to a LinkedIn user's question regarding ordered delivery options in BizTalk.
I thought it would be useful for people to understand some of the strategies for re-sequencing messages using BizTalk.
Often as an BizTalk Developer, you are required to integrate to line-of-business systems which are unchangeable. This can be for one or more of many different reasons. As an example, the cost of changing a system can be too high or the vendor license states that support may be withdrawn if the system is changed.
This would not normally represent a problem where the vendor has provided a thoughtfully designed API as a point-of-integration endpoint. However, as many Integration Developers quickly learn, this is very rarely the case.
What do I mean by a thoughtfully designed API? Well, aside from all the SODA principals (service composition, fault contracts etc.), the most important feature of an API is to support the consumption of data which arrives in the wrong order.
This is a fairly simple thing to do. For example, if you are a vendor and you provide a HTTP operation as your integration point then one of the fields you could expose on your operation is a time-stamp or, even better, a sequence number. This means that if a call is made with an out-of-date payload, the relevant compensating mechanism can kick-in - which can be as simple as discarding the data.
This article discusses what to do when the vendor has not built this feature into an API, and as an integrator you therefore are forced to implement end-to-end ordered delivery as part of your integration solution.
As stated in my response to the user's post on LinkedIn (see link above), in BizTalk ordered delivery in any but the simplest of scenarios is complicated at best and at worst can represent a huge cost in increased complexity, both in terms of development and support. The basic reason is that BizTalk is designed to be massively concurrent to enable high throughput, and there is a direct and unavoidable conflict between concurrency and ordering. Shoe-horning E2E ordered delivery into a BizTalk solution relies on artefacts such as singleton orchestrations which introduce complexity and increase both failure rate and cost-per-failure numbers.
A far better solution is to maintain concurrent processing to as near as possible to the line-of-business system endpoints, and then implement what is called a re-sequencer wrapper around each of the endpoints which require data to be delivered in the correct order.
How to implement such a wrapper in BizTalk depends on some factors, which are outlined in the following table:
|Sequencing |Messages|Database |Wrapper |
|field |are |integration?|strategy |
| |deltas? | | |
|--------------|--------|------------|----------------------------------|
|n of a total m| N | Y |Stored procedure |
|n of a total m| N | N |Singleton orchestration |
|n of a total m| Y | Y |Batched singleton orchestration |
|n of a total m| Y | N |Batched singleton orchestration |
|Timestamp | N | Y |Stored procedure |
|Timestamp | N | N |Singleton orchestration |
|Timestamp | Y | Y |Buffer table with staggered reader|
|Timestamp | Y | N |Buffer table with staggered reader|
The first factor Sequencing field relates to the idea that in order to implement any kind of re-sequencer wrapper, as a minimum you will require that your message data contains some sequencing information. This can take the form of a source time-stamp; however a better, though rarer, kind of sequencing information consists of a sequence number combined with the total number of messages, for example, message 1 of 10, 2 of 10, etc..
The second factor Messages are deltas? relates to whether or not the payload of your message contains a single state change to the data or the sum of all past changes to the data. Put another way, is it possible to reconstruct the full current state of the data from this message? If the message payload contains just a single change then it may not be possible to reconstruct the state of the data from the single message, and in this instance your message is a delta.
The third factor Database integration? relates to whether or not the integration-entry-point to a system is a database. The reason this matters is that integrating at the database layer is a fairly common integration scenario, and if available can greatly simplify handling re-sequencing.
The strategies from the above table are described in detail below:
Stored procedure wrapper
This is the simplest of the resequencing strategies. A new stored procedure is created which queries the target data before making a decision about whether to update the target data. The decision can be as simple as Is the data I have newer than the data in the target system?
Of course, in order to implement this strategy, the target data also has to include the sequencing field of the source data, although an approximation can be made if necessary by relying on existing time-stamps which may already exist in the target data. The stored procedure wrapper can be contained either in the target database or ideally in a separate database.
Singleton orchestration wrapper
The idea behind this strategy is the singleton orchestration. This is a pattern you can implement to ensure that only a single instance of the orchestration will exist at any one time. There are many articles on the web demonstrating how to implement this pattern in BizTalk.
The core of the idea is that the singleton simply keeps a track of the most recent successfully processed message sequence (or time-stamp). If the singleton receives a message which is older than the most recent sequence it is simply discarded. This works because the messages are non-deltas, so the target system can commit only the most recent of a number of messages and the data will be in the most recent state. Only when data is committed successfully is the most recent sequence held by the singleton updated.
Batched singleton orchestration wrapper
This strategy is based on the Singleton orchestration wrapper above, except it is more complex. Rather than only keep the most recent sequence information in memory the singleton is required to create and hold a working set of messages in memory which it will re-order and then process once all expected messages from the batch have arrived. This is because the messages are deltas so the target system MUST receive each message in the order they were intended. Once the batch has been sent successfully the singleton can terminate.
To do this it is a requisite of the source data that it contain a correlation identifier of some description which allows the batch of messages to be defined. For example, processing a defined set of orders from a customer, the inbound messages must contain an identifier for the customer. This can then be used to route the messages to the singleton orchestration instance correlated with this customer. Furthermore the message sequence field available must be of the n of a total m form.
Once the singleton is initialised it assembles a working set of messages in memory and proceeds to populate it as new messages arrive. One way I have seen this done is using a System.Collections.Generic.List as the container for the working set. Once the list has been fully populated (list length = m) then it is assumed all messages in the batch have been received and the orchestration then loops over the working set in sequence and processes the messages into the target system.
One of the benefits of the batched singleton orchestration wrapper is it allows concurrent processing by correlation identifier. In the example above this means that messages from two customers would be processed concurrently.
Buffer table with staggered reader wrapper
Arguably the most complex of the strategies presented, this solution is to be used when you have delta messaging with a time-stamp-based sequencing field. It can be implemented with a database of some description which acts as a re-sequencing buffer.
It is worth noting here that this re-sequencing wrapper does not guarantee ordered delivery, but used well it makes ordered delivery highly likely.
As messages arrive, they are written into the buffer and in the same operation the buffer is reordered, so that the order of messages held in the buffer are always correct.
To create the buffer reader, have a receive location which reads the messages in the buffer before passing the messages to a send port with ordered delivery enabled, which then will process the messages into the target system. You can also use a singleton orchestration as an intermediary if your target system's API semantics are too complex for a send port.
However, using this wrapper as I have described it above will not enable ordered delivery, as the messages will almost certainly be committed to the buffer in the wrong order, which will result in the messages being processed into the target system in the same (wrong) order. This is where the staggered query comes in. This is a fancy way of saying your buffer query needs to only select data at intervals of time T, AND only select those rows where the row-number is lower than buffer total row count minus C.
This has the effect of allowing sequencing to occur over an appropriate timespan. T will be familiar to most BizTalk developers as the polling interval of some adapters (such as the WCF-SQL adapter). C is slightly more difficult to set, but by increasing this number you are reducing the chances that when you poll, you will miss a message older than the most recent one in your retrieved data set.
What T and C are depends on many things, although these values should be based on your latency SLA and your message volume (or throughput). As a guideline, if you have a SLA to deliver data into your target system within 30 seconds and you process 10 messages per second then T should be around 10 seconds and C should be around 100 rows.
Of course this only works if your messages for a given correlation id are sent by the source system during a short space of time (ideally back-to-back). The longer the interval between sends, the greater the required value of C, and the less effective the wrapper becomes.
One of the benefits of this strategy is you can also perform de-duplication of messages in the buffer if your data source is prone to sending duplicate messages and your target system endpoint is not idempotent. You can also use the buffer to implement FILO and other non-standard queueing semantics.
Conclusions
The strategies I have discussed here are ways of bending BizTalk to a task which is wasn't designed to do. As a result each has caveats around cost and complexity to support, and also may not work in certain scenarios. I would like to hear from anyone who has implemented other patterns for ordered delivery in BizTalk.