I'm trying to use this API : https://wps.hereapi.com/v8/findsequence2
(documentation here https://developer.here.com/documentation/routing-waypoints/api-reference.html)
This variable : timingCompatibility (string)
is proposed but undocumented.
What should be passed in it?
What is its purpose?
I am trying to delay the departure of a car/truck to match first opening time on my sequence, I thought it might help but tried a a few values without success yet.
Setting timingCompatibility=true enforces WSE to synchronize with the travel times of the Router (WSE uses the Matrix-Router by default). The travel times are cross-checked after the sequence has been established and there will be no influence on the order of the waypoints. It is intended to have precisely consistent timings between WSE and the subsequent usage of Router (V7.2). My expectation is, that this parameter will not be necessary anymore with the migration of WSE to use Matrixrouter V8. This migration is progress.
Related
Currently I use the AcknoledgingMessageListener to implement a Kafka consumer using spring-Kafka. This implementation helps me listen on a specific topic and process messages with a manual ack.
I now need to build the following capability:
Let us assume that for an some environmental exception or some entry of bad data via this topic, I need to replay data on a topic from and to a specific offset. This would be a manual trigger (mostly via the execution of a Java class).
It would be ideal if I can retrieve the messages between those offsets and feed it is a replay topic so that a new consumer can process those messages thus keeping the offsets intact on the original topic.
CosumerSeekAware interface - if this is the answer how can I trigger this externally? Via let say a mvn -Dexec. I am not sure if this is even possible
Also let say that I have an crash time stamp with me, is it possible to introspect the topic to find the offset corresponding to the crash so that I can replay from that offset?
Can I find offsets corresponding to some specific data so that I can replay those specific offsets?
All of these requirements are towards building a resilience layer around our Kafka capabilities. I need all of these to be managed by a separate executable class that can be triggered manually providing the relevant data (like time stamps etc). This class should determine offsets and then seek to that offset, retrieve the messages corresponding to those offsets and post them to a separate topic. Can someone please point me in the right direction? I’m afraid I’m going around in circles.
so that a new consumer can process those messages thus keeping the offsets intact on the original topic.
Just create a new listener container with a different group id (new consumer) and use a ConsumerAwareRebalanceListener (or ConsumerSeekAware) to perform the seeks when the partitions are assigned.
Here is a sample CARL that seeks all assigned topics based on a timestamp.
You will need some mechanism to know when the new consumer should stop consuming (at which time you can stop() the new container). Maybe set max.poll.records=1 on the new consumer so he doesn't prefetch past the failure point.
I am not sure what you mean by #3.
I understand that sessionless operations are the preferred method of using gremlin. I'm wondering when is the sessioned approach better?
So I might be doing something like...
graph.addVertex("foo").property("name","bar")
graph.traversal().V().has("name","bar").as("f").addV("foo").property("name","baz").as("g").addE("test").from("f").to("g")
I'm doing this type of operation a lot. Often there's also a query (usually involving a coalesce) beforehand to check if a node (g in my example) exists, and create it if not.
So I'm wondering if a session might be better because I could hold a handle to the previous vertices and just attach new nodes to them without the expense of the lookup.
Feel free to tell me why I'm wrong in anything else that I'm doing.. Just trying to make things faster.
First of all, I would avoid use of addVertex() and stick to addV() - see more details here.
As to your question, I think the only time to leverage sessions is if you have some sort of loading operation that requires explicit control over transactions and you're not using a JVM based language. Even then, I might consider other options for dealing with that and just avoid sessions completely. You end up with a less portable solution as there are a number of graph systems which don't even support them directly (e.g. Neptune).
The cost to do a T.id based lookup should be really fast, so saving a vertex between requests in a session really shouldn't vastly improve performance. Even if you keep the vertex between requests you will still need to pass the vertex into your traversal so you still have the lookup anyway - I'm not sure I see the difference in cost there.
// first request
v = g.addV(...).property(...).next()
// second request
g.V(v).addE(....
// third request
g.V(v).addE(....
The above should not be that much faster than:
// first request - returns id=1
g.addV(...).property(...).id().next()
// second request - where "1" is just passed in on the next request as a parameter
g.V(1).addE(....
// third request
g.V(1).addE(....
I have a doubt about about whether I should consider a certain type of test functional or contract.
Let's say I have an API like /getToolType, that accepts a {object" "myObject"} as input, and returns at type in the form {type: "[a-z]+"}
It was agreed between client and server that the types returned will match a set of strings, let's say [hammer|knife|screwdriver], so the consumer decided to parse them in an enum, with a fallback value when the returned type is unknown.
Should the consumer include a test case for each type(hammer, knife, screwdriver) to ensure the producer is still following the agreement that it will always return , for instance , the lowercase string "hammer" when /getToolType is called with an hammer object?
Or would you consider such a test case as functional? And why?
IMO the short answer is 'no'.
Contract testing is more interested in structure, if we start boundary testing the API we move into functional test territory, which is best done in the provider code base. You can use a matcher to ensure only one of those three values is returned, this should ensure the Provider build can't return other values.
I would echo #J_A_X's comments - there is no right or wrong answer, just be wary of testing all permutations of input/output data.
Great question. Short answer: there's no right or wrong way, just how you want to do it.
Longer answer:
The point of Pact (and contract testing) is to test specific scenarios and making sure that they match up. You could simply, in your contract, create a regex that allows any string type for those enums, or maybe null, but only if your consumer simply doesn't care about that value. For instance, if the tool type had a brand, I wouldn't care about the brand, just that it's returned back as a string since I just display the brand verbatim on the consumer (front-end).
However, if it was up to me, from what I understand of your scenario, it seems like the tool type is actually pretty important considering the endpoint it's hitting, hence I would probably have specific tests and contracts for each enum to make sure that those particular scenarios on my consumer are valid (I call X with something and I expect Y to have tool type Z).
Both of these solutions are valid, what it comes down to is this: Do you think the specific tool type is important to the consumer? If it is, create contracts specific to it, if not, then just create a generic contract.
Hope that helps.
The proper state is that consumer consumes hammer, knife, and screwdriver, c=(hammer,knife,screwdriver) for short while producer produces hammer, knife, and screwdriver, p=(hammer,knife,screwdriver).
There are four regression scenarios:
c=(hammer,knife,screwdriver,sword), p=(hammer,knife,screwdriver)
c=(hammer,knife,screwdriver), p=(hammer,knife,screwdriver,sword)
c=(hammer,knife,screwdriver), p=(hammer,knife)
c=(hammer,knife), p=(hammer,knife,screwdriver)
1 and 3 break the contract in a very soft way.
In the 1st scenario, the customer declared a new type that is not (yet) supported by the producer.
In the 3rd scenario, the producer stops supporting a type.
The gravity of scenarios may of course wary, as something I consider soft regression, might be in a certain service in a business-critical process.
However, if it is critical then there is a significant motivation to cover it with a dedicated test case.
2nd and 4th scenarios are more severe, in both cases, the consumer may end up in an error, e.g. might be not able to deserialize the data.
Having a test case for each type should detect scenario 3 and 4.
In the 1st scenario, it may trigger the developer to create an extra test case that will fail on the producer site.
However, the test cases are helpless against the 2nd scenario.
So despite the relatively high cost, this strategy does not provide us with full test coverage.
Having one test case with a regex covering all valid types (i.e. hammer|knife|screwdriver) should be a strong trigger for the consumer developer to redesign the test case in 1st and 4th scenario.
Once the regex is adjusted to new consumer capabilities it can detect scenario 4 with probability p=1/3 (i.e. the test will fail if the producer selected screwdriver as sample value).
Even without regex adjustment, it will detect the 3rd scenario with p=1/3.
This strategy is helpless against the 1st and 2nd scenario.
However, on top of the regex, we can do more.
Namely, we can design the producer test case with random data.
Assuming that the type in question is defined as follows:
enum Tool {hammer,knife,screwdriver}
we can render the test data with:
responseBody = Arranger.some(Tool.class);
This piece of code uses test-arranger, but there are other libraries that can do the same as well.
It selects one of the valid enum values.
Each time it can be a different one.
What does it change?
Now we can detect the 2nd scenario and after regex adjustment the 4th one.
So it covers the most severe scenarios.
There is also a drawback to consider.
The producer test is nondeterministic, depending on the drawn value it can either succeed or fail which is considered to be an antipattern.
When some tests sometimes fail despite the tested code being correct, people start to ignore the results of the tests.
Please note that producer test case with random data is not the case, it is in fact the opposite.
It can sometimes succeed despite the tested code is not correct.
It still is far from perfect, but it is an interesting tradeoff as it is the first strategy that managed to address the very severe 2nd scenario.
My recommendation is to use the producer test case with random data supported with a regex on the customer side.
Nonetheless, there is no perfect solution, and you should always consider what is important for your services.
Specifically, if the consumer can safely ignore unknown values, the recommended approach might be not a perfect fit.
I would like to have pre-emption calls in Asterisk. I think there is no Asterisk support for this feature so i'm trying to implement it following a simliar algorithm like the one showed in this thread: Asterisk - Pre-emption calls
So I'm having problems in this step:
check if B in call with lower priority caller( ASTDB or REALTIME or fastagi script).
I know how to check if B is in a call using for example DEVICE_STATE(device) cmd, but i can't achieve to know who is the other caller in order to see his priority.
So, How can I know if one users is in a call and who is the other caller inside this call?
Thanks a lot.
You can read variables of any channel using
SHARED(varname[,channel])
-= Info about function 'SHARED' =-
[Synopsis]
Gets or sets the shared variable specified.
[Description]
Implements a shared variable area, in which you may share variables between
channels.
The variables used in this space are separate from the general namespace
of the channel and thus ${SHARED(foo)} and ${foo} represent two completely
different variables, despite sharing the same name.
Finally, realize that there is an inherent race between channels operating
at the same time, fiddling with each others' internal variables, which is
why this special variable namespace exists; it is to remind you that variables
in the SHARED namespace may change at any time, without warning. You should
therefore take special care to ensure that when using the SHARED namespace,
you retrieve the variable and store it in a regular channel variable before
using it in a set of calculations (or you might be surprised by the
result).
Sure you have set variables first.
You can set in variables or in ASTDB name of current speaking channel using in-call macro
General complexity of any solution like you want is above average, need person with at least 1-2 year of extensive experience with *.
We have a BizTalk application where the order of messages being inputted is very important and has to be kept, meaning they have to be outputted in the same order. Normally ordered delivery would do the trick here.
However I read that ordered delivery is only guaranteed when you connect a receive location directly to a send port. The moment you use orchestrations the order delivery isn't guaranteed anymore. Is there a way to work around or fix this? Because this kind of ruins our whole application and we've been working on this for months.
I read a work around from Microsoft where they use an extra field which has a counter and where they use an end orchestration which checks the counters. But this is way too much work for us to do now. So this work around is a no go. Plus not all messages are translated which creates holes in our flow and not all messages are coming from the same source either which makes this work around useless anyway.
Any other ideas?
Check out this page.
It explains that if you have an orchestration that follows the singleton pattern to ensure only one instance of the orchestration exists, and you make sure you set the orchestration's receive port to ordered delivery, than you should get a valid end-to-end ordered delivery scenario
To provide end-to-end ordered delivery the following conditions must be met:
Messages must be received with an adapter that preserves the order of the messages when submitting them to BizTalk Server. In BizTalk Server 2006, examples of such adapters are MSMQ, MQSeries, and MSMQT. In addition, HTTP or SOAP adapters can be used to submit messages in order, but in that case the HTTP or SOAP client needs to enforce the order by submitting messages one at a time.
You must subscribe to these messages with a send port that has the Ordered Delivery option set to True.
If an orchestration is used to process the messages, only a single instance of the orchestration should be used, the orchestration should be configured to use a sequential convoy, and the Ordered Delivery property of the orchestration's receive port should be set to True.
Resequencing strategies for ordered delivery in BizTalk:
I recently responded to a LinkedIn user's question regarding ordered delivery options in BizTalk.
I thought it would be useful for people to understand some of the strategies for re-sequencing messages using BizTalk.
Often as an BizTalk Developer, you are required to integrate to line-of-business systems which are unchangeable. This can be for one or more of many different reasons. As an example, the cost of changing a system can be too high or the vendor license states that support may be withdrawn if the system is changed.
This would not normally represent a problem where the vendor has provided a thoughtfully designed API as a point-of-integration endpoint. However, as many Integration Developers quickly learn, this is very rarely the case.
What do I mean by a thoughtfully designed API? Well, aside from all the SODA principals (service composition, fault contracts etc.), the most important feature of an API is to support the consumption of data which arrives in the wrong order.
This is a fairly simple thing to do. For example, if you are a vendor and you provide a HTTP operation as your integration point then one of the fields you could expose on your operation is a time-stamp or, even better, a sequence number. This means that if a call is made with an out-of-date payload, the relevant compensating mechanism can kick-in - which can be as simple as discarding the data.
This article discusses what to do when the vendor has not built this feature into an API, and as an integrator you therefore are forced to implement end-to-end ordered delivery as part of your integration solution.
As stated in my response to the user's post on LinkedIn (see link above), in BizTalk ordered delivery in any but the simplest of scenarios is complicated at best and at worst can represent a huge cost in increased complexity, both in terms of development and support. The basic reason is that BizTalk is designed to be massively concurrent to enable high throughput, and there is a direct and unavoidable conflict between concurrency and ordering. Shoe-horning E2E ordered delivery into a BizTalk solution relies on artefacts such as singleton orchestrations which introduce complexity and increase both failure rate and cost-per-failure numbers.
A far better solution is to maintain concurrent processing to as near as possible to the line-of-business system endpoints, and then implement what is called a re-sequencer wrapper around each of the endpoints which require data to be delivered in the correct order.
How to implement such a wrapper in BizTalk depends on some factors, which are outlined in the following table:
|Sequencing |Messages|Database |Wrapper |
|field |are |integration?|strategy |
| |deltas? | | |
|--------------|--------|------------|----------------------------------|
|n of a total m| N | Y |Stored procedure |
|n of a total m| N | N |Singleton orchestration |
|n of a total m| Y | Y |Batched singleton orchestration |
|n of a total m| Y | N |Batched singleton orchestration |
|Timestamp | N | Y |Stored procedure |
|Timestamp | N | N |Singleton orchestration |
|Timestamp | Y | Y |Buffer table with staggered reader|
|Timestamp | Y | N |Buffer table with staggered reader|
The first factor Sequencing field relates to the idea that in order to implement any kind of re-sequencer wrapper, as a minimum you will require that your message data contains some sequencing information. This can take the form of a source time-stamp; however a better, though rarer, kind of sequencing information consists of a sequence number combined with the total number of messages, for example, message 1 of 10, 2 of 10, etc..
The second factor Messages are deltas? relates to whether or not the payload of your message contains a single state change to the data or the sum of all past changes to the data. Put another way, is it possible to reconstruct the full current state of the data from this message? If the message payload contains just a single change then it may not be possible to reconstruct the state of the data from the single message, and in this instance your message is a delta.
The third factor Database integration? relates to whether or not the integration-entry-point to a system is a database. The reason this matters is that integrating at the database layer is a fairly common integration scenario, and if available can greatly simplify handling re-sequencing.
The strategies from the above table are described in detail below:
Stored procedure wrapper
This is the simplest of the resequencing strategies. A new stored procedure is created which queries the target data before making a decision about whether to update the target data. The decision can be as simple as Is the data I have newer than the data in the target system?
Of course, in order to implement this strategy, the target data also has to include the sequencing field of the source data, although an approximation can be made if necessary by relying on existing time-stamps which may already exist in the target data. The stored procedure wrapper can be contained either in the target database or ideally in a separate database.
Singleton orchestration wrapper
The idea behind this strategy is the singleton orchestration. This is a pattern you can implement to ensure that only a single instance of the orchestration will exist at any one time. There are many articles on the web demonstrating how to implement this pattern in BizTalk.
The core of the idea is that the singleton simply keeps a track of the most recent successfully processed message sequence (or time-stamp). If the singleton receives a message which is older than the most recent sequence it is simply discarded. This works because the messages are non-deltas, so the target system can commit only the most recent of a number of messages and the data will be in the most recent state. Only when data is committed successfully is the most recent sequence held by the singleton updated.
Batched singleton orchestration wrapper
This strategy is based on the Singleton orchestration wrapper above, except it is more complex. Rather than only keep the most recent sequence information in memory the singleton is required to create and hold a working set of messages in memory which it will re-order and then process once all expected messages from the batch have arrived. This is because the messages are deltas so the target system MUST receive each message in the order they were intended. Once the batch has been sent successfully the singleton can terminate.
To do this it is a requisite of the source data that it contain a correlation identifier of some description which allows the batch of messages to be defined. For example, processing a defined set of orders from a customer, the inbound messages must contain an identifier for the customer. This can then be used to route the messages to the singleton orchestration instance correlated with this customer. Furthermore the message sequence field available must be of the n of a total m form.
Once the singleton is initialised it assembles a working set of messages in memory and proceeds to populate it as new messages arrive. One way I have seen this done is using a System.Collections.Generic.List as the container for the working set. Once the list has been fully populated (list length = m) then it is assumed all messages in the batch have been received and the orchestration then loops over the working set in sequence and processes the messages into the target system.
One of the benefits of the batched singleton orchestration wrapper is it allows concurrent processing by correlation identifier. In the example above this means that messages from two customers would be processed concurrently.
Buffer table with staggered reader wrapper
Arguably the most complex of the strategies presented, this solution is to be used when you have delta messaging with a time-stamp-based sequencing field. It can be implemented with a database of some description which acts as a re-sequencing buffer.
It is worth noting here that this re-sequencing wrapper does not guarantee ordered delivery, but used well it makes ordered delivery highly likely.
As messages arrive, they are written into the buffer and in the same operation the buffer is reordered, so that the order of messages held in the buffer are always correct.
To create the buffer reader, have a receive location which reads the messages in the buffer before passing the messages to a send port with ordered delivery enabled, which then will process the messages into the target system. You can also use a singleton orchestration as an intermediary if your target system's API semantics are too complex for a send port.
However, using this wrapper as I have described it above will not enable ordered delivery, as the messages will almost certainly be committed to the buffer in the wrong order, which will result in the messages being processed into the target system in the same (wrong) order. This is where the staggered query comes in. This is a fancy way of saying your buffer query needs to only select data at intervals of time T, AND only select those rows where the row-number is lower than buffer total row count minus C.
This has the effect of allowing sequencing to occur over an appropriate timespan. T will be familiar to most BizTalk developers as the polling interval of some adapters (such as the WCF-SQL adapter). C is slightly more difficult to set, but by increasing this number you are reducing the chances that when you poll, you will miss a message older than the most recent one in your retrieved data set.
What T and C are depends on many things, although these values should be based on your latency SLA and your message volume (or throughput). As a guideline, if you have a SLA to deliver data into your target system within 30 seconds and you process 10 messages per second then T should be around 10 seconds and C should be around 100 rows.
Of course this only works if your messages for a given correlation id are sent by the source system during a short space of time (ideally back-to-back). The longer the interval between sends, the greater the required value of C, and the less effective the wrapper becomes.
One of the benefits of this strategy is you can also perform de-duplication of messages in the buffer if your data source is prone to sending duplicate messages and your target system endpoint is not idempotent. You can also use the buffer to implement FILO and other non-standard queueing semantics.
Conclusions
The strategies I have discussed here are ways of bending BizTalk to a task which is wasn't designed to do. As a result each has caveats around cost and complexity to support, and also may not work in certain scenarios. I would like to hear from anyone who has implemented other patterns for ordered delivery in BizTalk.