We have used Microsoft BizTalk 2006 to perform as a data hub for several interfaced system's information exchange.
Inside BizTalk, we use Receive Port, Receive Location, Flat-File Disassembler, Xml-Mapping, Send Port Group Filtering, Orchestration, Dynamic Send Port to determine the message flow.
Is there any idea on how to extract the information from BizTalk such that Admin can know what are the relationship between those interfaced systems (e.g. system A transfer budgetting data to system B etc)?
Thanks.
I don't believe there is any automated tool that can produce the documentation which demonstrates the relationship between interfaced systems as there just simply isn't enough metadata built into the product. However http://biztalk2010autodc.codeplex.com/ and http://biztalkdocumenter.codeplex.com/ are two good places to start at automating the documentation generation process.
I have seen the sort of documentation you are looking for in the past produced in Visio which provides an easy to understand overview of the various message routes/relationships etc.
Related
I will be asking a very subjective question, but it is important as I am looking to recover from failure to effectively use BlueZ programatically.
Basically I envision an IoT edge device that runs on a miniature computer (Ex: Raspberry pi or Intel Compute Stick). The device would then run AlpineLinux OS and interact with Cloud.
Since it is IoT environment, it is needless to mention the importance of Bluetooth BLE over ISM band. Hence the central importance of being able to customize and work with BlueZ.
I am looking to do several things with BlueZ BLE including but not limited to
Advertising
Pairing
Characteristic
Broadcast
Secure transport of data etc...
Since I will be needing full control over data, for data-processing and interacting with cloud (Edge AI or Data-science on Cloud) I am looking at three ways of using BlueZ:
Make DBus API calls to BlueZ Methods.
Modify BlueZ codebase and make install a custom bin.
(So that callback handlers can be registered and wealth of other bluez
methods can be invoked)
Invoke BlueZ using command line utils like hcitool/bluetoothctl inside a program using system() calls.
No 1 is where I have failed. It is exorbitant amount of effort to construct and export DBus objects and then to invoke BlueZ methods. Plus there is no guarantee that you will be able to take care of all BLE issues.
No 2 looks very promising and I want to fully explore how feasible it is to modify the BlueZ code to my needs.
No 3 is the least desirable option, but I want to have it as a fallback option nevertheless.
Given my problem statement, what is the most viable strategy forward? I am asking this aloud so that I do not make more missteps and cost myself time and efforts.
Your best strategy is to start with the second way (which you already found promising) as this is a viable solution and many developers go about this method in order to create their BlueZ programs. Here is what I would do:-
Write all the functionality of the system in some sort of flowchart or state machine. This helps you visualise your whole system and what needs to be done to reach your end goal.
Try to perform all the above functionality manually using bluetoothctl and btmgmt. This includes advertising, pairing, etc. I recommend steering away from legacy commands such as hcitool and hciconfig as these have been deprecated and have a very different code structure.
When stumbling upon something that is not the default in bluetoothctl/btmgmt or you want to tweak the functionality, update the source to do so.
Finally, once you manually get the system to perform the functionality that you need (it doesn't have to be all, it can just be a subset of the functions), you can move to automating the whole process. This involves modifying the source for bluetoothctl/btmgmt commands so that instead of manual intervention, everything would be event-driven.
This is a bonus, but if you can create automated tests using python or some other scripting language, then this would ensure that your system is robust and that previous functionality doesn't break when adding new ones.
By the end of this process, you'll have a much better understanding of the internals of bluetoothctl/btmgmt and D-BUS APIs that you might be able to completely detach your code from the original bluetoothctl/btmgmt or create the program from scratch.
You probably already know this, but when modifying the tools, this is the starting point for the source code:-
bluetoothctl - client/main.c
btmgmt - tools/btmgmt.c
For more references on using bluetoothctl commands and btmgmt, please see the links below:-
BlueZ D-Bus C or C++ Sample
Bluetoothctl set passkey
https://stackoverflow.com/a/51876272/2215147
Bluez Programming
Linux command line howto accept pairing for bluetooth device without pin
https://stackoverflow.com/a/52982329/2215147
Bluetooth Low Energy in C - using Bluez to create a GATT server
I hope this helps.
I am writing Corda App for financial transactions between companies. The incoming transaction initiates a business workflow at the receiving end. Can Corda flows be used as a general purpose BPMN software or is it a better design to use a separate BPMN software and trigger flow in that system?
It really depends on the kind of BPMN you have in mind.
However, within a flow you have full access to any JVM libraries. You can perform networking (see the Flow HTTP sample here), DB reads (see the Flow DB sample here), and anything else.
I am currently in the process of porting an existing application (BizTalk 2004) to a newer version of BizTalk. The current solution takes multiple types of EDI documents, modifies it if its necessary and sends it to our legacy system to be loaded and processed.
This process is developed using a combination of Receive Ports, Pipeline component, Send Ports and Maps, Schema and Message Queue Components. This solution uses 10 send & receive ports to handle various aspects of the process such as Bursting EDI into individual messages, Transforming Messages, Error handling, EDI Validation and Batching of EDI Messages. All the modification of EDI is done using Message Queue Components.
This solution does NOT use orchestration at all. I am considering implementing the current solution as a BizTalk orchestration. I have read up a bit on orchestrations and worked through few sample applications. But I am still very confused over what benefit of using orchestration, if a solution can be developed without it. I am sure I am missing something here. What additional benefit orchestration gives that the current solution does not?
Edit:...I should clarify the question...I can do this app without using Orchestration using content based routing & maps. My question is, if I am missing something by not using Orchestration?
If you can perform your task at hand with message based routing, an orchestration is overkill.
Orchestrations will help you with calling rules, or handling transactions. The following points can help you decide whether to use orchestration or not:
Is the handling Transactional
Is ordering of messages important
Are you going to process the message using business rules
Do you have to call external assemblies
A quote from "Microsoft BizTalk Server Pattern"
Orchestrations come at a considerable cost. Many of these costs manifest themselves as roundtrips to the messagebox, which means crossing a process boundary and writing to and reading from a database -the messagebox
An orchestration can potentially take twice as long for the same process. For example: A simple process of receiving a message and sending it will make 2 message hops with the messaging approach vs 4 with the orchestration.
Here are the steps for a messaging only solution
Receive the message via the adapter save it to the message box
Retrieve the message for the send port
vs:
Steps for Orchestration approach
Receive the message via the adapter and save it to the message box
Retrieve the message to start the orchestration
Do your mapping if you need to
Retrieve the item again for the send port.
Choose wisely
It sounds like you could re-implement the solution in a messaging only solution and don't need an Orchestration. If you can that's great, we prefer messaging only as they are simpler to maintain and generally more efficient. Orchestration are useful if you need to have a workflow of multiple actions, or special error handling that you can't easily do with a messaging only solution.
I am curious if it even makes consider BizTalk for the implementation of a pub/sub messaging architecture (basically what you can do with NServiceBus or MassTransit is all I really need). My manager tends to want to stick with frameworks provided directly from Microsoft and so as part of my due diligence as to which one to use I need to give a good set of pro/cons for both sides. Any guidance would be greatly appreciated!
One of the major cons of a Broker is that it is very difficult to version and upgrade. You would have to stop the flow of messages to upgrade a particular endpoint(s). A Service Bus allows endpoints to be autonomous and be upgraded independently.
Then there is a difference in terms of scale. With a Broker the tendency is to scale those up(vertical) versus a Service Bus which is built for scaling out(horizontal). You would also have to make the Broker highly available through some sort of HA setup(clustering usually). This combined with the cost of software to do so can become pretty cost prohibitive.
NSB in particular will be offering a paid support model so if your manager is nervous about not having someone on the other end of the line when something goes wrong, you can buy into support.
Biztalk is a broker and is more suited for EAI within the boundaries of different business services. A servicebus is fundamentally different. A comparison can be found here:
http://docs.particular.net/nservicebus/architecture/nservicebus-and-biztalk
If you can share some of your requirements I might be able to give some more guidance.
It's important to note that BizTalk is a server product for Enterprise Application Integration (EAI -- as Andreas mentioned). It's more involved and complicated than a framework.
Microsoft does have the Enterprise Service Bus Toolkit available to use within BizTalk, so that you can call your BizTalk environment your ESB. What they consider "ESB" may not be what you consider ESB. You can take a look at their ESB Toolkit page (http://msdn.microsoft.com/en-us/biztalk/dd876606.aspx), but it includes things like:
Dynamic (i.e., at run-time) message
transformation and translation.
Message routing can be content-based, itinerary-based, or context-based, and determined at run-time.
Of course, the publish-subscribe pattern is not the same thing as using a service bus.
BizTalk does do pub-sub well, whether or not you use the ESB Toolkit. It's extremely simple to publish a single message to the BizTalk "Message Box", and have the message routed to any and all subscribers. The pub-sub solution means that BizTalk acts as a broker, but this helps guarantee that messages do not get missed, and all messages are tracked. A BizTalk pub-sub solution has built-in extensibility points that allow us to add, change, or remove endpoints without impacting the rest of the solution.
All that being said, your requirements may not dictate extensive message reliability, monitoring, and tracking, so it could be that BizTalk is not the best fit for you. It is a large investment, and since the product can do so many different things all at once, it can be daunting at a first glance.
A new book was just published call Applied Architecture Patterns on the Microsoft Platform, which covers much of this. One of that book's authors, Richard Seroter, also published SOA Patterns with BIzTalk Server 2009, which would be essential reading if you do decide to go with BizTalk for your company.
I'm with Andreas on this - BizTalk is generally more suited to 'value add' integration and business process management, rather than ESB type activity. BizTalk is good at:
BPEL
Long Running / Compensated Transactions
EAI
Brokering / Mapping
Protocol changes (MQ to WCF, Flat File to SAP etc)
EDI, RFID
However, efforts have been made to use BizTalk as a service bus, notably the ESB Toolkit
I am looking into architectural patterns, Enterprise Services Bus (ESB) precisely. Upon reading this article Enterprise Integration, and with little to no experience I am wondering if BizTalk has is a ESB or is it just a EAI (Hub/Spokes or Bus)?
I found this NServiceBus and Biztalk, describing BizTalk as a central message broker.
Taking other ESB frameworks into account (NServiceBus and Rhino Service Bus). These frameworks have no central point to process messages.
Is Biztalk a EAI rather than an ESB?
Many thanks
BizTalk is punted by Microsoft as having ESB capabilities - see the BTS ESB toolkit
However, the term 'ESB' covers a very broad field, and there is a lot of subjectivity about an exact definition of an ESB. IMHO there are weak points in BizTalk's claim to be comprehensive as an ESB (in a > 2010 definition of the term).
BTS originated in the Hub-and-Spoke EAI era, before ESB became widespread.
BTS is more suited toward asynchronous processes than synchronous processes - latencies will vary depending on load on the system, throttling state, etc.
BTS is cumbersome when it comes to ease of versioning of services and schemas (new deployment is needed)
BTS is cumbersome when it comes to management of MANY services (e.g. Using BizTalk as a facade for all 5000 of your corporate SOA / Web Services will be painful)
FWIW we have found BTS a good fit for:
all of our synchronous and asynchronous EAI (i.e. formalized integration contracts between major LOB systems, and with trading partners), and the large number of adapters assists with integrating a wide number of protocols.
For Business Process and Business Monitoring capabilities
Addressing transactional and delivery reliablity - Biztalk has capability to retry, tracking and resumption of Suspended messages, which is useful over unreliable networks or when it comes to integration with unreliable systems.
Update, with some further comparative experiences
BTS is very centralised - ultimately, even a multi-server BizTalk cluster / group is dependent on Sql-Server. Queue based ESB products tend to be more decentralised (logically and physically), so loss of a few endpoint or queue servers should not pull the whole enterprise down.
Many queue based ESB's are built on open source technologies, with an eye on avoiding single vendor lock-in
Many contemporary ESB's seem to take a commodity-computing approach to scale out. Scaling out with products like BizTalk can become expensive.
On the plus side, the monitoring and administration capabilities of commercial offerings like BTS should not be underestimated - make sure any ESB you are considering has adequate auditing, instrumenting, retry, and diagnostic (WMI / SNMP / SCOM etc) capabilities - you'll need a dashboard to monitor the health of your bus, and there is nothing worse than not knowing where a message went. Here, centralisation administration and diagnosis is a plus.
BizTalk is a messaging and workflow orchestration platform, on which you can build ESB behaviours and capabilities. To make this easier, and standardise ESB implementation on BizTalk, Microsoft released the BizTalk ESB Toolkit - a set of guidelines, patterns and code.
The concepts of EAI and BPM have been around for a while, so there are many companies that have leveraged BizTalk to create solutions to these problems. Companies that host a full ESB on BizTalk server are far fewer, and adoption has certainly slowed in the advent of WCF/WF/NServiceBus and, of course, Azure Service Bus.
So in summary, BizTalk out of the box is nether EAI or ESB, but can do both with a number of developers applied to the problem.
By "EAI or ESB" I'm assuming you wanted to know if BizTalk follows the Hub&Spoke or the Bus architecture.
From an architecture patterns perspective, integration solutions roughly fall under one of the two patterns-
The Hub and spoke:
This involves a centralized message broker sending out messages to various receivers, while all the senders send their messages only to this broker.
Thus neither the senders nor the receivers need to be aware of each other.
This is typically what many people refer to as EAI (although it is absolutely possible to implement an EAI solution that follows the BUS pattern).
Solutions following this pattern are easy to develop and administer. All the routing logic is centrally managed at one place - in the hub.
But as you would have guessed, this has a glaring drawback - single point of failure. If the hub crashes everything comes to a halt. Also, this model doesn't scale very well.
BUS:
Enterprise Integration solutions developed around this pattern are generally referred to as ESB. There is no intelligent central authority here. All senders publish their messages on the bus. The receivers need to be intelligent enough to determine which messages are intended for them and take them off the bus.
Thus the senders and the receivers need only be aware of the bus. But here the routing logic is spread across the receivers so there is no single point of failure. Also this model is highly scalable. However such solutions are quite complex and difficult to administer.
Coming to the question which pattern does BizTalk follow- it is a hybrid of both these patterns.
the Hub-like appearance is very obvious with its centralized Messaging Engine, and a central MessageBox database. This gives one the simplicity and ease of administration which is typical of the hub approach.
But if you look at the BizTalk architecture, one can have a Host with its Host Instances spread across multiple servers. It is also possible to have the different BizTalk databases like MessageBox, Tracking, Ent SSO etc. configured on different servers. This makes BizTalk solutions more scalable and tolerant to faults than the run-of-the-mill hub implementations - which is a behavior usually attributed to the bus approach.
Hope this answers your question.
BizTalk is certainly an ESB. EAI is more of a loose concept - BizTalk can certainly be deployed to support EAI, and it can also do a lot more.
BizTalk is more than an ESB but certainly fits the bill. This link is a little old, but answers your exact question.
EDIT: Here is a more-recent MS link that gets into specifics of implementation.
BizTalk can be used as both EAI and ESB.
As for ESB, the BizTalk server architecture is publish-subscribed, a single message can be published to the messagebox which acts as the messaging backbone bus. That message can be received by one or more destination systems that are subscribed to that message. Of course there more capabilities and features that you can get by using BizTalk server like the mapper tool and the use of pipeline components for example.
For use as EAI, BizTalk offers you orchestrations that manage the business logic, LOB(Line of business) adatpers to connect to systems(also legacy), mapper tool, rules engine, and a lot of what you need in order to integrate the different systems in or outside your company.
Absolutely! Biztalk comes from an EIS background, which makes perfect sense for ESB as an infrastructure backplane for service-oriented architectures that span hybrid technical platforms.
At a previous company we chose Biztalk in preference to the IBM ESB product for reasons of functionality and lower cost.
It is Microsoft, so you get what you pay for, but still well worth looking into.
Biztalk Server withot "ESB Toolkit" Is not an ESB.
Because of the following:
Is a contract first, need to build you message types first.
Need to Plan the whole scenario first to minimize the impact of changes.
Changes requires Deployment which increases downtime.
Regarding to your qustion, Yes BizTalk Server is EAI Product
I agree with most of what's said here. It's a stretch to pitch BizTalk as an all inclusive EBS solution even with the EBS toolkit.
To address a couple points made here ...
•BTS is more suited toward asynchronous processes than synchronous
processes - latencies will vary depending on load on the system,
throttling state, etc.
BizTalk hosts with unchanged defaults are not ideal for a low latency. But those hosts are meant to be tuned. Out of the box configuration is not suitable for any situation where throughput is needed. In my experiences of walking into an organization where BizTalk has been shunned there is always an untuned single host setup sitting in the middle of it. It is somewhat analogous to making tables in a dbms with no indexes, getting performance issues and saying the dbms itself sucks.
•BTS is cumbersome when it comes to ease of versioning of services and
schemas (new deployment is needed)
Like with any development platform you need to have a deployment strategy. If schemas have version in the namespace you do not need to redeploy anything. A new version maybe deployed without taking anything down.
As far as the service endpoints are concerned BizTalk can host web services without the use of IIS (BizTalk can use HTTP.SYS to host just as IIS does). To host an inprocess service in BizTalk is merely a matter of importing a binding which can be done without stopping anything in BizTalk. In those end points you can implement versioning as well (like http:.../thing/v1, http:.../thing/v2, etc.).
Anyway ~5 years have passed I'm sure you have hit a conclusion before now :)
BizTalk can do both ESB and EAI, depends how you design your biztalk applications.