I'm looking to design a SOA service which in addition to its main requirements has a requirement for a small number of reports.
When should SOA services (a) include their own reporting functionality, and when should (b) reports be made available as part of a separate reporting service?
I guess (a) makes the service more self-contained, but (b) should probably be preferred when the organisation already has reporting services deployed?
You can/should report on individual services if the report is self contained and relates to data that changes frequently.
It is ok to use a centralised reporting when the data is immutable i.e. historic copy of the data. This way the services are still the owners of the data (and responsible for updating the data) and the data is still available for cross-service reports. This is a pattern I call aggregated reporting (you can see a draft of it here)
You can see an article I published on infoQ called bridging the gap between BI & SOA
Related
I design a data management strategy for a big IoT company. Our use case is fairly typical, we ingest large quantities of data, analyze them, and produce datasets that customers can query to learn about the insights they need.
I am looking at both Azure Data Explorer and the Data Warehouse side of Azure Synapse Analytics (a.k.a Azure SQL Data Warehouse) and find many commonalities. Yes, they use different languages and a different query engine on the backend, but both serve as a "serving layer" that customers use to query read-only data at a large scale.
I could not find any clear guidance from Microsoft about how to choose between the two, or maybe it makes sense to use them together? In that case, what is the best use case or type of data for each of the services?
If you can enlighten me please share your thoughts here. If you know about some guidance about the matter please reply with a link.
The classic and also the modern data warehouse pattern involve first designing a well curated data model, with documented entities and their attributes, creating a scheduled ETL pipeline that transforms and aggregates the raw data, big and small into the data model. Then you load and serve it. The curated data model provides stability, consistency and reliability when consuming these entities across an enterprise.
Azure Data Explorer was designed as an analytical data platform for telemetry. In this workload you do not aggregate the data first, but actually keep it close to the raw format as you do not want to lose data. It allows you to deal with the unexpected nature of security attacks, malfunctions, competitive behaviors, and in general the unknowns, as it allows looking at the fresh raw data from different angles and provide a lot flexibility.
This is why Azure Data Explorer is the storage for Microsoft Telemetry and also a growing set of analytical solutions like: Azure Monitor, Azure Security Center, Azure Sentinel, Azure Time Series Insights, IoT Central, PlayFab gaming analytics, Windows Intune Analytics, Customer Insights, Teams Education analytics and more.
Providing high performance analytics on raw data, with schema-on-read capability on textual, semi structured and structured data.
Quite a few of our partners and customers are adopting ADX for the same reasons.
Check out the overview webinar that describe these concepts in detail.
Azure Synapse Analytics packed SQL DW, ADF and Spark to have all the data warehouse pattern components highly integrated and easier to work with and manage. As we announced on the Azure Data Explorer Virtual Event, Azure Data Explorer is being integrated to Azure Synapse Analytics along side the SQL and Spark pools to cater for telemetry workloads - Real time analytics on high velocity, high volume, high variety data.
Check out some of the IoT cases Buhler, Daimler video,story, Bosch, AGL and there are more leading IoT platforms who are adopting Azure Data Explorer for this purpose. Reach out to us if you need additional help.
After reading this great article I thought about migrating our platform to micro-services architecture.
Our stack is Asp.Net Web API (Rest...) on the server.
Angular 2 in the front.
I wanted to make a little proof of concept to check if we should continue down this road.
As for my understanding, I need to take some chunks from our web app and slice it into micro services. As for the beginning, I want to take 2 screens I have, "Users" And "Purchase History" (each of them is too big to be micro service but this is just for the POC) and create each one of them as micro service.
I read that the UI should be part of the microservice, so should I create a new angular two app for each one of them?
If so, should I use rest to call for the rendered HTML?
Frontend and backend API, two services (components) – it’s already some kind of microservice architecture. The questions are how big your components, what kind of logic they have, will you have benefits if you split some logic to different services?
Per microservice architecture every service (component) of your system should have some dedicated logic (domain), solves some related problems, persists data to its own data-store, can be developed and deployed separately. In some cases, data-store can be shared between services.
So, the goal of splitting logic into different services is making your application easier to develop, maintenance, support and understand. Too many small services can bring a lot of overheads. To create a service, you need to spend additional development time, a service is also a deployment item, communication between services has network overheads. So, you should careful consider all pros and cons of splitting logic into services. Some balance should be found.
Going back to the question if "Users" and "Purchase History" are totally different, they don’t have common logic, can be stored in different databases and both are complicated enough, so you can split them into two services. The same about UI parts. The main thing is that splitting should bring you benefits - not overheads.
About using rest - it’s up to you, rest architecture is not required for microservice architecture but very often they are used together. Rest is about design of your services, how they expose API and so on.
I'm building a shiny app where users upload transaction data to get access to an analytics dashboard. Can I assure these people that their data is secure from sniffers/hackers and will be removed from the shiny server when their session expires? How does this actually work in Shiny? (Note that I'll be hosting my app on shinyapps.io)
This is not to do with shiny, but whatever server you're storing the data on, how you're using encryption/hashing, and software/app security methods you've used to protect against specific vulnerabilities.
Having said that, here's the (rather minimal, IMHO) security statement for shinyapps.io:
shinyapps.io is secure-by-design. Each Shiny application runs in its
own protected environment and access is always SSL encrypted. Standard
and Professional plans offer user authentication, preventing anonymous
visitors from being able to access your applications.
I would say that the burden will heavily fall on you to use good encryption and data storage practices.
There are many official and unofficial guidelines you can look to for guidance on data storage. One which big companies, particularlly companies going public, must follow is Sarbanes-Oxley.
From grtcorp.com:
The Sarbanes-Oxley Act (SOX Act) was passed by Congress and signed
into law in 2002 in response to major cases of financial fraud, of
which the rise and collapse of Enron is the best known. The overall
focus of the measure is on financial reporting responsibilities, and
ensuring that financial audits are genuinely independent.
However, SOX also includes provisions that relate to the security and
preservation of financial data. And the standards set out for its
implementation "recognized that senior management can't just certify
controls ON the system, these controls also have to control the way
financial information is generated, accessed, collected, stored,
processed, transmitted, and used through the system."
Senior management is thus held ultimately responsible for financial
data security, including putting in place appropriate controls and
procedures to ensure this data security. The good news is that
powerful tools, including data discovery and Data Masking, are
available to meet these standards.
I would also encourage you to familiarize yourself with OWASP's list of the top 10 major web app vulnerabilities:
https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
Both ESB and BPM tools that I have worked on take in some input , call multiple steps to fulfill a task. The difference that I have seen is that in ESB everything is automated - the process is automatically triggered and involves a number of external calls / data is transformed and sent to appropriate system for consumption. In case of BPM system , the process is either started manually or automatically and it involved series of decision steps some of which involve manual decision steps.Once the steps are done , the task is marked as complete. Is it possible to explain the clear distinction between BPM and ESB?
I think you are right that anything achievable with a BPM can be achieved just fine with an ESB and some Web UI that enables invocation of manual steps. But this is true if you are only looking strictly from the technical point of view. In a more mature SOA, where a lot of different parties and roles are involved, both ESB and BPM have their distinct place.
The distinction you're looking for is more "fuzzy" and it is about the focus of these tools, their intended end-users and the type of logic they compose. Here is my humble attempt at explaining the difference between ESB and BPM:
Focus and goals
ESB is more focused on enablement of interoperability, separation of concerns, and abstraction of technical details. It has much more of an infrastructural role, it also cares about monitoring, scalability performance, availability, state deferral. In the ESB your goal is to enable the creation of a federated interoperable layer, by abstracting all technical details and to exposing reusable functionality.
BPM is more business-focused and in a perfect world scenario it is managed by business people and business analysts themselves that modify processes without having any idea about any technical details. The BPMN language is all about workflows and is designed to be business-friendly. In the BPM your goal is to implement real business processes by using these building blocks.
Intended users
ESB services will be governed by architects and custodians (still, in accordance to requirements by business analysts).
BPM workflows will ideally be managed and modified by business people, business analysts and the like.
Composed logic
In a BPM the compositions (workflows) consist of business-oriented tasks (e.g. check customer loyalty level and give him a discount if user X approves and his level is gold).
In the ESB the compositions generally consist of more technical services (e.g. retrieve this from the database, combine with that from this component, transform with xslt). It is possible to have an orchestrated task that implements an entire workflow the way a BPM does, that is entirely business-centric and without any reusability whatsoever, but you don't have the handy tools and visualisation to be able to easily delegate the management of this business logic to business people.
Having said all the above, ideally if you have a mature SOA, you'll have a BPM layer on top of one or multiple ESBs and corresponding Service Inventories that have:
Entity and Utility services on the bottom (implemented in the ESBs)
Task, and in some cases Orchestrated Task services that compose said entity and utility services (implemented in the ESBs)
Workflows that use and reuse all these services in the BPM layer on top of the ESBs.
I hope this gave you a good initial idea of the differences. Feel free to ask if you need more information.
Plamen's answer is already very good. I disagree with the introduction
anything achievable with a BPM can be achieved just fine with an ESB
and some Web UI that enables invocation of manual steps
His later explanation puts this into perspective though.
From the top of my head some aspects a modern Business Process Management Suite (BPMS) handles (better) in comparison to an ESB:
Graphical modelling of the business process suitable for domain experts
No technical detail required, e.g. without service composition
the right granularity is reached when the task performer can be specific automated (system) vs manual (Human, possibly with system support).Below this granularity level the service composition start (ESB)
Simulation of the workflow (without or without services connectivity), based on assumptions or real-life audit data
Dashboard and Reporting features for operational control, tactical analysis and strategic continuous process improvement (all on business level / KPIs)
Organizational modeling, management of authorizations
task routing and assignment based on the business process model (e.g. roles) or dynamic based on conditions, business rules, decision tables, real-time analysis of user skills, workload and capacities, etc.
Management of the context of the business process, e.g. business objects, documents,references to data in external systems, references to other workflows belonging to the same business entity
Keeping an Audit Trail of all activities on business level (not a log file)
Comprehensive worklist management and search features
Features to operational management like definition and monitoring of business SLAs, priorities, benchmarks, criticalities, automated or manual task delegation
Organizational aspect like deputy management, business calendar
initiation of or changes to existing workflows based on defined internal or external technical or business events
BPMS and ESBs are complementary systems. The BPMS is the business layer which orchestrates the composite business services defined in the underlying ESB layer. The ESB layer is a technical mitigation layer which supports the definition of basics services, their aggregation into composite services and other aspects like transformation and standardization of data formats. Since the layers are close the products in both areas have adopted more and more features from the the other layer. The overlaps are increasing as the vendors extend their feature sets.
Depending on the complexity of the system landscape a comprehensive BPMS which covers many ESB features can make an ESB obsolete. An ESB which extends into the business layer is unlikely to reach the feature set and ease of use required by business users. If an ESB reached this business level then it would likely be rebranded and offered as a BPMS.
If you compare the website of ESB's like Mule and BPMS like Eclipse Stardust then the different focus (technical integration platform vs business process management: modeling, simulation, execution, reporting, analysis & improvement) should become evident.
I've been reading about Firebase and playing with it for a short while. The idea (BAAS) and implementation are impressive, and having programmed with Javascript it seems a viable choice. Not having to deal with scaling and other server side concerns makes it even more attractive.
My question is: generally speaking, is Firebase a first class back-end candidate for any average data-based application? e.g. billing, CRM, e-commerce, social, location based, etc. I do not include super light or heavy extremes such as a basic chat, or a nuclear plant monitor...
The answer may not be a clear yes/no, but was it built to support the general application space, or just stand out as a real-time read/write data service?
Would appreciate answers based on experience and existing production applications.
Thanks
Yes, Firebase is intended to be a first class back-end for any data based Web, iOS or Android application. The service offers real-time data reads and writes, but also comes with a powerful and flexible security system that allows you to write secure client-only apps, without needing any server code to enforce data boundaries.
There are several apps in production listed on the front page as customer and on the app showcase page on https://firebase.google.com/customers/
Firebase is now more capable and is considered as a full stand-alone back-end, especially after the introduction of cloud function. https://firebase.google.com/docs/functions/
Firebase may not have support for transaction spanning multiple business objects.
e.g. When a sales order is booked then it needs to update inventory for multiple items, update billing in receivables, give sales credit to multiple sales persons etc.
Firebase team is supposed to come up with a database trigger option which will make all these happen.