I could setup a cluster of wso2 ESB, following the dedicated doc, with a manager and two workers.
I am not sure about two points:
Does each worker node need it's own REGISTRY_LOCAL database ?
With both workers using the same db it works, but I'm not sure it is the way to do and the doc isn't clear about that.
Adding Data Services as a feature ?
Almost no docs about that, but not being able to fetch more than one row is a big limitation for me, so is it possible to add this feature in a clustered environment or is it better to separate the Data Services Servers from the ESB ones ?
If someone has experience in that kind of stuff I will really appreciate a feedback.
Thanks
You can use the same registry database for all the members in the cluster so that they can communicate with each other using that. You may refer my blog [1] for further information. Personally I have never used local registry DBs for each member in the cluster setup.
You cannot fetch multiple records when you install the DSS feature into the ESB. Also installing the feature will incur some performance overheads in the ESB. Therefore I thoroughly recommend you to use a separate DSS instance to get your work done. It also separates the concerns clearly which sounds good.
[1] http://ravindraranwala.blogspot.com/2015/09/wso2-esb-worker-manager-cluster-without.html
Related
Is it an anti pattern if multiple micro-services read/write to/from the same DynamoDB table?
Please note that:
We have a fixed schema which will not change any soon.
All read/write will go though REST services though API gateway + Lambda
micro-services are deployed on Kubernetes or Lambda
When you writing microservices it is advised to not share databases. That is there because it provides for easy scaling and provides each to services to have their own say when it comes to their set of data. It also enables one of these services to take a call on how data is to be kept and can change it at their will. This gives services flexibility.
Even if your schema is not changing you can be sure of the fact that one service when throttled will not impact the others.
If all your crud calls are channeled through a rest service you do have a service layer in front of a database and yes you can do that under microservices guidelines
I will still not call the approach an anti pattern. Its just that the collective experience says that its better not to talk to one databases for some of the reasons I mentioned above.
I have a different opinion here and I'll call this approach as anti-pattern. Few of the key principles which are getting violated here:
Each MS should have it's own bounded context, here if all are sharing the same data set then their business boundaries are blurred.
DB is single point of failure if DB goes down, all your services goes down.
If you try to scale single service and spawn multiple instances, it will impact DB performance and eventually, will effect the performance of other microservices.
Solution,..IMO,
First analyze if you have a case for MSA, if your data is very tightly coupled and dependent on each other then you don't need this architecture.
Explore CQRS pattern, you might like to have different DB for read and write and synchronize them via event pattern.
Hope this helps!!
I was reading through Nevatech Sentinet the last week and I'm currently asking myself the following question: "When NevaTech Sentinet exists, with all these named features, why should anyone use BizTalk with ESB Toolkit and extend it with Sentinet?"
Does I see there something wrong, but Sentinet is able to handle everything and more what BizTalk with ESB Toolkit is also able to do?
Essentially you are talking about 2 very different products here.
Sentinet is a very good tool if you are thinking about centralized API management. It is very good in what it does in that niche.
BizTalk on the other hand is a ESB, using a publish/subscribe architecture. BizTalk also has various ways of connecting to non-trivial systems like SAP, DB2, Siebel, MSMQ, etc... It can also do EDI/AS2/X12, flat file parsing and so on.
You can set it up as a ESB and/or message broker/hub/etc...
In your specific case (I'm guessing web services related?) it might seem that BizTalk and Sentinet are similar, yet the two are very different and actuallly complement each other rather nicely.
As you see these are 2 completely diverse products and together they actually might be a perfect match in your case.
Some more clarification after your comment:
Using BizTalk does not necessarily mean you have to use ESB Toolkit. BizTalk can perfectly act as an ESB without the ESB Toolkit.
The benefits of using Sentinet is definitely API management. What you are focusing on when "doing" API management with Sentinet is to form a single layer of API's within Sentinet. Often you would point all your clients (both internal or external) to Sentinet, where you would host all of your services virtually. You gain a lot of control that way and can add security, versioning, load balancing, SLA reporting, etc... to your existing services without any hassle.
Another thing Sentinet is quite good at is low-latency services. This is something BizTalk is not particulary good at, since it will persist everything to it's database to prevent losing messages. (I once used it in a POC and setup a virtual service calling an existing, external service with additional enrichment and easily covered 200+ trx/seconds).
BizTalk on the other hand is middleware. That's a whole other playing field.
It's very good in connecting different systems to each other using different protocols, mapping messages to other formats (xml, flat file or EDI), adding business logic to your flows, integration patters, long running flows, loose coupling, etc... you wouldn't want to use it as a virtual service since it will persist everything to it's message box!
Hopefully now you see they are both quite a different tool set.
They do play along nicely next to each other though: hosting your BizTalk web services in a virtual web service on Sentinet has a lot of advantages, especially in a fast-paced environment.
One addition to #Peter's great answer:
I almost always install the ESB ToolKit, if nothing else for the centralized exception handling (EsbExceptionDb). You may or may not want to use the itinerary and other services in the toolkit, but the exception DB is very handy when trying to debug and potentially resubmit messages.
I'm new to non-php web applications and to nosql databases. I was looking for a smart solution matching my application requirements and I was very surprised when I knew that there exist graph based db. Well I found neo4j very nice and very suitable for my application, but as I've already wrote I'm new to this and I have some limitations in understending how it works. I hope you guys could help me to learn.
If I embed neo4j in a servlet program then the database access I create is shared among the different threads of that servet right? so I need to put database creation in init() method and the shutdown in the destroy() right? And it will be thread safe.(every dot is a "right?") But what if I want to create a database shared among the whole application?
I heard that graph databases in general relies on a relational low level. Is that true for neo4j? But if it is then I see an high level interface to the real persistence layer, so what a Connection is in this case? Are there some techniques like connection pooling or these low level things are all managed by neo4j?
In my application I need to join some objects to users and many other classification stuff. any of these object has an unique id (a String). then If some one asks to view some stuff about object having id=QW then I need to load the vertex associate to object.QW. Is this an easy operation for graph datbases?
If I need to manage authentications, so as I receive the couple (usr,pwd) and I need to check whether exists this couple in my graph. Is the same problem as before or there exist some good variation for managing authentications?
thanks
If you're coming from PHP world in most cases you're better of running Neo4j in server mode and access it either via REST directly or use a client driver like https://github.com/jadell/neo4jphp. If you still want to embed Neo4j in a servlet environment, the GraphDatabaseService is a shared component, maybe stored within the ServletContext. On a per request (and therefore per-thread) basis you start and commit transactions.
Neo4j is a native graph database. The bare metal persistence layer is optimized for navigating from one node to its neighbors as fast as possible and written by the Neo4j devteam themselves. There are other graph databases out there reusing other persistence technologies for their underlying persistence.
Best thing is to run the Neo4j online course at http://www.neo4j.org/learn/online_course.
see SecurityRules
As the Neo4j is NoSql Graph Database,
Genration of the Unique ID you have to handle using the GUID(with 3.x autonincremented proery also supported for particular label),
as the Neo4j default genrated id is unique but can be realocated to the another object once the first assigned object is deleted,
I am .net developer in my project I used the Neo4j rest api it works well, i will sugesst you to go with that,as it is implemented using async-awit programing pattern, so long running operation you can pass to DB and utilize your web server resources in more prominent way.
I have been researching workflow foundation for a week or so now, but have been aware of it and the concepts and use cases for it for many years, just never had the chance to dedicate any time to going deeper.
We now have some projects where we would benifit from a centralized business logic exposed as services as these projects require many different interfaces on different platforms I can see the "Business Logic Silos" occuring.
I have had a play around with some proof of concepts to discover what is possible and how it can be achieved and I must say, its a bit of a fundamental phase shift for a regular C# developer.
There are 3 things that I want to achieve:
Runtime instanciated state machines
Customizable by the user (perform different tasks in different orders and have unique functions called between states).
WCF exposed
So I have gone down the route of testing state machine workflows, xamlx wcf services, appfabric hosted services with persistance and monitoring, loading xamlx services from the databse at runtime, etc, but all of these examples seem not to play nicely together. For example, a hosted state machine service, when in appfabric, has issues with the sequence of service method calls such as:
"Operation 'MethodName' on service instance with identifier 'efa6654f-9132-40d8-b8d1-5e611dd645b1' cannot be performed at this time. Please ensure that the operations are performed in the correct order and that the binding in use provides ordered delivery guarantees".
Also, if you call instancial workflow services at runtime from an sql store, they cannot be tracked in appfabric.
I would like to Thank Ron Jacobs for all of his very helpful Hands On Labs and blog posts.
Are there any examples out there that anyone knows of that will tie together all of these concepts?
Am I trying to do something that is not possible or am I attempting this in the right way?
Thanks for all your help and any comments that you can make to assist.
Nick
Regarding the error, it seems like you have modified the WF once deployed (is that #2 in your list?), hence the error you mention.
Versioning (or for this case, modifying a WF after it's been deployed) is something that will be improved in the coming version, but I don't think it will achieve what you need in #2 (if it is what I understood), as the same WF is used for every instance.
I am wondering if anyone has any thoughts on the best way to perform keyword searches on Amazon SimpleDB from an EC2 Asp.Net application.
A couple options I am considering are:
1) Add keywords to a multi-value attribute and search with a query like:
select id from keywordTable where keyword ='firstword' intersection keyword='secondword' intersection keyword = 'thirdword'
Amazon Query Example
2) Create a webservice frontend to Katta:
Katta on EC2
3) A queued Lucene.Net update service that periodically pushes the Lucene index to the cloud. (to get around the 'locking' issue)
Load balance Lucene(StackOverflow post)
Lucene on S3 (blog post)
If you are looking for a strictly SimpleDB solution (as per the question as stated) Katta and Lucene won't help you. If you are looking for merely an 'Amazon infrastructure' based solution then any of the choices will work.
All three options differ in terms of how much setup and management you'll have to do and deciding which is best depends on your actual requirements.
SimpleDB with a multi-valued attribute named Keyword is your best choice if you need simplicity and minimum administration. And if you don't need to sort by relevance. There is nothing to set up or administer and you'll only be charged for your actual cpu & bandwidth.
Lucene is a great choice if you need more than keyword searching but you'll have manage updates to the index yourself. You'll also have to manage the load balancing, backups and fail over that you would have gotten with SimpleDB. If you don't care about fail over and can tolerate down time while you do a restore in the event of EC2 crash then that's one less thing to worry about and one less reason to prefer SimpleDB.
With Katta on EC2 you'd be managing everything yourself. You'd have the most flexibility and the most work to do.
Just to tidy up this question... We wound up using Lightspeed's SimpleDB provider, Solr and SolrNet by writing a custom search provider for Lightspeed.
Info on implementing ISearchEngine interface for Lightspeed:
http://www.mindscape.co.nz/blog/index.php/2009/02/25/lightspeed-writing-a-custom-search-engine/
And this is the Solr Library we are using:
http://code.google.com/p/solrnet/
Since Solr can be easily scaled using EC2 machines, this made the most sense to us.
Simple Savant is an open-source .NET persistence library for SimpleDB which includes integrated support for full-text search using Lucene.NET (I'm the Simple Savant creator).
The full-text indexing approach is described here.