I've always been of the opinion an internal development group should really only be building/maintaining three applications.
An internal composite/pluggable/extendable application.
The company website.
(Optional) A mobile version of #1 for field employees.
I'm a consultant, and everywhere I go, my clients have dozens of one-off applications in the web and on the desktop for every need no matter how related to the others. Someone comes to IT and says "I need this", and IT developers turn around and write another one-off ASP.NET application, or another WinForms app.
What's your opionion? Should I embrace the "as many apps as we want/need" movement? I assume it's common; but is it sensible?
EDIT:
A colleague pointed out that it depends on the focus of the development - are you making apps or are you making a system? I guess to me, internal development is about making a system; development of shippable software products, like MS Word, iTunes, and Photoshop, is about making apps.
All of them?
Wow do I ever agree with you. The problem is that many one-off applications will (at some point) each have many one-off maintenance requests. Anything from business rule updates to requests for new reports. At some point the ratio of apps that need to be maintained to available development staff is going to be stretched/taxed.
From my perhaps (limited?) vantage point, I'm starting to think #1 and #3 could be boiled down to Sharepoint. Most one-off applications where I work (a large 500+ attorney law firm) consist of one or more of the following:
A wiki
A blog
Some sort of list (or lists joined together in some type of relationship), which can be sorted and arranged in different ways.
A report (either a Sharepoint data view or a SQL Server Report work just fine)
Or, the user just wants to "make a web page" and add content to it. But only they should be able to edit it. Except when they're out of the office, and then, etc...
Try to build any one of the above using [name your technology], and you've got lots of maintenance cycles to look forward to (versus a relatively minor Sharepoint change).
If I could restate what I think is your point: why not put most of your dev cycles to work improving and maintaining a single application that can support most of your business' one-off needs, rather than cranking out an unending stream of smallish speciality apps?
This question depends on so many things and is subjective besides. I've worked at companies that have needed several different apps because we do business in discreet silos. In that case, an internal group may not build and maintain apps, but may build several, with another group that is responsible for maintenance.
Also, what do you mean by "app"? If you broaden the term enough, then you could say "it's all just one big app".
In short, I think the main consideration is the capacity of the group and what business needs are.
I think there should be internal development teams that each has a system which may contain multiple applications within it. To take a few examples of what I mean by systems:
ERP - If you are a manufacturer of products, you may need a system to keep track of inventory, accounting of books and money, and other planning elements. There are a wide range of scales of such systems but I suspect in most cases there is some customization done and that is where a team is used and may end up just doing that over and over if the company is successful and a new system is needed to replace the previous one as these can take years to get fully up and running. The application for the shop floor is likely not the same one as what the CFO needs in order to write the quarterly earnings numbers to give two examples here.
CRM - How about tracking all customer interactions within an organization that can be useful for sales and marketing departments? Again, there are many different solutions and generally there is customizations done which is another team. The sales team may have one view of the data but if there is a support arm to the company they may want different data about a customer to help them.
CMS - Now, here I can see your three applications making sense, but note what else there is beyond simple content.
I don't think I'd want to work where everything is a home grown solution and there is no outside code used at all. Lots of code out there can be used in rather good ways such as tools but also components like DB servers or development IDEs.
So what's the alternative to several one-off applications? One super-huge application that runs everything and everything? That seems even worse to me...
Related
I am looking for some light in the complexity of architectural selection, before starting the development of a CMS or CRM or ERP.
I was able to find this similar question: A CRM architecture (open source app)
But it seems old enough.
I watch and read recently several conferences, discussions about monolith vs distrubuted, DDD philosophy, CQRS and event driven design, etc.
And I panic even more than before on the architectural choice, having taken into account the flaws of each (I think).
What I find unfortunate with all the examples of microservices and distributed systems that can be found easily on the net is that they always take e-commerce as an example (Customers, Orders, Products ...). And for this kind of example, several databases (in general, a NoSQL DB by microservice) exist.
I see the advantage (more or less) ==> to keep a minimalist representation of the necessary data for each context.
But how to go for a unique and relational database? I really think I need a single relational database, having worked in a company producing a CRM (without access to the source code of the machine, but the structure of the database), I could see the importance of relational: necessary for listings, reports, and consult the links between entities within the CRM (a contact can have several companies and conversely, each user has several actions, tasks, but each of his tasks can also be assigned to other users, or even be linked to other items such as: "contact", "company", "publication", "calendarDate", etc. And there can be a lot of records in each table (+ 100,000 rows), so the choice of indexes will be quite important, and transactions are omni-present because there will be a lot of concurrent access to data records).
What I'm saying to myself is that if I choose to use a microservice system, there will be a lot of microservices to do because there would really be a lot of different contexts, and a high probability of having a bunch of different domain models. And then I will end up having the impression of having to light each small bulb of a garland, with perhaps too much process running simultaneously.
To try to be precise and not go in all directions, I have 2 questions to ask:
Can we easily mix the DDD philosophy with a monolith system, while uncoupling very small quantity (for the eventual services that should absolutely be set apart, for various reasons)?
If so, could I ask for resources where I can learn a lot more about this?
Do we necessarily have to work with a multitude of databases, and should it necessarily be of the kind mongoDb, nosql?
I can imagine that the answer is no, but could I ask to elaborate a little more? Or redirect me to articles that will give me clear enough answers?
Thank you in advance !
(It would be .NET Core, draft is here: https://github.com/Jin-K/simple-cms)
DDD works perfectly as an approach in designing your CRM. I used it in my last project (a web-based CRM) and it was exactly what I needed. As a matter of fact, if I wouldn't have used DDD then it would have been impossible to manage. The CRM that I created (the only architect and developer) was very complex and very custom. It integrates with many external systems (i.e. with email server and phone calls system).
The first thing you should do is to discover the main parts of your system. This is the hardest part and you probably get them wrong the first time. The good thing is that this is an iterative process that should stabilize before it gets to production because then it is harder to refactor (i.e. you need to migrate data and this is painful). These main parts are called Bounded contexts (BC) in DDD.
For each BC I created a module. I didn't need microservices, a modular monolith was just perfect. I used the Conway's Law to discover the BCs. I noticed that every department had common but also different needs from the CRM.
There were some generic BCs that were common to each department, like email receiving/sending, customer activity recording, task scheduling, notifications. The behavior was almost the same for all departments.
The department specific BCs had very different behaviour for similar concepts. For example, the Sales department and Data processing department had different requirements for a Contract so I created two Aggregates named Contract that shared the same ID but they had other data+behavior. To keep them "synchronized" I used a Saga/Process manager. For example, when a Contract was activated (manually or after the first payment) then a DataProcessingDocument was created, containing data based on the contract's content.
Another important point of view is to discover and respect the sources of truth. For example, the source of truth for the received emails is the Email Server. The CRM should reflect this in its UI, it should be very clear that it is only a delayed reflection of what is happening on the Email Server; there may be received emails that are not shown in the CRM for technical reasons.
The source of truth for the draft emails is the CRM, with it's Email composer module. If a Draft is not shown anymore then it means that it has been deleted by a CRM user.
When the CRM is not the source of truth then the code should have little or no behavior and the data should be mostly immutable. Here you could have CRUD, unless you have performance problems (i.e. millions of entries) in which case you could use CQRS.
And there can be a lot of records in each table (+ 100,000 rows), so the choice of indexes will be quite important, and transactions are omni-present because there will be a lot of concurrent access to data records).
CQRS helped my a lot to have a performant+responsive system. You don't have to use it for each module, just where you have a lot of data and/or different behavior for write and read. For example, for the recording of the activity with the customers, I used CQRS to have performant listings (so I used CQRS for performance reasons).
I also used CQRS where I had a lot of different views/projections/interpretations of the same events.
Do we necessarily have to work with a multitude of databases, and should it necessarily be of the kind mongoDb, nosql? I can imagine that the answer is no, but could I ask to elaborate a little more? Or redirect me to articles that will give me clear enough answer
Of course not. Use whatever works. I used MongoDB in 95% of cases and Mysql only for the Search module. It was easier to manage only a database system and the performance/scalability/availability was good enough.
I hope these thoughts help you. Good luck!
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Our product is a web based course managemant system. We have 10+ clients and in future we may get more clients. (Asp.net,SQL Server)
Currently if one of our customers need extra functionality or customised business logic, we will change the db schema and code to meet the needs.
(we only have one branch code base and one database schema)
To make the change wont affect each others route, we use a client flag, which defined in a web config file, thus those extra fields and biz logic only applied to a particular customer's system.
if(ClientId = 'ABC')
{
//DO ABC Stuff
}
else
{
//Normal Route
}
One of our senior colleagues said, in this way, small company like us can save resources on supporting multiple resources.
But what I feel is, this strategy makes our code and database even harder to maintain.
Anyone there crossed similar situation? How do you handle that?
Update:If this is not a right question for SO, can someone move this question to a proper stackexchange site?
Update2:
you are right. The code is becoming smelly now, and I quite sure will be a nightmare sooner or later. Our company is doing the product and to save the effort, later products for other customers are based on the previous one. I know the ideal way is seperate the #e-j-brennan dev teams into 2 parts. One team works on core product and made it highly customisable, and team two works on customising for a particular client. However if since our company is so small, it is really a dilemma situation. :(
I think you need to decide if you sell custom software, that you tailor for each client, or 'off-the-shelf' software that is one-size-fits-all (and maybe customizable thru functionality you provide).
When you only have a handful of clients like you do now, you can get away with what you are doing, but I can almost guarantee that if you continue down this road, and your client base increases and the amount of client-specific customization's increases as well, you will have a nightmare on hand; I've been thru this many times for multiple clients, and it always ends the same way. It all is manageable until it is not, and then it is a royal pain-in-the-neck that could make your life very difficult indeed.
If you decide you are a custom company, and want to have multiple versions of the software and database, that is fine, just make sure you charge the full cost for it - i.e. factor in that you may need to maintain multiple levels of source code and databases and factor in that upgrades are going to take many multiples of effort to rollout as you will need to test each client's code base.
If you decide you want to be an 'off-the-shelf' type of product, then your best bet is provide the ability for each client to customize their experience, without the need for code changes - i.e. built in the customization capability thru config screens and tables that control how things work - but everyone will still use the same underlying code and database. Much more work upfront, but saves you boatloads of time down the road.
I have also been in your position, and I agree it is a difficult one. In my case, I was building custom single-product sites for clients. While each site followed a similar layout and workflow, there had to be enough flexibility for each to have a wholly custom design, custom rules around shipping and coupons, and different merchant gateways and configurations.
After some years, we did end up with something maintainable. First, we created libraries to house all of our common code and put those libraries into a TFS project simply called Common. Then, we created a new TFS project for each site (not client, as many clients had multiple products/sites) and branched the applicable projects into them from Common. Next, we created a VS Template project that contained a skeleton of the site, including "design-less" views, controllers, and their action methods (remember, each site had the same basic flow). Also, each site ran on its own database, which was cloned from an otherwise unused and mostly empty Template DB.
With each site running on its own branch and DB, modifications could be made to the original flow and design that was installed by the template (which would never need to be merged back in) without affecting any other site. For customizing business methods, like shipping calculations, we could create a subclass of the common class and override where needed. Part of what enabled this was converting all our code to use Dependency Injection. Specifically, each Controller had injected Services, and each Service had injected Repositories. Merchant Processing was also coded to an interface and injected. Also worth mentioning is that this allowed us to hard-code all of the upsell logic for each site (you bought product X, so we recommend Y), which was much easier to create and maintain compared to defining complex configuration rules in our old upsell rule engine. I don't know if you have anything like that...
Sometimes we would want to make a change to the Common code itself, which was usually prompted by a specific need for a specific site. In that case, we'd make the change on that branch, merge it to Common, and then merge it to the other sites at our convenience (great for "breaking" changes or changes that also required a change to the DB). Similarly for DB changes, we would update the Template DB and then write a little script to update the other site DBs with the same schema changes ( still had to be smart and careful about it).
An added benefit was that we also created Mock repositories that would be used/injected in a "Design" build configuration, which enabled the designers to jump around the application and work on screens without literally submitting themselves to the workflow. It also allowed them to start working on a site before there was anything done on the back-end, which was very important for those anxious clients who need to "see something".
10+ clients is definitely not a small number with what you're talking about. Three was pain enough for me. We had over 30 sites running at one time, maintained by three developers and two designers.
Finally, I know it's outside the scope of your question and a bit presumptuous, but getting "final" client sign-off on design before the designers actually went about implementing it (and before devs did their thing) also saved us a lot of costly rework. I know no design is final, but increasing efficiency on the implementation end gave the clients less time to change their minds about the design they approved.
I hope that at least gives you some approaches to think about.
People working with systems that have to change or be customized, have developed patterns to handle such concerns.
You should definitely start by reading a good book on Inversion of Control. In short, you can build your systems by defining building blocks (contracts, expressed as interfaces) and provide multiple implementations. There are multiple benefits of such approach but to mention just two:
- you can handle customizations by providing diffent implementations of the same interfaces
- you can reconfigure your application statically or dynamically but both approaches are far more clean than your "if"
When it comes to the data layer, study the repository pattern. It helps to organize the data access in a way that you can switch between different providers. It fits great wiht ioc.
And just a technical tip - nhibernate supports dynamic properties. You just provide additional columns in the mapping and nh is able to support it from the same code base. This way you can target different databases with slightly different db schemas.
Our company builds several (Java) applications that loosely communicate with eachother via web services, remote EJB and occasionally via shared data in a DB.
Each of those applications are build and maintained by their own teams. 1 or 2 persons for the smaller apps, and almost 10 for the largest one. The total amount of developers is approximately 25 FTE.
One problem we're facing is that there are some big egos among the teams. Historically the team of the largest app has set up a code convention and general guide lines. For instance our IDE is Netbeans, we use Hg for SCM, build with Ant and emphasize to first use as much from Java EE as possible, if that doesn't suffice use an external library and only resort to writing something yourself as a last resort. Writing things like yet another logging framework, orm, cms or web framework is pretty much not allowed following these guide lines.
Now some of the smaller teams go against this and start using Eclipse, Git and Maven and have an approach of writing as much as possible themselves and only look at existing things if time is short or they 'just don't feel like writing it themselves'. Where the main team uses log4j, one of the smaller teams just started writing their own logging framework.
There have been talks going on about all teams adhering to the same standards, but these have been 'troublesome' at best.
Now the big question I'd like to ask: does it actually matter that different teams do things differently? As long as each seperate app implements its requirements and provides the agreed upon interfaces, should we really force everyone to use Hg, Ant, the same code conventions, etc etc?
There is not much harm in letting each team use the technologies that work best for them. In fact if you restrict teams to the "standard" way of doing things you'll stifle innovation and have bad morale.
But you don't want things to diverge too much. There a few things you can do to prevent libraries and tools getting out of hand. The first thing is to have regular rotation of each member through the teams to cross pollinate ideas. In this way the best ideas will spread through the teams.
You can also enforce a "rule of 3", which simply says it is ok to introduce a second library, tool, logging approach, whatever. But as soon as you want to introduce a 3rd one, you have to remove one of the first two. In other words it is ok to have 2 competing logging frameworks but if there are 3 logging frameworks, choose one to kill.
A 3rd idea is to let developers run regular presentations to the entire developer group to demonstrate the pros and cons of each idea or approach. Encourage lots of discussion and constructive criticism. The purpose is to try many things and let everyone find the best way as a group.
Finally, Management 3.0 talks a lot more in depth about how teams make decisions. Well worth the read.
I have been tasked with automating some of the paper forms in HR. This might turn into "automate all forms" eventually, so I want to approach this in a way which will be best for the long term and will be a good framework as this project grows.
The first things that come to mind were:
-InfoPath/SharePoint (We currently don't use SharePoint now, and wouldn't be an option for the next two years.)
-Workflow Foundation (I've looked into this and does not seem too attractive or appropriate)
Option I'm considering at this point:
-Custom ASP.NET (VB.NET) & SQL Server, which is what my team mostly writes their apps with.
-Leverage Infopath for creating the forms electronically. Wondering if there is a good approach to integrating this with a custom built ASP.NET app.
-Considering creating the app as an MVC web app.
My question is this:
-Are there other options I might want to consider?
-Are there any starter kits or VB.NET based open source projects there which would be a starting point or could be used as a good reference. Here I'm mostly concerned with the workflow processing.
-Any comnments from those who have gone down this path?
This is going to sound really dumb, but in my many years of helping companies automate paper form-based processes is to understand the process first. You will most likely find that no single person understands the whole thing. You will need to role-play the many paths thru the process to get your head around it. And once you present your findings, everyone will be shocked because they had no idea it was that complex. Use that as an opportunity to streamline.
Automating a broken process only makes it screw up faster and tell a lot of people.
As far as tools, my experience dates me but try to go with something with these properties:
EASY to change. You WILL be changing it. So don't hard-code anything.
Possible revision control - changes to a process may or may not affect documents already in route?
Visual workflow editing. Everyone wants this but they'll all ask you to drive it. Still, nice tools.
Not sure if this helps or not - but 80% of success in automating processes is not technology.
This is slightly off topic, but related - defect tracking systems generally have workflow engines/state. (In fact, I think Joel or some other FC employee posted something about using FB for managing the initial emails and resume process)
I second the other advice about modeling the workflow before doing any coding or technology choices. You will also want this to be flexible.
as n8owl reminded us, automating a mess yields an automated mess - which is not an improvement. Many paper-forms systems have evolved over decades and can be quite redundant and unruly. Some may view "messing with the forms" as a violation of their personal fiefdoms, so watch your back ;-)
model the workflow in terms of the forms used by whom in what roles for what purposes; this documents the current process as a baseline. Get estimates of how long each step takes, both in terms of man-hours and calendar time
understand the workflow in terms of the information gathered, generated, and transmitted
consolidate the information on the forms into a new set of forms for minimal workflow
be prepared to be told "This is the way we've always done it and we're not going to change", and to gently (a) validate their feelings, (b) explain how less work is more efficient, and (c) show concrete benefits [vs.the baseline from step 1]
soft-code when possible; use processing rules when possible; web services and html forms (esp. w/jquery) will go a long way if you have an intranet
beware of canned packages (including sharepoint) unless you are absolutely certain they encompass your organization's current and future needs
good luck!
--S
I detect here a general tone of caution with regards to a workflow based approach and must agree. Be advised about the caveats of most workflow technologies which sacrifice usability for flexibility.
My work place recently started a SOA initiative. After a year-long examination of the biggest vendors (IBM and Oracle) they have decided which one to use and are now in the process investing quite a lot of money in the whole SOA stack (application servers, BAM, process servers, ESB, UDDI-like solution etc).
How many organizations are really using a fully-blown SOA stack? Did this technology show any proofs of being better? I'm afraid of a 'Silver Bullet' syndrome.
I work at a SOA shop (and we sell our own stack...perhaps you bought ours!), and it can really help businesses become more agile...if it's done well.
The problems come when:
People start making everything into a service, and you end up with just as many interconnections and interfaces as you had before you inserted an ESB. This makes change very difficult.
If you're using BPM with human interfaces: people don't 'get' portlets. Instead of making individual portlets do one task each, they make them do lots of things, which defeats the object of BPM. I can expand on this lots if necessary, but this might not be relevant to you.
It's all implemented at once. It's a massive system change, so try and do it slice by slice. (for example: just front your existing systems with web services, and build a new UI on top. Then gradually replace the UI calls that went to the old system with ones to the new system.) This will aid user acceptance as well as be a much safer way to do things. Management possibly won't want this approach (it's harder to manage) but emphasise the benefits. A lot.
It's sold as codeless development. This doesn't exist, and probably never will. Even if you don't have to write any code, if you don't know how code works then yours will be incomplete, ill-thought-out or unmaintainable.
From what I've seen, if SOA's done well then your business can turn on a sixpence, and it's cool. If it's done badly then it probably won't be worse than your legacy system, but it won't be better, and you'll have had an expensive and painful time in between for nothing.
I could give you a customer list but I dunno if I should, so I'll leave it. Suffice it to say we have some massive, everyday brandnames using our stack end to end.