I have been getting a bit lost in the creation of my program architecture and I want to take a step back to see if I'm approaching it correctly.
I am wondering if my setup makes sense. I'm starting to think it doesn't.
I am creating intranet applications (We were creating Internet applications, but now the scope has changed). We use an onsite Active Directory (Windows Server 2012 R2). We have a SQL Server Database.
I have been building Front End Angular applications and ASP.NET Web API's to push and pull data. I am now implementing Authentication with Auth0 and it's been a nightmare.
What kind of program architecture would you setup in this scenario?
Much Appreciated.
SQL Server + Asp.Net Web Api + Angular JS forms a perfect architecture for building Single Page Applications (SPAs). This architecture is useful for building desktop like web applications, i.e. apps that runs over web but works like desktop apps.
If you can be more specific about the problem you are facing, you will be able to get better recommendations from so.
This architecture is widely adopted in many scenarios such as SPAs. With it, you will be able to keep your front-end highly decoupled from your backend services being able to support multiple front-ends on the same set of services and run quite a few integration scenarios.
Some of the downsides of such an approach will be the extra layer of complexity added to the application (which might force you to write more tests and handle different failure scenarios that wouldn't happen otherwise, for an example) and authentication routines since you will need to authenticate two heterogeneous environments (the .NET/IIS one and the JS/Angular one).
As for the authentication pain, token-based auth schemes seem the current way to go (such as Auth0) since they let you keep and send an environment-agnostic token which will be used by different layers of your architecture.
In that sense, your architecture makes sense.
However, since you're feeling some pain in its implementation, you might want to ask yourself if you really needed all of these. When you choose an architecture, you do so trying to accomplish some specific goals (multiple front-ends? specific performance requirements? maintainability? auditability?) and the more goals you try to accommodate in your architecture the more complex will become up to a point where the pains start outweighing the benefits.
So, what were you trying to achieve in the first place?
I am after recommendations for a framework (or project template) for rapid application development using C# on the back-end. It must support the following:
User login/authentication
SPA
Responsive client
Easy to understand client and server (not a steep learning curve like AngularJS)
Clean/uncluttered project structure (both client and server). Some OOTB ASP.Net project templates are very cluttered (eg the the VS2012/2013 SPA MVC/Knockout/BackBone template) - maybe I could be convinced, but just looking at a new project it puts me off instantly.
Easy DB access.
Flexibility on the client (for custom functionality eg adding maps and other UI libraries like D3) - don't want a client framework that you have to wrangle to paint outside the lines.
Would appreciate any/all suggestions/opinions.
Thanks
Tim
Like anything in software, there are so many ways to do this. You are essentially asking for a complete system architecture. You could try to be more specific, but your question is going gather opinionated answers and this will probably end up getting closed.
But I'm bored and will throwin a couple of pennies before that happens:
C# most likely means you'll be going .NET on IIS (though maybe its docker on linux?? refer back to my first line above)
User login/authentication
If its IIS, you could be running Windows Auth, or if in an enterprise, Kerberos or Federation. Or if its going against social sites, maybe OAuth? This one depends on what authentication your users need and is really separate of the other areas below.
SPA / Responsive client / Easy to understand...
You listed some very popular frameworks (Angular/Knockout) used in modern SPAs. You may not like them but these are becoming industry standards. Responsive web is pretty much bootstrap or foundation. Though, angular material seems to be gaining popularity too.
Easy DB access.
Whats your definition of easy? Looking for an ORM? If so, Entity Framework is popular in .net apps. Or if you are more of an SQL person, maybe LINQ is easier. Many options here too.
Flexibility on the client ...
Though you may not like some of them, using popular frameworks means you get a lot of developer support behind it - and a lot more answers here on SO if you have problems. One man's sunset is another man's sunrise. What you consider difficult might be viewed as easy by another. Hard to say whats most flexible but going popular means you got more help.
My company has a fairly old fat client application written in Delphi. We are very interested in replacing it with a shiny new web application. This will make maintenance a breeze and many clients want a web application.
The application is extremely rich in domain knowledge, some of which is out of our control. Our clients use the program to manage their own clients and report them to the government. So an inaccurate program is a pretty big thing. The old program has no tests. We are not sure yet if we will implement automated testing with the new one.
We first planned to basically start from scratch. But we are short handed and wanting to basically get everyone on the web as soon as possible. So instead of starting from scratch we've decided to try to make use of the legacy fat-client database.
The database is SQL Server and can be used in SQL Server 2008 easily. It is very rich in stored procedures, functions, a few triggers, and lots of tables with over 80 columns... But it is decently normalized. We want for both the web application and fat client to be capable of using the same database. This is so that if something breaks badly in the web application, our clients can still use the fat client and connect to our servers. After the web application is considered "stable", we'd deprecate the fat client.
Has anyone else done this? What tips can you give? We want to, after getting everyone on the website, to slowly change the database structure to take care of some design deficiencies. What is the best way to keep this in a data access layer so that later changes are easy?
And what about actually making the screens? Is there any way easier than just rewriting an 80 field form in ASP.Net? Are there any tools that can make this easier?
The current plan is to use ASP.Net WebForms (.Net 3.5). I'd really like to use MVC, but no one on the team knows it including me.
We are not sure yet if we will implement automated testing with the
new one.
Implement automated testing. What's the point in replacing one buggy program with another?
Good question, but "Slowly change" the db structure after getting everyone on the website, sounds like a joke...
I would rather take the opportunity to create a fresh db structure, write a bulletproof migration script for you db, that you can try out and rewrite a zillion times without any side effect fro your clients, and then write whaterver you want (fat/web) on the new db, have it tested and migrate everyone when it's ready.
I have a couple suggestions:
1) create a service layer to abstract away the dependance on the DAL. In a situation as you describe having a layer of indirection for the UI and BLL to rely on makes DB changes much safer.
2) Create automated tests (both unit and integration), especially if you plan on making fairly significant changes to the Domain or Persistance layers (BLL/DAL). To make this really easy you should always try to program to an interface. This makes your code more flexible as well as letting you use mocking frameworks (Moq is one I like) to ensure your tests truely are unit tests and not integration tests.
3) Take a look at DDD (http://domaindrivendesign.org/) as it seems to fit pretty well with the given scenario. At the very least there are some very useful patterns that can help make your application more flexible.
4) MVC isnt very hard to learn at all, it is however an easy way to get unit testing setup for the UI as a result of the MVC architecture (testing the controller and not the view). That said, there is no reason you couldn't unit test web forms, its just a bit more work. MVC really is just a UI framework/design pattern (more Model2 but we can ignore that for now). It gets you closer to the metal so to speak as you will be writting a lot more HTML and using a Model (the 'M') for passing data around.
For DDD take a look at Eric Evans book: http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215/ref=sr_1_1?s=books&ie=UTF8&qid=1317333430&sr=1-1
Hope that helps
ASP.NEt forms is a no starter, is completely inappropriate for something like this. I recommend to start with something like Creating an OData API for StackOverflow including XML and JSON in 30 minutes, then build your Web app on top of that (ie. push it to the client, use JQuery/Silverlight).
We need to develop quite a powerful web application for an investment bank. The bank IT would like us to build it on top of the SharePoint platform, but we would prefer to do pure ASP.NET programming.
The web-app should have the following characteristics.
1) It will be a site for bank's clients that will allow them to view their stock portfolios, get miscellaneous reports with graphs and charts, etc.
2) The web-app will also allow clients to send orders to the bank to buy stocks and perform other financial operations.
3) The number of users will be approximately 3 000 000 (total) and 20 000 at any one time.
We have never made any SharePoint programming, but as far as I know, SharePoint is primarily designed to create intranet sites for colleagues to communicate with each other and work more efficiently, to maintain a document library, etc.
However, the bank IT told us that SharePoint has in fact lots of other features that will help us make the project more efficiently - for example, it seems that SharePoint has some built-in scalability and high availability technologies.
I heard saying that SharePoint development is very tedious, that the platform cannot be very easily customized, etc.
The question is: is it better to create our web-app on pure ASP.NET and deal with scalability and other issues ourselves, or base it on SharePoint - taking into account that the web-app we need to create is non-standard and complex?
Thank you,
Mikhail.
UPDATE
In the answers, someone suggested using ASP.NET MVC. My another question is: should we use "classic" ASP.NET or ASP.NET MVC for such project (if we leave out the SharePoint option)?
Do you need document management? Do you need version management? Do you need to create "sites"? Do you need audience filtering? Do you need ECM (fancy word for CMS), Do you need collaboration stuff on your site? If your answer is no then SharePoint is not for you.
You said "We have never made any SharePoint programming" and for that reason alone I think you should not use SharePoint. You also say that your app is going to be "non-standard" and complex, another reason not to use SharePoint.
Sounds like you know ASP.NET so I would advice to stick with ASP.NET or ASP.NET MVC.
Hope this helps
The answer is simple, you should go with what you know. If you prefer to do it in ASP.NET then, that is what you should go with. Trying to learn a new technology on that size of a project will almost certainty cause you severe problems when trying to develop it. Can sharepoint scale to that number of users, probably, but you don't know how to make it do that. That is the real key.
They are correct SharePoint does have a lot of functionality out of the box, but that doesn't mean that it will make you more efficient, because you don't know all of the APIs etc. to access.
Actually, if you want to know the way to cheat. If they force you into using it, you can run ASP.NET applications under SharePoint (well kind of). You can tell SharePoint to essentially ignore a path in the site and use regular ASP.NET as a web application just like any other site does. Really, this isn't using SharePoint, but it can get you out of a bind, in the "Needs to use SharePoint to make them happy scenario".
Mayo suggested contacting MS. I have a feeling they already have a relationship with the bank and have provided some insight about the project. I would contact: http://www.mindsharp.com/ and see if they can help you out. They are a training company, but I bet that the owners would be willing to help consult, and I haven't found anyone with more knowledge on SharePoint than Todd Bleeker.
I'll not go into the merits of sharepoint, but suffice it to say that I have been developing in sharepoint since it was known as "digital dashboard" - it was just a javascript-encrusted today page for outlook. With respect to its .NET incarnations, it has taken me about 3 years to become what some might call "expert" on SharePoint 2007/MOSS.
First up, let me give you some warnings concerning the politics of these kind of jobs. As a contractor, ALL of my jobs over the last 6 years - covering shaerpoint 2003 and 2007 - WITHOUT fail, have been getting about me on site with a client who has demanded sharepoint, and a development shop with decent ASP.NET developers who have become hopelessly lost and more than likely have blown 95% of the budget on the last 5% of the project because they have embarked on writing custom extensions to the platform without fully understanding the product.
If clients, and the shops who service them, spent more time understand the product and studied it to see how they could change/streamline their business processes & requirements slightly to suit sharepoint instead of being rigid in their specs (that were ALWAYS written with next to zero real experience of the platform) and deciding to get custom development done, then more sharepoint projects would be delivered on time and on budget. Sadly, this is not the case.
So, number one: SharePoint 2007 is an excellent product, but please, for the love of jeebus, find yourselves some top gun sharepoint developers who really understands the product before you embark on this journey. If you don't, you will all go down in flames.
-Oisin
What a load of CRAP that sharepoint isn't cut out for what the op wants to use it for. Especially the "Do not get yourself wrapped up in SharePoint" comment from ChaosPandion. Maybe he thought it to difficult and gave up...
Sure SharePoint development takes some getting used to, but it is able to what is wanted by the op most definately. SharePoint is built using ASP.NET so anything you do in ASP.NET can be used / ported to SharePoint. It is not a standalone product, but a DEVELOPMENT PLATFORM. It will scale to serve that many users, using multiple WFE's (Web Front Ends) and a SQL Cluster as backend.
The question here is: is sharepoint the most suited platform for building this site? Then I would have to answer, probably not, seeing as the wanted functionality is almost all custom development. If you plan on doing web content management as well, then yes, SharePoint is definately worth looking into. Also, SharePoint takes away all (or at least most :-D) authorisation and authentication wories. It is Department of Defense certified. And if the offered out of the box security is not enough, just write an authentication provider (seeing as SharePoint uses ASP.NET's provider model).
To answer your questions:
The bank IT told us that SharePoint has in fact lots of other features that will help us make the project more efficiently - for example, it seems that SharePoint has some built-in scalability and high availability technologies.
SharePoint is farm based, to which you can add machines, having each machine perform a different task, which means either app server, index server, WFE, document conversion services., WFE's can be behind a load balancer to distribute requests. Also I want to mention the web content management again.
I heard saying that SharePoint development is very tedious, that the platform cannot be very easily customized, etc.
Like I said, SharePoint is based on ASP.NET, so it is as much customizable as ASP.NET is. You could even create an ASP.NET web site, put all UI in Controls and then use those is SharePoint, maybe even have the controls use it's own database. As for it being tedious, not really. It's just DIFFERENT and deployment / testing is not like normal deployment / testing. SharePoint uses so called solution files (.wsp files), to package up functionality and deploy it to the server. This IMHO makes it possible to deploy functionality in a very modular way. Furthermore, there are loads of cool open source projects out there that make sharepoint development much easier and also provide cool extensions to "pimp" your site and make it more fun and easy to use for end-users.
Nuff said....
SharePoint development can be tedious but I'd hardly say the platform cannot be easily customized. I recently began developing with it full time and so far, I impressed at it's flexibility and suitability for my application but my needs are quite different from what you've described.
I understand 2007 is a vast improvement over 2003 so perhaps your information is only outdated. I hear 2010 is going to again be a significant improvement.
It's your job to deliver the functionality that the customer desires. If they desire a SharePoint solution, unless there's some particular reason why SharePoint really is a weaker model, that's what you should be able to deliver. In the event that SharePoint isn't a good fit, you need to be able to explain why to the bank's satisfaction. I'm not convinced "We don't know SharePoint" is an acceptable response in this situation: the bank's inclination should, at that point, be to find someone who knows both technologies well enough to deliver a product in SharePoint or better explain why SharePoint isn't actually what they want.
UPDATE: After looking at this more I would add that I do not believe that SharePoint is for you. As I mention below SharePoint is for collaboration. If the users that come to the site require an isolated experience then SharePoint is more overhead than you need.
SharePoint is built on top of ASP.NET so you have everything that you want to do with ASP.NET in addition to what SharePoint provides. Anyone who says that it is difficult is trying to make it that way. You can deploy stand alone custom pages with 100% of your own code and it will run under sharepoint, or you can create new application pages that also contain any code you want to write, or you can simply add your own webparts that can be added to any page you choose with 100% of your own code.
Here is just one example.
Creating an Application Page in Windows SharePoint Services 3.0
What SharePoint offers on top of that is a whole different paradigm on collaboration tools. If you wish to leverage it (if not the cost on return is somewhat limited) you can build amazingly complex and integrated solutions that is build around the aggregation of data from across an enterprise.
That being said, do not go into it lightly. If deployed wrong or with a half understanding of where SharePoint excels and where it does not will result in a diaster. Unless you have the time to understand the core concepts of SharePoint I would warn against it but your client is right. If you do build it in SharePoint you get a great deal more flexibility. One right off the bat is the ability to mix authentication modes. I designed a solution that mixed custom forms authentication with an LDAP backend with Windows Authentication. Anyone could visit the same pages but your authenticated account could come from two different locations.
This is a matter of what kind of concerns you want to have in the application:
Building it to look and function your way, go with sharepoint.
Building it to have infrastructure for authentication, permissions, http/web security, scalability, backup, database maintenance PLUS getting it to look and function your way (but now way more under your control), go with a more pure .NET approach.
I would pick the one I am best at, as Kevin said above.
Edit
More about Kevins post: you can also have your application under sharepoint but with full access to the API, in my projects we do it as a normal ASP.NET application, with own masterpages and everything, but we still use the authentication, lists and doc libraries for uploads, roleassignments for permissions etc. Its a very viable hybrid.
You said,
I heard saying that SharePoint
development is very tedious, that the
platform cannot be very easily
customized, etc.
You have been misinformed about SharePoint. All SharePoint pages are ASP.NET pages. You can customize any of them, either directly, or by using Microsoft Office SharePoint Designer, which is free.
Get started at http://msdn.microsoft.com/en-us/sharepoint/default.aspx.
SharePoint is a lot of work and with that amount of users I personally (and being a SharePoint developer) wouldn't bother.
I would go down the ASP.MVC route in all honesty and not because it's new and the latest buzz technology. I would use it because it's hands down faster. This site for example is written in ASP.NET MVC and it handles all these requests per day on I think 3 servers. 2 front end and 1 database. Correct my if I'm wrong with that.
The problem with asking whether Sharepoint is easy to customize is that there's a wide range of levels of customization people are experienced with. And for some reason, most people also seem to think that whatever level they customized Sharepoint to is the extent to which anyone else would also try to customize Sharepoint.
It's hard to talk about degrees of customization in concrete terms. What is "customization" to me is wrangling with the core DAL, fighting with bugs in the CAML to SQL query optimizers, overriding the SPListItem hydration pipeline, etc. To others, "customization" might mean building some web part widgets and deploying them in a WSP. If you find that there is some impedance mismatch between your logical model and Sharepoint's working model, you will have a really hard time reconciling the two.
Welcome to the dark land of politics.
It's worth making sure that your team properly evaluate and understand any compromises that SharePoint will have you make. Asking here is a good start. Things I'd look at include:
What's the whole solution going to include? Often the administration of a site can involve as much or more development work as the front end. While the 3M+ user front end is the glamorous part it may not be the bulk of the work.
Are there reference sites for 20K+ simultaneous user SharePoint sites? Honestly? What kind of hardware did that require? Is that available?
Get a small group of experienced contractors in for a few weeks to properly estimate the work, both on ASP.NET MVC and SharePoint. Make sure they've worked on large sites. (There's plenty of contractors around at the moment!)
Also, anticipate failure. Have a fall-back option:
If the MVC technologists win out, expect heat from senior management, and possibly even a skunk-works we'll-do-it-properly-anyway project that duplicates your efforts.
If you do end up with SharePoint, listen very carefully to users throughout the development process and be prepared to create Web parts, MVC pages or whathaveyou to address problem points.
I've been in a similar situation where it turned out that there was heavy vendor influence at a very senior level. The senior team had bought into SharePoint and required it to be used for all internal systems; the OCTO (Office of the Chief Technologist) had mandated open-source technologies. It was fun to watch the fur fly in the middle.
(Our option in the end was to use a service-based architecture based on REST, which effectively booted the current version of SharePoint out of the system altogether.)
I would build this on SharePoint. It is quite suitable for big sites and many sites have already been built on it: topsharepoint.com
SharePoint (like all complex applications) does require sufficient knowledge that you do not seem to have at the moment which is a big risk in my mind. Don't listen to the nay-sayers though.. lack of knowledge is a common problem for devs dealing with SharePoint but it doesn't mean you can't make it do whatever you want.
Regardless, what other options do you have? I think the days of building completely custom CMS's have passed just as building completely custom Intranets are not cost effective anymore. There are many competitors to what they want to do with SharePoint (Umbraco, Sitecore, Sitefinity, etc) and most of them seem better than 100% custom.
So the answer might be neither ASP.NET or Sharepoint..
I was involved in a SharePoint(WSS) project that was very data centric. The project consisted of more than 500 lists that has very complex relations between them. The client also asked for more than 350 Reports. Don't tell me why did you use SharePoint from the beginning. It was a managerial decision and we already delivered the project after 14 months of pain (this is 6 months overdue to the deadline)
When we first started the project, we didn't know anything about SharePoint development (believe it or not). The management said that they will take the risk. They were very convinced that SharePoint is the optimal solution for anything!!!(well, that proved wrong at the end of the project).
Anyway, we were learning SharePoint while we were developing. Our development were mainly based on SharePoint designer to customize all the AllItems/NewForm/EditForm/DispForm for every list to provide the needed logic/validation that the client asked for (Using JavaScript). We also implemented around 15 Custom Fields (e.g. master-details fields). We also made an event receiver to handle all the adding/updating/deleting... events for all the lists in the site. Plus around 40 ASP.Net user controls.
The main problems that faced us (we worked-around it but sadly in an inefficient way)
1- The Client asked for a search web part in each AllItems.aspx. The search web part should have multiple keys for the client to search with. We did that using Form Web parts using SPD no problem. But the real problem was how to search for a related field that was not in the current list. (So, in such cases we had to save these fields values in our list to be able to search for (crap, I know!!)). You might ask, why didn't you implement ASP.Net user controls for such task? Well, that would require us to forsake the default AllItems web part, and were already customized hundereds of AllItems.aspx pages with alot of customization that would take us a lot of time to reimplement them from the beginnging. Also, even if we used user controls, CAML is very inefficient in retrieving data from multiple related lists!
2- I think you can guess this one, if we've already faced a big time doing search web parts, how on earth will be able to do the 350 reports!!:D But we figured out a work-around (as usual :S) we made an Access DB file with links to all the 500 sharePoint lists, then we implemented a user control that has a report viewer control. This user controls takes an ordinary T-SQL Query to query on the Access DB, the Access DB retrieves the data from the SharePoint DB and pass it back to the user control which views the DataSet on the report viewer.
There are other administration related problems, but I would like to focus on the development here.
So, after I showed you the picture (sorry for the long post). What do you think was the best SharePoint development technique that we should have taken in such a data centric Project, if any?
I heard that some companies doesn't use lists at all in such projects, and builds there own SQL database tables instead of the SharePoint Database. But I can't keep my self from wondering, If I'm making my own DB, and hence implementing my CRUD web parts from scratch (We will also lose the security module benefits provided by SP Lists), what would be the benefits of SharePoint?
Once again I apologize for the long post.
I think you found out exactly what I did. Sharepoint just isn't good at handling large enterprise type applications. We ended up creating a custom database to house our data. We used Webparts for the user interface, but otherwise, the entire application was independent of Sharepoint.
In my opinion, Microsoft is overselling Sharepoint. It's actually good at team collaboration sites and simple Excel services applications, but anything beyond that it just isn't capable of handling.
I have to disagree with both Geoff and Abu in regards to SharePoint being a bad choice for large enterprise applications.
As you state yourself Abu your team were learning on the job as you had no SharePoint development experience, the issues you faced was more a Management error that a platform problem, management should have brought in SharePoint contractors to work alongside your team to help build what sounds to be a fairly complex system.
As a developer who has worked with SharePoint for a number of years many of the projects I have worked on some that I myself would not have believed suitable for SharePoint with in my first few years of developing on this platform, however now with more experience I know how to leverage the power of the platform far better and I realise the advantages gained using SharePoint for projects of that nature. That said I have a number of issues with parts of the platform but this is no different to any other platform I have worked on including parts of the ASP.Net platform.
If I was asked to develop a solution using a bespoke Java based system (or perhaps the new MVC platform) I am sure I would experience many problems similar to what you experienced where I simply don’t know what the right approach would be. That would not in any way be an issue with the platform but more with my inexperience.
I am sorry to hear that both of you experienced pains working within the bounds of the SharePoint platform that was forced on you by your management. Though I am disappointed that you are so fast to point the blame away from yourselves and your management.
Was SharePoint the best platform for your projects I can’t say, but that doesn’t make it a bad platform for enterprise applications.
I disagree with Geoff that SharePoint is no good for large enterprise type apps. You have to remember from the start that SharePoint is a development PLATFORM. This means it gives tyou a lot of functionality out of the box, but is extremely customisable.
Being a platform does not mean every bit of customisation needs to be done based on SharePoint lists. Seeing as it is built on ASP.NET you can do anything in SharePoint you could do in ASP.NET as well.
I have built a great deal of ASP.NET apps that were hosted in SharePoint, letting SharePoint do the authentication etc.
I have to say though, determining where SharePoint should stop being your base and you should switch to regular ASP.NET is sometimes hard..
If you are looking for a data-centric solution in SharePoint, the best solution is to use the Business Data Catalog (BDC). This keeps your rich data relationships and all of the SQL (or other DBMS) goodness you want where it should be - in a repository designed to be optimal at storing data.
For an overview of what the BDC functionality can do, see this post on the SharePoint Team Blog. For much more details, read the series on SharePoint Magazine. Note that these features requires an Enterprise license of SharePoint 2007.