I'm using asp.net and I need to build an application where we can easily create forms without recreating the database, and preferably without changing the create/read/update/delete queries. The goal is to allow customers to create their own forms with dropdowns, textboxes, checkboxes, even many-to-one relationship to another simple form (that's stretching it). The user does not have to be able to create the forms themselves, but I don't want to be adding tables, fields, queries, web page, etc. each time a new form is requested/modified.
2 questions:
1) How do I structure a flexible database to do this (in SQL Server)? I can think of two ways: a) Create a table for each datatype (int, varchar(x), smalldatetime, bit, etc). This would be very difficult to create the adequate queries. b) Create the form table with lots of extra fields and various datatypes in case the user needs 5 integers or 5 date fields. This seems the easiest, but is probably pretty inefficient use of space.
2) How do I build the forms? I thought about creating an xml sheet that had the validations, data type, control to display, etc. as a list. Then I would parse through the xml to build the form on the fly. Probably using css to do the layout (that would have to be manual, which is ok).
Is there a better/best way? Is there something out there that I could look at to get ideas? Any help is much appreciated.
This sounds like a potential candidate for an InfoPath solution. At first blush, it will do most/all of what you are asking.
This article gives a brief overview of creating an InfoPath form that is based on a SQL data source.
http://office.microsoft.com/en-us/infopath-help/design-a-form-template-based-on-a-microsoft-sql-server-database-HP010086639.aspx
I have built a completely custom solution like you are describing, and if I ever did it again I would probably opt for either 1) a third-party product or 2) less functionality. You can spend 90% of your time working on 10% of the feature set.
EDIT: reread your questions and here is additional feedback.
1 - Flexible data structure: A couple things to keep in mind are performance and the ability to write reports against the data. The more generic the data structure, the harder these will be to achieve (again, speaking from experience).
Somewhat contrary to both performance and report-readiness, Microsoft SharePoint uses XML fragments/documents in generic tables for maximum flexibility. I can't argue with the features of SharePoint, so this does get the job done and greatly simplifies the data structure. XML will perform well when done correctly, but most people will find it more difficult to write queries against XML. Even though XML is a "first class citizen" to SQL Server, it may or may not perform as well as an optimized table structure.
2 - Forms: I have implemented custom forms using XML transformed by XSLT. XML is often a good choice for storing form structure; XSLT is a monster unto itself, but it is very powerful. For what it's worth, InfoPath stores its form structure as XML.
I've also implemented dynamic forms using custom controls in .Net. This is a very object-oriented approach, but (depending on the complexity of the UI) can require a significant amount of code.
Lastly (again using SharePoint as an example), Microsoft implemented horrendously complicated XML list/form definitions in SharePoint 2007. The complexity defeats many of the benefits. In other words, if you go the XML route, make your structures clean and simple or you will have a maintenance nightmare on your hands.
EDIT #2: In reference to Scott's question below, here's a high-level data structure that will avoid duplicated data and doesn't rely on XML for the majority of the form definition.
Caveat: I just put this design together in SQL Management Studio...I only spent 10 minutes on it; developing a flexible form system is not a trivial task, so this is an over-simplification. It does not address the storage of user-entered data, just the definition of the form.
The tables:
Form - top-level form table which contains (as you would guess) the collection of fields that comprise the form.
Field - generic fields that could be reused across forms. For example, you don't want 50 different "Last Name" fields for 50 different forms. Note the "DataTypeId" column. You could store any type you wanted in this column, like "number, "free text", even a value that indicates the user should pick from a list.
FormField - allows a form to contain 0-many fields in its definition. You could even extend this table to indicate that the user can ADD as many of this field as they need.
Constraint - basically a lookup table that defines a constraint type (maybe it's max length, max occurrences, required, etc.)
FormFieldConstraint - relates a constraint to a particular instance of a form field. This table combines a specific form with a specific field with a specific constraint. Note the metadata column; this potentially would be a good use for XML to store the specifics of the constraint.
Essentially, I suggest building a normalized database with few or no null values and no duplicated data. A structure as I've described would get you on the path to that goal.
I think if you need truly dynamic forms saved into a database, you'd have to create a sort of "dictionary" data table.
For example...
UserForms
---------
FormID
FieldName
FieldValue
FormID relates back to the parent form (so you can aggregate all of the fields for one form. FieldName is the name of the text field entered from. FieldValue is the value entered/selected for that field.
Obviously this isn't a perfect solution. You could have issues typing your data dynamically, but I leave the implementation of that up to you.
Anyways, hopefully this gives you somewhere to start thinking about how you'd like to accomplish things. Good luck!
P.S. I've found using webforms with .NET to be a total pain when doing dynamic form generation. In the instances I had to do it, I ditched it almost entirely and used native HTML elements. Then rewired my form by using the necessary values from Request. Probably not a perfect solution either, but it worked the best for me.
We created a forms system like the one you're describing with a schema very similar to the one at the end of Tim's post. It has been pretty complicated, and we really had to wrestle with the standard WebForms controls like the DetailsView and GridView to make them be able to perform CRUD operations on groups of answers. They're used to binding straight to properties on an object, and we're forcing them to look up a field ID in a dictionary first. They don't like it. You may want to consider using MVC.
One tricky question is how to store the answers. We ended up using a table that's keyed on FieldId, InstanceId (for example, if 10 people are filling out your form, there are 10 instances), and RowNumber (because we support multi-row forms for things like employment history). Instead of doing it this way, I would recommend making your AnswerRow a first-class concept with its own table tied to an Instance, and having the answers be linked to the AnswerRow and Field.
In order to support different value types, we had to create multiple answer fields on our answer table (AnswerChar, AnswerDate, AnswerInt, AnswerDecimal). Each of these maps to one or more Field Types (Date, Time, Number, etc.). Each Field Type has its own logic to represent the value in the UI, and put the value into the appropriate property on our answer objects.
All in all, it's been a lot of work. It's worth it for us, since this is the crux of the product, but you will definitely want to keep the cost in mind before embarking on a project like this.
Related
We have a requirement to allow customising our core product and adding additional fields on a per client basis e.g. People entity some client wants to record their favourite colour etc. As far as I know we can't add properties to EF at runtime as it needs classes defined at startup. Each customer has their own database but we are deploying the same solution to all customers with all additional code. We are then detecting which customer they are and running customer specific services etc.
Now the last thing I want is to be forking my project or alternatively adding all fields for all clients. This would seem likely to become a nightmare. Also more often than not the extra fields would only be required in a very limited amount of place. Maybe some reports, couple of screens etc.
I found this article from Jermey Miller http://codebetter.com/jeremymiller/2010/02/16/our-extension-properties-story/ describing how they are adding extension properties and having them go from domain to the web front end.
Has anyone else implemented anything similar using EF? How did it work out? Are there any blogs/samples that anyone has seen? I am not sure if I am searching for the right thing even if someone could tell me the generic name for what we want to do that would help. I'm guessing it is a problem that comes up for other people.
Linked question still requires some forking or implementing all possible extensions in single solution because you are still creating strongly typed extensions upfront (= you know upfront what extensions customer wants). It is not generally extensible solution. If you want generic extensible solution you must leave strongly typed world and describe extensions as data.
You will need to use some metamodel. Your entity classes will contain only properties used by all customers and navigation property to special extension entity (additional table per every extensible entity) where you will be able to put additional properties as name / value pair (you can add other columns like type, validation, etc. if needed).
This will in general moves part of your model from hardcoded scenario to configuration based scenario and your customers will even be allowed to define extensions at runtime (if you implement such feature).
Most MVC tutorials I've been reading seem to create 4 View objects for each Model. For example, if my Model is "Foo", there seem to be 4 .cshtml files: Foo/Create, Foo/Delete, Foo/Details, and Foo/Edit. Using the VisualStudio "scaffolding" helper does this as well.
Is this really considered MVC best-practice? It just feels wrong to have 4 classes that are 80-90% identical to each other. When I add a new field to Foo, I need to edit all 4 .cshtml files. This sort of dual-maintenance (quad-maintenance?) just makes my OO skin crawl.
Please tell me: is there an expected/accepted best-practice which handles this differently? Or, if this really IS accepted best-practice, tell me why the quad-maintance shouldn't make me squirm.
I'm a reasonably skilled veteran of ASP.NET / c# / OO Design, but pretty new to MVC; so apologies if this is a noob question. Thanks in advance for your help!
Edit: thanks for all the replies! I marked the most thorough one as the answer, but upvoted all that were helpful.
You'll probably need between two and four different views:
List (for viewing many things)
View (for viewing a single thing. Might not be necessary, if it's OK to use Edit as View, or if List has room to show all properties)
Create
Edit (can be the same as create, if you code cleverly)
Thus, if your model doesn't have too many properties to show them all in a table, and if you're OK with not having a static, non-editable view for just examining, you can get well away with just List and Edit, and scrap the other two.
However, this doesn't solve your problem of double (or triple) maintenance if you update your model. There's other magic for that ;)
In ASP.NET MVC 3, there are extensions on HtmlHelper that let you do Html.DisplayForModel() and Html.EditorForModel(). These use predefined templates to nest themselves into your object and draw up display/edit fields for all public properites. If you pass DisplayForModel an IEnumerable<Foo>, it will create a table with column headers that are the property names of Foo (using the DisplayName attribute information if you supplied it) and where each row represent one Foo instance. If you give EditorForModel a Foo, it will create a <label> and an <input> for each public property on Foo.
All of the templates used by these powerful extension methods can be replaced by you, if you're not happy with the defaults. This can be done either on the level of Foo, in which case you'd be back in your double-maintenance scenario, or on lower levels (such as string or DateTime) to affect all editor/display fields generated with the templates.
For more information on exactly how this works, google "ASP.NET MVC 3 editor templates" and read a couple of tutorials. They'll explain the details much better than I could.
The views that ASP.NET MVC create for you don't necessarily need to be the views that you use in production. I found those just to be handy while developing quick prototypes or to test the database CRUD operations. Feel free to create whatever view(s) you would like to handle the operations.
I would generally just have 1 or 2 views to handle the basic operations and not use the built in views that are generated. For example, 1 view for adding, editing, or details and 1 view to show a list of objects.
It all depends on your application.
If you have a single item, you don't need a List view. If you can't edit it, then you don't need an edit view. Create and Edit can often be the same view, unless there are special things you need to do in one, but not the other.
In other words, use as many views as you need. There's no hard and fast rule here. The scaffolding is just there to help you on your way. Many kinds of apps will work just fine using the scaffolding, and won't require advanced HTML or Javascript.
Why would you want multiple views? Well, let's take the display and edit functions. You could create one view, in which you use if statements to determine the edit mode of the view, however this will complicated the view logic and views should be as simple as possible.
The reason to create seperate views is that its easier to maintain than one gigantic view with tons of conditional logic in it.
You can use exactly the same view when you are performing [HttpGet]. Given that you pass a proper ViewModel to this view, it will populate with appropriate data every time whether you are loading create, update, or delete Action.
The problem becomes apparent when you try to post that data to a specific Action.
Naturally View should have only one form, which will be used for posting data. When you declare this form, you specify which exactly Action to use for Post.
Having 3 different Submit buttons in that form will not make a difference since all of them will post the same form to the same Action.
You could do some javascript tweaking on OnClick event for these buttons to change Action to which data is posted, but this definitively would not be best practice.
Buttom line: having 4 different views for each of the CRUD actions is the best practice for MVC.
I tend to create the following for an object's CRUD ops:
index
_form (partial)
new
update
delete
view
As the same form is shared between new and update, there is very little difference between the two. It really depends on how much you want the variation to be, honestly.
As for delete, this is optional. I like to have a view in case javascript is disabled.
edit:
You mention view models and the guy above posted a long, convoluted (no offense) VM code sample.
Personally, I hate classes written to basically mirror domain objects and are only used to "move" data. I hate VMs. I hate DTOs. I hate everything that makes me have to write more code than is necessary.
I guess I've drank the coolaid of other frameworks (rails, sinatra, node.js) to the point where I can't stomach the idea of tossing DRY to the wind.
I personally say skip um.
Edit2 I forgot list..
I am currently working to rework the data system of our application. Basically, it is designed so that people can add all the custom fields they want, with only a few constant/always-there fields.
Our current design is giving us plenty of maintenance problems. What we do is dynamically(at runtime) add a column to the database for each field. We have to have a meta table and other cruft to maintain all of these dynamic columns.
Now we are looking at EAV, but it doesn't seem much better. Basically, we have many different types of fields, so there would be a StringValues, IntegerValues, etc table... which makes things that much worse.
I am wondering if using JSON or XML blobs in the database may be a better solution, specifically because in most use cases, when we retrieve anything out of these tables, we need the entire row. The problems is that we need to be able to create reports for this data as well.. No solution really makes custom queries look easy. And searching across such a blob database will surely be a performance nightmare when reports are ran.
Each "row" needs to have anywhere from about 15 to 100(possibly more) attributes/columns associated with it.
We are using SQL Server 2008 and our application interfacing with the database is a C# web application(so, ASP.Net).
what do you think? Use EAV or blobs or something else entirely? (Also, yes, I know a schema free database like MongoDB would be awesome here, but I can't convince my boss to use it)
What about the xml datatype? Advanced querying is possible against this type.
We've used the xml type with good success. We do most of our heavy lifting at the code level using linq to parse out values. Our schema is somewhat fixed, so that may not be an option for you.
One interesting feature of SQL server is the sql_variant type. It's fully supported in .NET and quite easy to use. The advantages is you don't need to create StringValue, IntValue, etc... columns, just one Value column that can contain all the simple types.
This very specific type favors the EAV option, IMHO.
It has some drawbacks though (sorting, distinct selects, etc...). So if you want to use it, make sure you read all the documentation and understand its limit.
Create a table with your known columns and "X" sparse columns using a sequential name such as DataColumn0001, DataColumn0002, etc. When there is a definition for a new column just rename a column and start inserting data. The great advantage to the sparse column is it is indexable.
More info at this link.
What you're doing is STUPID with a database that doesn't support your data type. You should work with a medium that meets your needs which include NoSQL databases such as RavenDB, MongoDB, DocumentDB, CouchBase or Postgres in RDMBS to name several.
You are inherently using the tool in a capacity it was neither designed for, and one it specifically attempts to limit you from achieving success. NoSQL database solutions frequently use JSON as an underlying storage because JSON is inherently schemaless. Want to add a property? Sure go ahead, want to add a whole sub collection? Sure go ahead. NoSQL databases were in part, created specifically to remove rigid schema requirements of RDBMS.
2015 Edit: Postgres now natively supports JSON. This is a viable option for RDBMS. My answer is still correct that you need to use the correct tool for the problem. It is a polygot persistence world.
Almost all the applications I worked on involve some look-up values. For example, a lot of times a list of languages ( English, French, etc...) need to be displayed on the WebForm for user to choose.
The common data structure for such look up values include an integer Id and a string as name. Since these look-up values are used so frequently, and they are unlikely to be changed. Most of time, instead of grabbing them from database, I just define a global enum in C# like this
enum Language : int { English = 1, French = 2}
I've been handling look-up values like this for years now. I knew it may not be the best way to handle them. For example, every time a new language is added to the system, somebody needs to remember to update that enum.
I guess this is a very common scenario, just wondering how you guys handle this type of look-up values.
Thanks,
I usually put them int he database anyway, because you never know if and when they'll need to change. There have been times when - by design - I know that the list will not change, and for those, I will use enums instead. All other cases, I will use a database table.
Depending on the data requirements, I typically handle small lookups like that in a configuration file often loaded singularly to be applied as an application variable for all instances of the app. The data is looked up once when the application is started or recycled and then remains useful until it is dead.
Although, I have a lot of intermediate data of similar nature that requires regular maintenance. It is easier to put this into a database because at least then you can build a small maintenance application to handle it. If you put them in an enum, you have to recompile and redeploy.
I ran into a similar issue with resource files. Since resource files are compiled, they require another redeployment.
You may benefit from T4 Templates. I haven't used these yet because we haven't had a requirement to meet the need, but I'd like to use them eventually.
Here's a great article for generating enums from a database table (the code is hard to read on the page, there is a link to the source at the end of the article):
http://www.thecodejunkie.com/2008/11/generate-enums-from-database-using-text.html
Also, Scott Hanselman has a list of great T4 templates/articles on his blog.
T4 Templates on MSDN
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
At my job, I have to implement web forms for loan applications with sometimes up to a hundred different input fields, and then save the application into the database for later retrieval.
The person whom I replaced created a sql table with 100s of columns where each row represents a loan application and there is a column for every field.
The problem with this is that I find myself having to type out the 100 fields a bunch of times, getting data from form, saving to database, retrieving from database, writing to output webform.
And then whenever there is a change to the application, I have to make the change in quite a few places.
So it can not only be cumbersome but error prone.
Is there a good design pattern that handles this?
See if you can come up with a better table design. 100 Cols is just too much IMO. If not you could use dynamic data. Also look for a better UI pattern or break it into more than 1 pages, looking at 100 fields to fill in could be over whelming for the users.
Code generation is the best bet. MyGeneration and CodeSmith are two very good tools.
I've had very similar projects and I used both generation tools and prefer CodeSmith.
100's of fields in a form is too many. Try to aim for max 20 per page.
If you can, try to isolate each part of this form into a separate page, and even better, into a wizard of some kind.
e.g. (example steps)
Step 1: Client Profile (name, age, SSN, etc.)
Step 2: Loan Data (Fixed, Variable, Amount, Terms, Special Deals...)
Step 3: Credit Check?, Pre-Approval?
Step 4: Variations/Calculations...
If you organize the data in structured groups, you should be able to make each portion work independently (e.g. you can input basic loan data, and get some preliminary numbers without needing to enter client data (yet).
More importantly, if the user enters a value in one of the 100 fields that causes the backend to not store (e.g. value out of range) what happens? does the user lose all the data they entered? - for usability make sure that nothing is lost when the user attempts to save/move to the next step.
Perhaps ASP.NET Dynamic Data would work for you. It is described as "a framework that enables you to quickly build a functional data-driven application"
I've worked with these loan applications before and I'll tell you straight out that your data model is broken. Are you gonna make a new row in the BigTable when a borrower comes in for a refi? What if the deal falls through but they want to try again for a different house? If you just edit the data in the row for that application then you're losing valuable metrics on your % of closed deals.
First, fix your data model. Then, embrace LINQ if at all possible.
With respect to the UI, I'd either do it as a wizard (which is pretty close to how these apps are filled out) or build it as one big form that the user scrolls through and adds stuff to (use AJAX in each section so that the user can save as they go along).
You also need to consider off the shelf solutions for this stuff. Don't reinvent the wheel here; these loan apps don't change often enough to throw a full time developer at them.
Edit: Sorry I was assuming that this was for residential mortgages. I don't know if that's true, but I still think your data model is broken. If you have the ability to change it you should.
Design pattern? But of course: The Adaptive Object-Model.
One thing that you can do is programmatically take the column names and put them on a file in the format you need. That will be to solve your immediate pain. Below is a piece of c# code that will get the column name from a oledbreader object.
internal List<string> GetFieldList(string sTableName)
{
tableName = sTableName;
BuildQuery("*");
command = SetupConnection();
List<string> FieldList = new List<string>();
OleDbDataReader reader = command.ExecuteReader();
for (int i = 0; i < reader.FieldCount; i++)
FieldList.Add(reader.GetName(i).ToString());
return FieldList;
}
To solve the situation in the long run, you probably want to do store procedures or functions with the fields as parameteres. you still have to put the values in each of the parameters, but is going to easy the management of each
he problem with this is that I find myself having to type out the 100 fields a bunch of times, getting data from form, saving to database, retrieving from database, writing to output webform.
hope it helps!!!
Ouch.
Is LINQ available to you? What version of .NET are we talking about here?
BTW, if it isn't obvious already, from a GUI perspective I suggest you hide most of these fields from the end user and reveal them in sections as the user progresses, which has the advantage that you can avoid showing what doesn't need to be and validate as you go.
And with some sort of progress indicator so they know there's an end in site (ha ha).
I've had issues like this before, and know there may be valid reasons for having so many columns. It's rare, but it can happen.
What you might want to do, is create a partial class in it's own file which contains all the data field code. This might be fields, properties, and possibly method parameters, and/or just your CRUD functionality. And generate that partial class, this will help ease your future maintanence.
Good Luck.