My dbml file is just getting bigger and bigger and more unwieldy:
I favoured an all-in-one approach as supposed to multiple data contexts because when I tried that it was near impossible to manage in code. I was advised it was better to have them all in one chart and the difficulty will be simply in managing this chart and not in code.
The chart I've got is becoming a pain to manage, if I want to even remove a table and re-add it it sometimes takes a little while to manually find it! There isn't even a list I can find in VS2010 of the objects you have in that chart!
Is there a better way of doing this?
Generally speaking, group tables related to the same concept in the same diagram, and create multiple diagrams, yes that means you have to MANAGE each diagram; but generally this is a GOOD thing. Here's why... Same database schema, different diagrams. Each diagram representing a specific subset of business. So product catalog section, an order section, a biling section, a returns section, a sales section.. etc.. Just make sure it groups up to a specific line of business. Yes this does mean that tables will be repeated on different diagrams
By segmenting the table structure into business logic groupings, you can quickly see all related tables to that groping. This is helpful to developers as they have to work in those specific sections; they understand the scope of work with out having to understand the entire database structure. When making a change, if you find a table is on multiple groupings/diagrams you can see what areas of the business are impacted by the change. This gives you an idea of the areas of the application which need to be tested and at a minimum considered when your making a change to the database structures. Ideally this type of modeling would be implemented in relation to services offered in a Service Management style of architecture. However starting to group your tables into business processes would help. IF you think this is unwieldy... try looking at an oracle db that has over 1500 tables in its schema.
The overall trick here is to only show those tables/views related to the business process/service someone would NEED to look at to support the system.
Good luck!
Related
After a watching a few videos regarding DynamoDB and its best practices, I decided to give it a try; however, I cannot help but feel what I'm doing may be an anti-pattern. As I understand it, the best practice is to leverage as few tables as possible while also taking advantage of GSIs to do some 'heavy' lifting. Unfortunately, I'm working with a use case that doesn't actually have strictly defined access patterns yet since we're still in early development.
Some early access patterns that we may see are:
Retrieve the number of wins for a particular game: rock paper scissors, boxing, etc. [1 quick lookup]
Retrieve the amount of coins a user has. [1 quick lookup]
Retrieve all the items that someone has purchased (don't care about date). [Not sure?]
Possibly retrieve all the attributes associated with a user (rps wins, box wins, coins, etc). [I genuinely don't know.]
Additionally, there may be 2 operations we will need to complete. For example, if the user wins a particular game they may receive "coins". Effectively, we'll need to add coins to the user "coins" attribute & update their number of wins for the game.
Do you think I should revisit this strategy? Additionally, we'll probably start creating 'logs' associated with various games and each individual play.
Designing a DynamoDB data model without fully understanding your applications access patterns is the anti-pattern.
Take the time to define your entities (Users, Games, Orders, etc), their relationship to one another and your applications key access patterns. This can be hard work when you are just getting started, but it's absolutely critical to do this when working with DynamoDB. How else can we (or you, or anybody) evaluate whether or not you're using DDB correctly?
When I first worked with DDB, I approached the process in a similar way you are describing. I was used to working with SQL databases, where I could define a few tables and rely on the magic of SQL to support my access patterns as my understanding of the application access patterns evolved. I quickly realized this was not going to work if I wanted to use DynamoDB!
Instead, I started from the front-end of my application. I sketched out the different pages in my app and nailed down the most important concepts in my application. Granted, I may not have covered all the access patterns in my application, but the exercise certainly nailed down the minimal access patterns I'd need to have a usable app.
If you need to rapidly prototype your application to get a better understanding of your acecss patterns, consider using the skills you and your team already have. If you already understand data modeling with SQL databses, go with that for now. You can always revisit DynamoDB once you have a better understanding of your access patterns and determine that your application can benefit from using a NoSQL databse.
For educational reasons I wish to build a functional, full, relational database. I'm aware LMDB was used to be the storage backend of sqlite, but I don't know C. I'm on .NET and I'm not interested in just duplicate a "traditional" RDBMS (so, for example, I not worry about implement a sql parser but my own custom scripting language that I'm building), but expose the full relational model.
Consider this question similar to "How I implement a programming language on top of LLVM" before worry about why I'm not using sqlite or similar.
From the material I read, LMDB look great, specially because It provide transactions and reliability, plus the low-level plumbing. How that translate to changes that could touch several rows at several tables is another question..
Exist material that explain how is implemented a relational layer on top of something like LMDB? Is using LMDB (or their competitors) optimal enough or exist another better way to get results?
Is possible to use LMDB to store other structures like hashtables, arrays and (the one I'm more interested for a columnar database) bitmap arrays?, ie, similar to redis?
P.D: Exist a forum or another place to talk more about this subject?
I had this idea too. You should realize that this is tons of work and most likely no one will care. I haven't built full-blown relational db as this is crazy to do for one person. You could check it out here
Anyway I've used leveldb (and later rocksdb) and so you have keys-values sorted by key, ability to get value by key, iterate keys, have atomic writes of many values (WriteBatch) and consistent view of data at given time - snapshots. These features are enough to build correct thread-safe reading of table rows (using snapshots), correct writing of data and related indexes - all or nothing (using writebatch) and even transactions.
Each column has it's on disk index - keys sorted by values - so you could efficiently do various operations on it and keys with values themselves so you could efficiently read values with given id.
This setup is efficient for writing and reading using available operations on tables with little data (say less than a million rows). However, if table grows iterating over many keys can become not so fast. To solve this and to add a group-by statement I've decided to add memory indexes, but that's another story. So all-in-all it might be fun idea but in reality a lot of work and often frustrating results - why would you want to do that?
My organization is just starting to dive into Dynamics CRM and one of the questions that has come up is when should we combine various applications into one instance and when should they be separated into multiple instances?
I know the answer to that question depends on the situation, so I'm trying to come up with a list of questions that can be asked to help determine which direction makes the most sense.
I'm having a surprisingly difficult time finding any discussion of this online, so thought I'd ask here. So, what questions do you ask when deciding whether a system/set of functionality should be in a separate instance?
Edit:
I wasn't very clear about our type of organization. I work for a City with several Departments that provide different services and serve different customers with often very different functionality required.
I'm concerned about the urge to put all of these different systems that have different functionality and track different "customers" into one system. I fear there will be issues about managing all of the various entities that apply to different systems and ensuring that requests for changes from one set of users doesn't cause problems for a different set of users.
I'm sure sometimes it will make sense to combine multiple systems into one instance, but I think there may be just as many times where we don't want to put them together, so I wanted to come up with a list of questions to ask.
Some basic ones would be:
1) Do systems share common data (e.g., same customers)?
2) Do systems share common functionality?
3) Do systems collect the same kind of data?
4) Are there requirements to report on combined data from these systems?
5) Will it be easier to manage security by separating instances or through user roles?
In my experience a single instance is the norm. The benefits of a single instance are very significant in my opinion.
A few points you may want to consider:
Do you want data in silos? If so, multi-instance provides a very easy way to achieve this. However a single instance with appropriate security modelling can also achieve this.
Do you want to combine data across applications into a single business process? If so, multi-instance means you have to build an integration between instances. Single instance does not have this problem.
Do you want to use custom built features in every instance? If so, a single instance provides this straight away. Multi-instance requires separate development and deployment to every instance which may increase costs.
Have you considered licensing? I'm not a licensing expert, but I believe if you are online multi-instance will attract a higher license cost.
As a rule of thumb I would say a single instance is the default position, as it allows you to easily combine data and processes. If you want to go multi-instance just have a good reason why and be sure its not something that can be provided by a single instance.
Here's the deal.
We have two departments with two completely different databases. Executives want a report to be printed out from both databases that are 100% identical. To accomplish this, the executives requested that I link my database to his and to create a single UNION query that combines the data so a single report can be generated off this data. This has proven quite difficult due to the difference in the systems, political considerations, maintenance considers and other factors.
The Executives are HIGHLY concerned with tiny formatting differences, which is why they won't let us create two isolated reports without the UNION query behind it.
Thus, my question is this: If I have a report in both databases, is there any simple way to make sure they do not drift in format? What if they were in the same database?
The only way I could think to do this would be to use an ETL tool to create an interim database with a 'union' of the data from the separate databases. Then from there run the report using the reporting tool of your choice.
See the following link for an example of an ETL tool with some videos explaining functionality.
http://www.cloveretl.com/
UPDATE: I had posted this on UI.stackexchange also for views on different kinds od visualisation. I am posting this here for finding out the programming techniques and tools required to do so.
Let us have the following three sets of information
Now I want to combine all of this data and show it all together. Telling it like a story. Giving inter-relations. Showing similarities in terms, concepts etc. to get the following (Note that in the diagram below, the colored relations may not be exact, they are merely indicative of a node of information)
Situation: I need to tell somebody the relation between two or more important things through the commonness of concepts, keywords, behaviours in those things.
One way that I figured out would be to use circles for concepts.
So that all concepts connected to thing A would be connected to it and all concept related to B would be connected to it. And the common concepts would be connected to both. That way 2 things can be easily compared.
Problem: To build such a graph/visualisation manually would be cumbersome. Especially to add, arrange, update and manipulate.
Question: Is there a good way to do it. Also, Is there a tool available for doing this?
I hope this make the question much more clear. :)
Where does this data (the concepts, keywords, and relations between them etc...) come from? If it's in a database somewhere you could write soem code to generate a graphiz file then open it in a graphiz visualizer. There might be some tools out there that allow interactive editing of a graphiz graph, it looks like WebDot may and there are probably others.
How to display the hierarchical data on User Interface
You're talking about Venn diagrams. I think there should be plenty of online and offline tools that can help making these.
graphiz has been mentioned already, although that would be used more to show a flow of a system, or a treeview.
When you're talking about software development and want to display a design through diagrams, a complete diagram solution already exist as UML. And there are plenty of UMT tools that can help here. A commercial version is Altova UModel, which has some very nice features. You could probably use Use Cases as the most logical diagram type.
Also see Wikipedia for more info about use case diagrams. Reconsidering the image you've added, I do tend to consider it to be a usecase. Since UML is based on XML, it should be possible to transform your data through a stylesheet to UML, then use a random UML tool to display the diagrams.To convert your data to XML, well... If it's in Excel then exporting it to XML should not be too difficult.
Why is your sample image an Use Case? Well, you have actors (Pinguin, Koala, Tulips) and you have actions. (well, kind of actions: Cause for concern, some kind of animal, linked to movie, bites your nose off...) And finally, there are associations between the actors and the actions connecting them all in some way. Thus Data--(export)->XML--(Styleheet)->UML--(UML tool)->Diagram.
D3: Data-Driven Documents JS library