This is a follow-up question to the answer here. In App Maker, if one has Model called Contacts, with a relational field called Owner (which refers to another Contact record within the Contact Model), how many levels deep can be accessed by a widget?
Could you have #datasource.item.Contact.Owner.Owner.Owner.Owner etc, to be able to show the contact and its owners many levels up?
It looks like there is no any limitation from browser side. I tried to create heavily nested object in Chrome and it allowed me to add 300K+ nesting levels and ate 5Gb+ of memory for that single object, before UI got frozen and tab died.
Taking into account that App Maker's objects are way more complex than my test one { x: { x: { x: { ... } } } }, we can be super conservative and lets say that browser can survive 1K+ nesting levels. So, you see there should not be limitation on browser side in case you'll load relations in async manner (explicitly by item._loadRelationName() or implicitly using bindings).
In case you want to load all those nested records at once using prefetches then we need to find out how many joins MySQL allows to make. According to this discussion it is somewhere about 60. Drive Tables is a separate story, maybe if I have some time I'll dig into it and try to find its limitations.
Related
This is a typical situation for 1 to many relationships: a chat group iOS app, a group table to record all the group chat related information, like group id, create time, thread title, etc.
To record the participants, of course, I would assume there is another 1:m table. So I was rather surprised to see the app just added another column called "participants" to record it, with each participant is separated by a delimiter (':' to be exact). The problem with that is quite obvious, mixing application code with sql code, e.g. no way to see how many groups a specific user is in with sql code, violated 1NF/2NF, etc.
But they said we understood all your points. But
as this is a mobile app, you always need to use objective c code to access sqlite tables, you won't use sql codes alone. So not a "big deal" to mix them together.
participants don't change often and normally are set when a group is created. If we have 100 participants we would rather just insert 1 record to group table instead of insert 100 records into another group-participants table.
The participant data will be used when someone wants to see who are in this chat group (by several taps on the menu) and when someone joins or leaves the chat group, assume it won't happen often.
So my question is in this particular situation what is the advantage I will gain if I use another 1:m table?
----- update -----
Except for the answer I got, Renzo kindly pointed this discussion to me, which is also very helpful!
It's hard to respond to "is this design better/worse" style questions without understanding the full context. I'm going to make some assumptions based on your question.
You appear to be building a mobile application, supporting "many to many" user chat. I'm picturing something like Slack.
Your application design is using the SQLite database for local storage.
Your local sqlite database on the phone is some kind of subset of the overall application data - like a cache, only showing the data for the current user.
If all that is true, the question really is down to style/maintainability on the one hand, and performance and scalability on the other.
From a "style" point of view, storing the data in a comma-separated value in a column is ugly. A new developer who joins the project, with a background in "regular" database design will consider it at best a hack. On the other hand, iOS developers may consider it perfectly normal.
From a performance point of view, it's probably not worth arguing about - parsing the CSV is probably just as slow as reading/writing from the database.
From a scalability point of view, you may have a problem. If the application design needs to capture in which order users joined the chat, or capture some kind of status (active/asleep, for instance), or provide a bit of history (user x exited at 21:20), you almost certainly end up re-designing the database.
I'm wondering if this ,the strategy I will explain, would be recommended to use in Firebase.
I will first explain what my goal is, since I'm sure tons of others have solved the same problem already, and maybe some of you can tell me how it's usually done.
The goal is to notify all users of an App when the friend in common "George" (based on their contacts) is now also a proud new user of the App.
So, my idea was to do so:
1- Build a collection with this structure:
{
"contacts":
{
"user1":
{
{"user239":true}
,
{"user23":false}
,
{"user732":true}
}
,
{
"user2" :
{
{"user23":false}
,
{"user96":false}
,
{"user88":true}
}
}
}
}
To save for each user a list of contacts.
Then the new user would query a list of contacts like this:
fbRef.child('contacts').orderByChild('user23').equalTo(false).once('value', showResults, console.error);
Then the user would save the results in a map, change the value to true, and then updateChildren() using that map.
Now, is this reasonable if we imagine that we aspire to have hundreds of thousands and even millions of users using the App?
How expensive would this be when we have 5M users and a few joining by the second?
Is there a known "best strategy" for this case?
Thanks
The real-time functions in Firebase are not only suited for, but designed for large data sets. The fact that records stream in real-time is perfect for this.
Performance is, as with any large data app, only as good as your implementation. So here are a few gotchas to keep in mind with large data sets.
So what you can do is Denormalise the data.
**/users/uid
/users/uid/profile
/users/uid/chat_messages
/users/uid/groups
/users/uid/audit_record**
**/user_profiles/uid
/user_chat_messages/uid
/user_groups/uid
/user_audit_records/uid**
Second approach is good for iterating large data sets that the first approach which is clearly visible.
Avoid calling the value on large data sets.Call it by the child_used
This helps to denormalize the data above.
Remember firebase can handle large amount of data but it depends upon the approach you follow.
For Example: if we want to store the 'last_logins' for any user we can directly store it under the specific object instance. It will provide ease of access when we want to access 'last_logins' for a particular user.
Maintaining Many to Many Relations
We have already seen that we cannot nest Users in Groups as it will not represent many to many relation and will leave redundant data. We can create an index of groups under a specific containing only the keys of the groups which a user belongs to. This will enable us to easily fetch list of groups to which a user belongs to.
Storing One to Many Relations or Static Lists
Here are the links to some of the best practices that must be used for a firebase design.
Performance
Indexing
How good/stupid would it be to use Firebase tree structure to directly represent a user-facing tree structure, like a "document outline" in "word processors"?
As opposed to e.g. doing an SQL-join parent-child type of relationship and then building the tree via a projection (which would probably be slow).
I know that there is a limit of 32 levels of nesting ( https://www.firebase.com/docs/web/guide/understanding-data.html ), which should be enough, as I cannot imagine a sane user wanting to do as many levels of nesting for a textual tree-outline...
Although maybe I need to divide 32 by two, because of each node needing to have sub nodes for its children and metadata, right?
I know that once a tree node is accessed via Firebase API, then all sub-nodes need to be fetched, which could be a performance problem if the user has a lot of data, but in the end I think this would not be a problem, since the data would mostly be a user-entered plaintext (short).
A performance problem could arise if the user pastes some very long chunks of text copied from somewhere (e.g. tens of kilobytes). But then I could separate those "TLOB-s" via a kind of "symlink" in firebase and fetch them on-demand from a different node, right? Same should apply for separating images and other heavy objects, right?
Although in a prototype and early stages, this should probably be ignored, for the sake of simplicity...
I could probably put in place a generic approach to "symlinking", to overcome the 32 levels limitation and the need to fetch all sub-nodes at once, right? Is there some best-practices approach for that (e.g. syntax for a firebase node which would symbolise a link to another node) ?
I have extracted the "symlinking" idea to a separate question: Firebase "symlink" to another node .
I could probably partition the topmost nodes into some kinds of projects/categories to prevent having to fetch absolutely everything the user has ever had...
Is my reasoning/approach correct?
Is there any consideration that I did not think of, e.g. innate limits on data size or performance or e.g. security rules?
Would I be better served by other technologies like Couchbase/Pouchbase ?
Further details: this is for a hybrid mobile app with some emphasis also on web access and offline access. I hope to do most of the logic in Javascript. The UI part of the question is here: HTML tree for hybrid mobile app .
I'm really new to firebase, want to try out a simple mix-client app on it - android, js. I have a users table and a tasks table. The very first question that comes to my mind is, how to store them (and thus how the url to be)? For example, based on the tasks table, should I use:
/tasks/{userid}/task1, /tasks/{userid}/task2, ...
Or
/{userid}/tasks/task1, /{userid}/tasks/task2, ...
The next question, based on the answer to the first one - why to use any of the versions?
In my opinion, the first version is good because domains are separated.
The second approach is good because data is stored per-user which may make some of the operations easier.
Any ideas/suggestions?
Update: For the current case, let's say there are following features:
show list of tasks for each user
add new task to the list
edit/delete a task by user.
Simple operations.
This answer might come in late, but here's how I feel about the question after a year's experience with Firebase.
For your very first question, it totally depends on which data your application will mostly read and how and in which order ( kind of like sorting ) you expect to read the data.
your first proposal of data structure, that is "/tasks/{userid}/task1", "taks/{userid}/task2"... is good if the application will oftentimes read the tasks as per users with an added advantage of possibly sorting the data by any task's "attribute" if I might call it so.
say each task has got a priority attribute then,
// get all of a user's tasks with a priority of 25.
var userTasksRef = firebase.database().ref("tasks/${auth.uid}");
userTasksRef.orderByChild("priority").equalTo(25).on(
"desired_event",
(snapshot) => {
//do something important here.
});
2. I'll highly advice against the second approach because generally most if not all of the data that is associated to that user will be stored under the "/{userid}/" node and with firebase's mechanism, should a situation be in which you need more than one datum at that path level, it will require you getting that data with all the other data that's associated to that user's node ( tasks and any other data included). I won't want that behavior on my database. Nonetheless, this approach still permits you to store the tasks as per the users or making multiple RESTfull requesting and collecting the required data datum after datum. Suggest fanning out the data structure if this situation is encountered. Totally valid data structure if there don't exist a use case in the application where in datum at the first level of the path is needed and only that datum is needed but rather the block of data available at that path level with all the data at the deriving paths at that level( that is 2nd 3rd ... levels).
As per the use cases you've described, and if the database structure you've given is exhaustive of your database structure, I'll say it isn't enough to cover your use cases.
Suggest reading the docs here. Great and exhaustive documentation of their's.
As a pick, the first approach is a better approach to modelling this data use case in NoSQL and more accurately Firebase's NoSQL database.
I have several graphs. The breadth and depth of each graph can vary and will undergo changes and alterations during runtime. See example graph.
There is a root node to get a hold on the whole graph (i.e. tree). A node can have several children and each child serves a special purpose. Furthermore a node can access all its direct children in order to retrieve certain informations. On the other hand a child node may not be aware of its own parent node, nor other siblings. Nothing spectacular so far.
Storing each graph and updating it with an object database (in this case DB4O) looks pretty straightforward. I could have used a relational database to accomplish data persistence (including database triggers, etc.) but I wanted to realize it with an object database instead.
There is one peculiar thing with my graphs. See another example graph.
To properly perform calculations some nodes require informations from other nodes. These other nodes may be siblings, children/grandchildren or related in some other kind. In this case a specific node knows the other relevant nodes as well (and thus can get the required informations directly from them). For the sake of simplicity the first image didn't show all potential connections.
If one node has a change of state (e.g. triggered by an internal timer or triggered by some other node) it will inform other nodes (interested obsevers, see also observer pattern) about the change. Each informed node will then take appropriate actions to update its own state (and in turn inform other observers as needed). A root node will not know about every change that occurs, since only the involved nodes will know that something has changed. If such a chain of events is triggered by the root node then of course it's not much of an issue.
The aim is to assure data persistence with an object database. Data in memory should be in sync with data stored within the database. What adds to the complexity is the fact that the graphs don't consist of simple (and stupid) data nodes, but that lots of functionality is integrated in each node (i.e. events that trigger state changes throughout a graph).
I have several rough ideas on how to cope with the presented issue (e.g. (1) stronger separation of data and functionality or (2) stronger integration of the database or (3) set an arbitrary time interval to update data and accept that data may be out of synch for a period of time). I'm looking for some more input and options concerning such a key issue (which will definitely leave significant footprints on a concrete implementation).
(edited)
There is another aspect I forgot to mention. A graph should not reside all the time in memory. Graphs that are not needed will be only present in the database and thus put in a state of suspension. This is another issue which needs consideration. While in suspension the update mechanisms will probably be put to sleep as well and this is not intended.
In the case of db4o check out "transparent activation" to automatically load objects on demand as you traverse the graph (this way the graph doesn't have to be all in memory) and check out "transparent persistence" to allow each node to persist itself after a state change.
http://www.gamlor.info/wordpress/2009/12/db4o-transparent-persistence/
Moreover you can use db4o "callbacks" to trigger custom behavior during db4o operations.
HTH
German
What's the exact question? Here a few comments:
As #German already mentioned: For complex object graphs you probably want to use transparent persistence.
Also as #German mentione: Callback can help you to do additional stuff when objects are read/written etc on the database.
To the Observer-Pattern. Are you on .NET or Java? Usually you don't want to store the observers in the database, since the observers are usually some parts of your business-logic, GUI etc. On .NET events are automatically not stored. On Java make sure that you mark the field holding the observer-references as transient.
In case you actually want to store observers, for example because they are just other elements in your object-graph. On .NET, you cannot store delegates / closures. So you need to introduce a interface for calling the observer. On Java: Often we use anonymous inner classes as listener: While db4o can store those, I would NOT recommend that. Because a anonymous inner class gets generated name which can change. Then db4o will not find that class later if you've changed your code.
Thats it. Ask more detailed questions if you want to know more.