So I just recently learn about denormalization and I am just trying to figure out the best way to structure my firebase database. What I currently have is I am displaying a users class schedule with the classes course number and intructor name. Firebase looks like this for me:
-root
-users
-uid1
-uid2
-schedules
-uid1 : true (match with users uid)
-uid2 : true (match with users uid)
-class
-classKey1
-users
-uid1 : true
-uid2 : true
-courseKey1 : true
-intructorKey1 : true
-classKey1
-users
-uid1
-courseKey2 : true
-intructorKey2 : true
-courses
-courseKey1
-intructors
-intructorKey1 : true
-intructorKey2 : true
-courseKey2
-similar to above
-intructors
-intructorKey1
-courses
-courseKey1: true
-courseKey2: true
-intructorKey2
-similar to above
Now, that is the basic structure of what I am working with, excluding all the unneeded information. Lets say I want to display all of the the schedule of the currently logged in user I will need to do the following.
_af.auth.subscribe(authState => {
let userID = authState.uid;
_af.database.list('/schedule/' + userID ).subscribe(schedule =>{
this.schedule = schedule; //global variable
this.filterClasses(); call function
});
});
_af.database.list('/class').subscribe(classes => {
this.masterClasses = classes; //gloabal variable
this.filterClasses();
});
Now because its all done asynchronously the only way I could think to do this is call the filterClasses function inside of each subscription.
filterClasses(): void {
if (this.scheduleKeys != null && this.masterClasses.length > 0) {
this.outputClasses = this.masterClasses.filter(value => {
let bool = false;
for (let i = 0; i < this.scheduleKeys.length; i++) {
if (this.scheduleKeys[i].$key === value.$key) {
bool = true;
break;
}
}
return bool;
});
this.outputClasses.forEach((value, i) => {
_af.database.object('courses/' + value.courseKey).subscribe(v => {
this.outputClasses[i]['courseNum'] = v.course;
})
_af.database.object('intructors/' + value.intructorKey).subscribe(v => {
this.outputClasses[i]['intructorName'] = v.name;
})
})
}
}
As you can see when I am done filtering my master list of classes into just the ones in my schedule I also now need to go and grab the course number firebase and the intructors name. Both of which require me to call firebase again. So to try and reduce researching firebase which seems to be causing some problems due to being async should I instead be denormalizing my data? Should in the class root of firebase instead of storing just courseKey1 should I store all the data associated to couseKey1? However, this will cause rooting in firebase because if I structure this way now in my courses root when I say the intructors key instead of just saving the key I would save everything which is just another level deeper firebase goes.
This is kind of an indirect answer:
Denormalizing data is a good thing but in same cases it may not be necessary.
For example in this case a student wants to know his courses, so you know the direct path to the student; users/uid_0 and then maybe a query for the semester you are interested in.
Here's a structure that goes with your courses and instructors
users
uid_0
schedules
-Yiaiajsdaoiojasd
semester: 1
courses
course1: true
course3: true
-Jyuhus99jsijskkss
semester: 2
courses
course2: true
course4: true
uid_1
schedules
-Yjs909k0s0s0ks0s
semester: 1
courses
course1: true
course3: true
-Jyuhus99jsijskkss
semester: 2
courses
course2: true
course4: true
This really isn't very deep since once you read in the user node, you have specific paths to the other data you need: /courses/courseKey1/instructors for example, then followed by instructors/instructor1/name.
Where you get into trouble is if you DON'T know the direct path to the data and have to pile up queries on queries - that's where denormalizing is most effective. Also, queries have lot of overhead vs just observing a node at a path you know.
Related
I have a firebase database like this structure:
-groups
--{group1id}
---groupname: 'group1'
---grouptype: 'sometype'
---groupmembers
----{uid1}:true
----{uid2}:true
--{group2id}
---groupname: 'group2'
---grouptype: 'someothertype'
---groupmembers
----{uid1}:true
----{uid3}:true
----{uid4}:true
Now, I am trying to pull groups of authenticated user. For example for uid1, it should return me group1id and group2id, and for example uid3 it should just return group2id.
I tried to do that with this code:
database().ref('groups/').orderByChild('groupMembers/' + auth().currentUser.uid).equalTo('true').on('value' , function(snapshot) {
console.log('GROUPS SNAPSHOT >> ' + JSON.stringify(snapshot))
})
but this returns null. if I remove "equalTo" and go it returns all childs under 'groups'.
Do you know any solution or better database structure suggestion for this situation ?
Your current structure makes it easy to retrieve the users for a group. It does not however make it easy to retrieve the groups for a user.
To also allow easy reading of the groups for a user, you'll want to add an additional data structure:
userGroups: {
uid1: {
group1id: true,
group2id: true
},
uid2: {
group1id: true,
group2id: true
},
uid3: {
group2id: true
},
uid3: {
group2id: true
}
}
Now of course you'll need to update both /userGroups and /groups when you add a user to (or remove them from) a group. This is quite common when modeling data in NoSQL databases: you may have to modify your data structure for the use-cases that your app supports.
Also see:
Firebase query if child of child contains a value
NoSQL data modeling
Many to Many relationship in Firebase
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
I am bit familiar with NoSQL and Firebase Realtime Database and also I know that it is not best solution to solve tasks where relational database should be more appropriate. I want to verify about structure of simple many to many relationship that I have.
I have events and users. I want to use Firebase for storing information about users participating in events, later I will need to
Get list of users for event knowing it's id and city
Get list of events for users knowing it's id and city
add or delete information about user attending to event
I would like to have first tree of events ids divided by cities.
events {
'city1' : {
event_id_1 : {'user_1', 'user_2'},
event_id_2 : {'user_3', 'user_4'}.
}
'city2' : {
event_id_3 : {'user_5', 'user_6'},
event_id_4 : {'user_7', 'user_7'}.
}
}
And second tree for users
users {
'user1' : {
'city1' : {event_id_1, event_id_2},
'city2' : {event_id_3, event_id_4},
'city3' : {event_id_3, event_id_4}
},
'user2' : {
'city1' : {event_id_1, event_id_2},
'city2' : {event_id_3, event_id_4},
'city3' : {event_id_3, event_id_4}
},
'user3' : {
'city1' : {event_id_1, event_id_2},
'city2' : {event_id_3, event_id_4},
'city3' : {event_id_3, event_id_4}
},
}
Would it be easy and fast to use and maintain?
Your structure looks pretty OK to me given the requirements listed. Most importantly: you store the data in both directions already, which is the biggest hurdle for many developers new to NoSQL data modeling.
A few notes about your data model, though most are on the level of typos:
Be sure to store the data as maps, not arrays. So event_id_1 : {'user_1': true, 'user_2': true }
If there is a many-to-many relationship between users and events, I'd usually have four top-level lists: users and events (for the primary information about each), and then userEvents and eventUsers (for connections between the two).
Adding a user to an event can be done with a single multi-location update, e.g.:
ref.update({
'/userEvents/userId1/eventId1': true,
'/eventUsers/eventId1/userId1': true
});
Unregistering them is a matter of doing the same with null as the value (which deletes the existing key):
ref.update({
'/userEvents/userId1/eventId1': null,
'/eventUsers/eventId1/userId1': null
});
Also see my answer here: Many to Many relationship in Firebase
You can do this:
List of user
Users
useruid
name:userx
email:userx#gmail.com
useruid
name:usery
email:usery#gmail.com
Events
eventid
useruid
name:userx
location: city1
eventname: party
eventid2
useruid1
name:usery
location: city2
eventname: Boring Party
Get list of users for event knowing it's id and city:
DatabaseReference ref=FirebaseDatabase.getInstance().getReference().child("Events").child(eventid);
ref.orderByChild("location").equalTo(city1);
//retrieve users using a listener
Get list of events for users knowing it's id and city:
DatabaseReference ref=FirebaseDatabase.getInstance().getReference().child("Events");
Query q=ref.orderByChild("location").equalTo(city1);
using a listener this can give you the events that are in location:city1
You can get the child count via
firebase_node.once('value', function(snapshot) { alert('Count: ' + snapshot.numChildren()); });
But I believe this fetches the entire sub-tree of that node from the server. For huge lists, that seems RAM and latency intensive. Is there a way of getting the count (and/or a list of child names) without fetching the whole thing?
The code snippet you gave does indeed load the entire set of data and then counts it client-side, which can be very slow for large amounts of data.
Firebase doesn't currently have a way to count children without loading data, but we do plan to add it.
For now, one solution would be to maintain a counter of the number of children and update it every time you add a new child. You could use a transaction to count items, like in this code tracking upvodes:
var upvotesRef = new Firebase('https://docs-examples.firebaseio.com/android/saving-data/fireblog/posts/-JRHTHaIs-jNPLXOQivY/upvotes');
upvotesRef.transaction(function (current_value) {
return (current_value || 0) + 1;
});
For more info, see https://www.firebase.com/docs/transactions.html
UPDATE:
Firebase recently released Cloud Functions. With Cloud Functions, you don't need to create your own Server. You can simply write JavaScript functions and upload it to Firebase. Firebase will be responsible for triggering functions whenever an event occurs.
If you want to count upvotes for example, you should create a structure similar to this one:
{
"posts" : {
"-JRHTHaIs-jNPLXOQivY" : {
"upvotes_count":5,
"upvotes" : {
"userX" : true,
"userY" : true,
"userZ" : true,
...
}
}
}
}
And then write a javascript function to increase the upvotes_count when there is a new write to the upvotes node.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.countlikes = functions.database.ref('/posts/$postid/upvotes').onWrite(event => {
return event.data.ref.parent.child('upvotes_count').set(event.data.numChildren());
});
You can read the Documentation to know how to Get Started with Cloud Functions.
Also, another example of counting posts is here:
https://github.com/firebase/functions-samples/blob/master/child-count/functions/index.js
Update January 2018
The firebase docs have changed so instead of event we now have change and context.
The given example throws an error complaining that event.data is undefined. This pattern seems to work better:
exports.countPrescriptions = functions.database.ref(`/prescriptions`).onWrite((change, context) => {
const data = change.after.val();
const count = Object.keys(data).length;
return change.after.ref.child('_count').set(count);
});
```
This is a little late in the game as several others have already answered nicely, but I'll share how I might implement it.
This hinges on the fact that the Firebase REST API offers a shallow=true parameter.
Assume you have a post object and each one can have a number of comments:
{
"posts": {
"$postKey": {
"comments": {
...
}
}
}
}
You obviously don't want to fetch all of the comments, just the number of comments.
Assuming you have the key for a post, you can send a GET request to
https://yourapp.firebaseio.com/posts/[the post key]/comments?shallow=true.
This will return an object of key-value pairs, where each key is the key of a comment and its value is true:
{
"comment1key": true,
"comment2key": true,
...,
"comment9999key": true
}
The size of this response is much smaller than requesting the equivalent data, and now you can calculate the number of keys in the response to find your value (e.g. commentCount = Object.keys(result).length).
This may not completely solve your problem, as you are still calculating the number of keys returned, and you can't necessarily subscribe to the value as it changes, but it does greatly reduce the size of the returned data without requiring any changes to your schema.
Save the count as you go - and use validation to enforce it. I hacked this together - for keeping a count of unique votes and counts which keeps coming up!. But this time I have tested my suggestion! (notwithstanding cut/paste errors!).
The 'trick' here is to use the node priority to as the vote count...
The data is:
vote/$issueBeingVotedOn/user/$uniqueIdOfVoter = thisVotesCount, priority=thisVotesCount
vote/$issueBeingVotedOn/count = 'user/'+$idOfLastVoter, priority=CountofLastVote
,"vote": {
".read" : true
,".write" : true
,"$issue" : {
"user" : {
"$user" : {
".validate" : "!data.exists() &&
newData.val()==data.parent().parent().child('count').getPriority()+1 &&
newData.val()==newData.GetPriority()"
user can only vote once && count must be one higher than current count && data value must be same as priority.
}
}
,"count" : {
".validate" : "data.parent().child(newData.val()).val()==newData.getPriority() &&
newData.getPriority()==data.getPriority()+1 "
}
count (last voter really) - vote must exist and its count equal newcount, && newcount (priority) can only go up by one.
}
}
Test script to add 10 votes by different users (for this example, id's faked, should user auth.uid in production). Count down by (i--) 10 to see validation fail.
<script src='https://cdn.firebase.com/v0/firebase.js'></script>
<script>
window.fb = new Firebase('https:...vote/iss1/');
window.fb.child('count').once('value', function (dss) {
votes = dss.getPriority();
for (var i=1;i<10;i++) vote(dss,i+votes);
} );
function vote(dss,count)
{
var user='user/zz' + count; // replace with auth.id or whatever
window.fb.child(user).setWithPriority(count,count);
window.fb.child('count').setWithPriority(user,count);
}
</script>
The 'risk' here is that a vote is cast, but the count not updated (haking or script failure). This is why the votes have a unique 'priority' - the script should really start by ensuring that there is no vote with priority higher than the current count, if there is it should complete that transaction before doing its own - get your clients to clean up for you :)
The count needs to be initialised with a priority before you start - forge doesn't let you do this, so a stub script is needed (before the validation is active!).
write a cloud function to and update the node count.
// below function to get the given node count.
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.userscount = functions.database.ref('/users/')
.onWrite(event => {
console.log('users number : ', event.data.numChildren());
return event.data.ref.parent.child('count/users').set(event.data.numChildren());
});
Refer :https://firebase.google.com/docs/functions/database-events
root--|
|-users ( this node contains all users list)
|
|-count
|-userscount :
(this node added dynamically by cloud function with the user count)
I have a snapshot for my reference in firebase like this:
"friendlist" : {
"user1" : {
"user3" : 1
},
"user2" : {
"user1" : 0
}
"user3" : {
"user1" : 1
}
}
The explanation for the reference:
Every user has an unique id, i'm using user's id for their friendlist unique id. In example above i have 3 users and every user have his own friendlist. Inside their friendlist, there's other user's id that already be friend with him. If the value is 1, the user already be friend. But when the value is 0, the user is requesting to be friend.
My problem is:
How to get all user's friendlist's id which have "user1" with value 0 inside their friendlist? Can i do that in just one query?
I think i need to iterate through all friendlist and orderbykey for every friendlist and looking for "user1". Or there's any good approach to do that?
Any answer would be appreciated, thanks!
It would help if you next time tell a bit more about what you've already tried. Or at the very least specify what language/environment you're targeting.
But in JavaScript, you can get those users with:
var ref = new Firebase('https://yours.firebaseio.com/friendlist');
var query = ref.orderByChild('user1').equalTo(0);
query.once('value', function(usersSnapshot) {
usersSnapshot.forEach(function(userSnapshot) {
console.log(userSnapshot.key());
});
});
With the sample data you specified, this will print:
user2
You should add (and will get a warning about) an index for efficiently performing this query:
{
"rules": {
"friendlist": {
".indexOn": ['user1']
}
}
}
Without this index, the Firebase client will just download all data to the client and do the filtering client-side. With the index, the query will be performed server-side.
A better data model
You'll likely want to search for any friend, which turns the index into:
".indexOn": ['user1', 'user2', 'user3']
But with this structure, you'll need to add an index whenever you add a user. Firebase SDKs don't have an API to add indexes, which is typically a good indication that your data structure is not fitting your needs.
When using a NoSQL database, your data structure should meet the needs of the application you're building. Since you are looking to query the friends of user1, you should store the data in that format too:
"friendlist" : {
"user1" : {
"user3" : 1
},
"user2" : {
"user1" : 0
}
"user3" : {
"user1" : 1
}
},
"friendsOf": {
"user1": {
"user2": 0,
"user3": 1
},
"user3": {
"user1": 1
}
}
As you can see, we now store two lists:
* friendList is your original list
* friendsOf is the inverse of your original list
When you need to know who friended user 1, you can now read that data with:
ref.child('friendsOf').child('user1').on('value'...
Note that we no longer need a query for this, which makes the operation a lot more scalable on the database side.
Atomic updates
With this new data model, you need to write data in two places when adding a friend relation. You can do this with two set()/update() operations. But in recent Firebase SDKs, you can also perform both writes in a single update like this:
function setRelationship(user1, user2, value) {
var updates = {};
updates['friendList/'+user1+'/'+user2] = value;
updates['friendsOf/'+user2+'/'+user1] = value;
ref.update(updates);
}
setRelationship('user3', 'user4', 1);
The above will send a single command to the Firebase server to write the relationship to both friendList and friendsOf nodes.