I have a complex application and I am using Firebase for my backend data.
I have an object that is used in two different contexts, one for viewing by the user and one for the server to do batch processing. Something like this:
users: {
USER_ID: {
...
txns: {
TXN_ID: {
// data I need
}
}
}
...
},
transactions: {
TXN_ID: {
USER_ID: {
// data I need
},
USER_ID2: {
// different data I need
}
}
}
In relational terms, txn_ids and user_ids are in a many-to-many relationship. So looking through the transactions node to find all the transactions that belong to a user is hard in FB and looking though all the users to find a transaction id is hard in FB. So I denormalize the data. I think this is correct.
But how do I keep them in sync?
I've been using MPUs, which is fine, but I'm worried about it being prone to bugs in the future.
I've considered writing a Firebase Cloud Function to keep them in sync, but I'm worried about that being more fragile than MPUs.
Advice?
Related
I'm wondering if this is possible, and if it's a good solution to my problem.
I want users to be able to subscribe to content. The content is associated with an id.. for instance:
'JavaScript': 1,
'C++': 2,
'Python': 3,
'Java': 4,
Let's say a user subscribes to 1, 3, and 4.
So their user json data would appear as:
'subscribed_to': [1,3,4]
Now in my firestore, I have posts. Each post gets assigned a content_id (1-4 for instance), and so when I query for the content that this user is subscribed to, how would I do that so as effectively as possible?
This is indeed a complex but common case, I would recommend to set a data structure similar to:
{
"subscriptions" {
javascript: { ... },
python: { ... },
java: { ... }
},
"users": {
1: {
subscribed_to: ['javascript', 'python']
}
}
}
It's very important that on your subscribed_to prop you use the doc name, cause this is the part that allows you to query them (the docs).
the big problem, how do I query this data? I don't have joins!
Case 1:
Assuming you have your user data when you apply load...
const fetchDocs = async (collection, docs) => {
const fetchs = docs.map(id => collection.doc(id).get())
const responses = await Promise.all(fetchs)
return responses.map(r => r.data())
}
const userSubscriptions = ['javascript', 'python']
const mySubscriptions = await fetchDocs(`subscriptions`, userSubscriptions)
Behind the scene, the sdk will group all the queries and do its best efforts to deliver em together. This works good, I'm 99% sure you still have to pay for each query individually each time the user logs in.
Case 2:
Create a view dashboard collection and pre-calculate the dashboard behind the scene, this approach involves cloud functions to listen or changes on the user changes (and maybe subscriptions as well) and copy each individual doc into another collection, let's say subscriptions_per_users. This is a more complex approach, I will require more time to explain but if you have a big application where costs are important and if the user is going to subscribe to a lot of things, then you might want to investigate about it.
Source: My own experience... a little of google can also help, there seems to be a lot of opinions about it, find what works best for you.
Wondering if I have the right mindset here... In firebase (with Vue) I have a userscollection:
users : {
uid: "09A09IQMSLDK0912",
name: "Gerard",
email: "gerard#mail.com"
}
If I want the user to add friends, should I add it to the userscollection?
users : {
uid: "09A09IQMSLDK0912",
name: "Gerard",
email: "gerard#mail.com",
friends: []
}
... or should I start a new collection (e.g. friendscollection)?
friends: {
{
userId : 09A09IQMSLDK0912,
friendId: 09A09IQMSLDAEAQS
}
}
Thanks for the advice!
You will want to add it as a new collection. You shouldn't model any of your collections to have an array property that can have an endless amount of items in it, otherwise, you will run into issues at scale.
Instead, you should create another collection and have them relate via ID as you mentioned.
I am not an expert on a Firebase DB but I know it functions very similarly to a MongoDB. For example, say the DB is a MongoDB. There is a limit to how large a collection item can be (BSON Limit: https://docs.mongodb.com/manual/reference/limits/) and if you allow a collection item to have an array property that can grow indefinitely, you will quickly reach this limit and you wont able to insert the item into the collection.
You will want to store each user's friends in your Firebase Database in a similar structure to this:
friends: {
userId1: {
friend1UserId: 123123123121 // timestamp of when the users became friends, or whatever value is useful to your application.
friend2UserId: 123123123123
friend3UserId: 213123123123
}
userId2: {
friend1UserId: 412124124124
friend2UserId: 213123213321
}
}
This is because Firebase is very limited with querying - and generally in a NoSQL database you would like to keep you data as wide as possible rather than going deeper.
I am stuck trying to allow an an array of admins access to their data.
I have a database structure like this:
{
"Respondents": {
"Acme Corp": {
"admins": ["mMK7eTrRL4UgVDh284HntNRETmx1", ""mx1TERNmMK7eTrRL4UgVDh284Hnt"],
"data": {data goes here...}
},
"Another Inc": {
"admins": ["Dh284HmMK7eTrRL4UgVDh284HntN", ""x1TERNmx1TERNmMK7eTrRL4UgVDh"],
"data": {their data goes here...}
}
}
}
And then I tried to set my rules like this
{
"rules": {
"Respondents": {
"$organisation" : {
".read": "root.child('Respondents').child($organisation).child('admins').val().includes(auth.id)",
".read": "root.child('Respondents').child($organisation).child('admins').val().includes(auth.id)"
}
}
}
}
..but that won't parse in the Firebase Database Rules editor
I get "Error saving rules - Line 7: No such method/property 'includes'", but I need something to match the user id with the array of admins.
Any experience or suggestions?
As you've found, there is no includes() operation in Firebase's security rules. This is because Firebase doesn't actually store the data as an array. If you look in the Firebase Database console or read this blog post you will see that Firebase stores it as a regular object:
"admins": {
"0": "mMK7eTrRL4UgVDh284HntNRETmx1",
"1": "mx1TERNmMK7eTrRL4UgVDh284Hnt"
}
And since that is a regular JavaScript object, there is no contains() method on it.
In general creating arrays are an anti-pattern in the Firebase Database. They're often the wrong data structure and when used are regularly the main cause of scalability problems.
In this case: you're not really looking to store a sequence of UIDs. In fact: the order of the UIDs doesn't matter, and each UID can be meaningfully present in the collection at most once. So instead of an array, you're looking to store set of uids.
To implement a set in Firebase, you use this structure:
"admins": {
"mMK7eTrRL4UgVDh284HntNRETmx1": true,
"mx1TERNmMK7eTrRL4UgVDh284Hnt": true
}
The value doesn't matter much. But since you must have a value to store a key, it is idiomatic to use true.
Now you can test whether a key with the relevant UID exists under admins (instead of checking whether it contains a value):
"root.child('Respondents').child($organisation).child('admins').child(auth.uid).exists()",
I'm trying to figure out how, if possible, to query a specifically named child node in all instances of a parent node in Firebase. It can be assumed that all parent nodes queried have this specifically named child node in it.
In this example, uid is a unique identifier for each user and I'm trying to get a list of displayNames:
{
users: {
uid: {
displayName: "stringOfSomeKind"
}
uid2: {
displayName: "moreStrings"
}
uid3: {
displayName: "evenMoreStrings"
}
...
}
}
The purpose of this is so I can check to see if a displayName is currently taken. (I can't use the displayName as the primary key because when a user logs in, I'll only have the uid available.)
How can I efficiently check to see if one of these displayNames is already taken? Do I have to denormalize my data to do so efficiently? If so, how?
Firebase world is quite different!
When such scenarios come you have to think to redesign your database structure, In your case uid is unique identifier so is displayName- speaking technically.
You will have to maintain additional data like:
{
users: {
uid: {
displayName: "stringOfSomeKind"
}
uid2: {
displayName: "moreStrings"
}
uid3: {
displayName: "evenMoreStrings"
}
...
}
displayNames: {
"display_name": "uid",
"display_name2": "uid2",
"display_name3": "uid3"
...
}
}
Happy Helping!
When you load a node from Firebase, you also get all data under that node. Assuming that you have more data per user than just their display name, that can indeed lead to needlessly loaded data.
If you only want to load a list of display names, you should indeed store a list of display names.
{
displayNames: {
"stringOfSomeKind": "uid",
"moreStrings": "uid2",
"evenMoreStrings": "uid3"
}
}
If you come from a background of relational/SQL databases, this may seem unnatural at first. For me it helped to realize that these structures are indexes, same as the "index by displayName" that you might add to your relational database. The difference is that in NoSQL/Firebase it is your code that maintains the index, while in most RDBMSs such indexes are maintained by the system.
I'm implementing an orbit.js adapter for firebase, orbit-firebase.
I'm looking for an efficient way to query for multiple records so that I can resolve relationships between objects e.g. course.participants
{
course: {
'c1': {
participants: ['p1', 'p2']
}
},
participant: {
'p1': {
name: "Jim"
},
'p2': {
name: "Mark"
}
}
}
Given I have the ids 'p1' and 'p2' what's an efficient way to query for both of them?
I can't use a query because I'm using security rules with the participants i.e. the user that's trying to resolve course.participants doesn't have access to all of the participants (bear in mind this is a contrived example).
I'd recommend that you move away from arrays in your JSON structures. These are nothing but pain in real-time, distributed data and don't work particularly well with security rules and situations like this.
Given this structure:
course: {
'c1': {
participants: {
'p1': true, 'p2': true
}
}
}
I could join these fairly easily. You can get a normalized ref that behaves just like a Firebase ref by using Firebase.util's NormalizedCollection:
var ref = new Firebase(...);
var coll = new Firebase.util.NormalizedCollection(
ref.child('course/c1/participants'),
ref.child('participant')
).select('participant.name').ref();
coll.on('child_added', function(snap) {
console.log('participant ' + snap.key(), snap.val());
});
Note that this data structure (sans the array) will also make it simpler to enforce read rules on participant data and the like by allowing you to directly reference the user ids under $courseid/participants/, since they are now keys that can match a $ variable.