I'm developing a shopping cart mobile application using ionic framework and FIREBASE as back-end . I have a requirement that I need to join JOIN, SORT, FILTER and PAGINATE by multiple data attributes.
Sample data structure is given below.
products : {
prod1:{
name: Samsung-s4,
type: pro,
category: Phone,
created_datetime: 1426472828282,
user: user1,
price: 400
},
prod2:{
name: iPhone 5s,
type: pro,
category: Phone,
created_datetime: 1426472635846,
user: user2,
price: 500
},
prod3:{
name: HP Laptop i3,
type: regular,
category: Computer,
created_datetime: 1426472111111,
user: user1,
price: 600
}
}
user_profiles : {
user1:{
name: abc_user,
display_name: ABC,
email: abc#mail.com
},
user2:{
name: xyz_user,
display_name: XYZ,
email: xyz#mail.com
}
}
I need to query the products using multiple ways. two simple examples given below.
1) Get all products with Pagination, where type is "pro" then join with user_profiles and SORT by created date.
2) Get all products with Pagination, filter by category then join with user_profiles and SORT by created date.
Like above there will be more and more filtering options coming in Eg: price.
My main problem is, I couldn't find straight forward way of doing this using FIREBASE query options. Also I referred firebase util documentation but there also I don't see a way of getting this done.
As far as I see only way to get this done is, do the majority of processing in client end by getting all the data (or majority of the data) in to client side and do the SORT / FILTER / PAGINATE in client end.
But we are expecting thousands of records in these schemas, therefore there will be a huge performance impact if we do the processing in client end !!
Appreciate your expertise/support to solve this problem.
Thank you in advance.
UPDATE:
I changed the data structure (as webduvet explained) and tried with firebase-util but failed to achieve what i want. CRITERIA :Products -> Filter By type/pro -> Sort By products.created_date
type: {
pro:{
product1: user1,
product3: user1,
...
},
regular:{
...
}
}
firebase-util - angularfire code
var list = $firebase(new Firebase.util.NormalizedCollection(
ref.child("products").orderByChild('created_datetime'),
ref.child('type').child('pro')
).select(
{key: "products.name" , alias: 'name'}
).ref()).$asArray();
"products" will have hundred thousand records, so we have to make sure we restrict as much possible in firebase end rather than handling in client end.
Please help !
This has to be done not by queries but by database design. You need to store data in de-normalised way for example:
type: {
pro:{
product1: user1,
product3: user1,
...
},
regular:{
...
}
}
the above structure will give you option to query all pro products and retrieve the user id as well. Firebase offer good sorting mechanism so there should not be a problem. The more complex query you require the more complex the data structure will be needed and the more denormalized data you will have.
But as pointed out by #Swordfish0321 sql type of db might suit you much better after all.
Related
So this is the structure of my Firebase DB right now, I am using the Firebase REST API:
"company": {
company1_id {
id: company_id,
userId: userid,
name: name
//someotherstuff
}
company2_id {
id: company_id,
userId: userid,
name: name,
//someotherstuff
}
}
Soo, right now I am getting the companies belonging to one user by calling :
"firebasedbname.firebaseio.com/company.json?orderBy="userId"&equalTo=userId"
This works perfectly fine and gets the corresponding data, but now I want it to order the companies alphabetically by name, and then i try this:
"firebasedbname.firebaseio.com/company.json?orderBy="name"&equalTo=userId"
But this time, it returns no data! Even though i have added .indexOn: "name" to the company node.Any help will be aprreciated.
As explained in the doc, if you want to filter data you need to first "specify how you want your data to be filtered using the orderBy parameter", and then you need to "combine orderBy with any of the other five parameters: limitToFirst, limitToLast, startAt, endAt, and equalTo".
So if you added "an .indexOn: "name" to the company node", it means that you intend to query as follows:
https://xxxx.firebaseio.com/company.json?orderBy="name"&equalTo="companyName"
You cannot order by (company) name and filter on userId.
If you want to get all the companies corresponding to a specific user and order them by the company name, you will need to use ?orderBy="userId"&equalTo=userId" and do the sorting in the client/application calling the REST API.
Wondering if I have the right mindset here... In firebase (with Vue) I have a userscollection:
users : {
uid: "09A09IQMSLDK0912",
name: "Gerard",
email: "gerard#mail.com"
}
If I want the user to add friends, should I add it to the userscollection?
users : {
uid: "09A09IQMSLDK0912",
name: "Gerard",
email: "gerard#mail.com",
friends: []
}
... or should I start a new collection (e.g. friendscollection)?
friends: {
{
userId : 09A09IQMSLDK0912,
friendId: 09A09IQMSLDAEAQS
}
}
Thanks for the advice!
You will want to add it as a new collection. You shouldn't model any of your collections to have an array property that can have an endless amount of items in it, otherwise, you will run into issues at scale.
Instead, you should create another collection and have them relate via ID as you mentioned.
I am not an expert on a Firebase DB but I know it functions very similarly to a MongoDB. For example, say the DB is a MongoDB. There is a limit to how large a collection item can be (BSON Limit: https://docs.mongodb.com/manual/reference/limits/) and if you allow a collection item to have an array property that can grow indefinitely, you will quickly reach this limit and you wont able to insert the item into the collection.
You will want to store each user's friends in your Firebase Database in a similar structure to this:
friends: {
userId1: {
friend1UserId: 123123123121 // timestamp of when the users became friends, or whatever value is useful to your application.
friend2UserId: 123123123123
friend3UserId: 213123123123
}
userId2: {
friend1UserId: 412124124124
friend2UserId: 213123213321
}
}
This is because Firebase is very limited with querying - and generally in a NoSQL database you would like to keep you data as wide as possible rather than going deeper.
I have three models, User, Project and ProjectMember. Keeping things simple, the models have the following attributes:
User
- id
Project
- id
- owner_id
- is_published
ProjectMember
- user_id
- project_id
Using sequelize.js, I want to find all projects where the project owner is a specific user, or where there is a project member for that project whose user is that user, or where the project is published. I imagine the raw SQL would look something like this:
SELECT p.*
FROM Project p
LEFT OUTER JOIN ProjectMember m
ON p.id = m.project_id
WHERE m.user_id = 2
OR p.owner_id = 2
OR p.is_published = true;
There are plenty of examples out there on how to perform a query on an association, but I can find none on how to do so conditionally. I have been able to query just the association using this code:
projModel.findAll({
where: { },
include: [{
model: memberModel,
as: 'projectMembers',
where: { 'user_id': 2 }
}]
})
How do I combine this where query in an $or to check the project's owner_id and is_published columns?
It's frustrating, I worked for hours to try to solve this problem, and as soon as I ask here I found a way to do it. As it turns out, sequelize.js developers recently added the ability to use raw keys in your where query, making it (at long last) possible to query an association inside of the main where clause.
This is my solution:
projModel.findAll({
where: {
$or: {
'$projectMembers.user_id$': 2,
owner_id: 2,
is_published: true
}
},
include: [{
model: memberModel,
as: 'projectMembers'
}]
})
Note: this solution breaks if you use 'limit' in the find options. As an alternative, you can fetch all results and then manually limit them afterwards.
im trying to use firebase to store and retrieve data for my application.. i know that it is recommended to denormalize data and that it may require data replication..
my scenario is as follows:
there are a number of users in the system..
there are a number of posts in the system..
any user should be able to get a list of posts for a particular user..
each posts has a number of users as participants..
i am tempted to use the following structure for this:
users: {
abc: {
name: 'UserA',
profilePicture: 'imageA.png'
},
pqr: {
name: 'UserB',
profilePicture: 'imageB.png'
},
xyz: {
name: 'UserC',
profilePicture: 'imageC.png'
},
...,
...,
...
},
posts: {
def: {
title: 'PostA',
users: {
abc: true,
def: true,
ghi: true,
...,
...,
...
}
},
stu: {
title: 'PostB',
users: {
abc: true,
xyz: true,
...,
...,
...
}
},
...,
...,
...
}
the issue with this is that if i need to show a list of users with each post, i will need to make a query to POST, and then make sequential calls to USER for each user inside that post to get the name/profilePicture data..
if i replicate the user info inside posts as well, the issue becomes that if a user later changes her profilePicture or name, then existing posts will still show the old data..
how can i structure this data better so these cases are efficient?
thanks..
Don't replicate data inside posts. Read Firebase Docs about structuring data
Best practices:
Avoid nesting data
Flatten data structures
if you include data in post you are breaking those 2 rules (and you don't want it).
Multiple calls are not bad.
I think I've wrapped my head around denormalization as a primary method of optimization when storing data in Firebase as mentioned in question like this one and in this blog post but I'm getting stuck on one small detail.
Assuming I have two things in my domain, users and posts as in the blog article I mentioned, I might have 20,000 users and 20,000 posts. Because I denormalized everything like a good boy, root/users/posts exists as does root/posts. root/users/posts has a set of post keys with a value of true so that I can get all post keys for a user.
users: {
userid: {
name: 'johnny'
posts: {
-Kofijdjdlehh: true,
-Kd9isjwkjfdj: true
}
}
}
posts: {
-Kofijdjdlehh: {
title: 'My hot post',
content: 'this was my content',
postedOn: '3987298737'
},
-Kd9isjwkjfdj: {
title: 'My hot post',
content: 'this was my content',
postedOn: '3987298737'
}
}
Now, I want to list the title of all posts a user has posted. I don't want to load all 20,000 posts in order to get the title. I can only think of the following options:
Query the root/posts path in some way using the subset of keys that are set to true in the root/users/posts path (if this is possible, I haven't figured out how)
Store the title in the root/users/posts so that each entry in that path has the title duplicated looking like this:
posts: {
-Kofijdjdlehh: true
}
becomes
posts: {
-Kofijdjdlehh: {
title: 'This was my content'
}
}
This seems reasonable, but I haven't seen a single example of doing this, so I'm concerned that it's some anti-pattern.
Another way I haven't been able to find
I appreciate any pointers you might have or documentation I might have missed on this use case.
Either are valid solutions. #1 would be more work for whoever is reading the data, while #2 would be more work when data is saved. Also for #2, you'd have to handle updates to post's titles, though this would be pretty easy with the new multi-path updates.
To implement #1, you'd have you essentially do two queries. Here's a really basic solution which only handles adding posts. It listens for posts being added to the user, and then hooks up a listener to each post's title.
var usersPosts = {};
ref.child('users').child(userId).child('posts').on('child_added', function(idSnap) {
var id = idSnap.key();
ref.child('posts').child(id).child('title').on('value', function(titleSnap) {
usersPosts[id] = titleSnap.val();
});
});
For a third solution, you could use firebase-util, which automagically handles the above scenario and more. This code would essentially do the same as the code above, except it comes with the bonus of giving you one ref to handle.
new Firebase.util.NormalizedCollection(
[ref.child('users').child(userId).child("posts"), "posts"],
[ref.child("posts"), "post"]
).select(
{
key: "posts.$value",
alias: "x"
},
{
key: "post.title",
alias: "title"
}
).ref();
Note that the x value will always be true. It's necessary to select that because firebase-util requires you to select at least one field from each path.