I am trying to make notification system for my website:
This is the table structure
notification
-----------------
id (pk)
userid
notification_type (for complexity like notifications for pictures, videos, apps etc.)
notification
time
notificationsRead
--------------------
id (pk) (i dont think this field is required, anyways)
lasttime_read
userid
Now my understanding is that when a notification is added, we need to find the users friends and insert all those rows in the notification table right ? If this is correct, then, what would be the best way to achieve this ?
Triggers?
Write T-SQL (sql query in server side) to Select all the friends and then use SQL bulk copy?
A good relational approach would be to set up your tables so that you have a people table. Each person has a primary key that uniquely identifies them. Then another tables links the peoplekeys together as friends. There are a few ways to go about this but as a rough example:
People
---------------------------------------
PeopleKey | Name | <other profile data>
Then you have a table that relates people to each other as friends.
Friendships
----------------------------------------------------------------------
PeopleKey | FriendKey (fk to peoplekey) | <details about friendship>
Then you have your notifications table that says a person has done something.
Notification
-----------------------------------------------------------
NotificationKey | PeopleKey | Date | <notification details>
With a query and your friendship relations you can obtain all the notifications for all the friends of a person:
select notification.* from notification inner join
friendships on friendships.peoplekey = notification.peoplekey
where date = #importantdate
The relationship can reveal a lot without having to store repetitive data. There are 1000 other ways to join, query, or link the tables to friendships and events. So for example, you could say show all events where peoplekey is one of my friends. Show all events where peoplekey is only one of my friends etc. etc. etc.
Related
First of all thank you to anybody reading through this and offering any advice and help. It is much appreciated.
I'm developing a small custom CRM (ouch) for my father's business (specialty contractor) and I'm using Firestore for my database. It is supposed to be very lean, with not much "bling" but stream lined to his speciality contracting business, which is very hard to to get any other custom CRM to be applied to his process. I have gotten quite far and have a decent size implementation, but am now running into some very fundamental issues as everything is expanding.
I admit that only having experience with relational databases (and not much of that either) left me scratching my head a few times when properly setting up my database structure and am running into some issues with Firestore. I'm also a fairly novice developer and I feel I'm tackling something that is just way out of my league. (but there's not much turning around now being a year into this journey)
As of right now I'm using Top Level Collections for what I am presenting here. I recently started using Sub-Collections for some other minor features and started questioning if I should apply that for everything.
A big problem that I foresee is because I want to query in a multitude of ways, I am already consuming almost 100 composite indexes at this time. There is still lots to add, so I need to reduce the amount of composite indexes that my current and future data structure needs.
So I am somewhat certain, that my data model probably is deeply flawed and needs to be improved/optimized/changed. (Which I don't mind doing, if that's what it takes, but I'm lost on "how") I don't need a specific solution, but maybe just some pointers, generally speaking, of what approaches are available. I think I might be lacking an "aha" moment. If I understand a pattern, I can usually apply that further in other areas.
I will make my "Sales Leads Collection" a central concern of this post, as it has the most variations of querying.
So I have a mostly top level collection structure like this, but also want to prefix, that besides writing the IDs to other Documents, I will "stash" an entire "Customer" or "Sales Rep" Object/Document with other Documents and I have Cloud Functions that will iterate through certain documents when there are updates, etc. (To avoid extra reads, i.e. when I read a SalesLead, I don't need to read the SalesRep and Customer Document, as they are also stashed/nested with the SalesLead)
| /sales_reps //SalesReps Collection
| /docId //Document ID
| + salesRepId (document id)
| + firstName
| + lastName
| + other employee/salesRep related info etc.
| /customers //Customers Collection
| /docId //Document ID
| + customerId (document id)
| + firstName
| + lastName
| + address + other customer specific related info such as contact info (phone, email) etc.
Logically Sales Leads are of course linked to a Customer (one to many, one Customer can have many leads).
All the Fields mentioned below I need to be able to "query" and "filter"
| /sales_leads //SalesLeads Collection
| /docId //Document ID
| + customerId (document id) <- this is what I would query by to look for leads for a specific customer
| + salesRepId (document id) <- this is what I would query by to look for leads for a specific sales Rep
| + status <- (String: "Open", "Sold", "Lost", "On Hold)
| + progress <- (String: "Started", "Appointment scheduled", "Estimates created", etc. etc., )
| + type <- (String: New Construction or Service/Repair)
| + jobTye <- (String: Different Types job Jobs related to what type of structures they are; 8-10 types right now)
| + reference <- (String: How the lead was referred to the company, i.e. Facebook, Google, etc. etc. );
| + many other (non queryable) data related to a lead, but not relevant here...
SalesEstimates are related to Leads in a one to many relationship. (one lead can have many estimates) But Estimates are not all that relevant for this discussion, but just wanted to include it anyhow. I query and filter estimates in a very similar way I do with leads, though. (similar fields etc.)
| /sales_estimates //SalesEstimates Collection
| /docId //Document ID
| + salesLeadId (document id) <- this is what I would query by to look for estimates for a specific lead
| + customerId (document id) <- this is what I would query by to look for estimates for a specific customer
| + salesRepId (document id) <- this is what I would query by to look for estimates for a specific sales Rep
| + specific sales Lead related data etc....
In my "Sales Lead List" on the client, I have some Drop Down Boxes as Filters, that contain Values (i.e. Sales Reps) but also haven an Option/Value "All" to negate any filtering.
So I would start assembling a query:
Query query = db.collection("sales_leads");
//Rep
if (!salesRepFilter.equals("All")) { //Typically only Managers/Supervisors woujld be able to see "all leads" whereas for a SalesRep this would be set on his own ID by default.
query = query = query.whereEqualTo("salesRepId", salesRepId);
}
//Lead Status (Open, Sold, Lost, On Hold)
if (!statusFilter.contains("All")) {
query = query.whereEqualTo("status", statusFilter);
}
//Lead Progress
if (!progressFilter.contains("All")) {
query = query.whereEqualTo("progress", progressFilter);
}
//Lead Type
if (!typeFilter.contains("All")) {
query = query.whereEqualTo("leadType", typeFilter);
}
//Job Type
if (!jobTypeFilter.contains("All")) {
query = query.whereArrayContains("jobTypes", jobTypeFilter);
}
//Reference
if (!referenceFilter.contains("All")) {
query = query.whereEqualTo("reference", referenceFilter);
}
Additionally I might want to reduce the whole query to a single customer (this typically means that all other filters are skipped and "all leads for this customer are shown). This would happen if the user opens the Customer Page/Details and clicks on something like "Show Leads for this customer".
//Filter by Customer (when entering my SalesLead List from a Customer Card/Page where user clicked on "Show Leads for this Customer")
if (filterByCustomer) {
query = query.whereEqualTo("customerId", customerFilter);
}
//And at last I want to be able to query the date Range (when the lead was created) and also sort by "oldest" or "newest"
//Date Range
query = query.whereGreaterThan("leadCreatedOnDate", filterFromDate);
.whereLessThan("leadCreatedOnDate", filterToDate;
//Sort Newest vs Oldest
if (sortByNewest) { //either newest or oldest
query = query.orderBy("leadCreatedOnDate", Query.Direction.ASCENDING);
} else {
query = query.orderBy("leadCreatedOnDate", Query.Direction.DESCENDING);
}
And that would complete my query on sales leads. Which that all works great right now but I am anxious about going forward and ultimately hitting the composite index limitation. I don't have an exact number, but I am probably entertaining 25-30 composite indexes just for my collection of sales_leads. (Yikes!)
Not only are there many fields to query by, the amount of composite indexes required is multiplied by the combination of possible filters set. (UGH)
I need to be able to query all leads and then filter them by the fields mentioned above (when describing my sales_leads collection).
So instead of keeping all these collections as top level collections I am guessing that somehow I should restructure my database by entertaining sub collections, but I tried modeling this with different approaches and always seem to hit a wall.
I suppose I could have "sales_leads" as a subcollection under each customer object and could use a collection group query to retrieve "all leads", but those require composite indexes, too right? So it would just be tradeoff for that one searchable field. (..hits wall..)
Sorry for the length. I hope it's readable. I appreciate any help, feedback and input. I'm in a very anxious and frustrated position.
If this doesn't work, I might need to consider professional consultation.
Thanks!
Here are a few things I think will help you.
First, watch the AWS re:Invent 2018: Amazon DynamoDB Deep Dive on YouTube. It's about DynamoDB but DynamoDB is a NoSQL database very similar to Firestore and the concepts universally apply. Midway through the video, Rick uses a company like yours as an example and you may be surprised to see how effectively he can reduce query count simply through data modeling.
Second, familiarize yourself with Firestore's index merging. In situations like yours, it may be better to manually create your composite indices, or at least manually audit them, because Firestore's automatic indexing doesn't guarantee the most efficient menu of composite indices. Remember, composite indices are automatically created based on the order you execute queries and if you execute a query later that could be better structured by voiding a previous index, Firestore will not go back and delete it for you—you have to.
I'm highly suspicious of the fact that the sales-lead query consumes 25-30 composite indices; that number seems far too high to me given how many fields in the documents are indexed. Before you do anything—after having watched the video and studied index merging, of course—I'd focus entirely on this collection. You must be completely certain of the maximum number of composite indices this collection needs to consume. Perhaps create a dummy collection and experiment with index merging and really understand how it works because this alone may solve all of your problems. I would be shocked if Firestore couldn't handle your company's use case.
Third, don't be afraid to denormalize your data. The fundamental premise of NoSQL is really denormalization—that is, data storage really should be your least concern and computation/operation really should be your greatest concern. If you can reduce your query count by duplicating data over multiple documents in multiple collections, that is simply what you must do if the alternative is hitting 200 composite indices.
I have a Google Firebase Realtime Database with the following structure:
project
|
- users
|
- uid_1
|
- name
- age
- ...
|
- uid_2
|
- name
- age
- ...
I'm new with Google Firebase. I can read all db-fields from a specific 'uid'. But how can I collect all names from all unsers? I want to make an file-uploader and to assing the file to an existing user, I want to use a dropdown-menu to select the user.
With the way you have this data structured, you're going to have to write code to query everything under "users", iterate the children, and pull the names out of each child.
Realtime Database doesn't offer any sort of SQL-like projection type query that will get you just the vales of certain children. Or to put it another way, you can't do anything like "select name from users". If getting only the name is a common operation for you, you will need to duplicate only the names into another parent node, and query that new node instead.
The Firebase Admin SDK allows retrieving the entire list of users in batches. But max result is 1000.
Or you can denormalizing your data.
My app has two Firestore Collection:
Events (collection)
Document
eventId
hostId (userId of organizer)
title
etc.
Users (Collection)
Document
userId
friends (array of userId's)
age
etc.
I would like for a user to query all events created by his friends.
I considered
do a seperate query for each friend and pull each friend events. This could get ugly with sorting on dates.
adding a hostFriends field in the event and use an array-contains user's Id query. But this will be problematic if someone add a friend after having created an event. I would have to sync this across continiously..
doing client side filtering, but this will lead to many unneccesary reads..
Would be happy to hear any additional idea's or if one of the above would make sense? many many thanks!
This is my first post, apologies if I made any formatting mistakes.
I wanted to ask for an advice on data structuring best practices for Cloud Firestore for the following scenario.
There's a booking/appointment app. Hotels rent out rooms. Each hotel has multiple rooms. Clients can search the rooms of all hotels by availability on specific days.
What is the best way to structure the availability data in Firestore so I could create a view of all available rooms throughout all hotels.
I thought of creating a separate collections where I would put all the reservations referencing the room ID and date of the reservation. However, it seems like I won't be able to search for available slots this way since Firestore can't perform 'not equals' queries.
So I thought I would create an array field for each room containing all the available dates as timestamps. This creates another problem. Even though I can use 'array_contains' query, users can't check availability for more than one day this way since 'array_contains' can only be used once per query.
What would be the most efficient way to structure the data in this case?
Thank you!
What is the best way to structure the availability data in Firestore so I could create a view of all available rooms throughout all hotels.
A possible database structure that can help you achieve what you want, might be this:
Firestore-root
|
--- hotels (collection)
| |
| --- hotelId (document)
| |
| --- //Hotel properties
|
|
--- rooms (collection)
| |
| --- hotelId (document)
| |
| --- hotelRooms (collection)
| |
| --- roomId (document)
| |
| --- available: true
| |
| --- hotel: "hotelId"
| |
| --- //Other room properties
|
|
--- availabeRooms (collection)
|
--- roomId (document)
|
--- available: true
|
--- hotel: "hotelId"
|
--- //Other room properties
As you can probably see, I have duplicate some data in order to achieve what you want. This practice is called denormalization and is a common practice when it comes to Firebase. For a better understanding, I recommend you see this video, Denormalization is normal with the Firebase Database. It's for Firebase realtime database but same principles apply to Cloud Firestore.
Also, when you are duplicating data, there is one thing that need to keep in mind. In the same way you are adding data, you need to maintain it. With other words, if you want to update/detele an item, you need to do it in every place that it exists.
Using this database schema, you can simply query the database to get all available rooms from all hotels by attaching a listener on availabeRooms reference and get all room objects. If you want to get the details of the hotel from which a particular room is apart, you need to make an extra call to get the hotel details. I have stored within the room object, only a reference of the hotel object which is as you can see, the hotelId. You can also store the entire hotel object but before taking a decision, I recommend you to be aware of some details that can be found in my answer from this post.
Furthermore, if a room becomes unavailable, simply change the value of the available property that exist under rooms -> hotelId -> hotelRooms -> roomId to false and remove the corresponding room from the availabeRooms collection. That's it!
P.S. If you want to get all the available rooms within a single hotel, just attach a listener on rooms -> hotelId -> hotelRooms and get all available rooms using a query that should look like this:
Query query = db.collection("rooms").document(hotelId)
.collection("hotelRooms").whereEqualTo("available", true);
Edit:
According to your comment regarding the date of the reservation, you should create a calendar of reservations for each room separately. Then just simply create a function, in Cloud Function that can be triggered using a cron job. This function can help you check the availability for each room daily. If the room is available, set the available to true otherwise, set the property to false and remove the room from the availabeRooms collection.
I'm trying to understand the best database structure to store and retrieve user to user conversations using the Firebase database for a chat app (web based).
My current plan is to give each chat its own ID which would be created by combining the unique Firebase IDs of the two chat participants (like UserID1_UserID2), for example: FQ5d0jwLQDcQLryzevBxKrP72Bb2_GSIbxEMi4jOnWhrZaq528KJKDbm8 this chat ID would be stored in the database, and would contain the messages sent between the two participants.
Example layout:
MYAPP
|_______conversations
| |_____UserID1_UserID2
| | |
| | |__OshwYF72Jhd9bUw56W7d
| | | |__name:"Jane"
| | | |__text:"Hello!"
| | |
| | |__KbHy4293dYgVtT9pdoW
| | |__PS8tgw53SnO892Jhweh
| | |__Qufi83bdyg037D7RBif
| | |__Gicuwy8r23ndoijdakr
| |
| |_____UserID5_UserID16
| |_____UserID8_UserID7
| |_____UserID3_UserID8
|
|_______users
Whenever a user signs into the app, they'll see a list of their contacts. When they select one to chat with, I would use some Javascript to combine their, and their selected friend's Firebase ID to generate the chat ID. This chat ID would then be either created in the database (if it's their first time to chat), or it would be used to load previous messages that they have exchanged (if they have chatted before), from the database.
My question is, is this the correct method to use? What issues might I run into if I use this method? For example, would I have problems if I try to implement group conversations (with more than 2 people) in the future?
I'd be really grateful for any help, or examples of the correct database layout logic for a person to person (and group) chat application using Firebase/a no SQL database.
Thank you in advance!
Something that I would like to point out as one of the most important "rules" to consider when creating a NoSQL database is that you must
Structure your data after the view.
This means that the data must be structured in such a way, that when you want to display it on your view (probably your html pages) you do a single read.
So in order to find the best way to structure your database, you must first look at your view. And try to imagine how you would read data (direct reads and queries).
Altought your current structure looks good (for what you're building now), yes, you might have some problems when creating group chats. I would recommend using something like this:
You wil notice that this way, you can actually add more participants to each chat easily. Note that this isn't supposed to be your only node. You would have to create other nodes like users (to store the user details) and messages (store each chat's messages), etc
To help you with that, I recommend watching David East's Firebase Database For SQL Developers.