Like the title suggests, I have a use case where I will write data to both firestore and realtime database. I am using the realtime database for operations that require live feedback to users and firestore to store data that will not really change but can be queried for more complex operations later on.
Due to my need of both databases, I would like to use the same UID when creating data in both databases to make it easy to retrieve in the future. The issue I have is determining which generated ID will satisfy the other service.
My thought process is since Realtime Database push ID is based on timestamp, it could create hot partitions for Firestore so indexing performance as data grows could get hurt in the future if I used the same ID there. But if I use firestore's generated ID in the realtime database, I will not have the data in the sorted fashion that realtime database creates pushed data.
I was wondering what solutions people used to tackle this use case and what options are available to me. Thanks!
If you need to order data, then simply store timestamps as fields instead of depending on the time-based sort order of Realtime Database push IDs. You can do this easily in both databases. Firestore makes obsolete the idea that unique IDs have any meaning other than simply being unique.
If you make sure your unique ID's are truly random like Firestore's, then you won't have any problems with indexing or writing documents.
Related
I have some documents in firestore have some fields in it. like collection "details" looks like this
{
id: "",
fields1: "",
userFK: Reference to users collection
}
Now I need to resolve userFK on the fly means that I don't want first fetch all the documents then query to userFk userFK.get()
Is there any method, its like doing a $lookup whick is supported in mongodb
Even In some case I want to fetch documents from "details" collection based of some specific fields in users
There is no way to get documents of multiple types from Firestore with a single read operation. To get the user document referenced by userFK you will have to perform a separate read operation.
This is normal when using NoSQL databases like Cloud Firestore, as they typically don't support any server-side equivalent of a SQL JOIN statement. The performance of loading these additional details is not as bad as you may think though, so be sure to measure how long it takes for your use-case before writing it off as not feasible.
If this additional load is prohibitive for a scenario, an alternative is to duplicate the necessary data of the user into each details document. So instead of only storing the reference to their document, you'd for example also store the user name.
This puts more work on the write operation, but makes the read operations simpler and more scalable. This is the common trade-off of space vs time, where in NoSQL databases you'll often find yourself trading time for space: so storing duplicate data.
If you're new to NoSQL data modeling, I highly recommend:
NoSQL data modeling
Getting to know Cloud Firestore
Edit: After posting the question I thought I could also make this post a quick reference for those of you needs a quick peek at some of the differences between these two technologies which might help you decide on one of them eventually. I will be editing this question and adding more info as I learn more.
I have decided to use firebase for the backend of my project. For firestore is says "the next generation of the realtime database". Now I am trying to decide which way to go. Realtime database or cloud firestore?
Billing:
At a first glance, it looks like firestore charges per number of results returned, number of reads, number of writes/updates etc. Real-time database charges based on the data transmitted. The number of read-write operations is irrelevant. They both also charge on the data stored on the google servers too (I think in this respect firestore is cheaper one). Why am I mentioning this price point? Because from my point of view, although it might a lower weight, it is also a point to consider while choosing the one over the other.
Scaling:
Cloudstore seems to scale horizontally seamlessly. I think this is not possible with the real-time database.
Edit:
In the real-time database, you need to shard your data yourself using multiple databases. And you can only do this if you are in BLAZE pracing plan.
ref: https://firebase.google.com/docs/database/usage/sharding
Performance & Indexing:
Another thing is the real-time database data structure is different in both. The real-time database stores the data as a JSON object in any way we structure them. Firestore structures the data as collections and documents. And hence the querying also changes between the two.
I think firestore does auto indexing which increases the read performance greatly too (which will decrease read performance). I am not sure if this is also the case with the real-time database.
Edit:
The real-time database does not automatically index your data. You need to do it yourself after a solid inspection of your data and your needs.
ref:https://firebase.google.com/docs/database/security/indexing-data
What other differences can you think of?
What would be (or has been) your choice for different types of projects?
Do you still go with the real-time database or have you migrated from that to the firestore? If so why?
And one last thing. How would you compare the SDKs of these two?
Thanks a lot!
What other differences can you think of?
what i think, ok. I use realtime-database for 6 months experience and difference is, firestore easy for sorting data. As Example, i want to retrieving user name based timestamp.
Query firstQuery = firestore.collection("Names").orderBy("timestamp", Query.Direction.DESCENDING).limit(10); // load 10 names
What would be (or has been) your choice for different types of
projects?
For me, Realtime-Database for Data Streaming when i work with Arduino, i want to store Drone Speed.
And Firestore for SMART OFFICE, like Air Conditioner, or light-room and Enterprise like Inventory Quantities, etc.
Do you still go with the real-time database or have you migrated from
that to the firestore? If so why?
still go with real-time because i need TREE for displaying streaming data strucure instead of query TABLE like firestore.
I have events in firebase database table where each event has certain fields. One of the field is event_type. What I want to achieve is to be able to visualize in graphical form, how many events of each type comes daily?
How do I do something like that in firebase database?
Q1. Is it possible to directly do this in firebase?
Q2. Do I need to move data to some other datasource (like Big query) and setup dashboard there?
It is definitely possible to create a dashboard with aggregate data directly on the Firebase Realtime Database. But you'll have to take a different approach than with e.g. BigQuery.
With relational databases, you'll create a dashboard by running aggregation queries. For example to show how many events of each type, you'll run something like SELECT type, COUNT(*) FROM events GROUP BY type.
The Firebase Realtime Database (and most NoSQL databases) don't have such a GROUP BY operation, not a COUNT() method. So that means that you'd have to load all data into your dashboard, and group/count it there, which is quite expensive. That why on NoSQL databases you'll typically keep a running count for each type in the database and update that on every write operation. While this puts an overhead on each write operation, the dashboard itself suddenly becomes very simply when you do this. For an example of a simple counter, see the function-samples repo.
This approach only works if you know up front what counters (and other aggregates) you want to show in the dashboard. If that isn't the case, many developers use the nightly backups from the Realtime Database to ingest the data into another system that lends itself more to exploratory querying, such as BigQuery.
Either approach can work fine. The right approach is a matter of your exact use-case (e.g. do you know the exact data you want in the dashboard, or are you still figuring that out?) and what you're most comfortable with.
I'm totally new to Firebase, and I'm trying to get my head round the best db model design for 'relational' data, both 1-1 and 1-many.
We are using the Firestore db (not the realtime db).
Say we have Projects which can contain many Users, and a User can be in multiple Projects
The UI needs to show a list of Users in a Project which shows things like email, firstname, lastname and department.
What is the best way to store the relationship?
An array of User ids in the Project document?
A map of Ids in the Project document?
Ive read the above approaches were recommended, but was that for realtime database? Firestore supports Sub Collections, which sound more appropriate...
A sub collection of Users in the Project document?
A separate collection mapping Project id to User id?
A Reference data type? I've read here https://firebase.google.com/docs/firestore/manage-data/data-types about Reference data type, which sounds like what I want, but I cant find any more on it!
If its just a map or array of Ids, how would you then retrieve the remaining data about the user? Would this have to sit in the application UI?
If its a sub collection of Users documents, is there any way to maintain data integrity? If a user changed their name, would the UI / a cloudFunction then have to update every entry of that users name in the Sub collections?
any help / pointers appreciated...
The approach for modeling many-to-many relationships in Firestore is pretty much the same as it was in Firebase's Realtime Database, which I've answered here: Many to Many relationship in Firebase. The only difference is indeed that you can store the lookup list in a sub-collection of each project/user.
Looking up the linked item is also the same as before, it indeed requires loading them individually from the client. Such a client-side join is not nearly as slow as you may initially expect, so test it before assuming it can't possibly be fast enough.
Ensuring data integrity can be accomplished by performing batched writes or using transactions. These either completely succeed or completely fail.
I have a collection, itemsCollection, which contains a very large amount of small itemDocs. Each itemDoc has a subcollection, statistics. Each itemDoc also has a field "owner" which indicates which user owns the itemDoc.
itemsCollection
itemDoc1
statistics
itemDoc2
statistics
itemDoc3
statistics
itemDoc4
statistics
...
I also have a collection, usersCollection, which contains basic user info.
usersCollection
user1
user2
user3
...
Since each itemDoc belongs to a specific user, it's necessary to display to each user which itemDocs they own. I have been using the query:
db.collection("itemsCollection").where("owner", "==", "user1")
I am wondering if this will scale effectively, i.e. whenever itemsCollection gets to be millions of records? If not, is the best solution to duplicate each itemDoc and its statistics subcollection as a subcollection in the user document, or should I be doing something else?
As Alex Dufter, the product manager from Firebase, explained in one of days at Firebase Dev Summit 2017 that Firestore was inspired in many ways by the feed-back that they had on the Firebase Realtime Database over the years. They faced two types of issues:
Data modelling and querying. Firebase Realtime Database cannot query over multiple properties because it ussaly involves duplication data or cliend-side filtering, which we all already know that is some kind of messy.
Realtime Database does not scale automatically.
With this new product, they say that you can now build an app and grow it to planetary scale without changing a single line of code. Cloud Firestore is also a NoSQL database that was build specifically for mobile and web app development. It's flexible to build all kinds of apps and scalable to grow to any size.
So because the new database was build knowing this iusses, duplication data is not nedeed anymore. So you will not have to worry about using that line of code, even if your data will grow to millions of records, it will scale automatically. But one thing you need to remember, if you will use multiple conditions, don't forget to use the indexes by simply adding them in the Firebase console. Here are two simple examples from the offical documentation:
citiesRef.whereEqualTo("state", "CO").whereEqualTo("name", "Denver");
citiesRef.whereEqualTo("state", "CA").whereLessThan("population", 1000000);