After i execute my Firestore-Test-App made with Flutter i look at the Firestore analytics to see how many request my Test-App made. It shows me that i make a minimum of 20 up to 60 document reads with 1 start of the Test. The problem is, the test should result in a maximum of ~1-3 reads as i understand it.
I've read https://firebase.google.com/docs/firestore/pricing. It did help to understand the billing logic of firestore, but following that logic in the article i should be making a maximum of ~5 reads.
This thread: Firestore - unexpected reads also suggests that maybe, the document reads come from the opened Firebase console, viewing the documents. So i closed it before the test and opened it 30 min after. This did not change the result. I also set breakpoints and the code did only execute once.
I opened a completly new Flutter Project to test it.
This is the only part making read requests:
CollectionReference dbUsers = dbInstance.collection("Users");
var user = dbUsers
.where("docId", isEqualTo: fireAppUser.user.uid)
.limit(1)
.snapshots();
var _userSub = user.listen((value) {
if (value.documents.isNotEmpty && value.documents.first.data != null)
print(value.documents.first.data);
});
_userSub.cancel()
Below my firestore rule, which is on default settings for now.
rules_version = '2';
service cloud.firestore {
match /databases/{database}/documents {
match /{document=**} {
allow read, write;
}
}
}
I have exactly 5 documents in my Database.
I expect to have a maximum of ~5 document reads. Please help me to undestand why this Snippet causes an unexpected number of reads. What could cause it?
Edit: forgot to append the cancel() to the snippet
That code can make a nearly unlimited number of reads, since it's adding a listener to the document. It will cost 1 read the first time you run it, and continue to incur reads as the listener remains added, as the document changes over time.
If the accounting doesn't make sense to you, contact Firebase support with your exact reproduction instructions.
Related
I'm really digging Firestore but it's hard to find answer to specific question so here I am. This is just to be sure I understood properly how security rules works :)
Here's my schema:
/databases/{database}/documents/Bases/Base1 {
roles: { // map
user1: {admin: true}
},
Items: { // SubCollection
item1: {
name: "Hello World"
},
...n,
item10: {
name: "Good Bye World"
}
}
}
I want my user1 to fetch all 10 items in Base1. Query is pretty simple db.collection('Bases').doc('Base1').collection('Items').get()
But I also want to be sure that user1 is an admin in Base1. So I'm setting this security rules:
match /bases/{baseId}/items/{itemId}{
allow read: if request.auth != null
&& get(/databases/$(database)/documents/bases/$(baseId)).data.roles[request.auth.id].admin == true
}
Which works, all good. Here're the questions:
1/ I understand this rules get() will cost me one read (which is very cheap I know). Is it one read per query OR one read per document that needs to be validate? ie. 10 reads in my case.
2/ I assume answer to 1/ is that it'll cost 10 reads. But, as I'm always querying the same $(baseId), cache will kick-in and even if not guarantee, it should drastically reduce the number of charged reads (theorically 1 read even if I'm fetchin 1000 docs)?
Any other advice on how to handle those kind of schemas are welcome. I know read ops are very cheap but I like to understand where I'm going :)
Thanks SO :)
The cost of a get() in security rules only applies once per query. It does not apply per document fetched.
Since you have one query, it will cost one read.
I'm having slow performance issues with Firestore while retrieving basic data stored in a document compared to the realtime database with 1/10 ratio.
Using Firestore, it takes an average of 3000 ms on the first call
this.db.collection(‘testCol’)
.doc(‘testDoc’)
.valueChanges().forEach((data) => {
console.log(data);//3000 ms later
});
Using the realtime database, it takes an average of 300 ms on the first call
this.db.database.ref(‘/test’).once(‘value’).then(data => {
console.log(data); //300ms later
});
This is a screenshot of the network console :
I'm running the Javascript SDK v4.50 with AngularFire2 v5.0 rc.2.
Did anyone experience this issue ?
UPDATE: 12th Feb 2018 - iOS Firestore SDK v0.10.0
Similar to some other commenters, I've also noticed a slower response on the first get request (with subsequent requests taking ~100ms). For me it's not as bad as 30s, but maybe around 2-3s when I have good connectivity, which is enough to provide a bad user experience when my app starts up.
Firebase have advised that they're aware of this "cold start" issue and they're working on a long term fix for it - no ETA unfortunately. I think it's a separate issue that when I have poor connectivity, it can take ages (over 30s) before get requests decide to read from cache.
Whilst Firebase fix all these issues, I've started using the new disableNetwork() and enableNetwork() methods (available in Firestore v0.10.0) to manually control the online/offline state of Firebase. Though I've had to be very careful where I use it in my code, as there's a Firestore bug that can cause a crash under certain scenarios.
UPDATE: 15th Nov 2017 - iOS Firestore SDK v0.9.2
It seems the slow performance issue has now been fixed. I've re-run the tests described below and the time it takes for Firestore to return the 100 documents now seems to be consistently around 100ms.
Not sure if this was a fix in the latest SDK v0.9.2 or if it was a backend fix (or both), but I suggest everyone updates their Firebase pods. My app is noticeably more responsive - similar to the way it was on the Realtime DB.
I've also discovered Firestore to be much slower than Realtime DB, especially when reading from lots of documents.
Updated tests (with latest iOS Firestore SDK v0.9.0):
I set up a test project in iOS Swift using both RTDB and Firestore and ran 100 sequential read operations on each. For the RTDB, I tested the observeSingleEvent and observe methods on each of the 100 top level nodes. For Firestore, I used the getDocument and addSnapshotListener methods at each of the 100 documents in the TestCol collection. I ran the tests with disk persistence on and off. Please refer to the attached image, which shows the data structure for each database.
I ran the test 10 times for each database on the same device and a stable wifi network. Existing observers and listeners were destroyed before each new run.
Realtime DB observeSingleEvent method:
func rtdbObserveSingle() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observeSingleEvent(of: .value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Realtime DB observe method:
func rtdbObserve() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observe(.value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore getDocument method:
func fsGetDocument() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).getDocument() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore addSnapshotListener method:
func fsAddSnapshotListener() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).addSnapshotListener() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Each method essentially prints the unix timestamp in milliseconds when the method starts executing and then prints another unix timestamp when each read operation returns. I took the difference between the initial timestamp and the last timestamp to return.
RESULTS - Disk persistence disabled:
RESULTS - Disk persistence enabled:
Data Structure:
When the Firestore getDocument / addSnapshotListener methods get stuck, it seems to get stuck for durations that are roughly multiples of 30 seconds. Perhaps this could help the Firebase team isolate where in the SDK it's getting stuck?
Update Date March 02, 2018
It looks like this is a known issue and the engineers at Firestore are working on a fix. After a few email exchanges and code sharing with a Firestore engineer on this issue, this was his response as of today.
"You are actually correct. Upon further checking, this slowness on getDocuments() API is a known behavior in Cloud Firestore beta. Our engineers are aware of this performance issue tagged as "cold starts", but don't worry as we are doing our best to improve Firestore query performance.
We are already working on a long-term fix but I can't share any timelines or specifics at the moment. While Firestore is still on beta, expect that there will be more improvements to come."
So hopefully this will get knocked out soon.
Using Swift / iOS
After dealing with this for about 3 days it seems the issue is definitely the get() ie .getDocuments and .getDocument. Things I thought were causing the extreme yet intermittent delays but don't appear to be the case:
Not so great network connectivity
Repeated calls via looping over .getDocument()
Chaining get() calls
Firestore Cold starting
Fetching multiple documents (Fetching 1 small doc caused 20sec delays)
Caching (I disabled offline persistence but this did nothing.)
I was able to rule all of these out as I noticed this issue didn't happen with every Firestore database call I was making. Only retrievals using get(). For kicks I replaced .getDocument with .addSnapshotListener to retrieve my data and voila. Instant retrieval each time including the first call. No cold starts. So far no issues with the .addSnapshotListener, only getDocument(s).
For now, I'm simply dropping the .getDocument() where time is of the essence and replacing it with .addSnapshotListener then using
for document in querySnapshot!.documents{
// do some magical unicorn stuff here with my document.data()
}
... in order to keep moving until this gets worked out by Firestore.
Almost 3 years later, firestore being well out of beta and I can confirm that this horrible problem still persists ;-(
On our mobile app we use the javascript / node.js firebase client. After a lot of testing to find out why our app's startup time is around 10sec we identified what to attribute 70% of that time to... Well, to firebase's and firestore's performance and cold start issues:
firebase.auth().onAuthStateChanged() fires approx. after 1.5 - 2sec, already quite bad.
If it returns a user, we use its ID to get the user document from firestore. This is the first call to firestore and the corresponding get() takes 4 - 5sec. Subsequent get() of the same or other documents take approx. 500ms.
So in total the user initialization takes 6 - 7 sec, completely unacceptable. And we can't do anything about it. We can't test disabling persistence, since in the javascript client there's no such option, persistence is always enabled by default, so not calling enablePersistence() won't change anything.
I had this issue until this morning. My Firestore query via iOS/Swift would take around 20 seconds to complete a simple, fully indexed query - with non-proportional query times for 1 item returned - all the way up to 3,000.
My solution was to disable offline data persistence. In my case, it didn't suit the needs of our Firestore database - which has large portions of its data updated every day.
iOS & Android users have this option enabled by default, whilst web users have it disabled by default. It makes Firestore seem insanely slow if you're querying a huge collection of documents. Basically it caches a copy of whichever data you're querying (and whichever collection you're querying - I believe it caches all documents within) which can lead to high Memory usage.
In my case, it caused a huge wait for every query until the device had cached the data required - hence the non-proportional query times for the increasing numbers of items to return from the exact same collection. This is because it took the same amount of time to cache the collection in each query.
Offline Data - from the Cloud Firestore Docs
I performed some benchmarking to display this effect (with offline persistence enabled) from the same queried collection, but with different amounts of items returned using the .limit parameter:
Now at 100 items returned (with offline persistence disabled), my query takes less than 1 second to complete.
My Firestore query code is below:
let db = Firestore.firestore()
self.date = Date()
let ref = db.collection("collection").whereField("Int", isEqualTo: SomeInt).order(by: "AnotherInt", descending: true).limit(to: 100)
ref.getDocuments() { (querySnapshot, err) in
if let err = err {
print("Error getting documents: \(err)")
} else {
for document in querySnapshot!.documents {
let data = document.data()
//Do things
}
print("QUERY DONE")
let currentTime = Date()
let components = Calendar.current.dateComponents([.second], from: self.date, to: currentTime)
let seconds = components.second!
print("Elapsed time for Firestore query -> \(seconds)s")
// Benchmark result
}
}
well, from what I'm currently doing and research by using nexus 5X in emulator and real android phone Huawei P8,
Firestore and Cloud Storage are both give me a headache of slow response
when I do first document.get() and first storage.getDownloadUrl()
It give me more than 60 seconds response on each request. The slow response only happen in real android phone. Not in emulator. Another strange thing.
After the first encounter, the rest request is smooth.
Here is the simple code where I meet the slow response.
var dbuserref = dbFireStore.collection('user').where('email','==',email);
const querySnapshot = await dbuserref.get();
var url = await defaultStorage.ref(document.data().image_path).getDownloadURL();
I also found link that is researching the same.
https://reformatcode.com/code/android/firestore-document-get-performance
I am attempting very simple thing and that is matching request.auth.uid to a field value in my transaction documents (like this resource.data.useruid) in Firebase security rule in order to get transactions of a particular logged in user. However, I don't get any documents while querying for them and get an error instead.
This is how the collection looks like - just one document there with useruid field.
The field's value is mapped to the users uid (screenshot taken in the Authentication -> Users tab.
And the rule looks like this
I should get the one document back but every time I query the documents with that user logged in (I am using angularfire2 for those purposes) I get Error: Missing or insufficient permissions.
If I modify the rule condition to return always true or if I only check for truthiness of request.auth.uid I get the query result alright. The funny thing though is that with resource.data involved - eg. checking for value of the amount field in the firebase rule - the condition is never met. I tried to write it like
allow read, write: if resource.data.amount == 3
and got the error again. Seems like I don't get the resource.data Map at all.
I feel like I am missing something obvious, although after reading the guides, it seems alright to me and I am already out of ideas. The debugging capabilities (or lack of) make the whole process very slow.
Could you please explain to me, why I don't get the resource.data Map in the firebase security rule or point me to a place where the problem might be?
You have most probably missed one specific point in the doc: your query fails "because it does not include the same constraints as your security rules". See https://firebase.google.com/docs/firestore/security/rules-query#secure_and_query_documents_based_on_authuid
The following, with your security rules works perfectly:
firebase.auth().signInWithEmailAndPassword("xxxx#xxxx.com", "xxxxx")
.then(function (info) {
db.collection("transactions").where("userid", "==", info.uid).get().then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
console.log(doc.id, " => ", doc.data());
});
});
});
If you remove the where clause, you get the exact error you are getting
I want to create a FireStore rule that grantes read privilages to documents after the current date has surpassed a timestamp value in the document.
This is for a blog web application.
E.G a blogger sets a blog post to be available to the public on a certain date.
From reading the documentation this should work, but It dosn't.
service cloud.firestore {
match /databases/{database}/documents {
match /Articles/{article}{
allow read: if request.time.date() < resource.data.date
}
}
}
What I am i missing ??
firebaser here
I tried the same thing a while ago, and found out it isn't currently possible.
It is possible to allow/deny read to a specific document based on a property of that document.
It is possible to allow a query that filters documents based on a property in that document, but currently that is only possible based on request.auth.
This means that unfortunately your filter currently can't be implemented with security rules. I recommend you file a feature request to chime in.
Update (2018-04-24): this might now be possible with request.time, but I haven't had a chance to test yet. Have a look here.
NOTE: As this is my first answer on Stack Overflow, I wasn't allowed to comment on Frank van Pueffelen's answer, so just as a heads-up, the credits for this solution are his!
The request has a request.time which is a timestamp, and Firestore allows for basic math operators on timestamp <> timestamp operations, so your request.time < resource.data.date will work ;)
service cloud.firestore {
match /databases/{database}/documents {
match /Articles/{article}{
allow read: if request.time < resource.data.date
}
}
}
This is based on my personal testing on 2018.09.29
trying switching the < to >.
request.time will be the time of accessing the document while resource.data.date should be the creation timestamp of the document.
try using this for your security rules:
allow read: if request.time > (resource.data.timestampPropertyName + duration.time(1, 0, 0, 0));
duration.time(4, 3, 2, 1) will create a four hour, three minute, two second, one nanosecond duration.
More information can be found at:
https://firebase.google.com/docs/firestore/reference/security/#timestamp
Do remember to wait for sometime after saving your security rules for it to take effect!
Answer of #user776914 is nice but what if there are diff timezones
Lets add +- 27 hours to be sure it was e.g. created in that day +- 1
duration.abs(request.time - request.resource.data.created) < duration.value(27, 'h')
What's max timezone offset
I wanted to do a similar thing for a game. I wanted to activate and deactivate a game object only after a particular date. I used google's cloud functions to do it. I deployed a function that runs every day to check the firestore documents and changes values according to the script.
This is more of a theoretical how database should be setup, and less about programming.
Lets say I have a news feed full of cards, which each contain a message and a like count. Each user is able to like a mesesage. I want it to be displayed to a user if they have already liked that particular card. (The same way you can see the post you like on facebook, even if you come back days later)
How would you implement that with this Firestore type database? Speed is definitely a concern..
storying it locally isn't an option, my guess would be on each card object, you would have to reference a collection that just kept a list of people who liked it. The only thing is that is a lot more querying.. which feels like it would be slow..
is there a better way to do this?
TL;DR
This approach requires more to setup, ie a cron service, knowledge of Firestore Security Rules and Cloud Functions for Firebase. With that said, the following is the best approach I've come up with. Please note, only pseudo-rules that are required are shown.
Firestore structure with some rules
/*
allow read
allow update if auth.uid == admin_uid and the
admin is updating total_likes ...
*/
messages/{message_key} : {
total_likes: <int>,
other_field:
[,...]
}
/*allow read
allow write if newData == {updated: true} and
docId exists under /messages
*/
messages_updated/{message_key} : {
updated: true
}
/*
allow read
allow create if auth.uid == liker_uid && !counted && !delete and
liker_uid/message_key match those in the docId...
allow update if auth.uid == admin_uid && the admin is
toggling counted from false -> true ...
allow update if auth.uid == liker_uid && the liker is
toggling delete ...
allow delete if auth.uid == admin_uid && delete == true and
counted == true
*/
likes/{liker_uid + '#' + message_key} : {
liker_uid:,
message_key:,
counted: <bool>,
delete: <bool>,
other_field:
[,...]
}
count_likes/{request_id}: {
message_key:,
request_time: <timestamp>
}
Functions
Function A
Triggered every X minutes to count message likes for potentially all messages.
query /messages_updated for BATCH_SIZE docs
for each, set its docId to true in a local object.
go to step 1 if BATCH_SIZE docs were retrieved (there's more to read in)
for each message_key in local object, add to /count_likes a doc w/ fields request_time and message_key.
Function B
Triggered onCreate of count_likes/{request_id}
Delete created docs message_key from /messages_updated.
let delta_likes = 0
query /likes for docs where message_key == created docs message_key and where counted == false.
for each, try to update counted to true (in parallel, not atomically)
if successful, increment delta_likes by 1.
query /likes for docs where message_key == created docs message_key, where delete == true and where counted == true.
for each doc, try to delete it (in parallel, not atomically)
if successful, decrement delta_likes by 1
if delta_likes != 0, transact the total likes for this message under
/messages by delta_likes.
delete this doc from /count_likes.
Function C (optional)
Triggered every Y minutes to delete /count_likes requests that were never met.
query docs under /count_likes that have request_time older than Z.
for each doc, delete it.
On the client
to see if you liked a message, query under /likes for a doc where liker_uid equals your uid, where message_key equals the message's key and where delete == false. if a doc exists, you have liked it.
to like a message, batch.set a like under /likes and batch.set a /messages_updated. if this batch fails, try a batch_two.update on the like by updating its delete field to false and batch_two.set its /messages_updated.
to unlike a message, batch.update on the like by updating its delete field to true and batch.set its /messages_updated.
Pros of this approach
this can be extended to counters for other things, not just messages.
a user can see if they've liked something.
a user can only like something once.
a user can spam toggle a like button and this still works.
any user can see who's liked what message by querying /likes by message_key.
any user can see all the messages any user has liked by querying /likes by liker_uid.
only a cloud function admin updates your like counts.
if a function is fired multiple times for the same event, this function is safe, meaning like counts will not be incremented multiple times for the same like.
if a function is not fired for some event, this approach still works. It just means that the count will not update until the next time someone else likes the same message.
likes are denormalized to only one root level collection, instead of the two that would be required if you had the like under the the message's likes subcollection and under the liker's messages_liked subcollection.
like counts for each message are updated in batches, ie if something has been liked 100 times, only 1 transaction of 100 is required, not 100 transactions of 1. this reduces write rate conflicts due to like counter transactions significantly.
Cons of this approach
Counts are only updated however often your cron job fires.
Relies on a cron service to fire and in general there's just more to set up.
Requires the function to authenticate with limited privileges to perform secure writes under /likes. In the Realtime Database this is possible. In Firestore, it's possible, but a bit hacky. If you can wait and don't want to use the hacky approach, use the regular unrestricted admin in development until Firestore supports authenticating with limited privileges.
May be costly depending on your standpoint. There are function invocations and read/write counts you should think about.
Things to consider
When you transact the count in Function B, you may want to consider trying this multiple times in case the max write rate of 1/sec is exceeded and the transaction fails.
In Function B, you may want to implement batch reading like in Function A if you expect to be counting a lot of likes per message.
If you need to update anything else periodically for the message (in another cron job), you may want to consider merging that function into Function B so the write rate of 1/sec isn't exceeded.