Recently I have migrated from firebase realtime database to firebase firestore because of the fact that it says the speed of the query depends on the size of the dataset(number of documents) being retreived from collection and not on number of documents in a collection. I checked with varying number of documents in a collection 100, 5000 and 10000 and I was retreiving 20 documents in one query. What I saw was the resulting time for the query increased when I moved from 100, 5000 and 10000 documents in a collection. Why is this happening ?
Is it because firestore is in beta ?
Querying on android (collection with 21000 docs)
collectionReference= FirebaseFirestore.getInstance().collection("_countries").document("+91").collection("_classes-exams").document(String.valueOf(mItem)).collection("_profiles").document(mOtherUserUid==null?uid:mOtherUserUid).collection("user_data");
collectionReference.document("_content").collection(articlesOrQuestions)
.orderBy("mTimeStampAsDate", Query.Direction.DESCENDING).limit(20).get().addOnCompleteListener(mCompleteListener)
.addOnFailureListener(new OnFailureListener() {
#Override
public void onFailure(#NonNull Exception e) {
Toast.makeText(getContext(), e.getMessage(), Toast.LENGTH_LONG).show();
}
});
Image of android monitor when querying the above collection reference: https://i.stack.imgur.com/QZaVX.jpg
You can see the query took almost one minute by looking at the heap(after one minute memory didn't changed much and remained constant and there is sudden spike in network section after 1 minute by which you can infer onComplete is called). What is happening between calling 'get()' function and 'onComplete' callback. This doesn't happen when querying small collections. and why the query on web is fast but slow on android ?
Link to jsbin: http://jsbin.com/fusoxoviwa/1/edit?html,css,js,console,output
Did you write these collections from the same Android client that's now loading a subset of documents from them? If so, that would explain.
The client-side cache in that case will contain information about all docs, and your app is spending time churning through that information.
If you try on a "clean client", it won't have any information cached and it should only spend time on documents that the client is requesting (or has requested before).
The behavior you're currently seeing should improve before Firestore exits beta, since the index will become more efficient, but also because it'll get some form of GC.
Related
I am currently setting up my 1st Firebase store and wondering how to best avoid unnecessary read/write costs (as well as how to create a nested structure...). Quite helpful was this answer.
However, if I listen to changes (caused by other persons) of a document, I assume I also get any change of myself again in return. So when using the logic of the todo example for bloc, I update a document. My listener recognizes this and fires an event to re-read the data from the repository.
#override
Stream<TodosState> mapEventToState(TodosEvent event) async* {
if (event is LoadTodos) {
yield* _mapLoadTodosToState();
} else if (event is TodosUpdated) {
yield* _mapTodosUpdateToState(event);
} else if (event is UpdateTodo) {
yield* _mapUpdateTodoToState(event);
}
Stream<TodosState> _mapLoadTodosToState() async* {
_todosSubscription?.cancel();
_todosSubscription = _todosRepository.todos().listen(
(todos) => add(TodosUpdated(todos)),
);
}
Stream<TodosState> _mapTodosUpdateToState(TodosUpdated event) async* {
yield TodosLoaded(event.todos);
}
Stream<TodosState> _mapUpdateTodoToState(UpdateTodo event) async* {
_todosRepository.updateTodo(event.updatedTodo);
}
Since I assume there may be multiple near time changes to a document by the same user in my app, is setting the source option to offline cache for 1min with each write access a proper option or are there better options?
And in case there isn't, can I somehow ensure that the data is sent when the user leaves the app (eg. when bringing another app upfront)?
And is there any overview how to use Firestone with Flutter? Unfortunately the coding examples in Google's documentation are for any language but Dart/Flutter. How would I, for example, set the source option with Flutter (haven't searched for it yet)?
You really can't have it both ways, you would either need to give up realtime or assume that this could generate more costs and as mentioned by #creativecreatorormaybenot you should still be using Listeners to deal with realtime updates.
The comments made in the accepted answer you shared are valid points to take better advantage of listeners with the use of cache and specially pagination, as you are very likely not required to have all of your documents in memory all the time.
Also, listeners are rather unexpensive for small projects, as you appears to be, and you have a free tier of 50.000 reads a day for Firestore, which will include read that come from listeners.
My App records food consumption in two places in the database and then discounts the inventory. When reviewing, the two records are being done correctly, but the inventory is not updating to the correct value. I would like to see a log of whether there was an error or how the inventory is being updated (I suspect that sometimes it is discounted twice), but I do not know how to do that since I cannot see the print in console (App is in Alpha tests with remote users).
Is there any way I can see or register what happens in the database (log)?
I'm using .push().set to write the two records in Firebase and I'm using ServerValue.increment to update the inventory in this way.
Future<bool> descontarAlimento(String idEmpresa, String idAlimento, double consumo) async {
try {
db.child('bodega/alimento/$idAlimento/cantidad')
.set(ServerValue.increment(-consumo.round()));
} catch (e) {
print(e);
}
return true;
}
I'm considering to add one line in the write as
db.child('logs').push().set('Inventory discounted in $consumo')
and one line following print(e) as
db.child('errors').push().set('Inventory error $e'), but, I'm not sure if it is a good idea.
What's the best way to view or record a "write operation log" in Firebase?
The Firebase Realtime Database does not keep a user-accessible log of operations, as that would become cost prohibitive quickly.
Writing a log from the client is definitely a valid option, as is adding a Cloud Function to do this replication for you.
If you can reasonably reproduce the problem, I'd recommend also running the database profiler to see if it shows any unexpected operations.
Imagine I have the following pseudo code in flutter/dart :
Stream<List<T>> list() {
Query query = Firestore.instance.collection("items");
return query.snapshots().map((snapshot) {
return snapshot.documents.map((doc) {
return standardSerializers.deserializeWith(serializer, doc.data());
}).toList();
});
}
I am listening to the whole collection of "items" in my database. Let's say for simplicity there are 10 documents in total and I constantly listen for changes.
I listen to this stream somewhere in my code. Let's say this query returns all 10 "items" the first time this is called for example. This counts as 10 reads, ok fine. If I modify one of these documents directly in the firestore web interface (or elsewhere), the listener is fired and I have the impression another 10 reads are counted, even though I only modified one document. I checked in the usage tab of my cloud project and I have this suspicion.
Is this the case that 10 document reads are counted even if just one document is modified for this query?
If the answer is yes, the next question would be "Imagine I wanted to have two calls to list(), one with orderBy "rating", another with orderBy "time" (random attributes), one of these documents changes, this would mean 20 reads for 1 update"?
Either I am missing something or firestore isn't adapted for my use or I should change my architecture or I miscounted.
Is there any way to just retrieve the changed documents? (I can obviously implement a cache, local db, and timestamp system to avoid useless reads if firestore does not do this)
pubspec.yaml =>
firebase_database: ^4.0.0
firebase_auth: ^0.18.0+1
cloud_firestore: ^0.14.0+2
This probably applies to all envs like iOS and Android as it is essentially a more general "firestore" question, but example in flutter/dart as that is what I am using just in case it has something to do with the flutterfire plugin.
Thank you in advance.
Q1: Is this the case that 10 document reads are counted even if just one document is modified for this query?
No, as detailed in the documentation:
When you listen to the results of a query [Note: (or a collection or subcollection)], you are charged for a read
each time a document in the result set is added or updated. You are
also charged for a read when a document is removed from the result set
because the document has changed. (In contrast, when a document is
deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for
example, if the user goes offline), you will be charged for reads as
if you had issued a brand-new query. [Note: So 10 reads in your example.]
Q2: If the answer is yes, the next question...
The answer to Q1 is "no" :-)
Q3: Is there any way to just retrieve the changed documents?
Yes, see this part of the doc, which explains how to catch the actual changes to query results between query snapshots, instead of simply using the entire query snapshot. For Flutter you should use the docChanges property.
Firebase: How to update an inventory with offline users?
I have an application for Agro built with Flutter and Firebase. The application works as follows:
You have a Food Inventory in the Warehouse
Users go out to feed the animals in offline mode (they are on a farm)
Then they return and the database writes are executed according to the Offline properties of Firebase
When users are online at feeding time, everything works perfectly, but when feeding in offline mode I have a problem specifically updating the inventory, since the offline user cached inventory may be different than the real inventory (either because new foods have been introduced or because of online updates from other users).
The way I am writing the data to Firebase (using the BLOC pattern) is as follows:
Future<bool> actualizarCantidad(String idAlimento, double cantidadActualizada) async {
try {
db.child('alimento/$idAlimento').update({ "cantidad": cantidadActualizada});
} catch (e) {
print(e);
}
return true;
}
The function where the inventory is read and the update of the database is ordered is the following:
Future<void> _submit() async {
_alimento = await alimentoBloc.cargarAlimento(alimentar.idAlimento);
//To read the inventory for the specific food type (alimentar.idAlimento)
final _cantAlimento = _alimento.cantidad - _consumoTotal;
//_alimento.cantidad is refered to the inventory
//_consumoTotal is the quantity to reduce (eaten food)
_alimentoBloc.actualizarCantidad(alimentar.idAlimento, _cantAlimento);
//Use the BLOC and PROVIDER pattern to Update the Inventory with a new Quantity (_cantAlimento)
}
What I would like to do in Firebase is that instead of assigning _cantAlimento quantity to Inventory, execute something like "decrease _consumoTotal of the number in Inventory" and that way it would not matter if the user is Offline or Online. Is this possible?
Another alternative that I have reviewed is to use Transactions to ensure that you are using the latest data, but transactions are lost when the user is offline so it is not a possibility.
How could I update the Inventory in the correct way, considering that my users are often offline?
Since a few months ago Firebase Realtime Database supports a ServerValue.increment() operation that can be used to atomically increment/decrement a value in the database, even when the client is not connected to the servers.
This new method also just landed in version 4.1 of the FlutterFire firebase_database plugin's ServerValue class. If you have issues with it, I'd file a bug or leave a comment on the feature request.
Based on #puf's answer, the procedure to use ServerValue.increment() to update an inventory or a number in Firebase is as follows:
db.child('alimento/$idAlimento/cantidad')
.set(ServerValue.increment(-consumo.round()));
It's important to note that ServerValue.increment() only support integers (until now, it doesn't support doubles). If you have a double number, you should round it up and you can try to use an smaller unit (grs instead Kgs in my case).
In the flutter firestore codebase you can find a comment about the stream it creates when you run snapshots() on a query.
// It's fine to let the StreamController be garbage collected once all the
// subscribers have cancelled; this analyzer warning is safe to ignore.
StreamController<QuerySnapshotPlatform> controller; // ignore: close_sinks
I want to wrap my resulting snapshot streams with a BehaviorSubject so I can keep track of the latest entry. This is useful when I have one stream that is at the top of a page that I want to be consumed through different widgets farther down in my tree without reloading the stream each time. Without keeping track in a BehaviorSubject or elsewhere if a new widget starts listening to that stream it does not get the most recent information from Firestore as it missed that event.
Can I also not worry about closing the behavior subject I am going to create as it will be garbage collected when there are no more listeners? Or is there another way to achieve what I am wanting?
I'm picturing code like this:
final snapshotStream = _firestore.collection('users').snapshots();
final behaviorSubjectStream = BehaviorSubject();
behaviorSubjectStream.addStream(snapshotStream);
return behaviorSubjectStream;
This will get a complaint that I don't close the behaviorSubjectStream. Is it ok to ignore?
That depends on how you listen to the subject.
From what you describe, it sounds safe to ignore the hint. When the subscriptions that listen to the subject are cancelled, the subject will be cancelled as well (when the garbage collector finds it).
There are situations where you have a subscription that is still listening, but you want the subject to stop emitting. In that case you will need to close() the subject.
You can test that the subject is correctly cancelled by adding
behaviorSubjectStream.onCancel = () {
print("onCancel");
};
Then you can test it by playing around with your app.