Removing all documents and collections from the Firestore - firebase

I am trying to clear a Firestore database which was filled with a lot of documents and subcollections for testing purposes. The Firebase CLI (firebase-tools#3.18.4) suggests the following possibility to delete data from Cloud Firestore:
Usage: firestore:delete [options] [path]
Options:
-r, --recursive Recursive. Delete all documents and subcollections. Any action which would result in the deletion of child documents will fail if this argument is not passed. May not be passed along with --shallow.
--shallow Shallow. Delete only parent documents and ignore documents in subcollections. Any action which would orphan documents will fail if this argument is not passed. May not be passed along with -r.
--all-collections Delete all. Deletes the entire Firestore database, including all collections and documents. Any other flags or arguments will be ignored.
-y, --yes No confirmation. Otherwise, a confirmation prompt will appear.
The problem is that it does not really work for me.
Executing firebase firestore:delete --all-collections produces the following output:
You are about to delete YOUR ENTIRE DATABASE. Are you sure? Yes
Deleting the following collections: 13OPlWrRit5PoaAbM0Rk, 17lHmJpTKVn1MBBbC169, 18LvlhhaCA1tygJYqIDt, 1DgDspzJwSEZrYxeM5G6, 1GQE7ySki4MhXxAeAzpx, 1MhoDe5JZY8Lz3yd7rVl, 1NOZ7OJeqSKl38dyh5Sw, 1Rxkjpgmr3gKvYhBJX29, 1S3mAhzQMd137Eli7qAp, 1S8FZxuefpIWBGx0hJW2, 1a7viEplYa79eNNus5xC, 1cgzMxAayzSkZv2iZf6e, 1dGjESrw6j12hEOqMpky, 1dbfgFD5teTXvQ6Ym897, 1eeYQgv2BJIS0aFWPksD, 1ehWNAZ0uKwg7mPXt3go, 1fDTkbwrXmGwZlFUl3zi, 1k5bk4aiMCuPw2KvCoAl, 1pxUSDh1YqkQAcuUH9Ie, 1rMSZ5Ru0cAfdcjY0Ljy
Deleted 92 docs (652 docs/s)
Even after executing the function multiple times an awful lot of documents and subcollections still remain in the Firestore database. Instead of deleting the ENTIRE DATABASE, only about 70-150 documents are deleted when the command is executed.
How can the entire database be deleted?

I've reported this as a bug and received the following answer:
Currently, this is an intended behavior. As stated in our documentation, deleting a collection of more than 500 documents requires multiple batched operations. So doing the iteration would be a good way to handle cases of partial deletion. I would also suggest that you check our docs regarding some of the callable function limitations for more details.
This means that firebase-tools always deletes a maximum of 500 documents in one operation. My solution to delete all collections and documents in the database is to use a while loop:
while firebase firestore:delete --all-collections --project MYPROJECT -y; do :; done
After some iterations you will see that there are no collections left and you can stop the script. Your Firestore DB is now completely empty.

You will want to use the admin sdk for this task. Use .listDocuments and .listCollections to build out simple iterations to perform your .delete operation(s) against.
If a documents .listCollections response is zero in length or null or empty, you know there's no subcollections and can iterate on / skip. Else, iterate that subcollections documents looking for deeper subcollections to delete.
let documentRef = firestore.doc('col/doc');
documentRef.listCollections().then(collections => {
for (let collection of collections) {
console.log(`Found subcollection with id: ${collection.id}`);
}
});
and
let collectionRef = firestore.collection('col');
return collectionRef.listDocuments().then(documentRefs => {
return firestore.getAll(documentRefs);
}).then(documentSnapshots => {
for (let documentSnapshot of documentSnapshots) {
if (documentSnapshot.exists) {
console.log(`Found document with data: ${documentSnapshot.id}`);
} else {
console.log(`Found missing document: ${documentSnapshot.id}`);
}
}
});

Related

Flutter Firebase local change doesn't update listener stream

I'm relying on Firebase Firestore offline capabilities, so I'm not using await on my queries as stated on the Access Data Offline Firebase doc. I'm expecting that when I write something I'll get an immediate reflection on my read stream, however, I'm only getting an update when the server/remote has been updated. Basically:
Update something in the DB. Note, I'm not using await
_db.doc(parentDoc).collection(DocInnerCollection).doc(childDoc).update({
"name": value,
});
I expect my listeners to be updated immediately. Note I've set the includeMetadataChanges to true as stated in the above doc.
_db.doc(parentDoc)
.collection(DocInnerCollection)
.orderBy('start_date', 'desc')
.limitToLast(1)
.snapshots(includeMetadataChanges: true)
.map((snapshot) {
print(snapshot.metadata.isFromCache)
});
However, I get no such update and instead I only get an update when the server has been updated.
You're requesting only one document with .limitToLast(1), yet are not providing a sort order for your query. This essentially means that you'll get a random document from your collection, and the chances of that being the newly updated document are close to zero.
If you want the latest (not just last) document, you need some ordering criteria to determine what latest means. Typically you'd do this by:
Adding a lastUpdated field to your documents, and setting that to firebase.firestore.FieldValue.serverTimestamp().
Ordering your query on that timestamp with orderBy('lastUpdated', 'desc').
And then limiting to the first result with limit(1).

What is the read count for realtime listening of cloud_firestore query and can it be done better?

Imagine I have the following pseudo code in flutter/dart :
Stream<List<T>> list() {
Query query = Firestore.instance.collection("items");
return query.snapshots().map((snapshot) {
return snapshot.documents.map((doc) {
return standardSerializers.deserializeWith(serializer, doc.data());
}).toList();
});
}
I am listening to the whole collection of "items" in my database. Let's say for simplicity there are 10 documents in total and I constantly listen for changes.
I listen to this stream somewhere in my code. Let's say this query returns all 10 "items" the first time this is called for example. This counts as 10 reads, ok fine. If I modify one of these documents directly in the firestore web interface (or elsewhere), the listener is fired and I have the impression another 10 reads are counted, even though I only modified one document. I checked in the usage tab of my cloud project and I have this suspicion.
Is this the case that 10 document reads are counted even if just one document is modified for this query?
If the answer is yes, the next question would be "Imagine I wanted to have two calls to list(), one with orderBy "rating", another with orderBy "time" (random attributes), one of these documents changes, this would mean 20 reads for 1 update"?
Either I am missing something or firestore isn't adapted for my use or I should change my architecture or I miscounted.
Is there any way to just retrieve the changed documents? (I can obviously implement a cache, local db, and timestamp system to avoid useless reads if firestore does not do this)
pubspec.yaml =>
firebase_database: ^4.0.0
firebase_auth: ^0.18.0+1
cloud_firestore: ^0.14.0+2
This probably applies to all envs like iOS and Android as it is essentially a more general "firestore" question, but example in flutter/dart as that is what I am using just in case it has something to do with the flutterfire plugin.
Thank you in advance.
Q1: Is this the case that 10 document reads are counted even if just one document is modified for this query?
No, as detailed in the documentation:
When you listen to the results of a query [Note: (or a collection or subcollection)], you are charged for a read
each time a document in the result set is added or updated. You are
also charged for a read when a document is removed from the result set
because the document has changed. (In contrast, when a document is
deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for
example, if the user goes offline), you will be charged for reads as
if you had issued a brand-new query. [Note: So 10 reads in your example.]
Q2: If the answer is yes, the next question...
The answer to Q1 is "no" :-)
Q3: Is there any way to just retrieve the changed documents?
Yes, see this part of the doc, which explains how to catch the actual changes to query results between query snapshots, instead of simply using the entire query snapshot. For Flutter you should use the docChanges property.

What is the difference between getDocuments() and snapshots() in Firestore?

I am a little confused about the difference between these two. My understanding is that getDocuments is a type of Future and seems to get the entire documents according to the query. while snapshots, on the other hand, is a type of Stream and, correct me if I'm wrong, I think it represents the results of the query? I need a more specific explanation of this issue. I will include some code snippets as an example for more clarification
getDocuments()
getUserById(String userId) async {
return await _firestore.collection("users").where("userId", isEqualTo: userId).getDocuments();
}
snapshots()
getUserById(String userId) async {
return await _firestore.collection("users").where("userId", isEqualTo: userId).snapshots();
}
So what's the difference?
When you call getDocuments(), the Firestore client gets the documents matching the query from the server once. Since this may take some time it returns a Future<QuerySnapshot>.
When you call snapshots() the Firestore client gets the documents, and then keeps watching the database on the server for changes that affect your query. So if document is written in the users collection that affects your query, your code gets called again. So this returns a stream of QuerySnapshot.
In both cases the results for the entire query are in the QuerySnapshot object.
I highly recommend reading the Firestore documentation on getting data once and on listening realtime updates. While they don't contain Flutter examples, the explanation in there applies equally to the Flutter libraries.
getDocuments():
It's used to provide data once. Cloud Firestore contains collections and inside these collections, you have documents that may contain subcollections or fields mapped to a value. To retrieve any of the doc fields to used it in widget this is used.
snapshots():
It will be called on every data change in your document query. For this StreamBuilder must be used to fetch fields as modified.
In short, it will do the job of setState() where it gives you the response for every modification so that UI can be updated.

How to keep an item counter in a collection when deleting with the Admin SDK

The function of deleting an item in firestore, returns correctly even if the item to be deleted don't exist.
If we delete a number of elements in a batch and we have a counter of elements of the collection that we want to update, in case some element that we are going to delete no longer exists, the counter would give a smaller number of the real one.
To avoid this, we use the following firestore rule:
       allow delete: if exists (/ databases / $ (database) / documents / ...);
The problem is that if we run the batch on the server, the Admin SDK ignores the firestore rules.
Any solution that does not involve transactions?
How about using a cloud function that listens for deletes?
The way this would work is you set up a listener that waits on specific collections to have a document deleted. This would be a great way to guarantee a delete leads to a function being run, and the function can do anything you'd like (e.g. increment/decrement, etc.).
This could very simply look like this:
exports.deletedItem = functions.firestore
.document('items/{itemId}')
.onDelete((snap, context) => {
const deletedValue = snap.data();
// Increment/decrement anything you'd like here.
// Perform any number of operations on a confirmed delete
});

Are changes in sub-collections are supposed to trigger onWrite function or not?

Can Firebase functions be triggered by a change in subcollections when the function is listening to a top level document? If not, what does the first line in the wildcard examples of the documentation mean?
What does this mean??? if a sub-collection changes would it trigger the function?
But it doesn't seem to and somewhere else in the same page says that too.
// Listen for changes in all documents and all sub-collections
...
exports.useWildcard = functions.firestore
.document('users/{userId}')
.onWrite((event) => {
// If we set `/users/marie` to {name: "marie"} then
event.params.userId == "marie"
// ... and ...
event.data.data() == {name: "Marie"}
});
What is the intended behavior for Firestore and function triggers?
I believe the comment in the sample code is incorrect for the sample whose document wildcard is this:
'users/{userId}'
Note that in the text below that sample it says:
If a document in users has subcollections, and a field in one of those
subcollections' documents is changed, the userId wildcard is not
triggered.
The sample below it with the following wildcards:
'users/{userId}/{messageCollectionId}/{messageId}'
will be triggered only when documents in any subcollection of documents in users are written (not the documents immediately within users).
Feel free to file a bug report for documentation or sample that you've for to be incorrect.

Resources