How to access deep value of realtime db with cloud function? - firebase

In childSnapshot.val().k I have this with cloud function:
{ '-LdmZIlKZh3O9cR8MOBU':
{ id: 'ceccomcpmoepincin3ipwc',
k: 'test',
p: 'somepath',
t: 1556700282278,
u: 'username' },
'-Llkocp3ojmrpemcpo3mc':
{ id: '[epc[3pc[3m,',
k: 'test2',
p: 'somepath2',
t: 1556700292290,
u: 'username2' }
}
I need each path value so I can delete that file from storage. How to access this value?
My cloud function for refreshing states, removing and deleting files from storage:
var db = admin.database();
var ref = db.ref('someref');
ref.once("value").then((snapshot) => {
var updates = {};
var patObject = {
fid: null,
ft: null,
ftr: null,
fu: null,
id: null,
lid: null,
lt: null,
ltr: null,
lu: null,
t: null,
tr: null,
v: null,
g: null,
l: null,
k: null
};
snapshot.forEach((childSnapshot) => {
if(childSnapshot.numChildren() >= 14){
var t = childSnapshot.val().t;
if((t===1 || t===5) && childSnapshot.val().tr > 0) {
if(childSnapshot.val().tr - 12 > 0){
updates[childSnapshot.key + '/tr'] = childSnapshot.val().tr - 12;
if(childSnapshot.val().k !== ""){
console.log('path: ', childSnapshot.val().k);
childSnapshot.val().k.snapshot.forEach(kpath => {
console.log('path: ', "path");
});
}
} else {
updates[childSnapshot.key] = patObject;
}
}
if(childSnapshot.val().tr<=0){
updates[childSnapshot.key] = patObject;
}
} else {
updates[childSnapshot.key] = patObject;
}
});
ref.update(updates);
res.send("");
return "";
}).catch(reason => {
res.send(reason);
})
return "";

If you want to delete all the files corresponding to the values of the ps, you need to use Promise.all() to execute in parallel the asynchronous deletion tasks (Since the delete() method returns a Promise). You need to iterate over the object that contains the p paths.
It is not easy to understand your code, so you'll find below the part corresponding to the above explanations. It's up to you to integrate it in your code!
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const defaultStorage = admin.storage(); //Note this line
//.....
exports.date = functions.https.onRequest((req, res) => { //I understand that you use an HTTP Cloud Function
//....
.then(...
// Somehow you get the object you mention in your question, through childSnapshot.val().k
const kObject = childSnapshot.val().k;
const bucket = defaultStorage.bucket(yourFileBucket);
const promises = [];
Object.keys(kObject).forEach(
//The values of the path p are obtained via kObject[key].p
//Based on that we push the Promise returned by delete() to the promises array
promises.push(bucket.file(kObject[key].p).delete());
);
return Promise.all(promises)
.then(results => {
//Here all the Promises that were in the promises array are resolved, which means that all the files are deleted
res.send({result: results.length + ' files(s) deleted'});
})
.catch(error => {
res.status(500).send(error);
});
});
Watch may be interested by watching the following official Firebase video by Doug Stevenson: https://youtu.be/7IkUgCLr5oA

Related

Firestore: remove document and collections and same time [duplicate]

How can you delete a Document with all it's collections and nested subcollections? (inside the functions environment)
In the RTDB you can ref.child('../someNode).setValue(null) and that completes the desired behavior.
I can think of two ways you could achieve the desired delete behavior, both with tremendously ghastly drawbacks.
Create a 'Super' function that will spider every document and delete them in a batch.
This function would be complicated, brittle to changes, and might take a lengthy execution time.
Add 'onDelete' triggers for each Document type, and make it delete any direct subcollections. You'll call delete on the root document, and the deletion calls will propagate down the 'tree'. This is sluggish, scales atrociously and is costly due to the colossal load of function executions.
Imagine you would have to delete a 'GROUP' and all it's children. It would be deeply chaotic with #1 and pricey with #2 (1 function call per doc)
groups > GROUP > projects > PROJECT > files > FILE > assets > ASSET
> urls > URL
> members > MEMBER
> questions > QUESTION > answers > ANSWER > replies > REPLY
> comments > COMMENT
> resources > RESOURCE > submissions > SUBMISSION
> requests > REQUEST
Is there a superior/favored/cleaner way to delete a document and all it's nested subcollections?
It ought to be possible considering you can do it from the console.
according to firebase documentation:
https://firebase.google.com/docs/firestore/solutions/delete-collections
Deleting collection with nested subcollections might be done easy and neat with node-JS on the server side.
const client = require('firebase-tools');
await client.firestore
.delete(collectionPath, {
project: process.env.GCLOUD_PROJECT,
recursive: true,
yes: true
});
Unfortunately, your analysis is spot on and indeed this use case does require a lot of ceremony. According to official documentation, there is no support for deep deletes in a single shot in firestore neither via client libraries nor rest-api nor the cli tool.
The cli is open sourced and its implementation lives here: https://github.com/firebase/firebase-tools/blob/master/src/firestore/delete.js. They basically implemented option 1. you described in your question, so you can take some inspiration from there.
Both options 1. and 2. are far from ideal situation and to make your solution 100% reliable you will need to keep a persistent queue with deletion tasks, as any error in the long running procedure will leave your system in some ill-defined state.
I would discourage to go with raw option 2. as recursive cloud function calls may very easily went wrong - for example, hitting max. limits.
In case the link changed, below the full source of https://github.com/firebase/firebase-tools/blob/master/src/firestore/delete.js:
"use strict";
var clc = require("cli-color");
var ProgressBar = require("progress");
var api = require("../api");
var firestore = require("../gcp/firestore");
var FirebaseError = require("../error");
var logger = require("../logger");
var utils = require("../utils");
/**
* Construct a new Firestore delete operation.
*
* #constructor
* #param {string} project the Firestore project ID.
* #param {string} path path to a document or collection.
* #param {boolean} options.recursive true if the delete should be recursive.
* #param {boolean} options.shallow true if the delete should be shallow (non-recursive).
* #param {boolean} options.allCollections true if the delete should universally remove all collections and docs.
*/
function FirestoreDelete(project, path, options) {
this.project = project;
this.path = path;
this.recursive = Boolean(options.recursive);
this.shallow = Boolean(options.shallow);
this.allCollections = Boolean(options.allCollections);
// Remove any leading or trailing slashes from the path
if (this.path) {
this.path = this.path.replace(/(^\/+|\/+$)/g, "");
}
this.isDocumentPath = this._isDocumentPath(this.path);
this.isCollectionPath = this._isCollectionPath(this.path);
this.allDescendants = this.recursive;
this.parent = "projects/" + project + "/databases/(default)/documents";
// When --all-collections is passed any other flags or arguments are ignored
if (!options.allCollections) {
this._validateOptions();
}
}
/**
* Validate all options, throwing an exception for any fatal errors.
*/
FirestoreDelete.prototype._validateOptions = function() {
if (this.recursive && this.shallow) {
throw new FirebaseError("Cannot pass recursive and shallow options together.");
}
if (this.isCollectionPath && !this.recursive && !this.shallow) {
throw new FirebaseError("Must pass recursive or shallow option when deleting a collection.");
}
var pieces = this.path.split("/");
if (pieces.length === 0) {
throw new FirebaseError("Path length must be greater than zero.");
}
var hasEmptySegment = pieces.some(function(piece) {
return piece.length === 0;
});
if (hasEmptySegment) {
throw new FirebaseError("Path must not have any empty segments.");
}
};
/**
* Determine if a path points to a document.
*
* #param {string} path a path to a Firestore document or collection.
* #return {boolean} true if the path points to a document, false
* if it points to a collection.
*/
FirestoreDelete.prototype._isDocumentPath = function(path) {
if (!path) {
return false;
}
var pieces = path.split("/");
return pieces.length % 2 === 0;
};
/**
* Determine if a path points to a collection.
*
* #param {string} path a path to a Firestore document or collection.
* #return {boolean} true if the path points to a collection, false
* if it points to a document.
*/
FirestoreDelete.prototype._isCollectionPath = function(path) {
if (!path) {
return false;
}
return !this._isDocumentPath(path);
};
/**
* Construct a StructuredQuery to find descendant documents of a collection.
*
* See:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/StructuredQuery
*
* #param {boolean} allDescendants true if subcollections should be included.
* #param {number} batchSize maximum number of documents to target (limit).
* #param {string=} startAfter document name to start after (optional).
* #return {object} a StructuredQuery.
*/
FirestoreDelete.prototype._collectionDescendantsQuery = function(
allDescendants,
batchSize,
startAfter
) {
var nullChar = String.fromCharCode(0);
var startAt = this.parent + "/" + this.path + "/" + nullChar;
var endAt = this.parent + "/" + this.path + nullChar + "/" + nullChar;
var where = {
compositeFilter: {
op: "AND",
filters: [
{
fieldFilter: {
field: {
fieldPath: "__name__",
},
op: "GREATER_THAN_OR_EQUAL",
value: {
referenceValue: startAt,
},
},
},
{
fieldFilter: {
field: {
fieldPath: "__name__",
},
op: "LESS_THAN",
value: {
referenceValue: endAt,
},
},
},
],
},
};
var query = {
structuredQuery: {
where: where,
limit: batchSize,
from: [
{
allDescendants: allDescendants,
},
],
select: {
fields: [{ fieldPath: "__name__" }],
},
orderBy: [{ field: { fieldPath: "__name__" } }],
},
};
if (startAfter) {
query.structuredQuery.startAt = {
values: [{ referenceValue: startAfter }],
before: false,
};
}
return query;
};
/**
* Construct a StructuredQuery to find descendant documents of a document.
* The document itself will not be included
* among the results.
*
* See:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/StructuredQuery
*
* #param {boolean} allDescendants true if subcollections should be included.
* #param {number} batchSize maximum number of documents to target (limit).
* #param {string=} startAfter document name to start after (optional).
* #return {object} a StructuredQuery.
*/
FirestoreDelete.prototype._docDescendantsQuery = function(allDescendants, batchSize, startAfter) {
var query = {
structuredQuery: {
limit: batchSize,
from: [
{
allDescendants: allDescendants,
},
],
select: {
fields: [{ fieldPath: "__name__" }],
},
orderBy: [{ field: { fieldPath: "__name__" } }],
},
};
if (startAfter) {
query.structuredQuery.startAt = {
values: [{ referenceValue: startAfter }],
before: false,
};
}
return query;
};
/**
* Query for a batch of 'descendants' of a given path.
*
* For document format see:
* https://firebase.google.com/docs/firestore/reference/rest/v1beta1/Document
*
* #param {boolean} allDescendants true if subcollections should be included,
* #param {number} batchSize the maximum size of the batch.
* #param {string=} startAfter the name of the document to start after (optional).
* #return {Promise<object[]>} a promise for an array of documents.
*/
FirestoreDelete.prototype._getDescendantBatch = function(allDescendants, batchSize, startAfter) {
var url;
var body;
if (this.isDocumentPath) {
url = this.parent + "/" + this.path + ":runQuery";
body = this._docDescendantsQuery(allDescendants, batchSize, startAfter);
} else {
url = this.parent + ":runQuery";
body = this._collectionDescendantsQuery(allDescendants, batchSize, startAfter);
}
return api
.request("POST", "/v1beta1/" + url, {
auth: true,
data: body,
origin: api.firestoreOrigin,
})
.then(function(res) {
// Return the 'document' property for each element in the response,
// where it exists.
return res.body
.filter(function(x) {
return x.document;
})
.map(function(x) {
return x.document;
});
});
};
/**
* Progress bar shared by the class.
*/
FirestoreDelete.progressBar = new ProgressBar("Deleted :current docs (:rate docs/s)", {
total: Number.MAX_SAFE_INTEGER,
});
/**
* Repeatedly query for descendants of a path and delete them in batches
* until no documents remain.
*
* #return {Promise} a promise for the entire operation.
*/
FirestoreDelete.prototype._recursiveBatchDelete = function() {
var self = this;
// Tunable deletion parameters
var readBatchSize = 7500;
var deleteBatchSize = 250;
var maxPendingDeletes = 15;
var maxQueueSize = deleteBatchSize * maxPendingDeletes * 2;
// All temporary variables for the deletion queue.
var queue = [];
var numPendingDeletes = 0;
var pagesRemaining = true;
var pageIncoming = false;
var lastDocName;
var failures = [];
var retried = {};
var queueLoop = function() {
if (queue.length == 0 && numPendingDeletes == 0 && !pagesRemaining) {
return true;
}
if (failures.length > 0) {
logger.debug("Found " + failures.length + " failed deletes, failing.");
return true;
}
if (queue.length <= maxQueueSize && pagesRemaining && !pageIncoming) {
pageIncoming = true;
self
._getDescendantBatch(self.allDescendants, readBatchSize, lastDocName)
.then(function(docs) {
pageIncoming = false;
if (docs.length == 0) {
pagesRemaining = false;
return;
}
queue = queue.concat(docs);
lastDocName = docs[docs.length - 1].name;
})
.catch(function(e) {
logger.debug("Failed to fetch page after " + lastDocName, e);
pageIncoming = false;
});
}
if (numPendingDeletes > maxPendingDeletes) {
return false;
}
if (queue.length == 0) {
return false;
}
var toDelete = [];
var numToDelete = Math.min(deleteBatchSize, queue.length);
for (var i = 0; i < numToDelete; i++) {
toDelete.push(queue.shift());
}
numPendingDeletes++;
firestore
.deleteDocuments(self.project, toDelete)
.then(function(numDeleted) {
FirestoreDelete.progressBar.tick(numDeleted);
numPendingDeletes--;
})
.catch(function(e) {
// For server errors, retry if the document has not yet been retried.
if (e.status >= 500 && e.status < 600) {
logger.debug("Server error deleting doc batch", e);
// Retry each doc up to one time
toDelete.forEach(function(doc) {
if (retried[doc.name]) {
logger.debug("Failed to delete doc " + doc.name + " multiple times.");
failures.push(doc.name);
} else {
retried[doc.name] = true;
queue.push(doc);
}
});
} else {
logger.debug("Fatal error deleting docs ", e);
failures = failures.concat(toDelete);
}
numPendingDeletes--;
});
return false;
};
return new Promise(function(resolve, reject) {
var intervalId = setInterval(function() {
if (queueLoop()) {
clearInterval(intervalId);
if (failures.length == 0) {
resolve();
} else {
reject("Failed to delete documents " + failures);
}
}
}, 0);
});
};
/**
* Delete everything under a given path. If the path represents
* a document the document is deleted and then all descendants
* are deleted.
*
* #return {Promise} a promise for the entire operation.
*/
FirestoreDelete.prototype._deletePath = function() {
var self = this;
var initialDelete;
if (this.isDocumentPath) {
var doc = { name: this.parent + "/" + this.path };
initialDelete = firestore.deleteDocument(doc).catch(function(err) {
logger.debug("deletePath:initialDelete:error", err);
if (self.allDescendants) {
// On a recursive delete, we are insensitive to
// failures of the initial delete
return Promise.resolve();
}
// For a shallow delete, this error is fatal.
return utils.reject("Unable to delete " + clc.cyan(this.path));
});
} else {
initialDelete = Promise.resolve();
}
return initialDelete.then(function() {
return self._recursiveBatchDelete();
});
};
/**
* Delete an entire database by finding and deleting each collection.
*
* #return {Promise} a promise for all of the operations combined.
*/
FirestoreDelete.prototype.deleteDatabase = function() {
var self = this;
return firestore
.listCollectionIds(this.project)
.catch(function(err) {
logger.debug("deleteDatabase:listCollectionIds:error", err);
return utils.reject("Unable to list collection IDs");
})
.then(function(collectionIds) {
var promises = [];
logger.info("Deleting the following collections: " + clc.cyan(collectionIds.join(", ")));
for (var i = 0; i < collectionIds.length; i++) {
var collectionId = collectionIds[i];
var deleteOp = new FirestoreDelete(self.project, collectionId, {
recursive: true,
});
promises.push(deleteOp.execute());
}
return Promise.all(promises);
});
};
/**
* Check if a path has any children. Useful for determining
* if deleting a path will affect more than one document.
*
* #return {Promise<boolean>} a promise that retruns true if the path has
* children and false otherwise.
*/
FirestoreDelete.prototype.checkHasChildren = function() {
return this._getDescendantBatch(true, 1).then(function(docs) {
return docs.length > 0;
});
};
/**
* Run the delete operation.
*/
FirestoreDelete.prototype.execute = function() {
var verifyRecurseSafe;
if (this.isDocumentPath && !this.recursive && !this.shallow) {
verifyRecurseSafe = this.checkHasChildren().then(function(multiple) {
if (multiple) {
return utils.reject("Document has children, must specify -r or --shallow.", { exit: 1 });
}
});
} else {
verifyRecurseSafe = Promise.resolve();
}
var self = this;
return verifyRecurseSafe.then(function() {
return self._deletePath();
});
};
module.exports = FirestoreDelete;
For those who don't want or can't use cloud functions, I found a recursiveDelete function in the admin sdk:
https://googleapis.dev/nodejs/firestore/latest/Firestore.html#recursiveDelete
// Recursively delete a reference and log the references of failures.
const bulkWriter = firestore.bulkWriter();
bulkWriter
.onWriteError((error) => {
if (error.failedAttempts < MAX_RETRY_ATTEMPTS) {
return true;
} else {
console.log('Failed write at document: ', error.documentRef.path);
return false;
}
});
await firestore.recursiveDelete(docRef, bulkWriter);
i don't know how much helpful for you but test it and compare the execution time which i get use it from fire store doc
/** Delete a collection in batches to avoid out-of-memory errors.
* Batch size may be tuned based on document size (atmost 1MB) and application requirements.
*/
void deleteCollection(CollectionReference collection, int batchSize) {
try {
// retrieve a small batch of documents to avoid out-of-memory errors
ApiFuture<QuerySnapshot> future = collection.limit(batchSize).get();
int deleted = 0;
// future.get() blocks on document retrieval
List<QueryDocumentSnapshot> documents = future.get().getDocuments();
for (QueryDocumentSnapshot document : documents) {
document.getReference().delete();
++deleted;
}
if (deleted >= batchSize) {
// retrieve and delete another batch
deleteCollection(collection, batchSize);
}
} catch (Exception e) {
System.err.println("Error deleting collection : " + e.getMessage());
}
}
As mentioned above, you need to write good bit of code for this. For each document that is to be deleted you need to check if it has one or more collections. If it does, then you need to queue those up for deletion too. I wrote the code below to do this. It's not tested to be scalable to large data sets, which is fine for me as I'm using it to clean up after small scale integration tests. If you need something more scalable, feel free to take this as a starting point and play around with batching more.
class FirebaseDeleter {
constructor(database, collections) {
this._database = database;
this._pendingCollections = [];
}
run() {
return new Promise((resolve, reject) => {
this._callback = resolve;
this._database.getCollections().then(collections => {
this._pendingCollections = collections;
this._processNext();
});
});
}
_processNext() {
const collections = this._pendingCollections;
this._pendingCollections = [];
const promises = collections.map(collection => {
return this.deleteCollection(collection, 10000);
});
Promise.all(promises).then(() => {
if (this._pendingCollections.length == 0) {
this._callback();
} else {
process.nextTick(() => {
this._processNext();
});
}
});
}
deleteCollection(collectionRef, batchSize) {
var query = collectionRef;
return new Promise((resolve, reject) => {
this.deleteQueryBatch(query, batchSize, resolve, reject);
});
}
deleteQueryBatch(query, batchSize, resolve, reject) {
query
.get()
.then(snapshot => {
// When there are no documents left, we are done
if (snapshot.size == 0) {
return 0;
}
// Delete documents in a batch
var batch = this._database.batch();
const collectionPromises = [];
snapshot.docs.forEach(doc => {
collectionPromises.push(
doc.ref.getCollections().then(collections => {
collections.forEach(collection => {
this._pendingCollections.push(collection);
});
})
);
batch.delete(doc.ref);
});
// Wait until we know if all the documents have collections before deleting them.
return Promise.all(collectionPromises).then(() => {
return batch.commit().then(() => {
return snapshot.size;
});
});
})
.then(numDeleted => {
if (numDeleted === 0) {
resolve();
return;
}
// Recurse on the next process tick, to avoid
// exploding the stack.
process.nextTick(() => {
this.deleteQueryBatch(query, batchSize, resolve, reject);
});
})
.catch(reject);
}
}
Solution using Node.js Admin SDK
export const deleteDocument = async (doc: FirebaseFirestore.DocumentReference) => {
const collections = await doc.listCollections()
await Promise.all(collections.map(collection => deleteCollection(collection)))
await doc.delete()
}
export const deleteCollection = async (collection: FirebaseFirestore.CollectionReference) => {
const query = collection.limit(100)
while (true) {
const snap = await query.get()
if (snap.empty) {
return
}
await Promise.all(snap.docs.map(doc => deleteDocument(doc.ref)))
}
}
There is now a simple way delete a document and all of its subcollections using NodeJS.
This was made available in nodejs-firestore version v4.11.0.
From the docs:
recursiveDelete()
Recursively deletes all documents and subcollections at and under the specified level.
import * as admin from 'firebase-admin'
const ref = admin.firestore().doc('my_document')
admin.firestore().recursiveDelete(ref)
You can write a handler which will recursive delete all nested descendants when triggers onDelete Firestore event.
Example of handler:
const deleteDocumentWithDescendants = async (documentSnap: FirebaseFirestore.QueryDocumentSnapshot) => {
return documentSnap.ref.listCollections().then((subCollections) => {
subCollections.forEach((subCollection) => {
return subCollection.get().then((snap) => {
snap.forEach((doc) => {
doc.ref.delete();
deleteDocumentWithDescendants(doc);
});
});
});
});
};
// On any document delete
export const onDocumentDelete = async (documentSnap: FirebaseFirestore.QueryDocumentSnapshot) => {
await deleteDocumentWithDescendants(documentSnap);
};
Tie it up with firestore event:
exports.onDeleteDocument = functions.firestore.document('{collectionId}/{docId}')
.onDelete(onDocumentDelete);
// You can add all the collection hierarchy to object
private collectionsHierarchy = {
groups: [
[
'groups',
'projects',
'files',
'assets',
'urls',
'members'
]
]
};
async deleteDocument(rootDocument: string) {
// if (!rootDocument.startsWith(`groups/${this.groupId()}`)) {
// rootDocument = `groups/${this.groupId()}/${rootDocument}`;
// }
const batchSize: number = 100;
let root = await this.db
.doc(rootDocument)
.get()
.toPromise();
if (!root.exists) {
return;
}
const segments = rootDocument.split('/');
const documentCollection = segments[segments.length - 2];
const allHierarchies = this.collectionsHierarchy[documentCollection];
for (let i = 0; i < allHierarchies.length; i = i + 1) {
const hierarchy = allHierarchies[i];
const collectionIndex = hierarchy.indexOf(documentCollection) + 1;
const nextCollections: [] = hierarchy.slice(collectionIndex);
const stack = [`${root.ref.path}/${nextCollections.shift()}`];
while (stack.length) {
const path = stack.pop();
const collectionRef = this.db.firestore.collection(path);
const query = collectionRef.orderBy('__name__').limit(batchSize);
let deletedIems = await this.deleteQueryBatch(query, batchSize);
const nextCollection = nextCollections.shift();
deletedIems = deletedIems.map(di => `${di}/${nextCollection}`);
stack.push(...deletedIems);
}
}
await root.ref.delete();
}
private async deleteQueryBatch(
query: firebase.firestore.Query,
batchSize: number
) {
let deletedItems: string[] = [];
let snapshot = await query.get();
if (snapshot.size === 0) {
return deletedItems;
}
const batch = this.db.firestore.batch();
snapshot.docs.forEach(doc => {
deletedItems.push(doc.ref.path);
batch.delete(doc.ref);
});
await batch.commit();
if (snapshot.size === 0) {
return deletedItems;
}
const result = await this.deleteQueryBatch(query, batchSize);
return [...deletedItems, ...result];
}
Another solution using Node.js Admin SDK with Batch.
const traverseDocumentRecursively = async (
docRef: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>,
accumulatedRefs: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>[],
) => {
const collections = await docRef.listCollections();
if (collections.length > 0) {
for (const collection of collections) {
const snapshot = await collection.get();
for (const doc of snapshot.docs) {
accumulatedRefs.push(doc.ref);
await traverseDocumentRecursively(doc.ref, accumulatedRefs);
}
}
}
};
import { chunk } from 'lodash';
const doc = admin.firestore().collection('users').doc('001');
const accumulatedRefs: FirebaseFirestore.DocumentReference<FirebaseFirestore.DocumentData>[] = [];
await traverseDocumentRecursively(doc, accumulatedRefs);
await Promise.all(
// Each transaction or batch of writes can write to a maximum of 500 documents
chunk(accumulatedRefs, 500).map((chunkedRefs) => {
const batch = admin.firestore().batch();
for (const ref of chunkedRefs) {
batch.delete(ref);
}
return batch.commit();
}),
);
Not sure if this is helpful for anyone here, but I am frequently facing the error "Fatal error deleting docs <list of docs>" when using firebase-tools.firestore.delete method (firebase-tools version 9.22.0).
I am currently handling these deletion failures using the returned error message in order to avoid rewriting the code cited at Oleg Bondarenko's answer. It uses admin.firestore to effectively delete the failed docs.
It's a poor solution since it relies on the error message, but at least it doesn't force us to copy the whole FirestoreDelete code to modify just a few lines of it:
firebase_tools.firestore
.delete(path, {
project: JSON.parse(process.env.FIREBASE_CONFIG!).projectId,
recursive: true,
yes: true,
token: getToken(),
})
.catch((err: Error) => {
if (err.name == "FirebaseError") {
// If recursive delete fails to delete some of the documents,
// parse the failures from the error message and delete it manually
const failedDeletingDocs = err.message.match(
/.*Fatal error deleting docs ([^\.]+)/
);
if (failedDeletingDocs) {
const docs = failedDeletingDocs[1].split(", ");
const docRefs = docs.map((doc) =>
firestore.doc(doc.slice(doc.search(/\(default\)\/documents/) + 19))
);
firestore
.runTransaction(async (t) => {
docRefs.forEach((doc) => t.delete(doc));
return docs;
})
.then((docs) =>
console.log(
"Succesfully deleted docs after failing: " + docs.join(", ")
)
)
.catch((err) => console.error(err));
}
}
});
If you are looking to delete user data, a solution to consider in 2022 is the Delete User Data Firebase Extension.
Once this is active, you can simply delete the user from Firebase Auth to trigger the recursive deletion of the user documents:
import admin from "firebase-admin";
admin.auth().deleteUser(userId);
You can call firebase.firestore().doc("whatever").set() and that will delete everything in that document.
The only way .set does not erase everything is if you set the merge flag to true.
See Firestore Documentation on Add Data
var cityRef = db.collection('cities').doc('BJ');
var setWithMerge = cityRef.set({
capital: true
}, { merge: true });

$match aggregate in mongoose

I am using $match to find a userid that is reference in my schema. When I pass in http://localhost:3001/api/orders/5d6f1611d3bcd442bc273211/2018-02-01/2019-09-01 I get the following:
5d6f1611d3bcd442bc273211
[0] 2018-02-01
[0] 2019-09-01
[0] ReferenceError: $match is not defined
Here is my code:
getOrderTotal: function (req, res) {
const userID = req.params.userid;
const firstDate = req.params.firstdate;
const secondDate = req.params.seconddate;
console.log(userID);
console.log(firstDate);
console.log(secondDate);
Order
.aggregate([{ $match: { 'user': userID } }])
.find({ "created_at": { "$gte": firstDate, "$lt": secondDate } })
.then(dbModel => res.json(dbModel))
.catch(err => res.status(422).json(err));
}
When I remove the line containing $match, I get the expected results. When I leave it in, I get an empty array.
convert userId to ObjectID
var mongoose = require('mongoose');
var id = mongoose.Types.ObjectId(userID);
Then use id in your $match aggregation
or
.aggregate([{ $match: { 'user': mongoose.Types.ObjectId(userID) } }])
You can use $find and $match at one time using expr.
const userID = req.params.userid;
const firstDate = req.params.firstdate;
const secondDate = req.params.seconddate;
console.log(userID);
console.log(firstDate);
console.log(secondDate);
Order.find({
"$expr": {
"$and": [
{ "$eq": ["$user", userID] },
{ "$gte": [{ "$created_at": firstDate }]},
{ "$lt": [{ "$created_at": secondDate }]}
]
}
}).exec((err , found)=>{
});
I am guessing this is what you want to achieve
Find document that has user matching userId
and has date greater than or equal to firstDate and less than secondDate
If you insist on using the aggregate pipeline plus $match operator, you must then be aware of the following
Note:
The documents returned are plain javascript objects, not mongoose documents (since any shape of document can be returned). More info here: https://mongoosejs.com/docs/api/aggregate.html
Example:
getOrderByDate: async (req, res) => {
// Get the parameter data from req.params
let { user, firstDate, secondDate } = req.params;
// Convert the user data type to mongoose ObjectId
user = mongoose.Types.ObjectId(user);
// Convert both firstDate and SecondDate to date
firstDate = new Date(firstDate);
firstDate.setHours(0, 0, 0, 0);
secondDate = new Date(secondDate);
secondDate.setHours(23, 59, 59, 999);
// See the output in your console if you will...
console.log({ user, firstDate, secondDate });
// Using async (you can still use promises if you will)
try {
const matchedDocuments = await OrderDocument.aggregate([{ $match: { user: user } }]);
// Filter returned document
const result = matchedDocuments.filter((docs) => {
if (docs.created_at >= firstDate && docs.created_at < secondDate) {
return true;
}
});
res.status(200).json({ result });
} catch (error) {
res.status(500).json({ error, message: 'Something went wrong' });
}
}

alexa sdk: can't get persitentAttributes

i'm trying to add persistent attributes to my lambda function.
i created a dynamoDB table and added it to the triggers of my lambda function.
i copied a sample code from github, but when i try to launch the skill i get an error. The console log shows:
{
"errorMessage": "Could not read item (amzn1.ask.account.AGIIYNRXWDLBD6XEPW72QS2BHGXNP7NWYBEWSH2XLSXZP64X3NCYEMVK233VFDWH77ZB6DAK6YJ53SZLNUFVQ56CYOVCILS7QFZI4CIRDWC3PAHS4QG27YUY5PTT6QEIK46YFNTJT54YAKNGOWV2UO66XZACFDQ5SEXKJYOBNFNIZNUXKNTIAAYZG4R5ZU4FMLPDZZN64KLINNA) from table (Spiele): The provided key element does not match the schema",
"errorType": "AskSdk.DynamoDbPersistenceAdapter Error",
"stackTrace": [
"Object.createAskSdkError (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/utils/AskSdkUtils.js:22:17)",
"DynamoDbPersistenceAdapter.<anonymous> (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:123:49)",
"step (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:44:23)",
"Object.throw (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:25:53)",
"rejected (/var/task/node_modules/ask-sdk-dynamodb-persistence-adapter/lib/attributes/persistence/DynamoDbPersistenceAdapter.js:17:65)",
"<anonymous>",
"process._tickDomainCallback (internal/process/next_tick.js:228:7)"
]
}
the table contains a primary key "name" and sort key "UserId". is that wrong?
here is my index.js:
const Alexa = require('ask-sdk');
// Define the skill features
let skill;
/**
* If this is the first start of the skill, grab the user's data from Dynamo and
* set the session attributes to the persistent data.
*/
const GetUserDataInterceptor = {
process(handlerInput) {
let attributes = handlerInput.attributesManager.getSessionAttributes();
if (handlerInput.requestEnvelope.request.type === 'LaunchRequest' && !attributes['isInitialized']) {
return new Promise((resolve, reject) => {
handlerInput.attributesManager.getPersistentAttributes()
.then((attributes) => {
attributes['isInitialized'] = true;
saveUser(handlerInput, attributes, 'session');
resolve();
})
.catch((error) => {
reject(error);
})
});
}
}
};
function saveUser(handlerInput, attributes, mode) {
if(mode === 'session'){
handlerInput.attributesManager.setSessionAttributes(attributes);
} else if(mode === 'persistent') {
console.info("Saving to Dynamo: ",attributes);
return new Promise((resolve, reject) => {
handlerInput.attributesManager.getPersistentAttributes()
.then((persistent) => {
delete attributes['isInitialized'];
handlerInput.attributesManager.setPersistentAttributes(attributes);
resolve(handlerInput.attributesManager.savePersistentAttributes());
})
.catch((error) => {
reject(error);
});
});
}
}
const LaunchHandler = {
canHandle(handlerInput) {
return handlerInput.requestEnvelope.request.type === 'LaunchRequest';
},
handle(handlerInput) {
console.info("LaunchRequest");
let attributes = handlerInput.attributesManager.getSessionAttributes();
console.info("Test the load: " + attributes['isInitialized']);
attributes['FOO'] = "BAR";
saveUser(handlerInput, attributes, 'persistent');
return handlerInput.responseBuilder
.speak('Hello')
.reprompt('Hello')
.getResponse();
}
}
exports.handler = Alexa.SkillBuilders.standard()
.addRequestHandlers(
LaunchHandler
)
.addRequestInterceptors(GetUserDataInterceptor)
.withTableName('Spiele')
.withAutoCreateTable(true)
.withDynamoDbClient()
.lambda();
can anyone tell me what i'm doing wrong?
please confirm the partition key is 'userId' not 'UserId' (notice the uppercase U).
Also I would suggest using 'this' object.
Let me know if that helps.
Cheers
Below code is for python lambda function
from ask_sdk_core.skill_builder import CustomSkillBuilder
from ask_sdk_dynamodb.adapter import DynamoDbAdapter
sb = SkillBuilder()
sb = CustomSkillBuilder(persistence_adapter = dynamodb_adapter)

How to dynamically update an attribute in a dynamodb item?

I created an item in dynamodb using Node js, the item has multiple attributes such as brand, category, discount, validity, etc. I am using uuid to generate ids for each item. Now let's say I want to update the validity attribute of the item, in which case I am currently sending the entire json object with the value of validity modified to the new value.
This is definitely not optimal, please help me find an optimal solution.
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: {
'#discount': 'discount',
},
ExpressionAttributeValues: {
':brand': data.brand,
':category': data.category,
':discount': data.discount,
':denominations': data.denominations,
":validity": data.validity,
":redemption": data.redemption
},
UpdateExpression: 'SET #discount = :discount, denominations = :denominations, brand = :brand, category = :category, validity = :validity, redemption = :redemption',
ReturnValues: 'ALL_NEW',
};
I want to send just the attribute I want to update with the new value, if I want to change the validity from 6 months to 8 months, I should just send something like:
{
"validity": "8 months"
}
And it should update the validity attribute of the item.
Same should apply to any other attribute of the item.
'use strict';
const AWS = require('aws-sdk');
const dynamoDb = new AWS.DynamoDB.DocumentClient();
module.exports.update = (event, context, callback) => {
const data = JSON.parse(event.body);
let attr = {};
let nameobj = {};
let exp = 'SET #';
let arr = Object.keys(data);
let attrname = {};
arr.map((key) => {attr[`:${key}`]=data[key]});
arr.map((key) => {
exp += `${key} = :${key}, `
});
arr.map((key) => {nameobj[`#${key}`]=data[key]});
attrname = {
[Object.keys(nameobj)[0]] : nameobj[Object.keys(nameobj)[0]]
}
const params = {
TableName: process.env.PRODUCT_TABLE,
Key: {
id: event.pathParameters.id,
},
ExpressionAttributeNames: attrname,
ExpressionAttributeValues: attr,
UpdateExpression: exp,
ReturnValues: 'ALL_NEW',
};
// update the todo in the database
dynamoDb.update(params, (error, result) => {
// handle potential errors
if (error) {
console.error(error);
callback(null, {
statusCode: error.statusCode || 501,
headers: { 'Content-Type': 'text/plain' },
body: 'Couldn\'t update the card',
});
return;
}
// create a response
const response = {
statusCode: 200,
body: JSON.stringify(result.Attributes),
};
callback(null, response);
});
};
Contrary to others comments, this is very possible, use the UpdateItem action.
Language agnostic API docs
JavaScript specific API docs
If you want to dynamically create the query, try something like this:
const generateUpdateQuery = (fields) => {
let exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(fields).forEach(([key, item]) => {
exp.UpdateExpression += ` #${key} = :${key},`;
exp.ExpressionAttributeNames[`#${key}`] = key;
exp.ExpressionAttributeValues[`:${key}`] = item
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1);
return exp
}
let data = {
'field' : { 'subfield': 123 },
'other': '456'
}
let expression = generateUpdateQuery(data)
let params = {
// Key, Table, etc..
...expression
}
console.log(params)
Output:
{
UpdateExpression: 'set #field = :field, #other = :other',
ExpressionAttributeNames: {
'#field': 'field',
'#other': 'other'
},
ExpressionAttributeValues: {
':field': {
'subfield': 123
},
':other': '456'
}
}
Using Javascript SDK V3:
Import from the right package:
import { DynamoDBClient PutItemCommandInput, UpdateItemCommandInput, UpdateItemCommand } from '#aws-sdk/client-dynamodb';
Function to dynamically do partial updates to the item:
(the code below is typescript can be easily converted to Javascript, just remove the types!)
function updateItem(id: string, item: any) {
const dbClient = new DynamoDBClient({region: 'your-region-here });
let exp = 'set ';
let attNames: any = { };
let attVal: any = { };
for(const attribute in item) {
const valKey = `:${attribute}`;
attNames[`#${attribute}`] = attribute;
exp += `#${attribute} = ${valKey}, `;
const val = item[attribute];
attVal[valKey] = { [getDynamoType(val)]: val };
}
exp = exp.substring(0, exp.length - 2);
const params: UpdateItemCommandInput = {
TableName: 'your-table-name-here',
Key: { id: { S: id } },
UpdateExpression: exp,
ExpressionAttributeValues: attVal,
ExpressionAttributeNames: attNames,
ReturnValues: 'ALL_NEW',
};
try {
console.debug('writing to db: ', params);
const command = new UpdateItemCommand(params);
const res = await dbClient.send(command);
console.debug('db res: ', res);
return true;
} catch (err) {
console.error('error writing to dynamoDB: ', err);
return false;
}
}
And to use it (we can do partial updates as well):
updateItem('some-unique-id', { name: 'some-attributes' });
What i did is create a helper class.
Here is a simple function : Add all the attribute and values that goes into, if the value is null or undefined it won't be in the expression.
I recommande to create a helper class with typescript and add more functions and other stuff like generator of expressionAttributeValues , expressionAttributeNames ... , Hope this help.
function updateExpression(attributes, values) {
const expression = attributes.reduce((res, attribute, index) => {
if (values[index]) {
res += ` #${attribute}=:${attribute},`;
}
return res;
}, "SET ");
return expression.slice(0, expression.length - 1)
}
console.log(
updateExpression(["id", "age", "power"], ["e8a8da9a-fab0-55ba-bae3-6392e1ebf624", 28, undefined])
);
You can use code and generate the params object based on the object you provide. It's just a JavaScript object, you walk through the items so that the update expression only contains the fields you have provided.
This is not really a DynamoDB question in that this is more a general JS coding question.
You can use UpdateItem; to familiarize yourself with DynamoDb queries I would suggest you DynamoDb NoSQL workbench:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.settingup.html
It can generate snippets for you based on your queries.
DynamoDb NoSQL workbench screenshot query

firebase snapshot change and angular2

i'm facing a little issue with firebase and angular2. I'm loading some little task and users can apply to them.
loadTask() {
let post = []
const publicationRef = this.rootRef.child('task').child('attente');
const usersR = this.rootRef.child('users')
return Observable.create((observer) => {
this.geoQuery.on("key_entered", function(key, location, distance){
publicationRef.child(key).on('value', snapshot => {
const taskkey = snapshot.key()
snapshot.forEach((data) => {
const datakey = data.key();
const dataVal = data.val();
usersR.orderByKey().equalTo(dataVal.user).on('child_added', snapshot => {
console.log({key: datakey,nom: snapshot.val().nom, distances: distance})
post.push({nom: dataVal.nom, key: datakey,unom: snapshot.val().nom, uprenom: snapshot.val().prenom, distance: Math.floor(distance), uimage:snapshot.val().image, budget: dataVal.budget, description: dataVal.description, date:dataVal.date, location: dataVal.location, user: dataVal.user, id1: taskkey, id:datakey})
observer.next(post)
})
})
})
})
})
}
all the task are displayed but when i apply to a task , in the dom it call a new push method , so a new task with the same value appear and disseapear when i reload the page. i tried to remove the push method
loadTask() {
const publicationRef = this.rootRef.child('task').child('attente');
const usersR = this.rootRef.child('users')
return Observable.create((observer) => {
this.geoQuery.on("key_entered", function(key, location, distance){
publicationRef.child(key).on('value', snapshot => {
const taskkey = snapshot.key()
snapshot.forEach((data) => {
const datakey = data.key();
const dataVal = data.val();
usersR.orderByKey().equalTo(dataVal.user).on('child_added', snapshot => {
console.log({key: datakey,nom: snapshot.val().nom, distances: distance})
const publicio = [{nom: dataVal.nom, key: datakey,unom: snapshot.val().nom, uprenom: snapshot.val().prenom, distance: Math.floor(distance), uimage:snapshot.val().image, budget: dataVal.budget, description: dataVal.description, date:dataVal.date, location: dataVal.location, user: dataVal.user, id1: taskkey, id:datakey}]
observer.next(publicio)
})
})
})
})
})
}
apply to a task doesnt add a new data with the same value , but in the dom it only load one set of data and not all of them
Try to clear post in the "on" method:
publicationRef.child(key).on('value', snapshot => {
post = [];
const taskkey = snapshot.key();
.
.
.

Resources