Looks like my Firestore requests are being throttled! (see picture below)
If I had to guess, I'd say it's not Firestore doing the throttling, it's actually Chrome... I've seen this kind of throttling before.
One way to get around Chrome's throttling here is to combine multiple requests into one.
As I sit here thinking "I wonder if Firestore can combine these read requests?", I notice that, coincidentally, the requests to Firestore are actually named batchGet! But oddly enough, each batchGet request only has one read inside it:
So, in an effort to avoid Chrome's throttling, my question is: how do I request multiple documents at the same time in Firestore? I suspect it's possible because, well, there's a server endpoint named batchGet. Is it possible?
Many thanks!
Just dug through the source a bit, looks like it's impossible. Transaction's .get() method looks like this:
Transaction.prototype.get = function (documentRef) {
var _this = this;
validateExactNumberOfArgs('Transaction.get', arguments, 1);
var ref = validateReference('Transaction.get', documentRef, this._firestore);
return this._transaction
.lookup([ref._key])
.then(function (docs) {
if (!docs || docs.length !== 1) {
return fail('Mismatch in docs returned from document lookup.');
}
var doc = docs[0];
if (doc instanceof NoDocument) {
return new DocumentSnapshot(_this._firestore, ref._key, null, false);
}
return new DocumentSnapshot(_this._firestore, ref._key, doc, false);
});
};
That .lookup() takes an array of all the documents to put into the batchGet request, and it's coded here to only ever have one element in that array.
So, looks like there's no way to get multiple documents in one batchGet request.
Related
With Flutter and Firestore, I am trying to get more than 10 documents into a Stream<List>. I can do this with a .where clause on a collection mapping the QuerySnapshot. However, the 10 limit is a killer.
I'm using the provider package in my app. So, in building a stream in Flutter with a StreamProvider, I can return a
Stream<List<Map from the entire collection. too expensive. 200 plus docs on these collections and too many users. Need to get more efficient.
Stream<List<Map using a .where from a Collection that returns a Stream List 10 max on the list...doesn't cut the mustard.
Stream<Map from a Document, that returns 1 stream of 1 document.
I need something in between 1 and 2.
I have a Collection with up to 500 Documents, and the user will choose any possible combination of those 500 to view. The user assembles class rosters to view their lists of users.
So I'm looking for a way to get individual streams of, say 30 documents, and then compile them into a List: But I need this List<Stream<Map to be a Stream itself so each individual doc is live, and I can also filter and sort this list of Streams. I'm using the Provider Package, and if possible would like to stay consistent with that. Here's where I am currently stuck:
So, my current effort:
Future<Stream<List<AttendeeData>>> getStreams() async {
List<Stream<AttendeeData>> getStreamsOutput = [];
for (var i = 0; i < teacherRosterList.length; i++) {
Stream thisStream = await returnTeacherRosterListStream(facility, teacherRosterList[i]);
getStreamsOutput.add(thisStream);
}
return StreamZip(getStreamsOutput).asBroadcastStream();
}
Feels like I'm cheating below: I get an await error if I put the snapshot directly in Stream thisStream above as Stream is not a future if I await, and if I don't await, it moves too fast and gets a null error.
Future<Stream<AttendeeData>> returnTeacherRosterListStream(String thisFacility, String thisID) async {
return facilityList.doc(thisFacility).collection('attendance').doc(thisID).snapshots().map(_teacherRosterListFromSnapshot);
}
}
Example of how I'm mapping in _teacherRosterListFromSnapshot (not having any problem here):
AttendeeData _teacherRosterListFromSnapshot(DocumentSnapshot doc) {
// return snapshot.docs.map((doc) {
return AttendeeData(
id: doc.data()['id'] ?? '',
authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
);
}
My StreamProvider Logic and the error:
return MultiProvider(
providers: [
StreamProvider<List<AttendeeData>>.value(
value: DatabaseService(
teacherRosterList: programList,
facility: user.claimsFacility,
).getStreams()),
]
Error: The argument type 'Future<Stream<List>>' can't be assigned to the parameter type 'Stream<List>'.
AttendeeData is my Map Class name.
So, the summary of questions:
Can I even do this? I'm basically Streaming a List of Streams of Maps....is this a thing?
If I can, how do I do it?
a. I can't get this into the StreamProvider because getStreams is a Future...how can I overcome this?
I can get the data in using another method from StreamProvider, but it's not behaving like a Stream and the state isn't updating. i'm hoping to just get this into Provider, as I'm comfortable there, and I can manage state very easily that way. However, beggars can't be choosers.
Solved this myself, and since there is a dearth of good start to finish answers, I submit my example for the poor souls who come after me trying to learn these things on their own. I'm a beginner, so this was a slog:
Objective:
You have any number of docs in a collection and you want to submit a list of any number of docs by their doc number and return a single stream of a list of those mapped documents. You want more than 10 (firestore limit on .where query), less than all the docs...so somewhere between a QuerySnapshot and a DocumentSnapshot.
Solution: We're going to get a list of QuerySnapshots, we're going to combine them and map them and spit them out as a single stream. So we're getting 10each in chunks (the max) and then some odd number left over. I plug mine into a Provider so I can get it whenever and wherever I want.
So from my provider I call this as the Stream value:
Stream<List<AttendeeData>> filteredRosterList() {
var chunks = [];
for (var i = 0; i < teacherRosterList.length; i += 10) {
chunks.add(teacherRosterList.sublist(i, i + 10 > teacherRosterList.length ? teacherRosterList.length : i + 10));
} //break a list of whatever size into chunks of 10.
List<Stream<QuerySnapshot>> combineList = [];
for (var i = 0; i < chunks.length; i++) {
combineList.add(*[point to your collection]*.where('id', whereIn: chunks[i]).snapshots());
} //get a list of the streams, which will have 10 each.
CombineLatestStream<QuerySnapshot, List<QuerySnapshot>> mergedQuerySnapshot = CombineLatestStream.list(combineList);
//now we combine all the streams....but it'll be a list of QuerySnapshots.
//and you'll want to look closely at the map, as it iterates, consolidates and returns as a single stream of List<AttendeeData>
return mergedQuerySnapshot.map(rosterListFromTeacherListDocumentSnapshot);
}
Here's a look at how I mapped it for your reference (took out all the fields for brevity):
List<AttendeeData> rosterListFromTeacherListDocumentSnapshot(List<QuerySnapshot> snapshot) {
List<AttendeeData> listToReturn = [];
snapshot.forEach((element) {
listToReturn.addAll(element.docs.map((doc) {
return AttendeeData(
id: doc.data()['id'] ?? '',
authorCreatedUID: doc.data()['authorCreatedUID'] ?? '',
);
}).toList());
});
return listToReturn;
}
I am trying to grab information from my firebase database after a particular intent is invoked in my conversation flow.
I am trying to make a function which takes a parameter of user ID, which will then get the highscore for that user, and then say that users highscore back to them.
app.intent('get-highscore', (conv) => {
var thisUsersHighestscore = fetchHighscoreByUserId(conv.user.id);
conv.ask('your highest score is ${thisUsersHighestScore}, say continue to keep playing.');
});
function fetchHighscoreByUserId(userId){
var highscoresRef = database.ref("highscores");
var thisUsersHighscore;
highscoresRef.on('value',function(snap){
var allHighscores= snap.val();
thisUsersHighscore = allHighscores.users.userId.highscore;
});
return thisUsersHighscore;
}
An example of the data in the database:
"highscores" : {
"users" : {
"1539261356999999924819020" : {
"highscore" : 2,
"nickname" : "default"
},
"15393362381293223232222738" : {
"highscore" : 78,
"nickname" : "quiz master"
},
"15393365724084067696560" : {
"highscore" : "32",
"nickname" : "cutie pie"
},
"45343453535534534353" : {
"highscore" : 1,
"nickname" : "friendly man"
}
}
}
It seems like it is never setting any value to thisUsersHighScore in my function.
You have a number of issues going on here - both with how you're using Firebase, how you're using Actions on Google, and how you're using Javascript. Some of these issues are just that you could be doing things better and more efficiently, while others are causing actual problems.
Accessing values in a structure in JavaScript
The first problem is that allHighscores.users.userId.highscore means "In an object named 'allHighscores', get the property named 'users', from the result of that, get the property named 'userId'". But there is no property named "userId" - there are just a bunch of properties named after a number.
You probably wanted something more like allHighscores.users[userId].highscore, which means "In an object named 'allHighscores', get the property named 'users', fromt he result of that, get the property named by the value of 'userId'".
But if this has thousands or hundreds of thousands of records, this will take up a lot of memory. And will take a lot of time to fetch from Firebase. Wouldn't it be better if you just fetched that one record directly from Firebase?
Two Firebase Issues
From above, you should probably just be fetching one record from Firebase, rather than the whole table and then searching for the one record you want. In firebase, this means you get a reference to the path of the data you want, and then request the value.
To specify the path you want, you might do something like
var userRef = database.ref("highscores/users").child(userId);
var userScoreRef = userRef.child( "highscore" );
(You can, of course, put these in one statement. I broke them up like this for clarity.)
Once you have the reference, however, you want to read the data that is at that reference. You have two issues here.
You're using the on() method, which fetches the value once, but then also sets up a callback to be called every time the score updates. You probably don't need the latter, so you can use the once() method to get the value once.
You have a callback function setup to get the value (which is good, since this is an async operation, and this is the traditional way to handle async operations in Javascript), but you're returning a value outside of that callback. So you're always returning an empty value.
These suggest that you need to make fetchHighScoreByUserId() an asynchronous function as well, and the way we have to do this now is to return a Promise. This Promise will then resolve to an actual value when the async function completes. Fortunately, the Firebase library can return a Promise, and we can get its value as part of the .then() clause in the response, so we can simplify things a lot. (I strongly suggest you read up on Promises in Javascript and how to use them.) It might look something like this:
return userScoreRef.once("value")
.then( function(scoreSnapshot){
var score = scoreSnapshot.val();
return score;
} );
Async functions and Actions on Google
In the Intent Handler, you have a similar problem as above. The call to fetchHighScoreByUserId() is async, so it doesn't finish running (or returning a value) by the time you call conv.ask() or return from the function. AoG needs to know to wait for an async call to finish. How can it do that? Promises again!
AoG Intent Handlers must return a Promise if there is an asyc call involved.
Since the modified fetchHighScoreByUserId() returns a Promise, we will leverage that. We'll also set our response in the .then() part of the Promise chain. It might look something like this:
app.intent('get-highscore', (conv) => {
return fetchHighscoreByUserId(conv.user.id)
.then( function(highScore){
conv.ask(`Your highest score is ${highScore}. Do you want to play again?`);
} );
});
Two asides here:
You need to use backticks "`" to define the string if you're trying to use ${highScore} like that.
The phrase "Say continue if you want to play again." is a very poor Voice User Interface. Better is directly asking if they want to play again.
I want to clean up this userPublic by deleting all of its child node which has isTesting == true. I am using Firebase's cloud function. My approach would be :
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", dataSnapshot => {
// ???
})
Since I can only call .remove() on reference and not snapshot but to filter the child I want it returns snapshot, how can I get the reference from snapshot? (I would like to know the key XXX-XXX-XXX of each filtered child, so I can concatenate with userPublic and .remove() them one by one)
Also, even if I can get all the references that I want to remove I think deleting them one by one by calling .remove() then wait for promise, then call the next one does not sounds like an optimal way. Are there any way to remove all of them in one go?
If it involves calling .update() on the top userPublic node, I would have to fetch everything, remove the one with isTesting and then put the remaining back for update. This sounds like it is not efficient compared to the filtering way. As eventually the one with .isTesting is only about 5% of all data. Or is this actually the approach everyone is using?
You're almost there. All that's left is to create a single multi-location update from the results of your query:
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", snapshot => {
var updates = {};
snapshot.forEach(function(child) {
updates["/userPublic/"+child.key] = null;
});
userPublic.update(updates);
})
Doing this with promises would not be too different:
testsInUserPublic.once("value", snapshot => {
var promises = [];
snapshot.forEach(function(child) {
promises.push(child.ref.remove());
});
Promise.all(promises); // this returns a promise that resolves once all deletes are done, or that rejects once one of them fails
})
Performance of this will be very similar, since Firebase pipelines the requests over a single connection. See http://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786
I'd like if someon could clarify this point about subscriptions in Meteor.js:
as written in the docs the find() function returns a cursor, not the data, that needs to be fetched;
let's say I have a publish function:
Meteor.publish('pages', function() {
return Pages.find()
});
now let's say I need only the page with code: "one"; in a template helper I can fetch that document:
Template.mytemplate.helpers({
data: function() {
return Pages.find({code: "one"});
}
});
The question is: is correct to do this, or if I need only one document would be better to subscribe only that document? (something like:
Meteor.publish('page', function(code) {
return Pages.find({code: code})
});
)
I mean: does it changes something in terms of efficiency/performance?
Yes, it does change a lot. If you subscribe to the entire collection, then the entire collection will be send to the client and kept in sync with the server. So by all means, if you only need one document, and do not need to cache any of the other documents (e.g., for switching to a new document), then only subscribe to the one you need.
I'd like to create a method that returns the count of a generic collection.
Calling the method would look something like this:
Meteor.call('getCollectionCount', 'COLLECTION_NAME');
And the result would be the collection count.
The server method code would look something like this:
getCollectionCount: function (collectionName) {
return window[collectionName].find().count();
}
This won't work because window isn't defined on the server, but is something similar possible?
Use global instead of window.
Note that this uses the variable name assigned to the collection object, not the name given to the collection. For this to work with Meteor.users you need to assign another variable name.
if (Meteor.isServer) {
users = Meteor.users;
}
if (Meteor.isClient) {
Meteor.call('count', 'users', function (err, res) {
// do something with number of users
});
}
Also probably a good idea to check that global[collectionName] is actually a collection.
I came up with this code which makes the following assumptions :
collections are declared in the global scope as top level objects.
collections are searched by collection name, not the collection variable identifier.
So client code should declare their collections like this :
MyCollection=new Meteor.Collection("my-collection");
And use the function like this :
var clientResult=Meteor.call("getCollectionCount","my-collection",function(error,result){
if(error){
console.log(error);
return;
}
console.log("actual server-side count is : ",result);
});
console.log("published subset count is : ",clientResult);
The method supports execution on the client (this is known as method stub or method simulation) but will only yield the count of the collection subset replicated client-side, to get the real count wait for server-side response using a callback.
/packages/my-package/lib/my-package.js
getCollection=function(collectionName){
if(collectionName=="users"){
return Meteor.users;
}
var globalScope=Meteor.isClient?window:global;
for(var property in globalScope){
var object=globalScope[property];
if(object instanceof Meteor.Collection && object._name==collectionName){
return object;
}
}
throw Meteor.Error(500,"No collection named "+collectionName);
};
Meteor.methods({
getCollectionCount:function(collectionName){
return getCollection(collectionName).find().count();
}
});
As Meteor.users is not declared as a top level variable you have to account for the special case (yes, this is ugly).
Digging into Meteor's collection handling code could provide a better alternative (getting access to a collection handle by collection name).
Final words on this : using a method call to count a collection documents is unfortunately non-reactive, so given the Meteor paradigm this might be of little use.
Most of the time you will want to fetch the number of documents in a collection for pagination purpose (something like a "Load more" button in a posts list for example), and as the rest of the Meteor architecture you'll want this to be reactive.
To count documents in a collection reactively you'll have to setup a slightly more complicated publication as showcased in the "counts-by-room" example in the docs.
http://docs.meteor.com/#meteor_publish
This is something you definitely want to read and understand.
This smart package is actually doing it right :
http://atmospherejs.com/package/publish-counts
It provides a helper function that is publishing the counts of any cursor.
Keep track of the collections on some other property that the server has access too. You could even call it window if you really wanted to.
var wow = new Meteor.Collection("wow");
collections["wow"] = wow;
getCollectionCount: function (collectionName) {
return collections[collectionName].find().count();
}
If you don't want the package users to change how they work with collections in the app then I think you should use MongoInternals to get collections by name from the db. Not tested but here is an example:
//on server
Meteor.methods({
count: function( name ){
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
var collection = db.collection( name );
return collection && collection.count({});
}
});
Another example of MongoInternals use is here. Documentation of the count() function available from the mongo driver is here.