Meteor with Angular2 , Fetching all entries from a collection in single shot - meteor

I have successfully integeraed meteor with angular2 but while fetching the data from collection facing difficulties in getting at one shot, here is the steps:
Collection Name : OrderDetails
No Of records : 1000
Server:
Created publication file to subcribe the collection:
Meteor.publish('orderFilter', function() {
return OrderLineDetails.find({});
});
Client:
this.dateSubscription =
MeteorObservable.subscribe('orderFilter').subscribe(()=> {
let lines = OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:
{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).fetch();
});
In this lines attribute fetches all the collection entries, but fails to subscribe for the changes
When I try with below one,
OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).zone().subscribe(results => {
// code to loop the results
});
In this am able to subscribe for the collection changes, but the results are looped for 1000 times , as 1000 entries in the colleciton.
Is there any way to get the whole collection entries in one single shot and mean time to subscribe the changes in the collection ?.

Yes, there are a couple of ways you can do it, mostly depending on how you want to handle the data.
If having everything at once is important, then use a Method such as:
MeteorObservable.call('getAllElements', (err, result) => {
// result.length === all elements
})
While on server side doing
Meteor.methods({
getAllElements:function(){return myCollection.find().fetch()}
})
Now, if you want to listen to changes, ofcourse you'll have to do a subscription, and if you want to lower the amount of subscriptions, use rxjs' debounceTime() function, such as (from your code):
this.theData.debounceTime(400).subscribe(value => ...., err =>)
This will wait a certain amount of time before subscribing to that collection.
Now, based on your intent: listening to changes and getting everything at once, you can combine both approaches, not the most efficient but can be effective.
As #Rager explained, observables are close to streams, so when you populate data on miniMongo (front end collection you use when you find() data and is populated when you subscribe to publications) it will start incrementing until the collection is in sync.
Since miniMongo is populated when you subscribe to a publication, and not when you query a cursor, you could either:
Try the debouceTime() approach
Use a Meteor.Method after subscribing to the publication, then sync both results, keeping the first response from the method as your starting point, and then using data from Collection.find().subscribe(collectionArray => ..., err=>) to do whatterver you want to do when changes apply (not that recommended, unless you have a specific use case for this)
Also, .zone() function is specific to force re-render on Angular's event cycle. I'd recomend not use it if you're processing the collections' data instead of rendering it on a ngFor* loop. And if you're using an ngFor* loop, use the async pipe instead ngFor="let entry of Collection | async"

I don't think that's possible. When you subscribe to an Observable it handles values as a "stream", not necessarily a loop. I have seen some makeshift helper methods that handle the data synchronously, though the time it takes to subscribe is not decreased. Check out this article for an under the hood look... A simple Observable implementation
However, you can set it up to only loop once.
The way that I've been setting up that scenario, the collection only gets looped through one time (in the constructor when the app starts) and detects changes in the collection. In your case it would look like:
values: YourModel[] = []; //this is an array of models to store the data
theData: Observable<YourModel[]>;
errors: string[];
subFinished: boolean = false;
constructor(){
this.theData = OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).zone();
MeteorObservable.subscribe('orderFilter').subscribe();
//push data onto the values array
this.theData.subscribe(
value => this.values = value,
error => this.errors.push("new error"),
() => this.subFinished = true
);
}
The "values" array is updated with whatever changes happen to the database.

Related

Firestore rule to only add/remove one item of array

To optimize usage, I have a Firestore collection with only one document, consisting in a single field, which is an array of strings.
This is what the data looks like in the collection. Just one document with one field, which is an array:
On the client side, the app is simply retrieving the entire status document, picking one at random, and then sending the entire array back minus the one it picked
var all = await metaRef.doc("status").get();
List tokens=all['all'];
var r=new Random();
int numar=r.nextInt(tokens.length);
var ales=tokens[numar];
tokens.removeAt(numar);
metaRef.doc("status").set({"all":tokens});
Then it tries to do some stuff with the string, which may fail or succeed. If it succeeds, then no more writing to the database, but if it fails it fetches that array again, adds the string back and pushes it:
var all = await metaRef.doc("status").get();
List tokens=all['all'];
List<String> toate=(tokens.map((element) => element as String).toList());
toate.add(ales.toString());
metaRef.doc("status").set({"all":toate});
You can use the methods associated with the Set object.
Here is an example to check that only 1 item was removed:
allow update: if checkremoveonlyoneitem()
function checkremoveonlyoneitem() {
let set = resource.data.array.toSet();
let setafter = request.resource.data.array.toSet();
return set.size() == setafter.size() + 1
&& set.intersection(setafter).size() == 1;
}
Then you can check that only one item was added. And you should also add additional checks in case the array does not exist on your doc.
If you are not sure about how the app performs the task i.e., successfully or not, then I guess it is nice idea to implement this logic in the client code. You can just make a simple conditional block which deletes the field from the document if the operation succeeds, either due to offline condition or any other issue. You can find the following sample from the following document regarding how to do it. Like this, with just one write you can delete the field which the user picks without updating the whole document.
city_ref = db.collection(u'cities').document(u'BJ')
city_ref.update({
u'capital': firestore.DELETE_FIELD
})snippets.py

Retrieve values from firebase database in conversation flow

I am trying to grab information from my firebase database after a particular intent is invoked in my conversation flow.
I am trying to make a function which takes a parameter of user ID, which will then get the highscore for that user, and then say that users highscore back to them.
app.intent('get-highscore', (conv) => {
var thisUsersHighestscore = fetchHighscoreByUserId(conv.user.id);
conv.ask('your highest score is ${thisUsersHighestScore}, say continue to keep playing.');
});
function fetchHighscoreByUserId(userId){
var highscoresRef = database.ref("highscores");
var thisUsersHighscore;
highscoresRef.on('value',function(snap){
var allHighscores= snap.val();
thisUsersHighscore = allHighscores.users.userId.highscore;
});
return thisUsersHighscore;
}
An example of the data in the database:
"highscores" : {
"users" : {
"1539261356999999924819020" : {
"highscore" : 2,
"nickname" : "default"
},
"15393362381293223232222738" : {
"highscore" : 78,
"nickname" : "quiz master"
},
"15393365724084067696560" : {
"highscore" : "32",
"nickname" : "cutie pie"
},
"45343453535534534353" : {
"highscore" : 1,
"nickname" : "friendly man"
}
}
}
It seems like it is never setting any value to thisUsersHighScore in my function.
You have a number of issues going on here - both with how you're using Firebase, how you're using Actions on Google, and how you're using Javascript. Some of these issues are just that you could be doing things better and more efficiently, while others are causing actual problems.
Accessing values in a structure in JavaScript
The first problem is that allHighscores.users.userId.highscore means "In an object named 'allHighscores', get the property named 'users', from the result of that, get the property named 'userId'". But there is no property named "userId" - there are just a bunch of properties named after a number.
You probably wanted something more like allHighscores.users[userId].highscore, which means "In an object named 'allHighscores', get the property named 'users', fromt he result of that, get the property named by the value of 'userId'".
But if this has thousands or hundreds of thousands of records, this will take up a lot of memory. And will take a lot of time to fetch from Firebase. Wouldn't it be better if you just fetched that one record directly from Firebase?
Two Firebase Issues
From above, you should probably just be fetching one record from Firebase, rather than the whole table and then searching for the one record you want. In firebase, this means you get a reference to the path of the data you want, and then request the value.
To specify the path you want, you might do something like
var userRef = database.ref("highscores/users").child(userId);
var userScoreRef = userRef.child( "highscore" );
(You can, of course, put these in one statement. I broke them up like this for clarity.)
Once you have the reference, however, you want to read the data that is at that reference. You have two issues here.
You're using the on() method, which fetches the value once, but then also sets up a callback to be called every time the score updates. You probably don't need the latter, so you can use the once() method to get the value once.
You have a callback function setup to get the value (which is good, since this is an async operation, and this is the traditional way to handle async operations in Javascript), but you're returning a value outside of that callback. So you're always returning an empty value.
These suggest that you need to make fetchHighScoreByUserId() an asynchronous function as well, and the way we have to do this now is to return a Promise. This Promise will then resolve to an actual value when the async function completes. Fortunately, the Firebase library can return a Promise, and we can get its value as part of the .then() clause in the response, so we can simplify things a lot. (I strongly suggest you read up on Promises in Javascript and how to use them.) It might look something like this:
return userScoreRef.once("value")
.then( function(scoreSnapshot){
var score = scoreSnapshot.val();
return score;
} );
Async functions and Actions on Google
In the Intent Handler, you have a similar problem as above. The call to fetchHighScoreByUserId() is async, so it doesn't finish running (or returning a value) by the time you call conv.ask() or return from the function. AoG needs to know to wait for an async call to finish. How can it do that? Promises again!
AoG Intent Handlers must return a Promise if there is an asyc call involved.
Since the modified fetchHighScoreByUserId() returns a Promise, we will leverage that. We'll also set our response in the .then() part of the Promise chain. It might look something like this:
app.intent('get-highscore', (conv) => {
return fetchHighscoreByUserId(conv.user.id)
.then( function(highScore){
conv.ask(`Your highest score is ${highScore}. Do you want to play again?`);
} );
});
Two asides here:
You need to use backticks "`" to define the string if you're trying to use ${highScore} like that.
The phrase "Say continue if you want to play again." is a very poor Voice User Interface. Better is directly asking if they want to play again.

How to know if Meteor Collection Subscription did not change. [Meteor + Blaze]

Below is the piece of subscription on UI.
Template.AssociateEmp.onCreated(function(){
this.validEmail = new ReactiveVar('');
const tpl = this;
this.autorun(() => {
var email = this.validEmail.get();
this.subscribe('GetUnassociatedUser', email, {
onReady: function () {},
onError: function () {}
});
});
});
Is there a way to know that even if the dynamic data changed (here validEmail), Meteor Subscription was unaffected and did not change its data on UI? Is there any flag or something that triggers when subscription data is unchanged?
Autorun, ReactiveVar and Subscriptions
In your code example the subscription itself will re-run the server's publication function as the subscription's input variable email depends on the reactive variable validEmail and thus triggers the autorun when validEmail changes.
You can easily check that on your server console by logging something to the console within the publication.
If validEmail remains unchanged than there is no reason for autorun to trigger (unless there are other reactive sources that may not be added to your code example).
What about the subscribed data
Now if something has caused the subscription to re-run and you want to know if the data of a collection has been changed you could easily check on collection.count() but this could be flawed.
Imagine your publication is parameterized to include different fields by different parameters, then the data that is transfered to the client side collection will be different.
You would then require a method to check on the client side collection's data integrity.
Use hashes to verify integrity
A possible help would be to generate hases from the dataset using the sha package.
You could for example create one hash of your whole collection:
// get data
const data = Collection.find().fetch();
// map data to strings
// and reduce to one string
const hashInput = data.map(doc => JSON.stringify(doc) ).reduce((a, b) => a + b);
// generate hash
const collectionHash = SHA256(hashInput);
After the next onReady you can generate a new hash of the collection and compare it with the previous hash. If they are different, then something has changed.
This also removes the need for iterating the collection's documents if you only want to know if the data has changed but it won't reveal which document has changed.
Hashing single documents
Hashing single documents gives you more insight about what has changed. To do that you only need to create a map of hashes of your collection:
// get data
const data = Collection.find().fetch();
// map data to strings
const hashes = data.map(doc => { _id: doc._id, hash: SHA256( JSON.stringify(doc) ) });
You can store these hashes together with a document's _id. If the hash of a document is different after a new subscription you can assume that the change is related to this document.
General notes
hashing is some kind of expensive operation so it might be difficult to keep up with performance on large collections
usually you should design your pub / sub and autorun in a way that when the input changes the output changes
code is cold-written so it may not work out of the box. Please let me know if something does not.

Good way to delete all data according to criteria/child's value in Firebase database admin?

I want to clean up this userPublic by deleting all of its child node which has isTesting == true. I am using Firebase's cloud function. My approach would be :
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", dataSnapshot => {
// ???
})
Since I can only call .remove() on reference and not snapshot but to filter the child I want it returns snapshot, how can I get the reference from snapshot? (I would like to know the key XXX-XXX-XXX of each filtered child, so I can concatenate with userPublic and .remove() them one by one)
Also, even if I can get all the references that I want to remove I think deleting them one by one by calling .remove() then wait for promise, then call the next one does not sounds like an optimal way. Are there any way to remove all of them in one go?
If it involves calling .update() on the top userPublic node, I would have to fetch everything, remove the one with isTesting and then put the remaining back for update. This sounds like it is not efficient compared to the filtering way. As eventually the one with .isTesting is only about 5% of all data. Or is this actually the approach everyone is using?
You're almost there. All that's left is to create a single multi-location update from the results of your query:
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", snapshot => {
var updates = {};
snapshot.forEach(function(child) {
updates["/userPublic/"+child.key] = null;
});
userPublic.update(updates);
})
Doing this with promises would not be too different:
testsInUserPublic.once("value", snapshot => {
var promises = [];
snapshot.forEach(function(child) {
promises.push(child.ref.remove());
});
Promise.all(promises); // this returns a promise that resolves once all deletes are done, or that rejects once one of them fails
})
Performance of this will be very similar, since Firebase pipelines the requests over a single connection. See http://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786

Meteor: Get count of collection by name. Accessing global scope on server

I'd like to create a method that returns the count of a generic collection.
Calling the method would look something like this:
Meteor.call('getCollectionCount', 'COLLECTION_NAME');
And the result would be the collection count.
The server method code would look something like this:
getCollectionCount: function (collectionName) {
return window[collectionName].find().count();
}
This won't work because window isn't defined on the server, but is something similar possible?
Use global instead of window.
Note that this uses the variable name assigned to the collection object, not the name given to the collection. For this to work with Meteor.users you need to assign another variable name.
if (Meteor.isServer) {
users = Meteor.users;
}
if (Meteor.isClient) {
Meteor.call('count', 'users', function (err, res) {
// do something with number of users
});
}
Also probably a good idea to check that global[collectionName] is actually a collection.
I came up with this code which makes the following assumptions :
collections are declared in the global scope as top level objects.
collections are searched by collection name, not the collection variable identifier.
So client code should declare their collections like this :
MyCollection=new Meteor.Collection("my-collection");
And use the function like this :
var clientResult=Meteor.call("getCollectionCount","my-collection",function(error,result){
if(error){
console.log(error);
return;
}
console.log("actual server-side count is : ",result);
});
console.log("published subset count is : ",clientResult);
The method supports execution on the client (this is known as method stub or method simulation) but will only yield the count of the collection subset replicated client-side, to get the real count wait for server-side response using a callback.
/packages/my-package/lib/my-package.js
getCollection=function(collectionName){
if(collectionName=="users"){
return Meteor.users;
}
var globalScope=Meteor.isClient?window:global;
for(var property in globalScope){
var object=globalScope[property];
if(object instanceof Meteor.Collection && object._name==collectionName){
return object;
}
}
throw Meteor.Error(500,"No collection named "+collectionName);
};
Meteor.methods({
getCollectionCount:function(collectionName){
return getCollection(collectionName).find().count();
}
});
As Meteor.users is not declared as a top level variable you have to account for the special case (yes, this is ugly).
Digging into Meteor's collection handling code could provide a better alternative (getting access to a collection handle by collection name).
Final words on this : using a method call to count a collection documents is unfortunately non-reactive, so given the Meteor paradigm this might be of little use.
Most of the time you will want to fetch the number of documents in a collection for pagination purpose (something like a "Load more" button in a posts list for example), and as the rest of the Meteor architecture you'll want this to be reactive.
To count documents in a collection reactively you'll have to setup a slightly more complicated publication as showcased in the "counts-by-room" example in the docs.
http://docs.meteor.com/#meteor_publish
This is something you definitely want to read and understand.
This smart package is actually doing it right :
http://atmospherejs.com/package/publish-counts
It provides a helper function that is publishing the counts of any cursor.
Keep track of the collections on some other property that the server has access too. You could even call it window if you really wanted to.
var wow = new Meteor.Collection("wow");
collections["wow"] = wow;
getCollectionCount: function (collectionName) {
return collections[collectionName].find().count();
}
If you don't want the package users to change how they work with collections in the app then I think you should use MongoInternals to get collections by name from the db. Not tested but here is an example:
//on server
Meteor.methods({
count: function( name ){
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
var collection = db.collection( name );
return collection && collection.count({});
}
});
Another example of MongoInternals use is here. Documentation of the count() function available from the mongo driver is here.

Resources