How to tell which descendants are changed with on("child_changed") - firebase

For example, I have following database structure:
/
+ users
+ 1
+ items
+ -xxx: "hello"
+ 2
+ items
Then;
var usersRef = new Firebase("https://mydb.firebaseio.com/users");
usersRef.on("child_changed", function(snapshot) {
utils.debug(JSON.stringify(snapshot.exportVal()));
});
If a value, "world", is pushed to "/users/1/items", I may get:
{"items": {"-xxx": "hello", "-yyy": "world"}}
So, how to tell which one is changed?
I need to on("child_added") every single ref to "/users/$id/items"?
NOTE: I'm trying to write an admin process in node.js.

The child_changed event only provides information on which immediate child has changed. If a node deeper in a data structure changed, you'll know which immediate child was affected but not the full path to the changed data. This is by design.
If you want granular updates about exactly what changed, you should attach callbacks recursively to all of the elements you care about. That way when an item changes, you'll know what the item was by which callback is triggered. Firebase is actually optimized for this use case; attaching large numbers of callbacks -- even thousands -- should work fine. Behind the scenes, Firebase aggregates all of callbacks together and only synchronizes the minimum set of data needed.
So, for your example, if you want to get alerted every time a new item is added for any user, you could do the following:
var usersRef = new Firebase("https://mydb.firebaseio.com/users");
usersRef.on("child_added", function(userSnapshot) {
userSnapshot.ref().child("items").on("child_added", function(itemSnapshot)
utils.debug(itemSnapshot.val());
});
});
If you are working with a very large number of users (hundreds of thousands or millions), and synchronizing all of the data is impractical, there's another approach. Rather than have your server listen to all of the data directly, you could have it listen to a queue of changes. Then when clients add items to their item lists, they could also add an item to this queue so that the server becomes aware of it.
This is what the client code might look like:
var itemRef = new Firebase("https://mydb.firebaseio.com/users/MYID/items");
var serverEventQueue = new Firebase("https://mydb.firebaseio.com/serverEvents");
itemRef.push(newItem);
serverEventQueue.push(newItem);
You could then have the server listen for child_added on that queue and handle the events when they come in.

Andrew Lee gave a nice answer, but I think you should try to use cloud functions. something like this should work:
exports.getPath = functions.database.ref('/users/{id}/items/{itemId}')
.onWrite(event => {
// Grab the current value of what was written to the Realtime Database.
const original = event.data.val();
console.log('user id', event.params.id);
console.log('item id', event.params.itemId);
});

Related

How to know if Meteor Collection Subscription did not change. [Meteor + Blaze]

Below is the piece of subscription on UI.
Template.AssociateEmp.onCreated(function(){
this.validEmail = new ReactiveVar('');
const tpl = this;
this.autorun(() => {
var email = this.validEmail.get();
this.subscribe('GetUnassociatedUser', email, {
onReady: function () {},
onError: function () {}
});
});
});
Is there a way to know that even if the dynamic data changed (here validEmail), Meteor Subscription was unaffected and did not change its data on UI? Is there any flag or something that triggers when subscription data is unchanged?
Autorun, ReactiveVar and Subscriptions
In your code example the subscription itself will re-run the server's publication function as the subscription's input variable email depends on the reactive variable validEmail and thus triggers the autorun when validEmail changes.
You can easily check that on your server console by logging something to the console within the publication.
If validEmail remains unchanged than there is no reason for autorun to trigger (unless there are other reactive sources that may not be added to your code example).
What about the subscribed data
Now if something has caused the subscription to re-run and you want to know if the data of a collection has been changed you could easily check on collection.count() but this could be flawed.
Imagine your publication is parameterized to include different fields by different parameters, then the data that is transfered to the client side collection will be different.
You would then require a method to check on the client side collection's data integrity.
Use hashes to verify integrity
A possible help would be to generate hases from the dataset using the sha package.
You could for example create one hash of your whole collection:
// get data
const data = Collection.find().fetch();
// map data to strings
// and reduce to one string
const hashInput = data.map(doc => JSON.stringify(doc) ).reduce((a, b) => a + b);
// generate hash
const collectionHash = SHA256(hashInput);
After the next onReady you can generate a new hash of the collection and compare it with the previous hash. If they are different, then something has changed.
This also removes the need for iterating the collection's documents if you only want to know if the data has changed but it won't reveal which document has changed.
Hashing single documents
Hashing single documents gives you more insight about what has changed. To do that you only need to create a map of hashes of your collection:
// get data
const data = Collection.find().fetch();
// map data to strings
const hashes = data.map(doc => { _id: doc._id, hash: SHA256( JSON.stringify(doc) ) });
You can store these hashes together with a document's _id. If the hash of a document is different after a new subscription you can assume that the change is related to this document.
General notes
hashing is some kind of expensive operation so it might be difficult to keep up with performance on large collections
usually you should design your pub / sub and autorun in a way that when the input changes the output changes
code is cold-written so it may not work out of the box. Please let me know if something does not.

Meteor with Angular2 , Fetching all entries from a collection in single shot

I have successfully integeraed meteor with angular2 but while fetching the data from collection facing difficulties in getting at one shot, here is the steps:
Collection Name : OrderDetails
No Of records : 1000
Server:
Created publication file to subcribe the collection:
Meteor.publish('orderFilter', function() {
return OrderLineDetails.find({});
});
Client:
this.dateSubscription =
MeteorObservable.subscribe('orderFilter').subscribe(()=> {
let lines = OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:
{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).fetch();
});
In this lines attribute fetches all the collection entries, but fails to subscribe for the changes
When I try with below one,
OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).zone().subscribe(results => {
// code to loop the results
});
In this am able to subscribe for the collection changes, but the results are looped for 1000 times , as 1000 entries in the colleciton.
Is there any way to get the whole collection entries in one single shot and mean time to subscribe the changes in the collection ?.
Yes, there are a couple of ways you can do it, mostly depending on how you want to handle the data.
If having everything at once is important, then use a Method such as:
MeteorObservable.call('getAllElements', (err, result) => {
// result.length === all elements
})
While on server side doing
Meteor.methods({
getAllElements:function(){return myCollection.find().fetch()}
})
Now, if you want to listen to changes, ofcourse you'll have to do a subscription, and if you want to lower the amount of subscriptions, use rxjs' debounceTime() function, such as (from your code):
this.theData.debounceTime(400).subscribe(value => ...., err =>)
This will wait a certain amount of time before subscribing to that collection.
Now, based on your intent: listening to changes and getting everything at once, you can combine both approaches, not the most efficient but can be effective.
As #Rager explained, observables are close to streams, so when you populate data on miniMongo (front end collection you use when you find() data and is populated when you subscribe to publications) it will start incrementing until the collection is in sync.
Since miniMongo is populated when you subscribe to a publication, and not when you query a cursor, you could either:
Try the debouceTime() approach
Use a Meteor.Method after subscribing to the publication, then sync both results, keeping the first response from the method as your starting point, and then using data from Collection.find().subscribe(collectionArray => ..., err=>) to do whatterver you want to do when changes apply (not that recommended, unless you have a specific use case for this)
Also, .zone() function is specific to force re-render on Angular's event cycle. I'd recomend not use it if you're processing the collections' data instead of rendering it on a ngFor* loop. And if you're using an ngFor* loop, use the async pipe instead ngFor="let entry of Collection | async"
I don't think that's possible. When you subscribe to an Observable it handles values as a "stream", not necessarily a loop. I have seen some makeshift helper methods that handle the data synchronously, though the time it takes to subscribe is not decreased. Check out this article for an under the hood look... A simple Observable implementation
However, you can set it up to only loop once.
The way that I've been setting up that scenario, the collection only gets looped through one time (in the constructor when the app starts) and detects changes in the collection. In your case it would look like:
values: YourModel[] = []; //this is an array of models to store the data
theData: Observable<YourModel[]>;
errors: string[];
subFinished: boolean = false;
constructor(){
this.theData = OrderDetails.find({expectedShipDate:{$in:strArr}},{fields:{"expectedShipDate":1,"loadNo":1},sort:{"expectedShipDate":1}}).zone();
MeteorObservable.subscribe('orderFilter').subscribe();
//push data onto the values array
this.theData.subscribe(
value => this.values = value,
error => this.errors.push("new error"),
() => this.subFinished = true
);
}
The "values" array is updated with whatever changes happen to the database.

Good way to delete all data according to criteria/child's value in Firebase database admin?

I want to clean up this userPublic by deleting all of its child node which has isTesting == true. I am using Firebase's cloud function. My approach would be :
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", dataSnapshot => {
// ???
})
Since I can only call .remove() on reference and not snapshot but to filter the child I want it returns snapshot, how can I get the reference from snapshot? (I would like to know the key XXX-XXX-XXX of each filtered child, so I can concatenate with userPublic and .remove() them one by one)
Also, even if I can get all the references that I want to remove I think deleting them one by one by calling .remove() then wait for promise, then call the next one does not sounds like an optimal way. Are there any way to remove all of them in one go?
If it involves calling .update() on the top userPublic node, I would have to fetch everything, remove the one with isTesting and then put the remaining back for update. This sounds like it is not efficient compared to the filtering way. As eventually the one with .isTesting is only about 5% of all data. Or is this actually the approach everyone is using?
You're almost there. All that's left is to create a single multi-location update from the results of your query:
const userPublic = admin.database().ref("/userPublic")
const testsInUserPublic = userPublic.orderByChild("isTesting").equalTo(true)
testsInUserPublic.once("value", snapshot => {
var updates = {};
snapshot.forEach(function(child) {
updates["/userPublic/"+child.key] = null;
});
userPublic.update(updates);
})
Doing this with promises would not be too different:
testsInUserPublic.once("value", snapshot => {
var promises = [];
snapshot.forEach(function(child) {
promises.push(child.ref.remove());
});
Promise.all(promises); // this returns a promise that resolves once all deletes are done, or that rejects once one of them fails
})
Performance of this will be very similar, since Firebase pipelines the requests over a single connection. See http://stackoverflow.com/questions/35931526/speed-up-fetching-posts-for-my-social-network-app-by-using-query-instead-of-obse/35932786#35932786

Re-create template while switching routes

How can we re-create template while switching routes?
For example, i have subscriber template. It detects when user scrolls down to a display and subscribes to more data. It takes several parameters.
Example:
amazing_page.html
{{#each}}
{{amazing_topic}}
{{/each}}
{{>subscriber name='topics' count=5}}
subscriber.js
//rough sample code
Template.subscriber.onCreated(function() {
var self = this;
var type = Template.currentData().name;
var count = Template.currentData().count;
var user = Template.currentData().user;
var skipCount = 0;
self.autorun(function(c){
self.subscribe(type, skipCount, user);
var block = true;
$(window).scroll(function() {
if (($(window).scrollTop() + $(window).height()) >= ($(document).height()) && block) {
block = false;
skipCount = skipCount + count;
console.log(type);
console.log(skipCount);
self.subscribe(type, skipCount, user, {
onReady: function() {
block = true;
},
onStop: function() {
console.log('stopped');
}
});
}
});
})
});
I use this template with different parameters in different routes.
The problem is if user switches some routes, and scrolls down in one page, all subscribers he gets in another pages will actualy work in this page. More, they will store increased values for them variables, and will do all included logic.
I found a bad decision when we use Route.getName (for example) comparing and name parameter of subscriber. It is not a best option. Can someone help me to find a good practice for that?:)
Simple Example:
We have 3 different routes:
1)News
2)Videos
3)Topics
These routes templates have included special subscriber-templates. And subscribtion works fine on scroll.
Ok, now let's visit all of them: News, Videos, Topics.
Good, now scroll down and... I have three instance of subscriber template what will subscribe on them own publications, because they not destroyed when we switch routes.
And, as a result - when user scrolling Topics page, he will call subscribtion for News and Videos too, and he will take data from these collections too;)
And - this is a problem:)
UPD:
Looks like we find a decision. If i use Template.instance (autorun/subscribe) it will start working expected, except some strange cases:)
First of all, when i go in another route in next iteration (scroll down) it returns me data from old, destroyed template + error. Next time (next iteration) it will start to subscribe to a correct data. Hmm...it looks like i have mistake in autorun section...or not?
Attached print screen from console
this
It sounds like you have multiple subscriptions to the same collection and that therefore the list of documents shown in various contexts can change in unexpected ways. Meteor manages multiple subscriptions on the same collection by synchronizing the union of the selected documents.
The simplest way to manage each of your views is to make sure that the data context for a particular view uses a .find() with the query you need. This will typically be the same query that your publication is using.
A different but less efficient approach is to .stop() the subscription when you leave a view.

how to discard initial data in a Firebase DB

I'm making a simple app that informs a client that other clients clicked a button. I'm storing the clicks in a Firebase (db) using:
db.push({msg:data});
All clients get notified of other user's clicks with an on, such as
db.on('child_added',function(snapshot) {
var msg = snapshot.val().msg;
});
However, when the page first loads I want to discard any existing data on the stack. My strategy is to call db.once() before I define the db.on('child_added',...) in order to get the initial number of children, and then use that to discard that number of calls to db.on('child_added',...).
Unfortunately, though, all of the calls to db.on('child_added',...) are happening before I'm able to get the initial count, so it fails.
How can I effectively and simply discard the initial data?
For larger data sets, Firebase now offers (as of 2.0) some query methods that can make this simpler.
If we add a timestamp field on each record, we can construct a query that only looks at new values. Consider this contrived data:
{
"messages": {
"$messageid": {
"sender": "kato",
"message": "hello world"
"created": 123456 // Firebase.ServerValue.TIMESTAMP
}
}
}
We could find messages only after "now" using something like this:
var ref = new Firebase('https://<your instance>.firebaseio.com/messages');
var queryRef = ref.orderBy('created').startAt(Firebase.ServerValue.TIMESTAMP);
queryRef.on('child_added', function(snap) {
console.log(snap.val());
});
If I understand your question correctly, it sounds like you only want data that has been added since the user visited the page. In Firebase, the behavior you describe is by design, as the data is always changing and there isn't a notion of "old" data vs "new" data.
However, if you only want to display data added after the page has loaded, try ignoring all events prior until the complete set of children has loaded at least once. For example:
var ignoreItems = true;
var ref = new Firebase('https://<your-Firebase>.firebaseio.com');
ref.on('child_added', function(snapshot) {
if (!ignoreItems) {
var msg = snapshot.val().msg;
// do something here
}
});
ref.once('value', function(snapshot) {
ignoreItems = false;
});
The alternative to this approach would be to write your new items with a priority as well, where the priority is Firebase.ServerValue.TIMESTAMP (the current server time), and then use a .startAt(...) query using the current timestamp. However, this is more complex than the approach described above.

Resources