Lazy Loaded List with Firebase - firebase

I need to build something like a lazy loaded list with Firebase, meaning I load an initial 10 items or something like that, whenever the user scrolls to the bottom of the list, I load 10 more items.
I have two problems:
How do I know when the user has scrolled to the bottom?
How do I update a query most effectively?
I currently query like this:
List myItems;
BehaviorSubject myItemsSubject = new BehaviorSubject();
int currentlyLoadedItems = 10;
// I call this in my constructor:
loadItems() {
Firestore.instance.collection('MyCollection').orderBy('position', descending: true).limit(currentlyLoadedItems).snapshots().listen((QuerySnapshot snaps) {
myItems = snaps.documents.map((DocumentSnapshot doc) {
return Workout.formDoc(doc);
}).toList();
myItems.add(myItems);
});
}
the when user scrolls to bottom I do this:
loadMoreItems() {
currentlyLoadedItems += 10;
loadItems();
}
The problem with this solution is, I always need to load the already loaded items again. Because when I have loaded the first 10 items and I call loadMoreItems() I load the first 20 items, so I have loaded the first 10 items twice.
Is there a better way to update a query?

Related

how to check if subscribe is done loading all existing rows?

I want to load all the items on start without showing any message, but once after loaded. I want to capture any new row in subscriber and show it to the desktop notification.
The problem is, I'm not sure how to check if all the previous items are loaded and if the row is new item or is it from previous existing item.
this.items = this.af.database.list('notifications/'+this.uid+'/');
this.items.subscribe(list => {
list.forEach(row => {
// doing something here...
});
// once all the rows are finished loading, then any new row, show desktop notification message
});
I have user lodash for the minimal code.
// this varible holds the initial loaded keys
let loadedKeys = [];
this.items = this.af.database.list('notifications/'+this.uid+'/');
this.items.subscribe((list)=>{
// we skip it while initial load
if(!_.isEmpty(loadedKeys)){
// different is the new keys
let newKeys = _.difference(_.map(list, "$key"), loadedKeys);
if(!_.isEmpty(newKeys)){
// ... notification code here
}
}
loadedKeys = _.map(list, "$key");
});
The behave you are looking for is the default Subject approach in RxJS.
Check this reactiveX url to follow the marble diagram of Publish Subject (the equivalent for Subject in RxJS).
So you have two easy options:
1) manually index witch rows you want to display like #bash replied
2) create a Rx.Subject() and assign only the newest's rows to it. Then you subscribe to this subject in your app workflow.
The advantage of method 2 is when a new .subscribe occur, it will not retrieve previous data.
Edit: I wrote this codepen as a guide to implement your custom RxJS Subject. Hope it helps.
Assuming your rows have something unique to match with previous rows you can do the following:
// A Row item has a unique identifier id
interface Row {
id: number;
}
this.rows: Row[];
this.items$ = this.af.database.list(`notifications/${this.uid}/`).pipe(
tap(list => {
// if rows is not array, first time...
if(!Array.isArray(this.rows)) {
// first time nothing to do
return;
}
// returns true if some item from list is not found in this.rows
const foundNewRow = list.some(item => {
return ! this.rows.find(row => row.id === item.id);
});
if (foundNewRow) {
// call method to show desktop message here
}
}
);
I used a pipe and a tap operator (that you will have to import). If you subscribe to this.items$ the tap operator will do the work:
this.items$.subscribe((items => this.rows = items));
If you do not want to set this.rows when normally subscribing than you can also do this in the tap operator. But that would assume you only use it for checking difference between existing and new items.

Re-create template while switching routes

How can we re-create template while switching routes?
For example, i have subscriber template. It detects when user scrolls down to a display and subscribes to more data. It takes several parameters.
Example:
amazing_page.html
{{#each}}
{{amazing_topic}}
{{/each}}
{{>subscriber name='topics' count=5}}
subscriber.js
//rough sample code
Template.subscriber.onCreated(function() {
var self = this;
var type = Template.currentData().name;
var count = Template.currentData().count;
var user = Template.currentData().user;
var skipCount = 0;
self.autorun(function(c){
self.subscribe(type, skipCount, user);
var block = true;
$(window).scroll(function() {
if (($(window).scrollTop() + $(window).height()) >= ($(document).height()) && block) {
block = false;
skipCount = skipCount + count;
console.log(type);
console.log(skipCount);
self.subscribe(type, skipCount, user, {
onReady: function() {
block = true;
},
onStop: function() {
console.log('stopped');
}
});
}
});
})
});
I use this template with different parameters in different routes.
The problem is if user switches some routes, and scrolls down in one page, all subscribers he gets in another pages will actualy work in this page. More, they will store increased values for them variables, and will do all included logic.
I found a bad decision when we use Route.getName (for example) comparing and name parameter of subscriber. It is not a best option. Can someone help me to find a good practice for that?:)
Simple Example:
We have 3 different routes:
1)News
2)Videos
3)Topics
These routes templates have included special subscriber-templates. And subscribtion works fine on scroll.
Ok, now let's visit all of them: News, Videos, Topics.
Good, now scroll down and... I have three instance of subscriber template what will subscribe on them own publications, because they not destroyed when we switch routes.
And, as a result - when user scrolling Topics page, he will call subscribtion for News and Videos too, and he will take data from these collections too;)
And - this is a problem:)
UPD:
Looks like we find a decision. If i use Template.instance (autorun/subscribe) it will start working expected, except some strange cases:)
First of all, when i go in another route in next iteration (scroll down) it returns me data from old, destroyed template + error. Next time (next iteration) it will start to subscribe to a correct data. Hmm...it looks like i have mistake in autorun section...or not?
Attached print screen from console
this
It sounds like you have multiple subscriptions to the same collection and that therefore the list of documents shown in various contexts can change in unexpected ways. Meteor manages multiple subscriptions on the same collection by synchronizing the union of the selected documents.
The simplest way to manage each of your views is to make sure that the data context for a particular view uses a .find() with the query you need. This will typically be the same query that your publication is using.
A different but less efficient approach is to .stop() the subscription when you leave a view.

How to delete all but most recent X children in a Firebase node?

Given a Firebase node lines filled with unique-ID children (from push() operations), such as this:
Firebase--
--lines
--K3qx02jslkdjNskjwLDK
--K3qx23jakjdz9Nlskjja
--K3qxRdXhUFmEJdifOdaj
--etc...
I want to be able to delete all children of lines except the most recently added 200 (or 100, or whatever). Basically this is a cleanup operation. Now I know I could do this by grabbing a snapshot of all children of lines on the client side, counting the entries, then using an endsAt(totalChildren-numToKeep) to grab the relevant data and run remove(). But I want to avoid grabbing all that data to the client.
Is there an alternative to my idea above?
Keep the most recent N items, is one of the trickier use-cases to implement. If you have any option to change it into "keep items from the past N hours", I recommend going that route.
The reason the use-case is tricky, is that you're counting items and Firebase does (intentionally) not have any count-based operations. Because of this, you will need to retrieve the first N items to know which item is N+1.
ref.child('lines').once('value', function(snapshot) {
if (snapshot.numChildren() > MAX_COUNT) {
var childCount = 0;
var updates = {};
snapshot.forEach(function (child) {
if (++childCount < snapshot.numChildren() - MAX_COUNT) {
updates[child.key()] = null;
}
});
ref.child('lines').update(updates);
}
});
A few things to note here:
this will download all lines
it performs a single update() call to remove the extraneous lines
One way to optimize this (aside from picking a different/time-based truncating strategy) is to keep a separate list of the "line ids".
lineids
--K3qx02jslkdjNskjwLDK
--K3qx23jakjdz9Nlskjja
--K3qxRdXhUFmEJdifOdaj
So you'll still keep the data for each line in lines, but also keep a list of just the ids. The code to then delete the extra ones then becomes:
ref.child('lineids').once('value', function(snapshot) {
if (snapshot.numChildren() > MAX_COUNT) {
var childCount = 0;
var updates = {};
snapshot.forEach(function (child) {
if (++childCount < snapshot.numChildren() - MAX_COUNT) {
updates['lineids/'+child.key()] = null;
updates['lines/'+child.key()] = null;
}
});
ref.update(updates);
}
});
This last snippet is slightly more involved, but prevents from having to download all lines data by just downloading the line ids.
There are many variations you can choose, but I hope this serves as enough inspiration to get started.

angularFire startAt querying and binding deletes new data

The application shows work-shifts for certain time-period. firebaseConn.getShifts is the API-function to get the shiftData for the given time period.
versions:
firebase: 2.0.6
angularFire: 0.9.0 (confirmed with 0.8.2 also)
This is my firebase schema:
And this is the code:
.factory('watchers', function(bunch-of-dependencies) {
var unbindShifts = function() {};
var inited = false;
var shifts = {};
... some irrelevant code in between ...
function initShifts() {
unbindShifts();
shifts.object = firebaseConn.getShifts( false, from, to, $scope );
$scope.shifts = shifts.object;
shifts.object.$bindTo($scope, "shifts").then(function(unbind) {
unbindShifts = unbind;
});
}
The firebase-queries (that have worked fine before adding the unbind / bind and possibly time-based querying might cause issues too):
firebaseConn.getShifts = function(asArray, from, to, scope) {
return cacheRequest(FBURL + "shifts", asArray, [from, to]);
};
function cacheRequest(url, asArray, limits) {
var type = asArray ? "array" : "object";
var startAt = limits ? limits[0] : undefined;
var endAt = limits ? limits[1] : undefined;
var retObj, FBRef;
cached[url] = cached[url] || {};
/* If there are limits-parameters we don't cache at all atm. Since those queries should be checked differently than static urls */
if(!limits && cached[url][type]) {
FBRef = cached[url][type];
} else {
FBRef = cached[url][type] = createFBRef(url, startAt, endAt);
}
if(asArray) {
retObj = FBRef.$asArray();
} else {
retObj = FBRef.$asObject();
}
return retObj;
}
function createFBRef(resourceURL, startAt, endAt) {
var modifiedObject = $firebase( createRef( resourceURL ).orderByKey().startAt(startAt).endAt(endAt) );
return modifiedObject;
}
function createRef(resourceURL) {
return new Firebase( resourceURL );
}
Now I have located the problem to be with the query limiting. If the from and to Dates are undefined, this works without problems. But I need to be able to limit the amount of data, since loading many years of workshift-data, to show a weeks time, won't be good :).
The actual problem is not displaying and fetching the data, everything works fine, it's related to the times and re-binding.
If I do any changes to e.g. "20150115"-table. For example I add another "groups"-child there. When i unbind and rebind, the whole "20150115"-table gets deleted and this holds true only to the latest changes. If I add multiple child to different dates e.g. "20150113", "20150114", "20150115" and the latest change is in "20150115" and then I unbind + re-bind another time from firebase, all the other root-paths will stay as they are, but the latest change in "20150115" will make the whole tree deleted.
I hope I make myself clear, so for safety I try to explain it again in simpler way.
- Changes to 1. "20150113", 2. "20150114", 3. "20150115" through the app.
- Changing timeline from UI causes: unbind + re-bind
- As a side-effect the whole "20150114" tree gets deleted.
The problem is somehow related to advanced querying with orderByKey().startAt(startAt).endAt(endAt) and binding.
Also for additional info. The data which is added through the UI gets added to the firebase database, but when the re-binding happens, the data is deleted from the database. Specifically on rebind, unbinding causes no issues, if I delay rebinding with timeout.
EDIT:
I have found the source of the actual issue. After the new binding is in place and everything seems to be in order, there is an angular watch event that kicks in. The event tries to save the last change user made before re-binding.
So if I have and active timeline for december (20141201 - 20141230) and I change "20141225"-data. Then change the timeline to 20150101 - 20150130, causing unbind and rebind (or manually fetching new data). There will be an event, after the binding has been done and everything seems to be in order, trying to save 20141225 data to either the new timeline (20150101 - 20150130) or the old one, not sure which one. This causes the firebase to actually delete the whole 20141225-tree, instead of saving the data.
The new data makes it into your Firebase fine, which you can see by either checking your Firebase dashboard or by running a quick snippet like this in your browser's dev console:
new Firebase("https://firebaseurl").once('value', function(s) { console.log(s.val()); })
The data even makes it back into your application. The only problem is that Angular doesn't know that new data has arrived, so it doesn't update the view with the new data.
Normally AngularFire's $asObject and $asArray methods take care of notifying AngularJS when new data arrives from Firebase. But since you are constantly creating new queries, you'll have to take care of that yourself.
There are a few ways to signal the new data to AngularJS and I'm definitely not an expert on which one is best. But if you add $scope.$apply(); to your setDays function it works:
function setDays(ref) {
var FBRange = setFBRange(ref, from, to);
var days;
unbindDays();
days = $firebase(FBRange).$asObject();
$scope.days = days;
days.$bindTo($scope, "days").then(function(unbind) {
unbindDays = unbind;
// As a result of the new binding entry gets mysteriously deleted from firebase
});
$scope.$apply(); // Tell AngularJS about the new data, so that it updates the view
function setFBRange(ref, from, to) {
return ref.orderByKey().startAt(""+from).endAt(from + to + "");
}
}
Updated Plunkr with this change (and some others to help in debugging): http://plnkr.co/edit/YZtkzUNtjQUCcw4xb2mj?p=preview

How do I reverse order based on my unique ids from push() [duplicate]

I'm trying to test out Firebase to allow users to post comments using push. I want to display the data I retrieve with the following;
fbl.child('sell').limit(20).on("value", function(fbdata) {
// handle data display here
}
The problem is the data is returned in order of oldest to newest - I want it in reversed order. Can Firebase do this?
Since this answer was written, Firebase has added a feature that allows ordering by any child or by value. So there are now four ways to order data: by key, by value, by priority, or by the value of any named child. See this blog post that introduces the new ordering capabilities.
The basic approaches remain the same though:
1. Add a child property with the inverted timestamp and then order on that.
2. Read the children in ascending order and then invert them on the client.
Firebase supports retrieving child nodes of a collection in two ways:
by name
by priority
What you're getting now is by name, which happens to be chronological. That's no coincidence btw: when you push an item into a collection, the name is generated to ensure the children are ordered in this way. To quote the Firebase documentation for push:
The unique name generated by push() is prefixed with a client-generated timestamp so that the resulting list will be chronologically-sorted.
The Firebase guide on ordered data has this to say on the topic:
How Data is Ordered
By default, children at a Firebase node are sorted lexicographically by name. Using push() can generate child names that naturally sort chronologically, but many applications require their data to be sorted in other ways. Firebase lets developers specify the ordering of items in a list by specifying a custom priority for each item.
The simplest way to get the behavior you want is to also specify an always-decreasing priority when you add the item:
var ref = new Firebase('https://your.firebaseio.com/sell');
var item = ref.push();
item.setWithPriority(yourObject, 0 - Date.now());
Update
You'll also have to retrieve the children differently:
fbl.child('sell').startAt().limitToLast(20).on('child_added', function(fbdata) {
console.log(fbdata.exportVal());
})
In my test using on('child_added' ensures that the last few children added are returned in reverse chronological order. Using on('value' on the other hand, returns them in the order of their name.
Be sure to read the section "Reading ordered data", which explains the usage of the child_* events to retrieve (ordered) children.
A bin to demonstrate this: http://jsbin.com/nonawe/3/watch?js,console
Since firebase 2.0.x you can use limitLast() to achieve that:
fbl.child('sell').orderByValue().limitLast(20).on("value", function(fbdataSnapshot) {
// fbdataSnapshot is returned in the ascending order
// you will still need to order these 20 items in
// in a descending order
}
Here's a link to the announcement: More querying capabilities in Firebase
To augment Frank's answer, it's also possible to grab the most recent records--even if you haven't bothered to order them using priorities--by simply using endAt().limit(x) like this demo:
var fb = new Firebase(URL);
// listen for all changes and update
fb.endAt().limit(100).on('value', update);
// print the output of our array
function update(snap) {
var list = [];
snap.forEach(function(ss) {
var data = ss.val();
data['.priority'] = ss.getPriority();
data['.name'] = ss.name();
list.unshift(data);
});
// print/process the results...
}
Note that this is quite performant even up to perhaps a thousand records (assuming the payloads are small). For more robust usages, Frank's answer is authoritative and much more scalable.
This brute force can also be optimized to work with bigger data or more records by doing things like monitoring child_added/child_removed/child_moved events in lieu of value, and using a debounce to apply DOM updates in bulk instead of individually.
DOM updates, naturally, are a stinker regardless of the approach, once you get into the hundreds of elements, so the debounce approach (or a React.js solution, which is essentially an uber debounce) is a great tool to have.
There is really no way but seems we have the recyclerview we can have this
query=mCommentsReference.orderByChild("date_added");
query.keepSynced(true);
// Initialize Views
mRecyclerView = (RecyclerView) view.findViewById(R.id.recyclerView);
mManager = new LinearLayoutManager(getContext());
// mManager.setReverseLayout(false);
mManager.setReverseLayout(true);
mManager.setStackFromEnd(true);
mRecyclerView.setHasFixedSize(true);
mRecyclerView.setLayoutManager(mManager);
I have a date variable (long) and wanted to keep the newest items on top of the list. So what I did was:
Add a new long field 'dateInverse'
Add a new method called 'getDateInverse', which just returns: Long.MAX_VALUE - date;
Create my query with: .orderByChild("dateInverse")
Presto! :p
You are searching limitTolast(Int x) .This will give you the last "x" higher elements of your database (they are in ascending order) but they are the "x" higher elements
if you got in your database {10,300,150,240,2,24,220}
this method:
myFirebaseRef.orderByChild("highScore").limitToLast(4)
will retrive you : {150,220,240,300}
In Android there is a way to actually reverse the data in an Arraylist of objects through the Adapter. In my case I could not use the LayoutManager to reverse the results in descending order since I was using a horizontal Recyclerview to display the data. Setting the following parameters to the recyclerview messed up my UI experience:
llManager.setReverseLayout(true);
llManager.setStackFromEnd(true);
The only working way I found around this was through the BindViewHolder method of the RecyclerView adapter:
#Override
public void onBindViewHolder(final RecyclerView.ViewHolder holder, int position) {
final SuperPost superPost = superList.get(getItemCount() - position - 1);
}
Hope this answer will help all the devs out there who are struggling with this issue in Firebase.
Firebase: How to display a thread of items in reverse order with a limit for each request and an indicator for a "load more" button.
This will get the last 10 items of the list
FBRef.child("childName")
.limitToLast(loadMoreLimit) // loadMoreLimit = 10 for example
This will get the last 10 items. Grab the id of the last record in the list and save for the load more functionality. Next, convert the collection of objects into and an array and do a list.reverse().
LOAD MORE Functionality: The next call will do two things, it will get the next sequence of list items based on the reference id from the first request and give you an indicator if you need to display the "load more" button.
this.FBRef
.child("childName")
.endAt(null, lastThreadId) // Get this from the previous step
.limitToLast(loadMoreLimit+2)
You will need to strip the first and last item of this object collection. The first item is the reference to get this list. The last item is an indicator for the show more button.
I have a bunch of other logic that will keep everything clean. You will need to add this code only for the load more functionality.
list = snapObjectAsArray; // The list is an array from snapObject
lastItemId = key; // get the first key of the list
if (list.length < loadMoreLimit+1) {
lastItemId = false;
}
if (list.length > loadMoreLimit+1) {
list.pop();
}
if (list.length > loadMoreLimit) {
list.shift();
}
// Return the list.reverse() and lastItemId
// If lastItemId is an ID, it will be used for the next reference and a flag to show the "load more" button.
}
I'm using ReactFire for easy Firebase integration.
Basically, it helps me storing the datas into the component state, as an array. Then, all I have to use is the reverse() function (read more)
Here is how I achieve this :
import React, { Component, PropTypes } from 'react';
import ReactMixin from 'react-mixin';
import ReactFireMixin from 'reactfire';
import Firebase from '../../../utils/firebaseUtils'; // Firebase.initializeApp(config);
#ReactMixin.decorate(ReactFireMixin)
export default class Add extends Component {
constructor(args) {
super(args);
this.state = {
articles: []
};
}
componentWillMount() {
let ref = Firebase.database().ref('articles').orderByChild('insertDate').limitToLast(10);
this.bindAsArray(ref, 'articles'); // bind retrieved data to this.state.articles
}
render() {
return (
<div>
{
this.state.articles.reverse().map(function(article) {
return <div>{article.title}</div>
})
}
</div>
);
}
}
There is a better way. You should order by negative server timestamp. How to get negative server timestamp even offline? There is an hidden field which helps. Related snippet from documentation:
var offsetRef = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/.info/serverTimeOffset");
offsetRef.on("value", function(snap) {
var offset = snap.val();
var estimatedServerTimeMs = new Date().getTime() + offset;
});
To add to Dave Vávra's answer, I use a negative timestamp as my sort_key like so
Setting
const timestamp = new Date().getTime();
const data = {
name: 'John Doe',
city: 'New York',
sort_key: timestamp * -1 // Gets the negative value of the timestamp
}
Getting
const ref = firebase.database().ref('business-images').child(id);
const query = ref.orderByChild('sort_key');
return $firebaseArray(query); // AngularFire function
This fetches all objects from newest to oldest. You can also $indexOn the sortKey to make it run even faster
I had this problem too, I found a very simple solution to this that doesn't involved manipulating the data in anyway. If you are rending the result to the DOM, in a list of some sort. You can use flexbox and setup a class to reverse the elements in their container.
.reverse {
display: flex;
flex-direction: column-reverse;
}
myarray.reverse(); or this.myitems = items.map(item => item).reverse();
I did this by prepend.
query.orderByChild('sell').limitToLast(4).on("value", function(snapshot){
snapshot.forEach(function (childSnapshot) {
// PREPEND
});
});
Someone has pointed out that there are 2 ways to do this:
Manipulate the data client-side
Make a query that will order the data
The easiest way that I have found to do this is to use option 1, but through a LinkedList. I just append each of the objects to the front of the stack. It is flexible enough to still allow the list to be used in a ListView or RecyclerView. This way even though they come in order oldest to newest, you can still view, or retrieve, newest to oldest.
You can add a column named orderColumn where you save time as
Long refrenceTime = "large future time";
Long currentTime = "currentTime";
Long order = refrenceTime - currentTime;
now save Long order in column named orderColumn and when you retrieve data
as orderBy(orderColumn) you will get what you need.
just use reverse() on the array , suppose if you are storing the values to an array items[] then do a this.items.reverse()
ref.subscribe(snapshots => {
this.loading.dismiss();
this.items = [];
snapshots.forEach(snapshot => {
this.items.push(snapshot);
});
**this.items.reverse();**
},
For me it was limitToLast that worked. I also found out that limitLast is NOT a function:)
const query = messagesRef.orderBy('createdAt', 'asc').limitToLast(25);
The above is what worked for me.
PRINT in reverse order
Let's think outside the box... If your information will be printed directly into user's screen (without any content that needs to be modified in a consecutive order, like a sum or something), simply print from bottom to top.
So, instead of inserting each new block of content to the end of the print space (A += B), add that block to the beginning (A = B+A).
If you'll include the elements as a consecutive ordered list, the DOM can put the numbers for you if you insert each element as a List Item (<li>) inside an Ordered Lists (<ol>).
This way you save space from your database, avoiding unnecesary reversed data.

Resources