How to load first 50 objects in firebase, stop the loading and then filter results? - firebase

My Firebase database is more than 800mb large and with more than 100.000 objects (news articles).
What I want to do is to fetch just the first 50 objects (most recent) and then to sort the objects got from the result of the first query according to child parameters.
So, for example, when the page is loaded, I need angularfire / firebase to load just first 50 objects and to stop loading the rest of objects in database. Then, I want to filter out just these 50 objects (articles) based on node category music.
So far, my first query seems to be fine (but if there is better way to ask firebase to load X objects and to stop, I would appreciate). But, the second part, I can’t figure it out because firebase throw an error.
The error is:
Query: Can't combine startAt(), endAt(), and limit(). Use limitToFirst() or limitToLast() instead
Here is my sample code:
var myArticlesRef = new Firebase(FIREBASE_URL + 'articles/');
var latestArticlesRef = myArticlesRef.limitToFirst(20); // is this the recommended way to ask firebase to stop
var latestArticlesOrder = latestArticlesRef.orderByChild('category').equalTo(‘Music’); // <- how to do something similar?
var latestArticlesInfo = $firebaseArray(latestArticlesOrder);
$scope.latestArticles = latestArticlesInfo;
console.log($scope.latestArticles);

This should work:
var query = myArticlesRef.orderByChild('category').equalTo(‘Music’).limitToFirst(20);
So you're asking Firebase to return the first 20 articles in the Music category.
While it is common to think of queries like this when coming from a relational/SQL mindset, I recommend that you consider this alternative data structure:
articles_by_category
music
article1: { }
article2: { }
article3: { }
...
technology
article4: { }
...
So instead of storing the articles in one big list, store them by category. That way to access the articles about music, you only have to do:
var query = ref.child('articles_by_category').child('music').limitToFirst(20);
With this approach the database doesn't have to execute any query and it can scale to a much higher number of users.
This is something you'll see regularly in a NoSQL world: you end up modeling your data for the way your application wants to query it. For a great introduction, see this article on NoSQL data modeling. Also read the Firebase documentation on data modeling and indexes.

Related

How to filter records by dateTime using firebase realtime database

In my firebase Realtime Database, every document saved has a datetime variable.
Example, dateTime:"2021-04-11T17:32:50.833523"
I have the following URL which fetches all documents. However, I now want to fetch only today's documents. I know that there are two options to achieving this. First one is to fetch all the documents from DB and then filter only today's records. Second option is to tweak the URL and put the filter while fetching records from the DB.
First I would like to know which approach yields faster results?
If it's the second approach, then I would like to know how to achieve it.
Here's the function I am using:
Future<void> fetchAllOrders() async {
final url ='https://fhgi-92211.firebaseio.com/orders.json?auth=$authToken';
}
According to the docs, something like this should work:
'https://fhgi-92211.firebaseio.com/orders.json?auth=$authToken&orderBy="dateTime"&startAt="2021-04-11"'
https://firebase.google.com/docs/database/rest/retrieve-data
Best solution - is n°2 : using & tweaking your URL - to do so I suggest :
First, tweaking your model by adding your date data in a simpler 2021-04-11 kinda of format
It will allow you much easier filtering
You can do so like this before writing your data to your RealTime Database :
import 'package:intl/intl.dart';
DateTime today = DateTime.now();
String simpleFormatToday = DateFormat('yyyy-MM-dd').format(today);
// returns a simple 2021-04-11 format of today's date
If your really need for some reason the full dateTime value you can write both dateTime:"2021-04-11T17:32:50.833523" & simpleDate:"$simpleFormatToday" to your object
Next,
To retrieve your data - use your function & tweak your URL to use Range Querying to retrieve only documents having simpleDate value equal to today - like so :
Future<void> fetchAllOrders() async {
final url ='https://fhgi-92211.firebaseio.com/orders.json?auth=$authToken&orderBy="simpleDate"&equalTo="$simpleFormatToday"';
}
First Approach
It will be useful if you want to get all the data and need to filter the dates several times from user input.
For example: If you have 100 records that you fetched using first approach i.e all you have is 100 records in your db and now user will make some filter on fetched records like getting 20-30 then 40-79, something like that. In this case making query using second approach wont be appropriate because of two reasons
Querying everytime on firebase would count read/write towards billing.
Managing data locally is more fast and reliable rather than making request to network frequently.
Second Approach
It will be useful in areas where you have to show specific time interval records in the one go and that's it, after fetching the records you dont need more of them to pull out from DB.
The Query will mainly revolve around what you want
equalsTo : It will give you only those records which will have exact date that you have passed as params.
orderBy : It will only work with equals to and will give you dates on bases of applied sorted filter (ex: by createdAt)
Future<void> fetchAllOrders() async {
//It will get today's date
var todaysDate = DateTime.now();
//From today's 12AM
var filterStartDate = DateTime(todaysDate.year,todaysDate.month,todaysDate.day,0,0,0);
//To second days' 11:59PM
var filterEndDate = DateTime(todaysDate.year,todaysDate.month,todaysDate.day,23,59,59);
final url ='https://fhgi-92211.firebaseio.com/orders.json?auth=$authToken&startAt="${filterStartDate.toIso8601String()}&endAt="${filterEndDate.toIso8601String()}';
}

Firebase firestore collection count with angularFire 2

I want to get the total number of the documents that exist in firestore.
I don't want to get the data only the total number of inside Products collection I have 200.000 items is that possible with Angular 4-5, not angular.js
Can someone expert tell me how I can achieve that ??
My code so far and is not work
get_total_messages() {
this.messages_collection = this.afs.collection<MessageEntity>('messages');
return this.messages_collection.snapshotChanges();
}
End this is how I try to get the data but is not what I want;
this.firebase_Service.get_total_messages().subscribe( data => {
console.log(data);
});
There is no API to get the count of the number of documents in a Firestore collection. This means that the only ways to get the count are:
Get all documents and count them client-side.
Store the count as a separate property and update that as you add/remove documents.
Both approaches are quite common in NoSQL databases, with the second of course being a lot more efficient as the number of documents grows.
Firebase provides a sample of using Cloud Functions to keep a counter. While this sample is written for the Firebase Realtime Database, it can easily be modified to work on Cloud Firestore too.
Firestore also provides documentation on running aggregation queries and running distributed counters. Both seem slightly more involved than the first sample I linked though.
this.firebase_Service.get_total_messages().subscribe( data=>this.totalnumber=data.length);
//now, you can get total number of messages
luckily , i've solved somehow using the code,
try this, and it works well .
this.db.collection('User').valueChanges()
.subscribe( result => {
console.log(result.length);
})

How can I use a gremlin query to filter based on a users permissions?

I am fairly new to graph databases, however I have used SQL Server and document databases (Lucene, DocumentDb, etc.) extensively. It's completely possible that I am approaching this query the wrong way, since I am new to graph databases. I am trying to convert some logic to a graph database (CosmosDB Graph via Gremlins to be specific) that we currently are using SQL Server for. The reason for the change is that this problem set is not really what SQL Server is great at and so our SQL query (which we have optimized as good as we can) is really starting to be the hot spot of our application.
To give a very brief overview of our logic, we run a web shop that allows admins to configure products and users with several levels of granular permissions (described below). Based on these permissions, we show the user only the products they are allowed to see.
Entities:
Region: A region consists of multiple countries
Country: A country has many markets and many regions
Market: A market is a group of stores in a single country
Store: A store is belongs to a single market
Users have the following set of permissions and each set can contain multiple values:
can-view-region
can-view-country
can-view-market
can-view-store
Products have the following set of permissions and each set can contain multiple values:
visible-to-region
visible-to-country
visible-to-market
visible-to-store
After trying for a few days, this is the query that I have come up with. This query does work and returns the correct products for the given user, however it takes about 25 seconds to execute.
g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country').in('in-market').hasLabel('store'),
__.out('can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup().union(
__.out('in-market').in('contains-country').in('visible-to-region').hasLabel('product'),
__.out('in-market').in('visible-to-country').hasLabel('product'),
__.out('in-market').in('visible-to-market').hasLabel('product'),
__.in('visible-to-store').hasLabel('product')
).dedup()
Is there a better way to do this? Is this problem maybe not best suited with a graph database?
Any help would be greatly appreciated!
Thanks,
Chris
I don't think this is going to help a lot, but here's an improved version of your query:
g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country','can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup().union(
__.out('in-market').union(
__.in('contains-country').in('visible-to-region'),
__.in('visible-to-country','visible-to-market')).hasLabel('product'),
__.in('visible-to-store').hasLabel('product')
).dedup()
I wonder if the hasLabel() checks are really necessary. If, for example, .in('in-market') can only lead a store vertex, then remove the extra check.
Furthermore it might be worth to create shortcut edges. This would increase write times whenever you mutate the permissions, but should significantly increase the read times for the given query. Since the reads are likely to occur way more often than permission updates, this might be a good trade-off.
CosmosDB Graph team is looking into improvements that can done on union step in particular.
Other options that haven't already been suggested:
Reduce the number of edges that are traversed per hop with additional predicates. e.g:
g.V('1').outE('market').has('prop', 'value').inV()
Would it be possible to split the traversal up and do parallel request in your client code? Since you are using .NET, you could take each result in first union, and execute parallel requests for the traversals in the second union. Something like this (untested code):
string firstUnion = #"g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country').in('in-market').hasLabel('store'),
__.out('can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup()"
string[] secondUnionTraversals = new[] {
"g.V({0}).out('in-market').in('contains-country').in('visible-to-region').hasLabel('product')",
"g.V({0}).out('in-market').in('visible-to-country').hasLabel('product')",
"g.V({0}).out('in-market').in('visible-to-market').hasLabel('product')",
"g.V({0}).in('visible-to-store').hasLabel('product')",
};
var response = client.CreateGremlinQuery(col, firstUnion);
while (response.HasMoreResults)
{
var results = await response.ExecuteNextAsync<Vertex>();
foreach (Vertex v in results)
{
Parallel.ForEach(secondUnionTraversals, (traversal) =>
{
var secondResponse = client.CreateGremlinQuery<Vertex>(col, string.Format(traversal, v.Id));
while (secondResponse.HasMoreResults)
{
concurrentColl.Add(secondResponse);
}
});
}
}

Firebase - Structuring Data For Efficient Indexing

I've read almost everywhere about structuring one's Firebase Database for efficient querying, but I am still a little confused between two alternatives that I have.
For example, let's say I want to get all of a user's "maxBenchPressSessions" from the past 7 days or so.
I'm stuck between picking between these two structures:
In the first array, I use the user's id as an attribute to index on whether true or false. In the second, I use userId as the attribute NAME whose value would be the user's id.
Is one faster than the other, or would they be indexed a relatively same manner? I kind of new to database design, so I want to make sure that I'm following correct practices.
PROGRESS
I have come up with a solution that will both flatten my database AND allow me to add a ListenerForSingleValueEvent using orderBy ONLY once, but only when I want to check if a user has a session saved for a specific day.
I can have each maxBenchPressSession object have a key in the format of userId_dateString. However, if I want to get all the user's sessions from the last 7 days, I don't know how to do it in one query.
Any ideas?
I recommend to watch the video. It is told about the structuring of the data very well.
References to the playlist on the firebase 3
Firebase 3.0: Data Modelling
Firebase 3.0: Node Client
As I understand the principle firebase to use it effectively. Should be as small as possible to query the data and it does not matter how many requests.
But you will approach such a request. We'll have to add another field to the database "negativeDate".
This field allows you to get the last seven entries. Here's a video -
https://www.youtube.com/watch?v=nMR_JPfL4qg&feature=youtu.be&t=4m36s
.limitToLast(7) - 7 entries
.orderByChild('negativeDate') - sort by date
Example of a request:
const ref = firebase.database().ref('maxBenchPressSession');
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })
Then add the user, and it puts all of its sessions.
const ref = firebase.database().ref('maxBenchPressSession/' + userId);
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })

How can I querying a key value without retriving all data in Firebase?

I'm learning angularjs, firebase, angularfire by building a sample app. It's a dictionary app that any one can add a word, add several explanation to the word, add several common usage to the word, add example centences to the word. People can freely add new word via this web app. OR people can freely realtime search a word and display the content of that word. When I build the app.
I found everytime when I search a word, I have to load all the data and then check if the query string exist in the data and then display the content if exist. So, it's quite heavy if the library become very big.
How can I query a string if it exist from the server side and if exist, just download that piece of data?
Here is a very simple example that is on the firebase site (Retreiving data). I suggest you take a look at that page, there is a lot more info there about this subject. Now the example:
var ref = new Firebase("https://dinosaur-facts.firebaseio.com/dinosaurs");
ref.orderByChild([the child node to compare to]).equalTo([what you are searching]).on("child_added", function(snapshot) {
//do something with the data in snapshot
});

Resources