Firebase database sort by deeper child - firebase

Considering the following structure of the Firebase database:
root
game1
$playerUidA
score: 50
$playerUidB
score: 10
.....
game2
$playerUidC
score: 20
$playerUidD
score: 30
.....
game3
.....
I want to run a query that will return all game nodes, where the children of each game node (the players) will be sorted based on the score. The game node contains the UID of each player, and each UID node contains the score of the player. I am storing other data as well, however, for the sake of this example I will be only using the score.
Can I do that with a single query? Something like
rootRef.child("root").orderByChild("score")? Unfortunately that doesn't seem to work.
Or the only way to achieve that is by manually sorting the items on the client?
#Puf - Hope you'll answer that :)

Although this question is rather old there might be people (like me) stumbling over it. Especially because it is pretty intuitive to structure the database in a similar way the author of the question did, to create (for example) a leaderboard system for a game. Since the answer is a bit outdated I wanted to add some things.
Some time ago the devs added the possibility to order by deeply nested children! (See Ordering by a specified child key) To do so you basically have to do the exact same thing the author did and additionally use the first part of the answer given by #adolfosrs. Basically you have to do two things:
Work with .indexOn (as described by #adolfosrs)
Use the OrderByChild() command.
To make this work on the example given by #steliosf you would have to do the following:
First set the .indexOn in your database rules:
{
"rules": {
"$gameId": {
".indexOn": "score",
"$playerUid": {
...
}
}
}
}
Second use the Command the author of the question already used:
rootRef.Child("root").OrderByChild("score")
I would recommend that you always add a LimitToFirst() or LimitToLast() command to avoid that you pull the whole database which might be a lot of data (depending on the size of your database of course). To get for example the top 10 scores you could use:
rootRef.Child("root").OrderByChild("score").LimitToLast(10)
Since the data is ordered in ascending order you need to use LimitToLast().

If you want all the games sorted by the player score all you need to do is to work with your .indexOn rule.
{
"rules": {
"$gameId": {
".indexOn": "score",
"$playerUid": {
...
}
}
}
}
This will keep your data sorted in the database so when you retrieve it you will have all the data ready. But keep in mind that if you want to retrieve all the games and all the players it means you will be fetching the whole database. So you should look at the needs of your application and maybe rethink the structure.
Something that could help when scaling is to iterate over the games and retrieve a limited amount of users with firebase.database().ref(gameId).limitToLast(10).
Update
For your example you will have all the games with the following request:
firebase.database().ref().once('value', snap => {
//prints all players for each game sorted ascending.
snap.forEach(game => {
game.forEach(player => {
console.log(player.val());
});
});
});

Related

Firebase database check if element exists in a ListField in Flutter

I have a real-time database on firebase which consists of ListFields. Among these fields, one field, participants is a list of strings and two usernames. I want to make a query to firebase database such that it will return the documents in which a particular username is present in the participants list.
The structure of my document is as follows :
I want to make a query such that Firebase returns all the documents in which the participants list consists aniruddh. I am using Flutter with the flutterfire plugins.
Your current data structure makes it easy to find the participants for a conversation. It does however not make it easy to find the conversations for a user.
One alternative data structure that makes this easier is to store the participants in this format:
imgUrls: {},
participants: {
"aniruddh": true,
"trubluvin": true
}
Now you can technically query for the the conversations of a user with something like:
db.child("conversations").orderByChild("participants/aniruddh").equalTo(true)
But this won't scale very well, as you'll need to define an index for each user.
The proper solution is to add a second data structure, known as an inverted index, that allows the look up of conversations for a user. In your case that could look like this:
userConversations: {
"aniruddh": {
"-LxzV5LzP9TH7L6BvV7": true
},
"trubluvin": {
"-LxzV5LzP9TH7L6BvV7": true
}
}
Now you can look up the conversations that a user is part of with a simple read operation. You could expand this data structure to contain more information on each conversation, such as the information you want to display in your list view.
Also see my answer heres:
Firebase query if child of child contains a value (for more explanation on why the queries won't work in your current structure, and why they won't scale in the first structure in my answer).
Best way to manage Chat channels in Firebase (for an alternative way of naming the chat rooms).

Google Cloud Firestore documents limit

I've been working with Google Cloud Firestore. I'm about to import 13000+ records from a CSV to the firestore back-end. I'll be using this collection for look up and auto-completion purposes.
I'm curious and concerned to know if this is a good idea. Also, I'm looking for some suggestions on what techniques should I be using to make retrieval of this this data as efficient as possible. I'm working with Angular 5 and using AngularFire2 to connect with Firestore.
The document itself is really small such as:
{
address: {
state: "NSW"
street: "19 XYZ Road"
suburb: "Darling Point"
},
user: {
name: "ABC",
company: "Property Management Company"
}
file_no: "AB996"
}
Most of the searching would be based on file_no property of the document.
Update
I just imported all 13k+ records to Firestore. It is really efficient. However, I have one issue. After importing the records, I'm getting the message on my Firestore console that my daily limit for Read Operations is reached (0.05 of 0.05 Million Ops). I just wrote data and displayed those records in a Data Table. I used the following query:
this.propertyService
.getSnapshotChanges()
.subscribe(properties => {
this.properties = properties;
this.loadingIndicator = false;
});
getSnapshotChanges(): Observable < any > {
return this.afs.collection(this.propertiesCollection).snapshotChanges()
.map((actions) => {
return actions.map((snapshot) => {
const data = snapshot.payload.doc.data();
data.id = snapshot.payload.doc.id;
return data;
});
});
}
How dos this makes my reading limit exceed?
The number of documents in a collection is of no consequence when you use Cloud Firestore. That's actually one of its bigger perks: no matter how many documents are in a collection, the queries will take the same amount of time.
Say you add 130 document and (for sake of example) it takes 1 second to get 10 documents out of it. That's the performance you'll get no matter how many documents are in the collection. So with 1300 documents it will also take 1 second, with 13K it will take 1 second, and with 13M, it will also take 1 second.
The problem more developers run into is to make their use-cases fit within the API of Firestore. For example: the only way to search for strings is with a so-called prefix match, there is no support for full-text search. This means that you can search for Prop* and find Property Management Company, but not for *Man* to find it.

firestore: representing a relationship

In firestore i have a collection called things.
Each thing is owned by a user.
Each thing can be shared by the owner with other specified users.
the structure of thing looks something like
{
id: "thing01",
sharedWith: {
"user1": true,
"user2": true,
},
dtCreated: 3458973948
}
When I want to retrieve all thing objects that are shared with user1, ordered by dtCreated desc,
i can't do this without having to create an index on things.thing.user1
i.e. for every unique userid i have to create an index on the things collection.
Obviously this is not practical. The docs talk about using full text search for this, but this doesn't seem like a problem we would want to use full text search for.
Is there a different way i should be structuring the data to achieve what i want?
Is firestore just the wrong technology choice for this?
It's working very well for storing the thing objects themselves.
---- update ----
this question is not a real duplicate of Firestore: Working with nested single queries because the answer provided there is very specific to the OP's context.

How can I use a gremlin query to filter based on a users permissions?

I am fairly new to graph databases, however I have used SQL Server and document databases (Lucene, DocumentDb, etc.) extensively. It's completely possible that I am approaching this query the wrong way, since I am new to graph databases. I am trying to convert some logic to a graph database (CosmosDB Graph via Gremlins to be specific) that we currently are using SQL Server for. The reason for the change is that this problem set is not really what SQL Server is great at and so our SQL query (which we have optimized as good as we can) is really starting to be the hot spot of our application.
To give a very brief overview of our logic, we run a web shop that allows admins to configure products and users with several levels of granular permissions (described below). Based on these permissions, we show the user only the products they are allowed to see.
Entities:
Region: A region consists of multiple countries
Country: A country has many markets and many regions
Market: A market is a group of stores in a single country
Store: A store is belongs to a single market
Users have the following set of permissions and each set can contain multiple values:
can-view-region
can-view-country
can-view-market
can-view-store
Products have the following set of permissions and each set can contain multiple values:
visible-to-region
visible-to-country
visible-to-market
visible-to-store
After trying for a few days, this is the query that I have come up with. This query does work and returns the correct products for the given user, however it takes about 25 seconds to execute.
g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country').in('in-market').hasLabel('store'),
__.out('can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup().union(
__.out('in-market').in('contains-country').in('visible-to-region').hasLabel('product'),
__.out('in-market').in('visible-to-country').hasLabel('product'),
__.out('in-market').in('visible-to-market').hasLabel('product'),
__.in('visible-to-store').hasLabel('product')
).dedup()
Is there a better way to do this? Is this problem maybe not best suited with a graph database?
Any help would be greatly appreciated!
Thanks,
Chris
I don't think this is going to help a lot, but here's an improved version of your query:
g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country','can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup().union(
__.out('in-market').union(
__.in('contains-country').in('visible-to-region'),
__.in('visible-to-country','visible-to-market')).hasLabel('product'),
__.in('visible-to-store').hasLabel('product')
).dedup()
I wonder if the hasLabel() checks are really necessary. If, for example, .in('in-market') can only lead a store vertex, then remove the extra check.
Furthermore it might be worth to create shortcut edges. This would increase write times whenever you mutate the permissions, but should significantly increase the read times for the given query. Since the reads are likely to occur way more often than permission updates, this might be a good trade-off.
CosmosDB Graph team is looking into improvements that can done on union step in particular.
Other options that haven't already been suggested:
Reduce the number of edges that are traversed per hop with additional predicates. e.g:
g.V('1').outE('market').has('prop', 'value').inV()
Would it be possible to split the traversal up and do parallel request in your client code? Since you are using .NET, you could take each result in first union, and execute parallel requests for the traversals in the second union. Something like this (untested code):
string firstUnion = #"g.V().has('user','username', 'john.doe').union(
__.out('can-view-region').out('contains-country').in('in-market').hasLabel('store'),
__.out('can-view-country').in('in-market').hasLabel('store'),
__.out('can-view-market').in('in-market').hasLabel('store'),
__.out('can-view-store')
).dedup()"
string[] secondUnionTraversals = new[] {
"g.V({0}).out('in-market').in('contains-country').in('visible-to-region').hasLabel('product')",
"g.V({0}).out('in-market').in('visible-to-country').hasLabel('product')",
"g.V({0}).out('in-market').in('visible-to-market').hasLabel('product')",
"g.V({0}).in('visible-to-store').hasLabel('product')",
};
var response = client.CreateGremlinQuery(col, firstUnion);
while (response.HasMoreResults)
{
var results = await response.ExecuteNextAsync<Vertex>();
foreach (Vertex v in results)
{
Parallel.ForEach(secondUnionTraversals, (traversal) =>
{
var secondResponse = client.CreateGremlinQuery<Vertex>(col, string.Format(traversal, v.Id));
while (secondResponse.HasMoreResults)
{
concurrentColl.Add(secondResponse);
}
});
}
}

do document IDs in Meteor need to be random or just unique?

i'm migrating data from a rails system, and it would be really convenient to assign the migrated objects IDs like post0000000000001, etc.
i've read here
Creating Meteor-friendly id's in Mongo?
that Meteor creates random 17 character strings from
23456789ABCDEFGHJKLMNPQRSTWXYZabcdefghijkmnopqrstuvwxyz
which looks to be chosen to avoid possibly ambiguous characters (omits 1 and I, etc.)
do the IDs need to be random for some reason? are there security implications to being able to guess a Meteor document's ID?! or it is just an easy way of generating unique IDs?
Mongo seems fine with sequential ids:
http://docs.mongodb.org/manual/core/document/#the-id-field
http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
so i would guess this would have to be a Meteor constraint if it exists.
The IDs just need to be unique.
Typically there is an element of order: Such as using integers, or timestamps, or something with sequentiality.
This can't work in Meteor since inserts can come from the client, they may be disconnected for a period, or clients clocks may be off/have varying latency. Also its not possible to know the previous _id (in the case of a sequential _id) at the time an _id is written owing to latency compensation (instant inserts).
The consequence of the lack of order in the DDP protocol is the decision to use entirely random ids. That is not to say you can't use your own _ids.
while there is a risk of a collision with this strategy it is minimal on the order of [number of docs in your collection]/[55^17] * 100 % or nearly impossible. In the event this occurs the client will temporarily insert it and cancel it once the server confirms the error with a Mongo Duplicate Key error.
Also when it comes to security with the other answer. It is not too much of an issue if the _id of the user is known. It is not possible to log in without a valid hashed login token or retrieve any information with it. This applies to the user collection only of course. If you have your own collection an easily guessable URL containing an id as a reference without publish method checks on the eligibility to read the data is a risk the high entropy random ids generated by Meteor can mitigate.
As long as they are unique it should be ok to use your own ids.
I am not an expert, but I suppose Mongo needs a unique ID so when it updates the document, it in fact creates a new version of the document of that same ID.
The real question is - I too whish to know - if we can change the ID without screwing Mongo mechanism and reliability, or we need to create a secondary attribute? (It can make a smaller index too I suppose)?
But me too, I can imagine that security wise, it is better if document IDs are difficult to guess, especially user IDs! Otherwise, could it be easy or possible to fake a user, knowing the ID? Anybody, correct me if I am wrong.
I don't think it's possible and desirable to change ID from Mongo.
But you can easily create a autoincrement ID with http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
I have created a package that does just that and that is configurable.
https://atmospherejs.com/stivaugoin/fluid-refno
var refNo = generateRefNo({
name: 'invoices', // default: 'counter'
prefix: 'I-', // default: ''
size: 5, // default: 5
filling: '0' // default: '0'
});
console.log(refNo); // output: "I-00001"
you now can use refNo to add in your document on Insert
maybe it will help you

Resources