Datastructure for multi-cast type of message broadcasting - firebase

We are thinking about migrating from Pusher to Firebase. We are having troubling thinking about how Pusher channels would be represented in Firebase.
In Pusher we have a channel per user. So a user might be in a user-1 channel, another might be in a user-2 channel.
Then our backend/server would send a message to both these users via Pusher.trigger(message, ['user-1', 'user-2']).
I think this would usually be done like this:
{
web_page_1: {
user_1: {
messages: [{}, {}, ..],
},
user_2: {
messages: [{}, {}, ..],
},
...
},
web_page_2: {
user_2: {
messages: [{}, {}, ..],
},
user_3: {
messages: [{}, {}, ..],
}
},
....
}
Here the problem is: User 1 and User 2 for the same page might have a lot of messages in common. Is there a way to reduce this duplication, since these messages can get rather large, sending and storing them per user can get expensive. Also User 1 should not be able to read the messages of User 2.
It would be nice to do something like this:
{
web_page_1: {
message_1: {
user_ids: [1,2,3]
content: {},
},
message_2: {
recipient_ids: [3,4,5]
content: {},
}
...
},
web_page_2: {
message_1: {
user_ids: [1,2,3]
content: {},
},
message_2: {
user_ids: [3,4,5]
content: {},
}
},
....
}
But then, how would the security policy be applied such that a message can only be read by the user_ids specified in it.
Any pointers would be really appreciated.

If multi-cast is your use-case and the messages get large, I would indeed split the messages from the users and add message-references to the users like you show.
Root
Users
provider:344923
Name: Akshay Rawat
Messages
1: true
2: true
3: true
provider:209103
Name: Frank van Puffelen
Messages
1: true
Messages
1: It's a beautiful day
2: The sun is shining
3: I feel good, I feel good
4: And nothing's gonna stop me now
In the above data you can see that you and I are users. The provider:... is our uid, but can be anything that allows you to identify the current user. You've received messages 1, 2 and 3, while I have only received message 3. Neither of us has received message 4.
I took the Web_page level out to simplify things a bit. If you really need that level, you can add it back. The basic approach will remain the same.
You security rules can then use these message-references to see if the use can read a specific message:
{
"rules": {
"Messages": {
"$message_id": {
".read": "root.child('Users/'+auth.uid+'/Messages').hasChild($message_id)"
}
}
}
This rule defines the security for any child under messages (identified by $message_id). We grant read access if the $message_id is references as a message for the current user (auth.uid).

Related

Firebase data structure for chat

I am trying to build an app for employers to chat with employees.
So I have employers, employees and messages.
I did it with $firebaseArray and childs:
recipient > sender > messages
I want to add sender data such as profile images and last message, I'm not how to do it.
- employer_1
- employee_1
- SKLDJLKDksdklJS
- content: "Hello"
- timestamp: 129081021
Is this the right way to do it or is there a better way? Thanks.
You might have issues of showing only latest messages or newer messages since any get on employer_1 -> employee_1 will load all messages.
Another alternate might be to have a structure like this:
{
"users":{
"employer_1":{
"profile-image":"<url>",
"last-message":"SKLDJLKDksdklJS",
...
},
"employee_1"{
"profile-image":"<url>",
"user-chat-list":{
"employer_1":{
"last-message":"SKLDJLKDksdklJS",
"message-list":{
"SKLDJLKDksdklJS" : 129081021,
"ASDCJLKDksdklJS" : 129081021
}
}
}
}
},
"messages":{
"SKLDJLKDksdklJS":{
"content": "Hello",
"sender":"employer_1",
"timestamp": 129081021
}
}
}
you won't need to fetch all the messages with content for a chat list.

Storing User Data Flattened Security

I'd like to create an app that has a lot of user data. Let's say that each user tracks their own time per task. If I were to store it flattened it would look like this:
{
users: {
USER_ID_1: {
name: 'Mat',
tasks: {
TASK_ID_1: true,
TASK_ID_2: true,
...
}
},
},
tasks: {
TASK_ID_1: {
start: 0,
end: 1
},
TASK_ID_2: {
start: 1,
end: 2
}
}
}
Now I'd like to query and get all the task information for the user. Right now the data is small. From their guides: https://www.firebase.com/docs/web/guide/structuring-data.html it says (near the end) "... Until we get into tens of thousands of records..." and then doesn't explain how to handle that.
So my question is as follows. I know we can't do filtering via security, but can I use security to limit what people have access to and then when searching base it off the user id? My structure would then turn to this:
{
users: {
USER_ID_1: {
name: 'Mat'
}
},
tasks: {
TASK_ID_1: {
user: USER_ID_1,
start: 0,
end: 1
},
TASK_ID_2: {
user: USER_ID_1,
start: 1,
end: 2
},
...
}
}
Then I would set up my security rules to only allow each task to be accessed by the user who created it, and my ref query would look like this:
var ref = new Firebase("https://MY_FIREBASE.firebaseio.com/");
$scope.tasks = $firebaseArray(ref.child('tasks/')
.orderByChild('user')
.startAt('USER_ID_1')
.endAt('USER_ID_1'));
Is that how I should structure it? My query works but I'm unsure if it'll work once I introduce security where one user can't see another users tasks.
You've already read that security rules can not be used to filter data. Not even creative data modeling can change that. :-)
To properly secure access to your tasks you'll need something like:
"tasks": {
"$taskid": {
".read": "auth.uid === data.child(user).val()"
}
}
With this each user can only read their own tasks.
But with these rules, your query won't work. At it's most core your query is reading from tasks here:
ref.child('tasks/')...some-filtering...on(...
And since your user does not have read permission on tasks this read operation fails.
If you'd give the user read permission on tasks the read and query would work, but the user could then also read all tasks that you don't want to give them access to.

Firebase security rule to prevent customers from circumventing usage tracking

We are offering a service that people will embed on their web site and we are hoping to use Firebase as our backend. We would like to base our subscription rates on page views or something similar. Right now we are stumped trying to figure out how to prevent customers from caching our client js code and omitting any portions that attempt to increment a page views counter.
What we need to do somehow is create a security rule that atomically prevents someone from reading from one location unless they have incremented the counter at another location. Any ideas on how to do this?
For example, assuming the following schema:
{
"comments" : {
"-JYlV8KQGkUk18-nnyHk" : {
"content" : "This is the first comment."
},
"-JYlV8KWNlFZHLbOphFO" : {
"content" : "This is a reply to the first.",
"replyToCommentId" : "-JYlV8KQGkUk18-nnyHk"
},
"-JYlV8KbT63wL9Sb0QvT" : {
"content" : "This is a reply to the second.",
"replyToCommentId" : "-JYlV8KWNlFZHLbOphFO"
},
"-JYlV8KelTmBr7uRK08y" : {
"content" : "This is another reply to the first.",
"replyToCommentId" : "-JYlV8KQGkUk18-nnyHk"
}
},
oldPageViews: 32498,
pageViews: 32498
}
What would be a way of only allowing read access to the comments if the client first incremented the pageViews field? At first I was thinking about having two fields (something like pageViews and oldPageViews) and starting out by incrementing pageViews, reading the comments, then incrementing oldPageViews to match, and only allowing read on comments if pageViews === oldPageViews + 1. However, unless this could be done atomically, the data could get into a corrupt state if the client started the process but didn't finish it.
Here is a codepen trying to test this idea out.
I would suggest a variation of Kato's rate limiting answer : https://stackoverflow.com/a/24841859/75644
Data:
{
"comments": {
"-JYlV8KQGkUk18-nnyHk": {
"content": "This is the first comment."
},
"-JYlV8KWNlFZHLbOphFO": {
"content": "This is a reply to the first.",
"replyToCommentId": "-JYlV8KQGkUk18-nnyHk"
},
"-JYlV8KbT63wL9Sb0QvT": {
"content": "This is a reply to the second.",
"replyToCommentId": "-JYlV8KWNlFZHLbOphFO"
},
"-JYlV8KelTmBr7uRK08y": {
"content": "This is another reply to the first.",
"replyToCommentId": "-JYlV8KQGkUk18-nnyHk"
},
"timestamp" : 1413555509137
},
"pageViews" : {
"count" : 345030,
"lastTs" : 1413555509137
}
}
Security Rules:
{
"rules": {
"pageViews": {
".validate": "newData.hasChildren(['count','lastTs'])",
"count": {
".validate": "newData.exists() && newData.isNumber() && newData.val() > data.val()"
},
"lastTs": {
// timestamp can't be deleted or I could just recreate it to bypass our throttle
".write": "newData.exists()",
// the new value must be at least 500 milliseconds after the last (no more than one message every five seconds)
// the new value must be before now (it will be since `now` is when it reaches the server unless I try to cheat)
".validate": "newData.isNumber() && newData.val() === now && (!data.exists() || newData.val() > data.val()+500)"
}
},
"comments": {
// The comments can't be read unless the pageViews lastTs value is within 500 milliseconds of now
".read": "root.child('pageViews').child('lastTs').val() > now - 501",
".write": true
}
}
}
NOTE : I haven't tested this so you need to play around with it a bit to see if it works.
Also, based on your sample data, I didn't deal with uid's. You need to make sure you're managing who can read/write here.
Justin's adaptation to the throttling code seems like a great starting point. There are a few annoying loopholes left, like forcing the counter to be updated, getting quantifiable metrics/analytics out of your counter (which requires hooking into a stats tool by some means and will be necessary for accurate billing reports and customer inquiries), and also being able to accurately determine when a visit "ends."
Building from Justin's initial ideas, I think a lot of this overhead can be omitted by simplifying the amount the client is responsible for. Maybe something like:
Only force the user to update a timestamp counter
Employ a node.js script to watch for updates to the counter
Let the node.js script "store" the audit data, preferably by
sending it to analytics tools like keen.io, intercom.io, etc.
Starting from this base, I'd adapt the security rules and structure as follows:
{
"rules": {
"count": {
// updated only from node.js script
// assumes our node worker authenticates with a special uid we created
// http://jsfiddle.net/firebase/XDXu5/embedded/result/
".write": "auth.uid === 'ADMIN_WORKER'",
".validate": "newData.exists() && newData.isNumber() && newData.val() > data.val()"
},
"lastTs": {
// timestamp can't be deleted or I could just recreate it to bypass our throttle
".write": "newData.exists()",
// the new value must be equal to now (i.e. Firebase.ServerValue.TIMESTAMP)
".validate": "newData.isNumber() && newData.val() === now"
},
"comments": {
// The comments can't be read unless the pageViews lastTs value is within 30 seconds
".read": "root.child('pageViews').child('lastTs').val() > now - 30000",
"$comment": {
".write": "???"
}
}
}
}
Now I would write a simple node script to perform the count and administrative tasks:
var Firebase = require('firebase');
var ref = new Firebase(URL);
ref.child('lastTs').on('value', heartbeatReceived);
var lastCheck = null;
function heartbeatReceived(snap) {
if( isNewSession(snap.val()) ) {
incrementCounter();
}
updateStatsEngine(snap);
}
function incrementCounter() {
ref.child('count').transaction(function(currVal) {
return (currVal||0) + 1;
});
}
function isNewSession(timestamp) {
// the criteria here is pretty arbitrary and up to you, maybe
// something like < 30 minutes since last update or the same day?
var res = lastCheck === null || timestamp - lastCheck > 30 * 60 * 1000;
lastCheck = timestamp;
return res;
}
function updateStatsEngine(snap) {
// contact keen.io via their REST API
// tell intercom.io that we have an event
// do whatever is desired to store quantifiable stats
// and track billing info
//
//var client = require('keen.io').configure({
// projectId: "<project_id>",
// writeKey: "<write_key>",
// readKey: "<read_key>",
// masterKey: "<master_key>"
//});
//
//client.addEvent("collection", {/* data */});
}
The downside of this approach is that if my admin script goes down, any events during that time are not logged. However, the wonderful thing about this script is its simplicity.
It's not going to have many bugs. Add monit, upstart, or another tool to make sure it stays up and does not crash. Job done.
It's also highly versatile. I can run it on my laptop or even my Android phone (as an HTML page) in a pinch.

Firebase Security API - Complex Data Structure - How to enforce relationships?

For the past few weeks i've been exploring Firebase and its features to build a web app, but I've kind of ran into a wall when it comes to security rules.
I've build a data structure on Firebase but I'm not sure if it follows best practices (if it doesn't, feel free to suggest anything different about it):
{
"groups" : {
<GROUP_KEY>
"name": "",
"rels": {
"users": {
<RELS_USERS_KEY>
"key":"" (USER_KEY)
},
"notes": {
<RELS_NOTES_KEY>
"key":"" (NOTE_KEY)
}
},
"isPrivate": true
},
"users": {
<USER_KEY>
"email": "",
"rels": {
"friends": {
<RELS_FRIENDS_KEY>
"key":"" (USER_KEY)
}
},
},
"notes": {
<NOTE_KEY>
"title": "",
"description": "",
"rels": {
"files": {
<RELS_FILES_KEY>
"key":"" (FILE_KEY)
}
}
},
"files": {
<FILE_KEY>
"mode": "",
"url": ""
}
}
The application flow is as follows:
The user signs up: a key is created on "users";
Is redirected to "Groups" view, where he should be shown only
groups that have his ID in RELS > USERS, or that has
"isPrivate":"false";
As the user creates a Group, a new group is added with his ID in RELS > USERS;
Entering the Group view, he should only see notes that are in RELS > NOTES for that group.
The rest of the logic follows the same principle, and I believe that if I can get through the first hurdle of understanding the Firebase security rules and applying them to this case, I can get through the rest.
I've tried a couple of rules, but I can't seem to get any feedback at all from the web application, debugging this has been a trial-and-error process, and its not really working.
Could someone help me at least understanding the logic behind it ? I've read all of their tutorials but they all seem very shallow with no deeper examples on complex structures.
Thanks for the help.
EDIT
I've added the debug:true flag to the login (thanks #Kato), but I'm still getting no feedback on the rules. With the rules as below, I still enter the "Groups" view, but get no feedback on the console, and the logged-in user sees groups he shouldn't:
{
"rules": {
"groups": {
".read": "data.child('rels').child('users/' + auth.user).exists()",
".write": "data.child('rels').child('users/' + auth.user).exists()"
}
}
}
As for the rules I've tried, they were countless, but this is the most recent one (still no feedback).
Maybe I'm missing something ?
Thanks again.
Rules cascade. That is, if any rule allows read, then you cannot revoke it later in a nested child. In this way, you can write rules like the following:
"$record": {
// I can write the entire record if I own it
".write": "data.child('owner').val() === auth.uid",
"foo": {
// anybody in my friends list can write to foo, but not anything else in $record
".write": "data.parent().child('friends/'+auth.uid).exists()"
},
"bar": {
// this is superfluous as permissions are only "granted" and never "revoked" by a child
".write": false
}
}
Note how, because I am the owner, I can also write to foo and to bar, even though bar has tried to revoke my read privilege.
So in your case above, your rules declaration lists read: true which allows full read access to the entire repo. Change that to false and you'll see better results.

Many-to-many using Firebase

Lets say I have two kinds of objects users and accounts. Users can have many Accounts and share them with other Users. So AccountA might be available to User1 and User2. While AccountB is only available to User1. So User1 has two accounts, and User2 has one Account.
What is the "firebase" way to structure this?
I was initially thinking that the users could each store an array of accounts they belong to.
users: {
1: {
name: 'Ted',
accounts: [1, 2]
}
2: {
name: 'Frank',
accounts: [1]
}
}
accounts: {
1: {
name: "Checking"
},
2: {
name: "Savings"
}
}
Or the Account would have an array of users.
users: {
1: {
name: 'Ted'
}
2: {
name: 'Frank'
}
}
accounts: {
1: {
name: "Checking",
users: [1, 2]
},
2: {
name: "Savings"
users: [1]
}
}
I'm wondering which way lends itself for me to easily find the accounts for a user so that when Ted logs in I can list the accounts he belong to without having to pull down the entire database. For security I don't want all that data on his machine anyway.
Question #1
Would the security rules take care of this? By that I mean I'm planning to set my security rule such that users can only the accounts they belong to. I'm hoping that if I query for "/accounts" I'll get back only those accounts that the user can access... no?
Question #2
Is there a mechanism for querying for a list of items. Like "/accounts/[1,2]" so that I get back the accounts "/accounts/1" and "/accounts/2"?
Thanks for any information you can share.
Before jumping into specifics, there are a few things you'll want to keep in mind:
You'll typically want to structure your data based on your reading patterns. So if you need to look up all accounts a user is in, you'll need to store that list somewhere. And if you want to look up all users associated with an account, you'll need to store that list somewhere as well. So regarding question #2, Firebase doesn't currently have any generic querying ability (like "/accounts[1,2]"), though this will probably come in the future.
Security rules can't be used to query data either. You're either allowed to read all of the data at a location, or none of it. So regarding your question #1, the security rules won't automatically query /accounts for you.
Security rules don't let you search the contents of an array for an element, but they can check for the existence of a key in an object. Therefore if users 1 and 4 have access to something, you'll probably want to store { 1: true, 4: true } rather than [1, 4].
With that in mind, I'd recommend storing the data like your first example, but without using arrays:
users: {
1: {
name: 'Ted',
accounts: {
1: true,
2: true
}
}
2: {
name: 'Frank',
accounts: {
1: true
}
}
}
accounts: {
1: {
name: "Checking"
},
2: {
name: "Savings"
}
}
This will let you easily get all of the accounts for a particular user. If you need to also be able to go the other direction (all of the users for an account), you'll have to duplicate that information and store it in the accounts as well. E.g.:
accounts: {
1: {
name: "Checking",
users: {
1: true,
2: true
}
},
2: {
name: "Savings",
users: {
1: true
}
}
}
Hope this helps!

Resources