Firebase security rules to check unique value of a child #AskFirebase - firebase

The structure of my firebase database is as shown above. How do I ensure the urls are unique and there are no duplicates? Since, these are urls I cannot use them directly as paths, I am forced to use them as values. So solutions like this won't work.

If you want something to be unique within the Firebase Database, you should store it as a key. That automatically guarantees uniqueness.
As you noted, some characters cannot be used in a key. In that case you'll need to encode the value to allow it in a key and ensure you don't lose the information that makes the value unique. A very simple case of this is when someone wants to store a unique email address in the database. Since a key cannot contain . characters, we need to encode that. A common encoding for this is to replace the . with a ,:
users: {
"uidOfPuf": {
name: "Frank van Puffelen",
email: "puf#firebaseui.com"
}
},
emailAddresses: {
"puf#firebaseui,com": "uidOfPuf"
}
Using a , is especially handy when it comes to email addresses, since an email address cannot contain a ,.
But in general all that matters is that the encoded value is "reasonably guaranteed to be unique" and that you still store the actual value somewhere too (such as the /users/$uid/email above).
For encoding URLs, I'd simply start with stripping all illegal characters:
var url = "http://stackoverflow.com/questions/39149216/firebase-security-rules-to-check-unique-value-of-a-child-askfirebase";
ref.child(url.replace(/[\.\/]/g, '')).set(url);
Stores:
"http:stackoverflowcomquestions39149216firebase-security-rules-to-check-unique-value-of-a-child-askfirebase": "http://stackoverflow.com/questions/39149216/firebase-security-rules-to-check-unique-value-of-a-child-askfirebase"
Update: I'm considering if using a simply hashcode for the key, which leads to more reasonably length keys:
// from http://stackoverflow.com/questions/7616461/generate-a-hash-from-string-in-javascript-jquery
String.prototype.hashCode = function(){
var hash = 0;
if (this.length == 0) return hash;
for (i = 0; i < this.length; i++) {
char = this.charCodeAt(i);
hash = ((hash<<5)-hash)+char;
hash = hash & hash; // Convert to 32bit integer
}
return hash;
}
var url = "http://stackoverflow.com/questions/39149216/firebase-security-rules-to-check-unique-value-of-a-child-askfirebase";
ref.child(url.hashCode()).set(url);
Leads to:
20397229: "http://stackoverflow.com/questions/39149216/firebase-security-rules-to-check-unique-value-of-a-child-askfirebase"

Related

What variables are available for use as "wildcards" in a Firestore Rules Match Statement?

The syntax for specifying the syntax of a rule that you might want to declare in order to protect documents in a Firestore collection are specified at https://firebase.google.com/docs/reference/security/storage. But this describes the wider context of Google Cloud storage and consequently, when you're trying to use these specifically for Firestore applications, the documentation seems a bit opaque.
The match statement used to specify the subset of documents in a collection myCollection to which the rule is to be applied typically takes the form:
match myCollection/{wildcard}
My question is "what variables are available for use as wildcards?". According to the Google document referenced above "A wildcard variable is declared in a path by wrapping a variable in curly braces: {variable}. This variable is accessible within the match statement as a string."
I now understand that if myCollection contains a field called email for example, a match statement of the form
match myCollection/{email}
will allow me to specify a rule to be applied to a document whose documentId is the value specified by the email field. In this case, a rule to allow only logged-in users to read their own record in myCollection would take the form:
match myCollection/{email}
allow read : if request.auth != null && request.auth.token.email == email;
where email is the user's email address filed in myCollection
But supposing that email isn't being used as the unique key to identify a myCollection record - ie that other fields identify documents and a user may have many records tagged with their email within myCollection? This is surely the more likely case but the Google documentation doesn't seem to give any clues to how you would specify a match to cover this situation.
A previous stackoverflow question at How write Firestore rule to match email field in document has directed me towards the keyword doc. This has allowed me to specify a more general rule that can be applied to all the records in a collection with a particular email value:
match /myCollection/{doc} {
allow read : if request.auth != null &&
request.auth.token.email == resource.data.email;
}
There are two concepts here that are new to me (at least in the context of Firestore rules): first the keyword doc and second the resource.data.email construct that gives my rule access to document fields. Can anybody tell me where these constructs are documented, and also if there are any more "wrinkles" to the "wildcard" concept (I already know about the documents=** one).
match myCollection/{wildcard} - This rule tells firebase to match all documents in the myCollection collection. Essentially, it says match all documents that match this path myCollection/*
wildcard above is simply a variable that holds the document path (the asterisks path). This variable can be anything (apart from the reserved strings). If this is your path:
'myCollection/tests', then wildcard variable is holds 'tests';
'myCollection/test#email.com', then wildcard variable holds 'test#email.com'
wildcard is simply a place holder for anything that matches this path: myCollection/*. Instead of wildcard, you can use anything e.g email (myCollection/{email} and email here becomes the variable that holds the document path).
resource.data is the data being fetched. So if your data has an 'email' field, then resource.data.email always holds the email field data (regardless of what the wildcard variable is). Do not use reserved strings e.g 'resource' as the wildcard variable.
what variables are available for use as wildcards?
Any variable you put in curly braces becomes the wildcard variable. It simply holds the document path. It is not a special string.
UPDATE #MartinJ.
For autogenerated ids. Lets assume this is your doc path myCollection/1s6SUqi8M6sC6DZdTKjt, (1s6SUqi8M6sC6DZdTKjt is autogenerated)
match /myCollection/{email} {
allow read : if email == '1s6SUqi8M6sC6DZdTKjt'; // true because
// email variable holds the value '1s6SUqi8M6sC6DZdTKjt'
}
match /myCollection/{doc} {
allow read : if doc == '1s6SUqi8M6sC6DZdTKjt'; // true because
// doc variable holds the value '1s6SUqi8M6sC6DZdTKjt'
}
match /myCollection/{stackoverflow} {
allow read : if stackoverflow == '1s6SUqi8M6sC6DZdTKjt'; // true
// because stackoverflow variable holds the value '1s6SUqi8M6sC6DZdTKjt'
}
// It does not matter what we use as wildcard variable (we can even
// use MartinJ), it will always hold the value of '1s6SUqi8M6sC6DZdTKjt'
// (doc id if the path matches).
// As #Frank said, these are simply variables, not keywords. You
// can call it anything and it will have the value of doc path.
}
if this is your document path sales/2K5CwGU9ZWwNoO9efyuv/payments/40ciObsicEntBZParkon (autogenerated document ids)
match /sales/{doc=**} {
allow write: if doc == '2K5CwGU9ZWwNoO9efyuv/payments/40ciObsicEntBZParkon';
// this will be true because doc=** (**with asterisks**) matches
// everything after sales
}
match /sales/{saleId} {
match /payment/{paymentId} {
allow write: if saleId == '2K5CwGU9ZWwNoO9efyuv' && paymentId == '40ciObsicEntBZParkon';
// this will be true because saleId holds the value
// '2K5CwGU9ZWwNoO9efyuv' and paymentId holds this value '40ciObsicEntBZParkon'
}
}

How to avoid store same values on DB

Considering my firebase DB:
How I ensure the email (or other field) uniqueness at firebase side (using rules)?
Thank you
I suggest you in stead of using the unique id generated by the push() method, to use the email address. Because Firebase does not accept symbols as . in the key, you'll need to encode the email address like this:
name#email.com -> name#email,com
As you probably see, the . symbol is changed with ,. To encode and decode the email address, i suggest you use this methods:
static String encodeUserEmail(String userEmail) {
return userEmail.replace(".", ",");
}
static String decodeUserEmail(String userEmail) {
return userEmail.replace(",", ".");
}
Than to verify the uniqueness of your users, just put a listener on the users node and use the exists() method on the dataSnapshot.
Hope it helps.

How to set auto increment field in DynamoDB? [duplicate]

I am new to dynamodb. I want to auto increment id value when I use putitem with dynamodb.
Is possible to do that?
This is anti-pattern in DynamoDB which is build to scale across many partitions/shards/servers. DynamoDB does not support auto-increment primary keys due to scaling limitations and cannot be guaranteed across multiple servers.
Better option is to assemble primary key from multiple indices. Primary key can be up to 2048 bytes. There are few options:
Use UUID as your key - possibly time based UUID which makes it unique, evenly distributed and carries time value
Use randomly generated number or timestamp + random (possibly bit-shifting) like: ts << 12 + random_number
Use another service or DynamoDB itself to generate incremental unique id (requires extra call)
Following code will auto-increment counter in DynamoDB and then you can use it as primary key.
var documentClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: 'sampletable',
Key: { HashKey : 'counters' },
UpdateExpression: 'ADD #a :x',
ExpressionAttributeNames: {'#a' : "counter_field"},
ExpressionAttributeValues: {':x' : 1},
ReturnValues: "UPDATED_NEW" // ensures you get value back
};
documentClient.update(params, function(err, data) {});
// once you get new value, use it as your primary key
My personal favorite is using timestamp + random inspired by Instagram's Sharding ID generation at http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
Following function will generate id for a specific shard (provided as parameter). This way you can have unique key, which is assembled from timestamp, shard no. and some randomness (0-512).
var CUSTOMEPOCH = 1300000000000; // artificial epoch
function generateRowId(shardId /* range 0-64 for shard/slot */) {
var ts = new Date().getTime() - CUSTOMEPOCH; // limit to recent
var randid = Math.floor(Math.random() * 512);
ts = (ts * 64); // bit-shift << 6
ts = ts + shardId;
return (ts * 512) + randid;
}
var newPrimaryHashKey = "obj_name:" + generateRowId(4);
// output is: "obj_name:8055517407349240"
DynamoDB doesn't provide this out of the box. You can generate something in your application such as UUIDs that "should" be unique enough for most systems.
I noticed you were using Node.js (I removed your tag). Here is a library that provides UUID functionality: node-uuid
Example from README
var uuid = require('node-uuid');
var uuid1 = uuid.v1();
var uuid2 = uuid.v1({node:[0x01,0x23,0x45,0x67,0x89,0xab]});
var uuid3 = uuid.v1({node:[0, 0, 0, 0, 0, 0]})
var uuid4 = uuid.v4();
var uuid5 = uuid.v4();
You probably can use AtomicCounters.
With AtomicCounters, you can use the UpdateItem operation to implement
an atomic counter—a numeric attribute that is incremented,
unconditionally, without interfering with other write requests. (All
write requests are applied in the order in which they were received.)
With an atomic counter, the updates are not idempotent. In other
words, the numeric value increments each time you call UpdateItem.
You might use an atomic counter to track the number of visitors to a
website. In this case, your application would increment a numeric
value, regardless of its current value. If an UpdateItem operation
fails, the application could simply retry the operation. This would
risk updating the counter twice, but you could probably tolerate a
slight overcounting or undercounting of website visitors.
Came across a similar issue, where I required auto-incrementing primary key in my table. We could use some randomization techniques to generate a random key and store it using that. But it won't be in a incremental fashion.
If you require something in incremental fashion, you can use Unix Time as your primary key. Not assuring, that you can get a accurate incrementation(one-by-one), but yes every record you put, it would be in incremental fashion, with respect to the difference in how much time each record in inserted in.
Not a complete solution, if you don't want to read the entire table and get it's last id and then increment it.
Following is the code for inserting a record in DynamoDB using NodeJS:
.
.
const params = {
TableName: RANDOM_TABLE,
Item: {
ip: this.ip,
id: new Date().getTime()
}
}
dynamoDb.put(params, (error, result) => {
console.log(error, result);
});
.
.
If you are using NoSQL Dynamo DB then using Dynamoose, you can easily set default unique id, here is the simple user create example
// User.modal.js
const dynamoose = require("dynamoose");
const { v4: uuidv4 } = require("uuid");
const userSchema = new dynamoose.Schema(
{
id: {
type: String,
hashKey: true,
},
displayName: String,
firstName: String,
lastName: String,
},
{ timestamps: true },
);
const User = dynamoose.model("User", userSchema);
module.exports = User;
// User.controller.js
exports.create = async (req, res) => {
const user = new User({ id: uuidv4(), ...req.body }); // set unique id
const [err, response] = await to(user.save());
if (err) {
return badRes(res, err);
}
return goodRes(res, reponse);
};
Update for 2022 :
I was looking for the same issue and came across following research.
DynamoDB still doesn't support auto-increment of primary keys.
https://aws.amazon.com/blogs/database/simulating-amazon-dynamodb-unique-constraints-using-transactions/
Also the package node-uuid is now deprecated. They recommend we use uuid package instead that creates RFC4122 compliant UUID's.
npm install uuid
import { v4 as uuidv4 } from 'uuid';
uuidv4(); // ⇨ '9b1deb4d-3b7d-4bad-9bdd-2b0d7b3dcb6d'
For Java developers, there is the DynamoDBMapper, which is a simple ORM. This supports the DynamoDBAutoGeneratedKey annotation. It doesn't increment a numeric value like a typical "Long id", but rather generates a UUID like other answers here suggest. If you're mapping classes as you would with Hibernate, GORM, etc., this is more natural with less code.
I see no caveats in the docs about scaling issues. And it eliminates the issues with under or over-counting as you have with the auto-incremented numeric values (which the docs do call out).

need help understanding how to use dictionaries

I am a novice with regards to the swift 3 programming language though I am familiar basically with C++.
I am trying to learn to use dictionaries in Swift 3; which I believe are similar to hashes in C++. I have a dictionary with several key:value pairs in it.
I want to take a certain, single key (which I won't know in advance) and extract from that dictionary the corresponding value.
I know there will be a single key with that name, although the same value will be associated with keys of different names.
After extracting that value from the key:value pair of that dictionary then I want to store that single value in a variable as a string.
What type of code could do that?
I found some code that seems it might be helpful but I'm not sure and I'm not sure too how to use that code(how to write it actually) to make it perform as I wish.
extension Dictionary where Value: Equatable {
func someKeyFor(value: Value) -> Key? {
guard let index = indexOf({ $0.1 == value }) else {
return nil
}
return self[index].0
}
}
This is straight-forward dictionary access; no need to use extensions.
var myDictionary = [String:String]()
// At some point strings are put into the dictionary
// e.g. myDictionary["SomeKey"] = "SomeString"
// then you can say
if let someString = myDictionary[key] {
// You can now do something with someString
}
Keys and objects don't have to be strings, of course
I would suggest you read the Swift book from Apple in iBooks and try things out in in a Swift Playground in Xcode.

Good way to replace invalid characters in firebase keys?

My use case is saving a user's info. When I try to save data to Firebase using the user's email address as a key, Firebase throws the following error:
Error: Invalid key e#e.ee (cannot contain .$[]#)
So, apparently, I cannot index user info by their email. What is the best practice to replace the .?
I've had success changing the . to a - but that won't cut it since some email's have -s in the address.
Currently, I'm using
var cleanEmail = email.replace('.','`');
but there are likely going to be conflicts down the line with this.
In the email address, replace the dot . with a comma ,. This pattern is best practice.
The comma , is not an allowable character in email addresses but it is allowable in a Firebase key. Symmetrically, the dot . is an allowable character in email addresses but it is not allowable in a Firebase key. So direct substitution will solve your problem. You can index email addresses without looping.
You also have another issue.
const cleanEmail = email.replace('.',','); // only replaces first dot
will only replace the first dot . But email addresses can have multiple dots. To replace all the dots, use a regular expression.
const cleanEmail = email.replace(/\./g, ','); // replaces all dots
Or alternatively, you could also use the split() - join() pattern to replace all dots.
const cleanEmail = email.split('.').join(','); // also replaces all dots
We've dealt with this issue many times and while on the surface it seems like using an email as a key is a simple solution, it leads to a lot of other issues: having to clean/parse the email so it can actually be used. What if the email changes?
We have found that changing the format of how the data is stored is a better path. Suppose you just need to store one thing, the user name.
john#somecompany.com: "John Smith"
changing it to
randomly_generated_node_name
email: "john#somecompany.com"
first: "John"
last: "Smith"
The randomly_generated_node_name is a string that Firebase can generate via childByAutoId, or really any type of reference that is not tied directly to the data.
This offers a lot of flexibility: you can now change the persons last name - say if they get married. Or change their email. You could add an 'index' child 0, 1, 2 etc that could be used for sorting. The data can be queried for any child data. All because the randomly_generated_node_name is a static reference to the variable child data within the node.
It also allows you to expand the data in the future without altering the existing data. Add address, favorite food, an index for sorting etc.
Edit: a Firebase query for email in ObjC:
//references all of the users ordered by email
FQuery *allUsers = [myUsersRef queryOrderedByChild:#"email"];
//ref the user with this email
FQuery *thisSpecificUser = [allUsers queryEqualToValue:#“john#somecompany.com”];
//load the user with this email
[thisSpecificUser observeEventType:FEventTypeChildAdded withBlock:^(FDataSnapshot *snapshot) {
//do something with this user
}];
I can think of two major ways to solve this issue:
Encode/Decode function
Because of the limited set of characters allowed in a Firebase key, a solution is to transform the key into an valid format (encode). Then have an inverse function (decode) to transform the encoded key back as the original key.
A general encode/decode function might be transforming the original key into bytes, then converting them to a hexadecimal representation. But the size of the key might be an issue.
Let's say you want to store users using the e-mail as key:
# path: /users/{email} is User;
/users/alice#email.com: {
name: "Alice",
email: "alice#email.com"
}
The example above doesn't work because of the dot in the path. So we use the encode function to transform the key into a valid format. alice#email.com in hexadecimal is 616c69636540656d61696c2e636f6d, then:
# path: /users/{hex(email)} is User;
/users/616c69636540656d61696c2e636f6d: {
name: "Alice",
email: "alice#email.com"
}
Any client can access that resource as long as they share the same hex function.
Edit: Base64 can also be used to encode/decode the key. May be more efficient than hexadecimals, but there are many different implementations. If clients doesn't share the exact same implementation, then they will not work properly.
Specialized functions (ex. that handles e-mails only) can also be used. But be sure to handle all the edge cases.
Encode function with original key stored
Doing one way transformation of the key is a lot easier. So, instead of using a decode function, just store the original key in the database.
A good encode function for this case is the SHA-256 algorithm. It's a common algorithm with implementations in many platforms. And the chances of collisions are very slim.
The previous example with SHA-256 becomes like this:
# path: /users/{sha256(email)} is User;
/users/55bf4952e2308638427d0c28891b31b8cd3a88d1610b81f0a605da25fd9c351a: {
name: "Alice",
email: "alice#email.com"
}
Any client with the original key (the e-mail) can find this entry, because the encode function is known (it is known). And, even if the key gets bigger, the size of the SHA-256 will always be the same, therefore, guaranteed to be a valid Firebase key.
I am using the following code for converting email to hash and then using the hash as key in firebase
public class HashingUtils {
public HashingUtils() {
}
//generate 256 bits hash using SHA-256
public String generateHashkeySHA_256(String email){
String result = null;
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(email.getBytes("UTF-8"));
return byteToHex(hash); // make it printable
}catch(Exception ex) {
ex.printStackTrace();
}
return result;
}
//generate 160bits hash using SHA-1
public String generateHashkeySHA_1(String email){
String result = null;
try {
MessageDigest digest = MessageDigest.getInstance("SHA-1");
byte[] hash = digest.digest(email.getBytes("UTF-8"));
return byteToHex(hash); // make it printable
}catch(Exception ex) {
ex.printStackTrace();
}
return result;
}
public String byteToHex(byte[] bytes) {
Formatter formatter = new Formatter();
for (byte b : bytes) {
formatter.format("%02x", b);
}
String hex = formatter.toString();
return hex;
}
}
code for adding the user to firebase
public void addUser(User user) {
Log.d(TAG, "addUser: ");
DatabaseReference userRef= database.getReference("User");
if(!TextUtils.isEmpty(user.getEmailId())){
String hashEmailId= hashingUtils.generateHashkeySHA_256(user.getEmailId());
Log.d(TAG, "addUser: hashEmailId"+hashEmailId);
userRef.child(hashEmailId).setValue(user);
}
else {
Log.d(TAG,"addUser: empty emailId");
}
}

Resources