We have a business rule that one account can only have 3 projects at any given time.
In order to keep it efficient, we track the number of projects in an "userData" field instead of doing a COUNT query
Consider the following example objects already in DynamoDB:
userData : { createdProjects : 2 }
project1 : { id : 1 }
project2 : { id : 2 }
In order to enforce this rule, we've done the following when creating a project (pseudo code)
in transaction:
putItem(key = "project3", object = { id : 3 })
updateItem(
key = "userData",
expression = "createdProjects = createdProjects + 1"
condition = "createdProjects < 3"
)
Now, if the user tries to create a project at the same time with two computers let's say, will DynamoDB guarantee that he won't be able to create more than 3?
I know there are similar questions like this one, but I wanted to know if this also works in a transaction, because my condition is in another object.
Also, is my pseudo code the best approach? open to other ways
You can use a transaction for this.
Just include the PutItem request and UpdateItem request with the condition in a transaction and either both will complete or none of them.
Transactions are the way to provide this all or nothing behavior.
With the transaction write API, you can group multiple Put, Update, Delete, and ConditionCheck actions. You can then submit the actions as a single TransactWriteItems operation that either succeeds or fails as a unit. The same is true for multiple Get actions, which you can group and submit as a single TransactGetItems operation.
— docs
Related
I can't manage to determine what is the better way of organizing my database for my app :
My users can create items identified by a unique ID.
The queries I need :
- Query 1: Get all the items created by a user
- Query 2 : From the UID of an item, get its creator
My database is organized as following :
Users database
user1 : {
item1_uid,
item2_uid
},
user2 : {
item3_uid
}
Items database
item1_uid : {
title,
description
},
item2_uid : {
title,
description
},
item3_uid : {
title,
description
}
For the query 2, its quite simple but for the query 2, I need to parse all the users database and list all the items Id to see if there is the one I am looking for. It works right now but I'm afraid that it will slow the request time as the database grows.
Should I add in the items data a row with the user id ? If yes the query will be simpler but I heard that I am not supposed to have twice the same data in the database because it can lead to conflicts when adding or removing items.
Should I add in the items data a row with the user id ?
Yes, this is a very common approach in the NoSQL world and is called denormalization. Denormalization is described, in this "famous" post about NoSQL data modeling, as "copying of the same data into multiple documents in order to simplify/optimize query processing or to fit the user’s data into a particular data model". In other words, the main driver of your data model design is the queries you plan to execute.
More concretely you could have an extra field in your item documents, which contain the ID of the creator. You could even have another one with, e.g., the name of the creator: This way, in one query, you can display the items and their creators.
Now, for maintaining these different documents in sync (for example, if you change the name of one user, you want it to be updated in the corresponding items), you can either use a Batched Write to modify several documents in one atomic operation, or rely on one or more Cloud Functions that would detect the changes of the user documents and reflect them in the item documents.
I have a real-time database on firebase which consists of ListFields. Among these fields, one field, participants is a list of strings and two usernames. I want to make a query to firebase database such that it will return the documents in which a particular username is present in the participants list.
The structure of my document is as follows :
I want to make a query such that Firebase returns all the documents in which the participants list consists aniruddh. I am using Flutter with the flutterfire plugins.
Your current data structure makes it easy to find the participants for a conversation. It does however not make it easy to find the conversations for a user.
One alternative data structure that makes this easier is to store the participants in this format:
imgUrls: {},
participants: {
"aniruddh": true,
"trubluvin": true
}
Now you can technically query for the the conversations of a user with something like:
db.child("conversations").orderByChild("participants/aniruddh").equalTo(true)
But this won't scale very well, as you'll need to define an index for each user.
The proper solution is to add a second data structure, known as an inverted index, that allows the look up of conversations for a user. In your case that could look like this:
userConversations: {
"aniruddh": {
"-LxzV5LzP9TH7L6BvV7": true
},
"trubluvin": {
"-LxzV5LzP9TH7L6BvV7": true
}
}
Now you can look up the conversations that a user is part of with a simple read operation. You could expand this data structure to contain more information on each conversation, such as the information you want to display in your list view.
Also see my answer heres:
Firebase query if child of child contains a value (for more explanation on why the queries won't work in your current structure, and why they won't scale in the first structure in my answer).
Best way to manage Chat channels in Firebase (for an alternative way of naming the chat rooms).
Consider I have an Events collection where it has startTimestamp and endTimestamp indicating when the event starts, ends respectively.
How to query in firestore to find out if the Event is live/finished/upcoming?
If both startTimestamp and endTimestamp properties exist in the database and are of type Date and not String or Number, then you can simply use a query to check if a particular date is within the bounds or not.
For example in Android, if you want to check if a particular date is within the bounds, you might think that a query like the one below will work:
eventsRef.whereGreaterThanOrEqualTo("startTimestamp", yourDate)
.whereLessThanOrEqualTo("endTimestamp", yourDate);
But it won't. You'll get an Exception with the following message:
All where filters other than whereEqualTo() must be on the same field. But you have filters on 'startTimestamp' and 'endTimestamp'
The only solutuin you have is to create three separate queries.
Edit:
According to your comment, one query should check if your yourDate is before startTimestamp
eventsRef.whereLessThanOrEqualTo("startTimestamp", yourDate);
If it is, it means it's an upcoming event.
The second one would be to see if it's grater than the startTimestamp:
eventsRef.whereGreaterThanOrEqualTo("startTimestamp", yourDate);
Where we have two cases. One case, you perform a new (third) query to check if the data is less than endTimestamp:
eventsRef.whereLessThanOrEqualTo("endTimestamp", yourDate);
If it is, it means that the event is within the bounds, so it's a live event otherwise is grater than that which means that the event is finished.
To get that data in realtime, you should use a snapshot listener for every query.
Here are the cases to handle this scenario. I'm pretty sure this is a very common problem but didn't find effective solutions for this anywhere.
Solution 1: Have all documents in a single collection called subscribedEvents
As suggested Alex, We need to do for the following status.
Upcoming : currentTimestamp < startTimestamp
Finished : currentTimestamp > endTimestamp
Live : currentTimestamp > startTimestamp in 1st Query and currentTimestamp < endTimestamp in second query.
Problem : I can have lots of documents (nearly 10,000) in subscribedTimestamp and Live condition is not scalable as I can't limit the results while querying. As it needs to be intersected from the two queries, I need to query with out filters.
Solution 2: This is a bit of hack but scalable. Don't have all the documents in a single subCollection. Separate Upcoming events and put those documents in subscribedEvents/others/Upcoming collection.
When a user subscribes, If its an upcoming event, you can directly store in the subscribedEvents/others/Upcoming collection.
Rest of the documents can go directly into subscribedEvents collection.
Upcoming : Query all the documents with a limit filter from subscribedEvents/others/Upcoming collection.
Finished : currentTimestamp > endTimestamp
Live : currentTimestamp < endTimestamp
The benefit with this structure is we can apply limit filter and lots of documents don't need to be read for your query and there will be only one query required for each status.
Now this step needs additionally a cron job to make sure the upcoming events from the upcoming sub-collection are moved back to subscribedEvents.
However, if you have lesser documents, Solution 1 is the way to go. But not in my case.
Hope it helps someone where they have to scale efficiently.
Using firebase real time database i want to move points from user to another but to keep conflicts away ( may user get coins from multi other users at the same time ) i have to use transactions.
My data structure :
{
uid-1:
{
points: 30
},
uid-2:
{
points:60
}
}
So i need two transactions one substracts uid-1 and second increases uid-2
But I'm afraid of that if one transaction success and other one fails .. any sol to revert the operation or update both same time?
There is no secure way to implement conditionality between multiple transactions.
If both operations depend on each other they should be run as a single transaction. That means you have an optimistic lock on the entire "users", but in your current data structure and solution that is required.
An alternative is to not update the balance, but just keep a list of transactions. In that case you can ensure both the addition for the first user and subtraction for the second user are written atomically by using a multi-location update. In JavaScript this would look something like:
ref = firebase.database().ref("users");
var updates = {};
let transactionID = ref.push().key;
updates["uid1/transactions/"+transactionID] = 20;
updates["uid2/transactions/"+transactionID] = -20;
ref.update(updates);
The above write operation will either succeed completely, or fail completely. This ensures your database is always correct.
i'm migrating data from a rails system, and it would be really convenient to assign the migrated objects IDs like post0000000000001, etc.
i've read here
Creating Meteor-friendly id's in Mongo?
that Meteor creates random 17 character strings from
23456789ABCDEFGHJKLMNPQRSTWXYZabcdefghijkmnopqrstuvwxyz
which looks to be chosen to avoid possibly ambiguous characters (omits 1 and I, etc.)
do the IDs need to be random for some reason? are there security implications to being able to guess a Meteor document's ID?! or it is just an easy way of generating unique IDs?
Mongo seems fine with sequential ids:
http://docs.mongodb.org/manual/core/document/#the-id-field
http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
so i would guess this would have to be a Meteor constraint if it exists.
The IDs just need to be unique.
Typically there is an element of order: Such as using integers, or timestamps, or something with sequentiality.
This can't work in Meteor since inserts can come from the client, they may be disconnected for a period, or clients clocks may be off/have varying latency. Also its not possible to know the previous _id (in the case of a sequential _id) at the time an _id is written owing to latency compensation (instant inserts).
The consequence of the lack of order in the DDP protocol is the decision to use entirely random ids. That is not to say you can't use your own _ids.
while there is a risk of a collision with this strategy it is minimal on the order of [number of docs in your collection]/[55^17] * 100 % or nearly impossible. In the event this occurs the client will temporarily insert it and cancel it once the server confirms the error with a Mongo Duplicate Key error.
Also when it comes to security with the other answer. It is not too much of an issue if the _id of the user is known. It is not possible to log in without a valid hashed login token or retrieve any information with it. This applies to the user collection only of course. If you have your own collection an easily guessable URL containing an id as a reference without publish method checks on the eligibility to read the data is a risk the high entropy random ids generated by Meteor can mitigate.
As long as they are unique it should be ok to use your own ids.
I am not an expert, but I suppose Mongo needs a unique ID so when it updates the document, it in fact creates a new version of the document of that same ID.
The real question is - I too whish to know - if we can change the ID without screwing Mongo mechanism and reliability, or we need to create a secondary attribute? (It can make a smaller index too I suppose)?
But me too, I can imagine that security wise, it is better if document IDs are difficult to guess, especially user IDs! Otherwise, could it be easy or possible to fake a user, knowing the ID? Anybody, correct me if I am wrong.
I don't think it's possible and desirable to change ID from Mongo.
But you can easily create a autoincrement ID with http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
I have created a package that does just that and that is configurable.
https://atmospherejs.com/stivaugoin/fluid-refno
var refNo = generateRefNo({
name: 'invoices', // default: 'counter'
prefix: 'I-', // default: ''
size: 5, // default: 5
filling: '0' // default: '0'
});
console.log(refNo); // output: "I-00001"
you now can use refNo to add in your document on Insert
maybe it will help you