Project requirement
A web UI where there is a spinning wheel. The wheel has 1000 possible prizes when the user spin the wheel, they will win that item the cursor is pointing. Each prize has a limit quantity, for this example, let's say each prize has 100 quantity. Thus total number of prizes available is 100,000, 1000 different items, each with 10 quantity. Even though spinning the wheel feels like luck is involved, but because there is a limit quantity, we will figure out which items are available and pre-deteremine what the user will get.
What I've done to get a prize from Firestore
I'm using Firestore to store each items are available. For each doc in prizes, it contains the following:
{
prize: 'PlayStation',
available: true,
}
Thus, I have 1000 docs, containing this schema. If the item has been picked, available will be false. We store each prize as one doc, because some prizes has only 1 quantity, while some other has 50.
On the click on the spin button, will send a request to the backend to get one item. This is the Firebase/Firestore logic I have currently:
try {
const collectionRef = collection(firestore, 'prizes');
const q = query(
collectionRef,
where('available', '==', true), // only get prizes that are available
limit(100) // added to reduce read
);
const querySnapshot = await getDocs(q);
// get available prizes ID
const availableIds = querySnapshot.docs.map((doc) => doc.id);
let data = null;
if (availableIds.length) {
// randomly picked on from the list
const documentId =
availableIds[Math.floor(Math.random() * availableIds.length)];
// to ensure that no 2 person picked the same prize, we use transaction
let res = await runTransaction(firestore, async (transaction) => {
const tDocRef = doc(collection(firestore, 'prizes'), documentId);
const tDoc = await transaction.get(tDocRef);
data = tDoc.data();
if (data.available == true) {
transaction.update(tDocRef, {
available: false,
});
return { data };
}
});
return res; // if we get a prize
}else{
return -1; // if there are no more prizes, send `-1` to frontend, tell user no more stocks
}
if(data==null){
return null; // might have 2 person picked the same prize, frontend auto try again
}
} catch (e) {
return null; // might have 2 person picked the same prize, frontend auto try again
}
How can I improve this?
do you see any issues I could face that I don't see it?
can this handle 10,000 people clicking the spin button at once?
can this ensure that no one get the same prize based on the doc, so to ensure that the total number of prizes for each item is correctly distributed
how can I improve the performance? (reducing the chance of getting null especially when the prizes are getting more limited.
thank you for reading, any ideas/thoughts are welcome
Related
My problem mainly is performance related, I have this code running on the main ElectronJS proccess :
ipcMain.handle('add_product', async (event, args)=>{
return new Promise((resolve, reject)=>{
try {
if(Array.isArray(args)){
args.forEach(prod =>{
const {name,barcode,stock,price,buy_price,image,alert} = prod
const stmt = db.prepare("INSERT INTO products VALUES (?,?,?,?,?,?,?)")
stmt.run(name, barcode, stock, alert, price, buy_price, image)
stmt.finalize()
})
resolve({text : `${args.length} product have been added to database!`})
}else{
// This code execute's only when adding a single product
// It is not relevant to the question
const {name,barcode,stock,price,buy_price,image,alert} = args
const stmt = db.prepare("INSERT INTO products VALUES (?,?,?,?,?,?,?)")
stmt.run(name, barcode, stock, alert, price, buy_price, image)
stmt.finalize()
resolve({text : `Product '${name}' have been saved!`})
}
}catch (error){
reject(error)
}
})
})
It receives an array of objects, each object contains a single product details. Now the above code works and successfully inserts rows inside the database. However when testing it with a substantial data sample (more than 5000 product) the whole application freezes for a couple of seconds while it is saving rows to the database before it becomes responsive again.
The dev stack is :
ElectronJS
ReactJS (using it for the VIEW)
SQLite
What is the optimal and performance driven way to make the application works fatser?
Okay so the way I formulated the query was that it would run 5000 times -once for each product- which significantly slowed the whole application.
I changed the code to :
ipcMain.handle('add_product', async (event, args)=>{
return new Promise((resolve, reject)=>{
try {
if(Array.isArray(args)){
let sql = `INSERT INTO products VALUES`
args.forEach((prod, i) =>{
const {name,barcode,price,buy_price,stock,alert,image} = prod
if(i === args.length - 1){
sql += `('${name}','${barcode}','${price}','${buy_price}','${stock}','${alert}','${image}')`
}else{
sql += `('${name}','${barcode}','${price}','${buy_price}','${stock}','${alert}','${image}'),`
}
})
db.exec(sql, (error)=>{
if(error){
reject(error)
}else{
resolve({text : `${args.length} product have been added to database!`})
}
})
}else{
const {name,barcode,price,buy_price,stock,alert,image} = args
const stmt = db.prepare("INSERT INTO products VALUES (?,?,?,?,?,?,?)")
stmt.run(name, barcode, stock, alert, price, buy_price, image)
stmt.finalize()
resolve({text : `Product '${name}' have been saved!`})
}
}catch (error){
reject(error)
}
})
})
Now the query runs only once (in an asynch fashion to not block the UI) but with all the products and it's much faster.
Objective
User on click a purchase button on the web frontend, it will send a POST request to the backend to create a purchase order. First, it will check the number of available stocks. If available is greater than 0, reduce available by 1 and then create the order.
The setup
Backend (NestJS) queries the Firestore for the latest available value, and reduce available by 1. For debugging, I will return the available value.
let available;
try {
await runTransaction(firestore, async (transaction) => {
const sfDocRef = doc(collection(firestore, 'items_available'), documentId);
const sfDoc = await transaction.get(sfDocRef);
if (!sfDoc.exists()) {
throw 'Document does not exist!';
}
const data = sfDoc.data();
available = data.available;
if(available>0){
transaction.update(sfDocRef, {
available: available-1,
});
}
});
} catch (e) {
console.log('Transaction failed: ', e);
}
return { available };
My stress test setup
Our goal is to see all API requests having different available value, this would mean that Firestore Transactions is reducing the value even though there are multiple requests coming in.
I wrote a simple multi-threaded program that queries the backend's create order API, it will query the available value and return the available value. This program will save the available value returned for each API request.
The stress test performed is about 10 transactions per second, as I have 10 concurrent processes querying the backend. Each process will http.get 20 queries:
const http = require('http');
function call(){
http.get('http://localhost:5000/get_item_available', res => {
let data = [];
res.on('data', chunk => {
data.push(chunk);
});
res.on('end', () => {
console.log('Response: ', Buffer.concat(data).toString());
});
}).on('error', err => {
console.log('Error: ', err.message);
});
}
for (var i=0; i<20; i++){
call();
}
The problem
Unfortunately, the available values I got from the requests contains repeated values, that is, having same available values instead of having unique available values.
What is wrong? Isn't Firestore Transactions meant to handle race conditions? Any suggestions on what I could change to handle multiple requests hitting the server and return a new value for each request?
You have a catch clause to handle when the transaction fails, but then still end up returning a value to the caller return { available }. In that situation you should return an error to the caller.
I have a Firebase web Realtime DB with users, each of whom has a jobs attribute whose value is an object:
{
userid1:
jobs:
guid1: {},
guid2: {},
userid2:
jobs:
guid1: {},
guid2: {},
}
I want to query to get the n users with the most jobs. Is there an orderby trick I can use to order the users by the number of values the given user has in their jobs attribute?
I specifically don't want to store an integer count of the number of jobs each user has because I need to update users' jobs attribute as a part of atomic updates that update other user attributes concurrently and atomically, and I don't believe transactions (like incrementing/decrementing counters) can be a part of those atomic transactions.
Here's an example of the kind of atomic update I'm doing. Note I don't have the user that I'm modifying in memory when I run the following update:
firebase.database().ref('/').update({
[`/users/${user.guid}/pizza`]: true,
[`/users/${user.guid}/jobs/${job.guid}/scheduled`]: true,
})
Any suggestions on patterns that would work with this data would be hugely appreciated!
Realtime Database transactions run on a single node in the JSON tree, so it would be quite difficult to integrate the update of a jobCounter node within your atomic update to several nodes (i.e. to /users/${user.guid}/pizza and /users/${user.guid}/jobs/${job.guid}/scheduled). We would need to update at /users/${user.guid} level and calculate the counter value, etc...
An easier approach is to use a Cloud Function to update a user's jobCounter node each time there is a change to one of the jobs nodes that implies a change in the counter. In other words, if a new job node is added or removed, the counter is updated. If an existing node is only modified, the counter is not updated, since there were no change in the number of jobs.
exports.updateJobsCounter = functions.database.ref('/users/{userId}/jobs')
.onWrite((change, context) => {
if (!change.after.exists()) {
//This is the case when no more jobs exist for this user
const userJobsCounterRef = change.before.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return 0;
});
} else {
if (!change.before.val()) {
//This is the case when the first job is created
const userJobsCounterRef = change.before.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return 1;
});
} else {
const valObjBefore = change.before.val();
const valObjAfter = change.after.val();
const nbrJobsBefore = Object.keys(valObjBefore).length;
const nbrJobsAfter = Object.keys(valObjAfter).length;
if (nbrJobsBefore !== nbrJobsAfter) {
//We update the jobsCounter node
const userJobsCounterRef = change.after.ref.parent.child('jobsCounter');
return userJobsCounterRef.transaction(() => {
return nbrJobsAfter;
});
} else {
//No need to update the jobsCounter node
return null;
}
}
}
});
Is there a way to retrieve the updated value of a document field updated using firestore.FieldValue.increment without asking for the document?
var countersRef = db.collection('system').doc('counters');
await countersRef.update({
nextOrderCode: firebase.firestore.FieldValue.increment(1)
});
// Get the updated nextOrderCode without asking for the document data?
This is not cost related, but for reliability. For example if I want to create a code that increases for each order, there is no guaranty that if >= 2 orders happen at the same time, will have different codes if I read the incremental value right after the doc update resolves, because if >= 2 writes happen before the first read, then at least 2 docs will have the same code even if the nextOrderCode will have proper advance increment.
Update
Possible now, check other answer.
It's not possible. You will have to read the document after the update if you want to know the value.
If you need to control the value of the number to prevent it from being invalid, you will have to use a transaction instead to make sure that the increment will not write an invalid value. FieldValue.increment() would not be a good choice for this case.
We can do it by using Firestore Transactions, like incremental worked before Field.increment feature:
try {
const orderCodesRef = admin.firestore().doc('system/counters/order/codes');
let orderCode = null;
await admin.firestore().runTransaction(async transaction => {
const orderCodesDoc = await transaction.get(orderCodesRef);
if(!orderCodesDoc.exists) {
throw { reason: 'no-order-codes-doc' };
}
let { next } = orderCodesDoc.data();
orderCode = next++;
transaction.update(orderCodesRef, { next });
});
if(orderCode !== null) {
newOrder.code = orderCode;
const orderRef = await admin.firestore().collection('orders').add(newOrder);
return success({ orderId: orderRef.id });
} else {
return fail('no-order-code-result');
}
} catch(error) {
console.error('commitOrder::ERROR', error);
throw errors.CantWriteDatabase({ error });
}
Had the same question and looks like Firestore Python client
doc_ref.update() returns WriteResult that has transform_results attribute with the updated field value
I don't get the concept of limits for query/scan in DynamoDb.
According to the docs:
A single Query operation can retrieve a maximum of 1 MB of data.This
limit applies before any FilterExpression is applied to the results.
Let's say I have 10k items, 250kb per item, all of them fit query params.
If I run a simple query, I get only 4 items?
If I use ProjectionExpression to retrieve only single attribute (1kb
in size), will I get 1k items?
If I only need to count items (select: 'COUNT'), will it count all
items (10k)?
If I run a simple query, I get only 4 items?
Yes
If I use ProjectionExpression to retrieve only single attribute (1kb in size), will I get 1k items?
No, filterexpressions and projectexpressions are applied after the query has completed. So you still get 4 items.
If I only need to count items (select: 'COUNT'), will it count all items (10k)?
No, still just 4
The thing that you are probably missing here is that you can still get all 10k results, or the 10k count, you just need to get the results in pages. Some details here. Basically when you complete your query, check the LastEvaluatedKey attribute, and if its not empty, get the next set of results. Repeat this until the attribute is empty and you know you have all the results.
EDIT: I should say some of the SDKs abstract this away for you. For example the Java SDK has query and queryPage, where query will go back to the server multiple times to get the full result set for you (i.e. in your case, give you the full 10k results).
For any operation that returns items, you can request a subset of attributes to retrieve; however, doing so has no impact on the item size calculations. In addition, Query and Scan can return item counts instead of attribute values. Getting the count of items uses the same quantity of read capacity units and is subject to the same item size calculations. This is because DynamoDB has to read each item in order to increment the count.
Managing Throughput Settings on Provisioned Tables
Great explanation by #f-so-k.
This is how I am handling the query.
import AWS from 'aws-sdk';
async function loopQuery(params) {
let keepGoing = true;
let result = null;
while (keepGoing) {
let newParams = params;
if (result && result.LastEvaluatedKey) {
newParams = {
...params,
ExclusiveStartKey: result.LastEvaluatedKey,
};
}
result = await AWS.query(newParams).promise();
if (result.count > 0 || !result.LastEvaluatedKey) {
keepGoing = false;
}
}
return result;
}
const params = {
TableName: user,
IndexName: 'userOrder',
KeyConditionExpression: 'un=:n',
ExpressionAttributeValues: {
':n': {
S: name,
},
},
ConsistentRead: false,
ReturnConsumedCapacity: 'NONE',
ProjectionExpression: ALL,
};
const result = await loopQuery(params);
Edit:
import AWS from 'aws-sdk';
async function loopQuery(params) {
let keepGoing = true;
let result = null;
let list = [];
while (keepGoing) {
let newParams = params;
if (result && result.LastEvaluatedKey) {
newParams = {
...params,
ExclusiveStartKey: result.LastEvaluatedKey,
};
}
result = await AWS.query(newParams).promise();
if (result.count > 0 || !result.LastEvaluatedKey) {
keepGoing = false;
list = [...list, ...result]
}
}
return list;
}
const params = {
TableName: user,
IndexName: 'userOrder',
KeyConditionExpression: 'un=:n',
ExpressionAttributeValues: {
':n': {
S: name,
},
},
ConsistentRead: false,
ReturnConsumedCapacity: 'NONE',
ProjectionExpression: ALL,
};
const result = await loopQuery(params);