If you have a lot of entries you would usually have a document that sums the price for each transaction. I could read that document to show the overall value in the overview.
My problem is that the price (price * amount) for each entry changes every 5 min. Because of that I can't save this sum value of all documents.
In order to calculate I would need the price I bought at that point in time and the amount.
Thats basically what im saving for each transaction and to save it again makes no sense anything else.
I can't just do a document with one value that I can update all the time because each transaction is bought for a different price and a different amount of this item.
I could have thousands of transactions and the 20k Firestore limit would not be enough for that summary document.
The view for the transactions shows the latest 50 and is paged, that's fine for Firestore but I can't read all documents for the overall sum price.
Is there no other option as reading all transactions? I thought maybe of using firebase cloud storage and saving there a document with all transactions just for the the summary page.
Related
Let's say i have an multi-restaurant food order app.
I'm storing orders in Firestore as documents.
Each order object/document contains:
total: double
deliveredByUid: str
restaurantId: str
I wanna see anytime during the day, the totals of every Driver to each Restaurant like so:
robert: mcdonalds: 10
kfc: 20
alex: mcdonalds: 35
kfc: 10
What is the best way of calculating the totals of all the orders?
I currently thinking of the following:
The safest and easiest method but expensive: Each time i need to know the totals i just query all the documents in that day and calculate them 1 by 1
Cloud Functions method: Each time an order has been added/removed modify a value in a Realtime database specific child: /totals/driverId/placeId
Manual totals: Each time a driver complete an order and write its id to the order object, make another write to the Realtime database specific child.
Edit: added the whole order object because i was asked to.
What I would most likely do is make sure orders are completely atomic (or as atomic as they can be). Most likely, I'd perform the order on the client within a transaction or batch write (both are atomic) that would not only create this document in question but also update the delivery driver's document by incrementing their running total. Depending on how extensible I wanted to get, I may even create subcollections within the user's document that represented chunks of time if I wanted to be able to record totals by month or year, or whatever. You really want to think this one through now.
The reason I'd advise against your suggested pattern is because it's not atomic. If the operation succeeds on the client, there is no guarantee it will succeed in the cloud. If you make both writes part of the same transaction then they could never be out of sync and you could guarantee that the total will always be accurate.
I'm creating a React firebase website that has a collection of documents that contains a rating from 1 to 10. All of these documents have an author attached. The average rating of all of the author's documents should be calculated and presented.
Here are my current two solutions:
Calculate the average from all the documents with the same author
Add the statistic to the author himself, such that every time the author adds a new document it will update his statistic
My thought process of the second one is such that the website doesn't have to calculate the average rating each time it's requested. Would this be a bad idea, or isn't there a problem in the first place, reading all the documents and calculating in the first place?
Your second approach is in fact a best practice when working with NoSQL databases. If you calculate the average on demand across a dynamic number of documents, the cost of that operation will grow as you add more documents to the database.
For this reason you'll want to calculate all aggregates on write, and store them in the database. With that approach looking up an aggregate value is a simple write.
Also see:
The Firebase documentation on aggregation queries
The Firebase documentation on distributed counters
How to get a count of number of documents in a collection with Cloud Firestore
Leaderboard ranking with Firebase
I have been using Postgres to store time-series sensor data but I am weighing the cost of using Firestore cause I prefer the serverless nature of Firestore. My only concern is the cost of Firestore because I am paying for every read. I want to be able to display this sensor information on my web app. Now, I am taking data every 10 seconds and theres over 400+ sensor points (400 columns per row in my postgres table)
Currently, if a user queries for a week's work of data that's about 60,000 rows of data, but I optimise it by just taking every nth value to "feather" the data. So by taking every 20th row for example, I have reduced the return of the data to 3000 rows which is manageable and still the chart shows a clear trend.
I want to be able to do this in Firestore to save costs, because if a user queries for a week's data, I am paying for 60000 document reads which I can't display all those data points on the web app anyway. I have tried searching for ways to query firestore to take the Nth row of data, but haven't found any concrete solutions.
Does anybody have any recommendation how I can optimise my Firestore costs for time series data or perhaps any other cheap serverless methods to manage this data?
Firestore doesn't offer any way to "feather" data from queries, as you say. What you could do instead is put an integer in each document that describes its "Nth" value, then query for only those "N" that you want.
I am making a Moneymanagement-App where the user can create Transfers for each day.
I am currently listing all the data on the mainscreen. At the moment that doesn't matter because there isn't much data but imagine a user who uses the app several years and tracking all his spendings.
My first thought was to cache all the available Data for that user but that would cause too many unnecessary reads because the user most likely won't need the data from lets say 5 years ago.
So I thought the solution would be to just implement pagination for that screen.
But :
The user can get statistics about his spendinghistory on another screen by selecting a category and a timeperiod.
Currently i am running a query on those parameters each time they change but this will obviously also lead to a lot of unnecessary reads.
So the problem is, if the user chooses to get statistics from 5 years ago, that Data wouldn't exist in the cache so i would still have to run a query for this time period and then end up with a incomplete cache of that period because i only got some of the Data based on the Query.
Would love to hear your thoughts on this. How would you handle it ?
In general: don't run aggregation queries from the client on demand. Instead store aggregated data in the database, and update it as data is written.
So say that you keep some annual totals, such as their balance at the start and end of the year, their total income and spend for that year, probably broken down by categories. That is all information that you could put in a document for each year.
You'd have a structure /users/$uid/totals/$year and you then have the totals in fields in that document. Every time you write a new transaction, you update the totals document for that user for the current year.
If you do this, you'll only need to read the totals document to show totals, and you'll only need to read individual transactions if you want to show individual transactions.
Also see: Is it possible to run aggregation queries in Firestore?
I am looking for a solution for a counter for specific documents in Firebase's Firestore database, but with the added requirement that I need it to keep count of the documents between specific timestamps.
So, for example, we have a bunch of documents in our "videos" collection which hold information related to each video. Each video document also has a subcollection "statistics" which holds a timestamped view for each time the given video is viewed. We do have an aggregate counter on the video document which is incremented each time a view is added to its statistics subcollection, but this only gives the number of total views. We need to be able to query the number of views in the last 7 days as well as the last 30 days.
As you can appreciate, querying all of the view documents in the subcollection using range filters is not a good solution server-side nor client-side because the number of view documents scales into the millions which would be costly.
Does anyone have a solution for a counter for views in the past x days? I just can't think of a way to ensure it's updated accurately without having to query the possibly millions of view documents.
Perhaps a collection called statistics that has a document for each video. Each video has a map of data with timestamps and values. I would bucket every few hours into a single timestamp range.
Perhaps every 3 hours:
Statistics
--- Video1
------ Plays
--------- TimestampRangeStart
--------- Play Count
--------- TimestampRangeStart
--------- Play Count
--------- TimestampRangeStart
--------- Play Count
Then you can query for all videos where TimestampRangeStart > 7 days ago to retrieve a list of all videos that have been played in the last week. This would return the document and then clientside you can just sum up the play counts for all ranges within the last 7 days.
If you're worried about space in the Firestore documents you can consider only adding a map entry if there's a view, so if nobody watches a video then you don't need to worry about an entry. Alternatively you can make the ranges bigger.