I want to log the following user actions on my Firebase app:
sign in/out
page in/out
timestamp of action
Right now, I use my own function to log actions to the database location "root > user-logs > [user's id]".
Each action is logged as
[time in milliseconds] : [action]
These logs put a lot of data in my database.
However, I won't be accessing data stored at the user-logs locations, so my belief is that this won't lower the speed of read operations at other locations in the database.
Question 1: Is the above belief true?
Question 2: Is there a better way to log customized user actions?
I first thought of creating a csv file in Cloud Storage and appending user actions to the file, but then realized that in order to write to a csv file, I would first have to download it, so I decided that writing to the database would be much faster (and easier).
Thanks.
If you write data to a location that you don't read any data from, then that write operation will not affect operations that read data from somewhere else in your database.
But storing data that you're never going to read is unlikely. Otherwise there probably wouldn't be a reason to store it. More likely you're going to want to read/query this data at some point.
Given the append-only, every-growing nature of your log data, it is unlikely that Firebase will offer the query capabilities that you need at that point. Therefor I'd recommend storing your data in a system that is more tailored towards the use-case: storing lots of data and querying that. A perfect example of such a system is Google's BigQuery.
A common way to get the data into BigQuery is to keep doing what you do now from the client: write it into the database. Then create a Cloud Function that triggers on incoming log data from your database, writes that data to BigQuery, and then deletes it from the database.
With this approach you're only using the Firebase Database for transient storage, and do the heavy lifting in BigQuery.
Related
I am currently working on an iOS App that uses Cloud Firestore from Firebase.
I was wondering: what is the best way (cost, efficiency and security-wise) to upload some data to multiple Firestore documents simultaneously (or almost simultaneously)?
* The data I have to upload consists of the following: there are two users (User A is the user currently using the app, User B is the one whose profile is currently being seen by User A). If User A saves User B's profile, I must upload User B's UID to User A's Firestore Document. Then, I have to increase a counter in User A's Firestore Document. Finally, I must add User A's UID to User B's Firestore Document. - Note that with Firestore Document I mean either a document Field or a document Subcollection.
The choices are:
Upload everything from the client: seems the best method, cost-wise: it doesn't require extra Cloud Functions usage. I would create a Batch Operation and upload all the data from there*. The downside is that the client must be able to access multiple unrelated collections and documents.
Update one document from the client, then update everything else from Cloud Functions: this method is the best one efficiency and security-wise; the client only uploads data to the user's document*, without accessing unrelated collections and documents. Also, the client only has to upload a fraction of the data that it had to upload in the previous method, saving bandwidth and cellular data / WiFi usage. The downside is that the usage of Cloud Functions would increase, eventually resulting in more costs.
Update one document from the client, update the counter* from the client and then update everything else form Cloud Functions: this method is somewhat a hybrid between the first two, as I think that updating the counter from the client is more secure (Cloud Functions' .onWrite trigger may happen twice or more, increasing the counter multiple times?).
My first thought was to go with method 2, as it's far more secured and efficient, but I would like to have someone else's advice too, before "wasting" too much time coding something wrong.
I hope this isn't any kind of duplicate, as I couldn't find anything that answered my question with enough specificity.
Any advice would be much appreciated. Thank you.
I would follow the third approach: updating from the client the current user collections (the saved_profiles collection and the counter field), which are private and only accessible by this user (configure Firestore Security Rules) and updating the other user's collection (users_who_saved_my_profile) with a triggered Cloud Function. As these operations are not controlled by security rules, they can access any part of the database. This way no unnecessary permissions are granted to any user.
I am building an application that has a single collection (itinerary data) that will have many (40,000+) entries. This data needs to be queryable and included in the firestore.
When I attempted to import the data set, I realized that executing so many writes would be costly and use up most of my allowance, so bulk importing this data isn't an option, unless there is a way to do so without executing so many writes.
My mentor floated the idea of serving the itinerary data as a separate API, and using firestore pulling it into firestore on demand. This would spread the burden of writes over time.
I'm curious about my options here, and would like some advice on how to execute.
What would my client side request look like? Would it involve using a cloud function? How do I ensure the data in the firestore is up to date if my API data changes?
Update: Editing the question title/body based on the suggestion.
Firebase store makes everything that is publicly readable also publicly accessible to the browser with a script, so nothing stops any user from just saying db.get('collection') and saving all the data as theirs.
In more traditional db setup where an app's frontend is pulling data from backend, and the user would have to at least go through the extra trouble of tweaking the UI and then scraping the front end to pull more-and-more data (think Twitter load more button).
My question was whether it was possible to limit users from accessing the entire database in a click, while also keeping the data publicly available.
Old:
From what I understand, any user who can see data coming out of a Firebase datastore can also run a query to extract all of that data. That is not desirable when data itself is of any value, and yet Firebase is such an easy to use tool, it's great for pretty much everything else.
Is there a way, or a best practice, for how to structure the data or access rules s.t. users see the data, but can't just run a script to download all of it entirely?
Thanks!
Kato once implemented a simplistic rate limit for writes in Realtime Database security rules: Firebase rate limiting in security rules?. Something similar could be possible in Cloud Firestore rules. But this approach won't work for reads, since you can't update the timestamp at the same time the read is performed.
You can however limit what queries a user can perform on your database. For example, to limit them to reading 50 documents at a time:
allow list: if request.query.limit <= 50;
My goal is to have a firebase cloud function track the upload of three separate files to the same storage bucket. These uploads are preceded by a write to the real time database which would preferably be the trigger for the cloud function to track the uploads.
The context is a user is adding an item to her shopping cart. The data is written to the RTDB and then a custom 3d model and 2 images are copied into a storage bucket. If any of these files don't successfully upload, I need to know that and conduct a rollback of the 3 files in the storage bucket and also remove the entry in the database. I could handle this client side, but that isn't ideal since usually if the uploads fail, its because the connection with the client has failed.
I haven't been able to find any sort of batch add or transaction-type uploads to firebase storage. Sorry for not having any code to show, but I'm not even really sure how to get started on this. Any suggestions would be much appreciated. Thanks!
There are no transactions that cross products like this. Nor are there any transactions offered by Cloud Storage. You're going to have to check errors and manually undo things previously done. Or, have some job that checks for orphaned data and deletes it later.
I need to get a user profile document, which then needs to access two other documents in separate collections, before it returns. At the moment I have implemented this client side but it takes a while. Should I/Can I run this using Cloud Functions, so that I just call one GET and retrieve everything in one go, rather than calling separate get functions sequentially from within my app?
The database retrieval from separate collections would take a similar amount of time whether it's done from the client or Cloud Function.
Collection queries should be very fast on your indexed fields, so probably your problem is the way you are handling asynchronicity. Are you waiting for the result from the first collection before starting the second query? You could dispatch both queries at the same time to cut your waiting time.
You can store all your documents in Firebase Storage and then concatenate the references from the files and download all the documents at the same time, plus you can access them quicker because you can store them into your SD card or internal storage.
Then, if the documents need to be rewritten there is not problem because if you download again from the storage it will auto replace them and the user will still have access to the documents. I tell you this because I'm doing something similar and it's working great!
Edit: As Sujil says, first make an authentication between the user and the database structure with Firebase, so only people logged in or authenticated in your app can read/write files.