I have a question in regards to the capabilities of Firebase in equivalence to MySQL features like:
Events
Triggers
Stored Procedures
In my case I want to migrate off of MySQL to Firebase but I need to know if the usecase can be replicated in Firebase.
My current MySQL DB I have a table that has a column called status which once it gets changed to 'Final' it triggers the execution of a stored procedure to do a calculation on an entire table.
So in other words I would have to be able to add a 'trigger' on the actual firebase data to then perform a 'stored procedure' to calculate something; is this possible with Firebase?!
You now have the capability to use Cloud Functions for Firebase to write code that triggers in response to changes in your Firebase project. There is lots of sample code that illustrates how to respond to writes to a particular location in your database, among many other types of events.
Related
Firebase Firestore: How to monitor read document count by collection?
So first of something similar like question was already asked almost a year ago so dont mark it duplicate cause I need some suggestions in detail.
So here is the scenario,
lets say I have some collections of database and in future I might need to perform some ML on the DB. According to the documents visit.
That is how many times a specific document is visited and for how much time.
I know the mentioned solution above indirectly suggests to perform a read followed by write operation to the database to append the read count every time I visit the database. But it seems this needs to be done from client side
Now if you see, lets say I have some documents and client is only allowed to read the document and not given with access for writing or updating. In this case either I will have to maintain a separate collection specifically to maintain the count, which is of course from client side or else I will have to expose a specific field in the parent document (actual documents from where I am showing the data to clients) to be write enabled and rest remaining field protected.
But fecthing this data from client side sounds an alarm for lot of things and parameters cause I want to collect this data even if the client is not authenticated.
I saw the documentation of cloud functions and it seems there is not trigger function which works as a watch dog for listening if the document is being fetched.
So I want some suggestions on how can we perform this in GCP by creating own custom trigger or hook in a server.
Just a head start will be so usefull.
You cannot keep track of read counts if you are using the Client SDKs. You would have to fetch data from Firestore with some secure env in the middle (Cloud Functions or your own server).
A callable function like this may be useful:
// Returns data for the path passed in data obj
exports.getData = functions.https.onCall(async (data, context) => {
const snapshot = admin.firestore().doc(data.path).get()
//Increment the read count
await admin.firestore().collection("uesrs").doc(context.auth.uid).update({
reads: admin.firestore.FieldValue.increment(1)
})
return snapshot.data()
});
Do note that you are using Firebase Admin SDK in this case which has complete access to all Firebase resources (bypasses all security rules). So you'll need to authorize the user yourself. You can get UID of user calling the function like this: context.auth.uid and then maybe some simple if-else logic will help.
One solution would be to use a Cloud Function in order to read or write from/to Firestore instead of directly interacting with Firestore from you front-end (with one of the Client SDKs).
This way you can keep one or more counters of the number of reads as well as calculate and apply specific access rights and everything is done in the back-end, not in the front-end. With a Callable Cloud Function you can get the user ID of authenticated users out of the box.
Note that by going through a Cloud Function you will loose some advantages of the Client SDKs, for example the ability to use a listener for real-time updates or the possibility to declare access rights through standard security rules. The following article covers the advantages and drawbacks of such approach.
I declare that I am a beginner in using Google Cloud Platform.
I am developing a web application in react using firebase, so all data is saved on firestore.
Now I need to have a relational database, and I am very confused as to which is the best between Cloud SQL and BigQuery.
My idea was to have one part of the data on Cloud SQL and the other part on Firestore.
When an event happens, the data from Cloud SQL and firestore are merged and uploaded to BigQuery for analysis.
Example:
On Firestore I have a product that has an array field where IDs are
stored. These IDs are related to the Database saved on Cloud SQL. When
an order is placed it is added to a collection on Firestore and
appended to the database on BigQuery.
My problem is that from what I have read there is no possibility of autoscaling on Cloud SQL, while on BigQuery it does.
So my question is can you autoscale on CloudSQL?
If it can't be done, is it correct to use BigQuery exclusively?
Is there another solution on GCP that allows you to have a relational database but with autoscaling?
Edit 1
This is the very simplified model of a part of the database on CloudSQL / BigQuery
I'll use a 2/3 inner join query to get all the values I need.
I don't know how to make it non-relational and therefore be able to use firestore without having a large duplication of data, I am open to any kind of advice
Not sure that I understood correctly, but I reckon you would like to get some data (from one data source), combine/process that data with the data from a Firestore collection, and load/stream the result into BigQuery. All of that - is operationally in run time. The question is about the choice of that data source - either a Cloud SQL or a BigQuery.
Am I right that from you point of view the main Cloud SQL drawback - is a lack of scalability (autoscale). And you would like to consider a BigQuery instead of the Cloud SQL due to the 'autoscale'?
It is not clear what is the rate of the request/queries you expect, and where the data is located (any requirements on a global access), so it may be difficult to discuss the situation. Anyway...
Thinking about BigQuery, in my opinion, - this is a great "database" (the best from my point of view), but mainly for analytical purposes... Each query has some 'initial' latency (the query job won't be executed faster than some threshold), which cannot be significantly minimised, and there is no binary indexes in BigQuery tables. It means that your query will take a few seconds (let's assume 3 or more) every time you run it (unless the result is taken from the cache). If the number of requests is significant - it may become expensive (in BigQuery) and expensive in the component, which is used to process that task (i.e. Cloud Function triggered by some event) - as the later has to wait (and do nothting) during the query time.
In addition, BigQuery is very good in loading or steeaming data into it, but not very good in regular data updates inside it - there are plenty of limitations. Thus, depending on your context, it may be not very good idea to maintain operational data in BigQuery.
If I rule out the BigQuery -
Can we sacrifice 'autoscalability' for the Cloud SQL?
Can we use a Firestore collection instead of the Cloud SQL (and sacrifice the 'relational' property?
Can we use Cloud SQl and handle the the amount of data in tables which are used for querying, so there is no delays?
Not sure if I managed to help, but at least I provided some thoughts about the problem.
'Now I need to have a relational database, and I am very confused as to which is the best between Cloud SQL and BigQuery.'
Please be aware that BigQuery cannot be used to substitute a relational Database, and it is oriented on running analytical queries, not for simple CRUD operations and queries (Like in Cloud SQL). That doesn’t mean BigQuery can’t handle normalized data and joins. It absolutely can. It just performs better on denormalized stuff because BigQuery is essentially an OLAP engine. So, denormalize whenever possible (please read here).
You can use read replications to scale Cloud SQL. Read Replica instances allow data from the master instance to be replicated to one or more slaves. This setup can provide increased read throughput. Please see this.
I can't figure out how to change a value of Firebase database and then change it back after a certain amount of time(30 min), doing everything on the server side and not by the actual device date.
I'm assuming i need Firebase functions.
In case i can't do it, is there any other way keeping Firebase as main Database?
I don't really need any code but just the logic behind it.
I would question your data model. Instead of using a boolean, you may want to consider using a timestamp.
For example, if your data model is currently something along the lines of:
Permissions
- user_id
- is_allowed (boolean)
You may want to use this instead:
Permissions
- user_id
- allow_until (timestamp)
You application code can then just check if the current time is earlier than the allow_until timestamp.
There is no logic in the Firebase Realtime Database to automatically change a value after a certain amount of time. You'll typically run such code in Cloud Functions, or in the apps in your client devices.
In both cases you can keep using the Firebase Realtime Database, as you'll just be interacting with that. From Cloud Functions you'll do that through the Admin SDK.
It's a few steps:
Create a Cloud Function that queries the database to find expired items, and changes the value on them. This code uses the Admin SDK for Node.js, but is very similar to what you'd otherwise run in a web client.
Tie that Cloud Function to a cron job that runs every minute or so (depending on how accurate you want the time-out to be). For some options, see Cloud Functions for Firebase trigger on time?
I recommend you also check out these similar questions:
Delete firebase data older than 2 hours
How to delete firebase data after "n" days (doing the same from an Android client)
How to purge old content in firebase realtime database
I want to log the following user actions on my Firebase app:
sign in/out
page in/out
timestamp of action
Right now, I use my own function to log actions to the database location "root > user-logs > [user's id]".
Each action is logged as
[time in milliseconds] : [action]
These logs put a lot of data in my database.
However, I won't be accessing data stored at the user-logs locations, so my belief is that this won't lower the speed of read operations at other locations in the database.
Question 1: Is the above belief true?
Question 2: Is there a better way to log customized user actions?
I first thought of creating a csv file in Cloud Storage and appending user actions to the file, but then realized that in order to write to a csv file, I would first have to download it, so I decided that writing to the database would be much faster (and easier).
Thanks.
If you write data to a location that you don't read any data from, then that write operation will not affect operations that read data from somewhere else in your database.
But storing data that you're never going to read is unlikely. Otherwise there probably wouldn't be a reason to store it. More likely you're going to want to read/query this data at some point.
Given the append-only, every-growing nature of your log data, it is unlikely that Firebase will offer the query capabilities that you need at that point. Therefor I'd recommend storing your data in a system that is more tailored towards the use-case: storing lots of data and querying that. A perfect example of such a system is Google's BigQuery.
A common way to get the data into BigQuery is to keep doing what you do now from the client: write it into the database. Then create a Cloud Function that triggers on incoming log data from your database, writes that data to BigQuery, and then deletes it from the database.
With this approach you're only using the Firebase Database for transient storage, and do the heavy lifting in BigQuery.
I'm planning a web application that requires user auth, plus the ability to display data for the users that is stored in a database. No interaction between the users is needed (yet), however the users should be able create objects and query their "own" objects. For example I list 10 book names (10 book objects), and User A should be able to pick a book and create a new object, call it userNoteObject that contains the name of a choosen book and a short note (that he/she writes).
With a basic pseudo code one book object would look like this:
bookObj = {"id": 1, "name": "book name"}
And the user's note object would be something like this:
userNoteObject = {
"id": 1,
"book_name": "random book name",
"owner_userid": "a1b2c3d",
"note": "some random string"
}
With MySQL I would create three tables, one for the users and one for the userNoteObject-s and another for the bookObj-s. Everytime an user saves a note, I would add it to the table that lists the saved notes. Then I can simply query the notes that belongs to X user based on the user's owner_userid. It's a quite simple functionality.
After reading about the possibilities I've made a decision to go with Firebase Auth (because in the future I might need Android and iOS compatibility) + Google Cloud Datastore or Firebase Realtime Database. However I'm a little bit scared about the Realtime Database of Firebase since I've never worked any DB like it. I also like to be able to modify records manually with something like PhpMyAdmin and I assume Cloud Datastore has a visual interface like that.
I'm familiar with JSON handling and creating JSON files, however the JSON based database is strange for me at the moment. Therefore I'm thinking about that maybe the other option would be a better choice. It's very important that I don't need realtime db features. I would load X number of entries into the table that holds the bookObj-s and sometimes update them. I assume when the user creates an userNoteObject it would be saved quickly with both and after deleting an userNoteObject I could refresh the page close to realtime with Datastore. But the table that holds the book objects must be able to store millions of entries easily.
So the important things:
One db table should be able to handle millions of records easily
Easy as possible querying
Visual interface for the DB (if it's possible)
I don't need realtime features like dynamic game score display/saving
Other info:
I would like to use Angular.js
I'm familiar with Python if it can help in something
So my question is that which database would be better for my needs? At the moment I say Datastore, but I'm totally new with these services so I'm not really against the Realtime Database, but Datastore looks more suitable since it has a visual interface. However I'm also not sure that how would work Datastore with Firebase. If there is a third option like combining both, Realtime Database for the objects save by the user and the static objects for Datastore for example, I would love to hear about it too. My overall goal is to be able to write and query the db easy and fast as it's possible and easily use it with Firebase auth.
UPDATE: I just discovered Firebase's Cloud Firestore, so if it can be more useful I could use it.
If you are going to use Firebase I would recommend you use Cloud Firestore instead of either Cloud Datastore or Firebase realtime database. You get the benefits of a real-time database plus a true document based JSON data store. The one downside is that you don`t have a UI to interact with the data. Datastore has one but its not as robust as say PHPMyAdmin. And since these are NoSQL datastores SQL support is pretty limited.
If you really want a true relational back-end you could try Cloud SQL which is basically MySQL running on Google Servers.
For the Firestore console/UI, see https://firebase.google.com/docs/firestore/using-console. Is that the kind of thing you're looking for?