I'm trying to log user activity in my appengine app, which will server both web and mobile users. I'm having a problem deciding which google product I should store the activities to. Two options I'm considering are (Fairly new to both of these):
Google Cloud Datastore as kinds (I think it's a different lingo for tables)
Google Cloud logging as JSON
Price-wise, it seems like Google Cloud logging is cheaper at $0.50/GiB(one time charge for the amount logged) and $0.01/GiB per month, whereas Google Cloud Datastore charge for Read, write, delete + $0.18/GiB/month of storage.
Interface wise, It seems easier to use Cloud Datastore during debugging process. I still haven't gotten a hang of Log explorer yet.
Both have the option to export the data out to CSV.
Which one is recommended for the purpose of activity logging in the long run?
I recommend using Cloud Logging, it is the best for logging any app activity for your application and Google Cloud as it designed for this kind of requirement. For long terms, you can configure your Cloud Logging retention period to retain your Logs up to 3650 days or 10 years. For pricing, the data retention will be priced at $0.01/GiB per month above default retention periods(30 days).
Related
My straight question is: since I am not interested in other Google Services, only Firestore, what is the difference between Firebase Cli and gcloud? Can I do all I need only with Firebase Cli or should I learn also gcloud?
I can easilly see that gcloud can handle more services than Firebase Cli. Also I can see Firebase Cli remenber me a lot the purpose of npm+ng cli when working with NodeJs/Angular (a simple analogy about scaffold/initialize/deploy). Nevertheless, our company will only use Firestore to take advantage of its Realtime Database pushing events to our mobile application.
I use Firebase Cli to type in my Windows command line and create a project in Firebase. I can see the result in Firebase Console and I also can see it in Console Cloud Plataform.
For me, who is total beginner with Google Cloud Plataform, it is a bit confused when prefer one over other. For instance, let's say I want to query data from Firebase Crashlytics from command line instead of logging from my Browser. Is it possible to use either Firebase Cli or gcloud? If both, which one is recommended for some company only interested in Firestore project?
This week I am struggling to figure out how to extract metrics regard my Firestore project and if I don't know what is the recommended tool for reach this task it make even harder to find the first steps. By extracting I don't mean see a beatifull dash in Google Console. I mean consume an endpoint providing me Firestore Metrics or, even better, subscribe my endpoint to Firestore project which will be called every time some issue happens (Other question asking exactly about getting Firestore metrics)
It is confusing and is partly the result of Google's acquisition of Firebase, the creation of what's now called Firestore (which is the unification of the original Firebase backend technology and Google Cloud Datastore), and that Google provides the Firestore service to both its Firebase and Google Cloud developer audiences.
As you've gathered, Google Cloud's CLI (Cloud SDK aka gcloud) and the Firebase CLI overlap in functionality. However, importantly:
both CLIs interact with a shared set of Google services (e.g. Firestore, identity, logging etc.).
there are some Google Cloud services not part of Firebase and vice versa
If you're focused on Firebase, you should be to able to stick with the Firebase CLI and continue to interact with Google's services through this Firebase lens.
If you include Google Cloud Platform services that aren't part of Firebase, you'll need to then consider using gcloud but importantly, you can either:
continue to use the Firebase CLI for the Firebase services;
use gcloud to interact with some (not all!) of the Firebase services (e.g. Firestore).
Over time, I assume (!) the remain discord between these 2 platforms will diminish.
If you consider both platforms as the result of (an ongoing?) unification since the acquisition, hopefully, these discrepancies will be more understandable and less frustrating.
Where you do find inconsistencies and frustrations, ensure Google knows by submitting feature requests and bugs.
I started using Firestore recently (Free plan) in my iOS app, the app is in Production now, and I see huge discrepancy in the number of Reads. App Engine Quota shows 1M reads of 0.05M (free quota) (Cloud Firestore Read Operations), while Firebase Console (Database tab) shows just a one hundred reads for today - 10,000 times difference!
The app can still read the data from the servers (it's not from cache as I tried to delete the app and reinstall), so the Firestore functionality is not limited by exceeding the free quota.
Each app user can read only one document (via listener) with applied security rules, and it reads own document only 3-5 times per session. There are about 50 users so far, so the expected number of reads matches Firebase console.
Is there a known bug in App Engine console?
Is there a way or free tool to understand the source of all those reads?
Which reporting tool is more reliable in general - Firebase Console or App Engine Console?
I appreciate your help guys!
I have an app which uses firebase storage and I am observing an unusual traffic to the storage. The bandwidth usage is unusually high, which shouldn't happen with the usual behavior of the app.
I need to isolate a user or a stored file which causes this unusual bandwidth usage. Is there any way I can get more information about the storage usage, other than the information shown at the Usage tab of firebase console?
Firebase Storage only tracks and shows aggregate data for the usage in your project. It does not tie this usage to specific Firebase Authentication users.
If you need that, consider adding your own tracking for this, for example by logging uploads/downloads to Google Analytics for Firebase.
You can also log to Stack Driver, although it might only be in aggregate there too. For the exact metrics, see the Google Cloud Platform documentation on its storage metrics.
Alternatively you could reach out to Firebase support for personalized help in troubleshooting. But I'm not sure if they have access to the type of data you're looking for.
Google announced Firestore, the new document datastore on the block.
I have been developing an application using Google Cloud Datastore for over six months now and after reading the blog, I feel Firestore seems to be a better choice.
The concept of the alternate collection-document-subcollection looks excellent to me because while designing schema for datastore I was aware I will be unable to query nested fields. Now with firestore subcollections, I get full query capabilities which is a game changer for me (I can get maximum data with minimum queries).
As a counter argument, the flowchart suggests me to use datastore because I do not have any mobile clients.
Will it be a good idea to use Firestore just like Datastore ?
(I will conveniently ignore the mobile client/realtime updates/syncing features!)
Update 2 (01/31/19)
As of today, Cloud Firestore is no longer in Beta and is Generally Available:
https://cloud.google.com/blog/products/databases/announcing-cloud-firestore-general-availability-and-updates
This means that Cloud Datastore is no longer an option for new projects (you can keep using it on existing projects). New projects that want to use the Datastore API can use Cloud Firestore in Datastore mode.
Update 1
As you we have noticed, we've expanded Cloud Firestore since this question was posted.
This means Cloud Firestore now has 2 modes:
The original launch was 'Native mode'
The new launch adds 'Datastore mode'
'Datastore mode' is the 3rd gen of Cloud Datastore. 1st was called Master/Slave Datastore, 2nd was High-Replication Datastore (HRD) that was rebranded as Cloud Datastore in 2013.
The below answer is still largely relevant since both modes are currently mutually exclusive, so you need to pick one or the other.
The main differences are the improves of Cloud Firestore in Datastore mode over Cloud Datastore. The biggest ones are:
Write through-put per entity group now unbounded (was 1 write/second)
Transactions no longer limited to 25 entity groups
All queries now strongly consistent.
Also note Cloud Firestore regardless of mode is beta, so the new Service-Level Agreement (SLA) doesn't go into effect until the product reaches General Availability (GA).
Original Answer
Cloud Datastore (CD) and Cloud Firestore (CF) are similar, however different in significant ways.
CF is mobile-centric with direct from mobile client functionality with the Firebase SDKs and Rules functionality. CD is server-centric with a wider range of server client libraries, as well as some mature frameworks on App Engine Standard that bundle in memcache functionality.
CF has a newer storage layer that is strongly consistency in the same way as Cloud Spanner, however, it's still in beta without an SLA. CD's storage layer is only strongly consistent within entity-groups and eventually consistent across entity-groups, however, it is GA with a 99.95% SLA for the Multi-Region locations.
CF is only available in the US Multi-Region at this time. CD is available Cloud across a dozen locations including places in the Americas, Europe, Asia, and Australia.
CF during beta has a guideline limit of 2500 writes/second while we build experience monitoring and tuning the system prior to GA, whereas CD will happily handle >1M writes/second (please reach out to your account rep first though).
CF and CD's set of query capabilities are overlapping but not the same. Overall CD has a broader set of query capabilities we haven't built in CF yet, so you'd have more flexibility in CD.
Overall, I'd consider this list to see if any of the differences make or break what you're trying to build then pick the DB that fits closest to your needs.
Firestore is the 3rd generation architecture and replacement for Datastore, essentially available in 2 modes: Native mode and Datastore mode.
Documentation regarding the choices: https://cloud.google.com/datastore/docs/firestore-or-datastore
Video overview: https://www.youtube.com/watch?v=SYG-BgXoJFQ
I'd say that Datastore is now a subset of Firestore:
Cloud Firestore is the next major version of Cloud Datastore and a re-branding of the product.
See Choosing between Cloud Firestore and Cloud Datastore
Cloud Firestore can operate in "Datastore mode", making it backwards- compatible with Cloud Datastore. Some time after Cloud Firestore is released for general availability, Google will begin contacting owners of existing Cloud Datastore databases to schedule an automatic upgrade to Cloud Firestore in Datastore mode. See auto upgrade
Google documentation says:
Firestore is the new version of Datastore and removes several
Datastore limitations.
I think cloud firestore also has nodejs client and its not centric to mobile. Actually thats the difference between Firebase realtime database which was mobile-centri vs Cloud Firestore which is anything centric.
I want to test Firebase analytics to monitor my app but i'm a little concerned about the free mode. Am I obligated to use Firebase storage, real time database, hosting and all that? or can I have my own and use Firebase just for analytics?
What I fear is giving up my current infra structure and end up with the app turned off during release because it got too many access or something like that.
You can use Firebase Analytics without using any of the other Firebase features. Firebase Analytics is free and unlimited.