I have a bunch of buckets in my Google Cloud Platform storage, and I'm not sure where they come from as I have never stored anything in the storage in the past.
What are these buckets for?
Cloud Storage is used as backend of several product. When you build a GCF (Cloud Functions), an APp Engine or you store a Container, the data behind the container created (for Cloud Functions and App Engine) need to reside somewhere. And it's on this special buckets.
You find the artifact buckets EU and US, you should have container in gcr.io and eu.gcr.io, you should have an app engine in US, and deployed at least 2 function, one in europe-west2 and one in us-central1.
Main problem today (and there is side discussion with Google on this): no automatic cleaning is performed on these buckets.
Related
I have a Firebase realtime database project that was originally created on the central1 server (USA).
The problem is that the applications using this database and the Cloud Functions related are all used in Europe (mainly France). Someone in the same situation changed the server to "europe-west1" and noticed a big improvement in term of speed for the Cloud Functions.
So I have several questions : will it really affect the speed? And also, do I have to change both the database and the Cloud Functions locations to make it work smoothly?
It is easy to change the Cloud Functions locations but it seems not possible to change the location of a realtime database once it is created.
I am afraid having a database located in USA and the Cloud Functions running from Europe would be even worse...
According to the Cloud Functions documentation the region recommended to deploy the Cloud Function if they are triggered by or uses a Realtime Database is always US-Central1.
An alternative to provide a better experience for the non-us users could be implementing a small function in the EU zone for all the non database related job that later connects to the Cloud Functions that does all the job related to the database.
I am trying to figure out why firebase storage usage is far above my expectation
I only have few photo files in my Firebase storage, just around 75 photos, 100kb for each photo. but my bytes stored and object counts is way above my expectation as you can see in the image above. in this case, maybe I find the answer from the documentation in here
When you deploy your function's source code to Cloud Functions, that
source is stored in a Cloud Storage bucket. Cloud Build then
automatically builds your code into a container image and pushes that
image to Container Registry. Cloud Functions accesses this image when
it needs to run the container to execute your function.
The process of building the image is entirely automatic and requires
no direct input from you
it is probably because I create a lot of cloud function. thats why the bytes stored and object count in my Firebase Storage is big
now I need to know why storage bandwidth is up to 20.2 GB in a month. I am still developing my app, the user is just me. I don't think I will hit 20.2 GB in a month, because in my Android app, I use cache when showing the image.
I am suspicious, the reason why my storage bandwidth usage is too high is because of cloud function. in August, I perform a lot of firebase deploy to my cloud function. will Cloud Function affect Firebase Storage bandwidth usage ?
I am in Indonesia, my Firebase Storage and Firestore are located in asia-southeast2, but my Cloud Function is located in asia-east2. my cloud function perform some operation to my firestore and images in storage. but still I don't think it will hit 20.2 GB per month
as you can see from the image above, the bandwidth usage is separated into 3 different parts
asia.artifacts.projectID.appspot.com
gcf-sources-5900904-asia-east2
projectID.appspot.com
the asia.artifacts.projectID.appspot.com seems way above the other, it is up to 4.3 GB
thats some information of my problem. so I need to know, will cloud function deployment/operation affect my Firebase Storage Bandwidth usage ?
I need to understand why this is happened, because I am worried that I will have unexpected cost if a lot of users use my app.
I was surprised by the bandwidth usage too.
First, Firebase Changed some policy
After August 17, 2020, each deployment operation will incur small-scale charges for the >storage space used for the function's container. For example, if your functions consume ?>1GB of storage via Container Registry, you'll be billed $0.026 per month. If your >development process depends on deploying functions for testing, you can further minimize >costs by using the Firebase Local Emulator Suite during development.
So the most reasonable explanation is cloud function build during npm install generates bandwidth usage of downloading node_modules and also takes up storage.
I would suggest use local emulator during development as much as possible, but many cases production test is inevitable, so sad.
The part of the documentation you quoted that's relevant here is:
Cloud Functions accesses this image when it needs to run the container to execute your function.
The built images are being accessed by Cloud Functions in order to run your code that's built into the image. Although the documentation doesn't specify, I would guess that every cold start of a function requires a download of the image from the artifacts bucket. This would explain the usage on that bucket.
I use Google's Firebase Storage through the Java Admin API to store files in a Spring Boot application. For security reasons, I create a separate storage bucket for each customer organization. Since I blow away my test environment frequently, I delete and create these storage buckets often. I'm looking for help to get these buckets quicker into Firebase.
Here's what I currently do:
My Spring Boot application creates a bucket with the Google Cloud Storage libraries per the Firebase Storage documentation.
I add the Google Cloud Storage buckets to Firebase by importing them in the Firebase Storage web console. I can import multiple buckets at once.
I apply the default security rules to each Firebase Storage bucket. I can only do this one bucket at a time.
I want to automate steps 2 & 3:
From what I can see in the Firebase docs, I can't do steps 2 and 3 with the Java Admin API.
I can probably automate step 3 through the Firebase CLI tool. For that, I need to set up a "deployment target" that includes all the buckets. I have to add the buckets by name, one by one, and can't use any wildcards here. But my bucket names include database IDs which will be pretty much the same across my environments. So I hope that this deployment target only changes when I add more customer organizations.
Does anybody know of better ways to automate steps 2 and 3?
I decided to change my approach: I now put all customer organizations into one bucket.
It's just too much work to deal with the different buckets now. With one bucket, I just delete all the folders in there which seems simple for now in the Firebase Storage web console.
I'm not sure how my project would behave with hundreds or thousands of buckets. On the contrary, it seems that having thousands of folders within a bucket is a valid use case.
The lack of functionality for Storage buckets both in the Firebase CLI and the Google Cloud SDK indicates to me that you shouldn't have many buckets in your project.
Hi I am using firebase and the firebase functions.
1 day ago I received an email from Google Cloud with the following content:
Starting April 20, 2020, Cloud Functions will use Google Cloud Build, Container Registry, and Google Cloud Storage to build and store your source code, and deploy container image(s) before running them on Google Cloud. You must enable the Cloud Build API for your project(s) to ensure your Cloud Function is built and deployed correctly. Once you enable the API, you may incur charges if your Cloud Build, Container Registry, or Cloud Storage usage exceeds the free tier limits for these products. You can find a list of your projects that are using Cloud Functions and may be affected by this change at the bottom of this email.
I tend to avoid tinkering the Google Cloud (or even logging in there) as I am using the Firebase as an all inclusive solution and so far I did not have the need to login there.
Now the question:
Do I need to follow the instructions and take action, or is this something separate from the Firebase Functions?
The "affected" project that is mentioned on the email is the Firebase project
To be able to use cloud functions after April 20, 2020 you must enable the Cloud Build API because this will be a new deployment framework.
You will not be able to use cloud functions if you do not enable the Cloud Build API after this date.
These changes will apply even for Firebase cloud functions.
I believe you will need to do it only if you are going to deploy Cloud Functions in the future, Cloud Functions already deployed will not be affected.
In case you are worried about billing, Cloud Build provides a free tier where only usage above 120 build-minutes/day will be charged. When your usage is within the free tier, you will not be charged for the Cloud Build portion of Cloud Function deployments. See Cloud Build pricing for more information.
Similarly, Cloud Storage and Container Registry share a free tier where only usage above 5GB-months will be charged. ( * Note : free tier only limited to US regions - US-WEST1, US-CENTRAL1, and US-EAST1 and aggregated over all 3 regions ) For example, if you have a large deployment that uses 100GB of storage, you will only be charged an additional $2.47 for storage/month (based on these particular U.S. regional storage prices).
You can monitor your usage and see whether you are getting close to hitting the free quotas.
This is a best effort from Google to communicate information that is necessary to the user’s continued use of the product or that is considered a necessary legal update and keep customers away from having future issues
Using Cloud Build, Container Registry and Cloud Storage provides the following benefits:
Detailed function build logs will be available in the GCP Console, aiding in debugging and increasing visibility.
The ability to get build time that exceeds the current build quota of 120 build-mins/day.
The ability to view a built container image for your function in Container Registry.
Google announced Firestore, the new document datastore on the block.
I have been developing an application using Google Cloud Datastore for over six months now and after reading the blog, I feel Firestore seems to be a better choice.
The concept of the alternate collection-document-subcollection looks excellent to me because while designing schema for datastore I was aware I will be unable to query nested fields. Now with firestore subcollections, I get full query capabilities which is a game changer for me (I can get maximum data with minimum queries).
As a counter argument, the flowchart suggests me to use datastore because I do not have any mobile clients.
Will it be a good idea to use Firestore just like Datastore ?
(I will conveniently ignore the mobile client/realtime updates/syncing features!)
Update 2 (01/31/19)
As of today, Cloud Firestore is no longer in Beta and is Generally Available:
https://cloud.google.com/blog/products/databases/announcing-cloud-firestore-general-availability-and-updates
This means that Cloud Datastore is no longer an option for new projects (you can keep using it on existing projects). New projects that want to use the Datastore API can use Cloud Firestore in Datastore mode.
Update 1
As you we have noticed, we've expanded Cloud Firestore since this question was posted.
This means Cloud Firestore now has 2 modes:
The original launch was 'Native mode'
The new launch adds 'Datastore mode'
'Datastore mode' is the 3rd gen of Cloud Datastore. 1st was called Master/Slave Datastore, 2nd was High-Replication Datastore (HRD) that was rebranded as Cloud Datastore in 2013.
The below answer is still largely relevant since both modes are currently mutually exclusive, so you need to pick one or the other.
The main differences are the improves of Cloud Firestore in Datastore mode over Cloud Datastore. The biggest ones are:
Write through-put per entity group now unbounded (was 1 write/second)
Transactions no longer limited to 25 entity groups
All queries now strongly consistent.
Also note Cloud Firestore regardless of mode is beta, so the new Service-Level Agreement (SLA) doesn't go into effect until the product reaches General Availability (GA).
Original Answer
Cloud Datastore (CD) and Cloud Firestore (CF) are similar, however different in significant ways.
CF is mobile-centric with direct from mobile client functionality with the Firebase SDKs and Rules functionality. CD is server-centric with a wider range of server client libraries, as well as some mature frameworks on App Engine Standard that bundle in memcache functionality.
CF has a newer storage layer that is strongly consistency in the same way as Cloud Spanner, however, it's still in beta without an SLA. CD's storage layer is only strongly consistent within entity-groups and eventually consistent across entity-groups, however, it is GA with a 99.95% SLA for the Multi-Region locations.
CF is only available in the US Multi-Region at this time. CD is available Cloud across a dozen locations including places in the Americas, Europe, Asia, and Australia.
CF during beta has a guideline limit of 2500 writes/second while we build experience monitoring and tuning the system prior to GA, whereas CD will happily handle >1M writes/second (please reach out to your account rep first though).
CF and CD's set of query capabilities are overlapping but not the same. Overall CD has a broader set of query capabilities we haven't built in CF yet, so you'd have more flexibility in CD.
Overall, I'd consider this list to see if any of the differences make or break what you're trying to build then pick the DB that fits closest to your needs.
Firestore is the 3rd generation architecture and replacement for Datastore, essentially available in 2 modes: Native mode and Datastore mode.
Documentation regarding the choices: https://cloud.google.com/datastore/docs/firestore-or-datastore
Video overview: https://www.youtube.com/watch?v=SYG-BgXoJFQ
I'd say that Datastore is now a subset of Firestore:
Cloud Firestore is the next major version of Cloud Datastore and a re-branding of the product.
See Choosing between Cloud Firestore and Cloud Datastore
Cloud Firestore can operate in "Datastore mode", making it backwards- compatible with Cloud Datastore. Some time after Cloud Firestore is released for general availability, Google will begin contacting owners of existing Cloud Datastore databases to schedule an automatic upgrade to Cloud Firestore in Datastore mode. See auto upgrade
Google documentation says:
Firestore is the new version of Datastore and removes several
Datastore limitations.
I think cloud firestore also has nodejs client and its not centric to mobile. Actually thats the difference between Firebase realtime database which was mobile-centri vs Cloud Firestore which is anything centric.