Firebase storage cost explosion (or how to prevent it) [duplicate] - firebase

Some functions in the Google Developers Console, like the Analytics API, are free until you reach a quota. Other functions, like Google Cloud Storage, create costs from the first click.
When I upload a file under https://console.developers.google.com/ > Storage > Cloud Storage > Storage Browser and I make this file publicly available, I pay about $0.12 per GB traffic.
But theoretically the traffic to this link could explode, e.g. because of sudden popularity. Therefore I would like to set something like a daily or monthly cost limit.
Q: How do I protect myself from overly high costs in the Google Developers Console?

You cannot. I asked Google about this, here's their response, from May 7 2016:
(GCE = Google cloud engine. No spending limits.
GAE = Google app engine — yes it has spending limits.)
... you are eligible for support on ... only ...
... [various helpful links] ...
That been said, at the moment there is no a feature that allows you to
configure a limited budget on GCE. This feature is certainly available
for GAE [1]. As you mentioned in your comments, you either can totally
shut down your VMs (will depend on your use case) or set the VMs to
send you alerts if they reach a certain traffic limit [2].
Sincerely,
Someone's first name
Technical Solutions Representative
Google Cloud Platform
[1] https://cloud.google.com/appengine/docs/quotas
[2] https://cloud.google.com/monitoring/support/notification-options
#wmdry, you wrote: "traffic to this link could explode" — I'm afraid of this too. That's why I asked Google about this. And I'm planning to avoid Google's CDN because of this, and use another CDN provider instead, which has spending limits. Because, unlike Nginx, I don't see any way for me to rate limit / throttle Google's CDN.
I do plan to use GCE (Google Cloud Engine) though. Therefore, right now I'm reading about how to rate limit my Nginx server. Because if I just configure Nginx correctly, then those $0.12 / GB you mentioned, cannot possible explode to ... like $10k in a month? What if Google sends a $10k bill when I'm back from an a few week's vacation, just because of my hobby project and a few people downloading a 1 MB movie over and over again forever (because: evil). Hmm, & the bigger & faster my servers, the higher the risk.
I hope Google will add spending limits, because I did want to use Google's CDN.
Update 2020: Apparently this does bite people from time to time — look here:
"Burnt $72k testing Firebase and Cloud Run and almost went bankrupt", Dec 08, 2020, https://news.ycombinator.com/item?id=25372336,
In that case, they could contact Google and in the end didn't need to pay.

As of July 2017 you can set budgets that send notifications via email but do not cap spending:
To set an alert-only budget, which will not cap spending:
Go to the Cloud Platform Console.
Open the console left side menu and click Billing
If you have more than one billing account, click the billing account name.
On the left, click Budgets & alerts.
Official help page: https://support.google.com/cloud/answer/6293540?hl=en

I found that Google's documentation now provides two methods to actually limit the cost of a GCP project. It involves the following setup:
Create a Cloud Function that checks the cost against the budget, and carries out a certain action if the cost exceeds the budget. Google's Documentation provides a sample code snip that can either shutdown all VM instances in a Project or disable the billing for a project. Shutting down all VMs would stop all VM-related cost but you get to keep your data (and still have to pay for the storage). Disabling the billing for a project would effectively zap all cost-related activities and you could lose data. You can name the Cloud Function "budget-enforcer".
The Google code snip as provided above has a hard coded ZONE variable. Remember to change it to match your zone!
Create a Service Account to run the Cloud Function "budget-enforcer". For shutting down VMs, the Service Account would need role "Compute Instance Admin (v1)". For disabling billing on a project, the Service Account would need role "Project Billing Manager".
Set a Topic for the Cloud Function (I call mine "proj-name-stop-vm" and "proj-name-disable-bill").
Set up a budget alert as usual, and connect it to one of the Pub/Sub topic above.
Please be noted that Google's documentation did mention that there could be a delay between the cost exceeds a budget and the function is triggered, so you should build in a buffer if you have an absolute hard cost limit. I use 90% of the budget as the trigger line for shutting down my instances.

The API usage can be limited with a hard limit:
Depending on the API, you can explicitly cap requests in a variety of
ways, including: requests per day, requests per 100 seconds, and
requests per 100 seconds per user. You might want to limit the
billable usage by setting caps. For example, to prevent getting billed
for usage beyond the free courtesy usage limits, you can set requests
per day caps
Source

You can combine budget pub/sub alerts with a cloud function that can disable billing on your entire account if a threshold is met.
Full Tutorial Here:
https://www.youtube.com/watch?v=KiTg8RPpGG4
GitHub Repo Here: https://github.com/aioverlords/Google-Cloud-Platform-Killswitch
To Disable Billing
const _disableBillingForProject = async projectName => {
const res = await billing.updateBillingInfo({
name: projectName,
resource: {
billingAccountName: ''
}, // Disable billing
});
console.log(res);
console.log("Billing Disabled");
return `Billing disabled: ${JSON.stringify(res.data)}`;
};

Simply go to the developer console:
https://console.developers.google.com/project
Select your project.
Select "billings & settings"
Enable billing.
Then go to Compute/AppEngine/Settings and set a daily budget.

Go to Google Cloud console, and then to Billing / Budgets and Alerts and create a new budget for one or all your projects. You can select which services should be included in the limit and set a monthly amount that should not be exceeded.

Related

Vision API quota/budget limit and API key security

I have never used Vision API before but recently I have found it very powerful for a project of mine. However I have two concerns regarding its budget limiting, in order to not get an unexpected bill:
Is it possible to set a monthly cost limit? I have been used to Compute Engine which gives me an almost exact cost of the month but this seems not possible here. Since I will be using the API for labelling I have set the label detections requests per minute and per user to a specific amount, also to be sure I have set the global request per minute and per user to the same amount, all the other quotas to 0. If I have understood correctly, setting the max calls quota per minute to 4, for example, should provide a maximum of 178560 calls per month, right? Should this limit my budget? Am I safe?
The API will be used as an API key in a mobile app. I have followed the code examples for iOS & Android and I have seen the key is written in the code. Is this safe? For a better security I have restricted the key to iOS/Android apps bundle and to Cloud Vision API only. Would it be a safe enough option?
Thanks everyone for any help!
Yes, it’s possible to set a monthly cost limit. Refer to this doc for more information about creating the budget, setting the budget scope, budget amount and threshold. Yes your understanding is correct by setting the max calls quota per minute to 4, it should provide a maximum of 178560 calls per month. It shouldn’t limit the maximum quotas.
API keys that are embedded in the code are not safe and secured.
Do not embed API keys directly in code. API keys that are embedded in code can be accidentally exposed to the public. For example, you may forget to remove the keys from code that you share. Instead of embedding your API keys in your applications, store them in environment variables or in files outside of your application's source tree.
Refer to this doc for more information about best practices for securing an API key.
Edit based on a question in the comment:
Can the quotas be seen as a hard limit?
The quotas might be seen as a hard limit only if you don't have any other resources running in your GCP project other than Vision API requests. Refer to this doc for more information about capping API usage.
If you want to set a hard limit and disable billing, configure a Cloud Function to call the Cloud Billing API that disables billing for the project as described in the GCP doc.
Note: Use this feature only if you want to stop the spending and might be willing to shutdown all your Google Cloud services and usage when your budget limit is reached.

How can you limit the billing in firebase? They used to have this possibility, it looks like they removed it [duplicate]

I'm currently working in a social network app and I need to do a search feature. Firestore does not support these kind of queries, so I need to use an external service like Algolia.
The problem is that the free plan does not support connecting to external websites/APIs other than Google's own ones, so I can't connect to Algolia to get my search system working.
I have read multiple stories about devs paying high bills because of loops or errors in their code, and as the Blaze plan is a pay-to-go plan, they get charged what they used. If a loop generated 10TB of files they will get charged for that.
I also know that Blaze plan's features are free as long as each of them (individually) stay below the limits of the free Spark plan.
So as my question says, is there a way to set limits? For example, I would like to tell Firebase to limit my cloud functions invocations to 100k per month. That way it would be free and I would never be able to get over 100k as it's limited, which means I'll never get billed for that.
Take into account that the only thing I need right now from a paid plan is the connection to external networks. I don't need anything else as we're just starting and the app is not in production, so there's no need for huge limits.
Every Firebase project is also a Google Cloud Platform project. This means that many of the advanced features of Google Cloud Platform are also available for your Firebase project.
For example, you can set up billing alert for your Firebase project, so that you are alerted when the usage reaches a certain level. While you can't configure it to switch off the project at some point, the alert should typically be quite good for alerting you to unusual usage patterns.
For more on this see:
Tracking your spending with budgets in a recent blog post.
The GCP documentation on how to set budget alerts, which is what Firebase uses under the hood.
The GCP documentation now also has a section on capping (disabling) billing to stop usage. This is a brute force approach though and may lead to data being lost, so I'd recommend investigating all other options first.
Update (December 2020): Firebase's Todd Kerpelman just released a series of videos where he disables billing using the process from the documentation mentioned above.
You cannot set spending limits to your app now.
As of December 12, 2019, you can no longer create spending limits, but
you can change or remove existing spending limits.
https://cloud.google.com/appengine/pricing#spending_limit
You can create budgets, which will alert you when reaching the budget. But it won't stop the usage when hitting the budget.
https://cloud.google.com/billing/docs/how-to/budgets#add-new-budget
The screenshot here seems to show a Spending Limit setting for Firebase projects: Firebase: Budget and Daily Spending Limit
That settings page is located here (the Spending Limit setting apparently only shows up once you set up billing for the project): https://console.cloud.google.com/appengine/settings
It's disabled in the poster's case, but I think that's only because he connected it up to a "NodeJS App Engine app", which isn't the case for many Firebase developers.
I haven't tried it yet myself, but will do so once I start a paid plan.
EDIT: Yep, the setting shows up once you switch to a paid plan. (in my case, Blaze) I don't have enough traffic yet to confirm that it works as expected, but if I find later that it doesn't, I'll give an update here.
"This example shows you how to cap costs and stops usage for a project by disabling Cloud Billing. This will cause all Google Cloud services to terminate non-free tier services for the project."
Google Cloud Source

understanding "action required" email from GCP, re: enable Cloud Build API

I use Firebase for my web apps, not Google Cloud Platform -- though I'm aware that behind the scenes, every Firebase project is also a GCP project.
I just received an email from GCP, saying (excerpted):
[Action required]: Enable the Cloud Build API for your projects before April 20, 2020, to ensure Cloud Function is built and deployed correctly ... You must enable the Cloud Build API for your project(s) to ensure your Cloud Function is built and deployed correctly. Once you enable the API, you may incur charges if your Cloud Build, Container Registry, or Cloud Storage usage exceeds the free tier limits for these products.
Does this impact the deployment or development process for Firebase at all? (e.g., right now I use firebase deploy )
As far as I can tell, the answer is no -- it just allows for a greater maximum build-time-per-day. But the email didn't mention Firebase at all, so I want to be sure I understand any other implications well before April 20.
I had the same questions and got in touch with the firebase support team, here are my questions and the answers I received:
"Is that mandatory?"
Only if you are going to deploy Cloud Functions in the future, already
deployed functions will not be affected.
"How do I know if my Cloud Build, Container Registry, or Cloud Storage usage will exceed the free tier limits for these products? is there any way how can check what I've been using so I'll be prepared?"
Cloud Build provides a free tier where only usage above 120
build-minutes/day will be charged. When your usage is within the free
tier, you will not be charged for the Cloud Build portion of Cloud
Function deployments. See Cloud Build pricing for more information.
Similarly, Cloud Storage and Container Registry share a free tier
where only usage above 5GB-months will be charged. ( * Note : free
tier only limited to US regions - US-WEST1, US-CENTRAL1, and US-EAST1
and aggregated over all 3 regions )? For example, if you have a large
deployment that uses 100GB of storage, you will only be charged an
additional $2.47 for storage/month (based on these particular U.S.
regional storage prices). You can monitor your usage and see whether
you are getting close to hitting the free quotas.
"Lastly, what does all this stuff do for my app? Why do I need to add this other features? If you were to sell me this, how would you present it for my use-case?"
This is a best effort from Google to communicate information that is
necessary to the user’s continued use of the product or that is
considered a necessary legal update and keep customers away from
having future issues. Using Cloud Build, Container Registry and Cloud
Storage provides the following benefits: Detailed function build logs
will be available in the GCP Console, aiding in debugging and
increasing visibility. The ability to get build time that exceeds the
current build quota of 120 build-mins/day. The ability to view a built
container image for your function in Container Registry.
I received the following response from Firebase Support:
You will not be able to use cloud functions if you do not enable the Cloud Build API after this date [April 20, 2020].
And, in a follow-up exchange:
as far as I know in terms of deployment everything will be the same
So, my interpretation of those responses with respect to the original question (will this affect the mechanics of development or deployment) is:
Deployment process: no change necessary
Development process: no change necessary*
*if your build time exceeds 120 minutes per day, and you wish to avoid associated charges, you could voluntarily investigate changing your deployment/development process to reduce build minutes.
Thanks also to Fran for his answer, which sheds light on other aspects of the change.

Firebase account creation limit

In Firebase Authentication Limits it states that new account creation limit is 100 accounts/IP address/hour. I could not find any further explanation on this, i.e. what happens if the limit is exceeded, because it is very likely that there might be more than 100 new user creation after the launch of the app or a successful marketing campaign.
It states that "You can schedule a temporary increase to the account creation limit in the Firebase console."
Is that a paid service? How much does it cost? How long is temporary? How often can it be scheduled?
To add to the Frank van Puffelen's answer, this is easily customizable via the Firebase console: Authentication tab -> Manage sign-up quota (at the very bottom) to get this:
The IP limit is in place to protect your project against abuse. Setting a temporary increase is available to all Firebase projects, free or paid.
I don't think there is any documented limit to how often you can increase this, although that too is monitored for abuse.

Firebase "Blaze" projects limit, multiple env and Spark plan outbound requests

I'm working with Firebase and quite enjoying it so far.
I'm working with DEV, PREPROD and PROD environments for each of my projects. For each env I've had to create a distinct firebase project.
Since my app is using Algolia and Cloud vision API, I apparently have to be on the Blaze plan because Spark plan doesn't allow outbound requests and Cloud vision API calls (if I'm correct).
The thing is we're limited with the numbers of Blaze projects we can have at the same time. Above a certain amount (6 or 7, I think) we have to request a "billing quota increase" and explain why we need more (sounds odd but ok).
So I did, but now Firebase is asking for a $50 transaction to increase the number of Blaze projects I can have.
So I have several questions:
- Am I right to think that in Spark plan I can't call the Algolia API in my cloud functions or call Cloud vision API ?
- Are these $50 a payment to unlock new projects slots or just credits that will be available if needed ?
- If I need even more projects in the future will I have to pay even more credits ?
- How am I supposed to handle separate environments on Firebase without creating a different project each time ?
Thanks a lot
On the Spark plan, with Cloud Functions, you can only make outgoing connections to services that Google fully controls. Algolia will not work.
Please read the FAQ regarding the number of projects you may have and the payment being asked to create a new project:
Why am I being asked to make a payment for more projects?
You may be asked to make a payment if your request for more projects
indicates that you need projects that will use paid cloud services.
The payment can be applied to any charges you incur in the future and
will be visible as a credit in your account.
This payment is required to ensure paid services will be available for
the projects you requested in the quota increase request form. This is
a common requirement, because Google Cloud Platform services are paid
(e.g., Compute Engine, Cloud SQL, and BigQuery).
The payment required varies depending on your billing history, the use
cases described in your request form, the number of projects you
request, and other factors.
So, the $50 you are being asked to pay will apply as credit to your project billing.
You should definitely create new projects for each environment.

Resources