I am new to DynamoDB. While creating a dynamodb table I can see that it also creates some cloudwatch alarms like TargetTraking-TableXX--ProvisionedCapacityHigh.
So my question is can I delete these alarms as it is adding cost. I have tried researching on this but dint find anything. I am afraid to delete this because I am not sure if the AWS itself is using these alarms for the dynamodb maintenance and whether if I delete them will the dynamodb malfunction by any means.
I would also like to know if these alarms are used for scaling up and down in case of autoscaling been enabled or ondemand provisioning is enabled for the dynamodb table.
In my case I am not using autoscaling or ondemand tables so is it fine if I delete those alarms?
The Cloudwatch alarms gets created if autoscaling is enabled for provisioned tables. Whereas On-demand tables are fully managed by AWS and we pay for what we use. If you wish not to have autoscaling then it is ok to delete the alarms or disable them. To revert back the change re-deploy of autoscaling will re-create the alarms.
NOTE: Autoscaling creates 4 read and 4 writes table. The alarms description also says do not EDIT or DELETE the alarm as it affects the autoscaling functionality.
Related
I have a large node in Realtime Database that I want to delete everyday using Cloud Functions schedule.
Is there a limit on how much data I can delete using Cloud Functions on Realtime Database? And where can I find the cost for delete?
I've read the billing doc (link) but I'm not sure where it is mentioned about delete cost.
I'll start by adding this link.
Combining the informations in your link and in this one I've added, the answer is: no, you'll not be billed if you just delete data. The important information here is: if you just delete it. You will still be billed if, before you delete it, you download it. In other words, if you get the reference to a node in your code, and then you just perform a remove(ref), you won't be billed.
There is a remote possibility that you can be billed for a huge CPU consumption. This could happen if the node you're deleting is really big, but you can estimate this by testing it out and checking the "Usage" tab in your Firebase console, under the voice "Load". If the load for a testing delete is low, you're good and you won't be billed.
The firebase documentation includes a warning that if you use a kill switch to stop using firebase when a budget cap is exceeded, as follows:
Warning: This example removes Cloud Billing from your project,
shutting down all resources. Resources might not shut down gracefully,
and might be irretrievably deleted. There is no graceful recovery if
you disable Cloud Billing. You can re-enable Cloud Billing, but there
is no guarantee of service recovery and manual configuration is
required.
I'm trying to investigate what gets irretrievably deleted. Does the datastore get deleted when the kill switch is activated? Is there any opportunity to save data previously stored in cloud firestore, before the deletion takes place? Is there a way to download the database so that I can keep a back up in this scenario?
Please review the following reply from Firebase Team member(samstern) to gain more clarity on this:
these things are handled on a per-product basis and each product has different thresholds for quota overages and different procedures for what happens to inactive resources in a bad state.
For instance I know in Realtime Database if your DB is too big for the
free plan after you downgrade from a paid plan we will not delete
your data automatically. Instead you'll just be stopped from using the
database until you restore your billing.
However that statement clearly says that some products may delete data
if you pull Cloud Billing. It could be for technical reasons, it could
be for policy reasons.
If you want to turn off a product without risking this pulling your
billing account is NOT the best way to do this. It's the nuclear
option if you get into a bad situation and want to get out at all
costs. Instead you should use per-product APIs to shut down the
individual product or service and prevent future usage. This could
include doing things like flipping off APIs in the APIs console,
changing security rules to prevent all further writes, deleting Cloud
Functions, etc
The best source of information I've been able to uncover in answer to this particular question is a discussion on reddit which indicates that you can't recover access to your data, until you pay the bill (including blow out charges) - so maybe that buys some time, but if you don't pay, the project gets deleted. There may also be lost data for things happening at the time the kill switch activates.
I'm building an offline-first mobile application using AWS Amplify, using the local DataStore and cloud sync. So far, I'm following the documentation without any variation (I think.)
As of now, I only have one model, lets call it at Thing. I noticed that after running amplify push, my environment contains not one, but two DynamoDB tables:
Thing-<app-id>-<env>
AmplifyDataStore-<app-id>-<env>
Whenever I save a Thing entity, it appears to be persisted redundantly in both tables. This effectively doubles my DynamoDB storage costs.
Is there a sound technical reason for this, or any way to avoid it? Or am I just making a mistake somewhere that is causing it to persist twice?
Assuming you have k models, then the Amplify DataStore will provision k + 1 tables. The extra table you're noticing is called the "delta sync table." It used to store incremental changes that have occurred since the last time the client synchronized fully with AppSync. The Delta Sync table carries a short TTL on the records, and they will get dropped if not utilized within that window of time.
To learn more about Delta Sync and DataStore generally, I recommend Ed Lima's AWS AppSync offline reference architecture – powered by the Amplify DataStore. See particularly the section labeled "The Delta Sync table."
Source: I'm an engineer on this product team.
I have a DynamoDB table that contains key value pairs that will be read by a number of applications. On startup each application will read the entire table and cache it in-memory.
The problem I'm trying to solve is that of getting the applications to update their cache if one or more items in the DynamoDB table have been modified.
DynamoDB streams initially seemed to be the right approach to solving the problem. I have implemented the consumer using Kinesis Client Library (KCL) as recommended by AWS. While implementing it, however, I have encountered some problems that make me believe that I'm on the wrong track. Specifically:
When I create a new consumer using KCL, it creates a new DynamoDB table to do the housekeeping of leases and checkpoints, such that when the application is restarted, KCL knows which records have been consumed and which have not. This is not what I need for this problem. Any stream records that are created while the application is offline is irrelevant, since the entire table is read upon application startup.
Several instances of the same application are running at the same time. Each of them needs to be notified of table updates. To implement that in KCL I need to assign a unique application name to each of them. Otherwise they will share the lease table and only one of the applications will get notified. One table for each application instance doesn't seem right. Also I would then need something to remove unused tables.
I also implemented it using the low level API instead. That works fine when there's a single shard. My implementation doesn't handle re-sharding like KCL, however, so it's too fragile. It seems wrong to have to implement handling of re-sharding for the simple problem I'm trying to solve.
I'm beginning to consider other solutions like:
Implementing a lambda function that gets triggered on updates to the table. The function sends a notification to an SNS topic. Consumers create SQS subscriptions on the topic and gets notified via that. This solution has too many moving parts for my liking.
Make the applications periodically re-read the entire table and determine themselves if changes have been made. This solution feels a bit primitive, but seems to be the simplest.
All solutions that I have considered so far have quite significant drawbacks. What am I missing?
It depends on how your KCL is pushing to the dependent apps but
I believe the SQS path is the correct choice.
You can add a presumably infinite number of consumers without being throttled.
When you do add another dependent app, it won't require changing your KCL to push to it, the new app will simply watch the SQS queue.
You gain the ability to monitor the queue when issues happen.
More moving parts to setup, but once you have the Streams -> SNS -> SQS pipe in place, it's basically bulletproof.
Just my 2¢.
Nowadays an AWS AppSync GraphQL API with subscriptions may be the simplest approach to power this type of application, with the least number of moving parts.
Whenever one of your applications starts up, it connects to your AppSync GraphQL API using the Amplify framework or AppSync SDK and subscribes to the updates its interested in. Then whenever an application updates information in the table via your GraphQL API, all your other applications will be notified of the change, along with the relevant changed data.
AppSync integrates well with DynamoDB out of the box, allowing you to generate DynamoDB tables with appropriate indexes alongside your GraphQL or generate GraphQL from your existing DynamoDB tables if you so choose. Amplify can even help you automatically generate an AppSync GraphQL API at a higher level with associated DynamoDB tables, indexes, entity relationships, and more like elasticsearch search capabilities by using their GraphQL transformers.
I'm looking for an example to delete items from Google Cloud Datastore by:
Key
Kind
Filter
Ancestry
ps: I couldn't find them here:
https://developers.google.com/datastore/docs/concepts/queries
Google Cloud Datastore only supports delete-by-key (and in general, does not support "update queries").
To delete a small number of entities, you can execute a (keys-only) RunQuery operation to fetch the keys and then a BlindWrite request to delete the entities.
Or, if the entities are in a single entity group, you can do the entire operation inside a transaction using BeginTransaction to create a new transaction, set the transaction handle in the query ReadOptions, and a Commit request to apply the mutation.
If you are deleting a large number of entities, you can use the above technique in a MapReduce.
If you are deleting all entities of a particular kind, you can use the App Engine Administration Console to delete entities in bulk.