In my Android app I use Amazon DynamoDB. I created 10 tables with Read capacity 10 and Write capacity 5. Today I received an email from Amazon. It costs me 11.36$.
I don't understand the meaning of free tier. Here is what I read from Amazon:
DynamoDB customers get 25 GB of free storage, as well as up to 25 write capacity units and 25 read capacity units of ongoing throughput capacity (enough throughput to handle up to 200 million requests per month) and 2.5 million read requests from DynamoDB Streams for free.
Please tell me more clearly about the meaning of free tier: 25 read and 25 write capacity units!
Amazon consider aggregates of read and write capacity of all tables, not the capacity of individual tables.
In your case the read capacity is 100 and the write capacity if 50. And you are charged for the 75 read capacity usage hours and 25 write capacity of usage hours.
Please plan properly for the read and write capacity of each table, otherwise you end up paying more bills.
Thanks.
Related
We are using a dynamodb with on-demand capacity mode. We are suddenly seeing few of our requests getting write throttled. During the same time frame there is sudden spike in Write Capacity Units (WCU).
I have checked the incoming traffic/ write request count and it is pretty much the same.
Does this mean the WCU increase is due to increase in item size?
How can I verify the size of my request?
Since our dynamodb is already provisioned in on-demand mode, why is it unable to auto scale and handle the requests given that traffic is pretty much the same.
It's possible
You can return the consumed capacity for each write operation which will allow you to understand which items are consuming more capacity. ReturnConsumedCapacity
On-demand mode tables can throttle for 3 reasons:
You exceed twice your previous peak in 30 mins
You exceed partition hard limit of 1000 WCU / 3000 RCU (hot partition)
You have GSI backoressure where you're GSI has a hot partition
You can enable CloudWatch Contributor Insights to understand if you have a hot key(s).
I want to set up a Cosmos DB account with a single write region and multiple read regions. I also want to use autoscaling RU provisioning, so I only pay for what I use (above the floor).
Now if there is zero load I expect the RU cost to be 400 RU multiplied by region count (since 400 RU is the cost floor for autoscaling).
If I perform a write charged at a specific RU cost that I can see in the response, is that only counted once (against the write region), and then the replication only incurs extra costs for egress and storage? Or will the RU cost be multiplied by the region count behind the scenes?
Similarly for reads, is that RU cost only counted once (against the read region), or is it multiplied by the region count?
Under Metrics (Classic), I see that Avg Throughput/s (RU/s) only changes in the write region when writing, but I'm not sure if this reflects the actual charge.
I felt that this was not answered clearly in: In Cosmos DB, how does geo-replication impact RU consumption of writes?
The throughput that you have configured for various Azure Cosmos
databases and containers will be reserved in each of the Azure regions
associated with your Azure Cosmos database account. If the sum of
provisioned throughput (RU/sec) configured across all the databases
and containers within your Azure Cosmos database account (provisioned
per hour) is T and the number of Azure regions associated with your
database account is N, then the total provisioned throughput for a
given hour, for your Azure Cosmos database account is equal to T x N
RU/sec.
Provisioned throughput (single write region) costs $0.008/hour per 100 RU/sec and provisioned throughput with multiple writable regions (multi-region writes config) costs $0.016/per hour per 100 RU/sec.
Source: Understand your Azure Cosmos DB bill
When you are doing your RU calculations for Cosmos DB, do you need to be calculating the max values of reads, inserts, updates and deletes or the average number per second?
Reason why I ask is because the average documents read (in current mongo db) is around 5500 but the maximum number of documents read (in on second) over my sampling period was 965880.
I have looked through all of Microsoft's documentation on Costing Cosmos DB and there is no clear guidance on whether the figure for RU throughput is average or max
As you said there's no MS document on 'average or max' for setting throughput, in my opinion, both average and max are meaningful, but we also need to look at the most common situation, for example, there's always around 5800 per second, and also usually 4500 per second, the min is 3000 and the max is 9000. 1 RU means '1KB doc read', if we set the max number as the throughput, it's expensive and waste, if we set the average, maybe the system usually 'in debt' as the answer said. That's why I say we also need to consider the 'most common' situation.
By the way, MS provides a web based tool for helping estimate the request unit requirements for typical operations. If admin also don't know the real situation about the database, I think this doc may help, in short for the doc, that says, if you're building a new application or a small application, you can start at the minimum RU/s to avoid over-provisioning in the beginning. After running the application for some time, maybe you can use azure monitor to determine if your traffic pattern is suitable.
To avoid throttling you need to provide the MAX throughput you need in RUs. Also, it depends how frequently you hit the max RUs. There are basically 3 ways to provision RUs- Provisioned throughput, Autoscale & Serverless(Preview).
If you provision standard (manual) RU/s at the entry point of 400 RU/s, you won't be able to consume above 400 RU/s, unless you manually change the throughput. You'll be billed for 400 RU/s at the standard (manual) provisioned throughput rate, per hour.
If you provision autoscale throughput at the entry point of max RU/s of 4000 RU/s, the resource will scale between 400 to 4000 RU/s. Since the autoscale throughput billing rate per RU/s is 1.5x of the standard (manual) rate, for hours where the system has scaled down to the minimum of 400 RU/s, your bill will be higher than if you provisioned 400 RU/s manually. However, with autoscale, at any time, if your application traffic spikes, you can consume up to 4000 RU/s with no user action required. In general, you should weigh the benefit of being able to consume up to the max RU/s at any time with the 1.5x rate of autoscale.
For small applications with a low expected traffic, you can also consider the serverless capacity mode, which bills you as you use.
Use the Azure Cosmos DB capacity calculator to estimate your throughput requirements.
Should definitely go through this and related pages of documentation- https://learn.microsoft.com/en-us/azure/cosmos-db/request-units
Our entire infrastructure is built in Linode, Singapore region. The problem is that as of now, Linode does not provide any block storage option. We have a 3 node Cassandra cluster which is growing at the rate of 4-5 gb per day. Each node has 192 GB SSD disk allotted to it. Adding a Cassandra node is simple , but it comes at the cost of maintaining it. The rate at which we are growing, we'd be needing 20-30 servers in 3 month time.
Digital Ocean, Singapore region, on the other hand, does have a block storage option. We can simply add more SSD to our servers rather than provision for 30 small servers.
Data is streamed from Kafka and stored in Cassandra.
What would be the pros and cons of having your Cassandra cluster in a different Data center but in the same country? The latency between the two is about 2 ms. The ratio between reads and writes are roughly 5% Read ops and 95 % write ops.
I have a topology running on aws. I use M3 xlarge machines with 15GB ram, 8 supervisors. My topology is simple, I read from
kafka spout -> [db o/p1] -> [db o/p2] -> [dynamo fetch] -> [dynamo write & kafka write] kafka
db o/ps are conditional. with latency around 100 - 150 ms.
But I have never been able to achieve a throughput of more than 300 msgs/sec.
What configuration changes are to be made so that I can get a throughput of more than 3k msgs/sec?
dynamo fetch bolt execute latency is around 150 - 220ms
and dynamo read bolt execute latency is also around this number.
four bolts with parallelism 90 each and one spout with parallelism 30 (30 kafka partitions)
overall latency is greater than 4 secs.
topology.message.timeout.secs: 600
worker.childopts: "-Xmx5120m
no. of worker ports per machine : 2
no of workers : 6
no of threads : 414
executor send buffer size 16384
executor receive buffer size 16384
transfer buffer size: 34
no of ackers: 24
Looking at the console snapshot I see...
1) The overall latency for the Spout is much greater than the sum of the execute latencies of the bolts, which implies that there's a backlog on one of the streams, and
2) The capacity for SEBolt is much higher that that of the other bolts, implying that Storm feels the need to run that bolt more than the others
So I think your bottleneck is the SEBolt. Look into increasing the parallelism hint on that one. If the total number of tasks is getting too high, reduce the parallelism hint for the other bolts to offset the increase for SEBolt.