Question on Hangfire
How to enable the container, created by hangfire on cosmos DB to be partitioned by jobId?
Can the container be created before invoking hangfire?
Assuming we're talking about .net and c# you can fairly easily create container using CosmosDB SDK. I recommend using 3.17+ as for the below versions there are some annoying memory leaks.
Azure allows portal UI to create container manually.
Code snippet for creation of jobs container in HangfireJobsDB:
var client = new CosmosClient("https://<your account>.documents.azure.com:443/", accessKey);
var database = await client.CreateDatabaseIfNotExistsAsync("HangfireJobsDB");
var container = await database.CreateContainerIfNotExistsAsync("jobs", "/jobId", 1000);
Note, this snippet will create container with provisioned 1k RUs throughput dedicated for it. CosmosDB another model of throughput allocation on database level which will share it across all containers in the DB. You should have at least 400 RUs per container in such model.
Related
I currently create a CosmosDB with the following properties:
cosmosDb = await azure.CosmosDBAccounts
.Define(cosmosDbResource.Name)
.WithRegion(cosmosDbResource.Region)
.WithExistingResourceGroup(cosmosDbResource.ResourceGroup.Name)
.WithKind(DatabaseAccountKind.GlobalDocumentDB)
.WithStrongConsistency()
.WithTags(cosmosDbResource.ResourceGroup.Tags)
.CreateAsync();
The only place I have seen to be able to set Zone Redundancy on is the ReadReplication database, like so:
cosmosDb = await azure.CosmosDBAccounts
.Define(cosmosDbResource.Name)
.WithRegion(cosmosDbResource.Region)
.WithExistingResourceGroup(cosmosDbResource.ResourceGroup.Name)
.WithKind(DatabaseAccountKind.GlobalDocumentDB)
.WithStrongConsistency()
.WithReadReplication(Region.USEast, true)
.WithTags(cosmosDbResource.ResourceGroup.Tags)
.CreateAsync();
The problem is that I don't care about a Read Replication database. I want to set Zone Redundancy on the initial database I create. I noticed that in the Azure Portal when I create a CosmosDB manually, it gives me the option to set Zone Redundancy. Is this not possible via the Azure Libraries for NET SDK?
To specify write region with Zone Redundancy do this below:
.WithWriteReplication(Region.USWest2, true)
PS: If at all possible I would recommend you use the Auto-rest generated version of this SDK. The fluent API is not generally as up to date as the Auto-rest generated API's. This gets built directly off our the Cosmos DB swagger spec and everything downstream is built upon this including ARM, PowerShell and CLI.
There is a repository with a fairly complete set of examples as well that you can use to help build your own management libraries. It also includes fluent samples but also out of date. Cosmos DB Samples
This is the repo for the Auto-rest generated SDK. Cosmos DB Management SDK for .NET
I'm using ADF pipeline for storing data from data lake into cosmos db sql api. On triggering pipeline run, I see the following error:
Currently I'm using Throughput 5000 RU/s for the cosmos db container. Please help me understand why it is showing this error?
Here is my pipeline:
For saving your cost, don't unlimited increase RUs setting of cosmos db. Based on the error message you provided in your answer,it indicates that cosmos db can't process so many rows in unit time. So i suggest you throttling the transfer data procedure. Please consider using below configuration for the sink dataset of your copy activity.
The default value is 10,000, please decrease the number here if you do not concern the speed of transmission.More details ,please refer to this document.
Additional, you could throttling the max concurrent connections:
I created several new pipelines in Azure Data Factory to process CosmosDB Change Feed (which go into Blob storage for ADF processing to on-prem SQL Server), and I'd like to "resnap" the data from the leases collection to force a full re-sync. Is there a way to do this?
For clarity, my set-up is:
Change Feed ->
Azure Function to process the changes -> Blob Storage to hold the JSON documents -> Azure Data Factory which picks up the Blob Storage documents and maps them to on-prem SQL Server stored proc inserts/updates.
The easiest and simplest way is to do it is to simply delete the lease documents and make sure that the StartFromBeginning setting is set to true. Once restarted the change feed service will recreate the leases (if the appropriate setting is configured to true) and reprocess all the documents.
The other way to do so is to update every single lease document and reset the Continuation token "checkpoint" to null, however I don't recommend this method since you might accidentally miss a lease which can lead to issues.
I'm using Azure Cosmos DB .NET SDK Version 3.0 and I want to create container programmatically without partition key. Is it possible? I always got error saying Value cannot be null.
Parameter name: partitionKey
I use method CosmosContainers.CreateContainerIfNotExistsAsync
Reproduce your issue on my side always.
Notice the exception is caused by below method:
Try to deserialize the dll source code and find the detailed logical code.
It seems we can't cross this judgement so far because cosmos db team is planning to deprecate ability to create non-partitioned containers, as they do not allow you to scale elastically.（Mentioned in my previous case:Is it still a good idea to create comos db collection without partition key?）
But you still could create non-partitioned containers with DocumentDB .net package or REST API.
Does it cost anything every time you drop/recreate a database in Cosmos DB (SQL API)?
Does it cost anything every time you drop/recreate a collection within a database in Cosmos DB (SQL API)?
Database, Collections, Offers, Documents etc all inherit from the same single object which is a Resource. Resources are basically a CosmosDB object which is represented in JSON.
Creating a collection or creating a database is essentially creating a Resource which is a document whose size doesn't exceed the 1kb so you will be charged the minimum price for the Create or Read of that data.
Keep in mind however that CosmosDB also charges you hourly per collection based on it's provisioned RUs. For example, if you create a collection even for a second, you will be charged the hourly rate of this collection's existence based on the provisioned RUs.
Based on the official doc, databases, users, permissions, collections ,documents and attachments are all resources. Just as what is mentioned in this doc, the billing unit for Cosmos DB is RUs. If you operate any resources in cosmos db , you will consume RUs so that costs will be incurred.
I tested code related to creating and dropping database via java cosmos db sdk.You could see the consumption of RUs with your operations.
public static void main(String[] args) throws Exception, DocumentClientException {
DocumentClient documentClient = new DocumentClient(END_POINT,
MASTER_KEY, ConnectionPolicy.GetDefault(),
ConsistencyLevel.Session);
Database database = new Database();
database.setId("hello");
ResourceResponse<Database> response = documentClient.createDatabase(database, null);
System.out.println(response.getRequestCharge());
ResourceResponse<Database> response1 = documentClient.deleteDatabase("dbs/hello", null);
System.out.println(response1.getRequestCharge());
}
Another details about price in cosmos db,please see this doc.
Hope it helps you.
Creating a Collection will incur one hour of billing at whatever RU/s throughput you have provisioned. Same as with scaling. IE if you scale up from 1000RU/s to 2000RU/s and then back down immediately you'll still be charged for one hour usage at 2000RU/s.
The Azure Pricing Calculator will let you break down the cost of a collection in hourly granularity based on the provisioned throughput.
Cosmos DB pricing completely depends on reserve RU per collection, You don't need to pay anything for creating Cosmos DB account and Database but when you will create a collection, minimum RU limit is 400 per collection, so if you use or not that collection you need to pay.