Our application uses Cosmos DB to store data. The DB is partitioned based on User ID and all backend services work fine using the User ID.
However, we have a UI where only admins have access and they are different from the users stored in DB. They fetch data from DB based on time. That is, get reports from DB for last 3 days (doesn't matter what user ID). In this case, the query needs to fan out to all partitions. Moreover, stored procedures are scoped per partition and cannot be used in this scenario. Even though we are able to fetch data through query, it's a huge performance hit.
Can anybody please advise if there is a way in Cosmos DB to workaround this?
Related
in the project i am working on, we have a database per tenant and each tenant consists of at least 1 department. One of the requirements we have is that when an admin user deletes a department using a custom frontend we've provided, the system should first archive the data of that department on a blob storage before the data is deleted. The same we have for the tenant, we need to archive the data before the database of that tenant is removed from the account.
Now, my question: is there any best practice to do this? We are planning to retrieve all the data from all collections, using a mongo query, based on the department id (which is also the partition key) and then send it to a blob storage. The challenge we have is the execution of the query to retrieve all the data because it can be a huge amount and the RUs required for that action may affect the performance of the system because other users may be using the system while we remove the data.
I looked at mongodump and mongoexport but these are applications so we cannot execute it from our code?
Any ideas? Thanks a lot.
I think one way to solve this is by using ChangeFeed, as it reallyhelps and simplifies writing a carbon copy somewhere else.
However, as of now the change feed processor won't notify you for deleted documents so you can't listen for them, this feature is planned as of now.
Your best bet is to write some custom application that does archiving using Query language support
I need to access Cosmos DB data through a middleware API that gives access to SQL queries but not the change feed (i.e. DocumentClient.CreateDocumentQuery() but not DocumentClient.CreateDocumentChangeFeedQuery()). Is it possible to query the change feed using regular SQL queries?
I was thinking about filtering documents on recent _ts but I am not sure timestamps are guaranteed to be monotonically increasing across entire collections due to potential clock drift across the VMs Cosmos DB runs on.
You cannot query the Change Feed using a SQL query. The Change Feed contains documents that have been inserted / updated, and any filtering needs to be done client-side after receiving such changes.
One colleague said that cosmos db will stop supporting collections without a partition key. But I can't find any information about this statement from Microsoft.
The application I'm working on has a collection of order records. A typical query returns 10s of thousands of these records. So if I use order id as partition key, it'll always run cross partition queries.... And the requirement is to get all records across all tenants, so partition by tenant id isn't an option, either.
I thought it'll be fine just create a collection without a partition key. I'll worry about archiving data later (probably with azure functions and change feed).
Is it a good idea to do so?
One colleague said that cosmos db will stop supporting collections
without a partition key. But I can't find any information about this
statement from Microsoft.
Based on the tips on the cosmos db portal,this message is confined to portal only so far.
You still could create non-partitioned collection by using sdk:
DocumentCollection collection = new DocumentCollection();
collection.set("id","jay");
ResourceResponse<DocumentCollection> createColl = client.createCollection("dbs/db",collection,null);
So,i think your service will not be affected by now. As for future trends, I suggest you pay more attention to Microsoft's official statement. If you have any special needs, you can submit feedback for help.
in my Azure Cosmos DB account, I can add multiple databases (containing multiple collections).
However, I only seem to find account-level connection strings (secrets), that are valid for each database. Differing only in the database name section.
I find this odd. Is this expected? If I want more granular control do I need to create separate accounts for each database?
PS: I'm using the Mongo API if it's somehow relevant.
Cheers
The account-level connection strings you mentioned in the question is master key.Based on this document, Azure Cosmos DB uses two types of keys to authenticate users and provide access to its data and resources.
Master keys cannot be used to provide granular access to containers and documents.
If you want more granular control,please get an idea of Resource Tokens which provides access to specific containers, partition keys, documents, attachments, stored procedures, triggers, and UDFs.More details,please refer to this link.
Im developing a database application using MariaDB where I have a table for clients registers. I want them to be able to query the database, but I don't know if I have to create a database user account for each one in order to allow simultanious connections without collitions(same user trying to query at the same time) or can I use the same user account for simultanieous queries.
I'm kind of new to MariaDB.
Thanks in advise.
Databases are good at handling 'simultaneous' accesses by multiple connections to the same table.
The question of "one" versus "many" user accounts is independent of simultaneous access.
Most accesses work just fine without any problem. In the extreme, I suggest you read about "deadlocks".