In Cosmos Db, I am using a document level Time to Live (TTL) and Cosmos does not appear to be expiring documents. Does this feature work in Cosmos Db using MongoDB API? If it does, what am I missing?
I am using Cosmos Db with the MongoDB API.
A "ttl" field is set in each document for my collection.
In Azure, Time to Live is set to "On (no default)" for my collection.
I am doing this without the emulator because the emulator defaults to the SQL API. In the emulator, I see "_ts" set and I do not see this field in Azure.
I can switch to collection level expiration by setting Time to Live to "On" and documents expire as expected. When I do this, my "ttl" field is ignored and the value I set for "second(s)" in Azure is followed. I still see my "ttl" field in the document.
Although I don't see a "_ts" field in my documents, an article about indexing mentions that it is a reserved property. This makes be think that it is set behind the scenes and it is not returned in queries.
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-indexing
"_ts is a Cosmos DB-specific field and is not accessible from MongoDB clients. It is a reserved (system) property that contains the timestamp of the document's last modification."
Update:
I checked the MongoDB support page (https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-feature-support) and it indicates that collection level TTL is available and says nothing about document level.
Azure Cosmos DB supports a relative time-to-live (TTL) based on the timestamp of the document. TTL can be enabled for MongoDB API collections through the Azure portal.
Update:
My Azure Portal Preview Features now show this:
I got document level Time to Live working in Cosmos Db using MongoDb API. I had to ask help from Microsoft support to get this working. Response from Microsoft Big data team was following.
Before enabling Document level TTL feature , I would like to
clarify following about Document TTL feature details here.
The TTL feature is controlled by TTL properties at two levels -
Collection level and the Document level.
Right now per Document level TTL for MongoDB accounts are not
available by default. However, we can enable this feature for specific
customers and this feature is set at an account level.
TTL is at a document level but the feature is enabled at an account
level which means for all collections under the account, if there is a
document with a TTL set, it will take effect. For other collections,
if the TTL value is not set for each document, it would not be
affected.
You needs to have an index on the _ts field for this to work.
To summarize this : - This feature works at Cosmos DB account level.
We need to enable Document TTL feature in Cosmos DB backend on our
side.
Related
Is it possible to review the history of changes to the setting of an Azure Cosmos database. I believe I am colliding with another them who is using the same Cosmos database. I would like to review all the setting that have changed since the database was created. Is that possible?
You can audit control plane operations for any Cosmos DB resource once it has been created. However, the account must be configured such that all control plane operations are done against the Cosmos DB resource provider and not through one of the data plane SDKs. See the link for details on this.
However, ARM does not support providing information on when a resource was created. This is a limitation of ARM, not Cosmos DB.
If you just wanna watch who did operations on your cosmos database, I think 'Activity Log' tab could help you. It records some message like screenshot below.
But I found that I can't get the details of the operations such as changing RU. By the way, I didn't find any documents saying how to review operations in cosmosdb. Or maybe you can provide more details about what information you wanna get from cosmosdb?
Does Cosmos DB automatically set PreferredLocations? For example when new regions are added/deleted.
Or users have to set it themselves?
According to the docs, the most optimal endpoint will be chosen by the SQL SDK to perform write and read operations: https://learn.microsoft.com/en-us/azure/cosmos-db/tutorial-global-distribution-sql-api#connecting-to-a-preferred-region-using-the-sql-api
This is based on the configuration of your account, region availability. If you don't specify the PreferredLocation property, then all requests (read and write) will be served from your account's current write region.
Hope this helps :)
PreferredLocations must be specified for high availability. EnableEndpointDiscovery along with PreferredLocations allows you to leverage Cosmos DB failover capabilities.
When the value of this EnableEndpointDiscovery is true, the SDK will automatically discover the current write and read regions to ensure requests are sent to the correct region based on the regions specified in the PreferredLocations property. Default value is true indicating endpoint discovery is enabled.
Is there any TTL option on documents for Firebase Firestore . Where documents get auto deleted after that amount time
Update (2022-07-26): Firestore just added the option to set a time-to-live policy on collection groups. I'm still leaving the custom approach below, as those give you control over the expunge moment which (for now) isn't possible with the built-in feature.
The easiest way to build it yourself is by:
Adding a expirationTimestamp property to your documents.
Denying read of documents whose expiration has passed in your security rules.
match /collection/{document} {
allow read: if resource.data.expirationTimestamp > request.time.date();
}
Unfortunately this means that you won't be able to query the collection anymore. You'll need to access the individual documents.
Periodically run Cloud Functions code to delete expired documents.
Also see Doug's excellent blog post describing this process: How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL).
Firestore TTL policies is now available in preview
https://cloud.google.com/firestore/docs/ttl
As of July 26th 2022, TTL Policies for Firestore were released as a preview feature (which means its not ready for production). Update Oct 2022: The TTL Policies feature has graduated from preview to General Availability, which means it should now be ready for production!
In order to use TTL Policies in Firestore, make sure that your documents have a field (of type Date & Time) to define the expiration date of that document (let's name the field expireAt for example).
And then follow the steps outlined in the documentation:
Go to the Cloud Firestore Time-to-live page in the Google Cloud Platform Console.
Go to the Time-to-live page.
Click Create Policy.
Enter a collection group name and the timestamp field name (expireAt in our example).
Click Create.
Is it possible to get the size of every partition in a Cosmos DB collection? I know the portal will show the top few partitions in a collection, in the Metrics blade, but I'm interesting in seeing the size of every partition.
I believe you should be able to get this data through the Cosmos DB REST API.
It doesn't seem to be exposed through the .NET SDK so you'd need to write some C# or PowerShell yourself to access the data however, it should be available.
Link is:
https://learn.microsoft.com/en-us/rest/api/cosmos-db-resource-provider/collectionpartition/listusages
Is there any TTL option on documents for Firebase Firestore . Where documents get auto deleted after that amount time
Update (2022-07-26): Firestore just added the option to set a time-to-live policy on collection groups. I'm still leaving the custom approach below, as those give you control over the expunge moment which (for now) isn't possible with the built-in feature.
The easiest way to build it yourself is by:
Adding a expirationTimestamp property to your documents.
Denying read of documents whose expiration has passed in your security rules.
match /collection/{document} {
allow read: if resource.data.expirationTimestamp > request.time.date();
}
Unfortunately this means that you won't be able to query the collection anymore. You'll need to access the individual documents.
Periodically run Cloud Functions code to delete expired documents.
Also see Doug's excellent blog post describing this process: How to schedule a Cloud Function to run in the future with Cloud Tasks (to build a Firestore document TTL).
Firestore TTL policies is now available in preview
https://cloud.google.com/firestore/docs/ttl
As of July 26th 2022, TTL Policies for Firestore were released as a preview feature (which means its not ready for production). Update Oct 2022: The TTL Policies feature has graduated from preview to General Availability, which means it should now be ready for production!
In order to use TTL Policies in Firestore, make sure that your documents have a field (of type Date & Time) to define the expiration date of that document (let's name the field expireAt for example).
And then follow the steps outlined in the documentation:
Go to the Cloud Firestore Time-to-live page in the Google Cloud Platform Console.
Go to the Time-to-live page.
Click Create Policy.
Enter a collection group name and the timestamp field name (expireAt in our example).
Click Create.