I tried to sign up for the "Analytical Storage" preview feature in the Azure Portal for my Cosmos DB account in North Europe. However it has been stuck in "Pending" state for two weeks now. Is this feature still available and if so how can I get it enabled?
Also how is the analytical storage accessed? According to https://learn.microsoft.com/en-us/azure/cosmos-db/globally-distributed-transactional-analytical-storage it's “Apache Spark”. Does this refer to the Cosmos DB Spark Connector or how do you query the underlying Parquet files? Also will it be possible to move the Parquet files to external storage (e.g. ADLS) for consumption by other BI tools?
Analytical Storage is currently a gated preview. This should go to public preview sometime this Summer.
The storage itself is fully managed so you will not be able to access the parquet files or storage directly. The only way to access this will be through the Synapse Analytics service.
thanks.
I have a cosmosDB with mongo API, however there are many features not supported in this API so I want to switch to SQL API?
As things stand today it is not possible to switch you account from one type to another.
This is what you need to do if you need to switch to the SQL API:
Create a SQL API Cosmos DB Account
Download the Cosmos DBmigration tool
Migrate all the data from the MongoDB account to the SQL API account
Delete the MongoDB account
I created Azure CosmosDb database and container for my documents.
I use NiFi as a main data ingestion tool and want to feed my container with documents from NiFi flow files.
Can anybody please share a way to post flowfile content to Azure Cosmos Db from NiFi?
Thanks in advance
UPDATE(2019.05.26):
In the end I used Python script and called it from NiFi to post messages. I passed a message as a parameter. The reason I chose python is because it has some examples on official Microsoft site with all the required connection settings and libraries, so it was easy to connect to Cosmos.
I tried Mongo component, but couldn't connect to Azure (security config didn't work), didn't really go too far with it as Python script worked just fine.
Azure CosmosDB exposes MongoDB API so you can use the following MongoDB processors which are available in NiFi to read/query/write to & from Azure CosmosDB using Apache NiFi.
DeleteMongo
GetMongo
PutMongo
PutMongoRecord
RunMonogAggregation
Useful Links
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-feature-support
Valeria. According to the components list supported by Apache Nifi related to Azure, you could only get Azure Blob Storage, Queue Storage, Event Hub etc,not including Cosmos DB.
So,I suggest you using PutAzureBlobStorage to feed azure blob container with documents from NiFi flow files. Then please create a copy activity pipeline in Azure Data Factory to transfer data from Azure Blob Storage into Azure Cosmos DB.
Google Cloud Firestore is going to replace the legacy Google Cloud Datastore soon. One then has the choice between using Cloud Firestore in "native mode" or in "datastore mode". The former allows access to Firestore through the usual Firestore SDK while the latter allows usage of the old Cloud Datastore SDK (which has no Web/Mobile APIs).
I am not yet familiar with Firestore. My question is: Apart from porting things to a new API, are there actually any things that can not be done with Firestore in "native mode" which could be done with the old Cloud Datastore (or its replacement: Firestore in "datastore mode")? Or any other advantage of using "datastore mode" (like costs, for example)?
If not, then it seems there is actually no advantage of using Firestore in "datastore mode" other than compatibility for older code using the old Cloud Datastore.
Am I right in my assumption that Firestore "datastore mode" has absolutely no advantage besides being able to use the legacy Datastore API (at the cost of not being able to use the newer and probably more feature-rich Firestore APIs, including mobile and web APIs)?
Cost.
Firestore in Datastore mode supports key-only and projection queries just like the original datastore. That means that the result set of those queries counts towards "Cloud Firestore Small Operations" which are free.
We accumulate billions of those small operations per day and not having projection queries would effectively 10-fold our datastore cost which would be unbearable.
As this feature is not available in Firestore native mode - and also with the strong consistency added in - we expected it to be less performant then in the original datastore, but in our tests this was not the case. Firestore in datastore mode consistently performed about twice as fast for our application across all types of operations.
There are advantages of using "datastore mode", even for a new project.
I am evaluating the two modes of firestore "Datastore mode" and "Native mode" for a migration project from MySQL.
On one hand, I consider using the "Datastore mode" for one back-end global repository because:
Server side only
Strong performance expectation on search queries across all entities
Query and sort on several properties altogether
Structured data model with one root kind and few second level kinds
A lot of write with limited transaction requirement, a huge number of read with projection within an entity group
On the other hand, the "Native mode" seems to fit some requirements for a user facing specific application because:
Web, iOS, Android, API interface with bi-directional sync
Several occasionally connected use cases
Few large polymorphic objects to sync and to persist
Mostly simple query on one properties (parent object)
Though, there is one reason which advocate for Datastore mode for the second project
Multi-tenancy with namespace
There are also common needs fulfilled by both mode, which support the decision to migration to NoSQL technologies
Scalability
No admin
Availability
Speed of development
The items 2 to 5 and 10 are based on features specific for the Datastore Mode, not possible in Native Mode. The items 6 to 9 are specific to Native Mode.
Update : March, 21st 2019
Six months after the evaluation described in my first answer, my team is using both Firestore (native) mode and Datastore mode.
2 projects based on Firestore. We are using a lot the concepts of collection, sub-collection and documents and the undelying segregation of data.
We also have implemented listeners in iOS and Android apps for sub-collections selected accordung to strong business and security rules, which is not possible with Datastore.
9 projects based on Datastore. For three 3 of them, we are are using a lot the concepts of namespace and kind. We also use the global indexing of kind's properties and the projection of properties server-side, which is not possible with Firestore.
PS: we are considering to open-source our python library for fast development of a common API running either on Firestore and on Datastore.
According to official documentation, although Cloud Firestore is backwards compatible with Cloud Datastore, the new data model, real-time updates, and mobile and web client library features are not.
Cloud Firestore in Datastore mode uses Cloud Datastore system behavior but accesses Cloud Firestore's storage layer, removing the following Cloud Datastore limitations:
Eventual consistency, all Cloud Datastore queries become strongly
consistent.
Transactions are no longer limited to 25 entity groups.
Writes to an entity group are no longer limited to 1 per second.
Datastore mode disables Cloud Firestore features that are not compatible with Cloud Datastore:
The project will accept Cloud Datastore API requests and deny Cloud Firestore API requests.
The project will use Cloud Datastore indexes instead of Cloud Firestore indexes.
You can use Cloud Datastore client libraries with this project but not Cloud Firestore client libraries.
Cloud Firestore real-time capabilities will not be available.
In the GCP console, the database will use the Cloud Datastore viewer.
Because Firestore in Datastore mode is optimized for server use cases and for App Engine, we recommend using Firestore in Datastore mode for databases that will be used primarily by App Engine apps. Firestore in Native mode is most useful for mobile and real-time notification use cases.
https://cloud.google.com/appengine/docs/flexible/go/using-cloud-datastore
When looking further into the docs after reading this post, I came across this official GCP page. It explains in plain English that Firestore "Datastore mode" recommended for primarily server-side use cases.
This answers my question perfectly, because my use case is only server-side. Note: I don't have any first hand experience with Firestore yet, but I appreciate the answers that do.
Hello and I am new to developing with Qt for cross platform development. My current goal is to be able to cache data from a CosmosDB with my app that I've developed in Qt. I've had no problem setting up the CosmosDB, but I am confused at what the best way to communicate with the database is.
Do I need to create my own API to talk with the database? Are there libraries out there already do what I want?
You can use either standard HTTP requests, or use SQL API.
Azure Cosmos DB exposes resources through the REST APIs that can be called by HTTP/HTTPS requests. How can I develop apps with the SQL API
Azure Cosmos DB REST API reference
Also Azure Cosmos DB offers a query language as an interface to query JSON documents. The language supports a subset of ANSI SQL grammar and adds deep integration of JavaScript object, arrays, object construction, and function invocation. Microsoft shows exmaples here