I created Azure CosmosDb database and container for my documents.
I use NiFi as a main data ingestion tool and want to feed my container with documents from NiFi flow files.
Can anybody please share a way to post flowfile content to Azure Cosmos Db from NiFi?
Thanks in advance
UPDATE(2019.05.26):
In the end I used Python script and called it from NiFi to post messages. I passed a message as a parameter. The reason I chose python is because it has some examples on official Microsoft site with all the required connection settings and libraries, so it was easy to connect to Cosmos.
I tried Mongo component, but couldn't connect to Azure (security config didn't work), didn't really go too far with it as Python script worked just fine.
Azure CosmosDB exposes MongoDB API so you can use the following MongoDB processors which are available in NiFi to read/query/write to & from Azure CosmosDB using Apache NiFi.
DeleteMongo
GetMongo
PutMongo
PutMongoRecord
RunMonogAggregation
Useful Links
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-feature-support
Valeria. According to the components list supported by Apache Nifi related to Azure, you could only get Azure Blob Storage, Queue Storage, Event Hub etc,not including Cosmos DB.
So,I suggest you using PutAzureBlobStorage to feed azure blob container with documents from NiFi flow files. Then please create a copy activity pipeline in Azure Data Factory to transfer data from Azure Blob Storage into Azure Cosmos DB.
Related
I have a project where my end users are using Alteryx to combine data sources and import into SQL server. We are investigating changing SQL server to CosmoDB but I'm not sure if connecting to CosmoDB is supported by Alteryx.
Do anyone know if there is a connector? Am I overthinking this? I don't know Alteryx very well.
There is currently no dedicated connector from Azure Cosmos DB.
However, the Azure Cosmos DB ODBC driver enables you to connect to Azure Cosmos DB using BI analytics tools.
Also, there is an existing REST API for Cosmos DB, it should be possible to access it through a Downloaded tool. You can suggest in the idea section of Alteryx Community to request one.
I am trying to connect Azure Data explorer plugin data source in grafana but we don’t have cluster setup for my azure application and I can’t provide cluster url in that plugin. I tried simple JSON plugin but it requires a http url and our Azure portal url is https.
Do we have any alternatives plugins or any other way to access Azure cosmos db data SQL API into grafana?
Thanks for your help!
Regards,
Sreenivasa
Sreeni. There is a Grafana use case for Azure and for Azure Cosmos DB but only through the Azure Monitor plug-in to view timeseries data of your Azure service. Grafana cannot be used as a client to the Azure SQL API, if I understand your question correctly.
Please see: Deploying Grafana for production deployments on Azure
Following the advice in this article, I have configured my applications to use region-specific Gremlin endpoints so that reads and writes are always directed to the master replica in the same data centre (the Cosmos DB account is multi-master and the applications are deployed to every region on AKS). My question is this: in the event of a regional Cosmos DB outage, what will the behaviour be when using region-specific Gremlin connection strings? Will applications that reference a regional endpoint that is affected by an outage be automatically redirected to a region where the Cosmos replica is healthy?
This depends upon the client SDK the application is using to connect and the connection string logic. If the application connection string is pointed to the .NET SDK URI, then you will want to implement either the .NET SDK v2 or .NET SDK v3 multi-master functionality. If you are using the Gremlin Endpoint, please follow the specific guidance: Regional endpoints for Cosmos DB for Graph Accounts
Once that is configured correctly, in the event of an outage, the routing will automatically be redirected to an available write region.
I'm new to CosmosDB and have used Data Factory to import some test data from a BLOB into a CosmoDB container. The Monitor screen tells me it was successful. I then went to the Azure portal, opened my container and clicked 'Documents' but this does not show me any data. I then clicked the refresh button in the sub-pane (the one on the 'load more' section) and it gave me the error:
Error while fetching page of documents:
{"code":400,"body":"Command find failed: Unknown server error occurred when processing this request.."}
I also could not find any good tutorials online or on Youtube that shoes step by step how to import a CSV from BLOB storage into ComosDB Document store via DataFactory, so unable to tell if I am doing it correctly.
I contacted Microsoft. Response was: "The Azure Data Factory loads data using the SQL API SDK, and does not support mMngo yet. The data loaded using the SQL API SDK would have to be in MongoDB BSON schema. Also, the Mongo DB Native driver expect the data in JSON schema and fails to deserialize triggerin the 400 error."
The MongoBulkExecutor API was recommended as an alternative but from what I can tell this really requires json too.
I also could not find any good tutorials online or on Youtube that
shoes step by step how to import a CSV from BLOB storage into ComosDB
Document store via DataFactory, so unable to tell if I am doing it
correctly.
In fact, you should check the below components when you import csv from blob storage into cosmos db.
1.You already have created cosmos db Linked Service and DataSet.
2.You already have created cosmos db Linked Service and DataSet.
You could do the above steps on the portal.
3.Create copy activity and fill the blob storage input and cosmos db output into the activity.
4.In addition,you need to know cosmos db sql api and cosmos db mongo api are different api,though they call named cosmos db. Based on Supported capabilities in document: Copy data to or from Azure Cosmos DB by using Azure Data Factory, Azure Cosmos DB connector only supports Copying data from and to the Azure Cosmos DB SQL API. So,please don't confuse.
If you do want to use Mongo api, you could choose mongo connector to do your jobs which is mentioned in this case:https://social.msdn.microsoft.com/Forums/security/en-US/52cddbf7-c132-490c-9088-65a38f9b7200/copy-activity-to-cosmosdb-with-mongo-api?forum=AzureDataFactory.
Hello and I am new to developing with Qt for cross platform development. My current goal is to be able to cache data from a CosmosDB with my app that I've developed in Qt. I've had no problem setting up the CosmosDB, but I am confused at what the best way to communicate with the database is.
Do I need to create my own API to talk with the database? Are there libraries out there already do what I want?
You can use either standard HTTP requests, or use SQL API.
Azure Cosmos DB exposes resources through the REST APIs that can be called by HTTP/HTTPS requests. How can I develop apps with the SQL API
Azure Cosmos DB REST API reference
Also Azure Cosmos DB offers a query language as an interface to query JSON documents. The language supports a subset of ANSI SQL grammar and adds deep integration of JavaScript object, arrays, object construction, and function invocation. Microsoft shows exmaples here