I want to monitor Azure Storage Account but I'm not getting any documentation on ARM Rest API for getting the details like availability, metrics that can be collected etc.
In my resource group I have a storage account, so when I form the URL and do get using:
https://management.azure.com/subscriptions/452529bb-083b-411a-a5c2-30c7352111d0/resourceGroups/SparkTraining/providers/Microsoft.Storage/storageAccounts/sparktrainingdisks521?api-version=2017-06-01
I get the response.
But how to monitor blob, table, queue and file? also, what are the metric's that can be collected for these?
We can get the storage metrics from the following azure storage tables.
And we can get the table entity with Azure storage table Query Entity RestAPI. How to construct the authorization Header please refer to this tutorials.
More info about these tables schema please refer to tutorials.
Related
I am confused as to how to connect to my Firebase DB using, say Postman to send the REST requests.
In the screenshot below, you can see my DB structure and GET request.
Specifically, the GET request in Postman looks like this,
https://fir-crud-api-b1cec.firebaseio.com/products/test.json?auth=xxxx
while the DB is structured as such
products > test.json
Would appreciate any advice to point me in the right direction. Thanks!
Your URL suggests that you want to use the Realtime Datbase REST API, but your screenshot suggests that your data is in Firestore. These are different databases with different APIs.
If you want to work with data in Firestore, you should use the Firestore REST API instead.
The screenshot on the left shows the Cloud Firestore database, while in PostMan you are trying to access the Realtime Database. While both databases are part of Firebase, they're completely separate, and have completely separate REST APIs.
If you want to access Cloud Firestore from PostMan, have a look at its REST API. Fair warning that this API is significantly more complex than the one for the Realtime Database.
If you want to continue using the Realtime Database REST API, make sure to create your data structure in the Realtime Database in the Firebase console.
I created Azure CosmosDb database and container for my documents.
I use NiFi as a main data ingestion tool and want to feed my container with documents from NiFi flow files.
Can anybody please share a way to post flowfile content to Azure Cosmos Db from NiFi?
Thanks in advance
UPDATE(2019.05.26):
In the end I used Python script and called it from NiFi to post messages. I passed a message as a parameter. The reason I chose python is because it has some examples on official Microsoft site with all the required connection settings and libraries, so it was easy to connect to Cosmos.
I tried Mongo component, but couldn't connect to Azure (security config didn't work), didn't really go too far with it as Python script worked just fine.
Azure CosmosDB exposes MongoDB API so you can use the following MongoDB processors which are available in NiFi to read/query/write to & from Azure CosmosDB using Apache NiFi.
DeleteMongo
GetMongo
PutMongo
PutMongoRecord
RunMonogAggregation
Useful Links
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-feature-support
Valeria. According to the components list supported by Apache Nifi related to Azure, you could only get Azure Blob Storage, Queue Storage, Event Hub etc,not including Cosmos DB.
So,I suggest you using PutAzureBlobStorage to feed azure blob container with documents from NiFi flow files. Then please create a copy activity pipeline in Azure Data Factory to transfer data from Azure Blob Storage into Azure Cosmos DB.
I'm new to CosmosDB and have used Data Factory to import some test data from a BLOB into a CosmoDB container. The Monitor screen tells me it was successful. I then went to the Azure portal, opened my container and clicked 'Documents' but this does not show me any data. I then clicked the refresh button in the sub-pane (the one on the 'load more' section) and it gave me the error:
Error while fetching page of documents:
{"code":400,"body":"Command find failed: Unknown server error occurred when processing this request.."}
I also could not find any good tutorials online or on Youtube that shoes step by step how to import a CSV from BLOB storage into ComosDB Document store via DataFactory, so unable to tell if I am doing it correctly.
I contacted Microsoft. Response was: "The Azure Data Factory loads data using the SQL API SDK, and does not support mMngo yet. The data loaded using the SQL API SDK would have to be in MongoDB BSON schema. Also, the Mongo DB Native driver expect the data in JSON schema and fails to deserialize triggerin the 400 error."
The MongoBulkExecutor API was recommended as an alternative but from what I can tell this really requires json too.
I also could not find any good tutorials online or on Youtube that
shoes step by step how to import a CSV from BLOB storage into ComosDB
Document store via DataFactory, so unable to tell if I am doing it
correctly.
In fact, you should check the below components when you import csv from blob storage into cosmos db.
1.You already have created cosmos db Linked Service and DataSet.
2.You already have created cosmos db Linked Service and DataSet.
You could do the above steps on the portal.
3.Create copy activity and fill the blob storage input and cosmos db output into the activity.
4.In addition,you need to know cosmos db sql api and cosmos db mongo api are different api,though they call named cosmos db. Based on Supported capabilities in document: Copy data to or from Azure Cosmos DB by using Azure Data Factory, Azure Cosmos DB connector only supports Copying data from and to the Azure Cosmos DB SQL API. So,please don't confuse.
If you do want to use Mongo api, you could choose mongo connector to do your jobs which is mentioned in this case:https://social.msdn.microsoft.com/Forums/security/en-US/52cddbf7-c132-490c-9088-65a38f9b7200/copy-activity-to-cosmosdb-with-mongo-api?forum=AzureDataFactory.
Azure CosmosDB - MongoDB provides keys for Read-Write and Read Only at account level.
CosmosDB SDK and API are there through which users can be created and access can be define at database and document level.
• But What I need to do is to create a pair of username and password with restricted access to a MongoDB database similar to one provided by installable MongoDB.
• How a user can connect only CosmosDB MongoDB database using RoboMongo.
Highly appreciate any help.
Amit -
Today, Cosmos DB access are provided by using two keys, Master Key and Read Only key. However, if you want to restrict user access per collection, per document etc, you have to use Resource Tokens. You can read more about it here and please take a look at CH9 video to see the implementation details. Resource Token service can be implemented as an Azure Function. Here is code to get you started.
But if you are using RoboMongo you have to Use the keys as define in this document. At this time you cannot define different users and Keys for a Database.
Is there any tool to monitor the data transfer to my real time database in firebase? To see how many user connect at live time and how much data is transferred?
There's a usage tab in your Firebase Database console that has most of that information.
We also just released a tool that allows you to profile your database usage. See that tool's blog post and reference documentation.