I have one parquet file ,trying to get the data in to the table. In one column it have json with multiple values. Can someone help me how to do in kusto?
Pasting the json's schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"path": {
"type": "string"
},
"partitionValues": {
"type": "object",
"properties": {
"deviceId": {
"type": "string"
},
"date": {
"type": "string"
}
},
"required": [
"deviceId",
"date"
]
},
"size": {
"type": "integer"
},
"modificationTime": {
"type": "integer"
},
"dataChange": {
"type": "boolean"
},
"stats": {
"type": "string"
}
},
"required": [
"path",
"partitionValues",
"size",
"modificationTime",
"dataChange",
"stats"
]
}
If I understood correctly, your PQ file contains a column with a JSON of the specified schema. If you want to ingest it as-is, ingest it into Kusto column with type "dynamic" and query later. If you'd like to ingest just part of this JSON data (like some inner fields), use ingestion mapping and provide appropriate JSON path.
Another option is to ingest as-is into a source table with a Retention policy with SoftDeletePeriod of zero. Define an Update Policy with a KQL query and the transformed data will be pushed into a target table.
Related
Here is the portion of my ARM file about the database:
{
"type": "Microsoft.Sql/servers/databases",
"name": "[concat(variables('SqlServerName'), '/', variables('dbName'))]",
"tags": {
"displayName": "Database"
},
"location": "[resourceGroup().location]",
"apiVersion": "2020-02-02-preview",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers', variables('SqlServerName'))]"
],
"properties": {
"collation": "[parameters('dbCollation')]",
"edition": "[parameters('dbEdition')]",
"maxSizeBytes": "53687091200",
"requestedServiceObjectiveName": "[parameters('dbRequestedServiceObjectiveName')]"
}
}
And here my parameters:
"dbEdition": {
"value": "Standard"
},
"dbRequestedServiceObjectiveName": {
"value": "S2"
}
But these database properties are ignored and the deployed database doesn't respect them.
What's wrong?
UPDATE
As #SwathiDhanwada pointed out in comments, the edition property may not exist as per the doc.
I should use sku defined like:
Yet the powershell script to get a list of available skus for a given location returns something like:
The headers of the table are not the same as the properties of the sku object 🤦♂️
Hey can someone help me with this,
This flow has been working okay but I tried using it today, but I got this error. kindly help me figure it out. attached is my error and the prior steps and the schema
{
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"link": {
"type": "string"
},
"id": {
"type": "string"
},
"type": {},
"size": {
"type": "integer"
},
"referenceId": {
"type": "string"
},
"driveId": {
"type": "string"
},
"status": {
"type": "integer"
},
"uploadSessionUrl": {}
},
"required": [
"name",
"link",
"id",
"type",
"size",
"referenceId",
"driveId",
"status",
"uploadSessionUrl"
]
}
}
I have tried changing the dynamic input, but I still get other errors stating that the items cannot be found. Here is the step prior. I have also tried to create a variable called ‘Test’ or anything else that makes it easy to identify. Then I set that variable to the body/r053… then when I run the flow I can see what content is going into that variable before the flow fails.
I have tried looking at some solved solutions online, but I still got nothing substantial to solve this.
I want to copy items from
CosmosDB databaseA/productCollection
to
CosmosDB databaseB/productCollection
Therefore I decided to use Azure Data Factory.
I actived also "Export as-is to JSON files or Cosmos DB collection".
The read operation works as expected.
Unfortunately, the write operation stops because of an error related to the data:
ErrorCode=InvalidTemplate, ErrorMessage=Unable to parse expression 'Currency'
{
"ProductName": "Sample",
"Price": {
"#Currency": "GBP",
"$": "2624.83"
}
}
I'm not able to change to input data itself.
The output data has to equal the input data.
Is there possiblity, that #Currency will not be interpreted as an expression
In ARM, this part is failling:
Price.{#Currency}
I had the same problem and I was able to resolve accordingly.
I am using a Pipeline with a Source that is a Dataset referencing JSON data.
Clicking the button highlighted below.
I had to change the JSON from
{
"name": "SourceDataset",
"properties": {
"linkedServiceName": {
"referenceName": "StorageAccountLink",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "Json",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "test-data"
}
},
"schema": {
"type": "object",
"properties": {
"#context": {
"type": "string"
},
"value": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
}
}
}
}
}
}
}
}
To ( Escaping the # with ## )
{
"name": "SourceDataset",
"properties": {
"linkedServiceName": {
"referenceName": "StorageAccountLink",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "Json",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "test-data"
}
},
"schema": {
"type": "object",
"properties": {
"##context": {
"type": "string"
},
"value": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
}
}
}
}
}
}
}
}
I tried to reproduce your issue but it works for me. I used copy activity to transfer data from account A to account B.
Additional, if this operation is just need to be executed once, please consider using Azure Cosmos DB Migration Tool. It's free for usage. You could export the data from cosmos db A as json file then import it into cosmos db B very simply.Also, it could be executed in the cmd so that it could be made as a scheduled job on the windows system.
I am preparing ARM template for "Schedule update deployment" in Update Management service. I want to add parameters like: "excludedKbNumbers" and "includedKbNumbers". I am deploying my templates using powershell. When I am passing KB numbers using mentioned parameters templates completed successfully. In case when I am putting KB number using one of the mentioned parameters, second is empty, template completed successfully. Problem is when I dont want to pass Included/Exluded KB numbers, in my powershell deployment command I am not putting parameter names "excludedKbNumbers" and "includedKbNumbers", and then I am receiving below error: "message": "{\"Message\":\"The request is invalid.\",\"ModelState\":{\"softwareUpdateConfiguration.properties.updateConfiguration\":[\"Software update configuration has same KbNumbers in
includedKbNumbers and excludedKbNumbers.\"]}}"
I am using this structure in my template json('null') and this is a problematic area.
extract from my template:
"parameters": {
"excludedKbNumbers": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Specify excluded KB numbers, required data structure: 123456"
}
},
"includedKbNumbers": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "Specify included KB numbers, required data structure: 123456"
}
},
"resources": [
{
"type": "Microsoft.Automation/automationAccounts/softwareUpdateConfigurations",
"apiVersion": "2017-05-15-preview",
"name": "[concat(parameters('automationAccountName'), '/', parameters('scheduleName'))]",
"properties": {
"updateConfiguration": {
"operatingSystem": "[parameters('operatingSystem')]",
"windows": {
"includedUpdateClassifications": "[parameters('Classification')]",
"excludedKbNumbers": [
"[if(empty(parameters('excludedKbNumbers')), json('null'), parameters('excludedKbNumbers'))]"
],
"includedKbNumbers": [
"[if(empty(parameters('includedKbNumbers')), json('null'), parameters('includedKbNumbers'))]"
],
"rebootSetting": "IfRequired"
},
"targets": {
"azureQueries": [
{
"scope": [
"[concat('/subscriptions', '/', parameters('subscriptionID'))]"
],
"tagSettings": {
"tags": {
"[parameters('tagKey')]": [
"[parameters('tagValue')]"
]
},
"filterOperator": "All"
},
"locations": []
}
]
},
"duration": "PT2H"
},
"tasks": {},
"scheduleInfo": {
"isEnabled": false,
"startTime": "2050-03-03T13:10:00+01:00",
"expiryTime": "2050-03-03T13:10:00+01:00",
"frequency": "OneTime",
"timeZone": "Europe/Warsaw"
}
}
}
],
try doing this:
"excludedKbNumbers": "[if(empty(parameters('excludedKbNumbers')), json('null'), array(parameters('excludedKbNumbers')))]",
"includedKbNumbers": "[if(empty(parameters('includedKbNumbers')), json('null'), array(parameters('includedKbNumbers')))]"
We've done quite a bit of swagger modelling, but one thing I am struggling with is how we could define that all our arrays will never be null, but will be an empty collection if we have no data.
Is there a way of explicitly defining this in Swagger?
Thanks
To define a model with a mandatory array, you can use required property :
{
"type": "object",
"required": [
"nonNullableArray"
],
"properties": {
"nonNullableArray": {
"type": "array",
"items": {
"type": "string"
}
},
"nullableArray": {
"type": "array",
"items": {
"type": "string"
}
}
}
}