consul batch KV pairs HTTP post using HTTP api (Import a huge json through HTTP API) - uri

I have application configuration in a json with upto 80 key/value pairs per app, which are to be stored in Consul KV Store. Generally they are java keyvalue properties, I have a simple shell script, which does encode value and convert it into import compatible for consul import command. I am trying to automate the consul KV import for the new apps that on-board (apps that start using consul for KV Store).
So far we are manually running the consul kv import #app_config.json however, I would like to call HTTP api rather importing through consul KV import. Any suggestions please?
example for app config in json format.
[
{
"key": "asia",
"flags": 0,
"value": "NDQzNg=="
},
{
"key": "asia/india",
"flags": 0,
"value": "MTMyNA=="
},
{
"key": "europe",
"flags": 0,
"value": "NzQzLjE="
},
{
"key": "europe/france",
"flags": 0,
"value": "NjYuOQ=="
},
{
"key": "europe/germany",
"flags": 0,
"value": "ODIuNjc="
}
]

Basically you need to send PUT /v1/txn request with payload containing the array of transaction operations, like
const url = 'http://127.0.0.1:8500/v1/txn';
const bulk = [
{
"KV": {
"Verb": "set",
"Key": "bulk/hello",
"Value": "aGVsbG8="
}
},
{
"KV": {
"Verb": "set",
"Key": "bulk/world",
"Value": "d29ybGQ="
}
}
];
const options = {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json;charset=UTF-8'
},
body: JSON.stringify(bulk)
};
await fetch(url, options);

Related

Ocelot Swagger MMLib.SwaggerForOcelot showing "No operations defined in spec!"

I am using Ocelot gateway and for swagger document using "MMLib.SwaggerForOcelot" library.
For some swagger Key, swagger UI is showing "No operations defined in spec!" and swagger JSON is coming without paths like
{
"openapi": "3.0.1",
"info": {
"title": "Admin API",
"version": "v1"
},
"paths": {},
"components": {
"schemas": {}
}
}
Ocelot Configuration Route is
{
"DownstreamPathTemplate": "/api/admin/v{version}/{everything} ",
"DownstreamScheme": "http",
"DownstreamHostAndPorts": [
{
"Host": "localhost",
"Port": 5000
}
],
"UpstreamPathTemplate": "/api/admin/v{version}/{everything}",
"UpstreamHttpMethod": [],
"QoSOptions": {
"ExceptionsAllowedBeforeBreaking": 3,
"DurationOfBreak": 1000,
"TimeoutValue": 900000
},
"SwaggerKey": "AdminAPI"
}
and Swagger Configuration is
"SwaggerEndPoints": [
{
"Key": "AdminAPI",
"Config": [
{
"Name": "Admin API",
"Version": "v1",
"Url": "http://localhost:5000/swagger/v1/swagger.json"
}
]
}
]
after reviewing the MMLib.SwaggerForOcelot source code, it looks like something to do with the version in the downstream path, any clue on how this can be fixed?
The issue is that MMLib.SwaggerForOcelot is not considering {version} while doing Ocelot transformation.
RouteOptions has a property TransformByOcelotConfig which is set to true by default, so once swagger JSON is obtained from downstream, the transformation will be done.
here, it will try to find the route from the route configuration like below and if not found it will delete the route from swagger JSON
private static RouteOptions FindRoute(IEnumerable<RouteOptions> routes, string downstreamPath, string basePath)
{
string downstreamPathWithBasePath = PathHelper.BuildPath(basePath, downstreamPath);
return routes.FirstOrDefault(p
=> p.CanCatchAll
? downstreamPathWithBasePath.StartsWith(p.DownstreamPathWithSlash, StringComparison.CurrentCultureIgnoreCase)
: p.DownstreamPathWithSlash.Equals(downstreamPathWithBasePath, StringComparison.CurrentCultureIgnoreCase));
}
The problem is StartsWith will return false since swagger JSON path will be like
/api/admin/v{version}/Connections
and route config is like
/api/admin/v{version}/{everything}
and version will replace with the version given in swagger options so that it will become
/api/admin/v1/{everything}
Fix to this problem will be
Either set "TransformByOcelotConfig":false in swagger option
"SwaggerEndPoints": [
{
"Key": "AdminAPI",
"TransformByOcelotConfig":false,
"Config": [
{
"Name": "Admin API",
"Version": "v1",
"Url": "http://localhost:5000/swagger/v1/swagger.json"
}
]
}
]
Or Change the route, just to have {everything} keyword
{
"DownstreamPathTemplate": "/api/admin/{everything} ",
"DownstreamScheme": "http",
"DownstreamHostAndPorts": [
{
"Host": "localhost",
"Port": 5000
}
],
"UpstreamPathTemplate": "/api/admin/{everything}",
"UpstreamHttpMethod": [],
"QoSOptions": {
"ExceptionsAllowedBeforeBreaking": 3,
"DurationOfBreak": 1000,
"TimeoutValue": 900000
},
"SwaggerKey": "AdminAPI"
}

Integromat app {{connection.param}} not working

I am trying to setup an own App in Integromat
What is required for my App is an URL (and later a Bearer Token) to be entered manually by the user who wants to use my App.
I have the Apps Base:
{
"baseUrl": "{{connection.url}}",
"log": {
"sanitize": ["request.headers.authorization"]
}
}
a Connection:
Parameters:
[
{
"name": "url",
"label": "url",
"type": "text",
"required": true,
"value":"https://my-server"
}
]
and the Scenario:
{
"url": "/api/endpoint",
"method": "GET",
"qs": {},
"headers": "{{connection.headers}}",
"response": {
"output": "{{body}}"
}
}
When i execute, the scenario from my App. The URL seems not to be correctly taken over from the one configured inside the connection parametrs.
Can someone help?
Everything was right. I had to delete the old Connection and create a new one.

How to insert empty string in DynamoDB using the output of a Lambda in Step Functions?

I'm trying to save the output of a Lambda which calls Lex to DynamoDB using Step Functions.
The intentName in a Lex response is sometimes null (unknown). The problem is that in the state (task) that saves the response to DynamoDB, because of this empty string I get an error from DynamoDB.
Is there any workaround, maybe using JsonPath or the state machine diagram of the Step Function, in order to insert null or maybe no insert that specific property in DynamoDB?
Here is the JSON for the state machine:
{
"StartAt": "ProcessLex",
"States": {
"ProcessLex": {
"Type": "Task",
"Resource": "arn:aws:lambda:<Region>:<Account Id>:function:getIntent",
"ResultPath": "$.lexResult",
"Next": "ChoiceIfIntent"
},
"SaveToDynamo": {
"Type": "Task",
"Resource": "arn:aws:states:::dynamodb:putItem",
"Parameters": {
"TableName": "MyTable",
"Item": {
"dateTime": {
"S.$": "$.dateTime"
},
"intentName": {
"S.$": "$.lexResult.intentName"
},
"analysis": {
"M.$": "$.lexResult.sentimentResponse"
}
}
},
"End": true
},
"Comprehend": {
"Comment": "To be implemented later",
"Type": "Pass",
"End": true
},
"ChoiceIfIntent": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.lexResult.intentName",
"StringGreaterThanEquals": "",
"Next": "SaveToDynamo"
}
],
"Default": "Comprehend"
}
}
}
The problem is not the null value, the problem is that in DynamoDB with the PutItem Api you cannot insert empty strings.
I know this is frustrating but the quickest solution is to replace "" with NULL.
The solution that I prefer is to set the convertEmptyValue to true in your DynamoDb client settings.
const dynamodb = new AWS.DynamoDB.DocumentClient({ convertEmptyValues: true })
UPDATE
Since yesterday, DynamoDB supports empty values for string!
Take a look here.

LinkedIn Assets API: No AWS key returned when registering multi-part upload

I'm trying to upload a video > 200MB via https://api.linkedin.com/v2/assets. When registering a MULTIPART_UPLOAD I do not receive the "x-amz-server-side-encryption" or "x-amz-server-side-encryption-aws-kms-key-id" headers information anywhere in the response. I do get those when registering a SINGLE_REQUEST_UPLOAD and I'm able to successfully upload a video file < 200MB using that mechanism.
Example response for registering a multi-part upload:
{
"value": {
"uploadMechanism": {
"com.linkedin.digitalmedia.uploading.MultipartUpload": {
"metadata": "{METADATA}",
"partUploadRequests": [
{
"headers": {
"Content-Length": "5242880",
"Content-Type": "application/octet-stream"
},
"urlExpiresAt": 1547231882996,
"byteRange": {
"lastByte": 5242879,
"firstByte": 0
},
"url": "{AWS_UPLOAD_URL}"
},
{
"headers": {
"Content-Length": "5242880",
"Content-Type": "application/octet-stream"
},
"urlExpiresAt": 1547231882996,
"byteRange": {
"lastByte": 10485759,
"firstByte": 5242880
},
"url": "{AWS_UPLOAD_URL}"
},
{
"headers": {
"Content-Length": "3585789",
"Content-Type": "application/octet-stream"
},
"urlExpiresAt": 1547231883023,
"byteRange": {
"lastByte": 580302588,
"firstByte": 576716800
},
"url": "{AWS_UPLOAD_URL}"
}
]
}
},
"mediaArtifact": "urn:li:digitalmediaMediaArtifact:(urn:li:digitalmediaAsset:{ASSET_URN},urn:li:digitalmediaMediaArtifactClass:aws-userUploadedVideo)",
"asset": "urn:li:digitalmediaAsset:{ASSET_URN}"
}
}
I have tried to upload to those urls without the two headers and get a 403 Forbidden response saying I'm missing signed headers. The "x-amz-server-side-encryption" and "x-amz-server-side-encryption-aws-kms-key-id" should be returned in the register response right? If not, how do I make requests to aws without them?

Google Cloud Endpoints: Transcoding messages with protobuf.Any fields

I have a grpc service that takes as input a message that contains fields of type protobuf.Any and I can't figure out the way to write correct json input for it. I am running on GKE, with cloud endpoint ESP and my service running in the same pod.
The protos look like:
message AnyArray {
repeated google.protobuf.Any value = 1;
}
message Metric {
string metric = 1;
int64 timestamp = 2;
double value = 3;
map<string, AnyArray> metadata = 4;
}
I have tried multiple combinations for the input json with no luck, most of the times cloud endpoints returns "Proto field is not repeating, cannot start list." Failed examples:
{
"metadata": {
"sample-key": {
"value": [1, "one"]
}
},
"metric": "request-count",
"timestamp": 1528425789,
"value": 0
}
{
"metadata": {
"sample-key": {
"value": [{
"#type": "type.googleapis.com/google.protobuf.Duration",
"value": "1.212s"
}, {
"#type": "type.googleapis.com/google.protobuf.Duration",
"value": "1.212s"
}]
}
},
"metric": "request-count",
"timestamp": 1528425789,
"value": 0
}
Response from ESP
{
"code": 3,
"message": "metadata[0].value: Proto field is not repeating, cannot start list.",
"details": [{
"#type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "internal"
}]
}
Any help would be greatly appreciated.
Thanks!
google.protobuf.Any means any proto message type so your first example doesn't work. For the second example, google.protobuf.Duration have a special JSON mapping, so following example should work:
{
"metadata": {
"sample-key": {
"value": [{
"#type": "type.googleapis.com/google.protobuf.Duration",
"value": "1.212s"
}, {
"#type": "type.googleapis.com/google.protobuf.Duration",
"value": "1.212s"
}]
}
},
"metric": "request-count",
"timestamp": 1528425789,
"value": 0
}
Note you will have to include all possible proto message types for the google.protobuf.Any in the proto descriptor set provided Cloud Endpoints service, otherwise it will fail to translate unknown types.
Another possible approach is use google.protobuf.Struct for your metadata, which eliminates the limitation above, but you will have to convert it to your expected types in your gRPC service.
Using protobuf.Struct instead of protobuf.Any solved my problem as #lizan recommended. Thanks!

Resources