How to execute ElasticSearch agg query from Apache Nifi using InvokeHTTP? - http

I want to execute the following query from Apache Nifi:
GET /myindex/_search
{
"size": 0,
"aggs": {
"range": {
"date_range": {
"field": "Datetime",
"ranges": [
{
"from": "2017-02-17T16:00:00Z||-1H/H",
"to": "2017-02-17T16:00:00Z||/H"
}
]
}
}
}
}
I want to get the value of doc_count.
I tried to use InvokeHTTP and directly pasted the above-defined query into the field HTTP Method. I also set Remote URL to http://localhost:9200. I connected InvokeHTTP with PutFile in order to save a response on the disk. The relationship is Response.
When I run InvokeHTTP, it does not give me any error. However, it neither outputs any result (FlowFile for Response). I am sure that the result is not an empty set, because I tested this query with curl.
What is wrong with my approach? Should I define HTTP Method in some different way?

For verbs like POST, PUT, and PATCH, you'd normally have the above JSON body as the content of a flow file and pass that to InvokeHttp, setting the correct verb and URL there. However the InvokeHttp documentation states that the message body will not be sent for a GET verb.
The good news is that the Elasticsearch REST API for the search endpoint supports both GET and POST. From their latest doc: "Both HTTP GET and HTTP POST can be used to execute search with body. Since not all clients support GET with body, POST is allowed as well."
I would set the content of a flow file to the above JSON body (perhaps using GenerateFlowFile or ReplaceText), then use POST as the verb.

There is a JsonQueryElasticsearch processor, which supports JSON elasticsearch queries.
Your query goes into the Query field.
The processor has hits and aggregations relations for you to process the result.

Related

What to do with an array of Salted Base64 strings?

I have been given this challange, to "hack" a website to register in it, with some given hints. It includes 3 steps.
1.site_url/login - a POST request with login credentials (username, password - I have these) in the body of the request. After this it gives a JSON response that looks like this:
{
"token": "U2FsdGVkX18VaaqQc/R3Xi3jQtMMlPNku0YJzn0KNMYX0GY2ZELDfA5smRduUs5Cf519WmgaQnA+j6MpwCsvi/699R5oaUdXHCrgzrsZEKM="
}
Every time I send the request I get a different token from the previous one.
site_url/keys - a GET request with the token in the Authorization field of the header. It returns an array of 500 base64 strings that look like this:
[
"U2FsdGVkX19UgyaPxxLVM2J5LIzQPR+FDjjMWkSWcOseMSfGPWTrnC4EAIzB6EbmKS9jewVBq9BCf9FiHQDlxipYADA3A2i+jTYt0028sOrd/dkrAZCXVJBbNUDWYy6+",
"U2FsdGVkX1/9YThiCftxiLRK6GpEY6iouivp5eGCzCfv+HVoKeaS8z/Ut7BFWAm4yVTUasl87MM2pR47EIVJZ8A62sPmfTtGabz9PMlOKCnf1UKRAZFr69dZzQy71jc7",
......
"U2FsdGVkX1/yN0jrC5VPyzbiLZ5HAiPREyojo9sb+dUw+pYcGmIUocoh9m8SeQsItKFElyVz/7xhaGkrBmpvOsdFNLFsIcfObVqZ1H7T9ZAPXoZibg9+tVRDYV/3VQWm"
]
Every time I send this it gives me a different strings from the previous ones.
3.site_url/register - a POST request with the token in the Authorization field of the header (without it responds "unathorized") and one of the keys from the aray in the body JSON as such:
{
"key": "U2FsdGVkX18vwo3TVGLIwbxvkJ4NIf1GhBBIkNw9deRciB9O6/aC9KkFxVZ09WrxzB2YFncchsNY/hZYec/Hxvj1wlCK+7iZAyqNaW0hIBm17lZEloIwJVVfjX9wlkVr"
}
It returns:
{ error: 'Forbidden: Invalid registration key' }
I am assuming that I need to "bruteforce" those 500 keys and see which one works, but so far I couldn't do that as after few requests the website gives error 503 or 502. From my side it seems like the website is getting down (even when I check on my phone with mobile data). After couple of minutes it again starts working. I am using node.js and fetch API.
List of things I tried so far:
-changed referrer in the fetch options
-generated random IP adress for 'X-Forwarded-For' in the request header
-put these in the fetch options credentials: "omit", cache: "no-store"
-tried doing step 2 and changin the Authorization field in the header for every key in the array
List of things I want to try but don't know how:
-try decrypting those salted Base64 strings
-try somehow combining those strings in the array into a file
Hope the explanation was clear.
What can I try else or what am I doing wrong?

Deleting all tags on an Azure resource/resgroup via REST

According to the Docs, I can update the tags on a resource group or resource by making a PATCH request and setting the tags field in the body. This works for setting one or more tags, or deleting one or more tags (I just leave the tagname out of the request).
However, when I try to delete all the tags by passing an empty tags field:
PATCH https://management.azure.com/{....}
{
"tags": {}
}
it doesn't work. For a resource, I get a 400 error Request must specify an account property to update while for a resource group, it just leaves the current tags unchanged.
Is there a way to delete all the tags on a resource/resource group via REST?
You could try to use this api: Resources - Update By Id, it should apply to both the resource and resource group. I have test it with the resource group and a resource(my sample is a web app), all work fine.
The request body:
{
tags: {}
}
Note: If you get a NoRegisteredProviderFound error, change the api-version depends on the error message.
This was actually due to a wart in how the httr package handles empty fields in a request body. If the body is a list, such fields are silently removed before being sent to the host.
The fix is to convert the data into JSON beforehand, rather than relying on httr to do it:
httr::PATCH("https://management.azure.com/{....}",
body=jsonlite::toJSON(body, auto_unbox=TRUE),
encoding="raw",
...)

How to verify an image contains in the response via pact test

I want to use pact test to verify if a provider could provide a image in a response. now the actual response looks like this
server response body which contains the picture
My Pact Json File looks like this
"interactions": [
{
"providerState": "there's a user has a portrait tiger.png",
"description": "Get the user's portrait",
"request": {
"method": "GET",
"path": "/api/Employees/v1/Employee/106656048406528/Attachments",
"headers": {
"Content-Type": "application/json",
"X-Employee-ID": "106656048406528",
"X-Tenant-ID": "26663977872256",
"X-User-ID": "1333"
}
},
"response": {
"status": 200,
"headers": {
"Content-Type": "image/jpeg;charset=UTF-8"
},
"body": "iVBORw0KGgoAAAANSU",
"matchingRules": {
"$.body": {
"match": "type"
}
}
}
}
],
the pact test always fails and say the body mismatch. I can understand the actual response is not a base64 format and not a string. it should some binaries file. In this case, how can I write the matching rules to valid it's a binary file?
The way to do this would be to ignore the body unless you really want to match the exact binary every time, which would be counter productive in my opinion.
When a binary file is sent, it does send the appropriate headers with it, like you already have in your contract with 'Content-Type'. You can always add a matching rule that the body has to be there, but you just don't care about the content, but I'm not positive if this is even needed since you could just look at the 'Content-Length' header and make sure it's bigger than 0.
What needs to be remembered is that Pact is not meant to be used to match everything. It's meant to match the critical parts of your system and certain expectations from your consumer, ie. as a consumer, I want to call this service which should give me an image, but I don't care what that image is since I just show it in the DOM. If you need an actual image to be given on the consumer side while testing, you can use the matcher query to resolve to anything by using regex, then as the generator, use the actual image output. I'd use something small though if I were you since Pact isn't exactly meant to be spewing out megabytes of binaries.
In my opinion, only the header check is enough. (Unless you want to verify entire binary content, which I think is not the best).

How to send a POST request for the Custom Location Extension

I am trying out the HERE Maps REST API's - Custom Location Extension and run into a problem. It was mentioned that a POST request using the Corridor Search Using Coordinates for Custom Locationscan be used instead of a GET request if I want to submit a request with a large coordinate set, but the example POST request was not in the documentation.
I tried to figure out how to submit a POST request and this was how far I got:
Request:
POST /v1/search/corridor HTTP/1.1
Host: customlocation.cit.api.here.com
?app_id={YOUR_APP_ID}
&app_code={YOUR_APP_CODE}
&route=52.51978,13.388211,
52.5198326,13.3882084,
52.5198402,13.3883495,
52.5147705,13.3891602,
52.514758,13.389155
&radius=100
&layerId=30
Response:
{
"message": "Required parameters have not been provided",
"status": "400 Bad Request",
"error": "Required String parameter 'route' is not present",
"targetLayerId": null
}
Notes:
I wrote it here using the HTTP format
This request was the same GET request in the documentation and I just tried to convert it as a POST request by moving the query parameter into the body and sending the request using POST.
Body datatype is set as RAW TEXT
Edit:
(Further trial)
I also tried to move all the query params from the body to the url except route.
POST /v1/search/corridor ?app_id=Qk8YkRrHMbbbpkVipwIZ
&app_code=tYSXrAAHGEOcUB_cxbPQSA&radius=100&layerId=852 HTTP/1.1
route=52.51978,13.388211, 52.5198326,13.3882084,
52.5198402,13.3883495,
52.5147705,13.3891602,
52.514758,13.389155
Response:
{
"message": "Required parameters have not been provided",
"status": "400 Bad Request",
"error": "Required String parameter 'route' is not present",
"targetLayerId": null
}
I poked at this a bit and was able to get a response via POSTing by removing the route from the querystring and sending it as a form key/value pair. I used the demo credentials, and am testing using the Chrome plugin "Postman". My POST URL looks like this:
https://customlocation.cit.api.here.com/v1/search/corridor?layerId=30&radius=100&app_id=DemoAppId01082013GAL&app_code=AJKnXv84fjrb0KIHawS0Tg
and then I set either a form-data or x-www-form-urlencoded key of "route" and then the value was a string of coordinates. And then it came back with a successful response. I did get the same error you mentioned when trying to put any payload as raw text.
Things I don't know: is there some limit on the form data values?
Good luck. I do agree that the official document is pretty light on the POST examples, especially since they explicitly suggest that as a method for sending large payloads.

What is the difference between PUT, POST and PATCH?

What is the difference between PUT, POST and PATCH methods in HTTP protocol?
Difference between PUT, POST, GET, DELETE and PATCH in HTTP Verbs:
The most commonly used HTTP verbs POST, GET, PUT, DELETE are similar to CRUD (Create, Read, Update and Delete) operations in database. We specify these HTTP verbs in the capital case. So, the below is the comparison between them.
Create - POST
Read - GET
Update - PUT
Delete - DELETE
PATCH: Submits a partial modification to a resource. If you only need to update one field for the resource, you may want to use the PATCH method.
Note:
Since POST, PUT, DELETE modifies the content, the tests with Fiddler for the below url just mimicks the updations. It doesn't delete or modify actually. We can just see the status codes to check whether insertions, updations, deletions occur.
URL: http://jsonplaceholder.typicode.com/posts/
GET:
GET is the simplest type of HTTP request method; the one that browsers use each time you click a link or type a URL into the address bar. It instructs the server to transmit the data identified by the URL to the client. Data should never be modified on the server side as a result of a GET request. In this sense, a GET request is read-only.
Checking with Fiddler or PostMan:
We can use Fiddler for checking the response. Open Fiddler and select the Compose tab.
Specify the verb and url as shown below and click Execute to check the response.
Verb: GET
url: http://jsonplaceholder.typicode.com/posts/
Response: You will get the response as:
"userId": 1, "id": 1, "title": "sunt aut...", "body": "quia et suscipit..."
In the “happy” (or non-error) path, GET returns a representation in XML or JSON and an HTTP response code of 200 (OK). In an error case, it most often returns a 404 (NOT FOUND) or 400 (BAD REQUEST).
2) POST:
The POST verb is mostly utilized to create new resources. In particular, it's used to create subordinate resources. That is, subordinate to some other (e.g. parent) resource.
On successful creation, return HTTP status 201, returning a Location header with a link to the newly-created resource with the 201 HTTP status.
Checking with Fiddler or PostMan:
We can use Fiddler for checking the response. Open Fiddler and select the Compose tab.
Specify the verb and url as shown below and click Execute to check the response.
Verb: POST
url: http://jsonplaceholder.typicode.com/posts/
Request Body:
data: {
title: 'foo',
body: 'bar',
userId: 1000,
Id : 1000
}
Response: You would receive the response code as 201.
If we want to check the inserted record with Id = 1000 change the verb to Get and use the same url and click Execute.
As said earlier, the above url only allows reads (GET), we cannot read the updated data in real.
3) PUT:
PUT is most-often utilized for update capabilities, PUT-ing to a known resource URI with the request body containing the newly-updated representation of the original resource.
Checking with Fiddler or PostMan:
We can use Fiddler for checking the response. Open Fiddler and select the Compose tab.
Specify the verb and url as shown below and click Execute to check the response.
Verb: PUT
url: http://jsonplaceholder.typicode.com/posts/1
Request Body:
data: {
title: 'foo',
body: 'bar',
userId: 1,
Id : 1
}
Response: On successful update it returns status 200 (or 204 if not returning any content in the body) from a PUT.
4) DELETE:
DELETE is pretty easy to understand. It is used to delete a resource identified by a URI.
On successful deletion, return HTTP status 200 (OK) along with a response body, perhaps the representation of the deleted item (often demands too much bandwidth), or a wrapped response (see Return Values below). Either that or return HTTP status 204 (NO CONTENT) with no response body. In other words, a 204 status with no body, or the JSEND-style response and HTTP status 200 are the recommended responses.
Checking with Fiddler or PostMan:
We can use Fiddler for checking the response. Open Fiddler and select the Compose tab.
Specify the verb and url as shown below and click Execute to check the response.
Verb: DELETE
url: http://jsonplaceholder.typicode.com/posts/1
Response: On successful deletion it returns HTTP status 200 (OK) along with a response body.
Example between PUT and PATCH
PUT
If I had to change my first name then send PUT request for Update:
{ "first": "Nazmul", "last": "hasan" }
So, here in order to update the first name we need to send all the parameters of the data again.
PATCH:
Patch request says that we would only send the data that we need to modify without modifying or effecting other parts of the data.
Ex: if we need to update only the first name, we pass only the first name.
Please refer the below links for more information:
https://jsonplaceholder.typicode.com/
https://github.com/typicode/jsonplaceholder#how-to
What is the main difference between PATCH and PUT request?
http://www.restapitutorial.com/lessons/httpmethods.html
The below definition is from the real world example.
Example Overview
For every client data, we are storing an identifier to find that client data and we will send back that identifier to the client for reference.
POST
If the client sends data without any identifier, then we will store the data and assign/generate a new identifier.
If the client again sends the same data without any identifier, then we will store the data and assign/generate a new identifier.
Note: Duplication is allowed here.
PUT
If the client sends data with an identifier, then we will check whether that identifier exists. If the identifier exists, we will update the resource with the data, else we will create a resource with the data and assign/generate a new identifier.
PATCH
If the client sends data with an identifier, then we will check whether that identifier exists. If the identifier exists, we will update the resource with the data, else we will throw an exception.
Note: On the PUT method, we are not throwing an exception if an identifier is not found. But in the PATCH method, we are throwing an exception if the identifier is not found.
Do let me know if you have any queries on the above.
Here is a simple description of all:
POST is always for creating a resource ( does not matter if it was duplicated )
PUT is for checking if resource exists then update, else create new resource
PATCH is always for updating a resource
PUT = replace the ENTIRE RESOURCE with the new representation provided
PATCH = replace parts of the source resource with the values provided AND|OR other parts of the resource are updated that you havent provided (timestamps) AND|OR updating the resource effects other resources (relationships)
https://laracasts.com/discuss/channels/general-discussion/whats-the-differences-between-put-and-patch?page=1
Simplest Explanation:
POST - Create NEW record
PUT - If the record exists, update else, create a new record
PATCH - update
GET - read
DELETE - delete
Think of it this way...
POST - create
PUT - replace
PATCH - update
GET - read
DELETE - delete
Request Types
create - POST
read - GET
create or update - PUT
delete - DELETE
update - PATCH
GET/PUT is idempotent
PATCH can be sometimes idempotent
What is idempotent -
It means if we fire the query multiple times it should not afftect the result of it.(same output.Suppose a cow is pregnant and if we breed it again then it cannot be pregnent multiple times)
get :-
simple get. Get the data from server and show it to user
{
id:1
name:parth
email:x#x.com
}
post :-
create new resource at Database. It means it adds new data. Its not idempotent.
put :-
Create new resource otherwise add to existing.
Idempotent because it will update the same resource everytime and output will be the same.
ex.
- initial data
{
id:1
name:parth
email:x#x.com
}
perform put-localhost/1
put email:ppp#ppp.com
{
id:1
email:ppp#ppp.com
}
patch
so now came patch request
PATCH can be sometimes idempotent
id:1
name:parth
email:x#x.com
}
patch name:w
{
id:1
name:w
email:x#x.com
}
HTTP Method
GET yes
POST no
PUT yes
PATCH no*
OPTIONS yes
HEAD yes
DELETE yes
Resources :
Idempotent -- What is Idempotency?
Main Difference Between PUT and PATCH Requests:
Suppose we have a resource that holds the first name and last name of a person.
If we want to change the first name then we send a put request for Update
{ "first": "Michael", "last": "Angelo" }
Here, although we are only changing the first name, with PUT request we have to send both parameters first and last.
In other words, it is mandatory to send all values again, the full payload.
When we send a PATCH request, however, we only send the data which we want to update. In other words, we only send the first name to update, no need to send the last name.
Quite logical the difference between PUT & PATCH w.r.t sending full & partial data for replacing/updating respectively. However, just couple of points as below
Sometimes POST is considered as for updates w.r.t PUT for create
Does HTTP mandates/checks for sending full vs partial data in PATCH? Otherwise, PATCH may be quite same as update as in PUT/POST
You may understand the restful HTTP methods as corresponding operations on the array in javascript (with index offset by 1).
See below examples:
Method
Url
Meaning
GET
/users
return users array
GET
/users/1
return users[1] object
POST
/users
users.push(body); return last id or index
PUT
/users
replace users array
PUT
/users/1
users[1] = body
PATCH
/users/1
users[1] = {...users[1], ...body }
DELETE
/users/1
delete users[1]
Reference to RFC: https://www.rfc-editor.org/rfc/rfc9110.html#name-method-definitions
POST - creates new object
PUT - updates old object or creates new one if not exist
PATCH - updates/modify old object. Primarly intended for for modificaiton.
There is few interpritation of RFC like mentioned before, but if you read carefully then you can notice that PUT and PATCH methods came after POST witch is common old fashion way to create native HTML Forms.
Therefore if you try to support all methods (like PATCH or DELETE), it can be suggested that the most approapreate way to use all methods is to stick to CRUD model:
Create - PUT
Read - GET
Update - PATCH
Delete - DELETE
Old HTML native way:
Read - GET
Create/Update/Delete - POST
Good Luck Coders! ;-)

Resources