I need to perform a get request from nifi to couchbase. The curl command is:
curl http://HOST:PORT/query/service -d "statement=select item.Date from bucket unnest bucket as item" -u USER:PASSWORD
I tried using InvokeHttp and ExecuteStreamCommand but it keeps returning errors(status code 400). The full error message is:
{ "requestID": "bff62c0b-36fd-401d-bca0-0959e0944323", "errors":
[{"code":1050,"msg":"No statement or prepared value"}], "status":
"fatal", "metrics": {"elapsedTime": "113.31µs","executionTime":
"74.321µs","resultCount": 0,"resultSize": 0,"errorCount": 1
It's important to say that I prefer that the http request will be triggered by an incoming flowfile. I tried using the processors in various of ways but non of them worked.
When I run the command from the nifi server it works fine.
Thanks for the help
the -d parameter of the curl utility forces HTTP POST command
and application/x-www-form-urlencoded mime-type.
so, in the nifi InvokeHTTP select the following parameters
HTTP Method = POST
Remote URL = <your url here>
Basic Authentication Username = <username>
Basic Authentication Password = <password>
Content-Type = application/x-www-form-urlencoded
and the body of the flow file should be
statement=select item.Date from bucket unnest bucket as item
I don't know nifi, but based on the error message, the "statement=" part of the request isn't being included, or you are not sending the request as a POST command.
Related
I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.
I am using the following function to get the signed url:
async getSignedUrl(file: File, expireAt: number): Promise<string> {
const [url] = await file
.getSignedUrl({
action: "read",
expires: expireAt
});
return url
}
which returns a signed URL in the following format:
https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).
My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.
What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?
If I understood correctly, the issue arises in two scenarios
When hitting your CF through curl with
curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')
According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:
curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'
I tried with the following with successfully result:
curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'
If I try with the content-type you shared with failed results.
2.
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
I tried a simple GET in postman using a Signed URL and it worked just fine
Command used in gsutil to get the signed URL:
gsutil signurl -d 10m key.json gs://BUCKET/aa.png
Then I tried a GET on postman and worked just fine.
I also tried with a Signed URL to upload a File in Postman and worked just fine.
My thoughts are that, according to Common MIME types
application/octet-stream is the default value for all other cases (not textual files).
When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream
curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.
I want to make a request to an API which expects a JSON to be sent in the Header field. I'm unable to do it in Python Request library.
I'm able to do it in cURL.
cURL code:
curl -v -X POST https://content.dropboxapi.com/2/files/download
--header "Authorization: Bearer abcdefgh12343567"
--header "Dropbox-API-Arg: {\"path\": \"/folder/file.mp4\"}" -o file.mp4
Python code:
import requests
import simplejson
r = requests.post(
'https://content.dropboxapi.com/2/files/download',
headers={
'Authorization':'Bearer abcdefgh12343567',
'Dropbox-API-arg': simplejson.dumps({'path': '/folder/file.mp4'})
})
Here the Header contains a JSON string.
I'm trying to use Dropbox's files/download API documented here.
Even though the request is sent, the JSON value seems to be wrong.
The above code seems to work correctly now. I'm not sure about what happened - I was receiving HTTP 409 error earlier.
I was doubtful about how Python Requests Library would handle JSON string inside the header. So, I created an API to see how the server would read the headers made by both cURL and Python Requests. Both headers are identical.
I would like to access HDFS files using webhdfs. Curl gives me an option of using --negotiate -u: user option to use existing kerberos token. How do we pass the negotiate option using HTTP request headers. I know that we can use "Authorization: Negotiate" option. However, I get the following error.
GSSException: Defective token detected
you can do like this:
kinit -kt ${your_keytab_file_full_path} ${your_principal}
curl --negotiate -u : -o ${your_keytab_file_full_path} ${URL}
Artifactory OSS
5.4.6 rev 50406900
Trying to get access token to work.
I created token...
e.g. curl -uadmin:adminpw -X POST "myserver:8081/artifactory/api/security/token" -d "username=moehoward"
Returned msg looks like success...
{
"scope" : "member-of-groups:readers api:*",
"access_token" : <very-long-string>
"expires_in" : 3600,
"token_type" : "Bearer"
}
I can see it in the gui (admin -> Security -> Access Tokens) with "Subject" = to the user ("moehoward" in the example above) and with a "Token ID" that's a lot shorter, something like...
f2eb693a-d4ff-4618-ba52-764dc975c497
To test, I tried to ping using example in the docs...
curl -umoehoward:<very-long-string> myserver:8081/artifactory/api/system/ping
Fails with a 401 (bad credentials).
I replace the token with the "token id" I see in the gui, same result.
I replace again with the hardcoded pw of the "moehoward" user and that works (responds with "OK").
I tried the "-H"Authentication: Bearer " approach using the long string and that worked. So I guess the very long string is the token and not the "Token ID" in the gui.
Q: Any idea why this works for Bearer" and not the user by name ?
So you are right that this is supposed to work for both standard authentication and the Authentication HTTP header.
I did the test on a server with the same version Artifactory OSS 5.4.6 and the following works fine here
Inject the proper variables
export SERVER=server-artifactory
export APIKEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Create and use an access token for user moehoward
curl -u "admin:$APIKEY" -X POST "http://$SERVER/artifactory/api/security/token" -d "username=moehoward" -d "scope=member-of-groups:readers" > token.log
export TOKEN=`cat token.log | jq -r '.access_token'`
curl -u "moehoward:$TOKEN" "http://$SERVER/artifactory/api/system/ping"
curl -H "Authorization: Bearer $TOKEN" "http://$SERVER/artifactory/api/system/ping"
I get "OK" from the last two commands. Can you run exactly these commands and report back?
I have personally experienced the same problem (Bearer header working, standard user credentials not working) with an old version of curl. The obvious workaround is to use Bearer, the more complex workaround is to upgrade curl (or use another tool).
What is the version of curl you use? Any curl from 2015 or more recent should work fine.
I am learning OpenStack now and I want to list all accessible volumes on my OpenStack controller by HTTP get request with x_auth_token:
http://{OpenStack controller IP}:8776/v3/{project_id}/volumes
(Reference URL: https://developer.openstack.org/api-ref/block-storage/v3/index.html?expanded=#list-accessible-volumes)
But, the request body always as below:
{
"badRequest":{
"message": "Malformed request url",
"code": 400
}
}
I have try many times but they all didn't work.
So, I want to know, what the request url should be here?
Thanks in advance.....
One of the possible reasons is that your project_id is incorrect or doesn't match your credential, take a look at this link. In order to solve this problem you can:
1. Try openstack command with debug option.
openstack --debug volume list
#or
cinder --debug list
you could get the request and response details:
REQ: curl -g -i -X GET http://ip:port/v3/{project_id}/volumes/detail -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {token_value}"
Make sure you have input every required headers(context-type/x-auth-token/...).
Try to ask for help in irc channel #openstack-cinder