I'm trying to use the airflow REST api (v1.10.2) but I'm having problems any time I try to query information about a specific dag.
Here's an example of the latest_runs endpoint working:
❯❯❯ curl -X GET http://192.168.99.100:30080/api/experimental/latest_runs
{
"items": [
{
"dag_id": "test_dag",
"dag_run_url": "/admin/airflow/graph?dag_id=test_dag&execution_date=2019-03-07+21%3A18%3A23.387031%2B00%3A00",
"execution_date": "2019-03-07T21:18:23.387031+00:00",
"start_date": "2019-03-07T21:18:23.683240+00:00"
}
]
}
However, when I try to query test_dag I get an error:
❯❯❯ curl -X GET "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
I've also tried looking in the source code and I found that there's a state param that can be used:
❯❯❯ curl -X GET "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs?state=success"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
but adding that in doesn't seem to help.
I find that triggering a DAG works:
❮❮❮ curl -X POST \
http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs \
-H 'Content-Type: application/json' \
-d '{}'
{
"message": "Created <DagRun test_dag # 2019-03-07 22:31:18+00:00: manual__2019-03-07T22:31:18+00:00, externally triggered: True>"
}
Anyone have an idea why I can't query the status of a DAG and get a 405?
EDIT:
here's some more investigation. I tried using all of the possible methods, none of which succeed:
4:21PM /Users/paymahn/solvvy/scheduler ✘ 130 train.models ✭ ✱ ◼
❯❯❯ curl -X GET "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
4:21PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X OPTIONS "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
4:21PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X TRACE "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
4:22PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X PATCH "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
4:22PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X DELETE "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
4:22PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X HEAD "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 178 bytes remaining to read
4:22PM /Users/paymahn/solvvy/scheduler ✘ 18 train.models ✭ ✱ ◼
❯❯❯ curl -X PUT "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
4:22PM /Users/paymahn/solvvy/scheduler train.models ✭ ✱ ◼
❯❯❯ curl -X POST "http://192.168.99.100:30080/api/experimental/dags/test_dag/dag_runs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
Turns out that this endpoint is only enabled in 1.10.2 in the RBAC UI mode: https://issues.apache.org/jira/browse/AIRFLOW-4043?focusedCommentId=16787946&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16787946.
Related
I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.
I am using the following function to get the signed url:
async getSignedUrl(file: File, expireAt: number): Promise<string> {
const [url] = await file
.getSignedUrl({
action: "read",
expires: expireAt
});
return url
}
which returns a signed URL in the following format:
https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).
My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.
What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?
If I understood correctly, the issue arises in two scenarios
When hitting your CF through curl with
curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')
According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:
curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'
I tried with the following with successfully result:
curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'
If I try with the content-type you shared with failed results.
2.
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
I tried a simple GET in postman using a Signed URL and it worked just fine
Command used in gsutil to get the signed URL:
gsutil signurl -d 10m key.json gs://BUCKET/aa.png
Then I tried a GET on postman and worked just fine.
I also tried with a Signed URL to upload a File in Postman and worked just fine.
My thoughts are that, according to Common MIME types
application/octet-stream is the default value for all other cases (not textual files).
When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream
curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.
Getting 302 for the JFrog REST api for listing docker tags.
Documentation :-
https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-ListDockerTags
Usage:
GET /api/docker/{repo-key}/v2/{image name}/tags/list?n=<n from the request>&last=<last tag value from previous response>
My query:-
repo-key - docker-local
My image name is like -> /eric/com.jfrog/test-app
So the query is,
https://jfrog.test.com/api/docker/docker-local/v2/eric/com.jfrog/test-app/tags/list
Response :-
<html>
<head><title>302 Found</title></head>
<body>
<center><h1>302 Found</h1></center>
<hr><center>nginx/1.17.5</center>
</body>
</html>
Artifactory API is exposed under /artifactory/api... path. At least for my Pro and JCR versions configured with nginx subdomains.
Try following paths:
Using general artifactory url:
curl -u user:pass https://jfrog.test.com/artifactory/api/docker/docker-local/v2/my-docker-image/tags/list?
If you're using subdomain and reverse proxies e.g. image is available at docker-local.test.com/my-docker-image:latest than following path should be correct as well:
curl -u user:pass https://docker-local.test.com/artifactory/api/docker/docker-local/v2/my-docker-image/tags/list?
For both cases /artifactory/api/docker are always. docker-local is the name of the repo (local or virtual) and my-docker-image is the name of the image. Probably for your path you should replace my-docker-image with eric/com.jfrog/test-app.
Instead using -u user:pass how to use the token or key to mask the password.
I would like to access HDFS files using webhdfs. Curl gives me an option of using --negotiate -u: user option to use existing kerberos token. How do we pass the negotiate option using HTTP request headers. I know that we can use "Authorization: Negotiate" option. However, I get the following error.
GSSException: Defective token detected
you can do like this:
kinit -kt ${your_keytab_file_full_path} ${your_principal}
curl --negotiate -u : -o ${your_keytab_file_full_path} ${URL}
after running command :-
curl -XPUT -H 'content-type: application/httpheader' 'http://localhost:8098/search/extract' --data-binary "#/home/cblap15/Desktop/google_packet.bin"
i am getting error as:-
<html><head><title>500 Internal Server Error</title></head><body><h1>Internal Server Error</h1>The server encountered an error while processing this request:<br><pre>{error,
{error,undef,
[{yz_httpheader_extractor,extract,
[<<"\nGET http://www.google.com HTTP/1.1\n\n">>],
[]},
{yz_wm_extract,extract,2,[{file,"src/yz_wm_extract.erl"},{line,91}]},
{webmachine_resource,resource_call,3,
[{file,"src/webmachine_resource.erl"},{line,186}]},
{webmachine_resource,do,3,
[{file,"src/webmachine_resource.erl"},{line,142}]},
{webmachine_decision_core,resource_call,1,
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
{webmachine_decision_core,accept_helper,1,
[{file,"src/webmachine_decision_core.erl"},{line,616}]},
{webmachine_decision_core,decision,1,
[{file,"src/webmachine_decision_core.erl"},{line,521}]},
{webmachine_decision_core,handle_request,2,
[{file,"src/webmachine_decision_core.erl"},{line,33}]}]}}</pre><P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></body></html>
How we resolve this??? while use of extractor in riak i am getting this.
I want to retrieve data from a classifier API. The documentation showed me:
curl -H "Authentication: Basic <my_secret>" "http://www.superapi.io/classifier/test
model_id=123&value=your+text"
How do I implement this in R? I tried with package RCurl:
getURL("http://www.superapi.io/classifier/test",
userpwd ="secret",
httpheader=list(model_id="123",value="your+text"))
but this was didnt work:
[1] "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>405 Method Not
Allowed</title>\n<h1>Method Not Allowed</h1>\n<p>The method is not allowed for the requested
URL </p>\n"
answer:
getURL("http://wwww.superapi.io/classifier/test?_model_id=123&value=TEST", userpwd="<secret>",
httpauth = 1L)