Get status job in cdsw - cloudera

I have some R and python scripts in CDSW "Cloudera-Data-Science-Workbench". I create a shell script to run this with curl -v -XPOST.
How to get the status of a job from the API CDSW?

Hi it's been a while since this question was posted but hopefully the answer can still be useful to someone :)
After you run:
curl -v -XPOST http://cdsw.example.com/api/v1/projects/<$USERNAME>/<$PROJECT_NAME>/jobs/<$JOB_ID>/start --user "API_KEY:" --header "Content-type: application/json"
You should be able to see in the output a URL that looks like this:
http://cdsw.example.com/api/v1/projects/<$USERNAME>/<$PROJECT_NAME>/dashboards/<$ID>
So then you can use it to retrieve the job status for example with piping the status using jq (or without it so you can also see the status in the output as well as other stuff returned):
curl -v http://cdsw.example.com/api/v1/projects/<$USERNAME>/<$PROJECT_NAME>/dashboards/<$ID> --user "API_KEY:" | jq '.status'

Related

How to get data with curl

I have a page which i need to submit a number and after that i get data from that number
page is like:
http://page/
and after submit form request URL changes to:
http://page/services/getData
but method is something like:
http://page/services/getData?method=search&1234
can someone helped me to pull data showed in http://page/services/getData?method=search&1234 with curl?
I've already tried:
curl -v -H "Content-Type: application/json" -X POST \ -d http://page/services/getData?method=search&1234
curl -o http://page/services/getData?method=search&1234
curl --data http://page/services/getData?method=search&1234
I was looking for hours but i just posted the question and i get what i wanted:
curl --request GET --url 'http://page/services/getData?method=search&1234'

CURL chunked POST from infinite pipe?

I want to continuously post an infinite binary data stream to a webserver.
So I'm using the following command:
curl -X POST -H "Transfer-Encoding: chunked" -d 'hello' http://127.0.0.1:9000
As a test, I'm piping the output of the 'yes' command:
$ yes | curl -X POST -H "Transfer-Encoding: chunked" -d '#-' http://127.0.0.1:9000
But it doesn't even connect to the webserver, and aborts with an out-of-memory error. It seems curl is trying to read the whole file into memory before starting the transfer.
Does curl supports continuously HTTP posting data coming from a pipe? (I'm using curl version 7.61.0).
Thanks!
Try with
yes | curl -T. http://127.0.0.1:9000
But you will probably get this known bug :
https://github.com/curl/curl/issues/2051
https://github.com/curl/curl/issues/932
Or
yes | curl -T- http://127.0.0.1:9000

how to post a json file with documents to couchdb

I am trying to post some documents to couchdb by curl and I have succeeded by choosing local file but not http-url... I have trying something like this:
curl -d #http://111.111.11.1/json/myjsonfile -X POST http://127.0.0.1:5984/MyTestDb/_bulk_docs -H "Content-Type: application/json"
I have been trying with many flags and tried many ways but I thing I am missing something. Is there anyone who can help?
The -d option for curl expects a local file only. You'll have to download it first. You could try piping the output of a curl download to a PUT to your CouchDB:
curl http://111.111.11.1/json/myjsonfile | curl -d #- -X PUT http://localhost:5984/MyTestDb....

is it possible to send curl another url as a param?

I know I can do this:
curl 'http://localhost:8080/myhandler' -F "myfile=#localfile"
so is it somehow possible to send a file that is on the web? Something like
curl 'http://localhost:8080/myhandler' -F "myfile=#SOMEURL"
I cannot find the right way to apply that (I am guessing it is possible?)
As per the curl documentation -F can read from stdin using -. So if you pipe output from the url as follows, I think it should work:
curl -vvv <url1> | curl -vvv 'http://localhost:8080/myhandler' -F "myfile=#-"

How to use pastebin from shell script?

Is it possible to use pastebin (may be via their "API" functionality) inside bash shell scripts? How do I send http-post? How do I get back the URL?
As pastebin.com closed their public api, I was looking for alternatives.
Sprunge is great. Usage:
<command> | curl -F 'sprunge=<-' http://sprunge.us
or, as I use it:
alias paste="curl -F 'sprunge=<-' http://sprunge.us"
<command> | paste
The documentation says that you need to submit a POST request to
http://pastebin.com/api_public.php
and the only mandatory parameter is paste_code, of type string is the paste that you want to make.
On success a new pastebin URL will be returned.
You can easily do this from your bash shell using the command curl.
curl uses the -d option to send the POST data to the specified URL.
Demo:
This demo will create a new paste with the code:
printf("Hello..I am Codaddict");
From your shell:
$ curl -d 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
http://pastebin.com/598VLDZp
$
Now if you see the URL http://pastebin.com/598VLDZp, you'll see my paste :)
Alternatively you can do it using the wget command which uses the option --post-data to sent POST values.
I've tried this command it works fine:
wget --post-data 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
Put the following in your .bashrc:
sprunge() {
if [[ $1 ]]; then
curl -F 'sprunge=<-' "http://sprunge.us" <"$1"
else
curl -F 'sprunge=<-' "http://sprunge.us"
fi
}
...and then you can run:
sprunge filename # post file to sprunge
...or...
some_command | sprunge # pipe output to sprunge
The API for posting to pastebin has changed, since posted by codaddict.
Details can be found at this link: https://pastebin.com/api
Example:
curl -d 'api_paste_code=printf("Hello..\n I am Codaddict");' \
-d 'api_dev_key=<get_your_own>' \
-d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
There are three essential fields as of now:
api_dev_key -> You need to create a login on pastebin.com in order to get that
api_option -> Format in which to post
api_paste_code -> Text you want to post
Two other answers (from circa 2014) point to http://sprunge.us, which is designed to be used like this...
curl --form 'sprunge=#yourfile.txt' sprunge.us
However, as of 2018, sprunge.us has a tendency to be overloaded and return 500 Internal Server Error to every request. For files up to at least 300 KB but not as high as 2.8 MB, I have had good luck with the very similar service at http://ix.io:
curl --form 'f:1=#yourfile.txt' ix.io
For files up to at least 2.8 MB (and maybe higher, I don't know), I've found the more highly polished https://transfer.sh. It recommends a slightly different and simpler command line, and requires https (it won't work without it):
curl --upload-file yourfile.txt https://transfer.sh
I have found that Sprunge is currently down, but dpaste.com has a simple API.
To post from STDIN
curl -s -F "content=<-" http://dpaste.com/api/v2/
from a file foo.txt
cat foo.txt | curl -s -F "content=<-" http://dpaste.com/api/v2/
to post a string
curl -s -F "content=string" http://dpaste.com/api/v2/
The response will be a plain text URL to the paste.
Nb: the trailing / in the URL http://dpaste.com/api/v2/ seems necessary
https://paste.c-net.org/ has a simpler API than all of them. Simply "POST" to it.
From the website:
Upload text using curl:
$ curl -s --data 'Hello World!' 'https://paste.c-net.org/'
Upload text using wget:
$ wget --quiet -O- --post-data='Hello World!' 'https://paste.c-net.org/'
Upload a file using curl:
$ curl --upload-file #'/tmp/file' 'https://paste.c-net.org/'
Upload a file using wget:
$ wget --quiet -O- --post-file='/tmp/file' 'https://paste.c-net.org/'
Upload the output of a command or script using curl:
$ ls / | curl --upload-file - 'https://paste.c-net.org/'
$ ./bin/hello_world | curl -s --data-binary #- 'https://paste.c-net.org/'
You can also simply use netcat. Unlike termbin, paste.c-net.org won't time out if your script takes more than 5 seconds to produce its output.
$ { sleep 10; ls /; } | nc termbin.com 9999
$ { sleep 10; ls /; } | nc paste.c-net.org 9999
https://paste.c-net.org/ExampleOne
Easiest way to post to pastebin
echo 'your message' | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key=<your_api_key>' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
Just change the <your_api_key> part and pipe whatever you want into it.
The sed invocations add the api_paste_code parameter to beginning of the message and add a newline at the end of each line so it can handle multiline input. The #- tells curl to read from stdin.
A Bash Function You Can Paste
For easy reuse, make it a bash function (copy and paste this into your terminal and set the API_KEY field appropriately:
pastebin () {
API_KEY='<your_api_key>'
if [ -z $1 ]
then
cat - | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
else
echo "$1" | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
fi
printf '\n'
}
You can run it with either:
pastebin 'your message'
or if you need to pipe a file into it:
cat your_file.txt | pastebin
To built upon Vishal's answer, pastebin has upgraded to only use HTTPS now:
curl -d 'api_paste_code=printf("Hello World");' \
-d 'api_dev_key=<your_key>' \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
You don't have to specify the -X POST parameter
Additional details can be found here:
https://pastebin.com/doc_api#1
Based on another answer on this page, I wrote the following script which reads from STDIN (or assumes output it piped into it).
This version allows for arbitrary data which is URI escaped (by jq).
#!/bin/bash
api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
curl -d "api_paste_code=$(jq -sRr #uri)" \
-d "api_dev_key=$api_key" \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
echo # By default, there's no newline
I am a bit late to this post, but I created a little tool to help with this.
https://pasteshell.com/
Feel free to check it out and let me know what you think.
Thanks,

Resources