I need to fetch data out of a cloud platform. The process to export data is 2 step:
First make a post call with the username/password details. This will return xml output with a jobid in the response.
Fetch the jobid from the first response and use this jobid, concatenate it to get a new url and then make a get call (execute curl again) using this new url, I will then get data in a json response.
What I did:
I am able to make the first API call and get the jobID. Next,I concatenated this jobId to get new url and saved complete curl statement in a variable (lets call the variable cmd_second_api_call). This variable 'cmd_second_api_call' contains the complete curl statement that I need to execute.
So I did a out=$($cmd_second_api_call), as I want to execute the second curl statement and store the output in a variable.
Problem:
When I execute out=$($cmd_second_api_call), I see out variable as empty. I verified $cmd_second_api_call actually outputs the curl command perfectly. If I execute the output of $cmd_second_api_call on my command prompt I see the output. What am I missing here? How to get the curl output in a variable?
Thanks!
r=$(curl -k -u user:password static_url -d <data I need to pass>)
jobid=$(echo $r | sed -n 's:.*<jobid>\(.*\)<\/jobid>.*:\1:p')
second_url="abc.com/${jobid}/result/ --get -d output=json"
cmd_second_api_call="curl -u user:password -k ${second_url}"
out=$($cmd_second_api_call)
echo $out
Putting a command in a variable or using variables without quotes can be dangerous.
I suggested
out=$(curl -u user:password -k ${second_url})
# or
out=$(curl -u user:password -k abc.com/${jobid}/result/ --get -d output=json)
# and
echo "$out"
Somehow this helped, together with a sleep 5 between both curl calls. You wouldn't expect a lag at the remote server between returning a valid jobid and enabling the interface for that jobid. Perhaps some kind of "defence" against unauthorized calls with random jobid's.
Related
I am using a script that gives me some data in json format, I want to send this data to splunk.
I can store the output of the script in a file but how can I send it to HTTP Event Collector?
Couple of things I tried but did not work:
FILE="output.json"
file1="cat answer.txt"
curl -k "https://prd-pxxx.splunkcloud.com:8088/services/collector" -H "Authorization: Splunk XXXXX" -d '{"event": "$file1", "sourcetype": "manual"}'
-----------------------------------------------------------
curl -k "https://prd-pxxx.splunkcloud.com:8088/services/collector" -H "Authorization: Splunk XXXXX" -d '{"event": "#output.json", "sourcetype": "manual"}'
curl -k "https://prd-p-w0gjo.splunkcloud.com:8088/services/collector" -H "Authorization: Splunk d70b305e-01ef-490d-a6d8-b875d98e689b" -d '{"sourcetype":"_json", "event": "#output.json", "source": "output.json}
-----------------------------------------------------------------
After trying this I understand that it literally sends everything specified in the event section. Is there a way I can send the content of the file or use a variable?
Thanks in advance!
(Note - I haven't tried this specifically, but it should get you close)
According to Docs.Splunk on HTTP Event Collector Examples #3, it would seem you can do something very similar to this:
curl -k "https://mysplunkserver.example.com:8088/services/collector/raw?channel=00872DC6-AC83-4EDE-8AFE-8413C3825C4C&sourcetype=splunkd_access&index=main" \
-H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \
-d < $FILE
Presuming the content of the file is formatted correctly, it should go straight in.
How is the file being created? Is it in a Deployment App on a managed endpoint? If so, it will likely be simpler to setup a scripted input for the UF to run on whatever schedule you choose.
I am working on providing support for sending http post requests from one of our tools. The tool basically executes a job via http requests.
The way this is achieved is the tool makes a call to RunScript.bat with a number of arguments. The script parses these args and makes a curl post request after validation. The Post request is handled by our internal Java Service.
What I want is my batch file should be able to fetch the Response Code (200/409 etc.) of the post request and based on this, the script should return 1 or 0 as exit value back to the tool.
The tool then marks the job pass/fail based on the return value.
Tool: Parses the returned value
RunScript.bat arg1 arg2 arg3
RunScript.bat
#echo off
SETLOCAL
ECHO %DATE% %TIME% Calling %0 with %*
SET "SCRIPT_DIR=%~dp0"
ECHO Script Dir %SCRIPT_DIR%
SET cmd=%1
SET val2=%2
SET val3=%3
%SCRIPT_DIR%\curl -v -X POST http://localhost:9500/%cmd%/%val2%/%val3%
Is it possible to fetch the Response Code of the curl request in the batch script and then return 1/0 based on the Response Code ?
The first thing we need is to get curl to output the HTTP status. For this I used pvandenberk's answer to Getting curl to output HTTP status code?. Next, we have to get that output into an environment variable so we can test it. For this we use the for /f %%a in ( 'command-to-execute' ) do ... form of the for command (see for /? for more details).
Incorporating these (and some other minor tweaks), something like the following should help:
#echo off
SETLOCAL
ECHO %DATE% %TIME% Calling %0 with %*
SET "SCRIPT_DIR=%~dp0"
ECHO Script Dir %SCRIPT_DIR%
SET "cmd=%1"
SET "val2=%2"
SET "val3=%3"
SET "URL=http://localhost:9500/%cmd%/%val2%/%val3%"
SET HTTP=
for /f %%a in ( '"%SCRIPT_DIR%\curl" -s -o nul -v -X POST -w "%%{http_code}" "%URL%"' ) do set HTTP=%%a
if "%HTTP%" == "200" (
exit /b 0
) else (
exit /b 1
)
Notes:
I've assumed cmd, val2 and val3 in your original URL were meant to be replaced with the contents of the earlier environment variables, so have changed them to %cmd% etc. when building URL.
In the for statement, the whole of the command to be executed is enclosed in single-quotes ('.....').
I've wrapped the invocation of curl in double-quotes ("%SCRIPT_DIR%\curl"): this should allow it to work even if the script-directory were to contain spaces.
I've added -s (silent) and -o nul (where to send the full response) to curl's command-line. This will help to ensure any output returned from the POST request does not interfere with the HTTP status code. If you need to see this output, use -o filename and TYPE the file before making the test on HTTP.
The percent-sign in -w "%%{http_code}" needs to be doubled because it has special meaning to the shell.
The output of -v (verbose) is sent to stderr so shouldn't interfere with capturing the HTTP response code.
Update
To get both the HTTP response code, and still see the output from the curl command itself you could try replacing the for /f ... line above with the one below:
for /f "delims=" %%a in ( '"%SCRIPT_DIR%\curl" -v -X POST -w "%%{http_code}" "%URL%"' ) do set "HTTP=%%a" && echo %%a
The "delims=" means that the lines of output from curl will not be split into separate tokens so %%a will hold the whole of each line.
I've removed the -s and -o nul options, so the original output from curl will still be generated.
The set "HTTP=%%a" command has had double-quotes added to prevent trailing spaces being added to the environment variable. Because the response code (e.g. 404) is the last line emitted by curl, the final value of HTTP will still be what we want.
The echo %%a will replay all lines generated by curl.
I want to know if I can have conflict in this scenario :
#!/usr/bin/env bash
curl -XPUT -d '{"bar":"baz"}' \
-H "Content-Type: application/json" \
http://127.0.0.1:8098/riak/obj/1
response=$(curl -I http://127.0.0.1:8098/riak/obj/1 | grep 'X-Riak-Vclock:' | egrep -o ' .*$')
curl -v -XPUT -d '{"bar":"foo"}' \
-H "Content-Type: application/json" \
-H "X-Riak-Vclock: $response" \
http://127.0.0.1:8098/riak/obj/1
In some words :
First I have no object for the key 1 I put the {"bar":"baz"} value with the PUT of the http api.
Then, I read the value with a get. And I store the vclock in variable.
And finaly I put a new value {"bar":"foo"} for the key 1
Is there a case where I can have {"bar":"baz"} for the key 1 ? If Riak has a conflict, it will be resolve with vclock ?
Thanks !
It depends how your Riak database is configured, either globally or if you changed the default configuration of the bucket you're using. If you keep the default config, your second PUT (with the vclock) might:
- fail, if someone updated the key behind your back (rare), and the vclock data you have is already obsolete. You need to re-read the value and update it. Best is to have a retry mechanism.
- fail, if the write consistency constrains you have is too strict, and too many nodes are down (rare). Usually the default read and write config are sane.
- succeed, if the vclock data is still valid for this key (most of the time)
In case it succeeds, it might be that the network topology was in a split-brain situation. In this case, Riak will solve the issue itself using v-clock data.
I need to write all the curl arguments in a separate file(myconfig.txt) to be executed and output for each different lines of command in output.txt, output1.txt, output2.txt...so on.
The webservices should be requested with post (multipart/form-data) method.
I am trying this command curl -K myconfig.txt -o output.txt
contents of
myconfig.txt include URL="http://1x2.2x1.x6.1x2:3000/latto/get_notifications"<option="to";mobile="+91999xx6xx3x"> into the output.txt - 'Cannot get /latto/get_notifications'. It seems like web-services are being requested as get method. Please anyone tell me the syntax to write in myconfig file. So that I can get the right output in file.
However, When I run this command sudo curl --form option="to" --form mobile="+9199999yyyxx" 1x2.2x1.x6.1x2:3000/latto/get_notifications, output is successfully printed in terminal.
When I do a curl -k config.txt with config.txt containing this:
URL="http://my.test.domain/index.php"
-d option="to"
-d mobile="+91999xx6xx3x"
I get the result:
POST
array(2) {
["option"]=>
string(4) ""to""
["mobile"]=>
string(15) "" 91999xx6xx3x""
}
So I don't known what you mean with <option ...> but when you put it into your config file as stated above, it should work.
Btw. here is the php script that answered:
<?php
echo $_SERVER['REQUEST_METHOD']."\n";
var_dump($_REQUEST);
?>
This webservice accepts request in POST method and enctype multipart/form-data
URL="http://my.test.domain/get_notifications"
-F option=to
-F mobile=+91999xx6xx3x
After a lot of hit and trial, I got this working: curl -k config.txt with config.txt containing the above.
Can anyone help me to find out a unix command that is used to upload/download a file on/from an URL?
Particular URL in which i'm trying to upload/download is protected with an user id and password.
I guess curl serves this purpose but not aware of how to use it? Could you please give me sugegstions on this?
curl has a command line argument named -d (for data) and you can use it like this to send a file(you need to add a # before a file-name to have curl treat it as a file and not a value:
curl -X POST -d #myfilename http://example.com/upload
You can add multiple -d arguments if you need to send a FORM value along with your file. Like so:
curl -X POST -d #myfilename -d name=MyFile http://example.com/upload