I read https://superuser.com/questions/272265/getting-curl-to-output-http-status-code . It mentioned that
curl -i
will print the HTTP response code. Is it possible to have curl print just the HTTP response code? Is there a generic way to get the HTTP status code for any type of request like GET/POST/etc?
I am using curl 7.54.0 on Mac OS High Sierra.
Thanks for reading.
This worked for me:
$ curl -s -w "%{http_code}\n" http://google.com/ -o /dev/null
curl -s -I http://example.org | grep HTTP/ | awk {'print $2'}
output: 200
Another solution:
curl -sI http://example.org | head -n 1 | cut -d ' ' -f 2
in this way you are:
getting the first HTTP response line (head -n 1), which must contain the response HTTP version, the response code and the response message (in this order), each one separated by a whitespace (as defined in the HTTP standard);
getting the 2° field of this line (cut -d ' ' -f 2), which is the status code
Related
I need to send a fax where the source file is coming from an HTTP URL. I have configured hylaFax. When trying a local file, it works fine. But with a URL it gives an error.
The command I am using is something like this:
sendfax -v -h faxhost -f kaur#xyz.com -D -d 1234567890 \
'http://kaur.dev.xyz.com:7771/app-name/proxy?bName=Test&oName=1.txt'
The error:
Error : 'Can not open file'
The file is downloading when connecting through browser.
sendfax will process stdin so you can pipe documents in:
wget -O - 'http://kaur.dev.xyz.com:7771/app-name/proxy?bName=Test&oName=1.txt' | sendfax -v -h faxhost -f kaur#xyz.com -D -d 1234567890
or
curl 'http://kaur.dev.xyz.com:7771/app-name/proxy?bName=Test&oName=1.txt' | sendfax -v -h faxhost -f kaur#xyz.com -D -d 1234567890
When downloading a file using curl, how would I follow a link location and use that for the output filename (without knowing the remote filename in advance)?
For example, if one clicks on the link below, you would download a filenamed "pythoncomplete.vim." However using curl's -O and -L options, the filename is simply the original remote-name, a clumsy "download_script.php?src_id=10872."
curl -O -L http://www.vim.org/scripts/download_script.php?src_id=10872
In order to download the file with the correct filename you would have to know the name of the file in advance:
curl -o pythoncomplete.vim -L http://www.vim.org/scripts/download_script.php?src_id=10872
It would be excellent if you could download the file without knowing the name in advance, and if not, is there another way to quickly pull down a redirected file via command line?
The remote side sends the filename using the Content-Disposition header.
curl 7.21.2 or newer does this automatically if you specify --remote-header-name / -J.
curl -O -J -L $url
The expanded version of the arguments would be:
curl --remote-name --remote-header-name --location $url
If you have a recent version of curl (7.21.2 or later), see #jmanning2k's answer.
I you have an older version of curl (like 7.19.7 which came with Snow Leopard), do two requests: a HEAD to get the file name from response header, then a GET:
url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename=$(curl -sI $url | grep -o -E 'filename=.*$' | sed -e 's/filename=//')
curl -o $filename -L $url
If you can use wget instead of curl:
wget --content-disposition $url
I wanted to comment to jmanning2k's answer but as a new user I can't, so I tried to edit his post which is allowed but the edit was rejected saying it was supposed to be a comment. sigh
Anyway, see this as a comment to his answer thanks.
This seems to only work if the header looks like filename=pythoncomplete.vim as in the example, but some sites send a header that looks like filename*=UTF-8' 'filename.zip' that one isn't recognized by curl 7.28.0
I wanted a solution that worked on both older and newer Macs, and the legacy code David provided for Snow Leopard did not behave well under Mavericks. Here's a function I created based on David's code:
function getUriFilename() {
header="$(curl -sI "$1" | tr -d '\r')"
filename="$(echo "$header" | grep -o -E 'filename=.*$')"
if [[ -n "$filename" ]]; then
echo "${filename#filename=}"
return
fi
filename="$(echo "$header" | grep -o -E 'Location:.*$')"
if [[ -n "$filename" ]]; then
basename "${filename#Location\:}"
return
fi
return 1
}
With this defined, you can run:
url="http://www.vim.org/scripts/download_script.php?src_id=10872"
filename="$(getUriFilename $url)"
curl -L $url -o "$filename"
Please note that certain malconfigured webservers will serve the name using "Filename" as key, where RFC2183 specifies it should be "filename". curl only handles the latter case.
I had the same Problem like John Cooper. I got no filename but a Location File name back. His answer also worked but are 2 commands.
This oneliner worked for me....
url="https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64&lang=de";url=$(curl -L --head -w '%{url_effective}' $url 2>/dev/null | tail -n1) ; curl -O $url
Stolen and added some stuff from
https://unix.stackexchange.com/questions/126252/resolve-filename-from-a-remote-url-without-downloading-a-file
An example using the answer above for Apache Archiva artifact repository to pull latest version. The curl returns the Location line and the filename is at the end of the line. Need to remove the CR at end of file name.
url="http://archiva:8080/restServices/archivaServices/searchService/artifact?g=com.imgur.backup&a=snapshot-s3-util&v=LATEST"
filename=$(curl --silent -sI -u user:password $url | grep Location | awk -F\/ '{print $NF}' | sed 's/\r$//')
curl --silent -o $filename -L -u user:password $url
instead of applying grep and other Unix-Fu operations, curl ships with a builtin "Write Out" option variable[1] specifically for such a case, e.g.
$ curl -OJsL "http://www.vim.org/scripts/download_script.php?src_id=10872" -w "%{filename_effective}"
pythoncomplete.vim
[1] https://everything.curl.dev/usingcurl/verbose/writeout#available-write-out-variables
Using the solution proposed above, I wrote this helper function curl2file.
[UPDATED]
function curl2file() {
url=$1
url=$(curl -o /dev/null -L --head -w '%{url_effective}' $url 2>/dev/null | tail -n1) ; curl -O $url
}
Usage:
curl2file https://cloud.tsinghua.edu.cn/f/4666d28af98a4e63afb5/?dl=1
Is it possible to use pastebin (may be via their "API" functionality) inside bash shell scripts? How do I send http-post? How do I get back the URL?
As pastebin.com closed their public api, I was looking for alternatives.
Sprunge is great. Usage:
<command> | curl -F 'sprunge=<-' http://sprunge.us
or, as I use it:
alias paste="curl -F 'sprunge=<-' http://sprunge.us"
<command> | paste
The documentation says that you need to submit a POST request to
http://pastebin.com/api_public.php
and the only mandatory parameter is paste_code, of type string is the paste that you want to make.
On success a new pastebin URL will be returned.
You can easily do this from your bash shell using the command curl.
curl uses the -d option to send the POST data to the specified URL.
Demo:
This demo will create a new paste with the code:
printf("Hello..I am Codaddict");
From your shell:
$ curl -d 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
http://pastebin.com/598VLDZp
$
Now if you see the URL http://pastebin.com/598VLDZp, you'll see my paste :)
Alternatively you can do it using the wget command which uses the option --post-data to sent POST values.
I've tried this command it works fine:
wget --post-data 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
Put the following in your .bashrc:
sprunge() {
if [[ $1 ]]; then
curl -F 'sprunge=<-' "http://sprunge.us" <"$1"
else
curl -F 'sprunge=<-' "http://sprunge.us"
fi
}
...and then you can run:
sprunge filename # post file to sprunge
...or...
some_command | sprunge # pipe output to sprunge
The API for posting to pastebin has changed, since posted by codaddict.
Details can be found at this link: https://pastebin.com/api
Example:
curl -d 'api_paste_code=printf("Hello..\n I am Codaddict");' \
-d 'api_dev_key=<get_your_own>' \
-d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
There are three essential fields as of now:
api_dev_key -> You need to create a login on pastebin.com in order to get that
api_option -> Format in which to post
api_paste_code -> Text you want to post
Two other answers (from circa 2014) point to http://sprunge.us, which is designed to be used like this...
curl --form 'sprunge=#yourfile.txt' sprunge.us
However, as of 2018, sprunge.us has a tendency to be overloaded and return 500 Internal Server Error to every request. For files up to at least 300 KB but not as high as 2.8 MB, I have had good luck with the very similar service at http://ix.io:
curl --form 'f:1=#yourfile.txt' ix.io
For files up to at least 2.8 MB (and maybe higher, I don't know), I've found the more highly polished https://transfer.sh. It recommends a slightly different and simpler command line, and requires https (it won't work without it):
curl --upload-file yourfile.txt https://transfer.sh
I have found that Sprunge is currently down, but dpaste.com has a simple API.
To post from STDIN
curl -s -F "content=<-" http://dpaste.com/api/v2/
from a file foo.txt
cat foo.txt | curl -s -F "content=<-" http://dpaste.com/api/v2/
to post a string
curl -s -F "content=string" http://dpaste.com/api/v2/
The response will be a plain text URL to the paste.
Nb: the trailing / in the URL http://dpaste.com/api/v2/ seems necessary
https://paste.c-net.org/ has a simpler API than all of them. Simply "POST" to it.
From the website:
Upload text using curl:
$ curl -s --data 'Hello World!' 'https://paste.c-net.org/'
Upload text using wget:
$ wget --quiet -O- --post-data='Hello World!' 'https://paste.c-net.org/'
Upload a file using curl:
$ curl --upload-file #'/tmp/file' 'https://paste.c-net.org/'
Upload a file using wget:
$ wget --quiet -O- --post-file='/tmp/file' 'https://paste.c-net.org/'
Upload the output of a command or script using curl:
$ ls / | curl --upload-file - 'https://paste.c-net.org/'
$ ./bin/hello_world | curl -s --data-binary #- 'https://paste.c-net.org/'
You can also simply use netcat. Unlike termbin, paste.c-net.org won't time out if your script takes more than 5 seconds to produce its output.
$ { sleep 10; ls /; } | nc termbin.com 9999
$ { sleep 10; ls /; } | nc paste.c-net.org 9999
https://paste.c-net.org/ExampleOne
Easiest way to post to pastebin
echo 'your message' | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key=<your_api_key>' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
Just change the <your_api_key> part and pipe whatever you want into it.
The sed invocations add the api_paste_code parameter to beginning of the message and add a newline at the end of each line so it can handle multiline input. The #- tells curl to read from stdin.
A Bash Function You Can Paste
For easy reuse, make it a bash function (copy and paste this into your terminal and set the API_KEY field appropriately:
pastebin () {
API_KEY='<your_api_key>'
if [ -z $1 ]
then
cat - | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
else
echo "$1" | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
fi
printf '\n'
}
You can run it with either:
pastebin 'your message'
or if you need to pipe a file into it:
cat your_file.txt | pastebin
To built upon Vishal's answer, pastebin has upgraded to only use HTTPS now:
curl -d 'api_paste_code=printf("Hello World");' \
-d 'api_dev_key=<your_key>' \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
You don't have to specify the -X POST parameter
Additional details can be found here:
https://pastebin.com/doc_api#1
Based on another answer on this page, I wrote the following script which reads from STDIN (or assumes output it piped into it).
This version allows for arbitrary data which is URI escaped (by jq).
#!/bin/bash
api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
curl -d "api_paste_code=$(jq -sRr #uri)" \
-d "api_dev_key=$api_key" \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
echo # By default, there's no newline
I am a bit late to this post, but I created a little tool to help with this.
https://pasteshell.com/
Feel free to check it out and let me know what you think.
Thanks,
I have the feeling that I'm missing the obvious, but have not succeeded with man [curl|wget] or google ("http" makes such a bad search term). I'm looking for a quick&dirty fix to one of our webservers that frequently fails, returning status code 500 with an error message. Once this happens, it needs to be restarted.
As the root cause seems to be hard to find, we're aiming for a quick fix, hoping that it will be enough to bridge the time until we can really fix it (the service doesn't need high availability)
The proposed solution is to create a cron job that runs every 5 minutes, checking http://localhost:8080/. If this returns with status code 500, the webserver will be restarted. The server will restart in under a minute, so there's no need to check for restarts already running.
The server in question is a ubuntu 8.04 minimal installation with just enough packages installed to run what it currently needs. There is no hard requirement to do the task in bash, but I'd like it to run in such a minimal environment without installing any more interpreters.
(I'm sufficiently familiar with scripting that the command/options to assign the http status code to an environment variable would be enough - this is what I've looked for and could not find.)
I haven't tested this on a 500 code, but it works on others like 200, 302 and 404.
response=$(curl --write-out '%{http_code}' --silent --output /dev/null servername)
Note, format provided for --write-out should be quoted.
As suggested by #ibai, add --head to make a HEAD only request. This will save time when the retrieval is successful since the page contents won't be transmitted.
I needed to demo something quickly today and came up with this. Thought I would place it here if someone needed something similar to the OP's request.
#!/bin/bash
status_code=$(curl --write-out %{http_code} --silent --output /dev/null www.bbc.co.uk/news)
if [[ "$status_code" -ne 200 ]] ; then
echo "Site status changed to $status_code" | mail -s "SITE STATUS CHECKER" "my_email#email.com" -r "STATUS_CHECKER"
else
exit 0
fi
This will send an email alert on every state change from 200, so it's dumb and potentially greedy. To improve this, I would look at looping through several status codes and performing different actions dependent on the result.
curl --write-out "%{http_code}\n" --silent --output /dev/null "$URL"
works. If not, you have to hit return to view the code itself.
Although the accepted response is a good answer, it overlooks failure scenarios. curl will return 000 if there is an error in the request or there is a connection failure.
url='http://localhost:8080/'
status=$(curl --head --location --connect-timeout 5 --write-out %{http_code} --silent --output /dev/null ${url})
[[ $status == 500 ]] || [[ $status == 000 ]] && echo restarting ${url} # do start/restart logic
Note: this goes a little beyond the requested 500 status check to also confirm that curl can even connect to the server (i.e. returns 000).
Create a function from it:
failureCode() {
local url=${1:-http://localhost:8080}
local code=${2:-500}
local status=$(curl --head --location --connect-timeout 5 --write-out %{http_code} --silent --output /dev/null ${url})
[[ $status == ${code} ]] || [[ $status == 000 ]]
}
Test getting a 500:
failureCode http://httpbin.org/status/500 && echo need to restart
Test getting error/connection failure (i.e. 000):
failureCode http://localhost:77777 && echo need to start
Test not getting a 500:
failureCode http://httpbin.org/status/400 || echo not a failure
Here is my implementation, which is a bit more verbose than some of the previous answers
curl https://somewhere.com/somepath \
--silent \
--insecure \
--request POST \
--header "your-curl-may-want-a-header" \
--data #my.input.file \
--output site.output \
--write-out %{http_code} \
> http.response.code 2> error.messages
errorLevel=$?
httpResponse=$(cat http.response.code)
jq --raw-output 'keys | #csv' site.output | sed 's/"//g' > return.keys
hasErrors=`grep --quiet --invert errors return.keys;echo $?`
if [[ $errorLevel -gt 0 ]] || [[ $hasErrors -gt 0 ]] || [[ "$httpResponse" != "200" ]]; then
echo -e "Error POSTing https://somewhere.com/somepath with input my.input (errorLevel $errorLevel, http response code $httpResponse)" >> error.messages
send_exit_message # external function to send error.messages to whoever.
fi
With netcat and awk you can handle the server response manually:
if netcat 127.0.0.1 8080 <<EOF | awk 'NR==1{if ($2 == "500") exit 0; exit 1;}'; then
GET / HTTP/1.1
Host: www.example.com
EOF
apache2ctl restart;
fi
To follow 3XX redirects and print response codes for all requests:
HTTP_STATUS="$(curl -IL --silent example.com | grep HTTP )";
echo "${HTTP_STATUS}";
i didn't like the answers here that mix the data with the status.
found this:
you add the -f flag to get curl to fail and pick up the error status code from the standard status var: $?
https://unix.stackexchange.com/questions/204762/return-code-for-curl-used-in-a-command-substitution
i don't know if it's perfect for every scenario here, but it seems to fit my needs and i think it's much easier to work with
this can help to evaluate http status
var=`curl -I http://www.example.org 2>/dev/null | head -n 1 | awk -F" " '{print $2}'`
echo http:$var
Another variation:
status=$(curl -sS -I https://www.healthdata.gov/user/login 2> /dev/null | head -n 1 | cut -d' ' -f2)
status_w_desc=$(curl -sS -I https://www.healthdata.gov/user/login 2> /dev/null | head -n 1 | cut -d' ' -f2-)
Here comes the long-winded – yet easy to understand – script, inspired by the solution of nicerobot, that only requests the response headers and avoids using IFS as suggested here. It outputs a bounce message when it encounters a response >= 400. This echo can be replaced with a bounce-script.
# set the url to probe
url='http://localhost:8080'
# use curl to request headers (return sensitive default on timeout: "timeout 500"). Parse the result into an array (avoid settings IFS, instead use read)
read -ra result <<< $(curl -Is --connect-timeout 5 "${url}" || echo "timeout 500")
# status code is second element of array "result"
status=${result[1]}
# if status code is greater than or equal to 400, then output a bounce message (replace this with any bounce script you like)
[ $status -ge 400 ] && echo "bounce at $url with status $status"
To add to #DennisWilliamson comment above:
#VaibhavBajpai: Try this: response=$(curl --write-out \n%{http_code} --silent --output - servername) - the last line in the result will be the response code
You can then parse the response code from the response using something like the following, where X can signify a regex to mark the end of the response (using a json example here)
X='*\}'
code=$(echo ${response##$X})
See Substring Removal: http://tldp.org/LDP/abs/html/string-manipulation.html
Assuming you have already implemented a stop and start script for your application. Create a script as follows which checks the http status of your application url and restarts in case of 502:
httpStatusCode=$(curl -s -o /dev/null -w "%{http_code}" https://{your_url}/)
if [ $httpStatusCode = 502 ]; then sh /{path_to_folder}/stopscript.sh sh /{path_to_folder}/startscript.sh fi
Implement a cron job to invoke this script every 5 mins. Assuming the script above has name checkBootAndRestart.sh. Then your crontab should look like- */5 * * * * /{path_to_folder}/checkBootAndRestart.sh
Im trying to extract a line from wget's result but having trouble with it.
This is my wget call:
$ wget -SO- -T 1 -t 1 http://myurl.com:15000/myhtml.html
Output:
--18:24:12-- http://xxx.xxxx.xxxx:15000/myhtml.html
=> `-'
Resolving xxx.xxxx.xxxx... xxx.xxxx.xxxx
Connecting to xxx.xxxx.xxxx|xxx.xxxx.xxxx|:15000... connected.
HTTP request sent, awaiting response...
HTTP/1.1 302 Found
Date: Tue, 18 Nov 2008 23:24:12 GMT
Server: IBM_HTTP_Server
Expires: Thu, 01 Dec 1994 16:00:00 GMT
Location: https://xxx.xxxx.xxxx/siteminderagent/...
Content-Length: 508
Keep-Alive: timeout=10, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
Location: https://xxx.xxxx.xxxx//siteminderagent/...
--18:24:13-- https://xxx.xxxx.xxxx/siteminderagent/...
=> `-'
Resolving xxx.xxxx.xxxx... failed: Name or service not known.
if I do this:
$ wget -SO- -T 1 -t 1 http://myurl.com:15000/myhtml.html | egrep -i "302" <br/>
It doesnt return me the line that contains the string. I just want to check if the site or siteminder is up.
The output of wget you are looking for is written on stderr. You must redirect it:
$ wget -SO- -T 1 -t 1 http://myurl.com:15000/myhtml.html 2>&1 | egrep -i "302"
wget prints the headers to stderr, not to stdout. You can redirect stderr to stdout as follows:
wget -SO- -T 1 -t 1 http://myurl.com:15000/myhtml.html 2>&1 | egrep -i "302"
The "2>&1" part says to redirect ('>') file descriptor 2 (stderr) to file descriptor 1 (stdout).
A bit enhanced version of already provided solution
wget -SO- -T 1 -t 1 http://myurl.com:15000/myhtml.html 2>&1 >/dev/null | grep -c 302
2>&1 >/dev/null will trim off unneeded output. This way egrep will parse only wget`s stderr, what eliminates possibility to catch strings containing 302 from stdout (where html file itself outputted + download proces bar with resulting bytes count e.t.c.) :)
egrep -c counts number of matched strings instead of simply output them. Enough to know how much strings egrep matched.
wget --server-response http://www.amazon.de/xyz 2>&1 | awk '/^ HTTP/{print $2}'
Just to explicate a bit. The -S switch in the original question is shorthand for --server-response.
Also, I know the OP specified wget, but curl is similar and defaults to STDOUT .
curl --head --silent $yourURL
or
curl -I -s $yourURL
The --silent switch is only needed for grep-ability: (-s turns off progress % meter)
I found this question trying to scrape response codes to large lists of URLs after finding curl very slow (5+s per request).
Previously, I was using this:
curl -o /dev/null -I --silent --head --write-out %{http_code} https://example.com
Building off Piotr and Adam's answers, I came up with this:
wget -Sq -T 1 -t 1 --no-check-certificate --spider https://example.com 2>&1 | egrep 'HTTP/1.1 ' | cut -d ' ' -f 4
This has a few bugs e.g. redirects return 302 200, but overall is greased lightning in comparison.