I'm trying to make a simple filtering script using AWS cli + jq (powershell or cmd in Windows).
aws iam list-roles | jq -c '.Roles[].RoleName | select(startswith ("blabla"))'
But getting this error:
jq: error: AD_/0 is not defined at <top-level>, line 1:
.Roles[].RoleName | select(startswith (AD_))
jq: 1 compile error
But using jqplay.org with same JSON everything works well.
Any thoughts?
thanks!
aws iam list-roles | jq -c '.Roles[].RoleName | select(startswith ("blabla"))'
blabla_rolename_1
It looks like you might be missing a string after the startswith function. Try adding a string in quotes after startswith:
aws iam list-roles | jq -c '.Roles[].RoleName | select(startswith("blabla"))'
If you still get an error, you can try running the jq command without the -c flag to see the full output from jq, which might give you more information about the error.
aws iam list-roles | jq '.Roles[].RoleName | select(startswith("blabla"))
AFAIK Powershell does not support single quotes. Use double quotes and escaped double quotes instead:
aws iam list-roles | jq -c ".Roles[].RoleName | select(startswith (\"blabla\"))"
I use cat file.json | jq -c to print out a minified json file in the logs in an AWS pipeline step. I've tested that it works locally but in the pipeline it fails and prints out the system usage:
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
...
Why does it fail in the pipeline but not locally?
That's not the right question. jq -c (with no further argument) should fail. According to the very output you obtained from jq, jq always requires at least one argument: a program.[1] That fact that it sometimes doesn't fail when no program is provided is a bug. One that appears to have been fixed in 1.6.[2]
If you simply want to reformat the JSON, you can use the trivial program ..
cat input.json | jq -c .
If the input really is a file, the following is better:
jq -c . input.json
Or the name of a file containing the program if -f is used.
Meaning I wasn't able to reproduce the lack of error in 1.6 after trying using Windows, Cygwin and Ubuntu builds.
The solution was to use cat file.json | jq -c . (note the dot that's added) in AWS.
After some experimenting, I found out that this happens only in jq 1.5 (I had jq 1.6 locally), and only when using jq -c instead of jq -c . in a detached terminal. My AWS pipeline uses jq 1.5.
I reproduced it locally by downloading jq 1.5 and running:
$ cat input.json | ./jq1.5 -c >output.txt &
[1] 822
jq - commandline JSON processor [version 1.5]
Usage: C:\Workspace\jq\jq1.5.exe [options] <jq filter> [file...]
Note that the ampersand executes it in the background which simulates a detached terminal.
What works:
cat input.json | ./jq1.6 -c >output.txt &
cat input.json | ./jq1.5 -c . >output.txt &
cat input.json | ./jq1.5 -c >output.txt
To sum up, if using jq1.5 in a detached terminal, also use the dot.
I am trying to use jq in the following to get a status on my pod "my pod":
curl '127.0.0.1:8080/api/v1/pods' | jq -r '.items[] | select(.metadata.name)'
This give me a ton of json, more specific how do you read json from k8s api - to get the status of a pod - running or not?
If I get items within a pod:
curl '127.0.0.1:8080/api/v1/pods' | jq -r 'select(.items[].metadata.name="go-test-volume1").items[]
Sure enough I get a list, but which one to chose - there is a status.phase in all of them?
In the case of a status of CrashLoopBackOff:
curl '127.0.0.1:8080/api/v1/pods' |
jq -r '.items[] | select(.metadata.name == "my-pod").status.phase'
Will show it as Running?
This is what you need :
curl '127.0.0.1:8080/api/v1/pods' |
jq -r '.items[] | select(.metadata.name == "my-pod").status.phase'
This will give you the right status:
curl '127.0.0.1:8080/api/v1/pods' |
jq -r '.items[] | select(.metadata.name == "my-pod).status.containerStatuses[0].state'
I'm using:
GET https://localhost/api/v4/search?scope=projects&search=test
to find project named "test" , but I get not only project named "test" but "qtest", "testot" or "test1" too.
Is it possible to get only exact name?
According to the Projects API, no. It will return all projects that contain your search string, but you should be able to filter the results once you retrieve it.
The available options are sort and order_by. You can order by the fields id, name, created_at, and last_activity_at
One may use jq to filter the returned results from a fuzzy search:
project_name="test"
curl --silent --show-error --location \
"https://gitlab.com/api/v4/search?scope=projects&search=${project_name}" \
--header "PRIVATE-TOKEN: ${GITLAB_COM_API_PRIVATE_TOKEN}" | jq \
--raw-output --arg project_name "${project_name}" '.[] | \
select(.name == $project_name)'
Of course, this returns 12 projects. Which one is mine?
It is far better if you can include a namespace.
project_path_with_namespace="$username/test"
curl --silent --show-error --location \
"https://gitlab.com/api/v4/search?scope=projects&search=${project_path_with_namespace}" \
--header "PRIVATE-TOKEN: ${GITLAB_COM_API_PRIVATE_TOKEN}" | jq \
--arg path_with_namespace "${project_path_with_namespace}" '.[] | \
select(.path_with_namespace == $path_with_namespace)'
Now the search is not fuzzy and will always return a precise result so long as such a project exists at that path.
Is it possible to use pastebin (may be via their "API" functionality) inside bash shell scripts? How do I send http-post? How do I get back the URL?
As pastebin.com closed their public api, I was looking for alternatives.
Sprunge is great. Usage:
<command> | curl -F 'sprunge=<-' http://sprunge.us
or, as I use it:
alias paste="curl -F 'sprunge=<-' http://sprunge.us"
<command> | paste
The documentation says that you need to submit a POST request to
http://pastebin.com/api_public.php
and the only mandatory parameter is paste_code, of type string is the paste that you want to make.
On success a new pastebin URL will be returned.
You can easily do this from your bash shell using the command curl.
curl uses the -d option to send the POST data to the specified URL.
Demo:
This demo will create a new paste with the code:
printf("Hello..I am Codaddict");
From your shell:
$ curl -d 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
http://pastebin.com/598VLDZp
$
Now if you see the URL http://pastebin.com/598VLDZp, you'll see my paste :)
Alternatively you can do it using the wget command which uses the option --post-data to sent POST values.
I've tried this command it works fine:
wget --post-data 'paste_code=printf("Hello..I am Codaddict");' 'http://pastebin.com/api_public.php'
Put the following in your .bashrc:
sprunge() {
if [[ $1 ]]; then
curl -F 'sprunge=<-' "http://sprunge.us" <"$1"
else
curl -F 'sprunge=<-' "http://sprunge.us"
fi
}
...and then you can run:
sprunge filename # post file to sprunge
...or...
some_command | sprunge # pipe output to sprunge
The API for posting to pastebin has changed, since posted by codaddict.
Details can be found at this link: https://pastebin.com/api
Example:
curl -d 'api_paste_code=printf("Hello..\n I am Codaddict");' \
-d 'api_dev_key=<get_your_own>' \
-d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
There are three essential fields as of now:
api_dev_key -> You need to create a login on pastebin.com in order to get that
api_option -> Format in which to post
api_paste_code -> Text you want to post
Two other answers (from circa 2014) point to http://sprunge.us, which is designed to be used like this...
curl --form 'sprunge=#yourfile.txt' sprunge.us
However, as of 2018, sprunge.us has a tendency to be overloaded and return 500 Internal Server Error to every request. For files up to at least 300 KB but not as high as 2.8 MB, I have had good luck with the very similar service at http://ix.io:
curl --form 'f:1=#yourfile.txt' ix.io
For files up to at least 2.8 MB (and maybe higher, I don't know), I've found the more highly polished https://transfer.sh. It recommends a slightly different and simpler command line, and requires https (it won't work without it):
curl --upload-file yourfile.txt https://transfer.sh
I have found that Sprunge is currently down, but dpaste.com has a simple API.
To post from STDIN
curl -s -F "content=<-" http://dpaste.com/api/v2/
from a file foo.txt
cat foo.txt | curl -s -F "content=<-" http://dpaste.com/api/v2/
to post a string
curl -s -F "content=string" http://dpaste.com/api/v2/
The response will be a plain text URL to the paste.
Nb: the trailing / in the URL http://dpaste.com/api/v2/ seems necessary
https://paste.c-net.org/ has a simpler API than all of them. Simply "POST" to it.
From the website:
Upload text using curl:
$ curl -s --data 'Hello World!' 'https://paste.c-net.org/'
Upload text using wget:
$ wget --quiet -O- --post-data='Hello World!' 'https://paste.c-net.org/'
Upload a file using curl:
$ curl --upload-file #'/tmp/file' 'https://paste.c-net.org/'
Upload a file using wget:
$ wget --quiet -O- --post-file='/tmp/file' 'https://paste.c-net.org/'
Upload the output of a command or script using curl:
$ ls / | curl --upload-file - 'https://paste.c-net.org/'
$ ./bin/hello_world | curl -s --data-binary #- 'https://paste.c-net.org/'
You can also simply use netcat. Unlike termbin, paste.c-net.org won't time out if your script takes more than 5 seconds to produce its output.
$ { sleep 10; ls /; } | nc termbin.com 9999
$ { sleep 10; ls /; } | nc paste.c-net.org 9999
https://paste.c-net.org/ExampleOne
Easiest way to post to pastebin
echo 'your message' | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key=<your_api_key>' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
Just change the <your_api_key> part and pipe whatever you want into it.
The sed invocations add the api_paste_code parameter to beginning of the message and add a newline at the end of each line so it can handle multiline input. The #- tells curl to read from stdin.
A Bash Function You Can Paste
For easy reuse, make it a bash function (copy and paste this into your terminal and set the API_KEY field appropriately:
pastebin () {
API_KEY='<your_api_key>'
if [ -z $1 ]
then
cat - | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
else
echo "$1" | sed '1s/^/api_paste_code=/g' | sed 's/$/\%0A/g' | curl -d #- -d 'api_dev_key='"$API_KEY"'' -d 'api_option=paste' 'http://pastebin.com/api/api_post.php'
fi
printf '\n'
}
You can run it with either:
pastebin 'your message'
or if you need to pipe a file into it:
cat your_file.txt | pastebin
To built upon Vishal's answer, pastebin has upgraded to only use HTTPS now:
curl -d 'api_paste_code=printf("Hello World");' \
-d 'api_dev_key=<your_key>' \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
You don't have to specify the -X POST parameter
Additional details can be found here:
https://pastebin.com/doc_api#1
Based on another answer on this page, I wrote the following script which reads from STDIN (or assumes output it piped into it).
This version allows for arbitrary data which is URI escaped (by jq).
#!/bin/bash
api_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
curl -d "api_paste_code=$(jq -sRr #uri)" \
-d "api_dev_key=$api_key" \
-d 'api_option=paste' 'https://pastebin.com/api/api_post.php'
echo # By default, there's no newline
I am a bit late to this post, but I created a little tool to help with this.
https://pasteshell.com/
Feel free to check it out and let me know what you think.
Thanks,