I have been working with an API call to structure it in JSON format so I might later push it into a database. Then code looks like this:
getPage() {
curl --fail -X GET 'https://api.app.com/v1/test?page=1&pageSize=1000&sort=desc' \
-H 'Authorization: Bearer 123abc456pickupsticks789' \
-H 'cache-control: no-cache'
}
getPage \
| jq -c '.items | .[] | {landing_id: .landing_id, submitted_at: .submitted_at, answers: .answers, email: .hidden.email}' \
> testpush.json
When I run it though, it produces this error: jq: error (at :0): Cannot iterate over null (null)
I've looked at solutions such as this one, or this one from this site, and this response.
The common solution seemed to be using a ? in front of [] and I tried it in the jq line towards the bottom, but it still does not work. It just produces an empty json file.
Am I misreading the takeaway from those other answers and not putting my ? in the right place?>
That means your field .items doesn't exist or is null if interpreting the output of your curl as a JSON.
As a side note, the shorter version of your query:
.items[] | {landing_id, submitted_at, answers, email: .hidden.email}
Related
This confused me so much today. Though I've got some ideas after doing some study, I would like to get some opinions here.
So here's one of the confusing API documents I've encoutered: https://console.faceplusplus.com.cn/documents/4888373. No need to read it carefully, I got confused when I saw this line:
Request Parameter
| Name | Type | Description
Optional | return_landmark | Int | Whether or not detect and return key points of facial features and contour. ...
How could there be a Int type in post body?
Another example I saw is:
parameter name | type | required | description
X-TC-Timestamp | Integer | Yes |Current UNIX timestamp to record the time an API request was made.For example, 1529223702.Note: if you miss the server time by more than 5 minutes, you will cause a signature expiration error.
And it's a custom content header required by the API. content header value, Integer?
Then I saw some demo code use it as a string...
curl -X POST https://somehost \
-H "Authorization: blablabla" \
-H "Content-Type: application/json; charset=utf-8" \
-H "Host: somehost" \
-H "X-TC-Action: DescribeInstances" \
-H "X-TC-Timestamp: 1551113065" \
-H "X-TC-Version: 2017-03-12" \
-d '{"Limit": 1, "Filters": [{"Values": ["\u672a\u547d\u540d"], "Name": "instance-name"}]}'
I'm running this query with zsh:
output=$(aws sagemaker describe-training-job \
--training-job-name $name \
--query '{S3ModelArtifacts:ModelArtifacts.S3ModelArtifacts,TrainingImage:AlgorithmSpecification.TrainingImage,RoleArn:RoleArn}')
But for the life of me I can't seem to individually extract out S3ModelArtifacts, TrainingImage, and RoleArn.
It seems to be neither an array nor an associative array? But it looks like it's json format when I do echo $output.
Ultimately I just want to be able to do something like var=${output[TrainingImage]} but this just gives me the whole response instead of just the TrainingImage value.
Any help appreciated.
You can use the command line tool jq to parse json output like so:
(19-11-27 10:25:38) <0> [~] printf %s "$output" | jq '.TrainingImage'
"123456789877.dkr.ecr.eu-west-1.amazonaws.com/kmeans:1"
Or, as this is a pretty simple query, you can use sed:
(19-11-27 10:25:43) <0> [~] printf %s "$output" | sed -n -e 's/^.*TrainingImage"://p'
"123456789877.dkr.ecr.eu-west-1.amazonaws.com/kmeans:1",
Here is the explanation of the sed command.
I have some json data and I want to interactively query it with fzf and jq, by sending the data through stdin and typing the jq query into the fzf query box.
My attempt so far is showing one result in the box, but editing the contents of the query box turns the results blank instead.
fzf-tmux --preview 'jq "$#" <<< {}' <<<'[{"x": 1}, {"y": 2}]'
A recent Hacker News post about using fzf as a REPL had me thinking it would be awesome to live-edit jq filters as well. Using the base implementation from that article, I ended up with:
echo '' | fzf --print-query --preview='jq {q} <(echo "[{"x": 1}, {"y": 2}]")'
You can clean up the quoting a bit, at the expense of some verbosity, by changing it to:
(export json='[{"x": 1}, {"y": 2}]'; echo '' | fzf --print-query --preview='jq {q} <(echo $json)')
or (somewhat safer for unvalidated input):
(export json='[{"x": 1}, {"y": 2}]'; echo '' | fzf --print-query --preview='jq {q} <(printf "%s" "$json")')
Final example, using the StackExchange API to retrieve this post:
(export json=$(curl -s --compressed -H "Accept-Encoding: GZIP" "https://api.stackexchange.com/2.2/posts/56744579?site=stackoverflow&filter=withbody"); echo '' | fzf --print-query --preview-window=wrap --preview='filter={q}; jq -M -r "${filter}" <(printf "%s" "$json")')
One more example, added around 18 months later. This is the same as the previous example, but for the fish shell. It also uses httpie to clean things up as well, since httpie automatically handles things like the encoding/compression. I also left in the color output on this one:
begin
set -lx jq_url 'https://api.stackexchange.com/2.2/posts/56744579?site=stackoverflow&filter=withbody'
echo '' | fzf --print-query --preview='set -x q {q}; jq -C {q} (http -b GET "$jq_url" | psub)'
end
Note: The begin/end block is only there to keep variables in a local scope. They really aren't required for the example to work, just to keep from polluting the namespace.
If you're expecting $# to be expanded by the shell, then the simple fix is to modify the quoting:
fzf-tmux --preview 'jq '"$#"' <<< {}'
If on the other hand, you want to use the {q} feature of fzf, which seems to be the case, then you may be out of luck, though whether that's because of a bug in fzf, or some incompatibility between jq and fzf, I cannot tell.
Navigating by paths
Let's suppose $JSON is a file containing a single JSON array or object. Then when running the following, you'll see the paths on the LHS, and the value at the selected path on the RHS:
jq -rc paths "$JSON" |
fzf-tmux --preview 'x={}; jq "getpath($x)" '"$JSON"
I am using the Twitter REST API to retrieve data in JSON format. Twitter's developer page makes it easy by providing a command that can be pasted directly into the terminal and executed. The following command works in the terminal.
curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key="####", oauth_nonce="####", oauth_signature="####", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1467719924", oauth_token="####", oauth_version="1.0"' --verbose
I am trying to get the JSON data into R, and would like to execute this same command in the R console. I have tried curlconverter using the exact same code, get an error that there is an unexpected symbol. However, the code is exactly the same. Is there a more suitable package for executing this code?
curlExample <- "curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key="####", oauth_nonce="####", oauth_signature="####", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1467719924", oauth_token="####", oauth_version="1.0"' --verbose"
I think you are going to need to escape those quotation marks. Try this instead:
curlExample <- "curl --get 'https://api.twitter.com/1.1/statuses/user_timeline.json' --data 'count=3200&screen_name=BernieSanders' --header 'Authorization: OAuth oauth_consumer_key=\"####\", oauth_nonce=\"####\", oauth_signature=\"####\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1467719924\", oauth_token=\"####\", oauth_version=\"1.0\"' --verbose"
That the quotes were not being escaped properly was actually evident in your question, because the Stack Overflow markup rendered the curlExample string in multiple colors.
I believe https://cran.r-project.org/web/packages/twitteR/twitteR.pdf is the way to do this in R.
But specifically in your case, you have unescaped " in your string.
I am struggling with the proper jq syntax for pulling out all the names from a curl call that looks like this:
my #repo = curl -s -R -D $tmp_fh_header -u $o_user:$o_auth https://api.github.com/repos/mycompany/$repo/commits| jq '.[].login';
In my case, it should report 10 names, but it only comes back with 5 nulls.
Can you point me in the right direction?
If you need the names for the author of the last commits, you can use:
curl -s https://api.github.com/repos/mycompany/$repo/commits | jq '.[].author.login'
If you need the names for the committer of the last commits, you can use:
curl -s https://api.github.com/repos/mycompany/$repo/commits | jq '.[].committer.login'
or if you need the unique names for that last call:
curl -s https://api.github.com/repos/mycompany/$repo/commits | jq '[.[].committer.login] | unique'
You can use pagination for more/less results as documented in https://developer.github.com/v3/#pagination