Why sub() can not perform a string substitution? - jq

I don't understand why the sub() command does not replace "%s" with "my string" in the first jq command below. How to make it work?
$ jq -r --arg format '|%s|' '$format | sub("%s"; .desc)' <<< '{"desc": "my string"}'
||
$ jq -r --arg format '|%s|' '$format | sub("%s"; "my string")' <<< '{"x": "y"}'
|my string|
$ jq -r .desc <<< '{"desc": "my string"}'
my string


You have lost the input context. Save it in a variable (eg. . as $dot) to reference it later (eg. $dot.desc):
$ jq -r --arg format '|%s|' '. as $dot | $format | sub("%s"; $dot.desc)' <<< '{"desc": "my string"}'
|my string|


You can use null input -n :
jq -nr --arg format '|%s|' '$format | sub("%s"; input.desc)' <<< '{"desc": "my string"}'

Related

Extract fields from json using jq

I am trying write a shell script that will get some json from URL and parse the json and extract fields.
This is what is done so far.
#!/bin/bash
token=$(http POST :3000/signin/frontm user:='{"email": "sourav#frontm.com", "password": "Hello_789"}' | jq -r '.data.id_token')
cred=$(http POST :3000/auth provider_name:frontm token:$token user:=#/tmp/user.json | jq '{ creds: .creds, userUuid: .user.userId }')
echo $cred
access=$(jq -r "'$cred'")
echo $access
So the output from echo $cred is a json:
Eg:
{ "creds": { "accessKeyId": "ASIAJPM3RDAZXEORAQ5Q", "secretAccessK
ey": "krg5GbU6gtQV+a5pz4ChL+ECVJm+wKogjglXOqr6", "sessionToken": "Ag
oGb3JpZ2luEAYaCXVzLWVhc3QtMSKAAmhOg7fedV+sBw+8c45HL9naPjqbC0bwaBxq
mQ9Kuvnirob8KtTcsiBkJA/OfCTpYNUFaXXYfUPvbmW5UveDJd+32Cb5Ce+3lAOkkL
aZyWJgvhM1u53WNuMekhcZX7SnlCcaO4e/A9TR74qMOsVptonw5jFB5zjbEI4hFsVX
UHXtkYMYpSyG+2P2LxWRqTg4XKcg2vT+qrLtiXu3XNK70wuCe0/L4/HjjzlLvChmhe
TRs8u8ZRcJvSim/j1sLqe85Sl1qrFv/7msCaxUa3gZ3dOcfHliH64+8NHfS1tkaVkS
iM2x4wxTdZI/SafduFDvGCsltxe9p5zQD0Jb1Qe02ccqpgUIWxAAGgw3NzE5NTYwMD
EyNDciDOQZkq8t+c7WatNLHyqDBahqpQwxpGsYODIC1Db/M4+PXmuYMdYKLwjv3Df2
JeTMw2RT1h8M0IOOPvyBWetwB42HLhv5AobIMkNVSw6tpGyZC/bLMGJatptB0hVMBg
/80VnI7pTPiSjb/LG46bbwlbJevPoorCEEqMZ3MlAJ2Xt2hMmA+sHBRRvV1hlkMnS8
NW6w9xApSGrD001zdfFkmBbHw+c4vmX+TMT7Bw0bHQZ5FQSpEBOw9M5sNOIoa+G/pP
p4WoHiYfGHzaXGQe9Iac07Fy36W/WRebZapvF7TWoIpBjAV+IrQKP3ShJdBi3Oa6py
lGUQysPa3EN0AF/gDuTsdz7TDsErzzUERfQHksK495poG92YoG2/ir8yqTQtUDvshO
7U4SbFpUrozCYT6vp7++BWnpe+miIRCvjy2spqBqv2RY6lhgC6QPfS/365T+QbSTMc
R+ZNes0gX/QrEG4q1sMoxyTltL4sXS2Dz9UXywPkg78AWCOr34ii72m/67Gqe1P3KA
vBe9xF9Hem4H1WbYAqBN76ppyJyG17qK8b2/r71c8rdY+1gYcskV1vUfTQUVCGE0y2
JXKV2UMFOwoTzy6SFIGcuTeOAHiYPgTkMZ6X7hNjf56ihzBIbhSHaST8U4eNBka8j8
Y949ilJwz9QO0l1kwdb2+fQSMblHgeYvF1P8HxBSpRA28gKkkXMf73Zk27I3O2DRGb
lcXS4tKRvan4ASTi4qkdrvVwMT5mwJI4mGIJZSiMJqPxjVh5E9OicFbIOCRcbcIRDE
mj5t9EvaSbIm4ELBMuyoFjmKJmesE03uFRcHkEXkPBxhkJbQwkJeUxHll5kR1IYzvA
K2A2EiZqjkhiSJC4NRekEuM+5WowwuWw1wU=" }, "userUuid": "mugqRKHmTPxk
obBAtwTmKk" }
So basically I am stuck here .. how do i parse this json in $cred further and basically want to get access to say accessKeyId using jq further?

I wonder if the variable $cred really holds a string formated in 67 columns, but if that so, tr might help to remove the newline and extract the accessKeyId using jq:
echo "$cred" | tr -d '\n' | jq -r '.creds.accessKeyId'

Using sed to enclose matches in double quotes

I'm trying to extract headers from emails and create a JSON fragment from them. I'm using sed to pull out the keys and values, but it's failing to put the trailing quote on each of the lines:
$ cat email1 | grep -i -e "^subject:" -e "^from:" -e "^to:" | \
sed -n 's/\^([^:]*\):[ ]*\(.*\)$/"\1":"\2"/gp'
"From":"Blah Blech <blah.blech#blahblech.com>
"To":"foo#bar.com
"Subject":"Yeah
I don't understand why the replacement pattern isn't working.

awk to the rescue!
$ awk -F": *" -vOFS=":" -vq="\"" 'tolower($0)~/^from|to|subject/
{print q$1q,q$2q}' email1
which combines cat or grep steps as well.

Stripping the carriage returns as #tripleee suggested fixed the issue with sed (using ctrl-v ctrl-m to capture the literal carriage return):
$ cat email1 | tr -d '^M' | grep -i -e "^subject:" -e "^from:" -e "^to:" | \
sed -n 's/^\([^:]*\):[ ]*\(.*\)$/"\1":"\2"/gp'
"From":"Blah Blech <blah.blech#blahblech.com>"
"To":"foo#bar.com"
"Subject":"Yeah"

Replace jq predefined variable with number

jq could have predefined variables and replace with --arg options. It works well with string. However, I don't know how to replace the variable as number.
$jq -n --arg number 3000 '{"number":$number}'
{
"number": "3000"
}
I would like to be able to generate something as following :
{
"number": 3000
}
Thanks in advance.

Or:
jq -n --arg number 300 '{"number": $number|tonumber}'
or (as Jeff pointed out) with jq 1.5:
jq -n --argjson number 300 '{"number": $number}'
or (with versions of jq after June 26, 2015):
jq -n --arg number 300 '{$number} | .number |= tonumber'

Here is the solution:
jq -n --argfile number <(printf '%d' 3000) '{"number":$number}'
Or
jq -n --arg number 3000 '{"number":$number|fromjson}'

Printing only the first field in a string

I have a date as 12/12/2013 14:32 I want to convert it into only 12/12/2013.
The string can be 1/1/2013 12:32 or 1/10/2013 23:41 I need only the date part.

You can do this easily with a variety of Unix tools:
$ cut -d' ' -f1 <<< "12/12/2013 14:32"
12/12/2013
$ awk '{print $1}' <<< "12/12/2013 14:32"
12/12/2013
$ sed 's/ .*//' <<< "12/12/2013 14:32"
12/12/2013
$ grep -o "^\S\+" <<< "12/12/2013 14:32"
12/12/2013
$ perl -lane 'print $F[0]' <<< "12/12/2013 14:32"
12/12/2013

$ echo "12/12/2013 14:32" | awk '{print $1}'
12/12/2013
print $1 --> Prints first column of the supplied string. 12/12/2013
print $2 --> Prints second column of the supplied string. 14:32
By default, awk treats the space character as the delimiter.

If your date string is stored in a variable, then you don't need to run an external program like cut, awk or sed, because modern shells like bash can perform string manipulation directly which is more efficient.
For example, in bash:
$ s="1/10/2013 23:41"
$ echo "${s% *}"
1/10/2013

How do I Get the distinct List of Special Characters from a File using GREP or SED?

I have a file which contains about 30000 Records delimited by '|'. I need to get a distinct list of special characters only from the file.
For Eg:
123|fasdf|%df&|pap,came|!
234|%^&asdf|34|'":|
My output should be:
|%&,!^'":
Any help would be greatly appreciated.
Thanks,
Velraj.

grep -o '[|%&,!^":]' input | sort -u
You have to list all your special characters inside brackets.
This will return each unique special character on its own line. If you really need a string with these characters you have to remove newlines afterwards, e.g.:
grep -o '[|%&,!^":]' input | sort -u | tr -d '\n'
UPDATE:
If you need to remove all characters which are not from 'a-zA-Z0-9' set then you can use this one:
grep -o '[^a-zA-Z0-9]' input | sort -u | tr -d '\n'

echo "123|fasdf|%df&|pap,came|! 234|%^&asdf|34|'\":|" \
| { tr -d '[[:alnum:]]'; printf "\n"; } \
| sed 's/\(.\)/\1_/g' \
| awk -v 'RS=_' '{print $0}' \
| sort -u \
| awk '{printf $0}END{printf "\n"}'
output
!"%&',:^||
You can replace the first line echo .... with cat fileName

Resources