I have a stream of JSON entities. Here is one of them called 'info':
null
null
{
"id": "qwefhu214o",
"number": "2346",
"date": "28.01.2019"
}
null
null
{
"id": "esg324lif",
"number": "1378",
"date": "29.05.2019"
}
{
"id": "gwrs853sdf",
"number": "4487",
"date": "20.12.2019"
}
I want to extract keys from nested json, so it looks like this:
null
null
{
"info_id": "qwefhu214o",
"info_number": "2346",
"info_date": "28.01.2019"
}
null
null
{
"info_id": "esg324lif",
"info_number": "1378",
"info_date": "29.05.2019"
}
{
"info_id": "gwrs853sdf",
"info_number": "4487",
"info_date": "20.12.2019"
}
I try this, but it doesn't work:
jqr::jq('.[].info |= with_entries(.key |= "info_" + .)')
Its says:
Error: lexical error: invalid char in json text.
NA
(right here) ------^
I guess it because of NULLs. How could i do that? Should i put somewhere "?" in code ? That code works for cases when there are np NULLs
The "nested JSON" shown is actually a stream of JSON entities,
for which the following filter will produce the desired results:
if . then with_entries(.key |= "info_" + .) else . end
You can easily modify this filter as required.
Related
I'm trying to transform below JSON using JQ. Wanted to transform it to a simple JSON which has city if relation's primary is true and relation address type is project. Wanted to populate it as empty string as "" in case source is not having a relation address type of project.
{
"miscellenous" : "apple",
"relations" : [
{
"primary": false,
"rel" : 123,
"address" : [
{
"type" : "project",
"city" : "xyz"
},
{
"type" : "mailing",
"city" : "abc"
}
]
},
{
"primary": true,
"rel" : 456,
"address" : [
{
"type" : "project",
"city" : "xyz"
},
{
"type" : "mailing",
"city" : "abc"
}
]
}
]
}
It's working if values are matching that is primary is true and address type is "project" available in source JSON then transformation is working fine with either of the below JQ logic. But it's not giving expected response if address type "project" is not available and it's giving blank JSON.
Using select
{
"miscellenous" : .miscellenous,
"city" : .relations[] | select(.primary == true) | .address[] | select(.type == "project") | .city
}
Using if else
{
"miscellenous" : .miscellenous,
"city" : .relations[] | (if .primary == true then .address[] | (if .type == "project" then .city else empty end) else empty end)
}
Expected output if address type "project" is available in source JSON - This is working as expected
{
"miscellenous": "apple",
"city": "xyz"
}
Expected output if address type "project" is not available in source JSON - This is not working as expected as it's blanking out whole JSON if "project" type address is not available in source.
{
"miscellenous": "apple",
"city": ""
}
You can use the alternative operator // to provide a default value:
{
miscellenous,
"city": (
(.relations[] | select(.primary) | .address[] | select(.type == "project") | .city) // ""
)
}
Note: Instead of {"miscellenous" : .miscellenous} you can just write {miscellenous}, and instead of select(.primary == true) you can just write select(.primary).
What does the below error mean in xquery :-
SVC-CODEPOINT: (err:FOCH0001) xdmp:unquote("{ "info": { "title": "Inventory...") -- Codepoint not legal
Getting the above error while running xquery file where I have defined a mapping file inside a variable like -
let $mapping := xdmp:unquote('{
"info": {
"title": "Inventory"
},
"InvTable": [
{
"prefix": "pdw",
"default": true
}
],
"ViewNames": {
"InvTable": {
"$ref": "pdw/DAU"
}
}
}')
It's basically saying you have an invalid character. A codepoint is the underlying numerical value for a character and there's something in your string that isn't legal to become JSON, probably. I suggest you simplify what you're unquoting until you find where that's coming from.
This question already has answers here:
How to deep copy a map and then clear the original?
(6 answers)
Closed 4 years ago.
What am I trying to do?
Copy a "default" struct into a new one when needed, keeping all it's values.
Details
I am trying to copy a Chat struct:
type ChatData struct {
User map[string]map[string]string `json:"user"`
Chat map[string]string `json:"chat"`
}
type Chat struct {
Settings map[string]map[string]interface{} `json:"settings"`
Data ChatData `json:"data"`
}
I only need to do this when a new chat is introduced, and I check for membership in a map[string]Chat.
//data is a map[string]Chat
if _, exists := data[event.Chat]; !exists {
data[event.Chat] = data["default"]
}
The full default struct is:
{
"default": {
"settings": {
"eightball": {
"messages": [
"yes",
"no",
"maybe"
]
},
"main": {
"blacklistedCommands": [],
"blacklistedUsers": [],
"error": "there was an error - ",
"maxConsecutive": 5,
"permissions": "You don't have permissions for that command.",
"success": "The command was successful.",
"whitelistedCommands": [],
"whitelistedUsers": []
}
},
"data": {
"user": {
"default": {
"admin": "false",
"consecutiveCommands": "0",
"nickname": "",
"sentMessages": "0"
},
"total": {
"admin": "false",
"consecutiveCommands": "0",
"nickname": "",
"sentMessages": "0"
}
},
"chat": {
"commandSender": "",
"lastMessage": "",
"lastSender": "",
"lastTimestamp": "",
"wasCommand":""
}
}
}
}
What have I tried
data[event.Chat] = data["default"]
// causes a reference
data[event.Chat] = &data["default"]
// cannot use &data["default"] (type *Chat) as type Chat in assignment
data[event.Chat] = *data["default"]
// invalid indirect of data["default"] (type Chat)
Do I need to change my use of map[string]Chat to map[string]*Chat and go with the second option? Pointers and references are not my specialty, so help would be appreciated.
Edit
whoever thought I was copying a map, what are you smoking?
I have found in previous cases, that an easy (though not the most efficient) way to copy a complex structure is to Marshal it, and then Unmarshal it into a new variable. This can be done with a variety of encodings. Two easy ones (included in the stdlib) would be json and gob.
There are also plenty of libraries using reflection to achieve a similar goal: https://github.com/jinzhu/copier
But like I said, though not as efficient, this is easy to reason about and takes just a single, simple function to achieve. If performance matters you should prefer gob over json. If performance REALLY matters, then you should prefer another solution altogether.
If you know the data structure it is best for you to fully define your structs. It will be much easier to manage than complex maps and interfaces with reflection to cast data types. Here is a tool to help you convert JSON to Go structs. It isn't perfect and you have to confirm the data types. You may even want to rip out the internal structs and create types for them for readability and maintenance, but it will save you some time.
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
)
var rawDataExample = []byte(`{
"default": {
"settings": {
"eightball": {
"messages": [
"yes",
"no",
"maybe"
]
},
"main": {
"blacklistedCommands": [],
"blacklistedUsers": [],
"error": "there was an error - ",
"maxConsecutive": 5,
"permissions": "You don't have permissions for that command.",
"success": "The command was successful.",
"whitelistedCommands": [],
"whitelistedUsers": []
}
},
"data": {
"user": {
"default": {
"admin": "false",
"consecutiveCommands": "0",
"nickname": "",
"sentMessages": "0"
},
"total": {
"admin": "false",
"consecutiveCommands": "0",
"nickname": "",
"sentMessages": "0"
}
},
"chat": {
"commandSender": "",
"lastMessage": "",
"lastSender": "",
"lastTimestamp": "",
"wasCommand":""
}
}
}
}
`)
type Settings struct {
Default struct {
Settings struct {
Eightball struct {
Messages []string `json:"messages"`
} `json:"eightball"`
Main struct {
BlacklistedCommands []string `json:"blacklistedCommands"`
BlacklistedUsers []string `json:"blacklistedUsers"`
Error string `json:"error"`
MaxConsecutive int `json:"maxConsecutive"`
Permissions string `json:"permissions"`
Success string `json:"success"`
WhitelistedCommands []string `json:"whitelistedCommands"`
WhitelistedUsers []string `json:"whitelistedUsers"`
} `json:"main"`
} `json:"settings"`
Data struct {
User struct {
Default struct {
Admin string `json:"admin"`
ConsecutiveCommands string `json:"consecutiveCommands"`
Nickname string `json:"nickname"`
SentMessages string `json:"sentMessages"`
} `json:"default"`
Total struct {
Admin string `json:"admin"`
ConsecutiveCommands string `json:"consecutiveCommands"`
Nickname string `json:"nickname"`
SentMessages string `json:"sentMessages"`
} `json:"total"`
} `json:"user"`
Chat struct {
CommandSender string `json:"commandSender"`
LastMessage string `json:"lastMessage"`
LastSender string `json:"lastSender"`
LastTimestamp string `json:"lastTimestamp"`
WasCommand string `json:"wasCommand"`
} `json:"chat"`
} `json:"data"`
} `json:"default"`
}
type Data struct {
Events map[string]Settings
}
func main() {
var settings Settings
err := json.Unmarshal(rawDataExample, &settings)
if err != nil {
log.Fatal(err)
}
event := "foo"
d := Data{
Events: make(map[string]Settings),
}
if _, ok := d.Events[event]; !ok {
// event doesn't exist
// add it
d.Events[event] = settings
fmt.Println("event added")
}
if _, ok := d.Events[event]; !ok {
// event exist, this will never be happen
// sanity check
fmt.Println("this will never be printed")
}
fmt.Printf("%+v\n", d)
}
Super simple fix, just wanted to get some input before I did it. Changed the map[string]Chat > map[string]*Chat and made a new *Chat, then copied the data.
data[event.Chat] = &Chat{}
*data[event.Chat] = *data["default"]
I have a set of timestamps formatted as seconds since the epoch. I'd like to insert to ElasticSearch as epoch_seconds but when querying would like to see the output as a pretty date, e.g. strict_date_optional_time.
My below mapping preserves the format that the input came in - is there any way to normalize the output to just one format via the mapping api?
Current Mapping:
PUT example
{
"mappings": {
"time": {
"properties": {
"time_stamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_second"
}
}
}
}
}
Example docs
POST example/time
{
"time_stamp": "2018-03-18T00:00:00.000Z"
}
POST example/time
{
"time_stamp": "1521389162" // Would like this to output as: 2018-03-18T16:05:50.000Z
}
GET example/_search output:
{
"total": 2,
"max_score": 1,
"hits": [
{
"_source": {
"time_stamp": "1521389162", // Stayed as epoch_second
}
},
{
"_source": {
"time_stamp": "2018-03-18T00:00:00.000Z"
}
}
]
}
Elasticsearch differentiates between the _source and the so called stored fields. The first one is supposed to represent your input.
If you actually use stored fields (by specifying store=true in your mapping) then specify multiple date formats this is easy: (emphasis mine)
Multiple formats can be specified by separating them with || as a separator. Each format will be tried in turn until a matching format is found. The first format will be used to convert the milliseconds-since-the-epoch value back into a string.
I have tested this with elasticsearch 5.6.4 and it works fine:
PUT /test -d '{ "mappings": {"doc": { "properties": {"post_date": {
"type":"date",
"format":"basic_date_time||epoch_millis",
"store":true
} } } } }'
PUT /test/doc/2 -d '{
"user" : "test1",
"post_date" : "20150101T121030.000+01:00"
}'
PUT /test/doc/1 -d '{
"user" : "test2",
"post_date" : 1525167490500
}'
Note how two different input-formats will result in the same format when using GET /test/_search?stored_fields=post_date&pretty=1
{
"hits" : [
{
"_index" : "test",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"fields" : {
"post_date" : [
"20150101T111030.000Z"
]
}
},
{
"_index" : "test",
"_type" : "doc",
"_id" : "1",
"_score" : 1.0,
"fields" : {
"post_date" : [
"20180501T093810.500Z"
]
}
}
]
}
If you want to change the input (in _source) you're not so lucky, the mapping-transform feature has been removed:
This was deprecated in 2.0.0 because it made debugging very difficult. As of now there really isn’t a feature to use in its place other than transforming the document in the client application.
If, instead of changing the stored data you are interested in formatting the output, have a look at this answer to Format date in elasticsearch query (during retrieval)
Suppose I have a contract like this specified in groovy:
org.springframework.cloud.contract.spec.Contract.make {
request {
method "GET"
url "/api/profiles"
headers {
header('Accept': 'application/json;charset=UTF-8')
header('Content-Type': 'application/json;charset=UTF-8')
}
}
response {
status 200
headers {
header('Content-Type': 'application/json;charset=UTF-8')
}
body(
value(
stub(
'''\
[
{
"profile": "profile1",
"myMap": {}
},
{
"profile": "profile2",
"myMap": {
"12345": "FOO",
"asdf": "BAR"
}
}
]
'''
),
test(
[
[
"profile" : regex(nonEmpty()),
"myMap": [
[
??
]
]
]
]
)
)
)
}
}
Now I want to test that the map contains String to String entries where the values must not be empty. The map itself may be empty.
How can I test for dynamic key name?
On the response side of the contract you have to chose whether you're using the map notation or the string notation. If you want to do assertions on pieces of the response you have to embed those assertions inside the body or use the test matchers.
You can put the body as a multiline string and then write the testMatchers section
testMatchers{
jsonPath('$.[*].myMap', byCommand('assertKeys($it)'))
}
then it's enough for you to provide the assertion in the assertKeys method.