Related
When deserializing a string, the curly braces are duplicating and this is disturbing the reading of some fields. I've tried everything, but I can't serialize correctly, without duplicate curly braces.
I already tried to do it like this:
dynamic values = JsonConvert.DeserializeObject<dynamic>(storedEvent.Data);
The storedEvent.Data property is of type string and contains this information:
"{\"PaisId\":31,\"PaisDivisaoAdministrativaNivelRemovedEventList\":[{\"Id\":6,\"PaisId\":31,\"PaisNomePtBr\":\"\",\"PaisDivisaoAdministrativaTipoId\":5,\"PaisDivisaoAdministrativaTipoNome\":\"Município\",\"PaisDivisaoAdministrativaTipoOrigemId\":5,\"Timestamp\":\"2022-11-24T20:16:15.6020289-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelRemovedEvent\",\"AggregateId\":6},{\"Id\":5,\"PaisId\":31,\"PaisNomePtBr\":\"\",\"PaisDivisaoAdministrativaTipoId\":1,\"PaisDivisaoAdministrativaTipoNome\":\"Estado\",\"PaisDivisaoAdministrativaTipoOrigemId\":null,\"Timestamp\":\"2022-11-24T20:16:15.6580242-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelRemovedEvent\",\"AggregateId\":5}],\"Timestamp\":\"2022-11-24T20:16:16.1892039-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelHierarquiasRemovedEvent\",\"AggregateId\":31}"
Result obtained (Duplicate curly braces):
{{
"PaisId": 31,
"PaisDivisaoAdministrativaNivelRemovedEventList": [
{
"Id": 6,
"PaisId": 31,
"PaisNomePtBr": "",
"PaisDivisaoAdministrativaTipoId": 5,
"PaisDivisaoAdministrativaTipoNome": "Município",
"PaisDivisaoAdministrativaTipoOrigemId": 5,
"Timestamp": "2022-11-24T20:16:15.6020289-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelRemovedEvent",
"AggregateId": 6
},
{
"Id": 5,
"PaisId": 31,
"PaisNomePtBr": "",
"PaisDivisaoAdministrativaTipoId": 1,
"PaisDivisaoAdministrativaTipoNome": "Estado",
"PaisDivisaoAdministrativaTipoOrigemId": null,
"Timestamp": "2022-11-24T20:16:15.6580242-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelRemovedEvent",
"AggregateId": 5
}
],
"Timestamp": "2022-11-24T20:16:16.1892039-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelHierarquiasRemovedEvent",
"AggregateId": 31
}}
Expected:
{
"PaisId": 31,
"PaisDivisaoAdministrativaNivelRemovedEventList": [
{
"Id": 6,
"PaisId": 31,
"PaisNomePtBr": "",
"PaisDivisaoAdministrativaTipoId": 5,
"PaisDivisaoAdministrativaTipoNome": "Município",
"PaisDivisaoAdministrativaTipoOrigemId": 5,
"Timestamp": "2022-11-24T20:16:15.6020289-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelRemovedEvent",
"AggregateId": 6
},
{
"Id": 5,
"PaisId": 31,
"PaisNomePtBr": "",
"PaisDivisaoAdministrativaTipoId": 1,
"PaisDivisaoAdministrativaTipoNome": "Estado",
"PaisDivisaoAdministrativaTipoOrigemId": null,
"Timestamp": "2022-11-24T20:16:15.6580242-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelRemovedEvent",
"AggregateId": 5
}
],
"Timestamp": "2022-11-24T20:16:16.1892039-03:00",
"MessageType": "PaisDivisaoAdministrativaNivelHierarquiasRemovedEvent",
"AggregateId": 31
}
Does anyone know of a solution?
Using dynamic is not a good option to work with Json objects dynamically
For this, you can use Jobject as follows
using Newtonsoft.Json.Linq;
string jsonStr= "{\"PaisId\":31,\"PaisDivisaoAdministrativaNivelRemovedEventList\":[{\"Id\":6,\"PaisId\":31,\"PaisNomePtBr\":\"\",\"PaisDivisaoAdministrativaTipoId\":5,\"PaisDivisaoAdministrativaTipoNome\":\"Município\",\"PaisDivisaoAdministrativaTipoOrigemId\":5,\"Timestamp\":\"2022-11-24T20:16:15.6020289-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelRemovedEvent\",\"AggregateId\":6},{\"Id\":5,\"PaisId\":31,\"PaisNomePtBr\":\"\",\"PaisDivisaoAdministrativaTipoId\":1,\"PaisDivisaoAdministrativaTipoNome\":\"Estado\",\"PaisDivisaoAdministrativaTipoOrigemId\":null,\"Timestamp\":\"2022-11-24T20:16:15.6580242-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelRemovedEvent\",\"AggregateId\":5}],\"Timestamp\":\"2022-11-24T20:16:16.1892039-03:00\",\"MessageType\":\"PaisDivisaoAdministrativaNivelHierarquiasRemovedEvent\",\"AggregateId\":31}";
var obj = JObject.Parse(jsonStr);
//access properties
var paisId = obj["PaisId"];
//Access to PaisDivisaoAdministrativaNivelRemovedEventList
var paisList = obj["PaisDivisaoAdministrativaNivelRemovedEventList"].ToList();
//search on PaisDivisaoAdministrativaNivelRemovedEventList
var result = paisList.FirstOrDefault(x => (int)x["Id"] == 6);
You have to fix the code that returns the double braces string. If you don't have access to this code, the only way to get rid of double braces is to use string function
storedEvent.Data=storedEvent.Data.Substring(1,Data.Length-2);
after this you can use your code if you like, but I don't like dynamic, IMHO it is better to use Parse and sqware brakets to get data.
Example json data:
{
"data": [
{
"place": "FM346",
"id": [
"7_day_A",
"7_day_B",
"7_day_C",
"7_day_D"
],
"values": [
0,
30,
23,
43
]
},
{
"place": "LH210",
"id": [
"1_day_A",
"1_day_B",
"1_day_C",
"1_day_D"
],
"values": [
4,
45,
100,
9
]
}
]
}
what i need to transform it into:
{
"data": [
{
"place": "FM346",
"7_day_A": {
"value": 0
},
"7_day_B": {
"value": 30
},
"7_day_C": {
"value": 23
},
"7_day_D": {
"value": 43
}
},
{
"place": "LH210",
"1_day_A": {
"value": 4
},
"1_day_B": {
"value": 45
},
"1_day_C": {
"value": 100
},
"1_day_D": {
"value": 9
}
}
]
}
i have tried this:
{
data:[.data |.[]|
{
place: (.place),
(.id[]):
{
value: (.values[])
}
}]
}
(in jqplay: https://jqplay.org/s/f4BBtN9gwmp)
and this:
{
data:[.data |.[]|
{
place: (.place),
test:
[{
(.id[]):
{
value: (.values[])
}
}]
}]
}
(in jqplay: https://jqplay.org/s/pKIvQe1CzgX)
but they arent grouped in the way i wanted and it gives each value to each id, not the corresponding one.
I have been trying for some time now, but im new to jq and have no idea how to transform it this way, thanks in advance for any answers.
You can use transpose here, which can play a key role in converting the arrays to key/value pairs
.data[] |= {place} +
([ .id, .values ] | transpose | map({(.[0]): { value: .[1] } }) | add)
The solution works by converting the array-of-arrays [.id, .values] by transposing them, i.e. converting
[["7_day_A","7_day_B","7_day_C","7_day_D"],[0,30,23,43]]
[["1_day_A","1_day_B","1_day_C","1_day_D"],[4,45,100,9]]
to
[["7_day_A",0],["7_day_B",30],["7_day_C",23],["7_day_D",43]]
[["1_day_A",4],["1_day_B",45],["1_day_C",100],["1_day_D",9]]
With the transformation done, we construct an object with key as the zeroth index element and value as an object comprising of the value of first index element, and combine the results together with add
Demo - jqplay
I have output brakeman scan results to a json file and want to flatten the structure.
Brakeman json output sample looks like this
{
"scan_info": {
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"end_time": "2021-11-01 14:45:13 +1100",
"brakeman_version": "5.1.2"
},
"warnings": [
{
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"file": "app/controllers/my/controller.rb",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to....",
"render_path": null,
"location": {
"type": "method",
"class": "My::Controller",
"method": "request_token"
},
"user_input": "my.base_url",
"confidence": "High"
},
{.....}
],
"ignored_warnings": [],
"errors": [],
"obsolete": ["xxx","yyyy"]
}
I want to flatten the output into an array of objects similar to this. I want most data from the warnings array except .location and .render_path.
[
{
"brakeman_version": "5.1.2",
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f0d2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to...",
"confidence": "High"
},
{
"brakeman_version": "5.1.2",
"warning_type": "SQL Injection",
"check_name": "SQL",
"message": "Possible SQL injection",
"warning_code": 35,
....
"confidence": "High"
}
]
Something like this gives a nested object for each item in the warnings array but I'd like to move them up a level
jq '{brakeman_version: .scan_info.brakeman_version, start_time: .scan_info.start_time, warnings: .warnings[]}' ./brakeman.json
Result
{
"brakeman_version": "5.1.2",
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": {
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "xxxx",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"confidence": "High"
},
{...}
}
Something like this flattens the .scan_info part similarly but still leaves me with an array for warnings which is what was there in the first place.
jq '{
rails_version: .scan_info.rails_version,
security_warnings: .scan_info.security_warnings,
start_time: .scan_info.start_time,
brakeman_version: .scan_info.brakeman_version,
warnings: [.warnings[] | {warning_type: .warning_type,
warning_code: .warning_code,
fingerprint: .fingerprint,
check_name: .check_name,
confidence: .confidence} ],
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
Results
{
"rails_version": "6.1.4.1",
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": [
{
"warning_type": "SQL Injection",
"warning_code": 0,
"fingerprint": "xx",
"check_name": "SQL",
"message": "Possible SQL injection",
"line": 178,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
"confidence": "Medium"
},
{
"warning_type": "Format Validation",
"warning_code": 30,
"fingerprint": "xxx",
"check_name": "ValidationRegex",
"line": 92,
"link": "https://brakemanscanner.org/docs/warning_types/format_validation/",
"code": null,
"user_input": null,
"confidence": "High"
},
]
}
Using jq how do I correctly flatten the structure to output as desired?
I've tried some suggestions found on SO like
jq '{
rails_version: .scan_info.rails_version,
brakeman_version: .scan_info.brakeman_version,
.warnings[] as $in | $in | del(.location, .render_path) as $in2 | $in2,
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
but have been going around in circles at this stage.
With assumptions based on your description provided, I guess you are looking to do this
.scan_info + ( .warnings[] | del(.location, .render_path) )
If you want the entire results packed into an array, enclose the above filter within square brackets([ ])
jqplay demo
I am trying to parse a json file using jq.
$cat ipres.txt
{
"start": {
"connected": [{
"socket": 5,
"local_host": "10.8.0.3",
"local_port": 36068,
"remote_host": "10.8.0.2",
"remote_port": 5201
}],
"version": "iperf 3.1.2",
"system_info": "Linux sid-Vostro-3700 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64",
"timestamp": {
"time": "Fri, 22 Apr 2016 13:08:10 GMT",
"timesecs": 1461330490
},
"connecting_to": {
"host": "10.8.0.2",
"port": 5201
},
"cookie": "sid-Vostro-3700.1461330486.892347.18",
"test_start": {
"protocol": "UDP",
"num_streams": 1,
"blksize": 8192,
"omit": 0,
"duration": 2,
"bytes": 0,
"blocks": 0,
"reverse": 0
}
},
"intervals": [{
"streams": [{
"socket": 5,
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"packets": 31,
"omitted": false
}],
"sum": {
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"packets": 31,
"omitted": false
}
}],
"end": {
"streams": [{
"udp": {
"socket": 5,
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"jitter_ms": 828.078,
"lost_packets": 0,
"packets": 24,
"lost_percent": 0,
"out_of_order": 0
}
}],
"sum": {
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"jitter_ms": 828.078,
"lost_packets": 0,
"packets": 24,
"lost_percent": 0
},
"cpu_utilization_percent": {
"host_total": 7.7914,
"host_user": 1.09745,
"host_system": 6.69392,
"remote_total": 0,
"remote_user": 0,
"remote_system": 0
}
},
"test": 0
}
But when i try to filter the end object, my command fails.
$cat ipres.txt | jq .end
error: syntax error, unexpected end, expecting $end
.end
^^^
1 compile error
If i change the end object to some other name for example end1, i can able to filter. What could be the problem?
It would seem that you are after:
jq '.end' ipres.txt
If you are still getting the same erroneous output - then you may be using an older / badly built version of jq.
You are evidently using an early version of jq -- perhaps jq 1.3, which is well out-of-date and which has some limitations with respect to the abbreviated syntax: .foo.
That is, you would have to write .["end"] in jq 1.3, since "end" is a keyword (as in: if ... then ... else ... end).
In jq 1.4 and jq 1.5, one can write .end
I have some json data that looks like:
{
"p": {
"d": {
"a" : {
"r": "foo",
"g": 1
},
"b": {
"r": "bar",
"g": 2
}
},
"c": {
"e": {
"r": "baz",
"g": 1
}
},
...
}
}
I want something like:
{
"d": [
"a",
"b"
],
"c": [
"e"
]
}
I can get the list of keys on the first level under "p" with jq '.p|keys', and the structure and keys on the second level with jq '.p|map(.|keys)', but I can't figure out how to combine it.
Use map_values instead of map to map the values of a JSON object while preserving the keys:
jq '.p | map_values(keys)'
On jq versions lower than 1.5, map_values is not defined: instead, you can use []|=:
jq '.p | . []|= keys'
In general
Top level keys:
curl -s https://crates.io/api/v1/crates/atty | jq '. |= keys'
[
"categories",
"crate",
"keywords",
"versions"
]
Two levels of keys:
curl -s https://crates.io/api/v1/crates/atty | jq '.| map_values(keys)'
{
"crate": [
"badges",
"categories",
"created_at",
"description",
"documentation",
"downloads",
"exact_match",
"homepage",
"id",
"keywords",
"links",
"max_version",
"name",
"newest_version",
"recent_downloads",
"repository",
"updated_at",
"versions"
],
"versions": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16
],
"keywords": [
0,
1,
2
],
"categories": []
}
Method versions
topLevelJsonKeys() {
curl -s $1 | jq '. |= keys'
# EXAMPLE:
# topLevelJsonKeys https://crates.io/api/v1/crates/atty
}
topLevelJsonKeys2() {
curl -s $1 | jq '.| map_values(keys)'
# EXAMPLE:
# topLevelJsonKeys2 https://crates.io/api/v1/crates/atty
}
Here is a solution which uses reduce and setpath
.p
| reduce keys[] as $k (
.
; setpath([$k]; .[$k] | keys)
)