Related
'''
{
"category": "process",
"status": 1,
"stacktrace": [],
"api": "CreateThread",
"return_value": 176,
"arguments": {
"thread_identifier": 1228,
"function_address": "0x004021ce",
"flags": 0,
"parameter": "0x0012fef0",
"stack_size": 0
},
"time": 1647881189.051382,
"tid": 1836,
"flags": {}
}
'''
this is part of report of cuckoo,what's the meaning of the "status":1,sometimes it's "status":0.
I have output brakeman scan results to a json file and want to flatten the structure.
Brakeman json output sample looks like this
{
"scan_info": {
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"end_time": "2021-11-01 14:45:13 +1100",
"brakeman_version": "5.1.2"
},
"warnings": [
{
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"file": "app/controllers/my/controller.rb",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to....",
"render_path": null,
"location": {
"type": "method",
"class": "My::Controller",
"method": "request_token"
},
"user_input": "my.base_url",
"confidence": "High"
},
{.....}
],
"ignored_warnings": [],
"errors": [],
"obsolete": ["xxx","yyyy"]
}
I want to flatten the output into an array of objects similar to this. I want most data from the warnings array except .location and .render_path.
[
{
"brakeman_version": "5.1.2",
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "cae2f0d2cfd8",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"code": "redirect_to...",
"confidence": "High"
},
{
"brakeman_version": "5.1.2",
"warning_type": "SQL Injection",
"check_name": "SQL",
"message": "Possible SQL injection",
"warning_code": 35,
....
"confidence": "High"
}
]
Something like this gives a nested object for each item in the warnings array but I'd like to move them up a level
jq '{brakeman_version: .scan_info.brakeman_version, start_time: .scan_info.start_time, warnings: .warnings[]}' ./brakeman.json
Result
{
"brakeman_version": "5.1.2",
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": {
"warning_type": "Redirect",
"warning_code": 18,
"fingerprint": "xxxx",
"check_name": "Redirect",
"message": "Possible unprotected redirect",
"line": 24,
"link": "https://brakemanscanner.org/docs/warning_types/redirect/",
"confidence": "High"
},
{...}
}
Something like this flattens the .scan_info part similarly but still leaves me with an array for warnings which is what was there in the first place.
jq '{
rails_version: .scan_info.rails_version,
security_warnings: .scan_info.security_warnings,
start_time: .scan_info.start_time,
brakeman_version: .scan_info.brakeman_version,
warnings: [.warnings[] | {warning_type: .warning_type,
warning_code: .warning_code,
fingerprint: .fingerprint,
check_name: .check_name,
confidence: .confidence} ],
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
Results
{
"rails_version": "6.1.4.1",
"security_warnings": 9,
"start_time": "2021-11-01 14:44:58 +1100",
"warnings": [
{
"warning_type": "SQL Injection",
"warning_code": 0,
"fingerprint": "xx",
"check_name": "SQL",
"message": "Possible SQL injection",
"line": 178,
"link": "https://brakemanscanner.org/docs/warning_types/sql_injection/",
"confidence": "Medium"
},
{
"warning_type": "Format Validation",
"warning_code": 30,
"fingerprint": "xxx",
"check_name": "ValidationRegex",
"line": 92,
"link": "https://brakemanscanner.org/docs/warning_types/format_validation/",
"code": null,
"user_input": null,
"confidence": "High"
},
]
}
Using jq how do I correctly flatten the structure to output as desired?
I've tried some suggestions found on SO like
jq '{
rails_version: .scan_info.rails_version,
brakeman_version: .scan_info.brakeman_version,
.warnings[] as $in | $in | del(.location, .render_path) as $in2 | $in2,
obsolete: (.obsolete | join(",")),
}' ./brakeman.json
but have been going around in circles at this stage.
With assumptions based on your description provided, I guess you are looking to do this
.scan_info + ( .warnings[] | del(.location, .render_path) )
If you want the entire results packed into an array, enclose the above filter within square brackets([ ])
jqplay demo
I've been using Forms Recognizer for some days now and can't get it to recognize the keys in my forms.
I want to use it to extract the answers given by students in a test...here is an example.
I can't change the structure of the sheet students fill because it is a national exam and I don't have access to who organizes it.
So I trained a model as recommended on Microsoft documentation and used it to "read" the forms and it gets most of the answers, but it all comes as values of a key "Tokens"
{
"key": [
{
"text": "__Tokens__",
"boundingBox": [
0,
0,
0,
0,
0,
0,
0,
0
]
}
],
"value": [
{
"text": "01",
"boundingBox": [
110.1,
826.6,
125.6,
826.6,
125.6,
816.8,
110.1,
816.8
],
"confidence": 1
},
{
"text": "A",
"boundingBox": [
148.2,
834.4,
160.6,
834.4,
160.6,
816.8,
148.2,
816.8
],
"confidence": 1
},
{
"text": "26",
"boundingBox": [
229.4,
828.6,
246,
828.6,
246,
816.8,
229.4,
816.8
],
"confidence": 1
},
{
"text": "B",
"boundingBox": [
268.6,
834.4,
277.8,
834.4,
277.8,
816.8,
268.6,
816.8
],
"confidence": 1
}
Then I recreated the structure on excel but with : after the numbers and trained another model. I also printed some copies of it and filled in to test and Form Recognizer understood the numbers as keys.
{
"key": [
{
"text": "01:",
"boundingBox": [
270.4,
1625.4,
313,
1625.4,
313,
1600.5,
270.4,
1600.5
]
}
],
"value": [
{
"text": "A",
"boundingBox": [
350.7,
1620.9,
368.8,
1620.9,
368.8,
1587,
350.7,
1587
],
"confidence": 1
}
]
},
{
"key": [
{
"text": "26:",
"boundingBox": [
520.2,
1624.2,
552.8,
1624.2,
552.8,
1600.5,
520.2,
1600.5
]
}
],
"value": [
{
"text": "E",
"boundingBox": [
604.6,
1618.8,
625.8,
1618.8,
625.8,
1587,
604.6,
1587
],
"confidence": 1
}
]
}
Does anyone know some way to recognize the number fields as keys without the : ?
Form Recognizer will not consider the row numbers as keys unless specifically marked as keys, hence it currently does not discover them as keys.
I have this file with JSON formatted data, but need this into a dataframe. Ultimately I would like to plot the geolocations onto a map, but can't seem to get this data into a df first.
json_to_df <- function(file){
file <- lapply(file, function(x) {
x[sapply(x, is.null)] <- NA
unlist(x)
})
df <- do.call("rbind", file)
return(df)
}
But I get only this error:
Error in fromJSON(file) :
STRING_ELT() can only be applied to a 'character vector', not a 'list'
The file structure looks like this (this is only part of the data):
{
"results": [
{
"utc_offset": 7200000,
"venue": {
"country": "nl",
"localized_country_name": "Netherlands",
"city": "Bergen",
"address_1": "16 Notweg",
"name": "FitClub Bergen",
"lon": 4.699218,
"id": 24632049,
"lat": 52.673046,
"repinned": false
},
"headcount": 0,
"distance": 22.46796989440918,
"visibility": "public",
"waitlist_count": 0,
"created": 1467149834000,
"rating": {
"count": 0,
"average": 0
},
"maybe_rsvp_count": 0,
"description": "<p>Start your week off right with a Monday Morning Bootcamp!!! The fresh air and peaceful dunes provide the perfect setting for a total body workout. Whether you are a beginner with brand spankin' new health goals and in need of some direction, or training for a race or competition, we're the trainers for you!!! See you at 8:50 for sign-in!</p>",
"event_url": "https://www.meetup.com/FitClubBergen/events/234936736/",
"yes_rsvp_count": 3,
"duration": 3600000,
"name": "Free Bootcamp in the Bergen Dunes",
"id": "glzqvlyvnbgc",
"time": 1477292400000,
"updated": 1477297999000,
"group": {
"join_mode": "open",
"created": 1441658286000,
"name": "FitClub Bergen Free Bootcamp in the Dunes",
"group_lon": 4.710000038146973,
"id": 18908751,
"urlname": "FitClubBergen",
"group_lat": 52.66999816894531,
"who": "FitClubbers"
},
"status": "past"
},
{
"utc_offset": 7200000,
"venue": {
"country": "nl",
"localized_country_name": "Netherlands",
"city": "Bergen",
"address_1": "16 Notweg",
"name": "FitClub Bergen",
"lon": 4.699218,
"id": 24632049,
"lat": 52.673046,
"repinned": false
},
"headcount": 0,
"distance": 22.46796989440918,
"visibility": "public",
"waitlist_count": 0,
"created": 1467149834000,
"rating": {
"count": 0,
"average": 0
},
"maybe_rsvp_count": 0,
"description": "<p>Start your week off right with a Monday Morning Bootcamp!!! The fresh air and peaceful dunes provide the perfect setting for a total body workout. Whether you are a beginner with brand spankin' new health goals and in need of some direction, or training for a race or competition, we're the trainers for you!!! See you at 8:50 for sign-in!</p> <p>ALWAYS FREE</p> <p>FOR ALL LEVELS OF FITNESS</p> <p>BRING: water bottle and energy</p>",
"event_url": "https://www.meetup.com/FitClubBergen/events/234936737/",
"yes_rsvp_count": 3,
"name": "Monday Morning Bootcamp in the Bergen Dunes",
"id": "flzqvlyvnbgc",
"time": 1477292400000,
"updated": 1477303926000,
"group": {
"join_mode": "open",
"created": 1441658286000,
"name": "FitClub Bergen Free Bootcamp in the Dunes",
"group_lon": 4.710000038146973,
"id": 18908751,
"urlname": "FitClubBergen",
"group_lat": 52.66999816894531,
"who": "FitClubbers"
},
"status": "past"
},
{
"utc_offset": 7200000,
"venue": {
"country": "nl",
"localized_country_name": "Netherlands",
"city": "Amsterdam",
"phone": "020 4275777",
"address_1": "Dijksgracht 2",
"address_2": "1019 BS ",
"name": "Klimmuur Central",
"lon": 4.91284,
"id": 1143381,
"lat": 52.376626,
"repinned": false
},
"headcount": 0,
"distance": 1.0689502954483032,
"visibility": "public",
"waitlist_count": 0,
"created": 1477215767000,
"rating": {
"count": 0,
"average": 0
},
"maybe_rsvp_count": 0,
"description": "<p>Climbing Right After Work: RAW.<br/>Quiet hall, pretty much every rope available; no rope chasing necessary. And.. still some time left to do other things later that evening. Take you gear and an extra sandwich to work and join me afterwards pulling some plastic.<br/>Some notes:<br/>- This events starts #17:00. If you can't make it that early, please comment the time you can.<br/>- Please fill in your belaying skills in your profile. If you've never climbed before or don't have belaying skills: follow an introduction course a the gym first! Safety above all!</p>",
"event_url": "https://www.meetup.com/The-Amsterdam-indoor-rockclimbing/events/235054729/",
"yes_rsvp_count": 3,
"name": "Monday's RAW Climb",
"id": "235054729",
"time": 1477321200000,
"updated": 1477334279000,
"group": {
"join_mode": "approval",
"created": 1358348565000,
"name": "The Amsterdam indoor rockclimbing",
"group_lon": 4.889999866485596,
"id": 6689952,
"urlname": "The-Amsterdam-indoor-rockclimbing",
"group_lat": 52.369998931884766,
"who": "Climbers"
},
"status": "past"
},
{
"utc_offset": 7200000,
"venue": {
"country": "nl",
"localized_country_name": "Netherlands",
"city": "Amstelveen",
"address_1": "Langs de Akker 3",
"name": "Emergohal",
"lon": 4.87967,
"id": 23816542,
"lat": 52.290199,
"repinned": false
},
"rsvp_limit": 12,
"headcount": 0,
"distance": 5.541957378387451,
"visibility": "public",
"waitlist_count": 0,
"created": 1474452073000,
"fee": {
"amount": 5.5,
"accepts": "cash",
"description": "per person",
"currency": "EUR",
"label": "price",
"required": "0"
},
"rating": {
"count": 0,
"average": 0
},
"maybe_rsvp_count": 0,
"description": "<p>We will play the Whole Season indoor soccer on Mondays from 18:00 - 19:00 starting 5 September until May 2017 in the Emergohal Amstelveen.</p> <p>Preferred payment is with Paypal EUR 5.50 (in advance)<br/>If this is not possible you may pay cash but then I will ask EUR 6,-<br/>(Please have the exact cash with you)</p> <p>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</p> <p>A couple of Unisys (ex)colleagues and football lovers are playing every Monday in the Emergohal Amstelveen at 6PM on a reasonable good level. We are looking for a compact group of players who are willing/able to play (almost) every Monday playing 5v5 (or 6v6).<br/>We are playing with the FIFA Futsal rules in mind:<br/>http://www.fifa.com/mm/document/footballdevelopment/refereeing/51/44/50/lawsofthegamefutsal2014_15_eneu_neutral.pdf</p> <p>The Emergohal has dressing rooms and a nice bar for after the game.</p> <p>Hope to see you on Mondays</p> <p>Cheers Jeroen</p> <p>For questions you may call me on[masked], send a text message (SMS) or leave a message on this meetup group.</p>",
"event_url": "https://www.meetup.com/Futsal_Emergohal_Monday_18-00/events/234290812/",
"yes_rsvp_count": 11,
"duration": 4500000,
"name": "Futsal",
"id": "234290812",
"time": 1477323900000,
"updated": 1477330559000,
"group": {
"join_mode": "approval",
"created": 1474445066000,
"name": "Futsal_Emergohal_Monday_18.00",
"group_lon": 4.860000133514404,
"id": 20450096,
"urlname": "Futsal_Emergohal_Monday_18-00",
"group_lat": 52.31999969482422,
"who": "Players"
},
"status": "past"
}],
"meta": {
"next": "https://api.meetup.com/2/open_events?and_text=False&offset=1&city=Amsterdam&sign=True&format=json&lon=4.88999986649&limited_events=False&photo-host=public&page=20&time=-24m%2C&radius=25.0&lat=52.3699989319&status=past&desc=False",
"method": "OpenEvents",
"total_count": 643,
"link": "https://api.meetup.com/2/open_events",
"count": 20,
"description": "Searches for recent and upcoming public events hosted by Meetup groups. Its search window is the past one month through the next three months, and is subject to change. Open Events is optimized to search for current events by location, category, topic, or text, and only lists Meetups that have **3 or more RSVPs**. The number or results returned with each request is not guaranteed to be the same as the page size due to secondary filtering. If you're looking for a particular event or events within a particular group, use the standard [Events](/meetup_api/docs/2/events/) method.",
"lon": ,
"title": "Meetup Open Events v2",
"url": "",
"signed_url": "{signed_url}",
"id": "",
"updated": 1479988687055,
"lat":
}
}
So I was wondering how I would put this in a dataframe or csv even to be able to extract geolocations later?
There is no need to write a parser yourself, there are a number of packages that can read JSON formatted data. The one I use, and #hrbrmstr linked, is jsonlite. This package provides a fromJSON function which can parse JSON into a data.frame:
fromJSON('file.json', flatten = TRUE)
note that the flatten argument here ensures the json is flattended into a nice data.frame.
I am trying to parse a json file using jq.
$cat ipres.txt
{
"start": {
"connected": [{
"socket": 5,
"local_host": "10.8.0.3",
"local_port": 36068,
"remote_host": "10.8.0.2",
"remote_port": 5201
}],
"version": "iperf 3.1.2",
"system_info": "Linux sid-Vostro-3700 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64",
"timestamp": {
"time": "Fri, 22 Apr 2016 13:08:10 GMT",
"timesecs": 1461330490
},
"connecting_to": {
"host": "10.8.0.2",
"port": 5201
},
"cookie": "sid-Vostro-3700.1461330486.892347.18",
"test_start": {
"protocol": "UDP",
"num_streams": 1,
"blksize": 8192,
"omit": 0,
"duration": 2,
"bytes": 0,
"blocks": 0,
"reverse": 0
}
},
"intervals": [{
"streams": [{
"socket": 5,
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"packets": 31,
"omitted": false
}],
"sum": {
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"packets": 31,
"omitted": false
}
}],
"end": {
"streams": [{
"udp": {
"socket": 5,
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"jitter_ms": 828.078,
"lost_packets": 0,
"packets": 24,
"lost_percent": 0,
"out_of_order": 0
}
}],
"sum": {
"start": 0,
"end": 2.0002,
"seconds": 2.0002,
"bytes": 253952,
"bits_per_second": 1.01571e+06,
"jitter_ms": 828.078,
"lost_packets": 0,
"packets": 24,
"lost_percent": 0
},
"cpu_utilization_percent": {
"host_total": 7.7914,
"host_user": 1.09745,
"host_system": 6.69392,
"remote_total": 0,
"remote_user": 0,
"remote_system": 0
}
},
"test": 0
}
But when i try to filter the end object, my command fails.
$cat ipres.txt | jq .end
error: syntax error, unexpected end, expecting $end
.end
^^^
1 compile error
If i change the end object to some other name for example end1, i can able to filter. What could be the problem?
It would seem that you are after:
jq '.end' ipres.txt
If you are still getting the same erroneous output - then you may be using an older / badly built version of jq.
You are evidently using an early version of jq -- perhaps jq 1.3, which is well out-of-date and which has some limitations with respect to the abbreviated syntax: .foo.
That is, you would have to write .["end"] in jq 1.3, since "end" is a keyword (as in: if ... then ... else ... end).
In jq 1.4 and jq 1.5, one can write .end