have a very large JSON data like below
{
"10.10.10.1": {
"asset_id": 1,
"referencekey": "ASSET-00001",
"hostname": "testDev01",
"fqdn": "ip-10-10.10.1.ap-northeast-2.compute.internal",
"network_zone": [
"DEV",
"Dev"
],
"service": {
"name": "TEST_SVC",
"account": "AWS_TEST",
"billing": "Testpay"
},
"aws": {
"tags": {
"Name": "testDev01",
"Service": "TEST_SVC",
"Usecase": "Dev",
"billing": "Testpay",
"OsVersion": "20.04"
},
"instance_type": "t3.micro",
"ami_imageid": "ami-e000001",
"state": "running"
}
},
"10.10.10.2": {
"asset_id": 3,
"referencekey": "ASSET-47728",
"hostname": "Infra_Live01",
"fqdn": "ip-10-10-10-2.ap-northeast-2.compute.internal",
"network_zone": [
"PROD",
"Live"
],
"service": {
"name": "Infra",
"account": "AWS_TEST",
"billing": "infra"
},
"aws": {
"tags": {
"Name": "Infra_Live01",
"Service": "Infra",
"Usecase": "Live",
"billing": "infra",
"OsVersion": "16.04"
},
"instance_type": "r5.large",
"ami_imageid": "ami-e592398b",
"state": "running"
}
}
}
Can I use JQ to make the conversion like below?
Or is there an easier way to solve it?
Thank you
Expected result
_key,asset_id,referencekey,hostname,fqdn,network_zone/0,network_zone/1,service/name,service/account,service/billing,aws/tags/Name,aws/tags/Service,aws/tags/Usecase,aws/tags/billing,aws/tags/OsVersion,aws/instance_type,aws/ami_imageid,aws/state
10.10.10.1,1,ASSET-00001,testDev01,ip-10-10.10.1.ap-northeast-2.compute.internal,DEV,Dev,TEST_SVC,AWS_TEST,Testpay,testDev01,TEST_SVC,Dev,Testpay,20.04,t3.micro,ami-e000001,running
10.10.10.2,3,ASSET-47728,Infra_Live01,ip-10-10-10-2.ap-northeast-2.compute.internal,PROD,Live,Infra,AWS_TEST,infra,Infra_Live01,Infra,Live,infra,16.04,r5.large,ami-e592398b,running
jq let's you do the conversion to CSV easily. The following code produces the desired output:
jq -r 'to_entries
| map([.key,
.value.asset_id, .value.referencekey, .value.hostname, .value.fqdn,
.value.network_zone[0], .value.network_zone[1],
.value.service.name, .value.service.account, .value.service.billing,
.value.aws.tags.Name, .value.aws.tags.Service, .value.aws.tags.Usecase, .value.aws.tags.billing, .value.aws.tags.OsVersion,
.value.aws.instance_type, .value.aws.ami_imageid, .value.aws.state])
| ["_key","asset_id","referencekey","hostname","fqdn","network_zone/0","network_zone/1","service/name","service/account","service/billing","aws/tags/Name","aws/tags/Service","aws/tags/Usecase","aws/tags/billing","aws/tags/OsVersion","aws/instance_type","aws/ami_imageid","aws/state"]
, .[]
| #csv' "$INPUT"
Remarks
If some nodes in the input JSON are missing, the code does not break but fills in empty values in the CSV file.
If more than two network zones are given, only the first two are covered in the CSV file
Related
From Wikidata, I get the following json:
# Sparql query
query=$(cat ./myquery.sparql)
response=$(curl -G --data-urlencode query="${query}" https://wikidata.org/sparql?format=json)
echo "${response}" | jq '.results.bindings'
[
{
"language": {
"type": "uri",
"value": "https://lingualibre.org/entity/Q100"
},
"wikidata": {
"type": "literal",
"value": "Q36157"
},
"code": {
"type": "literal",
"value": "lub"
}
},
{
"language": {
"type": "uri",
"value": "https://lingualibre.org/entity/Q101"
},
"wikidata": {
"type": "literal",
"value": "Q36284"
},
"code": {
"type": "literal",
"value": "srr"
}
}
]
I would like to have the keys directly paired with their values, such as :
[
{
"language": "https://lingualibre.org/entity/Q100",
"wikidata": "Q36157",
"iso": "lub"
},
{
"language": "https://lingualibre.org/entity/Q101",
"wikidata": "Q36284",
"iso": "srr"
}
]
I currently have a non-resilient code, which will break whenever the key names change :
jq 'map({"language":.language.value,"wikidata":.wikidata.value,"iso":.code.value})'
How to pair the keys with their values in a resilient way (not naming the keys) ?
I want to "prune" the child objects so to only keep the value.
You could use map_values which works like the outer map but for objects, i.e. it retains the object structure, including the field names:
jq 'map(map_values(.value))'
[
{
"language": "https://lingualibre.org/entity/Q100",
"wikidata": "Q36157",
"code": "lub"
},
{
"language": "https://lingualibre.org/entity/Q101",
"wikidata": "Q36284",
"code": "srr"
}
]
Note that this solution lacks the name conversion from code to iso.
I have the following structure:
{
"Subnets": [
{
"SubnetId": "foo1",
"Id": "bar1",
"Tags": [
{
"Key": "Name",
"Value": "foo"
},
{
"Key": "Status",
"Value": "dev"
}
]
},
{
"SubnetId": "foo2",
"Id": "bar2",
"Tags": [
{
"Key": "Name",
"Value": "foo"
},
{
"Key": "Status",
"Value": "dev"
}
]
}
]
}
I can extract multiple keys at the "top level" like so:
cat subnets.json| jq '.Subnets[] | "\(.Id) \(.SubnetId)"'
Anyone know how I can also display one of the tags by key name, let's say I also want the Status tag displayed on the same line as the Id and SubnetId.
Thx for any help,
Is this what you are looking for?
jq '.Subnets[] | "\(.Id) \(.SubnetId) \(.Tags | from_entries | .Status)"' subnets.json
I am trying to automate my project create process and would like as part of it to create a new jupyter notebook and populate it with some cells and content that I usually have in every notebook (i.e., imports, titles, etc.)
Is it possible to do this via python?
You can do it using nbformat. Below an example taken from Creating an IPython Notebook programatically:
import nbformat as nbf
nb = nbf.v4.new_notebook()
text = """\
# My first automatic Jupyter Notebook
This is an auto-generated notebook."""
code = """\
%pylab inline
hist(normal(size=2000), bins=50);"""
nb['cells'] = [nbf.v4.new_markdown_cell(text),
nbf.v4.new_code_cell(code)]
fname = 'test.ipynb'
with open(fname, 'w') as f:
nbf.write(nb, f)
This is absolutely possible. Notebooks are just json files. This
notebook for example is just:
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Header 1"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2016-09-16T16:28:53.333738",
"start_time": "2016-09-16T16:28:53.330843"
},
"collapsed": false
},
"outputs": [],
"source": [
"def foo(bar):\n",
" # Standard functions I want to define.\n",
" pass"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Header 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.10"
},
"toc": {
"toc_cell": false,
"toc_number_sections": true,
"toc_threshold": 6,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 0
}
While messy it's just a list of cell objects. I would probably create my template in an actual notebook and save it rather than trying to generate the initial template by hand. If you want to add titles or other variables programmatically, you could always copy the raw notebook text in the *.ipynb file into a python file and insert values using string formatting.
For an input below:
[{
"commit": {
"author": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"committer": {
"name": "Stephen Dolan",
"email": "mu#netsoc.tcd.ie",
"date": "2013-06-22T16:30:59Z"
},
"message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
"url":"https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f"
},
{
...
}
}]
How can JQ generate a delimited string from different objects as shown below?
"Stephen Dolan", "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "2013-06-22T16:30:59Z"
Collect the fields you want in an array and use #csv to convert to a CSV row. Make sure you get the raw output.
jq -r '.[] | [ .commit.author.name, .commit.url, .commit.author.date ] | #csv' input.json
$file = dirname(__FILE__) . '/myFile.xml';
$xml = simplexml_load_file("$file");
$json = json_encode($xml);
Here's my json code:
{
"Commands1": [{
"Name": "XLS1",
"Activated": "true",
"values": [{
"Name": "Cmd",
"default": "false"
}, {
"Name": "Ls",
"default": "false"
}, {
"Name": "rmdir",
"default": "false"
}],
"Commands2": [{
"Name": "SKA1",
"Activated": "true",
"values": [{
"Name": "Cp",
"default": "false"
}, {
"Name": "Tcpdump",
"default": "false"
}, {
"Name": "rmdir",
"default": "false"
}]
}]
}]
}
In my controller I open a XML file, and I'd like to know if it's possible to show its content in a table ? If so can someone give me a simple example ?
The result should be like
Name | Values
xls1 | cmd, ls,rmdir
SKA1 | cp, tcpdump,rmdir
The value of the JSON file decode it into a Associative Array and pass it to the view. From there act like you would acct with a normal AssocArray.
In controller:
$assoc_array_results = json_decode($json_file_result);
return $this->render('view.html.twig', array('results' => $assoc_array_results));
Well, it certainly is possible. You need to first decode it into an array.
$s = json_decode($json,true);
Then, comes the foreach() loop. You will have to consider each and every key-value pair and then iterate that loop to print your result.
PS: Apparently, I found a mistake in your Json String. Your Commands2 is becoming a subset of Commands1. Here is your correct Json.
{
"Commands1": [
{
"Name": "XLS1",
"Activated": "true",
"values": [
{
"Name": "Cmd",
"default": "false"
},
{
"Name": "Ls",
"default": "false"
},
{
"Name": "rmdir",
"default": "false"
}
]
}
],
"Commands2": [
{
"Name": "SKA1",
"Activated": "true",
"values": [
{
"Name": "Cp",
"default": "false"
},
{
"Name": "Tcpdump",
"default": "false"
},
{
"Name": "rmdir",
"default": "false"
}
]
}
]
}
And, Here is a reference, You can check How will your table look like when from JSON.http://json2table.com/
Now, Regarding your table, No one is gonna write the code for you on stack. So, I suggest go Baby steps if you wanna learn it. Here is a reference for you http://www.w3schools.com/php/php_looping_for.asp