Related
I'm trying to deduplicate all objects inside the array results that share the same key id, and merge their path arrays.
JSON input:
[
{
"type": "apple",
"results": [
{
"id": "apple1",
"name": "appleName1",
"path": "/some/path/a"
},
{
"id": "apple1",
"name": "appleName1",
"path": "/some/path/b"
},
{
"id": "apple2",
"name": "appleName2",
"path": "/some/path/c"
}
]
},
{
"type": "orange",
"results": [
{
"id": "orange1",
"name": "orangeName1",
"path": "/some/path/a"
},
{
"id": "orange1",
"name": "orangeName1",
"path": "/some/path/b"
},
{
"id": "orange2",
"name": "orangeName2",
"path": "/some/path/c"
}
]
}
]
Expected output:
[
{
"type": "apple",
"results": [
{
"id": "apple1",
"name": "appleName1",
"path": [
"/some/path/a",
"/some/path/b"
]
},
{
"id": "apple2",
"name": "appleName2",
"path": [
"/some/path/c"
]
}
]
},
{
"type": "orange",
"results": [
{
"id": "orange1",
"name": "orangeName1",
"path": [
"/some/path/a",
"/some/path/b"
]
},
{
"id": "orange2",
"name": "orangeName2",
"path": [
"/some/path/c"
]
}
]
}
]
I've managed to get an approximate solution using:
jq '[{type: .[].type, results: .[].results | group_by(.id) | map({id: .[0].id, name: .[0].name, path: (map(.path))})}]'
But my solution produces two additional elements that aren't supposed to be there.
I know there are some similar questions already answered but I didn't manage to get them to work with this example. Any help is appreciated!
You could group_by the .id field, then for each group take the first item and replace its .path field with a map on the .path fields of all group members:
jq 'map(.results |= (group_by(.id) | map(first + {path: map(.path)})))'
[
{
"type": "apple",
"results": [
{
"id": "apple1",
"name": "appleName1",
"path": [
"/some/path/a",
"/some/path/b"
]
},
{
"id": "apple2",
"name": "appleName2",
"path": [
"/some/path/c"
]
}
]
},
{
"type": "orange",
"results": [
{
"id": "orange1",
"name": "orangeName1",
"path": [
"/some/path/a",
"/some/path/b"
]
},
{
"id": "orange2",
"name": "orangeName2",
"path": [
"/some/path/c"
]
}
]
}
]
Demo
I want to convert a feature file to json so that I can pass it to a javascript function in an RMD file.
However, the toJSON function seems to flatten it and remove many of the fields and structures as below. How can I convert it and keep it in tact, as it does if I write to a file using sf::st_write?
url <- 'https://opendata.arcgis.com/api/v3/datasets/bf9d32b1aa9941af84e6c2bf0c54b1bb_0/downloads/data?format=geojson&spatialRefId=4326'
ukWardShapes <- sf::st_read(url) %>%
head(2)
# Looks OK when written out
sf::st_write(ukWardShapes, "wardShapes.geojson")
# Converting to json with toJSON seems drop other top level fields (type, name, crs) and list the objects within features object,
# but without type, and puts all fields in properties at the top level of object.
json_data <- jsonlite::toJSON(ukWardShapes)
# I want to do this as I need to pass it to javascript within an RMD like this
htmltools::tags$script(paste0("var ukWardShapes = ", json_data, ";"))
# Output from st_write - with all the objects and fields listed properly
{
"type": "FeatureCollection",
"name": "wardShapes",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { "OBJECTID": 1, "WD21CD": "E05000026", "WD21NM": "Abbey", "WD21NMW": " ", "BNG_E": 544433, "BNG_N": 184376, "LONG": 0.081276, "LAT": 51.53981, "SHAPE_Length": 0.071473941285613768, "SHAPE_Area": 0.00015225110241064838 }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 0.093628520000038, 51.53767283600007 ], [ 0.08163128800004, 51.539165094000055 ], [ 0.085507102000065, 51.537043160000053 ], [ 0.075954208000041, 51.533595714000057 ], [ 0.07333983500007, 51.537621201000036 ], [ 0.068771363000053, 51.536206993000064 ], [ 0.068303699000069, 51.544253423000043 ], [ 0.068361695000021, 51.544390390000046 ], [ 0.08006389600007, 51.544772356000067 ], [ 0.093628520000038, 51.53767283600007 ] ] ] ] } },
{ "type": "Feature", "properties": { "OBJECTID": 2, "WD21CD": "E05000027", "WD21NM": "Alibon", "WD21NMW": " ", "BNG_E": 549247, "BNG_N": 185196, "LONG": 0.150987, "LAT": 51.545921, "SHAPE_Length": 0.074652046036690151, "SHAPE_Area": 0.00017418950412786572 }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ 0.161601914000073, 51.543327754000074 ], [ 0.147931795000034, 51.541598449000048 ], [ 0.140256898000075, 51.54111542000004 ], [ 0.13420572800004, 51.540716652000071 ], [ 0.131925236000029, 51.543763455000033 ], [ 0.14633003900002, 51.546332889000041 ], [ 0.142816723000067, 51.550973604000035 ], [ 0.156378253000071, 51.551020271000027 ], [ 0.161601914000073, 51.543327754000074 ] ] ] ] } }
]
}
# Output from toJson which seems to have a lot of structure removed. Note, I'm not
# concerned about it being pretty and separated into lines
[{
"OBJECTID":1, "WD21CD":"E05000026", "WD21NM":"Abbey", "WD21NMW":" ", "BNG_E":544433, "BNG_N":184376, "LONG":0.0813, "LAT":51.5398, "SHAPE_Length":0.0715, "SHAPE_Area":0.0002, "geometry":{
"type":"MultiPolygon", "coordinates":[[[[0.0936, 51.5377], [0.0816, 51.5392], [0.0855, 51.537], [0.076, 51.5336], [0.0733, 51.5376], [0.0688, 51.5362], [0.0683, 51.5443], [0.0684, 51.5444], [0.0801, 51.5448], [0.0936, 51.5377]]]]
}
}, {
"OBJECTID":2, "WD21CD":"E05000027", "WD21NM":"Alibon", "WD21NMW":" ", "BNG_E":549247, "BNG_N":185196, "LONG":0.151, "LAT":51.5459, "SHAPE_Length":0.0747, "SHAPE_Area":0.0002, "geometry":{
"type":"MultiPolygon", "coordinates":[[[[0.1616, 51.5433], [0.1479, 51.5416], [0.1403, 51.5411], [0.1342, 51.5407], [0.1319, 51.5438], [0.1463, 51.5463], [0.1428, 51.551], [0.1564, 51.551], [0.1616, 51.5433]]]]
}
}]
As per #SymbolixAU's comment above, the answer is to use
geojsonsf::sf_geojson() instead of jsonlite::toJSON() as geojson is a specific structure of JSON for spatial data and it needs a specific parser for it.
So my line of code should be:
json_data <- geojsonsf::sf_geojson(ukWardShapes)
var k = [
{
"category": "Cars",
"products": [
{
"productName": "Aston Martin",
"quantity": 2,
"costPrice": 13500000,
"coverPhoto": "www.astonmartinphotoUrl.jpg"
},
{
"productName": "Mercedes",
"quantity": 1,
"costPrice": 220000,
"coverPhoto": "www.mercerdezphotoUrl.jpg"
}
]
},
{
"category": "Food",
"products": [
{
"productName": "Pizza",
"quantity": 50,
"costPrice": 30,
"coverPhoto": "www.pizzaphotoUrl.jpg"
},
{
"productName": "Pancake",
"quantity": 3,
"costPrice": 3,
"coverPhoto": "www.pancakephotoUrl.jpg"
}
]
}
];
Given the above code sample, I'm trying to add a new map to the nested list, product using the firebase.arrayUnion()
Map _map =
{
"productName": 'Tesla',
"quantity": 2,
"costPrice": 45000,
"coverPhoto": "www.teslaPhotoUrl.jpg"
};
I want to only add this map only where the key 'category' is equal to 'Cars'
_firebaseFirestoreRef.collection('data').doc(id).update({
"fieldPath to the the list where key category == Cars": FieldValue.arrayUnion([_map])
});
And I want final result to be like this on my firebase firestore database
[
{
"category": "Cars",
"products": [
{
"productName": "Aston Martin",
"quantity": 2,
"costPrice": 13500000,
"coverPhoto": "www.astonmartinphotoUrl.jpg"
},
{
"productName": "Mercedes",
"quantity": 1,
"costPrice": 220000,
"coverPhoto": "www.mercerdezphotoUrl.jpg"
},
{
"productName": 'Tesla',
"quantity": 2,
"costPrice": 45000,
"coverPhoto": "www.teslaPhotoUrl.jpg"
}
]
},
{
"category": "Food",
"products": [
{
"productName": "Pizza",
"quantity": 50,
"costPrice": 30,
"coverPhoto": "www.pizzaphotoUrl.jpg"
},
{
"productName": "Pancake",
"quantity": 3,
"costPrice": 3,
"coverPhoto": "www.pancakephotoUrl.jpg"
}
]
}
];
I know I need to use the FirebaseFirestore FieldPath and some query objects, but I don't know how to use it effectively to achieve this ...
Since your k is an array, you're trying to update an existing element in an array field, which is not possible in Firestore. You'll first need to read the document, get the k array from it, update it in your application code, and then write the resulting field back to the database.
This has been covered quite regularly before, so I recommend looking at some other questions about updating an item in an array.
You can also consider turning the top-level array into a map, using the category for the first-level field name:
products: {
"Cars": [{
"productName": "Aston Martin",
"quantity": 2,
"costPrice": 13500000,
"coverPhoto": "www.astonmartinphotoUrl.jpg"
}, ... ]
"Food": [{
...
}]
}
Now you can add an item to the Cars array with an array union on products.Cars.
As Dharmaraj commented, you could also consider putting the products into a subcollection. This will allow you to query the products separately, and allows you to read the parent document without reading all products.
I have this json format I'm making an API using ASP.NET.
{
"0": {
"order_id": 11748,
"complete_date": "2021-04-19 14:48:41",
"shipping_code": "aramex.aramex",
"awbs": [
{
"aramex_id": "1314",
"order_id": "11748",
"awb_number": "46572146154",
"reference_number": "11748",
"date_added": "2021-03-04 03:46:58"
}
],
"payment": {
"method": {
"name": "الدفع عند الاستلام",
"code": "cod"
},
"invoice": [
{
"code": "sub_total",
"value": "120.8700",
"value_string": "120.8700 SAR",
"title": "الاجمالي"
},
{
"code": "shipping",
"value": "0.0000",
"value_string": "0.0000 SAR",
"title": "ارمكس"
},
{
"code": "coupon",
"value": "-13.9000",
"value_string": "-13.9000 SAR",
"title": "قسيمة التخفيض(RMP425)"
},
{
"code": "cashon_delivery_fee",
"value": "5.0000",
"value_string": "5.0000 SAR",
"title": "رسوم الدفع عند الاستلام"
},
{
"code": "tax",
"value": "18.1300",
"value_string": "18.1300 SAR",
"title": " ضريبة القيمة المضافة (15%)"
},
{
"code": "total",
"value": "130.1000",
"value_string": "130.1000 SAR",
"title": "الاجمالي النهائي"
}
]
},
"product": [
{
"id": 69,
"name": "مخلط 4 أو دو بيرفيوم للجنسين - 100 مل",
"sku": "45678643230",
"weight": "0.50000000",
"quantity": 1,
"productDiscount": "",
"images": []
}
]
}
}
How can I reach order_id? I made an object let's say its name is obj1 I tried foreach obj1 and storing into a variable obj1.order_id;
It stored null in the variable. the {"0"} is the numbering of orders starts 0-1-2 etc.
You can deserialize that json to Dictionary<string,dynamic> without creating a new class as following:
var values = JsonConvert.DeserializeObject<Dictionary<string, dynamic>>(json);
var orderId = values["0"]["order_id"].ToString();
This will give you 11748 as a result.
I have a JSON document that looks as follows:
{
"Region": "Main",
"MarketLocations": [
{
"MarketName": "Central",
"MarketId": 1,
"SalesCategories": {
"Produce": [
{
"Type": "Apple",
"Name": "Granny Smith",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 24,
"Calories": 45,
"Price": 0.29
}
],
"BakedGoods": [
{
"DateMade": "2016-11-08T14:14:33.712Z",
"Name": "Apple Pie",
"Price": 14.25
}
],
"RestaurantItems": [
{
"Name": "Turkey Sandwich",
"Price": 4.85,
"PreparationTimeInMinutes": 20
}
],
"NonPerishable": [
{
"Name": "Honey Mustard",
"Type": "Condiments"
}
]
}
},
{
"MarketName": "Southern",
"MarketId": 2,
"SalesCategories": {
"Produce": [
{
"Type": "Apple",
"Name": "Granny Smith",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 24,
"Calories": 45,
"Price": 0.29
},
{
"Type": "Plums",
"Name": "Red Plums",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 12,
"Calories": 21,
"Price": 0.33
},
{
"Type": "Pears",
"Name": "Golden Nature",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 20,
"Calories": 40,
"Price": 0.45
}
],
"BakedGoods": [
{
"DateMade": "2016-11-08T14:14:33.712Z",
"Name": "Apple Pie",
"Price": 14.25
}
],
"RestaurantItems": [
{
"Name": "Turkey Sandwich",
"Price": 4.85,
"PreparationTimeInMinutes": 20
}
],
"NonPerishable": [
{
"Name": "Honey Mustard",
"Type": "Condiments"
}
]
}
},
{
"MarketName": "Western",
"MarketId": 3,
"SalesCategories": {
"Produce": [
{
"Type": "Plums",
"Name": "Red Plums",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 12,
"Calories": 21,
"Price": 0.33
},
{
"Type": "Pears",
"Name": "Golden Nature",
"DatePicked": "2016-11-08T14:14:33.712Z",
"ShelfLifeInDays": 20,
"Calories": 40,
"Price": 0.45
}
],
"BakedGoods": [
{
"DateMade": "2016-11-08T14:14:33.712Z",
"Name": "Plum Pie",
"Price": 18.25
}
],
"RestaurantItems": [
{
"Name": "Ham Sandwich",
"Price": 4.85,
"PreparationTimeInMinutes": 20
},
{
"Name": "Chicken Soup",
"Price": 2.25,
"PreparationTimeInMinutes": 5
}
],
"NonPerishable": [
{
"Name": "Mayo",
"Type": "Condiments"
},
{
"Name": "Syrup",
"Type": "Condiments"
},
{
"Name": "Ginger",
"Type": "Spices"
}
]
}
}
]
}
I have the following U-SQL, that processes this JSON file, running inside Visual Studio:
DECLARE #in string=#"/JsonDoc2.json";
DECLARE #out string=#"Output/JsonDoc2.csv";
#produce =
EXTRACT Name string,
DatePicked DateTime,
ShelfLifeInDays int,
Calories int,
Price decimal,
MarketId string,
MarketName string
FROM #in
USING new MultiLevelJsonExtractor("MarketLocations[*].SalesCategories.Produce[*]",
false,
"Name",
"DatePicked",
"ShelfLifeInDays",
"Calories",
"Price",
"MarketId",
"MarketName");
OUTPUT #produce
TO #out
USING Outputters.Csv(outputHeader : true);
This executes without error. The problem is that I am specifically specifying what sales category I want ('produce'). I'd like to change this query so that that all sales categories are included (produce, baked goods etc.) with the category name included. I've not been able to figure out a way to do this.
The JsonType method of the NewtonSoft JsonFunctions class, returns a MAP value which is a key-value pair. You can then reference the key to get the JSON property / object / array names, at least after a few other manipulations with CROSS APPLY and EXPLODE.
For your example, I got the following to work:
REFERENCE ASSEMBLY [Newtonsoft.Json];
REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];
USING Microsoft.Analytics.Samples.Formats.Json;
DECLARE #input string = #"/input/myinputfile.json";
DECLARE #output string = #"output/output.csv";
#json =
EXTRACT Region string,
MarketName string,
SalesCategories string // get the SalesCategories as JSON
FROM #input
USING new MultiLevelJsonExtractor("MarketLocations[*].SalesCategories",
true,
"Region",
"MarketName",
"SalesCategories"
);
// Convert the json string to tuple/MAP
#working =
SELECT Region,
MarketName,
JsonFunctions.JsonTuple(SalesCategories) AS x
FROM #json;
// Explode the tuple as key-value pair;
#working =
SELECT Region,
MarketName,
key,
value
FROM #working
CROSS APPLY
EXPLODE(x) AS y(key, value);
// Explode the value which is JSON
#working =
SELECT Region,
MarketName,
key,
JsonFunctions.JsonTuple(y) AS z
FROM #working
CROSS APPLY
EXPLODE(JsonFunctions.JsonTuple(value).Values) AS x(y);
// Prep the result, naming the items you want
#result =
SELECT Region,
MarketName,
key,
z["Type"] AS Type,
z["Name"] AS Name,
z["DatePicked"] AS DatePicked,
z["ShelfLifeInDays"] AS ShelfLifeInDays,
z["Calories"] AS Calories,
z["Price"] AS Price,
z["DateMade"] AS DateMade,
z["PreparationTimeInMinutes"] AS PreparationTimeInMinutes
FROM #working;
OUTPUT #result
TO #output
USING Outputters.Csv(quoting:false);
My results:
It feels like it could be simplified, but see how you get on. The samples for shredding JSON are in short supply but try here and here.