The answer to this question can be found in this link provided by #bdkosher:
kousenit.wordpress.com/2014/04/16/the-closure-of-no-return
I am attempting to write a recursive function that compares to complex objects (JSON objects) and reports on the differences of each. The code outputs correctly but the failure is not reported as subsequent comparisons take precedent over an earlier failure.
I'm new to groovy so I'm sure there are groovier ways to write it.
The code:
public boolean diffObjects(Object left, Object right)
{
if (left == null || right == null) {
println "Object comparison failure: One or both object null."
return false
}
if (left.getClass() != right.getClass()) {
println "Object comparison failure: Mismatch object classes."
return false
}
if (isArray(left)) {
if (left.size() != right.size()) {
println "Array comparison failure: Object size mismatch."
println "Left has " + left.size() + " items. Right has " + right.size() + " items."
println "Left Object:"
println left
println "Right Object:"
println right
return false
}
for(int i=0; i < left.size(); i++) {
// May detect matching items here if sort of objects is problem
diffObjects(left[i], right[i])
}
} else if (isLazyMap(left)) {
String[] leftKeys = left.keySet()
String[] rightKeys = right.keySet()
if (leftKeys != rightKeys) {
println "Map comparison failure: Left keys do not match right keys."
println "Left " + leftKeys.toString()
println "Right " + rightKeys.toString()
return false
}
leftKeys.each {
if (isArray(left[it])) {
// May detect matching items here if sort of objects is problem
diffObjects( left[it], right[it])
} else {
if (isValue(left[it])) {
if (left[it].toString() != right[it].toString()) {
println "String comparison failure: Left " + it + " value does not match right value."
println "Left " + left[it]
println "Right " + right[it]
return false
}
}
}
}
}
}
Sample data:
[
{
"productId": "141810",
"sizes": [
{
"sku": "11926",
"size": "L",
"gtin": "008206",
"localizedSize": "L",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "M",
"gtin": "0082065234",
"localizedSize": "M",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "S",
"gtin": "0082065234",
"localizedSize": "S",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": false
},
{
"sku": "1192",
"size": "XL",
"gtin": "0082065234",
"localizedSize": "XL",
"skuCountryItems": [
{
"country": "CN",
"commodityCode": null,
"vat": 17
}
],
"available": true
}
]
}
]
Any help would be appreciated. Thanks in advance.
Here's an alternative method:
def a = ['a', 'b', 'c']
def b = ['a', 'z', 'c']
def c = [
[sku: 'abc', size: 'L'],
[sku: '123', size: 'S']
]
def d = [
[sku: 'abc', size: 'L'],
[sku: 'xyz', size: 'S']
]
use(DiffMixin) {
assert a.diff(b) == [
[
[self:'b', other:'z'],
[self:[name:'bytes', value:[98]], other:[name:'bytes', value:[122]]]
]
]
println c.diff(d)
}
class DiffMixin {
static List diff(Object self, Object other) {
def diffs = []
if(self != other) {
diffs << [
'self': self.toString(),
'other': other.toString()
]
}
self.properties.inject(diffs) {list, entry ->
def key = "$entry.key"
if(self."$key" != other."$key") {
list << [
'self': [name: key, value: self."$key"],
'other': [name: key, value: other."$key"]
]
}
return list
}
}
static List diff(Collection self, Collection other) {
([self] + [other])
.transpose()
.inject([]) {list, pair ->
def diffs = diff(pair[0], pair[1])
if(diffs) list << diffs
return list
}
}
}
I couldn't get a working assert expression for the c-d comparison. The output looks like this:
[[[self:[sku:123, size:S], other:[sku:xyz, size:S]]]]
Explanation
The DiffMixin mixin implements comparison methods for the classes you want to compare. I only implemented two, Object and Collection, but it should give you the idea. The Object comparison compares toString() output and property values. The Collection comparison basically delegates most of the work, while collecting the diffs. These methods return a list describing the differences.
Think of this as a demo :)
Related
I have a deserialized object that I want to dynamically loop through to return the related results. The response package looks like so:
{"RatingResponse":
{"Success":"true",
"Message":"",
"QuoteID":"57451",
"LoadNum":"57451",
"Rates":
{"Rate":
[
{"SCAC":"test1",
"CarrierName":"TEST1",
"TransitTime":"1",
"ServiceLevel":"D",
"TotalCost":"1,031.82",
"ThirdPartyCharge":"1,031.82",
"Accessorials":
{"Accessorial":
[
{"Code":"400",
"Cost":"1,655.55",
"Description":"Freight"
},
{"Code":"DSC",
"Cost":"-952.77",
"Description":"Discount"
},
{"Code":"FUE",
"Cost":"329.04",
"Description":"Fuel Surcharge"
}
]
},
"QuoteNumber":""
},
{"SCAC":"test2",
"CarrierName":"TEST2",
"TransitTime":"1",
"ServiceLevel":"D",
"TotalCost":"1,031.82",
"ThirdPartyCharge":"1,031.82",
"Accessorials":
{"Accessorial":
[
{"Code":"400",
"Cost":"1,655.55",
"Description":"Freight"
},
{"Code":"DSC",
"Cost":"-952.77",
"Description":"Discount"
},
{"Code":"FUE",
"Cost":"329.04",
"Description":"Fuel Surcharge"
}
]
},
"QuoteNumber":""
}
]
},
"AverageTotalCost":"1,031.82"
}
}
I have parsed the response data so that there is less information to work with, especially since I only need the Accessorial Costs. The parsed response looks like
[
{
"SCAC": "test1",
"CarrierName": "TEST1",
"TransitTime": "1",
"ServiceLevel": "D",
"TotalCost": "1,031.82",
"ThirdPartyCharge": "1,031.82",
"Accessorials": {
"Accessorial": [
{
"Code": "400",
"Cost": "1,655.55",
"Description": "Freight"
},
{
"Code": "DSC",
"Cost": "-952.77",
"Description": "Discount"
},
{
"Code": "FUE",
"Cost": "329.04",
"Description": "Fuel Surcharge"
}
]
},
"QuoteNumber": ""
},
{
"SCAC": "test2",
"CarrierName": "TEST2",
"TransitTime": "1",
"ServiceLevel": "D",
"TotalCost": "1,031.82",
"ThirdPartyCharge": "1,031.82",
"Accessorials": {
"Accessorial": [
{
"Code": "400",
"Cost": "1,655.55",
"Description": "Freight"
},
{
"Code": "DSC",
"Cost": "-952.77",
"Description": "Discount"
},
{
"Code": "FUE",
"Cost": "329.04",
"Description": "Fuel Surcharge"
}
]
},
"QuoteNumber": ""
}
]
The problem I am facing is that I will never know how many Rate items will come back in the response data, nor will I know the exact amount of Accessorial Costs. I'm hoping to capture the Rate child node counts and the Accessorial child node counts per Rate. Here's what I have so far.
Root rootObject = Newtonsoft.Json.JsonConvert.DeserializeObject<Root>(responseFromServer);
//rate stores the parsed response data
JArray rate = (JArray)JObject.Parse(responseFromServer)["RatingResponse"]["Rates"]["Rate"];
var rate2 = rate.ToString();
//this for loop works as expected. it grabs the number of Rate nodes (in this example, 2)
for (int i = 0; i < rate.Count(); i++)
{
dynamic test2 = rate[i];
//this is where I'm struggling
dynamic em = (JArray)JObject.Parse(test2)["Accessorials"]["Accessorial"].Count();
for (int j = 0; j < em; j++)
{
string test3 = test2.Accessorials.Accessorial[j].Cost;
System.IO.File.AppendAllText(logPath, Environment.NewLine + test3 + Environment.NewLine);
}
}
I apologize in advance for the bad formatting and odd variable names - I'm obviously still testing the functionality, so I've been using random variables.
Where I'm struggling (as notated above) is getting to the Accessorial node to count how many items are in its array. I was thinking I could parse the first array (starting with SCAC data) and extend down to the Accessorial node, but I'm not having any luck.
Any help is GREATLY appreciated, especially since I am new to this type of code and have spent the majority of the day trying to resolve this.
you can try this
var rates = (JArray)JObject.Parse(json)["RatingResponse"]["Rates"]["Rate"];
var costs = rates.Select(r => new
{
CarrierName = r["CarrierName"],
Costs = ((JArray)((JObject)r["Accessorials"])["Accessorial"])
.Where(r => (string)r["Description"] != "Discount")
.Select(r => (double)r["Cost"]).Sum()
}).ToList();
result
[
{
"CarrierName": "TEST1",
"Costs": 1984.59
},
{
"CarrierName": "TEST2",
"Costs": 1984.59
}
]
I have the following object :
{
"db_credentials": {
"database": "greengrass",
"host": "localhost",
"password": "yZqXJzXHLUsLlPm",
"port": 7086,
"username": "greengrass"
},
"default_interval": 90000,
"fields_selected": [
{
"measurement": "ABPLCGD-GD_AB1AirFlowCalc",
"aggregation": "last",
"step": "10m",
"timeserie_physical": "null",
"timeserie_type": "null",
"timeserie_interpolation": "null",
"timeserie_unit": "null",
"timeserie_step": "10"
},
{
"measurement": "ABPLCGD-GD_AB1InletAirTempAct",
"aggregation": "last",
"step": "10m",
"timeserie_physical": "null",
"timeserie_type": "null",
"timeserie_interpolation": "null",
"timeserie_unit": "null",
"timeserie_step": "10"
}
]
}
and i wish to transform it into :
{
"db_credentials": {
"database": "greengrass",
"host": "localhost",
"password": "yZqXJzXHLUsLlPm",
"port": 7086,
"username": "greengrass"
},
"default_interval": 90000,
"fields_selected": [
{
"name": "ABPLCGD-GD_AB1AirFlowCalc",
"aggregation": {
"step": "10m",
"function": "last"
}
},
{
"name": "ABPLCGD-GD_AB1InletAirTempAct",
"aggregation": {
"step": "10m",
"function": "last"
}
}
]
}
i have tried multiple solution but this is the maximum where i can get :
jq ' .fields_selected = .fields_selected | map({name : .measurement , aggregation : {step: .step , function: .aggregation}})' config-it-client.json
but i got always this error that Cannot index number with string "measurement" and i cant figure out what i'm doing wrong
I'd use the following:
.fields_selected[] |= { name: .measurement, aggregation: { function: .aggregation, step } }
To get it working, you were just missing parens.
.fields_selected = .fields_selected | ...
means
( .fields_selected = .fields_selected ) | ...
It should be
.fields_selected = ( .fields_selected | ... )
This gives us
.fields_selected = (
.fields_selected |
map({
name: .measurement,
aggregation: {
function: .aggregation,
step: .step
}
})
)
Demo on jqplay
But we can improve this. foo = ( foo | ... ) can generally be written as foo |= ( ... ).
.fields_selected |= map({
name: .measurement,
aggregation: {
function: .aggregation,
step: .step
}
})
We could modify the objects in the fields array instead of the array itself.
.fields_selected[] |= {
name: .measurement,
aggregation: {
function: .aggregation,
step: .step
}
}
Finally, { foo: .foo, ... } can be shortened to { foo, ... }.
.fields_selected[] |= {
name: .measurement,
aggregation: {
function: .aggregation,
step
}
}
As a sh one-liner:
jq '.fields_selected[] |= { name: .measurement, aggregation: { function: .aggregation, step } }'
Demo on jqplay
I was able to figure out what i was doing wrong:
jq '.fields_selected = (.fields_selected | map({name : .measurement , aggregation : {step: .step , function: .aggregation}}))' config-it-client.json
We are trying to do DynamoDB migration from prod account to stage account.
In the source account, we are making use of "Export" feature of DDB to put the compressed .json.gz files into destination S3 bucket.
We have written a glue script which will read the exported .json.gz files and writes it to DDB table.
We are making the code generic, so we should be able to migrate any DDB table from prod to stage account.
As part of that process, while testing we are facing issues when we are trying to write a NUMBER SET data to target DDB table.
Following is the sample snippet which is raising ValidationException when trying to insert into DDB
from decimal import Decimal
def number_set(datavalue):
# datavalue will be ['0', '1']
set_of_values = set()
for value in datavalue:
set_of_values.add(Decimal(value))
return set_of_values
When running the code, we are getting following ValidationException
An error occurred while calling o82.pyWriteDynamicFrame. Supplied AttributeValue is empty, must contain exactly one of the supported datatypes (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: UKEU70T0BLIKN0K2OL4RU56TGVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
However, if instead of Decimal(value) if we use int(value) then no ValidationException is being thrown and the job succeeds.
I feel that write_dynamic_frame_from_options will try to infer schema based on the values the element contains, if the element has "int" values then the datatype would be "NS", but if the element contains all "Decimal type" values, then it is not able to infer the datatype.
The glue job we have written is
dyf = glue_context.create_dynamic_frame_from_options(
connection_type="s3",
connection_options={
"paths": [file_path]
},
format="json",
transformation_ctx = "dyf",
recurse = True,
)
def number_set(datavalue):
list_of_values = []
for value in datavalue:
list_of_values.append(Decimal(value))
print("list of values ")
print(list_of_values)
return set(list_of_values)
def parse_list(datavalue):
list_of_values = []
for object in datavalue:
list_of_values.append(generic_conversion(object))
return list_of_values
def generic_conversion(value_dict):
for datatype,datavalue in value_dict.items():
if datatype == 'N':
value = Decimal(datavalue)
elif datatype == 'S':
value = datavalue
elif datatype == 'NS':
value = number_set(datavalue)
elif datatype == 'BOOL':
value = datavalue
elif datatype == 'M':
value = construct_map(datavalue)
elif datatype == 'B':
value = datavalue.encode('ascii')
elif datatype == 'L':
value = parse_list(datavalue)
return value
def construct_map(row_dict):
ddb_row = {}
for key,value_dict in row_dict.items():
# value is a dict with key as N or S
# if N then use Decimal type
ddb_row[key] = generic_conversion(value_dict)
return ddb_row
def map_function(rec):
row_dict = rec["Item"]
return construct_map(row_dict)
mapped_dyF = Map.apply(frame = dyf, f = map_function, transformation_ctx = "mapped_dyF")
datasink2 = glue_context.write_dynamic_frame_from_options(
frame=mapped_dyF,
connection_type="dynamodb",
connection_options={
"dynamodb.region": "us-east-1",
"dynamodb.output.tableName": destination_table,
"dynamodb.throughput.write.percent": "0.5"
},
transformation_ctx = "datasink2"
)
can anyone help us in how can we unblock from this situation?
Record that we are trying to insert
{
"region": {
"S": "to_delete"
},
"date": {
"N": "20210916"
},
"number_set": {
"NS": [
"0",
"1"
]
},
"test": {
"BOOL": false
},
"map": {
"M": {
"test": {
"S": "value"
},
"test2": {
"S": "value"
},
"nestedmap": {
"M": {
"key": {
"S": "value"
},
"nestedmap1": {
"M": {
"key1": {
"N": "0"
}
}
}
}
}
}
},
"binary": {
"B": "QUFBY2Q="
},
"list": {
"L": [
{
"S": "abc"
},
{
"S": "def"
},
{
"N": "123"
},
{
"M": {
"key2": {
"S": "value2"
},
"nestedmaplist": {
"M": {
"key3": {
"S": "value3"
}
}
}
}
}
]
}
}
I have a tree object like below:
let data = [
{
id: 1,
children: [
{
id: 1.1,
children: [
{
id: 1.2,
children: []
},
{
id: 1.22,
children: []
}
]
}
]
},
{
id: 2,
children: []
}
]
I want to filter out id equal a specific value. In this case, I want to filter out id equal 1.2.
The rusult I want is like below:
let data = [
{
id: 1,
children: [
{
id: 1.1,
children: [
{
id: 1.22,
children: []
}
]
}
]
},
{
id: 2,
children: []
}
]
I have search a few question about filter nest deep object, But still don't know how. I need to use recursion to solve this.
This is my way:
function handleDelete (data) {
return data.filter(t => {
if (t.children.length) {
handleDelete(t.children)
})
} else {
return t.id !== '1.2'
}
})
}
let result = handleDelete(data)
delete a node and its descendants
Here's an effective technique using flatMap and mutual recursion1 -
del accepts an array of nodes, t, a query, q, and calls del1 on each node with the query
del1 accepts a single node, t, a query, q, and calls del on a node's children
const del = (t, q) =>
t.flatMap(v => del1(v, q)) // <-- 1
const del1 = (t, q) =>
q == t.id
? []
: { ...t, children: del(t.children, q) } // <-- 2
const data =
[{id:1,children:[{id:1.1,children:[{id:1.2,children:[]},{id:1.22,children:[]}]}]},{id:2,children:[]}]
const result =
del(data, "1.2")
console.log(result)
In the output, we see node.id 1.2 is removed -
[
{
"id": 1,
"children": [
{
"id": 1.1,
"children": [
{
"id": 1.22,
"children": []
}
]
}
]
},
{
"id": 2,
"children": []
}
]
preserving the descendants
In the program above, if a node.id matches our query, the node and all of its descendent children are removed. If we only want to delete the parent node and keep the children, we can make a single modification (!) to the program -
const del = (t, q) =>
t.flatMap(v => del1(v, q))
const del1 = (t, q) =>
q == t.id
? del(t.children, q) // <-- !
: { ...t, children: del(t.children, q) }
const data =
[{id:1,children:[{id:1.1,children:[{id:1.2,children:[]},{id:1.22,children:[]}]}]},{id:2,children:[]}]
const result =
del(data, "1") // <-- delete node.id equal to "1"
console.log(result)
Notice how the children for 1 are still included in the output -
[
{
"id": 1.1,
"children": [
{
"id": 1.2,
"children": []
},
{
"id": 1.22,
"children": []
}
]
},
{
"id": 2,
"children": []
}
]
without mutual recursion
Mutual recursion isn't the only way to do it, but it's the only way to avoid a dynamic type check, such as the one below. In this final revision, we remove a parent and all of its children, as we did in the first program, but this del is implemented using a single recursive function -
const del = (t, q) =>
Array.isArray(t) // <-- array
? t.flatMap(v => del(v, q))
: Object(t) === t // <-- object
? q == t.id
? []
: { ...t, children: del(t.children, q) }
: t // <-- neither (noop)
const data =
[{id:1,children:[{id:1.1,children:[{id:1.2,children:[]},{id:1.22,children:[]}]}]},{id:2,children:[]}]
const result =
del(data, "1.2")
console.log(result)
The output is the same as the first program, where 1.2 and all descendants are removed -
[
{
"id": 1,
"children": [
{
"id": 1.1,
"children": [
{
"id": 1.22,
"children": []
}
]
}
]
},
{
"id": 2,
"children": []
}
]
1. See this technique used on a different data set in this related Q&A.
2. All programs in this answer produce a new tree. The original input is not modified by del (or del1).
I have the following document,
{
"VehicleDetailId": 1,
"VehicleDetail": [
{
"Id": 1,
"Make": "BMW"
},
{
"Id": 1,
"Model": "ABDS"
},
{
"Id": 1,
"Trim": "5.6L/ASMD"
},
{
"Id": 1,
"Year": 2008
}
]
}
I want to give aliases for the array elements, something like this,
{
"VehicleDetailId": 1,
"Type": "VehicleDetail",
"VehicleDetail": [
{
"MakeId": 1,
"MakeValue": "BMW"
},
{
"ModelId": 1,
"ModelValue": "ABDS"
},
{
"TrimId": 1,
"TrimValue": "5.6L/ASMD"
},
{
"YearId": 1,
"YearValue": 2008
}
]
}
The following query seems to work fine, but since Id is common for all, it is repeating every time.
SELECT c.vehicleDetailId, ARRAY(SELECT v.Id AS MakeId, v.Make AS MakeValue,
v.Id AS ModelId, v.Model AS ModelValue,
v.Id AS TrimId, v.Trim AS TrimValue,
v.Id AS YearId, v.Year AS YearValue
FROM v IN c.VehicleDetail) AS VehicleDetail
FROM c
How should I write the query so that the Id does not repeat every time, and I can fetch an element from a specific position?
You could use UDF to implement your needs.
Udf code:
function userDefinedFunction(array){
var returnArray = [];
for(var i=0;i<array.length;i++){
var obj = array[i];
var map = {};
if(obj.Make){
map["MakeId"]= obj.Id;
map["MakeValue"]= obj.Make;
}else if(obj.Model){
map["ModelId"]= obj.Id;
map["ModelValue"]= obj.Model;
}else if(obj.Trim){
map["TrimId"]= obj.Id;
map["TrimValue"]= obj.Trim;
}else if(obj.Year){
map["YearId"]= obj.Id;
map["YearValue"]= obj.Year;
}
returnArray.push(map);
}
return returnArray;
}
Sql:
SELECT c.VehicleDetailId,udf.test(c.VehicleDetail) AS VehicleDetail
FROM c
Output: