I have this json:
{
"books": [
{
"bookId": "1",
"bookName": "name1",
"bookYear": "2010"
},
{
"bookId": "2",
"bookName": "name2",
"bookYear": "2010"
}
]
}
and I want to remove the list that contains "bookId": "1"
The result should be:
{
"books": [
{
"bookId": "2",
"bookName": "name2",
"bookYear": "2010"
}
]
}
I've tried this code:
import groovy.json.JsonSlurper
import groovy.json.JsonOutput
JsonSlurper slurper = new JsonSlurper()
String jsonString = vars.get("jsonOutput")
def parsedJson = slurper.parseText(jsonString)
log.info("Json before: " + jsonString)
int num = parsedJson.books.size()
for(int i=0;i<parsedJson.books.size();i++){
parsedJson.books[i].values().removeAll(value -> value.equals("1"))
}
def json = JsonOutput.toJson(parsedJson)
log.info("Json after: " + json.toString())
but it removes only the "bookId" from list:
Json before: {"books":[{"bookId":"1","bookName":"name1","bookYear":"2010"},{"bookId":"2","bookName":"name2","bookYear":"2010"}]}
Json after: {"books":[{"bookName":"name1","bookYear":"2010"},{"bookId":"2","bookName":"name2","bookYear":"2010"}]}
Please, could you help me on that?
You can filter the list of books like so:
def parsedJson = slurper.parseText(jsonString)
parsedJson.books = parsedJson.books.findAll { it.bookId != '1' }
def json = JsonOutput.toJson(parsedJson)
Related
We are trying to do DynamoDB migration from prod account to stage account.
In the source account, we are making use of "Export" feature of DDB to put the compressed .json.gz files into destination S3 bucket.
We have written a glue script which will read the exported .json.gz files and writes it to DDB table.
We are making the code generic, so we should be able to migrate any DDB table from prod to stage account.
As part of that process, while testing we are facing issues when we are trying to write a NUMBER SET data to target DDB table.
Following is the sample snippet which is raising ValidationException when trying to insert into DDB
from decimal import Decimal
def number_set(datavalue):
# datavalue will be ['0', '1']
set_of_values = set()
for value in datavalue:
set_of_values.add(Decimal(value))
return set_of_values
When running the code, we are getting following ValidationException
An error occurred while calling o82.pyWriteDynamicFrame. Supplied AttributeValue is empty, must contain exactly one of the supported datatypes (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: UKEU70T0BLIKN0K2OL4RU56TGVVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
However, if instead of Decimal(value) if we use int(value) then no ValidationException is being thrown and the job succeeds.
I feel that write_dynamic_frame_from_options will try to infer schema based on the values the element contains, if the element has "int" values then the datatype would be "NS", but if the element contains all "Decimal type" values, then it is not able to infer the datatype.
The glue job we have written is
dyf = glue_context.create_dynamic_frame_from_options(
connection_type="s3",
connection_options={
"paths": [file_path]
},
format="json",
transformation_ctx = "dyf",
recurse = True,
)
def number_set(datavalue):
list_of_values = []
for value in datavalue:
list_of_values.append(Decimal(value))
print("list of values ")
print(list_of_values)
return set(list_of_values)
def parse_list(datavalue):
list_of_values = []
for object in datavalue:
list_of_values.append(generic_conversion(object))
return list_of_values
def generic_conversion(value_dict):
for datatype,datavalue in value_dict.items():
if datatype == 'N':
value = Decimal(datavalue)
elif datatype == 'S':
value = datavalue
elif datatype == 'NS':
value = number_set(datavalue)
elif datatype == 'BOOL':
value = datavalue
elif datatype == 'M':
value = construct_map(datavalue)
elif datatype == 'B':
value = datavalue.encode('ascii')
elif datatype == 'L':
value = parse_list(datavalue)
return value
def construct_map(row_dict):
ddb_row = {}
for key,value_dict in row_dict.items():
# value is a dict with key as N or S
# if N then use Decimal type
ddb_row[key] = generic_conversion(value_dict)
return ddb_row
def map_function(rec):
row_dict = rec["Item"]
return construct_map(row_dict)
mapped_dyF = Map.apply(frame = dyf, f = map_function, transformation_ctx = "mapped_dyF")
datasink2 = glue_context.write_dynamic_frame_from_options(
frame=mapped_dyF,
connection_type="dynamodb",
connection_options={
"dynamodb.region": "us-east-1",
"dynamodb.output.tableName": destination_table,
"dynamodb.throughput.write.percent": "0.5"
},
transformation_ctx = "datasink2"
)
can anyone help us in how can we unblock from this situation?
Record that we are trying to insert
{
"region": {
"S": "to_delete"
},
"date": {
"N": "20210916"
},
"number_set": {
"NS": [
"0",
"1"
]
},
"test": {
"BOOL": false
},
"map": {
"M": {
"test": {
"S": "value"
},
"test2": {
"S": "value"
},
"nestedmap": {
"M": {
"key": {
"S": "value"
},
"nestedmap1": {
"M": {
"key1": {
"N": "0"
}
}
}
}
}
}
},
"binary": {
"B": "QUFBY2Q="
},
"list": {
"L": [
{
"S": "abc"
},
{
"S": "def"
},
{
"N": "123"
},
{
"M": {
"key2": {
"S": "value2"
},
"nestedmaplist": {
"M": {
"key3": {
"S": "value3"
}
}
}
}
}
]
}
}
I am trying to create a JSON array of objects using jsonlite in R.
The goal is a JSON like this:
{
"top":[
{
"master1": {
"item1": "value1"
}
},
{
"master2": {
"item2": "value2"
}
}
]
}
I tried list of lists and dataframes with list column but can't get the desired output.
Apart from that, converting the above with fromJSON/toJSON results in a different format:
library(jsonlite)
txt <- '{
"top":[
{
"master1": {
"item1": "value1"
}
},
{
"master2": {
"item2": "value2"
}
}]
}'
toJSON(fromJSON(txt), pretty = T)
# Output
{
"top": [
{
"master1": {
"item1": "value1"
},
"master2": {}
},
{
"master1": {},
"master2": {
"item2": "value2"
}
}
]
}
Do I need to set a parameter for this to work?
By default, the fromJSON call converts your input to a data frame, and therefore adds NA values, which result in the empty entries in your output JSON:
$top
item1 item2
1 value1 <NA>
2 <NA> value2
You need to add simplifyDataFrame = FALSE to the fromJSON call to prevent it from creating a data frame.
toJSON(fromJSON(txt, simplifyDataFrame = FALSE), pretty = T)
gives
{
"top": [
{
"master1": {
"item1": ["value1"]
}
},
{
"master2": {
"item2": ["value2"]
}
}
]
}
clients is an array inside my doc, the following query
SELECT
f.id, f.clients
FROM f
where f.id ="35fb0733-dfa1-4932-9690-3ee5b05d89ff"
Returns
[
"id": "35fb0733-dfa1-4932-9690-3ee5b05d89ff",
{
"clients": [
{
"firstname": "Benjamin",
"surname": "Bob",
},
{
"firstname": "Rachael",
"surname": "Smith",
}
]
}
]
But I would like clients to look like :
"firstnames": [ "Benjamin", "Rachael" ]
"surnames": [ "Bob", "Smith" ]
Is this possible?
You could use the ARRAY expression w/ a subquery to achieve that.
Try this query:
SELECT
ARRAY(SELECT VALUE client.firstname FROM client IN f.clients) AS firstnames,
ARRAY(SELECT VALUE client.surname FROM client IN f.clients) AS surnames
FROM f
Just give an additional option, you could use stored procedure to get the results you want.
function sample() {
var collection = getContext().getCollection();
var isAccepted = collection.queryDocuments(
collection.getSelfLink(),
'SELECT f.id, c.firstname,c.surname FROM f join c in f.clients where f.id ="1"',
function (err, feed, options) {
if (err) throw err;
var map = {};
var firstname = [];
var surname =[];
if (!feed || !feed.length) {
var response = getContext().getResponse();
response.setBody('no docs found');
}
else {
for(var i=0;i<feed.length;i++){
map["id"] = feed[i].id;
firstname.push(feed[i].firstname);
surname.push(feed[i].surname);
}
map["firstname"] = firstname;
map["surname"] =surname;
var response = getContext().getResponse();
response.setBody(map);
}
});
if (!isAccepted) throw new Error('The query was not accepted by the server.');
}
Test Output:
{
"id": "1",
"firstname": [
"Benjamin",
"Rachael"
],
"surname": [
"Bob",
"Smith"
]
}
I have a document like below, in my document, have a Array . in array have Objects. How can i update new object to an Array.
As you see below i can add document in a colletion wit an array, but when i tried to update it gives error
java.lang.IllegalArgumentException: Invalid data. Unsupported type: com.test.data.modal.Product
What i tried;
var pid = ""
btnAdd.setOnClickListener {
val list = ArrayList<Product>()
list.add(Product("u1", "1", 1))
list.add(Product("u2", "2", 1))
list.add(Product("u3", "3", 1))
val testObject = TestObject("Talha", "Kosen", list)
FirebaseFirestore.getInstance().collection("Test")
.add(testObject)
.addOnCompleteListener { task ->
pid = task.result.id
}
}
btnUpdate.setOnClickListener {
val list = ArrayList<Product>()
list.add(Product("u1", "1", 1))
list.add(Product("u2", "2", 1))
list.add(Product("u3", "3", 1))
list.add(Product("u4", "4", 4))
FirebaseFirestore.getInstance()
.collection("Test")
.document(pid)
.update("product", list)
}
document:
POJO:
#IgnoreExtraProperties
#Keep
class TestObject {
var name: String = ""
var surname: String = ""
lateinit var productList: List<Product>
constructor()
constructor(name: String, surname: String, productList: List<Product>) {
this.name = name
this.surname = surname
this.productList = productList
}
}
#IgnoreExtraProperties
#Keep
class Product {
var imagePath: String = ""
var productUrl: String = ""
var brand: Int = 0
constructor()
constructor(imagePath: String, productUrl: String, brand: Int) {
this.imagePath = imagePath
this.productUrl = productUrl
this.brand = brand
}
}
I have probably not the answer but this could help :
You try to update with an ArrayList but you store a List (in your TestObject)
Refer to the documentation to update fields in nested objects https://firebase.google.com/docs/firestore/manage-data/add-data#update_fields_in_nested_objects
It could be a good idea to store a collection in your TestObject document instead of a list ? You'll able to update easily your fields
From the following input:
{
"key1": {
"key_x": "1",
...
"key_z": "2"
},
"key2": {
"key_x": "2",
...
"key_z": "3"
}
}
I would like to exclude all keys with the name "key_x" so the result should be
{
"key1": {
...
"key_z": "2"
},
"key2": {
...
"key_z": "3"
}
}
You can use the del() function:
jq 'del(.[]|.key_x)' input.json