Transform a list of maps in Elixir - functional-programming

I have a list of maps, that I get from querying my database.It has a datetime field that I want to transform into another list of maps where the datetime field is transformed to it's corresponding epoch value.
What i have:
[
%{
m_id: 267,
end: #DateTime<2020-03-07 17:30:00Z>,
start: #DateTime<2020-03-07 14:30:00Z>,
type: "normal",
s_name: "smum",
w_id: 256
},
%{
m_id: 267,
end: #DateTime<2020-03-07 07:30:00Z>,
start: #DateTime<2020-03-07 04:30:00Z>,
type: "normal",
s_name: "smum",
w_id: 256
}
]
What i want to transform it to:
[
%{
m_id: 267,
end: 12356789, #some epoch value for eg
start: 12367576, #some epoch value for eg
type: "normal",
s_name: "smum",
w_id: 256
},
%{
m_id: 267,
end: 12334567, #some epoch value for eg
start: 12354767, #some epoch value for eg
type: "normal",
s_name: "smum",
w_id: 256
}
]

To transform a single map, you can do
%{map | end: DateTime.to_unix(map.end), start: DateTime.to_unix(map.start) }
So just Enum.map over the list to apply this to all list members:
Enum.map(list, fn map -> %{map | end: DateTime.to_unix(map.end), start: DateTime.to_unix(map.start) } end)
(I suspected there may be a problem using the map update syntax here because end is a reserved word, but I tested in https://www.jdoodle.com/execute-elixir-online/ and it works.)

I would go with Kernel.SpecialForms.for/1 comprehension.
for %{start: s, end: e} = map <- list do
%{map | start: DateTime.to_unix(s), end: DateTime.to_unix(e)}
end
It’s slightly different from Enum.map/2 solution, because it would discard those elements not having either start or end keys. To handle those properly, one should use Map.update/4 wisely.

Related

I'm having trouble with parsing a JSON file

I am attempting to use a .json file I found online, but I'm starting to think that there is an underlying issue with the file. I am not very knowledgeable in .json files, so I am trying to convert it into a CSV file. I have yet to find a website that can do that for me.
I've tried using R to convert the file since the file is also quite large and I can only assume that most websites have a size limit. I have tried flattening it in r with this code:
library(jsonlite)
library(tidyr)
library(tidyverse)
json_string <- readLines("data.json")
json_data <- fromJSON(json_string)
json_data <- flatten(json_data)
df <- as_data_frame(json_data)
write_csv(df, "output.csv")
but it returns this error:
! Tibble columns must have compatible sizes.
* Size 2: Columns `A-Alrund, God of the Cosmos // A-Hakka, Whispering Raven`, `A-Blessed Hippogriff // A-Tyr's Blessing`, `A-Emerald Dragon // A-Dissonant Wave`, `A-Monster Manual // A-Zoological Study`, `A-Rowan, Scholar of Sparks // A-Will, Scholar of Frost`, and 484 more.
* Size 3: Column `Smelt // Herd // Saw`.
* Size 5: Column `Who // What // When // Where // Why`.
* Size 6: Columns `Everythingamajig`, `Garbage Elemental`, `Ineffable Blessing`, `Knight of the Kitchen Sink`, `Scavenger Hunt`, and 4 more.
i Only values of size one are recycled.
Backtrace:
1. tibble::as_data_frame(json_data)
3. tibble:::as_tibble.list(x, ...)
4. tibble:::lst_to_tibble(x, .rows, .name_repair, col_lengths(x))
5. tibble:::recycle_columns(x, .rows, lengths)
Here is what the first 2 items of the .json file look like
{"data": {"\"Ach! Hans, Run!\"": [{"colorIdentity": ["G", "R"], "colors": ["G", "R"], "convertedManaCost": 6.0, "foreignData": [], "identifiers": {"scryfallOracleId": "a2c5ee76-6084-413c-bb70-45490d818374"}, "isFunny": true, "layout": "normal", "legalities": {}, "manaCost": "{2}{R}{R}{G}{G}", "manaValue": 6.0, "name": "\"Ach! Hans, Run!\"", "printings": ["UNH"], "purchaseUrls": {"cardKingdom": "https://mtgjson.com/links/84dfefe718a51cf8", "cardKingdomFoil": "https://mtgjson.com/links/d8c9f3fc1e93c89c", "cardmarket": "https://mtgjson.com/links/b9d69f0d1a9fb80c", "tcgplayer": "https://mtgjson.com/links/c51d2b13ff76f1f0"}, "rulings": [], "subtypes": [], "supertypes": [], "text": "At the beginning of your upkeep, you may say \"Ach! Hans, run! It's the . . .\" and the name of a creature card. If you do, search your library for a card with that name, put it onto the battlefield, then shuffle. That creature gains haste. Exile it at the beginning of the next end step.", "type": "Enchantment", "types": ["Enchantment"]}], "\"Brims\" Barone, Midway Mobster": [{"colorIdentity": ["B", "W"], "colors": ["B", "W"], "convertedManaCost": 5.0, "foreignData": [], "identifiers": {"scryfallOracleId": "c64c31f2-c1be-414e-9dff-c3b77ba97545"}, "isFunny": true, "layout": "normal", "leadershipSkills": {"brawl": false, "commander": true, "oathbreaker": false}, "legalities": {}, "manaCost": "{3}{W}{B}", "manaValue": 5.0, "name": "\"Brims\" Barone, Midway Mobster", "power": "5", "printings": ["UNF"], "purchaseUrls": {"cardKingdom": "https://mtgjson.com/links/d1e320bd9d6813c0", "cardKingdomFoil": "https://mtgjson.com/links/18f86e8a04682c34", "cardmarket": "https://mtgjson.com/links/d5a3d8cfb60767d4", "tcgplayer": "https://mtgjson.com/links/980f45f2bc8c3733"}, "rulings": [], "subtypes": ["Human", "Rogue"], "supertypes": ["Legendary"], "text": "When \"Brims\" Barone, Midway Mobster enters the battlefield, put a +1/+1 counter on each other creature you control that has a hat.\n\"Brims\" Barone, Midway Mobster has menace as long as you're wearing a hat.", "toughness": "4", "type": "Legendary Creature — Human Rogue", "types": ["Creature"]}]}
I am hoping that the resulting csv file has the keys as the column names, and the values to be assigned to the columns based on their keys.
EDIT:
I have now attached a screenshot of what the json_data structure looks like.Structure of json_data
Assuming it's one of the JSON dumps from scryfall, try this:
library(jsonlite)
library(tidyr)
library(tidyverse)
todo <- list.files(pattern = ".json")
json_data <- fromJSON(todo)
json_data_flat_jsl <- jsonlite::flatten(json_data)
df <- as_tibble(json_data_flat_jsl)
write_csv(df, "output.csv")

Json output from Vegeta HTTP load testing

I am using Vegeta to make some stress test but I am having some trouble while generating a json report. Running the following command I am able to see text results:
vegeta attack -targets="./vegeta_sagemaker_True.txt" -rate=10 -duration=2s | vegeta report -output="attack.json" -type=text
Requests [total, rate] 20, 10.52
Duration [total, attack, wait] 2.403464884s, 1.901136s, 502.328884ms
Latencies [mean, 50, 95, 99, max] 945.385864ms, 984.768025ms, 1.368113304s, 1.424427549s, 1.424427549s
Bytes In [total, mean] 5919, 295.95
Bytes Out [total, mean] 7104, 355.20
Success [ratio] 95.00%
Status Codes [code:count] 200:19 400:1
Error Set:
400
When I run the same command changing -type-text to -type=json I receive really weird numbers ad they don't make sense for me:
{
"latencies": {
"total": 19853536952,
"mean": 992676847,
"50th": 972074984,
"95th": 1438787021,
"99th": 1636579198,
"max": 1636579198
},
"bytes_in": {
"total": 5919,
"mean": 295.95
},
"bytes_out": {
"total": 7104,
"mean": 355.2
},
"earliest": "2019-04-24T14:32:23.099072+02:00",
"latest": "2019-04-24T14:32:25.00025+02:00",
"end": "2019-04-24T14:32:25.761337546+02:00",
"duration": 1901178000,
"wait": 761087546,
"requests": 20,
"rate": 10.519793517492838,
"success": 0.95,
"status_codes": {
"200": 19,
"400": 1
},
"errors": [
"400 "
]
}
Does anyone know why this should be happening?
Thanks!
These numbers are nanoseconds -- the internal representation of time.Duration in Go.
For example, the latencies.mean in the JSON is 992676847, which means 992676847 nanoseconds, that is 992676847/1000/1000 = 992.676847ms.
Actually, in vegeta, if you declare type as text (-type=text), it will use NewTextReporter, and print the time.Duration as a user-friendly string. If you declare type as json (-type=json), it will use NewJSONReporter and return time.Duration's internal representation:
A Duration represents the elapsed time between two instants as an int64 nanosecond count. The representation limits the largest representable duration to approximately 290 years.

How to process output from match function in jq?

I'm using js tool to parse some JSONs/strings. My minimal example is the following command:
echo '"foo foo"' | jq 'match("(foo)"; "g")'
Which results in the following output:
{
"offset": 0,
"length": 3,
"string": "foo",
"captures": [
{
"offset": 0,
"length": 3,
"string": "foo",
"name": null
}
]
}
{
"offset": 4,
"length": 3,
"string": "foo",
"captures": [
{
"offset": 4,
"length": 3,
"string": "foo",
"name": null
}
]
}
I want my final output for this example to be:
"foo,foo"
But in this case I get two separate objects instead of an array or similar that I could call implode on. I guess either the API isn't made for my UC or my understanding of it is very wrong. Please, advise.
The following script takes the string value from each of the separate objects with .string, wraps them in an array [...] and then joins the members of the array with commas using join.
I modified the regex because you didn't actually need a capture group for the given use case, but if you wanted to access the capture groups you could do .captures[].string instead of .string.
echo '"foo foo"' | jq '[match("foo"; "g").string] | join(",")'

Get json with pljson & plsql

I'm trying to chop out some lists from the following json using pljson
my_json := json('{"Order":
{"no": 1, "batch": 2,"id": 3,"quantity": 10,
"sm_pack": [
{
"no": 10,
"id": 1010,
"quantity": 2
},
{
"no": 11,
"id": 1040,
"quantity": 8
}
],
"sm_size": [
{ ....etc etc
However, I can't get it to work?
I can print the data using this syntax:
v_myjson.path('Order.sm_pack').print;
v_myjson.path('Order.sm_pack[1].no').print;
But how can I assing all those different lists to variables for further processing. i tried different versions of "v_list := json_list(my_json.get('Order.sm_pack')) .. my_json.get('sm_pack').. whatever I try its"NULL SELF" and I seem to have turned blind.
Regards
Printing json lists and objects differs from assigning them to some variables to manipulate them. I will try to answer your question through your example as follows:
DECLARE
obj json := json();
obj_1 json := json();
arr json_list := json_list();
val NUMBER;
BEGIN
/*Create your object*/
obj := json('{"Order":{"no":1,"batch":2,"id":3,"quantity":10,"sm_pack":[{"no":10,"id":1010,"quantity":2},{"no":11,"id":1040,"quantity":8}],"sm_size":[{"no":10,"id":1010,"quantity":2},{"no":11,"id":1040,"quantity":8}]}}');
/*Assign object*/
obj_1 :=json(obj.get('Order'));
/*Assign list from within the object*/
arr := json_list(obj_1.get('sm_pack'));
arr.print;
--or
arr := json_list(json(obj.get('Order')).get('sm_pack'));
arr.print;
/*Get object value from within list*/
val := json_ext.get_number(json(arr.get(2)), 'id');
DBMS_OUTPUT.PUT_LINE(VAL);
END;
/
Notice I used get_number function as your values without single quotes, otherwise, I would use get_string.
Hope that helps!

how to print recursively a Python dictionary and its subdictionaries with whitespace alignment into columns

I want to create a function that can take a dictionary of dictionaries such as the following
information = {
"sample information": {
"ID": 169888,
"name": "ttH",
"number of events": 124883,
"cross section": 0.055519,
"k factor": 1.0201,
"generator": "pythia8",
"variables": {
"trk_n": 147,
"zappo_n": 9001
}
}
}
and then print it in a neat way such as the following, with alignment of keys and values using whitespace:
sample information:
ID: 169888
name: ttH
number of events: 124883
cross section: 0.055519
k factor: 1.0201
generator: pythia8
variables:
trk_n: 147
zappo_n: 9001
My attempt at the function is the following:
def printDictionary(
dictionary = None,
indentation = ''
):
for key, value in dictionary.iteritems():
if isinstance(value, dict):
print("{indentation}{key}:".format(
indentation = indentation,
key = key
))
printDictionary(
dictionary = value,
indentation = indentation + ' '
)
else:
print(indentation + "{key}: {value}".format(
key = key,
value = value
))
It produces the output like the following:
sample information:
name: ttH
generator: pythia8
cross section: 0.055519
variables:
zappo_n: 9001
trk_n: 147
number of events: 124883
k factor: 1.0201
ID: 169888
As is shown, it successfully prints the dictionary of dictionaries recursively, however is does not align the values into a neat column. What would be some reasonable way of doing this for dictionaries of arbitrary depth?
Try using the pprint module. Instead of writing your own function, you can do this:
import pprint
pprint.pprint(my_dict)
Be aware that this will print characters such as { and } around your dictionary and [] around your lists, but if you can ignore them, pprint() will take care of all the nesting and indentation for you.

Resources