Merge two maps to resultant Map in Groovy - collections

I have two maps as:
firstMap = [totalProjectCount:53, activeProjectCount:29, completedProjectCount:1, userCount:85]
secondMap = [totalProjectCount:48, activeProjectCount:41, completedProjectCount:0, userCount:123]
and i need result such as:
resultMap = [totalProjectCount:101, activeProjectCount:70, completedProjectCount:1, userCount:208]

Yet another Groovy alternative:
def firstMap = [totalProjectCount:53, activeProjectCount:29, completedProjectCount:1, userCount:85, a:1]
def secondMap = [totalProjectCount:48, activeProjectCount:41, completedProjectCount:0, userCount:123, b:2]
def result = [firstMap, secondMap]*.keySet().flatten().toSet().collectEntries { k ->
[k, (firstMap[k] ?: 0) + (secondMap[k] ?: 0)]
}
assert result == [a:1, b:2, completedProjectCount:1, userCount:208, activeProjectCount:70, totalProjectCount:101]
This also works if you have values in one map that don't exist in the other (see a and b)

To make this a reusable part of your toolbelt, it's important, to extract the function you want to merge with.
The merging itself is then just the creation of a new map with all the keys of the maps and the values reduced with the passed function.
Map mergeWith(Closure fn, Map... maps) {
maps*.keySet().sum().collectEntries{k->
[k, maps.findAll{it.containsKey k}*.get(k).inject(fn)]
}
}
def m1 = [a:53, b:29, c:1, x:85]
def m2 = [a:48, b:41, c:0, y:123]
def m3 = [z:42]
// sum
assert mergeWith({a,b->a+b}, m1, m2, m3)==[a:101, b:70, c:1, x:85, y:123, z:42]
// product
assert mergeWith({a,b->a*b}, m1, m2, m3)==[a:2544, b:1189, c:0, x:85, y:123, z:42]
edit: as requested in the comments, some explanations
mergeWith takes a reducing function (a function, that gets called with
two params and returns one value) and some maps. This means, you can a)
use it for other operations than just the sum and b) you can use it for
more than one map.
maps*.keySet().sum() is the union of all keys. Then we
collectEntries (build a new map) over that set of keys. For the value
find all maps, that contains the key (this makes this work nicely even
in cases, where the actual value is falsey) and fetch the values of k
via the spread operator. Then we reduce (oddly named inject) the list
of values with the passed function.

Map additionJoin( Map firstMap, Map secondMap )
{
def resultMap = [:];
resultMap.putAll( firstMap );
resultMap.putAll( secondMap );
resultMap.each { key, value ->
if( firstMap[key] && secondMap[key] )
{
resultMap[key] = firstMap[key] + secondMap[key]
}
}
return resultMap;
}
def firstMap = [totalProjectCount:53, activeProjectCount:29, completedProjectCount:1, userCount:85]
def secondMap = [totalProjectCount:48, activeProjectCount:41, completedProjectCount:0, userCount:123]
def resultMap = additionJoin( firstMap , secondMap )
println resultMap

Try this :
Map<String, Integer> resultMap = new HashMap<>(firstMap);
secondMap.forEach((k, v) -> resultMap.merge(k, v, Integer::sum));

More efficiently, you could write it like below:
Map additionJoin(Map firstMap, Map secondMap)
{
secondMap.each { key, value ->
if( firstMap[key])
{
firstMap[key] = firstMap[key] + secondMap[key]
}
}
return firstMap;
}
def firstMap = [totalProjectCount:53, activeProjectCount:29, completedProjectCount:1, userCount:85]
def secondMap = [totalProjectCount:48, activeProjectCount:41, completedProjectCount:0, userCount:123]
def resultMap additionJoin( firstMap , secondMap )
println resultMap
This would be high in performance as it would save using a third map.
Also, if you understand value by reference you can do it like below as well:
void additionJoin(Map firstMap, Map secondMap)
{
secondMap.each { key, value ->
if( firstMap[key])
{
firstMap[key] = firstMap[key] + secondMap[key]
}
}
}
def firstMap = [totalProjectCount:53, activeProjectCount:29, completedProjectCount:1, userCount:85]
def secondMap = [totalProjectCount:48, activeProjectCount:41, completedProjectCount:0, userCount:123]
additionJoin( firstMap , secondMap )
println firstMap
Though later solution hampers the readability (have opinionated view) which could be achieved by additional statement as below:
def resultMap = firstMap
Hope it helps you do the same efficiently.

Related

gRPC: From Node Js, How to send Array of float by using repeated bytes of protobuff to python

I would like to send a list of float list via nodejs and receive it in python using protobuff's repeated bytes type.
The graph helps to understand the problem:
I tried with this configuration and what I get on the python side is not really what I expect:
tensors=[b'-TWW', b'-TWW', b'-TWW', b'-TWW']
Here is my test in node.
Client :
const PROTO_PATH = __dirname + '/route_guide.proto';
const async = require('async');
const grpc = require('#grpc/grpc-js');
const protoLoader = require('#grpc/proto-loader');
const packageDefinition = protoLoader.loadSync(
PROTO_PATH,
{
keepCase: true,
longs: String,
enums: String,
defaults: true,
oneofs: true
});
const routeguide = grpc.loadPackageDefinition(packageDefinition).routeguide;
const client = new routeguide.RouteGuide('localhost:50051',
grpc.credentials.createInsecure());
function runJoin(callback) {
const call = client.join();
call.on('data', function(receivedMessage) {
console.log('Got message "' + JSON.stringify(receivedMessage));
});
call.on('end', callback);
messageToSend = {
msg: 'parameters_res',
parameters_res: {
parameters: {
tensors: [
new Buffer.from(new Float64Array([45.1]).buffer),
new Buffer.from(new Float64Array([45.1, 84.5, 87.9, 87.1]).buffer),
new Buffer.from(new Float64Array([45.1, 84.5, 87.9, 87.1]).buffer),
new Buffer.from(new Float64Array([45.1, 84.5, 87.9, 87.1]).buffer)
],
tensor_type: 'numpy.ndarray'
}
}
}
console.log(messageToSend);
console.log(messageToSend.parameters_res.parameters.tensors)
call.write(messageToSend);
call.end();
}
function main() {
async.series([
runJoin
]);
}
if (require.main === module) {
main();
}
exports.runJoin = runJoin;
route_guide.proto:
syntax = "proto3";
option java_multiple_files = true;
option java_package = "io.grpc.examples.routeguide";
option java_outer_classname = "RouteGuideProto";
option objc_class_prefix = "RTG";
package routeguide;
service RouteGuide {
rpc Join(stream ClientMessage) returns (stream ClientMessage) {}
}
message RouteNote {
repeated bytes model = 1;
}
message ClientMessage {
message Disconnect { Reason reason = 1; }
message ParametersRes { Parameters parameters = 1; }
oneof msg {
Disconnect disconnect = 1;
ParametersRes parameters_res = 2;
}
}
message Parameters {
repeated bytes tensors = 1;
string tensor_type = 2;
}
enum Reason {
UNKNOWN = 0;
RECONNECT = 1;
POWER_DISCONNECTED = 2;
WIFI_UNAVAILABLE = 3;
ACK = 4;
}
Server:
const PROTO_PATH = __dirname + '/route_guide.proto';
const grpc = require('#grpc/grpc-js');
const protoLoader = require('#grpc/proto-loader');
const packageDefinition = protoLoader.loadSync(
PROTO_PATH,
{keepCase: true,
longs: String,
enums: String,
defaults: true,
oneofs: true
});
const routeguide = grpc.loadPackageDefinition(packageDefinition).routeguide;
function join(call) {
call.on('data', function(receivedMessage) {
console.log("SERVER RECEIVE:");
console.log(receivedMessage);
console.log(receivedMessage.parameters_res.parameters.tensors)
for (const element of receivedMessage.parameters_res.parameters.tensors) {
console.log(element)
}
call.write(receivedMessage);
});
call.on('end', function() {
call.end();
});
}
function getServer() {
var server = new grpc.Server();
server.addService(routeguide.RouteGuide.service, {
join: join
});
return server;
}
if (require.main === module) {
var routeServer = getServer();
routeServer.bindAsync('0.0.0.0:50051', grpc.ServerCredentials.createInsecure(), () => {
routeServer.start()
});
}
exports.getServer = getServer;
MyStartegy.py:
from logging import WARNING
from typing import Callable, Dict, List, Optional, Tuple, cast
import numpy as np
import flwr as fl
from flwr.common import (
EvaluateIns,
EvaluateRes,
FitIns,
FitRes,
Parameters,
Scalar,
Weights,
)
from flwr.common.logger import log
from flwr.server.client_manager import ClientManager
from flwr.server.client_proxy import ClientProxy
from flwr.server.strategy.aggregate import aggregate, weighted_loss_avg
from flwr.server.strategy import Strategy
from tensorflow import Tensor
DEPRECATION_WARNING = """
DEPRECATION WARNING: deprecated `eval_fn` return format
loss, accuracy
move to
loss, {"accuracy": accuracy}
instead. Note that compatibility with the deprecated return format will be
removed in a future release.
"""
DEPRECATION_WARNING_INITIAL_PARAMETERS = """
DEPRECATION WARNING: deprecated initial parameter type
flwr.common.Weights (i.e., List[np.ndarray])
will be removed in a future update, move to
flwr.common.Parameters
instead. Use
parameters = flwr.common.weights_to_parameters(weights)
to easily transform `Weights` to `Parameters`.
"""
class MyStrategy(Strategy):
"""Configurable FedAvg strategy implementation."""
# pylint: disable=too-many-arguments,too-many-instance-attributes
def __init__(
self,
fraction_fit: float = 0.1,
fraction_eval: float = 0.1,
min_fit_clients: int = 2,
min_eval_clients: int = 2,
min_available_clients: int = 2,
eval_fn: Optional[
Callable[[Weights], Optional[Tuple[float, Dict[str, Scalar]]]]
] = None,
on_fit_config_fn: Optional[Callable[[int], Dict[str, Scalar]]] = None,
on_evaluate_config_fn: Optional[Callable[[int], Dict[str, Scalar]]] = None,
accept_failures: bool = True,
initial_parameters: Optional[Parameters] = None,
) -> None:
"""Federated Averaging strategy.
Implementation based on https://arxiv.org/abs/1602.05629
Args:
fraction_fit (float, optional): Fraction of clients used during
training. Defaults to 0.1.
fraction_eval (float, optional): Fraction of clients used during
validation. Defaults to 0.1.
min_fit_clients (int, optional): Minimum number of clients used
during training. Defaults to 2.
min_eval_clients (int, optional): Minimum number of clients used
during validation. Defaults to 2.
min_available_clients (int, optional): Minimum number of total
clients in the system. Defaults to 2.
eval_fn (Callable[[Weights], Optional[Tuple[float, float]]], optional):
Function used for validation. Defaults to None.
on_fit_config_fn (Callable[[int], Dict[str, Scalar]], optional):
Function used to configure training. Defaults to None.
on_evaluate_config_fn (Callable[[int], Dict[str, Scalar]], optional):
Function used to configure validation. Defaults to None.
accept_failures (bool, optional): Whether or not accept rounds
containing failures. Defaults to True.
initial_parameters (Parameters, optional): Initial global model parameters.
"""
super().__init__()
self.min_fit_clients = min_fit_clients
self.min_eval_clients = min_eval_clients
self.fraction_fit = fraction_fit
self.fraction_eval = fraction_eval
self.min_available_clients = min_available_clients
self.eval_fn = eval_fn
self.on_fit_config_fn = on_fit_config_fn
self.on_evaluate_config_fn = on_evaluate_config_fn
self.accept_failures = accept_failures
self.initial_parameters = initial_parameters
def __repr__(self) -> str:
rep = f"FedAvg(accept_failures={self.accept_failures})"
return rep
def num_fit_clients(self, num_available_clients: int) -> Tuple[int, int]:
"""Return the sample size and the required number of available
clients."""
num_clients = int(num_available_clients * self.fraction_fit)
return max(num_clients, self.min_fit_clients), self.min_available_clients
def num_evaluation_clients(self, num_available_clients: int) -> Tuple[int, int]:
"""Use a fraction of available clients for evaluation."""
num_clients = int(num_available_clients * self.fraction_eval)
return max(num_clients, self.min_eval_clients), self.min_available_clients
def initialize_parameters(
self, client_manager: ClientManager
) -> Optional[Parameters]:
"""Initialize global model parameters."""
initial_parameters = self.initial_parameters
self.initial_parameters = None # Don't keep initial parameters in memory
if isinstance(initial_parameters, list):
log(WARNING, DEPRECATION_WARNING_INITIAL_PARAMETERS)
initial_parameters = self.weights_to_parameters(weights=initial_parameters)
return initial_parameters
def evaluate(
self, parameters: Parameters
) -> Optional[Tuple[float, Dict[str, Scalar]]]:
"""Evaluate model parameters using an evaluation function."""
if self.eval_fn is None:
# No evaluation function provided
return None
weights = self.parameters_to_weights(parameters)
eval_res = self.eval_fn(weights)
if eval_res is None:
return None
loss, other = eval_res
if isinstance(other, float):
print(DEPRECATION_WARNING)
metrics = {"accuracy": other}
else:
metrics = other
return loss, metrics
def configure_fit(
self, rnd: int, parameters: Parameters, client_manager: ClientManager
) -> List[Tuple[ClientProxy, FitIns]]:
"""Configure the next round of training."""
config = {}
if self.on_fit_config_fn is not None:
# Custom fit config function provided
config = self.on_fit_config_fn(rnd)
fit_ins = FitIns(parameters, config)
# Sample clients
sample_size, min_num_clients = self.num_fit_clients(
client_manager.num_available()
)
clients = client_manager.sample(
num_clients=sample_size, min_num_clients=min_num_clients
)
# Return client/config pairs
return [(client, fit_ins) for client in clients]
def configure_evaluate(
self, rnd: int, parameters: Parameters, client_manager: ClientManager
) -> List[Tuple[ClientProxy, EvaluateIns]]:
"""Configure the next round of evaluation."""
# Do not configure federated evaluation if fraction_eval is 0
if self.fraction_eval == 0.0:
return []
# Parameters and config
config = {}
if self.on_evaluate_config_fn is not None:
# Custom evaluation config function provided
config = self.on_evaluate_config_fn(rnd)
evaluate_ins = EvaluateIns(parameters, config)
# Sample clients
if rnd >= 0:
sample_size, min_num_clients = self.num_evaluation_clients(
client_manager.num_available()
)
clients = client_manager.sample(
num_clients=sample_size, min_num_clients=min_num_clients
)
else:
clients = list(client_manager.all().values())
# Return client/config pairs
return [(client, evaluate_ins) for client in clients]
def aggregate_fit(
self,
rnd: int,
results: List[Tuple[ClientProxy, FitRes]],
failures: List[BaseException],
) -> Tuple[Optional[Parameters], Dict[str, Scalar]]:
"""Aggregate fit results using weighted average."""
if not results:
return None, {}
# Do not aggregate if there are failures and failures are not accepted
if not self.accept_failures and failures:
return None, {}
# Convert results
print("\n\n aggregate_fit")
print(results)
weights_results = [
(self.parameters_to_weights(fit_res.parameters), fit_res.num_examples)
for client, fit_res in results
]
print("weights_results")
print(weights_results)
return self.weights_to_parameters(aggregate(weights_results)), {}
def aggregate_evaluate(
self,
rnd: int,
results: List[Tuple[ClientProxy, EvaluateRes]],
failures: List[BaseException],
) -> Tuple[Optional[float], Dict[str, Scalar]]:
"""Aggregate evaluation losses using weighted average."""
if not results:
return None, {}
# Do not aggregate if there are failures and failures are not accepted
if not self.accept_failures and failures:
return None, {}
loss_aggregated = weighted_loss_avg(
[
(evaluate_res.num_examples, evaluate_res.loss)
for _, evaluate_res in results
]
)
return loss_aggregated, {}
def weights_to_parameters(self, weights: Weights) -> Parameters:
"""Convert NumPy weights to parameters object."""
print('weights_to_parameters')
print(weights)
tensors = [self.ndarray_to_bytes(ndarray) for ndarray in weights]
return Parameters(tensors=tensors, tensor_type="numpy.nda")
def parameters_to_weights(self, parameters: Parameters) -> Weights:
"""Convert parameters object to NumPy weights."""
print('parameters_to_weights')
print(parameters)
return [self.bytes_to_ndarray(tensor) for tensor in parameters.tensors]
# pylint: disable=R0201
def ndarray_to_bytes(self, ndarray: np.ndarray) -> bytes:
"""Serialize NumPy array to bytes."""
print('ndarray_to_bytes')
print(ndarray)
return None
# pylint: disable=R0201
def bytes_to_ndarray(self, tensor: bytes) -> np.ndarray:
"""Deserialize NumPy array from bytes."""
print('bytes_to_ndarray')
print(tensor)
return None
# Start Flower server for three rounds of federated learning
fl.server.start_server(
server_address='localhost:5006',
config={"num_rounds": 2},
strategy=MyStrategy()
)
Is Float64Array the right type?
What should I use on the python side to deserialize the data?
I specify that I cannot modify the proto.
Thank you in advance for your explanations.

Lua 2d array defined in object uses pass by reference

I have the following object:
local Game = {}
function Game:new(aNumberOfPlayers, aPlayer1, aPlayer2)
self.NumberOfPlayers = aNumberOfPlayers
self.Player1 = aPlayer1
self.Player2 = aPlayer2
self.Id = HttpService:GenerateGUID(true)
self.Status = "Waiting"
self.Moves = {}
self.Position = GetInitialGamePosition()
return self
end
local Square = {}
function Square:new(x, y, index, color)
self.X = x
self.Y = y
self.Index = index
self.Color = color
return self
end
That uses following function to intialize the 2d array for Position
function GetInitialGamePosition()
local mt = {} -- create the matrix
for i=1,15 do
mt[i] = {}
for j=1,15 do
mt[i][j] = Square:new(i,j,GetIndexFromRowColumn(i,j),nil)
end
end
return mt
end
Problem here is that since tables pass by reference each element of the 2d array ends up being the same. In other words when i iterate over a Position every element has same row, column, and index. Not sure what the best way to get around this is?
function Square:new(x, y, index, color)
local o = {}
o.X = x
o.Y = y
o.Index = index
o.Color = color
setmetatable(o, self)
self.__index = self
return o
end

How to create a custom transformer with out any input column?

We have requirement where in we wanted to generate scores of our model with some random values in between 0-1.
To do that we wanted to have a custom transformer which will be generating random numbers with out any input fields.
So can we generate a transformer without input fields in mleap?
Like usually we do create as below:
import ml.combust.mleap.core.Model
import ml.combust.mleap.core.types._
case class RandomNumberModel() extends Model {
private val rnd = scala.util.Random
def apply(): Double = rnd.nextFloat
override def inputSchema: StructType = StructType("input" -> ScalarType.String).get
override def outputSchema: StructType = StructType("output" -> ScalarType.Double ).get
}
How to make it as input schema no necessary to put?
I have never tried that, but given how I achieved to have a custom transformer with multiple input fields ...
package org.apache.spark.ml.feature.mleap
import ml.combust.mleap.core.Model
import ml.combust.mleap.core.types._
import org.apache.spark.ml.linalg._
case class PropertyGroupAggregatorBaseModel (props: Array[String],
aggFunc: String) extends Model {
val outputSize = props.size
//having multiple inputs, you will have apply with a parameter Seq[Any]
def apply(features: Seq[Any]): Vector = {
val properties = features(0).asInstanceOf[Seq[String]]
val values = features(1).asInstanceOf[Seq[Double]]
val mapping = properties.zip(values)
val histogram = props.foldLeft(Array.empty[Double]){
(acc, property) =>
val newValues = mapping.filter(x => x._1 == property).map(x => x._2)
val newAggregate = aggFunc match {
case "sum" => newValues.sum.toDouble
case "count" => newValues.size.toDouble
case "avg" => (newValues.sum / Math.max(newValues.size, 1)).toDouble
}
acc :+ newAggregate
}
Vectors.dense(histogram)
}
override def inputSchema: StructType = {
//here you define the input
val inputFields = Seq(
StructField("input1" -> ListType(BasicType.String)),
StructField("input2" -> ListType(BasicType.Double))
)
StructType(inputFields).get
}
override def outputSchema: StructType = StructType(StructField("output" -> TensorType.Double(outputSize))).get
}
My suggestion would be, that the apply might already work for you. I guess if you define inputSchema as follows, it might work:
override def inputSchema: StructType = {
//here you define the input
val inputFields = Seq.empty[StructField]
StructType(inputFields).get
}

Kotlin - Filter/Map from two Collections

Is it possible to filter and map two Collections based on desired criteria match like the following:
fun main(args: Array<String>) {
val selectedDates = listOf("2018-08-12", "2018-08-13", "2018-08-14")
val expenses = listOf(Expense("Food", "2018-08-12"),
Expense("Transportation", "2018-08-15"),
Expense("Misc.", "2018-08-13"),
Expense("Uber", "2018-08-12"),
Expense("Clothing", "2018-08-16"))
val listOfExpensesInSelectedDate = mutableListOf<Expense>()
for (date in selectedDates){
listOfExpensesInSelectedDate.addAll(expenses.filter { it.date==date })
}
println(listOfExpensesInSelectedDate)
}
data class Expense(
val expense:String,
val date: String
)
Provided the given code above, I am trying to return a list of Expenses that match with the dates from another list of Strings. In the example above, I have used both for Loop and filter function to get my desired result. But is it possible to avoid the for loop and filter and map both the Collections in single line of code?
You could just use in to filter:
val listOfExpensesInSelectedDate = expenses.filter { it.date in selectedDates }
Edit: since hotkey posted the comment about optimal solution, I have tried this in my pc and anyone interested can try it too:
(1st) my answer as was posted:
val start = Date().time
for (i in 1..10000) {
val listOfExpensesInSelectedDate = expenses.filter { it.date in selectedDates }
}
val end= Date().time
println(end - start)
Average time result: 26ms (23ms-35ms)
(2nd) my answer with the hotkey's suggestion to use a set:
val start = Date().time
for (i in 1..10000) {
val expSet = selectedDates.toSet()
val listOfExpensesInSelectedDate = expenses.filter { it.date in expSet }
}
val end= Date().time
println(end - start)
Average time result: 70ms (50ms-86ms)
(3d) hotkey's answer:
val start = Date().time
for (i in 1..10000) {
val groups = expenses.groupBy { it.date }
val listOfExpensesInSelectedDate = selectedDates.flatMap { groups[it].orEmpty() }
}
val end= Date().time
println(end - start)
Average time result: 100ms (74ms-150ms)
You can simplify and optimize your code by first grouping expenses by the date and then selecting and merging the groups that have their key in the selectedDates, like this:
val selectedDates = listOf("2018-08-12", "2018-08-13", "2018-08-14")
val expenses: List<Expense> = TODO("content omitted")
val groups = expenses.groupBy { it.date }
val listOfExpensesInSelectedDate = selectedDates.flatMap { groups[it].orEmpty() }
See: groupBy, flatMap

How can I create new map with new values but same keys from an existing map?

I have an existing map in Groovy.
I want to create a new map that has the same keys but different values in it.
Eg.:
def scores = ["vanilla":10, "chocolate":9, "papaya": 0]
//transformed into
def preference = ["vanilla":"love", "chocolate":"love", "papaya": "hate"]
Any way of doing it through some sort of closure like:
def preference = scores.collect {//something}
You can use collectEntries
scores.collectEntries { k, v ->
[ k, 'new value' ]
}
An alternative to using a map for the ranges would be to use a switch
def grade = { score ->
switch( score ) {
case 10..9: return 'love'
case 8..6: return 'like'
case 5..2: return 'meh'
case 1..0: return 'hate'
default : return 'ERR'
}
}
scores.collectEntries { k, v -> [ k, grade( v ) ] }
Nice, functional style solution(including your ranges, and easy to modify):
def scores = [vanilla:10, chocolate:9, papaya: 0]
// Store somewhere
def map = [(10..9):"love", (8..6):"like", (5..2):"meh", (1..0):"hate"]
def preference = scores.collectEntries { key, score -> [key, map.find { score in it.key }.value] }
// Output: [vanilla:love, chocolate:love, papaya:hate]
def scores = ["vanilla":10, "chocolate":9, "papaya": 0]
def preference = scores.collectEntries {key, value -> ["$key":(value > 5 ? "like" : "hate")]}
Then the result would be
[vanilla:like, chocolate:like, papaya:hate]
EDIT: If you want a map, then you should use collectEntries like tim_yates said.

Resources