I am working through Miguel Morales Deep Reinforcement Learning book and I have come across some syntax that I am unfamilar with. I have looked at the tutorials for dictionary comprehension and lambda functions and I am yet to find what the square brackets does at the end of the lamda call. Can anyone help?
def policy_improvement(value, mdp, gamma=1.0):
''' Performs improvement of a given policy by evaluating the actions for each state and choosing greedily. '''
Q = np.zeros((len(mdp), len(mdp[0])), dtype=np.float64)
for state in range(len(mdp)):
for action in range(len(mdp[state])):
for transition_prob, state_prime, reward, done in mdp[state][action]:
Q[state][action] += transition_prob * (reward + gamma * value[state_prime] * (not done)) # Update the Q value for each action.
new_policy_pie = lambda state: {state:action for state, action in enumerate(np.argmax(Q, axis=1))}[state]
return new_policy_pie
The lambda is made of two parts:
# 1. Create a map of state to action
d = {state:action for state, action in enumerate(np.argmax(Q, axis=1))}
# 2. Return the value for argument `state`
d[state]
So the [state] bit is part of the lambda, it's the dictionary selector.
The intent might have been to write a fail-safe lambda in case the state does not exist, but then they should use .get(state) instead of [].
So in the end, your code above could be replaced with:
lambda state: np.argmax(Q, axis=1)[state]
Related
The following code comes from Stylish F# 6: Crafting Elegant Functional Code for .NET 6 listing 9-13:
let randomByte =
let r = System.Random()
fun () ->
r.Next(0, 255) |> byte
// E.g. A3-52-31-D2-90-E6-6F-45-1C-3F-F2-9B-7F-58-34-44-
for _ in 0..15 do
printf "%X-" (randomByte())
printfn ""
The author states, "Although we call randomByte() multiple times, only one System.Random() instance is created."
I understand randomByte returns a function that does not create a System.Random() instance, but it seems to me multiple System.Random() instances would be created each time through the for-do-loop anyway.
I would appreciate an explanation of how multiple instances of System.Random() are not created in this case.
The key point is that randomByte is not a function. It's a value with some complex initialization logic. Like, for example, I could write:
let x = 5
Or I could write:
let x =
let fourtyTwo = 42
let thirtySeven = 37
fourtyTwo - thirtySeven
And these would be equivalent. Both declare a value named x and equal to 5. I hope you can see how the expression fourtyTwo - thirtySeven is evaluated only once, not every time somebody gets the value of x.
And so it works with randomByte too: it's a value with non-trivial initialization logic. During that value's initialization, first it creates an instance of System.Random, and then it creates an anonymous function that closes over that instance, and this anonymous function becomes the value of randomByte.
I am seeing that Julia explicitly does NOT do classes... and I should instead embrace mutable structs.. am I going down the correct path here?? I diffed my trivial example against an official flux library but cannot gather how do I reference self like a python object.. is the cleanest way to simply pass the type as a parameter in the function??
Python
# Dense Layer
class Layer_Dense
def __init__(self, n_inputs, n_neurons):
self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
self.biases = np.zeros((1, n_neurons))
def forward(self, inputs):
pass
My JuliaLang version so far
mutable struct LayerDense
num_inputs::Int64
num_neurons::Int64
weights
biases
end
function forward(layer::LayerDense, inputs)
layer.weights = 0.01 * randn(layer.num_inputs, layer.num_neurons)
layer.biases = zeros((1, layer.num_neurons))
end
The flux libraries version of a dense layer... which looks very different to me.. and I do not know what they're doing or why.. like where is the forward pass call, is it here in flux just named after the layer Dense???
source : https://github.com/FluxML/Flux.jl/blob/b78a27b01c9629099adb059a98657b995760b617/src/layers/basic.jl#L71-L111
struct Dense{F, M<:AbstractMatrix, B}
weight::M
bias::B
σ::F
function Dense(W::M, bias = true, σ::F = identity) where {M<:AbstractMatrix, F}
b = create_bias(W, bias, size(W,1))
new{F,M,typeof(b)}(W, b, σ)
end
end
function Dense(in::Integer, out::Integer, σ = identity;
initW = nothing, initb = nothing,
init = glorot_uniform, bias=true)
W = if initW !== nothing
Base.depwarn("keyword initW is deprecated, please use init (which similarly accepts a funtion like randn)", :Dense)
initW(out, in)
else
init(out, in)
end
b = if bias === true && initb !== nothing
Base.depwarn("keyword initb is deprecated, please simply supply the bias vector, bias=initb(out)", :Dense)
initb(out)
else
bias
end
return Dense(W, b, σ)
end
This is an equivalent of your Python code in Julia:
mutable struct Layer_Dense
weights::Matrix{Float64}
biases::Matrix{Float64}
Layer_Dense(n_inputs::Integer, n_neurons::Integer) =
new(0.01 * randn(n_inputs, n_neurons),
zeros((1, n_neurons)))
end
forward(ld::Layer_Dense, inputs) = nothing
What is important here:
here I create an inner constructor only, as outer constructor is not needed; as opposed in the Flux.jl code you have linked the Dense type defines both inner and outer constructors
in python forward function does not do anything, so I copied it in Julia (your Julia code worked a bit differently); note that instead of self one should pass an instance of the object to the function as the first argument (and add ::Layer_Dense type signature so that Julia knows how to correctly dispatch it)
similarly in Python you store only weights and biases in the class, I have reflected this in the Julia code; note, however, that for performance reasons it is better to provide an explicit type of these two fields of Layer_Dense struct
like where is the forward pass call
In the code you have shared only constructors of Dense object are defined. However, in the lines below here and here the Dense type is defined to be a functor.
Functors are explained here (in general) and in here (more specifically for your use case)
I have following 2 forms of calling a collect operation, both return same result, but I still cannot depend fully on method references and need a lambda.
<R> R collect(Supplier<R> supplier,
BiConsumer<R,? super T> accumulator,
BiConsumer<R,R> combiner)
For this consider the following stream consisting on 100 random numbers
List<Double> dataList = new Random().doubles().limit(100).boxed()
.collect(Collectors.toList());
1) Following example uses pure lambdas
Map<Boolean, Integer> partition = dataList.stream()
.collect(() -> new ConcurrentHashMap<Boolean, Integer>(),
(map, x) ->
{
map.merge(x < 0.5 ? Boolean.TRUE : Boolean.FALSE, 1, Integer::sum);
}, (map, map2) ->
{
map2.putAll(map);
});
2) Following tries to use method references but 2nd argument still requires a lambda
Map<Boolean, Integer> partition2 = dataList.stream()
.collect(ConcurrentHashMap<Boolean, Integer>::new,
(map, x) ->
{
map.merge(x < 0.5 ? Boolean.TRUE : Boolean.FALSE, 1, Integer::sum);
}, Map::putAll);
How can I rewrite 2nd argument of collect method in java 8 to use method reference instead of a lambda for this example?
System.out.println(partition.toString());
System.out.println(partition2.toString());
{false=55, true=45}
{false=55, true=45}
A method reference is a handy tool if you have an existing method doing exactly the intended thing. If you need adaptations or additional operations, there is no special syntax for method references to support that, except, when you consider lambda expressions to be that syntax.
Of course, you can create a new method in your class doing the desired thing and create a method reference to it and that’s the right way to go when the complexity of the code raises, as then, it will get a meaningful name and become testable. But for simple code snippets, you can use lambda expressions, which are just a simpler syntax for the same result. Technically, there is no difference, except that the compiler generated method holding the lambda expression body will be marked as “synthetic”.
In your example, you can’t even use Map::putAll as merge function, as that would overwrite all existing mappings of the first map instead of merging the values.
A correct implementation would look like
Map<Boolean, Integer> partition2 = dataList.stream()
.collect(HashMap::new,
(map, x) -> map.merge(x < 0.5, 1, Integer::sum),
(m1, m2) -> m2.forEach((k, v) -> m1.merge(k, v, Integer::sum)));
but you don’t need to implement it by yourself. There are appropriate built-in collectors already offered in the Collectors class:
Map<Boolean, Long> partition2 = dataList.stream()
.collect(Collectors.partitioningBy(x -> x < 0.5, Collectors.counting()));
I am implementing simple modulus function on sage jupitar notebook. The function is as follows:
Mod2(v,b)=(v+b*(q-1)/2) mod q mod 2
The function is wriiten in sage as :
def modulus(v,b):
q=12289
c=[]
for i in range(len(v)):
c.append(mod(((v[i]+b[i]*(q-1)//2)%q),2))
return c
The function is executed as :
dimension = 1024 # degree of polynomials
modulus = 12289
R.<X> = PolynomialRing(GF(modulus),) Gaussian field of integers
Y.<x> = R.quotient(X^(dimension) + 1) # Cyclotomic field
pi=Y.random_element()
c=Y.random_element()
xi=Y.random_element()
sj=Y.random_element()
rj=Y.random_element()
gj=Y.random_element()
kj=((pi*c+xi)*(sj*d+rj)+(2*c*gj))
# Now, We are making another list named mon and calling the modulus function
mon=[1,2,6,5,8]
modulus(kj.list(),mon)
I get following error while executing the above code.
TypeError: 'sage.rings.integer.Integer' object is not callable
This kind of error nearly always happens when you try to do something that Sage translates as 1(3). In this case, you have redefined something!
def modulus(v,b):
versus
modulus = 12289
You can't overload things this way in Python. Sage will replace what modulus refers to by that number; your function is just gone now. So when you do
modulus(kj.list(),mon)
you are trying to call 12289 as a function.
I suggest calling your other modulus modulus1 or something like that. Do it consistently, and this problem should disappear. Good luck.
Just for learning purpose, I tried to set a dictionary as a global variable in accumulator the add function works well, but I ran the code and put dictionary in the map function, it always return empty.
But similar code for setting list as a global variable
class DictParam(AccumulatorParam):
def zero(self, value = ""):
return dict()
def addInPlace(self, acc1, acc2):
acc1.update(acc2)
if __name__== "__main__":
sc, sqlContext = init_spark("generate_score_summary", 40)
rdd = sc.textFile('input')
#print(rdd.take(5))
dict1 = sc.accumulator({}, DictParam())
def file_read(line):
global dict1
ls = re.split(',', line)
dict1+={ls[0]:ls[1]}
return line
rdd = rdd.map(lambda x: file_read(x)).cache()
print(dict1)
For anyone who arrives at this thread looking for a Dict accumulator for pyspark: the accepted solution does not solve the posed problem.
The issue is actually in the DictParam defined, it does not update the original dictionary. This works:
class DictParam(AccumulatorParam):
def zero(self, value = ""):
return dict()
def addInPlace(self, value1, value2):
value1.update(value2)
return value1
The original code was missing the return value.
I believe that print(dict1()) simply gets executed before the rdd.map() does.
In Spark, there are 2 types of operations:
transformations, that describe the future computation
and actions, that call for action, and actually trigger the execution
Accumulators are updated only when some action is executed:
Accumulators do not change the lazy evaluation model of Spark. If they
are being updated within an operation on an RDD, their value is only
updated once that RDD is computed as part of an action.
If you check out the end of this section of the docs, there is an example exactly like yours:
accum = sc.accumulator(0)
def g(x):
accum.add(x)
return f(x)
data.map(g)
# Here, accum is still 0 because no actions have caused the `map` to be computed.
So you would need to add some action, for instance:
rdd = rdd.map(lambda x: file_read(x)).cache() # transformation
foo = rdd.count() # action
print(dict1)
Please make sure to check on the details of various RDD functions and accumulator peculiarities because this might affect the correctness of your result. (For instance, rdd.take(n) will by default only scan one partition, not the entire dataset.)
For accumulator updates performed inside actions only, their value is
only updated once that RDD is computed as part of an action