I want to initialize the convolution layer by a specific kernel which is not defined in Keras. For instance, if I define the below function to initialize the kernel:
def init_f(shape):
ker=np.zeros((shape,shape))
ker[int(np.floor(shape/2)),int(np.floor(shape/2))]=1
return ker
And the convolution layer is designed as follows:
model.add(Conv2D(filters=32, kernel_size=(3,3),
kernel_initializer=init_f(3)))
I get the error:
Could not interpret initializer identifier
I have followed a similar issue at:
https://groups.google.com/forum/#!topic/keras-users/J46pplO64-8
But I could not adapt it to my code.
Could you please help me to define the arbitrary kernel in Keras?
A few items to fix. Let's start with the kernel initializer. From the documentation:
If passing a custom callable, then it must take the argument shape (shape of the variable to initialize) and dtype (dtype of generated values)
So the signature should become:
def init_f(shape, dtype=None)
The function will work without the dtype, but it's good practice to keep it there. That way you can specify the dtype to calls inside your function, e.g.:
np.zeros(shape, dtype=dtype)
This also addresses your second issue: the shape argument is a tuple, so you just need to pass it straight to np.zeros and don't need to make another tuple.
I'm guessing you're trying to initialize the kernel with a 1 in the middle, so you could also generalize your function to work with whatever shape it receives:
ker[tuple(map(lambda x: int(np.floor(x/2)), ker.shape))]=1
Putting it all together:
def init_f(shape, dtype=None):
ker = np.zeros(shape, dtype=dtype)
ker[tuple(map(lambda x: int(np.floor(x/2)), ker.shape))]=1
return ker
One last problem. You need to pass the function to the layer, not the result of the call:
model.add(Conv2D(filters=32, kernel_size=(3,3),
kernel_initializer=init_f))
The layer function will pass the arguments to init_f.
Related
I am trying to lean deep learning using Julia. In one of the tutorials, which is about MLP, use the below structure for modeling multiple layers in ANN. What does this code mean?
struct Chain
layers
Chain(layers...) = new(layers)
end
This definition in isolation doesn't really "mean" anything; it is just a user defined struct with one field (called layers) and one inner constructor. Usually custom structs like this is used for collecting some data and/or used to define operations on, e.g. you could define a function f operating on this struct like this:
function f(c::Chain)
# do something with the layers in the chain
end
but in order to understand what it is used for in this specific case you probably need to consult the documentation and/or the rest of the code.
One peculiarity in this example:
The inner constructor takes multiple arguments (layers...) and creates a tuple out of them, which is assigned to the property layers.
julia> c = Chain(1, 2, "foo")
Chain((1, 2, "foo"))
I am new to Julia and I am having trouble determining the type of the layer in the Flux's model. For the purpose of example, imagine that my model is just one neuron:
using Flux
m = Chain(Dense(1, 1, sigmoid))
I want to iterate my Chain and, depending on the type of the layer, make different actions (specifically, I want to add regularization for the Dense layers).
I come to Julia from Python, and my first guess was to compare the type of the layer to type of Dense. Contrary to my intuition, this gives me false:
for layer in m
println(typeof(layer) == typeof(Dense))
end
Why this does not work in Julia?
What is the proper way to make it work of Julia? Of course, I can check if the specific fields of the struct (in/out/sigmoid in case of Dense) exist for the given layer, but there would be no guarantee that it is not some other layer with analogous fields.
Use layers property of m instead and to check if a value is of a given type use isa instead. In summary this should work:
for layer in m.layers
if layer isa Dense
# do something with dense layer
else
# do something else
end
end
EDIT: indeed m supports iteration and indexing, which I did not know, so as #darsnack suggested this is enough:
for layer in m
if layer isa Dense
# do something with dense layer
else
# do something else
end
end
Now to clarify type checking:
if you have a value, and you want to check if its type is a subtype of a given type use isa as I have above
if you have two types you want to compare for subtyping use <:, so you could have written typeof(layer) <: Dense; for types == checks are not recommended, see this warning in the Julia manual
You can check out this section of the Julia manual to read more about it
Iterating with for layer in m should be fine. The reason you get false is because typeof(Dense) == UnionAll. You should change your code to:
for layer in m
println(typeof(layer) == Dense))
end
A more Julian approach is to dispatch on the layer type like so:
function processlayer(layer::Dense)
# do special thing for dense
end
function processlayer(layer)
# do other thing for anything else
end
for layer in m
processlayer(layer)
end
I was looking for a fast implementation of FWHT(Fast Walsh-Hadamard transformation) to understand it and implement it in python (implementation should be able to handle an n-dimensional array and should be able to apply the transformation on any specific set of dimensions).
I came across the Julia implementation (https://github.com/stevengj/Hadamard.jl) which seems to be pretty good but as I am new to Julia I am not able to understand a part of the code:
for (Tr,Tc,fftw,lib) in ((:Float64,:Complex128,"fftw",FFTW.libfftw),
(:Float32,:Complex64,"fftwf",FFTW.libfftwf))
#eval function Plan_Hadamard{N}(X::StridedArray{$Tc,N}, Y::StridedArray{$Tc,N},
region, flags::Unsigned, timelimit::Real,
bitreverse::Bool)
set_timelimit($Tr, timelimit)
dims, howmany = dims_howmany(X, Y, [size(X)...], region)
dims = hadamardize(dims, bitreverse)
plan = ccall(($(string(fftw,"_plan_guru64_dft")),$lib),
PlanPtr,
(Int32, Ptr{Int}, Int32, Ptr{Int},
Ptr{$Tc}, Ptr{$Tc}, Int32, UInt32),
size(dims,2), dims, size(howmany,2), howmany,
X, Y, FFTW.FORWARD, flags)
set_timelimit($Tr, NO_TIMELIMIT)
if plan == C_NULL
error("FFTW could not create plan") # shouldn't normally happen
end
return cFFTWPlan{$Tc,FFTW.FORWARD,X===Y,N}(plan, flags, region, X, Y)
end
#eval function Plan_Hadamard{N}(X::StridedArray{$Tr,N}, Y::StridedArray{$Tr,N},
region, flags::Unsigned, timelimit::Real,
bitreverse::Bool)
set_timelimit($Tr, timelimit)
dims, howmany = dims_howmany(X, Y, [size(X)...], region)
dims = hadamardize(dims, bitreverse)
kind = Array{Int32}(size(dims,2))
kind[:] = R2HC
plan = ccall(($(string(fftw,"_plan_guru64_r2r")),$lib),
PlanPtr,
(Int32, Ptr{Int}, Int32, Ptr{Int},
Ptr{$Tr}, Ptr{$Tr}, Ptr{Int32}, UInt32),
size(dims,2), dims, size(howmany,2), howmany,
X, Y, kind, flags)
set_timelimit($Tr, NO_TIMELIMIT)
if plan == C_NULL
error("FFTW could not create plan") # shouldn't normally happen
end
return r2rFFTWPlan{$Tr,(map(Int,kind)...),X===Y,N}(plan, flags, region, X, Y)
end
end
In the above code what is the plan variable, how is it used, and where can I find its implementation?
What are the inputs in the curly braces for the below line?
cFFTWPlan{$Tc,FFTW.FORWARD,X===Y,N}
This is constructing an FFTW "plan" to perform a multidimensional FFT. The cFFTWPlan type is a wrapper around the C fftw_plan pointer, and is implemented in the FFTW.jl module. The arguments in curly braces are Julia type parameters: in this case, indicating the number type (Tc), the FFTW transform direction FORWARD, whether the transform is in-place (X===Y), and the dimensionality of the transform (N). There are two methods here, one for an FWHT of complex-number data that creates a cFFTWPlan (which calls fftw_plan_guru_dft) and one for real-number data that creates an r2rFFTWPlan (which calls fftw_plan_guru_r2r). (These internal types of FFTW.jl are undocumented. The low-level C calls directly to the FFTW library are documented in the FFTW manual.
It should, in principle, be possible to make similar calls to FFTW for NumPy arrays. However, the existing pyFFTW wrappers don't seem to support FFTW's r2r transforms (needed for FWHTs of real data), so you'd have to add that.
Or you could call the Julia Hadamard.jl module from Python via the pyjulia package. Or you could use some other Python FWHT package, like https://github.com/FALCONN-LIB/FFHT
So here is the setting. I have multiple composite types defined with their own fields and constructors. Lets show two simplified components here:
type component1
x
y
end
type component2
x
y
z
end
Now I want to define a new type such that It can save an array of size K of previously defined composite types in it. So it is a parametric composite type with two fields: one is an integer K, and the other is an array of size K of the type passed.
type mixture{T}
components::Array{T, 1}
K::Int64
function mixture(qq::T, K::Int64)
components = Array{typeof(qq), K}
for k in 1:K
components[k] = qq
end
new(components, K)
end
end
But this is not the correct way to do it. Because all the K components are referring to one object and manipulating mixture.components[k] will affect all K components. In python I can remedy this with deepcopy. But the deepcopy in Julia is not defined for composite types. How do I solve this problem?
An answer to your specific question:
When you define a new type in Julia, it is common to extend some of the standard methods in Base to your new type, including deepcopy. For example:
type MyType
x::Vector
y::Vector
end
import Base.deepcopy
Base.deepcopy(m::MyType) = MyType(deepcopy(m.x), deepcopy(m.y))
Now you can call deepcopy over an instance of MyType and you will get a new, truly independent, copy of MyType as the output.
Note, my import Base.deepcopy is actually redundant, since I've referenced Base in my function definition, e.g. Base.deepcopy(m::MyType). However, I did both of these to show you the two ways of extending a method from Base.
Second note, if your type has lots of fields, you might instead iterate over the fields using deepcopy as follows:
Base.deepcopy(m::MyType) = MyType([ deepcopy(getfield(m, k)) for k = 1:length(names(m)) ]...)
A comment on your code:
First, it is standard practice in Julia to capitalize type names, e.g. Component1 instead of component1. Of course, you don't have to do this, but...
Second, from the Julia docs performance tips: Declare specific types for fields of composite types. Note, you can parameterize these declarations, e.g.
type Component1{T1, T2}
x::T1
y::T2
end
Third, here is how I would have defined your new type:
type Mixture{T}
components::Vector{T}
Mixture{T}(c::Vector{T}) = new(c)
end
Mixture{T}(c::Vector{T}) = Mixture{eltype(c)}(c)
Mixture(x, K::Int) = Mixture([ deepcopy(x) for k = 1:K ])
There are several important differences here between my code and yours. I'll go through them one at a time.
Your K field was redundant (I think) because it appears to just be the length of components. So it might be simpler to just extend the length method to your new type as follows:
Base.length(m::Mixture) = length(m.components)
and now you can use length(m), where m is an instance of Mixture to get what was previously stored in the K field.
The inner constructor in your type mixture was unusual. Standard practice is for the inner constructor to take arguments that correspond one-to-one (in sequence) to the fields of your type, and then the remainder of the inner constructor just performs any error checks you would like to be done whenever initialising your type. You deviated from this since qq was not (necessarily) an array. This kind of behaviour is better reserved for outer constructors. So, what have I done with constructors?
The inner constructor of Mixture doesn't really do anything other than pass the argument into the field via new. This is because currently there aren't any error checks I need to perform (but I can easily add some in the future).
If I want to call this inner constructor, I need to write something like m = Mixture{MyType}(x), where x is Vector{MyType}. This is a bit annoying. So my first outer constructor automatically infers the contents of {...} using eltype(x). Because of my first outer constructor, I can now initialise Mixture using m = Mixture(x) instead of m = Mixture{MyType}(x)
My second outer constructor corresponds to your inner constructor. It seems to me the behaviour you are after here is to initialise Mixture with the same component in every field of components, repeated K times. So I do this with a loop comprehension over x, which will work as long as the deepcopy method has been defined for x. If no deepcopy method exists, you'll get a No Method Exists error. This kind of programming is called duck-typing, and in Julia there is typically no performance penalty to using it.
Note, my second outer constructor will call my first outer constructor K times, and each time, my first outer constructor will call my inner constructor. Nesting functionality in this way will, in more complicated scenarios, massively cut-down on code duplication.
Sorry, this is a lot to take in I know. Hope it helps.
I've got a symbol that represents the name of a function to be called:
julia> func_sym = :tanh
I can use that symbol to get the tanh function and call it using:
julia> eval(func_sym)(2)
0.9640275800758169
But I'd rather avoid the 'eval' there as it will be called many times and it's expensive (and func_sym can have several different values depending on context).
IIRC in Ruby you can say something like:
obj.send(func_sym, args)
Is there something similar in Julia?
EDIT: some more details on why I have functions represented by symbols:
I have a type (from a neural network) that includes the activation function, originally I included it as a funcion:
type NeuralLayer
weights::Matrix{Float32}
biases::Vector{Float32}
a_func::Function
end
However, I needed to serialize these things to files using JLD, but it's not possible to serialize a Function, so I went with a symbol:
type NeuralLayer
weights::Matrix{Float32}
biases::Vector{Float32}
a_func::Symbol
end
And currently I use the eval approach above to call the activation function. There are collections of NeuralLayers and each can have it's own activation function.
#Isaiah's answer is spot-on; perhaps even more-so after the edit to the original question. To elaborate and make this more specific to your case: I'd change your NeuralLayer type to be parametric:
type NeuralLayer{func_type}
weights::Matrix{Float32}
biases::Vector{Float32}
end
Since func_type doesn't appear in the types of the fields, the constructor will require you to explicitly specify it: layer = NeuralLayer{:excitatory}(w, b). One restriction here is that you cannot modify a type parameter.
Now, func_type could be a symbol (like you're doing now) or it could be a more functionally relevant parameter (or parameters) that tunes your activation function. Then you define your activation functions like this:
# If you define your NeuralLayer with just one parameter:
activation(layer::NeuralLayer{:inhibitory}) = …
activation(layer::NeuralLayer{:excitatory}) = …
# Or if you want to use several physiological parameters instead:
activation{g_K,g_Na,g_l}(layer::NeuralLayer{g_K,g_Na,g_l} = f(g_K, g_Na, g_l)
The key point is that functions and behavior are external to the data. Use type definitions and abstract type hierarchies to define behavior, as is coded in the external functions… but only store data itself in the types. This is dramatically different from Python or other strongly object-oriented paradigms, and it takes some getting used to.
But I'd rather avoid the 'eval' there as it will be called many times and it's expensive (and func_sym can have several different values depending on context).
This sort of dynamic dispatch is possible in Julia, but not recommended. Changing the value of 'func_sym' based on context defeats type inference as well as method specialization and inlining. Instead, the recommended approach is to use multiple dispatch, as detailed in the Methods section of the manual.