Fom document of DifferentialEquations package, switching between sets of ODE functions can be done using a parameter as
function f(du,u,p,t)
if p==0
du[1] = 2u[1]
else
du[1] = - u[1]
end
du[2] = -u[2]
end
Is this possible to use dependent variable (state variable) instead of parameter p as the switch like
function f(du,u,p,t)
if (u[2]<=0 && du[2]>0)
du[1] = 2u[1]
else
du[1] = - u[1]
end
du[2] = -u[2]
end
Thank you in advance for your help.
Is this possible to use dependent variable (state variable) instead of parameter p as the switch like
Yes. It introduces a discontinuity, so it's not the best thing to do, but adaptivity will handle it. Sometimes performance can be improved by making a ContinuousCallback which rootfinds to that value as the condition, but then does nothing for the affect!. But yes, the code with the branch in it is fine.
Related
I'm trying to train a UNet in Julia with the help of Flux.
Flux.train!(loss, Flux.params(model), train_data_loader, opt)
batch_loss = loss(train_data, train_targets)
where the loss is
logitcrossentropy
and train_data_loader is
train_data_loader = DataLoader((train_data |> device, train_targets |> device), batchsize=batch_size, shuffle=true)
I dont understand how to take the loss from Flux.train out for printing loss (is that validation loss?). Evalcb will also trigger a call to calculate loss, so its not different. I was to skip extra calculation.
So What I did is call the loss function again and store it in a variable then print it per batch. Is there a way to print loss from Flux.train() instead of calling loss again?
Instead of altering train! like #Tomas suggested, the loss function can be instrumented to log the return value. Printing stuff during calculation sounds like a bad idea for decent performance, so I've made an example where the loss is logged into a global vector:
using ChainRulesCore
# returns another loss function which is the same as the function
# in parameter, but push!es the return value into global variable
# `loss_log_vec`
function logged_loss(lossfn, history)
return function _loss(args...)
err = lossfn(args...)
ignore_derivatives() do
push!(history, err)
end
return err
end
end
# initialize log vector
log_vec = Float32[]
# use function above to create logging loss function
newloss = logged_loss(loss, log_vec)
# run the training
Flux.train!(newloss, Flux.params(W, b), train_data, opt)
At this point, log_vec should include a record of return values from loss function. This is a rough solution, which uses annoying global variables. Interpreting the loss return values depends also on the nature of the optimizer. For my test, there was one call per epoch and it returned a decreasing loss until convergence. [This answer incorporates suggestions from #darsnack]
Note, since the log_vec is incorporated into the loss function, to clear the log, it must not be reassigned but clear!ed with clear!(log_vec).
Adding to #Dan's answer, you can also augment your loss function with logging on the fly using the do syntax:
using ChainRules
loss_history = Float32[]
Flux.train!(Flux.params(model), train_data_loader, opt) do x, y
err = loss(x, y)
ChainRules.ignore_derivatives() do
push!(loss_history, err)
end
return err
end
You would need to write your own version of Flux.train! using withgradient instead of gradient function. withgradient gives you the output of the loss (or a function which you are differentiating to be more precise). Flux.train! (https://github.com/FluxML/Flux.jl/blob/8bc0c35932c4a871ac73b42e39146cd9bbb1d446/src/optimise/train.jl#L123) is literaly few lines of code, therefore updating it to your version is very easy.
I am seeing that Julia explicitly does NOT do classes... and I should instead embrace mutable structs.. am I going down the correct path here?? I diffed my trivial example against an official flux library but cannot gather how do I reference self like a python object.. is the cleanest way to simply pass the type as a parameter in the function??
Python
# Dense Layer
class Layer_Dense
def __init__(self, n_inputs, n_neurons):
self.weights = 0.01 * np.random.randn(n_inputs, n_neurons)
self.biases = np.zeros((1, n_neurons))
def forward(self, inputs):
pass
My JuliaLang version so far
mutable struct LayerDense
num_inputs::Int64
num_neurons::Int64
weights
biases
end
function forward(layer::LayerDense, inputs)
layer.weights = 0.01 * randn(layer.num_inputs, layer.num_neurons)
layer.biases = zeros((1, layer.num_neurons))
end
The flux libraries version of a dense layer... which looks very different to me.. and I do not know what they're doing or why.. like where is the forward pass call, is it here in flux just named after the layer Dense???
source : https://github.com/FluxML/Flux.jl/blob/b78a27b01c9629099adb059a98657b995760b617/src/layers/basic.jl#L71-L111
struct Dense{F, M<:AbstractMatrix, B}
weight::M
bias::B
σ::F
function Dense(W::M, bias = true, σ::F = identity) where {M<:AbstractMatrix, F}
b = create_bias(W, bias, size(W,1))
new{F,M,typeof(b)}(W, b, σ)
end
end
function Dense(in::Integer, out::Integer, σ = identity;
initW = nothing, initb = nothing,
init = glorot_uniform, bias=true)
W = if initW !== nothing
Base.depwarn("keyword initW is deprecated, please use init (which similarly accepts a funtion like randn)", :Dense)
initW(out, in)
else
init(out, in)
end
b = if bias === true && initb !== nothing
Base.depwarn("keyword initb is deprecated, please simply supply the bias vector, bias=initb(out)", :Dense)
initb(out)
else
bias
end
return Dense(W, b, σ)
end
This is an equivalent of your Python code in Julia:
mutable struct Layer_Dense
weights::Matrix{Float64}
biases::Matrix{Float64}
Layer_Dense(n_inputs::Integer, n_neurons::Integer) =
new(0.01 * randn(n_inputs, n_neurons),
zeros((1, n_neurons)))
end
forward(ld::Layer_Dense, inputs) = nothing
What is important here:
here I create an inner constructor only, as outer constructor is not needed; as opposed in the Flux.jl code you have linked the Dense type defines both inner and outer constructors
in python forward function does not do anything, so I copied it in Julia (your Julia code worked a bit differently); note that instead of self one should pass an instance of the object to the function as the first argument (and add ::Layer_Dense type signature so that Julia knows how to correctly dispatch it)
similarly in Python you store only weights and biases in the class, I have reflected this in the Julia code; note, however, that for performance reasons it is better to provide an explicit type of these two fields of Layer_Dense struct
like where is the forward pass call
In the code you have shared only constructors of Dense object are defined. However, in the lines below here and here the Dense type is defined to be a functor.
Functors are explained here (in general) and in here (more specifically for your use case)
In Julia v1.01 I would like to create a function from a string.
Background: In a numerical solver, a testcase is defined via a JSON file. It would be great if the user could specify the initial condition in string form.
This results in the following situation: Assume we have (from the JSON file)
fcn_as_string = "sin.(2*pi*x)"
Is there a way to convert this into a function fcn such that I can call
fcn(1.0) # = sin.(2*pi*1.0)
Performance is not really an issue, as the initial condition is evaluated once and then the actual computation consumes most of the time.
Can't get my code displayed correctly in a comment so here's a quick fix for crstnbr's solution
function fcnFromString(s)
f = eval(Meta.parse("x -> " * s))
return x -> Base.invokelatest(f, x)
end
function main()
s = "sin.(2*pi*x)"
f = fcnFromString(s)
f(1.)
end
julia> main()
-2.4492935982947064e-16
The functions Meta.parse and eval allow you to do this:
fcn_as_string = "sin.(2*pi*x)"
fcn = eval(Meta.parse("x -> " * fcn_as_string))
#show fcn(1.0)
This return -2.4492935982947064e-16 (due to rounding errors).
I want to add a loss function to torch that calculates the edit distance between predicted and target values.
Is there an easy way to implement this idea?
Or do I have to write my own class with backward and forward functions?
If your criterion can be represented as a composition of existing modules and criteria, it's a good idea to simply construct such composition using containers. The only problem is that standard containers are designed to work with modules only, not criteria. The difference is in :forward method signature:
module:forward(input)
criterion:forward(input, target)
Luckily, we are free to define our own container which is able work with criteria too. For example, sequential:
local GeneralizedSequential, _ = torch.class('nn.GeneralizedSequential', 'nn.Sequential')
function GeneralizedSequential:forward(input, target)
return self:updateOutput(input, target)
end
function GeneralizedSequential:updateOutput(input, target)
local currentOutput = input
for i=1,#self.modules do
currentOutput = self.modules[i]:updateOutput(currentOutput, target)
end
self.output = currentOutput
return currentOutput
end
Below is an illustration of how to implement nn.CrossEntropyCriterion having this generalized sequential container:
function MyCrossEntropyCriterion(weights)
criterion = nn.GeneralizedSequential()
criterion:add(nn.LogSoftMax())
criterion:add(nn.ClassNLLCriterion(weights))
return criterion
end
Check whether everything is correct:
output = torch.rand(3,3)
target = torch.Tensor({1, 2, 3})
mycrit = MyCrossEntropyCriterion()
-- print(mycrit)
print(mycrit:forward(output, target))
print(mycrit:backward(output, target))
crit = nn.CrossEntropyCriterion()
-- print(crit)
print(crit:forward(output, target))
print(crit:backward(output, target))
Just to add to the accepted answer, you have to be careful that the loss function you define (edit distance in your case) is differentiable with respect to the network parameters.
Below is the code for my program. I'm attempting to find the value of the integral of 1/ln(x), and then evaluate the integral from 0 to x, with this as the integrand. I'm not exactly sure what I'm doing wrong, but I am quite new to Scilab.
t = input("t");
x=10; while x<t, x=x+10,
function y=f(x), y=(1/(log (x))), endfunction
I=intg(2,x,f);
function z=g(x), z=I, endfunction
W = intg(0,x,z);
W
end
I'm not entirely sure on what you are trying to achieve, but I reformatted your code and added some suggestions to documentation.
Maybe it will help you in finding the answer.
While loop
You can convert your while loop to a for loop
Your code
x=10;
while x<t
x=x+10
//some code
end
Could be
for x=10:10:t
//some code
end
Functions
In your code, you redeclare the two functions every single iteration of the while loop. You could declare them outside the while loop and call them inside the loop.
Reformatted
t = input("Please provide t: ");
// The function of 1/ln(x)
function y=f(x), y=1/log(x), endfunction
// Every time g(x) is called the current value of I is returned
function z=g(x), z=I, endfunction
for x=10:10:t
//Find definite integral of function f from 2 to x
I = intg(2,x,f);
//Find definite integral of I from 0 to x
W = intg(0,x,g);
disp( string(W) );
end
I know the question is porbably outdated; but the topic is still active. And I was looking for a code with double integral.
Here, it looks strange to use "intg" just to calculate the area of the rectangle defined by its diagonal ((0,0), (x,I)): the result is just x*I...
May be the initial aim was to consider "I" as a function of "x" (but in this case there is a convergence problem at x=1...); so restricting the integration of "I" to something above 1 gives the following code:
x=10:10:100;W2=integrate('integrate(''1/log(x2)'',''x2'',2,x1)','x1',1.001,x);
Note the use of integration variables x1 and x2, plus the use of quotes...