I'm using DifferentialEquations.jl to solve an ODE system as shown below. The result is not really relevant since p only contains test parameters for the purpose of producing a MWE, but the key is that I am seeing a lot of memory allocation despite using an in-place ODE function.
using DifferentialEquations
function ode_fun!(du,u,p,t)
a,b,c,d,e = p
X = #. u[1] * a * ((b-c)/b)
Y = #. u[2] * d * ((b-e)/b)
du[1] = -sum(X) + sum(Y) - u[1]*u[2]
du[2] = sum(X) - sum(Y) - u[1]*u[2]
end
#exemplary parameters
a = collect(10:-0.1:0.1)
b = a.^2
c = b*0.7
d = collect(0.01:0.01:1)
e = b*0.3
u0 = [1.0, 0.5]
p = [a,b,c,d,e]
tspan = [0.0, 100.0]
t = collect(0:0.01:100)
prob = ODEProblem(ode_fun!,u0,tspan,p,saveat=t)
#time sol = solve(prob)
1.837609 seconds (5.17 M allocations: 240.331 MiB, 2.31% gc time) #Julia 1.5.2
Since I need to solve this ODE system repeatedly I would like to reduce allocations as much as possible and am wondering if there is anything that can be done about them. I have been wondering if the issue lies with X and Y and have tried to preallocate these outside the ODE function, but have unfortunately not succeeded in reducing allocations that way.
I'm pretty sure this should be faster and half the allocations
function ode_fun!(du,u,p,t)
a,b,c,d,e = p
XmY = #. u[1] * a * (1-c/b) - u[2] * d * (1-e/b)
sXmY = sum(XmY)
du[1] = -sXmY - u[1]*u[2]
du[2] = sXmY - u[1]*u[2]
end
There's probably a way to get rid of all of them, but I'm not a DifferentialEquations expert.
Related
I am solving a differential equation in Julia the corresponding equations are given in the code below. Now by solving the differential equation what I am getting is the two variables only. But for my work, I also want the derivatives of the variables in each time step too after the integration is done, in which I am unable to proceed further.
using DelimitedFiles
using LightGraphs
using LinearAlgebra
using Random
using PyPlot
using BenchmarkTools
using SparseArrays
const N= 100;
Adj=readdlm("sf_simplicial_100.txt")
G=Graph(Adj)
global A = adjacency_matrix(G)
global deg=degree(G)
global omega=deg
mean(deg)
A2=zeros(N,N,N);
for i in 1:N
for j in 1:N
for k in 1:N
if (A[i,j]==1 && A[j,k]==1 && A[k,i]==1)
A2[i,j,k]=1
A2[i,k,j]=1
A2[j,k,i]=1
A2[j,i,k]=1
A2[k,i,j]=1
A2[k,j,i]=1
end;
end;
end;
end;
K= sum(p -> A2[:,:,p], 1:N)
deg_sim= sum(j -> K[:,j], 1:N)/2;
deg_sim2=2*deg_sim;
function kuramoto(du,u, pp, t)
u1 = #view u[1:N] #### θ
u2 = #view u[N+1:2*N] ####### λ
du1 = #view du[1:N] #### dθ
du2 = #view du[N+1:2*N] ####### dλ
α1=0.08
β1=0.04
σ1=1.0
σ2=1.0
λ0=pp
####### local_order
z1 = Array{Complex{Float64},1}(undef, N)
mul!(z1, A, exp.((u1)im))
z1 = z1 ./ deg
####### generalized_local_order
z2 = Array{Complex{Float64},1}(undef, N)
z2= (diag(A*Diagonal(exp.((u1)im))*A*Diagonal(exp.((u1)im))*A))
z2 = z2 ./ deg_sim2
####### equ of motion
#. du1 = omega + u2 *( σ1 * deg * imag(z1 * exp((-1im) * u1)) + σ2 * deg_sim * imag(z2 * exp((-1im) * 2*u1)))
#. du2 = α1 *(λ0-u2)- β1 * (abs(z1)+ abs(z2))/2.0
return nothing
end;
using DifferentialEquations
# setting up time steps and integration intervals
dt = 0.01 # time step
dts = 0.1 # save time
ti = 0.0
tt = 1000.0
tf = 5000.0
nt = Int(div(tt,dts))
nf = Int(div(tf,dts))
tspan = (ti, tf); # time interval
pp=0.75
ini=readdlm("N=100/initial_condition.txt")
u0=[ini;pp*ones(N)];
du = similar(u0);
prob = ODEProblem(kuramoto,u0, tspan, pp)
sol = solve(prob, RK4(), reltol=1e-4, saveat=dts,maxiters=1e10,progress=true)
Use the derivative of the interpolation sol(t,Val{1}). To do this you'll want to not use saveat. Otherwise you can use the SavingCallback.
I have the following coupled system of ODEs (that come from discretizing an integrodifferential PDE):
The xi's are points on an x-grid that I control. I can solve this with the following simple piece of code:
using DifferentialEquations
function ode_syst(du,u,p, t)
N = Int64(p[1])
beta= p[2]
deltax = 1/(N+1)
xs = [deltax*i for i in 1:N]
for j in 1:N
du[j] = -xs[j]^(beta)*u[j]+deltax*sum([u[i]*xs[i]^(beta) for i in 1:N])
end
end
N = 1000
u0 = ones(N)
beta = 2.0
p = [N, beta]
tspan = (0.0, 10^3);
prob = ODEProblem(ode_syst,u0,tspan,p);
sol = solve(prob);
However, as I make my grid finer, i.e. increase N, the computation time grows rapidly (I guess the scaling is quadratic in N). Is there any suggestion on how to implement this using either distributed parallelism or multithreading?
Additional information: I attach the profiling diagram that might be useful to understand where the program spends most of the time
I've looked into your code and found a few problems such as an accidentally introduced O(N^2) behavior due to recalculating the sum term.
My improved version uses the Tullio package to get further speed up from vectorization. Tullio also has tuneable parameters which would allow for multi threading if your system becomes big enough. See here what parameters you can tune in the options section. You might also see GPU support there, i didn't test that but it might yield further speed up or break hooribly. I also choose get the length from the acutal array which should make the use more economical and less error prone.
using Tullio
function ode_syst_t(du,u,p, t)
N = length(du)
beta = p[1]
deltax = 1/(N+1)
#tullio s := deltax*(u[i]*(i*deltax)^(beta))
#tullio du[j] = -(j*deltax)^(beta)*u[j] + s
return nothing
end
Your code:
#btime sol = solve(prob);
80.592 s (1349001 allocations: 10.22 GiB)
My code:
prob2 = ODEProblem(ode_syst_t,u0,tspan,[2.0]);
#btime sol2 = solve(prob2);
1.171 s (696 allocations: 18.50 MiB)
and the result more or less agree:
julia> sum(abs2, sol2(1000.0) .- sol(1000.0))
1.079046922815598e-14
Lutz Lehmanns solution i also benchmarked:
prob3 = ODEProblem(ode_syst_lehm,u0,tspan,p);
#btime sol3 = solve(prob3);
1.338 s (3348 allocations: 39.38 MiB)
However as we scale N to 1000000 with a tspan of (0.0, 10.0)
prob2 = ODEProblem(ode_syst_t,u0,tspan,[2.0]);
#time solve(prob2);
2.429239 seconds (280 allocations: 869.768 MiB, 13.60% gc time)
prob3 = ODEProblem(ode_syst_lehm,u0,tspan,p);
#time solve(prob3);
5.961889 seconds (580 allocations: 1.967 GiB, 11.08% gc time)
My code becomes more than twice as fast due to using the 2 cores in my old and rusty machine.
Analyze the formula. Obviously, the atomic terms repeat. So they should only be computed once.
function ode_syst(du,u,p, t)
N = Int64(p[1])
beta= p[2]
deltax = 1/(N+1)
xs = [deltax*i for i in 1:N]
term = [ xs[i]^(beta)*u[i] for i in 1:N]
term_sum = deltax*sum(term)
for j in 1:N
du[j] = -term[j]+term_sum
end
end
This should only linearly increase in N.
I wrote a main function which uses a stochastic optimization algorithm (Particle Swarm Optimization) to found optimal solution for a ODE system. I would run 50 times to make sure the optimal can be found. At first, it operates normally, but now I found the calculation time would increase with iteration increases.
It cost less than 300s for first ten calculations, but it would increase to 500s for final calculation. It seems that it would cost 3~5 seconds more for each calculation. I have followed the high performance tips to optimize my code but it doesn't work.
I am sorry I don't know quite well how to upload my code before, here is the code I wrote below. But in this code, the experimental data is not loaded, I may need to find a way to upload data. In main function, with the increase of i, the time cost is increasing for each calculation.
Oh, by the way, I found another interesting phenomenon. I changed the number of calculations and the calculation time changed again. For first 20 calculations in main loop, each calculation cost about 300s, and the memory useage fluctuates significantly. But something I don't know happend, it is speeding up. It cost 1/4 time less time for each calculation, which is about 80s. And the memory useage became a straight line like this:
I knew the Julia would do pre-heating for first run and then speed up. But this situation seems different. This situation looks like Julia run slowly for first 20 calculation, and then it found a good way to optimize the memory useage and speed up. Then the program just run at full speed.
using CSV, DataFrames
using BenchmarkTools
using DifferentialEquations
using Statistics
using Dates
using Base.Threads
using Suppressor
function uniform(dim::Int, lb::Array{Float64, 1}, ub::Array{Float64, 1})
arr = rand(Float64, dim)
#inbounds for i in 1:dim; arr[i] = arr[i] * (ub[i] - lb[i]) + lb[i] end
return arr
end
mutable struct Problem
cost_func
dim::Int
lb::Array{Float64,1}
ub::Array{Float64,1}
end
mutable struct Particle
position::Array{Float64,1}
velocity::Array{Float64,1}
cost::Float64
best_position::Array{Float64,1}
best_cost::Float64
end
mutable struct Gbest
position::Array{Float64,1}
cost::Float64
end
function PSO(problem, data_dict; max_iter=100,population=100,c1=1.4962,c2=1.4962,w=0.7298,wdamp=1.0)
dim = problem.dim
lb = problem.lb
ub = problem.ub
cost_func = problem.cost_func
gbest, particles = initialize_particles(problem, population, data_dict)
# main loop
for iter in 1:max_iter
#threads for i in 1:population
particles[i].velocity .= w .* particles[i].velocity .+
c1 .* rand(dim) .* (particles[i].best_position .- particles[i].position) .+
c2 .* rand(dim) .* (gbest.position .- particles[i].position)
particles[i].position .= particles[i].position .+ particles[i].velocity
particles[i].position .= max.(particles[i].position, lb)
particles[i].position .= min.(particles[i].position, ub)
particles[i].cost = cost_func(particles[i].position,data_dict)
if particles[i].cost < particles[i].best_cost
particles[i].best_position = copy(particles[i].position)
particles[i].best_cost = copy(particles[i].cost)
if particles[i].best_cost < gbest.cost
gbest.position = copy(particles[i].best_position)
gbest.cost = copy(particles[i].best_cost)
end
end
end
w = w * wdamp
if iter % 50 == 1
println("Iteration " * string(iter) * ": Best Cost = " * string(gbest.cost))
println("Best Position = " * string(gbest.position))
println()
end
end
gbest, particles
end
function initialize_particles(problem, population,data_dict)
dim = problem.dim
lb = problem.lb
ub = problem.ub
cost_func = problem.cost_func
gbest_position = uniform(dim, lb, ub)
gbest = Gbest(gbest_position, cost_func(gbest_position,data_dict))
particles = []
for i in 1:population
position = uniform(dim, lb, ub)
velocity = zeros(dim)
cost = cost_func(position,data_dict)
best_position = copy(position)
best_cost = copy(cost)
push!(particles, Particle(position, velocity, cost, best_position, best_cost))
if best_cost < gbest.cost
gbest.position = copy(best_position)
gbest.cost = copy(best_cost)
end
end
return gbest, particles
end
function get_dict_label(beta::Int)
beta_str = lpad(beta,2,"0")
T_label = "Temperature_" * beta_str
M_label = "Mass_" * beta_str
MLR_label = "MLR_" * beta_str
return T_label, M_label, MLR_label
end
function get_error(x::Vector{Float64}, y::Vector{Float64})
numerator = sum((x.-y).^2)
denominator = var(x) * length(x)
numerator/denominator
end
function central_diff(x::AbstractArray{Float64}, y::AbstractArray{Float64})
# Central difference quotient
dydx = Vector{Float64}(undef, length(x))
dydx[2:end] .= diff(y) ./ diff(x)
#views dydx[2:end-1] .= (dydx[2:end-1] .+ dydx[3:end])./2
# Forward and Backward difference
dydx[1] = (y[2]-y[1])/(x[2]-x[1])
dydx[end] = (y[end]-y[end-1])/(x[end]-x[end-1])
return dydx
end
function decomposition!(dm,m,p,T)
# A-> residue + volitale
# B-> residue + volatile
beta,A1,E1,n1,k1,A2,E2,n2,k2,m1,m2 = p
R = 8.314
rxn1 = -m1 * exp(A1-E1/R/T) * max(m[1]/m1,0)^n1 / beta
rxn2 = -m2 * exp(A2-E2/R/T) * max(m[2]/m2,0)^n2 / beta
dm[1] = rxn1
dm[2] = rxn2
dm[3] = -k1 * rxn1 - k2 * rxn2
dm[4] = dm[1] + dm[2] + dm[3]
end
function read_file(file_path)
df = CSV.read(file_path, DataFrame)
data_dict = Dict{String, Vector{Float64}}()
for beta in 5:5:21
T_label, M_label, MLR_label = get_dict_label(beta)
T_data = collect(skipmissing(df[:, T_label]))
M_data = collect(skipmissing(df[:, M_label]))
T = T_data[T_data .< 780]
M = M_data[T_data .< 780]
data_dict[T_label] = T
data_dict[M_label] = M
data_dict[MLR_label] = central_diff(T, M)
end
return data_dict
end
function initial_condition(beta::Int64, ode_parameters::Array{Float64,1})
m_FR_initial = ode_parameters[end]
m_PVC_initial = 1 - m_FR_initial
T_span = (300.0, 800.0) # temperature range
p = [beta; ode_parameters; m_PVC_initial]
m0 = [p[end-1], p[end], 0.0, 1.0] # initial mass
return m0, T_span, p
end
function cost_func(ode_parameters, data_dict)
total_error = 0.0
for beta in 5:5:21
T_label, M_label, MLR_label= get_dict_label(beta)
T = data_dict[T_label]::Vector{Float64}
M = data_dict[M_label]::Vector{Float64}
MLR = data_dict[MLR_label]::Vector{Float64}
m0, T_span, p = initial_condition(beta,ode_parameters)
prob = ODEProblem(decomposition!,m0,T_span,p)
sol = solve(prob, AutoVern9(Rodas5(autodiff=false)),saveat=T,abstol=1e-8,reltol=1e-8,maxiters=1e4)
if sol.retcode != :Success
# println(1)
return Inf
else
M_sol = #view sol[end, :]
MLR_sol = central_diff(T, M_sol)::Array{Float64,1}
error1 = get_error(MLR, MLR_sol)::Float64
error2 = get_error(M, M_sol)::Float64
total_error += error1 + error2
end
end
total_error
end
function main()
flush(stdout)
total_time = 0
best_costs = []
file_path = raw"F:\17-Fabric\17-Fabric (Smoothed) TG.csv"
data_dict = read_file(file_path)
dimension = 9
lb = [5, 47450, 0.0, 0.0, 24.36, 148010, 0.0, 0.0, 1e-5]
ub = [25.79, 167700, 5, 1, 58.95, 293890, 5, 1, 0.25]
problem = Problem(cost_func,dimension,lb,ub)
global_best_cost = Inf
println("-"^100)
println("Running PSO ...")
population = 50
max_iter = 1001
println("The population is: ", population)
println("Max iteration is:", max_iter)
for i in 1:50 # The number of calculation
start_time = Dates.now()
println("Current iteration is: ", string(i))
gbest, particles = PSO(problem, data_dict, max_iter=max_iter, population=population)
if gbest.cost < global_best_cost
global_best_cost = gbest.cost
global_best_position = gbest.position
end
end_time = Dates.now()
time_duration = round(end_time-start_time, Second)
total_time += time_duration.value
push!(best_costs, gbest.cost)
println()
println("The Best is:")
println(gbest.cost)
println(gbest.position)
println("The calculation time is: " * string(time_duration))
println()
println("-"^50)
end
println('-'^100)
println("Global best cost is: ", global_best_cost)
println("Global best position is: ", global_best_position)
println(total_time)
best_costs
end
#suppress_err begin
#time global best_costs = main()
end
So, what is the possible mechanism for this? Is there a way to avoid this problem? Because If I increase the population and max iterations of particles, the time increased would be extremely large and thus is unacceptable.
And what is the possible mechanism for speed up the program I mentioned above? How to trigger this mechanism?
As the parameters of an ODE optimizes it can completely change its characteristics. Your equation could be getting more stiff and require different ODE solvers. There are many other related ways, but you can see how changing parameters could give such a performance issue. It's best to use methods like AutoTsit5(Rodas5()) and the like in such estimation cases because it's hard to know or guess what the performance will be like, and thus adaptiveness in the method choice can be crucial.
I would like to minimize a distance function ||dz - z|| under the constraint that g(z) = 0.
I wanted to use Lagrange Multipliers to solve this problem. Then I used NLsolve.jl to solve the non-linear equation that I end up with.
using NLsolve
using ForwardDiff
function ProjLagrange(dz, g::Function)
λ_init = ones(size(g(dz...),1))
initial_x = vcat(dz, λ_init)
function gradL!(F, x)
len_dz = length(dz)
z = x[1:len_dz]
λ = x[len_dz+1:end]
F = Array{Float64}(undef, length(x))
my_distance(z) = norm(dz - z)
∇f = z -> ForwardDiff.gradient(my_distance, z)
F[1:len_dz] = ∇f(z) .- dot(λ, g(z...))
if length(λ) == 1
F[end] = g(z...)
else
F[len_dz+1:end] = g(z)
end
end
nlsolve(gradL!, initial_x)
end
g_test(x1, x2, x3) = x1^2 + x2 - x2 + 5
z = [1000,1,1]
ProjLagrange(z, g_test)
But I always end up with Zero: [NaN, NaN, NaN, NaN] and Convergence: false.
Just so you know I have already solved the equation by using Optim.jl and minimizing the following function: Proj(z) = b * sum(abs.(g(z))) + a * norm(dz - z).
But I would really like to know if this is possible with NLsolve. Any help is greatly appreciated!
Starting almost from scratch and wikipedia's Lagrange multiplier page because it was good for me, the code below seemed to work. I added an λ₀s argument to the ProjLagrange function so that it can accept a vector of initial multiplier λ values (I saw you initialized them at 1.0 but I thought this was more generic). (Note this has not been optimized for performance!)
using NLsolve, ForwardDiff, LinearAlgebra
function ProjLagrange(x₀, λ₀s, gs, n_it)
# distance function from x₀ and its gradients
f(x) = norm(x - x₀)
∇f(x) = ForwardDiff.gradient(f, x)
# gradients of the constraints
∇gs = [x -> ForwardDiff.gradient(g, x) for g in gs]
# Form the auxiliary function and its gradients
ℒ(x,λs) = f(x) - sum(λ * g(x) for (λ,g) in zip(λs,gs))
∂ℒ∂x(x,λs) = ∇f(x) - sum(λ * ∇g(x) for (λ,∇g) in zip(λs,∇gs))
∂ℒ∂λ(x,λs) = [g(x) for g in gs]
# as a function of a single argument
nx = length(x₀)
ℒ(v) = ℒ(v[1:nx], v[nx+1:end])
∇ℒ(v) = vcat(∂ℒ∂x(v[1:nx], v[nx+1:end]), ∂ℒ∂λ(v[1:nx], v[nx+1:end]))
# and solve
v₀ = vcat(x₀, λ₀s)
nlsolve(∇ℒ, v₀, iterations=n_it)
end
# test
gs_test = [x -> x[1]^2 + x[2] - x[3] + 5]
λ₀s_test = [1.0]
x₀_test = [1000.0, 1.0, 1.0]
n_it = 100
res = ProjLagrange(x₀_test, λ₀s_test, gs_test, n_it)
gives me
julia> res = ProjLagrange(x₀_test, λ₀s_test, gs_test, n_it)
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1000.0, 1.0, 1.0, 1.0]
* Zero: [9.800027199717013, -49.52026655749088, 51.520266557490885, -0.050887973682118504]
* Inf-norm of residuals: 0.000000
* Iterations: 10
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 11
* Jacobian Calls (df/dx): 11
I altered your code as below (see my comments in there) and got the following output. It doesn't throw NaNs anymore, reduces the objective and converges. Does this differ from your Optim.jl results?
Results of Nonlinear Solver Algorithm
* Algorithm: Trust-region with dogleg and autoscaling
* Starting Point: [1000.0, 1.0, 1.0, 1.0]
* Zero: [9.80003, -49.5203, 51.5203, -0.050888]
* Inf-norm of residuals: 0.000000
* Iterations: 10
* Convergence: true
* |x - x'| < 0.0e+00: false
* |f(x)| < 1.0e-08: true
* Function Calls (f): 11
* Jacobian Calls (df/dx): 11
using NLsolve
using ForwardDiff
using LinearAlgebra: norm, dot
using Plots
function ProjLagrange(dz, g::Function, n_it)
λ_init = ones(size(g(dz),1))
initial_x = vcat(dz, λ_init)
# These definitions can go outside as well
len_dz = length(dz)
my_distance = z -> norm(dz - z)
∇f = z -> ForwardDiff.gradient(my_distance, z)
# In fact, this is probably the most vital difference w.r.t. your proposal.
# We need the gradient of the constraints.
∇g = z -> ForwardDiff.gradient(g, z)
function gradL!(F, x)
z = x[1:len_dz]
λ = x[len_dz+1:end]
# `F` is memory allocated by NLsolve to store the residual of the
# respective call of `gradL!` and hence doesn't need to be allocated
# anew every time (or at all).
F[1:len_dz] = ∇f(z) .- λ .* ∇g(z)
F[len_dz+1:end] .= g(z)
end
return nlsolve(gradL!, initial_x, iterations=n_it, store_trace=true)
end
# Presumable here is something wrong: x2 - x2 is not very likely, also made it
# callable directly with an array argument
g_test = x -> x[1]^2 + x[2] - x[3] + 5
z = [1000,1,1]
n_it = 10000
res = ProjLagrange(z, g_test, n_it)
# Ugly reformatting here
trace = hcat([[state.iteration; state.fnorm; state.stepnorm] for state in res.trace.states]...)
plot(trace[1,:], trace[2,:], label="f(x) inf-norm", xlabel="steps")
Evolution of inf-norm of f(x) over iteration steps
[Edit: Adapted solution to incorporate correct gradient computation for g()]
I am trying to speedup the following code in R, which is calling uniroot on 1000 different cases using mapply (one for each equation form with each element of the vectors a, ce and w -- see below).
In order to run this code faster, I would like to use my GPU (a NVIDIA card) in order to parallelize the computation. Here is a simple example (of the actual computation):
a = exp(seq(log(1), log(401), length.out = 1000)) - 1
ce = runif(1000, min = 0.9, max = 4)
w = runif(1000, min = 0.9, max = 4)
fun1 = function(ce, w, a) {
return(uniroot(function(h) ce * 0.32 * (1 - h)^-0.35 - 0.8 * ((0.0093 + 0.01) / 0.33)^(0.33 / (0.33 - 1))*(1 - 0.04) * (w * h + 0.0093 * a)^-0.16, c(0.5, 0.99999), extendInt = "yes")$root)
}
out = mapply(fun1, ce = ce, w = w, a = a)
I read here that you can use Newton-Raphson algorithm with GPU, but I don't get how to do it. Can someone give me a hint (or a working example) on how to parallelize this?
Thank you in advance!
Note: I call this mapply function lots of time, so the actual time used by it is considerably.