Is the objective function in Lagrange multipliers a convex function?
personally, I guess not. Coz the dual function of the primal function must be a concave function, that is why we feel hard to get the result when the objective function of the primal problem is not convex?
Is my understanding correct?
Any ideas?
Generally, there is no limitation on the objective function.
Consider the optimization problem:
maximize f(x, y) subject to g(x, y) = 0.
If f and g have continuous first partial derivatives, then we can define
L(x,y,lambda) = f(x,y)-lambda*g(x,y)
for more information, see Lagrange_multiplier in [wiki]
Related
I need to calculate Hessian of my function numerically using my gradient function (programmed by formula, not numeric). Packages like numDeriv or rootSolve calculate hessian using numerical gradient that do not satisfy my needs. The thing I need performed internally (no separate method I can call) in optim package, however the only method that handles my optimization task well implemented in nlopt package and passing its optimal value to optim in order to get hessian is too expensive for my program.
So I need some function that calculates Hessian using not numeric gradient (see for example these formulas https://neos-guide.org/content/difference-approximations). I can't make such a function as I don't understand how to choose a parameter h (increment), that my function is very sensitive to. Could I find such a function in R or retrieve it somehow from optim package? Or may someone at least explain how to choose error minimizing value for h so then I will post this function by myself?
If I understand you correctly, you should use numDeriv::jacobian(), which takes a vector-valued function and computes a matrix (the derivative of each element with respect to each input), and apply it to your analytical gradient function. jacobian() does use numerical approximation (Richardson extrapolation, to be precise), but I don't see any other way that you could get from a black-box gradient function to a Hessian?
You do need to specify (or use the default value) of a numerical 'delta' function (1e-4 by default). On the other hand, the internal code that optim() uses to compute the Hessian also uses finite differences: see here and here ...
Below I define a function, its gradient function, and its Hessian; this code shows that jacobian(grad(x)) is the same as the Hessian (for a particular test case).
library(numDeriv)
x1 <- c(1,1,1)
Test that I didn't screw up the gradient and Hessian functions:
all.equal(grad(f,x1),g(x1)) ## TRUE
all.equal(h(x1),hessian(f,x1)) ## TRUE
Numerical equivalence of my Hessian and the Jacobian of the gradient:
all.equal(h(x1),jacobian(g,x1)) ## TRUE
Test functions:
f <- function(x) {
sin(x[1])*exp(x[2])*x[3]^2
}
g <- function(x) {
c(cos(x[1])*exp(x[2])*x[3]^2,
sin(x[1])*exp(x[2])*x[3]^2,
sin(x[1])*exp(x[2])*2*x[3])
}
h <- function(x) {
m <- matrix(NA,3,3)
m[lower.tri(m,diag=TRUE)] <-
c(-sin(x[1])*exp(x[2])*x[3]^2,
cos(x[1])*exp(x[2])*x[3]^2,
cos(x[1])*exp(x[2])*2*x[3],
# col 2
sin(x[1])*exp(x[2])*x[3]^2,
sin(x[1])*exp(x[3])*2*x[3],
# col 3
sin(x[1])*exp(x[2])*2)
m <- Matrix::forceSymmetric(m,"L")
m <- unname(as.matrix(m))
return(m)
}
I am trying to solve the following inequality constraint:
Given time-series data for N stocks, I am trying to construct a portfolio weight vector to minimize the variance of the returns.
the objective function:
min w^{T}\sum w
s.t. e_{n}^{T}w=1
\left \| w \right \|\leq C
where w is the vector of weights, \sum is the covariance matrix, e_{n}^{T} is a vector of ones, C is a constant. Where the second constraint (\left \| w \right \|) is an inequality constraint (2-norm of the weights).
I tried using the nloptr() function but it gives me an error: Incorrect algorithm supplied. I'm not sure how to select the correct algorithm and I'm also not sure if this is the right method of solving this inequality constraint.
I am also open to using other functions as long as they solve this constraint.
Here is my attempted solution:
data <- replicate(4,rnorm(100))
N <- 4
fn<-function(x) {cov.Rt<-cov(data); return(as.numeric(t(x) %*%cov.Rt%*%x))}
eqn<-function(x){one.vec<-matrix(1,ncol=N,nrow=1); return(-1+as.numeric(one.vec%*%x))}
C <- 1.5
ineq<-function(x){
z1<- t(x) %*% x
return(as.numeric(z1-C))
}
uh <-rep(C^2,N)
lb<- rep(0,N)
x0 <- rep(1,N)
local_opts <- list("algorithm"="NLOPT_LN_AUGLAG,",xtol_rel=1.0e-7)
opts <- list("algorithm"="NLOPT_LN_AUGLAG,",
"xtol_rel"=1.0e-8,local_opts=local_opts)
sol1<-nloptr(x0,eval_f=fn,eval_g_eq=eqn, eval_g_ineq=ineq,ub=uh,lb=lb,opts=opts)
This looks like a simple QP (Quadratic Programming) problem. It may be easier to use a QP solver instead of a general purpose NLP (NonLinear Programming) solver (no need for derivatives, functions etc.). R has a QP solver called quadprog. It is not totally trivial to setup a problem for quadprog, but here is a very similar portfolio example with complete R code to show how to solve this. It has the same objective (minimize risk), the same budget constraint and the lower and upper-bounds. The example just has an extra constraint that specifies a minimum required portfolio return.
Actually I misread the question: the second constraint is ||x|| <= C. I think we can express the whole model as:
This actually looks like a convex model. I could solve it with "big" solvers like Cplex,Gurobi and Mosek. These solvers support convex Quadratically Constrained problems. I also believe this can be formulated as a cone programming problem, opening up more possibilities.
Here is an example where I use package cccp in R. cccp stands for
Cone Constrained Convex Problems and is a port of CVXOPT.
The 2-norm of weights doesn't make sense. It has to be the 1-norm. This is essentially a constraint on the leverage of the portfolio. 1-norm(w) <= 1.6 implies that the portfolio is at most 130/30 (Sorry for using finance language here). You want to read about quadratic cones though. w'COV w = w'L'Lw (Cholesky decomp) and hence w'Cov w = 2-Norm (Lw)^2. Hence you can introduce the linear constraint y - Lw = 0 and t >= 2-Norm(Lw) [This defines a quadratic cone). Now you minimize t. The 1-norm can also be replaced by cones as abs(x_i) = sqrt(x_i^2) = 2-norm(x_i). So introduce a quadratic cone for each element of the vector x.
I need to optimize a function, say g(x), where x is vector and g is an analytic function.
The problem is that I need to optimize with a constraint (analytic) function c(x) which gives a scalar as an output, i.e the constraint is c(x) > k for some k > 0.
constrOptim only allows to give a constrain for every field separately.
Advices?
Found the right tool - nloptr package. A very robust package where I can define functions for the optimization (g) and constraint (c). Also I can define upper and lower bounds for each of the variables separately, and use different king of optimization methods.
I am trying to solve the following inequality constraint:
Given time-series data for N stocks, I am trying to construct a portfolio weight vector to minimize the variance of the returns.
the objective function:
min w^{T}\sum w
s.t. e_{n}^{T}w=1
\left \| w \right \|\leq C
where w is the vector of weights, \sum is the covariance matrix, e_{n}^{T} is a vector of ones, C is a constant. Where the second constraint (\left \| w \right \|) is an inequality constraint (2-norm of the weights).
I tried using the nloptr() function but it gives me an error: Incorrect algorithm supplied. I'm not sure how to select the correct algorithm and I'm also not sure if this is the right method of solving this inequality constraint.
I am also open to using other functions as long as they solve this constraint.
Here is my attempted solution:
data <- replicate(4,rnorm(100))
N <- 4
fn<-function(x) {cov.Rt<-cov(data); return(as.numeric(t(x) %*%cov.Rt%*%x))}
eqn<-function(x){one.vec<-matrix(1,ncol=N,nrow=1); return(-1+as.numeric(one.vec%*%x))}
C <- 1.5
ineq<-function(x){
z1<- t(x) %*% x
return(as.numeric(z1-C))
}
uh <-rep(C^2,N)
lb<- rep(0,N)
x0 <- rep(1,N)
local_opts <- list("algorithm"="NLOPT_LN_AUGLAG,",xtol_rel=1.0e-7)
opts <- list("algorithm"="NLOPT_LN_AUGLAG,",
"xtol_rel"=1.0e-8,local_opts=local_opts)
sol1<-nloptr(x0,eval_f=fn,eval_g_eq=eqn, eval_g_ineq=ineq,ub=uh,lb=lb,opts=opts)
This looks like a simple QP (Quadratic Programming) problem. It may be easier to use a QP solver instead of a general purpose NLP (NonLinear Programming) solver (no need for derivatives, functions etc.). R has a QP solver called quadprog. It is not totally trivial to setup a problem for quadprog, but here is a very similar portfolio example with complete R code to show how to solve this. It has the same objective (minimize risk), the same budget constraint and the lower and upper-bounds. The example just has an extra constraint that specifies a minimum required portfolio return.
Actually I misread the question: the second constraint is ||x|| <= C. I think we can express the whole model as:
This actually looks like a convex model. I could solve it with "big" solvers like Cplex,Gurobi and Mosek. These solvers support convex Quadratically Constrained problems. I also believe this can be formulated as a cone programming problem, opening up more possibilities.
Here is an example where I use package cccp in R. cccp stands for
Cone Constrained Convex Problems and is a port of CVXOPT.
The 2-norm of weights doesn't make sense. It has to be the 1-norm. This is essentially a constraint on the leverage of the portfolio. 1-norm(w) <= 1.6 implies that the portfolio is at most 130/30 (Sorry for using finance language here). You want to read about quadratic cones though. w'COV w = w'L'Lw (Cholesky decomp) and hence w'Cov w = 2-Norm (Lw)^2. Hence you can introduce the linear constraint y - Lw = 0 and t >= 2-Norm(Lw) [This defines a quadratic cone). Now you minimize t. The 1-norm can also be replaced by cones as abs(x_i) = sqrt(x_i^2) = 2-norm(x_i). So introduce a quadratic cone for each element of the vector x.
General problem:
I have a function that takes another function as an argument, of form:
F <- function(x, FUN){FUN(x)}
I could easily pass a simple function to it:
f1 <- function(x){plot(x, 1/x)}
F(-5:5, f1)
would display a plot of 1/x.
Supposing I had another, more complex function:
f2 <- function(x, a){plot(x, 1/x^a)}
f2 has 2 arguments, so can't be passed directly to F. But I might want to retain the flexibility in a so that, without defining lots of different functions, I can quickly plot 1/x^a for whatever value of a I fancy. I've tried, for a = 2:
F(-5:5, f2(, 2))
F(-5:5, f2(, a=2))
F(-5:5, f2(x, 2))
F(-5:5, f2(a=2))
But none of these work. Does anyone have a solution? (I could set a default for a in f2, but then I could not run it with a different value of a).
Specific context:
I have a function that will find the inverse Laplace Transform of a function, taking a function as its argument which is expected to have one argument (the Laplace variable, p). I can invert a function like f1 above. But I am trying to invert a function for contaminant transport in groundwater. This process depends upon lots of other parameters such as the water velocity and the distance being travelled. So I would like to be able to pass a multi-parameter function for Laplace inversion in such a way that all parameters apart from the Laplace parameter p is fixed. Ultimately I would like to do this process many times with different values for velocity etc., so I need a fluid way to change the "fixed" parameters being used.
Thanks in advance for any help,
Christopher
Just define a generator of function:
genFunc = function(a)
{
function(x) plot(x, 1/x^a)
}
F(-5:5, genFunc(2))
Or use Curry from functional package to fix parameters you want and spice your meals:
library(functional)
F(-5:5, Curry(f2, a=2))