I'm estimating a Phillips Curve Model - and, as such, need to take into account the unemployment Gap, which is the difference between actual unemployment and the Nairu (here, Nairu is the unobservable variable).
I need to impose the following constraint: some of the coefficients (say, beta 1 and beta 2) in the Z matrix (which relationates to the Nairu) must be the same that in the D matrix (which accounts for the unemployment). However, it doesn't seem possible to impose such a constraint simultaneously in the Z and D matrix.
Can you guys help me?
I've tried setting the same "names" for the coefficients in the Z and D matrix, but that didn't work.
Related
I'm trying to understand Cramer V and if it fits my data (after CHI squared returns a significant association between variables) and also looking for alternatives (I've heard a Spearman correlation might be better in some cases..?)
1st example: I have a yes/no response from users and there are 2 conditions (2x2 matrix)
library(rcompanion)
x <- matrix(c(144,120,66,90), nrow = 2)
colnames(x) <- c("No", "Yes")
rownames(x) <- c("Cond1", "Cond2")
cramerV(x, conf = 95)
2nd example: similar but the options for the answers are 3 (so a 3x2 matrix).
library(rcompanion)
x <- matrix(c(57,27,87,93,66,90), nrow = 2)
colnames(x) <- c("NA", "No", "Yes")
rownames(x) <- c("Cond1", "Cond2")
cramerV(x, conf = 95)
is this the correct implementation of the Cramer V? I'm struggling to understand what the p-value is actually telling me. how does a single p-value represent how strong is the association between all those values? are there better alternatives? thanks
General information:
Cramer's V gives us information about the statistical relationship between two or more nominally scaled variables (e.g. eye color: green, blue, brown)
When determining Cramer's V, the chi-square value (X²) is standardized. This enables us to compare relationships between variables using Cramer’s V. (Similar to Pearson contingency coefficient that is also a standardized measure of correlation based on chi-square (X²).
Your example 1:
Here we want to assess the association of condition(cond1/cond2) and answer (yes/no)
We need the chi-square value to calculate Cramer’s V, therefore
we first determine chi-square value and then convert into a value for Cramer’s V.
We get Cramer V 0.1183
Interpretation:
Cramer's V is always between 0 and 1.
0 means no association
1 means complete/very strong association
Rule of thumb:
Cramer's V
0 - 0.2 -> weak association
0.2 - 0.6 -> middle association
0.6 - 1 -> strong association
In this example with a Cramer V of 0.12 there is a weak association between condition and answer.
Note:
As we use nominal scaled variables we can assess the strength of the association but not the direction!
Same interpretation for example 2.
It does not matter how many nominal scaled variables you assess.
The association here is 0.19 -> weak.
In your examples you can use Carmer's V to assess the strength of association.
I have an analytically derrived formula that looks something like this;
y = ((b/x)^n )* e^(-c(x-b)) * D
In the formula x is my independent variable. Alle the other lettres on the right side of the equation are parameters that I want to estimate.
I have a dataset with y and x. Is there a way by which I can get the most optimal values of the parameters b,c,d and n that minimizes square error of y.
I also have alot of constraints, such as b being inbetween 0 and 10, and c being inbetween 0 and 0.5 etc.
Is there way I can perform such an optimization problem in R preferrably? And furthermore, is it possible to have parameters b,c and d predicted by other variables in my dataset. This then becomes more of a ML problem, but I want the model to be written in the same way as the first mentioned formula.
Thanks in advance
In machine learning, PCA is used to reduce the dimensionality of training data. However, from the above picture, I can't understand where is the reduction?
The input data x_i has D dimensions:
The output data x still has D dimensions:
The crucial element here is misunderstanding what is the output, in this pseudocode the output is y (equation 29), not x (equation 30), consequently you do reduce your data to d dimensions, the final equation shows you that if you would like to move back to original space, you can do it (obviously data will be recovered with some errors, since in meantime we dropped a lot of information when going to d dimensions).
The important thing to understand while using PCA is the covariance matrix C(x) and its corresponding spectral decomposition. The obtained eigenvalues and eigenvector of the decomposition is used to reduce the dimensionality.
For a D dimensional training set, we have D number of eigenvalues and their corresponding eigenvectors. But in practice (specially image related applications) many of the eigenvectors are correlated; in other words many of them are redundant basis vectors. So discarding those vector from the basis space doesn't result in significant information loss.
Now, if you want to reduce the dimension of your input data from original D to d < D dimension, you can project the input data into d dominant eigenvectors (from the d largest eigenvalues). Eq~29 gives the project input data into the d dimensional space. Eq~30 is used to reconstruct the original data; here reconstruction errors depend on d (number of eigenvectors)
Given a set of variables, x's. I want to find the values of coefficients for this equation:
y = a_1*x_1 +... +a_n*x_n + c
where a_1,a_2,...,a_n are all unknowns. Thinking this in perspective of data frame, I want to create this value of y for every rows in the data.
My question is: for y, a_1...a_n and c are all unknown, is there a way for me to find a set of solutions a_1,...,a_n under the condition that corr(y,x_1), corr(y,x_2) .... corr(y,x_n) are all greater than 0.7. For simplicity take correlation here as Pearson correlation. I know there would no be unique solution. But how can I construct a set of solutions for a_1,...,a_n to fulfill this condition?
Spent a day to search the idea but could not get any information out of it. Any programming language to tackle this problem is welcomed or at least some reference for this.
No, it is not possible in general. It may be possible in some special cases.
Given x₁, x₂, ... you want to find y = a₁x₁ + a₂x₂ + ... + c so that all the correlations between y and the x's are greater than some target R. Since the correlation is
Corr(y, xi) = Cov(y, xi) / Sqrt[ Var(y) * Var(xi) ]
your constraint is
Cov(y, xi) / Sqrt[ Var(y) * Var(xi) ] > R
which can be rearranged to
Cov(y, xi)² > R² * Var(y) * Var(xi)
and this needs to be true for all i.
Consider the simple case where there are only two columns x₁ and x₂, and further assume that they both have mean zero (so you can ignore the constant c) and variance 1, and that they are uncorrelated. In that case y = a₁x₁ + a₂x₂ and the covariances and variances are
Cov(y, x₁) = a₁
Cov(y, x₂) = a₂
Var(x₁) = 1
Var(x₂) = 1
Var(y) = (a₁)² + (a₂)²
so you need to simultaneously satisfy
(a₁)² > R² * ((a₁)² + (a₂)²)
(a₂)² > R² * ((a₁)² + (a₂)²)
Adding these inequalities together, you get
(a₁)² + (a₂)² > 2 * R² * ((a₁)² + (a₂)²)
which means that in order to satisfy both of the inequalities, you must have R < Sqrt(1/2) (by cancelling common factors on both sides of the inequality). So the very best you could do in this simple case is to choose a₁ = a₂ (the exact value doesn't matter as long as they are equal) and both of the correlations Corr(y,a₁) and Corr(y,a₂) will be equal to 0.707. You cannot achieve correlations higher than this between y and all of the x's simultaneously in this case.
For the more general case with n columns (each of which has mean zero, variance 1 and zero correlation between columns) you cannot simultaneously achieve correlations greater than 1 / sqrt(n) (as pointed out in the comments by #kazemakase).
In general, the more independent variables there are, the lower the correlation you will be able to achieve between y and the x's. Also (although I haven't mentioned it above) the correlations between the x's matter. If they are in general positively correlated, you will be able to achieve a higher target correlation between y and the x's. If they are in general uncorrelated or negatively correlated, you will only be able to achieve low correlations between y and the x's.
I am not expert in this field so read with extreme prejudice!
I am a bit confused by your y
Your y is a single constant and you want to have the correlation between it and all the x_i values be > 0.7 ? I am no math/statistics expert but my feelings for this are that this is achievable only if the correlation between x_i,x_j upholds the same condition. in that case you can simply do the average of x_i like this:
y=(x_1+x_2+x_3+...+x_n)/n
so the a_i=1.0/n and c=0.0 But still the question is:
What meaning has a correlation between 2 numbers only?
More reasonable would be if y is a function dependent on x
for example like this:
y(x) = a_1*(x-x_1)+... +a_n*(x-x_n) + c
or any other equation (hard to make any without knowing where it came from and for what purpose). Then you can compute the correlation between two sets
X = { x_1 , x_2 ,..., x_n }
Y = { y(x_1),y(x_2),...y(x_n) }
In that case I would give try approximation search for the c,a_i constants to maximize correlation between X,Y, but the results complexity for the whole thing would be insane. So instead I would tweak just one constant. at the time
set some safe c,a_1,a_2,... constants
tweak a_1
so compute correlation for (a_1-delta) and (a_1+delta) and then choose the direction which is in favor of correlation. then keep going in that direction until the correlation coefficient start to drop.
Then you can recursively to this again with smaller delta. Btw this is exactly what my approx class does from the link above.
loop #2 through all the a_i
loop this whole few times to enhance precision
May be you could compute the c after each run to minimize the distance between X,Y sets.
Objective function to be maximized : pos%*%mu where pos is the weights row vector and mu is the column vector of mean returns of d stocks
Constraints: 1) ones%*%pos = 1 where ones is a row vector of 1's of size 1*d (d is the number of stocks)
2) pos%*%cov%*%t(pos) = rb^2 # where cov is the covariance matrix of size d*d and rb is risk budget which is the free parameter whose values will be changed to draw the efficient frontier
I want to write a code for this optimization problem in R but I can't think of any function or library for help.
PS: solve.QP in library quadprog has been used to minimize covariance subject to a target return . Can this function be also used to maximize return subject to a risk budget ? How should I specify the Dmat matrix and dvec vector for this problem ?
EDIT :
library(quadprog)
mu <- matrix(c(0.01,0.02,0.03),3,1)
cov # predefined covariance matrix of size 3*3
pos <- matrix(c(1/3,1/3,1/3),1,3) # random weights vector
edr <- pos%*%mu # expected daily return on portfolio
m1 <- matrix(1,1,3) # constraint no.1 ( sum of weights = 1 )
m2 <- pos%*%cov # constraint no.2
Amat <- rbind(m1,m2)
bvec <- matrix(c(1,0.1),2,1)
solve.QP(Dmat= ,dvec= ,Amat=Amat,bvec=bvec,meq=2)
How should I specify Dmat and dvec ? I want to optimize over pos
Also, I think I have not specified constraint no.2 correctly. It should make the variance of portfolio equal to the risk budget.
(Disclaimer: There may be a better way to do this in R. I am by no means an expert in anything related to R, and I'm making a few assumptions about how R is doing things, notably that you're using an interior-point method. Also, there is likely an R package for what you're trying to do, but I don't know what it is or how to use it.)
Minimising risk subject to a target return is a linearly-constrained problem with a quadratic objective, looking like this:
min x^T Q x
subject to sum x_i = 1
sum ret_i x_i >= target
(and x >= 0 if you want to be long-only).
Maximising return subject to a risk budget is quadratically-constrained, however; it looks like this:
max ret^T x
subject to sum x_i = 1
x^T Q x <= riskbudget
(and maybe x >= 0).
Convex quadratic terms in the objective impose less of a computational cost in an interior-point method compared to introducing a convex quadratic constraint. With a quadratic objective term, the Q matrix just shows up in the augmented system. With a convex quadratic constraint, you need to optimise over a more complicated cone containing a second-order cone factor and you need to be careful about how you solve the linear systems that arise.
I would suggest you use the risk-minimisation formulation repeatedly, doing a binary search on the target parameter until you've found a portfolio approximately maximising return subject to your risk budget. I am suggesting this approach because it is likely sufficient for your needs.
If you really want to solve your problem directly, I would suggest using an interface Todd, Toh, and Tutuncu's SDPT3. This really is overkill; SDPT3 permits you to formulate and solve symmetric cone programs of your choosing. I would also note that portfolio optimisation problems are particularly special cases of symmetric cone programs; other approaches exist that are reportedly very successful. Unfortunately, I'm not studied up on them.