R - fastest way to look at all unique pairs of columns - r

Question

I have a data frame, M, and I want to calculate all pairwise correlations between the columns of M. I can do this easily using apply functions, e.g.
pvals = laply(M, function(x) llply(M, function(y) cor.test(x, y)$p.value))
However, this solution is doing 2x the required work because the correlation between x and y is the same as the correlation between y and x.
I am looking for a fast, simple way to calculate all correlations among unique pairs of columns. I would like the result to be an NxN matrix, where N=ncol(M). I've searched on Stack Overflow for a long time, but couldn't find anything that did this. Thanks!

Answer 1

for the iris data, you can do:
data(iris)
r <- cor(iris[1:4])
to get the correlation matrix.
You can look at what cor.test actually does with stats:::cor.test and find this...
df <- n - 2L
ESTIMATE <- c(cor = r)
PARAMETER <- c(df = df)
STATISTIC <- c(t = sqrt(df) * r/sqrt(1 - r^2))
p <- pt(STATISTIC, df)
which is all vectorized, so you can just run it.
There's a good discussion of the different tests on wikipedia: http://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient

Answer 2

You could use combn:
#Some data:
DF <- USJudgeRatings
#transform to matrix for better subset performance:
m <- as.matrix(DF)
#use combn and its `FUN` argument:
res <- matrix(nrow=ncol(DF), ncol=ncol(DF))
res[lower.tri(res)] <- combn(seq_along(DF), 2, function(ind) cor.test(m[, ind[[1]]], m[, ind[[2]]])$p.value)
res[upper.tri(res)] <- t(res)[upper.tri(res)]
diag(res) <- 0
Benchmarks:
corpRoland <- function(DF) {
m <- as.matrix(DF)
res <- matrix(nrow=ncol(DF), ncol=ncol(DF))
res[lower.tri(res)] <- combn(seq_along(DF), 2, function(ind) cor.test(m[, ind[[1]]], m[, ind[[2]]])$p.value)
res[upper.tri(res)] <- t(res)[upper.tri(res)]
diag(res) <- 0
res}
corpNeal <- function(DF) {
cors <- cor(DF)
df <- nrow(DF)-2
STATISTIC <- c(t = sqrt(df) * cors/sqrt(1 - cors^2))
p <- pt(STATISTIC, df)
matrix(2 * pmin(p, 1 - p),nrow=ncol(DF))}
library(microbenchmark)
DF <- as.data.frame(matrix(rnorm(1e3), ncol=10))
microbenchmark(corpRoland(DF), corpNeal(DF))
#Unit: microseconds
# expr min lq median uq max neval
# corpRoland(DF) 14021.003 14228.2040 14950.212 15157.27 17013.574 100
# corpNeal(DF) 342.631 351.6775 373.636 385.34 467.773 100
DF <- as.data.frame(matrix(rnorm(1e4), ncol=100))
microbenchmark(corpRoland(DF), corpNeal(DF), times=10)
# Unit: milliseconds
# expr min lq median uq max neval
# corpRoland(DF) 1595.878487 1601.221980 1615.391891 1633.746678 1637.373231 10
# corpNeal(DF) 8.359662 8.751755 9.021532 9.509576 9.753154 10
So, you should use the answer by #NealFultz.