In R - generate pairwise data.frame from all rows in data.frame - r

Question

I have a data.frame called df with a 8 million observations on 4 columns:
name <- c("Pablo", "Christina", "Steve", "Diego", "Ali", "Brit", "Ruth", "Mia", "David", "Dylan")
year <- seq(2000, 2009, 1)
v1 <- sample(1:10, 10, replace=T)
v2 <- sample(1:10, 10, replace=T)
df <- data.frame(year, v1)
> df
name year v1 v2
1 Pablo 2000 2 9
2 Christina 2001 5 3
3 Steve 2002 8 9
4 Diego 2003 7 6
5 Ali 2004 2 4
6 Brit 2005 1 1
7 Ruth 2006 10 9
8 Mia 2007 6 7
9 David 2008 10 9
10 Dylan 2009 3 2
I want to generate a data.frame output with all pair-wise combination of the rows in df that looks like this:
>output
name year v1 v2 name_2 year_2 v1_2 v2_2
1 Pablo 2000 2 9 Christina 2001 5 3
2 Pablo 2000 2 9 Steve 2002 8 9
3 Pablo 2000 2 9 Diego 2003 7 6
etc.
What are the fastest ways to do this?

Answer 1

tidyr::crossing will return all combinations of observations, but you'll need to set names with setNames or the like. If you don't want self-matches, you can remove them by calling dplyr::filter on any unique ID column.
library(tidyverse)
df_crossed <- df %>%
setNames(paste0(names(.), '_2')) %>%
crossing(df) %>%
filter(name != name_2)
head(df_crossed)
## name_2 year_2 v1_2 v2_2 name year v1 v2
## 1 Pablo 2000 5 5 Christina 2001 7 3
## 2 Pablo 2000 5 5 Steve 2002 1 9
## 3 Pablo 2000 5 5 Diego 2003 2 8
## 4 Pablo 2000 5 5 Ali 2004 9 5
## 5 Pablo 2000 5 5 Brit 2005 8 5
## 6 Pablo 2000 5 5 Ruth 2006 8 1
Another way to fix names would be to use janitor::clean_names after crossing, though it's an extra package.

Answer 2

Hopefully this will give the result the post owner was looking for.
name <- c("Pablo", "Christina", "Steve", "Diego", "Ali", "Brit", "Ruth", "Mia", "David", "Dylan")
year <- seq(2000, 2009, 1)
v1 <- sample(1:10, 10, replace=T)
v2 <- sample(1:10, 10, replace=T)
df <- data.frame(name, year, v1, v2, stringsAsFactors=FALSE)
print(df)
rows = nrow(df)
n <- rows * (rows - 1) / 2
ndf <- data.frame(
name1=character(n),year1=numeric(n), v1_1=numeric(n),v2_1=numeric(n),
name2=character(n),year2=numeric(n), v1_2=numeric(n),v2_2=numeric(n),
stringsAsFactors=FALSE
)
k <- 1
for (i in 1:(rows-1))
{
for (j in (i+1):rows)
{
ndf[k,] <- c(df[i,], df[j,])
k <- k + 1
}
}
print(ndf)
# name year v1 v2
#1 Pablo 2000 4 9
#2 Christina 2001 2 1
#3 Steve 2002 2 9
#4 Diego 2003 5 5
#5 Ali 2004 10 4
#6 Brit 2005 5 2
#7 Ruth 2006 7 10
#8 Mia 2007 6 7
#9 David 2008 4 10
#10 Dylan 2009 7 3
# name1 year1 v1_1 v2_1 name2 year2 v1_2 v2_2
#1 Pablo 2000 4 9 Christina 2001 2 1
#2 Pablo 2000 4 9 Steve 2002 2 9
#3 Pablo 2000 4 9 Diego 2003 5 5
#4 Pablo 2000 4 9 Ali 2004 10 4
#5 Pablo 2000 4 9 Brit 2005 5 2
#6 Pablo 2000 4 9 Ruth 2006 7 10
#7 Pablo 2000 4 9 Mia 2007 6 7
#8 Pablo 2000 4 9 David 2008 4 10
#9 Pablo 2000 4 9 Dylan 2009 7 3
#10 Christina 2001 2 1 Steve 2002 2 9
#...

Answer 3

Not to add to the noise but consider a base R cross join with merge on same dataframe that also filters out reverse duplicates. Do note, cross join before filter will return a 8 mill X 8 mill records dataset, so hopefully your RAM is sufficient for such an operation.
df <- data.frame(name = c("Pablo", "Christina", "Steve", "Diego", "Ali",
"Brit", "Ruth", "Mia", "David", "Dylan"),
year = seq(2000, 2009, 1),
v1 =sample(1:10, 10, replace=T),
v2 =sample(1:10, 10, replace=T),
stringsAsFactors = FALSE)
# MERGE ON KEY, THEN REMOVE KEY COL
df$key <- 1
dfm <- merge(df, df, by="key")[,-1]
# FILTER OUT SAME NAME AND REVERSE DUPS, THEN RENAME COLUMNS
dfm <- setNames(dfm[(dfm$name.x < dfm$name.y),],
c("name_p1", "year_p1", "V1_p1", "V2_p1",
"name_p2", "year_p2", "V1_p2", "V2_p2"))
# ALL PABLO PAIRINGS
dfm[dfm$name_p1=='Pablo' | dfm$name_p2=='Pablo',]
# name_p1 year_p1 V1_p1 V2_p1 name_p2 year_p2 V1_p2 V2_p2
# 3 Pablo 2000 7 8 Steve 2002 3 1
# 7 Pablo 2000 7 8 Ruth 2006 8 4
# 11 Christina 2001 10 10 Pablo 2000 7 8
# 31 Diego 2003 4 9 Pablo 2000 7 8
# 41 Ali 2004 5 3 Pablo 2000 7 8
# 51 Brit 2005 2 4 Pablo 2000 7 8
# 71 Mia 2007 7 7 Pablo 2000 7 8
# 81 David 2008 1 7 Pablo 2000 7 8
# 91 Dylan 2009 9 2 Pablo 2000 7 8
If somehow this large set derived from an SQL compliant database, I can provide the counterpart in SQL which may be more efficient as the filter runs with join process and not separately after.

Answer 4

This extension of #alistaires solution shows a crossed matrix used as index. The question as stated wants the full crossed output which
will be very large (~64 million rows for 8 million items) so there
is really no way around the memory requirement. However, if the
the real-world use of this is to deal with subsets, the indexing technique
shown here may be a way to reduce memory use. Its possible that crossing the integers only uses less memory during the crossing operation.
library(dplyr)
library(tidyr)
crossed <- as.matrix(crossing(1:nrow(df), 1:nrow(df)))
# bind and name in one step (may be inefficient) so that filter can be applied in one step
output <- as.data.frame(cbind(df[crossed[, 1],],
data.frame(name_2 = df[crossed[, 2], 1],
year_2 = df[crossed[, 2], 2],
v1_2 = df[crossed[, 2], 3],
v2_2 = df[crossed[, 2], 4]) )) %>%
filter(!(name == name_2 & year == year_2))
# estimated sized for 8 million rows gine this 10 row sample
format(object.size(output) / (10 / 8e6), units="MB")
#[1] "5304 Mb"

Answer 5

You could cross join the name column to itself, using data.table and remove repeated cases. This will result in a smaller structure on which to merge in data rather than doing the full merge, then filtering. You can add the rest of the data with two merges: once to merge data associated with the first name column and again to merge in data associated with the second column.
name <- c("Pablo", "Christina", "Steve", "Diego", "Ali", "Brit", "Ruth", "Mia", "David", "Dylan")
year <- seq(2000, 2009, 1)
v1 <- sample(1:10, 10, replace=T)
v2 <- sample(1:10, 10, replace=T)
# stringsAsFactors = FALSE in order for pmin to work properly
df <- data.frame(name, year, v1, v2, stringsAsFactors = FALSE)
library(data.table)
setDT(df)
setkey(df)
# cross-join name column to itself while removing duplicates and redundancies
name_cj <- setnames(
CJ(df[, name], df[, name])[V1 < V2], # taking a hint from Parfait's clever solution
c("name1", "name2"))
# perform 2 merges, once for the 1st name column and
# again for the 2nd name colum
name_cj <- merge(
merge(name_cj, df, by.x = "name1", by.y = "name"),
df,
by.x = "name2", by.y = "name", suffixes = c("_1", "_2"))
# reorder columns as desired with setorder()
head(name_cj)
# name2 name1 year_1 v1_1 v2_1 year_2 v1_2 v2_2
#1: Brit Ali 2004 3 8 2005 4 5
#2: Christina Ali 2004 3 8 2001 9 8
#3: Christina Brit 2005 4 5 2001 9 8
#4: David Ali 2004 3 8 2008 5 2
#5: David Brit 2005 4 5 2008 5 2
#6: David Christina 2001 9 8 2008 5 2