I'm new to reshaping data in R and can't figure out how to use reshape() (or another package) to create a panel data. There are two time observations for each geographical unit, however each of the time observations is formatted in a variable. For example:
subdistrict <- 1:4
control_t1 <- 5:8
control_t2 <- 9:12
motivation_t1 <- 12:15
motivation_t2 <- 16:19
data_mat <- as.data.frame(cbind(subdistrict, control_t1, control_t2, motivation_t1, motivation_t2))
data_mat
subdistrict control_t1 control_t2 motivation_t1 motivation_t2
1 1 5 9 12 16
2 2 6 10 13 17
3 3 7 11 14 18
4 4 8 12 15 19
Here, control_t1 and control_t2 each refer to a different period. My goal is to reshape the data such that a time variable can be established and the named variable can be collapsed so to produce the following frame:
subdistrict time control motivation
1 1 1 12
1 2 5 16
2 1 2 13
2 2 6 17
3 1 3 14
3 2 7 18
4 1 4 15
4 2 8 19
I'm not sure how to create the new time variable, and collapse and rename the variables to reshape the data as such. Thanks for any help.
You just have to use the reshape() function with option direction = "long". Here is the code :
district <- 1:4
control_t1 <- 5:8
control_t2 <- 9:12
relax_t1 <- 12:15
relax_t2 <- 16:19
data_mat <- as.data.frame(cbind(district, control_t1, control_t2, relax_t1, relax_t2))
reshape(data = data_mat, direction = "long", idvar = "district", timevar = "time", varying = list(c(2:3), c(4:5)))
# district time control_t1 relax_t1
# 1.1 1 1 5 12
# 2.1 2 1 6 13
# 3.1 3 1 7 14
# 4.1 4 1 8 15
# 1.2 1 2 9 16
# 2.2 2 2 10 17
# 3.2 3 2 11 18
# 4.2 4 2 12 19
Have a look at the R Programming wikibooks to learn more.
A simple answer is to split and rebind the data frame into your new form, like so:
new_Data <- data.frame(
subdistrict=data_mat[,1],
control=unlist(data_mat[,2:3]),
motivation=unlist(data_mat[,4:5]))
All we are doing here is collapsing the two columns of 'control' and 'motivation' into single columns of data by using the 'unlist' function and then binding it all into a new data frame. The 'subdistrict' data repeats, so there is no reason to specify it twice.
Related
I want to subtract one column from another and create a new one using the corresponding suffix in the first column. I have approx 50 columns
I can do it "manually" as follows...
df$new1 <- df$col_a1 - df$col_b1
df$new2 <- df$col_a2 - df$col_b2
What is the easiest way to create a loop that does the job for me?
We can use grep to identify columns which has "a" and "b" in it and subtract them directly.
a_cols <- grep("col_a", names(df))
b_cols <- grep("col_b", names(df))
df[paste0("new", seq_along(a_cols))] <- df[a_cols] - df[b_cols]
df
# col_a1 col_a2 col_b1 col_b2 new1 new2
#1 10 15 1 5 9 10
#2 9 14 2 6 7 8
#3 8 13 3 7 5 6
#4 7 12 4 8 3 4
#5 6 11 5 9 1 2
#6 5 10 6 10 -1 0
data
Tested on this data
df <- data.frame(col_a1 = 10:5, col_a2 = 15:10, col_b1 = 1:6, col_b2 = 5:10)
Suppose I have the next data frame:
df<-data.frame(step1=c(1,2,3,4),step2=c(5,6,7,8),step3=c(9,10,11,12),step4=c(13,14,15,16))
step1 step2 step3 step4
1 1 5 9 13
2 2 6 10 14
3 3 7 11 15
4 4 8 12 16
and what I have to do is something like the following:
df2<-data.frame(col1=c(1,2,3,4,5,6,7,8,9,10,11,12),col2=c(5,6,7,8,9,10,11,12,13,14,15,16))
col1 col2
1 1 5
2 2 6
3 3 7
4 4 8
5 5 9
6 6 10
7 7 11
8 8 12
9 9 13
10 10 14
11 11 15
12 12 16
How can I do that? consider that more steps can be included (example, 20 steps).
Thanks!!
We can design a function to achieve this task. df_final is the final output. Notice that bin is an argument that the users can specify how many columns to transform together.
# A function to conduct data transformation
trans_fun <- function(df, bin = 3){
# Calculate the number of new columns
new_ncol <- (ncol(df) - bin) + 1
# Create a list to store all data frames
df_list <- lapply(1:new_ncol, function(num){
return(df[, num:(num + bin - 1)])
})
# Convert each data frame to a vector
dt_list2 <- lapply(df_list, unlist)
# Convert dt_list2 to data frame
df_final <- as.data.frame(dt_list2)
# Set the column and row names of df_final
colnames(df_final) <- paste0("col", 1:new_ncol)
rownames(df_final) <- 1:nrow(df_final)
return(df_final)
}
# Apply the trans_fun
df_final <- trans_fun(df)
df_final
col1 col2
1 1 5
2 2 6
3 3 7
4 4 8
5 5 9
6 6 10
7 7 11
8 8 12
9 9 13
10 10 14
11 11 15
12 12 16
Here is a method using dplyr and reshape2 - this assumes all of the columns are the same length.
library(dplyr)
library(reshape2)
Drop the last column from the dataframe
df[,1:ncol(df)-1]%>%
melt() %>%
dplyr::select(col1=value) -> col1
Drop the first column from the dataframe
df %>%
dplyr::select(-step1) %>%
melt() %>%
dplyr::select(col2=value) -> col2
Combine the dataframes
bind_cols(col1, col2)
This should do the work:
df2 <- data.frame(col1 = 1:(length(df$step1) + length(df$step2)))
df2$col1 <- c(df$step1, df$step2, df$step3)
df2$col2 <- c(df$step2, df$step3, df$step4)
Things to point:
The important thing to see in the first line of the code, is the need for creating a table with the right amount of rows
Calling a columns that does not exist will create one, with that name
Deleting columns in R should be done like this df2$col <- NULL
Are you not just looking to do:
df2 <- data.frame(col1 = unlist(df[,-nrow(df)]),
col2 = unlist(df[,-1]))
rownames(df2) <- NULL
df2
col1 col2
1 1 5
2 2 6
3 3 7
4 4 8
5 5 9
6 6 10
7 7 11
8 8 12
9 9 13
10 10 14
11 11 15
12 12 16
I'm operating with a dataset that contains the values of same variables at different points in time. In the example below I have the values of variables a and b at time points 1 and 2.
> set.seed(1)
> data <- data.frame(matrix(sample(16), ncol = 4))
> names(data) <- paste(rep(c("a", "b"), each = 2), 1:2, sep = "")
> data
a1 a2 b1 b2
1 5 3 14 13
2 6 10 1 8
3 9 11 2 4
4 12 15 7 16
Now, suppose I want to calculate a new variable for both time points so that it would contain the sum of a and b (instead of the NAs as in example below). Since my actual dataset contains about 15 different variables and 10 time points (so 150 columns), I want to automate this calculation of 10 new variables.
> data[, paste("ab", 1:2, sep = "")] <- NA
> data
a1 a2 b1 b2 ab1 ab2
1 5 3 14 13 NA NA
2 6 10 1 8 NA NA
3 9 11 2 4 NA NA
4 12 15 7 16 NA NA
I've previously used Stata where I could create a simple 'foreach' loop to do this. Something like below.
foreach t of numlist 1/2 {
generate ab`t' = a`t' + b`t'
}
But I've learned that using loops in R is not feasible, nor have I any idea how to loop over variable names like that in R.
So what would be the correct solution for my problem in R?
This will replicate the same foreach loop you used in Stata.
for(i in 1:2){
data[, paste("ab", i, sep="")] <-
data[,paste("a", i, sep="")] + data[, paste("b", i, sep="")]
}
The output looks like this:
> data
a1 a2 b1 b2 ab1 ab2
1 15 1 16 12 31 13
2 10 7 14 3 24 10
3 2 5 9 4 11 9
4 6 8 13 11 19 19
to do this the R way,
make use of some native iteration via a *apply function
use the built-in rowSums (as in #Sotos) answer
make use of assignment into the data.frame, that is `]`<-
all together
data[paste0('ab', 1:2)] <- sapply(1:2,
function(i)
rowSums(data[paste0(c('a', 'b'), i)]))
data
# a1 a2 b1 b2 ab1 ab2
# 1 5 3 14 13 19 16
# 2 6 10 1 8 7 18
# 3 9 11 2 4 11 15
# 4 12 15 7 16 19 31
ps, in a program use vapply instead, you'll need to provide an additional argument specifying the shape of the output but its safer and sometimes faster
You can do without iteration:
data$ab1 <- data$a1 + data$b1
data$ab2 <- data$a2 + data$b2
or
data <- transform(data, ab1=a1+b1, ab2=a2+b2)
BTW:
It is better not to name an object data because data= is often a parameter in functions.
Here is one way to do it. We iterate over the unique values of the column names and we calculate the rowSums when those unique values match the colname values.
sapply(unique(sub('\\D', '', names(data))),
function(i) rowSums(data[,grepl(i, sub('\\D', '', names(data)))]))
# 1 2
#[1,] 17 23
#[2,] 24 22
#[3,] 14 10
#[4,] 15 11
I have a dataframe, df1:
Type CA AR Total
alpha 2 3 5
beta 1 5 6
gamma 6 2 8
delta 8 1 9
and a dataframe, df2:
Type AR CA Total
alpha 3 4 7
beta 2 6 8
gamma 9 1 10
delta 4 1 5
I want to add the two dataframes such that the values under "CA" are added together and that the values under "AR" are added together. Basically, the values under each heading should be added together.
The resulting df should look like this:
Type AR CA Total
alpha 6 6 12
beta 7 7 14
gamma 11 7 18
delta 5 9 14
For example: (AR, gamma) = 2 + 9 = 11
The safest way would probably be to bind and aggregate
aggregate(.~Type, rbind(df1,df2), sum)
# Type CA AR Total
# 1 alpha 6 6 12
# 2 beta 7 7 14
# 3 delta 9 5 14
# 4 gamma 7 11 18
The rbind.data.frame function pays attention to column names so it will properly stack your values.
I'll repeat my suggestion from the comments last time -- consider putting Type in rownames:
DF1 <- data.frame(df1[-1],row.names=df1$Type)
DF2 <- data.frame(df2[-1],row.names=df2$Type)
From here, adding is straightforward:
DF1 + DF2[names(DF1)]
# CA AR Total
# alpha 6 6 12
# beta 7 7 14
# gamma 7 11 18
# delta 9 5 14
A couple of caveats: If your rows are not ordered the same way, this will not work correctly (that's why #MrFlick's approach is "safe"). Also, the extension to more data frames isn't so elegant here:
Reduce(`+`,lapply(list(DF2,DF3,DF4),`[`,order(names(DF1))),init=DF1) # here
aggregate(.~Type, rbind(df1,df2,df3,df4), sum) # #MrFlick
You can consider storing your data in a "long" form instead, which would make further operations more straightforward.
If you have your data.frames in a list, you can easily use melt from "reshape2" to get a "long" data.frame. For example:
melt(list(df1, df2), id.vars = "Type")
Once the data are in the long form, you can reshape it to a "wide" form using dcast, and perform whatever aggregation you want to at that stage.
Furthermore, you can generalize the creation of the list if you have similarly named data.frames in your workspace by using mget.
Here's an example, assuming we have two data.frames, one named "df1", and one named "df2":
library(reshape2)
dcast(melt(mget(ls(pattern = "df\\d+")), id.vars = "Type"),
Type ~ variable, value.var = "value", fun.aggregate = sum)
# Type CA AR Total
# 1 alpha 6 6 12
# 2 beta 7 7 14
# 3 delta 9 5 14
# 4 gamma 7 11 18
Apologies for the seemingly simple question, but I can't seem to find a solution to the following re-arrangement problem.
I'm used to using read.csv to read in files with a header row, but I have an excel spreadsheet with two 'header' rows - cell identifier (a, b, c ... g) and three sets of measurements (x, y and z; 1000s each) for each cell:
a b
x y z x y z
10 1 5 22 1 6
12 2 6 21 3 5
12 2 7 11 3 7
13 1 4 33 2 8
12 2 5 44 1 9
csv file below:
a,,,b,,
x,y,z,x,y,z
10,1,5,22,1,6
12,2,6,21,3,5
12,2,7,11,3,7
13,1,4,33,2,8
12,2,5,44,1,9
How can I get to a data.frame in R as shown below?
cell x y z
a 10 1 5
a 12 2 6
a 12 2 7
a 13 1 4
a 12 2 5
b 22 1 6
b 21 3 5
b 11 3 7
b 33 2 8
b 44 1 9
Use base R reshape():
temp = read.delim(text="a,,,b,,
x,y,z,x,y,z
10,1,5,22,1,6
12,2,6,21,3,5
12,2,7,11,3,7
13,1,4,33,2,8
12,2,5,44,1,9", header=TRUE, skip=1, sep=",")
names(temp)[1:3] = paste0(names(temp[1:3]), ".0")
OUT = reshape(temp, direction="long", ids=rownames(temp), varying=1:ncol(temp))
OUT
# time x y z id
# 1.0 0 10 1 5 1
# 2.0 0 12 2 6 2
# 3.0 0 12 2 7 3
# 4.0 0 13 1 4 4
# 5.0 0 12 2 5 5
# 1.1 1 22 1 6 1
# 2.1 1 21 3 5 2
# 3.1 1 11 3 7 3
# 4.1 1 33 2 8 4
# 5.1 1 44 1 9 5
Basically, you should just skip the first row, where there are the letters a-g every third column. Since the sub-column names are all the same, R will automatically append a grouping number after all of the columns after the third column; so we need to add a grouping number to the first three columns.
You can either then create an "id" variable, or, as I've done here, just use the row names for the IDs.
You can change the "time" variable to your "cell" variable as follows:
# Change the following to the number of levels you actually have
OUT$cell = factor(OUT$time, labels=letters[1:2])
Then, drop the "time" column:
OUT$time = NULL
Update
To answer a question in the comments below, if the first label was something other than a letter, this should still pose no problem. The sequence I would take would be as follows:
temp = read.csv("path/to/file.csv", skip=1, stringsAsFactors = FALSE)
GROUPS = read.csv("path/to/file.csv", header=FALSE,
nrows=1, stringsAsFactors = FALSE)
GROUPS = GROUPS[!is.na(GROUPS)]
names(temp)[1:3] = paste0(names(temp[1:3]), ".0")
OUT = reshape(temp, direction="long", ids=rownames(temp), varying=1:ncol(temp))
OUT$cell = factor(temp$time, labels=GROUPS)
OUT$time = NULL