I Have multiple csv files that i have already read into R. Now I want to append all these into one file. I tried few things but getting different errors. Can anyone please help me with this?
TRY 1:
mydata <- rbind(x1,x2,x3,x4,x5,x6,x7,x8)
WHERE XI,X2....X8 Are the CSV files I read into R, error I am getting is
ERROR 1 :In [<-.factor(*tmp*, ri, value = c(NA, NA, NA, NA, NA, NA, NA, :
invalid factor level, NA generated
TRY 2: Then I try this in another way :
mydata1<- c(x1,x2,x3,x4,x5,x6,x7,x8)
> mydata2 <- do.call('rbind',lapply(mydata1,read.table,header=T))
Error 2: in FUN(X[[i]], ...) :
'file' must be a character string or connection
can anyone please help me know what is the right way to do this?
How to import all files from a single folder at once and bind by row (e.g., same format for each file.)
library(tidyverse)
list.files(path = "location_of/data/folder_you_want/",
pattern="*.csv",
full.names = T) %>%
map_df(~read_csv(.))
If there is a file that you want to exclude then
list.files(path = "location_of/data/folder_you_want/",
pattern="*.csv",
full.names = T) %>%
.[ !grepl("data/folder/name_of_file_to_remove.csv", .) ] %>%
map_df(~read_csv(.))
Sample CSV Files
Note
CSV files to be merged here have
- equal number of columns
- same column names
- same order of columns
- number of rows can be different
1st csv file abc.csv
A,B,C,D
1,2,3,4
2,3,4,5
3,4,5,6
1,1,1,1
2,2,2,2
44,44,44,44
4,4,4,4
4,4,4,4
33,33,33,33
11,1,11,1
2nd csv file pqr.csv
A,B,C,D
1,2,3,40
2,3,4,50
3,4,50,60
4,4,4,4
5,5,5,5
6,6,6,6
List FILENAMES of CSV Files
Note
The path below E:/MergeCSV/ has just the files to be merged. No other csv files. So in this path, there are only two csv files, abc.csv and pqr.csv
## List filenames to be merged.
filenames <- list.files(path="E:/MergeCSV/",pattern="*.csv")
## Print filenames to be merged
print(filenames)
## [1] "abc.csv" "pqr.csv"
FULL PATH to CSV Files
## Full path to csv filenames
fullpath=file.path("E:/MergeCSV",filenames)
## Print Full Path to the files
print(fullpath)
## [1] "E:/MergeCSV/abc.csv" "E:/MergeCSV/pqr.csv"
MERGE CSV Files
## Merge listed files from the path above
dataset <- do.call("rbind",lapply(filenames,FUN=function(files){ read.csv(files)}))
## Print the merged csv dataset, if its large use `head()` function to get glimpse of merged dataset
dataset
# A B C D
# 1 1 2 3 4
# 2 2 3 4 5
# 3 3 4 5 6
# 4 1 1 1 1
# 5 2 2 2 2
# 6 44 44 44 44
# 7 4 4 4 4
# 8 4 4 4 4
# 9 33 33 33 33
# 10 11 1 11 1
# 11 1 2 3 40
# 12 2 3 4 50
# 13 3 4 50 60
# 14 4 4 4 4
# 15 5 5 5 5
# 16 6 6 6 6
head(dataset)
# A B C D
# 1 1 2 3 4
# 2 2 3 4 5
# 3 3 4 5 6
# 4 1 1 1 1
# 5 2 2 2 2
# 6 44 44 44 44
## Print dimension of merged dataset
dim(dataset)
## [1] 16 4
The accepted answer above generates the error shown in the comments because the do.call requires the "fullpath" parameter. Use the code as shown to use in the directory of your choice:
dataset <- do.call("rbind",lapply(fullpath,FUN=function(files){ read.csv(files)}))
You can use a combination of lapply(), and do.call().
## cd to the csv directory
setwd("mycsvs")
## read in csvs
csvList <- lapply(list.files("./"), read.csv, stringsAsFactors = F)
## bind them all with do.call
csv <- do.call(rbind, csvList)
You can also use fread() function from the data.table package and rbindlist() instead for a performance boost.
I just have a data frame and want to split the data frame by rows, assign the several new data frames to new variables and save them as csv files.
a <- rep(1:5,each=3)
b <-rep(1:3,each=5)
c <- data.frame(a,b)
# a b
1 1 1
2 1 1
3 1 1
4 2 1
5 2 1
6 2 2
7 3 2
8 3 2
9 3 2
10 4 2
11 4 3
12 4 3
13 5 3
14 5 3
15 5 3
I want to split c by column a. i.e all rows are 1 in column a are split from c and assign it to A and save A as A.csv. The same to B.csv with all 2 in column a.
What I can do is
A<-c[c$a%in%1,]
write.csv (A, "A.csv")
B<-c[c$a%in%2,]
write.csv (B, "B.csv")
...
If I have 1000 rows and there will be lots of subsets, I just wonder if there is a simple way to do this by using for loop?
The split() function is very useful to split data frame. Also, you can use lapply() here - it should be more efficient than a loop.
dfs <- split(c, c$a) # list of dfs
# use numbers as file names
lapply(names(dfs),
function(x){write.csv(dfs[[x]], paste0(x,".csv"),
row.names = FALSE)})
# or use letters (max 26!) as file names
names(dfs) <- LETTERS[1:length(dfs)]
lapply(names(dfs),
function(x){write.csv(dfs[[x]],
file = paste0(x,".csv"),
row.names = FALSE)})
for(i in seq_along(unique(c$a))){
write.csv(c[c$a == i,], paste0(LETTERS[i], ".csv"))}
You should consider, however, what happens if you have more than 26 subsets. What will those files be named?
When I do:
var = names(df)[2]
df$var
I get NULL. I think that var is a string inside quotes and that is why this is happening. How could get the columns in a dataframe and dynamically query them?
It has been suggested that I use df[var], but what if my dataframe has another dataframe within it? df[var][x] or df[var]$x won't work.
Get a column of a data frame or item in a list by value of a variable by doing:
df[[var]]
It's hard to know what error-inducing situation has been constructed without dput-output on the offending dataframe. It's modestly difficult to get a column name as described (with actual quotes in the column name, but its possible. First we can try and fail to get such a beast:
df2 <- data.frame("\"col1\""=1:10)
df2[["\"col1\""]]
#NULL
df2
# the data.frame function coerced it to a valid column name with no quotes
X.col1.
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
10 10
So we can bypass the validity checks. Now we need escapes preceding the quotes:
df2 <- data.frame("\"col1\""=1:10, check.names=FALSE)
> df2[["\"col1\""]]
[1] 1 2 3 4 5 6 7 8 9 10
If the df[[var]]$x approach worked for you, then the answer is more likely that df is not a dataframe but rather is an ordinary R named list and that it is x that is a dataframe. You should check this by doing:
str(df)
You could make such a structure very simply with:
> df3 <- list( item=data.frame(x=1:10, check.names=FALSE))
> var1 = "item"
> df3[[var1]]$x
[1] 1 2 3 4 5 6 7 8 9 10
> str(df3)
List of 1
$ item:'data.frame': 10 obs. of 1 variable:
..$ x: int [1:10] 1 2 3 4 5 6 7 8 9 10
I want to read a matrix (all values, no null or empty column) from a tab-separated text file of integers and name the columns automatically (based on the titles in the first line):
a b c
9 2 3
2 9 6
3 2 4
5 3 3
I have tried read.csv(), read.table() and scan() methods and read the file, but I want something that:
1- Automatically identifies the column names (no need to mention the
names one by one).
2- I would be able to treat them as matrix of integers; run rcorr(data) and quantile(data$a, 0.9) instead of rcorr(as.matrix(data)) and quantile(as.matrix(data$a), 0.9) any time.
Any ideas on the simplest (yet efficient) way?
How about read.table?
read.table(text="a b c
9 2 3
2 9 6
3 2 4
5 3 3", header=TRUE)
> a b c
1 9 2 3
2 2 9 6
3 3 2 4
4 5 3 3
it also has options to input file, declare the separator, etc.. see help(read.table)
data <-- as.matrix(read.table("c:\\temp\\inFile.tsv", header=TRUE))
Note that I got the following error when there was special characters (#) in the header line:
Error in read.table("..."), : more columns than column names
So there shouldn't be special characters in the header line. Also it automatically detects the separator ("\t").
I would like to rename a large number of columns (column headers) to have numerical names rather than combined letter+number names. Because of the way the data is stored in raw format, I cannot just access the correct column numbers by using data[[152]] if I want to interact with a specific column of data (because random questions are filtered completely out of the data due to being long answer comments), but I'd like to be able to access them by data$152. Additionally, approximately half the columns names in my data have loaded with class(data$152) = NULL but class(data[[152]]) = integer (and if I rename the data[[152]] file it appropriately allows me to see class(data$152) as integer).
Thus, is there a way to use the loop iteration number as a column name (something like below)
for (n in 1:415) {
names(data)[n] <-"n" # name nth column after number 'n'
}
That will reassign all my column headers and ensure that I do not run into question classes resulting in null?
As additional background info, my data is imported from a comma delimited .csv file with the value 99 assigned to answers of NA with the first row being the column names/headers
data <- read.table("rawdata.csv", header=TRUE, sep=",", na.strings = "99")
There are 415 columns with headers in format Q001, Q002, etc
There are approximately 200 rows with no row labels/no label column
You can do this without a loop, as follows:
names(data) <- 1:415
Let me illustrate with an example:
dat <- data.frame(a=1:4, b=2:5, c=3:6, d=4:7)
dat
a b c d
1 1 2 3 4
2 2 3 4 5
3 3 4 5 6
4 4 5 6 7
Now rename the columns:
names(dat) <- 1:4
dat
1 2 3 4
1 1 2 3 4
2 2 3 4 5
3 3 4 5 6
4 4 5 6 7
EDIT : How to access your new data
#Ramnath points out very accurately that you won't be able to access your data using dat$1:
dat$1
Error: unexpected numeric constant in "dat$1"
Instead, you will have to wrap the column names in backticks:
dat$`1`
[1] 1 2 3 4
Alternatively, you can use a combination of character and numeric data to rename your columns. This could be a much more convenient way of dealing with your problem:
names(dat) <- paste("x", 1:4, sep="")
dat
x1 x2 x3 x4
1 1 2 3 4
2 2 3 4 5
3 3 4 5 6
4 4 5 6 7