I have a number of csv files and my goal is to find the number of complete cases for a file or set of files given by id argument. My function should return a data frame with column id specifying the file and column obs giving the number of complete cases for this id. However, my function overwrites the previous value of nobs in each loop and the resulting data frame gives me only its last value. Do you have any idea how to get the value of nobs for each value of id?
myfunction<-function(id=1:20) {
files<-list.files(pattern="*.csv")
myfiles = do.call(rbind, lapply(files, function(x) read.csv(x,stringsAsFactors = FALSE)))
for (i in id) {
good<-complete.cases(myfiles)
newframe<-myfiles[good,]
cases<-newframe[newframe$ID %in% i,]
nobs<-nrow(cases)
}
clean<-data.frame(id,nobs)
clean
}
Thanks.
We can do all inside lapply(), something like below (not tested):
myfunction <- function(id = 1:20) {
files <- list.files(pattern = "*.csv")[id]
do.call(rbind,
lapply(files, function(x){
df <- read.csv(x,stringsAsFactors = FALSE)
df <- df[complete.cases(df), ]
data.frame(ID=x,nobs=nrow(df))
}
)
)
}
Related
I have a bunch of csv files that I'm trying to read into R all at once, with each data frame from a csv becoming an element of a list. The loops largely work, but they keep overriding the list elements. So, for example, if I loop over the first 2 files, both data frames in list[[1]] and list[[2]] will contain the data frame for the second file.
#function to open one group of files named with "cores"
open_csv_core<- function(year, orgtype){
file<- paste(year, "/coreco.core", year, orgtype, ".csv", sep = "")
df <- read.csv(file)
names(df) <- tolower(names(df))
df <- df[df$ntee1 %in% c("C","D"),]
df<- df[!(df$nteecc %in% c("D20","D40", "D50", "D60", "D61")),]
return(df)
}
#function to open one group of files named with "nccs"
open_csv_nccs<- function(year, orgtype){
file2<- paste(year, "/nccs.core", year, orgtype, ".csv", sep="")
df2 <- read.csv(file2)
names(df2) <- tolower(names(df2))
df2 <- df2[df2$ntee1 %in% c("C","D"),]
df2<- df2[!(df2$nteecc %in% c("D20","D40", "D50", "D60", "D61")),]
return(df2)
}
#############################################################################
yrpc<- list()
yrpf<- list()
yrco<- list()
fname<- vector()
file_yrs<- as.character(c(1989:2019))
for(i in 1:length(file_yrs)){
fname<- list.files(path = file_yrs[i], pattern = NULL)
#accessing files in a folder and assigning to the proper function to open them based on how the file is named
for(j in 1:length(fname)){
if(grepl("pc.csv", fname[j])==T) {
if(grepl("nccs", fname[j])==T){
a <- open_csv_nccs(file_yrs[j], "pc")
yrpc[[paste0(file_yrs[i], "pc")]] <- a
} else {
b<- open_csv_core(file_yrs[j], "pc")
yrpc[[paste0(file_yrs[i], "pc")]] <- b
}
} else if (grepl("pf.csv", fname[j])==T){
if(grepl("nccs", fname[j])==T){
c <- open_csv_nccs(file_yrs[j], "pf")
yrpf[[paste0(file_yrs[i], "pf")]] <- c
} else {
d<- open_csv_core(file_yrs[j], "pf")
yrpf[[paste0(file_yrs[i], "pf")]] <- d
}
} else {
if(grepl("nccs", fname[j])==T){
e<- open_csv_nccs(file_yrs[j], "co")
yrco[[paste0(file_yrs[i], "co")]] <- e
} else {
f<- open_csv_core(file_yrs[j], "co")
yrco[[paste0(file_yrs[i], "co")]] <- f
}
}
}
}
Actually, both of your csv reading functions do exactly the same,
except that the paths are different.
If you find a way to list your files with abstract paths instead of relative
paths (just the file names), you wouldn't need to reconstruct the paths like
you do. This is possible by full.names = TRUE in list.files().
The second point is, it seems there is never from same year and same type
a "nccs.core" file in addition to a "coreco.core" file. So they are mutually
exclusive. So then, there is no logics necessary to distinguish those cases, which simplifies our code.
The third point is, you just want to separate the data frames by filetype ("pc", "pf", "co") and years.
Instead of creating 3 lists for each type, I would create one res-ults list, which contains for each type an inner list.
I would solve this like this:
years <- c(1989:2019)
path_to_type <- function(path) gsub(".*(pc|pf|co)\\.csv", "\\1", path)
res <- list("pc" = list(),
"pf" = list(),
"co" = list())
lapply(years, function(year) {
files <- list.files(path = year, pattern = "\\.csv", full.names = TRUE)
dfs <- lapply(files, function(path) {
print(path) # just to signal that the path is getting processed
df <- read.csv(path)
file_type <- path_to_type(path)
names(df) <- tolower(names(df))
df <- df[df$ntee1 %in% c("C", "D"), ]
df <- df[!(df$nteecc %in% c("D20", "D40", "D50", "D60", "D61")), ]
res[[file_type]][[year]] <- df
})
})
Now you can call from result's list by file_type and year
e.g.:
res[["co"]][[1995]]
res[["pf"]][[2018]]
And so on.
Actually, the results of the lapply() calls in this case are
not interesting. Just the content of res ... (result list).
It seems that in your for(j in 1:length(fname)){... you are creating one of 4 variable a, b, c or d. And you're reusing these variable names, so they are getting overwritten.
The "correct" way to do this is to use lapply in place of the for loop. Pass the list of files, and the required function (i.e. open_csv_core, etc) to lapply, and the return value that you get back is a list of the results.
I am new to R and not sure why I have to rename data frame column names at the end of the program though I have defined data frame with column names at the beginning of the program. The use of the data frame is, I got two columns where I have to save sequence under ID column and some sort of number in NOBS column.
complete <- function(directory, id = 1:332) {
collectCounts = data.frame(id=numeric(), nobs=numeric())
for(i in id) {
fileName = sprintf("%03d",i)
fileLocation = paste(directory, "/", fileName,".csv", sep="")
fileData = read.csv(fileLocation, header=TRUE)
completeCount = sum(!is.na(fileData[,2]), na.rm=TRUE)
collectCounts <- rbind(collectCounts, c(id=i, completeCount))
#print(completeCount)
}
colnames(collectCounts)[1] <- "id"
colnames(collectCounts)[2] <- "nobs"
print(collectCounts)
}
Its not quite clear what your specific problem is, as you did not provide a complete and verifiable example. But I can give a few pointers on improving the code, nonetheless.
1) It is not recommended to 'grow' a data.frame within a loop. This is extremely inefficient in R, as it copies the entire structure each time. Better is to assign the whole data.frame at the outset, then fill in the rows in the loop.
2) R has a handy functionpaste0 that does not require you to specify sep = "".
3) There's no need to specify na.rm = TRUE in your sum, because is.na will never return NA's
Putting this together:
complete = function(directory, id = 1:332) {
collectCounts = data.frame(id=id, nobs=numeric(length(id)))
for(i in 1:length(id)) {
fileName = sprintf("%03d", id[i])
fileLocation = paste0(directory, "/", fileName,".csv")
fileData = read.csv(fileLocation, header=TRUE)
completeCount = sum(!is.na(fileData[, 2]))
collectCounts[i, 'nobs'] <- completeCount
}
}
Always hard to answer questions without example data.
You could start with
collectCounts = data.frame(id, nobs=NA)
And in your loop, do:
collectCounts[i, 2] <- completeCount
Here is another way to do this:
complete <- function(directory, id = 1:332) {
nobs <- sapply(id, function(i) {
fileName = paste0(sprintf("%03d",i), ".csv")
fileLocation = file.path(directory, fileName)
fileData = read.csv(fileLocation, header=TRUE)
sum(!is.na(fileData[,2]), na.rm=TRUE)
}
)
data.frame(id=id, nobs=nobs)
}
This question already has answers here:
What's wrong with my function to load multiple .csv files into single dataframe in R using rbind?
(6 answers)
Closed 5 years ago.
I am quite new to R and I need some help. I have multiple csv files labeled from 001 to 332. I would like to combine all of them into one data.frame. This is what I have done so far:
filesCSV <- function(id = 1:332){
fileNames <- paste(id) ## I want fileNames to be a vector with the names of all the csv files that I want to join together
for(i in id){
if(i < 10){
fileNames[i] <- paste("00",fileNames[i],".csv",sep = "")
}
if(i < 100 && i > 9){
fileNames[i] <- paste("0", fileNames[i],".csv", sep = "")
}
else if (i > 99){
fileNames[i] <- paste(fileNames[i], ".csv", sep = "")
}
}
theData <- read.csv(fileNames[1], header = TRUE) ## here I was trying to create the data.frame with the first csv file
for(a in 2:332){
theData <- rbind(theData,read.csv(fileNames[a])) ## here I wanted to use a for loop to cycle through the names of files in the fileNames vector, and open them all and add them to the 'theData' data.frame
}
theData
}
Any help would be appreciated, Thanks!
Hmm it looks roughly like your function should already be working. What is the issue?
Anyways here would be a more idiomatic R way to achieve what you want to do that reduces the whole function to three lines of code:
Construct the filenames:
infiles <- sprintf("%03d.csv", 1:300)
the %03d means: insert an integer value d padded to length 3 zeroes (0). Refer to the help of ?sprintf() for details.
Read the files:
res <- lapply(infiles, read.csv, header = TRUE)
lapply maps the function read.csv with the argument header = TRUE to each element of the vector "infiles" and returns a list (in this case a list of data.frames)
Bind the data frames together:
do.call(rbind, res)
This is the same as entering rbind(df1, df2, df3, df4, ..., dfn) where df1...dfn are the elments of the list res
You were very close; just needed ideas to append 0s to files and cater for cases when the final data should just read the csv or be an rbind
filesCSV <- function(id = 1:332){
library(stringr)
# Append 0 ids in front
fileNames <- paste(str_pad(id, 3, pad = "0"),".csv", sep = "")
# Initially the data is NULL
the_data <- NULL
for(i in seq_along(id)
{
# Read the data in dat object
dat <- read.csv(fileNames[i], header = TRUE)
if(is.null(the_data) # For the first pass when dat is NULL
{
the_data <- dat
}else{ # For all other passes
theData <- rbind(theData, dat)
}
}
return(the_data)
}
I have a file in CSV format which contains a table with column "id", "timestamp", "action", "value" and "location".
I want to apply a function to each row of the table and I've already written the code in R as follows:
user <- read.csv(file_path,sep = ";")
num <- nrow(user)
curLocation <- "1"
for(i in 1:num) {
row <- user[i,]
if(user$action != "power")
curLocation <- row$value
user[i,"location"] <- curLocation
}
The R script works fine and now I want to apply it SparkR. However, I couldn't access the ith row directly in SparkR and I couldn't find any function to manipulate every row in SparkR documentation.
Which method should I use in order to achieve the same effect as in the R script?
In addition, as advised by #chateaur, I tried to code using dapply function as follows:
curLocation <- "1"
schema <- structType(structField("Sequence","integer"), structField("ID","integer"), structField("Timestamp","timestamp"), structField("Action","string"), structField("Value","string"), structField("Location","string"))
setLocation <- function(row, curLoc) {
if(row$Action != "power|battery|level"){
curLoc <- row$Value
}
row$Location <- curLoc
}
bw <- dapply(user, function(row) { setLocation(row, curLocation)}, schema)
head(bw)
Then I got an error:
I looked up the warning message the condition has length > 1 and only the first element will be used and I found something https://stackoverflow.com/a/29969702/4942713. It made me wonder whether the row parameter in the dapply function represent an entire partition of my data frame instead of one single row? Maybe dapply function is not a desirable solution?
Later, I tried to modify the function as advised by #chateaur. Instead of using dapply, I used dapplyCollect which saves me the effort of specifying the schema. It works!
changeLocation <- function(partitionnedDf) {
nrows <- nrow(partitionnedDf)
curLocation <- "1"
for(i in 1:nrows){
row <- partitionnedDf[i,]
if(row$action != "power") {
curLocation <- row$value
}
partitionnedDf[i,"location"] <- curLocation
}
partitionnedDf
}
bw <- dapplyCollect(user, changeLocation)
Scorpion775,
You should share your sparkR code. Don't forget that data isn't manipulated the same way in R and sparkR.
From : http://spark.apache.org/docs/latest/sparkr.html,
df <- read.df(csvPath, "csv", header = "true", inferSchema = "true", na.strings = "NA")
Then you can look at dapply function here : https://spark.apache.org/docs/2.1.0/api/R/dapply.html
Here is a working example :
changeLocation <- function(partitionnedDf) {
nrows <- nrow(partitionnedDf)
curLocation <- as.integer(1)
# Loop over each row of the partitionned data frame
for(i in 1:nrows){
row <- partitionnedDf[i,]
if(row[1] != "power") {
curLocation <- row[2]
}
partitionnedDf[i,3] <- curLocation
}
# Return modified data frame
partitionnedDf
}
# Load data
df <- read.df("data.csv", "csv", header="false", inferSchema = "true")
head(collect(df))
# Define schema of dataframe
schema <- structType(structField("action", "string"), structField("value", "integer"),
structField("location", "integer"))
# Change location of each row
df2 <- dapply(df, changeLocation, schema)
head(df2)
I would like to design a function. Say I have files file1.csv, file2.csv, file3.csv, ..., file100.csv. I only want to read some of them every time I call the function by specifying an integer vector id, e.g., id = 1:10, then I will read file1.csv,...,file10.csv.
After reading those csv files, I would like to row combine them into a single variable. All csv files have the same column structure.
My code is below:
namelist <- list.files()
for (i in id) {
assign(paste0( "file", i ), read.csv(namelist[i], header=T))
}
As you can see, after I read in all the data matrix, I stuck at combining them since they all have different variable names.
You should read in each file as an element of a list. Then you can combine them as follows:
namelist <- list.files()
df <- vector("list", length = length(id))
for (i in id) {
df[[i]] <- read.csv(namelist[i], header = TRUE)
}
df <- do.call("rbind", df)
Or more concisely:
df <- do.call(rbind, lapply(list.files(), read.csv))
I do this, which is more R like without the for loop:
## assuming you have a folder full of .csv's to merge
filenames <- list.files()
all_files <- Reduce(rbind, lapply(filenames, read.csv))
If I understand correctly what you want to do then this is all you need:
namelist <- list.files()
singlevar = c()
for (i in id) {
singlevar = rbind(singlevar, read.csv(namelist[i], header=T))
}
Since in the end you want one single object to contain all the partial information from the single files, rbind as you go.