I run into issues reading in csv files with dynamic names and avoiding hard coding the file path. I'd like short tidy code (non-hardcoded). If I hardcode the full path (everything before the "~") it reads in the files fine. But soft-coding (if that is the opposite of hard coding) the file path it gives the error (despite showing the correct path in the error. I have two variable parts of the file name that I paste into the file name before reading it in. If I avoid paste and just type a path per individual it also works.
#dynamic part I usually have in a loop with all the options.
part_a <- "outside" #other options here in my loop include "inside"
part_b <- "late" # other option "early" or "preterm"
#reading in the df
df <-read.csv(paste0("~/Data/FromR/clean_",part_a,part_b,"_2016.csv"),
check.names=FALSE, na.strings="null")
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'C:/Users/myname/Documents/Data/FromR/clean_outsidelate_2016.csv': No such file or directory
if I use getwd() in the first part of the paste in place of ~ as suggested here it works by producing this string "C:/Users/myname/Documents/MyR_Projects/Specific_R_project/" at the beginning of the paste. But how can I get it to work with the "~"? when using the ~ it stops at the "Documents" folder...
The desired outcome is to read in the file without error perform functions and repeat with other files. My loop works fine hardcoded, and I only wanted to make it more general or softcoded.
I just tried to read a file (testFile.txt) in my home from a different wdand it works fine with ~
myFile <- "testFile
mymy <- ".txt"
ciao <- read.delim(paste0("~/",myFile,mymy))
In powershell you can use %~% (have a look here tread), but I am not sure how to expand the $HOME in R.
#-------- edit
Have a look here and here. Basically any variable defined in your .Renviron should be accessible.
I am running a script for some batch copying/renaming operations using R. The script is executed using a .bat file for executing using command prompt in MS Windows. Note that the destination computer where this will be run does not allow external connections (internet of otherwise) to install new packages so the solution should be in base R.
I can print comments to screen using cat and the output shows up in the .Rout file generated after the script is run. The .Rout file gets overwritten each time the script is executed and I would like to create a separate log file instead.
Here's the relevant of code:
if(copy_files){
# tried different modes for the file statement below
log_con <- file(paste(format(Sys.time(), '%d-%b-%Y %H:%M:%S'),
'move duplicates.txt'),
open = 'at')
cat('Parent directory:\t\t' , file = log_con, append= F)
cat(parent_folder , file = log_con, append= T)
cat('\nSubfolder to copy files to:\t ', file = log_con, append= T)
cat(subfolder_old, file = log_con, append= T)
cat('\n\n', file = log_con, append= T)
# copying operations here - omitted for brevity
}
Using only the cat statements without the file and append argument works fine but the above code returns the following error messages:
> log_con <- file(paste(format(Sys.time(), '%d-%b-%Y %H:%M:%S'), 'move duplicates.txt'), open = 'at')
Error in file(paste(format(Sys.time(), "%d-%b-%Y %H:%M:%S"), "move duplicates.txt"), :
cannot open the connection
In addition: Warning message:
In file(paste(format(Sys.time(), "%d-%b-%Y %H:%M:%S"), "move duplicates.txt"), :
cannot open file '07-Aug-2018 15:50:36 move duplicates.txt': Invalid argument
> cat('Parent directory:\t\t' , file = log_con, append= F)
Error in cat("Parent directory:\t\t", file = log_con, append = F) :
cannot open the connection
In addition: Warning message:
In cat("Parent directory:\t\t", file = log_con, append = F) :
cannot open file '07-Aug-2018 15:48:11 move duplicates.txt': Invalid argument
From what I understand the error stems from the fact that the log file does not exist at the beginning and the connection can not be opened.
Looking at the documentation and the answer for How to create periodically send text to a "log file" while printing normal output to console? seems to suggest that including append = F in the first cat statement ought to work. I have tried versions where different/no mode is specified for file command with the same result. Answers for Add lines to a file seem to suggest the same. Am I missing something?
I could create a file and have R append lines to it each time but I want a unique log for each time the script is run.
one way to solve this is to use sink, at the start of the code put sink("R_code.log") once the whole process is done put sink() at the last line of code .I hope this will solve your problem and if you want to name it dynamically use paste0 inside sink function.
I'm new to R studio and was not well aware of this portal T&C, so was blocked for questing for 5 days.
I have a code for importing multiple files from any directory to R.
Using this code for doing so, but the problem is this code runs sometime and sometime it gets failed with mentioned error.
I tried to found the solution of this but yet not found any solution.
library(data.table)
t = setwd("/home/dp/vishan/olp_data/19164/1/")
files <- file.info(list.files(path = t,pattern = "", full.names=TRUE))
files = rownames(files)[files$size > 0]
temp <- lapply(files, fread, sep=",")
Error:
Error in FUN(X[[i]], ...) :
'input' can not be a directory name, but must be a single character string containing a file name, a command, full path to a file, a URL starting 'http[s]://', 'ftp[s]://' or 'file://', or the input data itself.
Thanks in advance!
try using
files <- file.info(list.files(path = t,pattern = "", full.names=TRUE))
files <- subset(files, !isdir & size > 0)
temp <- lapply(rownames(files), fread, sep=',')
since list.files also shows directories. The data.frame you create in files can be easily subset on the isdir column which indicates if this is a directory or a file.
I am trying and failing to write a process that will download a .zip archive, extract a particular Excel file from that archive, and load that Excel file into my R workspace without ever writing any of those files (the .zip or the .xls) to my hard drive.
I have written a version of this process that works for zipped .csvs, but it doesn't work for .xls. Here's how that version goes, using one of the URLs I'm targeting in my current project and using readWorksheetFromFile() instead of read.csv() at the appropriate moment:
library(XLConnect)
waed.old.link <- "http://eventdata.parusanalytics.com/data.dir/pitf.world.19950101-20121231.xls.zip"
waed.old.file <- "pitf.world.19950101-20121231.xls"
tmp <- tempfile()
download.file(waed.old.link, tmp)
tmp2 <- tempfile()
tmp2 <- unz(tmp, waed.old.file)
WAED.old <- readWorksheetFromFile(tmp2, sheet = 1, startRow = 3, startCol = 1, endCol = 73)
unlink(tmp)
unlink(tmp2)
And here's what pops up after line 8, the one that tries to ingest the spreadsheet as WAED.old:
Error in path.expand(filename) : invalid 'path' argument
I also tried read_excel() at that step and got the same result:
> WAED.old <- read_excel(tmp2, skip = 2)
Error in file.exists(path) : invalid 'file' argument
I gather that this has something to do with pointing readWorksheetFromFile() at a connection rather than a file, but I'm not sure that's right, and I don't know how to fix it if it is. I searched stackoverflow and the web for an answer but couldn't find one that was right on point. I'd really appreciate some help.
As you say, it is because unz returns a connection object for the file within the zip (but does not explicitly unzip that file), while readWorksheetFromFile expects a path to a file.
Use unzip to explicitly unzip the file.
tmp2 <- unzip(zipfile=tmp, files = waed.old.file, exdir=tempdir())
# readWorksheetFromFile(tmp2, ...)
This question already has answers here:
How to open CSV file in R when R says "no such file or directory"?
(16 answers)
Closed 2 years ago.
I'm new to R, and after researching this error extensively, I'm still not able to find a solution for it. Here's the code. I've checked my working directory, and made sure the files are in the right directory. Appreciate it. Thanks
pollutantmean <- function(directory, pollutant = "nitrate", id= 1:332)
{ if(grep("specdata",directory) ==1)
{
directory <- ("./specdata")
}
mean_polldata <- c()
specdatafiles <- as.character(list.files(directory))
specdatapaths <- paste(directory, specdatafiles, sep="")
for(i in id)
{
curr_file <- read.csv(specdatapaths[i], header=T, sep=",")
head(curr_file)
pollutant
remove_na <- curr_file[!is.na(curr_file[, pollutant]), pollutant]
mean_polldata <- c(mean_polldata, remove_na)
}
{
mean_results <- mean(mean_polldata)
return(round(mean_results, 3))
}
}
The error I'm getting is below:
Error in file(file, "rt") : cannot open the connection
file(file, "rt")
read.table(file = file, header = header, sep = sep, quote = quote,
dec = dec, fill = fill, comment.char = comment.char, ...)
read.csv(specdatapaths[i], header = T, sep = ",")
pollutantmean3("specdata", "sulfate", 1:10)
In addition: Warning message:
In file(file, "rt") :
cannot open file './specdata001.csv': No such file or directory
You need to change directory <- ("./specdata") to directory <- ("./specdata/")
Relative to your current working directory, you are looking for the file 001.csv, which is in your specdata directory.
This question is nearly impossible to answer without any context, since you have not provided us with the structure of your working directory here. Fortunately for you, I have already taken R Programming on Coursera, so I already did this homework question.
The reason why you see this error I guess is because RStudio lost the path of your working directory.
(1) Go to session...
(2) Set working directory...
(3) Choose directory...
--> Then you can see a window pops up.
--> Choose the folder where you store your data.
This is the way without any code that you change your working directory.
Hope this can help you.
Set your working directory one level/folder higher. For example, if it is already set as:
setwd("C:/Users/Z/Desktop/Files/RStudio/Coursera/specdata")
go up one level back and set it as:
setwd("C:/Users/Z/Desktop/Files/RStudio/Coursera")
In other words, do not make "specdata" folder as your working directory.
I just spent a lot of time trying to understand what was wrong on my code too...
And it seems to be simple if you are using windows.
When you name your file "blabla.txt" then windows name it "blabla.txt.txt"...
That's the same with .CSV files so windows create a file named "001.csv.csv" if you called it "001.csv"
So, when you create your .csv file, just rename it "001" and open it in R using read.table("/absolute/path/of/directory/with/required/001.csv")
It works for me.
close your R studio and run it again as an administrator. That did the magic for me. Hope it works for you and anyone going through this too.
One better check that can be done if you get such error while accessing a file is to use the file.exists("file_path/file_name") function. This function will return TRUE if the file is existing and accessible, else False.
If running on Windows try running R or R Studio as administrator to avoid Windows OS file system constraints.
Use setwd() to change to appropriate directory.
Use only filename to access any file in the working directory.
Navigate a folder above by using "../<filename>".
Got this error and found that RStudio on my Windows machine try to use \ as escape symbol, so had to replace it with \\ to deal with it.
Try file.exists function with your path, e.g.:
file.exists("D:\\R\\path_to_file.csv")
Error in file(file, "rt") :
I just faced the same error and resolved by removing spacing in address using paste0 instead of paste
filepath=paste0(directory,"/",filename[1],sep="")
I got this same error message and fixed it in the easiest way I could. I put my .csv file into a file folder on my desktop, opened desktop in the window next to console on RStudio, and then opened my file there, and checked the box next to my .csv file, then I used the "more" pull down menu at the top of this window to set this as my working directory...probably the easiest thing for SUPER beginners like me :)
This error also occurs when you try to use the result of getwd() directly in the path. The problem is the missingness of the "/" at the end. Try the following code:
projectFolder <- paste(getwd(), "/", sep = '')
The paste() is to concatenate the forward slash at the end.
I got my R code file from a friend and was not able to run read.csv command but If I copy the same command(read.csv ) to a new R script file, it ran fine.
Below command was not running in R code file shared by my friend, working directory,file name etc were all correct because If I created a new R script file and ran below command ,it worked.
df <- read.csv("file.csv",header=TRUE,stringsAsFactors = FALSE,strip.white =
TRUE,sep = ',')
issue/resolution:
I right clicked the R code file and unblocked the file and click save button and issue got resolved.
I your R code file is in Downloads folder in windows , then move to some other folder.
I had a same issue .I removed the extension from the file name.Example my file name was saved as xyz. csv. i saved it as xyz.
Error in file(file, "rt")
Created a .r file and saved it in Desktop together with a sample_10000.csv file.
Once trying to read it
heisenberg <- read.csv(file="sample_100000.csv")
was getting the same error as you
heisenberg <- read.csv(file="sample_10000")
Error in file(file, "rt") : cannot open the connection In addition: Warning message:
In file(file, "rt") : cannot open file 'sample_10000': No such file or
directory
I knew at least two ways to fix this, one using the absolute path and the other changing the working directory.
Absolute path
I fixed it adding the absolute path to the file, more precisely
heisenberg <- read.csv(file="C:/Users/tiago/Desktop/sample_100000.csv")
Working directory
This error shows up because RStudio has a specific working directory defined which isn't necessarily the place the .r file is at.
So, to fix using this approach I've gone to Session > Set Working Directory > Chose Directory (CTRL + Shift + H) and selected Desktop, where the .csv file was at. That way running the following command also worked
heisenberg <- read.csv(file="sample_100000.csv")
I was getting the same error when trying to import 10,000 files. I tried opening a single file with Excel and Excel gave me an error message: "the file path is too long".
The file was buried in 6 nested folders, which is a problem of itself. Moving all the files to a desktop folder solved the issue without having to change any code.
I came across a solution based on a few answers popped in the thread.
(windows 10)
setwd("D:/path to folder where the files are")
directory <- getwd()
myfile<- "a_file_in_the_folder.txt"
filepath=paste0(directory,"/",myfile[1],sep="")
table <- read.table(table, sep = "\t", header=T, row.names = 1, dec=",")