I have a large file stored in linux. I don't want to transfer the file onto my laptop and then read into R. I was hoping I can read the large file into R without storing the file on my laptop (as my storage is nearly full). The file I want to read into R Studio is located in my university file path: /data/genes/h3/PROs_GWAS/output_PROs.bgen
The file is not a txt file but a genotype file e.g. ending is .bgen
I have tried the command below:
d = read.table( pipe ('ssh hkj7#spectre2.le.ac.uk "ls /data/genes/h3/PROs_GWAS/output_PROs.bgen"'), header = T )
However, this prompts me to a password but then an error which I am assuming is because of the read.table thinking the file is a txt file.
Error in read.table(pipe("ssh hkj7#spectre2.le.ac.uk \"ls /data/genes/h3/PROs_GWAS/output_PROs.bgen\""), :
no lines available in input
I am not sure how to get round this.
Any help will be greatly appreciated!
Related
I have an excel file that I want to open in R. I tried both of these commands after saving the excel file as a csv file or a text file.
read.table() or read.csv()
I think part of the problem is where the file is located. I have it saved on the desk top. What am I missing here?
Here is the R output
In file(file, "rt") :
cannot open file 'Rtrial.csv': No such file or directory
> help.search("read.csv")
> read.csv("Rtrial.csv")
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'Rtrial.csv': No such file or directory
> read.table("tab")
To throw out another option, why not set the working directory (preferably via a script) to the desktop using setwd('C:\John\Desktop') and then read the files just using file names
Try
f <- file.choose()
to choose the file interactively and save the name in f.
Then run read.csv on the saved filename
d <- read.csv(f)
Sound like you just have an issue with the path. Include the full path, if you use backslashes they need to be escaped: "C:\\folder\\folder\\Desktop\\file.csv" or "C:/folder/folder/Desktop/file.csv".
myfile = read.csv("C:/folder/folder/Desktop/file.csv") # or read.table()
It may also be wise to avoid spaces and symbols in your file names, though I'm fairly certain spaces are OK.
I had to combine Maiasaura and Svun answers to get it to work: using setwd and escaping all the slashes and spaces.
setwd('C:\\Users\\firstname\ lastname\\Desktop\\folder1\\folder2\\folder3')
data = read.csv("file.csv")
data
This solved the issue for me.
Here is one way to do it. It uses the ability of R to construct file paths based on the platform and hence will work on both Mac OS and Windows. Moreover, you don't need to convert your xls file to csv, as there are many R packages that will help you read xls directly (e.g. gdata package).
# get user's home directory
home = setwd(Sys.getenv("HOME"));
# construct path to file
fpath = file.path(home, "Desktop", "RTrial.xls");
# load gdata library to read xls files
library(gdata);
# read xls file
Rtrial = read.xls(fpath);
Let me know if this works.
Save as in excel will keep the file open and lock it so you can't open it. Close the excel file or you won't be able to use it in R.
Give the full path and escape backslashes read.csv("c:\\users\\JoeUser\\Desktop\\JoesData.csv")
I have experienced that this error occurs when you either move the excel file to the destination other than where your r file is located or when you move your r file to the destination other than where your excel file is located.
Good Practice:
Keep your .r and .csv files in the same directory.
open your .r file from getting into its directory instead of opening the r file from rstuio's open file option.
You also have import Dataset option at Environment Block, just click there and get your required packages installed & from next time use this option to read datasets. You will not get this error again.
I also appreciate the above provided answers.
Another way of reading Excel including the new format xlsx could be the package speedR (https://r-forge.r-project.org/projects/speedr/). It is an interactive and visual data importer. Besides importing you can filter(subset) the existing objects from the R workspace.
My issue was very simple, the working directory was not the "Source" directory that was printed when the file ran. To fix this, you can use getwd() and setwd() to get your relative links working, or just use a full path when opening the csv.
print(getwd()) # Where does the code think it is?
setwd("~/Documents") # Where do I want my code to be?
dat = read.csv("~/Documents/Data Visualization/expDataAnalysis/one/ac1_survey.csv") #just make it work!
MAC OS It happened to me as well. I simply chose from the R toolbar MISC and then chose Change Working Directory. I was able to choose the directory that the .csv file was saved in. When I went back to the command line and typed getwd() the full directory was updated and correct and the read.csv function finally worked.
I had the same problem and when I checked the properties of the file on file explorer, it shows me the next message:
"Security: This file came from another computer and might be blocked to help protect this computer"
You click on the "Unblock" button and... you can access to the file from R without any problem, just using read.csv() function and from the directory specified as your working directory, even if is not the same as the file’s directory you are accessing to.
I just had this problem and I first switched to another directory and then switched back and the problem was fixed.
this work for me, accesing data from root. use double slash to access address.
dataset = read.csv('C:\\Users\\Desktop\\Machine Learning\\Data.csv')
Kindly check whether the file name has an extension for example:
abc.csv
if so remove the .csv extension.
set wd to the folder containing the file (~)
data<-read.csv("abc.csv")
Your data has been read the data object
In my case this very problem was raised by wrong spelling, lower case 'c:' instead of upper case 'C:' in the path. I corrected spelling and problem vanished.
You can add absolute path to the file
heisenberg <- read.csv(file="C:/Users/tiago/Desktop/sample_100000.csv")
If really want to run something like
heisenberg <- read.csv(file="sample_100000.csv")
then you'll have to change the working directory to match the place the .CSV file is at. More about it here.
I currently saved some data as a csv file on my computer. It has 581 rows, but when I try to open the saved file on my mac, the dataframe has been altered and the numbers app from which I am looking at my csv from says some data was deleted. Is there a way to fix this? Or is there a different type of file I can save my data as that would adjust for the number of rows?
This is how I am writing the csv. I'm trying to manually add my file to a github repo after it has been saved to my computer.
write.csv(coords, 'Top_50_Distances.csv', row.names = FALSE)
I have an excel file that I want to open in R. I tried both of these commands after saving the excel file as a csv file or a text file.
read.table() or read.csv()
I think part of the problem is where the file is located. I have it saved on the desk top. What am I missing here?
Here is the R output
In file(file, "rt") :
cannot open file 'Rtrial.csv': No such file or directory
> help.search("read.csv")
> read.csv("Rtrial.csv")
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'Rtrial.csv': No such file or directory
> read.table("tab")
To throw out another option, why not set the working directory (preferably via a script) to the desktop using setwd('C:\John\Desktop') and then read the files just using file names
Try
f <- file.choose()
to choose the file interactively and save the name in f.
Then run read.csv on the saved filename
d <- read.csv(f)
Sound like you just have an issue with the path. Include the full path, if you use backslashes they need to be escaped: "C:\\folder\\folder\\Desktop\\file.csv" or "C:/folder/folder/Desktop/file.csv".
myfile = read.csv("C:/folder/folder/Desktop/file.csv") # or read.table()
It may also be wise to avoid spaces and symbols in your file names, though I'm fairly certain spaces are OK.
I had to combine Maiasaura and Svun answers to get it to work: using setwd and escaping all the slashes and spaces.
setwd('C:\\Users\\firstname\ lastname\\Desktop\\folder1\\folder2\\folder3')
data = read.csv("file.csv")
data
This solved the issue for me.
Here is one way to do it. It uses the ability of R to construct file paths based on the platform and hence will work on both Mac OS and Windows. Moreover, you don't need to convert your xls file to csv, as there are many R packages that will help you read xls directly (e.g. gdata package).
# get user's home directory
home = setwd(Sys.getenv("HOME"));
# construct path to file
fpath = file.path(home, "Desktop", "RTrial.xls");
# load gdata library to read xls files
library(gdata);
# read xls file
Rtrial = read.xls(fpath);
Let me know if this works.
Save as in excel will keep the file open and lock it so you can't open it. Close the excel file or you won't be able to use it in R.
Give the full path and escape backslashes read.csv("c:\\users\\JoeUser\\Desktop\\JoesData.csv")
I have experienced that this error occurs when you either move the excel file to the destination other than where your r file is located or when you move your r file to the destination other than where your excel file is located.
Good Practice:
Keep your .r and .csv files in the same directory.
open your .r file from getting into its directory instead of opening the r file from rstuio's open file option.
You also have import Dataset option at Environment Block, just click there and get your required packages installed & from next time use this option to read datasets. You will not get this error again.
I also appreciate the above provided answers.
Another way of reading Excel including the new format xlsx could be the package speedR (https://r-forge.r-project.org/projects/speedr/). It is an interactive and visual data importer. Besides importing you can filter(subset) the existing objects from the R workspace.
My issue was very simple, the working directory was not the "Source" directory that was printed when the file ran. To fix this, you can use getwd() and setwd() to get your relative links working, or just use a full path when opening the csv.
print(getwd()) # Where does the code think it is?
setwd("~/Documents") # Where do I want my code to be?
dat = read.csv("~/Documents/Data Visualization/expDataAnalysis/one/ac1_survey.csv") #just make it work!
MAC OS It happened to me as well. I simply chose from the R toolbar MISC and then chose Change Working Directory. I was able to choose the directory that the .csv file was saved in. When I went back to the command line and typed getwd() the full directory was updated and correct and the read.csv function finally worked.
I had the same problem and when I checked the properties of the file on file explorer, it shows me the next message:
"Security: This file came from another computer and might be blocked to help protect this computer"
You click on the "Unblock" button and... you can access to the file from R without any problem, just using read.csv() function and from the directory specified as your working directory, even if is not the same as the file’s directory you are accessing to.
I just had this problem and I first switched to another directory and then switched back and the problem was fixed.
this work for me, accesing data from root. use double slash to access address.
dataset = read.csv('C:\\Users\\Desktop\\Machine Learning\\Data.csv')
Kindly check whether the file name has an extension for example:
abc.csv
if so remove the .csv extension.
set wd to the folder containing the file (~)
data<-read.csv("abc.csv")
Your data has been read the data object
In my case this very problem was raised by wrong spelling, lower case 'c:' instead of upper case 'C:' in the path. I corrected spelling and problem vanished.
You can add absolute path to the file
heisenberg <- read.csv(file="C:/Users/tiago/Desktop/sample_100000.csv")
If really want to run something like
heisenberg <- read.csv(file="sample_100000.csv")
then you'll have to change the working directory to match the place the .CSV file is at. More about it here.
I'm trying to write a table into a macro-enabled Excel file (.xlsm) through the R. The write.xlsx (openxlsx) and writeWorksheetToFile (XLconnect) functions don't work.
When I used the openxlsx package, as seen below, the resulting .xlsm files ended up getting corrupted.
Code:
library(XLConnect)
library(openxlsx)
for (i in 1:3){
write.xlsx(Input_Files[[i]], Inputs[i], sheetName="Input_Sheet")
}
#Input_Files[[i]] are the R data.frames which need to be inserted into the .xslm file
#Inputs[i] are the excel files upon which the tables should be written into
Corrupted .xlsm file error message after write.xlsx:
Excel cannot open the file 'xxxxx.xslm' because the file format or file extension is not valid. Verify that the file has not been corrupted and that the file extension matches the format of the file
After researching this problem extensively, I found that the XLConnect connect package offers the writeWorksheetToFile function which works with .xlsm, albeit after running it a few times it yields an error message that there is no more free space. It also runs for 20+ minutes for tables with approximately 10,000 lines. I tried adding xlcFreeMemory at the beginning of the for loop, but it doesn't solve the issue.
Code:
library(XLConnect)
library(openxlsx)
for (i in 1:3){
xlcFreeMemory()
writeWorksheetToFile(Inputs[i], Input_Files[[i]], "Input_Sheet")
}
#Input_Files[[i]] are the R data.frames which need to be inserted into the .xslm file
#Inputs[i] are the excel files upon which the tables should be written into
Could anyone recommend a way to easily and quickly transfer an R table into an xlsm file without corrupting it?
I'm trying to use feather (v. 0.0.1) in R to read a fairly large (3.5 GB) csv file with 21178665 rows and 16 columns.
I use the following lines to load the file:
library(feather)
path <- "pp-complete.csv"
df <- read_feather(path)
But I get the following error:
Error: Invalid: File is too small to be a well-formed file
There's no explanation in the documentation of read_feather so I'm not sure what's the problem. I guess this function expects a different file form but I'm not sure what that would be.
Btw, I can read the file with read_csv in readr library but it takes a while.
The feather file format is distinct from a CSV file format. They are not interchangeable. The read_feather function cannot read simple CSV files.
If you want to read CSV files quickly, your best bets are probably readr::read_csv or data.table::fread. For large files, it will still usually take a while just to read it from disc.
After you've loaded the data into R, you can create a file in the feather format with write_feather so you can read it with read_feather the next time.