download nc files with R - r

I am trying to download an nc file with R. It downloads well but I get this error when trying to open it:
Error in R_nc4_open: NetCDF: Unknown file format Error in
nc_open("SM_D2010323_Map_SATSSS_data_1day.nc") : Error in nc_open
trying to open file SM_D2010323_Map_SATSSS_data_1day.nc
(return_on_error= FALSE )
url <- "https://www.star.nesdis.noaa.gov/data/socd1/coastwatch/products/miras/nc/SM_D2010323_Map_SATSSS_data_1day.nc"
destfile <- "***/SM_D2010323_Map_SATSSS_data_1day.nc"
download.file(url, destfile)
nc_data <- nc_open('SM_D2010323_Map_SATSSS_data_1day.nc')
But when I use the same URL on my web browser, I can open the file without any problems with R.

Related

Reading zip file off the internet: Error in open.connection(file, "rt") : cannot open the connection

I have a url I'm trying to directly read into R using the following code:
url <- "https://oehha.ca.gov/media/downloads/calenviroscreen/document/ces3results.xlsx"
temp <- tempfile()
download.file(url, temp)
data <- read.table(unz(temp, 'ces3results.xlsx'))
unlink(temp)
and I get the following error:
Error in open.connection(file, "rt") : cannot open the connection
In addition: Warning message:
In open.connection(file, "rt") :
cannot open zip file '/var/folders/94/1ldxgg4d3tzg4sqznhj8m2qr0000gn/T//RtmpcHqBEX/fileb6f920054f3a'
From what I've researched, this may be a matter of the website not allowing people to download data directly from their url. Is it that or is my code just faulty?

Your URL does not seem to point to a .zip archive, and you can actually open it directly using openxlsx::read.xlsx.
url <- "https://oehha.ca.gov/media/downloads/calenviroscreen/document/ces3results.xlsx"
data1 <- openxlsx::read.xlsx(url)
To first download, then open it you may adapt your method a little. Use mode="wz", leave out the unz, finally method as above.
temp <- tempfile()
download.file(url, destfile = temp, mode="wb")
data2 <- openxlsx::read.xlsx(temp)
unlink(temp) ## cleanup
stopifnot(all.equal(data1, data2))

Download NASA satellite data using RCurl in R

I am trying to download a ncdf file using rCurl. Can anyone provide any advice on why this is not working?
require(RCurl)
require(ncdf4)
url <- "https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Seasonal_Climatology/4km/sst/"
filename <-"A20021722014263.L3m_SCSU_NSST_sst_4km.nc"
download.file(paste0(url, filename),destfile = paste0("~/Desktop/", filename), method="curl")
setwd("~/Desktop/")
files<-dir(pattern="*.nc")
f<-nc_open(files[1])
Error in R_nc4_open: NetCDF: Unknown file format
Error in nc_open(files[1]) :
Error in nc_open trying to open file A20021722014263.L3m_SCSU_NSST_sst_4km.nc

It appears that the file downloaded is an error file in XML format? If you open it in Notepad, you'll see it contains stuff like
Sorry, an error has occurred. Use the back button to return to the previous page or go to the Ocean Color Home Page
Are you sure that the filename you're wanting to download actually exists in that URL?

RStudio : Problem with downloading a ZIP file from an URL and read CSV files from ZIP file

I am relatively new to R programming. I am trying to download a few zip files which contain CSV files using URL and read them. Below are the code, URL and the errors. From errors I suspect it is only downloading some text or html code and not the ZIP file (the download is only 10KB as against 396KB for the ZIP file as shown on website). I have tried downloading a few other datsets from same site, but having the same issue. Appreciate if someone can help. Please note, I can directly download the ZIP files, extract and view the CSV files.
tempdl <- tempfile()
download.file("https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016/downloads/suicide-rates-overview-1985-to-2016.zip",tempdl, mode="wb")
unzip(tempdl, "master.csv")
data <- read.table("master.csv", sep=",")
the error I get is:
> download.file("https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016/downloads/suicide-rates-overview-1985-to-2016.zip",tempdl, mode="wb")
trying URL 'https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016/downloads/suicide-rates-overview-1985-to-2016.zip'
Content type 'text/html; charset=utf-8' length unknown
downloaded 10 KB
> unzip(tempdl, "master.csv")
Warning message:
In unzip(tempdl, "master.csv") : error 1 in extracting from zip file
> data <- read.table("master.csv", sep=",")
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
cannot open file 'master.csv': No such file or directory

r unable to download file from server

I am trying to download a pdf file that is stored in an internal server.
The url for this file is like this below
file_location <- "file://dory.lisa.org/research/data/test.pdf"
I tried downloading this file using the download.file option
download.file(file_location, "test.pdf",method='curl')
and i am getting an error.
curl: (37) Couldn't open file /research/data/test.pdf
Warning message:
In download.file(file_location, "test.pdf", method = "curl") :
download had nonzero exit status
I tried
url <- ('http://cran.r-project.org/doc/manuals/R-intro.pdf')
download.file(url, 'introductionToR.pdf')
And i have no problem downloading this file, but somehow it shows an error when I try to use the same approach to download a file on my server.

I'm guessing that the file does not exist at that location on your local drive. When I executed the couple of lines that downloaded from CRAN I get a pdf file in my User directory/folder. I then get success with this code:
url <- ('file://~/introductionToR.pdf')
download.file(url, 'NewintroductionToR.pdf')

How to download file from internet via R

I have an url, and I want to download the file via R, I notice that download.file would be helpful, but my problem seems different:
url <- "http://journal.gucas.ac.cn/CN/article/downloadArticleFile.do?attachType=PDF&id=11771"
destfile <- "myfile.pdf"
download.file(url, destfile)
It doesn't work! I notice that if my url is in the form of xxx.pdf, then the code above is no problem, otherwise the file that is downloaded is corrupt.
Does anyone know how to solve this problem?

Setting the mode might be required to treat the file as binary data while saving it. If I leave that argument out, I get a blank file, but this way works for me:
url <- "http://journal.gucas.ac.cn/CN/article/downloadArticleFile.do?
attachType=PDF&id=11771"
destfile <- "myfile.pdf"
download.file(url, destfile, mode="wb")

I am trying to download an nc file with R. It downloads well but I get this error when trying to open it:
Error in R_nc4_open: NetCDF: Unknown file format Error in
nc_open("SM_D2010323_Map_SATSSS_data_1day.nc") : Error in nc_open
trying to open file SM_D2010323_Map_SATSSS_data_1day.nc
(return_on_error= FALSE )
url <- "https://www.star.nesdis.noaa.gov/data/socd1/coastwatch/products/miras/nc/SM_D2010323_Map_SATSSS_data_1day.nc"
destfile <- "***/SM_D2010323_Map_SATSSS_data_1day.nc"
download.file(url, destfile)
nc_data <- nc_open('SM_D2010323_Map_SATSSS_data_1day.nc')
But when I use the same URL on my web browser, I can open the file without any problems with R.

Resources