Are there any workarounds to the problem with ncdf4::nc_open not being able to access some .nc files from a URL? I would like to avoid having to download the file first since this is for a shiny app deployed on a server and so I want to avoid users being able to download files to the server.
Some URLs work e.g. this OPeNDAP URL from a THREDDS server:
library(ncdf4)
nc <- nc_open("https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc")
But others do not e.g. this NetCDF Subset Service URL from a THREDDS server:
nc <- nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1")
# Error in nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1") :
# Error in nc_open trying to open file https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1
or this file directly from a website:
nc <- nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc")
# syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
# context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p></body></html>
# Error in R_nc4_open: NetCDF: file not found
# Error in nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc") :
# Error in nc_open trying to open file https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc
What could be the reason why some methods work and some don't, and are there any ways to fix this?
I ran into the same issue and contacted the creator of ncdf4. Here is his reply:
The ncdf4 R package is an interface to the underlying netcdf library on your machine. The R package cannot avoid limitations or problems with the netcdf library itself. You can always see how the underlying netcdf library handles a URL by just trying to use "ncdump" with the URL. For example,
% ncdump "https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc"
Gives:
netcdf Test_pixel_count {
dimensions:
lat = 283 ;
lon = 451 ;
variables:
byte crs ;
crs:_Unsigned = "false" ;
crs:grid_mapping_name = "latitude_longitude" ;
crs:long_name = "CRS definition" ;
crs:longitude_of_prime_meridian = 0. ;
crs:semi_major_axis = 6378137. ;
crs:inverse_flattening = 298.257222101 ;
crs:spatial_ref = "GEOGCS[\"GEOCENTRIC DATUM of AUSTRALIA\",DATUM[\"GDA94\",SPHEROID[\"GRS80\",6378137,298.257222101]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433],AXIS[\"Longitude\",EAST],AXIS[\"Latitude\",NORTH]]" ;
crs:GeoTransform = "141.4607205456306 0.0075 0 -22.95189554231917 0 -0.0075 " ;
etc. I.e., the netcdf library itself can read this URL/file.
However when I do this:
% ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"
I get this error:
ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"
ncdump: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1:
NetCDF: Malformed or unexpected Constraint
So the netcdf library itself cannot access that URL/file, and therefore ncdf4 cannot do so. There's not anything I would be able to do to the ncdf4 library to make up for this.
You might try contacting the netcdf library folks and ask if this is expected behavior or a bug.
Related
Trying to pull in and scale a Landsat image but receive an error, this seems to be a somewhat common issue given the older questions and various ways of creating a RasterLayer object.
After loading the libraries I need I get to this and hit an issue:
Directory <- "D:/Geo Files/LANDSAT"
prefix <- "CU_LC08.001_"
suffix <- "_doy2020222_aid0001.tif"
## Get band 2 reflectance (blue, 0.45-0.515 micron)
sr2 <- raster(paste0(Directory,prefix,"SRB2",suffix))
Error in .local(.Object, ...) :
*Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)*
> sr2 = sr2*0.0001 #Scale
Error: object 'sr2' not found
The full name of the first (SRB2) file I want to use is:
CU_LC08.001_SRB2_doy2020222_aid0001.tif - the name of the file as it's downloaded, unchanged.
The rest of the code continues through the other files to go through each SRB and later calculate NDVI, etc
i.e. ## Get band 3 reflectance (green, 0.533-0.590 micron) sr3 <- raster(paste0(Directory,prefix,"SRB3",suffix)) sr3 = sr3*0.0001 #Scale
Edit 1: I tried using try using \ instead of forward slashes and got exactly the same error.
R version is R-4.0.3
Any help would be appreciated, this was provided to us in a hackathon and I am new to R, it should be simple to run but if the file cannot be read..?
EDIT 2: The issue was with the end of the path, it needed "\"
Directory <- "D:\\Geo Files\\LANDSAT\\"
But when trying to get another file to work from a different folder later in the code, the same issue appears, and the above fix did nothing:
Directory <- "D:\\Geo Files\\ECOSTRESS\\USCities\\"
LST_ECO <- raster(paste0(Directory,"SDS_LST_doy2020260040819_aid0001.tif"))
Error in .local(.Object, ...) :
Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)
EDIT 3: Changed \Geo Files\ to "\GeoFiles\" as suggested, tried again but problem still occurs as above.
I had the same issue and the following was the process that finally worked.
# location of the file
loc_1 = "D:\\Geo Files\\ECOSTRESS\\USCities\\SDS_LST_doy2020260040819_aid0001.tif"
# check if the file can be read into r
tif1 = readr::read_file(loc_1)
It raised the following error (edited):
Error: Cannot read file "D:\\Geo Files\\ECOSTRESS\\USCities\\SDS_LST_doy2020260040819_aid0001.tif": The cloud file provider is not running.
The reason was because R could not read the file as it was in an online folder.
Confirm if the file is available offline. If it can be read by read_file function then the raster function should work.
I had the same issue
i was loading a raster file from my local storage
I added ".tif" extension to the name of the file and it worked
I need the data of the .rdata file for text mining. These are my dataset. I don't know exactly what's in that file. The problem is i can't load it.
I tried to open the file with different windows computers but with the same errors. I used RStudio in the updated version. I google the error-Information but nothing worked. Because I can open other rdata files there should be no registry problem. I wanted to check in an other basic windows Editor to look what is in the file but there were only signs like: ‹ ìùuPo³6
’#p ÜÝ‚»»»[pw—…»»»»»†àînÁuA°E`!‡ßûîýí}æÌLMÕÌùæŸÝõTñÈ}÷ÝrõÕ½
¢hXˆ
I tried different possibilities to open the file in RStudio with different error informations as followed:
with load()
require("readr")
setwd("C:/Users/..")
options(stringsAsFactors = F)
load("file")
# Error in load("file") :
# bad restore file magic number (file may be corrupted) -- no data loaded
# In addition: Warning message:
# file ‘.rdata’ has magic number ''
# Use of save versions prior to 2 is deprecated
with source()
require("readr")
setwd("C:/Users/..")
options(stringsAsFactors = F)
source("file")
# Error in source("file") :
# file.rdata:1:1: unexpected input
# 1:
# ^
readRDS
setwd("C:/Users/..")
options(stringsAsFactors = F)
readRDS("file")
# Error in readRDS("file") : unknown input format
I am new to hdf5 files. Trying to read some sample files from the below URL..
https://support.hdfgroup.org/ftp/HDF5/examples/files/exbyapi/
while trying to reading one of the .h5 files in R environment
library(rhdf5)
h5ls("h5ex_d_sofloat.h5")
I am getting the below error
Error in H5Fopen(file, "H5F_ACC_RDONLY") :
HDF5. File accessability. Unable to open file.
help is appreciated.
There was some issues with windows itself which was encrypting the hdf5 file while downloading it with default arguments. while downloading the just change the mode as "wb"..
file_url <- "http://support.hdfgroup.org/ftp/HDF5/examples/files/exbyapi/h5ex_d_sofloat.h5"
library(rhdf5)
download.file(url = file_url,destfile = "h5ex_d_sofloat.binary.h5",mode = "wb")
h5ls("h5ex_d_sofloat.binary.h5")
> group name otype dclass dim
0 / DS1 H5I_DATASET FLOAT 64 x 32
i got this solution from biocondutor itself...
https://support.bioconductor.org/p/97311/#97362
When I run "send2cy" function in R studio, I got error.
# Basic setup
library(igraph)
library(RJSONIO)
library(httr)
dir <- "/currentdir/"
setwd(dir)
port.number = 1234
base.url = paste("http://localhost:", toString(port.number), "/v1", sep="")
print(base.url)
# Load list of edges as Data Frame
network.df <- read.table("./data/eco_EM+TCA.txt")
# Convert it into igraph object
network <- graph.data.frame(network.df,directed=T)
# Remove duplicate edges & loops
g.tca <- simplify(network, remove.multiple=T, remove.loops=T)
# Name it
g.tca$name = "Ecoli TCA Cycle"
# This function will be published as a part of utility package, but not ready yet.
source('./utility/cytoscape_util.R')
# Convert it into Cytosccape.js JSON
cygraph <- toCytoscape(g.tca)
send2cy(cygraph, 'default%20black', 'circular')
Error in file(con, "r") : cannot open the connection
Called from: file(con, "r")
But I didn't find error when I use "send2cy" function from terminal R (Run R from terminal just calling by "R").
Any advice is welcome.
I tested your script with local copies of the network data and utility script, and with updated file paths. The script ran fine for me in R Studio.
Given the error message you are seeing "Error in file..." I suspect the issue is with your local files and file paths... somehow in an R Studio-specific way?
FYI: an updated, consolidated and update set of R scripts for Cytoscape are available here: https://github.com/cytoscape/cytoscape-automation/tree/master/for-scripters/R. I don't think anything has significantly changed, but perhaps trying in a new context will resolve the issue you are facing.
I'm trying to import a SRTM dataset into R. I've downloaded the data in a tif file however am having trouble reading it in "R".
Ive tried using the following code:
t = readTIFF("srtm_56_06/srtm_56_06.tif", as.is=TRUE)
load('srtm_56_06/srtm_56_06.tif')
read_file<-as.matrix(raster("srtm_56_06/srtm_56_06.tif")
However I am still getting error messages:
load('srtm_56_06/srtm_56_06.tif')
# Error: bad restore file magic number (file may be corrupted) -- no data loaded
# In addition: Warning message:
# file ‘srtm_56_06.tif’ has magic number 'II*'
# Use of save versions prior to 2 is deprecated
library(raster)
t = readTIFF("srtm_56_06/srtm_56_06.tif", as.is=TRUE)
# Error: could not find function "readTIFF"
read_file<-as.matrix(raster("srtm_56_06/srtm_56_06.tif") + min(read_file)
# Error: unexpected symbol in:
# "read_file<-as.matrix(raster("srtm_56_06/srtm_56_06.tif")
# min"
Can anyone help me with the commands to import this data. I'm a novice at "R" and a little lost.
Just read it with raster, but note you depend on rgdal being installed as well to read a .tif.
library(raster)
library(rgdal)
r <- raster("srtm_56_06/srtm_56_06.tif")
If that works, try
plot(r)
r
If it's really a "TIFF" then that should be fine, if it's really a GeoTIFF then you'll have a sensible map as well. (If it's something else that GDAL can read you might get a good result anyway, remember the extension of a file is not a reliable indicator of its contents).
The SRTM clue suggests that this is a single band DEM file from the tiled global SRTM data set. If it's somehow a "multi-band image" then you could read that with brick and plot with plotRGB (but I really doubt that is the case here). Note that there is a native binary format for SRTM that raster/rgdal could read as well but either they distributed .tif as well or someone else converted it.
There are a number of misconceptions in your code:
load is for a particular file type created from R (not these .tifs)
readTIFF is not in package raster
read_file would be a sensible matrix, if you have rgdal installed (which raster must use to load a .tif), but why throw away the spatial metadata?