I would like to create a RasterLayer from GeoTIFF data provided by a server. I'll query the server for this data using a httr::GET call (the data is provided on-demand, so in the application there won't be a url ending in .tif but a query url).
After writing the result of this call to disk as a GeoTIFF file it's easy enough to create the RasterLayer from the resulting GeoTIFF file on disk:
library(httr)
library(raster)
url <- 'http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif'
geotiff_file <- tempfile(fileext='.tif')
httr::GET(url,httr::write_disk(path=geotiff_file))
my_raster <- raster(geotiff_file)
my_raster
However, I would like to skip the write to disk part and create the raster straight from the in-memory server response.
response <- httr::GET(url,httr::write_memory())
response
The content of the response is a raw string which I would need to interpret as geoTIFF data.
str(httr::content(response))
However, I can only find raster or rgdal functions to read from a file. Any suggestions on translating this raw string to a raster?
Thanks!
GDAL has some cool virtual file system driver, one of which is /vsicurl that
allows on-the-fly random reading of files available through HTTP/FTP web protocols, without prior download of the entire file. It requires GDAL to be built against libcurl.
Since the raster package builds on rgdal you can simply do this:
library(raster)
r <- raster('/vsicurl/http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif')
plot(r)
Related
I have a set of raster geotif files in a bucket in google cloud storage that I would like to read and use in a web platform (like shiny).
To connect and download the data I am using googleCloudStorageR library and to display the raster I am using raster library.
Below is an example of the code to download a tif and then read it with raster library as any regular tif file which works fine.
library(googleCloudStorageR)
library(raster)
...authenthication and set bucket of interest (data omitted)...
#Download to a tif file and plot
download <- gcs_get_object(objects$name[[post]], bucket=bucket, saveToDisk = filegeo1.tif, overwrite = TRUE)
r1<-raster(filegeo1.tif)
plot(r1)
However, what I want is to able to read into memory the data and plot without saving into a tif file locally.
#Download to raw in memory
raw_raster <- gcs_get_object(objects$name[[post]], bucket=bucket, saveToDisk = NULL)
str(raw_raster)
raw_raster
saveRDS(raw_raster, file = "raw_raster_from_googlecloudstorage.Rds")
The result of the above commands can be seen like:
Screenshot of console result
I saved the file as Rds so anyone can have a use it for testing purposes in this open link to download file in mega storage. If I could transform from this raw data to geotif it would be great and would solve my situation.
I have read other question reported about reading geotiff data straight from web URL but is not the same since I don't have an url for each file.
So what advice can you provide? I appreciate the help.
Thank you,
I have just downloaded some climate data in grib format. I want to use "R" to convert it to NetCDF format.
Furthermore, as the file consists of different variables, I would like to extract one variable at a time into individual files.
It's hard to answer this without your specific file. You should look into producing reproducible examples, especially if you're posting to the R board.
For R, check out library(raster) and library(ncdf4). I just grabbed the first grib1 file I saw, and put together a quick example.
library(raster)
library(ncdf4)
download.file(url = 'ftp://ftp.hpc.ncep.noaa.gov/grib/20130815/p06m_2013081500f030.grb', destfile = 'test.grb')
(r <- raster('test.grb'))
n <- writeRaster(r, filename = 'netcdf_in_youR_comp.nc', overwrite = TRUE)
1. RNOMADS
The package Rnomads has a function readgrib providing wrappers to external libraries allowing one to read grib files
2. converting to netcdf
If the GRIB data is on a regular lat-lon grid, then probably an easier way is to convert to netcdf as the support for reading that is more developed (and you are probably already used to using it)
You can convert grib in several ways, two of the easiest are
CDO:
cdo -f nc copy test.grb test.nc
Use "-f nc4" if you want netcdf4 conventions.
ECCODES (on a mac install with brew install eccodes)
grib_to_netcdf -o test.nc test.grb
you can use ncl installed on you computer
library(ncdf)
system(ncl_convert2nc xxxx.grb, internal = TRUE)
my.nc <- open.ncdf("result.nc")
print(my.nc)
Is it possible to load ftp files directly to the R workspace without downloading it?
I have 700+ files each around 1.5 Gb and I want to extract approx 0.1 % of the information from every files and add them into a single dataframe.
I had a look at Download .RData and .csv files from FTP using RCurl (or any other method), could not get it to work.
Edit: After some reading, i managed to get the files into R
library(httr)
r <- GET("ftp://ftp.ais.dk/ais_data/aisdk_20141001.csv", write_memory())
when i try to read the body i use
content(r, "text")
but the output is gibberish. It might be because of the encoding, but how do i know which encoding the server uses. Any ideas on how to get the original data from the ftp?
I found a solution, which is very simple, but works nonetheless:
library(data.table)
r <- fread("ftp://ftp.ais.dk/ais_data/aisdk_20141001.csv")
This blog was helpfull
I don't know how to deal with save.image()and saveRDS()with raster data in R. I have understood that raster package open a connexion with the image file using raster() function, so it doesn't really open the file into R workspace.
I want to save my workspace (data.frame, list, raster, etc) with save.image() function (or similar) and open it in a different computer. If I try to plot or process a raster object saved in a different computer, always have the same issue:
Error in .local(.Object, ...) :
`C:\path\to\file.tif' does not exist in the file system,
and is not recognised as a supported dataset name.
Is there a way to save a raster object (opened as external file) in R format? I don't mean raster format as tiff nor grid and others.
At your own risk, you can use the readAll function to load the raster into memory before saving. e.g.
r <- raster(system.file("external/test.grd", package="raster"))
r <- readAll(r) # force data into memory
save(r, file = 'r.RData')
It can be loaded on a different machine as mentioned
load('r.Rdata`)
Beware, this will be problematic for very large rasters on memory limited systems
You can save rasters, like other R objects, using the save command.
save(r,file="r.Rdata")
On a different computer, you can load that file using
load("r.Rdata")
which will bring back the raster r in your workspace.
I have tried this across Windows and Linux and it never gives problems
I have just downloaded some climate data in grib format. I want to use "R" to convert it to NetCDF format.
Furthermore, as the file consists of different variables, I would like to extract one variable at a time into individual files.
It's hard to answer this without your specific file. You should look into producing reproducible examples, especially if you're posting to the R board.
For R, check out library(raster) and library(ncdf4). I just grabbed the first grib1 file I saw, and put together a quick example.
library(raster)
library(ncdf4)
download.file(url = 'ftp://ftp.hpc.ncep.noaa.gov/grib/20130815/p06m_2013081500f030.grb', destfile = 'test.grb')
(r <- raster('test.grb'))
n <- writeRaster(r, filename = 'netcdf_in_youR_comp.nc', overwrite = TRUE)
1. RNOMADS
The package Rnomads has a function readgrib providing wrappers to external libraries allowing one to read grib files
2. converting to netcdf
If the GRIB data is on a regular lat-lon grid, then probably an easier way is to convert to netcdf as the support for reading that is more developed (and you are probably already used to using it)
You can convert grib in several ways, two of the easiest are
CDO:
cdo -f nc copy test.grb test.nc
Use "-f nc4" if you want netcdf4 conventions.
ECCODES (on a mac install with brew install eccodes)
grib_to_netcdf -o test.nc test.grb
you can use ncl installed on you computer
library(ncdf)
system(ncl_convert2nc xxxx.grb, internal = TRUE)
my.nc <- open.ncdf("result.nc")
print(my.nc)