Related
I would like to write a RasterStack object and preserve names and metadata of the individual layers. How to preserve names is explained here. Is there a way to preserve metadata of individual layers when writing a RasterStack object?
Here is replicable code:
# load library
library(raster)
# create example rasters
ras_1 <- raster(nrows=180, ncols=360, xmn=-180, xmx=180, ymn=-90, ymx=90, resolution=, vals=1)
ras_2 <- raster(nrows=180, ncols=360, xmn=-180, xmx=180, ymn=-90, ymx=90, resolution=, vals=2)
ras_3 <- raster(nrows=180, ncols=360, xmn=-180, xmx=180, ymn=-90, ymx=90, resolution=, vals=3)
# assign names
names(ras_1) <- "raster_A"
names(ras_2) <- "raster_B"
names(ras_3) <- "raster_C"
# assign metadata
metadata(ras_1) <- list("metadata_raster_A")
metadata(ras_2) <- list("metadata_raster_B")
metadata(ras_3) <- list("metadata_raster_C")
# check
ras_1
ras_2
ras_3
metadata(ras_1)
metadata(ras_2)
metadata(ras_3)
# create and check stack
raster_stack <- stack(ras_1,
ras_2,
ras_3)
raster_stack
raster_stack[[1]]
metadata(raster_stack[[1]])
# write raster stack to disk
setwd("~")
# load library
library(terra)
# create rast object
raster_stack_terr <- rast(raster_stack)
# write raster stack
terra::writeRaster(raster_stack_terr, "raster_stack_terr_test.tif")
# load and check raster stack
raster_stack_check <- stack("raster_stack_terr_test.tif")
raster_stack_check
raster_stack_check[[1]]
names(raster_stack_check[[1]])
metadata(raster_stack_check[[1]])
Use terra to preseve names according to the 3rd answer from here.
When opening the RasterStack from disk, the metadata is not preserved. See console output:
> metadata(raster_stack_check[[1]])
list()
How to preserve metadata of individual layers when writing and re-loading a RasterStack object? Thanks!
It does not seem like {terra} offers an equivalent to raster::metadata(). However, from my perspective, the use cases would be limited here, because you would only be able to store structured information in corresponding format-specific tags (at least, this is my understanding) when writing to disk.
TIFF files (c.f. here) seem to offer the following tags:
TIFFTAG_DOCUMENTNAME
TIFFTAG_IMAGEDESCRIPTION
TIFFTAG_SOFTWARE
TIFFTAG_DATETIME
TIFFTAG_ARTIST
TIFFTAG_HOSTCOMPUTER
TIFFTAG_COPYRIGHT
TIFFTAG_XRESOLUTION
TIFFTAG_YRESOLUTION
TIFFTAG_RESOLUTIONUNIT
ESRI-Grids, on the other hand, do not offer any possibilities to store metadata except for the known header and maybe the filename as far as I know.
If you only wanted to store certain metadata with your raster object, you might as well make use of attr(r, "meta") <- "foobar". However, I don't see how this (random) information can be stored in specific formats and restored afterwards.
You already noticed names() when using {terra}, but there is also time() to be mentioned. Maybe this already suits your needs, since you did not specify what exactly you intend to store.
# set up a raster stack with three layers
library(terra)
#> terra 1.6.17
# create raster
r <- rast(nrows = 10, ncols = 10)
values(r) <- rnorm(100)
# set metadata
names(r) <- "foo"
time(r) <- as.Date("2000-01-01")
attr(r, "meta") <- "bar"
# inspect
r
#> class : SpatRaster
#> dimensions : 10, 10, 1 (nrow, ncol, nlyr)
#> resolution : 36, 18 (x, y)
#> extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84
#> source : memory
#> name : foo
#> min value : -2.503790
#> max value : 1.998731
#> time (days) : 2000-01-01
# write to disk
writeRaster(r, "sample.tif", overwrite = TRUE)
# read from disk
r2 <- rast("sample.tif")
r2
#> class : SpatRaster
#> dimensions : 10, 10, 1 (nrow, ncol, nlyr)
#> resolution : 36, 18 (x, y)
#> extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84 (EPSG:4326)
#> source : sample.tif
#> name : foo
#> min value : -2.503790
#> max value : 1.998731
#> time (days) : 2000-01-01
# try to access attributes
attr(r2, "meta")
#> NULL
As expected, data stored as attribute has been lost whereas information provided via names() and time() was sustained.
I need to project longitude/latitude coordinates in the terra package, but I don't believe it is working correctly, as I am trying to extract data from a raster with this projection, but the data is not being extracted correctly.
Here's my lon/lat points and the code I am using to try to project them.
latlon_df <- structure(list(Lon = c(-103.289, -96.6735, -96.9041, -96.76864,
-102.4694, -96.6814, -97.7504, -99.6754, -96.4802, -103.0007,
-96.8897, -101.8539, -103.9717, -101.253, -99.1134, -96.5849,
-98.0301, -99.9537, -99.4601, -99.7122, -103.8278, -98.931, -102.1081,
-101.7162, -100.115, -101.3448, -100.7805, -103.5606, -96.5302,
-99.4156, -103.281, -100.0063, -97.9928, -100.7208, -98.5289,
-96.762, -96.9218, -97.1024, -103.3793, -101.0841, -102.6745,
-96.9188, -97.5154, -100.7435, -98.6938), Lat = c(45.5194, 44.3099,
43.0526, 44.3252, 45.5183, 43.7316, 45.6796, 45.4406, 44.7154,
44.0006, 43.7687, 43.9599, 43.4737, 44.9875, 45.0292, 44.0867,
45.5735, 44.9895, 44.5256, 43.5938, 43.7343, 45.7163, 45.9189,
43.1672, 45.6716, 45.9154, 45.7963, 44.6783, 44.5073, 43.7982,
43.3784, 44.2912, 43.3841, 43.2002, 44.8579, 43.5048, 43.5033,
45.1055, 44.4245, 45.4167, 44.5643, 44.304, 45.2932, 43.5601,
43.7321)), class = "data.frame", row.names = c(NA, -45L))
latlons <- terra::vect(latlon_df,geom=c('Lon','Lat'),crs="+proj=longlat")
lcc <- terra::project(latlons,"+proj=lcc +lat_0=38.5 +lon_0=262.5 +lat_1=38.5 +lat_2=38.5 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs")
var_df <- terra::extract(grib_data,lcc)[,-1]
The raster data (grib_data) I am using comes from here (it is way too big for me to put on here). https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.20210612/conus/hrrr.t00z.wrfsubhf00.grib2
I am not sure what I am doing wrong here, as I have used this method previously, and it seemed to work fine. Any help would be wonderful.
EDIT: The specific problem I am having is that I am not getting any different values for each lon/lat pair. The value for each variable is different, but all the values for the stations (different lon/lats are the same).
Why do you think it has to do with the projection? Either way, it appears to work for me.
url <- "https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.20210612/conus/hrrr.t00z.wrfsubhf00.grib2"
if (!file.exist(basename(url))) download.file(url, basename(url), mode="wb")
url <- paste0(url, ".idx")
if (!file.exist(basename(url))) download.file(url, basename(url), mode="wb")
library(terra)
r <- rast("hrrr.t00z.wrfsubhf00.grib2")
r
#class : SpatRaster
#dimensions : 1059, 1799, 49 (nrow, ncol, nlyr)
#resolution : 3000, 3000 (x, y)
#extent : -2699020, 2697980, -1588806, 1588194 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=lcc +lat_0=38.5 +lon_0=262.5 +lat_1=38.5 +lat_2=38.5 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs
#source : hrrr.t00z.wrfsubhf00.grib2
#names : 0[-] ~here", 0[-] ~tops", 0[-] ~here", 0[-] ~here", 0[-] ~face", 1000[~ound", ...
You can check of the points overlap with the raster data
plot(r, 1)
points(lcc)
And extract. It takes very long with grib files, but it does appear to work
e <- extract(r, lcc)
head(e[,c(1,6,9)])
# ID 0[-] SFC="Ground or water surface" 0[-] SFC="Ground or water surface".1
#1 1 85100 11.775471
#2 2 54400 11.087971
#3 3 79300 9.900471
#4 4 49200 10.712971
#5 5 70800 9.212971
#6 6 56600 11.400471
Make sure you have the current (CRAN) version, or perhaps the development version that you can install like this:
install.packages('terra', repos='https://rspatial.r-universe.dev')
You can speed things up a lot by doing a single read from disk (by adding zero in this example)
e <- extract(r+0, lcc)
That is not always possible and I need to do some optimization behind the scences.
I have 365 .nc files located in a folder containing daily soil moisture information. I want to extract data at five-coordinate locations for the whole year and write them into five separate csv files. My code is attached below. However, I am getting this error after the line:
s <- stack(ff)
>Error in if (is.na(get("has_proj_def.dat", envir = .RGDAL_CACHE))) { : argument is of length zero In addition: Warning message: In .varName(nc, varname, warn = warn) : varname used is: sm If that is not correct, you can set it to one of: sm, sm_noise, flag, sensor
No idea how to proceed further.
library(raster)
library(ncdf4)
ptf <- "D://SMOS_ECV_SM//SMOS_ECV_SM//ECV_SM_Data_1978_2010//1978"
ff <- list.files(path=ptf, pattern="[.]nc$", full.names=TRUE)
s <- stack(ff)
points <- rbind(c(0,1), c(100,120), c(80,5), c(85,4), c(82,4))
v <- extract(s, points)
for (i in 1:ncol(v)) {
write.csv(v[,i,drop=FALSE], paste0("file", i, ".csv"))
}
library(raster)
#Loading required package: sp
f <- list.files("try", full=T)
First try for a single file
r <- raster(f[1])
#Loading required namespace: ncdf4
#Warning message:
#In .varName(nc, varname, warn = warn) : varname used is: sm
#If that is not correct, you can set it to one of: sm, sm_noise, flag, sensor
To get rid of the warning:
r <- raster(f[1], varname="sm")
Now for all files
s <- stack(f, varname="sm")
s
#class : RasterStack
#dimensions : 720, 1440, 1036800, 2 (nrow, ncol, ncell, nlayers)
#resolution : 0.25, 0.25 (x, y)
#extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#crs : +proj=longlat +datum=WGS84 +no_defs
#names : Soil.Moisture.1, Soil.Moisture.2
Extract values
points <- rbind(c(-96.7, 47), c(34.55, 54.85))
v <- extract(s, points)
v
# Soil.Moisture.1 Soil.Moisture.2
#[1,] 0.3254 0.3018
#[2,] 0.3386 0.3386
I have been trying to use the gdal_grid in R and while running the example data set I recieve a raster which has only zero values. I have tried experimenting with this with my own data, and have searched the forums with no luck. Can others get the example to work?
I have tried to explicitly call my path to my GDAL library, and have updated the version of GDAL. I am running R studio with version 3.3.1.
library(raster)
library(rgeos)
library(gdalUtils)
# We'll pre-check to make sure there is a valid GDAL install
# and that raster and rgdal are also installed.
# Note this isn't strictly neccessary, as executing the function will
# force a search for a valid GDAL install.
gdal_setInstallation()
valid_install <- !is.null(getOption("gdalUtils_gdalPath"))
if(require(raster) && valid_install)
{
# Create a properly formatted CSV:
temporary_dir <- tempdir()
tempfname_base <- file.path(temporary_dir,"dem")
tempfname_csv <- paste(tempfname_base,".csv",sep="")
pts <- data.frame(
Easting=c(86943.4,87124.3,86962.4,87077.6),
Northing=c(891957,892075,892321,891995),
Elevation=c(139.13,135.01,182.04,135.01)
)
write.csv(pts,file=tempfname_csv,row.names=FALSE)
# Now make a matching VRT file
tempfname_vrt <- paste(tempfname_base,".vrt",sep="")
vrt_header <- c(
'<OGRVRTDataSource>',
'\t<OGRVRTLayer name="dem">',
'\t<SrcDataSource>dem.csv</SrcDataSource>',
'\t<GeometryType>wkbPoint</GeometryType>',
'\t<GeometryField encoding="PointFromColumns" x="Easting" y="Northing" z="Elevation"/>',
'\t</OGRVRTLayer>',
'\t</OGRVRTDataSource>'
)
vrt_filecon <- file(tempfname_vrt,"w")
writeLines(vrt_header,con=vrt_filecon)
close(vrt_filecon)
tempfname_tif <- paste(tempfname_base,".tiff",sep="")
# Now run gdal_grid:
setMinMax(gdal_grid(src_datasource=tempfname_vrt,
dst_filename=tempfname_tif,a="invdist:power=2.0:smoothing=1.0",
txe=c(85000,89000),tye=c(894000,890000),outsize=c(400,400),
of="GTiff",ot="Float64",l="dem",output_Raster=TRUE))
}
r<-raster(tempfname_tif)
r
#class : RasterLayer
#dimensions : 400, 400, 160000 (nrow, ncol, ncell)
#resolution : 10, 10 (x, y)
#extent : 85000, 89000, 890000, 894000 (xmin, xmax, ymin, ymax)
#coord. ref. : NA
#data source : C:\Users\m.modeler\AppData\Local\Temp\RtmpW6HvOc\dem.tiff
#names : dem
#min values : 0
#max values : 0
plot(r)
Raster results plot with zero values:
Thanks much,
I have got the code to run by changing the path from the temp directory to a folder on my hard drive. Example below.
# change to a path on your computer
setwd("C:\\Users\\m.modeler\\Documents\\R\\gdal_Examples")
#######################################################
#create XYZ csv
pts <- data.frame(
Easting=c(86943.4,87124.3,86962.4,87077.6),
Northing=c(891957,892075,892321,891995),
Elevation=c(139.13,135.01,182.04,135.01))
write.csv(pts,file="dem.csv",row.names=FALSE)
#######################################################
#create VRT
fn_vrt<-"dem.vrt"
# Now make a matching VRT file
vrt_header <- c(
'<OGRVRTDataSource>',
'\t<OGRVRTLayer name="dem">',
'\t<SrcDataSource>dem.csv</SrcDataSource>',
'\t<GeometryType>wkbPoint</GeometryType>',
'\t<GeometryField encoding="PointFromColumns" x="Easting" y="Northing" z="Elevation"/>',
'\t</OGRVRTLayer>',
'\t</OGRVRTDataSource>')
vrt_filecon <- file(fn_vrt,"w")
writeLines(vrt_header,con=vrt_filecon)
close(vrt_filecon)
#######################################################
#create interpolated DEM
fn_tif <- "dem.tif"
# Now run gdal_grid:
r.dem <- setMinMax(gdal_grid(src_datasource=fn_vrt,
dst_filename=fn_tif,a="invdist:power=2.0:smoothing=1.0",
txe=c(85000,89000),tye=c(894000,890000),outsize=c(400,400),
of="GTiff",ot="Float64",l="dem",output_Raster=TRUE,verbose=TRUE))
plot(r.dem)
Through attempting to get an intersect result from a single point and a single polygon I have found what I believe can only be a bug in the R raster package intersect function.
I have 1 polygon and 1 point, and use intersect as follows:
intersect(a_point, a_polygon)
Where a_point contains an id attribute. This fails with the error:
Error in j[, 2] : incorrect number of dimensions
However, if I reverse the arguments and do:
intersect(a_polygon, a_point)
It works fine, but doesn't return the id from the point shape file as part of the result which I require. This is expected behaviour, so fine but I need it to work the other way around.
To rule out there being some peculiarity with my polygon or point data, I created a single polygon and single point spatial object and tested the same hypothesis, and the same result occurred as above with these 'raw' objects.
The following is the code for generating these two 'fake' objects for completeness and so that it can be reproduced:
test_list_x = list(530124, 530125) #For when I use 2 points
test_list_y = list(176949, 176950) #For when I use 2 points
data_frame_object = data.frame(530124, 176950)
names(data_frame_object) = c("Longitude", "Latitude")
coordinates(data_frame_object)=~Longitude+Latitude
proj4string(data_frame_object)=CRS("+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +datum=OSGB36 +units=m +no_defs +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894")
fake_point_shape_object=SpatialPointsDataFrame(data_frame_object, data.frame(id=1:length(data_frame_object)))
coords = matrix( nrow=5, ncol=2)
coords[1,1] = 530106.8
coords[1,2] = 176953.3
coords[2,1] = 530127.5
coords[2,2] = 176953.3
coords[3,1] = 530127.5
coords[3,2] = 176933.3
coords[4,1] = 530106.8
coords[4,2] = 176933.3
coords[5,1] = 530106.8
coords[5,2] = 176953.3
my_fake_polygon = Polygon(coords)
polygon_list = list(my_fake_polygon)
polygon_set <- lapply(seq_along(polygon_list), function(i) Polygons(list(polygon_list[[i]]), i ))
new_polygons <- SpatialPolygons(polygon_set)
new_polygons#proj4string = CRS("+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +datum=OSGB36 +units=m +no_defs +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894")
df <- data.frame("1")
names(df) = "id"
my_fake_polygon <- SpatialPolygonsDataFrame(new_polygons,df)
Now here's the thing, if I create 2 points next to each other (so they are both within the polygon) instead of just one, it works fine, no error. Suggesting there is a bug associated with intersection between 1 point and 1 polygon, WHEN the point carries an attribute to be returned in the intersection process.
You might ask why do you actually need to have the attribute returned if there is just one point, this is because it is an iterative process in which it may not be just one point, it could be none or many.
I would appreciate somebody explaining this error or confirming my findings.
Here are your example data in a more concise way.
library(raster)
pnt <- SpatialPoints(cbind(530124, 176950))
pol <- spPolygons(matrix(c(530106.8, 530127.5, 530127.5, 530106.8, 530106.8, 176953.3, 176953.3, 176933.3, 176933.3, 176953.3), ncol=2))
Now illustrate the problem.
intersect(pol, pnt)
#class : SpatialPolygons
#features : 1
#extent : 530106.8, 530127.5, 176933.3, 176953.3 (xmin, xmax, ymin, ymax)
#coord. ref. : NA
# this fails
intersect(pnt, pol)
#Loading required namespace: rgeos
#Error in j[, 2] : incorrect number of dimensions
# but it works with two points!
intersect(bind(pnt, pnt), pol)
#class : SpatialPoints
#features : 2
#extent : 530124, 530124, 176950, 176950 (xmin, xmax, ymin, ymax)
#coord. ref. : NA
This was another drop=TRUE bug caused by the R default of "dropping" matrices to vectors when a single row is selected. This was fixed in raster version 2.6-11 (not on CRAN yet).
Sorry i can't answer your intersect bug question, but it might be simpler for now to use sp::over to return polygon attributes to points
# dummy polygon
xym <- as.matrix(data.frame(x=c(16.48438,17.49512,24.74609,22.59277,16.48438),
y=c(59.73633,55.12207,55.03418,61.14258,59.73633)))
# make into SpatialPolygon
p = Polygon(xym)
ps = Polygons(list(p),1)
sps = SpatialPolygons(list(ps))
# Promote to SPDF and give an attribute
SPDF = SpatialPolygonsDataFrame(sps, data.frame(N = "hello", row.names = 1))
# make 2 points, one inside the polygon and one outside
p <- data.frame(x=c(16,18),y=c(58,58))
coordinates(p) <- ~x + y
# plot to check
plot(sps)
plot(p,add=T)
# perform the over, returns a named vector for every point in the SpatialPoints
res <- unname(over(p,SPDF))
# promote points to SpatialPointsDataFrame and put in new polygon attribute
data <- data.frame(ID=row.names(p),pol=res)
sp <- SpatialPointsDataFrame(p, data)