I have multiple NetCDF files that contain fire weather data. The files are rotated and also, other R packages (e.g., stars) and programs (e.g., cdo) do not recognize their crs but I can see the crs as WGS84 when I opened them with the raster package.
I'm looking into rotating the data to the common/regular lonlat structure, update the crs, crop the data to the extent of my study area, and writing out each NetCDF file (containing 365/366 layers) just as they were loaded. Example data here: https://www.filemail.com/t/JtIAB1zC
Here's the code that I'm working with:
all_nc = list.files("C:/file_path/",
pattern = ".nc$", recursive = F, full.names = T)
for(i in seq_along(all_nc)){
r = stack(all_nc[i])
r2 = raster::rotate(brick(r))
projectRaster(r2, crs = crs("+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0"))
r2 <- terra::crop(r2, terra::extent(-141.0069, -123.7893, 60, 69.64794))
writeRaster(r, file.path('D:/path/', names(r)),
force_v4 = T, overwrite=TRUE, format="CDF", compression = 7)
}
Error in if (filename == "") { : the condition has length > 1
I can get individual layers with all the changes that I want by tweaking the code but that's not what I'm hoping to achieve. I don't use loops often. So, I suspect that I'm screwing up the indexing somehow.
You can do that like this
fin <- list.files("C:/file_path/", pattern = ".nc$", full.names=TRUE)
fout <- gsub("C:/file_path/", "D:/path/", fin)
library(terra)
e <- ext(-141.0069, -123.7893, 60, 69.64794)
for(i in seq_along(fin)){
r <- rast(fin[i])
crs(r) <- "+proj=longlat +datum=WGS84"
r <- rotate(r)
r <- crop(r, e)
writeCDF(r, filename=fout[i], compression=7)
}
Instead of
r <- crop(r, e)
writeCDF(r, filename=fout[i], compression=7)
writeCDF is a better netCDF writer than writeRaster, but you could also use
r <- crop(r, e, filename=fout[i], gdal=c(COMPRESS="DEFLATE", ZLEVEL="7"))
## or
r <- crop(r, e)
writeRaster(r, filename=fout[i], gdal=c(COMPRESS="DEFLATE", ZLEVEL="7"))
Note that you are mixing up a lot of "raster" and "terra" code. In the above I only use "terra". Also could you point to or share one of the files. I would like to find out why GDAL does not detect the crs.
Related
I'm having trouble trying to read a plus 2Gb NetCDF file from links in R
If I try to read the whole file R returns me a message that I don't have enough memory,
'Error: cannot allocate vector of size 31.3 Gb'
as it was reported by the post link
Following the this post advice, I decided to read only the part of the file based on the geographic coordinates of the a given area. Here is my code:
#load any packages
library(ncdf4)
set path and filename
ncpath <- "C:\Users\Me\Documents\Science\GIS\Global Land Cover\"
ncname <- "C3S-LC-L4-LCCS-Map-300m-P1Y-2018-v2.1.1"
ncfname <- paste(ncpath, ncname, ".nc", sep="")
dname <-"lccs_class"
open a netCDF file
ncin <- nc_open(ncfname)
print(ncin)
get longitude and latitude
lon <- ncvar_get(ncin,"lon")
nlon <- dim(lon)
head(lon)
lat <- ncvar_get(ncin,"lat")
nlat <- dim(lat) head(lat)
print(c(nlon,nlat))
#' create a bounding box to work with a subset
LonIdx <- c(841, 842, 844,845,846,847,848)
LatIdx <- c(93,94,95,96)
However, when I try to execute the code for the Subset:
Susbset <- ncvar_get(ncin, dname,
start = c(LatIdx[1], LonIdx[1]),
count = c(length(LatIdx),length(LonIdx)))
I get the error:
> Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval,
> addOffset, : Error: variable has 3 dims, but start has 2 entries. They
> must match!
Can anyone help me? Much appreciated.
It seems like there is still a third dimension to specify before you can procceed.
Let's suppose it is a time variable:
start = c(LatIdx[1], LonIdx[1], DesiredTimeIdx),
count = c(length(LatIdx),length(LonIdx), 1))
This should read all lats an all lons in that specific time.
I have some code in R which does the following:
Uses lapply to bring in files in a set folder e.g. 1997 data
Makes file list into a brick - they are NetCDF files, so I've used brick function
Stacks the bricks into one raster stack of months for each year.
Calculate the mean from the stack
Crops the new mean raster to the Area of Interest (AOI).
I've got a working code, see below, but it is clunky and I feel could be better in one loop to then run through each year's folder (I have data from 1997 to 2018). Could anyone aid in streamlining this into a simple looped code I could run by changing the filepath? I've used loops a bit before but not from scratch.
# Packages:
library(raster)
library(parallel) # Check cores in PC
library(lubridate) # needed for lapply
library(dplyr) # ""
library(sf) # For clipping data
library(rgdal)
# ChlA
# Set file paths for input and outputs:
usingfp <- "/filepath/GIS/ChlA/1997/"
the_dir_ex <- "Data/CHL/1997"
# List all NETCDF files in folder:
CHL_1997 <- list.files(path = usingfp, pattern = "\\.nc$", full.names = TRUE,
recursive = FALSE)
# Make file list into brick
CHL_1997_brick <- lapply(CHL_1997,
FUN = brick,
the_dir = the_dir_ex)
# Stack bricks
s <- stack(CHL_1997_brick)
# Calculate mean from stack
mean <- calc(s, fun = mean, na.rm = T)
plot(mean)
# Load vector boundary to "crop" to
AOI <- readOGR("/filepath/AOI/AOI.shp")
plot(AOI,
main = "Shapefile imported into R - crop extent",
axes = TRUE,
border = "blue",
add = T)
# crop the raster using the vector extent
CHL_1997_mean <- crop(mean, AOI)
plot(CHL_1997_mean, main = "Cropped mean CHL - 1997")
# add shapefile on top of the existing raster
plot(AOI, add = TRUE)
Thanks very much.
Something like this should work
library(raster)
AOI <- shapefile("/filepath/AOI/AOI.shp")
path <- "/filepath/GIS/ChlA/"
years <- 1997:2018
for (yr in years) {
fp <- file.path(path, yr)
fout <- file.path(fp, paste0(year, ".tif"))
print(fout); flush.console()
# if (file.exists(fout)) next
files <- list.files(path=fp, pattern="\\.nc$", full.names=TRUE)
b <- lapply(files, brick)
s <- stack(b)
s <- mean(s)
s <- crop(s, AOI, filename=fout) #, overwrite=TRUE)
}
Notes:
mean(s) is more efficient than calc(s, mean)
If the AOI is relatively small, it can be more efficient to first
use crop, then mean (and then use writeRaster)
You can also use terra like this:
library(terra)
AOI <- vect("/filepath/AOI/AOI.shp")
path <- "/filepath/GIS/ChlA/"
years <- 1997:2018
for (yr in years) {
fp <- file.path(path, year)
fout <- file.path(fp, paste0(year, ".tif"))
print(fout); flush.console()
# if (file.exists(fout)) next
files <- list.files(path=fp, pattern="\\.nc$", full.names=TRUE)
r <- rast(files)
s <- mean(r)
s <- crop(s, AOI, filename=fout) #, overwrite=TRUE)
}
R Programming Language (New to this)
I am attempting to loop through a number of tiled rasters that have been output by splitRaster. During the loop I want to carry out some processes on each raster.
But the following code throws an error.
library(ForestTools)
library(raster)
library(sp)
library(rgdal)
library(SpaDES)
rm(list = ls())
tmpdir <- file.path(tempdir(), "splitRaster")
lin <- function(x){x * 0.1 + 0.6}
inCHM <- raster("input raster path and name.tif")
split <- splitRaster(inCHM, 5, 5, c(0.05, 0.05), tmpdir)
files <- list.files(path=tmpdir, pattern="*.grd", full.names=FALSE, recursive=FALSE)
file.names <- dir(tmpdir, pattern ="*.grd")
for(file.names in files ){
name <- file.names
ttops <- vwf(name, winFun = lin, minHeight = 5)
writeOGR(ttops, "output folder", name, driver = "ESRI Shapefile")
}
and this is the error
[1] "Xrastername_tile1.grd"
Error in CRS(x) :
PROJ4 argument-value pairs must begin with +: Xrastername_tile1.grd
More to the problem (24/7/2020),
I have removed the loop for trouble shooting instead just choosing one of the splitRasters outputs that would be used in the loop ie files[[3]]
When I run the following code the error is the same;
library(ForestTools)
library(raster)
library(sp)
library(rgdal)
library(SpaDES)
rm(list = ls())
# set temp directory
tmpdir <- "C:\\R-Test\\Temp_Output"
# get raster
r <- raster("C:\\Lidar\\grid_treeheight_max_1m_nofill.tif")
# define projection
projection(r) <- "+proj=utm +zone=50 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs"
# split raster brick
y <- splitRaster(r, 8, 8, c(0.05, 0.05), tmpdir)
# Get the complete file locations with full.names = T
files <- list.files(path=tmpdir, pattern="*.grd", full.names=FALSE, recursive=FALSE)
tmpfile <- paste(tmpdir, "\\", files[[3]], sep="")
lin <- function(x){x * 0.06 + 0.6}
ttops <- vwf(tmpfile, winFun = lin, minHeight = 5)
This is the error
Error in CRS(x) :
PROJ4 argument-value pairs must begin with +: D:\R-Test\Temp_Output\Xgrid_treeheight_max_1m_nofill_tile11.grd
When I run the following code using one of the splitRaster outputs (files[[3]]) from the above code it runs error free and I am able to plot ttops.
rm(list = ls())
# set temp directory
tmpdir <- "D:\\R-Test\\Temp_Output"
# get raster
r <- raster("D:\\R-Test\\Temp_Output\\Xgrid_treeheight_max_1m_nofill_tile11.grd")
lin <- function(x){x * 0.06 + 0.6}
ttops <- vwf(r, winFun = lin, minHeight = 5)
Why is the PROJ4 error occurring?
This seems to be the error that is causing the loop to fail?
I think the problem is that you are trying to feed the vwf function with the file name instead of a raster object. I would also recommend using lapply instead of for for the loop. Here is a code that should work
library(raster)
library(ForestTools)
library(rgdal)
# Get the complete file locations with full.names = T
files <- list.files(path=tmpdir, pattern="*.grd", full.names=T, recursive=FALSE)
# Loop over each item of the list, i.e., each raster
lapply(files, function(x){
# Load the image as raster
image <- raster(x)
# Calculate vwf (I added a dummy function for winFun)
ttops <- vwf(image, winFun = function(x){x * 0.06 + 0.5}, minHeight = 5)
# Write the file with the name of each raster
writeOGR(ttops, "output_dir", names(x), driver = "ESRI Shapefile")
})
In the R Raster package, when I load a netCDF file into a raster and attempt to mask or crop the raster with a shapefile, I am getting the error:
"*Error in `colnames<-`(`*tmp*`, value = names(object)) :
attempt to set 'colnames' on an object with less than two dimensions*"
Here is example code
library(ncdf4)
library(raster)
nc <- brick("in.netcdf.nc", varname = "diff", values=TRUE )
crs(nc) <- "+proj=utm +zone=17 +ellps=GRS80 +datum=NAD83 +units=m +no_defs"
everShp <- shapefile("EVERareas")
nc.sub <- mask(nc, everShp)
Plotting shows both nc and everShp are 2D and spatially overlap. Thanks for any suggestions with this error.
if I replace
nc <- brick("in.netcdf.nc", varname = "diff", values=TRUE )
with
nc <- raster("in.netcdf.nc", varname = "diff", values=TRUE )
the mask and crop commands work as expected.
I'm getting an error when merging raster tiles inside a custom function. The following code works every time:
files = list.files("data_folder/")
tiles = list()
for(f in files) tiles = c(tiles, list(raster(readGDAL(paste0(dir,f)), layer=1, values=T)))
ras = do.call(merge, tiles)
But when I try to run this inside a function:
read_and_merge_rasters = function(dir){
files = list.files(dir)
tiles = list()
for(f in files) tiles = c(tiles, list(raster(readGDAL(paste0(dir,f)), layer=1, values=T)))
return(do.call(merge, tiles))
}
.. it reads the files in ok (confirmed by print report) but then fails with the error: Error in as.data.frame.default(x) :
cannot coerce class "structure("RasterLayer", package = "raster")" to a data.frame
I can't think where/why as.data.frame is being called. Any idea why this is happening? Thanks in advance.
You can try this approach (not tested, since you didn't not provide a reproductible example).
read_and_merge_rasters <- function(dir) {
library(raster)
files <- list.files(dir, full.names = TRUE)
tiles <- lapply(files, raster)
do.call(merge, tiles)
}
You also have check that your differents raster in your directory are comparable (extent, origin and resolution).
There are multiple merge functions. do.call will tend to send it to merge for data.frames unless you set the names of the first two arguments:
tiles <- lapply(files, raster)
names(tiles)[1:2] <- c('x', 'y')
x <- do.call(merge, tiles)