R Programming Language (New to this)
I am attempting to loop through a number of tiled rasters that have been output by splitRaster. During the loop I want to carry out some processes on each raster.
But the following code throws an error.
library(ForestTools)
library(raster)
library(sp)
library(rgdal)
library(SpaDES)
rm(list = ls())
tmpdir <- file.path(tempdir(), "splitRaster")
lin <- function(x){x * 0.1 + 0.6}
inCHM <- raster("input raster path and name.tif")
split <- splitRaster(inCHM, 5, 5, c(0.05, 0.05), tmpdir)
files <- list.files(path=tmpdir, pattern="*.grd", full.names=FALSE, recursive=FALSE)
file.names <- dir(tmpdir, pattern ="*.grd")
for(file.names in files ){
name <- file.names
ttops <- vwf(name, winFun = lin, minHeight = 5)
writeOGR(ttops, "output folder", name, driver = "ESRI Shapefile")
}
and this is the error
[1] "Xrastername_tile1.grd"
Error in CRS(x) :
PROJ4 argument-value pairs must begin with +: Xrastername_tile1.grd
More to the problem (24/7/2020),
I have removed the loop for trouble shooting instead just choosing one of the splitRasters outputs that would be used in the loop ie files[[3]]
When I run the following code the error is the same;
library(ForestTools)
library(raster)
library(sp)
library(rgdal)
library(SpaDES)
rm(list = ls())
# set temp directory
tmpdir <- "C:\\R-Test\\Temp_Output"
# get raster
r <- raster("C:\\Lidar\\grid_treeheight_max_1m_nofill.tif")
# define projection
projection(r) <- "+proj=utm +zone=50 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs"
# split raster brick
y <- splitRaster(r, 8, 8, c(0.05, 0.05), tmpdir)
# Get the complete file locations with full.names = T
files <- list.files(path=tmpdir, pattern="*.grd", full.names=FALSE, recursive=FALSE)
tmpfile <- paste(tmpdir, "\\", files[[3]], sep="")
lin <- function(x){x * 0.06 + 0.6}
ttops <- vwf(tmpfile, winFun = lin, minHeight = 5)
This is the error
Error in CRS(x) :
PROJ4 argument-value pairs must begin with +: D:\R-Test\Temp_Output\Xgrid_treeheight_max_1m_nofill_tile11.grd
When I run the following code using one of the splitRaster outputs (files[[3]]) from the above code it runs error free and I am able to plot ttops.
rm(list = ls())
# set temp directory
tmpdir <- "D:\\R-Test\\Temp_Output"
# get raster
r <- raster("D:\\R-Test\\Temp_Output\\Xgrid_treeheight_max_1m_nofill_tile11.grd")
lin <- function(x){x * 0.06 + 0.6}
ttops <- vwf(r, winFun = lin, minHeight = 5)
Why is the PROJ4 error occurring?
This seems to be the error that is causing the loop to fail?
I think the problem is that you are trying to feed the vwf function with the file name instead of a raster object. I would also recommend using lapply instead of for for the loop. Here is a code that should work
library(raster)
library(ForestTools)
library(rgdal)
# Get the complete file locations with full.names = T
files <- list.files(path=tmpdir, pattern="*.grd", full.names=T, recursive=FALSE)
# Loop over each item of the list, i.e., each raster
lapply(files, function(x){
# Load the image as raster
image <- raster(x)
# Calculate vwf (I added a dummy function for winFun)
ttops <- vwf(image, winFun = function(x){x * 0.06 + 0.5}, minHeight = 5)
# Write the file with the name of each raster
writeOGR(ttops, "output_dir", names(x), driver = "ESRI Shapefile")
})
Related
I have a similar question to u/Ananas here: Sentinel3 OLCI (chl) Average of netcdf files on Python
I am running into similar problems, in so much that I cannot seem to extract the necessary information from the .nc-files and then merge them to create a time-series. In my case,I am trying to do this in R. My current code, which I have followed and customised from here: https://www.youtube.com/watch?v=jWRszWCVWLc&t=1504s , returns an error:
Error in `[<-.data.frame`(`*tmp*`, variable, value = c(0, 0, 0, 0, 0, :
replacement has 1927 rows, data has 2202561
Maybe I am going at it the wrong way from the start and R-s capabilities wiht .nc files are not suited for this? Any suggestions are welcomed.
Here is my code
extract_variable_from_netcdf<- function(nc,variable){
tryCatch(
{
result<-var.get.nc(nc,variable)
return(result)
},
error=function(cond){
message(paste(variable,"attribute not found"))
message("Here is the original error message")
message(cond)
}
)
}
extract_global_attribute_from_netcdf<- function(nc,global_attribute){
tryCatch(
{
result<-att.get.nc(nc,"NC_GLOBAL",global_attribute)
return(result)
},
error=function(cond){
message(paste(global_attribute,"attribute not found"))
message("Here is the original error message")
message(cond)
}
)
}
folder<- "path to folder"
files<- list.files(folder, pattern= ".nc", full.names = TRUE)
variables<- c("conc_chl", "iop_bpart","lat", "lon") #variables I need to extract
global_attrs<- c("start_date", "stop_date")
headers<-c(global_attrs,variables)
df<-data.frame(matrix(ncol=length(headers), nrow=0))
colnames(df)<- headers
for(file in files) {
nc<- open.nc(file)
chl<- var.get.nc(nc, "conc_chl")
num_chl<- length(chl)
newdf<- data.frame(matrix(ncol=length(headers), nrow=num_chl))
colnames(newdf)<- headers
for (global_attribute in global_attrs) {
newdf[global_attribute]<-extract_global_attribute_from_netcdf(nc,global_attribute)
}
for (variable in variables) {
newdf[variable]<-extract_variable_from_netcdf(nc,variable)
}
df<-merge(df,newdf,all=TRUE)
}
The way I have used ".nc" files with satellite data, in R. Have been reading it in with the "raster" library as a raster file.
library(raster)
r <- raster("yuor_file.nc")
plot(r) # quick plot to see if everything is as it should be
The way I read in my timeseries was with a loop, and in addition I used a function found from this site somewhere, to covert the raster into a sensible r-data frame
stack overflow function, to convert the loaded raster to data frame
gplot_data <- function(x, maxpixels = 50000) {
x <- raster::sampleRegular(x, maxpixels, asRaster = TRUE)
coords <- raster::xyFromCell(x, seq_len(raster::ncell(x)))
## Extract values
dat <- utils::stack(as.data.frame(raster::getValues(x)))
names(dat) <- c('value', 'variable')
dat <- dplyr::as.tbl(data.frame(coords, dat))
if (!is.null(levels(x))) {
dat <- dplyr::left_join(dat, levels(x)[[1]],
by = c("value" = "ID"))
}
dat
}
Read in one file at a time, convert with function and return data.frame
files<- list.files(folder, pattern= ".nc", full.names = TRUE)
fun <- function(i) {
#read in one file at a time
r <- raster(files[i])
#convert to normal data frame
temp <- gplot_data(r)
temp #output
}
dat <- plyr::rbind.fill(lapply(1:length(files), fun)) #bind each iteration
Here a plot using ggplot2 and ggforce.
ggplot() +
geom_tile(data = dat,
aes(x = x, y = y, fill = value))
Alternatively if you do not know the context of you file, the following, from the "ncdf4" package, will help you inspect it. https://towardsdatascience.com/how-to-crack-open-netcdf-files-in-r-and-extract-data-as-time-series-24107b70dcd
library(ncdf4)
our_nc_data <- nc_open("/your_file.nc")
print(our_nc_data)
# look for the variable names and assign them to vectors that can be bound together in dataframes
lat <- ncvar_get(our_nc_data, "lat") #names of latitude column
lon <- ncvar_get(our_nc_data, "lon") #name of longitude column
time <- ncvar_get(our_nc_data, "time") #the time was called time
tunits <- ncatt_get(our_nc_data, "time", "units")# check units
lswt_array <- ncvar_get(our_nc_data, "analysed_sst") #select the relevant variable, this is temperature named "analysed_sst"
Is it possible to write a matrix of string variables to a netCDF file in R? When I run the following the R session aborts.
library(ncdf4)
# define a small set of netCDF dimensions
lon3 <- seq(0, 2.0, 0.5)
nlon3 <- length(lon3)
lat3 <- seq(0, 0.5, 0.5)
nlat3 <- length(lat3)
# Make the matrix of strings
id_matrix <- matrix(c("top_extreme_left", "top_left", "top_centre", "top_right", "top_extreme_right", "bottom_extreme_left", "bottom_left", "bottom_centre", "bottom_right", "bottom_extreme_right"), nrow=2, byrow = TRUE)
# create and write NetCDF file
# path and file name, set dname
ncpath <- "C:/"
ncname <- "tiny_test_netcdf"
ncfname <- paste(ncpath, ncname, ".nc", sep="")
# create and write the netCDF file -- ncdf4 version
# define dimensions
londim <- ncdim_def("lon","degrees_east",as.double(lon3))
latdim <- ncdim_def("lat","degrees_north",as.double(lat3))
# dimension for each char in the strings (max string length of 25 characters)
dimchar <- ncdim_def("nchar", "", 1:25, create_dimvar = FALSE)
# define variables
dlname <- "test string matrix data"
id_def <- ncvar_def("op_id", "", list(londim, latdim, dimchar), longname = dlname, prec="char")
# create netCDF file and put arrays
ncout <- nc_create(ncfname, list(id_def), force_v4=TRUE)
# Put the matrix of id strings into the netCDF file.
ncvar_put(ncout, id_def, id_matrix, start=c(1,1,1), count = c(5, 2, 25), verbose = TRUE)
# Close the file to write the data
nc_close(ncout)
I'm attempting to extend the example given in the help for ncvar_put which shows the process for a vector of strings. Any idea how/if it's possible to do the same with a matrix?
Eventually found a way by formatting the strings to write into a vector first rather than a matrix. You just have to be mindful that the writing will place the strings into the correct X,Y locations (X grows fastest).
#----------------------------------------------------------------------
# Illustrate creating a character type variable
#----------------------------------------------------------------------
cnames <- c("bottom_extreme_left", "bottom_left", "bottom_centre", "bottom_right", "bottom_extreme_right", "top_extreme_left", "top_left", "top_centre", "top_right", "top_extreme_right")
nstrings <- length(cnames)
max_length <- max(nchar(cnames))
dimX <- ncdim_def( "X", "lon", c(1,2,3,4,5))
dimY <- ncdim_def( "Y", "lat", c(1,2))
#--------------------------------------------------------------
# Make dimensions. Setting "dimnchar" to have a length equa to max string length in data
dimnchar <- ncdim_def("nchar", "", 1:max_length, create_dimvar=FALSE )
#------------------------------------------------------------------------
# NOTE in the following call that units is set to the empty string (""),
# which suppresses creation of a units attribute, and the missing value
# is entirely omitted, which suppresses creation of the missing value att
#------------------------------------------------------------------------
varlocations <- ncvar_def("locations", "", list(dimnchar, dimX, dimY),
prec="char" )
# path and file name
ncpath <- "C:/"
ncname <- "tiny_test_netcdf"
ncfname <- paste(ncpath, ncname, ".nc", sep="")
ncid <- nc_create(ncfname, list(varlocations) )
ncvar_put( ncid, "locations", cnames, verbose=TRUE )
nc_close( ncid )
I have some code in R which does the following:
Uses lapply to bring in files in a set folder e.g. 1997 data
Makes file list into a brick - they are NetCDF files, so I've used brick function
Stacks the bricks into one raster stack of months for each year.
Calculate the mean from the stack
Crops the new mean raster to the Area of Interest (AOI).
I've got a working code, see below, but it is clunky and I feel could be better in one loop to then run through each year's folder (I have data from 1997 to 2018). Could anyone aid in streamlining this into a simple looped code I could run by changing the filepath? I've used loops a bit before but not from scratch.
# Packages:
library(raster)
library(parallel) # Check cores in PC
library(lubridate) # needed for lapply
library(dplyr) # ""
library(sf) # For clipping data
library(rgdal)
# ChlA
# Set file paths for input and outputs:
usingfp <- "/filepath/GIS/ChlA/1997/"
the_dir_ex <- "Data/CHL/1997"
# List all NETCDF files in folder:
CHL_1997 <- list.files(path = usingfp, pattern = "\\.nc$", full.names = TRUE,
recursive = FALSE)
# Make file list into brick
CHL_1997_brick <- lapply(CHL_1997,
FUN = brick,
the_dir = the_dir_ex)
# Stack bricks
s <- stack(CHL_1997_brick)
# Calculate mean from stack
mean <- calc(s, fun = mean, na.rm = T)
plot(mean)
# Load vector boundary to "crop" to
AOI <- readOGR("/filepath/AOI/AOI.shp")
plot(AOI,
main = "Shapefile imported into R - crop extent",
axes = TRUE,
border = "blue",
add = T)
# crop the raster using the vector extent
CHL_1997_mean <- crop(mean, AOI)
plot(CHL_1997_mean, main = "Cropped mean CHL - 1997")
# add shapefile on top of the existing raster
plot(AOI, add = TRUE)
Thanks very much.
Something like this should work
library(raster)
AOI <- shapefile("/filepath/AOI/AOI.shp")
path <- "/filepath/GIS/ChlA/"
years <- 1997:2018
for (yr in years) {
fp <- file.path(path, yr)
fout <- file.path(fp, paste0(year, ".tif"))
print(fout); flush.console()
# if (file.exists(fout)) next
files <- list.files(path=fp, pattern="\\.nc$", full.names=TRUE)
b <- lapply(files, brick)
s <- stack(b)
s <- mean(s)
s <- crop(s, AOI, filename=fout) #, overwrite=TRUE)
}
Notes:
mean(s) is more efficient than calc(s, mean)
If the AOI is relatively small, it can be more efficient to first
use crop, then mean (and then use writeRaster)
You can also use terra like this:
library(terra)
AOI <- vect("/filepath/AOI/AOI.shp")
path <- "/filepath/GIS/ChlA/"
years <- 1997:2018
for (yr in years) {
fp <- file.path(path, year)
fout <- file.path(fp, paste0(year, ".tif"))
print(fout); flush.console()
# if (file.exists(fout)) next
files <- list.files(path=fp, pattern="\\.nc$", full.names=TRUE)
r <- rast(files)
s <- mean(r)
s <- crop(s, AOI, filename=fout) #, overwrite=TRUE)
}
I'm reading multiple .tif files in R and am converting raster value less than 6000 as NA, and then plotting the raster and saving it in folder as .png. Everything works fine except that output png raster plots are empty. Here is my code which I'm using -
library(raster)
#get the path
path <- "C:/lab/fire/photo_2/gif_images/gif_images_2"
#list all files
files <- list.files(path, pattern = "tif$", full.names = TRUE)
files
for (i in 1:length(files)){
# load one raster
r <- raster(files[i])
values(r)[values(r) < 6000] = NA
png(paste("plot_", i, ".png", sep = ""), width=600, height=500, res=120)
plot(r)
dev.off()
}
I can do this on single rasters quite easily but need to run the loop to account for 300 .tifs.
You have to print your plot to make it visible in png:
An example with fake data:
library(raster)
r <- s <- t <- u <- v <- z <- raster()
r[] <- s[] <- t[] <- u[] <- v[] <- z[] <- 1:ncell(r)
rast.list <- list(r,s,t,u,v,z)
for (i in 1:length(rast.list)){
r <- rast.list[[i]]
r <- clamp(r, -Inf, 6000)
png(paste("plot_", i, ".png", sep = ""), width=600, height=500, res=120)
print(plot(r))
dev.off()
}
-
I am beginner in R and I have made a script where I convert a .grd file into .tif file in a for loop (I have quite a lot of .grd files that should be converted). The script is as you can see below.
# Set your work directory
setwd("xxx")
library(rgdal)
library(sp)
library(raster)
# Data: .grd files in order to reproduce the code below
g1 <- raster(ncol=10, nrow=10)
vals <- 1:100
g1 <- setValues(g1, vals)
writeRaster(g1, filename="TEST_G1.grd", overwrite=TRUE)
g2 <- raster(ncol=50, nrow=50)
vals <- 1
g2 <- setValues(g2, vals)
writeRaster(g2, filename="TEST_G2.grd", overwrite=TRUE)
# Convert .grd to geotif in a for loop
rlist <- list.files(pattern=".grd$")
for (i in rlist)
{
#Read raster
x <- raster(rlist[i])
#make new file name
filename <- rlist[i]
n <- unlist(strsplit(filename, split='.', fixed=TRUE))[1]
name <- paste(n, ".tif", sep="")
#write the raster as GTiff
writeRaster(x, filename=name, format="GTiff", overwrite=TRUE)
}
I have run all sentences in the script separately and managed to get geotif file. However, when I run the lobe I get the following error:
Error in .local(.Object, ...) :
Error in .rasterObjectFromFile(x, band = band, objecttype = "RasterLayer", :
Cannot create a RasterLayer object from this file. (file does not exist)
The file does exist, and when I just run the last line separately the tif file is created... so I don't understand what is wrong with the loop.
If I e.g. write "for (i in 1:2)" instead of "for (i in rlist)" it works. but I would prefer not to count the number of files in every directory, which is the reason I was trying to use "for (i in rlist)"
Thanks a lot for you help!