How can I automize raster processing in R - r

I have 90 rasters that I need to clip and ressample to match my template raster. Is there a way for me to "automize this"? (i.e., only have to run the process once, so that I can, for instance, leave it running overnight)
I know that the answer is probably a "loop", but I'm not sure how to do it because each original file has a different name and the final product needs to have that same name, but end in "_final.asc"
This is the code I've been using:
library(raster)
#load original world raster (the one to be processed)
mau <- raster("PATH/name_of_the_raster.asc")
#load bounding box raster made by me (to clip the original and therefore make the next steps easier)
rastergrande <- raster("PATH")
#clip original raster by bounding box raster
intermedio <- crop(mau, rastergrande)
#load my template raster (the one that has the extent and cell size I want)
template <- raster("PATH")
#resample raster to template parameters
novoraster <- resample(intermedio, template, "bilinear")
#assign NA's to where template is NA
values(novoraster)[is.na(values(template))] <- NA
#Export to ascii
writeRaster(novoraster, "PATH/name_of_the_raster_final.asc", overwrite=TRUE)

Variations of this question have been asked a number of times. Here is one example
The general approach can be to get a vector with all your filenames, typically with list.files, and use that to make a vector of output filenames by changing the names or the path. Something like this:
input <- list.files(path=".", pattern="asc$", full.names=TRUE)
output <- gsub(".tif", "_out.tif", input)
Or change the folder
output <- gsub("this/path", "that/path", input)
And then use a loop
for (i in 1:length(input)) {
# read input[i]
# write output[i]
}
A working minimal example (using a single file):
library(raster)
input <- system.file("external/test.grd", package="raster")
# write to current working directory, but with asc extensions
output <- gsub(".grd", ".asc", basename(input))
for (i in 1:length(input)) {
r <- raster(input[i])
writeRaster(r, output[i], overwrite=TRUE)
}
Also, not directly related to your question, but this:
values(novoraster)[is.na(values(template))] <- NA
Should be
novoraster <- mask(novoraster, template)

You can use lapply. You would need a vector of file names, maybe with
fn_list <- list.files("PATH", pattern="asc$")
Your workhorse function could look like
one_file <- function(fn, template, rastergrande) {
mau <- raster(fn)
intermedio <- crop(mau, rastergrande)
novoraster <- resample(intermedio, template, "bilinear")
values(novoraster)[is.na(values(template))] <- NA
fn_out <- gsub("asc$", "_final.asc", fn)
writeRaster(novoraster, fn_out)
}
You can then apply one_file on fn_list:
library(raster)
library(usdm)
library(ggplot2)
library(rgdal)
rastergrande <- raster("PATH")
template <- raster("PATH")
lapply(fn_list, one_file, template=templ, rastergrande=rastergrande)
By the way, take a look at the terra package: https://www.youtube.com/watch?v=O8V4QJ13Gjc

Here you can work with a for loop or apply (lapply and mapply)
file_names <- list.files("PATH/", pattern = ".tif")
r_list <- lapply(file_names,raster)
rastergrande <- raster("PATH")
template <- raster("PATH")
intermedio <- list()
novoraster <- list()
Creating a vector with rasters names
rasternames <- c("name1","name2" ...)
For loop to process all rasters
for(i in 1:length(file_names){
intermedio[[i]] <- crop(r_list[[i]], rastergrande)
novoraster[[i]] <- resample(intermedio[[i]], template, "bilinear")
values(novoraster[[i]])[is.na(values(template))] <- NA
writeRaster(novoraster[[i]], paste0("PATH/", rasternames[i], ".asc"), overwrite=TRUE)
}

Related

New rasters from crop of rasters by one polygon in loop R

From 19 climatic rasters, I try to crop only the raster area in the polygon (in France). For this, I try to generate a loop that crops each rasters one by one with the polygon, like this:
# my list of raster
rlist <- list.files(path=mydir, pattern='tif$', full.names=FALSE)
# loop to generate the 19 raster
for(i in rlist) {assign(unlist(strsplit(i, "[.]"))[1], raster(i)) }
# import my polygon (France)
France <- readOGR("./QGIS/France/Shapefile/France.shp")
# crop and mask one raster by polygon France
c1 <- crop(raster1, extent(France))
c2 <- mask(c1, France)
It works, but now I'm trying to find a loop to crop and mask the 19 rasters at once, and create new raster from this crop and mask that I would have named. Any suggestions?
I don't know if it's clear, tell me if you need more info. Thanks.
There is no need for a loop
library(terra)
ff <- list.files(path=mydir, pattern='tif$', full.names=FALSE)
x <- rast(ff)
france <- vect("./QGIS/France/Shapefile/France.shp")
xf <- crop(x, france, mask=TRUE)
If xf has 19 layers and you want to write each to a separate file (as you indicate in the comments), you can do something like
fout <- paste0("bio_", 1:19)
writeRaster(xf, fout, filetype="GTiff")
Also, you should never, ever, use assign. In a case like this, you could make a list instead, so that you can later iterate over its items (with for or lapply. What you were doing could be written as
x <- lapply(ff, rast)
y <- lapply(x, \(i) crop(i, France, mask=TRUE))
z <- rast(y)
But that is still rather clumsy
I recommend you to switch to terra. It's from the same author than raster library but it's faster:
library(terra)
files <- list.files(path=mydir, pattern='tif$', full.names=TRUE)
v <- vect("./QGIS/France/Shapefile/France.shp")
l <- list()
for(i in seq_along(files)){
rtemp <- rast(files[i])
l[[i]] <- crop(rtemp, v, mask=TRUE)
}
s <- rast(l)

Is there a faster way to mask multiple rasters using multiple polygons?

Basically I have 12 multispectral images, and I want to mask them using 2 polygons (small waterbodies). The 2 polygons are in one shapefile, but I can break them up if it would make the process easier. With the help of some nice users on here, I tested this all out using the 12 images on one polygon and it works just fine, but I'll eventually need to do this for multiple polygons so I want to adapt my code.
The loop to crop all rasters using a single polygon:
#The single polygon
mask <- st_read(here::here("data", "mask.shp") %>%
st_as_sf()
#Creates list of input files and their paths
crop_in <- list.files(here::here("data", "s2_rasters"), pattern="tif$", full.names=TRUE)
#Creates list of output files and their directory.
crop_out <- gsub(here::here("data", "s2_rasters"), here::here("data", "s2_cropped"), crop_in)
for (i in seq_along(crop_in)) {
b <- brick(crop_in[i])
crop(b, mask, filename = crop_out[i])
}
Like I said this works just fine, but I want to mask instead of crop. Additionally, I need to mask using multiple polygons.
My working loop to do the same thing but for multiple (2) polygons:
masks_2 <- st_read(here::here("data", "multiple_masks.shp")) %>%
st_as_sf()
for (i in seq_along(crop_in)) {
b <- brick(crop_in[i])
mask(b, masks_2, filename = crop_out[i], overwrite = TRUE)
}
This took around 2 hours (which makes me suspicious) and I think it lost the polygon id somewhere along the way. When I tried plotting the results the plot was empty. My final output should be 24 rasterstacks, 12 for each polygon. I will need to do further image analysis so I will need to keep the names. I hope this makes sense and thank you!
Here is a minimal, self-contained, reproducible example using terra because it is much faster than raster (make sure you are using the current version)
Raster dataset with 12 layers
library(terra)
f <- system.file("ex/elev.tif", package="terra")
r <- rast(f)
r <- rep(r, 12) * 1:12
names(r) <- paste0("band", 1:12)
Two "lakes"
v <- vect(system.file("ex/lux.shp", package="terra"))
v <- v[c(1,12)]
Solution:
x <- mask(r, v)
And always try things for a single case before running the loop.
So if you have 12 files, you can do something like
inf <- list.files("data/s2_rasters", pattern="tif$", full.names=TRUE)
outf <- gsub(".tif$", "_masked.tif", inf)
for (for i in 1:length(inf)) {
r <- rast(inf[i])
m <- mask(r, v, filename=outf[i])
}
It might be a little faster to instead do this (only rasterize the polygons once)
msk <- rast(inf[1])
msk <- rasterize(v, msk)
for (for i in 1:length(inf)) {
r <- rast(inf[i])
m <- mask(r, msk, filename=outf[i])
}
Or make one object/file, if that is practical.
rr <- rast(inf)
mm <- mask(rr, v)

Create a list of all loaded Rasters in a for loop in R

I want to merge 3252 .hgt SRTM raster using the mosaic_rasters function from rgdal, therefore i need a list of all my raster files.
I started to load all my raster files into my Global Environment using following for loop:
list_SRTM3_Africa <- list.files("C://.../01_data/01_SRTM/version2_1_SRTM3/Africa")
for(i in list_SRTM3_Africa) {
assign(unlist(strsplit(i, "[.]"))[1], raster(i))
}
But unfortunately i could not figure out how to create a list from the loaded grids.
And i can not do it like this for all 3252 files:
raster_list_SRTM3_Africa <- c(S01E006, S01E007,..., S35E0253)
Then i create a large empty Raster witch i want to fill with the mosaic_rasters command.
template <- raster(extent_Africa_canvas)
projection(template) <- coord_ref_Africa_canvas
writeRaster(template, file="SRTM3_Africa.tif", format="GTiff")
mosaic_rasters(gdalfile=raster(raster_list_SRTM3_Africa),dst_dataset="SRTM3_Africa.tif",of="GTiff", output_Raster = T)
gdalinfo("SRTM3_Africa.tif")
You can use lapply for tasks like that
library(raster)
list_SRTM3_Africa <- list.files("C://.../01_data/01_SRTM/version2_1_SRTM3/Africa")
rlist <- lapply(list_SRTM3_Africa, raster)
Generally, you can then do something like:
x <- do.call(merge, rlist)
I have done it using the append command.
raster_list <- c() # initialise the list of rasters
for(i in list_SRTM3_Africa) {
raster_list <- append(raster_list, assign(unlist(strsplit(i, "[.]"))[1], raster(i)))
}

How to crop a raster stack using multiple shapefiles in r

I am new to r so I am still figuring out the basics. I am trying to a take a list of multiple shapefiles and use them to crop several rasters stack. Then use the cropped raster stacks and create histograms from them.
I know I have to make lists of shapefiles which I have already figured out how to do. I also figured out how to individually crop rasters and make histograms. Unfortunately, the looping is throwing me for a loop. Does anyone have any suggestions?
Here's what I have so far:
library('rgdal')
library('raster')
library('landsat')
library('ggplot2')
library('rasterVis')
library('spatial.tools')
library('plyr')
library('reshape')
library('reshape2')
library('GGally')
library('lattice')
library('sp')
library('maptools')
library('rgeos')
library('gdata')
#Create folder for shapefiles located and rasters
foldershp <- paste0("F:/2926/Hayes/crp/hnw/")
setwd("F:/2926/Hayes/crp/hnw/")
fileName<-list.files(path= ".", pattern = ".shp$")
folderout<- paste0("F:/Indices/Histograms/")
#list for Rasters
ras.root<- paste0("F:/2926/July/")
ras.list <- c("mtvi", "ndsvi", "ndvi", "ndwi", "satvi", "swir32")
#Loop
for (i in ras.list) {
s <- stack()
setwd(paste(ras.root,i,sep=""))
files <- list.files(path = getwd(), pattern = "*.tif$")
s <- stack(files)
for (fileName in fileNames){
setwd(foldershp)
data <- readOGR(dsn=".",substr(fileName,1,nchar(fileName)-4))
names(data) <- substr(fileName,1,nchar(fileName)-4)
m <- mask(s,data)
setwd(folderout)
png(paste(i,"_",names(data),"_hist_plot",sep=""))
hist(m)
dev.off()
png(paste(i,"_clipped_",names(data),sep=""))
plot(m)
dev.off()
}
}
I'll give this a shot, though i'm sure there are more elegant ways....
set environments, read in the shapefiles as a list of names, create a character vector of where the rasters are kept (in your case, "mtvi","ndvi" etc)
require(raster)
require(gdata)
require(rgdal)
setwd("/home/st/R/shapes")
fileNames <- list.files(path = ".", pattern = "*.shp$")
ras.fols <- c("one","two","three")
setwd("/home/st/R/rasters")
now loop through the raster folders vector, setting the working directory to that folder and make a stack of the rasters inside it.
Then the nested loop will call your list of shapefiles and, one by one, mask the first rasterstack by the shapefiles and save a histogram and plot image of the masked data.
It then returns to the original loop, moves onto the next raster folder and performs the same action.
for (i in ras.fols) {
s <- stack()
setwd(paste("/home/st/R/rasters/",i,sep=""))
files <- list.files(path = getwd(), pattern = "*.tif$")
s <- stack(files)
for (fileName in fileNames){
setwd("/home/st/R/shapes")
data <- readOGR(dsn=".",substr(fileName,1,nchar(fileName)-4))
names(data) <- substr(fileName,1,nchar(fileName)-4)
m <- mask(s,data)
setwd("/home/st/R/write")
png(paste(i,"_",names(data),"_hist_plot",sep=""))
hist(m)
dev.off()
png(paste(i,"_clipped_",names(data),sep=""))
plot(m)
dev.off()
}
}
hope there's something that can help. You can tidy it up, put the shapefile folder and character vector of raster folders to the top, replace some 'setwd9("adfad/asdad/asdasd")' with some tidier stuff etc

Merging multiple rasters in R

I've been trying to find a time-efficient way to merge multiple raster images in R. These are adjacent ASTER scenes from the southern Kilimanjaro region, and my target is to put them together to obtain one large image.
This is what I got so far (object 'ast14dmo' representing a list of RasterLayer objects):
# Loop through single ASTER scenes
for (i in seq(ast14dmo.sd)) {
if (i == 1) {
# Merge current with subsequent scene
ast14dmo.sd.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
} else if (i > 1 && i < length(ast14dmo.sd)) {
tmp.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
ast14dmo.sd.mrg <- merge(ast14dmo.sd.mrg, tmp.mrg, tolerance = 1)
} else {
# Save merged image
writeRaster(ast14dmo.sd.mrg, paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg", sep = ""), format = "GTiff", overwrite = TRUE)
}
}
As you surely guess, the code works. However, merging takes quite long considering that each single raster object is some 70 mb large. I also tried Reduce and do.call, but that failed since I couldn't pass the argument 'tolerance' which circumvents the different origins of the raster files.
Anybody got an idea of how to speed things up?
You can use do.call
ast14dmo.sd$tolerance <- 1
ast14dmo.sd$filename <- paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg.tif", sep = "")
ast14dmo.sd$overwrite <- TRUE
mm <- do.call(merge, ast14dmo.sd)
Here with some data, from the example in raster::merge
r1 <- raster(xmx=-150, ymn=60, ncols=30, nrows=30)
r1[] <- 1:ncell(r1)
r2 <- raster(xmn=-100, xmx=-50, ymx=50, ymn=30)
res(r2) <- c(xres(r1), yres(r1))
r2[] <- 1:ncell(r2)
x <- list(r1, r2)
names(x) <- c("x", "y")
x$filename <- 'test.tif'
x$overwrite <- TRUE
m <- do.call(merge, x)
The 'merge' function from the Raster package is a little slow. For large projects a faster option is to work with gdal commands in R.
library(gdalUtils)
library(rgdal)
Build list of all raster files you want to join (in your current working directory).
all_my_rasts <- c('r1.tif', 'r2.tif', 'r3.tif')
Make a template raster file to build onto. Think of this a big blank canvas to add tiles to.
e <- extent(-131, -124, 49, 53)
template <- raster(e)
projection(template) <- '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs'
writeRaster(template, file="MyBigNastyRasty.tif", format="GTiff")
Merge all raster tiles into one big raster.
mosaic_rasters(gdalfile=all_my_rasts,dst_dataset="MyBigNastyRasty.tif",of="GTiff")
gdalinfo("MyBigNastyRasty.tif")
This should work pretty well for speed (faster than merge in the raster package), but if you have thousands of tiles you might even want to look into building a vrt first.
You can use Reduce like this for example :
Reduce(function(...)merge(...,tolerance=1),ast14dmo.sd)
SAGA GIS mosaicking tool (http://www.saga-gis.org/saga_tool_doc/7.3.0/grid_tools_3.html) gives you maximum flexibility for merging numeric layers, and it runs in parallel by default! You only have to translate all rasters/images to SAGA .sgrd format first, then run the command line saga_cmd.
I have tested the solution using gdalUtils as proposed by Matthew Bayly. It works quite well and fast (I have about 1000 images to merge). However, after checking with document of mosaic_raster function here, I found that it works without making a template raster before mosaic the images. I pasted the example codes from the document below:
outdir <- tempdir()
gdal_setInstallation()
valid_install <- !is.null(getOption("gdalUtils_gdalPath"))
if(require(raster) && require(rgdal) && valid_install)
{
layer1 <- system.file("external/tahoe_lidar_bareearth.tif", package="gdalUtils")
layer2 <- system.file("external/tahoe_lidar_highesthit.tif", package="gdalUtils")
mosaic_rasters(gdalfile=c(layer1,layer2),dst_dataset=file.path(outdir,"test_mosaic.envi"),
separate=TRUE,of="ENVI",verbose=TRUE)
gdalinfo("test_mosaic.envi")
}
I was faced with this same problem and I used
#Read desired files into R
data_name1<-'file_name1.tif'
r1=raster(data_name1)
data_name2<-'file_name2.tif'
r2=raster(data_name2)
#Merge files
new_data <- raster::merge(r1, r2)
Although it did not produce a new merged raster file, it stored in the data environment and produced a merged map when plotted.
I ran into the following problem when trying to mosaic several rasters on top of each other
In vv[is.na(vv)] <- getValues(x[[i]])[is.na(vv)] :
number of items to replace is not a multiple of replacement length
As #Robert Hijmans pointed out, it was likely because of misaligned rasters. To work around this, I had to resample the rasters first
library(raster)
x <- raster("Base_raster.tif")
r1 <- raster("Top1_raster.tif")
r2 <- raster("Top2_raster.tif")
# Resample
x1 <- resample(r1, crop(x, r1))
x2 <- resample(r2, crop(x, r2))
# Merge rasters. Make sure to use the right order
m <- merge(merge(x1, x2), x)
# Write output
writeRaster(m,
filename = file.path("Mosaic_raster.tif"),
format = "GTiff",
overwrite = TRUE)

Resources