Basically I have 12 multispectral images, and I want to mask them using 2 polygons (small waterbodies). The 2 polygons are in one shapefile, but I can break them up if it would make the process easier. With the help of some nice users on here, I tested this all out using the 12 images on one polygon and it works just fine, but I'll eventually need to do this for multiple polygons so I want to adapt my code.
The loop to crop all rasters using a single polygon:
#The single polygon
mask <- st_read(here::here("data", "mask.shp") %>%
st_as_sf()
#Creates list of input files and their paths
crop_in <- list.files(here::here("data", "s2_rasters"), pattern="tif$", full.names=TRUE)
#Creates list of output files and their directory.
crop_out <- gsub(here::here("data", "s2_rasters"), here::here("data", "s2_cropped"), crop_in)
for (i in seq_along(crop_in)) {
b <- brick(crop_in[i])
crop(b, mask, filename = crop_out[i])
}
Like I said this works just fine, but I want to mask instead of crop. Additionally, I need to mask using multiple polygons.
My working loop to do the same thing but for multiple (2) polygons:
masks_2 <- st_read(here::here("data", "multiple_masks.shp")) %>%
st_as_sf()
for (i in seq_along(crop_in)) {
b <- brick(crop_in[i])
mask(b, masks_2, filename = crop_out[i], overwrite = TRUE)
}
This took around 2 hours (which makes me suspicious) and I think it lost the polygon id somewhere along the way. When I tried plotting the results the plot was empty. My final output should be 24 rasterstacks, 12 for each polygon. I will need to do further image analysis so I will need to keep the names. I hope this makes sense and thank you!
Here is a minimal, self-contained, reproducible example using terra because it is much faster than raster (make sure you are using the current version)
Raster dataset with 12 layers
library(terra)
f <- system.file("ex/elev.tif", package="terra")
r <- rast(f)
r <- rep(r, 12) * 1:12
names(r) <- paste0("band", 1:12)
Two "lakes"
v <- vect(system.file("ex/lux.shp", package="terra"))
v <- v[c(1,12)]
Solution:
x <- mask(r, v)
And always try things for a single case before running the loop.
So if you have 12 files, you can do something like
inf <- list.files("data/s2_rasters", pattern="tif$", full.names=TRUE)
outf <- gsub(".tif$", "_masked.tif", inf)
for (for i in 1:length(inf)) {
r <- rast(inf[i])
m <- mask(r, v, filename=outf[i])
}
It might be a little faster to instead do this (only rasterize the polygons once)
msk <- rast(inf[1])
msk <- rasterize(v, msk)
for (for i in 1:length(inf)) {
r <- rast(inf[i])
m <- mask(r, msk, filename=outf[i])
}
Or make one object/file, if that is practical.
rr <- rast(inf)
mm <- mask(rr, v)
Related
From 19 climatic rasters, I try to crop only the raster area in the polygon (in France). For this, I try to generate a loop that crops each rasters one by one with the polygon, like this:
# my list of raster
rlist <- list.files(path=mydir, pattern='tif$', full.names=FALSE)
# loop to generate the 19 raster
for(i in rlist) {assign(unlist(strsplit(i, "[.]"))[1], raster(i)) }
# import my polygon (France)
France <- readOGR("./QGIS/France/Shapefile/France.shp")
# crop and mask one raster by polygon France
c1 <- crop(raster1, extent(France))
c2 <- mask(c1, France)
It works, but now I'm trying to find a loop to crop and mask the 19 rasters at once, and create new raster from this crop and mask that I would have named. Any suggestions?
I don't know if it's clear, tell me if you need more info. Thanks.
There is no need for a loop
library(terra)
ff <- list.files(path=mydir, pattern='tif$', full.names=FALSE)
x <- rast(ff)
france <- vect("./QGIS/France/Shapefile/France.shp")
xf <- crop(x, france, mask=TRUE)
If xf has 19 layers and you want to write each to a separate file (as you indicate in the comments), you can do something like
fout <- paste0("bio_", 1:19)
writeRaster(xf, fout, filetype="GTiff")
Also, you should never, ever, use assign. In a case like this, you could make a list instead, so that you can later iterate over its items (with for or lapply. What you were doing could be written as
x <- lapply(ff, rast)
y <- lapply(x, \(i) crop(i, France, mask=TRUE))
z <- rast(y)
But that is still rather clumsy
I recommend you to switch to terra. It's from the same author than raster library but it's faster:
library(terra)
files <- list.files(path=mydir, pattern='tif$', full.names=TRUE)
v <- vect("./QGIS/France/Shapefile/France.shp")
l <- list()
for(i in seq_along(files)){
rtemp <- rast(files[i])
l[[i]] <- crop(rtemp, v, mask=TRUE)
}
s <- rast(l)
I have 90 rasters that I need to clip and ressample to match my template raster. Is there a way for me to "automize this"? (i.e., only have to run the process once, so that I can, for instance, leave it running overnight)
I know that the answer is probably a "loop", but I'm not sure how to do it because each original file has a different name and the final product needs to have that same name, but end in "_final.asc"
This is the code I've been using:
library(raster)
#load original world raster (the one to be processed)
mau <- raster("PATH/name_of_the_raster.asc")
#load bounding box raster made by me (to clip the original and therefore make the next steps easier)
rastergrande <- raster("PATH")
#clip original raster by bounding box raster
intermedio <- crop(mau, rastergrande)
#load my template raster (the one that has the extent and cell size I want)
template <- raster("PATH")
#resample raster to template parameters
novoraster <- resample(intermedio, template, "bilinear")
#assign NA's to where template is NA
values(novoraster)[is.na(values(template))] <- NA
#Export to ascii
writeRaster(novoraster, "PATH/name_of_the_raster_final.asc", overwrite=TRUE)
Variations of this question have been asked a number of times. Here is one example
The general approach can be to get a vector with all your filenames, typically with list.files, and use that to make a vector of output filenames by changing the names or the path. Something like this:
input <- list.files(path=".", pattern="asc$", full.names=TRUE)
output <- gsub(".tif", "_out.tif", input)
Or change the folder
output <- gsub("this/path", "that/path", input)
And then use a loop
for (i in 1:length(input)) {
# read input[i]
# write output[i]
}
A working minimal example (using a single file):
library(raster)
input <- system.file("external/test.grd", package="raster")
# write to current working directory, but with asc extensions
output <- gsub(".grd", ".asc", basename(input))
for (i in 1:length(input)) {
r <- raster(input[i])
writeRaster(r, output[i], overwrite=TRUE)
}
Also, not directly related to your question, but this:
values(novoraster)[is.na(values(template))] <- NA
Should be
novoraster <- mask(novoraster, template)
You can use lapply. You would need a vector of file names, maybe with
fn_list <- list.files("PATH", pattern="asc$")
Your workhorse function could look like
one_file <- function(fn, template, rastergrande) {
mau <- raster(fn)
intermedio <- crop(mau, rastergrande)
novoraster <- resample(intermedio, template, "bilinear")
values(novoraster)[is.na(values(template))] <- NA
fn_out <- gsub("asc$", "_final.asc", fn)
writeRaster(novoraster, fn_out)
}
You can then apply one_file on fn_list:
library(raster)
library(usdm)
library(ggplot2)
library(rgdal)
rastergrande <- raster("PATH")
template <- raster("PATH")
lapply(fn_list, one_file, template=templ, rastergrande=rastergrande)
By the way, take a look at the terra package: https://www.youtube.com/watch?v=O8V4QJ13Gjc
Here you can work with a for loop or apply (lapply and mapply)
file_names <- list.files("PATH/", pattern = ".tif")
r_list <- lapply(file_names,raster)
rastergrande <- raster("PATH")
template <- raster("PATH")
intermedio <- list()
novoraster <- list()
Creating a vector with rasters names
rasternames <- c("name1","name2" ...)
For loop to process all rasters
for(i in 1:length(file_names){
intermedio[[i]] <- crop(r_list[[i]], rastergrande)
novoraster[[i]] <- resample(intermedio[[i]], template, "bilinear")
values(novoraster[[i]])[is.na(values(template))] <- NA
writeRaster(novoraster[[i]], paste0("PATH/", rasternames[i], ".asc"), overwrite=TRUE)
}
Given a netcdf file, I am trying to extract all pixels to form a data.frame for later export to .csv
a=brick(mew.nc)
#get coordinates
coord<-xyFromCell(a,1:ncell(a))
I can extract data for all pixels using extract(a,1:ncell(a)). However, I run into memory issues.
Upon reading through various help pages, I found that one can speed up things with:
beginCluster(n=30)
b=extract(a, coord)
endCluster()
But I still run out of memory. Our supercomputer has more than 1000 nodes, each node has 32 cores.
My actual rasterbrick has 400,000 layers
I am not sure how to parrallize this task without running into memory issues.
Thank you for all your suggestions.
Sample data of ~8MB can be found here
You can do something along these lines to avoid memory problems
library(raster)
b <- brick(system.file("external/rlogo.grd", package="raster"))
outfile <- 'out.csv'
if (file.exists(outfile)) file.remove(outfile)
tr <- blockSize(b)
b <- readStart(b)
for (i in 1:tr$n) {
v <- getValues(b, row=tr$row[i], nrows=tr$nrows[i])
write.table(v, outfile, sep = ",", row.names = FALSE, append = TRUE, col.names=!file.exists(outfile))
}
b <- readStop(b)
To parallelize, you could do this by layer, or groups of layers; and probably all values in one step for each subset of layers. Here for one layer at a time:
f <- function(d) {
filename <- extension(paste(names(d), collapse='-'), '.csv')
x <- values(d)
x <- matrix(x) # these two lines only needed when using
colnames(x) <- names(d) # a single layer
write.csv(x, filename, row.names=FALSE)
}
# parallelize this:
for (i in 1:nlayers(b)) {
f(b[[i]])
}
or
x <- sapply(1:nlayers(b), function(i) f(b[[i]]))
You should not be using extract. The question I have is what you would want such a large csv file for.
I have a question about stack() Raster Layers.
Usually I stack() Raster Layers like that:
stack(RasterLayer1,RasterLayer2,RasterLayer3) # e.g. for 3 Layers
My Question is, how can I stack() Raster Layers without typing in every Raster Layer?
For example: n is the amount of Raster Layers (e.g. 12), all named band.
I created n-Raster Layers and now I want to stack all without typing n-times the Name of the Raster Layers. So instead of typing:
stack(band1,band2,band3,band4,band5,band6,band7,band8,band9,band10,band11,band12)
I want to short that by stack(band[n]), but that doesn't work.
And if I create a list of all bands, I can't stack that list, because they don't appear in my Working Directory because I just created them.
Can anyone help me, please?
If your data is in a directory, you can use a search pattern (for example: *.tif, *.grd,...) and store it in a variable.
bands <- list.files(path=".",pattern="*.tif",full.names=TRUE,recursive=TRUE)
now assume that your data is called:
band_01.tif
band_02.tif
band_03.tif
band_04.tif
band_05.tif
band_06.tif
band_07.tif
then you can stack for example:
data_stack <- stack(bands) #stack all data
data_stack <- stack(bands[1:3]) #stack 1,2 and 3 data
data_stack <- stack(bands[c(1,3,5,7)])
I would recommend not to save them under separate variables like band1,band2,... but instead store them in a list. Here an example:
#Create empty rasters
ras1<- raster()
ras2<- raster()
#Initialise and append to list
list_ras <- list()
list_ras[[1]] <- ras1
list_ras[[2]] <- ras2
#Stack single bands
ras_stack <- stack(list_ras[[1]], list_ras[[2]])
#Stack all bands
ras_stack <- stack(list_ras)
Here is an other approach using mget:
# Generate some data
library(raster)
r <- raster()
r[] <- runif(ncell(r))
for (i in 1:10) assign(paste0("r", i), r)
# create a stack
stack(mget(ls(pattern = "^r.+")))
I've been trying to find a time-efficient way to merge multiple raster images in R. These are adjacent ASTER scenes from the southern Kilimanjaro region, and my target is to put them together to obtain one large image.
This is what I got so far (object 'ast14dmo' representing a list of RasterLayer objects):
# Loop through single ASTER scenes
for (i in seq(ast14dmo.sd)) {
if (i == 1) {
# Merge current with subsequent scene
ast14dmo.sd.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
} else if (i > 1 && i < length(ast14dmo.sd)) {
tmp.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
ast14dmo.sd.mrg <- merge(ast14dmo.sd.mrg, tmp.mrg, tolerance = 1)
} else {
# Save merged image
writeRaster(ast14dmo.sd.mrg, paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg", sep = ""), format = "GTiff", overwrite = TRUE)
}
}
As you surely guess, the code works. However, merging takes quite long considering that each single raster object is some 70 mb large. I also tried Reduce and do.call, but that failed since I couldn't pass the argument 'tolerance' which circumvents the different origins of the raster files.
Anybody got an idea of how to speed things up?
You can use do.call
ast14dmo.sd$tolerance <- 1
ast14dmo.sd$filename <- paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg.tif", sep = "")
ast14dmo.sd$overwrite <- TRUE
mm <- do.call(merge, ast14dmo.sd)
Here with some data, from the example in raster::merge
r1 <- raster(xmx=-150, ymn=60, ncols=30, nrows=30)
r1[] <- 1:ncell(r1)
r2 <- raster(xmn=-100, xmx=-50, ymx=50, ymn=30)
res(r2) <- c(xres(r1), yres(r1))
r2[] <- 1:ncell(r2)
x <- list(r1, r2)
names(x) <- c("x", "y")
x$filename <- 'test.tif'
x$overwrite <- TRUE
m <- do.call(merge, x)
The 'merge' function from the Raster package is a little slow. For large projects a faster option is to work with gdal commands in R.
library(gdalUtils)
library(rgdal)
Build list of all raster files you want to join (in your current working directory).
all_my_rasts <- c('r1.tif', 'r2.tif', 'r3.tif')
Make a template raster file to build onto. Think of this a big blank canvas to add tiles to.
e <- extent(-131, -124, 49, 53)
template <- raster(e)
projection(template) <- '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs'
writeRaster(template, file="MyBigNastyRasty.tif", format="GTiff")
Merge all raster tiles into one big raster.
mosaic_rasters(gdalfile=all_my_rasts,dst_dataset="MyBigNastyRasty.tif",of="GTiff")
gdalinfo("MyBigNastyRasty.tif")
This should work pretty well for speed (faster than merge in the raster package), but if you have thousands of tiles you might even want to look into building a vrt first.
You can use Reduce like this for example :
Reduce(function(...)merge(...,tolerance=1),ast14dmo.sd)
SAGA GIS mosaicking tool (http://www.saga-gis.org/saga_tool_doc/7.3.0/grid_tools_3.html) gives you maximum flexibility for merging numeric layers, and it runs in parallel by default! You only have to translate all rasters/images to SAGA .sgrd format first, then run the command line saga_cmd.
I have tested the solution using gdalUtils as proposed by Matthew Bayly. It works quite well and fast (I have about 1000 images to merge). However, after checking with document of mosaic_raster function here, I found that it works without making a template raster before mosaic the images. I pasted the example codes from the document below:
outdir <- tempdir()
gdal_setInstallation()
valid_install <- !is.null(getOption("gdalUtils_gdalPath"))
if(require(raster) && require(rgdal) && valid_install)
{
layer1 <- system.file("external/tahoe_lidar_bareearth.tif", package="gdalUtils")
layer2 <- system.file("external/tahoe_lidar_highesthit.tif", package="gdalUtils")
mosaic_rasters(gdalfile=c(layer1,layer2),dst_dataset=file.path(outdir,"test_mosaic.envi"),
separate=TRUE,of="ENVI",verbose=TRUE)
gdalinfo("test_mosaic.envi")
}
I was faced with this same problem and I used
#Read desired files into R
data_name1<-'file_name1.tif'
r1=raster(data_name1)
data_name2<-'file_name2.tif'
r2=raster(data_name2)
#Merge files
new_data <- raster::merge(r1, r2)
Although it did not produce a new merged raster file, it stored in the data environment and produced a merged map when plotted.
I ran into the following problem when trying to mosaic several rasters on top of each other
In vv[is.na(vv)] <- getValues(x[[i]])[is.na(vv)] :
number of items to replace is not a multiple of replacement length
As #Robert Hijmans pointed out, it was likely because of misaligned rasters. To work around this, I had to resample the rasters first
library(raster)
x <- raster("Base_raster.tif")
r1 <- raster("Top1_raster.tif")
r2 <- raster("Top2_raster.tif")
# Resample
x1 <- resample(r1, crop(x, r1))
x2 <- resample(r2, crop(x, r2))
# Merge rasters. Make sure to use the right order
m <- merge(merge(x1, x2), x)
# Write output
writeRaster(m,
filename = file.path("Mosaic_raster.tif"),
format = "GTiff",
overwrite = TRUE)