I am new to r so I am still figuring out the basics. I am trying to a take a list of multiple shapefiles and use them to crop several rasters stack. Then use the cropped raster stacks and create histograms from them.
I know I have to make lists of shapefiles which I have already figured out how to do. I also figured out how to individually crop rasters and make histograms. Unfortunately, the looping is throwing me for a loop. Does anyone have any suggestions?
Here's what I have so far:
library('rgdal')
library('raster')
library('landsat')
library('ggplot2')
library('rasterVis')
library('spatial.tools')
library('plyr')
library('reshape')
library('reshape2')
library('GGally')
library('lattice')
library('sp')
library('maptools')
library('rgeos')
library('gdata')
#Create folder for shapefiles located and rasters
foldershp <- paste0("F:/2926/Hayes/crp/hnw/")
setwd("F:/2926/Hayes/crp/hnw/")
fileName<-list.files(path= ".", pattern = ".shp$")
folderout<- paste0("F:/Indices/Histograms/")
#list for Rasters
ras.root<- paste0("F:/2926/July/")
ras.list <- c("mtvi", "ndsvi", "ndvi", "ndwi", "satvi", "swir32")
#Loop
for (i in ras.list) {
s <- stack()
setwd(paste(ras.root,i,sep=""))
files <- list.files(path = getwd(), pattern = "*.tif$")
s <- stack(files)
for (fileName in fileNames){
setwd(foldershp)
data <- readOGR(dsn=".",substr(fileName,1,nchar(fileName)-4))
names(data) <- substr(fileName,1,nchar(fileName)-4)
m <- mask(s,data)
setwd(folderout)
png(paste(i,"_",names(data),"_hist_plot",sep=""))
hist(m)
dev.off()
png(paste(i,"_clipped_",names(data),sep=""))
plot(m)
dev.off()
}
}
I'll give this a shot, though i'm sure there are more elegant ways....
set environments, read in the shapefiles as a list of names, create a character vector of where the rasters are kept (in your case, "mtvi","ndvi" etc)
require(raster)
require(gdata)
require(rgdal)
setwd("/home/st/R/shapes")
fileNames <- list.files(path = ".", pattern = "*.shp$")
ras.fols <- c("one","two","three")
setwd("/home/st/R/rasters")
now loop through the raster folders vector, setting the working directory to that folder and make a stack of the rasters inside it.
Then the nested loop will call your list of shapefiles and, one by one, mask the first rasterstack by the shapefiles and save a histogram and plot image of the masked data.
It then returns to the original loop, moves onto the next raster folder and performs the same action.
for (i in ras.fols) {
s <- stack()
setwd(paste("/home/st/R/rasters/",i,sep=""))
files <- list.files(path = getwd(), pattern = "*.tif$")
s <- stack(files)
for (fileName in fileNames){
setwd("/home/st/R/shapes")
data <- readOGR(dsn=".",substr(fileName,1,nchar(fileName)-4))
names(data) <- substr(fileName,1,nchar(fileName)-4)
m <- mask(s,data)
setwd("/home/st/R/write")
png(paste(i,"_",names(data),"_hist_plot",sep=""))
hist(m)
dev.off()
png(paste(i,"_clipped_",names(data),sep=""))
plot(m)
dev.off()
}
}
hope there's something that can help. You can tidy it up, put the shapefile folder and character vector of raster folders to the top, replace some 'setwd9("adfad/asdad/asdasd")' with some tidier stuff etc
Related
I have 90 rasters that I need to clip and ressample to match my template raster. Is there a way for me to "automize this"? (i.e., only have to run the process once, so that I can, for instance, leave it running overnight)
I know that the answer is probably a "loop", but I'm not sure how to do it because each original file has a different name and the final product needs to have that same name, but end in "_final.asc"
This is the code I've been using:
library(raster)
#load original world raster (the one to be processed)
mau <- raster("PATH/name_of_the_raster.asc")
#load bounding box raster made by me (to clip the original and therefore make the next steps easier)
rastergrande <- raster("PATH")
#clip original raster by bounding box raster
intermedio <- crop(mau, rastergrande)
#load my template raster (the one that has the extent and cell size I want)
template <- raster("PATH")
#resample raster to template parameters
novoraster <- resample(intermedio, template, "bilinear")
#assign NA's to where template is NA
values(novoraster)[is.na(values(template))] <- NA
#Export to ascii
writeRaster(novoraster, "PATH/name_of_the_raster_final.asc", overwrite=TRUE)
Variations of this question have been asked a number of times. Here is one example
The general approach can be to get a vector with all your filenames, typically with list.files, and use that to make a vector of output filenames by changing the names or the path. Something like this:
input <- list.files(path=".", pattern="asc$", full.names=TRUE)
output <- gsub(".tif", "_out.tif", input)
Or change the folder
output <- gsub("this/path", "that/path", input)
And then use a loop
for (i in 1:length(input)) {
# read input[i]
# write output[i]
}
A working minimal example (using a single file):
library(raster)
input <- system.file("external/test.grd", package="raster")
# write to current working directory, but with asc extensions
output <- gsub(".grd", ".asc", basename(input))
for (i in 1:length(input)) {
r <- raster(input[i])
writeRaster(r, output[i], overwrite=TRUE)
}
Also, not directly related to your question, but this:
values(novoraster)[is.na(values(template))] <- NA
Should be
novoraster <- mask(novoraster, template)
You can use lapply. You would need a vector of file names, maybe with
fn_list <- list.files("PATH", pattern="asc$")
Your workhorse function could look like
one_file <- function(fn, template, rastergrande) {
mau <- raster(fn)
intermedio <- crop(mau, rastergrande)
novoraster <- resample(intermedio, template, "bilinear")
values(novoraster)[is.na(values(template))] <- NA
fn_out <- gsub("asc$", "_final.asc", fn)
writeRaster(novoraster, fn_out)
}
You can then apply one_file on fn_list:
library(raster)
library(usdm)
library(ggplot2)
library(rgdal)
rastergrande <- raster("PATH")
template <- raster("PATH")
lapply(fn_list, one_file, template=templ, rastergrande=rastergrande)
By the way, take a look at the terra package: https://www.youtube.com/watch?v=O8V4QJ13Gjc
Here you can work with a for loop or apply (lapply and mapply)
file_names <- list.files("PATH/", pattern = ".tif")
r_list <- lapply(file_names,raster)
rastergrande <- raster("PATH")
template <- raster("PATH")
intermedio <- list()
novoraster <- list()
Creating a vector with rasters names
rasternames <- c("name1","name2" ...)
For loop to process all rasters
for(i in 1:length(file_names){
intermedio[[i]] <- crop(r_list[[i]], rastergrande)
novoraster[[i]] <- resample(intermedio[[i]], template, "bilinear")
values(novoraster[[i]])[is.na(values(template))] <- NA
writeRaster(novoraster[[i]], paste0("PATH/", rasternames[i], ".asc"), overwrite=TRUE)
}
I have 120 county shapefiles in a directory "Counties". I want to use R to read in each shapefile and, for each shapefile, perform zonal statistics (mean) using a single raster layer "NOAA_normal_crop."
I was able to create a script that reads in all of the shapefiles as a list:
library(rgdal)
library(raster)
library(sf)
library(maptools)
NOAA_normal <- raster("C:/path/to/raster/noaa_normal.tif")
input_path <- "C:/path/to/Counties"
files <- list.files(dir, pattern="[.]shp$", full.names=TRUE)
allShapes <- lapply(files, readOGR)
But I still need to create a loop that goes through each individual shapefile and performs the zonal statistics. The loop below is one I tried, but it only gives me a single value in return whereas I want the mean value for each polygon.
for (i in 1:length(allShapes)){
ex <- extract(NOAA_normal_crop, allShapes[[i]], fun=mean, na.rm=TRUE, df=TRUE)
}
EDIT: I have also tried using lapply again, but this isn't working either.
lapply(allShapes, extract(NOAA_normal_crop, allShapes, fun=mean, na.rm=TRUE, df=TRUE))
# Error in round(y) : non-numeric argument to mathematical function
Thanks in advance for any advice you might have.
I believe your loop is correct. However it is not a list or array, so every iteration the value is being overwritten.
You should try:
ex = vector()
for (i in 1:length(allShapes)){
ex[i] <- extract(NOAA_normal_crop, allShapes[[i]], fun=mean, na.rm=TRUE, df=TRUE)
}
I would like to average several rasters by months. The files are named as follows:
africa_arc.20160101
africa_arc.20160102
africa_arc.20160103
Here is my current code, its not grabbing the files by their names:
require(raster)
require(maptools)
require(rgdal)
iDir <- "D:/RainfallFTP_unzipped"
oDir <- "D:/RainfallFTP_unzipped_AVG"
yrLs <- c("2016", "2017")
mthLs <- c(paste0("0",1:9), 10:12)
dyLs <- seq(as.Date('2016-01-01'),as.Date('2017-12-31'),by = 1)
dLs <- format(dyLs,"%d")
for (yr in yrLs){
for (mth in mthLs){
r.stk <- stack(paste0(iDir, "/", "africa_arc.", yr, mth, dLs, ".tif", sep=""))
}
}
Where am I going wrong?
I think you should always avoid (double) loops when possible.
Here's a simple solution, with only a couple of lines:
library (raster)
iDir <- "D:/RainfallFTP_unzipped/"
# list rasters in iDir
rs <- list.files(iDir,full.names = T)
# extract year-month and use lapply to grep and stack
r.stk <- lapply(sub('.+(\\d{6})\\d{2}','\\1',rs),function(x) stack(grep(x,rs,value = T)))
This will leave you with a list with each element being a stack of rasters for each year-month combination.
Please note that this assumes your filenames are as in your examples (so they end in the date without extension) and the rasters are located in iDir, not in any sub-directories of iDir.
If your rasters have a tif extension, as I think they have looking at your code, you just have to substitute the sub call within lapply to
sub('.+(\\d{6}).+tif$','\\1',rs)
to account for the file ending.
I have a question about stack() Raster Layers.
Usually I stack() Raster Layers like that:
stack(RasterLayer1,RasterLayer2,RasterLayer3) # e.g. for 3 Layers
My Question is, how can I stack() Raster Layers without typing in every Raster Layer?
For example: n is the amount of Raster Layers (e.g. 12), all named band.
I created n-Raster Layers and now I want to stack all without typing n-times the Name of the Raster Layers. So instead of typing:
stack(band1,band2,band3,band4,band5,band6,band7,band8,band9,band10,band11,band12)
I want to short that by stack(band[n]), but that doesn't work.
And if I create a list of all bands, I can't stack that list, because they don't appear in my Working Directory because I just created them.
Can anyone help me, please?
If your data is in a directory, you can use a search pattern (for example: *.tif, *.grd,...) and store it in a variable.
bands <- list.files(path=".",pattern="*.tif",full.names=TRUE,recursive=TRUE)
now assume that your data is called:
band_01.tif
band_02.tif
band_03.tif
band_04.tif
band_05.tif
band_06.tif
band_07.tif
then you can stack for example:
data_stack <- stack(bands) #stack all data
data_stack <- stack(bands[1:3]) #stack 1,2 and 3 data
data_stack <- stack(bands[c(1,3,5,7)])
I would recommend not to save them under separate variables like band1,band2,... but instead store them in a list. Here an example:
#Create empty rasters
ras1<- raster()
ras2<- raster()
#Initialise and append to list
list_ras <- list()
list_ras[[1]] <- ras1
list_ras[[2]] <- ras2
#Stack single bands
ras_stack <- stack(list_ras[[1]], list_ras[[2]])
#Stack all bands
ras_stack <- stack(list_ras)
Here is an other approach using mget:
# Generate some data
library(raster)
r <- raster()
r[] <- runif(ncell(r))
for (i in 1:10) assign(paste0("r", i), r)
# create a stack
stack(mget(ls(pattern = "^r.+")))
I've been trying to find a time-efficient way to merge multiple raster images in R. These are adjacent ASTER scenes from the southern Kilimanjaro region, and my target is to put them together to obtain one large image.
This is what I got so far (object 'ast14dmo' representing a list of RasterLayer objects):
# Loop through single ASTER scenes
for (i in seq(ast14dmo.sd)) {
if (i == 1) {
# Merge current with subsequent scene
ast14dmo.sd.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
} else if (i > 1 && i < length(ast14dmo.sd)) {
tmp.mrg <- merge(ast14dmo.sd[[i]], ast14dmo.sd[[i+1]], tolerance = 1)
ast14dmo.sd.mrg <- merge(ast14dmo.sd.mrg, tmp.mrg, tolerance = 1)
} else {
# Save merged image
writeRaster(ast14dmo.sd.mrg, paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg", sep = ""), format = "GTiff", overwrite = TRUE)
}
}
As you surely guess, the code works. However, merging takes quite long considering that each single raster object is some 70 mb large. I also tried Reduce and do.call, but that failed since I couldn't pass the argument 'tolerance' which circumvents the different origins of the raster files.
Anybody got an idea of how to speed things up?
You can use do.call
ast14dmo.sd$tolerance <- 1
ast14dmo.sd$filename <- paste(path.mrg, "/AST14DMO_sd_", z, "m_mrg.tif", sep = "")
ast14dmo.sd$overwrite <- TRUE
mm <- do.call(merge, ast14dmo.sd)
Here with some data, from the example in raster::merge
r1 <- raster(xmx=-150, ymn=60, ncols=30, nrows=30)
r1[] <- 1:ncell(r1)
r2 <- raster(xmn=-100, xmx=-50, ymx=50, ymn=30)
res(r2) <- c(xres(r1), yres(r1))
r2[] <- 1:ncell(r2)
x <- list(r1, r2)
names(x) <- c("x", "y")
x$filename <- 'test.tif'
x$overwrite <- TRUE
m <- do.call(merge, x)
The 'merge' function from the Raster package is a little slow. For large projects a faster option is to work with gdal commands in R.
library(gdalUtils)
library(rgdal)
Build list of all raster files you want to join (in your current working directory).
all_my_rasts <- c('r1.tif', 'r2.tif', 'r3.tif')
Make a template raster file to build onto. Think of this a big blank canvas to add tiles to.
e <- extent(-131, -124, 49, 53)
template <- raster(e)
projection(template) <- '+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs'
writeRaster(template, file="MyBigNastyRasty.tif", format="GTiff")
Merge all raster tiles into one big raster.
mosaic_rasters(gdalfile=all_my_rasts,dst_dataset="MyBigNastyRasty.tif",of="GTiff")
gdalinfo("MyBigNastyRasty.tif")
This should work pretty well for speed (faster than merge in the raster package), but if you have thousands of tiles you might even want to look into building a vrt first.
You can use Reduce like this for example :
Reduce(function(...)merge(...,tolerance=1),ast14dmo.sd)
SAGA GIS mosaicking tool (http://www.saga-gis.org/saga_tool_doc/7.3.0/grid_tools_3.html) gives you maximum flexibility for merging numeric layers, and it runs in parallel by default! You only have to translate all rasters/images to SAGA .sgrd format first, then run the command line saga_cmd.
I have tested the solution using gdalUtils as proposed by Matthew Bayly. It works quite well and fast (I have about 1000 images to merge). However, after checking with document of mosaic_raster function here, I found that it works without making a template raster before mosaic the images. I pasted the example codes from the document below:
outdir <- tempdir()
gdal_setInstallation()
valid_install <- !is.null(getOption("gdalUtils_gdalPath"))
if(require(raster) && require(rgdal) && valid_install)
{
layer1 <- system.file("external/tahoe_lidar_bareearth.tif", package="gdalUtils")
layer2 <- system.file("external/tahoe_lidar_highesthit.tif", package="gdalUtils")
mosaic_rasters(gdalfile=c(layer1,layer2),dst_dataset=file.path(outdir,"test_mosaic.envi"),
separate=TRUE,of="ENVI",verbose=TRUE)
gdalinfo("test_mosaic.envi")
}
I was faced with this same problem and I used
#Read desired files into R
data_name1<-'file_name1.tif'
r1=raster(data_name1)
data_name2<-'file_name2.tif'
r2=raster(data_name2)
#Merge files
new_data <- raster::merge(r1, r2)
Although it did not produce a new merged raster file, it stored in the data environment and produced a merged map when plotted.
I ran into the following problem when trying to mosaic several rasters on top of each other
In vv[is.na(vv)] <- getValues(x[[i]])[is.na(vv)] :
number of items to replace is not a multiple of replacement length
As #Robert Hijmans pointed out, it was likely because of misaligned rasters. To work around this, I had to resample the rasters first
library(raster)
x <- raster("Base_raster.tif")
r1 <- raster("Top1_raster.tif")
r2 <- raster("Top2_raster.tif")
# Resample
x1 <- resample(r1, crop(x, r1))
x2 <- resample(r2, crop(x, r2))
# Merge rasters. Make sure to use the right order
m <- merge(merge(x1, x2), x)
# Write output
writeRaster(m,
filename = file.path("Mosaic_raster.tif"),
format = "GTiff",
overwrite = TRUE)