Assume that I have large RasterBrick, and I would like to do consecutive difference, which means layer2-layer1, layer3-layer2, layer4-layer3, ……..
I was thinking to use Raster package for this, but i am not able to do it. Could any one suggest me a working (guessing) code please ? I am not providing reproducible data thinking that the issue is clear.
Thanks in advance.
The following seems to work:
library(raster)
r1 = raster()
r2 = raster()
r3 = raster()
dim(r2)
r1[] = runif(180*360)
r2[] = runif(180*360)
r3[] = runif(180*360)
r = stack(r1,r2,r3)
dim(r)
rd = calc(r, fun = diff)
dim(rd)
Related
I'm trying to aggregate a raster r of global extent from a ~300m*300m (10 arc‐seconds, 7.4GB) resolution to a ~10km resolution (0.083333 decimal degrees), i.e. a factor of 30.
Both the aggregate functions from the raster and the velox packages do not seem to handle such large dataset. I very much welcome recommendations!
# sample rasters
r <- raster(extent(-180,180, -90 , 90))
res(r)<-c(0.5/6/30, 0.5/6/30)
r <- setValues(r, runif(ncell(r))) # Error: cannot allocate vector of size 62.6 Gb
# velox example
devtools::install_github('hunzikp/velox')
library(velox)
vx <- velox(r) # the process aborts in linux
vx$aggregate(factor=30, aggtype='mean')
# raster example
r_agg <- aggregate(r, fact=30)
You say that raster cannot handle a large raster like that, but that is not true. The problem is that you are trying to create a very large data set in memory --- more memory than your computer has available. You can use the init function instead. I show that below but not using a global 300 m raster to make the example run a bit faster.
library(raster)
r <- raster(ymn=80, res=0.5/6/30)
r <- init(r, "col")
r_agg <- aggregate(r, fact=30)
You get better mileage with terra
library(terra)
rr <- rast(ymin=80, res= 0.5/6/30)
rr <- init(rr, "col")
rr_agg <- aggregate(rr, fact=30)
In addition to Robert's suggestion, I'd resample the rast with a template so the ext and crs would be precise.
r <- terra::rast("your_rast.tif") %>%
aggregate(., fact = 30) %>%
resample(., template_rast, filename ="sth.tif",
wopt = list(gdal = c("COMPRESS=LZW", "TFW=YES", "BIGTIFF=YES"),
tempdir = "somewhere_you_have_a_lot_of_space", todisk = TRUE))
Those wopt options might help you a lot with large rasters.
I have a few very small country-level polygon and point shapefiles that I would like to rasterize in R. The final product should be one global binary raster (indicating whether grid cell center is covered by a polygon / point lies within cell or not). My approach is to loop over the shapefiles and do the following for each shapefile:
# load shapefile
shp = sf::read_sf(shapefile_path)
# create a global raster template with resolution 0.0083
ext = extent(-180.0042, 180.0042, -65.00417, 75.00417)
gridsize = 0.008333333
r = raster(ext, res = gridsize)
# rasterize polygon or point shapefile to raster
rr = rasterize(shp, r, background = 0) #all grid cells that are not covered get 0
# convert to binary raster
values(rr)[values(rr)>0] = 1
Here, rr is the raster file where the polygons / points in shp are coded as 1 and all other grid cells are coded as 0. Afterwards, I take the sum over all rr to arrive at one global binary raster file including all polygons / points.
The final two steps are incredibly slow. In addition, I get RAM problems when I try to replace the all positive values in rr with 1 as the cell count is very large due to the fine resolution. I was wondering whether it is possible to come up with a smarter solution for what I'd like to achieve.
I have already found the fasterize package that has a speedy implementation of rasterize which works fine. I think it would be of great help if someone has a solution where rasterize directly returns a binary raster.
This is how you can do this better with raster. Note the value=1 argument, and also that that I changed your specification of the extent -- as what you do is probably not correct.
library(raster)
v <- shapefile(shapefile_path)
ext <- extent(-180, 180, -65, 75)
r <- raster(ext, res = 1/120)
rr <- rasterize(v, r, value=1, background = 0)
There is no need for your last step, but you could have done
rr <- clamp(rr, 0, 1)
# or
rr <- rr > 0
# or
rr <- reclassify(rr, cbind(1, Inf, 1))
raster::calc is not very efficient for simple arithmetic like this
It should be much faster to rasterize all vector data in one step, rather than in a loop, especially with large rasters like this (for which the program may need to write a temp file for each iteration).
To illustrate this solution with example data
library(raster)
cds1 <- rbind(c(-180,-20), c(-140,55), c(10, 0), c(-140,-60))
cds2 <- rbind(c(-10,0), c(140,60), c(160,0), c(140,-55))
cds3 <- rbind(c(-125,0), c(0,60), c(40,5), c(15,-45))
v <- spLines(cds1, cds2, cds3)
r <- raster(ncols=90, nrows=45)
r <- rasterize(v, r, field=1)
To speed things up, you can use terra (the replacement for raster)
library(raster)
f <- system.file("ex/lux.shp", package="terra")
v <- as.lines(vect(f))
r <- rast(v, ncol=75, nrow=100)
x <- rasterize(v, r, field=1)
Something that seems to work computationally and significantly improves computation time is to
Create one large shapefile shp instead of working with individual rasterized shapefiles.
Use the fasterize package to rasterize the merged shapefile.
Use raster::calc to avoid memory problems.
ext = extent(-180.0042, 180.0042, -65.00417, 75.00417)
gridsize = 0.008333333
r = raster(ext, res=gridsize)
rr = fasterize(shp, r, background = 0) #all not covered cells get 0, others get sum
# convert to binary raster
fun = function(x) {x[x>0] <- 1; return(x) }
r2 = raster::calc(rr, fun)
I have been trying to get a graph from using distance() in the raster package. The raster dimensions are inherited from a SpatialPointsDataFrame. The raster works fine until I try distance(raster) and get the following warning:
Warning message:
In matrix(v, ncol = tr$nrows[1] + 3) :
data length [8837790] is not a sub-multiple or multiple of the number of rows [4384]
The bizarre thing is the raster works at smaller resolution but not large. The error can be replicated below:
Fails:
library(raster)
r <- raster(ncol=4386,nrow=6039)
r[] <- NA
r[500] <- 1
dist <- distance(r)
plot(dist / 1000)
Works:
r <- raster(ncol=438.6,nrow=603.9)
r[] <- NA
r[500] <- 1
dist <- distance(r)
plot(dist / 1000)
Why? Have I missed something really obvious?
An update to raster_2.4-20 solved the problem. Thanks Pascal and RobertH for pointing me in the right direction.
I'm a total beginner and have what I believe is a rather simple problem, but I am kind of stuck...
I have a data table with three columns: x, y and height values. I've imported the csv file and extracted the values:
data <- read.csv("C:/Data/heights.csv")
mydata <- data[, c("x", "y", "height")]
I then need to put the height values into a grid with these coordinates, which I created as a raster:
grid <- raster(ncol=2001, nrow=2001, xmn=479975, xmx=500025, ymn=119975, ymx=140025)
I think you are very close. I'd suggest you reading the raster package material, the vignette and description.
You'll have to pay attention to your data and how it is organized. By row? Column? raster cells values will be populated by row.
Using something simmilar to the example provided in the help ?raster
library(raster)
vals <- 1:100
r1 <- raster(nrows=10, ncols=10, xmn=479975, xmx=500025, ymn=119975, ymx=140025)
r1[] <- vals
plot(r1)
But the procedure may be different approaching with a spatial object
library(sp)
x <- 1:10
y <- 1:10
df <- data.frame(expand.grid(x=x,y=y), v=1:100)
coordinates(df)=~x+y
gridded(df) = TRUE
r2 <- raster(df)
plot(r2)
I think this may be the best approach but it would be better to have an idea about your data and how it is organized.
You could try this:
library(raster)
x <- rasterFromXYZ(mydata)
extent(x) <- c(479975, 500025, 119975, 140025)
I have two 30m x 30m raster files which I would like to sample points from. Prior to sampling, I would like to remove the clouded areas from the images. I turned to R and Hijman's Raster package for the task.
Using the drawPoly(sp=TRUE) command, I drew in 18 different polygons. The function did not seem to allow 18 polygons as one sp object, so I drew them all separately. I then gave the polygons a proj4string matching the rasters', and set them into a list. I ran the list through a lapply function to convert them to rasters (rasterize function in Hijman's package) with the polygon areas set to NA, and the rest of the image set to 1.
My end goal is one raster layer with the 18 areas set to NA. I have tried stacking the list of rasterized polygons, and subsetting it to put set a new raster to NA in the same areas. My reproducible code is below.
library(raster)
r1 <- raster(nrow=50, ncol = 50)
r1[] <- 1
r1[4:10,] <- NA
r2 <- raster(nrow=50, ncol = 50)
r2[] <- 1
r2[9:15,] <- NA
r3 <- raster(nrow=50, ncol = 50)
r3[] <- 1
r3[24:39,] <- NA
r4 <- raster(nrow=50, ncol = 50)
r4[] <- 1
s <- stack(r1, r2, r3)
test.a.cool <- calc(s, function(x){r4[is.na(x)==1] <- NA})
For whatever reason, the darn testacool is a blank plot, where I'm aiming to have it as a raster with all values except for the NAs in the stack, s, equal to 1.
Any tips?
Thanks.
Doing sum(s) will work, as sum() returns NA for any grid cell with even one NA value in the stack.
To see that it works, compare the figures produced by the following:
plot(s)
plot(sum(s))
I posted this question on the R-Sig-Geo forum, as well, and received a response from the package author. The two simplest solutions:
Use the sp package to rbind my polygons into one, then rasterize the polygon.
p <- rbind(p1, p2, p3...etc., makeUniqueIDs = TRUE)
r4 <- raster(nrow=50, ncol = 50)
r4[] <- 1
mask <- rasterize(p, r4)
mask[mask %in% 1:18] <- 1
#The above code produces a single raster file with
#my polygons as unique values, ready for masking.
And the second simple solution, as just pointed out by Josh O'Brien:
m <- sum(s)
test <- mask(r4, m)
The R community rocks. Problem solved (twice) within an hour. Thanks.
I'm not familiar with the package you are using, however looking at the final line in your code, I think the issue might be here:
function(x){r4[is.na(x)==1] <- NA})
It doesn't look like calc will do much with that. It is setting the values of r4 indexed by the NAs of x and setting those to NA.
What then? If anything, maybe:
function(x){r4[is.na(x)==1] <- NA; return(r4) })
Although, it's not clear if that is even what you are after.
You were on the right track. The [ operator is defined for rasters and raster stacks, so you could just use the single line:
r4[ any(is.na(s) ) ] <- NA
plot(r4)
If you wanted to use calc you could have used it like this:
r4 <- calc( s, function(x){ ( ! any( is.na(x) ) ) } )
r4[is.na(r4)] <- NA
plot(r4)