disaggregating raster causes insufficient disk space - r

I am trying to disaggregate a raster using terra package. My original raster is:
library(terra)
my_raster
class : SpatRaster
dimensions : 180, 360, 1 (nrow, ncol, nlyr)
resolution : 1, 1 (x, y)
extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84
source : EHF2_2020_max.nc_rotated.nc
varname : X2020
name : X2020
I want to drop the resolution to the following target raster
target_raster
class : SpatRaster
dimensions : 21600, 43200, 1 (nrow, ncol, nlyr)
resolution : 0.008333333, 0.008333333 (x, y)
extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84
source : GDP_PPP_30arcsec_v3.nc
varname : GDP_PPP (Gross Domestic Production (GDP) (PPP))
name : GDP_PPP_3
unit : constant 2011 international US dollar
When I did this, I get below error:
disagg_raster <- disagg(my_raster, fact = c(21600,43200))
Error: [disagg] insufficient disk space (perhaps from temporary files?)
I have a fresh R session and other than these two objects, nothing else is loaded in my environment. What is causing this error?
However, when I disaggregate raster using resample, I do not get any memory issue
resample_raster <- resample(my_raster, target_raster, method='bilinear')

I get below error:
disagg_raster <- disagg(my_raster, fact = c(21600,43200))
# Error: [disagg] insufficient disk space (perhaps from temporary files?)
That says there is not enough disk space to write the file. Output SpatRasters are are written to disk if they are deemed too large to keep in memory. As you do not provide a filename argument, the file would go to the tempdir() folder and that does not have enough disk space.
This is happening because you use trying to create a monster of a raster (easy to do and see for a raster that has no cell values):
library(terra)
r <- rast()
disagg(r, fact = c(21600,43200))
#class : SpatRaster
#dimensions : 3888000, 15552000, 1 (nrow, ncol, nlyr)
#resolution : 2.314815e-05, 4.62963e-05 (x, y)
#extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : lon/lat WGS 84
It appears that you misunderstood the fact argument. To get the spatial resolution of your target raster you can do:
disagg(r, fact = 120)
#class : SpatRaster
#dimensions : 21600, 43200, 1 (nrow, ncol, nlyr)
#resolution : 0.008333333, 0.008333333 (x, y)
#extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : lon/lat WGS 84
That is still big enough, but it should not create problems.

Related

terra and raster package gives flip warning for reading the same raster

I have a raster in working directory:
When I read this using terra package, it reads fine
r1 <- terra::rast(my_rast)
r1
class : SpatRaster
dimensions : 6000, 6000, 8 (nrow, ncol, nlyr)
resolution : 0.0008333333, 0.0008333333 (x, y)
extent : 59.99958, 64.99958, 24.99958, 29.99958 (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326)
source : n25e060.tif
names : n25e060_1, n25e060_2, n25e060_3, n25e060_4, n25e060_5, n25e060_6, ...
But If I read the same raster using the raster package, I get the following warning:
r2 <- raster::raster(my_rast)
Warning message:
In .rasterFromGDAL(x, band = band, objecttype, ...) :
data seems flipped. Consider using: flip(x, direction='y')
class : RasterLayer
band : 1 (of 8 bands)
dimensions : 6000, 6000, 3.6e+07 (nrow, ncol, ncell)
resolution : 0.0008333333, 0.0008333333 (x, y)
extent : 59.99958, 64.99958, 25.00042, 30.00042 (xmin, xmax, ymin, ymax)
crs : +proj=longlat +datum=WGS84 +no_defs
source : n25e060.tif
names : n25e060
values : 0, 255 (min, max)
Why is this happening?
That suggests that either {terra} can directly use "flipped" raster data, whereas that {raster} cannot or that {terra} does not notice that the data are flipped. I believe the former to be true, but, while this would be easy for you to verify, I do not have the file so I cannot be 100% sure.

match extent of two rasters in R

I have two raster:
raster1
class : SpatRaster
dimensions : 21600, 43200, 1 (nrow, ncol, nlyr)
resolution : 0.008333333, 0.008333333 (x, y)
extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
raster2
class : SpatRaster
dimensions : 720, 1440, 1 (nrow, ncol, nlyr)
resolution : 0.25, 0.25 (x, y)
extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84
I want to run zonal stastics to calculate sum of smaller raster raster1 on bigger raster raster2:
terra::zonal(raster1, raster2, fun = sum, as.raster=T, filename = 'zonal.tif')
Error: [zonal] dimensions and/or extent do not match
I wasn't sure why the extent are not matching until I did this
terra::ext(raster1)
SpatExtent : -180.000001017276, 180.000001017276, -90.0000010172997, 90.0000010172997 (xmin, xmax, ymin, ymax)
terra::ext(raster2)
SpatExtent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
which shows that raster1 extent have some imprecision. What are different ways I can fix this?
EDIT: I tried the suggestion in the comment
terra::crs(raster2) <- sf::st_crs(4326)$wkt
terra::crs(raster1) <- sf::st_crs(4326)$wkt
terra::ext(raster2)
SpatExtent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
terra::ext(raster1)
SpatExtent : -180.000001017276, 180.000001017276, -90.0000010172997, 90.0000010172997 (xmin, xmax, ymin, ymax)
But my extent are still not matching
The error message is
Error: [zonal] dimensions and/or extent do not match
In this case, that clearly refers to the difference is dimensions, which are
#dimensions : 21600, 43200, 1 (nrow, ncol, nlyr)
#dimensions : 720, 1440, 1 (nrow, ncol, nlyr)
I do not think the small difference in the extent is relevant at all.
To make the dimensions match you can use disagg or aggregate with a factor of 30.
If the crs of raster2 is different from that of raster1 you can (in this case, where you clearly have two global lon/lat rasters) fix that with
crs(raster2) <- crs(raster1)
And for good measure, you could in this case also do
ext(raster2) <- ext(raster1)
Mismatching extents don't stop zonal working, so I think that error message is wrong. Here's two rasters that only differ in their CRS:
> library(terra)
terra 1.5.21
> raster1 = rast()
> raster1[]=1:360
> raster2 = rast()
> raster2[]=1:360
> crs(raster1) = ""
> zonal(raster1, raster2)
Error: [zonal] dimensions and/or extent do not match
I guess it could be argued that a different CRS means a different extent in the same way that "23" is different to "23 km". But the extent and the dimensions are all the same according to these tests:
> ext(raster1) == ext(raster2)
[1] TRUE
> dim(raster1) == dim(raster2)
[1] TRUE TRUE TRUE
Rasters with a different extent work fine with zonal (once I've set the CRSs to match):
> ext(raster2) = c(-180.01, 180.01, -90.01, 90.01)
> crs(raster2) = ""
> z = zonal(raster1, raster2)
>
If you do want to change an extent, you can use ext(r) = ... as above, but you should try and figure out why extents don't match. Likely its because you have data on points and you may have cells extending out from points...

convert raster from coarse to finer resolution

This is my raster properties:
class : RasterLayer
dimensions : 3001, 3000, 9003000 (nrow, ncol, ncell)
resolution : 0.12, 0.034 (x, y)
extent : -180, 180, -40.034, 62 (xmin, xmax, ymin, ymax)
crs : +proj=longlat +datum=WGS84 +no_defs
source : test.tif
names : test
I want to convert this into a raster with a resolution of 0.008333333 in both x and y direction
library(raster)
tar_res <- 0.008333333
disagg_raster <- disaggregate(my_raster,
fact = c(res(my_raster)[1]/tar_res, res(my_raster)[2]/tar_res))
disagg_raster
class : RasterLayer
dimensions : 12004, 42000, 504168000 (nrow, ncol, ncell)
resolution : 0.008571429, 0.0085 (x, y)
extent : -180, 180, -40.034, 62 (xmin, xmax, ymin, ymax)
crs : +proj=longlat +datum=WGS84 +no_defs
How do I get the resolution to be 0.008333333 in both x and y direction?
If you cannot get there by (dis) aggregating you can use resample instead.
# example data
library(terra)
r <- rast(nrow=3001, ncol=3000, ymin=-40.034, ymax=62)
# create empty template with the desired geometry
tmp <- rast(r)
res(tmp) <- 1/120
ymax(tmp) <- ymax(r)
# resample
x <- resample(r, tmp)
See here for a more general and longer answer.

R - Calculate mean of rasterbrick

I am trying to calculate the mean of a rasterbrick over specific timesteps in R.
Subsetting the dataset shows a lay-out that appears to be correct (to me), however R does not allow any computations to follow up.
> r_sub <- b[[1699:1862]]
> r_sub
class : RasterStack
dimensions : 127, 147, 18669, 164 (nrow, ncol, ncell, nlayers)
resolution : 5, 5 (x, y)
extent : 603299.4, 604034.4, 6615598, 6616233 (xmin, xmax, ymin, ymax)
crs : NA
names : X2019.02.09.19, X2019.02.09.20, X2019.02.09.21, X2019.02.09.22, X2019.02.09.23, X2019.02.09.24, X2019.02.10.1, X2019.02.10.2, X2019.02.10.3, X2019.02.10.4, X2019.02.10.5, X2019.02.10.6, X2019.02.10.7, X2019.02.10.8, X2019.02.10.9, ...
> r_mean <- calc(r_sub, mean)
Error in Rsx_nc4_get_vara_double: NetCDF: Index exceeds dimension bound
Var: SWIT Ndims: 3 Start: 1698,0,0 Count: 1,127,147
Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset, :
C function R_nc4_get_vara_double returned error
How to solve this specific error?
I am guessing that this is related to a problem with your file, not with the code.
I did this and it worked well
library(raster)
b <- brick("filename.nc")
r_sub <- b[[1699:1862]]
r_sub
#class : RasterStack
#dimensions : 160, 320, 51200, 164 (nrow, ncol, ncell, nlayers)
#resolution : 1.125, 1.121277 (x, y)
#extent : -0.5625, 359.4375, -89.70216, 89.70216 (xmin, xmax, ymin, ymax)
#crs : +proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0
mean(r_sub)
#class : RasterLayer
#dimensions : 160, 320, 51200 (nrow, ncol, ncell)
#resolution : 1.125, 1.121277 (x, y)
#extent : -0.5625, 359.4375, -89.70216, 89.70216 (xmin, xmax, ymin, ymax)
#crs : +proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0
#source : memory
#names : layer
#values : 3.033032e-07, 0.0002482847 (min, max)
Also, you might want to look into using stackApply

Reproject MODIS data using R (results in NAs or no spatial extent)

I am using GLASS albedo data stored here for pre-2000 (AVHRR) data and here for post-2000 data (MODIS). My end goal is to create a raster stack of each month that contains white sky albedo data from 1982-2015. The problem I have run into is that the MODIS and AVHRR data are in different spatial reference systems and I can't seem to reproject them to be in the same system.
I convert from hdf to tif using R like this:
fileavhrr <- ".../GLASS02B05.V04.A1990161.2018062.hdf"
filemodis<-".../GLASS02B06.V04.A2013169.2017128.hdf"
gdal_translate(get_subdatasets(filemodis)[10], dst_dataset =
".../modis.tif")
gdal_translate(get_subdatasets(fileavhrr)[8], projwin = c(-180,90,180,50), dst_dataset = ".../avhrr.tif") #ideally I'd only like data north of 50 degrees
avhrr<- raster(".../avhrr.tif")
#class : RasterLayer
#dimensions : 800, 7200, 5760000 (nrow, ncol, ncell)
#resolution : 0.05, 0.05 (x, y)
#extent : -180, 180, 50, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +ellps=clrk66 +no_defs
#values : -32768, 32767 (min, max)
modis<- raster(".../modis.tif")
#class : RasterLayer
#dimensions : 3600, 7200, 25920000 (nrow, ncol, ncell)
#resolution : 154.4376, 308.8751 (x, y)
#extent : -20015109, -18903159, 8895604, 10007555 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=sinu +lon_0=0 +x_0=0 +y_0=0 +a=6371007.181
+b=6371007.181 +units=m +no_defs
#values : -32768, 32767 (min, max)
Here are things I have tried:
1.) Use the MODIS Reprojection Tool. For whatever reason, this tool seems to think the subdatasets of the MODIS .hdf files are only one tile (the upper left most tile, tile 0,0) and not the global dataset. My understanding is that the MODIS data are global (not in tiles?), so I do not know why the MRT is doing this.
2.) Use the raster package in R.
projectedMODIS <- projectRaster(modis,avhrr,method="bilinear")
This returns a raster with values that are all NA:
class : RasterLayer
dimensions : 800, 7200, 5760000 (nrow,> ncol, ncell)
resolution : 0.05, 0.05 (x, y)
extent : -180, 180,> 50, 90 (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +ellps=clrk66 +no_defs
values : NA, NA (min, max)
3.) Use the gdalUtils package in R:
gdalwarp(srcfile=get_subdatasets(filemodis)[10], dstfile= ".../gdalMODIS_avhrr.tif", s_srs = crs(modis), t_srs =crs(avhrr) )
This returns a raster with essentially no spatial extent.
gdalMODISavhrr<-raster(".../gdalMODIS_avhrr.tif")
#class : RasterLayer
#dimensions : 357, 12850, 4587450 (nrow, ncol, ncell)
#resolution : 0.02801551, 0.02801573 (x, y)
#extent : -180, 179.9993, 79.99838, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +ellps=clrk66 +no_defs
#values : -32768, 32767 (min, max)
Any ideas on why reprojecting this MODIS data is so difficult?
I have not tried this, but from looking at the package gdalUtils, the function gdalwarp() might do what you need?

Resources