Trouble reading a netcdf file and convert it into a SpatRaster - r

I am struggling to open a NetCDF file and convert it into a raster using R. The
data is supposed to be on a regular grid of 25 km by 25 km. It contains sea
ice concentration in the Arctic.
library(terra)
#> terra 1.5.21
library(ncdf4)
file <- "~/Downloads/data_sat_Phil_changt_grid/SIC_SMMR_month_2015.nc"
I am getting a warning about the extent not found.
r <- rast(file)
#> Error in R_nc4_open: No such file or directory
#> Warning: [rast] GDAL did not find an extent. Cells not equally spaced?
We can see that there is a problem with the coordinates/extent.
r
#> class : SpatRaster
#> dimensions : 448, 304, 12 (nrow, ncol, nlyr)
#> resolution : 0.003289474, 0.002232143 (x, y)
#> extent : 0, 1, 0, 1 (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84
#> source : SIC_SMMR_month_2015.nc:sic
#> varname : sic
#> names : sic_1, sic_2, sic_3, sic_4, sic_5, sic_6, ...
I can open the nc file with nc_open() and I see that the coordinates are present.
nc <- nc_open(file)
names(nc$var)
#> [1] "lat" "lon" "sic"
lat <- ncvar_get(nc, "lat")
lon <- ncvar_get(nc, "lon")
dim(lat)
#> [1] 304 448
dim(lon)
#> [1] 304 448
dim(r)
#> [1] 448 304 12
Is it possible to assemble this data (the SIC values and the coordinates) to create a SpatRaster?
The nc file can be downloaded here: https://easyupload.io/pfth0s
Created on 2022-05-20 by the reprex package (v2.0.1)

The data are gridded, but the file does not specify the coordinates, nor the coordinate reference system. The file specifies the lon/lat values associated with the cells, but does not help us much, as these are clearly not on a regular grid. That is easy to see from plot(r)
NAflag(r) = -9999
plot(r,1)
And also from
p = cbind(as.vector(lon), as.vector(lat))
plot(p, cex=.1, xlab="lon", ylab="lat")
So what you need to find out, is which coordinate reference system (crs) is used, clearly some kind of polar crs. And what the extent of the data set is.
From the website you point to, I take it we can use:
crs(r) = "+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +k=1 +x_0=0 +y_0=0 +a=6378273 +b=6356889.449 +units=m +no_defs"
ext(r) = c(-3850000, 3750000, -5350000, 5850000)
r
#class : SpatRaster
#dimensions : 448, 304, 12 (nrow, ncol, nlyr)
#resolution : 25000, 25000 (x, y)
#extent : -3850000, 3750000, -5350000, 5850000 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +x_0=0 +y_0=0 +a=6378273 +b=6356889.449 +units=m +no_defs
#source : SIC_SMMR_month_2015.nc:sic
#varname : sic
#names : sic_1, sic_2, sic_3, sic_4, sic_5, sic_6, ...
The results look good:
g = geodata::gadm("Greenland", level=0, path=".")
gg = project(g, crs(r))
plot(r,1)
lines(gg)
But this is of course not a good way to do such things; the ncdf file should have contained all the metadata required.

Related

Save raster to USGS DEM Format in R

Similar to this question:
I would like to know how to do the reverse and save an .img raster image into a USGS DEM format.
Based on GDAL docs, it seems like it would be possible but when I run rgdal::getGDALDriverNames() in R I get the following:
name long_name create copy isRaster
139 USGSDEM USGS Optional ASCII DEM (and CDED) FALSE TRUE TRUE
which seems to imply that it won't create these files?
I was hoping to do something like:
library(raster)
# read
img <- raster("Raster_100ft_2022_10_18.img")
# convert to DEM
writeRaster(img, 'test.dem')
But raster doesn't seem to recognize that output format.
Is there some other method to save as USGS DEM files?
Thanks
For me it works with terra. If that's proper "USGSDEM" file, that's another question. From gdal reference it should save the file as well: https://gdal.org/drivers/raster/usgsdem.html
f <- system.file("ex/elev.tif", package="terra")
r <- terra::rast(f)
terra::writeRaster(r, filename = "test.dem", filetype = "USGSDEM", overwrite = TRUE)
raster::raster("test.dem")
#> class : RasterLayer
#> dimensions : 90, 95, 8550 (nrow, ncol, ncell)
#> resolution : 0.008333333, 0.008333333 (x, y)
#> extent : 5.741667, 6.533333, 49.44167, 50.19167 (xmin, xmax, ymin, ymax)
#> crs : +proj=longlat +datum=WGS84 +no_defs
#> source : test.dem
#> names : elevation
#> values : 141, 547 (min, max)
Created on 2022-10-20 with reprex v2.0.2

Projecting points with terra package R

I need to project longitude/latitude coordinates in the terra package, but I don't believe it is working correctly, as I am trying to extract data from a raster with this projection, but the data is not being extracted correctly.
Here's my lon/lat points and the code I am using to try to project them.
latlon_df <- structure(list(Lon = c(-103.289, -96.6735, -96.9041, -96.76864,
-102.4694, -96.6814, -97.7504, -99.6754, -96.4802, -103.0007,
-96.8897, -101.8539, -103.9717, -101.253, -99.1134, -96.5849,
-98.0301, -99.9537, -99.4601, -99.7122, -103.8278, -98.931, -102.1081,
-101.7162, -100.115, -101.3448, -100.7805, -103.5606, -96.5302,
-99.4156, -103.281, -100.0063, -97.9928, -100.7208, -98.5289,
-96.762, -96.9218, -97.1024, -103.3793, -101.0841, -102.6745,
-96.9188, -97.5154, -100.7435, -98.6938), Lat = c(45.5194, 44.3099,
43.0526, 44.3252, 45.5183, 43.7316, 45.6796, 45.4406, 44.7154,
44.0006, 43.7687, 43.9599, 43.4737, 44.9875, 45.0292, 44.0867,
45.5735, 44.9895, 44.5256, 43.5938, 43.7343, 45.7163, 45.9189,
43.1672, 45.6716, 45.9154, 45.7963, 44.6783, 44.5073, 43.7982,
43.3784, 44.2912, 43.3841, 43.2002, 44.8579, 43.5048, 43.5033,
45.1055, 44.4245, 45.4167, 44.5643, 44.304, 45.2932, 43.5601,
43.7321)), class = "data.frame", row.names = c(NA, -45L))
latlons <- terra::vect(latlon_df,geom=c('Lon','Lat'),crs="+proj=longlat")
lcc <- terra::project(latlons,"+proj=lcc +lat_0=38.5 +lon_0=262.5 +lat_1=38.5 +lat_2=38.5 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs")
var_df <- terra::extract(grib_data,lcc)[,-1]
The raster data (grib_data) I am using comes from here (it is way too big for me to put on here). https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.20210612/conus/hrrr.t00z.wrfsubhf00.grib2
I am not sure what I am doing wrong here, as I have used this method previously, and it seemed to work fine. Any help would be wonderful.
EDIT: The specific problem I am having is that I am not getting any different values for each lon/lat pair. The value for each variable is different, but all the values for the stations (different lon/lats are the same).
Why do you think it has to do with the projection? Either way, it appears to work for me.
url <- "https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.20210612/conus/hrrr.t00z.wrfsubhf00.grib2"
if (!file.exist(basename(url))) download.file(url, basename(url), mode="wb")
url <- paste0(url, ".idx")
if (!file.exist(basename(url))) download.file(url, basename(url), mode="wb")
library(terra)
r <- rast("hrrr.t00z.wrfsubhf00.grib2")
r
#class : SpatRaster
#dimensions : 1059, 1799, 49 (nrow, ncol, nlyr)
#resolution : 3000, 3000 (x, y)
#extent : -2699020, 2697980, -1588806, 1588194 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=lcc +lat_0=38.5 +lon_0=262.5 +lat_1=38.5 +lat_2=38.5 +x_0=0 +y_0=0 +R=6371229 +units=m +no_defs
#source : hrrr.t00z.wrfsubhf00.grib2
#names : 0[-] ~here", 0[-] ~tops", 0[-] ~here", 0[-] ~here", 0[-] ~face", 1000[~ound", ...
You can check of the points overlap with the raster data
plot(r, 1)
points(lcc)
And extract. It takes very long with grib files, but it does appear to work
e <- extract(r, lcc)
head(e[,c(1,6,9)])
# ID 0[-] SFC="Ground or water surface" 0[-] SFC="Ground or water surface".1
#1 1 85100 11.775471
#2 2 54400 11.087971
#3 3 79300 9.900471
#4 4 49200 10.712971
#5 5 70800 9.212971
#6 6 56600 11.400471
Make sure you have the current (CRAN) version, or perhaps the development version that you can install like this:
install.packages('terra', repos='https://rspatial.r-universe.dev')
You can speed things up a lot by doing a single read from disk (by adding zero in this example)
e <- extract(r+0, lcc)
That is not always possible and I need to do some optimization behind the scences.

Unmatch raster and polygon data in R

The polygons (read in from a shapefile) are :
class : SpatialPolygonsDataFrame
features : 653
extent : -7115213, 4895580, 1368240, 7805331 (xmin, xmax, ymin,
ymax)
coord. ref. : +proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154
+x_0=0 +y_0=0 +datum=NAD27 +units=us-ft +no_defs +ellps=clrk66
+nadgrids=#conus,#alaska,#ntv2_0.gsb,#ntv1_can.dat
variables : 6
names : cat, NAME, AREA_MI, lccount, lcsum,
lcmean
min values : 1, Alaska, 1.006402e+00, 1.000000e+01, 0.000000e+00,
0.00000000
max values : 99, Alaska, 9.945810e-01, 9.900000e+01, 9.960000e+02,
12.00000000
The raster (elevation) data is
class : RasterLayer
dimensions : 6800, 9200, 62560000 (nrow, ncol, ncell)
resolution : 1305.521, 946.6311 (x, y)
extent : -7115213, 4895580, 1368240, 7805331 (xmin, xmax, ymin,
ymax)
coord. ref. : +proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154
+x_0=0 +y_0=0 +datum=NAD27 +units=us-ft +no_defs +ellps=clrk66
+nadgrids=#conus,#alaska,#ntv2_0.gsb,#ntv1_can.dat
data source : /Users/hong/Documents/GitHub/Alaska/akshd300m.tif
names : akshd300m
values : 0, 255 (min, max)
Does anyone how to solve this? I use tmap to plot this figure
Although the coordinate reference system is reported as +proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154 +x_0=0 +y_0=0 +datum=NAD27 +units=us-ft +no_defs +ellps=clrk66 for both data sets, clearly that cannot be true. It should be easy to figure out which one is correct by transforming known polygons to that crs. For example
library(raster)
library(rgdal)
usa <- getData('GADM', country='USA', level=1)
ak <- usa[usa$NAME_1 == 'Alaska', ]
ak_asa <- spTransform(ak, '+proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154 +x_0=0 +y_0=0 +datum=NAD27 +units=us-ft +no_defs +ellps=clrk66')
And plotting that one as well. Having established which one is correct (hopefully one of the two is), you can then try to trace back what went wrong with the other one. With the information you have provided we cannot figure that out for you.

Loading multiple rasters in R results in some with attributes

I am trying to load multiple rasters (87 in total) into R for analysis. The rasters are being loaded direct to a stack using this code:
filelist <- list.files(path="mypath", pattern="tif$", full.names=TRUE)
stk1 <- raster::stack(filelist)
The data largely load correctly apart from 9 problematic bands that seem to have the same issue. These bad bands are loading with 'attributes'. See example below
goodband
> goodband
class : RasterLayer
dimensions : 1768, 3674, 6495632 (nrow, ncol, ncell)
resolution : 40, 40 (x, y)
extent : 135800.4, 282760.4, 22297.69, 93017.69 (xmin, xmax, ymin, ymax)
coord. ref. : +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +datum=OSGB36 +units=m +no_defs +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894
data source : goodband.tif
names : Good
values : 0.5, 36 (min, max)
badband
> badband
class : RasterLayer
dimensions : 1768, 3674, 6495632 (nrow, ncol, ncell)
resolution : 40, 40 (x, y)
extent : 135800.4, 282760.4, 22297.69, 93017.69 (xmin, xmax, ymin, ymax)
coord. ref. : +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +datum=OSGB36 +units=m +no_defs +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894
data source : badband.tif
names : Bad
values : 33, 6824 (min, max)
attributes :
ID OID Value Count
from: 0 0 33 433
to : 529 529 6824 1307
As you can see the badband has additional 'attributes' at the bottom. When trying to analyse bad bands they are just full of null values and don't operate like the good bands.
Could you advise how best to fix this problem?

"Failure during raster IO" when converting raster to data.frame?

Anyone have any idea what may be happening? Here is my console printout. What's weird is that the first four as.data.frame lines run without issue but the last one gives me the "Failure during raster IO" error.
library("raster", lib.loc="~/R/win-library/3.3")
library("rgdal", lib.loc="~/R/win-library/3.3")
#> rgdal: version: 1.1-10, (SVN revision 622)
#> Geospatial Data Abstraction Library extensions to R successfully loaded
#> Loaded GDAL runtime: GDAL 2.0.1, released 2015/09/15
#> Path to GDAL shared files: C:/Users/JV-Desktop/Documents/R/win-library/3.3/rgdal/gdal
#> Loaded PROJ.4 runtime: Rel. 4.9.2, 08 September 2015, [PJ_VERSION: 492]
#> Path to PROJ.4 shared files: C:/Users/JV-Desktop/Documents/R/win-library/3.3/rgdal/proj
liMax <- as.data.frame(raster("D:/LiDAR/LiDAR/30_m/m_30_max.tif"), xy=TRUE)
CanopyBase <- as.data.frame(raster("D:/LiDAR/LiDAR/30_m/m_30_avg.tif") - raster("D:/LiDAR/LiDAR/30_m/m_30_std.tif"), xy=TRUE)
rawData.li <- merge(liMax,CanopyBase)
processed.li <- rawData.li[complete.cases(rawData.li),]
satMax <- as.data.frame(stack("D:/LiDAR/Landsat/For_Joe/Max_2015.tif"), xy=TRUE)
satMed <- as.data.frame(stack("D:/LiDAR/Landsat/For_Joe/Med_2015.tif"), xy=TRUE)
#> Error: Failure during raster IO
In case it helps here is the printout for the info on the last two files:
stack("D:/LiDAR/Landsat/For_Joe/Max_2015.tif")
# class : RasterStack
# dimensions : 5640, 8763, 49423320, 7 (nrow, ncol, ncell, nlayers)
# resolution : 30, 30 (x, y)
# extent : -1491300, -1228410, 2709840, 2879040 (xmin, xmax, ymin, ymax)
# coord. ref. : +proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
# names : Max_2015.1, Max_2015.2, Max_2015.3, Max_2015.4, Max_2015.5, Max_2015.6, Max_2015.7
stack("D:/LiDAR/Landsat/For_Joe/Med_2015.tif")
# class : RasterStack
# dimensions : 5640, 8763, 49423320, 7 (nrow, ncol, ncell, nlayers)
# resolution : 30, 30 (x, y)
# extent : -1491300, -1228410, 2709840, 2879040 (xmin, xmax, ymin, ymax)
# coord. ref. : +proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
# names : Med_2015.1, Med_2015.2, Med_2015.3, Med_2015.4, Med_2015.5, Med_2015.6, Med_2015.7
# min values : -0.3208517, -0.5767694, -0.5474564, ?, ?, ?, ?
# max values : 1, 1, 1, ?, ?, ?, ?
I hope this is enough info and I am asking the question correctly. I am new to stackoverflow and relatively new to R and coding in general. Thanks for any insight you can give.

Resources