I am trying to import a netCDF file into a RasterBrick in R. The netCDF file has 3 dimensions.
library(ncdf)
nc <- open.ncdf("fm100_2003.nc");
print(nc)
[1] "file fm100_2003.nc has 3 dimensions:"
[1] "lon Size: 1386"
[1] "lat Size: 585"
[1] "day Size: 365"
[1] "------------------------"
[1] "file fm100_2003.nc has 1 variables:"
[1] "short dead_fuel_moisture_100hr[day,lon,lat] Longname:dead_fuel_moisture_100hr Missval:-9999"
The size of the day dimension correspond to daily fuel moisture for one year (365 days). I'd like to import these into a RasterBrick for additional analysis which is pretty straightforward with,
r <- "fm100_2003.nc"
b <- brick(r,varname="dead_fuel_moisture_100hr")
However, the issue is that the ncol and nlayers in the RasterBrick are switched, which results in an incorrect rasterLayer for each layer in the brick. The dimensions of the RasterBrick should read 1386, 585, 505890, 365 instead of the dimensions below:
class : RasterBrick
dimensions : 1386, 365, 505890, 585 (nrow, ncol, ncell, nlayers)
resolution : 1, 0.04166667 (x, y)
extent : 37619.5, 37984.5, -124.793, -67.043 (xmin, xmax, ymin, ymax)
coord. ref. : NA
data source : fm100_2003.nc
names : X49.3960227966309, X49.3543561299642, X49.3126894632975, X49.2710227966309, X49.2293561299642, X49.1876894632975, X49.1460227966309, X49.1043561299642, X49.0626894632975, X49.0210227966309, X48.9793561299642, X48.9376894632975, X48.8960227966309, X48.8543561299642, X48.8126894632975, ...
degrees_north: 25.0626894632975, 49.3960227966309 (min, max)
varname : dead_fuel_moisture_100hr
I am wondering if there is any way to specify the dimensions when creating the RasterBrick to avoid this problem?
It's rather strange, why the dimension is not correct. You could explore your ncdf file by using some command below:
# Open your dimension
# following by '$' you can use tab to see the next available command (R Studio)
r$dim
# REad value for each dimension
lon = get.var.ncdf(r, varid='lon')
lat = get.var.ncdf(r, varid='lat')
time = get.var.ncdf(r, varid='day')
I was able to figure out a solution (or perhaps work around) to the above problem.
I first import the netCDF file into an array in R.
dname <- "dead_fuel_moisture_100hr"
array1 <- get.var.ncdf(nc, dname)
dim(array1)
[1] 365 1386 585
The dimensions of array1 are: days, columns, rows. However, I can change the dimensions of the array:
array2<-aperm(array1, c(3, 2, 1))
dim(array2)
[1] 585 1386 365
Now the array is organized properly: rows, columns, days. At this point I can access the depth range I need (days 1 through 365) as a matrix:
fm.day.001<-array2[,,1]
...
fm.day.365<-array2[,,365]
The matrix can be converted to a raster too:
r2<-raster(nrow=585,ncol=1386,vals=fm.day.001, xmn=-124.7722, xmx=-67.06383, ymn=25.06269, ymx=49.39602)
You can try the dims argument. Something like:
b <- brick("fm100_2003.nc", varname="dead_fuel_moisture_100hr", dims=3:1)
This is experimental. Even if it creates the correct object, the object may not work for subsequent operations.
Related
I have an object called data (representing a field of wind strengths at geographical locations) obtained by using some code to read in a .grib file:
> data
ecmf : u-component of wind
Time:
2020/07/09 z00:00 0-0 h
Domain summary:
601 x 351 domain
Projection summary:
proj= latlong
NE = ( 50 , 75 )
SW = ( -10 , 40 )
Data summary:
-89.06099 -50.08242 -41.13694 -43.42623 -34.77617 -25.03278
data is 601 x 351 array of doubles:
> typeof(data)
[1] "double"
> is.array(data)
[1] TRUE
> dim(data)
[1] 601 351
but, as shown above, it also has extra information attached beyond the numerical values of the array elements (Time:, Projection summary etc). How do I extract these? Attempts such as data$time do not seem to work.
As suggested in the comments to the question, I was able to access the values I wanted using attributes(). attributes(data) returns a list of all the relevant elements.
I'd like to read in a GRIB2 file to R, but have been unable to install wgrib2 (after several hours of struggling) meaning that rNOMADS is not an option. That's okay, as GRIB2 files can be read by both the raster and rgdal packages. The problem I run into is that the names of the layers are stripped when reading in the file.
Here's an example.
# Load libraries
library(raster)
library(rgdal)
# Name of file
file_name <- "https://dd.weather.gc.ca/model_gem_regional/coupled/gulf_st-lawrence/grib2/00/001/CMC_coupled-rdps-stlawrence-ocean_latlon0.02x0.03_2020120100_P001.grib2"
# Load as raster brick
b <- brick(file_name)
# Get layer names
names(b)
# [1] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.1"
# [2] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.2"
# [3] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.3"
# [4] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.4"
# [5] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.5"
# [6] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.6"
# [7] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.7"
# [8] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.8"
# [9] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.9"
#[10] "CMC_coupled.rdps.stlawrence.ocean_latlon0.02x0.03_2020120100_P001.10"
As you can see, the name are just generic defaults. Next, I tried rgdal.
# Load using rgdal
r <- readGDAL(file_name)
# Get names
names(r)
# [1] "band1" "band2" "band3" "band4" "band5" "band6" "band7" "band8"
# [9] "band9" "band10"
Once again, default names. But, if I use the command line utility ncl_convert2nc to convert the GRIB2 file to NetCDF and then read in the NetCDF file with ncdf4 – an additional conversion step that I don't want to include in my workflow if it can be avoided – there are definitely variable names present.
# [1] "UOGRD_P0_L160_GLL0" "VOGRD_P0_L160_GLL0" "ICEC_P0_L1_GLL0"
# [4] "ICETK_P0_L1_GLL0" "UICE_P0_L1_GLL0" "VICE_P0_L1_GLL0"
# [7] "ICETMP_P0_L1_GLL0" "ICEPRS_P0_L1_GLL0" "CICES_P0_L1_GLL0"
#[10] "WTMP_P0_L1_GLL0"
QUESTION: Is there a way to extract or retain the variable/layer names when using rgdal or raster to read a GRIB2 file?
PS The reason I need to get the variable names from the file is because the layers don't match up with the order of layers as specified on the website when loaded with (e.g.) raster. This is apparent from the variable values. While I could use the variable names gleaned from the NetCDF file shown above, if the order of the layers changed this would break my package.
You can use the terra package instead of raster.
file_name <- "https://dd.weather.gc.ca/model_gem_regional/coupled/gulf_st-lawrence/grib2/00/001/CMC_coupled-rdps-stlawrence-ocean_latlon0.02x0.03_2020120100_P001.grib2"
b <- basename(file_name)
if (!file.exists(b)) download.file(file_name, b, mode="wb")
library(terra)
r <- rast(b)
r
#class : SpatRaster
#dimensions : 325, 500, 10 (nrow, ncol, nlyr)
#resolution : 0.03, 0.02 (x, y)
#extent : -71.015, -56.015, 45.49, 51.99 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +R=6371229 +no_defs
#source : CMC_coupled-rdps-stlawrence-ocean_latlon0.02x0.03_2020120100_P001.grib2
#names : 0[-] SFC="Ground or water surface", 0[-] SFC="Ground or water surface", 0[-] SFC="Ground or water surface", 0[-] SFC="Ground or water surface", 0[m] DBSL="Depth below sea level", 0[m] DBSL="Depth below sea level", ...
But the variable names do not match yours
names(r)
# [1] "0[-] SFC=\"Ground or water surface\"" "0[-] SFC=\"Ground or water surface\"" "0[-] SFC=\"Ground or water surface\""
# [4] "0[-] SFC=\"Ground or water surface\"" "0[m] DBSL=\"Depth below sea level\"" "0[m] DBSL=\"Depth below sea level\""
# [7] "0[-] SFC=\"Ground or water surface\"" "0[-] SFC=\"Ground or water surface\"" "0[-] SFC=\"Ground or water surface\""
#[10] "0[-] SFC=\"Ground or water surface\""
You can set the names to other pieces of the metadata
nms <- trimws(grep("GRIB_ELEMENT=", desc(b), value=TRUE))
names(r) <- gsub("GRIB_ELEMENT=", "", nms)
r
#class : SpatRaster
#dimensions : 325, 500, 10 (nrow, ncol, nlyr)
#resolution : 0.03, 0.02 (x, y)
#extent : -71.015, -56.015, 45.49, 51.99 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +R=6371229 +no_defs
#source : CMC_coupled-rdps-stlawrence-ocean_latlon0.02x0.03_2020120100_P001.grib2
#names : ICEC, ICETK, UICE, VICE, UOGRD, VOGRD, ..
names(r)
#[1] "ICEC" "ICETK" "UICE" "VICE" "UOGRD" "VOGRD" "WTMP" "ICET" "ICEPRS" "CICES"
I can change the behavior of terra such that it uses "GRIB_ELEMENT" (please let me know if that makes sense). But it is not clear to me how to get to the names you show. For example, below is the GDAL metadat for the first layer. You show ICEC_P0_L1_GLL0. All names have P0 and GLL0 so at least for this file, these seem redundant. But what does L1 refer to?
d <-desc(b)
d[35:46]
# [1] " GRIB_COMMENT=Ice cover [Proportion]"
# [2] " GRIB_DISCIPLINE=10(Oceanographic_Products)"
# [3] " GRIB_ELEMENT=ICEC"
# [4] " GRIB_FORECAST_SECONDS=3600 sec"
# [5] " GRIB_IDS=CENTER=54(Montreal) SUBCENTER=0 MASTER_TABLE=4 LOCAL_TABLE=0 SIGNF_REF_TIME=1(Start_of_Forecast) REF_TIME=2020-12-01T00:00:00Z PROD_STATUS=0(Operational) TYPE=1(Forecast)"
# [6] " GRIB_PDS_PDTN=0"
# [7] " GRIB_PDS_TEMPLATE_ASSEMBLED_VALUES=2 0 2 56 56 0 0 1 1 1 0 0 255 -127 -2147483647"
# [8] " GRIB_PDS_TEMPLATE_NUMBERS=2 0 2 56 56 0 0 0 1 0 0 0 1 1 0 0 0 0 0 255 255 255 255 255 255"
# [9] " GRIB_REF_TIME= 1606780800 sec UTC"
#[10] " GRIB_SHORT_NAME=0-SFC"
#[11] " GRIB_UNIT=[Proportion]"
#[12] " GRIB_VALID_TIME= 1606784400 sec UTC"
I see that "UOGRD" and "VOGRD" have L160 and have the number 160 in "GRIB_PDS_TEMPLATE_ASSEMBLED_VALUE" and "GRIB_PDS_TEMPLATE_NUMBERS" where the others have 1.
The metadata structure is described here but I could use some guidance about where to look to understand what to extract from the metadata.
test <- brick(file.name)
My data is very large and it would be difficult to upload.
I read in a NetCDF file using this code. The NetCDF file has lat longs included i the file. I know this because I can extract them (see below). However, when I read the netCDF file in as a brick, the brick's extent does not reflect the lat long that are in the netCDF file. It simply seems to center the raster in the center of the continental US (see first plot below):
library(ncdf4)
library(raster)
test <- brick(file.name)
test
class : RasterBrick
dimensions : 141, 240, 33840, 7273 (nrow, ncol, ncell, nlayers)
resolution : 25000, 25000 (x, y)
extent : -3e+06, 3e+06, -1762500, 1762500 (xmin, xmax, ymin, ymax)
crs : NA
source : C:/Users/anfangen/Box/Environmental-Data/Model-Pollen-Data/Raw-Data/RAGcount_1997-2016.nc
names : X1997.01.02.00.00.00, X1997.01.03.00.00.00, X1997.01.04.00.00.00, X1997.01.05.00.00.00, X1997.01.06.00.00.00, X1997.01.07.00.00.00, X1997.01.08.00.00.00, X1997.01.09.00.00.00, X1997.01.10.00.00.00, X1997.01.11.00.00.00, X1997.01.12.00.00.00, X1997.01.13.00.00.00, X1997.01.14.00.00.00, X1997.01.15.00.00.00, X1997.01.16.00.00.00, ...
Date/time : 1997-01-02 00:00:00, 2016-12-29 00:00:00 (min, max)
varname : count
If I pull the lat longs out using I can see that they exist:
test_nc_open <- nc_open(file_name.nc')
## Get the lat longs for the ragweed file. These are in matrix form. It is as yet unclear to me why we need them in this form.
lat <- ncvar_get(test_nc_open, varid = 'xlat')
lon <- ncvar_get(test_nc_open, varid = 'xlon')
lat
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,] 18.86005 19.05895 19.25808 19.45742 19.65697 19.85674 20.05672 20.25690 20.45730 20.65790 20.85870 21.05970
[2,] 18.92987 19.12906 19.32847 19.52810 19.72794 19.92799 20.12826 20.32874 20.52942 20.73031 20.93141 21.13270
[3,] 18.99921 19.19868 19.39837 19.59828 19.79841 19.99875 20.19931 20.40007 20.60105 20.80223 21.00361 21.20520
[4,] 19.06806 19.26781 19.46778 19.66797 19.86839 20.06901 20.26985 20.47091 20.67217 20.87363 21.07531 21.27719
[5,] 19.13641 19.33644 19.53669 19.73717 19.93786 20.13877 20.33989 20.54123 20.74278 20.94453 21.14649 21.34866
[6,] 19.20427 19.40458 19.60511 19.80586 20.00683 20.20802 20.40943 20.61105 20.81288 21.01492 21.21716 21.41962
[7,] 19.27163 19.47221 19.67302 19.87405 20.07530 20.27677 20.47846 20.68036 20.88247 21.08479 21.28732 21.49006
I tried to set the extent manually, but when I plot the result it does not match up with other spatial layers:
## Get state boundaries
states <- shapefile("GIS Data/State_boundaries/cb_2018_us_state_20m.shp")
test <- brick(file.name)
[1] "vobjtovarid4: error #F: I could not find the requsted var (or dimvar) in the file!"
[1] "var (or dimvar) name: crs"
[1] "file name: file.name.nc"
test <-
setExtent(test, c(min(apply(lon, 2, min)),
max(apply(lon, 2, max)),
min(apply(lat, 2, min)),
max(apply(lat, 2, max)) ), keepres =
FALSE, snap = FALSE)
crs(test) <- crs(states)
plot(test[[1]])
plot(states, add=TRUE)
I do not understand what I am doing wrong. I have looked at several other similar questions and cannot get this to work using that information.
print(test)
3 variables (excluding dimension variables):
float xlat[jx,iy]
long_name: Latitude on Cross Points
standard_name: latitude
units: degrees_north
grid_mapping: crs
float xlon[jx,iy]
long_name: Longitude on Cross Points
standard_name: longitude
units: degrees_east
grid_mapping: crs
float count[jx,iy,time]
cell_methods: time: point
grid_mapping: crs
coordinates: xlat xlon
units: particles m-3
standard_name: near-surface_particle_count
long_name: Near-surface particle count
3 dimensions:
time Size:7273 *** is unlimited ***
long_name: time
standard_name: time
units: hours since 1949-12-01 00:00:00 UTC
calendar: gregorian
bounds: time_bnds
iy Size:141
long_name: y-coordinate in Cartesian system
standard_name: projection_y_coordinate
units: m
axis: Y
_CoordinateAxisType: GeoY
jx Size:240
long_name: x-coordinate in Cartesian system
standard_name: projection_x_coordinate
units: m
axis: X
_CoordinateAxisType: GeoX
It is not clear why you are doing what you are doing. In this case, the solution you are looking for depends on what your goal is.
The file has a coordinate reference system (crs) that is not lon/lat. It seems that it does not specify what it is, or, if it does, raster does not understand it. You can probably get the crs from whoever provided you the file. To see if there info in the file you can do print(test)
The file also stores the longitude and latitude of the cells. But these are not on a regular grid, so you cannot use those. If you really need the raster data to be in lon/lat you can use something like
x <- projectRaster(test, "+proj=longlat +datum=WGS84")
But that leads to loss of precision, so it is best avoided. To extract values by point or polygon from the raster data, you should first transform those to the crs of the raster data. You can use sp::spTransform for that. After that, you can use raster::extract
So with SpatialPoints p, you can do something like this (but with the correct crs, utm 43 is certainly wrong on this case)
putm <- spTransform(p, CRS("+proj=utm +zone=43 +datum=WGS84"))
e <- extract(test, putm)
I have a raster stack/brick in R containing 84 layers and I am trying to name them according to year and month from 199911 to 200610 (November 1999 to October 2006). However for some reason R keeps adding an "X" onto the beginning of any names I give my layers.
Does anyone know why this is happening and how to fix it? Here are some of the ways I've tried:
# Import raster brick
rast <- brick("rast.tif")
names(rast)[1:3]
[1] "MonthlyRainfall.1" "MonthlyRainfall.2" "MonthlyRainfall.3"
## Method 1
names(rast) <- paste0(rep(1999:2006, each=12), 1:12)[11:94]
names(rast)[1:3]
[1] "X199911" "X199912" "X20001"
## Method 2
# Create a vector of dates
dates <- format(seq(as.Date('1999/11/1'), as.Date('2006/10/1'), by='month'), '%Y%m')
dates[1:3]
[1] "199911" "199912" "200001"
# Set names
rast <- setNames(rast, dates)
names(rast)[1:3]
[1] "X199911" "X199912" "X200001"
## Method 3
names(rast) <- paste0("", dates)
names(rast)[1:3]
[1] "X199911" "X199912" "X200001"
## Method 4
substr(names(rast), 2, 7)[1:3]
[1] "199911" "199912" "200001"
names(rast) <- substr(names(rast), 2, 7)
names(rast)[1:3]
[1] "X199911" "X199912" "X200001"
To some extent I have been able to work around the problem by adding "X" to the beginning of some of my other data but now its reached the point where I can't do that any more. Any help would be greatly appreciated!
R won't allow the column to begin with a numeral so it prepends a character to avoid that restriction.
I have been trying plot the following gridded netcdf file: "air.1999.nc" found at the following website:
http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.html
I have tried the code below based on answers I have found here and elsewhere, but no luck.
library(ncdf);
temp.nc <- open.ncdf("air.1999.nc");
temp <- get.var.ncdf(temp.nc,"air");
temp.nc$dim$lon$vals -> lon
temp.nc$dim$lat$vals -> lat
lat <- rev(lat)
temp <- temp[nrow(temp):1,]
temp[temp==-32767] <- NA
temp <- t(temp)
image(lon,lat,temp)
library(maptools)
data(wrld_simpl)
plot(wrld_simpl, add = TRUE)
This code was from modified from the one found here: The variable from a netcdf file comes out flipped
Does anyone have any ideas or experience with using these type of netcdf files? Thanks
In the question you linked the whole part from lat <- rev(lat) to temp <- t(temp) was very specific to that particular OP dataset and have absolutely no universal value.
temp.nc <- open.ncdf("~/Downloads/air.1999.nc")
temp.nc
[1] "file ~/Downloads/air.1999.nc has 4 dimensions:"
[1] "lon Size: 144"
[1] "lat Size: 73"
[1] "level Size: 12"
[1] "time Size: 365"
[1] "------------------------"
[1] "file ~/Downloads/air.1999.nc has 2 variables:"
[1] "short air[lon,lat,level,time] Longname:Air temperature Missval:32767"
[1] "short head[level,time] Longname:Missing Missval:NA"
As you can see from these informations, in your case, missing values are represented by the value 32767 so the following should be your first step:
temp <- get.var.ncdf(temp.nc,"air")
temp[temp=="32767"] <- NA
Additionnaly in your case you have 4 dimensions to your data, not just 2, they are longitude, latitude, level (which I'm assuming represent the height) and time.
temp.nc$dim$lon$vals -> lon
temp.nc$dim$lat$vals -> lat
temp.nc$dim$time$vals -> time
temp.nc$dim$level$vals -> lev
If you have a look at lat you see that the values are in reverse (which image will frown upon) so let's reverse them:
lat <- rev(lat)
temp <- temp[, ncol(temp):1, , ] #lat being our dimension number 2
Then the longitude is expressed from 0 to 360 which is not standard, it should be from -180 to 180 so let's change that:
lon <- lon -180
So now let's plot the data for a level of 1000 (i. e. the first one) and the first date:
temp11 <- temp[ , , 1, 1] #Level is the third dimension and time the fourth.
image(lon,lat,temp11)
And then let's superimpose a world map:
library(maptools)
data(wrld_simpl)
plot(wrld_simpl,add=TRUE)