How to read GRIB files with R / GDAL? - r

I'm trying to read a GRIB file wavedata.grib with wave heights from the ECMWF ERA-40 website, using an R function. Here is my source code until now:
mylat = 43.75
mylong = 331.25
# read the GRIB file
library(rgdal)
library(sp)
gribfile<-"wavedata.grib"
grib <- readGDAL(gribfile)
summary = GDALinfo(gribfile,silent=TRUE)
save(summary, file="summary.txt",ascii = TRUE)
# >names(summary): rows columns bands ll.x ll.y res.x res.y oblique.x oblique.y
rows = summary[["rows"]]
columns = summary[["columns"]]
bands = summary[["bands"]]
# z=geometry(grib)
# Grid topology:
# cellcentre.offset cellsize cells.dim
# x 326.25 2.5 13
# y 28.75 2.5 7
# SpatialPoints:
# x y
# [1,] 326.25 43.75
# [2,] 328.75 43.75
# [3,] 331.25 43.75
myframe<-t(data.frame(grib))
# myframe[bands+1,3]=331.25 myframe[bands+2,3]=43.75
# myframe[1,3]=2.162918 myframe[2,3]=2.427078 myframe[3,3]=2.211989
# These values should match the values read by Degrib (see below)
# degrib.exe wavedata.grib -P -pnt 43.75,331.25 -Interp 1 > wavedata.txt
# element, unit, refTime, validTime, (43.750000,331.250000)
# SWH, [m], 195709010000, 195709010000, 2.147
# SWH, [m], 195709020000, 195709020000, 2.159
# SWH, [m], 195709030000, 195709030000, 1.931
lines = rows * columns
mycol = 0
for (i in 1:lines) {
if (mylat==myframe[bands+2,i] & mylong==myframe[bands+1,i]) {mycol = i+1}
}
# notice mycol = i+1 in order to get values in column to the right
myvector <- as.numeric(myframe[,mycol])
sink("output.txt")
cat("lat:",myframe[bands+2,mycol],"long:",myframe[bands+1,mycol],"\n")
for (i in 1:bands) { cat(myvector[i],"\n") }
sink()
The wavedata.grib file has grided SWH values, in the period 1957-09-01 to 2002-08-31. Each band refers to a pair of lat/long and has a series of 16346 SWH values at 00h of each day (1 band = 16346 values at a certain lat/long).
myframe has dimensions 16438 x 91. Notice 91 = 7rows x 13columns. And the number 16438 is almost equal to number of bands. The additional 2 rows/bands are long and lat values, all other columns should be wave heights corresponding to the 16436 bands.
The problem is I want to extract SWH (wave heights) at lat/long = 43.75,331.25, but they don't match the values I get reading the file with Degrib utility at this same lat/long.
Also, the correct values I want (2.147, 2.159, 1.931, ...) are in column 4 and not column 3 of myframe, even though myframe[16438,3]=43.75 (lat) and myframe[16437,3]=331.25 (long). Why is this? I would like to know to which lat/long do myframe[i,j] values actually correspond to or if there is some data import error in the process. I'm assuming Degrib has no errors.
Is there any R routine to easily interpolate values in a matrix if I want to extract values between grid points? More generally, I need help in writing an effective R function to extract wave heights like this:
SWH <- function (latitude, longitude, date/time)
Please help.

Related

Reading a grib2 dataset without loss of metata with R

My question follows this previous question, that has been partially answered : Reading a grib2 dataset with 4 dimensions and 2 variables with R
I am trying to read a GRIB2 file with R. This file is a probabilistic meteorological forecast : 10 variables, 1 lead time, 17 longitudes, 23 latitudes, and 51 members.
I can extract that thanks to the terra package and this script :
require(terra)
require(dplyr)
require(data.table)
require(stats)
destfile <- "C:/Users/XXX/Documents/example_grib_file_3"
##Downloading file
grib_data <- terra::rast(destfile)
print(grib_data)
## Convert to data frame
df <- as.data.frame(grib_data, xy=TRUE)
## Colnames is a combination of members (50) X time (57) X variables (2)
colNames <- paste(names(grib_data), as.character(time(grib_data)), sep="_")
colnames(df) <- c("lon", "lat", colNames)
df2 <- data.table::melt(as.data.table(df), c("lon", "lat"))
## Split variable and time
df2$time_UTC <- sub(".*_", "", df2$variable)
df2$variable <- sub("_.*", "", df2$variable)
## Add members
df2 <- df2 %>% group_by(lon, lat, variable, time_UTC) %>% mutate(member=(1:length(value)))
##Convert to array
df_array <- stats::xtabs(value~lon+lat+variable+member+time_UTC, df2, drop=F)
The BIG problem is that I can't retrieve the metadata concerning the perturbation number (member). For now, variables and members are mixed, giving 500 columns, with the variable name repeated 50 times. The member is not always at the same position along the other dimension (variable). For example, one particular member is in position 6 for temperature data, and position 50 for precipitation data.
So the line "Add member" in my code is totally irrelevant and needs the "perturbation number" to arrange the array. If I use eccodes, there is a field called "perturbationNumber", how can I retrieve it from terra and R ?
The example file : https://drive.google.com/file/d/1kUfTdtAdNJpugMpPhdQ6tY4QZaZlUQcA/view?usp=sharing
The example file in grib2 : https://drive.google.com/file/d/1Zsf8uajm5AOXTiCus6PyANZAsJ5cKsc1/view?usp=sharing
These are the different parameters retrieved using eccodes with Python :
grib_ls -l 48.5,3.5,1 -p paramId,name,typeOfLevel,level,dataDate,stepRange,dataType,shortName,step,perturbationNumber PATH_TO_FILE
paramId name typeOfLevel level dataDate stepRange dataType shortName step perturbationNumber value
167 2 metre temperature surface 0 20230124 0 pf 2t 0 35 273.517
167 2 metre temperature surface 0 20230124 0 pf 2t 0 3 273.589
167 2 metre temperature surface 0 20230124 0 pf 2t 0 50 273.229
etc...
Thanks for any help
With your example file
f <- "example_grib_file_3"
library(terra)
#terra 1.7.5
x <- rast(f)
This is the metadata that is available
names(x)[1]
#[1] "SFC (Ground or water surface); 2 metre temperature [C]"
time(x)[1]
#[1] "2023-01-24 UTC"
units(x)[1]
#[1] "C"
And with version 1.7-5 (currently the development version) you can also get raw metadata
metadata(x)[[1]]
# [,1] [,2]
#[1,] "GRIB_COMMENT" "2 metre temperature [C]"
#[2,] "GRIB_ELEMENT" "2T"
#[3,] "GRIB_FORECAST_SECONDS" "0"
#[4,] "GRIB_REF_TIME" "1674518400"
#[5,] "GRIB_SHORT_NAME" "0-SFC"
#[6,] "GRIB_UNIT" "[C]"
#[7,] "GRIB_VALID_TIME" "1674518400"
I do not know if there are ways to get other metdata from this file but in principle that is possible: see your previous question.

Calculating measure of spatial segregation?

There is five polygons for five different cities (see attached file in the link, it's called bound.shp). I also have a point file "points.csv" with longitude and latitude where for each point I know the proportion of people belonging to group m and group h.
I am trying to calculate the spatial segregation proposed by Reardon and O’Sullivan, “Measures of Spatial Segregation”
There is a package called "seg" which should allow us to do it. I am trying to do it but so far no success.
Here is the link to the example file: LINK. After downloading the "example". This is what I do:
setwd("~/example")
library(seg)
library(sf)
bound <- st_read("bound.shp")
points <- st_read("points.csv", options=c("X_POSSIBLE_NAMES=x","Y_POSSIBLE_NAMES=y"))
#I apply the following formula
seg::spseg(bound, points[ ,c(group_m, group_h)] , smoothing = "kernel", sigma = bandwidth)
Error: 'x' must be a numeric matrix with two columns
Can someone help me solve this issue? Or is there an alternate method which I can use?
Thanks a lot.
I don't know what exactly spseg function does but when evaluating the spseg function in the seg package documentation;
First argument x should be dataframe or object of class Spatial.
Second argument data should be matrix or dataframe.
After evaluating the Examples for spseg function, it should have been noted that the data should have the same number of rows as the id number of the Spatial object. In your sample, the id is the cities that have different polygons.
First, let's examine the bound data;
setwd("~/example")
library(seg)
library(sf)
#For the fortify function
library(ggplot2)
bound <- st_read("bound.shp")
bound <- as_Spatial(bound)
class(bound)
"SpatialPolygonsDataFrame"
attr(,"package")
"sp"
tail(fortify(bound))
Regions defined for each Polygons
long lat order hole piece id group
5379 83.99410 27.17326 972 FALSE 1 5 5.1
5380 83.99583 27.17339 973 FALSE 1 5 5.1
5381 83.99705 27.17430 974 FALSE 1 5 5.1
5382 83.99792 27.17552 975 FALSE 1 5 5.1
5383 83.99810 27.17690 976 FALSE 1 5 5.1
5384 83.99812 27.17700 977 FALSE 1 5 5.1
So you have 5 id's in your SpatialPolygonsDataFrame. Now, let's read the point.csv with read.csv function since the data is required to be in matrix format for the spseg function.
points <- read.csv("c://Users/cemozen/Downloads/example/points.csv")
tail(points)
group_m group_h x y
950 4.95 78.49000 84.32887 26.81203
951 5.30 86.22167 84.27448 26.76932
952 8.68 77.85333 84.33353 26.80942
953 7.75 82.34000 84.35270 26.82850
954 7.75 82.34000 84.35270 26.82850
955 7.75 82.34000 84.35270 26.82850
In the documentation and the example within, it has been strictly stated that; the row number of the points which have two attributes (group_m and group_h in our data), should be equal to the id number (which is the cities). Maybe, you should calculate a value by using the mean for each polygon or any other statistics for each city in your data to be able to get only one value for each polygon.
On the other hand, I just would like to show that the function is working properly after feeding with a matrix that has 5 rows and 2 groups.
sample_spseg <- spseg(bound, as.matrix(points[1:5,c("group_m", "group_h")]))
print(sample_spseg)
Reardon and O'Sullivan's spatial segregation measures
Dissimilarity (D) : 0.0209283
Relative diversity (R): -0.008781
Information theory (H): -0.0066197
Exposure/Isolation (P):
group_m group_h
group_m 0.07577679 0.9242232
group_h 0.07516285 0.9248372
--
The exposure/isolation matrix should be read horizontally.
Read 'help(spseg)' for more details.
first: I do not have experience with the seg-package and it's function.
What I read from your question, is that you want to perform the spseg-function, om the points within each area?
If so, here is a possible apprach:
library(sf)
library(tidyverse)
library(seg)
library(mapview) # for quick viewing only
# read polygons, make valif to avoid probp;ems later on
areas <- st_read("./temp/example/bound.shp") %>%
sf::st_make_valid()
# read points and convert to sf object
points <- read.csv("./temp/example/points.csv") %>%
sf::st_as_sf(coords = c("x", "y"), crs = 4326) %>%
#spatial join city (use st_intersection())
sf::st_join(areas)
# what do we have so far??
mapview::mapview(points, zcol = "city")
# get the coordinates back into a data.frame
mydata <- cbind(points, st_coordinates(points))
# drop the geometry, we do not need it anymore
st_geometry(mydata) <- NULL
# looks like...
head(mydata)
# group_m group_h city X Y
# 1 8.02 84.51 2 84.02780 27.31180
# 2 8.02 84.51 2 84.02780 27.31180
# 3 8.02 84.51 2 84.02780 27.31180
# 4 5.01 84.96 2 84.04308 27.27651
# 5 5.01 84.96 2 84.04622 27.27152
# 6 5.01 84.96 2 84.04622 27.27152
# Split to a list by city
L <- split(mydata, mydata$city)
# loop over list and perform sppseg function
final <- lapply(L, function(i) spseg(x = i[, 4:5], data = i[, 1:2]))
# test for the first city
final[[1]]
# Reardon and O'Sullivan's spatial segregation measures
#
# Dissimilarity (D) : 0.0063
# Relative diversity (R): -0.0088
# Information theory (H): -0.0067
# Exposure/Isolation (P):
# group_m group_h
# group_m 0.1160976 0.8839024
# group_h 0.1157357 0.8842643
# --
# The exposure/isolation matrix should be read horizontally.
# Read 'help(spseg)' for more details.
spplot(final[[1]], main = "Equal")

Correct variable values in a dataframe applying a function using variable-specific values in another dataframe in R

I have a df called 'covs' with sites on rows and in columns, 9 different environmental variables for each of these sites. I need to recalculate the value of each cell using the function x - center_values(x)) / scale_values(x). However, 'center_values' and 'scale_values' are different for each environmental covariate, and they are located in another df called 'correction'.
I have found many solutions for applying a function for a whole df, but not for applying specific values according to the id of the value to transform.
covs <- read.table(text = "X elev builtup river grip pa npp treecov
384879-2009 1 24.379101 25188.572 1241.8348 1431.1082 5.705152e+03 16536.664 60.23175
385822-2009 2 29.533478 32821.770 2748.9053 1361.7772 2.358533e+03 15773.115 62.38455
385823-2009 3 30.097059 28358.244 2525.7627 1073.8772 4.340906e+03 14899.451 46.03269
386765-2009 4 33.877861 40557.891 927.4295 1049.4838 4.580944e+03 15362.518 53.08151
386766-2009 5 38.605156 36182.801 1479.6178 1056.2130 2.517869e+03 13389.958 35.71379",
header= TRUE)
correction <- read.table(text = "var_name center_values scale_values
1 X 196.5 113.304898393671
2 elev 200.217889868483 307.718211316278
3 builtup 31624.4888660664 23553.2438790344
4 river 1390.41023742909 1549.88661649406
5 grip 5972.67361738244 6996.57793554527
6 pa 2731.33431010861 4504.71055521749
7 npp 10205.2997576655 2913.19658598938
8 treecov 47.9080656134352 17.7101565911347
9 nonveg 7.96755640452006 4.56625351682905", header= TRUE)
Could someone help me write a code to recalculate the environmental covariate values in 'covs' using the specific covariate values reported in 'correction'? E.g. For each value in the column 'elev' of the df 'covs', I need to substract the 'center_value' reported for 'elev' in the 'corrected' df, and then divided by the 'scale_value' of 'elev' reported in 'corrected' df. Thank you for your kind help.
You may assign var_name to row names, then loop over the names of covs to do the calculations in an sapply.
rownames(correction) <- correction$var_name
res <- as.data.frame(sapply(names(covs), function(x, y)
(covs[, x] - correction[x, "center_values"])/correction[x, "scale_values"]))
res
# X elev builtup river grip pa npp treecov
# 1 -1.725433 -0.5714280 -0.27324970 -0.09586213 -0.6491124 0.66015733 2.173339 0.6958541
# 2 -1.716607 -0.5546776 0.05083296 0.87651254 -0.6590217 -0.08275811 1.911239 0.8174114
# 3 -1.707781 -0.5528462 -0.13867495 0.73253905 -0.7001703 0.35730857 1.611340 -0.1058927
# 4 -1.698956 -0.5405596 0.37928543 -0.29871910 -0.7036568 0.41059457 1.770295 0.2921174
# 5 -1.690130 -0.5251972 0.19353224 0.05755748 -0.7026950 -0.04738713 1.093183 -0.6885470
Check e.g. "elev":
(covs[,"elev"] - correction["elev", "center_values"]) / correction["elev", "scale_values"]
# [1] -0.5714280 -0.5546776 -0.5528462 -0.5405596 -0.5251972

Association between two coordinates in R

I have a problem with spatial data.
I need to extract temperature data from a NetCDF file; then I need to associate this temperature at given latitude and longitude to another set of latitude and longitude contained in a different dataframe.
This is the code I used to extract my variables:
myfile <- nc_open(paste(wd, 'myfile.nc', sep=''))
timearr = ncvar_get(myfile, "time")
temp <- ncvar_get(myfile, 'temp_srf')
lat <- ncvar_get(myfile, 'lat_rho')
lon <- ncvar_get(myfile, 'lon_rho')
dim(temp)
[1] 27 75 52 # which means: 27 longitude * 75 latitudes * 52 time steps
I chose to work on the first time step of temperature for now. So:
> t1 <- as.vector(temp[,,1])
Then I created a data.frame including lat, lon and temperature in the first time step:
lat1 <- as.vector(lat)
lon1 <- as.vector(lon)
df1 <- as.data.frame(cbind(lon1, lat1, t1))
head(df1)
lon1 lat1 t1
1 18.15338 40.48656 13.96225
2 18.24083 40.55126 14.36726
3 18.32845 40.61589 14.53822
4 18.41627 40.68045 14.78643
5 18.50427 40.74495 14.88624
6 18.59246 40.80938 14.95925
In another data frame (df2) I have some random points of latitude and longitude, that I have to associate to the closest latitude and longitude of the previous data.frame:
> df2 <- read.csv(paste(id, "myfile.csv", sep=""), header=TRUE, sep=",")
> head(df2)
LONs LATs
1 14.13189 43.41072
2 14.13342 43.34871
3 14.09980 43.40822
4 14.05338 43.72771
5 13.91311 43.88051
6 13.98500 43.91164
I was thinking to get the distance between each point and get the lowest one, but I don't know how to do it. Not sure if there are other solutions.
I am assuming your data are projected coordinates, and that you need to calculate great circle distances. You can use a formula yourself (see my answer here), or you can use rdist.earth from the package fields. For each entry in df2, calculate the distance from all entries in df1, find the index of the minimum distance in that vector, and use that index to select the appropriate row df1 to assign temp to df2. It only takes one line (but it might be clearer to seperate the steps over a few commands):
require( fields )
df2["Temp"] <- df1[ sapply( seq_len( nrow(df2) ) , function(x){ which.min( rdist.earth( df2[x,] , as.matrix( df1[ c("lon1" , "lat1") ] ) , miles = FALSE, R = 6371 ) ) } ) , "t1" ]
And the results using your data:
df1
# lon1 lat1 t1
# 1 18.15338 40.48656 13.96225
# 2 18.24083 40.55126 14.36726
# 3 18.32845 40.61589 14.53822
# 4 18.41627 40.68045 14.78643
# 5 18.50427 40.74495 14.88624
# 6 18.59246 40.80938 14.95925
df2
# LONs LATs Temp
# 1 14.13189 43.41072 13.96225
# 2 14.13342 43.34871 13.96225
# 3 14.09980 43.40822 13.96225
# 4 14.05338 43.72771 14.53822
# 5 13.91311 43.88051 14.53822
# 6 13.98500 43.91164 14.78643
It looks like your distances are at least a Km apart (>300km in this data) so you should get good accuracy with the Great Circle formula. If they are smaller than 1km you may want to use the Haversine formula.
Two formulas I like for getting the distance between two lat/long coordinates are the Haversine formula and Vincenty's formula. The Haversine formula is a simpler formula that assumes Earth is a perfect sphere. You will probably get accuracy to a few feet. If you need a higher level of accuracy, try Vincenty's formula. It's spheroid based which attempts to account for Earth's imperfect sphere shape. The samples on the links aren't in R but it shouldn't be difficult to rewrite them in R.

Peak detection in Manhattan plot

The attached plot (Manhattan plot) contains on the x axis chromosome positions from the genome and on the Y axis -log(p), where p is a p-value associated with the points (variants) from that specific position.
I have used the following R code to generate it (from the gap package) :
require(gap)
affy <-c(40220, 41400, 33801, 32334, 32056, 31470, 25835, 27457, 22864, 28501, 26273,
24954, 19188, 15721, 14356, 15309, 11281, 14881, 6399, 12400, 7125, 6207)
CM <- cumsum(affy)
n.markers <- sum(affy)
n.chr <- length(affy)
test <- data.frame(chr=rep(1:n.chr,affy),pos=1:n.markers,p=runif(n.markers))
oldpar <- par()
par(cex=0.6)
colors <- c("red","blue","green","cyan","yellow","gray","magenta","red","blue","green", "cyan","yellow","gray","magenta","red","blue","green","cyan","yellow","gray","magenta","red")
mhtplot(test,control=mht.control(colors=colors),pch=19,bg=colors)
> head(test)
chr pos p
1 1 1 0.79296584
2 1 2 0.96675136
3 1 3 0.43870076
4 1 4 0.79825513
5 1 5 0.87554143
6 1 6 0.01207523
I am interested in getting the coordinates of the peaks of the plot above a certain threshold (-log(p)) .
If you want the indices of the values above the 99th percentile:
# Add new column with log values
test = transform(test, log_p = -log10(test[["p"]]))
# Get the 99th percentile
pct99 = quantile(test[["log_p"]], 0.99)
...and get the values from the original data test:
peaks = test[test[["log_p"]] > pct99,]
> head(peaks)
chr pos p log_p
5 1 5 0.002798126 2.553133
135 1 135 0.003077302 2.511830
211 1 211 0.003174833 2.498279
586 1 586 0.005766859 2.239061
598 1 598 0.008864987 2.052322
790 1 790 0.001284629 2.891222
You can use this with any threshold. Note that I have not calculated the first derivative, see this question for some pointers:
How to calculate first derivative of time series
after calculating the first derivative, you can find the peaks by looking at points in the timeseries where the first derivative is (almost) zero. After identifying these peaks, you can check which ones are above the threshold.
Based on my experience after plotting the graph you can use following R code to find the peak coordinate
plot(x[,1], x[,2])
identify(x[,1], x[,2], labels=row.names(x))
note here x[,1] refers to x coordinate(genome coordinate and x[,2] would be #your -log10P value
at this time use point you mouse to select a point and hit enter which #will give you peak location and then type the following code to get the #coordinate
coords <- locator(type="l")
coords

Resources