NetCDF - converting into raster and projection issues - r

I have the following NetCDF file - I am trying to convert into raster but something is not right. The projection of the NetCDF file is not given but based on the software I received it from it should LatLong but might be cylindrical equal area. I tried both, but I keep getting this distortion which makes it impossible to query for the values at the right locations. I know the spacing of the grid is not even, not sure if that affects the end result (here visual from ArcGIS but in R it is the same problem unless plotted with levelplot function).
library(raster)
library(ncdf4)
library(lattice)
library(RColorBrewer)
setwd("D:/Results")
climexncdf <- nc_open("ResultsSO_month.nc")
lon <- ncvar_get(climexncdf,"Longitude")
nlon <- dim(lon)
head(lon)
lat <- ncvar_get(climexncdf,"Latitude")
nlat <- dim(lat)
head(lat)
dname <- "Weekly Growth Index"
t <- ncvar_get(climexncdf,"Step")
tmp_array <- ncvar_get(climexncdf,dname)
tmp_stack <- vector("list",length(t))
for (i in 1:length(t)) {
tmp_stack[[i]] <- tmp_array[,,i]
}
YearData <- vector("list",52)
for (i in 1:4) {
YearData[[i]] <- tmp_array[,,i]
}
Month1 <- YearData[c(1,2,3,4)]
# Calculate monthly averages
M1Avg <- Reduce("+",Month1)/length(Month1)
# Replace 0's with NA's
M1Avg[M1Avg==0] <- NA
# Piece of code that gives me what I need:
grid <- expand.grid(lon=lon, lat=lat)
cutpts <- seq(0,1,0.1)
# Convert to raster - work to include lat and long
M1Avg_reorder <- M1Avg[ ,order(lat) ]
M1Avg_reorder <- apply(t(M1Avg_reorder),2,rev)
M1AvgRaster <- raster(M1Avg_reorder,
xmn=min(lon),xmx=max(lon),
ymn=min(lat),ymx=max(lat),
crs=CRS("+proj=longlat +datum=WGS84"))
#crs=CRS("+proj=cea +lat_0=0 +lon_0=0"))
r <- projectRaster(M1AvgRaster,crs=CRS("+proj=longlat +datum=WSG84"))
plot(M1AvgRaster)
# Location file not included but any locations can be entered
locations <- read.csv("Locations.csv", header=T)
coordinates(locations) <- c("y","x")
data <- extract(M1AvgRaster,locations)
writeRaster(M1AvgRaster, "M1AvgRaster_Globe_projWGSTest", format = "GTiff")

the python version shows that after reordering at least the location of data seems correct. However, the data file seems strange, I saw data actually getting corrupted in the python netcdf library, which I've never seen before with quite a lot of different NetCDF files. Also, the chunking and compression settings are strange, better not to apply them at all.
But minimal python example to get the plot is here:
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
from netCDF4 import Dataset
ff = Dataset('ResultsSO_month.nc')
test_var = np.copy(ff.variables['Maximum Temperature'][:])
## reorder latitudes
latindex = np.argsort(ff.variables['Latitude'][:])
## Set up map and compute map coordinates
m = Basemap(projection='cea', llcrnrlat=-90, urcrnrlat=90,
llcrnrlon=-180, urcrnrlon=180, resolution='c')
grid_coords = np.meshgrid(ff.variables['Longitude'[:],ff.variables['Latitude'][latindex])
X,Y = m(grid_coords[0],grid_coords[1])
## Plot
m.pcolormesh(X,Y,test_var[0,latindex,:])
m.drawcoastlines()
plt.colorbar()
plt.show()

Related

Extract Raster Pixels Values Using Vector Polygons in R

I have been struggling with this for hours.
I have a shapefile (called "shp") containing 177 polygons i.e. 177 counties. This shapefile is overlaid on a raster. My raster (called "ras") is made of pixels having different pollution values.
Now I would like to extract all pixel values and their number of occurrences for each polygon.
This is exactly what the QGIS function "zonal histogram" is doing. But I would like to do the exact same thing in R.
I tried the extract() function and I managed to get a mean value per county, which is already a first step, but I would like to make a pixels distribution (histogram).
Could someone give me a hand ?
Many thanks,
Marie-Laure
Thanks a lot for your help. Next time I promise I will be careful and explain my issue more in details.
With your help I managed to find a solution.
I also used this website : http://zevross.com/blog/2015/03/30/map-and-analyze-raster-data-in-r/
For information, first I had to uninstall the "tidyr" package because there was a conflict with the extract function.
In case it can help someone, here is the final code :
# Libraries loading
library(raster)
library(rgdal)
library(sp)
# raster layer import
ras=raster("C:/*.tif")
# shapefile layer import
shp<-shapefile("C:/*.shp")
# Extract the values of the pixels raster per county
ext <- extract(ras, shp, method='simple')
# Function to tabulate pixel values by region & return a data frame
tabFunc <- function(indx, extracted, region, regname) {
dat <- as.data.frame(table(extracted[[indx]]))
dat$name <- region[[regname]][[indx]]
return(dat)
}
# run through each county & compute a table of the number
# of raster cells by pixel value. ("CODE" is the county code)
tabs <- lapply(seq(ext), tabFunc, ext, shp, "CODE")
# assemble into one data frame
df <- do.call(rbind, tabs)
# to see the data frame in R
print(df)
# table export
write.csv(df,"C:/*.csv", row.names = FALSE)
Here is a minimal, self-contained, reproducible example (almost literally from ?raster::extract, so not difficult to make)
library(raster)
r <- raster(ncol=36, nrow=18, vals=rep(1:9, 72))
cds1 <- rbind(c(-180,-20), c(-160,5), c(-60, 0), c(-160,-60), c(-180,-20))
cds2 <- rbind(c(80,0), c(100,60), c(120,0), c(120,-55), c(80,0))
polys <- spPolygons(cds1, cds2)
Now you can do
v <- extract(r, polys)
par(mfrow=c(1,2))
z <- lapply(v, hist)
Or more fancy
mains <- c("first", "second")
par(mfrow=c(1,2))
z <- lapply(1:length(v), function(i) hist(v[[i]], main=mains[i]))
Or do you want a barplot
z <- lapply(1:length(v), function(i) barplot(table(v[[i]]), main=mains[i]))

Line density function in R equivalent to Line density tool in ArcMap (arcpy)

I need to calculate the magnitude-per-unit area of polylines that fall within a radius around each cell. Essentially I need to calculate a km/km2 road density within a 500m pixel search radius. ArcMap has a quick and easy tool that handles this, but I need a pure R solution.
Here is a link on how line density works: http://desktop.arcgis.com/en/arcmap/10.3/tools/spatial-analyst-toolbox/how-line-density-works.htm
And this is how to use it in a python (arcpy) script: http://desktop.arcgis.com/en/arcmap/10.3/tools/spatial-analyst-toolbox/line-density.htm
I currently execute a backwards approach using raster::focal function, calculating a density of burned in road features. I then convert the km2/km2 output to km/km2.
#Import libraries
library(raster)
library(rgdal)
library(gdalUtils)
#Read-in an already created raster mask (cells are all set to 0)
mask <- raster("x://path to raster mask...")
#Make a copy of the mask to burn features in, keeping the original untouched
roads_mask <- file.copy(mask, "x://output path ...//roads.tif")
#Read-in road features (shapefile format)
roads_sldf <- readOGR("x://path to shapefile" , "roads")
#Rasterize spatial lines data frame ie. burn road features into mask
#Where road features get a value of 1, mask extent gets a value of 0
roads_raster <- gdalUtils::gdal_rasterize(src_datasource = roads_sldf,
dst_filename = "x://output path ...//roads.tif", b = 1,
burn = 1, l = "roads", output_Raster = TRUE)
#Run a 1km circular radius density function (be mindful of edge effects)
weight <- raster::focalWeight(roads_raster,1000,type = "circle")
1km_rdDensity <- raster::focal(roads_raster, weight, fun=sum, filename = '',
na.rm=TRUE, pad=TRUE, NAonly=FALSE, overwrite=TRUE)
#Convert km2/km2 road density to km/km2
#Set up the moving window
weight <- raster::focalWeight(roads_raster,1000,type = "circle")
#Count how many records in each column of the moving window are > 0
columnCount <- apply(weight,2,function(x) sum(x > 0))
#Get the sum of the column count
number_of_cells <- sum(columnCount)
#multiply km2/km2 density by number of cells in the moving window
step1 <- roads_raster * number_of_cells
#Rescale step1 output with respect to cell size(30m) and radius of a circle
final_rdDensity <- (step1*0.03)/3.14159265
#Write out final km/km2 road density raster
writeRaster(final_rdDensity,"X://path to output...", datatype = 'FLT4S', overwrite = TRUE)
After some more research I think I may be able to use a kernel function, however I don't want to apply the smoothing algorithm... As well the output is an 'im' object which I would need to write to as a 'tif'
#Import libraries
library(spatstat)
library(rgdal)
#Read-in road features (shapefile format)
roads_sldf <- readOGR("x://path to shapefile" , "roads")
#Convert roads spatial lines data frame to psp object
psp_roads <- as.psp(roads_sldf)
#Apply kernel density, however this is where I am unsure of the arguments
road_density <- spatstat::density.psp(psp_roads, sigma = 0.01, eps = 500)
Cheers.
See this question https://gis.stackexchange.com/questions/138861/calculating-road-density-in-r-using-kernel-density
Tried to mark as a duplicate but doesn't work because the other Q is on gis stack exchange
Short answer is use spatstat.geom::pixellate()
I also needed spatstat.geom::as.psp(sf::st_geometry(x)) to convert an sf lines object to the correct format and maptools::as.im.RasterLayer(r) to convert a raster. I was able to convert the result to RasterLayer with raster::raster(pix_res)
Perhaps you can use terra::rasterizeGeom which is available in the development version that you can install with install.packages('terra', repos='https://rspatial.r-universe.dev')
Example data
library(terra)
f <- system.file("ex/lux.shp", package="terra")
v <- vect(f) |> as.lines()
r <- rast(v, res=.1)
Solution
x <- rasterizeGeom(v, r, fun="length", "km")
And then use focal sum, but you would not have a perfect circle.
What you could do instead, if your dataset is not too large, is create a circle for each grid cell and use intersect. Something like this:
p <- xyFromCell(r, 1:ncell(r)) |> vect(crs="+proj=longlat")
p$id <- 1:ncell(r)
b <- buffer(p, 10000)
values(v) <- NULL
i <- intersect(v, b)
x <- aggregate(perim(i), list(id=i$id), sum)
r[x$id] <- x[,2]

Extract shapefiles from longitude/latitude gridded data

I have some gridded data of sea surface temperature values in the Mediterranean to which I've applied clustering. I have 420 files with three columns structure (long,lat,value). The data for a particular file looks like this map
Now I want to extract the cluster areas as shapefile for postprocessing. I have found this post (https://gis.stackexchange.com/a/187800/9227) and tried to use its code like this
# Packages
library(sp)
library(rgdal)
library(raster)
# Paths
ruta_datos<-"/home/meteo/PROJECTES/VERSUS/OUTPUT/DATA/CLUSTER_MED/"
setwd("~/PROJECTES/VERSUS/temp")
# File list
files <- list.files(path = ruta_datos, pattern = "SST-cluster-mitja-mensual")
for (i in 1:length(files)){
datos<-read.csv(paste0(ruta_datos,files[i],sep=""),header=TRUE)
nclusters<-max(datos$cluster)
for (j in 1:nclusters){
clust.dat<-subset(datos, cluster == j)
coordinates(clust.dat)=~longitud+latitud
proj4string(clust.dat)=CRS("+init=epsg:4326")
pts = spTransform(clust.dat,CRS("+init=epsg:4326"))
gridded(pts) = TRUE
r = raster(pts)
projection(r) = CRS("+init=epsg:4326")
# make all values the same. Either do
s <- r > -Inf
# convert to polygons
pp <- rasterToPolygons(s, dissolve=TRUE)
# save shapefile
shname<-paste("SST-shape-",substr(files[i],27,32),"-",j,sep="")
writeOGR(pp, dsn = '.', layer = shname, driver = "ESRI Shapefile")
}
}
But the code stops for with this error message
gridded(pts) = TRUE
suggested tolerance minimum: 1
Error in points2grid(points, tolerance, round) : dimension 2
: coordinate intervals are not constant
Warning message: In points2grid(points, tolerance, round) : grid has empty
column/rows in dimension 1
I don't understand that at a certain file it says that coordinate intervals are not constant while they indeed are, original SST data from which clustering was derived are on a regular grid over the whole globe. All cluster data files have the same size, 4248 points. A sample data file is available here
What does the tolerance suggestion means? I've been looking for a solution and found some suggestion to use SpatialPixelsDataFrame but couldn't find out how to apply.
Any help would be appreciated. Thanks.
I am not an expert of geospatial data but for me, if you filter on cluster, data are indeed not on a grid. So far as I understand, you start from a grid (convex set of regularly distant points).
I tried following modifications to your code and some files are generated but I can't test whether they are correct or not.
Principle is to build the grid on all data then only filter on cluster before calling raster.
This gives:
files <- list.files(path = ruta_datos, pattern = "SST-cluster-mitja-mensual")
for (i in 1:length(files)){
datos<-read.csv(paste0(ruta_datos,files[i],sep=""),header=TRUE)
nclusters<-max(datos$cluster)
for (j in 1:nclusters){
## clust.dat<-subset(datos, cluster == j)
clust.dat <- datos
coordinates(clust.dat)=~longitud+latitud
proj4string(clust.dat)=CRS("+init=epsg:4326")
pts = spTransform(clust.dat,CRS("+init=epsg:4326"))
gridded(pts) = TRUE
## r = raster(pts)
r= raster(pts[pts$cluster==j,])
projection(r) = CRS("+init=epsg:4326")
# make all values the same. Either do
s <- r > -Inf
# convert to polygons
pp <- rasterToPolygons(s, dissolve=TRUE)
# save shapefile
shname<-paste("SST-shape-",substr(files[i],27,32),"-",j,sep="")
writeOGR(pp, dsn = '.', layer = shname, driver = "ESRI Shapefile")
}
}
So, two lines in comment and update just the line below.

How to create SpatialPixelsDataFrame object in R (compatible with adehabitat package)

My problem is simple. I have found very good package called adehabitat in R. To use it I need to transform my data into specificaly structured object containing raster map data and coordinates of an animal. To see it please type:
# example data in adahabitat package
data(bauges)
bauges
str(bauges)
How do I convert my data (bellow) into such structure? I figured out how to convert $locs into SpatialPoints, but I don't know how to convert map (in my example are raster values categorical codes of individual types of habitat -i.e. not continuous variable).
# My example data:
library(raster)
library(adehabitatHS)
# map
habitat_type_temp <- matrix(c(1,1,1,1,1,1,1,1,2,2,
1,1,2,2,1,1,1,2,2,2,
1,2,2,2,3,3,3,2,2,2,
2,2,2,1,1,1,3,2,2,1,
2,2,1,1,1,1,3,2,1,1,
2,1,1,1,1,1,3,3,1,1,
2,1,1,1,1,3,3,3,3,1,
1,1,1,1,1,1,1,3,3,3), 10)
habitat_type <- t(habitat_type_temp)
# coordinates
animal_coords <- data.frame(x = c(2,4,5,5,6,9),
y = c(2,8,3,2,4,3))
# see the situation
plot(raster(habitat_type, xmn=1, xmx=10, ymn=1, ymx=8))
points(animal_coords$x, animal_coords$y)
# creating object which could be manipulated in adehabitat package
my.hab <- list()
my.hab$map <- SpatialPixelsDataFrame(...)
my.hab$locs <- SpatialPoints(animal_coords)
Is it even possible to insert such manually fabricated data into such specific type of object, or I need some original tiff with specific CRS?
You could just drop the location somewhere to produce the SpatialPixelsDataFrame, I think this is roughly Iowa:
x <- 93+rep(1:8,each=10)/100
y <- rep(seq(42.01,42.1,by=0.01), 8)
z <- c(1,1,1,1,1,1,1,1,2,2,
1,1,2,2,1,1,1,2,2,2,
1,2,2,2,3,3,3,2,2,2,
2,2,2,1,1,1,3,2,2,1,
2,2,1,1,1,1,3,2,1,1,
2,1,1,1,1,1,3,3,1,1,
2,1,1,1,1,3,3,3,3,1,
1,1,1,1,1,1,1,3,3,3)
xy.df <- data.frame(x,y)
xy.coords <- SpatialPixels(SpatialPoints(xy.df))
llCRS <- CRS("+proj=utm +zone=15 +ellps=WGS84")
xy.sp <- SpatialPoints(xy.coords, proj4string = llCRS)
xyz <- as.data.frame(cbind(x,y,z))
xyz.spdf <- SpatialPixelsDataFrame(xy.coords, xyz)
plot(xyz.spdf)
Your spatialpoints would have to be changed similarly.

Applying d3.js Density map of homicides example to own data fails

We tried to reproduce the beautiful example of bl.ocks.org/diegovalle/5166482, using d3.js and leaflet, but with our own data, which is on a regular lon-lat grid.
In R, we first retrieve the data from a mysql table, and write them to a shapefile:
lonmin <- -10; lonmax <- 10
latmin <- 30; latmax <- 50
dlon <- dlat <- 0.5
lon <- seq(from=lonmin, to=lonmax, by=dlon)
lat <- seq(from=latmin, to=latmax, by=dlat)
nlon <- length(lon); nlat <- length(lat)
# cl <- a mysql request
solRad <- matrix(cl$solRad, ncol=nlon, nrow=nlat)
# Plot the data
levels=seq(from=-40, to=1000, by=40)
filled.contour(solRad, x=lon, y=lat, levels=levels, col=col)
# Write a shapefile
require(maptools); require(rgdal)
writeOGR(ContourLines2SLDF(contourLines(lon, lat, solRad, levels=levels)),
"solRad.shp", "contours", "ESRI Shapefile")
You can look at the filled.contour output ![here] http://www.jonxion.ch/5166482/solRad.png. We then transform the shape file to a topojson file, which you can find by replacing the .png in the above link by .json.
Finally, we render it with D3.js and Leaflet, leading to this faulty result [here] http://www.jonxion.ch/5166482/
We browsed many tutorials and other examples without finding the cue to our problem. What are we doing wrong?
Might it be a d3.js limitation? We recognise that our data is more complex, but Diegovalle's data contains unclosed contours too (see its upper left corner). Would writing ContourPolygones instead of ContourLines solve our problem? Does such routines exist? Or, is there an alternative technique to d3.js? Thank's in advance for your help!

Resources