I am trying to convert a large number (>500) of text files into shapefiles. I can successfully convert a single .csv into a projected shapefile. And I can get lapply and 'for' loops to work when just loading, cleaning up, and exporting the text files. But the code fails when I add in steps to convert to shapefiles within the loops. Below are two ways I've tried tackling the problem and the associated error messages:
General processing/definitions-
library(rgdal)
library(sp)
crs.geo<-CRS("+proj=utm +zone=14 +ellps=GRS80 +datum=NAD83 +units=m +no_defs ") #define projection
coltype=c("character","character","character","numeric","numeric") #define data types for input .csv (x,y UTM coords are columns 4,5)
setwd("C:/.../testdata/out")
all.the.filenames <- list.files(pattern= "\\.csv") #create list of files to batch process
head(exampledata,2)
Point Location Time easting northing
1 Trackpoint 14 S 661117 3762441 12/1/2008 5:57:02 AM 661117 3762441
2 Trackpoint 14 S 661182 3762229 12/1/2008 5:58:02 AM 661182 3762229
Batch conversion with a 'for' loop
names <- substr(all.the.filenames, 1, nchar(all.the.filenames)-4) #clean up file names
for(i in names) {
filepath <- file.path("../out",paste(i,".csv",sep=""))
assign(i, read.table(filepath, colClasses=coltype, header=TRUE, sep=",", na.strings=c("NA","")))
coordinates(i) <- c(4,5) #coords in columns 4,5
proj4string(i) <- crs.geo
writeOGR(i,"C:/Users/Seth/Documents/testdata/out","*",driver="ESRI Shapefile") }
R returns this error message:
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘coordinates<-’ for signature ‘"character"’
If I end the 'for' loop after the 'assign' line, it successfully imports all .csv files as separate objects in R. The problem seems to be that function 'coordinates' is not seeing the coords as numeric, and I get the same error message no matter how explicitly I try to define them as such (e.g., coordinates(i) <- c(as.numeric("easting","northing")) Also, these lines of code work successfully when applied to a single .csv file, the problem is when I subset within a loop.
Batch conversion using lapply
files.to.process <- lapply(all.the.filenames, function(x) read.csv(x, colClasses=coltype, header=TRUE))
lapply(files.to.process, function(c) coordinates(c)<-c("easting","northing"))
[[1]]
[1] "easting" "northing"
[[2]]
[1] "easting" "northing"
[[3]]
[1] "easting" "northing"
[[4]]
[1] "easting" "northing"
[[5]]
[1] "easting" "northing"
lapply(files.to.process, function(p) proj4string(p) <- crs.geo)
which returns the error message:
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘proj4string<-’ for signature ‘"data.frame", "CRS"’
#Double-check if function 'coordinates' worked
class(files.to.process) == "SpatialPoints"
[1] FALSE
Conclusion/problem
With both approaches the problem seems to be in the 'coordinates' step to make a spatial object. What am I doing wrong in the loops? Thanks much for any help!
Seth H.
In your first attempt, the object i inside the loop is a character object. So,
coordinates(get(i))
would work better; I don't have a batch of csv files to test it on.
In the second attempt using lapply(), I'm not exactly sure what's going on, but
class(files.to.process)
should be "list", so what you want to do is
lapply(files.to.process,class)
and that will tell you if the objects are of class spatialpoints. I'm guessing they are data.frames, and you need one more step in between.
Related
I have 14 raster files from each year of land use. I would like to crop these rasters using my shapefile area of interest. With these cut rasters I would like to create a list where each element represents each year cut by the shapefile.
I tried in a very simple and fast way using the for, crop and mask commands, but without success.
my data exemplo:
raster and shape files : https://drive.google.com/file/d/1IOGWZ3_ckKj3UoZVd7ikR5l9n04xd6He/view?usp=sharing
my code
library(raster)
library(rgdal)
library(sf)
setwd('dir')
raster_solo_2005<-raster('utm-50-2005.tif')
raster_solo_2006<-raster('utm-50-2006.tif')
raster_solo_2007<-raster('utm-50-2007.tif')
raster_solo_2008<-raster('utm-50-2008.tif')
raster_solo_2009<-raster('utm-50-2009.tif')
raster_solo_2010<-raster('utm-50-2010.tif')
raster_solo_2011<-raster('utm-50-2011.tif')
raster_solo_2012<-raster('utm-50-2012.tif')
raster_solo_2013<-raster('utm-50-2013.tif')
raster_solo_2014<-raster('utm-50-2014.tif')
raster_solo_2015<-raster('utm-50-2015.tif')
raster_solo_2016<-raster('utm-50-2016.tif')
raster_solo_2017<-raster('utm-50-2017.tif')
raster_solo_2018<-raster('utm-50-2018.tif')
#list raster
list_raster<-as.list(raster_solo_2005, raster_solo_2006, raster_solo_2007, raster_solo_2008, raster_solo_2009, raster_solo_2010, raster_solo_2011, raster_solo_2012, raster_solo_2013, raster_solo_2014, raster_solo_2015, raster_solo_2016, raster_solo_2017, raster_solo_2018)
#shapefile
shp_area<-sf::st_read('500m_utm.shp')
#creat list empity
list_raster_cut<-list()
#loop operation
for(i in 1:10){
# crop e mask
list_raster_cut[[i]] <- list_raster %>%
raster::crop(shp_area)%>%
raster::mask(shp_area)
}
Update: exit error
Error in h(simpleError(msg, call)) : error evaluating argument 'x' when selecting method for function 'mask': 'unable to find an inherited method for function 'crop' for signature '"list" , "sf"''
The problem is that you need to also subset the list_raster object inside the for loop.
list_raster_cut[[i]] <- list_raster[[i]] %>%
raster::crop(shp_area)%>%
raster::mask(shp_area)
I am trying to calculate the centroids of a set of polygons.
My dataset, geodata, contains five columns including one geometry column of class sfc_GEOMETRY, with 45759 rows.
When I run sf::st_centroid(geodata), I get the following message
Error in CPL_geos_op("centroid", x, numeric(0), integer(0), numeric(0), : Evaluation error: IllegalArgumentException: Points of LinearRing do not form a closed linestring.
In addition: Warning messages:
1: In st_centroid.sf(geodata) :
st_centroid assumes attributes are constant over geometries of x
2: In st_centroid.sfc(st_geometry(x), of_largest_polygon = of_largest_polygon) :
st_centroid does not give correct centroids for longitude/latitude data
Should I run a loop to detect which geometry is not closed?
Is this a problem with the class of my geometry? Should it be sfc_MULTIPOLYGON?
Possible solution:
I was encountering this problem when reading in a list of files through a loop. The loop would read in the files and then rbind them together into geodata, and then calculate the centroid:
for(i in 1:length(list)){
file <- st_read(list[i])
geodata <- rbind(geodata, file) #geodata is here a void sf object
}
geocent <- st_centroid(geodata)
When I calculated the centroids within the loop (for each file in the list), the error disappeared.
for(i in 1:length(list)){
file <- st_read(list[i])
file <- st_centroid(file)
geocent <- rbind(geodata, file) #geodata is here a void sf object
}
Hence, I think the problem lay in the binding operation.
Perhaps I had not defined my void sf object in the right manner.
Perhaps rbind was not the appropriate function, or I should have specified its parameters.
There's no need to run a loop to find which geometry is bad. The st_is_valid() function should tell you which row(s) have the problem.
It looks like one of your geometries might be made up of an incorrect number of points.
More info about finding and fixing the problem at r-spatial: https://www.r-spatial.org/r/2017/03/19/invalid.html
I am trying to extract summed raster cell values from a single big file for various SpatialPolygonsDataFrames (SPDF) objects in R stored in a list, then add the extracted values to the SPDF objects attribute tables. I would like to iterate this process, and have no idea how to do so. I have found an efficient solution for multiple polygons stored in a single SPDF object (see: https://gis.stackexchange.com/questions/130522/increasing-speed-of-crop-mask-extract-raster-by-many-polygons-in-r), but do not know how to apply the crop>mask>extract procedure to a LIST of SPDF objects, each containing multiple polygons. Here is a reproducible example:
library(maptools) ## For wrld_simpl
library(raster)
## Example SpatialPolygonsDataFrame
data(wrld_simpl) #polygon of world countries
bound <- wrld_simpl[1:25,] #country subset 1
bound2 <- wrld_simpl[26:36,] #subset 2
## Example RasterLayer
c <- raster(nrow=2e3, ncol=2e3, crs=proj4string(wrld_simpl), xmn=-180,
xmx=180, ymn=-90, ymx=90)
c[] <- 1:length(c)
#plot, so you can see it
plot(c)
plot(bound, add=TRUE)
plot(bound2, add=TRUE, col=3)
#make list of two SPDF objects
boundl<-list()
boundl[[1]]<-bound1
boundl[[2]]<-bound2
#confirm creation of SPDF list
boundl
The following is what I would like to run for the entire list, in a forloop format. For a single SPDF from the list, the following series of functions seem to work:
clip1 <- crop(c, extent(boundl[[1]])) #crops the raster to the extent of the polygon, I do this first because it speeds the mask up
clip2 <- mask(clip1, boundl[[1]]) #crops the raster to the polygon boundary
extract_clip <- extract(clip2, boundl[[1]], fun=sum)
#add column + extracted raster values to polygon dataframe
boundl[[1]]#data["newcolumn"] = extract_clip
But when I try to isolate the first function for the SPDF list (raster::crop), it does not return a raster object:
crop1 <- crop(c, extent(boundl[[1]])) #correctly returns object class 'RasterLayer'
cropl <- lapply(boundl, crop, c, extent(boundl)) #incorrectly returns objects of class 'SpatialPolygonsDataFrame'
When I try to isolate the mask function for the SPDF list (raster::mask), it returns an error:
maskl <- lapply(boundl, mask, c)
#Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘mask’ for signature ‘"SpatialPolygonsDataFrame", "RasterLayer"’
I would like to correct these errors, and efficiently iterate the entire procedure within a single loop (i.e., crop>mask>extract>add extracted values to SPDF attribute tables. I am really new to R and don't know where to go from here. Please help!
One approach is to take what is working and simply put the desired "crop -> mask -> extract -> add" into a for loop:
for(i in seq_along(boundl)) {
clip1 <- crop(c, extent(boundl[[i]]))
clip2 <- mask(clip1, boundl[[i]])
extract_clip <- extract(clip2, boundl[[i]], fun=sum)
boundl[[i]]#data["newcolumn"] <- extract_clip
}
One can speed-up the loop with parallel execution, e.g., with the R package foreach. Conversely, the speed gain of using lapply() instead of the for loop will be small.
Why the error occurs:
cropl <- lapply(boundl, crop, c, extent(boundl))
applies the function crop() to each element of the list boundl. The performed operation is
tmp <- crop(boundl[[1]], c)
## test if equal to first element
all.equal(cropl[[1]], tmp)
[1] TRUE
To get the desired result use
cropl <- lapply(boundl, function(x, c) crop(c, extent(x)), c=c)
## test if the first element is as expected
all.equal(cropl[[1]], crop(c, extent(boundl[[1]])))
[1] TRUE
Note:
Using c to denote an R object is a bade choice, because it can be easily confused with c().
I am using John Baumgrtner's gdal_polygonizeR (https://johnbaumgartner.wordpress.com/2012/07/26/getting-rasters-into-shape-from-r/) to covert rasters to polygons in R. Aside - I tried raster pkg rasterToPolygons function and it took forever; gdal_polygonizeR is way faster. Anyways, I have a list of 533 raster files (different extents) that I want to convert to polygons. The gdal_polygonizeR function works when a single list element is called, but I have tried to use it on all list elements using lapply and get an error message. See code below:
#path to folder containing all .tif raster files
dir <- "/path/to/raster/files"
#create a list of the files in the folder
files <- list.files(path = dir, pattern = ".tif$")
#use lapply to import/create list of all files in folder
rasterl_50 <- lapply(paste0(dir, files), raster)
#test gdal_polygonizeR function on single list element
gdal_polygonizeR(rasterl_50[[1]]) #works properly
#loop thru all elements in list
lapply(rasterl_50, gdal_polygonizeR)
Output = the first six (6) elements seem to run OK, but I get the following error msg at [[7]]:
wfp1 <- gdal_polygonizeR(rasterl_50[[1]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d4dc99d8d.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp2 <- gdal_polygonizeR(rasterl_50[[2]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d7698a853.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp3 <- gdal_polygonizeR(rasterl_50[[3]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d30d4d703.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp4 <- gdal_polygonizeR(rasterl_50[[4]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d24036d07.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp5 <- gdal_polygonizeR(rasterl_50[[5]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d4683ed87.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp6 <- gdal_polygonizeR(rasterl_50[[6]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d4e23b4d1.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
wfp7 <- gdal_polygonizeR(rasterl_50[[7]])
Creating output /var/folders/s9/pm92gdl94h18k4n6026cb8x00000gn/T//RtmpvRRvA4/file23d6791d108.shp of format ESRI Shapefile.
0...10...20...30...40...50...60...70...80...90...100 - done.
Error in readOGR(dirname(outshape), layer = basename(outshape), verbose = !quiet) :
no features found
In addition: Warning message:
In ogrFIDs(dsn = dsn, layer = layer) :
Show Traceback
Rerun with Debug
Error in readOGR(dirname(outshape), layer = basename(outshape), verbose = !quiet) :
no features found
#
If anyone has ideas for a solution using lapply or a for loop etc., please reply. Thanks
Solution: I had to run gdal_polygonizeR on each individual list element, and found that several raster files in the list contained no values (this resulted from reclassify function applied to rasters prior). I removed these files from the list, and lapply worked. Here is the code:
#remove 'no value' elements from the list
new_rastlist <-
rasterlist[c(-7,-14,-36,-89,-191,-310,-432,-436,-476,-493,-494,-501)]
#then try again to use lapply
polyl <- lapply(rastlist, gdal_polygonizeR)
UPDATE:
Even better, remove rasters with all NAs first with this:
batch_reclass <- function(rastlist){
for (i in 1:length(wfrastlist)) {
#read in raster
r <-raster(paste0("/path/to/rasterfiles/", rastlist[i]))
#perform the reclassifcation
rc <- reclassify(r, rclmat)
#write each reclass to a new file
if (!is.na(minValue(rc))) {
writeRaster(rc, filename = paste0("/path/to/new/rasterfiles/", "rc_",
rastlist[i]), format="GTiff", overwrite=TRUE)
}}
}
#run the function
batch_reclass(rastlist)
I also posted this question on stack gis 1. From the netcdf4 data that have sub categories, I want to be able to read "Retrieval/fs" variable. I also want to read them and convert to raster girds, but it seems that raster doesn't support netcdf4. I appreciate any suggestions.
library(ncdf4)
library(raster)
file <- "http://140906_B7101Ar_150909171225s.nc4"
names(file$var)
"latitude" ... "longitude"... "Retrieval/fs"
lat <- raster(file, varname="latitude")
lon <- raster(file, varname="longitude")
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘raster’ for signature ‘"ncdf4"’
raster does work with ncdf4 You are now showing actual code. file is a character vector. You cannot do names(file$var) with that (at least you won't get "latitude" ... "longitude"... "Retrieval/fs". So file is probably ncdf4 object (see the error message), while the raster function expects a filename (but not a url).
If you download the file and then do
library(raster)
x <- brick(filename, var="Retrieval/fs")
Things should work if the ncdf file had regular raster data.
However, it does not so you cannot directly import this as a raster. Instead you can get the lat and lon and values from the files, treat these as points and then rasterize (interpolate) these to get a regular raster.
Here is the answer to the question I asked. Since the data is not gridded, I retrieve the lon and lat information along with the variables to create a dataframe.
fs <- ncvar_get(ncfile, "Retrieval/fs")
xlon <- ncvar_get(ncfile, "longitude")
xlat <- ncvar_get(ncfile, "latitude")
d <- data.frame( as.vector(xlon),as.vector(xlat), as.vector(fs))# create a dataframe
coordinates(d) <- c("xlon","xlat")
proj4string(d) <- CRS("+proj=longlat")
spoint <- SpatialPoints(coords = d) #create a spatial point object