Is it possible to use R to crop out a subsection from a raster currently stored on a network drive without the need to download the file to the local drive first?
It is possible to read a raster from a network drive using the call:
img <- raster("Z:/path/file.bsq")
However, the raster I am working with exceeds the file size limit of my network drive. I would have thought that because raster() does not actually read the entire file into memory it should be possible to just read in the metadata and then use:
img_crop <- crop(x=img, y=extent)
To read in a cropped part of the image into the memory. Of course, this fails at the stage of reading the raster so I can't even try and crop it. I guess that crop has no reason to know in advance how large the cropped raster will be, which is part of the reason this doesn't work.
Is there a way of doing this or do I just need to increase the file size limit of my network drive?
Related
Summary:
I'm trying to calculate the area of a large number of polygons in R. I've read a few posts about how I might do this (Example #1 & Example #2) but the problem I'm having is that my shapefile is too large (1.7gb) to import. Given I can't import the file, I can't calculate the area of the polygons.
Extended Explanation:
I'm actually trying to calculate the area of properties in Victoria, Australia. The polygons represent these properties. I downloaded the simplified models 1 and 2 of VicMaps from Spatial Datamart for all of Victoria.
However, given the size of the shapefiles, I had to narrow my search to just one local government area (LGA) and calculated the polygon areas (just for testing). The shapefile was 15.5MB.
library(raster)
x <- shapefile("D:/Downloads/SDM616230/ll_gda94/shape/lga_polygon/ballarat/VMPROP/PROPERTY_PRIMARY_APPROVED.shp")
crs(x)
x$area_sqkm <- area(x) / 1000000
This worked but its not a practical solution to my problem given there's many LGAs in Victoria and I plan to eventually follow the same process for Queensland and NSW.
However, trying to load a larger shapefile doesn't work and results in the below error code "Error: memory exhausted (limit reached?)".
I've tried using readShapePoly, readogr, st_read and read_sf to get the large shapefile into R but they don't work. I think the file is just too large. I tried using a select query within read_sf in an effort to reduce the size of the file I was reading but that didn't work either. I've read online that I should seek to split the shapefile into just the data I need to reduce the size - but I have no idea how to do that.
Hope you can help.
Obviously the file is too big for a single box. I think the options then are either
1) split the files into smaller ones, process one by one. See
https://gis.stackexchange.com/questions/195508/split-a-shapefile-into-smaller-files-on-linux-command-line
2) use some dbms or data warehouse to do it, they do such batching automatically.
Our office does scanning of data entry forms, and we lack any proprietary software that is able to do automated double-entry (primary entry is done by hand, of course). We are hoping to provide a tool for researchers to highlight regions on forms and use scanned versions to determine what participant entry was.
To do this, all I need for a very rough attempt is a file to read in PDFs as raster files, with coordinates as X, Y components, and B&W white "intensities" as a Z-axis.
We use R mainly for statistical analysis and data management, so options in R would be great.
You could use the raster package from R. However, it doesnt support .pdf files, but .tif,.jpg,.png (among many others).
But coverting your pdfs into pngs shouldn't be a big problem: Look here for more information.
Once you have your png files ready, you can do the following:
png <- raster("your/png/file.png")
and then use the extract() function to get your brigthness value from the picture. I.e. let's say your png is 200x200px and you want to extract a pixel value from row 100 and column 150:
value <- extract(png, c(150,100))
I created some networks through R and the IGRAPH package in .png files. My goal would be to make a little video to show how the topology of the network changes through the time. Only that the resolution of the .png files is really low. Is there a way that would generate the same file but with an higher resolution? My colleague with his PC and using the same R script generated networks with an higher resolution.
Here is my code:
wd<-getwd()
setwd(wd)
library(Matrix)
library(igraph)
library(slam)
plotname<-sprintf("g_communities_t%03d.png",t)
png(filename=plotname, height=640, width=640)
plot(community.infomap[[t]],g1[[t]],layout=layout.fruchterman.reingold,vertex.label=NA,edge.arrow.size=1,edge.curved=TRUE,vertex.size=2+sqrt(vertex.weight[[t]]))
dev.off()
Thanks to all
Use png's res parameter to adjust nominal resolution. Note that is a good idea to adjust values vor height and width accordingly in order to preserve text size etc.:
resfactor = 3
png(filename=plotname, res = 72*resfactor, height=640*resfactor, width=640*resfactor)
? png gives you more details (also see ? dev.copy to copy the graphics contents of the current device to another).
I have a huge scatter plot matrix to generate and save into a zoom-able image. I takes a bunch of hours to draw and then I got some errors like:
"Server Error Unabe to establish connection with R session".
Any ideas? The problem is obviously memory, but there must be a way to get around this.
I've managed to save the file as a pdf format of 28.7 MB, it takes a lot of time to display and makes inkscape crash. I know that people who generate fractals are able to make images of infinite resolution without consuming a lot of memory since the image is generate as u zoom into it. Problem is fractals are self similar and scatterplots are not, so I'm not sure if there's a smart way to get around this issue.
A possible way to get around this "information overload" is to plot variables in pairs using qplot() and then save the file using ggsave(), for example in bmp on jpeg files.
Here, I found a landscape GIS layer is really attractive, especially for presenting species/samples distributions. I would like to know if it can be reached in R or any other resources?
The GIS layer were used in Fig 1. in this article (http://onlinelibrary.wiley.com/doi/10.1111/j.1469-8137.2010.03479.x/full).
This Fig 1 image is here:
http://onlinelibrary.wiley.com/store/10.1111/j.1469-8137.2010.03479.x/asset/image_t/NPH_3479_f1_thumb.gif?v=1&t=gsk5sbhs&s=e5e2e4bbb194f799f7ab9bec85a416e295405784
I have ever tried to submit this question in R-sig-geo. But, I failed. I expect to get some helps/directions here.
Thanks a lots for any directions.
Best wishes,
It is very possible to download this file and read it in with R, configure it to have the correct geo-coordinates so that overplotting works easily, and showing the image with the right colour scheme and so on. But, automating getting all of the data you need is not so easy.
You need the colour table from the GIF file so that you can plot the correct set of RGB values for each pixel (the information is in the file, but I'm not sure if this can be obtained directly with R, I will check - it certainly can be with GDAL, but extracting those values in an automated way depends on various tools being available).
UPDATE: It turns out that the raster package gets hold of the colour information correctly and plots it, see below.
You also need the geo-spatial information, i.e. the coordinates of a reference pixel (say, the top left pixel corner), and the scale (the geographic width and height of the pixels) and this information is not stored in the file. Also, the coordinate system of the file is not in the file, and very likely not provided explicitly with the image data.
If the colours and the coordinate system were stored with the file, then it would all be easy and something like the following would be enough.
(Note this worked for me once, but then I think subsequent requests are blocked by the server, so try to only download the file one time).
u <- "http://onlinelibrary.wiley.com/store/10.1111/j.1469-8137.2010.03479.x/asset/image_n/NPH_3479_f1.gif?v=1&t=gskxvi17&s=0f13fa9dae78bd6837aeee594065c6ca112864d2"
imfile <- paste(tempfile(), ".gif", sep = "")
download.file(u, imfile, mode = "wb")
library(raster) ## rgdal also required for this file format
library(rgdal)
im <- raster(imfile)
plot(im)
This looks fine but now see that there is no "real-world" coordinate system, this is just an axis from pixel 1 to the number in the X dimension (and same for Y).
axis(1, pos = 2)
So, still we need manually work to discover appropriate reference coordinates for the image - and guesses here can work fine, but still they are only guesses and you may end up creating a lot of pain for something seemingly simple.
If plot points interactively is enough for you, then you might use locator in conjunction with points and lines and text, and related plotting functions.
Feng,
if I read the Google docs correctly, you can modify the labels and displayed features with the extra parameters style and element.
I did not include custom parameters for these in the RgoogleMaps package, however, you can easily pass ANY addition parameters via the path argument !
If you read the help file for GetMap carefully, you will note the following example:
note that since the path string is just appended to the URL you can "abuse" the path argument to pass anything to the query, e.g. the style parameter:
#The following example displays a map of Brooklyn where local roads have been changed to bright green and the residential areas have been changed to black:
## Not run: GetMap(center='Brooklyn', zoom=12, maptype = "roadmap", path = "&style=feature:road.local|element:geometry|hue:0x00ff00|saturation:100&style=feature:landscape|element:geometry|lightness:-100", sensor='false', destfile = "MyTile4.png", RETURNIMAGE = FALSE);
Hope this helps,
Markus Loecher
If you just want data like this image, then there are packages to access imagery directly, again utilizing the tools in sp and rgdal. This example is close using gmap in the dismo package.
library(dismo)
e <- extent(-7, 5, 38, 44)
gm <- gmap(e, type = "terrain")
plot(gm)
Note that while we specify the extents in "longlat" the image comes back in its native (Google) Mercator.
print(gm)
See ?gmap for more options on transforming your own data to match the image's projection, or the broader function set in raster, rgdal and sp for other options. There are other imagery providers that might be preferable, and quite a few options in the R suite of contributed packages.