I'm trying to write a function that loops over rows of a dataframe and uses information about other rows to determine the output for each loop.
Consider the following dataframe, which is meant to represent people who have a longitude coordinate, a latitude coordinate, and a value to represent if they are or are not sick:
game.mat<-as.data.frame(matrix(0, nrow = 100, ncol = 3))
colnames(game.mat)<-c("PosX","PosY","Sick")
game.mat[,"PosX"]<-sample(x = c(1:100), 100, replace = TRUE)
game.mat[,"PosY"]<-sample(x = c(1:100), 100, replace = TRUE)
game.mat[,"Sick"]<-sample((c(rep(0,8),rep(1,2))),100,replace=TRUE)
Some minority of people will be sick at baseline. My function is meant to infect people who have neighboring x-y coordinates with a sick person (so anyone who is adjacent to someone who is sick). I considered embedding a function like this in an ifelse statement:
search_sick<-function(d,corx,cory){
d2<-d[d$PosX<corx+2&d$PosX>corx-2&d$PosY<cory+2&d$PosY>cory-2,]
if(sum(d2$Sick>0)){
d$Sick<-1
} else{
d$Sick<-0
}
}
But it makes everyone sick, perhaps because it gives everyone a value of 1 if anyone is next to a sick person. I also considered using an apply function. But from what I understand about apply, it will only operate within the a single row at a time so it will be impossible to retrieve information about whether other rows have neighboring coordinate values.
I hope this makes sense. Happy to provide any additional information.
Here's an example using apply
set.seed(1)
game.mat<-as.data.frame(matrix(0, nrow = 100, ncol = 3))
colnames(game.mat)<-c("PosX","PosY","Sick")
game.mat[,"PosX"]<-sample(x = c(1:100), 100, replace = TRUE)
game.mat[,"PosY"]<-sample(x = c(1:100), 100, replace = TRUE)
game.mat[,"Sick"]<-sample((c(rep(0,8),rep(1,2))),100,replace=TRUE)
#plot the sick individuals in red
plot(PosY~PosX, data=game.mat, col=as.factor(Sick), pch=16)
We'll modify your function to have a flexible search radius "r", and to return the indices of the newly infected individuals
search_sick<-function(d, corx, cory, r){
indx<-which(d$PosX<corx+r & d$PosX>corx-r & d$PosY<cory+r & d$PosY>cory-r)
}
contagious<-game.mat[game.mat$Sick==1,]
infected<-apply(contagious, 1, function(x) {
search_sick(game.mat, x[1], x[2], r=10)
})
game.mat$T1<-game.mat$Sick
game.mat$T1[unique(unlist(infected))]<-1
#circle points which have become sick
points(PosY~PosX, data=game.mat[game.mat$Sick==0 & game.mat$T1==1,], col="red", cex=2)
Related
I have a big raster with some NA cells that need to be filled. I want to fill it by Inverse Distance Weighting (IDW), by considering the 9 nearest [valid] cells. I used the idw function from the gstat package, but although it works, it takes ages to complete the task (my original raster comprises 6232186 cells that I include in the gstat call, and I have ~14000 gaps to be filled). As I have to repeat this task with several rasters, I'm looking for a faster solution. Does anyone have a suggestion?
I was thinking about using the focal from the raster or terra packages, but to be sincere I didn't understood very well how to set a matrix of weights to get a result like the IDW... Also, I would like to get the nearest valid cells (thus, suppose that in a square focal does not find valid cells, it would look further away to find more valid cells).
Just to give an example, suppose that in the following raster I need to fill the cells of number 310 and 330:
r <- raster(nrow = 20, ncol = 20)
r[1:300] <- sample(1:4, size = 300, replace = T)
plot(r)
gaps <- xyFromCell(r, c(310, 330))
points(gaps)
By using focal with a 3x3 square I would get the mean for just the cell 310 (and without the inverse weighting and also without getting 9 valid cells):
filed <- raster::focal(r, matrix(1, nrow = 3, ncol = 3), fun = mean, NAonly = T, na.rm = T)
plot(filed);points(gaps)
I appreciate any help/suggestion!
One approach would be to use a while loop to increase the window/matrix of the focal function until all NA cells are filled.
With terra it would be like this:
library(terra)
r <- rast(nrow = 20, ncol = 20)
r[1:300] <- sample(1:4, size = 300, replace = T)
gaps <- xyFromCell(r, c(310, 330))
w <- 1
filled <- r # just in case you want to keep the original
to_fill <- any(is.na(values(filled)))
# for big rasters you could use (same inside loop)
# to_fill <- global(filled, function(x) any(is.na(x)))[,1]
while(to_fill) {
w <- w + 2
filled <- focal(filled, w = w, fun = mean, na.policy = "only", na.rm = T)
to_fill <- any(is.na(values(filled)))
}
plot(filled)
points(gaps)
I have a 3d array as below:
prob = array(0,c(7,7,7))
Now, i need to refill it by random numbers as below:
pop = sample(1:100, 7**3, replace=TRUE)
pop = pop/sum(pop)
if simply assign the value to it then it will remove all the dimentions of prob :
prob = pop
print(dim(prob))
The output of the print is:
> print(dim(prob))
NULL
Therefore, apparently the prob = pop erase the dimensions.
How can i assign data but keep the 3d dimensions?
You can perform subset assignment as follows:
prob[] = pop
This will replace the values but preserve dimensions and other attributes.
However, this seems unnecessary in your case: why assign after the fact, when you can initialise?
pop = sample(1 : 100, 7 ** 3, replace = TRUE)
prob = array(pop / sum(pop), c(7, 7, 7))
There’s no need to pre-assign prob as a zero array, and in fact I’d consider that an anti-pattern: in general you should treat variables as read-only, unless there are specific reasons to reassign/modify them (and there rarely are).
I am working on an ecological problem, involving species distribution models. I have a raster which is essentially a landscape of probabilities of presence per cell, so to speak. I want to calculate a new raster, based on the old one, where each cell is equal to the mean of itself and all 8 adjacent cells. This is not the same as aggregating the cells by mean, which results in the border between the newly aggregated cells being calculated incorrectly.
I can do this with the bit of code provided, but the raster I am working with is way, way too big to run this calculation, as it uses too much memory. If I subdivide the raster, it will still take days to do. Does anyone have a more efficient way of calculating this? I have created a small version of the raster as an example, albeit somewhat clumsily:
require(raster)
## create raster called "ras" rather clumsily
## create raster called "ras" rather clumsily
# (UTM coordinates and a probability value for each cell, not really
# important)
s.x = seq(249990, by = 30, length.out = 20)
s.y = seq(6189390, by = 30, length.out = 20)
x.l = lapply(1:20, function(x){
rep(s.x[x], 20)
})
x.l2 = as.vector(c(x.l[[1]], x.l[[2]], x.l[[3]], x.l[[4]], x.l[[5]],
x.l[[6]], x.l[[7]], x.l[[8]], x.l[[9]], x.l[[10]],
x.l[[11]], x.l[[12]], x.l[[13]], x.l[[14]], x.l[[15]],
x.l[[16]],x.l[[17]], x.l[[18]], x.l[[19]], x.l[[20]]))
df = as.data.frame(cbind(x.l2, rep(s.y, 20), rnorm(20*20, 0.5, 0.2)))
colnames(df) = c("x", "y", "P")
coordinates(df) <- ~ x + y
gridded(df) <- TRUE
ras = raster(df)
# for each cell, make a vector of the values at
# the cell and all <=8 adjacent cells:
vl = lapply(1:length(ras), function(x){
extract(ras,
(c(x,(adjacent(ras, x, directions=8, pairs=F, sorted=F)))))
})
# find the mean for each cell
vm = sapply(1:length(ras), function(x){
as.vector(mean(vl[[x]], na.rm = T))
})
# create raster template
templ = ras/ras
# multiply into template for new raster
ras = vm*templ
I have two sets of points stored in R as sf objects. Point object x contains 204,467 and point y contains 5,297 points.
In theory, I would want to calculate the distance from all points in x to all points in y. I understand that this would create a beast of a matrix, but it is doable using st_distance(x, y, by_element=FALSE) in the sf package in about 40 minutes on my i7 desktop.
What I want to do is to calculate the distance from all of the points in x to all of the points in y, then I want to convert this into a data.frame, that contains all variables for the respective x and y pair of points. This is because I want flexibility in terms of aggregation using dplyr, for instance, I want to find the number of points in y, that is within 10, 50, 100 km from x, and where x$year < y$year.
I successfully created the distance matrix, which has around 1,083,061,699 cells. I know this is a very inefficient way of doing this, but it gives flexibility in terms of aggregation. Other suggestions are welcome.
Below is code to create two sf point objects, and measure the distance between them. Next, I would want to convert this into a data.frame with all variables from x and y, but this is where I fail to proceed.
If my suggested workflow is unfeasible, can someone provide an alternative solution to measure distance to all points within a predefined radius, and create a data.frame of the result with all variables from x and y?
# Create two sf point objects
set.seed(123)
library(sf)
pts1 <- st_as_sf(x = data.frame(id=seq(1,204467,1),
year=sample(seq(from = 1990, to = 2018, by = 1), size = 204467, replace = TRUE),
xcoord=sample(seq(from = -180, to = 180, by = 1), size = 204467, replace = TRUE),
ycoord=sample(seq(from = -90, to = 90, by = 1), size = 204467, replace = TRUE)),
coords=c("xcoord","ycoord"),crs=4326)
pts2 <- st_as_sf(x = data.frame(id=seq(1,5297,1),
year=sample(seq(from = 1990, to = 2018, by = 1), size = 5297, replace = TRUE),
xcoord=sample(seq(from = -180, to = 180, by = 1), size = 5297, replace = TRUE),
ycoord=sample(seq(from = -90, to = 90, by = 1), size = 5297, replace = TRUE)),
coords=c("xcoord","ycoord"),crs=4326)
distmat <- st_distance(pts1,pts2,by_element = FALSE)
I would consider approaching this differently. Once you have your distmat matrix, you can do the types of calculation you describe without needing a data.frame. You can use standard subsetting to find which points meet your specified criteria.
For example, to find the combinations of points where pts1$year is greater than pts2$year we can do:
subset_points = outer(pts1$year, pts2$year, `>`)
Then, to find how many of these are separated more than 100 km, we can do
library(units)
sum(distmat[subset_points] > (100 * as_units('km', 1)))
A note on memory usage
However you approach this with sf or data.frame objects, the chances are that you will start to bump up against RAM limits with 1e9 floating points in each matrix or column of a data.table. You might think about instead converting your distance matrix to a raster. Then the raster can be stored on disk rather than in memory, and you can utilise the memory-safe functions in the raster package to crunch your way through.
How we might use rasters to work from disk and save RAM
We can use memory-safe raster operations for the very large matrices like this, for example:
library(raster)
# convert our matrices to rasters, so we can work on them from disk
r = raster(matrix(as.numeric(distmat), length(pts1$id), length(pts2$id)))
s = raster(subset_points)
remove('distmat', 'subset_points')
# now create a raster equal to r, but with zeroes in the cells we wish to exclude from calculation
rs = overlay(r,s,fun=function(x,y){x*y}, filename='out1.tif')
# find which cells have value greater than x (1e6 in the example)
Big_cells = reclassify(rs, matrix(c(-Inf, 1e6, 0, 1e6, Inf, 1), ncol=3, byrow=TRUE), 'out.tiff', overwrite=T)
# and finally count the cells
N = cellStats(Big_cells, sum)
Suppose you have two lines, L1 and L2, which for each x value (x1 and x2 for example) they have known points at L1={(x1,L1_y1), (x2,L1_y2)}, and L2={(x1,L2_y1), (x2,L2_y2)}. By joining these points they may or may not have an intersection at some x3 where x1
Now suppose you want to know the maximum at any x value (not restricted to just x1, x2 etc, but anywhere along the axis) of both of these lines. Obviously it is often trivial to calculate for just a few lines, and a few different x value, but in my case I have several tens of thousand x values and a few lines to check it against, so it can't be done manually.
In R, is there some code which will calculate the maximum at any given point x3?
An example of this can be seen here with L1={(1,1), (2,4)}, and L2={(1,4),(2,1)}, illustrated by:
Here the intersection of these lines is at (1.5, 2.5). L2 is the maximum before this, and L1 after. This maximum line is shown in red below.
As you can see, it isn't enough just to take the max at every point and join these up, and so it will need to consider the lines as some form of function, and then take the maximum of this.
Also, as mention before as there are several thousand x values it will need to generalise to larger data.
To test the code further if you wish you can randomly generate y values for some x values, and it will be clear to see from a plot if it works correctly or not.
Thanks in advance!
Defining points constituting your lines from the example
L1 <- list(x = c(1, 2), y = c(1, 4))
L2 <- list(x = c(1, 2), y = c(4, 1))
defining a function taking a pointwise maximum of two functions corresponding to the lines
myMax <- function(x)
pmax(approxfun(L1$x, L1$y)(x), approxfun(L2$x, L2$y)(x))
This gives
plot(L1$x, L1$y, type = 'l')
lines(L2$x, L2$y, col = 'red')
curve(myMax(x), from = 1, to = 2, col = 'blue', add = TRUE)
Clearly this extends to more complex L1 and L2 as approxfun is just a piecewise-linear approximation. Also, you may add L3, L4, and so on.