I have a dataframe (test) in R. Inside one of the columns contains coordinates in this list structure:
> dput(test$coordinates)
list(structure(list(x = c(-1.294832, -1.294883, -1.294262,
-1.249478), y = c(54.61024, 54.61008, 54.610016, 54.610006
)), .Names = c("x", "y"), row.names = c(NA, -284L), class = c("tbl_df",
"tbl", "data.frame")))
I've reduced the number of coordinates for clarity.
Ultimately I wish to convert the dataframe into a spaitial lines dataframe but to do that I need the test$coordinates in a lines form. However, I get the following error
> lines(test$coordinates)
Error in xy.coords(x, y) :
'x' is a list, but does not have components 'x' and 'y'
I have tried to convert the test$coordinates to other forms but it usually results in some error. How do I transform this list into a line?
Extra info this is a follow up question to
Convert data frame to spatial lines data frame in R with x,y x,y coordintates
UPDATE as requested dput(head(test)):
> dput(head(test))
structure(list(rid = 1, start_id = 1L, start_code = "E02002536",
end_id = 106L, end_code = "E02006909", strategy = "fastest",
distance = 12655L, time_seconds = 2921L, calories = 211L,
document.id = 1L, array.index = 1L, start = "Geranium Close",
finish = "Hylton Road", startBearing = 0, startSpeed = 0,
start_longitude = -1.294832, start_latitude = 54.610241,
finish_longitude = -1.249478, finish_latitude = 54.680691,
crow_fly_distance = 8362, event = "depart", whence = 1473171787,
speed = 20, itinerary = 419956, clientRouteId = 0, plan = "fastest",
note = "", length = 12655, time = 2921, busynance = 42172,
quietness = 30, signalledJunctions = 3, signalledCrossings = 2,
west = -1.300074, south = 54.610006, east = -1.232447, north = 54.683814,
name = "Geranium Close to Hylton Road", walk = 0, leaving = "2016-09-06 15:23:07",
arriving = "2016-09-06 16:11:48", grammesCO2saved = 2359,
calories2 = 211, type = "route", coordinates = list(structure(list(
x = c(-1.294832, -1.294883, -1.294262, -1.294141, -1.29371,
-1.293726, -1.293742, -1.29351, -1.293368, -1.292816,
-1.248019, -1.249478), y = c(54.61024, 54.61008, 54.610016,
54.610006, 54.610038, 54.610142, 54.610247, 54.610262,
54.681238, 54.680975, 54.680601, 54.680404
)), .Names = c("x", "y"), row.names = c(NA, -284L), class = c("tbl_df",
"tbl", "data.frame")))), .Names = c("rid", "start_id", "start_code",
"end_id", "end_code", "strategy", "distance", "time_seconds",
"calories", "document.id", "array.index", "start", "finish",
"startBearing", "startSpeed", "start_longitude", "start_latitude",
"finish_longitude", "finish_latitude", "crow_fly_distance", "event",
"whence", "speed", "itinerary", "clientRouteId", "plan", "note",
"length", "time", "busynance", "quietness", "signalledJunctions",
"signalledCrossings", "west", "south", "east", "north", "name",
"walk", "leaving", "arriving", "grammesCO2saved", "calories2",
"type", "coordinates"), row.names = c(NA, -1L), class = c("tbl_df",
"tbl", "data.frame"))
lines is a plotting function. I'm assuming you want sp::SpatialLines. See ?"SpatialLines-class" for how to construct such an object.
Here's for your case, provided you don't have a "corrupt" data.frame (see at the bottom of this post).
library(sp)
coords <- as.data.frame(xy$coordinates[[1]])[1:12, ]
out <- SpatialLines(list(Lines(list(Line(coords)), ID = 1)))
An object of class "SpatialLines"
Slot "lines":
[[1]]
An object of class "Lines"
Slot "Lines":
[[1]]
An object of class "Line"
Slot "coords":
x y
1 -1.294832 54.61024
2 -1.294883 54.61008
3 -1.294262 54.61002
4 -1.294141 54.61001
5 -1.293710 54.61004
6 -1.293726 54.61014
7 -1.293742 54.61025
8 -1.293510 54.61026
9 -1.293368 54.68124
10 -1.292816 54.68097
11 -1.248019 54.68060
12 -1.249478 54.68040
Slot "ID":
[1] "1"
Slot "bbox":
min max
x -1.294883 -1.248019
y 54.610006 54.681238
Slot "proj4string":
CRS arguments: NA
To add data to this object, you should use
SpatialLinesDataFrame(out, data = yourdata)
but see this example for more info.
There's a warning when I tried to coerce your coordinates to a data.frame. Hopefully this isnt' the case for your dataset.
> as.data.frame(xy$coordinates[[1]])
x y
1 -1.294832 54.61024
2 -1.294883 54.61008
3 -1.294262 54.61002
...
281 <NA> <NA>
282 <NA> <NA>
283 <NA> <NA>
284 <NA> <NA>
Warning message:
In format.data.frame(x, digits = digits, na.encode = FALSE) :
corrupt data frame: columns will be truncated or padded with NAs
Related
I've got a list of dataframes. I'd like to cbind them by the index column, sample_id. Each table has the same column headings, so I can't just cbind them otherwise I won't know which list item the columns came from. The name of the list item gives the measure used to generate them, so I'd like to suffix the column headings with the list item name.
Here's a simplified demo list of dataframes:
list_of_tables <- list(number = structure(list(sample_id = structure(1:3, levels = c("CSF_1",
"CSF_2", "CSF_4"), class = "factor"), total = c(655, 331, 271
), max = c(12, 5, 7)), row.names = c(NA, -3L), class = c("tbl_df",
"tbl", "data.frame")), concentration_cm_3 = structure(list(sample_id = structure(1:3, levels = c("CSF_1",
"CSF_2", "CSF_4"), class = "factor"), total = c(121454697, 90959097,
43080697), max = c(2050000, 2140000, 915500)), row.names = c(NA,
-3L), class = c("tbl_df", "tbl", "data.frame")), volume_nm_3 = structure(list(
sample_id = structure(1:3, levels = c("CSF_1", "CSF_2", "CSF_4"
), class = "factor"), total = c(2412783009, 1293649395, 438426087
), max = c(103500000, 117400000, 23920000)), row.names = c(NA,
-3L), class = c("tbl_df", "tbl", "data.frame")), area_nm_2 = structure(list(
sample_id = structure(1:3, levels = c("CSF_1", "CSF_2", "CSF_4"
), class = "factor"), total = c(15259297.4, 7655352.2, 3775922
), max = c(266500, 289900, 100400)), row.names = c(NA, -3L
), class = c("tbl_df", "tbl", "data.frame")))
You'll see it's a list of 4 tables, and the list item names are "number", "concentration_cm_3", "volume_nm_3", and "area_nm_2".
Using join_all from plyr I can merge them all by sample_id. However, how do I suffix with the list item name?
merged_tables <- plyr::join_all(stats_by_measure, by = "sample_id", type = "left")
we could do it this way:
The trick is to use .id = 'id' in bind_rows which adds the name as a column. Then we could pivot:
library(dplyr)
library(tidyr)
bind_rows(list_of_tables, .id = 'id') %>%
pivot_wider(names_from = id,
values_from = c(total, max))
sample_id total_number total_concentration_cm_3 total_volume_nm_3 total_area_nm_2 max_number max_concentration_cm_3 max_volume_nm_3 max_area_nm_2
<fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 CSF_1 655 121454697 2412783009 15259297. 12 2050000 103500000 266500
2 CSF_2 331 90959097 1293649395 7655352. 5 2140000 117400000 289900
3 CSF_4 271 43080697 438426087 3775922 7 915500 23920000 100400
Probably, we may use reduce2 here with suffix option from left_join
library(dplyr)
library(purrr)
nm <- names(list_of_tables)[1]
reduce2(list_of_tables, names(list_of_tables)[-1],
function(x, y, z) left_join(x, y, by = 'sample_id', suffix = c(nm, z)))
Or if we want to use join_all, probably we can rename the columns before doing the join
library(stringr)
imap(list_of_tables, ~ {
nm <- .y
.x %>% rename_with(~str_c(.x, nm), -1)
}) %>%
plyr::join_all( by = "sample_id", type = "left")
Or use a for loop
tmp <- list_of_tables[[1]]
names(tmp)[-1] <- paste0(names(tmp)[-1], names(list_of_tables)[1])
for(nm in names(list_of_tables)[-1]) {
tmp2 <- list_of_tables[[nm]]
names(tmp2)[-1] <- paste0(names(tmp2)[-1], nm)
tmp <- left_join(tmp, tmp2, by = "sample_id")
}
tmp
Let's say I have 2 different functions to apply. For example, these functions are max and min . After applying bunch of functions I am getting outputs below. I want to assign a function to each output.
Here is my data and its structure.
data<-structure(list(Apr = structure(list(`a1` = structure(list(
date = c("04-01-2036", "04-02-2036", "04-03-2036"), value = c(0,
3.13, 20.64)), .Names = c("date", "value"), row.names = 92:94, class = "data.frame"),
`a2` = structure(list(date = c("04-01-2037", "04-02-2037",
"04-03-2037"), value = c(5.32, 82.47, 15.56)), .Names = c("date",
"value"), row.names = 457:459, class = "data.frame")), .Names = c("a1",
"a2")), Dec = structure(list(`d1` = structure(list(
date = c("12-01-2039", "12-02-2039", "12-03-2039"), value = c(3,
0, 11)), .Names = c("date", "value"), row.names = 1431:1433, class = "data.frame"),
`d2` = structure(list(date = c("12-01-2064", "12-02-2064",
"12-03-2064"), value = c(0, 5, 0)), .Names = c("date", "value"
), row.names = 10563:10565, class = "data.frame")), .Names = c("d1",
"d2"))), .Names = c("Apr", "Dec"))
I applied these functions:
drop<-function(y){
lapply(y, function(x)(x[!(names(x) %in% c("date"))]))
}
q1<-lapply(data, drop)
q2<-lapply(q1, function(x) unlist(x,recursive = FALSE))
daily_max<-lapply(q2, function(x) lapply(x, max))
dailymax <- data.frame(matrix(unlist(daily_max), nrow=length(daily_max), byrow=TRUE))
row.names(dailymax)<-names(daily_max)
max_value <- apply(dailymax, 1, which.max)
And I'm getting
Apr Dec
2 1
And I am applying any random function to both Apr[2] and Dec[1] like:
Map(function(x, y) sum(x[[y]]), q2, max_value)
So, the function will be executed considering the outputs (to Apr's second element which is a1, Dec's first element which is a2.) As you can see, there are outputs as numbers 1 and 2.
What I want
What I want is assigning specific functions to 1 and 2. If output is 1 then max function; if it is 2, min function will be executed. In conclusion, max function will be applied to Apr[2] and min function will be applied to Dec[1].
I will get this:
min(q2$Apr$a2.value)
[1] 5.32
max(q2$Dec$d2.value)
[1] 5
How can I achieve this automatically for all my functions?
You can take help of switch here to apply a function based on number in max_value.
apply_function <- function(x, num) switch(num, `1` = max, `2` = min)(x)
Map(function(x, y) apply_function(x[[y]], y), q2, max_value)
#$Apr
#[1] 5.32
#$Dec
#[1] 11
Map returns a list if you want a vector output use mapply.
I want to apply any function to elements which I found its location with which.max function. For example, my sample data is below:
$Apr
$Apr$`04-2036`
date value
92 04-01-2036 0.00
93 04-02-2036 3.13
94 04-03-2036 20.64
$Apr$`04-2037`
date value
457 04-01-2037 5.32
458 04-02-2037 82.47
459 04-03-2037 15.56
$Dec
$Dec$`04-2039`
date value
1431 12-01-2039 3
1432 12-02-2039 0
1433 12-03-2039 11
$Dec$`04-2064`
date value
10563 12-01-2064 0
10564 12-02-2064 5
10565 12-03-2064 0
data<-structure(list(Apr = structure(list(`04-2036` = structure(list(
date = c("04-01-2036", "04-02-2036", "04-03-2036"), value = c(0,
3.13, 20.64)), .Names = c("date", "value"), row.names = 92:94, class = "data.frame"),
`04-2037` = structure(list(date = c("04-01-2037", "04-02-2037",
"04-03-2037"), value = c(5.32, 82.47, 15.56)), .Names = c("date",
"value"), row.names = 457:459, class = "data.frame")), .Names = c("04-2036",
"04-2037")), Dec = structure(list(`04-2039` = structure(list(
date = c("12-01-2039", "12-02-2039", "12-03-2039"), value = c(3,
0, 11)), .Names = c("date", "value"), row.names = 1431:1433, class = "data.frame"),
`04-2064` = structure(list(date = c("12-01-2064", "12-02-2064",
"12-03-2064"), value = c(0, 5, 0)), .Names = c("date", "value"
), row.names = 10563:10565, class = "data.frame")), .Names = c("04-2039",
"04-2064"))), .Names = c("Apr", "Dec"))
I have found locations of maximum values for each element in lists of list using the functions below.
drop<-function(y){
lapply(y, function(x)(x[!(names(x) %in% c("date"))]))
}
q1<-lapply(data, drop)
q2<-lapply(q1, function(x) unlist(x,recursive = FALSE))
daily_max<-lapply(q2, function(x) lapply(x, max))
dailymax <- data.frame(matrix(unlist(daily_max), nrow=length(daily_max), byrow=TRUE))
row.names(dailymax)<-names(daily_max)
apply(dailymax, 1, which.max)
Locations of max. values of each element is computed as it is seen below;
Apr Dec
2 1
Now, I want to apply any function to these elements automatically for all my data(it is Apr 2 = Apr$04-2037 and Dec$2039).
You can subset and keep only the max value data in each list.
max_value <- apply(dailymax, 1, which.max)
Map(`[[`, data, max_value)
#$Apr
# date value
#457 04-01-2037 5.32
#458 04-02-2037 82.47
#459 04-03-2037 15.56
#$Dec
# date value
#1431 12-01-2039 3
#1432 12-02-2039 0
#1433 12-03-2039 11
Let's say you want to apply the function fn to this list.
fn <- function(x) {x$value <- x$value * 2;x}
You can change the Map function as -
Map(function(x, y) fn(x[[y]]), data, max_value)
#$Apr
# date value
#457 04-01-2037 10.64
#458 04-02-2037 164.94
#459 04-03-2037 31.12
#$Dec
# date value
#1431 12-01-2039 6
#1432 12-02-2039 0
#1433 12-03-2039 22
I know the subject is widely covers but I didn't find the code working for my case... I have a dataframe of this type:
V1 V2 V3
1: label1 alias_fr alias_fr
2: label1 triplet triplet
3: label1 Q9327 Q3122270
4: label2 NULL NULL
5: label3 alias_fr NULL
6: label3 triplet NULL
7: label3 Q678 NULL
This dataframe is generated after mapping a json output to a query from a df input:
df <- Map(rbind, originalDF$input,out) #I first used Map(c,..) but it seems to be more difficult to reshape than rbind
df <- rbind.fill(lapply(df,function(y){as.data.frame(t(y),stringsAsFactors=FALSE)}))
class(df)
[1] "data.frame"
The example given is simplified though, as I have more than 3 columns, and some values are lists. Nevertheless when I have non-NULL values for a label I have always the same number of rows within a column (3 in my example: alias_fr, triplet, Qxx).
And I would like to have V2 and V3 values in row for each V1 value:
V1 var1 var2 var3
label1 alias_fr triplet Q9327
label1 alias_fr triplet Q3122270
label2 NULL NULL NULL
label3 alias_fr triplet Q678
I try to start melt: melt(df,id="V1"), but then I am stuck.
I also tried reshape, cast, dcast, without any success, and I am more and more confuse with all reshaping stuff... If a reshape master is around, I would be very gratefull ;)
[Edit]: real objects to clarify my issue
Ok so this is an extract of the real dataset I’m working with:
#original dataset (actually it’s one column of the dataset)
originalDF <- c("Guy de Maupassant", "J.-J. Goldman", "Poitou-Charentes")
#output of the API query from the text in the orginalDF
out <- list(structure(list(`_index` = c("alias_fr", "alias_fr"), `_type` = c("triplet",
"triplet"), `_id` = c("Q9327", "Q3122270"), `_score` = c(NA,
NA), sort = list(-4.95263021255079, -6.65910164747673), `_source.types` = list(
structure(list(id = c("Q5", "dbPedia.Person"), value = c("être humain",
"personne")), .Names = c("id", "value"), class = "data.frame", row.names = 1:2),
structure(list(id = c("Q11424", "dbPedia.Film"), value = c("film",
"film")), .Names = c("id", "value"), class = "data.frame", row.names = 1:2)),
`_source.pageRank` = c(-4.95263021255079, -6.65910164747673
), `_source.subTypes` = list(structure(list(id = c("Q1930187",
"Q36180", "Q15949613", "Q6625963", "Q214917"), value = c("journaliste",
"écrivain", "nouvelliste", "romancier", "dramaturge")), .Names = c("id",
"value"), class = "data.frame", row.names = c(NA, 5L)), NULL),
`_source.label` = c("Guy de Maupassant", "Guy de Maupassant"
), `_source.id` = c("Q9327", "Q3122270")), .Names = c("_index",
"_type", "_id", "_score", "sort", "_source.types", "_source.pageRank",
"_source.subTypes", "_source.label", "_source.id"), class = "data.frame", row.names = 1:2),
list(), structure(list(`_index` = "alias_fr", `_type` = "triplet",
`_id` = "Q17009", `_score` = NA, sort = list(-5.0448283638424),
`_source.types` = list(structure(list(id = "Q22670030",
value = "ancienne région française"), .Names = c("id",
"value"), class = "data.frame", row.names = 1L)), `_source.pageRank` = -5.0448283638424,
`_source.label` = "Poitou-Charentes", `_source.id` = "Q17009"), .Names = c("_index",
"_type", "_id", "_score", "sort", "_source.types", "_source.pageRank",
"_source.label", "_source.id"), class = "data.frame", row.names = 1L))
#df object (generated from Map, then rbind.fill)
df <- structure(list(V1 = list("Guy de Maupassant", "Guy de Maupassant",
"Guy de Maupassant", "Guy de Maupassant", "Guy de Maupassant",
"Guy de Maupassant", "Guy de Maupassant", "Guy de Maupassant",
"Guy de Maupassant", "Guy de Maupassant", "J.-J. Goldman",
"Poitou-Charentes", "Poitou-Charentes", "Poitou-Charentes",
"Poitou-Charentes", "Poitou-Charentes", "Poitou-Charentes",
"Poitou-Charentes", "Poitou-Charentes", "Poitou-Charentes"),
V2 = list("alias_fr", "triplet", "Q9327", NA_character_,
-4.95263021255079, structure(list(id = c("Q5", "dbPedia.Person"
), value = c("être humain", "personne")), .Names = c("id",
"value"), class = "data.frame", row.names = 1:2), "-4.95263021255079",
structure(list(id = c("Q1930187", "Q36180", "Q15949613",
"Q6625963", "Q214917"), value = c("journaliste", "écrivain",
"nouvelliste", "romancier", "dramaturge")), .Names = c("id",
"value"), class = "data.frame", row.names = c(NA, 5L)),
"Guy de Maupassant", "Q9327", NULL, "alias_fr", "triplet",
"Q17009", NA_character_, -5.0448283638424, structure(list(
id = "Q22670030", value = "ancienne région française"), .Names = c("id",
"value"), class = "data.frame", row.names = 1L), "-5.0448283638424",
"Poitou-Charentes", "Q17009"), V3 = list("alias_fr",
"triplet", "Q3122270", NA_character_, -6.65910164747673,
structure(list(id = c("Q11424", "dbPedia.Film"), value = c("film",
"film")), .Names = c("id", "value"), class = "data.frame", row.names = 1:2),
"-6.65910164747673", NULL, "Guy de Maupassant", "Q3122270",
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL), V4 = list(NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL)), .Names = c("V1", "V2",
"V3", "V4"), row.names = c(NA, 20L), class = "data.frame")
For your particular example (each label has a maximum of three rows), one approach would be
require(dplyr)
df <- data.frame(label = c(rep('a',3),'b', rep('c',3)), id1 = c(1,2,3,NA, 1,2,4), id2 = c(1,2,5,NA,NA,NA,NA))
#I used different names than you, because I have the impression that each column belongs to another measurement or so
df1 <- select(df, label, id1)#subsets your df into two data frames
df2 <- select(df, label, id2)
df1a <- df1 %>% group_by(label) %>% summarise (var1 = id1[1], var2 = id1[2], var3 = id1[3])
df2b <- df2 %>% group_by(label) %>% summarise (var1 = id2[1], var2 = id2[2], var3 = id2[3])
#this groups the columns after your label and then you can force the creation of NA values for the observations that do not have a row in your df
df_final <- rbind(df1a,df2b)
> df_final
# A tibble: 6 x 4
label var1 var2 var3
<fctr> <dbl> <dbl> <dbl>
1 a 1 2 3
2 b NA NA NA
3 c 1 2 4
4 a 1 2 5
5 b NA NA NA
6 c NA NA NA
I know that this is not elegant, and not generalisable, because you manually assign a new column for each row that you have/ do not have in your df, but it should work with your example.
After reading in a large data set with read.csv.ffdf, one of the columns is time. Such as 2014-10-18 00:01:02, for 1 million rows in that column. That column is a factor. How do I convert it to POSIXct supported by ff? Simply using as.POSIXct() just turns the values into NA
Or when I read in the data set in the beginning, can I specify that column to be POSIXct?
My goal is to get the month and days (or even hour). So I'm open to solutions other than converting to POSIXct.
For example, we have 9 by 2 table,
test <- read.csv.ffdf(file="test.csv", header=T, first.rows=-1)
Two columns are ID (numeric class), and time (factor class)
Here is dput
structure(list(virtual = structure(list(VirtualVmode = c("integer",
"integer"), AsIs = c(FALSE, FALSE), VirtualIsMatrix = c(FALSE,
FALSE), PhysicalIsMatrix = c(FALSE, FALSE), PhysicalElementNo = 1:2,
PhysicalFirstCol = c(1L, 1L), PhysicalLastCol = c(1L, 1L)), .Names = c("VirtualVmode",
"AsIs", "VirtualIsMatrix", "PhysicalIsMatrix", "PhysicalElementNo",
"PhysicalFirstCol", "PhysicalLastCol"), row.names = c("ID", "time"
), class = "data.frame", Dim = c(9L, 2L), Dimorder = 1:2), physical = structure(list(
ID = structure(list(), physical = <pointer: 0x000000000821ab20>, virtual = structure(list(), Length = 9L, Symmetric = FALSE), class = c("ff_vector",
"ff")), time = structure(list(), physical = <pointer: 0x000000000821abb0>, virtual = structure(list(), Length = 9L, Symmetric = FALSE, Levels = c("10/17/2003 0:01",
"12/5/1999 0:02", "2/1/2000 0:01", "3/23/1998 0:01", "3/24/2013 0:00",
"5/29/2004 0:00", "5/9/1985 0:01", "6/14/2010 0:01", "6/25/2008 0:02"
), ramclass = "factor"), class = c("ff_vector", "ff"))), .Names = c("ID",
"time")), row.names = NULL), .Names = c("virtual", "physical",
"row.names"), class = "ffdf")
You can use with from package ffbase as shown below on a toy example. Best.
require(ff)
x <- data.frame(id = 1:100000, timepoint = seq(from = Sys.time(), by = "sec", length.out = 100000))
x$timepoint <- as.factor(x$timepoint)
xff <- as.ffdf(x)
class(xff)
require(ffbase)
xff$time <- with(xff, as.POSIXct(as.character(timepoint)), by = 10000)
ramclass(xff$time)
[1] "POSIXct" "POSIXt"
str(xff[1:10, ])
'data.frame': 10 obs. of 3 variables:
$ id : int 1 2 3 4 5 6 7 8 9 10
$ timepoint: Factor w/ 100000 levels "2014-10-20 09:14:10",..: 1 2 3 4 5 6 7 8 9 10
$ time : POSIXct, format: "2014-10-20 09:14:10" "2014-10-20 09:14:11" "2014-10-20 09:14:12" "2014-10-20 09:14:13" ...
Use colClasses when reading in the data.
e.g. with your example of two columns: ID (numeric class), and time (factor class):
test <- read.csv.ffdf(file="test.csv", header=T, first.rows=-1,colClasses = c("integer","POSIXct"))