I have a CSV file containing times per competitor of each section of a triathlon. I am having trouble reading the data so that R can use it. Here is an example of how the data looks (I've removed some columns for clarity):
"Place","Division","Gender","Swim","T1","Bike","T2","Run","Finish"
1, "40-49","M","7:45","0:55","27:07","0:29","18:53","55:07"
2, "UNDER 18","M","5:41","0:28","30:41","0:28","18:38","55:55"
3, "40-49","M","6:27","0:26","29:24","0:40","20:16","57:11"
4, "40-49","M","7:57","0:35","29:19","0:23","19:20","57:32"
5, "40-49","M","6:28","0:32","31:00","0:34","19:19","57:51"
6, "40-49","M","7:42","0:30","30:02","0:37","19:11","58:02"
....
250 ,"18-29","F","13:20","3:23","1:06:40","1:19","38:00","2:02:40"
251 ,"30-39","F","13:01","2:42","1:02:12","1:20","43:45","2:02:58"
252 ,50 ,"F","20:45","1:33","58:09","3:17","40:14","2:03:56"
253 ,"30-39","M","13:14","1:14","DNF","1:11","25:10","DNF bike"
254 ,"40-49","M","10:04","1:41","56:36","2:32",,"D.N.F"
My first naive attempt to plot the data went like this.
> tri <- read.csv(file.choose(), header=TRUE, as.is=TRUE)
> pairs(~ Bike + Run + Swim, data=tri)
The times are not being imported in a sensible way so the charts don't make sense.
I have found the difftime type and have tried to use it to parse the times in the data file.
There are some rows with DNF or similar in place of times, I'm happy for rows with times that can't be parsed to be discarded. There are two formats for the times "%M:%S" and "%H:%M:%S"
I think I need to create a new data frame from the data but I am having trouble parsing the times. This is what I have so far.
> tri <- read.csv(file.choose(), header=TRUE, as.is=TRUE)
> str(tri)
'data.frame': 254 obs. of 12 variables:
$ Place : num 1 2 3 4 5 6 7 8 9 10 ...
$ Race.. : num 237 274 268 226 267 247 264 257 273 272 ...
$ First.Name: chr ** removed names ** ...
$ Last.Name : chr ** removed names ** ...
$ Division : chr "40-49" "UNDER 18" "40-49" "40-49" ...
$ Gender : chr "M" "M" "M" "M" ...
$ Swim : chr "7:45" "5:41" "6:27" "7:57" ...
$ T1 : chr "0:55" "0:28" "0:26" "0:35" ...
$ Bike : chr "27:07" "30:41" "29:24" "29:19" ...
$ T2 : chr "0:29" "0:28" "0:40" "0:23" ...
$ Run : chr "18:53" "18:38" "20:16" "19:20" ...
$ Finish : chr "55:07" "55:55" "57:11" "57:32" ...
> as.numeric(as.difftime(tri$Bike, format="%M:%S"), units="secs")
This converts all the times that are under one hour, but the hours are interpreted as minutes for any times over an hour. Substituting "%H:%M:%S" for "%M:%S" parses times over an hour but produces NA otherwise. What is the best way to convert both types of times?
EDIT: Adding a simple example as requested.
> times <- c("27:07", "1:02:12", "DNF")
> as.numeric(as.difftime(times, format="%M:%S"), units="secs")
[1] 1627 62 NA
> as.numeric(as.difftime(times, format="%H:%M:%S"), units="secs")
[1] NA 3732 NA
The output I would like would be 1627 3732 NA
Here's a quick hack at a solution, although there may be a better one:
cdifftime <- function(x) {
x2 <- gsub("^([0-9]+:[0-9]+)$","00:\\1",x) ## prepend 00: to %M:%S elements
res <- as.difftime(x2,format="%H:%M:%S")
units(res) <- "secs"
as.numeric(res)
}
times <- c("27:07", "1:02:12", "DNF")
cdifftime(times)
## [1] 1627 3732 NA
You can apply this to the relevant columns:
tri[4:9] <- lapply(tri[4:9],cdifftime)
A couple of notes from trying to replicate your example:
you may want to use na.strings="DNF" to set "did not finish" values to NA automatically
you need to make sure strings are not read in as factors, e.g. (1) set options(stringsAsFactors="FALSE"); (2) use stringsAsFactors=FALSE when calling read.csv; (3) use as.is=TRUE, ditto.
Related
I am attempting to do a survival analysis, with "time to loss to follow up".
I have tried to fix the error by ensuring that the column is numeric (see strings as factors and colClasses in the .csv read function below), but it has not solved the error.
I have trawled stack overflow and other sites for answers, but I am stuck.
Can anyone help, please?
library(tidyverse)
library(gtsummary)
library(data.table)
library(tidyr)
library(dplyr)
library(survival)
survdat <- fread("221121_HBV_Followup_survivalanalysis.csv", stringsAsFactors=FALSE,
colClasses = c("Time to LTFU"="numeric"))
#Create censoring variable (right censoring)
survdat$censored[survdat$`LTFU confirmed` == 'Yes']<- 1
survdat$censored[survdat$`LTFU confirmed` == 'No'] <-0
#specify KM analysis model
km1 <- survfit(Surv('Time to LTFU', censored) ~ 1,
data=survdat,
type="kaplan-meier")
#I get the following error
> km1 <- survfit(Surv('Time to LTFU', censored) ~ 1,
+ data=survdat,
+ type="kaplan-meier")
Error in Surv("Time to LTFU", censored) : Time variable is not numeric
str(survdat)
````
NB Have removed some of the variables for confidentiality
Classes ‘data.table’ and 'data.frame': 43 obs. of 10 variables:
$ Date screened : chr "19/10/2021" "07/07/2021" "18/01/2022" "07/05/2021" ...
$ Last date seen : chr "21/11/2022" "21/11/2022" "21/11/2022" "21/11/2022" ...
$ Time to LTFU : num 398 502 307 563 564 605 516 29 118 118 ...
$ LTFU confirmed : chr "No" "No" "No" "No" ...
$ censored : num 0 0 0 0 0 0 0 1 1 0 ...
As you can see, the "Time to LTFU" variable IS numeric!
Please help!
Thanks
Time to LTFU needs to be between backticks, not single quotes, otherwise you are supplying a string (character variable) to the function.
km1 <- survfit(Surv(`Time to LTFU`, censored) ~ 1,
data=survdat,
type="kaplan-meier")
Can someone please help how to get the list of built-in data sets and their dependency packages?
There are several ways to find the included datasets in R:
1: Using data() will give you a list of the datasets of all loaded packages (and not only the ones from the datasets package); the datasets are ordered by package
2: Using data(package = .packages(all.available = TRUE)) will give you a list of all datasets in the available packages on your computer (i.e. also the not-loaded ones)
3: Using data(package = "packagename") will give you the datasets of that specific package, so data(package = "plyr") will give the datasets in the plyr package
If you want to know in which package a dataset is located (e.g. the acme dataset), you can do:
dat <- as.data.frame(data(package = .packages(all.available = TRUE))$results)
dat[dat$Item=="acme", c(1,3,4)]
which gives:
Package Item Title
107 boot acme Monthly Excess Returns
I often need to also know which structure of datasets are available, so I created dataStr in my misc package.
dataStr <- function(package="datasets", ...)
{
d <- data(package=package, envir=new.env(), ...)$results[,"Item"]
d <- sapply(strsplit(d, split=" ", fixed=TRUE), "[", 1)
d <- d[order(tolower(d))]
for(x in d){ message(x, ": ", class(get(x))); message(str(get(x)))}
}
dataStr()
Please mind that the output in the console is quite long.
This is the type of output:
[...]
warpbreaks: data.frame
'data.frame': 54 obs. of 3 variables:
$ breaks : num 26 30 54 25 70 52 51 26 67 18 ...
$ wool : Factor w/ 2 levels "A","B": 1 1 1 1 1 1 1 1 1 1 ...
$ tension: Factor w/ 3 levels "L","M","H": 1 1 1 1 1 1 1 1 1 2 ...
WorldPhones: matrix
num [1:7, 1:7] 45939 60423 64721 68484 71799 ...
- attr(*, "dimnames")=List of 2
..$ : chr [1:7] "1951" "1956" "1957" "1958" ...
..$ : chr [1:7] "N.Amer" "Europe" "Asia" "S.Amer" ...
WWWusage: ts
Time-Series [1:100] from 1 to 100: 88 84 85 85 84 85 83 85 88 89 ...
Edit: To get more informative output and use it for unloaded packages or all the packages on the search path, please use the revised online version with
source("https://raw.githubusercontent.com/brry/berryFunctions/master/R/dataStr.R")
Here is a comprehensive R packages datasets list maintained by Prof. Vincent Arel-Bundock.
https://vincentarelbundock.github.io/Rdatasets/
Rdatasets is a collection of 1892 datasets that were originally
distributed alongside the statistical software environment R and some
of its add-on packages. The goal is to make these data more broadly
accessible for teaching and statistical software development.
Run
help(package = "datasets")
in the R Studio console and you'll get all available datasets in the tidy Help tab on the right.
In a large dataset of US stocks I have a integer variable containing SIC codes. https://www.sec.gov/info/edgar/siccodes.htm
I would like to create a dummy variable indicating the major group of 50, i.e. a variable that takes on 1 for durable goods and 0 otherwise.
I tried the code:
data$durable <- as.integer(grepl(pattern = "50", x = data$sic))
But this, of course, does not take the hierarchical structure of SIC into account. I want to get the "50" only for the first two digits.
(New to R)
/Alex
Use either the division, or pad zero to left and check the first two letters.
code <- c(100, 102, 501, 5010)
# approach 1
as.integer(as.integer(code/100) == 50)
# approach 2
as.integer(substring(sprintf("%04d", code), 1, 2) == "50")
library(readxl)
library(dplyr)
library(stringi)
data_sic <- read_excel("./sic_example.xlsx")
data_sic$temp1 <- stri_sub(data_sic$SIC,1,2)
data_sic <- mutate(data_sic, durable_indicator =
ifelse(temp1 == "50", 1, 0))
str(data_sic)
Output:
str(data_sic)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 6 obs. of 4 variables:
$ SIC : num 4955 4961 4991 5000 5010 ...
$ Industry Title : chr "HAZARDOUS WASTE MANAGEMENT" "STEAM & AIR-CONDITIONING SUPPLY" "COGENERATION SERVICES & SMALL POWER PRODUCERS" "WHOLESALE-DURABLE GOODS" ...
$ temp1 : chr "49" "49" "49" "50" ...
$ durable_indicator: num 0 0 0 1 1 1
Addendum:
There are multiple ways to approach this problem.
I would suggest reviewing the stringi package Link to documentation for string editing.
As well as, the caret package - documentation for dummification of variables and other statistical transformations.
I need to do some computations with the data stored in a dataframe. ANd put the result into new column of this dataframe.
Initial dataframe:
> str(mydf)
'data.frame': 1122 obs. of 6 variables:
$ MMSI : num 2.73e+08 2.73e+08 2.73e+08 2.73e+08 2.73e+08 ...
$ MMSI.1 : num 2.73e+08 2.72e+08 2.72e+08 2.72e+08 6.67e+08 ...
$ LATITUDE : num 46.9 46.9 46.9 46.9 46.9 ...
$ LONGITUDE : num 32 32 32 32 32 ...
$ LATITUDE.1 : num 46.9 46.9 46.9 46.9 46.9 ...
$ LONGITUDE.1: num 32 32 32 32 32 ...
Now I need to add a new column which contain the result of operation with the data of the current raw..
running next code:
library(geosphere)
> mydf$distance <- with(mydf, distGeo(c(mydf$LONGITUDE,mydf$LATITUDE),c(mydf$LONGITUDE,mydf$LATITUDE)))
Error in .pointsToMatrix(p1) : Wrong length for a vector, should be 2
I understand that's the structure of the data for function distGeo should be different.
How to fix this error or how to change the code to get the distances between point in a new column?
Without having data to look at, it looks like you are trying to calculate the distance between a single point. The second point should likely include .1 at the end of the column name
library(geosphere)
mydf$distance <- with(mydf, distGeo(c(LONGITUDE, LATITUDE), c(LONGITUDE.1, LATITUDE.1)))
update
It looks like the error is that you're passing the entire data frame instead of each row individually. Try this
apply(mydf, 1, function(x) distGeo(x[c("LONGITUDE","LATITUDE")],x[c("LONGITUDE.1","LATITUDE.1")]))
Or just pass specific data columns to function as it accepts a matrix
distGeo(mydf[,c("LONGITUDE", "LATITUDE")], mydf[,c("LONGITUDE.1", "LATITUDE.1")])
I have a set of data frames belonging to many countries consisting of 3 variables (year, AI, OAD). The example for Zimbabwe is shown as below,
>str(dframe_Zimbabwe_1955_1970)
'data.frame': 16 obs. of 3 variables:
$ year: chr "1955" "1956" "1957" "1958" ...
$ AI : chr "11.61568161" "11.34114927" "11.23639317" "11.18841409" ...
$ OAD : chr "5.740789488" "5.775882473" "5.800441036" "5.822536579" ...
I am trying to change the data type of the variables in data frame to below so that I can model the linear fit using lm(dframe_Zimbabwe_1955_1970$AI ~ dframe_Zimbabwe_1955_1970$year).
>str(dframe_Zimbabwe_1955_1970)
'data.frame': 16 obs. of 3 variables:
$ year: int 1955 1956 1957 1958 ...
$ AI : num 11.61568161 11.34114927 11.23639317 11.18841409 ...
$ OAD : num 5.740789488 5.775882473 5.800441036 5.822536579 ...
The below static code able to change AI from character (chr) to numeric (num).
dframe_Zimbabwe_1955_1970$AI <- as.numeric(dframe_Zimbabwe_1955_1970$AI)
However when I tried to automate the code as below, AI still remains as character (chr)
countries <- c('Zimbabwe', 'Afghanistan', ...)
for (country in countries) {
assign(paste('dframe_',country,'_1955_1970$AI', sep=''), eval(parse(text = paste('as.numeric(dframe_',country,'_1955_1970$AI)', sep=''))))
}
Can you advice what I could have done wrong?
Thanks.
42: Your code doesn't work as written but with some edits it will. in addition to the missing parentheses and wrong sep, you can't use $'column name' in assign, but you don't need it anyway
for (country in countries) {
new_val <- get(paste( 'dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
remove(new_val)
}
proof it works:
dframe_Zimbabwe_1955_1970 <- data.frame(year = c("1955", "1956", "1957"),
AI = c("11.61568161", "11.34114927", "11.23639317"),
OAD = c("5.740789488", "5.775882473", "5.800441036"),
stringsAsFactors = F)
str(dframe_Zimbabwe_1955_1970)
'data.frame': 3 obs. of 3 variables:
$ year: chr "1955" "1956" "1957"
$ AI : chr "11.61568161" "11.34114927" "11.23639317"
$ OAD : chr "5.740789488" "5.775882473" "5.800441036"
countries <- 'Zimbabwe'
for (country in countries) {
new_val <- get(paste( 'dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
remove(new_val)
}
str(dframe_Zimbabwe_1955_1970)
'data.frame': 3 obs. of 3 variables:
$ year: num 1955 1956 1957
$ AI : num 11.6 11.3 11.2
$ OAD : num 5.74 5.78 5.8
It's going to be considered fairly ugly code by teh purists but perhaps this:
for (country in countries) {
new_val <- get(paste('dframe_',country,'_1955_1970', sep=''))
new_val[] <- lapply(new_val, as.numeric) # the '[]' on LHS keeps dataframe
assign(paste('dframe_',country,'_1955_1970', sep=''), new_val)
}
Using the get('obj_name') function is considered cleaner than eval(parse(text=...)). It would get handled more R-naturally had you assembled these dataframes in a list.