I have two dataframes with each two columns c("price", "size") with different lengths.
Each price must be linked to its size. It's two lists of trade orders. I have to discover the differences between the two dataframes knowing that the two databases can have orders that the other doesn't have and vice versa. I would like an output with the differences or two outputs, it doesn't matter. But I need the row number in the output to find where are the differences in the series.
Here is sample data :
> out
price size
1: 36024.86 0.01431022
2: 36272.00 0.00138692
3: 36272.00 0.00277305
4: 36292.57 0.05420000
5: 36292.07 0.00403948
---
923598: 35053.89 0.30904890
923599: 35072.76 0.00232000
923600: 35065.60 0.00273000
923601: 35049.36 0.01760000
923602: 35037.23 0.00100000
>bit
price size
1: 37279.89 0.01340020
2: 37250.84 0.00930000
3: 37250.32 0.44284049
4: 37240.00 0.00056491
5: 37215.03 0.99891906
---
923806: 35053.89 0.30904890
923807: 35072.76 0.00232000
923808: 35065.60 0.00273000
923809: 35049.36 0.01760000
923810: 35037.23 0.00100000
For example, I need to know if the first row of the database out is in the database bit.
I've tried many functions : comparedf()
summary(comparedf(bit, out, by = c("price","size"))
but I've got error:
Error in vecseq(f__, len__, if (allow.cartesian || notjoin ||
!anyDuplicated(f__, :
I've tried compare_df() :
compareout=compare_df(out,bit,c("price","size"))
But I know the results are wrong, I've only 23 results and I know that there are more than 200 differences minimum.
I've tried match(), which() functions but it doesn't get the results I search.
If you have any other methods, I will take them.
Perhaps you could just do inner_join on out and bit by price and size? But first make id variable for both data.frame's
library(dplyr)
out$id <- 1:nrow(out)
bit$id <- 1:nrow(bit)
joined <- inner_join(bit, out, by = c("price", "size"))
Now we can check which id from out and bit are not present in joined table:
id_from_bit_not_included_in_out <- bit$id[!bit$id %in% joined$id.x]
id_from_out_not_included_in_bit <- out$id[!out$id %in% joined$id.y]
And these ids are the rows not included in out or bit, i.e. variable id_from_bit_not_included_in_out contains rows present in bit, but not in out and variable id_from_out_not_included_in_bit contains rows present in out, but not in bit
First attempt here. It will be difficult to do a very clean job with this data tho.
The data I used:
out <- read.table(text = "price size
36024.86 0.01431022
36272.00 0.00138692
36272.00 0.00277305
36292.57 0.05420000
36292.07 0.00403948
35053.89 0.30904890
35072.76 0.00232000
35065.60 0.00273000
35049.36 0.01760000
35037.23 0.00100000", header = T)
bit <- read.table(text = "price size
37279.89 0.01340020
37250.84 0.00930000
37250.32 0.44284049
37240.00 0.00056491
37215.03 0.99891906
37240.00 0.00056491
37215.03 0.99891906
35053.89 0.30904890
35072.76 0.00232000
35065.60 0.00273000
35049.36 0.01760000
35037.23 0.00100000", header = T)
Assuming purely that row 1 of out should match with row 1 of bit a simple solution could be:
df <- cbind(distinct(out), distinct(bit))
names(df) <- make.unique(names(df))
However judging from the data you have provided I am not sure if this is the way to go (big differences in the first few rows) so maybe try sorting the data first?:
df <- cbind(distinct(out[order(out$price, out$size),]), distinct(bit[order(bit$price, bit$size),]))
names(df) <- make.unique(names(df))
Related
I'm trying to convert dat data to data frame by converting the data and applying xml functions to extract. However facing the error different number of rows. The attached data was first reduced from the dat and converted to string. As being a beginner to R any help will be highly appreciated
Data:
dat <- '<d2lm:d2LogicalModel extensionVersion="2.0" extensionName="NTIS Published Services"
modelBaseVersion="2" xmlns:ns4="http://www.thalesgroup.com/NTIS/Datex2Extensions/1.0Beta1"
xmlns:ns3="http://datex2.eu/schema/2/2_0/inrix" xmlns:d2lm="http://datex2.eu/schema/2/2_0">
<d2lm:exchange><d2lm:supplierIdentification><d2lm:country>gb</d2lm:country>
<d2lm:nationalIdentifier>NTIS</d2lm:nationalIdentifier></d2lm:supplierIdentification></d2lm:exchange>
<d2lm:payloadPublication xsi:type="d2lm:SituationPublication" lang="en"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><d2lm:feedType>Event Data</d2lm:feedType>
<d2lm:publicationTime>2020-05-10T00:00:44.778+01:00</d2lm:publicationTime><d2lm:publicationCreator>
<d2lm:country>gb</d2lm:country><d2lm:nationalIdentifier>NTIS</d2lm:nationalIdentifier>
</d2lm:publicationCreator><d2lm:situation version="" id="2922904"><d2lm:headerInformation>
<d2lm:areaOfInterest>national</d2lm:areaOfInterest>
<d2lm:confidentiality>restrictedToAuthoritiesTrafficOperatorsAndPublishers</d2lm:confidentiality>
<d2lm:informationStatus>real</d2lm:informationStatus></d2lm:headerInformation><d2lm:situationRecord
xsi:type="d2lm:RoadOrCarriagewayOrLaneManagement" version="" id="2922904">
<d2lm:situationRecordCreationReference>UF-20-05-09-600215</d2lm:situationRecordCreationReference>
<d2lm:situationRecordCreationTime>2020-05-09T21:04:28.000+01:00</d2lm:situationRecordCreationTime>
<d2lm:situationRecordVersionTime>2020-05-10T00:00:39.677+01:00</d2lm:situationRecordVersionTime>
<d2lm:probabilityOfOccurrence>certain</d2lm:probabilityOfOccurrence>
<d2lm:severity>highest</d2lm:severity><d2lm:source>
<d2lm:sourceIdentification>RCCs</d2lm:sourceIdentification><d2lm:sourceExtension>
<d2lm:sourceSituation><d2lm:sourceSituationId>SL</d2lm:sourceSituationId>
<d2lm:sourceSituationCreationTime>2020-05-09T20:48:00.000+01:00</d2lm:sourceSituationCreationTime>
</d2lm:sourceSituation></d2lm:sourceExtension></d2lm:source><d2lm:validity>
<d2lm:validityStatus>active</d2lm:validityStatus><d2lm:validityTimeSpecification>
<d2lm:overallStartTime>2020-05-09T21:03:19.755+01:00</d2lm:overallStartTime>
<d2lm:overallEndTime>2020-05-10T02:41:00.000+01:00</d2lm:overallEndTime>
</d2lm:validityTimeSpecification></d2lm:validity><d2lm:impact>
<d2lm:capacityRemaining>0.0</d2lm:capacityRemaining>
<d2lm:numberOfLanesRestricted>0</d2lm:numberOfLanesRestricted>
<d2lm:numberOfOperationalLanes>0</d2lm:numberOfOperationalLanes>
<d2lm:originalNumberOfLanes>3</d2lm:originalNumberOfLanes>
<d2lm:trafficConstrictionType>carriagewayBlocked</d2lm:trafficConstrictionType><d2lm:impactExtension>
<d2lm:impactDetails><d2lm:changeInCapacity>-100.0</d2lm:changeInCapacity><d2lm:individualLanesStatus>
<d2lm:individualLane><d2lm:laneIdentifier>hardShoulder</d2lm:laneIdentifier>
<d2lm:laneStatus>closed</d2lm:laneStatus></d2lm:individualLane><d2lm:individualLane>
<d2lm:laneIdentifier>lane1</d2lm:laneIdentifier><d2lm:laneStatus>closed</d2lm:laneStatus>
</d2lm:individualLane><d2lm:individualLane><d2lm:laneIdentifier>lane2</d2lm:laneIdentifier>
<d2lm:laneStatus>closed</d2lm:laneStatus></d2lm:individualLane><d2lm:individualLane>
<d2lm:laneIdentifier>lane3</d2lm:laneIdentifier><d2lm:laneStatus>closed</d2lm:laneStatus>
</d2lm:individualLane></d2lm:individualLanesStatus><d2lm:returnToNormalStatus>
<d2lm:predictedTimeToClear>2020-05-10T02:41:03.536+01:00</d2lm:predictedTimeToClear>
<d2lm:predictedReturnToProfile>2020-05-10T02:41:00.000+01:00</d2lm:predictedReturnToProfile>
</d2lm:returnToNormalStatus></d2lm:impactDetails></d2lm:impactExtension></d2lm:impact>
<d2lm:generalPublicComment><d2lm:comment><d2lm:values><d2lm:value>Traffic is being diverted via the
exit and entry slips</d2lm:value></d2lm:values></d2lm:comment>
<d2lm:commentType>internalNote</d2lm:commentType></d2lm:generalPublicComment>
<d2lm:generalPublicComment><d2lm:comment><d2lm:values><d2lm:value>TYPE : GDP</d2lm:value><
<d2lm:value>Location : The M5 southbound at junction J27 </d2lm:value><d2lm:value>Reason : Road
Management</d2lm:value><d2lm:value>Status : Currently Active</d2lm:value><d2lm:value>Time To Clear :
The event is expected to clear between 02:30 and 02:45 on 10 May 2020</d2lm:value><d2lm:value>Return
To Normal : Normal traffic conditions are expected between 02:30 and 02:45 on 10 May
2020</d2lm:value><d2lm:value>Lanes Closed : All lanes are closed</d2lm:value></d2lm:values>
</d2lm:comment></d2lm:generalPublicComment><d2lm:groupOfLocations
xsi:type="d2lm:NonOrderedLocationGroupByList"><d2lm:locationContainedInGroup xsi:type="d2lm:Point">
<d2lm:locationForDisplay><d2lm:latitude>50.9196</d2lm:latitude>
<d2lm:longitude>-3.3536696</d2lm:longitude></d2lm:locationForDisplay></d2lm:locationContainedInGroup>
<d2lm:locationContainedInGroup xsi:type="d2lm:LocationByReference"><d2lm:predefinedLocationReference
targetClass="PredefinedLocation" version="12.10" id="101002501"/></d2lm:locationContainedInGroup>
</d2lm:groupOfLocations><d2lm:management><d2lm:lifeCycleManagement><d2lm:cancel>false</d2lm:cancel>
<d2lm:end>false</d2lm:end></d2lm:lifeCycleManagement><d2lm:managementExtension>
<d2lm:situationManagement>
<d2lm:confirmedBy>RCCs</d2lm:confirmedBy><d2lm:areaTeamInvolved>Area 2</d2lm:areaTeamInvolved>
<d2lm:rccInformation><d2lm:allocatedRcc>South West RCC</d2lm:allocatedRcc>
<d2lm:rccSituationId>1004</d2lm:rccSituationId></d2lm:rccInformation><d2lm:emergencyServicesInvolved>
<d2lm:emergencyServiceStatus><d2lm:serviceType>police</d2lm:serviceType>
<d2lm:serviceStatus>attending</d2lm:serviceStatus></d2lm:emergencyServiceStatus>
<d2lm:emergencyServiceStatus><d2lm:serviceType>fire</d2lm:serviceType>
<d2lm:serviceStatus>none</d2lm:serviceStatus></d2lm:emergencyServiceStatus>
<d2lm:emergencyServiceStatus>
<d2lm:serviceType>ambulance</d2lm:serviceType><d2lm:serviceStatus>none</d2lm:serviceStatus>
</d2lm:emergencyServiceStatus><d2lm:emergencyServiceStatus>
<d2lm:serviceType>airAmbulance</d2lm:serviceType><d2lm:serviceStatus>none</d2lm:serviceStatus>
</d2lm:emergencyServiceStatus></d2lm:emergencyServicesInvolved><d2lm:peopleAndVehiclesInvolved>
<d2lm:totalNumberOfPeopleInvolved>0</d2lm:totalNumberOfPeopleInvolved>
<d2lm:totalNumberOfVehiclesInvolved>0</d2lm:totalNumberOfVehiclesInvolved>
<d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles><d2lm:vehicleCharacteristics>
<d2lm:vehicleType>articulatedVehicle</d2lm:vehicleType></d2lm:vehicleCharacteristics>
</d2lm:groupOfVehiclesInvolved><d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles><d2lm:vehicleCharacteristics>
<d2lm:vehicleType>lorry</d2lm:vehicleType></d2lm:vehicleCharacteristics>
</d2lm:groupOfVehiclesInvolved><d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles><d2lm:vehicleCharacteristics>
<d2lm:vehicleType>car</d2lm:vehicleType></d2lm:vehicleCharacteristics>
</d2lm:groupOfVehiclesInvolved><d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles><d2lm:vehicleCharacteristics>
<d2lm:vehicleType>motorcycle</d2lm:vehicleType></d2lm:vehicleCharacteristics>
</d2lm:groupOfVehiclesInvolved><d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles><d2lm:vehicleCharacteristics>
<d2lm:vehicleType>other</d2lm:vehicleType></d2lm:vehicleCharacteristics>
</d2lm:groupOfVehiclesInvolved><d2lm:groupOfVehiclesInvolved>
<d2lm:numberOfVehicles>0</d2lm:numberOfVehicles>
<d2lm:vehicleCharacteristics><d2lm:vehicleType>trailer</d2lm:vehicleType>
</d2lm:vehicleCharacteristics></d2lm:groupOfVehiclesInvolved><d2lm:groupOfPeopleInvolved>
<d2lm:numberOfPeople>0</d2lm:numberOfPeople><d2lm:injuryStatus>slightlyInjured</d2lm:injuryStatus>
</d2lm:groupOfPeopleInvolved><d2lm:groupOfPeopleInvolved><d2lm:numberOfPeople>0</d2lm:numberOfPeople>
<d2lm:injuryStatus>seriouslyInjured</d2lm:injuryStatus></d2lm:groupOfPeopleInvolved>
</d2lm:peopleAndVehiclesInvolved></d2lm:situationManagement></d2lm:managementExtension>
</d2lm:management>
<d2lm:complianceOption>mandatory</d2lm:complianceOption>
<d2lm:roadOrCarriagewayOrLaneManagementType>other</d2lm:roadOrCarriagewayOrLaneManagementType>
</d2lm:situationRecord></d2lm:situation></d2lm:payloadPublication></d2lm:d2LogicalModel>
'
Code So Far:
library(XML)
require(plyr)
library(stringr)
datDF <- data.frame(
tags = unlist(str_extract_all(dat, "<([^>]*)>(?=[^>]*</\\1>)")),
values = unlist(str_extract_all(dat, "(?<=<([^>]{1,100})>).*(?=</\\1>)"))
)
datDF
Many thanks
All variables in a data.frame need to have the same length. tags and values differ in length. You can make them the same length by adding NAs to the shorter vector and then combine them:
library(XML)
require(plyr)
library(stringr)
library(xml2)
tags = unlist(str_extract_all(dat, "<([^>]*)>(?=[^>]*</\\1>)"))
values = unlist(str_extract_all(dat, "(?<=<([^>]{1,100})>).*(?=</\\1>)"))
values <- c(values, rep(NA, length(tags)-length(values)))
datDF <- data.frame(
tags,
values
)
But I would be careful, because this assumes that you parsed the document correctly. I think this is not the case since the tag roadOrCarriagewayOrLaneManagementType is on the second-to-last line but the corresponding value "other" is on the last line of value.
All.
I've been trying to solve a problem on a large data set for some time and could use some of your wisdom.
I have a DF (1.3M obs) with a column called customer along with 30 other columns. Let's say it contains multiple instances of customers Customer1 thru Customer3000. I know that I have issues with 30 of those customers. I need to find all the customers that are NOT the customers I have issues and replace the value in the 'customer' column with the text 'Supported Customer'. That seems like it should be a simple thing...if it werent for the number of obs, I would have loaded it up in Excel, filtered all the bad customers out and copy/pasted the text 'Supported Customer' over what remained.
Ive tried replace and str_replace_all using grepl and paste/paste0 but to no avail. my current code looks like this:
#All the customers that have issues
out <- c("Customer123", "Customer124", "Customer125", "Customer126", "Customer127",
"Customer128", ..... , "Customer140")
#Look for everything that is NOT in the list above and replace with "Enabled"
orderData$customer <- str_replace_all(orderData$customer, paste0("[^", paste(out, collapse =
"|"), "]"), "Enabled Customers")
That code gets me this error:
Error in stri_replace_all_regex(string, pattern, fix_replacement(replacement), :
In a character range [x-y], x is greater than y. (U_REGEX_INVALID_RANGE)
I've tried the inverse of this approach and pulled a list of all obs that dont match the list of out customers. Something like this:
in <- orderData %>% filter(!customer %in% out) %>% select(customer) %>%
distinct(customer)
This gets me a much larger list of customers that ARE enabled (~3,100). Using the str_replace_all and paste approach seems to have issues though. At this large number of patterns, paste no longer collapses using the "|" operator. instead I get a string that looks like:
"c(\"Customer1\", \"Customer2345\", \"Customer54\", ......)
When passed into str_replace_all, this does not match any patterns.
Anyways, there's got to be an easier way to do this. Thanks for any/all help.
Here is a data.table approach.
First, some example data since you didn't provide any.
customer <- sample(paste0("Customer",1:300),5000,replace = TRUE)
orderData <- data.frame(customer = sample(paste0("Customer",1:300),5000,replace = TRUE),stringsAsFactors = FALSE)
orderData <- cbind(orderData,matrix(runif(0,100,n=5000*30),ncol=30))
out <- c("Customer123", "Customer124", "Customer125", "Customer126", "Customer127", "Customer128","Customer140")
library(data.table)
setDT(orderData)
result <- orderData[!(customer %in% out),customer := gsub("Customer","Supported Customer ",customer)]
result
customer 1 2 3 4 5 6 7 8 9
1: Supported Customer 134 65.35091 8.57117 79.594166 84.88867 97.225276 84.563997 17.15166 41.87160 3.717705
2: Supported Customer 225 72.95757 32.80893 27.318046 72.97045 28.698518 60.709381 92.51114 79.90031 7.311200
3: Supported Customer 222 39.55269 89.51003 1.626846 80.66629 9.983814 87.122153 85.80335 91.36377 14.667535
4: Supported Customer 184 24.44624 20.64762 9.555844 74.39480 49.189537 73.126275 94.05833 36.34749 3.091072
5: Supported Customer 194 42.34858 16.08034 34.182737 75.81006 35.167769 23.780069 36.08756 26.46816 31.994756
---
i've gone through several answers and tried the following but each either yields an error or an un-wanted result:
here's the data:
Network Campaign
Moburst_Chartboost Test Campaign
Moburst_Chartboost Test Campaign
Moburst_Appnext unknown
Moburst_Appnext 1065
i'd like to replace "Test Campaign" with "1055" whenever "Network" == "Moburst_Chartboost". i realize this should be very simple but trying out these:
dataset = read.csv('C:/Users/User/Downloads/example.csv')
for( i in 1:nrow(dataset)){
if(dataset$Network == 'Moburst_Chartboost') dataset$Campaign <- '1055'
}
this yields an error: Warning messages:
1: In if (dataset$Network == "Moburst_Chartboost") dataset$Campaign <- "1055" :
the condition has length > 1 and only the first element will be used
2: In if (dataset$Network == "Moburst_Chartboost") dataset$Campaign <- "1055" :
the condition has length > 1 and only the first element will be used
etc.
then i tried:
within(dataset, {
dataset$Campaign <- ifelse(dataset$Network == 'Moburst_Chartboost', '1055', dataset$Campaign)
})
this turned ALL 4 values in row "Campaign" into "1055" over running what was there even when condition isn't met
also this:
dataset$Campaign[which(dataset$Network == 'Moburst_Chartboost')] <- 1055
yields this error, and replaced the values in the two first rows of "Campaign" with NA:
Warning message:
In `[<-.factor`(`*tmp*`, which(dataset$Network == "Moburst_Chartboost"), :
invalid factor level, NA generated
scratching my head here. new to R but this shouldn't be so hard :(
In your first attempt, you're trying to iterate over all the columns, when you only want to change the 2nd column.
In your second, you're trying to assign the value "1055" to all of the 2nd column.
The way to think about it is as an if else, where if the condition in col 1 is met, col 2 is changed, otherwise it remains the same.
dataset <- data.frame(Network = c("Moburst_Chartboost", "Moburst_Chartboost",
"Moburst_Appnext", "Moburst_Appnext"),
Campaign = c("Test Campaign", "Test Campaign",
"unknown", "1065"))
dataset$Campaign <- ifelse(dataset$Network == "Moburst_Chartboost",
"1055",
dataset$Campaign)
head(dataset)
Network Campaign
1 Moburst_Chartboost 1055
2 Moburst_Chartboost 1055
3 Moburst_Appnext unknown
4 Moburst_Appnext 1065
You may also try dataset$Campaign[dataset$Campaign=="Test Campaign"]<-1055 to avoid the use of loops and ifelse statements.
Where dataset
dataset <- data.frame(Network = c("Moburst_Chartboost", "Moburst_Chartboost",
"Moburst_Appnext", "Moburst_Appnext"),
Campaign = c("Test Campaign", "Test Campaign",
"unknown", 1065))
Try the following
dataset = read.csv('C:/Users/User/Downloads/example.csv', stringsAsFactors = F)
for( i in 1:nrow(dataset)){
if(dataset$Network[i] == 'Moburst_Chartboost') dataset$Campaign[i] <- '1055'
}
It seems your forgot the index variable. Without [i] you work on the whole vector of the data frame, resulting in the error/warning you mentioned.
Note that I added stringsAsFactors = F to the read.csv() function to make sure the strings are indeed interpreted as strings and not factors. Using factors this would result in an error like this
In `[<-.factor`(`*tmp*`, i, value = c(NA, 2L, 3L, 1L)) :
invalid factor level, NA generated
Alternatively you can do the following without using a for loop:
idx <- which(dataset$Network == 'Moburst_Chartboost')
dataset$Campaign[idx] <- '1055'
Here, idx is a vector containing the positions where Network has the value 'Moburst_Chartboost'
thank you for the help! not elegant, but since this lingered with me when going to sleep last night i decided to try to bludgeon this with some ugly code but it worked too - just as a workaround...separated to two data frames, replaced all values and then binded back...
# subsetting only chartboost
chartboost <- subset(dataset, dataset$Network=='Moburst_Chartboost')
# replace all values in Campaign
chartboost$Campaign <-sub("^.*", "1055",chartboost$Campaign)
#subsetting only "not chartboost"
notChartboost <-subset(dataset, dataset$Network!='Moburst_Chartboost')
# binding back to single dataframe
newSet <- rbind(chartboost, notChartboost)
Ugly as a duckling but worked :)
Instead of showing more2 or less2 in the column, it only shows up as NA. Why aren't the character names appearing instead?
careermore2 <- vector(mode="character",length=length(mlb$careeryrs))
"less2" <- careermore2[mlb$careeryrs<=2]
"more2" <- careermore2[mlb$careeryrs>=2]
No.seasons <- factor(careermore2,levels=c("more2","less2"),exclude=NA,ordered=TRUE)
mlb2 <- cbind(mlb,No.seasons)
str(mlb2$No.seasons)
head(mlb2$No.seasons)
mlb2[mlb2$No.seasons=="more2",]
Looking at careermore2 I would say you've got these the wrong way round:
"less2" <- careermore2[mlb$careeryrs<=2]
"more2" <- careermore2[mlb$careeryrs>=2]
That creates two objects. You really meant:
careermore2[mlb$careeryrs<=2] = "less2"
careermore2[mlb$careeryrs>=2] = "more2"
ie set the corresponding values in careermore2. And you probably want <2 or >2 rather than have = in both...
I use data from the NHANES periodontal dataset (https://wwwn.cdc.gov/Nchs/Nhanes/2009-2010/OHXPER_F.htm) and after cleaning it to only keep the "pc" variables, I have a df=setPD 168 columns that include 6 measurements (pcd, pcm, pcs, pcp, pcl, pca) around 28 teeth numbered from #02 to #31
#names(setPD)
[1] "ohx02pcd" "ohx02pcm" "ohx02pcs" "ohx02pcp" "ohx02pcl" "ohx02pca" "ohx03pcd" "ohx03pcm" "ohx03pcs" "ohx03pcp" "ohx03pcl" "ohx03pca"
[13] "ohx04pcd" "ohx04pcm" "ohx04pcs" "ohx04pcp" "ohx04pcl" "ohx04pca" "ohx05pcd" "ohx05pcm" "ohx05pcs" "ohx05pcp" "ohx05pcl" "ohx05pca"
[25] "ohx06pcd" "ohx06pcm" "ohx06pcs" "ohx06pcp" "ohx06pcl" "ohx06pca" "ohx07pcd" "ohx07pcm" "ohx07pcs" "ohx07pcp" "ohx07pcl" "ohx07pca"
[37] "ohx08pcd" "ohx08pcm" "ohx08pcs" "ohx08pcp" "ohx08pcl" "ohx08pca" "ohx09pcd" "ohx09pcm" "ohx09pcs" "ohx09pcp" "ohx09pcl" "ohx09pca"
[49] "ohx10pcd" "ohx10pcm" "ohx10pcs" "ohx10pcp" "ohx10pcl" "ohx10pca" "ohx11pcd" "ohx11pcm" "ohx11pcs" "ohx11pcp" "ohx11pcl" "ohx11pca"
[61] "ohx12pcd" "ohx12pcm" "ohx12pcs" "ohx12pcp" "ohx12pcl" "ohx12pca" "ohx13pcd" "ohx13pcm" "ohx13pcs" "ohx13pcp" "ohx13pcl" "ohx13pca"
[73] "ohx14pcd" "ohx14pcm" "ohx14pcs" "ohx14pcp" "ohx14pcl" "ohx14pca" "ohx15pcd" "ohx15pcm" "ohx15pcs" "ohx15pcp" "ohx15pcl" "ohx15pca"
[85] "ohx18pcd" "ohx18pcm" "ohx18pcs" "ohx18pcp" "ohx18pcl" "ohx18pca" "ohx19pcd" "ohx19pcm" "ohx19pcs" "ohx19pcp" "ohx19pcl" "ohx19pca"
[97] "ohx20pcd" "ohx20pcm" "ohx20pcs" "ohx20pcp" "ohx20pcl" "ohx20pca" "ohx21pcd" "ohx21pcm" "ohx21pcs" "ohx21pcp" "ohx21pcl" "ohx21pca"
[109] "ohx22pcd" "ohx22pcm" "ohx22pcs" "ohx22pcp" "ohx22pcl" "ohx22pca" "ohx23pcd" "ohx23pcm" "ohx23pcs" "ohx23pcp" "ohx23pcl" "ohx23pca"
[121] "ohx24pcd" "ohx24pcm" "ohx24pcs" "ohx24pcp" "ohx24pcl" "ohx24pca" "ohx25pcd" "ohx25pcm" "ohx25pcs" "ohx25pcp" "ohx25pcl" "ohx25pca"
[133] "ohx26pcd" "ohx26pcm" "ohx26pcs" "ohx26pcp" "ohx26pcl" "ohx26pca" "ohx27pcd" "ohx27pcm" "ohx27pcs" "ohx27pcp" "ohx27pcl" "ohx27pca"
[145] "ohx28pcd" "ohx28pcm" "ohx28pcs" "ohx28pcp" "ohx28pcl" "ohx28pca" "ohx29pcd" "ohx29pcm" "ohx29pcs" "ohx29pcp" "ohx29pcl" "ohx29pca"
[157] "ohx30pcd" "ohx30pcm" "ohx30pcs" "ohx30pcp" "ohx30pcl" "ohx30pca" "ohx31pcd" "ohx31pcm" "ohx31pcs" "ohx31pcp" "ohx31pcl" "ohx31pca"
I am trying to apply a conditional selection in each group of six columns. This is:
transmute(setPD,PD02 = ifelse(setPD$ohx02pcd >5 |
setPD$ohx02pcm>5 |setPD$ohx02pcs >5|
setPD$ohx02pcp >5 | setPD$ohx02pcl >5 |
setPD$ohx02pca >5, 1, 0))
Then for the next tooth (03) I have to write again:
transmute(setPD,PD03 = ifelse(setPD$ohx03pcd >5 |
setPD$ohx03pcm>5|setPD$ohx03pcs >5|
setPD$ohx03pcp >5|setPD$ohx03pcl >5|
setPD$ohx03pca >5, 1, 0))
I tried to firstly do that conditional selection in a more efficient way, something like:
transmute(setPD,PD02 = ifelse(list(setPD$ohx02pcd:setPD$ohx02pcp) >5, 1, 0))
but it does not work.
Then I am looking for a way to write a loop that does that over each tooth without needing to write this 28 times!!
I thought of applying the select function of dplyr in a for loop but I don't know how to do that.
At the end I want to get all the new columns I made with transmute and say that if at least 2 of the 28 columns are 1, then I have disease, if <2 are 1 then I have health. ANy help would be appreciated.
**Note: If you want to get the dataset, it is open access from CDC.org:
https://wwwn.cdc.gov/Nchs/Nhanes/2009-2010/OHXPER_F.htm **
First, it is useful to point out that the logical statements of the form is A true OR is B true OR is C true are equivalent to asking is ANY of A,B,C true? We can use this to simplify the statements setPD$ohx02pcd >5 | setPD$ohx02pcm>5 |setPD$ohx02pcs >5| ... to ask if for any of these columns it is true that their value is larger than 5.
For example, let us focus on tooth number 02 first. To get all columns that concern this tooth, we can use grep to get a vector of column names. This can be achieved with
current_tooth <- grep("02", names(setPD), value = T)
Note that if there are any other columns in the data that contain the string 02, these columns will also show up. This does not appear to be the case in your data, but it is worthwhile pointing out here in case someone else uses it and this applies in other datasets.
Now, we can use these names to subset the dataframe. For instance,
setPD[,current_tooth]
will give you the corresponding columns. In each row, we want to check if any of the above mentioned conditions are true. Given a vector of logical statements, we can check if any of them is true with the function any. To go through a dataframe by row and apply a function, we can use apply, such as in
setPD$PD02 <-
apply(setPD[,grep("02", names(setPD), value = T)], 1, function(x) any(x>5))
Now, the above applies to one tooth only, namely 02. One way of doing it for all teeth is to create a vector with all tooth indicators and use this to loop over the above lines, replacing the "02" in the above grep call in each iteration and using assign or something similar to get the variable name right. A more elegant and more efficient way is to use the same principle on long data. Consider the following:
library(reshape2)
library(dplyr)
m <- melt(setPD, id.vars="SEQN")
m$num <- substr(m$variable, 4,5) # be careful here and check output!
m <- m %>% group_by(num) %>% mutate(PS = any(value>5))
m$num <- paste0("PS", m$num)
md <- dcast(m, SEQN ~ num, value.var = "PS")
setPD <- merge(setPD, md, by="SEQN")
This melts your data first and creates a variable num that indicates your tooth. Again, make sure that this works. I have used the fact that in your data, the tooth number all appear in the 4th and 5th place in the character string. Make sure this is true, and adjust the code otherwise. Then I create a variable PS which indicates whether any of the columns that contain the tooth identifer has a value larger than 5. Last but not least I recast the data so that you have the values of PD02, PD03, etc in columns again, before I merge this to the old dataset. The line with paste0 merely creates the variable names that you want to have.