Key ordering vs. ordering of original columns with gather() - r

Does key ordering depend on whether I first list the columns to gather vs. those not to gather?
This is my data.frame:
library(tidyr)
wide_df <- data.frame(c("a", "b"), c("oh", "ah"), c("bla", "ble"), stringsAsFactors = FALSE)
colnames(wide_df) <- c("first", "second", "third")
wide_df
first second third
1 a oh bla
2 b ah ble
First I gather all columns in a specific order, and my ordering is respected in the key listing as second, first, although the columns are actually ordered as first, second:
long_01_df <- gather(wide_df, my_key, my_value, second, first, third)
long_01_df
my_key my_value
1 second oh
2 second ah
3 first a
4 first b
5 third bla
6 third ble
Then I decide to exclude one column from gathering:
long_02_df <- gather(wide_df, my_key, my_value, second, first, -third)
long_02_df
third my_key my_value
1 bla second oh
2 ble second ah
3 bla first a
4 ble first b
The keys are again ordered as second, first. Then I code it like this, believing to be doing the exact same thing:
long_03_df <- gather(wide_df, my_key, my_value, -third, second, first)
long_03_df
And I get the keys ordered according to the real column order in the original data.frame:
third my_key my_value
1 bla first a
2 ble first b
3 bla second oh
4 ble second ah
This behavior does not even change, when I call the function with factor_key = TRUE. What I am missing?

Summary
The reason for this is that you can not mix negative and positive indices. (You also should not: it simply makes no sense.) If you do that, gather() will ignore some of the indices.
Detailed answer
Also for standard indexing you are not allowed to mix positive and negative indices:
x <- 1:10
x[c(4, -2)]
## Error in x[c(4, -2)] : only 0's may be mixed with negative subscripts
It makes sense that this is the case: Indexing with 4 tells R to only keep the fourth element. There is no need to tell it explicitly to throw away the second element in addition.
According to the documentation of gather(), selecting columns works the same way as in dplyr's select(). So let's play with that. I'll work with a subset of mtcars:
mtcars <- mtcars[1:2, 1:5]
mtcars
## mpg cyl disp hp drat
## Mazda RX4 21.0 6 160 110 3.90
## Mazda RX4 Wag 21.0 6 160 110 3.90
You can use positive and negative indexing with select():
select(mtcars, mpg, cyl)
## mpg cyl
## Mazda RX4 21 6
## Mazda RX4 Wag 21 6
select(mtcars, -mpg, -cyl)
## disp hp drat
## Mazda RX4 160 110 3.9
## Mazda RX4 Wag 160 110 3.9
Also for select(), mixing positive and negative indices makes no sense. But instead of throwing an error, select() seems to ignore all indices that have a different sign than the first one:
select(mtcars, mpg, -hp, cyl)
## mpg cyl
## Mazda RX4 21 6
## Mazda RX4 Wag 21 6
select(mtcars, -mpg, hp, -cyl)
## disp hp drat
## Mazda RX4 160 110 3.9
## Mazda RX4 Wag 160 110 3.9
As you can see, the results are exactly the same as before.
For your examples with gather(), you use these two lines:
long_02_df <- gather(wide_df, my_key, my_value, second, first, -third)
long_03_df <- gather(wide_df, my_key, my_value, -third, second, first)
According to what I've shown above, these lines are identical to:
long_02_df <- gather(wide_df, my_key, my_value, second, first)
long_03_df <- gather(wide_df, my_key, my_value, -third)
Note that there is nothing in the second line that would indicate your preferred ordering of the keys. It only says that third should be omitted.

Related

Proper Syntax for Filtering Expressions for Arrow Datasets in R

I am attempting to use the arrow package (relatively recently implemented) DataSet API to to read a directory of files into memory, and leverage the c++ back-end to filter rows and columns. I would like to use the arrow package functions directly, not the wrapper functions for dplyr style verbs. These functions are very early in their lifecycle as of today, so I'm having a hard time tracking down some examples that illustrate the syntax.
In order to understand the syntax, I have created a very minimal example for testing. The first two queries work as expected.
library(arrow) ## version 4.0.0
write.csv(mtcars,"ArrowTest_mtcars/mtcars.csv")
## Define a dataset object
DS <- arrow::open_dataset(sources = "ArrowTest_mtcars", format = "text")
## Generate a basic scanner
AT <- DS$NewScan()$UseThreads()$Finish()$ToTable()
head(as.data.frame(AT), n = 3)
## mpg cyl disp hp drat wt qsec vs am gear carb
## 1 Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
## 2 Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
## 3 Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
## Generate a basic scanner with projection to select columns
AT <- DS$NewScan()$UseThreads()$Project(c("mpg","cyl"))$Finish()$ToTable()
head(as.data.frame(AT), n = 3)
# mpg cyl
#1 21.0 6
#2 21.0 6
#3 22.8 4
However, I have not yet been able to figure out the proper syntax to implement a filtering expression. I've tried a number of things, but my best guess still isn't working, and causes a segfault when I execute the Filt <- Expression$create(...) line.
## Generate a basic scanner with filtering where column `cyl` = 6
## My best guess at what might work, but causes a segfault instead
Filt <- Expression$create("==",args = list(Expression$field_ref("cyl"), Scalar$create(6L)))
AT <- DS$NewScan()$UseThreads()$Filter(Filt)$Finish()$ToTable()
head(as.data.frame(AT))
What is the proper syntax to implement row based filtering?
The documentation is quite awful on this. But a bit of trying and testing actually got me something that might lead you to the right answer. The problem I found was with Scalar$create and knowing which function to name to use:
Filt = Expression$create('or',
args = list(Expression$field_ref("cyl") == 6L,
Expression$field_ref('cyl') == 4L))
AT <- DS$NewScan()$UseThreads()$Filter(Filt)$Finish()$ToTable()
head(as.data.frame(AT))
However, note that for a single condition just using Expression$field_ref(...) == x works directly in filter
AT <- DS$NewScan()$UseThreads()$Filter(Expression$field_ref("cyl") == 6L)$Finish()$ToTable()
head(as.data.frame(AT))
help(open_dataset)
" Call ‘open_dataset()’ to point to a directory of data files and return a
‘Dataset’, then use ‘dplyr’ methods to query it."'
Reproducible example sharding the iris dataset by species, then opening only one shard for the setosa species.
library(arrow)
library(dplyr)
# Set up directory for examples
tf <- tempfile()
dir.create(tf)
on.exit(unlink(tf))
data <- dplyr::group_by(iris, Species)
write_dataset(data, tf)
# open only the setosa part
setosa <- open_dataset(tf) %>%
filter(Species == "setosa") %>%
collect()

Cannot use a variable named with numbers in R

I have some dataframes named as:
1_patient
2_patient
3_patient
Now I am not able to access its variables. For example:
I am not able to obtain:
2_patient$age
If I press tab when writing the name, it automatically gets quoted, but I am still unable to use it.
Do you know how can I solve this?
It is not recommended to name an object with numbers as prefix, but we can use backquote to extract the value from the object
`1_patient`$age
If there are more than object, we can use mget to return the objects in a list and then extract the 'age' column by looping over the list with lapply
mget(ls(pattern = "^\\d+_mtcars$"))
#$`1_mtcars`
# mpg cyl disp hp drat wt qsec vs am gear carb
#Mazda RX4 21 6 160 110 3.9 2.620 16.46 0 1 4 4
#Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
lapply(mget(ls(pattern = "^\\d+_patient$")), `[[`, 'age')
Using a small reproducible example
data(mtcars)
`1_mtcars` <- head(mtcars, 2)
1_mtcars$mpg
Error: unexpected input in "1_"
`1_mtcars`$mpg
#[1] 21 21

is there a way to use the ggplot aes callout without inputing the column name but by just inputting the column #?

EXAMPLE DATASET:
mtcars
mpg cyl disp hp drat wt ...
Mazda RX4 21.0 6 160 110 3.90 2.62 ...
Mazda RX4 Wag 21.0 6 160 110 3.90 2.88 ...
Datsun 710 22.8 4 108 93 3.85 2.32 ...
............
Recommended ggplot way:
ggplot(mtcars,aes(x=mpg)) + geom_histogram
They way I want to do it:
ggplot(mtcars,aes(x=[,1]) +geom_histogram
or
ggplot(mtcars,aes(x=[[1]]))+geom_histogram
Why can't ggplot let me call out my variable by its column? I need to call it out by column number not name. Why is ggplot so strict here? Any work around for this?
The problem you're facing is that the ggplot aes argument evaluates within the data.frame that you pass it. A column name is a string, and can't be properly evaluated the same way.
Fortunately, there is a solution: use the aes_string option, as follows:
library(ggplot2)
my_data <- mtcars
names(my_data)
ggplot(my_data, aes_string(x=names(my_data)[1]))+
geom_histogram()
This works because names(my_data)[1] returns a string, and is perfectly acceptable for the aes_string option.

R flag cases with missingness from regression analysis

When running a regression analysis in R (using glm) cases are removed due to 'missingness' of the data. Is there any way to flag which cases have been removed? I would ideally like to remove these from my original dataframe.
Many thanks
The model fit object returned by glm() records the row numbers of the data that it excludes for their incompleteness. They are a bit buried but you can retrieve them like this:
## Example data.frame with some missing data
df <- mtcars[1:6, 1:5]
df[cbind(1:5,1:5)] <- NA
df
# mpg cyl disp hp drat
# Mazda RX4 NA 6 160 110 3.90
# Mazda RX4 Wag 21.0 NA 160 110 3.90
# Datsun 710 22.8 4 NA 93 3.85
# Hornet 4 Drive 21.4 6 258 NA 3.08
# Hornet Sportabout 18.7 8 360 175 NA
# Valiant 18.1 6 225 105 2.76
## Fit an example model, and learn which rows it excluded
f <- glm(mpg~drat,weight=disp, data=df)
as.numeric(na.action(f))
# [1] 1 3 5
Alternatively, to get the row indices without having to fit the model, use the same strategy with the output of model.frame():
as.numeric(na.action(model.frame(mpg~drat,weight=disp, data=df)))
# [1] 1 3 5
Without a reproducible example I can't provide code tailored to your problem, but here's a generic method that should work. Assume your data frame is called df and your variables are called y, x1, x2, etc. And assume you want y, x1, x3, and x6 in your model.
# Make a vector of the variables that you want to include in your glm model
# (Be sure to include any weighting or subsetting variables as well, per Josh's comment)
glm.vars = c("y","x1","x3","x6")
# Create a new data frame that includes only those rows with no missing values
# for the variables that are in your model
df.glm = df[complete.cases(df[ , glm.vars]), ]
Also, if you want to see just the rows that have at least one missing value, do the following (note the addition of ! (the "not" operator)):
df[!complete.cases(df[ , glm.vars]), ]

Adding objects together in R (like ggplot layers)

I'm doing OOP R and was wondering how to make it so the + can be used to add custom objects together. The most common example of this I've found is in ggplot2 w/ adding geoms together.
I read through the ggplot2 source code and found this
https://github.com/hadley/ggplot2/blob/master/R/plot-construction.r
It looks like "%+%" is being used, but it's not clear how that eventually translates into the plain + operator.
You just need to define a method for the generic function +. (At the link in your question, that method is "+.gg", designed to be dispatched by arguments of class "gg"). :
## Example data of a couple different classes
dd <- mtcars[1, 1:4]
mm <- as.matrix(dd)
## Define method to be dispatched when one of its arguments has class data.frame
`+.data.frame` <- function(x,y) rbind(x,y)
## Any of the following three calls will dispatch the method
dd + dd
# mpg cyl disp hp
# Mazda RX4 21 6 160 110
# Mazda RX41 21 6 160 110
dd + mm
# mpg cyl disp hp
# Mazda RX4 21 6 160 110
# Mazda RX41 21 6 160 110
mm + dd
# mpg cyl disp hp
# Mazda RX4 21 6 160 110
# Mazda RX41 21 6 160 110

Resources