Recently, I've faced a behaviour in table function that was not what I was expected:
For example, let take the following vector:
ex_vec <- c("Non", "Non", "Nan", "Oui", "NaN", NA)
If I check for NA values in my vector, "NaN" is not considered one (as expected):
is.na(ex_vec)
# [1] FALSE FALSE FALSE FALSE FALSE TRUE
But if I tried to get the different values frequencies:
table(ex_vec)
#ex_vec
#Nan Non Oui
# 1 2 1
"NaN" does not appear in the table.
However, if I "ask" table to show the NA values, I get this:
table(ex_vec, useNA="ifany")
#ex_vec
# Nan NaN Non Oui <NA>
# 1 1 2 1 1
So, the character strings "NaN" is treated as a NA value inside table call, while being treated in the ouput as a not NA value.
I know (it would be better and) I could solve my problem by converting my vector to a factor but nonetheless, I'd really like to know what's going on here. Does anyone have an idea?
When factor matches up levels for a vector it converts its exclude list to the same type as the input vector:
exclude <- as.vector(exclude, typeof(x))
so if your exclude list has NaN and your vector is character, this happens:
as.vector(exclude, typeof(letters))
[1] NA "NaN"
Oh dear. Now the real "NaN" strings will be excluded.
To fix, use exclude=NA in table (and factor if you are making factors that might hit this).
I do love this in the docs for factor:
There are some anomalies associated with factors that have ‘NA’ as
a level. It is suggested to use them sparingly, e.g., only for
tabulation purposes.
Reassuring...
First idea coming to my mind was to have a look at table definition which start by:
> table
function (..., exclude = if (useNA == "no") c(NA, NaN), useNA = c("no",
"ifany", "always"), dnn = list.names(...), deparse.level = 1)
{
Sounds logical, by default table exclude NA and NaN.
Digging within table code we see that if xis not a factor it coerce it to a factor (nothing new here, it's said in the doc).
else {
a <- factor(a, exclude = exclude)
I didn't find anything else which could have impacted the input to coerce "NaN" into NA values.
So looking into factor to get the why we find the root cause:
> factor
function (x = character(), levels, labels = levels, exclude = NA,
ordered = is.ordered(x), nmax = NA)
{
[...] # Snipped for brievety
exclude <- as.vector(exclude, typeof(x))
x <- as.character(x)
levels <- levels[is.na(match(levels, exclude))] # defined in the snipped part above, is the sorted unique values of input vector, coerced to char.
f <- match(x, levels)
[...]
f
}
Here we got it, the exclude parameter, even being NA values is coerced into a character vector.
So what happens is:
> ex_vec <- c("Non", "Non", "Nan", "Oui", "NaN", NA)
> excludes<-c(NA,NaN)
> as.vector(excludes,"character")
[1] NA "NaN"
> match(ex_vec,as.vector(excludes,"character"))
[1] NA NA NA NA 2 1
We do match character "NaN" as the exclude vector as been coerced to character before comparison.
Related
assume we have data.table d1 with 6 rows:
d1 <- data.table(v1 = c(1,2,3,4,5,6), v2 = c(5,5,5,5,5,5))
we add a column to d1 called test, and fill it with NA
d1$test <- NA
the external vector rows gives the index of rows we want to fill with values contained in vals
rows <- c(5,6)
vals <- c(6,3)
how do you do this in data table syntax? i have not been able to figure this out from the documentation.
it seems like this should work, but it does not:
d1[rows, test := vals]
the following error is returned:
Warning: 6.000000 (type 'double') at RHS position 1 taken as TRUE when assigning to type 'logical' (column 3 named 'test')
This is my desired outcome:
data.table(v1 = c(1,2,3,4,5,6), v2 = c(5,5,5,5,5,5), test = c(NA,NA,NA,NA,6,3))
Let's walk through this:
d1 <- data.table(v1 = c(1,2,3,4,5,6), v2 = c(5,5,5,5,5,5))
d1$test <- NA
rows <- c(5,6)
vals <- c(6,3)
d1[rows, test := vals]
# Warning in `[.data.table`(d1, rows, `:=`(test, vals)) :
# 6.000000 (type 'double') at RHS position 1 taken as TRUE when assigning to type 'logical' (column 3 named 'test')
class(d1$test)
# [1] "logical"
class(vals)
# [1] "numeric"
R can be quite "sloppy" in general, allowing one to coerce values from one class to another. Typically, this is from integer to floating point, sometimes from number to string, sometimes logical to number, etc. R does this freely, at times unexpectedly, and often silently. For instance,
13 > "2"
# [1] FALSE
The LHS is of class numeric, the RHS character. Because of the different classes, R silently converts 13 to "13" and then does the comparison. In this case, a string-comparison is doing a lexicographic comparison, which is letter-by-letter, meaning that it first compares the "1" with the "2", determines that it is unambiguously not true, and stops the comparison (since no other letter will change the results). The fact that the numeric comparison of the two is different, nor the fact that the RHS has no more letters to compare (lengths themselves are not compared) do not matter.
So R can be quite sloppy about this; not all languages are this allowing (most are not, in my experience), and this can be risky in unsupervised (automated) situations. It often produces unexpected results. Because of this, many (including devs of data.table and dplyr, to name two) "encourage" (force) the user to be explicit about class coersion.
As a side note: R has at least 8 different classes of NA, and all of them look like NA:
str(list(NA, NA_integer_, NA_real_, NA_character_, NA_complex_,
Sys.Date()[NA], Sys.time()[NA], as.POSIXlt(Sys.time())[NA]))
# List of 8
# $ : logi NA
# $ : int NA
# $ : num NA
# $ : chr NA
# $ : cplx NA
# $ : Date[1:1], format: NA
# $ : POSIXct[1:1], format: NA
# $ : POSIXlt[1:1], format: NA
There are a few ways to fix that warning.
Instantiate the test column as a "real" (numeric, floating-point) version of NA:
# starting with a fresh `d1` without `test` defined
d1$test <- NA_real_
d1[rows, test := vals] # works, no warning
Instantiate the test column programmatically, matching the class of vals without using the literal NA_real_:
# starting with a fresh `d1` without `test` defined
d1$test <- vals[1][NA]
d1[rows, test := vals] # works, no warning
Convert the existing test column in its entirety (not subsetted) to the desired class:
d1$test <- NA # this one is class logical
d1[, test := as.numeric(test)] # converts from NA to NA_real_
d1[rows, test := vals] # works, no warning
Things that work but are still being sloppy:
replace allows us to do this, but it is silently internally coercing from logical to numeric:
d1$test <- NA # logical class
d1[, test := replace(test, .I %in% rows, vals)]
This works because the internals of replace are simple:
function (x, list, values)
{
x[list] <- values
x
}
The reassignment to x[list] causes R to coerce the entire vector from logical to numeric, and it returns the whole vector at once. In data.table, assigning to the whole column at once allows this, since it is a common operation to change the class of a column.
As a side note, some might be tempted to use replace to fix things here. Using base::ifelse, this works, but further demonstrates the sloppiness of R here (and more so in ifelse, which while convenient, it is broken in a few ways).
base::ifelse doesn't work here out of the box because we'd need vals to be the same length as number of rows in d1. Even if that were the case, though, ifelse also silently coerces the class of one or the other. Imagine these scenarios:
ifelse(c(TRUE, TRUE), pi, "pi")
# [1] 3.141593 3.141593
ifelse(c(TRUE, FALSE), pi, "pi")
# [1] "3.14159265358979" "pi"
The moment one of the conditions is false in this case, the whole result changes from numeric to character, and there was no message or warning to that effect. It is because of this that data.table::fifelse (and dplyr::if_else) will fail preemptively:
fifelse(c(TRUE, TRUE), pi, "pi")
# Error in fifelse(c(TRUE, TRUE), pi, "pi") :
# 'yes' is of type double but 'no' is of type character. Please make sure that both arguments have the same type.
(There are other issues with ifelse, not just this, caveat emptor.)
I encountered an unexpected output when I used %in% in a condition whilst recoding a categorical variable.
When an element of a vector on the left is NA, the condition evaluates as FALSE, whilst I expected it to be NA.
The expected behaviour is the more verbose statement with two == conditions separated by an |
dt <- data.frame(colour = c("red", "orange", "blue", NA))
# Expected
dt$is_warm1 <- ifelse(dt$colour == "red" | dt$colour == "orange", TRUE, FALSE)
# Unexpected
dt$is_warm2 <- ifelse(dt$colour %in% c("red", "orange"), TRUE, FALSE)
dt
#> colour is_warm1 is_warm2
#> 1 red TRUE TRUE
#> 2 orange TRUE TRUE
#> 3 blue FALSE FALSE
#> 4 <NA> NA FALSE
This is quite unhelpful when recoding categorical variables because it silently fills missing values. Why does this happen, and are there any alternatives that don't involve listing all the == conditions? (Imagine that colour contains thirty possible levels).
a %in% b is just shorthand for match(a, b, nomatch = 0) > 0 (check the source code for %in% to satisfy yourself that this is the case).
You can get your expected result by removing the nomatch = 0 argument:
match(dt$colour, c("red", "orange")) > 0
#> [1] TRUE TRUE NA NA
Which of course doesn't require the ifelse
%in% checks to see if NA is in the list. Consider these two scenarios
NA %in% 1:3
# [1] FALSE
NA %in% c(1:3, NA)
# [1] TRUE
This allows you to check of NA is in the vector or not.
If you want to preserve NA values, you could write your own alternative
`%nain%` <- function(val, list) {
ifelse(is.na(val), NA, val %in% list)
}
And then you can use
dt$is_warm3 <- dt$colour %nain% c("red", "orange")
Here is some info from the help documentation ?%in%
So you can see in the last line %in% never returns NA so that is why it returns FALSE and not NA. It is checking for missing values as #MrFlick mentioned in his answer
Exactly what matches what is to some extent a matter of definition.
For all types, NA matches NA and no other value. For real and complex
values, NaN values are regarded as matching any other NaN value, but
not matching NA, where for complex x, real and imaginary parts must
match both (unless containing at least one NA).
Character strings will be compared as byte sequences if any input is
marked as "bytes", and otherwise are regarded as equal if they are in
different encodings but would agree when translated to UTF-8 (see
Encoding).
That %in% never returns NA makes it particularly useful in if
conditions.
I generally prefer to code R so that I don't get warnings, but I don't know how to avoid getting a warning when using as.numeric to convert a character vector.
For example:
x <- as.numeric(c("1", "2", "X"))
Will give me a warning because it introduced NAs by coercion. I want NAs introduced by coercion - is there a way to tell it "yes this is what I want to do". Or should I just live with the warning?
Or should I be using a different function for this task?
Use suppressWarnings():
suppressWarnings(as.numeric(c("1", "2", "X")))
[1] 1 2 NA
This suppresses warnings.
suppressWarnings() has already been mentioned. An alternative is to manually convert the problematic characters to NA first. For your particular problem, taRifx::destring does just that. This way if you get some other, unexpected warning out of your function, it won't be suppressed.
> library(taRifx)
> x <- as.numeric(c("1", "2", "X"))
Warning message:
NAs introduced by coercion
> y <- destring(c("1", "2", "X"))
> y
[1] 1 2 NA
> x
[1] 1 2 NA
In general suppressing warnings is not the best solution as you may want to be warned when some unexpected input will be provided.
Solution below is wrapper for maintaining just NA during data type conversion. Doesn't require any package.
as.num = function(x, na.strings = "NA") {
stopifnot(is.character(x))
na = x %in% na.strings
x[na] = "0"
x = as.numeric(x)
x[na] = NA_real_
x
}
as.num(c("1", "2", "X"), na.strings="X")
#[1] 1 2 NA
I have slightly modified the jangorecki function for the case where we may have a variety of values that cannot be converted to a number. In my function, a template search is performed and if the template is not found, FALSE is returned.! before gperl, it means that we need those vector elements that do not match the template. The rest is similar to the as.num function. Example:
as.num.pattern <- function(x, pattern){
stopifnot(is.character(x))
na = !grepl(pattern, x)
x[na] = -Inf
x = as.numeric(x)
x[na] = NA_real_
x
}
as.num.pattern(c('1', '2', '3.43', 'char1', 'test2', 'other3', '23/40', '23, 54 cm.'))
[1] 1.00 2.00 3.43 NA NA NA NA NA
Following is related to R language.
x1 <- c(1, 4, 3, NA, 7)
is.na(x1) <- which(x1 == 7)
I don't undertand, the LHS in last line gives you a vector of boolean and RHS is a value(index where x ==7, 5 in this case). So what does it mean to assign a boolean vector a value of 5?
is.na from the docs returns:
The default method for is.na applied to an atomic vector returns a logical vector of the same length as its argument x, containing TRUE for those elements marked NA or, for numeric or complex vectors, NaN, and FALSE otherwise.
Therefore, by making a logical vector(you're in essence saying wherever an index is TRUE, this should be an NA.
By "matching" these indices to the corresponding index from which, you're turning the latter into NAs wherever FALSE hence the change.
To put it in practice:
This is the output from is.na(x1):
is.na(x1)
[1] FALSE FALSE FALSE TRUE FALSE
The corresponding output from which(x==7):
which(x1 == 7)
[1] 5
Combining, the element at position 5 will now become an NA because it has been given the logical is.na() which returns TRUE
is.na(x1) <- which(x1 == 7)
x1
[1] 1 4 3 NA NA
The above turns the first index into an NA and appends two more NAs so as to make index 7 and NA.
This can be best seen by:
is.na(x1) <- c(1,7)
x1
[1] NA 4 3 NA 7 NA NA
Compare with this example from the docs:
(xx <- c(0:4))
is.na(xx) <- c(2, 4)
xx
[1] 0 NA 2 NA 4
From the above, it is clear that c(2,4) follows the original indices in xx hence the rest become NAs.
The code is like this
x <- 1:5
x[NA]
Why does it produce 5 NAs?
The answer to this question has two sides:
How is NA interpreted when indexing matrices?
In one of the links provided by #alexis_laz, I found a very well structured explanation of how TRUE, FALSE and NA are interpreted when indexing matrices:
Logical indices tell R which elements to include or exclude.
You have three options: TRUE, FALSE and NA
They serve to indicate whether or not the index represented in that position should be included. In other words:
TRUE == "Include the elment at this index"
FALSE == "Do not include the element at this index"
NA == "Return NA instead of this index" # loosely speaking
For example:
x <- 1:6
x[ c(TRUE, FALSE, TRUE, NA, TRUE, FALSE)]
# [1] 1 3 NA 5
An important detail is that the default storage mode for an isolated NA value is logical (try typeof(NA)). You can choose the storage mode of the NA by using NA_integer_, NA_real_ (for double), NA_complex_ or NA_character_.
Why 5 NA and not just 1?
When the length of the indices is smaller than the length of vector x, the indexing will start over to also index the values in x that have not been indexed yet. In other words, R will automatically "recycle" the indices:
(...) However, standard recycling rules apply. So in the previous example, if we drop the last FALSE, the index vector is recycled, the first element of the index is TRUE, and hence the 6th element of x is now included
x <- 1:6
x[c(TRUE, FALSE, TRUE, NA, TRUE)]
# [1] 1 3 NA 5 6
Recall the detail about the storage mode from the previous section. If you type x[NA_integer_], then you will find a different result.