I'm working with a data set that includes first names entered in all capital letters. I need to work with the names as character variables, not as factors.
One person in the data set has the first name "NA". Can I get R to accept "NA" as a legitimate character value? My work-around solution was to rename that person NAA, but I am interested to see if there is a better way.
As a demonstration of my comment, consider the following sample CSV file:
x <- tempfile()
cat("v1,v2", "NA,1", "AB,3", sep = "\n", file = x)
cat(readLines(x), sep = "\n")
# v1,v2
# NA,1
# AB,3
Here's the str of a basic read.csv. Note the NA is seen as NA
str(read.csv(x))
# 'data.frame': 2 obs. of 2 variables:
# $ v1: Factor w/ 1 level "AB": NA 1
# $ v2: int 1 3
Now, specify a different character as your na.strings argument:
str(read.csv(x, na.strings = ""))
# 'data.frame': 2 obs. of 2 variables:
# $ v1: Factor w/ 2 levels "AB","NA": 2 1
# $ v2: int 1 3
Related
I have a data.frame with 3 columns. The structure of the data.fame is as below
str(data)
'data.frame': 76971772 obs. of 3 variables:
$ V1: chr "XH104_AACGAGAGCTAAACTAGCCCTA" "XH104_AACGAGAGCTAAACTAGCCCTA" "XH104_AACGAGAGCTAAACTAGCCCTA" "XH104_AACGAGAGCTAAACTAGCCCTA" ...
$ V2: chr "10:100175000-100180000" "10:101065000-101070000" "10:101550000-101555000" "10:101585000-101590000" ...
$ V3: int 2 2 2 2 10 1 2 2 2 2 ...
I am trying to convert it into sparseMatrix such that the row name of sparseMatrix is data$V1 and the column name is data$V2. I am using the command given below to do that.
sparse.data <- with(data, sparseMatrix(i=as.numeric(V1), j=as.numeric(V2), x=V3, dimnames=list(levels(V1), levels(V2))))
I keep getting the this error.
Error in sparseMatrix(i = as.numeric(V1), j = as.numeric(V2), x = V3, :
NA's in (i,j) are not allowed
I realized that when I use i=as.numeric(V1) in my command, all the values of V1 become NA.
Can someone suggest how can I solve this error?
I have a dataset where there are many columns with numbers in quotes which indicates that a variable is a factor. (ex: "8").
read.table automatically converts them in numerical variables even if stringsAsFactor is set as true.
Suppose I cannot convert them manually with as.factor, how can I import this dataset with those variables coded directly as factor?
That's because of the quote option. Set quote="". Example:
t <- '"1" "3"
"2" "4"'
> str(read.table(text=t))
'data.frame': 2 obs. of 2 variables:
$ V1: int 1 2
$ V2: int 3 4
> str(read.table(text=t, quote=""))
'data.frame': 2 obs. of 2 variables:
$ V1: Factor w/ 2 levels "\"1\"","\"2\"": 1 2
$ V2: Factor w/ 2 levels "\"3\"","\"4\"": 1 2
I'm trying to read in an excel file to R using read_excel(it's a xlsx file), I have columns that contain letters and numbers, for example things like P765876. These columns also have cells with just numbers i.e 234654, so when it reads in to R it reads as an Unknown (not character or numeric) but this means that it gives any cell which has a letter and number a value of NA, how can I read this in correctly?
My code at the moment is
tenant<-read_excel("C:/Users/MPritchard/Repairs Projects/May 2017/Tenant Info/R data 1.xlsx")
Would also recommend to use the col_types argument, by specifying it as "text" you should avoid getting NAs introduced by coercion. So your code would be like:
tenant<-read_excel("C:/Users/MPritchard/Repairs Projects/May 2017/Tenant Info/R data 1.xlsx", col_types = "text")
Please let me know if this solved your problem.
Regards,
/Michael
Not really an answer but too much for a comment...
1:
> library(xlsx)
> tenant <- read.xlsx("returns.xlsx", sheetIndex = 1)
> str(tenant)
'data.frame': 9 obs. of 3 variables:
$ only_integer: num 1 2 34 5 546931 ...
$ int_char : Factor w/ 9 levels "2545","2a","2d",..: 6 4 9 3 5 1 7 2 8
$ only_char : Factor w/ 6 levels "af","dd","e",..: 2 1 5 6 3 2 4 3 1
2:
> library(readxl)
> tenant2 <- read_excel("returns.xlsx")
> str(tenant2)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 9 obs. of 3 variables:
$ only_integer: num 1 2 34 5 546931 ...
$ int_char : chr "d5" "5" "ff2ad2f" "2d" ...
$ only_char : chr "dd" "af" "h" "ha" ...
The column int_char is a mixture of both, starting/ending with numbers or characters
I have 2 variables in a data frame with 300 observations.
$ imagelike: int 3 27 4 5370 ...
$ user: Factor w/ 24915 levels "\"0.1gr\"","\"008bla\"", ..
I then tried to remove the duplicates, such as "- " appears 2 times:
testclean <- data1[!duplicated(data1), ]
This gives me the warning message:
In Ops.factor(left): "-"not meaningful for factors
I have then converted it to a maxtrix:
data2 <- data.matrix(data1)
testclean2 <- data2[!duplicated(data2), ]
This does the trick - however - it converts the userNames to a numeric.
=========================================================================
I am new but I have tried looking at previous posts on this topic (including the one below) but it did not work out:
Convert data.frame columns from factors to characters
Some sample data, from your image (please don't post images of data!):
data1 <- data.frame(imageLikeCount = c(3,27,4,4,16,103),
userName = c("\"testblabla\"", "test_00", "frenchfries", "frenchfries", "test.inc", "\"parmezan_pizza\""))
str(data1)
# 'data.frame': 6 obs. of 2 variables:
# $ imageLikeCount: num 3 27 4 4 16 103
# $ userName : Factor w/ 5 levels "\"parmezan_pizza\"",..: 2 5 3 3 4 1
To fix the problem with factors as well as the embedded quotes:
data1$userName <- gsub('"', '', as.character(data1$userName))
str(data1)
# 'data.frame': 6 obs. of 2 variables:
# $ imageLikeCount: num 3 27 4 4 16 103
# $ userName : chr "testblabla" "test_00" "frenchfries" "frenchfries" ...
Like #DanielWinkler suggested, if you can change how the data is read-in or defined, you might choose to include stringsAsFactors = FALSE (this argument is accepted in many functions, including read.csv, read.table, and most data.frame functions including as.data.frame and rbind):
data1 <- data.frame(imageLikeCount = c(3,27,4,4,16,103),
userName = c("\"testblabla\"", "test_00", "frenchfries", "frenchfries", "test.inc", "\"parmezan_pizza\""),
stringsAsFactors = FALSE)
str(data1)
# 'data.frame': 6 obs. of 2 variables:
# $ imageLikeCount: num 3 27 4 4 16 103
# $ userName : chr "\"testblabla\"" "test_00" "frenchfries" "frenchfries" ...
(Note that this still has embedded quotes, so you'll still need something like data1$userName <- gsub('"', '', data1$userName).)
Now, we have data that looks like this:
data1
# imageLikeCount userName
# 1 3 testblabla
# 2 27 test_00
# 3 4 frenchfries
# 4 4 frenchfries
# 5 16 test.inc
# 6 103 parmezan_pizza
and your need to remove duplicates works:
data1[! duplicated(data1), ]
# imageLikeCount userName
# 1 3 testblabla
# 2 27 test_00
# 3 4 frenchfries
# 5 16 test.inc
# 6 103 parmezan_pizza
Try
data$userName <- as.character(data$userName)
And then
data<-unique(data)
You could also pass the argument stringAsFactor = FALSE when reading the data. This is usually a good idea.
Is there a way for fread to mimic the behaviour of read.table whereby the class of the variable is set by the data that is read in.
I have numeric data with a few comments underneath the main data. When i use fread to read in the data, the columns are converted to character. However, by setting the nrow in read.table` i can stop this behaviour. Is this possible in fread. (I would prefer not to alter the raw data or make an amended copy). Thanks
An example
d <- data.frame(x=c(1:100, NA, NA, "fff"), y=c(1:100, NA,NA,NA))
write.csv(d, "test.csv", row.names=F)
in_d <- read.csv("test.csv", nrow=100, header=T)
in_dt <- data.table::fread("test.csv", nrow=100)
Which produces
> str(in_d)
'data.frame': 100 obs. of 2 variables:
$ x: int 1 2 3 4 5 6 7 8 9 10 ...
$ y: int 1 2 3 4 5 6 7 8 9 10 ...
> str(in_dt)
Classes ‘data.table’ and 'data.frame': 100 obs. of 2 variables:
$ x: chr "1" "2" "3" "4" ...
$ y: int 1 2 3 4 5 6 7 8 9 10 ...
- attr(*, ".internal.selfref")=<externalptr>
As a workaround I thought i would be able to use read.table to read in one line, get the class and set the colClasses, but i am misunderstanding.
cl <- read.csv("test.csv", nrow=1, header=T)
cols <- unname(sapply(cl, class))
in_dt <- data.table::fread("test.csv", nrow=100, colClasses=cols)
str(in_dt)
Using Windows8.1
R version 3.1.2 (2014-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Option 1: Using a system command
fread() allows the use of a system command in its first argument. We can use it to remove the quotes in the first column of the file.
indt <- data.table::fread("cat test.csv | tr -d '\"'", nrows = 100)
str(indt)
# Classes ‘data.table’ and 'data.frame': 100 obs. of 2 variables:
# $ x: int 1 2 3 4 5 6 7 8 9 10 ...
# $ y: int 1 2 3 4 5 6 7 8 9 10 ...
# - attr(*, ".internal.selfref")=<externalptr>
The system command cat test.csv | tr -d '\"' explained:
cat test.csv reads the file to standard output
| is a pipe, using the output of the previous command as input for the next command
tr -d '\"' deletes (-d) all occurrences of double quotes ('\"') from the current input
Option 2: Coercion after reading
Since option 1 doesn't seem to be working on your system, another possibility is to read the file as you did, but convert the x column with type.convert().
library(data.table)
indt2 <- fread("test.csv", nrows = 100)[, x := type.convert(x)]
str(indt2)
# Classes ‘data.table’ and 'data.frame': 100 obs. of 2 variables:
# $ x: int 1 2 3 4 5 6 7 8 9 10 ...
# $ y: int 1 2 3 4 5 6 7 8 9 10 ...
# - attr(*, ".internal.selfref")=<externalptr>
Side note: I usually prefer to use type.convert() over as.numeric() to avoid the "NAs introduced by coercion" warning triggered in some cases. For example,
x <- c("1", "4", "NA", "6")
as.numeric(x)
# [1] 1 4 NA 6
# Warning message:
# NAs introduced by coercion
type.convert(x)
# [1] 1 4 NA 6
But of course you can use as.numeric() as well.
Note: This answer assumes data.table dev v1.9.5
Ok, the customer is abusing CSV format to intentionally write out trailing string rows to an integer column, yet without those rows starting with a comment.char (#).
Then you somehow expect you can override fread()'s type inference to read those as integer, by using nrow to try to limit it to just see the integer rows. read.csv(..., nrow) will accept this, however fread() always uses all rows for type-inference (not just the ones specified by nrow, skip, header), and even if they start with comment.char (that's a bug).
Sounds like an abuse of CSV. Your comment rows should be prepended with #
Yes, fread() needs a fix/enhance to ignore comment rows for type inference.
For now, you can workaround with fread() by post-processing the data-table read in.
It's arguable whether fread() should be changed to support the behavior you want: using nrows to limit what gets exposed to type-inference. It might fix your (pretty unique) case and break some others.
I don't see why you (EDIT: the customer) can't write your comments to a separate .txt/README/data-dictionary file to accompany the .csv. The practice of using a separate data-dictionary file is pretty well-established.
I've never seen someone do this to a CSV file. At least move the comments to the header, not a footer.