Unable to find the solution to my problem.
I am trying to read a text file using r. File contains a single row and separated by number of characters.
000341656.0000000000000000004.6000000000000000009.0000000000000000050.9566787004000000052.0000000000000000072.8621215573000000007.0000000000000000050.0361010830000000047.2490974729000000054.5560183531000000006.0000000000000000049.9711191336000000047.0397111913000000043.1488475260000000023.0000000000000000046.6281588448000000040.1516245487000000038.4653540241000000002.0000000000000000046.2129963899000000041.9963898917000000037.3850068798000000030.0000000000000000046.0144404332000000040.0324909747000000027.0930952140000000003.0000000000000000043.3971119134000000032.4801444043000000010.4757238771
First value is of 20 digit floating.9 digit followed by 10 decimal digit.
The file contains between 22 and 30 values, each 20 digits long (decimal digit set to '.')
What i am unable to figure out how to get rid of this extra 0.
Any lead of help is highly appreciable.
You can read data in a fixed-width format with read.fwf:
> read.fwf("./d.txt", widths=rep(20,30))
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12
1 341656 4.6 9 50.95668 52 72.86212 7 50.0361 47.2491 54.55602 6 49.97112
2 341656 4.6 9 50.95668 52 72.86212 7 50.0361 47.2491 54.55602 6 49.97112
V13 V14 V15 V16 V17 V18 V19 V20 V21 V22
1 47.03971 43.14885 23 46.62816 40.15162 38.46535 2 46.213 41.99639 37.38501
2 47.03971 43.14885 23 46.62816 40.15162 38.46535 2 46.213 41.99639 37.38501
V23 V24 V25 V26 V27 V28 V29 V30
1 30 46.01444 40.03249 27.0931 3 43.39711 32.48014 10.47572
2 30 46.01444 40.03249 27.0931 3 43.39711 32.48014 10.47572
You need to know how many fields and how big they are. You didnt say how many lines in the file but I copied your line in twice (hence the duplication).
library(stringi)
stri_match_all_regex(
"000341656.0000000000000000004.6000000000000000009.0000000000000000050.9566787004000000052.0000000000000000072.8621215573000000007.0000000000000000050.0361010830000000047.2490974729000000054.5560183531000000006.0000000000000000049.9711191336000000047.0397111913000000043.1488475260000000023.0000000000000000046.6281588448000000040.1516245487000000038.4653540241000000002.0000000000000000046.2129963899000000041.9963898917000000037.3850068798000000030.0000000000000000046.0144404332000000040.0324909747000000027.0930952140000000003.0000000000000000043.3971119134000000032.4801444043000000010.4757238771",
".{20}"
) %>%
unlist() %>%
as.numeric()
## [1] 341656.00000 4.60000 9.00000 50.95668 52.00000
## [6] 72.86212 7.00000 50.03610 47.24910 54.55602
## [11] 6.00000 49.97112 47.03971 43.14885 23.00000
## [16] 46.62816 40.15162 38.46535 2.00000 46.21300
## [21] 41.99639 37.38501 30.00000 46.01444 40.03249
## [26] 27.09310 3.00000 43.39711 32.48014 10.47572
Also:
as.numeric(readChar("~/Data/20.txt", rep(20, file.size("~/Data/20.txt")/20)))
I wanna skip the first three columns. Couldn't quite understand the posts about colClasses because I'm new to R.
YDL025C YDL025C 1 -0.1725 -0.5375 -0.4970 -0.3818 -0.5270 -0.4260 -0.6929 -0.4020 -0.3263 -0.3373 -0.3532 -0.2771 -0.2732 -0.3307 -0.4660 -0.4314 -0.3135
YKL032C YKL032C 1 -0.2364 0.0794 0.1678 0.2389 0.3847 0.2625 0.1889 0.2681 0.0363 -0.1992 -0.0521 -0.0307 0.0584 0.2817 0.2239 -0.0253 0.0751
If you have to use read.table and you want to filter on the way in, you can use col.classes as follows. You have 20 columns. Say the first 2 are character, rest are numeric and you want to drop 4,5,6. You construct a vector of length 20 detailing that information. The NULL will not pull in those columns.
x<- read.table(file="datat.txt",
colClasses = c(rep("character", 2),
rep("numeric", 1),
rep("NULL", 3),
rep("numeric", 14)),
header = FALSE)
x
V1 V2 V3 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
1 YDL025C YDL025C 1 -0.3818 -0.5270 -0.4260 -0.6929 -0.4020 -0.3263 -0.3373 -0.3532 -0.2771 -0.2732 -0.3307 -0.4660 -0.4314 -0.3135
2 YKL032C YKL032C 1 0.2389 0.3847 0.2625 0.1889 0.2681 0.0363 -0.1992 -0.0521 -0.0307 0.0584 0.2817 0.2239 -0.0253 0.0751
As commented above, easier to remove the columns after reading in. For example:
mydf <- read.table("mydf.txt")
Then,
mydf[, 4:ncol(mydf)]
will remove the first 3 columns.
I would like to create a table of column means by Strain factors
I have the following data:
Age Strain 103 3 163 39
V2 28 101CD -3.4224173012 -0.3360570164 -9.2417448649 -3.6094766494
V3 28 101CD -3.6487198656 -0.7948262475 -4.6350611123 -1.9232938265
V4 28 101CD -7.0936427264 -0.1981243536 -9.2063428591 -3.367139071
V5 28 101CD -5.9245254437 -0.1161875584 -7.3830396092 -4.7980771085
V6 30 101HFD -9.4618204696 -5.0355557149 -3.9915005349 -0.9271933496
V7 30 101HFD -8.805867863 -2.667103793 -2.2489197384 -1.5169130813
V8 30 101HFD -10.9841335945 -2.9617657815 -3.3460597574 -1.121806194
V9 30 101HFD -10.4612747952 -4.3759351258 -4.4322637085 -0.772499965
V10 30 101HFD -9.2871507889 -1.2664335711 -4.3142098012 -1.3791233817
V11 30 101HFD -10.9443983294 -2.4651954898 -4.7759052834 -1.0954401254
V12 29 103CD -2.7492530803 -2.0659306194 -2.5698186069 -1.4978280502
V13 29 103CD -6.4401905692 -2.1098420514 -3.4349220483 -0.8836564768
V14 29 103CD -6.479929929 -2.4792621691 -3.368774934 -0.7756932376
V15 29 103CD -3.6586850957 -1.9145944032 -3.0911223702 -1.2730896376
V16 29 103CD -7.1377230731 -1.413139617 -2.9203340711 -1.3152010161
V17 29 103HFD -9.4624093184 -1.3265834556 -4.1871313168 -1.0108235293
V18 29 103HFD -7.336764023 -0.8712499419 -4.204313727 -1.4450582002
V19 29 103HFD -7.036723106 -0.7546877382 -6.0432957599 -1.4161366956
V20 29 103HFD -9.4449207581 -0.9226067311 -4.6305567775 -1.320094489
V21 29 103HFD -9.6383454033 -1.9620356763 -3.0214290407 -0.8602682738
And, I want to end up with this:
Age Strain 103 3 163 39
V1 28 101CD -3.4224173012 -0.3360570164 -9.2417448649 -3.6094766494
V2 30 101HFD -9.4618204696 -5.0355557149 -3.9915005349 -0.9271933496
V3 29 103CD -2.7492530803 -2.0659306194 -2.5698186069 -1.4978280502
V4 29 103HFD -9.4624093184 -1.3265834556 -4.1871313168 -1.0108235293
Where [1,] is the mean of all columns for all samples with Strain=101CD, [2:3] is the mean of all columns for samples with Strain=101HFD, etc.
I have attempted to use:
> ave <- aggregate(data, as.list(factor(data$Age)), mean)
Error in aggregate.data.frame(data, as.list(factor(data$Age)), mean) : arguments must have same length
and
> ave <- sapply(split(data, data$Strain), mean)
101CD 101HFD 103CD 103HFD 32CD 40CD 40HFD 43CD 43HFD 44CD 44HFD
NA NA NA NA NA NA NA NA NA NA NA
...
97HFD 98CD 98HFD 99CD 99HFD
NA NA NA NA NA
There were 50 or more warnings (use warnings() to see the first 50)
and
> ave <- daply(data, data$Strain, mean)
Error in parse(text = x) : <text>:1:4: unexpected symbol
1: 101CD
I feel like there should be a fairly straightforward way to accomplish this, but I have been unable to find a solution.
You can use dplyr. Here we group_by Strain, then use summarise_each to summarise each column, with the function mean with na.rm set to TRUE:
library(dplyr)
data %>% group_by(Strain) %>%
summarise_each(funs(mean(., na.rm=TRUE)))
Source: local data frame [4 x 6]
Strain Age X103 X3 X163 X39
(fctr) (dbl) (dbl) (dbl) (dbl) (dbl)
1 101CD 28 -5.022326 -0.3612988 -7.616547 -3.424497
2 101HFD 30 -9.990774 -3.1286649 -3.851476 -1.135496
3 103CD 29 -5.293156 -1.9965538 -3.076994 -1.149094
4 103HFD 29 -8.583833 -1.1674327 -4.417345 -1.210476
Exploit the fact that a data.frame is a special kind of list.
aggregate(data, data[, "Age", drop = FALSE], mean)
drop = FALSE is required so that the result of the selection remains a data.frame. data[, "Age"] is equivalent to data[, "Age", drop = TRUE] and will return a vector.
I have the CSV file like below:
data,key
"VA1,VA2,20140524,,0,0,5969,20140523134902,S7,S1147,140,20140523134902,m/t",4503632376496128
"VA2,VA3,20140711,,0,0,8824,20140601095714,S1,S6402,175,20140601095839,m/t",4503643113914368
I try to read it with R, but I don't need key value and data value should be read to separate columns. With the following code I get almost what I need:
data <- read.csv(fileCSV, header = FALSE, sep = ",", skip = 1, comment.char = "", quote = "")
I skip header line there (skip = 1), say that I don't have it (header = FALSE), and say that I don't have quotes (quote = ""). But in result I get quote characters in V1 and V13 columns and extra V14 column:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14
1 "VA1 VA2 20140524 NA 0 0 5969 2.014121e+13 S7 S1147 140 2.014121e+13 m/t" 4.503608e+15
Should I delete it somehow after reading csv? Or, is there any better way to read such csv files?
Upd. I use the following approach to delete quotes:
data[,"V1"] = sub("^\"", "", data[,"V1"])
data[,"V13"] = sub("\"$", "", data[,"V13"])
But factor type is changed to character for these columns.
How about a system command with fread()?
writeLines(
'data,key
"VA1,VA2,20140524,,0,0,5969,20140523134902,S7,S1147,140,20140523134902,m/t",4503632376496128
"VA2,VA3,20140711,,0,0,8824,20140601095714,S1,S6402,175,20140601095839,m/t",4503643113914368', "x.txt"
)
require(bit64)
data.table::fread("cat x.txt | rev | cut -d '\"' -f2 | rev | tail -n +2")
# V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13
# 1: VA1 VA2 20140524 NA 0 0 5969 20140523134902 S7 S1147 140 20140523134902 m/t
# 2: VA2 VA3 20140711 NA 0 0 8824 20140601095714 S1 S6402 175 20140601095839 m/t
Here's a test on the two methods, as requested.
## 150k lines
writeLines(c("data,key\n", rep_len(
'"VA1,VA2,20140524,,0,0,5969,20140523134902,S7,S1147,140,20140523134902,m/t",4503632376496128\n', 1.5e5)),
"test.txt"
)
## fread() in well under 1 second (with bit64 loaded)
system.time({
dt <- data.table::fread(
"cat test.txt | rev | cut -d '\"' -f2 | rev | grep -e '^V'"
)
})
# user system elapsed
# 0.945 0.108 0.547
## your current read.csv() method in just over two seconds
system.time({
df <- read.csv("test.txt", header = FALSE, sep = ",", skip = 1, comment.char = "", quote = "")
df[,"V1"] = sub("^\"", "", df[,"V1"])
df[,"V13"] = sub("\"$", "", df[,"V13"])
})
# user system elapsed
# 2.134 0.000 2.129
dim(dt)
# [1] 150000 13
dim(df)
# [1] 150000 14
Hi I am using the following r script to build a random forest:
# load the necessary libraries
library(randomForest)
testPP<-numeric()
# load the dataset
QdataTrain <- read.csv('train.csv',header = FALSE)
QdataTest <- read.csv('test.csv',header = FALSE)
QdataTrainX <- subset(QdataTrain,select=-V1)
QdataTrainY<-as.factor(QdataTrain$V1)
QdataTestX <- subset(QdataTest,select=-V1)
QdataTestY<-as.factor(QdataTest$V1)
mdl <- randomForest(QdataTrainX, QdataTrainY)
where I am getting the following error:
Error in randomForest.default(QdataTrainX, QdataTrainY) :
NA not permitted in predictors
however i see no occurence of NA in my data.
for reference here is my data:
https://docs.google.com/file/d/0B0iDswLYaZ0zUFFsT01BYlRZU0E/edit
does anyone know why this error is being thrown? I'll keep looking in the mean time.
Thanks in advance for any help!
The given data does contain some missing values (7 in particular):
sapply(QdataTrainX, function(x) sum(is.na(x)))
## V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16
## 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
## V17 V18 V19 V20 V21 V22 V23 V24 V25 V26 V27 V28 V29
## 0 0 0 0 0 0 1 1 1 1 1 1 1
Therefore columns V23 to V29 have one missing value each
which(is.na(QdataTrainX$V23))
## 318
Gives the row number for that.