I have a 33 rows list data,
as below:
enter link description here
And I was using the code as below to create a correlation coefficient:
mpg.df <- as.data.frame(mpg)
cor(mpg, method = "pearson", use = "complete.obs")
However, I am getting the same error:
Error in cor(mpg, method = "pearson", use = "complete.obs") :
'x' must be numeric
And I used typeof(mpg.df), the result was "list" still.
Any suggestions would be highly appreciated!!!
It is likely that your example data frame has one or more columns that are not numeric. Since your example data frame looks like a subset of the pre-defined mtcars data frame in R, I will just create this subset in R as follows.
# Select some columns
mtcars2 <- mtcars[, c("mpg", "cyl", "disp", "hp", "drat", "wt")]
# View the first six rows
head(mtcars2)
# mpg cyl disp hp drat wt
# Mazda RX4 21.0 6 160 110 3.90 2.620
# Mazda RX4 Wag 21.0 6 160 110 3.90 2.875
# Datsun 710 22.8 4 108 93 3.85 2.320
# Hornet 4 Drive 21.4 6 258 110 3.08 3.215
# Hornet Sportabout 18.7 8 360 175 3.15 3.440
# Valiant 18.1 6 225 105 2.76 3.460
Now take a look at the structure mtcars2, you can see that all columns are numeric.
# Show the class of each column
str(mtcars2)
# 'data.frame': 32 obs. of 6 variables:
# $ mpg : num 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
# $ cyl : num 6 6 4 6 8 6 8 4 4 6 ...
# $ disp: num 160 160 108 258 360 ...
# $ hp : num 110 110 93 110 175 105 245 62 95 123 ...
# $ drat: num 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
# $ wt : num 2.62 2.88 2.32 3.21 3.44 ...
Since all columns are numeric, we can thus use the cor function to do correlation analysis.
# Do correlation analysis
cor(mtcars2)
# mpg cyl disp hp drat wt
# mpg 1.0000000 -0.8521620 -0.8475514 -0.7761684 0.6811719 -0.8676594
# cyl -0.8521620 1.0000000 0.9020329 0.8324475 -0.6999381 0.7824958
# disp -0.8475514 0.9020329 1.0000000 0.7909486 -0.7102139 0.8879799
# hp -0.7761684 0.8324475 0.7909486 1.0000000 -0.4487591 0.6587479
# drat 0.6811719 -0.6999381 -0.7102139 -0.4487591 1.0000000 -0.7124406
# wt -0.8676594 0.7824958 0.8879799 0.6587479 -0.7124406 1.0000000
Related
For the application of a function to multiple smaller datasets from a larger dataset, I need to perform a split of the large dataset by multiple variables. However, for further use of the child datasets, I want to store them in a nested list with the different grouping variables as list node names (to be used with rapply).
An example:
head_mtcars <- head(mtcars, 10)
I know from here that I can split the data set using list(data$V1, data$V2), but the generated list unfortunately only keeps the grouping variable in the same level. I would be wishing for list nodes like $6$3, $8$3 etc.:
split(head_mtcars, list(head_mtcars$cyl, head_mtcars$gear), drop = T)
$`6.3`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
$`8.3`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
Duster 360 14.3 8 360 245 3.21 3.57 15.84 0 0 3 4
$`4.4`
mpg cyl disp hp drat wt qsec vs am gear carb
Datsun 710 22.8 4 108.0 93 3.85 2.32 18.61 1 1 4 1
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.90 1 0 4 2
$`6.4`
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
I also tried to change the separator but this does not help:
## only changes the naming separator to a $ but does not actually create a new list level:
split(head_mtcars, list(head_mtcars$cyl, head_mtcars$gear), drop = T, sep = "$")
$`6$3`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
$`8$3`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
Duster 360 14.3 8 360 245 3.21 3.57 15.84 0 0 3 4
$`4$4`
mpg cyl disp hp drat wt qsec vs am gear carb
Datsun 710 22.8 4 108.0 93 3.85 2.32 18.61 1 1 4 1
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.90 1 0 4 2
$`6$4`
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
I also tried to modify the code from here to be used with multiple splitting variables, but this moves the group variables to dimnames, from which I don't know how (if possible at all) to convert to nested list levels (it works perfectly when using only one grouping variable).
by(head_mtcars, list(head_mtcars$cyl, head_mtcars$gear), identity, simplify = FALSE)
: 4
: 3
NULL
-------------------------------------------------------------
: 6
: 3
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
-------------------------------------------------------------
: 8
: 3
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
Duster 360 14.3 8 360 245 3.21 3.57 15.84 0 0 3 4
-------------------------------------------------------------
: 4
: 4
mpg cyl disp hp drat wt qsec vs am gear carb
Datsun 710 22.8 4 108.0 93 3.85 2.32 18.61 1 1 4 1
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.90 1 0 4 2
-------------------------------------------------------------
: 6
: 4
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
-------------------------------------------------------------
: 8
: 4
NULL
I also tried various tidyverse approaches but also none of them really solved the problem.
In the end, I would like to have a nested list with the levels from $cyl as the first level and the levels from $gear as the level below. Any advice?
Here is a method to nest the splits an arbitrary number of times using reduce() and map_depth().
Note that the formula interface for split() is a relatively recent feature so if it doesn't work you may have to upgrade to a more recent version.
library(purrr)
head_mtcars <- head(mtcars, 10)
fms <- list(~cyl, ~gear, ~carb)
reduce(.x = fms, .f = ~ map_depth(.x, .depth = vec_depth(.x) - 2, split, .y), .init = head_mtcars)
$`4`
$`4`$`4`
$`4`$`4`$`1`
mpg cyl disp hp drat wt qsec vs am gear carb
Datsun 710 22.8 4 108 93 3.85 2.32 18.61 1 1 4 1
$`4`$`4`$`2`
mpg cyl disp hp drat wt qsec vs am gear carb
Merc 240D 24.4 4 146.7 62 3.69 3.19 20.0 1 0 4 2
Merc 230 22.8 4 140.8 95 3.92 3.15 22.9 1 0 4 2
$`6`
$`6`$`3`
$`6`$`3`$`1`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
$`6`$`4`
$`6`$`4`$`4`
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4
Merc 280 19.2 6 167.6 123 3.92 3.440 18.30 1 0 4 4
$`8`
$`8`$`3`
$`8`$`3`$`2`
mpg cyl disp hp drat wt qsec vs am gear carb
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
$`8`$`3`$`4`
mpg cyl disp hp drat wt qsec vs am gear carb
Duster 360 14.3 8 360 245 3.21 3.57 15.84 0 0 3 4
This question already has answers here:
Subsetting R data frame results in mysterious NA rows
(7 answers)
Closed 1 year ago.
I am removing rows with certain values with
dat14a <- dat14[dat14${MYVARIABLE}<80, ]
dat14 looks like this
However, after using a/m code dat14a is also affected in other colums showing then the following
How can I avoid that?
Thanks!!
That is most likely because you have NA's in MYVARIABLE column.
Using a reproducible example from mtcars.
df <- mtcars[1:5, 1:5]
df$mpg[1:2] <- NA
df
# mpg cyl disp hp drat
#Mazda RX4 NA 6 160 110 3.90
#Mazda RX4 Wag NA 6 160 110 3.90
#Datsun 710 22.8 4 108 93 3.85
#Hornet 4 Drive 21.4 6 258 110 3.08
#Hornet Sportabout 18.7 8 360 175 3.15
df[df$mpg > 22, ]
# mpg cyl disp hp drat
#NA NA NA NA NA NA
#NA.1 NA NA NA NA NA
#Datsun 710 22.8 4 108 93 3.85
To fix the issue use an additional !is.na(..)
df[df$mpg > 22 & !is.na(df$mpg), ]
# mpg cyl disp hp drat
#Datsun 710 22.8 4 108 93 3.85
I wanted to import multiple csv files from a folder and sort them into distinct data frames based on the file name.
The pattern of my file name is chX_imgN_chYROI, where X & Y = 1, 2 & 3, N = 1,2,3,4 & 5. The 'N' does not matter as I want to combine .csv files based on distinct combinations of X and Y (eg ch1_ch2ROI <– ch1_img1_ch2ROI, ch1_img2_ch2ROI..... ch1_img5_ch2ROI)
I'm a novice and any suggestions/insights will be helpful. Thanks!
The first part of this question (import multiple csv files) is really a duplicate of How do I make a list of data frames?.
But the second part -- combining some frames -- is a little different. I'll generate some sample data.
From the duplicate part, you'd probably use something like below to read in the files:
alldat <- sapply(list.files(somedir, pattern = "ch.*_img.*_ch.*.csv", full.names = TRUE),
read.csv, stringsAsFactors = FALSE,
simplify = FALSE)
Even if you just use this code blindly, I still recommend you read over the answers in How do I make a list of data frames?, as the advice and methodology are efficient and very idiomatic to R. Done correctly, they can make many workflows significantly easier to visualize, understand, and maintain.
To mimic import process, I'll use this fake data:
alldat <- list(
"ch1_img1_ch1ROI" = mtcars[1:2,],
"ch1_img1_ch2ROI" = mtcars[3:4,],
"ch1_img2_ch1ROI" = mtcars[5:6,],
"ch2_img1_ch1ROI" = mtcars[7:8,],
"ch2_img1_ch2ROI" = mtcars[9:10,],
"ch2_img2_ch2ROI" = mtcars[11:12,]
)
alldat
# $ch1_img1_ch1ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Mazda RX4 21 6 160 110 3.9 2.620 16.46 0 1 4 4
# Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
# $ch1_img1_ch2ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
# Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
# $ch1_img2_ch1ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
# Valiant 18.1 6 225 105 2.76 3.46 20.22 1 0 3 1
# $ch2_img1_ch1ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Duster 360 14.3 8 360.0 245 3.21 3.57 15.84 0 0 3 4
# Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
# $ch2_img1_ch2ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Merc 230 22.8 4 140.8 95 3.92 3.15 22.9 1 0 4 2
# Merc 280 19.2 6 167.6 123 3.92 3.44 18.3 1 0 4 4
# $ch2_img2_ch2ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Merc 280C 17.8 6 167.6 123 3.92 3.44 18.9 1 0 4 4
# Merc 450SE 16.4 8 275.8 180 3.07 4.07 17.4 0 0 3 3
By your logic, we have some combinations of X/Y that are unique and some that have multiple N's. Let's group solely by X/Y combinations.
First, we'll extract the X and Y components into a unique string for each filename:
gsub(".*ch([0-9]+)_.*ch([0-9]+).*", "\\1_\\2", names(alldat))
# [1] "1_1" "1_2" "1_1" "2_1" "2_2" "2_2"
Notice that we have some frames that need to be combined, namely elements 1 and 3, and elements 5 and 6.
split the list of frames by this string. Notice how we have a list of 4 elements, each of which is a nested list of 1 or more frames.
spllists <- split(alldat, gsub(".*ch([0-9]+)_.*ch([0-9]+).*", "\\1_\\2", names(alldat)))
str(spllists, max.level = 2)
# List of 4
# $ 1_1:List of 2
# ..$ ch1_img1_ch1ROI:'data.frame': 2 obs. of 11 variables:
# ..$ ch1_img2_ch1ROI:'data.frame': 2 obs. of 11 variables:
# $ 1_2:List of 1
# ..$ ch1_img1_ch2ROI:'data.frame': 2 obs. of 11 variables:
# $ 2_1:List of 1
# ..$ ch2_img1_ch1ROI:'data.frame': 2 obs. of 11 variables:
# $ 2_2:List of 2
# ..$ ch2_img1_ch2ROI:'data.frame': 2 obs. of 11 variables:
# ..$ ch2_img2_ch2ROI:'data.frame': 2 obs. of 11 variables:
Iterate (lapply) over the outer list, combining the inner lists. To do the inner row-combining, we'd use
spllists[[1]]
# $ch1_img1_ch1ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Mazda RX4 21 6 160 110 3.9 2.620 16.46 0 1 4 4
# Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
# $ch1_img2_ch1ROI
# mpg cyl disp hp drat wt qsec vs am gear carb
# Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
# Valiant 18.1 6 225 105 2.76 3.46 20.22 1 0 3 1
do.call(rbind, spllists[[1]])
# mpg cyl disp hp drat wt qsec vs am gear carb
# ch1_img1_ch1ROI.Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
# ch1_img1_ch1ROI.Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
# ch1_img2_ch1ROI.Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
# ch1_img2_ch1ROI.Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
So to do this for all elements in the spllists, we'll use
alldat2 <- lapply(spllists, function(x) do.call(rbind, x))
alldat2
# $`1_1`
# mpg cyl disp hp drat wt qsec vs am gear carb
# ch1_img1_ch1ROI.Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
# ch1_img1_ch1ROI.Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
# ch1_img2_ch1ROI.Hornet Sportabout 18.7 8 360 175 3.15 3.440 17.02 0 0 3 2
# ch1_img2_ch1ROI.Valiant 18.1 6 225 105 2.76 3.460 20.22 1 0 3 1
# $`1_2`
# mpg cyl disp hp drat wt qsec vs am gear carb
# ch1_img1_ch2ROI.Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
# ch1_img1_ch2ROI.Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
# $`2_1`
# mpg cyl disp hp drat wt qsec vs am gear carb
# ch2_img1_ch1ROI.Duster 360 14.3 8 360.0 245 3.21 3.57 15.84 0 0 3 4
# ch2_img1_ch1ROI.Merc 240D 24.4 4 146.7 62 3.69 3.19 20.00 1 0 4 2
# $`2_2`
# mpg cyl disp hp drat wt qsec vs am gear carb
# ch2_img1_ch2ROI.Merc 230 22.8 4 140.8 95 3.92 3.15 22.9 1 0 4 2
# ch2_img1_ch2ROI.Merc 280 19.2 6 167.6 123 3.92 3.44 18.3 1 0 4 4
# ch2_img2_ch2ROI.Merc 280C 17.8 6 167.6 123 3.92 3.44 18.9 1 0 4 4
# ch2_img2_ch2ROI.Merc 450SE 16.4 8 275.8 180 3.07 4.07 17.4 0 0 3 3
I am importing data from multiple sheet Excel workbook using rio package in R. The code is super simple below:
library(rio)
my <- import_list("test.xls")
This is a list of data-frames. The problem is that the first row automatically becomes a header while I do not have any headers and it's just a data. In the description of package I didn't find the way to read worksheet with
header = FLASE
So, how can I convert this header to data row?
Assuming you can't import your data properly using that function (and I strongly recommend that you read the documentation for that function throughly, as the argument you're looking for is very likely to exist - it likely just has a different name than in read.table) you can access the "header" using colnames, then just rbind it on top of your data:
df2 <- rbind(colnames(mtcars), mtcars)
head(df2)
mpg cyl disp hp drat wt qsec vs am gear carb
1 mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21 6 160 110 3.9 2.62 16.46 0 1 4 4
Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.32 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
Then you can assign new column names with colnames(df2) <- ...:
# Assign numbers as column names
colnames(df2) <- paste0('V', seq_len(ncol(df2)))
head(df2)
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
1 mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21 6 160 110 3.9 2.62 16.46 0 1 4 4
Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.32 18.61 1 1 4 1
Hornet 4 Drive 21.4 6 258 110 3.08 3.215 19.44 1 0 3 1
Hornet Sportabout 18.7 8 360 175 3.15 3.44 17.02 0 0 3 2
I am working on this predifend dataset in R called mtcars. The head of this dataset looks like:
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.320 18.61 1 1 4 1
on the left side there is the name of each car. how come it doesn't have data type such as num or factor ? how can I apply that on a simillar datset?
the structure is as :
$ mpg : num 21 21 22.8 21.4 18.7 18.1 14.3 24.4 22.8 19.2 ...
$ cyl : num 6 6 4 6 8 6 8 4 4 6 ...
$ disp: num 160 160 108 258 360 ...
$ hp : num 110 110 93 110 175 105 245 62 95 123 ...
$ drat: num 3.9 3.9 3.85 3.08 3.15 2.76 3.21 3.69 3.92 3.92 ...
$ wt : num 2.62 2.88 2.32 3.21 3.44 ...
$ qsec: num 16.5 17 18.6 19.4 17 ...
$ vs : num 0 0 1 1 0 1 0 1 1 1 ...
$ am : num 1 1 1 0 0 0 0 0 0 0 ...
$ gear: num 4 4 4 3 3 3 3 4 4 4 ...
$ carb: num 4 4 1 1 2 1 4 2 2 4 ...
where name of the cars doesn't appear.
From what you share, the name of car was read in and assign as rowname of the dataset. As str() only print out the summary of columns in dataset so it will not print any thing related with the row names.
So if you want to have the car name as a column just add a column:
mtcars_trial <- mtcars
mtcars_trial$carname <- rownames(mtcars)
Regard to comment:
# this will assign the column as rowname for dataset
rownames(mtcars_trial) <- mtcars_trial$carname
# this will remove the carname column
mtcars_trial$carname <- NULL
The reason behind this is that here, the names of cars are stored as rownames and NOT as a column in the data.frame.
Based on Kunal Puri,
## values
mpg <- c(21.0, 21.0,22.8)
cyl <- c(6,6,4)
disp <- c(160,160,108)
hp <- c(110, 110, 93)
drat <- c(3.90,3.90,3.85)
wt <- c(2.620, 2.875, 2.320)
qsec <- c(16.46, 17.02,18061)
vs <- c(0,0,1)
am <- c(1,1,1)
gear <- c(4,4,4)
crab <- c(4,4,1)
## data.frame
df <- data.frame(mpg, cyl, disp, hp, drat, wt, qsec, vs, am, gear, crab, row.names = c("Mazda RX4", "Mazda RX4 Wag", "Datsun 710"))
## give a column name, take rownames values
df$cars <- rownames(df)
## row names removed
rownames(df) <- NULL
## rearranged df
df <- data.frame(df[12], df[1], df[2], df[3], df[4], df[5], df[6],df[7], df[8], df[9], df[10], df[11])
print(df)
output:
cars mpg cyl disp hp drat wt qsec vs am gear crab
1 Mazda RX4 21.0 6 160 110 3.90 2.620 16.46 0 1 4 4
2 Mazda RX4 Wag 21.0 6 160 110 3.90 2.875 17.02 0 1 4 4
3 Datsun 710 22.8 4 108 93 3.85 2.320 18061.00 1 1 4 1
Does it help?I guess it is the same thing as previous solution.