mk.test() results to tabble/matrix R - r

I want to apply mk.test() to the large dataset and get results in a table/matrix.
My data look something like this:
Column A
Column B
...
ColumnXn
1
2
...
5
...
...
...
...
3
4
...
7
So far I managed to perform mk.test() for all columns and print the results:
for(i in 1:ncol(data)) {
print(mk.test(as.numeric(unlist(data[ , i]))))
}
I got all the results printed:
.....
Mann-Kendall trend test
data: as.numeric(unlist(data[, i]))
z = 4.002, n = 71, p-value = 6.28e-05
alternative hypothesis: true S is not equal to 0
sample estimates:
S varS tau
7.640000e+02 3.634867e+04 3.503154e-01
Mann-Kendall trend test
data: as.numeric(unlist(data[, i]))
z = 3.7884, n = 71, p-value = 0.0001516
alternative hypothesis: true S is not equal to 0
sample estimates:
S varS tau
7.240000e+02 3.642200e+04 3.283908e-01
....
However, I was wondering if it is possible to get results in a table/matrix format that I could save as excel.
Something like this:
Column
z
p-value
S
varS
tau
Column A
4.002
0.0001516
7.640000e+02
3.642200e+04
3.283908e-01
...
...
...
...
...
...
ColumnXn
3.7884
6.28e-05
7.240000e+02
3.642200e+04
3.283908e-01
Is it possible to do so?
I would really appreciate your help.

Instead of printing the test results you can store them in a variable. This variable holds the various test statistics and values. To find the names of the properties you can perform the test on the first row and find the property names using a string conversion:
testres = mk.test(as.numeric(unlist(data[ , 1])))
str(testres)
List of 9
$ data.name : chr "as.numeric(unlist(data[, 1]))"
$ p.value : num 0.296
$ statistic : Named num 1.04
..- attr(*, "names")= chr "z"
$ null.value : Named num 0
..- attr(*, "names")= chr "S"
$ parameter : Named int 3
..- attr(*, "names")= chr "n"
$ estimates : Named num [1:3] 3 3.67 1
..- attr(*, "names")= chr [1:3] "S" "varS" "tau"
$ alternative: chr "two.sided"
$ method : chr "Mann-Kendall trend test"
$ pvalg : num 0.296
- attr(*, "class")= chr "htest"
Here you see that for example the z-value is called testres$statistic and similar for the other properties. The values of S, varS and tau are not separate properties but they are grouped together in the list testres$estimates.
In the code you can create an empty dataframe, and in the loop add the results of that run to this dataframe. Then at the end you can convert to csv using write.csv().
library(trend)
# sample data
mydata = data.frame(ColumnA = c(1,3,5), ColumnB = c(2,4,1), ColumnXn = c(5,7,7))
# empty dataframe to store results
results = data.frame(matrix(ncol=6, nrow=0))
colnames(results) <- c("Column", "z", "p-value", "S", "varS", "tau")
for(i in 1:ncol(mydata)) {
# store test results in variable
testres = mk.test(as.numeric(unlist(mydata[ , i])))
# extract elements of result
testvars = c(colnames(mydata)[i], # column
testres$statistic, # z
testres$p.value, # p-value
testres$estimates[1], # S
testres$estimates[2], # varS
testres$estimates[3]) # tau
# add to results dataframe
results[nrow(results)+1,] <- testvars
}
write.csv(results, "mannkendall.csv", row.names=FALSE)
The resulting csv file can be opened in Excel.

Related

R - Alteryx - All columns in a tibble must be vectors

I'm using R on Alteryx to perform some statical analysis from my data.
It appears the error message " ! All Columns in a tibble must be vectors." as the following error message:
Does anybody can help me?
Below is my entire code:
library("tibble")
# Calling Data from Connection #1
data <- read.Alteryx("#1")
average_wilcox <- c("1","1","1","1","1","1","1")
# Creating data frame for in case it comes an empty table
df <- data.frame(average_wilcox)
#Verify if p-value is empty
# In case is different that empty, executes the steps for the Hypothesis Test for non-normal data
if (length(data$p.value) == 0) {
write.Alteryx(df, 1)
} else if (data$p.value != '') {
Week1 <- read.Alteryx("#2", mode="data.frame")
"&"
Week2 <- read.Alteryx("#3", mode="data.frame")
# MANN WHITNEY TEST (AVERAGE TEST FOR NON NORMAL)
Week1_data <- Week1$Wk1_feature_value
Week2_data <- Week2$Wk2_feature_value
# DEFINE VECTORS
week1 <- c(Week1_data)
week2 <- c(Week2_data)
merge(cbind(Week1, X=1:length(week1)),
cbind(Week2, X=1:length(week2)), all.y =T) [-1]
# MANN WHITNEY TEST (MEAN TEST FOR NON NORMAL)
average_wilcox <- wilcox.test(week1,week2, alternative='two.sided', conf.level=.95)
average_test <- tibble(average_wilcox)
average_test[] <- lapply(average_test, as.character)
write.Alteryx(average_test, 1)
}
#### NORMAL HYPOTHESIS TEST ####
# Calling Data from Connection #4
data1 <- read.Alteryx("#4")
df1 <- data.frame(Date=as.Date(character()),"p.value"=character(),User=character(),stringsAsFactors=FALSE)
# Verify if p-value is empty
# In case if different than empty, executes the steps for the Hypothesis Test for normal data
if(length(data1$p.value) == 0) {
write.Alteryx(df1, 3)
} else if (data1$p.value != '') {
Week1 <- read.Alteryx("#2", mode="data.frame")
"&"
Week2 <- read.Alteryx("#3", mode="data.frame")
# T TEST (MEAN TEST FOR NORMAL)
Week1_data <- Week1$Wk1_feature_value
Week2_data <- Week2$Wk2_feature_value
# DEFINE VECTORS
week1 <- c(Week1_data)
week2 <- c(Week2_data)
# T TEST (MEAN TEST FOR NORMAL)
t_test <- t.test(week1,week2, alternative='two.sided',conf.level=.95)
write.Alteryx(t_test,3)
}
Please, anybody knows what I have to do?
Many thanks,
Wil
Reason is that both wilcox.test and t.test returns a list of vectors, which may have difference in length. So, using that list in write.Alteryx is triggering the error as it expects a data.frame/tibble/data.table. e.g.
> str(t.test(1:10, y = c(7:20)))
List of 10
$ statistic : Named num -5.43
..- attr(*, "names")= chr "t"
$ parameter : Named num 22
..- attr(*, "names")= chr "df"
$ p.value : num 1.86e-05
$ conf.int : num [1:2] -11.05 -4.95
..- attr(*, "conf.level")= num 0.95
$ estimate : Named num [1:2] 5.5 13.5
..- attr(*, "names")= chr [1:2] "mean of x" "mean of y"
$ null.value : Named num 0
..- attr(*, "names")= chr "difference in means"
$ stderr : num 1.47
$ alternative: chr "two.sided"
$ method : chr "Welch Two Sample t-test"
$ data.name : chr "1:10 and c(7:20)"
- attr(*, "class")= chr "htest"
> x <- c(0.80, 0.83, 1.89, 1.04, 1.45, 1.38, 1.91, 1.64, 0.73, 1.46)
> y <- c(1.15, 0.88, 0.90, 0.74, 1.21)
> str(wilcox.test(x, y, alternative = "g") )
List of 7
$ statistic : Named num 35
..- attr(*, "names")= chr "W"
$ parameter : NULL
$ p.value : num 0.127
$ null.value : Named num 0
..- attr(*, "names")= chr "location shift"
$ alternative: chr "greater"
$ method : chr "Wilcoxon rank sum exact test"
$ data.name : chr "x and y"
- attr(*, "class")= chr "htest"
An option is to convert the output from both t.test and wilcox.test to a data.frame/tibble. tidy/glance from broom does this
...
library(broom)
average_wilcox <- tidy(wilcox.test(week1,week2, alternative='two.sided', conf.level=.95))
write.Alteryx(average_wilcox, 1)
...
t_test <- tidy(t.test(week1,week2, alternative='two.sided',conf.level=.95))
write.Alteryx(t_test,3)

Return value of fitdistr function explanation

I have the following generated dataset
library(MASS)
df <- data.frame(product= sample(x= c("toyota","honda","nissan","bmw"),size = 1000 ,replace = TRUE),
parameter = sample(x= c("X","Y", "A"),size = 1000 ,replace = TRUE),
value= rgamma(1000, shape = 5, rate = 0.1))
and I want to fit the lognormal distribution on column "value" and
I use the following code
dist_par <- fitdistr(unlist(df["value"]), "lognormal")
the result is something like below:
meanlog sdlog
3.8416 0.4292
(0.0458) (0.0324)
I have two questions:
I read the help and I guess that the meanlog and sdlog estimations are shown on the first row:
meanlog sdlog
3.8416 0.4292
but the second row of numbers (numbers in parentheses) are confusing, what are they?
meanlog sdlog
.... ....
(0.0458) (0.0324)
I know the result of fitdistr is a list but I don't know how to have access to those four values. For instance how can I get 3.8416 ?
If I run
dist_par[1]
then I get
meanlog sdlog
3.842 0.429
and if I run:
dist_par[1,1]
then I get the following error:
Error in dist_par[1, 1] : incorrect number of dimensions
According to the ?fitdistr documentation
An object of class "fitdistr", a list with four components,
estimate - the parameter estimates,
sd - the estimated standard errors,
vcov - the estimated variance-covariance matrix, and
loglik - the log-likelihood.
This would be evident if we check the structure
str(out)
List of 5
$ estimate: Named num [1:2] 3.801 0.455
..- attr(*, "names")= chr [1:2] "meanlog" "sdlog"
$ sd : Named num [1:2] 0.0144 0.0102
..- attr(*, "names")= chr [1:2] "meanlog" "sdlog"
$ vcov : num [1:2, 1:2] 0.000207 0 0 0.000103
..- attr(*, "dimnames")=List of 2
.. ..$ : chr [1:2] "meanlog" "sdlog"
.. ..$ : chr [1:2] "meanlog" "sdlog"
i.e. the print method returns the 'estimatte and inside the parentheses the sd and as they are list the [1,1] doesn't work, we need to use standard extraction methods i.e. either $ or [[
> out
meanlog sdlog
3.80075311 0.45468543
(0.01437842) (0.01016708)
> out$estimate
meanlog sdlog
3.8007531 0.4546854
> out$estimate[["meanlog"]]
[1] 3.800753
> out$sd
meanlog sdlog
0.01437842 0.01016708
i.e inside the list, the elements are just named vectors, so use the [ or [[ to extract by name

Extract a column from lme4 summary in R

I was wondering what is the most efficient way to extract (not print like HERE) only the Std.Dev. column from the vc object below as a vector?
library(lme4)
library(nlme)
data(Orthodont, package = "nlme")
fm1 <- lmer(distance ~ age + (age|Subject), data = Orthodont)
vc <- VarCorr(fm1) ## extract only the `Std.Dev.` column as a vector
The structure of 'vc' suggests it is a list with single element 'Subject' and the 'stddev' is an attribute
str(vc)
#List of 1
# $ Subject: num [1:2, 1:2] 6.3334 -0.3929 -0.3929 0.0569
# ..- attr(*, "dimnames")=List of 2
# .. ..$ : chr [1:2] "(Intercept)" "age"
# .. ..$ : chr [1:2] "(Intercept)" "age"
# ..- attr(*, "stddev")= Named num [1:2] 2.517 0.239 ####
So, extract the attribute directly
attr(vc$Subject, "stddev")
and the residual standard deviation is an outside attribute
attr(vc, "sc")
#[1] 1.297364
If we combine them with c, we get a single vector
c(attr(vc$Subject, "stddev"), attr(vc, "sc"))
# (Intercept) age
# 2.5166317 0.2385853 1.2973640
Wrap with as.numeric/as.vector to remove the names as it is a named vector
Or use attributes
c(attributes(vc)$sc, attributes(vc$Subject)$stddev)
If you want the three elements in the column, you can use:
as.numeric(c(attr(vc[[1]], "stddev"), attr(vc, "sc")))

How to make contingency table over a list in r

The main exposure variable is aff. I want to get contingency tables for aff and all variables in the varlist. Then I want to do chi-square test using these contingency tables. My codes are following:
name=names(data)
varlist=name[11:40]
models=lapply(varlist, function(x) {
chisq.test(table(substitute(data$i,list(i = as.name(x))),data$aff))
})
lapply(models, summary)
But I got error
Error in unique.default(x, nmax = nmax) :
unique() applies only to vectors
How to fix this?
I think you're over-complicating things by using substitute and such. Without your data, I'll try with mtcars, using cyl as the exposure variable.
data <- mtcars
name <- names(data)
ev <- "cyl"
varlist <- name[ name != ev ]
models <- lapply(varlist, function(nm) {
chisq.test(table(data[[nm]], data[[ev]]))
})
# Warning messages:
# 1: In chisq.test(table(data[[nm]], data[[ev]])) :
# Chi-squared approximation may be incorrect
(because I'm using a bad example for the test, there are a lot of warnings here; this can be ignored when using mtcars because it is really not a good dataset for this test.)
summaries <- lapply(models, summary)
str(summaries[1:2])
# List of 2
# $ : 'summaryDefault' chr [1:9, 1:3] " 1" " 1" " 1" " 1" ...
# ..- attr(*, "dimnames")=List of 2
# .. ..$ : chr [1:9] "statistic" "parameter" "p.value" "method" ...
# .. ..$ : chr [1:3] "Length" "Class" "Mode"
# $ : 'summaryDefault' chr [1:9, 1:3] " 1" " 1" " 1" " 1" ...
# ..- attr(*, "dimnames")=List of 2
# .. ..$ : chr [1:9] "statistic" "parameter" "p.value" "method" ...
# .. ..$ : chr [1:3] "Length" "Class" "Mode"
Supposing your data is like mtcars, where vs, am, gear, and carb are categorical variables, you can create a function like so:
df_list_f <- function(x) chisq.test(table(df2$cyl, x))
df2 <- mtcars[,8:11] # df2 contains the columns vs, am, gear and carb
lapply(df2, df_list_f)

Rename x,y vectors in t-test R

I am performing a t-test in R
out <- t.test(x=input1, y=input2, alternative=c("two.sided","less","greater"), mu=0, paired=TRUE, conf.level = 0.95)
It gives the result
Paired t-test
data: input1 and input2
t = -1.1469, df = 7, p-value = 0.2891
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.15100900 0.05236717
sample estimates:
mean of the differences
-0.04932091
I need to change the names of the data in result. e.g.,
data: Fruits and Vegetables
Please, anyone give me an idea to include some attribute in t.test to change the data names.
With some dummy data
set.seed(1)
input1 <- rnorm(20, mean = -1)
input2 <- rnorm(20, mean = 5)
It would be easier just to rename or create objects with the desired names:
Fruits <- input1
Vegetables <- input2
t.test(x = Fruits, y = Vegetables, paired = TRUE, alternative = "two.sided")
Paired t-test
data: Fruits and Vegetables
t = -18.6347, df = 19, p-value = 1.147e-13
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-6.454791 -5.151218
sample estimates:
mean of the differences
-5.803005
But if you really want to do this after the fact, then grab the object returned by t.test():
tmp <- t.test(x = input1, y = input2, paired = TRUE, alternative = "two.sided")
Look at the structure of the object tmp
> str(tmp)
List of 9
$ statistic : Named num -18.6
..- attr(*, "names")= chr "t"
$ parameter : Named num 19
..- attr(*, "names")= chr "df"
$ p.value : num 1.15e-13
$ conf.int : atomic [1:2] -6.45 -5.15
..- attr(*, "conf.level")= num 0.95
$ estimate : Named num -5.8
..- attr(*, "names")= chr "mean of the differences"
$ null.value : Named num 0
..- attr(*, "names")= chr "difference in means"
$ alternative: chr "two.sided"
$ method : chr "Paired t-test"
$ data.name : chr "input1 and input2"
- attr(*, "class")= chr "htest"
and note the data.name component. We can replace that with a character string:
tmp$data.name <- "Fuits and Vegetables"
The print tmp:
> tmp
Paired t-test
data: Fuits and Vegetables
t = -18.6347, df = 19, p-value = 1.147e-13
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-6.454791 -5.151218
sample estimates:
mean of the differences
-5.803005

Resources