I have 5 raters who have rated 10 subjects. I've chosen to use the Light's Kappa to calculate inter-rater reliability because I have multiple raters. My issue is that when there is strong agreement between the raters, Light's kappa cannot be calculated due to lack of variability, and I've followed the updated post here which suggests using the Raters package in R when there is strong agreement.
My issue is that the Raters package calculates Fleiss' kappa, which from my understanding,is not suitable for inter-rater reliability where the same raters rate all subjects (such as in my case). My question is what type of kappa statistic I should be calculating in cases where there is strong agreement?
#install.packages("irr")
library(irr)
#install.packages('raters')
library(raters)
#mock dataset
rater1<- c(1,1,1,1,1,1,1,1,0,1)
rater2<- c(1,1,1,1,1,1,1,1,1,1)
rater3<- c(1,1,0,1,1,0,1,1,1,1)
rater4<- c(1,1,1,1,1,1,1,1,1,1)
rater5<- c(1,1,1,1,1,1,0,1,1,1)
df <- data.frame(rater1, rater2, rater3, rater4, rater5)
#light's kappa
kappam.light(df)
#kappa using raters package
data(df)
concordance(df, test = 'Normal')
Related
I have a table which has been generated in one of the publication and I am trying to do similar analysis for another study. please help me out to find CI by non-parametric percentile bootstrap method.
here is the table
I want to know how the ci for 1-RR is calculated.
here is the statistical explanation for it:
We used the Kaplan-Meier estimator18 to construct
cumulative incidence curves and to estimate the risk for
each outcome. The risks were compared via ratios and
differences. We estimated the risk ratio for each outcome
using only matched pairs in which both individuals were
still at risk 7 days after receipt of the third vaccine dose in
those vaccinated. We analysed outcomes in the full
population and in subgroups defined by strata of age,
sex, and number of comorbidities. 95% CIs were
calculated using the nonparametric percentile bootstrap
method with 1000 repetitions. The effectiveness of the
third dose was estimated as 1 – risk ratio. As a sensitivity
analysis, vaccine effectiveness was also estimated as
1 – incidence rate ratio derived from a Poisson regression
using the same dataset, with no further adjustment.
Analyses were done using R software (version 4.0.4)."
How to calculate CI.
I have 4 raters who have rated 10 subjects. Because I have multiple raters (and in my actual dataset, these 4 raters have rated the 10 subjects on multiple variables) I've chosen to use the Light's Kappa to calculate inter-rater reliability. I've run the light's kappa code shown below and included an example of my data.
My question is that the resulting kappa value (kappa=0.545) is fairly low even though the raters agree on almost all ratings? I'm not sure if there is some other way to calculate inter-rater reliability (e.g., pairwise combinations between raters?)
Any help is appreciated.
subjectID<- c(1,2,3,4,5,6,7,8,9,10)
rater1<- c(3,2,3,2,2,2,2,2,2,2)
rater2<-c(3,2,3,2,2,2,2,2,2,2)
rater3<- c(3,2,3,2,2,2,2,2,2,2)
rater4<-c(3,2,1,2,2,2,2,2,2,2)
df <- data.frame(subjectID, rater1, rater2, rater3, rater4)
kappam.light(df)
For more context, the raters evaluate each target on multiple dimensions where some of them are categorical and others are ordinal. We have at least 2 raters (may have more) who are evaluating these targets and we want to see if the raters are reliable.
Is there an R package that would help me analyze the interrater reliability on a multiquestion assessment?
I am conducting a propensity score match analysis on the outcome of two different new cancer treatments where the outcome is binary (cancer-free or not cancer free). Following successful matching I get my paired 2x2 contingency table for my outcome between my matched pairs which looks like below;
**Treatment 1**
Not-Cancer Free Cancer Free
**Treatment 2** Not-Cancer Free 50 39
Cancer Free 53 60
I'd like to compare the outcomes to figure out if one treatment is better than the other by comparing odds ratios of being cancer free. I've been advised to conduct a McNemar's test due to the matched nature of the data which I do and get a p-value of 0.17 (non-significant). However, I've also been advised that instead of simply using the odds ratio normally used for such 2x2 tables (B/C --> 39/53 = 0.78 OR) that I should calculate the odds ratio and 95% confidence intervals using the methods shown in Agresti Alan, Min Yongyi. Effects and non‐effects of paired identical observations in comparing proportions with binary matched‐pairs data. Statistics in medicine. 2004 Jan 15;23(1):65-75. as it accounts for the matched nature of the data.
Unfortunately after reading this paper numerous times (especially it's odds ratio section) I can't figure out what the equations given for the odds ratio and 95% CI calculations are that they are referring to but know that they must be in there somewhere as other papers have cited this paper when referring to their odds ratios but don't share their methodology making it difficult to traceback.
If anyone has read this paper or has experience with odds ratios for matched binary data, can you please let me know how I can go about to get matched pair odds ratios. Thank you incredibly much in advance!
You can use McNemar exact test for the paired data. A point they are making in the paper and what the exact test uses are the off-diagonal elements (b,c) in the calculations. You can use exact2x2 package (https://cran.r-project.org/web/packages/exact2x2/exact2x2.pdf) to get the test results with 95%CI:
library(exact2x2)
# Set up your data as matrix
x<-matrix(c(50,53,39,60),2,2)
mcnemar.exact(x)
Gives:
Exact McNemar test (with central confidence intervals)
data: x
b = 39, c = 53, p-value = 0.175
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.4738071 1.1339142
sample estimates:
odds ratio
0.7358491
I'm trying to perform propensity score matching on survey data. I'm aware of the package MatchIt which is able to make the matching procedure but can I include in some ways the individual weights? because if I don't consider them, a less relevant observation can be match with a more relevant one. Thank you!
Update 2020-11-25 below this answer.
Survey weights cannot be used with matching in this way. You might consider using weighting, which can accommodate survey weights. With weighting, you estimate the propensity score weights using a model that accounts for the survey weights, and then multiply the estimated weights by the survey weights to arrive at your final set of weights.
This can be done using the weighting companion to the MatchIt package, WeightIt (of which I am the author). With your treatment A, outcome Y (I assume continuous for this demonstration), covariates X1 and X2, and sampling weights S, you could run the following:
#Estimate the propensity score weights
w.out <- weightit(A ~ X1 + X2, data = data, s.weights = "S",
method = "ps", estimand = "ATT")
#Combine the estimated weights with the survey weights
att.weights <- w.out$weights * data$S
#Fit the outcome model with the weights
fit <- lm(Y ~ A, data = data, weights = att.weights)
#Estimate the effect of treatment and its robust standard error
lmtest::coeftest(fit, vcov. = sandwich::vcovHC)
It's critical that you assess balance after estimating the weights; you can do that using the cobalt package, which works with WeightIt objects and automatically incorporates the sampling weights into the balance statistics. Prior to estimating the effect, you would run the following:
cobalt::bal.tab(w.out, un = TRUE)
Only if balance was achieved would you continue on to estimating the treatment effect.
There are other ways to estimate weights besides using logistic regression propensity scores. WeightIt provides support for many methods, and almost all of them support sampling weights. The documentation for each method explains whether sampling weights are supported.
MatchIt 4.0.0 now supports survey weights through the s.weights, just like WeightIt. This supplies survey weights to the model used to estimate the propensity scores but otherwise does not affect the matching. If you want units to be paired with other units that have similar survey weights, you should enter the survey weights as a variable to match on or to place a caliper on.