Extend HITs until agreement MTurkR - r

Question

I am using MTurkR to post HITs to mTurk and I am having trouble extending the HITs until there is consensus among workers (or until a total of 5 HITs have been posted).
When two different responses are entered, the HIT is not extended. My code follows from MTurkR documentation pg. 45. My code is as follows:
# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
#format for sandbox question. Get this from MTURK site
annotation.v="Question1"
assignments.v="2"
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)
# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("Question1"),
QuestionAgreementThreshold=49, #at least 50% agree
ExtendIfHITAgreementScoreIsLessThan=50,
ExtendMinimumTimeInSeconds=3600,
ExtendMaximumAssignments=5,
DisregardAssignmentIfRejected=TRUE)
policya=do.call(GenerateHITReviewPolicy,TurkAgreement)
# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
hits.i=CreateHIT(
hitlayoutid=layout,
hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
annotation=annotation.v[i],
assignments=assignments.v,
title=title.v,
description=description.v,
reward=reward.v,
duration=duration.v,
expiration=expiration.v,
keywords=keywords.v,
auto.approval.delay=auto.approval.delay.v,
qual.req=qualReqs,
hit.review.policy=policya,
sandbox=sandbox.v)
hits=rbind(hits,hits.i)}
The code generates 2 HITs (as specified by assignments.v) but the HIT doesn't extend.
My session info is below:
> sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] MTurkR_0.6
loaded via a namespace (and not attached):
[1] digest_0.6.4 RCurl_1.95-4.3 tcltk_3.1.1 XML_3.98-1.1

Answer 1

First a couple of key points about the code you posted:
Because you are adding the hit.review.policy you don't need the assignments argument. Two HITs will immediately populate with just this statement.
Before you define the terms for the hit.review.policy take a look at Amazon's documentation for this. Pay particular attention to the values for QuestionAgreementThreshold. Because values greater than this value are considered agreed answers, your original code indicated values greater than 49% were agreed answers. Also examine ExtendIfHITAgreementScoreIsLessThan this value should probably +1 of the value for QuestionAgreementThreshold in order to capture all possible agreement values.
Finally, be sure the answer values for HIT responses correspond to the QuestionIds object. When I view the answers to my HITs, the answers are referred to as "question", so I've replaced this value here.
With all of that said, here is the revised code that now properly extends the HITs:
# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
#format for sandbox question. Get this from MTURK site
annotation.v="question" #NOTE CHANGE HERE
#assignments.v="2" #NOTE CHANGE HERE (COMMENTED THIS OUT)
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)
# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("question"), #NOTE CHANGE HERE
QuestionAgreementThreshold=50, #at least 50% agree #NOTE CHANGE HERE
ExtendIfHITAgreementScoreIsLessThan=51, #NOTE CHANGE HERE
ExtendMinimumTimeInSeconds=3600,
ExtendMaximumAssignments=5,
DisregardAssignmentIfRejected=TRUE)
policya=do.call(GenerateHITReviewPolicy,TurkAgreement)
# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
hits.i=CreateHIT(
hitlayoutid=layout,
hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
annotation=annotation.v[i],
assignments=assignments.v,
title=title.v,
description=description.v,
reward=reward.v,
duration=duration.v,
expiration=expiration.v,
keywords=keywords.v,
auto.approval.delay=auto.approval.delay.v,
qual.req=qualReqs,
hit.review.policy=policya,
sandbox=sandbox.v)
hits=rbind(hits,hits.i)}
Also note that the documentation on the MTurkR Package pg. 44 (end) is confusing. The document provides the following example:
lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 49, # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 50,
...
But the argument QuestionAgreementThreshold actually specifies at least 49% agreement (i.e., the HIT will not extend if 2 Turks split the answers). Unless this is the intention, it might be better to use the following code:
lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 50, #NOTE CHANGE HERE # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 51, #NOTE CHANGE HERE
...