Error when attempting to submit form with Rvest - r

Question

Adapting this SO answer, I'm trying to use rvest to generate a form to scrape the resulting page. I keep coming up with an error.
library(rvest)
url <- "https://iemweb.biz.uiowa.edu/pricehistory/pricehistory_SelectContract.cfm?market_ID=214"
pg.session <- html_session(url)
pg.form <- html_form(html(pg.session))
filled_form <- set_values(pg.form[[1]],
Month = "8",
Year = "1")
out <- submit_form(session = pg.session, pg.form)
returns this error
Submitting with ''
Error in if (!(submit %in% names(submits))) { :
argument is of length zero
What am I doing wrong?

Answer 1

Well, for one thing, you are not submitting the form you actually filled in and you are also attempting to pass in a list of forms rather than a form, but also it appears there may be a bug in the code that doesn't recognize submit buttons with upper case tags. In this case, the HTML has the code
<INPUT TYPE="SUBMIT" VALUE="Get Prices">
and the submit_form codes calls submit_request which looks for submit buttons via
submits <- Filter(function(x) identical(x$type, "submit"),
form$fields)
and since it checks for values identical to "submit" it's not finding "SUBMIT"
sapply(pg.form[[1]]$fields, function(x) x$type)
# $Market_ID
# [1] "HIDDEN"
# $Month
# NULL
# $Year
# NULL
# $`NULL`
# [1] "SUBMIT"
The easiest thing might be to change it ourselves
filled_form <- set_values(pg.form[[1]],
Month = "08",
Year = "2007")
filled_form$fields[[4]]$type <- "submit"
The other problem is that this version has a bug in the way the URL for the form us resolved. we can fix it with
# incorrectly was: url <- XML::getRelativeURL(session$url, form$url)
body(submit_form)[[3]]<-quote(url <- XML::getRelativeURL(form$url, session$url))
And now finally we can submit the request
out <- submit_form(session = pg.session, filled_form)
# out %>% html_table()
(Tested with rvest_0.2.0.9000)