I am trying to scrape my trip history data on Capital Bikeshare Website. I have to log in and go to the trips menu to see the data. but i get this error:
> `No encoding supplied: defaulting to UTF-8.
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘readHTMLTable’ for signature ‘"xml_document"’
Here's my code.
> library(httr)
> library(XML)
> handle <- handle("https://www.capitalbikeshare.com/")
> path <-"profile/trips"
> login <- list( profile_login="myemail", profile_pass ="mypassword", profile_redirect_url="https://secure.capitalbikeshare.com/profile/trips/QNURCMF2Q6")
> response <- POST(handle = handle, path = path, body = login)
> readHTMLTable(content(response))
I also tried using rvest but then I kept getting the "Error: Unknown field names: _username, _password" error. Which field should I use here? I tried Id, name, etc and still didn't work.
For a start the member login page is different than the intro page which you have listed above:
This may not be correct but try this as a possible rvest starting point:
login<-"https://secure.capitalbikeshare.com/profile/login"
library(rvest)
pgsession<-html_session(login)
pgform<-html_form(pgsession)[[1]]
#update user id and password in the next line
filled_form<-set_values(pgform, "_username"="myemail#gmail.com", "_password"="password")
submit_form(pgsession, filled_form)
Once you login in then one can use the jump_to function to move to the desired pages:
page<-jump_to(pgsession, newurl) #newurl will be the address where to go to next.
Hope this helps, if this does not work, leave a comment and I'll delete the post.
Related
I am trying to access a table that shows the level of adoption of microsoft licenses, I developed a script but when I ran it the answer gave me an error.
Does anybody know how I can fix it?
library(httr)
library(XML)
handle <- handle("https://login.microsoftonline.com/")
path <- "common/login"
login <- list(
amember_login = "username"
,amember_pass = "password*"
,amember_redirect_url =
"https://partner.microsoft.com/es-es/dashboard/analytics/usageanalytics"
)
response <- POST(handle = handle, path = path, body = login)
readHTMLTable(content(response))
the error is the following
Error in (function (classes, fdef, mtable) :
unable to find an inherited method for function ‘readHTMLTable’ for signature ‘"xml_document"’
I have some problem with the google_geocode function in googleway package. At first I've just tried it with a simple address:
key <- MyKey
df <- google_geocode(address = "5 Rue J-F Kennedy 8332 Luxembourg",
key = key,
simplify = TRUE)
And got the following message:
Error in value[[3L]](cond) :
There was an error downloading results. Please manually check the following URL is valid by entering it into a browswer. If valid, please file a bug report citing this URL (note: your API key has been removed, so you will need to add that back in)
https://maps.googleapis.com/maps/api/geocode/json?&address=5+rue+j-f+kennedy+8332+luxembourg&key=
I'd pasted it to my browser, and it was OK. Any idea?
I am trying to scrape the http://www.emedexpert.com/lists/brand-generic.shtml web page for brand and generic drug names
library(httr)
library(rvest)
session <- read_html("http://www.emedexpert.com/lists/brand-generic.shtml")
form1 <- html_form(session)[[2]]
form2 <- set_values(form1, brand = "tylenol")
submit_form(session, form2)
however this results in the error message:
Error in xml2::url_absolute(form$url, session$url) :
not compatible with STRSXP
Therefore, based on this answer to the same error message ("Error: not compatible with STRSXP" on submit_form with rvest) I added a session$url as follows:
session$url <- "http://www.emedexpert.com/lists/brand-generic.shtml" # added from S.Ov
but I still get the same error message. So I tried also adding various permutations of also adding form2$url such as these
form2$url <- "http://www.emedexpert.com/lists/brand-generic.shtml"
form2$url <- ""
form2$url <- "/"
submit_form(session, form2)
At this point, the error message goes away and I obtain a web page which contain most of the desired web page. However it seems to completely lack the table of brand and generic names.
Any suggestions?
Yes #hackR, RSelenium is not always the answer.
library(rvest)
url<-"http://www.emedexpert.com/lists/bg.php?myc"
page<-html_session(url)
table<-html_table(read_html(page))[[1]]
This could help you I hope.
I've searched around stackoverflow and github but haven't seen a solution to this one.
session <- read_html("http://www.whitepages.com")
form1 <- html_form(session)[[1]]
form2 <- set_values(form1, who = "john smith")
submit_form(session, form)
After the submit form line, I get the following:
Submitting with '<unnamed>'
Error: not compatible with STRSXP
I've pieced together that this error is usually from mismatched types (strings and numeric, for example), but I can't tell where that might be happening.
Any help would be greatly appreciated!
I just had this problem myself, and I found that the error was happening when submit_form() called the function rvest:::submit_request(), which tries to run this line:
xml2::url_absolute(form$url, session$url)
In this line, R tries to create an absolute url which throws an error because either form$url or session$url is NULL. In my case, session$url was NULL for some reason. So you should probably try:
session$url <- "http://www.whitepages.com"
submit_form(session, form2)
Try to change the URL of your form into an empty string
form2$url <- "" before submitting it.
I am trying and failing to use RCurl to automate the process of fetching a spreadsheet from a web site, China Labour Bulletin's Strike Map.
Here is the URL for the spreadsheet with the options set as I'd like them:
http://strikemap.clb.org.hk/strikes/api.v4/export?FromYear=2011&FromMonth=1&ToYear=2015&ToMonth=6&_lang=en
Here is the code I'm using:
library(RCurl)
temp <- tempfile()
temp <- getForm("http://strikemap.clb.org.hk/strikes/api.v4/export",
FromYear="2011", FromMonth="1",
ToYear="2015", ToMonth="6",
_lang="en")
And here is the error message I get in response:
Error: unexpected input in:
" ToYear=2015, ToMonth=6,
_"
Any suggestions on how to get this to work?
Try enclosing _lang with a backtick.
temp <- getForm("http://strikemap.clb.org.hk/strikes/api.v4/export",
FromYear="2011",
FromMonth="1",
ToYear="2015",
ToMonth="6",
`_lang`="en")
I think R has trouble on the argument starting with an underscore. This seems to have worked for me.