I can see the contents of the API call in webbrowser, but getting this error with jsonlite package: read_json.
Error in open.connection(con, "rb") : connection cannot be opened
Añso: Warning message:
In open.connection(con, "rb") :
cannot open URL 'https://www.plazavea.com.pe/api/catalog_system/pub/products/search?fq=C:/678/687/&_from=21&_to=41&O=OrderByScoreDESC&': HTTP status was '206 Partial Content'
Code::
library(rvest)
library(tidyverse)
library(jsonlite)
api_request <- "https://www.plazavea.com.pe/api/catalog_system/pub/products/search?fq=C:/678/687/&_from=21&_to=41&O=OrderByScoreDESC&"
product_data <- jsonlite::read_json(api_request)
Use httr and then extract as = 'text' and pass over to parse_json(), or simply specify as = 'parsed' in the content() call on the response object.
library(httr)
api_request <- "https://www.plazavea.com.pe/api/catalog_system/pub/products/search?fq=C:/678/687/&_from=21&_to=41&O=OrderByScoreDESC&"
product_data <- content(httr::GET(api_request), as = 'parsed')
Related
I like to use a Web Feature Service (WFS) [1] to get soil informations. But I am very unsure how to do it right, maybe you can give me a hint on how to get all available informations.
I can query bmf:Musterstueck and convert it to a Simple Feature. But if I do the same with so:SoilHorizon or so:ObservedSoilProfile I get the error message like
Error: Cannot open "https://via.bund.de/bmf/inspire/so/wfs?service=wfs&version=2.0.0&request=GetFeature&typename=so%3ASoilHorizon"; The file doesn't seem to exist.
In addition: Warning message:
In CPL_read_ogr(dsn, layer, query, as.character(options), quiet, :
GDAL Error 1: HTTP error code : 403
Thank you
Christof
[1] https://via.bund.de/bmf/inspire/so/wfs?service=wfs&request=getcapabilities
name title
bmf:Musterstueck Musterstücke der Bodenschätzung
so:SoilHorizon INSPIRE SoilHorizon
so:ObservedSoilProfile INSPIRE ObservedSoilProfile
library(tidyverse)
library(sf)
library(ows4R)
library(httr)
wfs = WFSClient$new("https://via.bund.de/bmf/inspire/so/wfs", "2.0.0", logger = "INFO")
wfs$getFeatureTypes(pretty = TRUE)
url <- parse_url("https://via.bund.de/bmf/inspire/so/wfs")
url$query <- list(service = "wfs",
version = "2.0.0",
request = "GetFeature",
typename = "bmf:Musterstueck")
request <- build_url(url)
request
bmf_musterstueck <- read_sf(request)
bmf_musterstueck
How can I navigate to a link, click a link, and scrape data from there?
I tried this code, without success.
library("RSelenium")
startServer()
mybrowser <- remoteDriver()
mybrowser$open()
mybrowser$navigate("https://finance.yahoo.com/quote/SBUX/balance-sheet?p=SBUX")
# click 'Quarterly' button...
Something that is kind of close, is this.
Testing updated code now; results below.
> rm(list=ls())
>
>
> library("RSelenium")
> startServer()
Error: startServer is now defunct. Users in future can find the function in
file.path(find.package("RSelenium"), "examples/serverUtils"). The
recommended way to run a selenium server is via Docker. Alternatively
see the RSelenium::rsDriver function.
> mybrowser <- remoteDriver()
> mybrowser$open()
[1] "Connecting to remote server"
Error in checkError(res) :
Undefined error in httr call. httr output: Failed to connect to localhost port 4444: Connection refused
> mybrowser$navigate("https://finance.yahoo.com/quote/SBUX/balance-sheet?p=SBUX")
Error in checkError(res) :
Undefined error in httr call. httr output: length(url) == 1 is not TRUE
> mybrowser$findElement("xpath", "//button[text() = '
+
+ OK
+ ']")$clickElement()
Error in checkError(res) :
Undefined error in httr call. httr output: length(url) == 1 is not TRUE
> mybrowser$findElement("xpath", "//span[text() = 'Quarterly']")$clickElement()
Error in checkError(res) :
Undefined error in httr call. httr output: length(url) == 1 is not TRUE
>
I think it might be the case you run into this on the website.
You can just "click" the OK button via:
mybrowser$findElement("xpath", "//button[text() = '
OK
']")$clickElement()
And then you can click "quarterly" via:
mybrowser$findElement("xpath", "//span[text() = 'Quarterly']")$clickElement()
(Hint: To identify these kind of errors it can be helpful to check the current state of the browser via: remDr$screenshot(TRUE).)
I am not sure it is up to date, but certain data is also available via the API, you could check the quantmod package to get an easier access.
Full example:
library("RSelenium")
startServer()
mybrowser <- remoteDriver()
mybrowser$open()
mybrowser$navigate("https://finance.yahoo.com/quote/SBUX/balance-sheet?p=SBUX")
mybrowser$findElement("xpath", "//button[text() = '
OK
']")$clickElement()
mybrowser$findElement("xpath", "//span[text() = 'Quarterly']")$clickElement()
i'm trying to run a API request for a number of parameters with the lapply function in R.
However, when i run this function, i get the error " Error in file(con, "r") : cannot open the connection"
Google suggests using setInternet2(TRUE) to fix this issue, however, i get the error: Error: 'setInternet2' is defunct.
See help("Defunct"
localisedDestinationNameForGivenLang <- function (LocationId) {
gaiaURL <- paste0("https://URL/",LocationId, "?geoVersion=rwg&lcid=", "1036",
"&cid=geo&apk=explorer")
print(LocationId)
localisation <- fromJSON(gaiaURL)
}
lapply(uniqueLocationId, localisedDestinationNameForGivenLang)
Can someone suggest a fix please?
Here's a sample of how you could identify which sites are throwing errors while still getting response from the ones that don't:
urls = c("http://citibikenyc.com/stations/test", "http://citibikenyc.com/stations/json")
grab_data <- function(url) {
out <- tryCatch(
{fromJSON(url)},
error=function(x) {
message(paste(url, x))
error_msg = paste(url, "threw an error")
return(error_msg)
})
return(out)
}
result <- lapply(urls, grab_data)
result will be a list that contains API response for urls that work, and error msg for those that don't.
I wan to download a file using the following request:
request <- 'http://earthserver.ecmwf.int/rasdaman/ows?service=WCS&version=2.0.1&request=ProcessCoverages&query=for c in (geff_fire_weather_index) return encode(c[Lat(35:47),Long(6:18),ansi("2015-01-01T00:00":"2015-12-31T00:00")],"netcdf")'
download.file(url = request, destfile = "rea.nc")
If I run this using the browser it works no problem but in R it fails with the following error message:
downloaded 0 bytes
Error in download.file(url = request2, destfile = "rea.nc") :
cannot download all files
In addition: Warning message:
In download.file(url = request2, destfile = "rea.nc") :
URL 'http://earthserver.ecmwf.int/rasdaman/ows?service=WCS&version=2.0.1&request=ProcessCoverages&query=for c in (geff_fire_weather_index) return encode(c[Lat(35:47),Long(6:18),ansi("2015-01-01T00:00":"2015-12-31T00:00")],"netcdf")': status was 'Couldn't connect to server'
Is there a way to fix this?
When I execute htmlTreeParser(), I am getting this error:
Error in htmlTreeParse(webpage, error = function(...) { :
error in creating parser for
In addition: Warning message:
XML content does not seem to be XML: ''
Kindly someone help me to debug it.
library(RCurl)
library(XML)
theurl <- "http://www.forbes.com/powerful-brands/list/"
webpage <- getURL(theurl)
webpage <- readLines(tc <- textConnection(webpage)); close(tc)
pagetree <- htmlTreeParse(webpage, error=function(...){}, useInternalNodes = TRUE)
The problem is clearly stated in the error message. The url you provided does not point directly to an XML page.
Try this:
theurl <- "https://www.forbes.com/powerful-brands/list/#tab:rank"