I am trying to learn how to collect data from the Web into R. There's a website from Brazilian Ministery of Health that is sharing the numbers of the disease here in Brazil, it is a public portal.
COVIDBRASIL
So, on this page, I am interested in the graph that displays the daily reporting of cases here in Brazil. Using the inspector on Google Chrome I can access the JSON file feeding the data to this chart, my question is how could I get this file automatically with R. When I try to open the JSON in a new tab outside the inspector "Response" tab, I get an "Unauthorized" message. There is any way of doing this or every time I would have to manually copy the JSON from the inspector and update my R script?
In my case, I am interested in the "PortalDias" response. Thank you.
URL PORTAL DIAS
You need to set some headers to prevent this "Unauthorized" message. I copied them from the 'Headers' section in the browser 'Network' window.
library(curl)
library(jsonlite)
url <- "https://xx9p7hp1p7.execute-api.us-east-1.amazonaws.com/prod/PortalDias"
h <- new_handle()
handle_setheaders(h, Host = "xx9p7hp1p7.execute-api.us-east-1.amazonaws.com",
`Accept-Encoding` = "gzip, deflate, br",
`X-Parse-Application-Id` = "unAFkcaNDeXajurGB7LChj8SgQYS2ptm")
fromJSON(rawToChar(curl_fetch_memory(url, handle = h)$content))
# $results
# objectId label createdAt updatedAt qtd_confirmado qtd_obito
# 1 6vr9rUPbd4 26/02 2020-03-25T16:25:53.970Z 2020-03-25T22:25:42.967Z 1 123
# 2 FUNHS00sng 27/02 2020-03-25T16:27:34.040Z 2020-03-25T22:25:55.169Z 0 34
# 3 t4qW51clpj 28/02 2020-03-25T19:08:36.689Z 2020-03-25T22:26:02.427Z 0 35
# ...
Related
I am learning how to use API in R and it is going well for the most part, but I am having trouble getting any data from the league of legends API.
For reference, I used this article as a start (https://www.dataquest.io/blog/r-api-tutorial/) and cop
res <- GET("http://api.open-notify.org/astros.json")
res
This worked just fine and has a 200 status, but I am not interested in that data.
What I want is data about league of legends, so I am trying to use:
base.url <- "https://na1.api.riotgames.com"
path <- "/lol/champion-mastery/v4/champion-masteries/by-summoner/"
API_Key <- read.table("riotkey.txt")
API_KEY <- API_Key$V1
Summoner_ID <- read.table("summonerID.txt")
SUMMONER_ID <- Summoner_ID$V1
path <- paste0(path,SUMMONER_ID)
LoL_API_Test <- GET(base.url, path = path,
add_headers(Authorization = API_KEY))
LoL_API_Test
This is Riot's explanation for the 403 error - Forbidden. "This error indicates that the server understood the request but refuses to authorize it. There is no distinction made between an invalid path or invalid authorization credentials (e.g., an API key)"
I am certain that my API key and summoner ID are correct.
So I assume the issue has to be with how I am requesting the data.
What am I doing wrong?
This particular API expects the API key to be passed in a header called "X-Riot-Token", not "Authorization". Change your call to
LoL_API_Test <- GET(base.url, path = path,
add_headers("X-Riot-Token" = API_KEY))
I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
{"section":[1,4],"period":20170501}
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
I am looking to scrape the web pages of several golfers on wikipedia. Some of the player names are identical to other people and, in that case, the URL has to have the text '(golfer)' appended to reach the right page
I am looking to try every player with this addition and if the page does not exist then revert to the plain name
There may be a better approach, but I had hoped that I could obtain the content size of the response. If it did not reach a certain level e.g. 2kb then that was not a valid page
library(httr)
base_url <- "https://en.wikipedia.org/w/api.php"
query_params <- list(action = "parse",
page = "Patrick Reed_(golfer)",
format = "xml")
resp <- GET(url = base_url, query = query_params)
resp
Response [https://en.wikipedia.org/w/api.php?
action=parse&page=Patrick%20Reed_%28golfer%29&format=xml]
Date: 2018-04-08 08:35
Status: 200
Content-Type: text/xml; charset=utf-8
Size: 402 B
So a suitably low size gets reported but I am not sure how to get to it when I expand the list, resp. According to httr quickstart there is something in the headers referencing content-length but I am unable to discover it
TIA
i'm trying to make an order through the Binance API in R, all of my other endpoints work but this,I'm getting error HTTP/1.1 400 Bad Request ....please help.
**url="https://api.binance.com/api/v3/order/test"
timestamp <- GET(
url = "https://api.binance.com",
path = "/api/v1/time")
timestamp=content(timestamp, as="parsed")
timestamp=timestamp$serverTime
signature <- openssl::sha256(postmsg, key=secretKey)
postmsg <- paste0("timestamp=", timestamp, "&recvWindow=", recvWindow,"&symbol=",symbol,
"&side=",side,"&type=",type,"&quantity=",quantity,"&timeInForce=",timeInForce)
signature <- openssl::sha256(postmsg, key=secretKey)
order = POST(
url = url,
content_type('application/json'), add_headers(.headers = c("X-MBX-APIKEY"=apiKey)),
query=list(signature=signature),
encode = 'json'
)**
The most common cause of this (https://github.com/ccxt/ccxt/issues/663) is that each order needs to be a minimum cost - either 0.001 BTC, 0.01 ETH or 1 BNB or 1 USDT depending upon which one of the 4 quote symbols you're using.
I also find myself running into this when submitting amounts less than 1 on symbol pairs that only accept integers.
You might also find their developer telegram group useful.
I wish to use read.csv to read a google doc spreadsheet.
I try using the following code:
data_url <- "http://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv"
read.csv(data_url)
Which results in the following error:
Error in file(file, "rt") : cannot open the connection
I'm on windows 7. And the code was tried on R 2.12 and 2.13
I remember trying this a few months ago and it worked fine.
Any suggestion what might be causing this or how to solve it?
Thanks.
It might have something to do with the fact that Google is reporting a 302 temporarily moved response.
> download.file(data_url, "~/foo.csv", method = "wget")
--2011-04-29 18:01:01-- http://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv
Resolving spreadsheets0.google.com... 74.125.230.132, 74.125.230.128, 74.125.230.130, ...
Connecting to spreadsheets0.google.com|74.125.230.132|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv [following]
--2011-04-29 18:01:01-- https://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv
Connecting to spreadsheets0.google.com|74.125.230.132|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: `/home/gavin/foo.csv'
[ <=> ] 41 --.-K/s in 0s
2011-04-29 18:01:02 (1.29 MB/s) - `/home/gavin/foo.csv' saved [41]
> read.csv("~/foo.csv")
column1 column2
1 a 1
2 b 2
3 ds 3
4 d 4
5 f 5
6 ga 5
I'm not sure R's internal download code is capable of responding to such redirects:
> download.file(data_url, "~/foo.csv")
trying URL 'http://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv'
Error in download.file(data_url, "~/foo.csv") :
cannot open URL 'http://spreadsheets0.google.com/spreadsheet/pub?hl=en&hl=en&key=0AgMhDTVek_sDdGI2YzY2R1ZESDlmZS1VYUxvblQ0REE&single=true&gid=0&output=csv'
I ran into the same problem and eventually found a solution in a forum thread. Using my own public CSV file:
library(RCurl)
tt = getForm("https://spreadsheets.google.com/spreadsheet/pub",
hl ="en_US", key = "0Aonsf4v9iDjGdHRaWWRFbXdQN1ZvbGx0LWVCeVd0T1E",
output = "csv",
.opts = list(followlocation = TRUE, verbose = TRUE, ssl.verifypeer = FALSE))
holidays <- read.csv(textConnection(tt))
Check the solution on http://blog.forret.com/2011/07/google-docs-infamous-moved-temporarily-error-fixed/
So what is the solution: just add “&ndplr=1” to your URL and you will
skip the authentication redirect. I’m not sure what the NDPLR
parameter name stands for, let’s just call it: “Never Do Published
Link Redirection“.