I'm attempting to request data from a google spreadsheet using the googleAuthR. I need to use this library instead of Jenny Bryan's googlesheets because the request is part of a shiny app with multiple user authentication. When the request range does not contain spaces (e.g. "Sheet1!A:B" the request succeeds. However, when the tab name contains spaces (e.g. "'Sheet 1'!A:B" or "\'Sheet 1\'!A:B", the request fails and throws this error:
Request Status Code: 400
Error : lexical error: invalid char in json text.
<!DOCTYPE html> <html lang=en>
(right here) ------^
Mark Edmondson's googleAuthR uses jsonlite for parsing JSON. I assume this error is coming from jsonlite, but I'm at a loss for how to fix it. Here is a minimal example to recreate the issue:
library(googleAuthR)
# scopes
options("googleAuthR.scopes.selected" = "https://www.googleapis.com/auth/spreadsheets.readonly")
# client id and secret
options("googleAuthR.client_id" = "XXXX")
options("googleAuthR.client_secret" = "XXXX")
# request
get_data <- function(spreadsheetId, range) {
l <- googleAuthR::gar_api_generator(
baseURI = "https://sheets.googleapis.com/v4/",
http_header = 'GET',
path_args = list(spreadsheets = spreadsheetId,
values = range),
pars_args = list(majorDimension = 'ROWS',
valueRenderOption = 'UNFORMATTED_VALUE'),
data_parse_function = function(x) x)
req <- l()
req
}
# authenticate
gar_auth(new_user = TRUE)
# input
spreadsheet_id <- "XXXX"
range <- "'Sheet 1'!A:B"
# get data
df <- get_data(spreadsheet_id, range)
How should I format range variable for the request to work? Thanks in advance for the help.
Use URLencode() to percent-encode spaces.
Details:
Using options(googleAuthR.verbose = 1) shows that the GET request was of the form:
GET /v4/spreadsheets/.../values/'Sheet 1'!A:B?majorDimension=ROWS&valueRenderOption=UNFORMATTED_VALUE HTTP/1.1
I had assumed the space would be encoded, but I guess not. In this github issue from August 2016, Mark states URLencode() was going to be the default for later versions of googleAuthR. Not sure if that will still happen, but it's an easy fix in the meantime.
Related
Let me start by saying that I understand how to do a POST request using "httr" and "crul" packages. I am working on developing an asynchronous method to sending multiple POST request with unique JSON body requests using the basic "curl" package. I have legitimate reasons for trying this with this package, but more importantly I'm just determined to get it to work. This may not be possible, or I may even be trying to wrong functions in "curl"...but wanted to see if anyone had any ideas.
I am trying to send a post request using curl_fetch_multi() as a POST request with a JSON in the body like this...
{
"configuration": {
"Id": 4507
},
"age": 0,
"zip": 32411,
"Date": "2020-12-23"
}
I have succeeded in at least getting getting error messages back form the API indicating an invalid body input using something along the lines of starting with an object containing each body i need to submit
library(curl)
library(jsonlite)
library(magrittr)
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# Create request for each body
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setopt(post = TRUE) %>%
handle_setheaders("Content-Type"="application/vnd.v1+json") %>%
handle_setform(body = df$body[[i]]) ###df$body[[i]] is a JSON string
)
}
# This actually performs requests
out <- multi_run(pool = pool)
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
....
I'm 90% positive it has to do with how it's attempting to call the JSON in handle_setform() setting of the handle. This is about where I am over my head and documentation is scarce.
Also, I am pretty sure the JSON is structured properly, as I can use them in other packages with no problem.
Any assistance would be greatly appreciated.
Found the solution!!
Needed to use following settings with handle_setopts()
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setheaders("Content-Type"="application/v1+json") %>%
handle_setopt(customrequest = "POST") %>%
handle_setopt(postfields = df$body[[i]]) #df$body is list of JSON
)
}
out <- multi_run(pool = pool)
I have some url, some raw data that looks like json and auth creds.
Everything together works great via postman, but now i need to automate it in R.
For example:
url1 = 'https://site.io/v2/passes/?values=true'
bod = paste0('{"values": [{"value": "0","label": "label1"},{"value": "5","label": "label2"}] }')
I've already tried this one
r <- POST(url1, body = bod, encode = "raw", httr::add_headers(.headers=headers)
, httr::authenticate("user", "password", type="digest"))
But it seems incorrect (and also getting 400 error)
I assume your bod is a text string. When you use encode="raw", you have to also specify the Content-encoding header if the host is expecting one (and in this case, I assume it's expecting "application/json":
headers <- c(headers, `Content-encoding`="application/json")
POST(url1, body=bod, encode="raw", add_headers(.headers=headers), ...)
I am trying to scrape the table in : WEB TABLE
I have tried copying the xpath but it does not return anything:
require("rvest")
url = "https://www.barchart.com/options/stocks-by-sector?page=1"
pg = read_html(url)
pg %>% html_nodes(xpath="//*[#id=main-content-column]/div/div[4]/div/div[2]/div")
EDIT
I found the following link and feel I am getting closer....
So by using the same process I found the updated link by watching the XHR updates:
url = paste0("https://www.barchart.com?access_token=",token,"/proxies/core-api/v1/quotes/",
"get?lists=stocks.optionable.by_sector.all.us&fields=symbol%2CsymbolName",
"%2ClastPrice%2CpriceChange%2CpercentChange%2ChighPrice%2ClowPrice%2Cvolume",
"%2CtradeTime%2CsymbolCode%2CsymbolType%2ChasOptions&orderBy=symbol&orderDir=",
"asc&meta=field.shortName%2Cfield.type%2Cfield.description&hasOptions=true&page=1&limit=100&raw=1")
Where the token is found within the scope:
token = "eyJpdiI6IjJZMDZNOGYwUDk4dE1OcVc4ekdnUGc9PSIsInZhbHVlIjoib2lYcWtzRi9VN3ovbzdER2NhQlg0KzJQL1ZId2ZOeWpwSTF5YThlclN1SW9YSEtJbG9kR0FLbmRmWmtNcmd1eCIsIm1hYyI6ImU4ODA3YzZkZGUwZjFhNmM1NTE4ZjEzNmZkNThmZDY4ODE1NmM0YTM1Yjc2Y2E2OWVkNjZiZTE3ZDcxOGFlZjMifQ"
However, I do not know if I am placing the token where I should in the URL, but when I ran:
fixture <- jsonlite::read_json(url,simplifyVector = TRUE)
I received the following error:
Error in parse_con(txt, bigint_as_char) :
lexical error: invalid char in json text.
<!doctype html> <html itemscope
(right here) ------^
The token needs to be sent as a request header named x-xsrf-token not by pass to the parameters:
Also, the token value might change over sessions so you need to get it in the cookie. After that, convert the data to a data frame and get the result:
library(rvest)
pg <- html_session("https://www.barchart.com/options/stocks-by-sector?page=1")
cookies <- pg$response$cookies
token <- URLdecode(dplyr::recode("XSRF-TOKEN", !!!setNames(cookies$value, cookies$name)))
pg <-
pg %>% rvest:::request_GET(
"https://www.barchart.com/proxies/core-api/v1/quotes/get?lists=stocks.optionable.by_sector.all.us&fields=symbol%2CsymbolName%2ClastPrice%2CpriceChange%2CpercentChange%2ChighPrice%2ClowPrice%2Cvolume%2CtradeTime%2CsymbolCode%2CsymbolType%2ChasOptions&orderBy=symbol&orderDir=asc&meta=field.shortName%2Cfield.type%2Cfield.description&hasOptions=true&page=1&limit=1000000&raw=1",
config = httr::add_headers(`x-xsrf-token` = token)
)
data_raw <- httr::content(pg$response)
data <-
purrr::map_dfr(
data_raw$data,
function(x){
as.data.frame(x$raw)
}
)
I am trying to write an R script to programmatically update a Google Tag Manager container via API and I have hit a bit of a wall getting it to work, as it keeps returning an invalid argument error. The problem is that I can't quite figure out what the problem is.
The documentation for the API call is here:
https://developers.google.com/tag-manager/api/v2/reference/accounts/containers/update
Here's the code:
library(httr)
url_base <- 'https://www.googleapis.com/tagmanager/v2'
url_path <- paste('accounts',account_id,'containers',container_id,sep='/')
api_url <- paste(url_base,url_path,sep='/')
#since the instructions indicate that the request body parameters are all optional, let's just send a new name
call <- PUT(api_url,
add_headers(Authorization = paste("Bearer", gtm_token$credentials$access_token)),
encode = 'json',
body = list(name = 'new name'))
call_content <- content(call,'parsed')
This is a pretty standard API call to the GTM API, and in fact I have written a bunch of functions for other GTM API methods that work in the same way, so I am a bit perplexed as to why this one keeps failing:
$error
$error$errors
$error$errors[[1]]
$error$errors[[1]]$domain
[1] "global"
$error$errors[[1]]$reason
[1] "invalidArgument"
$error$errors[[1]]$message
[1] "Bad Request"
$error$code
[1] 400
$error$message
[1] "Bad Request"
It seems like the issue is in the message body, but it's not clear if the issue is down to the API expecting different information / more parameters, when the documentation suggests that all of the parameters are optional.
OK, so the documentation is lacking here. This works if you include a name at least. Here's a working function:
gtm_containers_update <- function(account_id,container_id,container_name,usage_context,domain_name,notes,token) {
require(httr)
token$refresh()
#create the post url
api_url <- paste('https://www.googleapis.com/tagmanager/v2','accounts',account_id,'containers',container_id,sep='/')
#create the list with required components
call_body <- list(name = container_name,
usageContext = list(usage_context),
notes = notes,
domainName = domain_name)
call <- POST(url,
add_headers(Authorization = paste("Bearer", token$credentials$access_token)),
encode = 'json',
body = call_body)
print(paste('Status code:',call$status_code))
}
I'm using R to interact with the Gemini exchange API (https://docs.gemini.com/rest-api/) for private endpoints. I've been able to reduce my problem to endpoints which require more than 2 parameters in the payload. In particular I'm attempting to query the /v1/mytrades endpoint (https://docs.gemini.com/rest-api/#get-past-trades) which I believe requires the 'request', 'nonce' and 'symbol' parameters at a minimum. The error code I receive is HTTP 400 which Gemini describes as:
Auction not open or paused, ineligible timing, market not open, or the request was malformed; in the case of a private API request, missing or malformed Gemini private API authentication headers
I have no issues with other endpoints which require only the 'request' and 'nonce' parameters, so I'm struggling to understand which step is a problem since those queries require similar steps to create a base64 encoding of the payload, a signature of that encoding using the API secret and headers with that data plus the API key.
Below is my example code where MY_API_SECRET and MY_API_KEY are placeholders for the actual secret and key strings
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/mytrades"
# Create the symbol parameter
symbol <- 'btcusd'
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(
request = geminiEndpoint,
nonce = currentTimeNonce,
symbol = symbol
)) %>% gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
For comparison, the following code which requests the /v1/orders endpoint (https://docs.gemini.com/rest-api/#get-active-orders) does indeed come back with a response:
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/orders"
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(request = geminiEndpoint, nonce = currentTimeNonce)) %>%
gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
So in the latter code the only difference is the geminiEndpoint and payload construction which as mentioned above only has 2 required parameters (request and nonce). To expand further, I'm successfully hitting other endpoints such as /v1/tradevolume, /v1/heartbeat, and /v1/balances that also require those 2 parameters only while /v1/order/status is another example of an endpoint requiring at least 3 parameters that doesn't work for me.
I appreciate any help to understand where I'm going wrong with this. Thank you in advance!
As mentioned in the comments, I started to work on an equivalent bash script based on r2evans' reply when I found my issue. It was the base64 encoding step in R that resulted in some unusual output. As seen in my original scripts, I was using the "base64_enc()" function which is part of the jsonlite package. As a simple check I was trying to confirm that the encoding from R was equal to an equivalent encoding using base64 in shell so I started by trying to decode the R result.
In R, the encoding of the payload for the 3 parameter example was coming out with a backslash character '\' which is not valid Base64 and points to my misunderstanding of what the base64_enc function is doing. I replaced this function with base64_encode from the openssl package and now my 3 parameter queries are coming back with results.