R: Azure Table entity query via REST call fails - r

I am working with Azure, Databricks and R. I am trying to call Azure Table Storages from R cell in Databricks notebook via ATS REST API. I am able to create/delete tables and I also can insert new entities into existing tables. However, if I try to query entities I always get an 403 error. Here is my code that tries to filter on "RowKey=1" and select "Name" property.
library(httr)
key <- "myStorageKey"
url <- "https://myAccount.table.core.windows.net/myTable()?$filter=(RowKey%20eq%20'1')?$select=Name"
canonResource <- paste0("/", account, "/myTable()?$filter=(RowKey%20eq%20'1')?$select=Name")
Now I am using SharedKeyLite authentiction as described here
requestdate <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
signStr <- paste(requestdate, canonResource, sep = "\n")
auth <- paste0("SharedKeyLite myAccount:",
RCurl::base64(digest::hmac(key = RCurl::base64Decode(key, mode = "raw"),
object = enc2utf8(signStr),
algo = "sha256", raw = TRUE)))
Now, according to the query entities description, i need to form a header like that:
header <- httr::add_headers(#`x-ms-version` = "2015-12-11 ",
`x-ms-date` = requestdate,
Authorization = auth,
#Accept = "application/json;odata=nometadata",
#`Accept-Charset` = "UTF-8",
DataServiceVersion = "3.0;NetFx",
#Date = requestdate,
MaxDataServiceVersion = "3.0;NetFx")
However, calling
httr::GET(url = url, config = header, verbose())
gives me
HTTP/1.1 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Does anyone see what I am missing or doing wrong? I played around with the header un-/commenting out several optional ones, but it still fails. As said, other calls on that api work for me. What I find a bit confusing is, that the documentation of the query call say I must specify the (Max)DataServiceVersion header, even if it is an optional header!

The issue was with how you're creating your canonResource.
Based on the documentation:
Shared Key Lite and Table service format for 2009-09-19 and later
This format supports Shared Key and Shared Key Lite for all versions
of the Table service, and Shared Key Lite for version 2009-09-19 and
later of the Blob and Queue services and version 2014-02-14 and later
of the File service. This format is identical to that used with
previous versions of the storage services. Construct the
CanonicalizedResource string in this format as follows:
Beginning with an empty string (""), append a forward slash (/),
followed by the name of the account that owns the resource being
accessed.
Append the resource's encoded URI path. If the request URI addresses a
component of the resource, append the appropriate query string. The
query string should include the question mark and the comp parameter
(for example, ?comp=metadata). No other parameters should be included
on the query string.
Based on this, your canonResource should be
canonResource <- paste0("/", account, "/myTable()")
Here's the complete code I used:
library(httr)
library(RCurl)
library(digest)
key <- "<account-key>"
account <- "<account-name>"
url <- "https://<account-name>.table.core.windows.net/myTable()?$filter=RowKey%20eq%20'1'&$select=Name"
canonResource <- paste0("/", account, "/myTable()")
print(canonResource)
requestdate <- format(Sys.time(),"%a, %d %b %Y %H:%M:%S %Z", tz="GMT")
print(requestdate)
signStr <- paste(requestdate, canonResource, sep = "\n")
print(signStr)
auth <- paste0("SharedKeyLite ", account, ":",
RCurl::base64(digest::hmac(key = RCurl::base64Decode(key, mode = "raw"),
object = enc2utf8(signStr),
algo = "sha256", raw = TRUE)))
print(auth)
header <- httr::add_headers(`x-ms-version` = "2015-12-11 ",
`x-ms-date` = requestdate,
Authorization = auth,
Accept = "application/json;odata=nometadata",
#`Accept-Charset` = "UTF-8",
DataServiceVersion = "3.0;NetFx",
#Date = requestdate,
MaxDataServiceVersion = "3.0;NetFx")
print(header)
httr::GET(url = url, config = header, verbose())

Related

paste0() adds an escape forward backslash in call to API?

I need to dynamically make a call to an API using the following format:
auth_secret <- paste0("Bearer ", secret)
headers = c(
`Authorization` = auth_secret,
`Notion-Version` = '2022-02-22',
`Content-Type` = 'application/json' )
res <- httr::PATCH(url = paste0('https://api.notion.com/v1/pages/', id),
httr::add_headers(.headers = headers),
body = payload,
encode = "json")
d <- httr::content(res)
This payload works:
payload <- "{\"properties\":{\"Project\":{\"relation\":[{\"id\":\"1d148a9e-783d-47a7-b3e8-2d9c34210355\"}]}}}"
But if I want to create it dynamically, using a paste0 (so it is inside of a function), I get some backslashes added before and after:
payload <- paste0('"{\"properties\":{\"',property_name,'\":{\"relation\":[{\"id\":\"',value,'\"}]}}}"')
print(payload)
"\"{\"properties\":{\"%7CAK%5E\":{\"relation\":[{\"id\":\"8cb9519e72ca4bbe9e0448807acb8e10\"}]}}}\""
I presume this is due to some weird escape character being added but have run out of ideas. I've added two \ and have gotten same issue. Call fails as JSON is not passed correctly. Is there a workaround?
This is obviously something to do with the fact that paste0 is escaping even the double quotes you do not want to escape.
No doubt someone is aware of the details. However, when I get weird stuff with paste0, I just use sprintf, which seems to work in this case:
property_name = "Projects"
value = "1d148a9e-783d-47a7-b3e8-2d9c34210355"
payload <- sprintf(
"{\"properties\":{\"%s\":{\"relation\":[{\"id\":\"%s\"}]}}}",
property_name, value
)
print(payload)
# [1] "{\"properties\":{\"Projects\":{\"relation\":[{\"id\":\"1d148a9e-783d-47a7-b3e8-2d9c34210355\"}]}}}"

Running a POST request to get a Service Ticket inside a for loop

I'm working with the NIH/NLM REST API and attempting to programmatically pull lots of data at once. I've never worked with an API that validates with Service Tickets (TGT and ST) instead of OAUTH, that need to be refreshed for every GET request you make, so I'm not sure if I"m even going about this the right way. Any help much appreciated.
Here's the code I currently have:
library(httr)
library(jsonlite)
library(xml2)
UTS_API_KEY <- 'MY API KEY'
# post to the CAS endpoint
response <- POST('https://utslogin.nlm.nih.gov/cas/v1/api-key', encode='form', body=list(apikey = 'MY API KEY'))
# print out the status_code and content_type
status_code(response)
headers(response)$`content-type`
doc <- content(response)
action_uri <- xml_text(xml_find_first(doc, '//form/#action'))
action_uri
# Service Ticket
response <- POST(action_uri, encode='form', body=list(service = 'http://umlsks.nlm.nih.gov'))
ticket <- content(response, 'text')
ticket #this is the ST I need for every GET request I make
# build search_uri using the paste function for string concatenation
version <- 'current'
search_uri <- paste('https://uts-ws.nlm.nih.gov/rest/search/', version, sep='')
# pass the the query params into httr GET to get the response
query_string <- 'diabetic foot'
response <- GET(search_uri, query=list(ticket=ticket, string=query_string))
## print out some of the results
search_uri
status_code(response)
headers(response)$`content-type`
search_results_auto_parsed <- content(response)
search_results_auto_parsed
class(search_results_auto_parsed$result$results)
search_results_data_frame <- fromJSON(content(response,'text'))
search_results_data_frame
This code works perfectly for just a handful of GET requests, however, I'm attempting to pull 300-something medical terms. For example, in query string, I'd like to loop through an array of strings (e.g., "diabetes", "blood pressure", "cardiovascular care", "EMT", etc.). I'd need to make the POST request and pass the ST into the GET parameter for every string in the array.
I've played around with this code:
for (i in 1:length(Entity_Subset$Entities)){
ent = Entity_Subset$Entities[i] #Entities represents my df of strings
url <- paste(' https://uts-ws.nlm.nih.gov/rest/search/current?string=',
ent,'&ticket=', sep = "")
print(url)
}
But haven't had much luck piecemealing together the POST and GET requests after putting the strings into the (GET) HTTPS request.
Sidebar: I also attempted writing some pre-scripts in Postman, but oddly the Service Ticket doesn't return as JSON (no key-value pair to grab and pass). Just plain text.
Thank you for any advice you can provide!
I think you can simply wrap both POST and GET requests in a function. Then, lapply that function to a list of characters.
library(httr)
library(jsonlite)
library(xml2)
fetch_data <- function(query_string = 'diabetic foot', UTS_API_KEY = 'MY API KEY', version = 'current') {
response <- POST('https://utslogin.nlm.nih.gov/cas/v1/api-key', encode='form', body=list(apikey = UTS_API_KEY))
# print out the status_code and content_type
message(status_code(response), "\n", headers(response)$`content-type`)
action_uri <- xml_text(xml_find_first(content(response), '//form/#action')); message(action_uri)
# Service Ticket
response <- POST(action_uri, encode = 'form', body=list(service = 'http://umlsks.nlm.nih.gov'))
ticket <- content(response, 'text'); message(ticket)
# build search_uri using the paste function for string concatenation
search_uri <- paste0('https://uts-ws.nlm.nih.gov/rest/search/', version)
# pass the the query params into httr GET to get the response
response <- GET(search_uri, query=list(ticket=ticket, string=query_string))
## print out some of the results
message(search_uri, "\n", status_code(response), "\n", headers(response)$`content-type`)
fromJSON(content(response, 'text'))
}
# if you have a list of query strings, then
lapply(Entity_Subset$Entities, fetch_data, UTS_API_KEY = "blah blah blah")
# The `lapply` above is logically equivalent to
result <- vector("list", length(Entity_Subset$Entities))
for (x in Entity_Subset$Entities) {
result[[x]] <- fetch_data(x, "blah blah blah")
}

Problem with authorization to COINAPI REST API with custom header and key in R

I would like to connect to COINAPI resources. They provide two types of authorization. https://docs.coinapi.io/#authorization
Custom authorization header named X-CoinAPI-Key
Query string parameter named apikey
When I am using the first method, it is working with basic requests. But respond with an error in more advanced.
endpoint<-"/v1/exchangerate/BTC?apikey="
But when I specify endpoint like this:
endpoint <- "/v1/trades/BITSTAMP_SPOT_BTC_USD/history?time_start=2016-01-01T00:00:00/?apikey="
I got error 401.
The second method is not working so far, I do not really understand how can I specify custom header name here.
I need to get data from here:
https://rest.coinapi.io/v1/ohlcv/BTC/USD/history?period_id=1DAY&time_start=2017-01-02T00:00:00.0000000Z&time_end=2019-01-02T00:00:00.0000000Z&limit=10000&include_empty_items=TRUE
I would appreciate any help on this issue.
1. method (working)
library(httr)
library(jsonlite)
base <- "https://rest.coinapi.io"
endpoint <- "/v1/exchangerate/BTC?apikey="
api_key <- <KEY>
call <- paste0(base, endpoint, api_key)
call
get_prices <- GET(call)
http_status(get_prices)
class(get_prices)
get_prices_text <- content(get_prices, "text", encoding = 'UTF-8')
get_prices_json <- fromJSON(get_prices_text, flatten = TRUE)
names(get_prices_json)
get_prices_json$asset_id_base
head(get_prices_json$rates)
data<-as.data.frame(get_prices_json)
2. method (not working)
key<-<KEY>
GET(
url = sprintf("https://rest.coinapi.io/v1/exchangerate/BTC"),
add_headers(`Authorization` = sprintf("X-CoinAPI-Key: ", key))
) -> res
http_status(res)
From reading the examples in the documentation, it looks like it's just looking for a simple header, not an "Authorization" header specifically. Try this
GET(
url = sprintf("https://rest.coinapi.io/v1/exchangerate/BTC"),
add_headers(`X-CoinAPI-Key` = key)
) -> res
http_status(res)

Access Coinbase API using HTTR and R

I am trying to pull a list of Coinbase accounts into R using their API. I receive an authentication error saying "invalid signature". Something is clearly wrong when I create my sha256 signature but I can't figure out what the issue is. I have not had this same issue when accessing the GDAX API using a sha256 signature.
API Key Documentation
API key is recommend if you only need to access your own account. All API >key requests must be signed and contain the following headers:
CB-ACCESS-KEY The api key as a string
CB-ACCESS-SIGN The user generated message signature (see below)
CB-ACCESS-TIMESTAMP A timestamp for your request
All request bodies should have content type application/json and be valid JSON.
The CB-ACCESS-SIGN header is generated by creating a sha256 HMAC using the secret key on the prehash string timestamp + method + requestPath + body (where + represents string concatenation). The timestamp value is the same as the CB-ACCESS-TIMESTAMP header.
My Code
library(httr)
library(RCurl)
library(digest)
coinbase_url <- "https://api.coinbase.com"
coinbase_reqPath <- "/v2/accounts/"
coinbase_fullPath <- paste(coinbase_url, coinbase_reqPath,sep = "")
coinbase_key <- "XXXXMYKEYXXX"
coinbase_secret <- "XXXXXMYSECRETKEYXXXX"
cb_timestamp <- format(as.numeric(Sys.time()), digits=10)
coinbase_message <- paste0(cb_timestamp,"GET", coinbase_reqPath)
coinbase_sig <- hmac(key = coinbase_secret, object = coinbase_message, algo = "sha256", raw = F)
coinbase_acct <- content(GET(coinbase_fullPath,
add_headers(
"CB-ACCESS-KEY" = coinbase_key,
"CB-ACCESS-SIGN" = coinbase_sig,
"CB-ACCESS-TIMESTAMP" = cb_timestamp,
"Content-Type"="application/json")))
Sorry for not updating this earlier. The answer is simple, I just needed to remove the final forward slash in "/v2/accounts/" when specifying my request path.

Posting to and Receiving data from API using httr in R

I'm trying to access the US Census Geocoder batch address API, found here: http://geocoding.geo.census.gov/geocoder/
I've also gone through the documentation for the API here: http://geocoding.geo.census.gov/geocoder/Geocoding_Services_API.pdf
I'm trying to use the httr package in R to post a batch csv file of formatted addresses using this format: Unique ID, Street address, City, State, ZIP
I've tried the single address request version using getURL from RCurl and that works fine, but postForm doesn't seem to submit the file in the right way. The code I'm using right now seems to post the request correctly, but I'm not getting any geocoded data back.
curlhandle <- handle("http://geocoding.geo.census.gov/geocoder/geographies/addressbatch",
cookies = TRUE)
# using fileUpload from RCurl instead of upload_file from httr
upload3 <- fileUpload(contents = address100, contentType = "file")
test <- POST(url="http://geocoding.geo.census.gov/geocoder/geographies/addressbatch",
body = list(addressFile = upload3,
benchmark = "Public_AR_Census2010",
vintage="Census2010_Census2010"),
encode = "multipart",
handle = curlhandle,
followLocation = TRUE,
verbose = TRUE)
Is my request missing something? I'm not sure if I should be using writefunction & writedata in this case. Any help would be appreciated!
You seem to have a weird mix of RCurl and httr. Here's how I'd write the request:
req <- POST(
"http://geocoding.geo.census.gov/geocoder/geographies/addressbatch",
body = list(
addressFile = upload_file(address100),
benchmark = "Public_AR_Census2010",
vintage = "Census2010_Census2010"
),
encode = "multipart",
verbose())
stop_for_status(req)
content(req)
(also "file" is not a valid mime type)

Resources