I am using R to call on Cybersource API. When I use the GET request I obtain a successful response of 200. When I read the body of the the response rather than get the csv data back I get a path to a csv file path. I wonder what I am doing wrong.
content(request) gives
"/space/download_reports/output/dailyreports/reports/2018/10/27/testrest/TRRReport-7931d82d-cf4a-71fa-e053-a2588e0ab27a.csv"
The result of content(request) should be the data not a file path.
Here is a the code
library('httr')
merchant<-'testrest'
vcdate<-'Wed, 29 May 2019 10:09:48 GMT'
ho<-'apitest.cybersource.com'
URL<-'https://apitest.cybersource.com/reporting/v3/report-downloads?organizationId=testrest&reportDate=2018-10-27&reportName=TRRReport'
sign<-'keyid="08c94330-f618-42a3-b09d-e1e43be5efda", algorithm="HmacSHA256", headers="host (request-target) v-c-merchant-id", signature="7cr6mZMa1oENhJ5NclGsprmQxwGlo1j3VjqAR6xngxk="'
req<-GET(URL, add_headers(.headers=c('v-c-merchant-id'=merchant, 'v-c-date'=vcdate, 'Host'=ho, 'Signature'=sign)))
content(req)
Here is Cybersource test api where you can verify the data produced.
https://developer.cybersource.com/api/reference/api-reference.html
I am trying to download a report under the reporting tab
Honestly I'm not exactly sure what's going on here, but I do think I was able to get something to work.
The API seems to like to return data in application/hal+json format which is not one that httr normally requests. You can say you'll accept anything with accept("*") So you can do your request with:
req <- GET(URL, add_headers('v-c-merchant-id'=merchant,
'v-c-date'=vcdate,
'Host'=ho,
'Signature'=sign), accept("*"))
Now we are actually getting the data back that we want, but httr doesn't know how to automatically parse it. So we need to parse it ourselves. This seems to do the trick
readr::read_csv(rawToChar(content(req)), skip=1)
There seems to be a header row which we skip with skip= and then we parse the rest as a CSV file with readr::read_csv.
Related
I am trying to connect to Qualtrics API using Rstudio Cloud "httr" package to download mailing lists. After a review of the API documentation I was unable to download the data, getting the following error after running the code:
"{"meta":{"httpStatus":"400 - Bad Request","error":{"errorMessage":"Expected authorization in headers, but none provided.","errorCode":"ATP_2"},"requestId":"8fz33cca-f9ii-4bca-9288-5tc69acaea13"}}"
This does not makes me any sense since I am using a inherit auth from parent token. Here is the code:
install.packages("httr")
library(httr)
directoryId<-"POOL_XXXXX"
mailingListId <- "CG_XXXXXX"
apiToken<-"XXXX"
url<- paste("https://iad1.qualtrics.com/API/v3/directories/",directoryId,
"/mailinglists/",mailingListId,"/optedOutContacts", sep = "")
response <- VERB("GET",url, add_headers('X_API-TOKEN' = apiToken),
content_type("application/octet-stream"))
content(response, "text")
Any help will be appreciated.
Thanks in advance.
Your call to httr::VERB breaks the API token and the content type into two arguments to the function, but they should be passed together in a vector to a single "config" argument. Also, content_type isn't a function, it's just the name of an element in that header vector. This should work:
response <- VERB("GET", url, add_headers(c(
'X_API-TOKEN' = apiToken,
'content_type' = "application/octet-stream")))
Note that mailing lists will be returned by Qualtrics as lists that will include both a "meta" element and a "result" element, both of which will themselves be lists. If the list is long, the only the first 100 contacts on the list will be returned; there will be an element response$result$nextpage that will provide the URL required to access the next 100 results. The qualtRics::fetch_mailinglist() function does not work with XM Directory contact lists (which is probably why you got a 500 error when using it), but the code for unpacking the list and looping over each "nextpage" element might be helpful.
I have a bunch of weather data files I want to download, but there's a mix of website url's that have data and those that don't. I'm using the download.file function in R to download the text files, which is working fine, but I'm also downloading a lot of empty text files because all the url's are valid, even if no data is present.
For example, this url provides good data.
http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=2021&MONTH=12&FROM=3000&TO=3000&STNM=72645
But this one doesn't.
http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=1970&MONTH=12&FROM=3000&TO=3000&STNM=72645
Is there a way to check to see if there's valid data in the text file before I download it? I've looked for something in the RCurl package, but didn't see what I needed. Thank you.
You can use httr::HEAD to determine the data size before downloading it. Note that this saves you the "pain" of downloading; if there is any cost on the server side, it feels the query-pain even if you do not download it. (These two seem quick enough, perhaps it's not a problem.)
# good data
res1 <- httr::HEAD("http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=2021&MONTH=12&FROM=3000&TO=3000&STNM=72645")
httr::headers(res1)$`content-length`
# [1] "9435"
# no data
res2 <- httr::HEAD("http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=1970&MONTH=12&FROM=3000&TO=3000&STNM=72645")
httr::headers(res2)$`content-length`
# NULL
If the API provides a function for estimating size (or at least presence of data), then it might be nicer to the remote end to use that instead of using this technique. For example: let's assume that an API call requires a 20 second SQL query. A call to HEAD will take 20 seconds, just like a call to GET, the only difference being that you don't get the data. If you see that you will get data and then subsequently call httr::GET(.), then you'll wait another 20 seconds (unless the remote end is caching queries).
Alternatively, they may have a heuristic to find presence of data, perhaps just a simple yes/no, that only takes a few seconds. In that case, it would be much "nicer" for you to make a 3 second "is data present" API call before calling the 20-second full query call.
Bottom line: if the API has a "data size" estimator, use it, otherwise HEAD should work fine.
As an alternative to HEAD, just GET the data, check the content-length, and save to file only if found:
res1 <- httr::GET("http://weather.uwyo.edu/cgi-bin/sounding?region=naconf&TYPE=TEXT%3ALIST&YEAR=2021&MONTH=12&FROM=3000&TO=3000&STNM=72645")
stuff <- as.character(httr::content(res1))
if (!is.null(httr::headers(res1)$`content-length`)) {
writeLines(stuff, "somefile.html")
}
# or do something else with the results, in-memory
I'm receiving the data I'm requesting but don't understand how to sufficiently extract the data. Here is the POST request:
library(httr)
url <- "http://tools-cluster-interface.iedb.org/tools_api/mhci/"
body <- list(method="recommended", sequence_text="SLYNTVATLYCVHQRIDV", allele="HLA-A*01:01,HLA-A*02:01", length="8,9")
data <- httr::POST(url, body = body,encode = "form", verbose())
If I print the data with:
data
..it shows the request details followed by a nicely formatted table. However if I try to extract with:
httr::content(data, "text")
This returns a single string with all the values of the original table. The output looks delimited by "\" but I couldn't str_replace or tease it out properly.
I'm new to requests using R (and httr) and assume it's an option I'm missing with httr. Any advice?
API details here: http://tools.iedb.org/main/tools-api/
The best way to do this is to specify the MIME type:
content(data, type = 'text/tab-separated-values')
I'm having trouble accessing the Energy Information Administration's API through R (https://www.eia.gov/opendata/).
On my office computer, if I try the link in a browser it works, and the data shows up (the full url: https://api.eia.gov/series/?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json).
I am also successfully connected to Bloomberg's API through R, so R is able to access the network.
Since the API is working and not blocked by my company's firewall, and R is in fact able to connect to the Internet, I have no clue what's going wrong.
The script works fine on my home computer, but at my office computer it is unsuccessful. So I gather it is a network issue, but if somebody could point me in any direction as to what the problem might be I would be grateful (my IT department couldn't help).
library(XML)
api.key = "e122a1411ca0ac941eb192ede51feebe"
series.id = "PET.MCREXUS1.M"
my.url = paste("http://api.eia.gov/series?series_id=", series.id,"&api_key=", api.key, "&out=xml", sep="")
doc = xmlParse(file=my.url, isURL=TRUE) # yields error
Error msg:
No such file or directoryfailed to load external entity "http://api.eia.gov/series?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json"
Error: 1: No such file or directory2: failed to load external entity "http://api.eia.gov/series?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json"
I tried some other methods like read_xml() from the xml2 package, but this gives a "could not resolve host" error.
To get XML, you need to change your url to XML:
my.url = paste("http://api.eia.gov/series?series_id=", series.id,"&api_key=",
api.key, "&out=xml", sep="")
res <- httr::GET(my.url)
xml2::read_xml(res)
Or :
res <- httr::GET(my.url)
XML::xmlParse(res)
Otherwise with the post as is(ie &out=json):
res <- httr::GET(my.url)
jsonlite::fromJSON(httr::content(res,"text"))
or this:
xml2::read_xml(httr::content(res,"text"))
Please note that this answer simply provides a way to get the data, whether it is in the desired form is opinion based and up to whoever is processing the data.
If it does not have to be XML output, you can also use the new eia package. (Disclaimer: I'm the author.)
Using your example:
remotes::install_github("leonawicz/eia")
library(eia)
x <- eia_series("PET.MCREXUS1.M")
This assumes your key is set globally (e.g., in .Renviron or previously in your R session with eia_set_key). But you can also pass it directly to the function call above by adding key = "yourkeyhere".
The result returned is a tidyverse-style data frame, one row per series ID and including a data list column that contains the data frame for each time series (can be unnested with tidyr::unnest if desired).
Alternatively, if you set the argument tidy = FALSE, it will return the list result of jsonlite::fromJSON without the "tidy" processing.
Finally, if you set tidy = NA, no processing is done at all and you get the original JSON string output for those who intend to pass the raw output to other canned code or software. The package does not provide XML output, however.
There are more comprehensive examples and vignettes at the eia package website I created.
I need to send a CSV file using Curl in Windows to an R program using the plumber API. I have Curl installed in Windows.
I've tried the following, but it doesn't work. I'm new to R and APIs, sorry.
The Curl command on Windows:
curl -X GET -d '#C:\Users\hp\Document\Final_Training.csv'
http://localhost:8000/data1
gives the error:
"error":["500 - Internal server error"],"message":["Error in is.data.frame(x):
argument \"data1\" is missing, with no default\n"]}
My R plumber code is:
#* #get /data1
function(data1) {
write.csv(data1, file="Final_Test.csv", sep=",", row.names=FALSE,
col.names=TRUE)
}
There are two issues. First, when "form data" is sent over HTTP, it's sent as a sequence of key/value pairs. In particular, in a GET request, it's actually embedded in the URL. For example, in the URL:
https://www.google.com/intl/en/about/?fg=1&utm_source=google.com&...
the key/value pairs are fg=1, utm_source=google.com, etc.
When you write Plumber code, the functions take arguments that correspond to the keys, so if Google was going to decide to deploy Plumber for their "About" page, they might write:
#* #get /intl/en/about
function(fg, utm_source, etc) {
...
}
In your code, you have Plumber code for URL /data1 that also expects a data1 key containing the CSV contents:
#* #get /data1 <-- the URL
function(data1) { <-- which expects a `?data1=...` query parameter
write.csv(data1, file="Final_Test.csv", sep=",", row.names=FALSE,
col.names=TRUE)
}
To get Curl to do this (provide the data from a file using a key data1 and supply it encoded in the URL for a GET request), you need to use a slightly different command line than you're using:
curl -X GET --data-urlencode 'data1#C:\Users\hp\Document\Final_Training.csv'
http://localhost:8000/data1
That should get the data actually sent to the Plumber process. On the Windows side, you should see an empty Javascript object {} response (so no error). On the R side, you'll see some weird warnings about attempts to set col.names and sep being ignored or something. In addition, what gets written to Final_Test.csv won't be a proper CSV file.
That leads to the second issue. In your plumber code, data1 will be received as a plain character string containing the contents of the CSV file. It will not be a data frame or anything like that. If you just want to write the uploaded file to a CSV file, you'd just write the string data1 to the file directly:
function(data1) {
cat(data1, file="Final_Test.csv")
}
If you want to load it into a data frame, you'd want to parse the string as a CSV file using something like:
#* #get /data1
function(data1) {
t <- textConnection(data1)
X <- read.csv(t)
close(t)
print(summary(X))
list(result=jsonlite::unbox("success"))
}