I am trying to upload a file to a platform via an API
CUrl request from terminal works fine
curl -v --trace-ascii trace.txt -H "Authorization: Bearer XXXX-XXXX-XXXX-XXXX" -F "file=#test1234.txt;type=text/txt" https://xxx.YYYY/upload
But in R when I use the Post I get status 500 error
outfilename="/Volumes/Work/texttest.txt"
POST(url="https://xxx.YYYY/upload",body = upload_file(
path = outfilename,
type = 'text/txt'),
verbose(),add_headers(Authorization=paste0("Bearer ",btoken$access_token)))
or
postForm(uri="https://xxx.YYYY/upload",file = fileUpload(
filename = outfilename, contentType = 'text/txt'),
add_headers(Authorization=paste0("Bearer ",btoken$access_token)))
Can anyone help please ?
EDIT: Got it to work with postForm, will be great if someone helps on why it does not work with POST
httpheader <- c(Authorization=paste0("Bearer ",btoken$access_token))
status<-postForm(uri=paste0(server,"upload"),file = fileUpload(filename = outfilename),.opts=list(httpheader=httpheader))
For POST, the most similar translation is
POST(url="https://xxx.YYYY/upload",
body = list(file=upload_file(
path = outfilename,
type = 'text/txt')
),
verbose(),
add_headers(Authorization=paste0("Bearer XXXX-XXXX-XXXX-XXXX"))
)
Notice how the body= is a list. This submits the data as form-data which is what the curl -F option does.
Note that I find that using a site like http://requestb.in/ can make it easier to troubleshoot these problems. By posting to that site, you can see exactly what is sent to the server and see how the request different between the two methods.
Related
I'm learning how to fetch data using an API in R. I understand that the aim of httr is to provide a wrapper for the curl package.
The documentation I'm following so that I make requests to the API has the following HTTP request format. This code below will be used to generate a token
curl -s \
-d "client_id=clientid” \
-d "username=user” \
-d "password=pwd” \
-d "grant_type=password" \
-d "scope=openid email" \
"https://auth.com/token"
Afterward, I'll use the token to now communicate with the API using this request
curl --header "Content-Type: application/json" \
--header "Accept: application/+json" \
--header "Authorization: Bearer token_goes_here“ \
--request GET \
--url "https://api-sitename.org/sections?parent_id=0"
Initially, I run these two requests in a terminal and they were successful, I got a response in JSON format. My question is, how do I run these requests in an R script such that I get a responses and they're it's stored in R studio global environment? My goal is to finally load the dataset from the API to the Rstudio working environment.
T
Here is something to get you started:
library(httr)
resp <- POST("https://auth.com/token",
body=list(client_id="clientid",
username="user",
password="pwd",
grant_type="password",
scope="openid email")
)
#parse for auth token here
content(resp, "text")
get_resp <- GET("https://api-sitename.org/sections?parent_id=0",
add_headers("Content-Type"="application/json",
Accept="application/+json",
"Authorization"=paste("Bearer", token))
I was able to successfully get my API call in R by replacing the content in header to body.
Here is my code
#' Th base url
base_url <- "your/url/endpoint/for/token"
# base64 encoded client id, my end-point requires to encone the client id to base64
c_id <- RCurl::base64(txt = "clinetid:sceret", mode = "character")
#' headers
headers <- httr::add_headers(
"Authorization" = paste("Basic",c_id, sep = " ")
)
# move everything else to the body. grant_type and password were requested by the endpoint
body <- list(
username = "your username",
password = "your password",
grant_type = "password",
scope = "read"
)
#' post call to get the token
httr::POST(
url = base_url,
body = body,
config = headers,
httr::accept_json()
)
When I had the user name and password in the body, I received 400 and 403 errors. Once I moved them o the body received 200 status and the token was successfully retrieved. If you can provide what you tried in R, can help you troubleshoot.
I'm trying to get some appointment data from a practice management software. I have an API key but I have no experience in the area.
I have tried to convert Curl code with little success. The api documentation is here https://github.com/redguava/cliniko-api
I am trying to convert this curl code
curl https://api.cliniko.com/v1/appointments \
-u API_KEY: \
-H 'Accept: application/json' \
-H 'User-Agent: APP_VENDOR_NAME (APP_VENDOR_EMAIL)'
What I've tried: (yes this is from a curl to r converter)
require(httr)
headers = c(
`Accept` = 'application/json',
`User-Agent` = 'APP_VENDOR_NAME (APP_VENDOR_EMAIL)'
)
res <- httr::GET(url = 'https://api.cliniko.com/v1/appointments',
httr::add_headers(.headers=headers),
httr::authenticate('API_KEY', 'INSERTED MY API KEY'))
Any ideas would be greatly appreciated
httr::authenticate takes input username and password in the form httr::authenticate(username,password).
Curl's authenticate takes argument username and password joined by by a :, i.e. username:password.
In the example from the API documentation the curl command authenticates the username:password combination API_KEY:. Looking closely, we can see that after the : is blank. From this we can determine the username field should be 'API_KEY' and the password field should be ''.
So you should change your curl command to:
require(httr)
headers = c(
`Accept` = 'application/json',
`User-Agent` = 'APP_VENDOR_NAME (APP_VENDOR_EMAIL)'
)
res <- httr::GET(url = 'https://api.cliniko.com/v1/appointments',
httr::add_headers(.headers=headers),
httr::authenticate('API_KEY', ''))
Where API_KEY is your provided API key.
i am new to using curl and I am trying to execute a https post request using curl. and it doest seem to work like other json post requests so i was hoping someone can explain this to me
I am not sure if i understood your answer well enough but you can take a look at this Package or this one the latter one provides provides more flexibility and features like Interceptors etc.
You can use this package Curl https://pub.dev/packages/curl
example code
import 'package:curl/curl.dart';
import 'package:http/http.dart';
final req1 = new Request("GET", "https://exyui.com/endpoint");
print(toCurl(req1));
// will print out:
// curl 'https://exyui.com/endpoint' --compressed --insecure
final req2 = new Request("PUT", "https://exyui.com/endpoint");
req2.body = "This is the text of body😅, \\, \\\\, \\\\\\";
print(req2);
// will print out:
// curl 'https://exyui.com/endpoint' -X PUT -H 'content-type: text/plain; charset=utf-8' --data-binary \$'This is the text of body\\ud83d\\ude05, \\, \\\\, \\\\\\' --compressed --insecure
final req3 = new Request("POST", "https://exyui.com/endpoint");
final part1 = "This is the part one of content";
final part2 = "This is the part two of content😅";
final expectQuery = "part1=This%20is%20the%20part%20one%20of%20content&part2=This%20is%20the%20part%20two%20of%20content%F0%9F%98%85";
req3.bodyFields = {
"part1": part1,
"part2": part2,
};
print(toCurl(req3));
I am trying to make async requests to a REST API from R. The below curl command illustrates the parameters that I need to the pass to the api. I'm giving you guys the linux curl command as I'm hoping that will make it clear:
curl -v -X POST https://app.example.com/api/ \
-H 'Authorization: somepwd' \
-H "Content-Type: application/json" \
-d {key1: value1, key2: value2}
Right now, I'm accomplishing the same thing in R by executing the following:
library(httr)
library(jsonlite)
content(POST('https://app.example.com/api/'
,add_headers(Authorization = 'somepwd')
,body = toJSON(rDataFrame)
,content_type_json()
)
)
The goal is to submit the above POST request from R but to vary the json string that is sent in the body, and do that asynchronously.
I have been searching for packages that will help me make asynchronous requests rather than making requests serially. The closest thing I could find is the getURIAsynchronous() function from the RCurl package (https://cran.r-project.org/web/packages/RCurl/RCurl.pdf) but do not understand how to submit a PUT request with headers and a body using their function. I would really like to make the above POST request from R but asynchronously where the URI is the same, but the data sent is different for each request.
I found this http://www.omegahat.org/RCurl/concurrent.html
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
complete(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
My idea is that I could replace for (i in uris) with for(i in jsons) where I am looping over the different data that I want to send to the same URL, however I am having trouble understanding the following concepts from the RCurl Package:
How do I pass a header as part of a PUT request. How do I pass data in the body of the request? This was pretty straight forward using the httr package as I have illustrated above.
I tried passing in the header in the curl options and alternatively the header. The thing is I don't understand where to pass the the component parts of the post request: authentication, header, and body within the getURIAsynchronous() function, or any of the resources I have described above.
Does anyone know how to accomplish this? An example would be incredibly helpful.
The curl package has been recently updated to handle async requests (see here)
Using the curl, magrittr and jsonlite packages you can create asynchronous post requests by:
Creating a generic handle with your header and body content using the handle_setform function
Writing a call back function to retrieve your results
Initializing a pool and adding your concurrent requests to it
Running your pool via multi_run
Example code is below:
library(curl)
library(jsonlite)
library(magrittr)
#create a handle object
h <- new_handle() %>%
handle_setheaders(Authorization = "somepwd",
"Content-Type" = "application/json") %>%
handle_setform(body = toJSON(iris))
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# example vector of uris to loop through
uris <- c("https://app.example.com/api/endpoint1"
,"https://app.example.com/api/endpoint2"
,"https://app.example.com/api/endpoint3")
# all scheduled requests are performed concurrently
sapply(uris, curl_fetch_multi, done=cb, pool=pool)
# This actually performs requests
out <- multi_run(pool = pool)
I am new to using R to post forms and then download data off the web. I have a question that is probably very easy for someone out there to spot what I am doing wrong, so I appreciate your patience. I have a Win7 PC and Firefox 23.x is my typical browser.
I am trying to post the main form that shows up on
http://www.aplia.com/
I have the following R script:
your.username <- 'username'
your.password <- 'password'
setwd( "C:/Users/Desktop/Aplia/data" )
require(SAScii)
require(RCurl)
require(XML)
agent="Firefox/23.0"
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))
curl = getCurlHandle()
curlSetOpt(
cookiejar = 'cookies.txt' ,
useragent = agent,
followlocation = TRUE ,
autoreferer = TRUE ,
curl = curl
)
# list parameters to pass to the website (pulled from the source html)
params <-
list(
'userAgent' = agent,
'screenWidth' = "",
'screenHeight' = "",
'flashMajor' = "",
'flashMinor' = "",
'flashBuild' = "",
'flashPatch' = "",
'redirect' = "",
'referrer' = "http://www.aplia.com",
'txtEmail' = your.username,
'txtPassword' = your.password
)
# logs into the form
html = postForm('https://courses.aplia.com/', .params = params, curl = curl)
html
# download a file once form is posted
html <-
getURL(
"http://courses.aplia.com/af/servlet/mngstudents?ctx=filename" ,
curl = curl
)
html
But from there I can tell that I am not getting the page I want, as what is returned into html is a redirect message that appears to be asking me to login again (?):
"\r\n\r\n<html>\r\n<head>\r\n <title>Aplia</title>\r\n\t<script language=\"JavaScript\" type=\"text/javascript\">\r\n\r\n top.location.href = \"https://courses.aplia.com/af/servlet/login?action=form&redirect=%2Fservlet%2Fmngstudents%3Fctx%3Dfilename\";\r\n \r\n\t</script>\r\n</head>\r\n<body>\r\n Click here to continue.\r\n</body>\r\n</html>\r\n"
Although I do believe there are a series of redirects that occur once the form is posted successfully (manually, in a browser). How can I tell the form was posted correctly?
I am quite sure that once I can get the post working correctly, I won't have a problem directing R to download the files I need (online activity reports for each of my 500 students this semester). But spent several hours working on this and got stuck. Maybe I need to set more options with the RCurl package that have to do with cookies (as the site does use cookies) ---?
Any help so much appreciated!! I typically use R to handle statistical data so am new to these packages and functions.
The answer ends up being very simple. For some reason, I didn't see one option that needs to be included in postForm:
html = postForm('https://courses.aplia.com/', .params = params, curl = curl, style="POST")
And that's it...