How do I convert this command:
curl -v -u abcdefghij1234567890:X -H "Content-Type: application/json" -X GET 'https://domain.freshdesk.com/api/v2/tickets'
to
curl command in Rcurl?
The dev version of curlconverter (devtools::install_github("hrbrmstr/curlconverter") can convert curl command-line strings with authentication and verbose params now:
Copy your URL to the clipboard:
curl -v -u abcdefghij1234567890:X -H "Content-Type: application/json" -X GET 'https://domain.freshdesk.com/api/v2/tickets'
Then run:
library(curlconverter)
req <- make_req(straighten())[[1]]
The following will now be in your clipboard:
httr::VERB(verb = "GET", url = "https://domain.freshdesk.com/api/v2/tickets",
httr::authenticate(user = "abcdefghij1234567890",
password = "X"), httr::verbose(),
httr::add_headers(), encode = "json")
but req is now also a callable function. You can see that by doing:
req
## function ()
## httr::VERB(verb = "GET", url = "https://domain.freshdesk.com/api/v2/tickets",
## httr::authenticate(user = "abcdefghij1234567890", password = "X"),
## httr::verbose(), httr::add_headers(), encode = "json")
or by actually calling it:
req()
I usually reformat the function source to make it more readable:
httr::VERB(verb = "GET",
url = "https://domain.freshdesk.com/api/v2/tickets",
httr::authenticate(user = "abcdefghij1234567890", password = "X"),
httr::verbose(),
httr::add_headers(),
encode = "json")
and you can easily translate that to a plain GET call without namespacing:
GET(url = "https://domain.freshdesk.com/api/v2/tickets",
authenticate(user = "abcdefghij1234567890", password = "X"),
verbose(),
add_headers(),
encode = "json"))
We can validate it working with authenticated curl command-lines by a small substitution in your example:
curl_string <- 'curl -v -u abcdefghij1234567890:X -H "Content-Type: application/json" -X GET "https://httpbin.org/basic-auth/abcdefghij1234567890/X"'
make_req(straighten(curl_string))[[1]]()
## -> GET /basic-auth/abcdefghij1234567890/X HTTP/1.1
## -> Host: httpbin.org
## -> Authorization: Basic YWJjZGVmZ2hpajEyMzQ1Njc4OTA6WA==
## -> User-Agent: libcurl/7.43.0 r-curl/1.2 httr/1.2.1
## -> Accept-Encoding: gzip, deflate
## -> Accept: application/json, text/xml, application/xml, */*
## ->
## <- HTTP/1.1 200 OK
## <- Server: nginx
## <- Date: Tue, 30 Aug 2016 14:13:12 GMT
## <- Content-Type: application/json
## <- Content-Length: 63
## <- Connection: keep-alive
## <- Access-Control-Allow-Origin: *
## <- Access-Control-Allow-Credentials: true
## <-
## Response [https://httpbin.org/basic-auth/abcdefghij1234567890/X]
## Date: 2016-08-30 14:13
## Status: 200
## Content-Type: application/json
## Size: 63 B
## {
## "authenticated": true,
## "user": "abcdefghij1234567890"
## }
You can use httr to do this as follows:
require(httr)
GET('https://domain.freshdesk.com/api/v2/tickets',
verbose(),
authenticate("user", "passwd"),
content_type("application/json"))
Related
So i am scrapping the nse india results calendar site via using curl command and even after that its giving me "Resource not Found" error . Here's my code
url = f"https://www.nseindia.com/api/event-calendar?index=equities"
header1 ="Host:www.nseindia.com"
header2 = "User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0"
header3 ="Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
header4 ="Accept-Language:en-US,en;q=0.5"
header5 ="Accept-Encoding:gzip, deflate, br"
header6 ="DNT:1"
header7 ="Connection:keep-alive"
header8 ="Upgrade-Insecure-Requests:1"
header9 ="Pragma:no-cache"
header10 ="Cache-Control:no-cache"
def run_curl_command(curl_command, max_attempts):
result = os.popen(curl_command).read()
count = 0
while "Resource not found" in result and count < max_attempts:
result = os.popen(curl_command).read()
count += 1
time.sleep(1)
print("API Read")
result = json.loads(result)
result = pd.DataFrame(result)
def init():
max_attempts = 100
curl_command = f'curl "{url}" -H "{header1}" -H "{header2}" -H "{header3}" -H "{header4}" -H "{header5}" -H "{header6}" -H "{header7}" -H "{header8}" -H "{header9}" -H "{header10}" --compressed '
print(f"curl_command : {curl_command}")
run_curl_command(curl_command, max_attempts)
init()
httr 1.4.1
R version 3.6.1 (also tried with 3.5.3)
Edit (adding verbose()) output.
I've got a request as follows:
r <- GET("https://my.cool.domain",add_headers(.headers = c('x-api-key' = 'abcdefg', 'Accept' = "text/csv")), verbose())
On my machine it responds with:
-> GET / HTTP/1.1
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.54.0 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/1.1 200 OK
<- Date: Tue, 26 Nov 2019 17:50:15 GMT
<- Content-Type: text/csv
<- Content-Length: 24902
<- Connection: keep-alive
<- x-amzn-RequestId: ...
<- Content-Encoding: deflate
<- x-amz-apigw-id: ...
<- X-Amzn-Trace-Id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:20
Status: 200
Content-Type: text/csv
Size: 209 kB
cats,dogs...
yes,no...
yes,yes...
no,no...
However on my colleague's machine (same version of httr and R, and also with an updated version of R) I get the following:
-> GET / HTTP/2
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.64.1 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/2 200
<- date: Tue, 26 Nov 2019 17:46:17 GMT
<- content-type: application/json
<- content-length: 21501
<- x-amzn-requestid: ...
<- content-encoding: deflate
<- x-amz-apigw-id: ...
<- x-amzn-trace-id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:30
Status: 200
Content-Type: application/json
Size: 377 kB
I'm working with the developer of the https://my.cool.domain domain and I can confirm that the request header params (x-api-key and 'Accept' = "text/csv") are perfect. And the request works on my machine, and several others, but not this one colleague's.
What's going wrong here and how can I debug this?
Thanks
This was fixed by doing httr::set_config(httr::config(http_version = 1.1)) to force 1.1.
curl req:
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -d '{"username":"emailId","password":"passwrd"}' -X POST https://central.vizury.com/-/api/login
res:
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json; charset=utf-8
Date: Wed, 06 Sep 2017 10:47:00 GMT
Expires: 0
Pragma: no-cache
Set-Cookie: viz.sess3=SessionCookieHere; path=/; expires=Wed, 06 Sep 2017 10:49:01 GMT; secure; httponly
Set-Cookie: AWSELB=someval;PATH=/;EXPIRES=Wed, 06 Sep 2017 10:49:01 GMT;SECURE;HTTPONLY
Vary: Accept-Encoding
X-Powered-By: Express
Content-Length: 226
Connection: keep-alive
{"status":"OK","results":{"username":"email","role":"role","products":["webConvert","mobiConvert"],"needsNewPassword":false},"homePath":"/webConvert/#/dashboard/campaignName"}
I need to perform the same action in R:
This is what I have tried so far:
h <- basicHeaderGatherer()
loginUrl <- "https://central.vizury.com/-/api/login"
params <- list('username' = 'username',
'password' = 'password')
loginRes <- postForm(loginUrl, .params=params, style="POST", .opts=curlOptions(headerfunction=h$update, verbose=TRUE))
print("loginres")
print(loginRes)
In response,
print(h$value()['Set-Cookie'] )
I can access Set-Cookie. But how do I access the value of viz.sess3?
Example using curl package:
h <- curl::new_handle()
login_url <- 'https://central.vizury.com/-/api/login'
curl::handle_setform(
handle = h,
username = 'username',
password = 'password'
)
resp <- curl::curl_fetch_memory(login_url, handle = h)
message(resp$status_code)
jsonlite::fromJSON(rawToChar(resp$content))
I'm following the official manual of opencpu package in R. In chapter 4.3 Calling a function It uses curl to test API:
curl http://your.server.com/ocpu/library/stats/R/rnorm -d "n=10&mean=100"
and the sample output is:
/ocpu/tmp/x032a8fee/R/.val
/ocpu/tmp/x032a8fee/stdout
/ocpu/tmp/x032a8fee/source
/ocpu/tmp/x032a8fee/console
/ocpu/tmp/x032a8fee/info
I can use curl to get similar result, but when I try to send this http request using httr package in R, I don't know how to replicate the result. Here is what I tried:
resp <- POST(
url = "localhost/ocpu/library/stats/R/rnorm",
body= "n=10&mean=100"
)
resp
the output is:
Response [HTTP://localhost/ocpu/library/stats/R/rnorm]
Date: 2015-10-16 00:51
Status: 400
Content-Type: text/plain; charset=utf-8
Size: 30 B
No Content-Type header found.
I guess I don't understand what's the equivalence of curl -d parameter in httr, how can I get it correct?
Try this :)
library(httr)
library(jsonlite)
getFunctionEndPoint <- function(url, format) {
return(paste(url, format, sep = '/'))
}
resp <- POST(
url = getFunctionEndPoint(
url = "https://public.opencpu.org/ocpu/library/stats/R/rnorm",
format = "json"),
body = list(n = 10, mean = 100),
encode = 'json')
fromJSON(rawToChar(resp$content))
I'm trying to POST data to JIRA Project using R and I keep getting: Error Bad Request. At first I thought it must be the JSON format that I created. So I wrote the JSON to file and did a curl command from console (see below) and the POST worked just fine.
curl -D- -u fred:fred -X POST -d #sample.json -H "Content-Type: application/json" http://localhost:8090/rest/api/2/issue/
Which brings the issue to my R code. Can someone tell me what am I doing wrong with the RCurl postForm?
Source:
library(RJSONIO)
library(RCurl)
x <- list (
fields = list(
project = c(
c(key="TEST")
),
summary="The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name="Task")
)
)
curl.opts <- list(
userpwd = "fred:fred",
verbose = TRUE,
httpheader = c('Content-Type' = 'application/json',Accept = 'application/json'),
useragent = "RCurl"
)
postForm("http://jirahost:8080/jira/rest/api/2/issue/",
.params= c(data=toJSON(x)),
.opts = curl.opts,
style="POST"
)
rm(list=ls())
gc()
Here's the output of the response:
* About to connect() to jirahost port 80 (#0)
* Trying 10.102.42.58... * connected
* Connected to jirahost (10.102.42.58) port 80 (#0)
> POST /jira/rest/api/2/issue/ HTTP/1.1
User-Agent: RCurl
Host: jirahost
Content-Type: application/json
Accept: application/json
Content-Length: 337
< HTTP/1.1 400 Bad Request
< Date: Mon, 07 Apr 2014 19:44:08 GMT
< Server: Apache-Coyote/1.1
< X-AREQUESTID: 764x1525x1
< X-AUSERNAME: anonymous
< Cache-Control: no-cache, no-store, no-transform
< Content-Type: application/json;charset=UTF-8
< Set-Cookie: atlassian.xsrf.token=B2LW-L6Q7-15BO- MTQ3|bcf6e0a9786f879a7b8df47c8b41a916ab51da0a|lout; Path=/jira
< Connection: close
< Transfer-Encoding: chunked
<
* Closing connection #0
Error: Bad Request
You might find it easier to use httr which has been constructed with
the needs of modern APIs in mind, and tends to set better default
options. The equivalent httr code would be:
library(httr)
x <- list(
fields = list(
project = c(key = "TEST"),
summary = "The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name = "Task")
)
)
POST("http://jirahost:8080/jira/rest/api/2/issue/",
body = RJSONIO::toJSON(x),
authenticate("fred", "fred", "basic"),
add_headers("Content-Type" = "application/json"),
verbose()
)
If that doesn't work, you'll need to supply the output from a successful
verbose curl on the console, and a failed httr call in R.