I'm having no joy getting through the corporate web proxy with RCurl.
I'm using R 3.1.2 and RCurl 1.95.4.5 on Windows 7. I've researched the existing stackoverflow workarounds but none of them work.
Here is my code which I expect to work :
curl <- getCurlHandle()
curlSetOpt(.opts = list(proxy = 'proxyIP:proxyPort',
proxyuserpwd = "domain\\username:password",
proxyauth="ntlm"
), curl = curl)
Res <- getURL('http://yahoo.com', curl=curl)
After hitting this wall I tried with Curl to diagnose more. I actually got the request working with Curl using this:
curl -x proxyIP:port --proxy-ntlm -U domain\username:password http://yahoo.com
I verified with curl (using -v) that NTLM was being used to authenticate.
I don't understand why the RCurl options aren't working as I'm sure I've used the correct setting names.
The error message I see in R is a 407 Proxy Authenticate page.
Is this an RCurl bug?
I sorted it. With the last throw of the dice :
From :
curlSetOpt(.opts = list(proxy = 'proxyIP:proxyPort',
proxyuserpwd = "domain\\username:password",
proxyauth="ntlm"
), curl = curl)
To
curlSetOpt(.opts = list(proxy = 'proxyIP:proxyPort',
proxyusername = "domain\\username",
proxypassword = "password",
proxyauth="ntlm"
), curl = curl)
Related
I am trying to get data from an API with a POST request. The request works well with a direct shell command :
system(sprintf('curl POST -k --tlsv1.2 -v "https://api-gateway.inpi.fr/services/apidiffusion/api/marques/search" -H "X-XSRF-TOKEN: %s" -H \'accept: application/xml\' -H "Content-Type: application/json" -H "Cookie: XSRF-TOKEN=%s; access_token=%s; session_token=%s" -d \'%s\' > test.xml',token,token,access_token,refresh_token,json_request))
However, I would like to use httr for many reasons. I use the following code :
test <- httr::POST(
"https://api-gateway.inpi.fr/services/apidiffusion/api/marques/search",
httr::set_config(config(ssl_verifypeer = 0L)),
config = (add_headers(
"X-XSRF-TOKEN" = token,
"accept" = "application/xml",
"Content-Type" = "application/json",
"Cookie" = sprintf("XSRF-TOKEN=%s; access_token=%s; session_token=%s",token,access_token,refresh_token)
))
,set_cookies(`X-XSRF-TOKEN` = token,
`XSRF-TOKEN` = token,
access_token = access_token,
session_token = refresh_token)
,body = json_request
)
But this returns a 403 Forbidden error (my_token being the token I use) :
$error
[1] "access_denied"
$error_description
[1] "Invalid CSRF Token '*my_token*' was found on the request parameter '_csrf' or header 'X-XSRF-TOKEN'.
It seems like httr did not take into account my cookies because the token is different inside the test object I create :
> test2$cookies
domain flag path secure expiration name value
1 api-gateway.inpi.fr FALSE / FALSE <NA> XSRF-TOKEN *another_token*
Any idea ? I am sorry that I can't create a reproducible example for obvious security reasons.
Thank you !
The solution was wierd.
I had to rid off from httr, I used UNIX system commands instead, and it worked with the same request.
system(sprintf('curl POST -k --tlsv1.2 "https://api-gateway.inpi.fr/services/apidiffusion/api/marques/search" -H "X-XSRF-TOKEN: %s" -H \'accept: application/json\' -H "Content-Type: application/json" -H "Cookie: XSRF-TOKEN=%s; access_token=%s; session_token=%s" -d \'%s\' > %s/res.json',tokens$xsrf_token,tokens$xsrf_token,tokens$access_token,tokens$refresh_token,json_request,tempdir()))
It seems like httr tries to handle cookies by its own, so maybe that's what caused my problem.
I am currently trying to upload a CSV file to a proprietary service via httr::POST(). Unfortunately the Admins are not experienced in R and can give only little support.
This is an example on how it should look like in the command line:
curl -X POST --header 'Content-Type: multipart/form-data' \
-F file=#"member-1-test.csv.gz" 'https://some/api/endpoint'
So, in the following code I just try to stick with the example (and additionally provide a token).
> library(tidyverse)
> library(httr)
# Provide some test data with characters specifically quoted
> test_file <- tibble::tribble(
~keytype, ~key, ~action, ~segment,
6,"\"https://www.google.com\"", 0, 37372818,
6,"\"https://www.sport1.de\"" , 0, 37372818
)
> data.table::fwrite(test_file, "test.csv", quote = FALSE)
> file <- upload_file(path = "C:/R/projects/DefaultWorkingDirectory/test.csv")
> res <- POST(
url = "https://some/api/endpoint",
body = list(file = file),
add_headers(.headers = c('Content-Type' = "multipart/form-data", "Authorization" = token))
)
This gives me the follwing error:
> res
Response [https://some/api/endpoint]
Date: 2020-11-18 09:35
Status: 503
Content-Type: text/plain
Size: 95 B
> content(res, encoding = "UTF8")
"upstream connect error or disconnect/reset before headers. reset reason: connection termination"
Any help or guidance on how to move forward with this issue is very much appreciated. Thanks!
Directly after posting the question it was already solved by one of the Admins :)
The issue was that the encoding needs to be set to "multipart" and that a specific content type needs to be provided which is similar to JSON but needs to be added to the "Accept"-header field.
Here the answer for any future references:
> res <- POST(
url = "https://some/api/endpoint",
body = list(
file = upload_file("test.csv")
),
add_headers(c(
"Authorization" = token,
"Accept" = "specific_format_for_the_application"
)),
encode = "multipart",
verbose()
)
This is, I presume, some really simple curl code that I am trying to translate into a httr format.
curl -X POST \
--user '<email>:<password>' \
--header 'user-key: <user_key>' \
--url https://api.m.com/v1/clients
So far I have tried
library(httr)
POST(url = "https://api.m.com/v1/clients",
add_headers('user-key' = "userkey",
user = 'email:password'))
But without success. Any hints on what is wrong here? Is there an httr equivalent to --user in the curl code?
library(httr)
username <- 'my_user_name'
password <- 'my_password'
POST(url = "https://api.m.com/v1/clients",
config = authenticate(username, password), add_headers("Content-Type: application/json"),
body = upload_file('./my_file.json'),
encode = 'json')
I'm looking to send a PUT command to a server from R. The following unix commandline snippet works fine:
curl -i -k -u "username:pass" -X PUT -d "description=test+test+test" https://somewebsite.com/application
I've tried various iterations on httpPUT(), and I'm now working with the inner function getURLContent:
curl_opts = curlOptions(username = "username", password = "pass", httpauth=AUTH_BASIC, ssl.verifypeer = FALSE, headerfunction=h$update)
url=paste("https://somewebsite.com/", "application", sep="")
info <- "description=test+test+test"
val <- charToRaw(info)
getURLContent(url, infilesize = length(val), readfunction = val, upload = TRUE, customrequest = "PUT", .opts=curl_opts)
The server is sending back a search like response, as if I did not actually post the data. Any thoughts on how to recreate the curl commandline PUT in RCurl?
Thanks!
I want to use the following curl command using RCurl
curl -X POST http://test.reco4j.org:7474/db/data/ext/Reco4jRecommender/node/248/get_recommendations -H "Content-Type: application/json" -d '{"type":"0"}'
so i am using the following R code
library(RCurl)
library(RJSONIO)
postForm("http://test.reco4j.org:7474/db/data/ext/Reco4jRecommender/node/248/get_recommendations",
.opts = list(postfields = toJSON(list(id = "0")),
httpheader = c('Content-Type' = 'application/json', ssl.verifypeer = FALSE)
))
But I get an "Internal server errror", so i am not sure my R code is wrong or it is a windows problem.
The reason I am mentioning this, is that the original curl command fails in windows but works on mac and Linux, so I am not sure the R failure is a windows issue or an R issue.
You have an error in your code. The pair you need to send is "type":"0" you are sending "id":"0".
library(RCurl)
library(RJSONIO)
res <- postForm("http://test.reco4j.org:7474/db/data/ext/Reco4jRecommender/node/248/get_recommendations",
.opts = list(postfields = toJSON(list(type = "0")),
httpheader = c('Content-Type' = 'application/json', ssl.verifypeer = FALSE)
))
out <- fromJSON(rawToChar(res))
> head(out[[1]])
$outgoing_relationships
[1] "http://test.reco4j.org:7474/db/data/node/2285/relationships/out"
$data
$data$movieId
[1] 1342
$data$title
[1] "Convent, The (Convento, O) (1995)"
$data$releaseDate
[1] "14-Jun-1996"
It looks like a bad URL. I get an HTTP ERROR 500: INTERNAL_SERVER_ERROR trying to access that URL in firefox.
EDIT: Disregard, you're right: the curl command worked in a shell prompt. Sorry for doubting you.