httr and TLS -- sometimes it works, sometimes it doesn't - r

I've been using httr to export data from REDCap databases into R for a few months now. We recently upgraded our R Studio Server to the most recent version (v0.98.1049) and upgraded to R 3.1.1 at the same time. After that upgrade, my httr::POST calls stopped working, sometimes. The error I keep getting is
Error in function (type, msg, asError = TRUE) :
GnuTLS recv error (-9): A TLS packet with unexpected length was received.
At first I thought it might be an SSL issue, but the error only occurs in certain databases, and in those databases, I can still download the data using RCurl. That is, this code will work
RCurl::postForm(uri=[URL],
.params=list(token=[TOKEN],
content='record',
format='csv'))
But this code will not
httr::POST(url=[URL],
body=list(token=[TOKEN],
content='record',
format='csv'))
Adding further to my confusion, even though I can't export data in projects where this error occurs, I can import data.
I'm out of ideas on where to start. I'd much appreciate any ideas on what might be going wrong here.
(I'd like to provide a reproducible example, but I'm afraid I'm working with healthcare data. sorry)
As requested, here's the verbose() output. It's a slightly different call, but produces the same error. (I used a call that wouldn't risk exposing confidential information)
> httr::POST(url=whi$url,
+ body=list(token=whi$token,
+ content='metadata',
+ format='csv'),
+ httr::verbose(data_in=TRUE, info=TRUE))
* Hostname was found in DNS cache
* Hostname in DNS cache was stale, zapped
* Trying 172.26.30.4...
* Connected to [URL] (172.26.30.4) port 443 (#7)
* found 153 certificates in /home/nutterb/R/x86_64-unknown-linux-gnu-library/3.1/httr/cacert.pem
* SSL re-using session ID
* server certificate verification OK
* common name: [URL] (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: OU=Domain Control Validated,CN=[URL]
* start date: Thu, 10 Apr 2014 17:06:17 GMT
*
* expire date: Sat, 21 Mar 2015 16:35:07 GMT
*
* issuer: C=US,ST=Arizona,L=Scottsdale,O=Starfield Technologies\, Inc.,OU=http://certs.starfieldtech.com/repository/,CN=Starfield Secure Certificate Authority - G2
* compression: NULL
* cipher: AES-128-CBC
* MAC: SHA1
-> POST /redcap/api/ HTTP/1.1
-> User-Agent: curl/7.35.0 Rcurl/1.95.4.1 httr/0.5.0.9000
-> Host: [URL]
-> Accept-Encoding: gzip
-> accept: application/json, text/xml, */*
-> Content-Length: 374
-> Expect: 100-continue
-> Content-Type: multipart/form-data; boundary=------------------------05c968969cc362a9
->
<- HTTP/1.1 100 Continue
>> --------------------------05c968969cc362a9
>> Content-Disposition: form-data; name="token"
>>
>> [TOKEN]
>> --------------------------05c968969cc362a9
>> Content-Disposition: form-data; name="content"
>>
>> metadata
>> --------------------------05c968969cc362a9
>> Content-Disposition: form-data; name="format"
>>
>> csv
>> --------------------------05c968969cc362a9--
<- HTTP/1.1 200 OK
<- Date: Sat, 06 Sep 2014 09:55:43 GMT
<- Expires: 0
<- cache-control: no-store, no-cache, must-revalidate
<- Pragma: no-cache
<- Access-Control-Allow-Origin: *
<- Vary: Accept-Encoding
<- Content-Type: text/html; charset=utf-8
<- Connection: close
<- Content-Encoding: gzip
<-
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
<< NA
* GnuTLS recv error (-9): A TLS packet with unexpected length was received.
* Closing connection 7
Error in function (type, msg, asError = TRUE) :
GnuTLS recv error (-9): A TLS packet with unexpected length was received.
In addition: There were 11 warnings (use warnings() to see them)
> warnings()
Warning messages:
1: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
2: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
3: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
4: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
5: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
6: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
7: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
8: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
9: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
10: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
11: In strsplit(x, "\n", fixed = TRUE) : input string 1 is invalid in this locale
>

Related

httr GET returns wrong content-type

httr 1.4.1
R version 3.6.1 (also tried with 3.5.3)
Edit (adding verbose()) output.
I've got a request as follows:
r <- GET("https://my.cool.domain",add_headers(.headers = c('x-api-key' = 'abcdefg', 'Accept' = "text/csv")), verbose())
On my machine it responds with:
-> GET / HTTP/1.1
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.54.0 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/1.1 200 OK
<- Date: Tue, 26 Nov 2019 17:50:15 GMT
<- Content-Type: text/csv
<- Content-Length: 24902
<- Connection: keep-alive
<- x-amzn-RequestId: ...
<- Content-Encoding: deflate
<- x-amz-apigw-id: ...
<- X-Amzn-Trace-Id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:20
Status: 200
Content-Type: text/csv
Size: 209 kB
cats,dogs...
yes,no...
yes,yes...
no,no...
However on my colleague's machine (same version of httr and R, and also with an updated version of R) I get the following:
-> GET / HTTP/2
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.64.1 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/2 200
<- date: Tue, 26 Nov 2019 17:46:17 GMT
<- content-type: application/json
<- content-length: 21501
<- x-amzn-requestid: ...
<- content-encoding: deflate
<- x-amz-apigw-id: ...
<- x-amzn-trace-id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:30
Status: 200
Content-Type: application/json
Size: 377 kB
I'm working with the developer of the https://my.cool.domain domain and I can confirm that the request header params (x-api-key and 'Accept' = "text/csv") are perfect. And the request works on my machine, and several others, but not this one colleague's.
What's going wrong here and how can I debug this?
Thanks
This was fixed by doing httr::set_config(httr::config(http_version = 1.1)) to force 1.1.

SOAP request in R - Content-length = 0

Can someone wise point out to me where i am going wring with this SOAP request in R? I can get a valid response from the server when sending request with Python but in R i get an empty body in the response (Content-length = 0).
Here is the example request: http://www.bom.gov.au/waterdata/wiski-web-public/GetCapabilities%20example%20request.xml
and the example response: http://www.bom.gov.au/waterdata/wiski-web-public/GetCapabilities%20example%20response.xml
library(RCurl)
headerFields =
c(Accept = "text/xml",
'Content-Type' = "text/xml; charset=utf-8",
SOAPAction = "")
TXbody = '<?xml version="1.0" encoding="UTF-8"?>
<soap12:Envelope xmlns:soap12="http://www.w3.org/2003/05/soap-envelope" xmlns:sos="http://www.opengis.net/sos/2.0" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:fes="http://www.opengis.net/fes/2.0" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:swes="http://www.opengis.net/swes/2.0" xsi:schemaLocation="http://www.w3.org/2003/05/soap-envelope http://www.w3.org/2003/05/soap-envelope/soap-envelope.xsd http://www.opengis.net/sos/2.0 http://schemas.opengis.net/sos/2.0/sos.xsd">
<soap12:Header>
<wsa:To>http://www.ogc.org/SOS</wsa:To>
<wsa:Action>
http://www.opengis.net/def/serviceOperation/sos/core/2.0/GetCapabilities
</wsa:Action>
<wsa:ReplyTo>
<wsa:Address>http://www.w3.org/2005/08/addressing/anonymous</wsa:Address>
</wsa:ReplyTo>
<wsa:MessageID>0</wsa:MessageID>
</soap12:Header>
<soap12:Body>
<sos:GetCapabilities service="SOS"/>
</soap12:Body>
</soap12:Envelope>'
h = basicTextGatherer()
R <- curlPerform(url = "http://www.bom.gov.au/waterdata/services?service=SOS",
httpheader = headerFields,
postfields = TXbody, verbose=TRUE,
writefunction = h$update)
RXbody <- h$value()
The response i get is:
Trying 104.99.8.39...
Connected to www.bom.gov.au (104.99.8.39) port 80 (#0)
POST /waterdata/services?service=SOS HTTP/1.1 Host: www.bom.gov.au Accept: text/xml Content-Type: text/xml; charset=utf-8 Content-Length:
1018
upload completely sent off: 1018 out of 1018 bytes < HTTP/1.1 200 OK < Access-Control-Allow-Origin: * < Content-Type: text/plain;
charset=UTF-8 < Server: Apache-Coyote/1.1 < X-UA-Compatible: IE=Edge <
Content-Length: 0 < Expires: Tue, 12 Mar 2019 08:11:53 GMT <
Cache-Control: max-age=0, no-cache, no-store < Pragma: no-cache <
Date: Tue, 12 Mar 2019 08:11:53 GMT < Connection: keep-alive <
Connection #0 to host www.bom.gov.au left intact
I have tried this: SOAP request in R
and this: SOAP request failure in R

character vector and JSON in R

I called an API from R using getURL that returns a JSON response.
When I check with typeof in R, it gives me [1] "character".
I am trying to have my data in JSON format as it should be, to be able to convert it to a DataTable. What could be the reason that it is a character list and how do I fix it?
This is what I am getting in the data returned from the API:
[1] "HTTP/1.1 200 OK\r\nDate: Thu, 04 Jan 2018 20:38:50 GMT\r\nContent-Type: application/json; charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nSet-Cookie: __cfduid=d6bbf45645c3bd5332f83d25d06d8b8ca1515098329; expires=Fri, 04-Jan-19 20:38:49 GMT; path=/; domain=.onesignal.com; HttpOnly\r\nStatus: 200 OK\r\nCache-Control: public, max-age=7200\r\nAccess-Control-Allow-Origin: *\r\nX-XSS-Protection: 1; mode=block\r\nX-Request-Id: bd2552de-bf7d-4a0c-94d6-ff1b6856002a\r\nAccess-Control-Allow-Headers: SDK-Version\r\nETag: W/\"47580e0a23e806945b01f1237219175c\"\r\nX-Frame-Options: SAMEORIGIN\r\nX-Runtime: 0.112902\r\nX-Content-Type-Options: nosniff\r\nX-Powered-By: Phusion Passenger 5.1.4\r\nCF-Cache-Status: REVALIDATED\r\nExpires: Thu, 04 Jan 2018 22:38:50 GMT\r\nServer: cloudflare-nginx\r\nCF-RAY: 3d8100f109c6a23f-ICN\r\n\r\n{\"total_count\":2057,\"offset\":0,\"limit\":50,\"notifications\":[{\"adm_big_picture\":\"\",\"adm_group\":\"\",\"adm_group_message\":{\"en\":\"\... <truncated>
If I try to use fromJSON function with this data,
I get:
Error in file(con, "r") : cannot open the connection
jsonlite::fromJSON works great for parsing JSON. Your problem is that you have a bunch of stuff in front of your JSON. (Maybe after too, can't tell...)
I think the JSON starts at the first {, so we'll remove everything before that. Calling your data x:
x = sub('^[^\\{]*\\{', '{', x)
jsonlite::fromJSON(x)
Type the unescaped version of the patter into the Regex101 tool for an explanation. (Unescaped version uses single not double backslashes: ^[^\{]*\{ . In R strings we need to double the backslashes.)
Here's a working example based on your data:
x = 'HTTP/1.1 200 OK
Date: Thu, 04 Jan 2018 20:38:50 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d6bbf45645c3bd5332f83d25d06d8b8ca1515098329; expires=Fri, 04-Jan-19 20:38:49 GMT; path=/; domain=.onesignal.com; HttpOnly
Status: 200 OK
Cache-Control: public, max-age=7200
Access-Control-Allow-Origin: *
X-XSS-Protection: 1; mode=block
X-Request-Id: bd2552de-bf7d-4a0c-94d6-ff1b6856002a
Access-Control-Allow-Headers: SDK-Version
ETag: W/\"47580e0a23e806945b01f1237219175c\"
X-Frame-Options: SAMEORIGIN
X-Runtime: 0.112902
X-Content-Type-Options: nosniff
X-Powered-By: Phusion Passenger 5.1.4
CF-Cache-Status: REVALIDATED\r\nExpires: Thu, 04 Jan 2018 22:38:50 GMT
Server: cloudflare-nginx
CF-RAY: 3d8100f109c6a23f-ICN
{\"total_count\":2057,\"offset\":0,\"limit\":50,\"notifications\":[{\"adm_big_picture\":\"\",\"adm_group\":\"\"}]}'
y = gsub('^[^\\{]*\\{', '{', x)
jsonlite::fromJSON(sub('^(^\\{)*\\{', '{', y))
# $total_count
# [1] 2057
#
# $offset
# [1] 0
#
# $limit
# [1] 50
#
# $notifications
# adm_big_picture adm_group
# 1
You can use the rjson package to transform your input into a json. Using simplifyDataFrame parameter fromJSON should output a dataframe object.
Importing data from a JSON file into R
[edit]
Your data is returning with some header, you can overcome it removing it from the string and passing to fromJSON
library(stringr)
library(rjson)
json <- str_sub(str_extract(data, "ICN\\r\\n\\r\\n.*"), 8)
df <- as.data.frame(fromJSON(json))
> head(df)
total_count
1 2057

Wrong oauth_signature

Please, can you help me to figure out what is wrong with the signature? I really, can't understand what is the problem. I have been searching for the answer for over a month. If you give me an example, of the similar working
request or help me to understand the problem I would appreciate it so much!
library(RCurl)
library(httr)
library(rvest)
cons_key<-"ae1dd19212a84c4299f4b157462d32d7"
shared_secret<-"c0bc9938ea754f4ba6e926865b347fae"
encoded_url<-URLencode("http://platform.fatsecret.com/rest/server.api",reserved = T,repeated = T)
encoded_string<-URLencode("method=GET&oauth_consumer_key=cons_key&oauth_nonce=asdas4&oauth_signature_method=HMAC-SHA1&oauth_timestamp=15138110238&oauth_version=1.0",reserved = T,repeated = T)
text_string<-paste0("GET&",encoded_url,"&",encoded_string)
signature_oauth<-sha1_hash(key = "ae1dd19212a84c4299f4b157462d32d7&",string = text_string)
getForm("http://platform.fatsecret.com/rest/server.api",
method="GET",
oauth_consumer_key=cons_key,
oauth_signature=signature_oauth,
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1513811023",
oauth_nonce="asdas4",
oauth_version="1.0",.opts = list(verbose = TRUE))
The answer is:
Trying 34.225.169.9...
* Connected to platform.fatsecret.com (34.225.169.9) port 80 (#0)
> GET /rest/server.api method=GET&oauth_consumer_key=ae1dd19212a84c4299f4b157462d32d7&oauth_signature=rB2IYo5tZ%2BAUIsk%2BjlP0ahtn%2Bhk%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1513811023&oauth_nonce=asdas4&oauth_version=1.0 HTTP/1.1
Host: platform.fatsecret.com
Accept: */*
< HTTP/1.1 200 OK
< Date: Wed, 20 Dec 2017 23:05:44 GMT
< Content-Type: text/xml; charset=utf-8
< Content-Length: 377
< Connection: keep-alive
< Cache-Control: private
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
<
* Connection #0 to host platform.fatsecret.com left intact
[1] "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r\n<error
xmlns=\"http://platform.fatsecret.com/api/1.0/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://platform.fatsecret.com/api/1.0/ http://platform.fatsecret.com/api/1.0/fatsecret.xsd\">\r\n\t<code>8</code>\r\n\t<message>Invalid signature: oauth_signature 'rB2IYo5tZ+AUIsk+jlP0ahtn+hk='</message>\r\n</error>\r\n"
attr(,"Content-Type")
charset
"text/xml" "utf-8"

RCurl JSON data to JIRA REST add issue

I'm trying to POST data to JIRA Project using R and I keep getting: Error Bad Request. At first I thought it must be the JSON format that I created. So I wrote the JSON to file and did a curl command from console (see below) and the POST worked just fine.
curl -D- -u fred:fred -X POST -d #sample.json -H "Content-Type: application/json" http://localhost:8090/rest/api/2/issue/
Which brings the issue to my R code. Can someone tell me what am I doing wrong with the RCurl postForm?
Source:
library(RJSONIO)
library(RCurl)
x <- list (
fields = list(
project = c(
c(key="TEST")
),
summary="The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name="Task")
)
)
curl.opts <- list(
userpwd = "fred:fred",
verbose = TRUE,
httpheader = c('Content-Type' = 'application/json',Accept = 'application/json'),
useragent = "RCurl"
)
postForm("http://jirahost:8080/jira/rest/api/2/issue/",
.params= c(data=toJSON(x)),
.opts = curl.opts,
style="POST"
)
rm(list=ls())
gc()
Here's the output of the response:
* About to connect() to jirahost port 80 (#0)
* Trying 10.102.42.58... * connected
* Connected to jirahost (10.102.42.58) port 80 (#0)
> POST /jira/rest/api/2/issue/ HTTP/1.1
User-Agent: RCurl
Host: jirahost
Content-Type: application/json
Accept: application/json
Content-Length: 337
< HTTP/1.1 400 Bad Request
< Date: Mon, 07 Apr 2014 19:44:08 GMT
< Server: Apache-Coyote/1.1
< X-AREQUESTID: 764x1525x1
< X-AUSERNAME: anonymous
< Cache-Control: no-cache, no-store, no-transform
< Content-Type: application/json;charset=UTF-8
< Set-Cookie: atlassian.xsrf.token=B2LW-L6Q7-15BO- MTQ3|bcf6e0a9786f879a7b8df47c8b41a916ab51da0a|lout; Path=/jira
< Connection: close
< Transfer-Encoding: chunked
<
* Closing connection #0
Error: Bad Request
You might find it easier to use httr which has been constructed with
the needs of modern APIs in mind, and tends to set better default
options. The equivalent httr code would be:
library(httr)
x <- list(
fields = list(
project = c(key = "TEST"),
summary = "The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name = "Task")
)
)
POST("http://jirahost:8080/jira/rest/api/2/issue/",
body = RJSONIO::toJSON(x),
authenticate("fred", "fred", "basic"),
add_headers("Content-Type" = "application/json"),
verbose()
)
If that doesn't work, you'll need to supply the output from a successful
verbose curl on the console, and a failed httr call in R.

Resources