Very strange issue. I am trying to connect to some API (inside organization) which require first to POST key & code to some url and then use that cookie to get the desired data (json).
Running the POST request returns status 200 which is good - but no cookie is returned.
Running the same request in Firefox using "httprequester" returns a cookie as expected and works fine.
url<-"https://some_url"
login <- list(
Key="some_key",
Code="some_code"
)
try_temp<-POST(url = url,body=login,encode="form",verbose())
Result is:
-> POST /api/Service/Login HTTP/1.1
-> Host: **************
-> User-Agent: libcurl/7.53.1 r-curl/2.5 httr/1.2.1
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/x-www-form-urlencoded
-> Content-Length: 43
->
>> Key=*****&Code=*******
<- HTTP/1.1 200 OK
<- Content-Type: text/html; charset="utf-8"
<- Content-Length: 6908
<- Connection: Close
<-
Thing is, that the same request works when down in browser.
GET request (after I know the cookie, I use GET in httr passing the cookie i've got. I get the same log as above.
BTW When instead of GET i use BROWSE - R opens the default browser and i see the expected data returned.
I suspect that some of the settings for R are not the same as for Firefox (or any other browser). We don't use PROXY but rather automatic configuration script.
Tnx
Related
I'm trying to upload large files to SharePoint Online directly through R using the Microsoft Graph API. To this end, I'm using the API's createUploadSession, documentation here.
My code looks like this:
httr::PUT(url = "https://graph.microsoft.com/v1.0/sites/.../createUploadSession",
headers = add_headers(.headers = headers),
body = httr::upload_file(file),
encode = mime::guess_type(file),
verbose())
(where 'headers' include authentication and host name, here graph.microsoft.com)
And the resultant request looks like this:
-> PUT /v1.0/sites/.../createUploadSession HTTP/1.1
-> Host: graph.microsoft.com
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: text/plain
-> Content-Length: 4543954542
Of course, this fails:
<- HTTP/1.1 413 Request Entity Too Large
<- Content-Type: text/html
<- Server: Microsoft-IIS/10.0
<- Strict-Transport-Security: max-age=31536000
<- Date: Fri, 02 Oct 2020 12:32:29 GMT
<- Connection: close
<- Content-Length: 67
<-
since as the documentation says, we need to upload in 327,680 byte chunks. However, I was under the assumption that httr's upload_file allows for streaming. This is where I'm stuck: it looks like my request is still trying to upload this all at once, so how do I 'invoke' this streaming behavior? And is some kind of while loop required to continue sending the next chunk of data?
This functionality is now available in the Microsoft365R package. Full disclosure: I'm the author of this package.
site <- get_sharepoint_site("sitename")
site$get_drive()$upload_file("path/to/file", "destfilename")
I am trying to use R to access a web page in our organization using httr GET.
However i get a message that "Access is denied due to invalid credentials".
I can do the desired action manually.
it seems that authorization is done automatically when i use internet explorer to get to the web site but the access is blocked when trying to do the same action through R.
This is how i'm trying to do it:
(I can't supply the exact address because it's an intranet address which can be used only inside the organization the same for the proxy address)
library(httr)
r <- GET(myurl, useproxy(myproxyid, 80), verbose())
-> GET http: //myurl
-> host: xxx
-> User-Agent : libcurl...
-> Accept-Encoding: gzip, deflate
-> Proxy-Connection: Keep-Alive
-> Accept: application/json, text/xml, application/xml, *.*
<- HTTP/1.1 401 Unauthorized
<- Content-Type: text/html
<- Server: Microsoft-IIS/8.5
<- WWW-Authenticate: Negotiate
<- WWW-Authenticate: NTLM
<- X-Powered-By: ASP.NET
r
Response [myurl]
Date
Status: 401
...
<title>401 - Unauthorized: Access is denied due to invalid credentials.</title>
....
I understand that i somehow have to send my credentials with my request.
is it possible to somehow use automatic authentication?
Thanks
Rafael
OK
I finally made it.
I had to supply authentication data to my get command like this:
library(httr)
r <- GET(myurl,
useproxy(myproxyid, 80),
verbose(),
authenticate(user = "myuserid", password = "mypassword", type = "ntlm"))
I hope it helps anybody
Thanks
Rafael
I have been using WHO package in R to get data from WHO database without problems for the last several weeks. Yesterday I found that I could no longer do it. I reproduced the error in different machines using different R versions, running from R itself and from Rstudio, Mac and Windows alike...
Example with 2 of the variables I want to request.
library(WHO)
socio_econ <- c("WHS7_143", "WHS9_95")
SECON <- lapply(socio_econ, function(t) get_data(t))
Here's the error:
Error in get_result(url) : Internal Server Error (HTTP 500).
NOT AN ANSWER JUST SHOWING A DEBUGGING METHOD FOR THE OP
httr::with_verbose(get_data("WHS7_143"))
-> GET /gho/athena/api/GHO/WHS7_143?format=json&profile=simple HTTP/1.1
-> Host: apps.who.int
-> User-Agent: libcurl/7.51.0 r-curl/2.0 httr/1.2.1
-> Accept-Encoding: gzip, deflate
-> Cookie: TS01ac0ef4=015dd60f3e63259629be28ff562fb98a7b99c500697d6a49e2671ad07b50034231788b7dd97944f7f6fd363c9ef2b32a1a34c37a22
-> Accept: application/json, text/xml, application/xml, */*
->
<- HTTP/1.1 500 Internal Server Error
<- Date: Mon, 01 May 2017 20:46:40 GMT
<- Content-Type: text/html;charset=utf-8
<- Content-Length: 1298
<- Via: 1.1 ghodata.who.int
<- Connection: close
<- Set-Cookie: TS01ac0ef4=015dd60f3e63259629be28ff562fb98a7b99c500697d6a49e2671ad07b50034231788b7dd97944f7f6fd363c9ef2b32a1a34c37a22; Path=/
<-
Error in get_result(url) : Internal Server Error (HTTP 500).
It's definitely something happening server-side, but it could be that the API changed and the pkg has not updated yet.
Hitting the inferred URL directly:
suggests it's an Java server error on their end.
I have a program that activate an chip for racing results. (Its just a piece of hardware).
I watch with Fiddler (Sniffing program) the in and outgoing traffic from my pc when I connect the chip with my computer.
The program sends the following HTTP Request:
POST http://example.com/index.php HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Content-Length: 185
Content-Type: application/x-www-form-urlencoded
Host: example.com
Pragma: no-cache
User-Agent: SomeProgram 1.2.3
Data==%0D%0AAjlFNEEw-SOMELONGSECRETKEY-RGAw%3D%3D%0D%0A
I receive the following response:
<?xml version="1.0" encoding="utf-8"?>
<message type="3" result="1" txid="someid" activationdate="" availablecredits="732" firstname="John" lastname="Doe" email="JohnDoe#outlook.com" phonenumber="00123445" notification_email="1" notification_text="1"/>
Is it possible to edit the response so that when the programs check for the availablecredits variable, he get the value 9999 instead of 732.
Im working on a Windows 8 laptop.
Definitely - Fiddler allows you to modify requests and responses by adding rules to FiddlerScript. Citing Fiddler documentation:
To make custom changes to web requests and responses, use
FiddlerScript to add rules to Fiddler's OnBeforeRequest or
OnBeforeResponse function. Which function is appropriate depends on
the objects your code uses: OnBeforeRequest is called before each
request, and OnBeforeResponse is called before each response.
So, all you have to do is to add to OnBeforeResponse the logic for replacing the availablecredits attribute value with any value you desire.
yI'm using the {httr} package to log into an internal web application (Theradoc on IIS7.5) in order to scrape some html (infection) data.
library(httr)
POST("http://ahdc372n2", authenticate("foo", "bar"), encode="multipart"), verbose())
The verbose console output says,
-> POST /theradoc/login/index.cfm HTTP/1.1
-> Authorization: Basic Y2xpbmludGVsbDowMWRFbmdsaXNo
-> User-Agent: curl/7.19.6 Rcurl/1.95.4.3 httr/0.4
-> Host: ahdc372n2.phs-sfalls.amck.net
-> Accept: */*
-> Accept-Encoding: gzip
-> Cookie: JSESSIONID=843052421c871dec2ac3a263b136d475a4a6
->
<- HTTP/1.1 411 Length Required
<- Content-Type: text/html; charset=us-ascii
<- Server: Microsoft-HTTPAPI/2.0
<- Date: Mon, 08 Sep 2014 15:53:02 GMT
<- Connection: close
<- Content-Length: 344
<-
* Closing connection #0
And ultimately I get an ">HTTP Error 411. The request must be chunked or have a content length."
I've reviewed this older post without and useful pointers.
Is there a way to force the Content-Length in the httr POST request?
UPDATE : Manually installing httr_0.5 from the zip archive seems to have solved the problem. Thank you hrbmstr.