Establish a connection using Kerberos Authentication - r

I'm trying to establish a connection using kerberos authentication. I think the question I have does not depend on the type of server (in my case it's a cognos tm1 server) nor the language (in my case R with use of the package httr (or RCurl)) since it's more a general http(s) thing.
I do not have much experience using kerberos. According to my understanding there is some negotiation between the client and server following the following steps (here get-requests). The only thin I need to pass is a username, no password is needed.
get(url) -> returning a "WWW-Authenticate: Kerberos" header telling this authmethod is supported.
get(url, header = Authentification: "Negotiate" + token) --> second request, this time with header information "Negotiate" plus token
Server returns some authentification details.
Received details can be sent in the header again and the requested data is sent back
httr (type = gssnegotiate) or curl (4 = CURLAUTH_NEGOTIATE) allow to enter negotiation types. I thought, this should do the negotiation process described above and return the requested data straight ahead. This does not seem to be the case:
library(httr)
httr::set_config(config( ssl_verifypeer = 0L))
httr::set_config(config( ssl_verifyhost = 0L))
GET(url, authenticate(user = "user", password = "", type = "gssnegotiate"), verbose = TRUE)
does not return the desired result. The log says:
-> GET /api/v1/Dimensions('Time')/Hierarchies('Time')/Subsets('Yesterday')/Elements HTTP/1.1
-> Host: myhostaddress.com:20049
-> User-Agent: libcurl/7.47.1 r-curl/1.2 httr/1.2.1
-> Accept-Encoding: gzip, deflate
-> Cookie: TM1SessionId=tbQcdXh4PsIHUQdkW_UyNQ
-> Accept: application/json, text/xml, application/xml, */*
->
<- HTTP/1.1 401 Unauthorized
<- Content-Type: text/plain
<- Content-Length: 0
<- Connection: keep-alive
<- OData-Version: 4.0
<- WWW-Authenticate: Kerberos
<-
* Connection #0 to host myhostaddress.com left intact
I tried the same using (R)curl
library(RCurl)
getURL(url, user = "username", userpwd="", httpauth = 4, verbose = TRUE, ssl.verifypeer = FALSE, ssl.verifyhost = FALSE)
Unfortunately, this wasn't successful as well:
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain
< Content-Length: 0
< Connection: keep-alive
< OData-Version: 4.0
< Set-Cookie: TM1SessionId=WMSrJHGTps0RIbmjCCaW5w; Path=/api/; HttpOnly; Secure
< WWW-Authenticate: Kerberos
Do you have any hints how I could get the desired data? I was also thinking about implementing the steps described above manually. By I'm stuck in step 2, because I do not have a token to send in the negotiation header (and do also not know where to get it from).

This won't work because the server requires WWW-Authenticate: Kerberos, but curl only talks SPNEGO. Modify your server to request WWW-Authenticate: Negotiate and it will work.
Note: no major browser supports pure Kerberos over HTTP, so don't expect any other library to do so.

On Windows, you can use
library(httr)
GET(url, authenticate(user=":", password="", type="gssnegotiate"), verbose = TRUE)
Or if no proxy is required for an internal website, state no proxy explictly as follows:
library(httr)
GET(url, use_proxy(""), authenticate(user=":", password="", type="gssnegotiate"), verbose = TRUE)

Related

Using httr + graph API for large files

I'm trying to upload large files to SharePoint Online directly through R using the Microsoft Graph API. To this end, I'm using the API's createUploadSession, documentation here.
My code looks like this:
httr::PUT(url = "https://graph.microsoft.com/v1.0/sites/.../createUploadSession",
headers = add_headers(.headers = headers),
body = httr::upload_file(file),
encode = mime::guess_type(file),
verbose())
(where 'headers' include authentication and host name, here graph.microsoft.com)
And the resultant request looks like this:
-> PUT /v1.0/sites/.../createUploadSession HTTP/1.1
-> Host: graph.microsoft.com
-> User-Agent: libcurl/7.64.1 r-curl/4.3 httr/1.4.2
-> Accept-Encoding: deflate, gzip
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: text/plain
-> Content-Length: 4543954542
Of course, this fails:
<- HTTP/1.1 413 Request Entity Too Large
<- Content-Type: text/html
<- Server: Microsoft-IIS/10.0
<- Strict-Transport-Security: max-age=31536000
<- Date: Fri, 02 Oct 2020 12:32:29 GMT
<- Connection: close
<- Content-Length: 67
<-
since as the documentation says, we need to upload in 327,680 byte chunks. However, I was under the assumption that httr's upload_file allows for streaming. This is where I'm stuck: it looks like my request is still trying to upload this all at once, so how do I 'invoke' this streaming behavior? And is some kind of while loop required to continue sending the next chunk of data?
This functionality is now available in the Microsoft365R package. Full disclosure: I'm the author of this package.
site <- get_sharepoint_site("sitename")
site$get_drive()$upload_file("path/to/file", "destfilename")

Getting error 401 when using httr get

I am trying to use R to access a web page in our organization using httr GET.
However i get a message that "Access is denied due to invalid credentials".
I can do the desired action manually.
it seems that authorization is done automatically when i use internet explorer to get to the web site but the access is blocked when trying to do the same action through R.
This is how i'm trying to do it:
(I can't supply the exact address because it's an intranet address which can be used only inside the organization the same for the proxy address)
library(httr)
r <- GET(myurl, useproxy(myproxyid, 80), verbose())
-> GET http: //myurl
-> host: xxx
-> User-Agent : libcurl...
-> Accept-Encoding: gzip, deflate
-> Proxy-Connection: Keep-Alive
-> Accept: application/json, text/xml, application/xml, *.*
<- HTTP/1.1 401 Unauthorized
<- Content-Type: text/html
<- Server: Microsoft-IIS/8.5
<- WWW-Authenticate: Negotiate
<- WWW-Authenticate: NTLM
<- X-Powered-By: ASP.NET
r
Response [myurl]
Date
Status: 401
...
<title>401 - Unauthorized: Access is denied due to invalid credentials.</title>
....
I understand that i somehow have to send my credentials with my request.
is it possible to somehow use automatic authentication?
Thanks
Rafael
OK
I finally made it.
I had to supply authentication data to my get command like this:
library(httr)
r <- GET(myurl,
useproxy(myproxyid, 80),
verbose(),
authenticate(user = "myuserid", password = "mypassword", type = "ntlm"))
I hope it helps anybody
Thanks
Rafael

R POST/GET Issues not returning cookie

Very strange issue. I am trying to connect to some API (inside organization) which require first to POST key & code to some url and then use that cookie to get the desired data (json).
Running the POST request returns status 200 which is good - but no cookie is returned.
Running the same request in Firefox using "httprequester" returns a cookie as expected and works fine.
url<-"https://some_url"
login <- list(
Key="some_key",
Code="some_code"
)
try_temp<-POST(url = url,body=login,encode="form",verbose())
Result is:
-> POST /api/Service/Login HTTP/1.1
-> Host: **************
-> User-Agent: libcurl/7.53.1 r-curl/2.5 httr/1.2.1
-> Accept-Encoding: gzip, deflate
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/x-www-form-urlencoded
-> Content-Length: 43
->
>> Key=*****&Code=*******
<- HTTP/1.1 200 OK
<- Content-Type: text/html; charset="utf-8"
<- Content-Length: 6908
<- Connection: Close
<-
Thing is, that the same request works when down in browser.
GET request (after I know the cookie, I use GET in httr passing the cookie i've got. I get the same log as above.
BTW When instead of GET i use BROWSE - R opens the default browser and i see the expected data returned.
I suspect that some of the settings for R are not the same as for Firefox (or any other browser). We don't use PROXY but rather automatic configuration script.
Tnx

r-project {httr} POST authentication gets HTTP ERROR 411 from IIS7.5

yI'm using the {httr} package to log into an internal web application (Theradoc on IIS7.5) in order to scrape some html (infection) data.
library(httr)
POST("http://ahdc372n2", authenticate("foo", "bar"), encode="multipart"), verbose())
The verbose console output says,
-> POST /theradoc/login/index.cfm HTTP/1.1
-> Authorization: Basic Y2xpbmludGVsbDowMWRFbmdsaXNo
-> User-Agent: curl/7.19.6 Rcurl/1.95.4.3 httr/0.4
-> Host: ahdc372n2.phs-sfalls.amck.net
-> Accept: */*
-> Accept-Encoding: gzip
-> Cookie: JSESSIONID=843052421c871dec2ac3a263b136d475a4a6
->
<- HTTP/1.1 411 Length Required
<- Content-Type: text/html; charset=us-ascii
<- Server: Microsoft-HTTPAPI/2.0
<- Date: Mon, 08 Sep 2014 15:53:02 GMT
<- Connection: close
<- Content-Length: 344
<-
* Closing connection #0
And ultimately I get an ">HTTP Error 411. The request must be chunked or have a content length."
I've reviewed this older post without and useful pointers.
Is there a way to force the Content-Length in the httr POST request?
UPDATE : Manually installing httr_0.5 from the zip archive seems to have solved the problem. Thank you hrbmstr.

log into a website to grab the data using RCurl

I wanted to login to the website using RCurl and grab the data from the web (The data cannot be seen without logging in.)
I wanted to export this (for example) "http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone" into R after I log in using RCurl. The issue is I cannot log in using RCurl. I haven't tried this before so mostly I referred to http://www.omegahat.org/RCurl/philosophy.html.
So here's what I tried. (here, 'me#gmail.com' is my user ID and '9999' is my Password - i just made it up.)
library(RJSONIO)
library(rjson)
library(RCurl)
appannie <- getURL("http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/.json?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone, userpwd = me#gmail.com:9999", verbose = TRUE)
But this gave me the message below :
About to connect() to www.appannie.com port 80 (#0)
* Trying 69.167.138.64... * connected
* Connected to www.appannie.com (69.167.138.64) port 80 (#0)
> GET /app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone HTTP/1.1
Host: www.appannie.com
Accept: */*
< HTTP/1.1 403 FORBIDDEN
< Server: nginx/1.1.19
< Date: Fri, 01 Mar 2013 23:41:32 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=10
< Vary: Accept-Encoding
< Vary: Cookie,Accept-Encoding
<
* Connection #0 to host www.appannie.com left intact
So, I went back and read this http://www.omegahat.org/RCurl/philosophy.html again and didn't know what to do, so I tried this after I saw the similar question from stackoverflow.
getURL("http://www.appannie.com/app/ios/instagram/ranking/history/chart_data/?s=2010-10-06&e=2012-06-04&c=143441&f=ranks&d=iphone",.opts=list(userpwd=me#gmail.com:9999"))
But this gives me below output.
[1] ""
Can anyone give me a hint? (After a bunch of different trial, the website starts to send me warnings =(
This is some sort of authentication issue not anything you did wrong with RCurl most likely.
You got through to the server but either your login was incorrect, it wasn't valid or the data is not available via the API.
http://en.wikipedia.org/wiki/HTTP_403

Resources