Login with rvest/httr - r

I am trying manage logging in with httr.
I have tried the following:
username<-"email"; password<-"password"
page<-html_session("https://www.crowdcube.com/login")
page<-request_POST(page,url=page$response$url,
body=list("email"=username,
"password"=password
,"redirect_url"="https://www.crowdcube.com"),
encode='json',verbose()
)
and simply this (I have tried to provide a url to redirect to, but that did not work either):
x <- POST("https://www.crowdcube.com/login", authenticate(username, password),config(followlocation = 0), encode="json", verbose())
He is what the latter produced (the first one also triggers 302 error):
-> POST /login HTTP/1.1
-> Host: www.crowdcube.com
-> Authorization: Basic iYmlAbWFpbC5ydToxUnVzc2
-> User-Agent: libcurl/7.64.1 r-curl/4.0 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Cookie: CROWDCUBE=g59tvlaq5v5lrpsgogak9; __cfduid=d18fd4953c89e16e5713cff6ba34262
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Length: 0
->
<- HTTP/1.1 302 Found
<- Date: Thu, 19 Nov 2020 23:59:49 GMT
<- Content-Type: text/html; charset=UTF-8
<- Transfer-Encoding: chunked
<- Connection: keep-alive
<- X-XSS-Protection: 1; mode=block
<- Strict-Transport-Security: max-age=15552000
<- Cache-control: no-store, no-cache
<- Cache-Control: no-cache
<- Location: /login
<- X-Request-Id: 3bf2-5d1b-483d-9f81-a78af46a
<- x-frame-options: SAMEORIGIN
<- CF-Cache-Status: DYNAMIC
<- cf-request-id: 4600000015f00a81e00000
<- Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
<- X-Content-Type-Options: nosniff
<- Server: cloudflare
<- CF-RAY: 31cc99215f0-ARN
Rseleniumcan probably manage the problem, but I am trying to avoid it
Edit in response to #MrFlick. The first version produces the following:
-> POST /login HTTP/1.1
-> Host: www.crowdcube.com
-> User-Agent: libcurl/7.64.1 r-curl/4.0 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Cookie: AWSALBCORS=fLpF/isCxxMpnt/e2/lc1CGgyln7VFK6uuHNe94mwJKERlhwo8tgXwd0rDlEwZOGdKr6RzYFAvHhk82tgefXSBVqRE7q4le+1H02+QS+a0/eyMlT6vKMBIkpDi5k; CROWDCUBE=rc279ndia8vjstjb95g0l2pvt1; AWSALB=fLpF/isCxxMpnt/e2/lc1CGgyln7VFK6uuHNe94mwJKERlhwo8tgXwd0rDlEwZOGdKr6RzYFAvHhk82tgefXSBVqRE7q4le+1H02+QS+a0/eyMlT6vKMBIkpDi5k; __cfduid=d81f8fe2764634872724f3ea67e9226581605830327
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/json
-> Content-Length: 92
->
>> {"email":"eeeee","password":"ppppp","redirect_url":"https://www.crowdcube.com"}
<- HTTP/1.1 302 Found
<- Date: Fri, 20 Nov 2020 00:44:39 GMT
<- Content-Type: text/html; charset=UTF-8
<- Transfer-Encoding: chunked
<- Connection: keep-alive
<- X-XSS-Protection: 1; mode=block
<- Strict-Transport-Security: max-age=15552000
<- Cache-control: no-store, no-cache
<- Cache-Control: no-cache
<- Location: /login
<- X-Request-Id: 5eaeee34-f6e1-474c-96bf-f9185226fc78
<- x-frame-options: SAMEORIGIN
<- CF-Cache-Status: DYNAMIC
<- cf-request-id: 0684b5538d00001c06f2970000000001
<- Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
<- X-Content-Type-Options: nosniff
<- Server: cloudflare
<- CF-RAY: 5f4e24cc1f471c06-OSL
<-
-> GET /login HTTP/1.1
-> Host: www.crowdcube.com
-> User-Agent: libcurl/7.64.1 r-curl/4.0 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> Referer: https://www.crowdcube.com/login
-> Cookie: AWSALBCORS=fLpF/isCxxMpnt/e2/lc1CGgyln7VFK6uuHNe94mwJKERlhwo8tgXwd0rDlEwZOGdKr6RzYFAvHhk82tgefXSBVqRE7q4le+1H02+QS+a0/eyMlT6vKMBIkpDi5k; CROWDCUBE=rc279ndia8vjstjb95g0l2pvt1; AWSALB=fLpF/isCxxMpnt/e2/lc1CGgyln7VFK6uuHNe94mwJKERlhwo8tgXwd0rDlEwZOGdKr6RzYFAvHhk82tgefXSBVqRE7q4le+1H02+QS+a0/eyMlT6vKMBIkpDi5k; __cfduid=d81f8fe2764634872724f3ea67e9226581605830327
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/json
->
<- HTTP/1.1 200 OK
<- Date: Fri, 20 Nov 2020 00:44:39 GMT
<- Content-Type: text/html; charset=UTF-8
<- Transfer-Encoding: chunked
<- Connection: keep-alive
<- X-XSS-Protection: 1; mode=block
<- Strict-Transport-Security: max-age=15552000
<- Cache-control: no-store, no-cache
<- Cache-Control: no-cache
<- X-Request-Id: fda74a48-75a0-4420-98b5-d565b1dc8df8
<- x-frame-options: SAMEORIGIN
<- CF-Cache-Status: DYNAMIC
<- cf-request-id: 0684b5542300001c06f7b5d000000001
<- Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
<- X-Content-Type-Options: nosniff
<- Server: cloudflare
<- CF-RAY: 5f4e24cd08101c06-OSL
<- Content-Encoding: gzip

Related

httr GET returns wrong content-type

httr 1.4.1
R version 3.6.1 (also tried with 3.5.3)
Edit (adding verbose()) output.
I've got a request as follows:
r <- GET("https://my.cool.domain",add_headers(.headers = c('x-api-key' = 'abcdefg', 'Accept' = "text/csv")), verbose())
On my machine it responds with:
-> GET / HTTP/1.1
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.54.0 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/1.1 200 OK
<- Date: Tue, 26 Nov 2019 17:50:15 GMT
<- Content-Type: text/csv
<- Content-Length: 24902
<- Connection: keep-alive
<- x-amzn-RequestId: ...
<- Content-Encoding: deflate
<- x-amz-apigw-id: ...
<- X-Amzn-Trace-Id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:20
Status: 200
Content-Type: text/csv
Size: 209 kB
cats,dogs...
yes,no...
yes,yes...
no,no...
However on my colleague's machine (same version of httr and R, and also with an updated version of R) I get the following:
-> GET / HTTP/2
-> Host: https://my.cool.domain
-> User-Agent: libcurl/7.64.1 r-curl/4.2 httr/1.4.1
-> Accept-Encoding: deflate, gzip
-> x-api-key: abcdefg
-> Accept: text/csv
->
<- HTTP/2 200
<- date: Tue, 26 Nov 2019 17:46:17 GMT
<- content-type: application/json
<- content-length: 21501
<- x-amzn-requestid: ...
<- content-encoding: deflate
<- x-amz-apigw-id: ...
<- x-amzn-trace-id: ...
Response [https://my.cool.domain]
Date: 2019-11-26 17:30
Status: 200
Content-Type: application/json
Size: 377 kB
I'm working with the developer of the https://my.cool.domain domain and I can confirm that the request header params (x-api-key and 'Accept' = "text/csv") are perfect. And the request works on my machine, and several others, but not this one colleague's.
What's going wrong here and how can I debug this?
Thanks
This was fixed by doing httr::set_config(httr::config(http_version = 1.1)) to force 1.1.

What part of the Set-Cookie should be sent to the server?

I have this http request:
> Header contents:
> HTTP/1.1 200 OK
> Cache-Control: max-age=0, no-store
> Date: Wed, 17 Jul 2019 08:55:45 GMT
> Content-Length: 22049
> Content-Type: text/html; charset=utf-8
> Expires: Wed, 17 Jul 2019 08:54:46 GMT
> P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
> Server: Microsoft-IIS/10.0
> Set-Cookie: uaid=d796f99cd2d0412896d89ff49e17f558; domain=login.live.com;secure= ;path=/;HTTPOnly= ;version=1
> Set-Cookie: MSPRequ=lt=1563353746&co=1&id=N; secure= ;path=/;HTTPOnly=;version=1
> Set-Cookie: MSCC=185.114.139.243-UA; expires=Mon, 10-Aug-2020 08:55:46 GMT;domain=login.live.com;secure= ;path=/;HTTPOnly=
> ;version=1
> Set-Cookie: OParams=11DYc12zw!m0m7KBb!DgFPYsJXqpdnsnm6nN3KgjlWTBX2lFFtVrtpWXroTonZhmkMvYlU!f3nP2nNmxl1e06ODEOqVKryMr2c8oUmEwI6X!pkrwGqp*d3yzT3VgKmGYMM8kan2MCPifbPjx!8Ww3OhjXTuJz6OCXClnzjw6BP2S8xT8jhvTPxOwiLO1aKqS65f2TXcEHLKmR7yHPQp773y8M$;
> domain=login.live.com;secure= ;path=/;HTTPOnly= ;version=1
> Set-Cookie: MSPOK=$uuid-3a02e0a2-6a86-42f0-ab2a-d8252a1a9567; domain=login.live.com;secure= ;path=/;HTTPOnly= ;version=1
> X-Frame-Options: deny
> PPServer: PPV: 30 H: BL02PF05B1B33B1 V: 0
> X-Content-Type-Options: nosniff
> Strict-Transport-Security: max-age=31536000
> X-XSS-Protection: 1; mode=block
What part of the Set-Cookie should be sent to the server?
I didn't find something about MSPRequ, MSCC, MSPOK. Thank for your answer.

SOAP request in R - Content-length = 0

Can someone wise point out to me where i am going wring with this SOAP request in R? I can get a valid response from the server when sending request with Python but in R i get an empty body in the response (Content-length = 0).
Here is the example request: http://www.bom.gov.au/waterdata/wiski-web-public/GetCapabilities%20example%20request.xml
and the example response: http://www.bom.gov.au/waterdata/wiski-web-public/GetCapabilities%20example%20response.xml
library(RCurl)
headerFields =
c(Accept = "text/xml",
'Content-Type' = "text/xml; charset=utf-8",
SOAPAction = "")
TXbody = '<?xml version="1.0" encoding="UTF-8"?>
<soap12:Envelope xmlns:soap12="http://www.w3.org/2003/05/soap-envelope" xmlns:sos="http://www.opengis.net/sos/2.0" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:fes="http://www.opengis.net/fes/2.0" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:swes="http://www.opengis.net/swes/2.0" xsi:schemaLocation="http://www.w3.org/2003/05/soap-envelope http://www.w3.org/2003/05/soap-envelope/soap-envelope.xsd http://www.opengis.net/sos/2.0 http://schemas.opengis.net/sos/2.0/sos.xsd">
<soap12:Header>
<wsa:To>http://www.ogc.org/SOS</wsa:To>
<wsa:Action>
http://www.opengis.net/def/serviceOperation/sos/core/2.0/GetCapabilities
</wsa:Action>
<wsa:ReplyTo>
<wsa:Address>http://www.w3.org/2005/08/addressing/anonymous</wsa:Address>
</wsa:ReplyTo>
<wsa:MessageID>0</wsa:MessageID>
</soap12:Header>
<soap12:Body>
<sos:GetCapabilities service="SOS"/>
</soap12:Body>
</soap12:Envelope>'
h = basicTextGatherer()
R <- curlPerform(url = "http://www.bom.gov.au/waterdata/services?service=SOS",
httpheader = headerFields,
postfields = TXbody, verbose=TRUE,
writefunction = h$update)
RXbody <- h$value()
The response i get is:
Trying 104.99.8.39...
Connected to www.bom.gov.au (104.99.8.39) port 80 (#0)
POST /waterdata/services?service=SOS HTTP/1.1 Host: www.bom.gov.au Accept: text/xml Content-Type: text/xml; charset=utf-8 Content-Length:
1018
upload completely sent off: 1018 out of 1018 bytes < HTTP/1.1 200 OK < Access-Control-Allow-Origin: * < Content-Type: text/plain;
charset=UTF-8 < Server: Apache-Coyote/1.1 < X-UA-Compatible: IE=Edge <
Content-Length: 0 < Expires: Tue, 12 Mar 2019 08:11:53 GMT <
Cache-Control: max-age=0, no-cache, no-store < Pragma: no-cache <
Date: Tue, 12 Mar 2019 08:11:53 GMT < Connection: keep-alive <
Connection #0 to host www.bom.gov.au left intact
I have tried this: SOAP request in R
and this: SOAP request failure in R

Wrong oauth_signature

Please, can you help me to figure out what is wrong with the signature? I really, can't understand what is the problem. I have been searching for the answer for over a month. If you give me an example, of the similar working
request or help me to understand the problem I would appreciate it so much!
library(RCurl)
library(httr)
library(rvest)
cons_key<-"ae1dd19212a84c4299f4b157462d32d7"
shared_secret<-"c0bc9938ea754f4ba6e926865b347fae"
encoded_url<-URLencode("http://platform.fatsecret.com/rest/server.api",reserved = T,repeated = T)
encoded_string<-URLencode("method=GET&oauth_consumer_key=cons_key&oauth_nonce=asdas4&oauth_signature_method=HMAC-SHA1&oauth_timestamp=15138110238&oauth_version=1.0",reserved = T,repeated = T)
text_string<-paste0("GET&",encoded_url,"&",encoded_string)
signature_oauth<-sha1_hash(key = "ae1dd19212a84c4299f4b157462d32d7&",string = text_string)
getForm("http://platform.fatsecret.com/rest/server.api",
method="GET",
oauth_consumer_key=cons_key,
oauth_signature=signature_oauth,
oauth_signature_method="HMAC-SHA1",
oauth_timestamp="1513811023",
oauth_nonce="asdas4",
oauth_version="1.0",.opts = list(verbose = TRUE))
The answer is:
Trying 34.225.169.9...
* Connected to platform.fatsecret.com (34.225.169.9) port 80 (#0)
> GET /rest/server.api method=GET&oauth_consumer_key=ae1dd19212a84c4299f4b157462d32d7&oauth_signature=rB2IYo5tZ%2BAUIsk%2BjlP0ahtn%2Bhk%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1513811023&oauth_nonce=asdas4&oauth_version=1.0 HTTP/1.1
Host: platform.fatsecret.com
Accept: */*
< HTTP/1.1 200 OK
< Date: Wed, 20 Dec 2017 23:05:44 GMT
< Content-Type: text/xml; charset=utf-8
< Content-Length: 377
< Connection: keep-alive
< Cache-Control: private
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
<
* Connection #0 to host platform.fatsecret.com left intact
[1] "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\r\n<error
xmlns=\"http://platform.fatsecret.com/api/1.0/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation=\"http://platform.fatsecret.com/api/1.0/ http://platform.fatsecret.com/api/1.0/fatsecret.xsd\">\r\n\t<code>8</code>\r\n\t<message>Invalid signature: oauth_signature 'rB2IYo5tZ+AUIsk+jlP0ahtn+hk='</message>\r\n</error>\r\n"
attr(,"Content-Type")
charset
"text/xml" "utf-8"

curl request in R

curl req:
curl -i -H "Accept: application/json" -H "Content-Type: application/json" -d '{"username":"emailId","password":"passwrd"}' -X POST https://central.vizury.com/-/api/login
res:
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json; charset=utf-8
Date: Wed, 06 Sep 2017 10:47:00 GMT
Expires: 0
Pragma: no-cache
Set-Cookie: viz.sess3=SessionCookieHere; path=/; expires=Wed, 06 Sep 2017 10:49:01 GMT; secure; httponly
Set-Cookie: AWSELB=someval;PATH=/;EXPIRES=Wed, 06 Sep 2017 10:49:01 GMT;SECURE;HTTPONLY
Vary: Accept-Encoding
X-Powered-By: Express
Content-Length: 226
Connection: keep-alive
{"status":"OK","results":{"username":"email","role":"role","products":["webConvert","mobiConvert"],"needsNewPassword":false},"homePath":"/webConvert/#/dashboard/campaignName"}
I need to perform the same action in R:
This is what I have tried so far:
h <- basicHeaderGatherer()
loginUrl <- "https://central.vizury.com/-/api/login"
params <- list('username' = 'username',
'password' = 'password')
loginRes <- postForm(loginUrl, .params=params, style="POST", .opts=curlOptions(headerfunction=h$update, verbose=TRUE))
print("loginres")
print(loginRes)
In response,
print(h$value()['Set-Cookie'] )
I can access Set-Cookie. But how do I access the value of viz.sess3?
Example using curl package:
h <- curl::new_handle()
login_url <- 'https://central.vizury.com/-/api/login'
curl::handle_setform(
handle = h,
username = 'username',
password = 'password'
)
resp <- curl::curl_fetch_memory(login_url, handle = h)
message(resp$status_code)
jsonlite::fromJSON(rawToChar(resp$content))

Resources