I'm trying to access a site that sets a cookie on the basis of what type of visitor:
library(httr)
url <- "https://www.blackrock.com/ca/individual/en/products/product-list#categoryId=1&lvl2=overview"
Essentially everything I've tried returns the this...:
GET(url)
Response [https://www.blackrock.com/ca/individual/en/site-entry?targetUrl=%2Fca%2Findividual%2Fen%2Fproducts%2Fproduct-list%3FsiteEntryPassthrough%3Dtrue]
Status: 200
Content-type: text/html;charset=UTF-8
Size: 270 kB
<!DOCTYPE html><html xmlns="http://www.w3.org/1999/xhtml" prefix="og: http://ogp.me/ns#" lang="en" xml:lang="en"><head><meta content="text/html;charset=UTF-8" http-equiv="Content-type...
<meta name="description" content="" />
<meta name="robots" content="noindex,noarchive,nocache" />
<meta property="og:title" content="Site Entry" />
<meta property="og:type" content="website" />
<meta property="og:image" content="/amer-retail-assets/include/common/images/blackrock_logo.png" />
<meta property="og:site_name" content="BlackRock" />
<meta property="og:locale" content="en_CA" />
<meta property="og:url" content="https://www.blackrock.com/ca/individual/en/site-entry?locale=en_CA" />
<link rel="canonical" href="https://www.blackrock.com/ca/individual/en/site-entry?locale=en_CA" />
So, instead I tried copying the cookie after manually passing the gateway or site-entry page in chrome:
cookie = c(`JSESSION_amer-retail01` = "AC91C17F269D8F6A136E576531FA6E05",
`UnicaID` = "10.39.18.249-1408392737241856",
`UnicaNIODID` = "NQHKNKQYKDm-YxUEYgX",
`_ga` = "GA1.2.127078965.1408392737",
`blkUserType-ca-one` = "institutional",
`s_pers` = "%20s_fid%3D14569E53C0F4EE51-3982590542B2DDA4%7C1471566685393%3B%20gpv%3Dca-one%257Cproducts%257Cproduct%2520list%7C1408410085400%3B%20s_nr%3D1408408285405-Repeat%7C1439944285405%3B%20s_pers_eVar15%3Dprospect%7C1411000285408%3B%20s_pers_prop19%3Danonymous%7C1411000285411%3B",
`s_sess` = "%20s_cc%3Dtrue%3B%20s_sq%3D%3B",
`s_vi` = "[CS]v1|29F92F108507B6C6-6000010A8000E2ED[CE]",
`ts-ca-one-locale` = "en_CA")
GET(url, set_cookies(.cookies = cookie), user_agent("Mozilla/5.0"))
This returns the same result.
However, when I look at the examples for httr I think the results I'm getting for setting the cookie are incorrect.
For example:
> example(set_cookies)
st_cks> set_cookies(a = 1, b = 2)
Config:
List of 1
$ cookie:"a1=;b2="
st_cks> set_cookies(.cookies = c(a = "1", b = "2"))
Config:
List of 1
$ cookie:"a1=;b2="
st_cks> GET("http://httpbin.org/cookies")
Response [http://httpbin.org/cookies]
Status: 200
Content-type: application/json
Size: 19 B
{
"cookies": {}
st_cks> GET("http://httpbin.org/cookies", set_cookies(a = 1, b = 2))
Response [http://httpbin.org/cookies]
Status: 200
Content-type: application/json
Size: 50 B
{
"cookies": {
"a1": "",
"b2": ""
}
I would have expected httpbin.org to return a = 1 and b = 1 instead of a1 = and b2 =. Am I doing something wrong?
You're not doing anything wrong (at least IMO). Taking a look at set_cookies, there's a bit of code there:
cookie <- paste0(names(cookies), cookies_str, sep = "=", collapse = ";")
which is not working as intended (i.e. it's turning a = "1" into a1= instead of a=1).
You can use this one:
sc2 <- function (..., .cookies = character(0)) {
cookies <- c(..., .cookies)
stopifnot(is.character(cookies))
cookies_str <- vapply(cookies, RCurl::curlEscape, FUN.VALUE = character(1))
cookie <- paste(names(cookies), cookies_str, sep = "=",
collapse = ";")
config(cookie = cookie)
}
GET("http://httpbin.org/cookies", sc2(a = 1, b = 2))
## Response [http://httpbin.org/cookies]
## Status: 200
## Content-type: application/json
## Size: 50 B
## {
## "cookies": {
## "a": "1",
## "b": "2"
## }
that uses paste instead of paste0 temporarily until that's fixed (I'll file an issue on the httr github repo).
Related
On my site i have element with ::after css pseudo that uses the "⌙" character as content:
.tags_checks label::after{
content:"⌙"
}
Now it works great on mac chrome. but when displayed on windows chrome it is flipped horizontally (and it is like that in the source code):
Mac (Chrome DevTools):
Windows (Chrome DevTools):
Why is that and how can i make it consist on both mac and windows?
I have written a Python script for tesing purposes and here's the result (FontGlyphsHtml.py 0x2319 on Windows 10, default browser Chrome)
The script (partially documented in comments):
import unicodedata
import sys
import os
from fontTools.ttLib import TTFont, TTCollection
def char_in_font(unicode_char, font):
for cmap in font['cmap'].tables:
if cmap.isUnicode() or cmap.getEncoding() == 'utf_16_be':
if ord(unicode_char) in cmap.cmap:
auxcn = cmap.cmap[ord(unicode_char)]
return auxcn if auxcn != '' else '<nil>'
return ''
def checkfont(char,font,fontdict,fontpath):
nameID_index = 1 # works generally (not always)
for i,f in enumerate(font['name'].names):
# An Introduction to TrueType Fonts: A look inside the TTF format
# https://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=IWS-Chapter08
# 1 = Font Family name, 2 = Font SubFamily name, 4 = Full font name
if f.nameID == 1:
nameID_index = i
break
fontname = font['name'].names[nameID_index].toStr()
if fontname not in fontdict.keys():
aux = char_in_font(char, font)
if aux != '':
fontdict[fontname] = "{} ({}) {} [{}] '{}'".format(
char,
'0x{:04x}'.format(ord(char)),
unicodedata.name(char, '???'),
aux,
fontname
)
def testfont(char):
fontdict = {}
for fontpath in fontsPaths:
font = TTFont(fontpath) # specify the path to the font
checkfont(char,font,fontdict,fontpath)
for fontpath in fontcPaths: # specify the path to the font collection
fonts = TTCollection(fontpath)
for ii in range(len(fonts)):
font = TTFont(fontpath, fontNumber=ii) # fontfile and index
checkfont(char,font,fontdict,fontpath)
return fontdict
def testprint(char):
chardesc = '0x{:04x}'.format(ord(char)), unicodedata.name(char, '???')
charline = '{} {} '.format( char, chardesc)
print('') # empty line for better readability
print(charline)
outfile.write('\n' + htmchar.format(char, chardesc) + '\n')
fontarray = testfont(char)
for x in fontarray.keys():
print(fontarray[x])
outfile.write(htmline.format(x, fontarray[x]) + '\n')
### if __name__ == '__main__':
fontsPaths = []
fontcPaths = []
fontsdirs = [ os.path.join( os.getenv('SystemRoot'), 'Fonts')
, r"D:\Downloads\MathJax-TeX-fonts-otf"
, r"D:\Downloads\Fonty"
# , r"D:\Downloads\Fonty\KrutiDev010\k010"
# , os.path.join( os.getenv('LOCALAPPDATA'), r'Microsoft\Windows\Fonts')
# , os.path.join( os.getenv('ProgramFiles'), r'WindowsApps\Microsoft.WindowsTerminal_1.4.3243.0_x64__8wekyb3d8bbwe')
]
print(fontsdirs, file=sys.stderr)
for fontsdir in fontsdirs:
for root,dirs,files in os.walk( fontsdir ):
for file in files:
if file.endswith(".ttf") or file.endswith(".otf") or file.endswith(".ttc"):
tfile = os.path.join(root,file)
if file.endswith(".ttc"):
fontcPaths.append(tfile)
else:
fontsPaths.append(tfile)
htmfile = 'CharGlyphs.html' ### ### ### ###
htmpref = '''
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta name="generator" content="{}"><title></title>
</head><body>
'''.format(os.path.basename(__file__))
htmchar = '<BR>{} <font color="Red">{}</font><BR>' # line
# htmchar = '<BR>{}<BR>' # line
htmline = '<font face="{}">{}</font><BR>' # font, line
htmsuff = '</body></html>'
outfile = open(htmfile, 'w', encoding='utf-8')
outfile.write(htmpref + '\n')
if len(sys.argv) == 1:
testprint(u"अ") # अ 0x0905 Devanagari Letter A
# testprint(u"😺") # 😺 0x1f63a SMILING CAT FACE WITH OPEN MOUTH
# testprint(u"🐈") # 🐈 0x1f408 CAT
# surprising results (characters from the `Private Use` area)
# testprint("\uf0a3")# 0xf0a3 ???
# testprint("\uf098")# 0xf098 ???
# testprint("\uEE79\uED5F") # [uniEE79] 'Segoe MDL2 Assets'
else:
for i in range( 1, len(sys.argv) ):
if len(sys.argv[i]) >=2:
try:
chars = chr(int(sys.argv[i])) # 0x042F or 1071
except:
try:
chars = chr(int(sys.argv[i],16)) # 042F
except:
chars = (sys.argv[i].
encode('raw_unicode_escape').
decode('unicode_escape')) # ➕🐈\U00010A30\u042F\xFE
else:
chars = sys.argv[i] # Я (Cyrillic Capital Letter Ya)
for char in chars:
testprint(char)
outfile.write(htmsuff)
outfile.close()
os.startfile(htmfile)
You can test yourself all Not Signs excerpted from UnicodeData.Txt:
Char CodePoint Category Description
---- --------- -------- -----------
¬ U+00AC Sm-MathSymbol Not Sign
⌐ U+2310 So-OtherSymbol Reversed Not Sign
⌙ U+2319 So-OtherSymbol Turned Not Sign
⫬ U+2AEC Sm-MathSymbol Double Stroke Not Sign
⫭ U+2AED Sm-MathSymbol Reversed Double Stroke Not Sign
￢ U+FFE2 Sm-MathSymbol Fullwidth Not Sign
I am trying to get data from Eikon Elektron cloud platform.
I ran following codes from https://github.com/Refinitiv/websocket-api/blob/master/Applications/Examples/R/market_price_authentication.R:
library(curl)
> content = paste("grant_type=", "password","&username=", user, "&password=", password, sep="")
> h <- new_handle(copypostfields = content)
> h
<curl handle> (empty)
> handle_setheaders(h,
+ "Content-Type" = "application/x-www-form-urlencoded"
+ )
> handle_setopt(h, ssl_verifypeer = FALSE, ssl_verifyhost = FALSE)
> auth_url = paste("https://", auth_hostname, sep="")# ":", auth_port, "/getToken", sep="")
> auth_url
[1] "https://api.refinitiv.com/auth/oauth2/v1/token"
> req <- curl_fetch_memory(auth_url, **handle = h**)
> req
$url
[1] "https://api.refinitiv.com/auth/oauth2/v1/token"
$status_code
[1] 400
$type
[1] "application/json"
**> h
<curl handle> (https://api.refinitiv.com/auth/oauth2/v1/token)**
> res_headers = parse_headers(req$headers)
> auth_json_string = rawToChar(req$content)
> auth_json = fromJSON(auth_json_string)
> cat(toJSON(auth_json, pretty=TRUE, auto_unbox=TRUE))
{
"error": "invalid_request"
}
As you can see, I got invalid request error. I think the problem lies in curl_fetch_memory and that the handle=h is using same input as auth_url, however it should use something similiar to the input of content. What can I change in my code to make it work?
I found solution how to access the URL. In github was wrongly written that app_id in R file should equal to the code from the App Key generator in Reuters. However, app_id and client_id are different things and you should add client_id=value from App Key generator (not app_id).+ also do not forget to include trapi and etc.. to your content.
I am very newbie in HTTR and connection with API by R. I cannot connect with F.A.C.E API (documentation: https://face-api.sightcorp.com/dev-documentation/)
library(httr)
library("XML")
library("jsonlite")
library(RCurl)
key = 'XXX'
img = 'C:\\my_image.jpg'
The query:
my_json <- POST('https://api-face.sightcorp.com/api/detect/', add_headers(app_key = key, img = img, ethnicity = TRUE ))
print(my_json)
and the answer:
Response [https://api-face.sightcorp.com/api/detect/]
Date: 2019-02-19 20:26
Status: 200
Content-Type: application/json
Size: 88 B
{
"error_code" : 1200,
"description" : "__all__ : img or url parameter is needed."
How shall I implement img or url adress into the code to obtain API analysis?
The solution is as follows:
my_body = list('app_key'=key, 'img'= upload_file(img.url))
my_json <- POST('https://api-face.sightcorp.com/api/detect/',body = my_body)
I'm following the official manual of opencpu package in R. In chapter 4.3 Calling a function It uses curl to test API:
curl http://your.server.com/ocpu/library/stats/R/rnorm -d "n=10&mean=100"
and the sample output is:
/ocpu/tmp/x032a8fee/R/.val
/ocpu/tmp/x032a8fee/stdout
/ocpu/tmp/x032a8fee/source
/ocpu/tmp/x032a8fee/console
/ocpu/tmp/x032a8fee/info
I can use curl to get similar result, but when I try to send this http request using httr package in R, I don't know how to replicate the result. Here is what I tried:
resp <- POST(
url = "localhost/ocpu/library/stats/R/rnorm",
body= "n=10&mean=100"
)
resp
the output is:
Response [HTTP://localhost/ocpu/library/stats/R/rnorm]
Date: 2015-10-16 00:51
Status: 400
Content-Type: text/plain; charset=utf-8
Size: 30 B
No Content-Type header found.
I guess I don't understand what's the equivalence of curl -d parameter in httr, how can I get it correct?
Try this :)
library(httr)
library(jsonlite)
getFunctionEndPoint <- function(url, format) {
return(paste(url, format, sep = '/'))
}
resp <- POST(
url = getFunctionEndPoint(
url = "https://public.opencpu.org/ocpu/library/stats/R/rnorm",
format = "json"),
body = list(n = 10, mean = 100),
encode = 'json')
fromJSON(rawToChar(resp$content))
I'm trying to POST data to JIRA Project using R and I keep getting: Error Bad Request. At first I thought it must be the JSON format that I created. So I wrote the JSON to file and did a curl command from console (see below) and the POST worked just fine.
curl -D- -u fred:fred -X POST -d #sample.json -H "Content-Type: application/json" http://localhost:8090/rest/api/2/issue/
Which brings the issue to my R code. Can someone tell me what am I doing wrong with the RCurl postForm?
Source:
library(RJSONIO)
library(RCurl)
x <- list (
fields = list(
project = c(
c(key="TEST")
),
summary="The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name="Task")
)
)
curl.opts <- list(
userpwd = "fred:fred",
verbose = TRUE,
httpheader = c('Content-Type' = 'application/json',Accept = 'application/json'),
useragent = "RCurl"
)
postForm("http://jirahost:8080/jira/rest/api/2/issue/",
.params= c(data=toJSON(x)),
.opts = curl.opts,
style="POST"
)
rm(list=ls())
gc()
Here's the output of the response:
* About to connect() to jirahost port 80 (#0)
* Trying 10.102.42.58... * connected
* Connected to jirahost (10.102.42.58) port 80 (#0)
> POST /jira/rest/api/2/issue/ HTTP/1.1
User-Agent: RCurl
Host: jirahost
Content-Type: application/json
Accept: application/json
Content-Length: 337
< HTTP/1.1 400 Bad Request
< Date: Mon, 07 Apr 2014 19:44:08 GMT
< Server: Apache-Coyote/1.1
< X-AREQUESTID: 764x1525x1
< X-AUSERNAME: anonymous
< Cache-Control: no-cache, no-store, no-transform
< Content-Type: application/json;charset=UTF-8
< Set-Cookie: atlassian.xsrf.token=B2LW-L6Q7-15BO- MTQ3|bcf6e0a9786f879a7b8df47c8b41a916ab51da0a|lout; Path=/jira
< Connection: close
< Transfer-Encoding: chunked
<
* Closing connection #0
Error: Bad Request
You might find it easier to use httr which has been constructed with
the needs of modern APIs in mind, and tends to set better default
options. The equivalent httr code would be:
library(httr)
x <- list(
fields = list(
project = c(key = "TEST"),
summary = "The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name = "Task")
)
)
POST("http://jirahost:8080/jira/rest/api/2/issue/",
body = RJSONIO::toJSON(x),
authenticate("fred", "fred", "basic"),
add_headers("Content-Type" = "application/json"),
verbose()
)
If that doesn't work, you'll need to supply the output from a successful
verbose curl on the console, and a failed httr call in R.