Automate multiple requests to a web search form using R - r

I am trying to create a dataframe of combinations of place of residence and tax using https://swisstaxcalculator.estv.admin.ch/#/calculator/income-wealth-tax
I have seen How to automate multiple requests to a web search form using R and Scrape tables by passing multiple search requests using R but both those cases work by building new URLs and I don't think this works for admin.ch (the URL always stays the same).
Using the browser inspector I can extract query from a random calculation and it looks like this:
curl 'https://swisstaxcalculator.estv.admin.ch/delegate/ost-integration/v1/lg-proxy/operation/c3b67379_ESTV/API_calculateSimpleTaxes' \
-H 'Connection: keep-alive' \
-H 'Accept: application/json, text/plain, */*' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: https://swisstaxcalculator.estv.admin.ch' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Referer: https://swisstaxcalculator.estv.admin.ch/' \
-H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7' \
-H 'Cookie: SCDID_S=ji-7X06i9sLI29sYA_JLmUSOd6JpMHsgFG_8ilp1l6EZz8h7y2PMjA$$' \
--data-binary '{"SimKey":null,"TaxYear":2019,"TaxLocationID":100000000,"Relationship":1,"Confession1":5,"Children":[],"Confession2":0,"TaxableIncomeCanton":30000,"TaxableIncomeFed":30000,"TaxableFortune":0}' \
--compressed
It's quite clear I should change the part after --data-binary so I create a vector of taxable incomes and then a vector of --data-binary:
taxable_inc <- seq(30000, 150000, by = 1000)
data <- paste0('{"SimKey":null,"TaxYear":2019,"TaxLocationID":100000000,"Relationship":1,"Confession1":5,"Children":[],"Confession2":0,"TaxableIncomeCanton":',
taxable_inc,
',"TaxableIncomeFed":',
taxable_inc,
',"TaxableFortune":0}')
The next part would be to pass/execute this query and capture the answer but I do not know how to to this. Since the query starts with curl I looked at packages curl and RCurl but I really do not understand the link between handles, urls and queries....

You can use httr to make the call and specify encode = "json" to automatically set the header Content-Type: application/json like in curl and encode the body parameter as a JSON object :
library(httr)
url <- 'https://swisstaxcalculator.estv.admin.ch/delegate/ost-integration/v1/lg-proxy/operation/c3b67379_ESTV/API_calculateSimpleTaxes'
r <- POST(url, body = list(
SimKey = NULL,
TaxYear = 2019,
TaxLocationID = 100000000,
Relationship = 1,
Confession1 = 5,
Children = array(),
Confession2 = 0,
TaxableIncomeCanton = 30000,
TaxableIncomeFed = 30000,
TaxableFortune = 0
), encode = "json", verbose())
data = content(r, "parsed")
print(str(data))
which gives :
{"response":{"IncomeSimpleTaxCanton":1747,"FortuneTaxCanton":0,"IncomeSimpleTaxCity":1747,"IncomeTaxChurch":0,"IncomeTaxCity":1380,"IncomeSimpleTaxFed":119,"PersonalTax":0,"FortuneTaxCity":0,"FortuneSimpleTaxCanton":0,"IncomeTaxFed":119,"FortuneSimpleTaxCity":0,"IncomeTaxCanton":2699,"Location":{"TaxLocationID":100000000,"ZipCode":"1000","BfsID":5586,"CantonID":23,"BfsName":"Lausanne","City":"Lausanne","Canton":"VD"},"FortuneTaxChurch":0}}

Related

Posting httr request to API with xmlquery in body

I am trying to retrieve data from an API with stock data from this site: nasdaqomxnordic.com.
So far, I have succesfully constructed a curl call by inspecting the page in Google Chrome. The curl call works great when I send it from reqbin.com and I receive the expected data.
However, when I try to translate it into an httr request in R, I can't make it work. It seems that httr connects succesfully to the server, but nothing happens after that - the command just keeps running forever in r, until I cancel it manually. I suspect that the issue lies in how the body of the request (the xmlquery) is sent so I have tried various formats and tried curlconverter.com, but with no luck (see code below).
Do you guys have any idea about what could be wrong?
The curl code looks like this (this works perfectly on reqbin.com):
curl 'https://www.nasdaqomxnordic.com/webproxy/DataFeedProxy.aspx' \
-H 'authority: www.nasdaqomxnordic.com' \
-H 'accept: */*' \
-H 'accept-language: se-SE,se;q=0.9,en-US;q=0.8,en;q=0.7' \
-H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \
-H 'origin: https://www.nasdaqomxnordic.com' \
-H 'referer: https://www.nasdaqomxnordic.com/aktier/microsite?Instrument=CSE1158&name=Novo%20Nordisk%20B&ISIN=DK0060534915' \
-H 'sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "Windows"' \
-H 'sec-fetch-dest: empty' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-site: same-origin' \
-H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' \
-H 'x-requested-with: XMLHttpRequest' \
--data-raw 'xmlquery=%3Cpost%3E%0A%3Cparam+name%3D%22Exchange%22+value%3D%22NMF%22%2F%3E%0A%3Cparam+name%3D%22SubSystem%22+value%3D%22History%22%2F%3E%0A%3Cparam+name%3D%22Action%22+value%3D%22GetDataSeries%22%2F%3E%0A%3Cparam+name%3D%22AppendIntraDay%22+value%3D%22no%22%2F%3E%0A%3Cparam+name%3D%22Instrument%22+value%3D%22CSE1158%22%2F%3E%0A%3Cparam+name%3D%22FromDate%22+value%3D%222021-12-25%22%2F%3E%0A%3Cparam+name%3D%22ToDate%22+value%3D%222022-12-25%22%2F%3E%0A%3Cparam+name%3D%22hi__a%22+value%3D%220%2C5%2C6%2C3%2C1%2C2%2C4%2C21%2C8%2C10%2C12%2C9%2C11%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt%22+value%3D%22%2FnordicV3%2Fhi_csv.xsl%22%2F%3E%0A%3Cparam+name%3D%22OmitNoTrade%22+value%3D%22true%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_lang%22+value%3D%22en%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_options%22+value%3D%22%2Cadjusted%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttype%22+value%3D%22application%2Fms-excel%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttypefilename%22+value%3D%22NOVO_B-2021-12-25-2022-12-25.csv%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_hiddenattrs%22+value%3D%22%2Civ%2Cip%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_tableId%22+value%3D%22historicalTable%22%2F%3E%0A%3Cparam+name%3D%22DefaultDecimals%22+value%3D%22false%22%2F%3E%0A%3Cparam+name%3D%22app%22+value%3D%22%2Faktier%2Fmicrosite%22%2F%3E%0A%3C%2Fpost%3E' \
--compressed
My best try at an httr equivalent look like this:
require(httr)
headers = c(
`authority` = 'www.nasdaqomxnordic.com',
`accept` = '*/*',
`accept-language` = 'se-SE,se;q=0.9,en-US;q=0.8,en;q=0.7',
`content-type` = 'application/x-www-form-urlencoded; charset=UTF-8',
`origin` = 'https://www.nasdaqomxnordic.com',
`referer` = 'https://www.nasdaqomxnordic.com/aktier/microsite?Instrument=CSE1158&name=Novo%20Nordisk%20B&ISIN=DK0060534915',
`sec-ch-ua` = '"Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"',
`sec-ch-ua-mobile` = '?0',
`sec-ch-ua-platform` = '"Windows"',
`sec-fetch-dest` = 'empty',
`sec-fetch-mode` = 'cors',
`sec-fetch-site` = 'same-origin',
`user-agent` = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
`x-requested-with` = 'XMLHttpRequest'
)
data = 'xmlquery=%3Cpost%3E%0A%3Cparam+name%3D%22Exchange%22+value%3D%22NMF%22%2F%3E%0A%3Cparam+name%3D%22SubSystem%22+value%3D%22History%22%2F%3E%0A%3Cparam+name%3D%22Action%22+value%3D%22GetDataSeries%22%2F%3E%0A%3Cparam+name%3D%22AppendIntraDay%22+value%3D%22no%22%2F%3E%0A%3Cparam+name%3D%22Instrument%22+value%3D%22CSE1158%22%2F%3E%0A%3Cparam+name%3D%22FromDate%22+value%3D%222021-12-25%22%2F%3E%0A%3Cparam+name%3D%22ToDate%22+value%3D%222022-12-25%22%2F%3E%0A%3Cparam+name%3D%22hi__a%22+value%3D%220%2C5%2C6%2C3%2C1%2C2%2C4%2C21%2C8%2C10%2C12%2C9%2C11%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt%22+value%3D%22%2FnordicV3%2Fhi_csv.xsl%22%2F%3E%0A%3Cparam+name%3D%22OmitNoTrade%22+value%3D%22true%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_lang%22+value%3D%22en%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_options%22+value%3D%22%2Cadjusted%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttype%22+value%3D%22application%2Fms-excel%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttypefilename%22+value%3D%22NOVO_B-2021-12-25-2022-12-25.csv%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_hiddenattrs%22+value%3D%22%2Civ%2Cip%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_tableId%22+value%3D%22historicalTable%22%2F%3E%0A%3Cparam+name%3D%22DefaultDecimals%22+value%3D%22false%22%2F%3E%0A%3Cparam+name%3D%22app%22+value%3D%22%2Faktier%2Fmicrosite%22%2F%3E%0A%3C%2Fpost%3E'
http_response <- httr::POST(url = 'https://www.nasdaqomxnordic.com/webproxy/DataFeedProxy.aspx', httr::add_headers(.headers=headers), body = data, verbose())
I have also tried this alternate formatting of the data-part of the request, but with no better luck:
data = list(
`xmlquery` = '<post>\n<param name="Exchange" value="NMF"/>\n<param name="SubSystem" value="History"/>\n<param name="Action" value="GetDataSeries"/>\n<param name="AppendIntraDay" value="no"/>\n<param name="Instrument" value="CSE1158"/>\n<param name="FromDate" value="2021-12-25"/>\n<param name="ToDate" value="2022-12-25"/>\n<param name="hi__a" value="0,5,6,3,1,2,4,21,8,10,12,9,11"/>\n<param name="ext_xslt" value="/nordicV3/hi_csv.xsl"/>\n<param name="OmitNoTrade" value="true"/>\n<param name="ext_xslt_lang" value="en"/>\n<param name="ext_xslt_options" value=",adjusted,"/>\n<param name="ext_contenttype" value="application/ms-excel"/>\n<param name="ext_contenttypefilename" value="NOVO_B-2021-12-25-2022-12-25.csv"/>\n<param name="ext_xslt_hiddenattrs" value=",iv,ip,"/>\n<param name="ext_xslt_tableId" value="historicalTable"/>\n<param name="DefaultDecimals" value="false"/>\n<param name="app" value="/aktier/microsite"/>\n</post>'
)
```

request within browser is fine,but not in postman

I'm testing a page request in postman(desktop app), but it redirect all requests to captcha page;
While i replay in browser, it works fine even i clean all cookies && local storage, there's no redirections, simply response with 200 code status;
Then I copy all browser's header or cookie(another try), nothing worked!
I have search some Q&A, including ssl verify close in postman, or login in user cache ,not my case at all.
This is a link of my case
Is there anything i omitted? Please help me to know
I can just put a postman request export as bash curl,sorry:
curl --location --request GET 'https://www.fwrd.com/product-burberry-lola-backpack/BURF-WY121/?d=Womens&itrownum=16&itcurrpage=2&itview=05' \
--header 'authority: www.fwrd.com' \
--header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
--header 'accept-language: zh-CN,zh;q=0.9' \
--header 'cache-control: max-age=0' \
--header 'cookie: viewNumR1=100; isPopupEnabledR1=true; pocketViewR1=front; currency=USD; currencyOverride=USD; _fbp=fb.1.1658202232315.1459169565; countryCodePref=US; userLanguagePref=en; _gid=GA1.2.1936343211.1658202234; _pxvid=0273b224-0715-11ed-9b1c-4d6e63515557; __rtbh.lid=%7B%22eventType%22%3A%22lid%22%2C%22id%22%3A%22Bpefyig8l0wxukHxQ1QF%22%7D; requestBrowserIdMapping=1; _px_f394gi7Fvmc43dfg_user_id=MDM3YzUxNjEtMDcxNS0xMWVkLWFjNTItMWRiODkxNDdiYmRi; _scid=88365d6e-855a-4b9b-b456-ed90a888f464; rskxRunCookie=0; rCookie=82jkrqh5x5p7yx1jm7r0lyl5rmsq6u; _sctr=1|1658160000000; visitor-cookie30=true; visitor-cookie1=true; _gcl_au=1.1.467830307.1658204129; _pin_unauth=dWlkPU1qbGhNR1pqWXpFdFkyUTNOUzAwWkdJd0xUaGxNMlV0WmpKaE5UQTFZekkwTkdGaw; recentlyViewedItems=0; sortByR2=featuredF; altexp=%7B%22896%22%3A0%2C%221409%22%3A1%2C%221412%22%3A0%2C%221157%22%3A0%2C%221031%22%3A1%2C%221292%22%3A1%2C%221421%22%3A0%2C%221295%22%3A1%2C%221424%22%3A1%2C%221298%22%3A0%2C%221301%22%3A0%2C%221430%22%3A1%2C%221304%22%3A0%2C%221433%22%3A0%2C%221179%22%3A0%2C%221436%22%3A0%2C%22926%22%3A0%2C%221439%22%3A1%2C%221442%22%3A1%2C%221188%22%3A0%2C%22677%22%3A0%2C%221445%22%3A1%2C%22936%22%3A0%2C%221448%22%3A0%2C%221194%22%3A1%2C%221322%22%3A0%2C%221197%22%3A1%2C%221457%22%3A0%2C%22946%22%3A0%2C%221334%22%3A0%2C%22951%22%3A1%2C%221081%22%3A0%2C%221210%22%3A0%2C%221466%22%3A1%2C%221340%22%3A0%2C%221469%22%3A1%2C%221086%22%3A1%2C%221472%22%3A1%2C%221346%22%3A1%2C%221475%22%3A1%2C%22836%22%3A1%2C%221349%22%3A0%2C%221478%22%3A1%2C%221481%22%3A1%2C%221226%22%3A0%2C%221355%22%3A0%2C%221484%22%3A0%2C%221358%22%3A1%2C%221487%22%3A0%2C%221232%22%3A0%2C%221490%22%3A1%2C%221235%22%3A1%2C%221493%22%3A1%2C%221496%22%3A1%2C%221370%22%3A0%2C%221499%22%3A1%2C%221502%22%3A0%2C%221247%22%3A1%2C%221382%22%3A0%2C%221514%22%3A1%2C%22876%22%3A0%2C%221388%22%3A1%2C%221517%22%3A0%2C%221391%22%3A1%2C%221520%22%3A0%2C%221011%22%3A0%2C%221139%22%3A1%2C%221268%22%3A1%2C%221016%22%3A0%2C%221403%22%3A0%2C%221406%22%3A0%7D; pxcts=a80a374b-07d9-11ed-96b1-636858726351; userSeenFwrdNtfDialogDate=2022-07-19; JSESSIONID=BD3FB79F2CE2EEBB32A4565060EBED39.tc-nova_tomcat4; _ga_MRQYJ2518M=GS1.1.1658293460.7.0.1658293460.60; requestSessionID=4254930521; __cf_bm=O0MTs.5O4z3O.Lz7tV4epamTsK4T1BHxmhi10rmax0Y-1658293459-0-AZMBFcXc0ZL7DtIgX2gvkyH7rG54t0KaUItmqPesO/BchuY7GsvNY/f44PNnN+sHQmfUgQma+uQBOFDsidSC6i8=; _ga=GA1.2.1509327558.1658202234; _gat=1; browserID=QhdWmBQ7WgUehyiYRX1EygeH7OOqck; bb_PageURL=%2Ffw%2Fcontent%2Fproduct%2FstyledWithMarkup%3Fcode%3DBURF-WY121%26d%3DF; _uetsid=038da160071511ed89ea4336c7c04aeb; _uetvid=038dbb20071511eda26e25fc715b0a9c; cto_bundle=eFjkml9qcFJnY3E0WlZVVTFzTWNMRXVWOE9nalBTZnNlM0JjbWR3V0xIQXVSc3hOR0FKVUx1UFZ6dFJ1Y0JiRjQxJTJCeVZuUiUyRk4xdUwlMkJrWjBLeklmTzRZN2FodjJFNDZHdXgybVA5UEhJQmJ5RXk4cTRqNVFJc3BVaGlENFRtVEw3U1YlMkJDR1V2WFc4ZDVmc2NGOFhYMmFrUTFoUSUzRCUzRA; lastRskxRun=1658293471138; _px2=eyJ1IjoiNmM0M2EzZTAtMDdlOS0xMWVkLTgzMTctNWZjNjg1MDMxYzMyIiwidiI6IjAyNzNiMjI0LTA3MTUtMTFlZC05YjFjLTRkNmU2MzUxNTU1NyIsInQiOjE2NTgyOTM3ODM4MjcsImgiOiJkODA4NjI1OGRmYWE1ZTE1OTFmYjdlZTIyYWNiYmI4MDg5OGI4NjkwZWNjMzIxZDYxMTI0ZTNiYzFhNWU2MDAwIn0=; __cf_bm=xGlLbY0GGGMACBg4Z8TtXSuxuW4yk9hjpz8e41d2inQ-1658293709-0-AT1Kf+tBGczWEcAu+1rDk5hhnDdbHleJS/slXdDvGm6E4BrVrSFy2/Ezkiw7aSPjpYyLkLzMpUH2YtPpwXPV6Tw=' \
--header 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "Windows"' \
--header 'sec-fetch-dest: document' \
--header 'sec-fetch-mode: navigate' \
--header 'sec-fetch-site: none' \
--header 'sec-fetch-user: ?1' \
--header 'upgrade-insecure-requests: 1' \
--header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'

Post request with form data to login to a website return 400 - Bad request

I am trying to login into this website with below credentials (no worry, it's anonymous):
user = 'MattieBailey#mailinator.com'
password = 'thx9jD3SkrssPKY'
When I logged in manually, I saw in Chrome inspector view there are sequentially a POST request and a GET request.
However, when I tried to simulate these requests with python, I got 400 - Bad request for the POST request. The GET request returned 200 - OK and sent me the content of the login webpage. In the Chrome Inspector View, all these 2 requests returned 302 status code with some response headers contains authorized id_token to use for further requests.
The same result was obtained when I tried to import these two requests to Postman so the problem might not be my code.
My ultimate goal is to get the id_token parameter in the response header of the GET request.
I appreciate your help!
POST request
curl 'https://auth.fiintrade.vn/Account/Login?ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3D19e5bb5a7df14a2bab8ded1f265a7bda%26nonce%3D1547e70f8a2940b89dac75565e032046%26clientType%3DWEBCLIENT%26fp%3D54404ac5041ff5b042c29fb397bef44d%26type%3Dredirect' \
-H 'Connection: keep-alive' \
-H 'Cache-Control: max-age=0' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Origin: null' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' \
-H 'Cookie: _ga=GA1.2...' \
--data-raw 'ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3D19e5bb5a7df14a2bab8ded1f265a7bda%26nonce%3D1547e70f8a2940b89dac75565e032046%26clientType%3DWEBCLIENT%26fp%3D54404ac5041ff5b042c29fb397bef44d%26type%3Dredirect&Username=MattieBailey%40mailinator.com&Password=thx9jD3SkrssPKY&button=login&__RequestVerificationToken=CfDJ8KNrfTjG6sBLmncBE9MMk0dDAX-OFwPXUy1uT0AqrEyhngGIImEM99JGxVKsssw62kRxLY2tHfTqfH8M8Mphqxh80YdwwHM6vpZIb-yVMnma4nS0QCkOf51FMFFu8pPwMM6FY52WEPSWy_ZWuaiAeCU' \
--compressed
GET request
curl 'https://auth.fiintrade.vn/connect/authorize/callback?client_id=StoxPlus.FiinTrade.SPA&redirect_uri=https%3A%2F%2Ffiintrade.vn%2Fsignin-callback&response_type=id_token%20token&scope=openid%20FiinTrade.Market%20FiinTrade.Core%20FiinTrade.Realtime%20FiinTrade.Fundamental&state=19e5bb5a7df14a2bab8ded1f265a7bda&nonce=1547e70f8a2940b89dac75565e032046&clientType=WEBCLIENT&fp=54404ac5041ff5b042c29fb397bef44d&type=redirect' \
-H 'Connection: keep-alive' \
-H 'Cache-Control: max-age=0' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' \
-H 'Cookie: _ga=GA1.2...' \
--compressed
Response header that contains id_token
Location: https://fiintrade.vn/signin-callback#id_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IkY4MDA1REIxQkI0MzY4Q0Q3RkJFMUUxRTlEQjkwMTg2MUY5NEFGMTEiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItQUJkc2J0RGFNMV92aDRlbmJrQmhoLVVyeEUifQ.eyJuYmYiOjE2MjkxOTM4MTgsImV4cCI6MTYyOTIyMjYxOCwiaXNzIjoiaHR0cHM6Ly9hdXRoLmZpaW50cmFkZS52biIsImF1ZCI6IlN0b3hQbHVzLkZpaW5UcmFkZS5TUEEiLCJub25jZSI6IjE1NDdlNzBmOGEyOTQwYjg5ZGFjNzU1NjVlMDMyMDQ2IiwiaWF0IjoxNjI5MTkzODE4LCJhdF9oYXNoIjoiOVJNNmdEOWFJZkprdGFCSENHRHpwQSIsInNpZCI6IjQ0MWVkZmM1YTc0MzhhOWNkYzYzYTkwYTM3NmQwOGI1Iiwic3ViIjoiMTcwMzU5IiwiYXV0aF90aW1lIjoxNjI5MTkzODE4LCJpZHAiOiJsb2NhbCIsInVzZXJfaWQiOiIxNzAzNTkiLCJ1c2VyX25hbWUiOiJNYXR0aWVCYWlsZXlAbWFpbGluYXRvci5jb20iLCJuYW1lIjoiIiwiZ2l2ZW5fbmFtZSI6Ik1hdHRpZSIsImZhbWlseV9uYW1lIjoiQmFpbGV5IiwibWlkZGxlX25hbWUiOiIiLCJlbWFpbCI6Ik1hdHRpZUJhaWxleUBtYWlsaW5hdG9yLmNvbSIsInNlcnZpY2VfdHlwZSI6IkZpaW5Hcm91cC5GaWluVHJhZGUiLCJsaXN0X3BhY2thZ2UiOiJGaWluVHJhZGUuVHJpYWwiLCJsaXN0X2ZlYXR1cmUiOiIiLCJsaXN0X2FwaSI6IiIsInJvbGUiOiJDVVNUT01FUiIsImdyb3VwX25hbWUiOiJJbmRpdmlkdWFsIiwic3RhcnRfZGF0ZSI6IjEyLzA4LzIwMjEiLCJlbmRfZGF0ZSI6IjI2LzA4LzIwMjEiLCJoaXRjb3VudF9wZXJtb250aCI6IjAiLCJjb21ncm91cF9saW1pdCI6IiIsInRpY2tlcl9saW1pdCI6IiIsInRpbWVyYW5nZV9saW1pdCI6IjAiLCJkYXRhcmFuZ2VfbGltaXQiOiIwIiwicGVyX21pbnV0ZSI6IjAiLCJwZXJfaG91ciI6IjAiLCJwZXJfZGF5IjoiMCIsInBlcl9tb250aCI6IjAiLCJlbmFibGVkIjoiVHJ1ZSIsImxhc3RfYXR0ZW1wdCI6IjgvMTIvMjAyMSAxMTo1ODoxNyBQTSIsImxhc3RfYXR0ZW1wdF9zdGF0dXMiOiJTVUNDRVNTIiwiZmluZ2VycHJpbnQiOiI1NDQwNGFjNTA0MWZmNWIwNDJjMjlmYjM5N2JlZjQ0ZCIsImNsaWVudHR5cGUiOiJXRUJDTElFTlQiLCJhbXIiOlsicHdkIl19.bObyB2sb-kkAxbxhhLc2hQsEtB6YAvzdtd2OwlLGKrzikqoGPwaQYlA41YK_v9OJLqUPRvKvkSZPBwfbguYXp8KnOAHxyPwM4lrbWc7u_OMIHarANxIsA50ckkj1gEszaGZtN6vtz4QYkxLMdbhHcpTnYd4XidFBVoyMQ3csLFaEifJEVsKfGStqgXW6etIhug5yuPnhjnZbCWdUPh0887auIuB_r7Nc7x6faHLtqr2in8c_TTKoI9YoM6LYBNkB7G7AtYkxlVImbyrdczHqsh6_ajLITuuSKYnwws2STwzPQMrZsIbHo_97ce3AVJcdUzVMXeY343FOYBDZeFteJg&access_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IkY4MDA1REIxQkI0MzY4Q0Q3RkJFMUUxRTlEQjkwMTg2MUY5NEFGMTEiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItQUJkc2J0RGFNMV92aDRlbmJrQmhoLVVyeEUifQ.eyJuYmYiOjE2MjkxOTM4MTgsImV4cCI6MTYyOTIyMjQxOCwiaXNzIjoiaHR0cHM6Ly9hdXRoLmZpaW50cmFkZS52biIsImF1ZCI6WyJodHRwczovL2F1dGguZmlpbnRyYWRlLnZuL3Jlc291cmNlcyIsIkZpaW5UcmFkZS5NYXJrZXQiLCJGaWluVHJhZGUuQ29yZSIsIkZpaW5UcmFkZS5SZWFsdGltZSIsIkZpaW5UcmFkZS5GdW5kYW1lbnRhbCJdLCJjbGllbnRfaWQiOiJTdG94UGx1cy5GaWluVHJhZGUuU1BBIiwic3ViIjoiMTcwMzU5IiwiYXV0aF90aW1lIjoxNjI5MTkzODE4LCJpZHAiOiJsb2NhbCIsInVzZXJfaWQiOiIxNzAzNTkiLCJ1c2VyX25hbWUiOiJNYXR0aWVCYWlsZXlAbWFpbGluYXRvci5jb20iLCJuYW1lIjoiIiwiZ2l2ZW5fbmFtZSI6Ik1hdHRpZSIsImZhbWlseV9uYW1lIjoiQmFpbGV5IiwibWlkZGxlX25hbWUiOiIiLCJlbWFpbCI6Ik1hdHRpZUJhaWxleUBtYWlsaW5hdG9yLmNvbSIsInNlcnZpY2VfdHlwZSI6IkZpaW5Hcm91cC5GaWluVHJhZGUiLCJsaXN0X3BhY2thZ2UiOiJGaWluVHJhZGUuVHJpYWwiLCJsaXN0X2ZlYXR1cmUiOiIiLCJsaXN0X2FwaSI6IiIsInJvbGUiOiJDVVNUT01FUiIsImdyb3VwX25hbWUiOiJJbmRpdmlkdWFsIiwic3RhcnRfZGF0ZSI6IjEyLzA4LzIwMjEiLCJlbmRfZGF0ZSI6IjI2LzA4LzIwMjEiLCJoaXRjb3VudF9wZXJtb250aCI6IjAiLCJjb21ncm91cF9saW1pdCI6IiIsInRpY2tlcl9saW1pdCI6IiIsInRpbWVyYW5nZV9saW1pdCI6IjAiLCJkYXRhcmFuZ2VfbGltaXQiOiIwIiwicGVyX21pbnV0ZSI6IjAiLCJwZXJfaG91ciI6IjAiLCJwZXJfZGF5IjoiMCIsInBlcl9tb250aCI6IjAiLCJlbmFibGVkIjoiVHJ1ZSIsImxhc3RfYXR0ZW1wdCI6IjgvMTIvMjAyMSAxMTo1ODoxNyBQTSIsImxhc3RfYXR0ZW1wdF9zdGF0dXMiOiJTVUNDRVNTIiwiZmluZ2VycHJpbnQiOiI1NDQwNGFjNTA0MWZmNWIwNDJjMjlmYjM5N2JlZjQ0ZCIsImNsaWVudHR5cGUiOiJXRUJDTElFTlQiLCJzY29wZSI6WyJvcGVuaWQiLCJGaWluVHJhZGUuTWFya2V0IiwiRmlpblRyYWRlLkNvcmUiLCJGaWluVHJhZGUuUmVhbHRpbWUiLCJGaWluVHJhZGUuRnVuZGFtZW50YWwiXSwiYW1yIjpbInB3ZCJdfQ.RSMsirz2etpYNVx9Pq1W1qExw1ETwCt5XUxglHwreyQX0WDZV3nUDOf5mxgNKjQx_sPCBQuF6SBH8IdNGcuv9ajvd2XbKWdcsNljXidiRHNGogo-JKIZIqadzLAEAmdXoYP_w2ToqOYkWmtEzm2SMpgbVzklRftciQg4h-K_jYW4YpteX23Az78yT3_RIirfBfszaTp5v3BEOR_StHG16bHPauOlIGmch6o6cWUSX1RPnqyAUVaNGELPygNQcmoYVgyQhwoloXyuDY7LGjhK3EgMm9iCuDx5JasFi5A0PQrAfgs1g7UftC-B_7XvJG4pw4qbevj3PehvZ9Kp1RshXw&token_type=Bearer&expires_in=28600&scope=openid%20FiinTrade.Market%20FiinTrade.Core%20FiinTrade.Realtime%20FiinTrade.Fundamental&state=19e5bb5a7df14a2bab8ded1f265a7bda&session_state=dV3zKwxom21hMpGaoJweGLKUlj2rO2gGSFWbw6bIw4U.ab1d0e2f49d55710dba656b0c958891d
I finally figured it out using Selenium. Here is the code:
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium import webdriver
LOGIN_URL = 'https://auth.fiintrade.vn/Account/Login?ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3Dd454e517f7ff44709a797670cd357be7%26nonce%3D890d02d7b58f42e88197818b035ad6d9%26clientType%3DWEBCLIENT%26fp%3De292f9093f0b9abc9904b5fba1d948c6%26type%3Dredirect'
def generate_token() -> bool:
service = Service(cfg.CHROME_DRIVER_PATH)
options = Options()
options.headless = True # do not show new window
driver = webdriver.Chrome(service=service, options=options)
driver.get(LOGIN_URL)
username_textbox = driver.find_element(By.ID, "exampleInputEmail1")
username_textbox.send_keys(cfg.USERNAME)
password_textbox = driver.find_element(By.ID, "exampleInputPassword1")
password_textbox.send_keys(cfg.PASSWORD)
login_button = driver.find_element(By.NAME, "button")
login_button.click()
assert 'access_token=' in driver.current_url, 'Failed to get access token'
resp = driver.current_url
# get token from resp
resp = resp.split('access_token=')[1].split('&token_type=')
id_token = resp[0]
token_type = resp[1].split('&expires_in=')[0]
auth_token = token_type + ' ' + id_token
with open(cfg.AUTH_TOKEN_SAVE_PATH, 'w') as f:
f.write(auth_token)
return 0

Reading API - code from Curl to R

I am trying to read a json from an authenticated API using R, but not sucessfully.
I have the Curl code and tried to convert it to R using "curlconverter" library and also tried to get it using "httr" library.
curl -X GET \
'https://api.cartolafc.globo.com/auth/liga/gurudocartola-com?orderBy=campeonato&page=1' \
-H 'Cache-Control: no-cache' \
-H 'x-glb-token: mytoken'
I would appreciate a solution to write this code in R.
library(curlconverter) # devtools::install_github("hrbrmstr/curlconverter")
u <- "curl -X GET 'https://api.cartolafc.globo.com/auth/liga/gurudocartola-com?orderBy=campeonato&page=1' -H 'Cache-Control: no-cache' -H 'x-glb-token: mytoken'"
straighten(u) %>%
make_req()
That makes:
httr::VERB(verb = "GET", url = "https://api.cartolafc.globo.com/auth/liga/gurudocartola-com?orderBy=campeonato&page=1",
httr::add_headers(`Cache-Control` = "no-cache",
`x-glb-token` = "mytoken"))
which very straightforwardly (if one has done any research before posting a question) translates to:
httr::GET(
url = "https://api.cartolafc.globo.com/auth/liga/gurudocartola-com",
httr::add_headers(
`Cache-Control` = "no-cache",
`x-glb-token` = "mytoken"
),
query = list(
`orderBy` = "campeonato",
`page` = 1L
)
)
The back-ticks are there solely to remind me they are parameters (and, they sometimes contain dashes or other chars which force a back-tick quote).

Rcurl with http data post

I would like to move the following curl call to Rcurl:
curl 'http://myserver.org/stream'
-H 'Authorization: Basic XXXXXXXX' -H 'Connection: keep-alive' --data-binary '{"limit": 20}'
-H 'Content-Type: application/json;charset=UTF-8'
This is one of my R tests:
library(RCurl)
url.opts <- list(httpheader = list(Authorization ="Basic XXXXXXXX",
Connection = "keep-alive",
"Content-Type" = "application/json;charset=UTF-8"))
getURLContent('http://myserver.org/stream', .opts=url.opts)
Now I am missing an attribute to add --data or --data-binary. How to add this option?
Thanks a lot
Markus
Thanks a lot #hadley.
This is my working solution:
library(httr)
user <- "test"
password <- "test123"
POST(url = 'http://myserver.org/stream',
config = c(authenticate(user, password, "basic"),
add_headers(Connection = "keep-alive"),
accept_json()),
body = "{'username':'test'}")

Resources