I am trying to retrieve data from an API with stock data from this site: nasdaqomxnordic.com.
So far, I have succesfully constructed a curl call by inspecting the page in Google Chrome. The curl call works great when I send it from reqbin.com and I receive the expected data.
However, when I try to translate it into an httr request in R, I can't make it work. It seems that httr connects succesfully to the server, but nothing happens after that - the command just keeps running forever in r, until I cancel it manually. I suspect that the issue lies in how the body of the request (the xmlquery) is sent so I have tried various formats and tried curlconverter.com, but with no luck (see code below).
Do you guys have any idea about what could be wrong?
The curl code looks like this (this works perfectly on reqbin.com):
curl 'https://www.nasdaqomxnordic.com/webproxy/DataFeedProxy.aspx' \
-H 'authority: www.nasdaqomxnordic.com' \
-H 'accept: */*' \
-H 'accept-language: se-SE,se;q=0.9,en-US;q=0.8,en;q=0.7' \
-H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' \
-H 'origin: https://www.nasdaqomxnordic.com' \
-H 'referer: https://www.nasdaqomxnordic.com/aktier/microsite?Instrument=CSE1158&name=Novo%20Nordisk%20B&ISIN=DK0060534915' \
-H 'sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "Windows"' \
-H 'sec-fetch-dest: empty' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-site: same-origin' \
-H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' \
-H 'x-requested-with: XMLHttpRequest' \
--data-raw 'xmlquery=%3Cpost%3E%0A%3Cparam+name%3D%22Exchange%22+value%3D%22NMF%22%2F%3E%0A%3Cparam+name%3D%22SubSystem%22+value%3D%22History%22%2F%3E%0A%3Cparam+name%3D%22Action%22+value%3D%22GetDataSeries%22%2F%3E%0A%3Cparam+name%3D%22AppendIntraDay%22+value%3D%22no%22%2F%3E%0A%3Cparam+name%3D%22Instrument%22+value%3D%22CSE1158%22%2F%3E%0A%3Cparam+name%3D%22FromDate%22+value%3D%222021-12-25%22%2F%3E%0A%3Cparam+name%3D%22ToDate%22+value%3D%222022-12-25%22%2F%3E%0A%3Cparam+name%3D%22hi__a%22+value%3D%220%2C5%2C6%2C3%2C1%2C2%2C4%2C21%2C8%2C10%2C12%2C9%2C11%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt%22+value%3D%22%2FnordicV3%2Fhi_csv.xsl%22%2F%3E%0A%3Cparam+name%3D%22OmitNoTrade%22+value%3D%22true%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_lang%22+value%3D%22en%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_options%22+value%3D%22%2Cadjusted%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttype%22+value%3D%22application%2Fms-excel%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttypefilename%22+value%3D%22NOVO_B-2021-12-25-2022-12-25.csv%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_hiddenattrs%22+value%3D%22%2Civ%2Cip%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_tableId%22+value%3D%22historicalTable%22%2F%3E%0A%3Cparam+name%3D%22DefaultDecimals%22+value%3D%22false%22%2F%3E%0A%3Cparam+name%3D%22app%22+value%3D%22%2Faktier%2Fmicrosite%22%2F%3E%0A%3C%2Fpost%3E' \
--compressed
My best try at an httr equivalent look like this:
require(httr)
headers = c(
`authority` = 'www.nasdaqomxnordic.com',
`accept` = '*/*',
`accept-language` = 'se-SE,se;q=0.9,en-US;q=0.8,en;q=0.7',
`content-type` = 'application/x-www-form-urlencoded; charset=UTF-8',
`origin` = 'https://www.nasdaqomxnordic.com',
`referer` = 'https://www.nasdaqomxnordic.com/aktier/microsite?Instrument=CSE1158&name=Novo%20Nordisk%20B&ISIN=DK0060534915',
`sec-ch-ua` = '"Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"',
`sec-ch-ua-mobile` = '?0',
`sec-ch-ua-platform` = '"Windows"',
`sec-fetch-dest` = 'empty',
`sec-fetch-mode` = 'cors',
`sec-fetch-site` = 'same-origin',
`user-agent` = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
`x-requested-with` = 'XMLHttpRequest'
)
data = 'xmlquery=%3Cpost%3E%0A%3Cparam+name%3D%22Exchange%22+value%3D%22NMF%22%2F%3E%0A%3Cparam+name%3D%22SubSystem%22+value%3D%22History%22%2F%3E%0A%3Cparam+name%3D%22Action%22+value%3D%22GetDataSeries%22%2F%3E%0A%3Cparam+name%3D%22AppendIntraDay%22+value%3D%22no%22%2F%3E%0A%3Cparam+name%3D%22Instrument%22+value%3D%22CSE1158%22%2F%3E%0A%3Cparam+name%3D%22FromDate%22+value%3D%222021-12-25%22%2F%3E%0A%3Cparam+name%3D%22ToDate%22+value%3D%222022-12-25%22%2F%3E%0A%3Cparam+name%3D%22hi__a%22+value%3D%220%2C5%2C6%2C3%2C1%2C2%2C4%2C21%2C8%2C10%2C12%2C9%2C11%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt%22+value%3D%22%2FnordicV3%2Fhi_csv.xsl%22%2F%3E%0A%3Cparam+name%3D%22OmitNoTrade%22+value%3D%22true%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_lang%22+value%3D%22en%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_options%22+value%3D%22%2Cadjusted%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttype%22+value%3D%22application%2Fms-excel%22%2F%3E%0A%3Cparam+name%3D%22ext_contenttypefilename%22+value%3D%22NOVO_B-2021-12-25-2022-12-25.csv%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_hiddenattrs%22+value%3D%22%2Civ%2Cip%2C%22%2F%3E%0A%3Cparam+name%3D%22ext_xslt_tableId%22+value%3D%22historicalTable%22%2F%3E%0A%3Cparam+name%3D%22DefaultDecimals%22+value%3D%22false%22%2F%3E%0A%3Cparam+name%3D%22app%22+value%3D%22%2Faktier%2Fmicrosite%22%2F%3E%0A%3C%2Fpost%3E'
http_response <- httr::POST(url = 'https://www.nasdaqomxnordic.com/webproxy/DataFeedProxy.aspx', httr::add_headers(.headers=headers), body = data, verbose())
I have also tried this alternate formatting of the data-part of the request, but with no better luck:
data = list(
`xmlquery` = '<post>\n<param name="Exchange" value="NMF"/>\n<param name="SubSystem" value="History"/>\n<param name="Action" value="GetDataSeries"/>\n<param name="AppendIntraDay" value="no"/>\n<param name="Instrument" value="CSE1158"/>\n<param name="FromDate" value="2021-12-25"/>\n<param name="ToDate" value="2022-12-25"/>\n<param name="hi__a" value="0,5,6,3,1,2,4,21,8,10,12,9,11"/>\n<param name="ext_xslt" value="/nordicV3/hi_csv.xsl"/>\n<param name="OmitNoTrade" value="true"/>\n<param name="ext_xslt_lang" value="en"/>\n<param name="ext_xslt_options" value=",adjusted,"/>\n<param name="ext_contenttype" value="application/ms-excel"/>\n<param name="ext_contenttypefilename" value="NOVO_B-2021-12-25-2022-12-25.csv"/>\n<param name="ext_xslt_hiddenattrs" value=",iv,ip,"/>\n<param name="ext_xslt_tableId" value="historicalTable"/>\n<param name="DefaultDecimals" value="false"/>\n<param name="app" value="/aktier/microsite"/>\n</post>'
)
```
Related
I'm testing a page request in postman(desktop app), but it redirect all requests to captcha page;
While i replay in browser, it works fine even i clean all cookies && local storage, there's no redirections, simply response with 200 code status;
Then I copy all browser's header or cookie(another try), nothing worked!
I have search some Q&A, including ssl verify close in postman, or login in user cache ,not my case at all.
This is a link of my case
Is there anything i omitted? Please help me to know
I can just put a postman request export as bash curl,sorry:
curl --location --request GET 'https://www.fwrd.com/product-burberry-lola-backpack/BURF-WY121/?d=Womens&itrownum=16&itcurrpage=2&itview=05' \
--header 'authority: www.fwrd.com' \
--header 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
--header 'accept-language: zh-CN,zh;q=0.9' \
--header 'cache-control: max-age=0' \
--header 'cookie: viewNumR1=100; isPopupEnabledR1=true; pocketViewR1=front; currency=USD; currencyOverride=USD; _fbp=fb.1.1658202232315.1459169565; countryCodePref=US; userLanguagePref=en; _gid=GA1.2.1936343211.1658202234; _pxvid=0273b224-0715-11ed-9b1c-4d6e63515557; __rtbh.lid=%7B%22eventType%22%3A%22lid%22%2C%22id%22%3A%22Bpefyig8l0wxukHxQ1QF%22%7D; requestBrowserIdMapping=1; _px_f394gi7Fvmc43dfg_user_id=MDM3YzUxNjEtMDcxNS0xMWVkLWFjNTItMWRiODkxNDdiYmRi; _scid=88365d6e-855a-4b9b-b456-ed90a888f464; rskxRunCookie=0; rCookie=82jkrqh5x5p7yx1jm7r0lyl5rmsq6u; _sctr=1|1658160000000; visitor-cookie30=true; visitor-cookie1=true; _gcl_au=1.1.467830307.1658204129; _pin_unauth=dWlkPU1qbGhNR1pqWXpFdFkyUTNOUzAwWkdJd0xUaGxNMlV0WmpKaE5UQTFZekkwTkdGaw; recentlyViewedItems=0; sortByR2=featuredF; altexp=%7B%22896%22%3A0%2C%221409%22%3A1%2C%221412%22%3A0%2C%221157%22%3A0%2C%221031%22%3A1%2C%221292%22%3A1%2C%221421%22%3A0%2C%221295%22%3A1%2C%221424%22%3A1%2C%221298%22%3A0%2C%221301%22%3A0%2C%221430%22%3A1%2C%221304%22%3A0%2C%221433%22%3A0%2C%221179%22%3A0%2C%221436%22%3A0%2C%22926%22%3A0%2C%221439%22%3A1%2C%221442%22%3A1%2C%221188%22%3A0%2C%22677%22%3A0%2C%221445%22%3A1%2C%22936%22%3A0%2C%221448%22%3A0%2C%221194%22%3A1%2C%221322%22%3A0%2C%221197%22%3A1%2C%221457%22%3A0%2C%22946%22%3A0%2C%221334%22%3A0%2C%22951%22%3A1%2C%221081%22%3A0%2C%221210%22%3A0%2C%221466%22%3A1%2C%221340%22%3A0%2C%221469%22%3A1%2C%221086%22%3A1%2C%221472%22%3A1%2C%221346%22%3A1%2C%221475%22%3A1%2C%22836%22%3A1%2C%221349%22%3A0%2C%221478%22%3A1%2C%221481%22%3A1%2C%221226%22%3A0%2C%221355%22%3A0%2C%221484%22%3A0%2C%221358%22%3A1%2C%221487%22%3A0%2C%221232%22%3A0%2C%221490%22%3A1%2C%221235%22%3A1%2C%221493%22%3A1%2C%221496%22%3A1%2C%221370%22%3A0%2C%221499%22%3A1%2C%221502%22%3A0%2C%221247%22%3A1%2C%221382%22%3A0%2C%221514%22%3A1%2C%22876%22%3A0%2C%221388%22%3A1%2C%221517%22%3A0%2C%221391%22%3A1%2C%221520%22%3A0%2C%221011%22%3A0%2C%221139%22%3A1%2C%221268%22%3A1%2C%221016%22%3A0%2C%221403%22%3A0%2C%221406%22%3A0%7D; pxcts=a80a374b-07d9-11ed-96b1-636858726351; userSeenFwrdNtfDialogDate=2022-07-19; JSESSIONID=BD3FB79F2CE2EEBB32A4565060EBED39.tc-nova_tomcat4; _ga_MRQYJ2518M=GS1.1.1658293460.7.0.1658293460.60; requestSessionID=4254930521; __cf_bm=O0MTs.5O4z3O.Lz7tV4epamTsK4T1BHxmhi10rmax0Y-1658293459-0-AZMBFcXc0ZL7DtIgX2gvkyH7rG54t0KaUItmqPesO/BchuY7GsvNY/f44PNnN+sHQmfUgQma+uQBOFDsidSC6i8=; _ga=GA1.2.1509327558.1658202234; _gat=1; browserID=QhdWmBQ7WgUehyiYRX1EygeH7OOqck; bb_PageURL=%2Ffw%2Fcontent%2Fproduct%2FstyledWithMarkup%3Fcode%3DBURF-WY121%26d%3DF; _uetsid=038da160071511ed89ea4336c7c04aeb; _uetvid=038dbb20071511eda26e25fc715b0a9c; cto_bundle=eFjkml9qcFJnY3E0WlZVVTFzTWNMRXVWOE9nalBTZnNlM0JjbWR3V0xIQXVSc3hOR0FKVUx1UFZ6dFJ1Y0JiRjQxJTJCeVZuUiUyRk4xdUwlMkJrWjBLeklmTzRZN2FodjJFNDZHdXgybVA5UEhJQmJ5RXk4cTRqNVFJc3BVaGlENFRtVEw3U1YlMkJDR1V2WFc4ZDVmc2NGOFhYMmFrUTFoUSUzRCUzRA; lastRskxRun=1658293471138; _px2=eyJ1IjoiNmM0M2EzZTAtMDdlOS0xMWVkLTgzMTctNWZjNjg1MDMxYzMyIiwidiI6IjAyNzNiMjI0LTA3MTUtMTFlZC05YjFjLTRkNmU2MzUxNTU1NyIsInQiOjE2NTgyOTM3ODM4MjcsImgiOiJkODA4NjI1OGRmYWE1ZTE1OTFmYjdlZTIyYWNiYmI4MDg5OGI4NjkwZWNjMzIxZDYxMTI0ZTNiYzFhNWU2MDAwIn0=; __cf_bm=xGlLbY0GGGMACBg4Z8TtXSuxuW4yk9hjpz8e41d2inQ-1658293709-0-AT1Kf+tBGczWEcAu+1rDk5hhnDdbHleJS/slXdDvGm6E4BrVrSFy2/Ezkiw7aSPjpYyLkLzMpUH2YtPpwXPV6Tw=' \
--header 'sec-ch-ua: ".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "Windows"' \
--header 'sec-fetch-dest: document' \
--header 'sec-fetch-mode: navigate' \
--header 'sec-fetch-site: none' \
--header 'sec-fetch-user: ?1' \
--header 'upgrade-insecure-requests: 1' \
--header 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
I am trying to login into this website with below credentials (no worry, it's anonymous):
user = 'MattieBailey#mailinator.com'
password = 'thx9jD3SkrssPKY'
When I logged in manually, I saw in Chrome inspector view there are sequentially a POST request and a GET request.
However, when I tried to simulate these requests with python, I got 400 - Bad request for the POST request. The GET request returned 200 - OK and sent me the content of the login webpage. In the Chrome Inspector View, all these 2 requests returned 302 status code with some response headers contains authorized id_token to use for further requests.
The same result was obtained when I tried to import these two requests to Postman so the problem might not be my code.
My ultimate goal is to get the id_token parameter in the response header of the GET request.
I appreciate your help!
POST request
curl 'https://auth.fiintrade.vn/Account/Login?ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3D19e5bb5a7df14a2bab8ded1f265a7bda%26nonce%3D1547e70f8a2940b89dac75565e032046%26clientType%3DWEBCLIENT%26fp%3D54404ac5041ff5b042c29fb397bef44d%26type%3Dredirect' \
-H 'Connection: keep-alive' \
-H 'Cache-Control: max-age=0' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Origin: null' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' \
-H 'Cookie: _ga=GA1.2...' \
--data-raw 'ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3D19e5bb5a7df14a2bab8ded1f265a7bda%26nonce%3D1547e70f8a2940b89dac75565e032046%26clientType%3DWEBCLIENT%26fp%3D54404ac5041ff5b042c29fb397bef44d%26type%3Dredirect&Username=MattieBailey%40mailinator.com&Password=thx9jD3SkrssPKY&button=login&__RequestVerificationToken=CfDJ8KNrfTjG6sBLmncBE9MMk0dDAX-OFwPXUy1uT0AqrEyhngGIImEM99JGxVKsssw62kRxLY2tHfTqfH8M8Mphqxh80YdwwHM6vpZIb-yVMnma4nS0QCkOf51FMFFu8pPwMM6FY52WEPSWy_ZWuaiAeCU' \
--compressed
GET request
curl 'https://auth.fiintrade.vn/connect/authorize/callback?client_id=StoxPlus.FiinTrade.SPA&redirect_uri=https%3A%2F%2Ffiintrade.vn%2Fsignin-callback&response_type=id_token%20token&scope=openid%20FiinTrade.Market%20FiinTrade.Core%20FiinTrade.Realtime%20FiinTrade.Fundamental&state=19e5bb5a7df14a2bab8ded1f265a7bda&nonce=1547e70f8a2940b89dac75565e032046&clientType=WEBCLIENT&fp=54404ac5041ff5b042c29fb397bef44d&type=redirect' \
-H 'Connection: keep-alive' \
-H 'Cache-Control: max-age=0' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'sec-ch-ua: "Chromium";v="92", " Not A;Brand";v="99", "Google Chrome";v="92"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'Accept-Language: en-US,en;q=0.9,vi;q=0.8' \
-H 'Cookie: _ga=GA1.2...' \
--compressed
Response header that contains id_token
Location: https://fiintrade.vn/signin-callback#id_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IkY4MDA1REIxQkI0MzY4Q0Q3RkJFMUUxRTlEQjkwMTg2MUY5NEFGMTEiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItQUJkc2J0RGFNMV92aDRlbmJrQmhoLVVyeEUifQ.eyJuYmYiOjE2MjkxOTM4MTgsImV4cCI6MTYyOTIyMjYxOCwiaXNzIjoiaHR0cHM6Ly9hdXRoLmZpaW50cmFkZS52biIsImF1ZCI6IlN0b3hQbHVzLkZpaW5UcmFkZS5TUEEiLCJub25jZSI6IjE1NDdlNzBmOGEyOTQwYjg5ZGFjNzU1NjVlMDMyMDQ2IiwiaWF0IjoxNjI5MTkzODE4LCJhdF9oYXNoIjoiOVJNNmdEOWFJZkprdGFCSENHRHpwQSIsInNpZCI6IjQ0MWVkZmM1YTc0MzhhOWNkYzYzYTkwYTM3NmQwOGI1Iiwic3ViIjoiMTcwMzU5IiwiYXV0aF90aW1lIjoxNjI5MTkzODE4LCJpZHAiOiJsb2NhbCIsInVzZXJfaWQiOiIxNzAzNTkiLCJ1c2VyX25hbWUiOiJNYXR0aWVCYWlsZXlAbWFpbGluYXRvci5jb20iLCJuYW1lIjoiIiwiZ2l2ZW5fbmFtZSI6Ik1hdHRpZSIsImZhbWlseV9uYW1lIjoiQmFpbGV5IiwibWlkZGxlX25hbWUiOiIiLCJlbWFpbCI6Ik1hdHRpZUJhaWxleUBtYWlsaW5hdG9yLmNvbSIsInNlcnZpY2VfdHlwZSI6IkZpaW5Hcm91cC5GaWluVHJhZGUiLCJsaXN0X3BhY2thZ2UiOiJGaWluVHJhZGUuVHJpYWwiLCJsaXN0X2ZlYXR1cmUiOiIiLCJsaXN0X2FwaSI6IiIsInJvbGUiOiJDVVNUT01FUiIsImdyb3VwX25hbWUiOiJJbmRpdmlkdWFsIiwic3RhcnRfZGF0ZSI6IjEyLzA4LzIwMjEiLCJlbmRfZGF0ZSI6IjI2LzA4LzIwMjEiLCJoaXRjb3VudF9wZXJtb250aCI6IjAiLCJjb21ncm91cF9saW1pdCI6IiIsInRpY2tlcl9saW1pdCI6IiIsInRpbWVyYW5nZV9saW1pdCI6IjAiLCJkYXRhcmFuZ2VfbGltaXQiOiIwIiwicGVyX21pbnV0ZSI6IjAiLCJwZXJfaG91ciI6IjAiLCJwZXJfZGF5IjoiMCIsInBlcl9tb250aCI6IjAiLCJlbmFibGVkIjoiVHJ1ZSIsImxhc3RfYXR0ZW1wdCI6IjgvMTIvMjAyMSAxMTo1ODoxNyBQTSIsImxhc3RfYXR0ZW1wdF9zdGF0dXMiOiJTVUNDRVNTIiwiZmluZ2VycHJpbnQiOiI1NDQwNGFjNTA0MWZmNWIwNDJjMjlmYjM5N2JlZjQ0ZCIsImNsaWVudHR5cGUiOiJXRUJDTElFTlQiLCJhbXIiOlsicHdkIl19.bObyB2sb-kkAxbxhhLc2hQsEtB6YAvzdtd2OwlLGKrzikqoGPwaQYlA41YK_v9OJLqUPRvKvkSZPBwfbguYXp8KnOAHxyPwM4lrbWc7u_OMIHarANxIsA50ckkj1gEszaGZtN6vtz4QYkxLMdbhHcpTnYd4XidFBVoyMQ3csLFaEifJEVsKfGStqgXW6etIhug5yuPnhjnZbCWdUPh0887auIuB_r7Nc7x6faHLtqr2in8c_TTKoI9YoM6LYBNkB7G7AtYkxlVImbyrdczHqsh6_ajLITuuSKYnwws2STwzPQMrZsIbHo_97ce3AVJcdUzVMXeY343FOYBDZeFteJg&access_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IkY4MDA1REIxQkI0MzY4Q0Q3RkJFMUUxRTlEQjkwMTg2MUY5NEFGMTEiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItQUJkc2J0RGFNMV92aDRlbmJrQmhoLVVyeEUifQ.eyJuYmYiOjE2MjkxOTM4MTgsImV4cCI6MTYyOTIyMjQxOCwiaXNzIjoiaHR0cHM6Ly9hdXRoLmZpaW50cmFkZS52biIsImF1ZCI6WyJodHRwczovL2F1dGguZmlpbnRyYWRlLnZuL3Jlc291cmNlcyIsIkZpaW5UcmFkZS5NYXJrZXQiLCJGaWluVHJhZGUuQ29yZSIsIkZpaW5UcmFkZS5SZWFsdGltZSIsIkZpaW5UcmFkZS5GdW5kYW1lbnRhbCJdLCJjbGllbnRfaWQiOiJTdG94UGx1cy5GaWluVHJhZGUuU1BBIiwic3ViIjoiMTcwMzU5IiwiYXV0aF90aW1lIjoxNjI5MTkzODE4LCJpZHAiOiJsb2NhbCIsInVzZXJfaWQiOiIxNzAzNTkiLCJ1c2VyX25hbWUiOiJNYXR0aWVCYWlsZXlAbWFpbGluYXRvci5jb20iLCJuYW1lIjoiIiwiZ2l2ZW5fbmFtZSI6Ik1hdHRpZSIsImZhbWlseV9uYW1lIjoiQmFpbGV5IiwibWlkZGxlX25hbWUiOiIiLCJlbWFpbCI6Ik1hdHRpZUJhaWxleUBtYWlsaW5hdG9yLmNvbSIsInNlcnZpY2VfdHlwZSI6IkZpaW5Hcm91cC5GaWluVHJhZGUiLCJsaXN0X3BhY2thZ2UiOiJGaWluVHJhZGUuVHJpYWwiLCJsaXN0X2ZlYXR1cmUiOiIiLCJsaXN0X2FwaSI6IiIsInJvbGUiOiJDVVNUT01FUiIsImdyb3VwX25hbWUiOiJJbmRpdmlkdWFsIiwic3RhcnRfZGF0ZSI6IjEyLzA4LzIwMjEiLCJlbmRfZGF0ZSI6IjI2LzA4LzIwMjEiLCJoaXRjb3VudF9wZXJtb250aCI6IjAiLCJjb21ncm91cF9saW1pdCI6IiIsInRpY2tlcl9saW1pdCI6IiIsInRpbWVyYW5nZV9saW1pdCI6IjAiLCJkYXRhcmFuZ2VfbGltaXQiOiIwIiwicGVyX21pbnV0ZSI6IjAiLCJwZXJfaG91ciI6IjAiLCJwZXJfZGF5IjoiMCIsInBlcl9tb250aCI6IjAiLCJlbmFibGVkIjoiVHJ1ZSIsImxhc3RfYXR0ZW1wdCI6IjgvMTIvMjAyMSAxMTo1ODoxNyBQTSIsImxhc3RfYXR0ZW1wdF9zdGF0dXMiOiJTVUNDRVNTIiwiZmluZ2VycHJpbnQiOiI1NDQwNGFjNTA0MWZmNWIwNDJjMjlmYjM5N2JlZjQ0ZCIsImNsaWVudHR5cGUiOiJXRUJDTElFTlQiLCJzY29wZSI6WyJvcGVuaWQiLCJGaWluVHJhZGUuTWFya2V0IiwiRmlpblRyYWRlLkNvcmUiLCJGaWluVHJhZGUuUmVhbHRpbWUiLCJGaWluVHJhZGUuRnVuZGFtZW50YWwiXSwiYW1yIjpbInB3ZCJdfQ.RSMsirz2etpYNVx9Pq1W1qExw1ETwCt5XUxglHwreyQX0WDZV3nUDOf5mxgNKjQx_sPCBQuF6SBH8IdNGcuv9ajvd2XbKWdcsNljXidiRHNGogo-JKIZIqadzLAEAmdXoYP_w2ToqOYkWmtEzm2SMpgbVzklRftciQg4h-K_jYW4YpteX23Az78yT3_RIirfBfszaTp5v3BEOR_StHG16bHPauOlIGmch6o6cWUSX1RPnqyAUVaNGELPygNQcmoYVgyQhwoloXyuDY7LGjhK3EgMm9iCuDx5JasFi5A0PQrAfgs1g7UftC-B_7XvJG4pw4qbevj3PehvZ9Kp1RshXw&token_type=Bearer&expires_in=28600&scope=openid%20FiinTrade.Market%20FiinTrade.Core%20FiinTrade.Realtime%20FiinTrade.Fundamental&state=19e5bb5a7df14a2bab8ded1f265a7bda&session_state=dV3zKwxom21hMpGaoJweGLKUlj2rO2gGSFWbw6bIw4U.ab1d0e2f49d55710dba656b0c958891d
I finally figured it out using Selenium. Here is the code:
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium import webdriver
LOGIN_URL = 'https://auth.fiintrade.vn/Account/Login?ReturnUrl=%2Fconnect%2Fauthorize%2Fcallback%3Fclient_id%3DStoxPlus.FiinTrade.SPA%26redirect_uri%3Dhttps%253A%252F%252Ffiintrade.vn%252Fsignin-callback%26response_type%3Did_token%2520token%26scope%3Dopenid%2520FiinTrade.Market%2520FiinTrade.Core%2520FiinTrade.Realtime%2520FiinTrade.Fundamental%26state%3Dd454e517f7ff44709a797670cd357be7%26nonce%3D890d02d7b58f42e88197818b035ad6d9%26clientType%3DWEBCLIENT%26fp%3De292f9093f0b9abc9904b5fba1d948c6%26type%3Dredirect'
def generate_token() -> bool:
service = Service(cfg.CHROME_DRIVER_PATH)
options = Options()
options.headless = True # do not show new window
driver = webdriver.Chrome(service=service, options=options)
driver.get(LOGIN_URL)
username_textbox = driver.find_element(By.ID, "exampleInputEmail1")
username_textbox.send_keys(cfg.USERNAME)
password_textbox = driver.find_element(By.ID, "exampleInputPassword1")
password_textbox.send_keys(cfg.PASSWORD)
login_button = driver.find_element(By.NAME, "button")
login_button.click()
assert 'access_token=' in driver.current_url, 'Failed to get access token'
resp = driver.current_url
# get token from resp
resp = resp.split('access_token=')[1].split('&token_type=')
id_token = resp[0]
token_type = resp[1].split('&expires_in=')[0]
auth_token = token_type + ' ' + id_token
with open(cfg.AUTH_TOKEN_SAVE_PATH, 'w') as f:
f.write(auth_token)
return 0
I am trying to create a dataframe of combinations of place of residence and tax using https://swisstaxcalculator.estv.admin.ch/#/calculator/income-wealth-tax
I have seen How to automate multiple requests to a web search form using R and Scrape tables by passing multiple search requests using R but both those cases work by building new URLs and I don't think this works for admin.ch (the URL always stays the same).
Using the browser inspector I can extract query from a random calculation and it looks like this:
curl 'https://swisstaxcalculator.estv.admin.ch/delegate/ost-integration/v1/lg-proxy/operation/c3b67379_ESTV/API_calculateSimpleTaxes' \
-H 'Connection: keep-alive' \
-H 'Accept: application/json, text/plain, */*' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36' \
-H 'Content-Type: application/json;charset=UTF-8' \
-H 'Origin: https://swisstaxcalculator.estv.admin.ch' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Referer: https://swisstaxcalculator.estv.admin.ch/' \
-H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7' \
-H 'Cookie: SCDID_S=ji-7X06i9sLI29sYA_JLmUSOd6JpMHsgFG_8ilp1l6EZz8h7y2PMjA$$' \
--data-binary '{"SimKey":null,"TaxYear":2019,"TaxLocationID":100000000,"Relationship":1,"Confession1":5,"Children":[],"Confession2":0,"TaxableIncomeCanton":30000,"TaxableIncomeFed":30000,"TaxableFortune":0}' \
--compressed
It's quite clear I should change the part after --data-binary so I create a vector of taxable incomes and then a vector of --data-binary:
taxable_inc <- seq(30000, 150000, by = 1000)
data <- paste0('{"SimKey":null,"TaxYear":2019,"TaxLocationID":100000000,"Relationship":1,"Confession1":5,"Children":[],"Confession2":0,"TaxableIncomeCanton":',
taxable_inc,
',"TaxableIncomeFed":',
taxable_inc,
',"TaxableFortune":0}')
The next part would be to pass/execute this query and capture the answer but I do not know how to to this. Since the query starts with curl I looked at packages curl and RCurl but I really do not understand the link between handles, urls and queries....
You can use httr to make the call and specify encode = "json" to automatically set the header Content-Type: application/json like in curl and encode the body parameter as a JSON object :
library(httr)
url <- 'https://swisstaxcalculator.estv.admin.ch/delegate/ost-integration/v1/lg-proxy/operation/c3b67379_ESTV/API_calculateSimpleTaxes'
r <- POST(url, body = list(
SimKey = NULL,
TaxYear = 2019,
TaxLocationID = 100000000,
Relationship = 1,
Confession1 = 5,
Children = array(),
Confession2 = 0,
TaxableIncomeCanton = 30000,
TaxableIncomeFed = 30000,
TaxableFortune = 0
), encode = "json", verbose())
data = content(r, "parsed")
print(str(data))
which gives :
{"response":{"IncomeSimpleTaxCanton":1747,"FortuneTaxCanton":0,"IncomeSimpleTaxCity":1747,"IncomeTaxChurch":0,"IncomeTaxCity":1380,"IncomeSimpleTaxFed":119,"PersonalTax":0,"FortuneTaxCity":0,"FortuneSimpleTaxCanton":0,"IncomeTaxFed":119,"FortuneSimpleTaxCity":0,"IncomeTaxCanton":2699,"Location":{"TaxLocationID":100000000,"ZipCode":"1000","BfsID":5586,"CantonID":23,"BfsName":"Lausanne","City":"Lausanne","Canton":"VD"},"FortuneTaxChurch":0}}
I actualy try to replicate a http request with a msgpack body request. I have the original request that works (several times testet, generates a http 200 statuscode)
If i replicate the request for example in insomnia or in PowerShell or with curl i get an http error 500. I tried it with a binary file also as Body.
Any ideas?
Following curl command can be use:
curl --request POST \
--url https://4af7c20b-7646-4fb7-b64f-ae0a8c51c1f1.register.collector.scopely.io/register \
--header 'Accept: application/x-msgpack' \
--header 'Accept-Encoding: gzip' \
--header 'Cache-Control: no-cache' \
--header 'Connection: Keep-Alive' \
--header 'Content-Type: application/x-msgpack' \
--header 'Host: 4af7c20b-7646-4fb7-b64f-ae0a8c51c1f1.register.collector.scopely.io' \
--header 'User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1.1; SM-N950N Build/NMF26X)' \
--header 'X-Unity-Version: 5.6.4p3' \
--header 'cache-control: no-cache' \
--header 'content-length: 672' \
--header 'x-scopely-titan-auth: 4af7c20b-7646-4fb7-b64f-ae0a8c51c1f1' \
--body 3gAQrHN5cy5wbGF0Zm9ybadhbmRyb2lkqXN5cy5zdG9yZaZHb29nbGW3c3lzLmNsaWVudF9nZW5lcmF0ZWRfaWTZIGQ3NDYyNDJhMzk3NjQ0N2E5ZTg3NmRlNmUxNDViZTgzr3N5cy5zZGtfdmVyc2lvbqUzLjAuN6pzeXMuYnVuZGxltGNvbS5zY29wZWx5LnN0YXJ0cmVrr3N5cy5hcHBfdmVyc2lvbqswLjU0My4xMDQ0Na5zeXMub3NfdmVyc2lvbr41LjEuMS9hcGktMjIobm1mMjZ4LzUwMDE5MDMzMSmpc3lzLm1vZGVssHNhbXN1bmcgU00tTjk1ME6qc3lzLmxvY2FsZaVkZV9ERalzeXMuYnVpbGSqUHJvZHVjdGlvbqtzeXMuY2FycmllcrB2b2RhZm9uZSBkMiBnbWJoqHN5cy5pbWVprzM1MTU2NDA4MDIwMjIyOapzeXMuc2VyaWFssDQ5NWE2ZGQzODk4YmM1Zmauc3lzLmFuZHJvaWRfaWSwYmM1ZmY0OTVhNmRkMzg5OK9zeXMubWFjX2FkZHJlc3OxQkM6NUY6RjQ6OTU6QTY6RES5c3lzLmdvb2dsZV9hZHZlcnRpc2luZ19pZNkkZTRiMThhN2EtOGQ2Yi00ZTFiLTgwYWItOWM5MzY4NGVkNTBm
I am trying to translate this code into R programming:
curl -v -X POST -H "X-IBM-Client-Id:YOUR_CLIENT_ID" -H "X-IBM-Client-Secret:YOUR_CLIENT_SECRET" -H "Authorization: Bearer YOUR_ACCESS_TOKEN" -H "Content-Type:application/json" -H "Accept:application/json" https://api.ibm.com/watsonanalytics/run/data/v1/datasets -d '{ "description" :"1234" , "name" :"1234" }'
But when i tried, it gives me an Unauthorized error.
Here it is the code I am using:
library(RCurl)
library(RJSONIO)
httpPOST("https://api.ibm.com/watsonanalytics/run/data/v1/datasets",
content= c("description"="Test1",
"name"="Test1"),
.opts = list(httpheader = c('Content-Type' = 'application/json',
'Accept' = 'application/json',
'X-IBM-Client-Secret' = '123',
'X-IBM-Client-Id' = '123',
'Authorization: Bearer'="")))
I really appreciate your help
Thank you