below link gives scanned list of stocks based on certain conditions
https://chartink.com/screener/mwdbscans
i can see that below is the url i should use for POST request
https://chartink.com/screener/process
i am facing problem when i want to send a POST request to get the list in python
from the developer tools, i could see the request headers and form data but when i encode the data and send the POST request i get error 500 (internal server error)
the data body of POST request i tried is,
Host: chartink.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0)
Gecko/20100101 Firefox/61.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://chartink.com/screener/nsk-ohl-long
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-CSRF-TOKEN: PsJC0BUBkL7gE8YyglD6uYU8TpjqXLvN3whpdM9v
X-Requested-With: XMLHttpRequest
Content-Length: 179
Cookie: _ga=GA1.2.1391629072.1532845865; XSRF-
Connection: keep-alive
This is because mismatch of send data and the server side deserialize. you should send data from correct form to the server side. then server will be able to deserialise your data correctly.
Related
when investigating network behavior, I usually use postman for sending HTTP requests, however
I need the option to send a raw HTTP request (via clear text), or at least only the headers, and it seems that postman does not support to edit your request via clear HTTP text. (buy the way the opposite is possible, you can read the raw http text of the requests you constructed in postman but you can't edit them)
for example:
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: he-IL,he;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 21114
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: csrftoken=0alLaljTasofjCWZv7gcmukXuz6gMxfzlWpV691hzZZ1hTBcdVJ3mH8ozRDnO6hu; tk_or=%22%22; tk_lr=%22%22; session_id_12211=ff6a58b0baf98005748ce5a3c6a732aef33b750f; splunkweb_csrf_token_12211=10024448868272708216; token_key=10024448868272708216; experience_id=4852e1c6-726b-1ab3-bafa-f0a735d3f708; splunkd_12211=NjcrwAj_TLgz5JalVh2HTynLdbp_CPnfHFKi8qmsODiH40HI2urbPvAvJ9uvDKKoM3nATXEkS6dGytD0TvfiOtAUGJhk7Od25on_gJcZrQwcePQZ8HQaCmGScm^RXmOdDa^KVvN
Host: localhost:12211
Origin: http://localhost:12211
Pragma: no-cache
sec-ch-ua: "Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
X-Requested-With: XMLHttpRequest
X-Splunk-Form-Key: 10024448898272708216
does postman allow editing the raw HTML? if not, there is other tool that can?
What would cause Internet Explorer to replace the HTTP header
Authorization : Bearer <server-provided-token>
with
Authorization : Negotiate <some token>
when making an AJAX request?
In Internet Explorer, some AJAX requests that are configured to contain the header Authorization: Bearer ... are being sent by Internet Explorer with the header Authorization: Negotiate ... instead.
For example, Fiddler shows that the first two of three requests contain the Authorization : Bearer... header, while the third suddenly contains the Authorization : Negotiate... header. The first two requests are successful, and the third fails because the request can't be properly authenticated.
All of the requests are constructed using the same client-side code, and are made one after another (within the span of a second). I have verified that the Authorization header correctly contains the Bearer token in all three cases up until the point the request is provided to the browser.
Also, I'm not seeing the same behavior in Chrome; it's only occurring in IE.
Request 1
GET http://localhost/myapp/api/User HTTP/1.1
Accept: application/json, text/plain, */*
Authorization: Bearer oEXS5IBu9huepzW6jfh-POMA18AUA8yWZsPfBPZuFf_JJxq-DKIt0JDyPXSiGpmV_cpT8FlL3D1DN-Tv5ZbT73MTuBOd5y75-bsx9fZvOeJgg04JcO0cUajdCH2h5QlMP8TNwgTpHg-TR9FxyPk3Kw6bQ6tQCOkOwIG_FmEJpP89yrOsoYJoCfrAoZ7M4PVcik9F9qtPgXmWwXB2eHDtkls44wITF_yM_rPm5C47OPCvMVTPz30KwoEPi6fHUcL3qHauP-v9uypv2e48TyPHUwLYmNFxyafMhBx4TkovnRcsdLHZiHmSjMq0V9a2Vw70
Referer: http://localhost/client/login.html
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: localhost
DNT: 1
Connection: Keep-Alive
Request 2
POST http://localhost/myapp/api/Permissions HTTP/1.1
Referer: http://localhost/client/#/Dashboard
Content-Type: application/json
Authorization: Bearer oEXS5IBu9huepzW6jfh-POMA18AUA8yWZsPfBPZuFf_JJxq-DKIt0JDyPXSiGpmV_cpT8FlL3D1DN-Tv5ZbT73MTuBOd5y75-bsx9fZvOeJgg04JcO0cUajdCH2h5QlMP8TNwgTpHg-TR9FxyPk3Kw6bQ6tQCOkOwIG_FmEJpP89yrOsoYJoCfrAoZ7M4PVcik9F9qtPgXmWwXB2eHDtkls44wITF_yM_rPm5C47OPCvMVTPz30KwoEPi6fHUcL3qHauP-v9uypv2e48TyPHUwLYmNFxyafMhBx4TkovnRcsdLHZiHmSjMq0V9a2Vw70
Accept: application/json, text/plain, */*
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: localhost
Content-Length: 1419
DNT: 1
Connection: Keep-Alive
Pragma: no-cache
<Post Data Removed>
Request 3
GET http://localhost/myapp/api/UserPreferences/Dashboard HTTP/1.1
Referer: http://localhost/client/#/Dashboard
Content-Type: application/json
Authorization: Negotiate YHsGBisGAQUFAqBxMG+gMDAuBgorBgEEAYI3AgIKBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHqI7BDlOVExNU1NQAAEAAACXsgjiBgAGADMAAAALAAsAKAAAAAYBsR0AAAAPVk1ERVZFTlYtU1JTQ0VSSVM=
Accept: application/json, text/plain, */*
Accept-Language: en-US
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Connection: Keep-Alive
DNT: 1
Host: localhost
The requests are being made via the AngularJS $http service, and the back-end is ASP.NET Web API hosted in IIS.
I have tried fixing this issue with
document.execCommand('ClearAuthenticationCache', 'false');"
But this leads to another issue. That is if I log in using form based authentication in my application then on page refresh the IIS picks up the windows authenticated user and changes the current logged in user to window's authenticated user.
Despite what the title may suggest, this is not related to the common "preventing double post request" issue.
In my application, I add some data on outgoing HTTP traffic, and with some some websites, I randomly encounter an HTTP POST request that has a double-header, resulting in a server termination, as I assume the server doesn't understand the request and decides to cut me off. As the title says, I'm literally seeing two POST headers in a single request. This only happens when I append some custom HTTP fields to the header. For example, I came across this today when I followed a surveygizmo.com link, as seen in the trace at the bottom of this post.
I cannot understand if it's the browser that's doing something funky because it noticed I've modified some data, or it's something in my LSP application that causes this to happen.
When I debug my application, I only see the intercepted request the first time, which is when I inject the custom data. After that, I don't see the request anywhere except in Wireshark, so it's not like I can remediate the double headers by deleting the redundant data.
Things to note looking at the trace:
The data I'm appending is 'Custom-FieldN:'
Two almost-identical headers
Three double-CRLF's in one single request header (how is that possible?)
The Request:
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
accept-encoding: gzip,deflate
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
------WebKitFormBoundaryaQraA7ZABICMT6jO
Content-Disposition: form-data; name="sg_navchoice"
I tried to make a GET request for the netflix home page with command prompt because the response returned to me was a 302/301 all the time. So I connected to netflix via the following:
telnet signup.netflix.com 80
Then the request I made was
GET / HTTP/1.1
Host: signup.netflix.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
As copied exactly from LiveHTTPHeaders when I visit netflix however I removed the Cookie part because I don't know where the browser (FireFox) is getting these values
Netflix responds with
Why don't I get a 200 OK status code? Is it because I'm not sending any cookies?
Its doing a redirect to https://signup.netflix.com/?tcw=2. i.e., it wants you to resend the request with the twc=2 variable in the query string and more importantly, with the cookie it just gave you through the Set-Cookie header.
I'm sending an http request to a server that requests an image but using range header with several ranges, the request text is:
GET /images/nav_logo102.png HTTP/1.1
Host: www.google.com.eg
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Cookie: PREF=ID=8aacc11c670a5a37:U=cba349de64cbf880:FF=0:LD=en:TM=1310392358:LM=1327944471:S=zc8_vfTdF5U3C-XN; NID=56=15iHWUMVXBGpdEkEcMCeu32GUL6GlK3aEB5vmocRT4kdILhwGpe9mn6DqDrnZBYWJzA2g4YzfXP8IP3tH7Hw4CzC6FwDrV3uqgv3XhCqmrWI0TBD52Vs3nbdth5YzkOR; SID=DQAAAMUAAAC5feWAhXsCT9NT6ObU32dOLBTU_KtRzxc7Ug6QWIhQW_bpXNvzQtoddtt-a2BbUefv89ZjwNwgrgCiCse3INYkeCyfR7PVPaoDPEmUWQ_0sHXBgbf5U0JBg0XxT8KZMmY9kLOZfEKGg0UGtfNNk8uJKWwPXWlkCYjGalQFyVinUNwiYqayYHGvnA0vtiftBCgqnpHawkyVWoo9hCk3vfDKrRHSdO-GQZoxbL21VlH2dbsJJrnJBaVlR-y8KTg14eLgYpt2swZJxcVZwHThhaF-; HSID=AJLTtk1qkIE6L4aLe; GZ=Z=1
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:5.0) Gecko/20100101 Firefox/5.0
Range: bytes=2048-4095,4096-6143,6144-8191,8192-10239,10240-12287,12288-14335,14336-16383,16384-18431,18432-20479,20480-22527,22528-24575,24576-26623,26624-26722
but the server is responding with
HTTP/1.1 200 OK
and sending the whole image
if i try the same thing but with only one range, it works and i get only the requested range.
any idea how the request should be made?
You should add \r\n\r\n after last header.
REQUEST:
GET /images/djsBox_2.jpg HTTP/1.1\r\n
Range: bytes=0-2,10-20\r\n
Host: www.djsoft.net\r\n
Accept: */*\r\n
\r\n
I checked it with WFetch tool, and the response is HTTP/1.1 206 Partial Content\r\n
And the server actually returned 2 parts of data:
Accept-Ranges: bytes\r\n
Content-Length: 196\r\n
Content-Type: multipart/byteranges; boundary=4b9e42cd78baa9\r\n
\r\n
\r\n
--4b9e42cd78baa9\r\n
Content-type: image/jpeg\r\n
Content-range: bytes 0-2/4543\r\n
\r\n
ัะจั\r\n
--4b9e42cd78baa9\r\n
Content-type: image/jpeg\r\n
Content-range: bytes 10-20/4543\r\n
\r\n
\x000\x001\x001\x001\x000`\x000`\x000\x000ั\r\n
--4b9e42cd78baa9--\r\n