I am trying to create a GET request using HttpRequester ( addon in firefox ). And I am analyzing packet using the Http Fox ( addon in firefox ).
I have created a GET packet with following parameters
url :-http://enquiry.indianrail.gov.in/ntes
Headers
Host :- enquiry.indianrail.gov.in
Referer :- http://enquiry.indianrail.gov.in/ntes/
When I submit this request. I get a response code of 200. In the HttpFox add on, When I analyze my packet, I see that there is additional field in header named
cookie with value _ga=GA1.3.150104442.1441509203.
Relevant Information
Before sending the request deleted all the cookies for enquiry.indianrail.gov.in .
Running all this behind a proxy server.
I get the respone 200 in HttpRequester, while 302 in HttpFox
I want to know, If I am not attaching cookie in my header,than Why HttpFox shows cookie in the header ( with response code 302 ) ?
The _ga cookie is a google tracking cookie. It is a client cookie created by google analytics.js running in your browser. The analytics.js is included by common.js, which is included in the /ntes home page.
HttpRequester will not execute the javascript logic which creates the client side _ga cookie. It may not automatically load the analytics.js either. If you are trying to automate a page that needs to execute javascript, one simple way is to use a headless browser, such as phantomjs
Related
I have set up a section on my Drupal 7 site to show press releases that are fetched from a remote service through a REST API. It works fine to fetch and show all press items via a very specific POST request.
But I have no luck when I want to fetch just one specific press item, which according to the API docs shall be done with a GET request.
In order to see what happens I want to view the GET request to the remote URL in Chrome, but I can't figure out how.
This is how I do it: when opening the page where I want to show only one full press item:
On my press item page...
https://examplesite.com/newsroom/pressrelease
... I have a custom block where this code is run:
$item_id = (int)$_GET['item_id'];
// Fetch specific pressRelease from remote API with custom token
$headers = array('Custom-Token'=>'[custom-token]', 'Accept'=>'text/json', 'Content-Type'=>'text/json');
$url = "https://[remote newsroom url/]$item_id";
dpm($url);
$options=array(
'method'=>'GET',
'headers'=>$headers
);
$result=drupal_http_request($url, $options);
dpm($result);
$data = json_decode($result->data, true);
dpm($data);
With my dpm statements I can see the request on the page, but I would really like to see the actual request and response in Chrome dev tools (under the Network tab). But there is nothing to be found. I have tried using extensions like Live HTTP Headers, with no luck. Only the request to view the local page is shown not the remote one.
How can I see the http request to the remote URL in Chrome or Firefox?
EDITED... the problem might be that the request is done while the page is generated on the server side, before it is loaded on my site?
The code that you are executing, lies on the server and cannot be viewed from network tools of the browser, The browser developer tools will be able to track only those request that are made directly from your browser, in the form of AJAX request etc.
In order to view the actual request, you need to use a network proxy like Charles or Fiddler on your server which will track all HTTP network traffic from your server.
I'm writing some http client code to interact with a website, and I need to set some cookies. Simply visiting the website sets 4 cookies (as seen in Chrome Settings).
However, when I look at the HTTP response headers for when those cookies were set (using Live HTTP Headers extension), there is no Set-Cookie header anywhere. How were those cookies set? Is there another way than through Set-Cookie?
Edit: Some of the cookies are HttpOnly.
If you load a site in your browser, it might also load other assets that can also set cookies (given that they are on the same domain).
But there is a second way to set cookies: with Javascript via document.cookies.
As far as I know, if your javascript or python code sets a cookie for that domain, then the response will include the SET-COOKIE field. You can view that from at least the inspect console.
So I see that you're using HTTP live extension, but it doesn't look like it shows that field in the response.
I tried looking for other extensions that could show it, but I wasn't able to find one as far as I know. I suppose we both can always fall back to the chrome inspect console. If you go to the network tab, you should actually see the req-resp.
I am trying to work out how cookies are being set on a website, I have scoured the page source and can see how most of them are being generated.
However, there is one cookie that appears on page load that I can't track down.
Is it possible that a cookie is being set when an image is being requested from a remote server? If so, can I inspect that http request response with a tool to find out if it contains the cookie?
Any HTTP-Request can set a cookie, if the server says so.
Cookies are set using the Set-Cookie HTTP header, sent in an HTTP response from the web server.
https://en.wikipedia.org/wiki/HTTP_cookie#Setting_a_cookie
A request for an image is basically the same as a request for a html page. It uses the same request/response structure. So yes you can set a cookie on an image request.
The request/response can be seen in most modern browsers. In FireFox there is under tools -> Web Developer -> Network a tool that shows the requests/responses from all calls being made on a page. Opera and Chrome have similar functionality.
I am trying to simulate a form submission on an ASPX.NET site.
The flow of the website when accessed in a browser is as follows:
1) In a browser the user visits http://mysite.com/ which is configured with Basic Authentication
2) Upon correct credentials, the user is shown a form with one input text box and a button (URL stays http://mysite.com/ but the form being served is Default.aspx)
3)User enters some text and presses submit...
4) The page reloads... URL is still http://mysite.com/... but there is a timer which triggers after 10 secs and downloads a file from http://mysite.com/Downloader
I am trying to simulate this flow in my program using HTTPClient.
1) Do a GET on http://mysite.com
2) Extract hidden form fields __EVENTVALIDATION and __VIEWSTATE
3) Create a POST request with above two and other form fields and POST it to http://mysite.com RESULTS in Invalid Viewstate exception.
How do I achieve this in HTTPClient?
The usual way to do this is as follows: First, record the HTTP traffic using WireShark or Fiddler while you are using the website from the browser. Second, analyze the packet trace in detail, and collect every HTTP header and every HTTP payload from every GET and POST message sent by the browser. Third, try to send the same messages from your code. After sending an HTTP request, you will have to analyze the response of the server, and extract all pieces of data you need to insert into the next request. Don't forget to set the referer field, for example. Add each request to your code one by one, and record the traffic when you run the code. If you assemble your HTTP requests correctly, then your request packets should look like the requests of the browser.
I'm in the same scenario, I have to create a POST request to an external ASPX page.
I have captured the traffic using FIDDLER and tryed to simulare the call using online post request tool like https://www.codepunker.com
I have not been able to recreate the request...
In my opinion (and this require time) we have to:
Create a basic webrequest to the source form
Collect all the form elements with value
Create a POST request submitting all the elements including VIEWSTATE
NOTE: may be that you need to use a webclient that accepts cookies, check:
Accept Cookies in WebClient?
Good luck
I would like to load cookies everytime and everywhere in my website because when my RoR application receives and accepts an "external" HTTP request (ex: REST API), cookies are not loaded (see RFC2109). So their values are inaccessible.
Cookies are accessible only when the HTTP request is made "internally" in my application.
new_cookies = {"Cookie" => "mycookie=1234;myothercookie=4567"}
Net::HTTP.get( URI.parse( http: //app1.website.com/users ), new_cookies)
All browsers will automatically send any cookies you set from your domain, you can check them simply by calling request.cookies from any controller method. It doesn't matter if the request was initiated from within your application (such as a 302 redirect) or not.
I just tried this with Firecookie:
Created a cookie "mycoolcookie" for the domain ".stackoverflow.com"
Went to stackoverflow.com, firebug showed that the cookie was sent in the request header.
Went to meta.stackoverflow.com, firebug showed that the cookie was sent in the request header.
Went to chat.stackoverflow.com, firebug showed that the cookie was sent in the request header.
A cookie is sent automatically by the browser, the server can never request for a cookie to be sent to it.
REST APIs are generally stateless, therefore you should avoid the use of server-side sessions or client-side cookies. If you want to indicate that a user only grabs resources belonging to them, use the Rails nested resources approach, that results in a call like:
http://abc.com/user/user001/books
For all books that belong to user001.
If you are looking to implement security, first you have to use HTTPS instead of HTTP. For the actual implementation you can use Basic Authentication and set the username/password in the request header or you can use something like OAuth which sets up a token for the user that they pass in with each request.