I installed some chrome extension that pop ups a modal box when I'm on a certain domain.
If I click the button in that box, I see in the Network tab of chrome developer tools, that the extension makes an HTTP Post request to the website.
The request contains some request cookies from the domain: A,B,C,D.
And response cookies from the domain: A,B,C - without D.
When the request is done (and the extension finished doing its "magic"), I discovered that the value of cookie D has changed, even though D was not in the response cookies. I tested it several times.
How is this possible? Can the extension make something in the background that is hidden from the network tab, that will cause the cookie D from the domain to change?
I want to be able to capture and document this Cookie D generation behavior, and don't know how to do that.
Using the chrome.cookies API, a Chrome extension can manipulate the cookies that are stored in the browser without the need to perform an HTTP request. The extension will need the cookies permission to access this API.
You will not be able to capture, or intercept, the extension's calls to the chrome.cookies API.
In addition, through the chrome.webRequest API, a chrome extension can modify the request headers, including cookies, which are sent or received without directly changing the cookies which are stored in the browser. The extension will need the webRequest and webRequestBlocking permissions to make such changes.
Related
I am trying to build a web scraper for the following site:
https://webdiplomacy.net/gamelistings.php
When I visit there on my browser, my initial request includes a Cookie in the headers.
Cookie: __utma=56936876.27553852.1525640664.1525640664.1525640664.1; __utmc=56936876; __utmz=56936876.1525640664.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt=1; __utmb=56936876.4.10.1525640664
How does my browser generate this cookie, and how does it know to include a cookie - given that I am visiting this page in an incognito window without having visited another page?
There is no set-cookie header in the response.
gamelistings.php is the first file accessed in the request. How does my browser know to include specific data when it is first accessing the site?
Surely the procedure for generating a cookie must be contained in the website, but this cannot be the case since my browser is generating a cookie before it ever actually receives any data (since this is the first request).
How could I possibly generate such a cookie with a web scraper?
They are Google Analytics cookie. Are you sure you have not visited another site, not even google search? I can't reproduce your problem on ubuntu, firefox.
When developing endpoints I need to follow a rather complex/cumbersome login process to authenticate with our gatekeeper.
For this I open the developer login page in Paw. It opens inside paw and renders the content properly. If I then click the button to actually login and an external browser is opened causing Paw to loose track of the proper cookies.
I wonder if it would be possible to keep the request inside Paws embedded browser so the cookies are not lost and I can continue testing my endpoints.
To be able to login using cookie-based auth, in Paw 2.1, you can use this hack: go to your web browser with developer mode enabled and login on the website. Then, in the Inspector / Debugger, go to Cookies (in Chrome/Safari: inspector is Cmd+Option+I, then tab "Resources" then "Cookies"). Copy the session/login cookie. Go back to Paw and add a "Cookie: mySessionCookie=value" header. It should do the trick.
Note: that's a hacky method. As mentioned in my earlier comment, a nicer way is to come.
I want to automate logging into a website and retrieving certain data.
I thought the way to do this would be to sniff the HTTP requests so I know where the login form is being POSTed to so I can do the same using NodeJS/Java/Python.
However I can't seem to find the HTTP request that handles it.
The site seems to use some Java-applet and a lot of Javascript.
This is the site: link
Should I have a different approach?
Also also wonder about storing a cookie session, and sending it with each HTTP request after logging in.
I'm sorry if I am not to clear, I will try to explain myself further and edit this post if needed.
You can use the developer console (hit F12) in Chrome (this works also in other browsers) and then click the "Network" tab. There you see all network calls.
To detect what http requests are performed from a mobile device, you can use a proxy like Charles Proxy.
Also be aware that if you post from nodejs the cookies won't be set in the users browser.
I was using Fiddler see on-the-field how web sites use cookies in their login systems. Although I have some HTTP knowledge, I'm just just learning about cookies and how they are used within sites.
Initially I assumed that when submitting the form I'd see no cookies sent, and that the response would contain some cookie info that would then be saved by the browser.
In fact, just the opposite seems to be the case. It is the request that's sending in info, and the server returns nothing.
When fiddling about the issue, I noticed that even with a browser cleaned of cookies, the client seems to always be sending a RequestVerificationToken to the server, even when just looking around withot being signed in.
Why is this so?
Thanks
Cookies are set by the server with the Set-Cookie HTTP response header, and they can also be set through JavaScript.
A cookie has a path. If the path of a cookie matches the path of the document that is being requested, then the browser will include all such cookies in the Cookie HTTP request header.
You must make sure to be careful when setting or modifying cookies in order to avoid XSS attacks against your users. As such, it might be useful to include a hidden and unique secret within your login forms, and use such secret prior to setting any cookies. Alternatively, you can simply check that HTTP Referer header matches your site. Otherwise, a malicious site can copy your form fields, and create a login form to your site on their site, and do form.submit(), effectively logging out your user, or performing a brute-force attack on your site through unsuspecting users that happen to be visiting the malicious web-site.
The RequestVerificationToken that you mention has nothing to do with HTTP Cookies, it sounds like an implementation detail that some sites written in some specific site-scripting language use to protect their cookie-setting-pages against XSS attacks.
When you hit a page on a website, usually the response(the page that you landed on) contains instructions from the server in the http response to set some cookies.
Websites may use these to track information about your behavior or save your preferences for future or short term.
Website may do so on your first visit to any page or on you visit to a particular page.
The browser would then send all cookies that have been set with subsequent request to that domain.
Think about it, HTTP is stateless. You landed on Home Page and clicked set by background to blue. Then you went to a gallery page. The next request goes to your server but the server does not have any idea about your background color preference.
Now if the request contained a cookie telling the server about your preference, the website would serve you your right preference.
Now this is one way. Another way is a session. Think of cookies as information stored on client side. But what if server needs to store some temporary info about you on server side. Info that is maybe too sensitive to be exposed in cookies, which are local and easily intercepted.
Now you would ask, but HTTP is stateless. Correct. But Server could keep info about you in a map, whose is the session id. this session id is set on the client side as a cookie or resent with every request in parameters. Now server is only getting the key but can lookup information about you, like whether you are logged in successfully, what is your role in the system etc.
Wow, that a lot of text, but I hope it helped. If not feel free to ask more.
I have a Drupal site with user logins. Embedded within this site is a Flash application that shows some data to everybody but allows extra functionality if the user is logged on. From within Flash, I'm using XMLRPC to access the system.check method (to determine whether the user is logged in) and the user.login method (to log in a user from within Flash).
Within Flash, everything works fine. However the Flash login state does not seem to correspond to the rest of the site. For example, if I invoke user.login via XMLRPC, subsequent calls to system.check show that I am logged in, but the Drupal user page still says I'm logged out. Or if I am logged in both places and then I log out via the Drupal user page, the next call to system.check still indicates that I am logged in.
How can I make the login state consistent between the Drupal GUI and my embedded Flash app?
(Note: I am not using any XMLRPC library, I am just constructing the requisite XML manually and sending POST methods using a URLRequest object.)
EDIT: I have confirmed via this question and also via testing with a Web Proxy that the SESS cookie returned by the user login page is being picked up and sent back by the flash application.
EDIT: And now I have shown experimentally that even though Flash (via Safari) is sending the same cookie, it gets back a DIFFERENT cookie when it connects to the XMLRPC service than when it requests and HTML page. In other words, Drupal just doesn't support this kind of synchronization and I'm stuck. I'm accepting the answer below that put me on the right track.
EDIT: AMHPHP is not fully released for Drupal6 as of this writing, but it turned out to be installed on the site anyway. Using the DrupalSite library, I was very easily able to log into and out of the site from flash, and the login remained consistent between flash and HTML.
I'm not terribly familar with Flash, but do the URLRequest objects "inherit" state from the browser session, including the user login cookies? If not, you'll need to explicitly send the login cookie with your hand-built request or Drupal will think that it's just coming from another web browser at the same IP address.
If you're not quite sure, using the Firebug plugin might be useful. It lets you inspect any requests that are being piped through the browser, examine their headers, and look at the raw HTTP response object that comes back.
Update: Even more important than the flash widget getting a session cookie is the flash widget getting the SAME session cookie as the web browser itself. Drupal allows users to log in from multiple machines simultaneously, so if the browser is creating one session and the flash widget is creating another, you'd see the behavior you're describing...
It can't be done.
(For details, please see my final edit to the original question and the equivalent information in my comment to #Eaton.)