Please note this question is related to Selenium.
Before a HTML form submit i.e., selenium.click("//button[#type='submit']");
I want to inject a name value pair at native level in the HTTP Post back to the Server e.g.
Change HTTP Post from:
POSTDATA=register=true&accountType=customer
To:
POSTDATA=register=true&accountType=customer&mynewfield=true
Working with Selenium commands its not obvious how to intercept and modify what is posted back to the server.
Any ideas of how to achieve desired result in Selenium or something that can be called from Selenium? Kindly appreciated NJ
In theory you could use javascript or jQuery to alter the page. For example, using jQuery you could add a hidden form element with a default value or pre-set value that will then be passed upon form submission. (if i understand your question right - emulating TamperData?)
Related
Is it possible for me to scrape the data from the pop up appears after clicking the link.the website is https://ngodarpan.gov.in/index.php/home/statewise_ngo/61/35
Of course it's possible, it's just a table with pagination.
But you'd better check the legal part before scraping a website, moreover on a governmental one.
Yes, you have to follow exactly what browser does. See network behaviour from your browser.
First, you have to send request to https://ngodarpan.gov.in/index.php/ajaxcontroller/get_csrf in order to get token like this :{"csrf_token":"0d1c59184c7df788dc4b8759f6da40c6"}
After, send another POST request to https://ngodarpan.gov.in/index.php/ajaxcontroller/show_ngo_info. As parameters you have to mention csrf_test_name which which is equals to csrf_token and id which is found from onclick attribute of each link.
You will get JSON as response and just to parse it as you need.
Im running some tests on a app and Ive stumbled upon an odd thing that happends when I sniff the traffic between me and the app server using wireshark:
In a scenario when Im making the post request through the app's html, it looks like that:
But when Im requesting the same thing using the chrome extension "postman", it looks like that:
why are the parameters are now shown at the top of the request? I mean, what changed here?
Im trying to find a clue why is it working in the first option and refuses to work on the second. thats why I need to investigate every little thing..
Edit:
I wrote a short html page to illustrate this and the second option happens here as well:
...
<form action="http://x.x.x.x/page.cgi?id=1726931735&host_name=blah" method="post">
....
postman will send an empty body by default, you need to enable the rows of key-value fields for them to be added to the request. postman does not read the form on the page, its parameters must be inserted into the plugin fields. Otherwise postman will send an empty request just like you see.
You need to enter the fields in the Body tab in postman:
It's very odd. The first one is correct. The second is using get semantics but doing a post with no content. I'd try something else instead of this chrome extension.
The overall goal is to perform a search on the following webpage http://www.cma-cgm.com/eBusiness/Tracking/Default.aspx with a container value of CMAU1173561. I have tried two approaches, the php extension cURL and python's mechanized. The php approached involves a performing a POST submit using the input fields found on the page (NOTE: These are really ugly on the asp.net page). The returned page does not contain any of the search results. The second approaches involves using python's mechanize module. In this approach I load the page, select the form, then change the text field ctl00$ContentPlaceBody$TextSearch to the container value. When I load the response again no search results.
I am at a really dead end. Any help would be appreciate because as it stands my next step is to become a asp.net expertm which i perfer not to.
The source of that page is pretty scary (giant viewstate, tables all over the place, inline CSS, styles that look like they were copied from Word).
Regardless...an ASP.Net form still passes the same raw data to the server as any other form (though it is abstracted to the developer).
It's very possible that you are missing the cookies which go along with the request. If the search page (or any piece of the site) uses session state, the ASP.Net session cookie must be included in the request. You will be able to tell it from its name (contains "asp.net" and "session").
I assume that you have used a tool like Firebug or Chrome to view the complete outgoing request when the page is submitted. From my quick test, it looks like the request may be performed with a GET, not a POST. I submitted a form, looked at the request, and pasted the URL into a new browser window.
Example: http://www.cma-cgm.com/eBusiness/Tracking/Default.aspx?ContNum=CMAU1173561&T=57201202648
This may be all you need to do.
I am working on screen scraping, its easy when filteration in query string, but the problem in AJAX based filteration,
e.g. here is an sample URL
When you open this page, enter hotel name and click Go, Ajax filter work and show the result accordingly or you click on Next Page, it will shown next record using AJAX based.
please suggest me, how to handle these kind of issues when working in Screen Scraping?
Thank alot
You may want to try 2 Firefox add-ons. They are "Firebug" and "Tamper Data".
The "Console" window of Firebug shows the AJAX request and response.
You can then write scripts using the PHP/cURL library to mimic the request.
Do a http request as you normally do for any link or form sumbit but use the url used with ajax. Sometimes you may need to read the javascript source to determine how the url is built.
How do I send an HTTP GET request without a form? This request is performed on a web page.
For more details on what I'm trying to accomplish and why, see my question here: http get/post request and google geolocation api
Any link is a GET request, so, just make a link, add the extra info at the end, done.
http://someurl.net/somelink?value1=avalue&value2=anothervalue
A normal hyperlink does a GET request when the user clicks on it. Loading an image is also a GET request, as are most of the other ways of embedding things in a page. If you're trying to do it via JavaScript, you can use an XMLHttpRequest.
You can use Ajax and jQuery's get method:
$.get("someURL.php?variables=true");
Note, that you will only be able to make requests to the same domain OR the target domain must return JSON formatted results.
In plain HTML you can do that e.g. requesting an image:
<img src="http://example.com/?bla=bla">
< a href="www.google.com">google</a>
I agree with altCognito points out using jquery but I'd rather use
$.get("someurl.php", { variable: value });
I like this way better because it allows me to send objects instead of concatenate strings.
you can see a couple of samples here