Select dropdown option and submit form - python-requests

Question

I have written code to get tables from a website, but I need to get access to an additional table by:
selecting a time frame from a dropdown.
submitting by clicking a 'filter' button.
The website is:
http://www.myfxbook.com/forex-market/currencies/nzdchf-historical-data
Inspect element on the dropdown shows this:
<select id="timeScales" class="selectField" onfocus="borderChange(this); "onblur="borderReturn(this);" style="border: 1px solid rgb(226, 225, 225);">
<option id="timeScale1" value="1">1 Minute</option>
<option id="timeScale5" value="5">5 Minutes</option>
<option id="timeScale15" value="15">15 Minutes</option>
<option id="timeScale30" value="30">30 Minutes</option>
<option id="timeScale60" value="60">1 Hour</option>
<option id="timeScale240" value="240">4 Hours</option>
<option id="timeScale1440" value="1440" selected="">1 Day</option>
<option id="timeScale10080" value="10080">1 Week</option>
<option id="timeScale43200" value="43200">1 Month</option>
From the timeframe dropdown, I want to select 4 hours, then click on the filter button.
I know how to get HTML using the requests.get method, but the problem I am having here is that the URL does not change when I select a dropdown value.
I have tried using requests.post, but I clearly don't know how to use it.
I am writing this program using "HOW TO AUTOMATE THE BORING STUFF USING PYTHON" as a tutorial, and the author does not explain how to programmatically click dropdown menus and click buttons.
He does explain the use of Selenium, but I have to repeat this process over 30 times, and having to open a new page every time would take too long.
I'd appreciate it if you could show how this could be achieved using requests module (if possible).

Answer 1

You cannot click a button with requests, what you can do is mimic requests that get the data, when you choose a time period click filter an ajax request is made:
So to mimic that and get the html we need to pass the params below:
import requests
params = {"start": "2016-02-02 00:00",
"end": "2016-06-20 00:00",
"symbol": "NZDCHF",
"timeScale": "240",
"userTimeFormat": "0"}
head = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36",
"X-Requested-With": "XMLHttpRequest"}
url = "http://www.myfxbook.com/getHistoricalDataByDate.json"
js = requests.get(url, params=params, headers=head).json()
Then to get the table, access js["content"]["historyData"]:
print(js["content"]["historyData"])
Which gives you the table:
<table id="symbolMarket" class="sortable maxWidth center dataTable">
<tr style="height:26px">
<th sort="string">
<a class="dotted font11 pointer">Date</a><span id="symbolMarketImg0"></span>
</th>
<th><span class="font11">Open</span></th>
<th><span class="font11">High</span></th>
<th><span class="font11">Low</span></th>
<th width="63"><span class="font11">Close</span></th>
<th sort="float" >
<a class="dotted font11 pointer">Change (Pips)</a><span id="symbolMarketImg5"></span>
</th>
<th sort="float" >
<a class="dotted font11 pointer">Change (%)</a><span id="symbolMarketImg6"></span>
</th>
</tr>
<tr onmouseover="this.className='normalActive';" onmouseout="this.className='';">
<td><span name="timeNZDCHF">Jun 20, 2016 00:00</span></td>
<td><span name="openNZDCHF">0.67771</span></td>
<td><span name="highNZDCHF">0.68133</span></td>
<td><span name="lowNZDCHF">0.67771</span></td>
<td><span name="closeNZDCHF">0.68057</span></td>
<td class="bold"><span name="changePipsNZDCHF"><span class='green'>+28.6</span></span></td>
<td class="bold"><span name="changePercentNZDCHF"><span class='green'>+0.42%</span></span></td>
</tr>
......................................
You could easily parse the original page to get the values from the tag using Beautifulsoup:
In [7]: from bs4 import BeautifulSoup
In [8]: h = """<select id="timeScales" class="selectField" onfocus="borderChange(this); "onblur="borderReturn(this);" style="border: 1px solid rgb(226, 225, 225);">
...: <option id="timeScale1" value="1">1 Minute</option>
...: <option id="timeScale5" value="5">5 Minutes</option>
...: <option id="timeScale15" value="15">15 Minutes</option>
...: <option id="timeScale30" value="30">30 Minutes</option>
...: <option id="timeScale60" value="60">1 Hour</option>
...: <option id="timeScale240" value="240">4 Hours</option>
...: <option id="timeScale1440" value="1440" selected="">1 Day</option>
...: <option id="timeScale10080" value="10080">1 Week</option>
...: <option id="timeScale43200" value="43200">1 Month</option>"""
In [9]: soup = BeautifulSoup(h,"html.parser")
In [10]: durations = ([opt["value"] for opt in soup.select("#timeScales option")])
In [11]: durations
Out[11]: [u'1', u'5', u'15', u'30', u'60', u'240', u'1440', u'10080', u'43200']
But I don't see the logic behind that as the durations are in minutes and you would need to know your duration for the data.