HTML DOM not working with JavaScript page - web-scraping

Question

Getting HTML DOM from websites that work with client-side (JavaScript)

Answer 1

It doesn't work because
<div class="currTableContainer" id="divCurrTableContainer">
is empty. There is nothing in it. You wont find a table in it. Hence the nullpointer. The HTML you're looking for is generated by a script(ajax).
Check the loaded HTML to confirm.
Because I was bored I decided to finish your homework. You can't use HAP in this case. You have to use the ajax service they provided. Also better since the data is prone to change a lot. The code that is used on the site(ajax script) looks like this:
$("document").ready(function () {
$.ajax({
type: "POST",
url: '/_layouts/15/LINKDev.CIB.CurrenciesFunds/FundsCurrencies.aspx/GetCurrencies',
async: true,
data: "{'lang':'" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnLang").value + "'}",
contentType: "application/json; charset=utf-8",
dataType: "json",
success: function (msg) {
if (msg.d != null && msg.d.length > 0) {
var contentHTML = "<table class='currTable' cellspacing='0' rules='all' style='border-width:0px;border-collapse:collapse;'>"
+ "<tbody><tr class='currHeaderRow'>"
+ "<th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnCurrency").value + "</th><th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnBuy").value + "</th><th scope='col'>" + document.getElementById("ctl00_ctl48_g_5d7fc52f_a66d_4aa2_8d6c_c01fb4b38cb2_hdnSell").value + "</th>"
+ "</tr>";
for (var i = 0; i < msg.d.length; i++) {
if (msg.d[i].CurrencyID.length > 0) {
contentHTML += "<tr class='currRow'>"
+ "<td>" + msg.d[i].CurrencyID + "</td><td>" + msg.d[i].BuyRate + "</td><td class='lastCell'>" + msg.d[i].SellRate + "</td>"
+ "</tr>";
}
}
contentHTML += "</tbody></table>";
$("#divCurrTableContainer").html(contentHTML);
if ($(".bannerElements").length > 0)
FixCurrenciesRatesScroll();
}
},
error: function (msg) {
}
});
});
As you can see the script uses an URL from a different part of the site to update their currency. In order to fetch it you just make http request with the following JSON {"lang":"en"}. I have translated the script into it's equivalent in C# below. The response from the http request will be JSON formatted. Which means you will have to create a class that can be used to serialize the data. I recommend you look at Newtonsoft for this since I can't do all your homework.
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.Net;
namespace test
{
class Program
{
public static void Main(string[] args)
{
try
{
string webAddr = "http://www.cibeg.com/_layouts/15/LINKDev.CIB.CurrenciesFunds/FundsCurrencies.aspx/GetCurrencies";
var httpWebRequest = (HttpWebRequest)WebRequest.Create(webAddr);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
httpWebRequest.Expect = "application/json";
string datastring = "{\"lang\":\"en\"}";
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] data = encoder.GetBytes(datastring);
httpWebRequest.ContentLength = data.Length;
httpWebRequest.GetRequestStream().Write(data, 0, data.Length);
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var responseText = streamReader.ReadToEnd();
Console.WriteLine(responseText);
//Now you have your response.
//or false depending on information in the response
}
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
Console.ReadKey();
}