webscraping java by hovering mouse. Dynamic data not displayed after scraping - web-scraping

Question

I want to scrape data from a graph of a particular website.
This information in graph is available only if you hover mouse on the graph.But after I scrape, I am unable to see the data in output even though it is visible under 'Inspect Element'.
I have tried to scrape using JSoup but when I scrape the data, the data that changes by hovering mouse is not displayed.
How can I do this?
Below is the information which I have to scrape. I have to scrape the dynamically changing value '184'.
The value 184 is dynamically changing when you hover mouse on graph wit h RGB values displyaed in the above line. Even these RGB values changes by hovering mouse on graph.
After scraping, the output of document by Jsoup looks like the below:
The number 184 and rgb values are not appeared. How are these fields disappeared in output? Does this not appear because it is a dynamic data by mouse hovering?
I actually have to scrape information from the following graph which displays 'Carbon Intensity' value from the graph "Carbon Intensity in the last 24 hours" only by hovering mouse on it.
I am stuck with this problem since two days and has not found any helpful solution. I am using Jsoup on linux.Could some one suggest me how can I do this.
Thanks in advance!

Answer 1

To do that you should use Selenium and add it to Maven if you are using it, or to whatever dependency manager you are using. Once you do that you need to add this .exe (https://github.com/mozilla/geckodriver/releases) to your project folder to get the Firefox support for Selenium, you can also use Google Chrome following this tutorial (https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver).
You have a lot of tutorials on how to force the JS of a web page to get its content, but it could be something like this, to set the mouse over an item from the HTML:
WebDriver webDriver = new FirefoxDriver();
JavascriptExecutor js = (JavascriptExecutor)webDriver;
webDriver.get(URL); // You have to place the URL you are crawling here
Actions action = new Actions(webDriver);
WebElement webElement = webDriver.findElement(By.id("country-emission-rect));
// using By you have a lot more options to select HTML content, I guess you want to place the mouse over that item in particular, but you can change if it it's another one
action.moveToElement(webElement).perform();
WebDriverWait webDriverWait = new WebDriverWait(webDriver, 15); // wait max 15 seconds
// wait until the element with class name: "country-emission-intensity" is loaded
webDriverWait.until(ExpectedConditions.visibilityOfElementLocated(By.className("country-emission-intensity")));
// get the HTML generate after the mouse over that now has the text you want to get
String fullHtml = webDriver.getPageSource();
webDriver.quit();
If you want to keep using JSOUP instead of Selenium for the scrapping you can now do:
Document document = Jsoup.parse(fullHtml);
Remember to place the .exe in your project folder and to install correctly all the Selenium dependencies (enabling auto-import if you are using Maven).
Hope it helped you! If you need anything else feel free to ask!