Web scraping for hidden content - web-scraping

Question

I am trying to scrape the price data from this website: https://fuelkaki.sg/home
However, the data does not appear to be in the HTML code of the page. Upon inspecting, the data seems to be nested in the tag, for instance under Caltex for the retailer name, and similarly under multiple nested tags for the price data, which I am unable to scrape with the following code (there are no results to be found).
Any help would be much appreciated.
import requests
from bs4 import BeautifulSoup
URL = 'https://fuelkaki.sg/home'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
results = soup.find('div', class_='fuel-name')

Answer 1

The table is behind JS (JavaScript) so BeautifulSoup won't see it.
Here's how I'd do it:
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
url = "https://fuelkaki.sg/home"
driver.get(url)
time.sleep(3)
element = driver.find_element_by_xpath('//*[#class="table"]')
print(element.text)
driver.close()
Output:
Diesel
92
95
98
Others
(V-Power, etc)
Caltex
3‬‬‬‬‬‬0‬‬‬‬‬‬ ‬‬‬‬‬‬S‬‬‬‬‬‬e‬‬‬‬‬‬p‬‬‬‬‬‬t‬‬‬‬‬‬e‬‬‬‬‬‬m‬‬‬‬‬‬b‬‬‬‬‬‬e‬‬‬‬‬‬r‬‬‬‬‬‬ ‬‬‬‬‬‬2‬‬‬‬‬‬0‬‬‬‬‬‬2‬‬‬‬‬‬0‬‬‬‬‬‬,‬‬‬‬‬‬ ‬‬‬‬‬‬0‬‬‬‬‬‬2‬‬‬‬‬‬:‬‬‬‬‬‬0‬‬‬‬‬‬5‬‬‬‬‬‬p‬‬‬‬‬‬m
S$ 1‬‬‬‬.‬‬‬‬7‬‬‬‬3
S$ 2‬‬‬.‬‬‬0‬‬‬2
S$ 2‬‬‬.‬‬‬0‬‬‬6
N.A.
S$ 2‬‬‬‬‬‬.‬‬‬‬‬‬6‬‬‬‬‬‬1
and so on...
EDIT:
If you want the table in a Dataframe try this:
import time
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)
url = "https://fuelkaki.sg/home"
driver.get(url)
time.sleep(3)
html = driver.page_source
soup = BeautifulSoup(html, "html.parser").select_one(".table")
df = pd.read_html(str(soup))
df = pd.concat(df).rename(columns={"Unnamed: 0": ""})
df.to_csv("fuel_data.csv", index=False)
driver.close()
Outputs a .csv file with the table's data: