Verifying Bs4 Parsing Output from a Website - web-scraping

Question

I was trying to scrape this site when I was running into errors due to tags that I thought existed, but did not exist in the scraped html from Bs4.
Site: https://en.thejypshop.com/category/cdlp/59/
I manually verified that the parsed output from Bs4 was giving me a completely different view of the html than when I inspected the site itself; here is a comparison of the two (copied relevant html in the two pastebin links). I also tried scraping with different parsing options such as 'lxml', 'html.parser', etc. but to no avail.
(Bs4 Output): https://pastebin.com/tg4P5DFh
<div class="thumbnail">
<div class="prdImg">
<a href="/product/stray-kids-mini-album-maxident-case-ver/842/category/59/display/2/" name="anchorBoxName_842">
<img alt="" id="eListPrdImage842_2" src="https://cafe24img.poxo.com/jyp3602022/web/product/medium/202210/ca01b08c39232296f482b657be53aa4b.jpg" />
</a>
<span class="wish">
<img alt="Before add to wish list" categoryno="59" class="icon_img ec-product-listwishicon" icon_status="off" individual-set="F" login_status="F" productno="842" src="/web/upload/icon_202204271744355800.png" />
</span>
</div>
<div class="icon">
<div class="promotion"></div>
<div class="button">
<div class="option"></div>
<img alt="Add to cart" class="ec-admin-icon cart" onclick="category_add_basket('842','59', '2', 'A0000', false, '1', 'P0000BGK', 'B', 'T', '20');" src="/web/upload/icon_202204271744303700.png" />
<img alt="View larger image" onclick="zoom('842', '59', '2','', '');" src="//img.echosting.cafe24.com/design/skin/admin/en_US/btn_prd_zoom.gif" style="cursor:pointer" />
</div>
</div>
</div>
(html from Site): https://pastebin.com/2xfi4XTA
<div class="thumbnail">
<div class="prdImg">
<a href="/product/stray-kids-mini-album-maxident-case-ver/842/category/59/display/1/">
<img src="https://cafe24img.poxo.com/jyp3602022/web/product/medium/202210/ca01b08c39232296f482b657be53aa4b.jpg" id="eListPrdImage842_1" alt="">
</a>
</div>
<span class="pro_icon">
<img src="/web/upload/icon_202204271744355800.png" class="icon_img ec-product-listwishicon" alt="Before add to wish list" productno="842" categoryno="59" icon_status="off" login_status="F" individual-set="F">
<img src="/web/upload/icon_202204271744303700.png" onclick="category_add_basket('842','59', '1', 'A0000', false, '1', 'P0000BGK', 'B', 'T', '20');" alt="Add to cart" class="ec-admin-icon cart">
</span>
<span class="soldout_icon"></span>
</div>
Note that the <span class="soldout_icon"></span> tag does not appear in what Bs4 sees, among other things.
My guess as to why this is the case;
I am not using a headless browser, so some websites such as this one might not display the same thing.
There is some JS running in the background that Bs4 does not pick up on
Please let me know if any of my guesses are incorrect and what is actually going on!

Answer 1

Yes, you are right as
the second page is beeing built dynamicaly so you can't get the real html with bs4. Try to use combination of selenium and bs4 to get what you need. Here is a small script that finds some hidden divs and print them out. You should get deeper insight and simulate web surfing to catch the html when the page is fully developed. This one below is still in the process of construction.
import time
from bs4 import BeautifulSoup
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome(options = options)
urls = ['https://en.thejypshop.com/category/cdlp/59/', 'https://pastebin.com/2xfi4XTA']
for url in urls:
data = driver.get(url)
time.sleep(1)
pg_html = driver.page_source
pg_html = pg_html.replace('<', '<').replace('>', '>')
soup = BeautifulSoup(pg_html, 'html.parser')
dv = soup.find_all('div', attrs={'class': 'thumbnail'})
dv1 = soup.find_all('span', attrs={'class': 'soldout_icon'})
try:
print(60 * '-')
print(dv[0])
except:
pass
print(60 * '-')
try:
print(dv1[0])
print(60 * '-')
except:
pass
''' R e s u l t :
------------------------------------------------------------
<div class="thumbnail">
<div class="prdImg">
<img alt="" id="eListPrdImage842_2" src="https://cafe24img.poxo.com/jyp3602022/web/product/medium/202210/ca01b08c39232296f482b657be53aa4b.jpg"/>
<span class="wish"><img alt="Before add to wish list" categoryno="59" class="icon_img ec-product-listwishicon" icon_status="off" individual-set="F" login_status="F" productno="842" src="/web/upload/icon_202204271744355800.png"/></span>
</div>
<div class="icon">
<div class="promotion"> </div>
<div class="button">
<div class="option"></div> <img alt="Add to cart" class="ec-admin-icon cart" onclick="category_add_basket('842','59', '2', 'A0000', false, '1', 'P0000BGK', 'B', 'T', '20');" src="/web/upload/icon_202204271744303700.png"/> <img alt="View larger image" onclick="zoom('842', '59', '2','', '');" src="//img.echosting.cafe24.com/design/skin/admin/en_US/btn_prd_zoom.gif" style="cursor:pointer"/> </div>
</div>
</div>
------------------------------------------------------------
<span class="soldout_icon"></span>
------------------------------------------------------------
------------------------------------------------------------
<div class="thumbnail">
</div>
------------------------------------------------------------
<span class="soldout_icon"></span>
------------------------------------------------------------
'''
Regards...