How to follow links if in HTML element in href attribute we have href='#' in scrapy? - web-scraping

Question

I am trying to scrape Niche.com website to extract all schools and details of schools which are present in each school links but if we try to follow the school link in href attribute we have href = "#" so scrapy unable to get inside each school page and collect the data
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class NicheschoolsSpider(scrapy.Spider):
name = 'nicheschools'
allowed_domains = ['www.niche.com']
start_urls = ['https://www.niche.com/k12/search/best-schools/s/wisconsin/']
def parse(self, response):
schoollink = response.xpath("//div[#class='search-result__title-wrapper']/h2")
for school in schoollink:
name= school.xpath(".//text()").get()
link = school.xpath(".//#href").get()
yield {
'name':name,
'link':link
}
yield response.follow(url=link,callback =self.parse_schools)
def parse_schools(self,response):
name = response.xpath("//h1[#class='postcard__title postcard__title--claimed']/text()").get()
website = response.xpath("(//a[#class='profile__website__link']/#href)[1]").get()
address = response.xpath("(//address[#class='profile__address--compact']/text())[1]").get()
yield{
'name':name,
"website":website,
'address':address
}
OUTPUT FOR ONE ENTRY:
2023-01-25 16:33:10 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.niche.com/k12/search/best-schools/s/wisconsin/%5C\>
{'name': 'Brookfield Central High School', 'link': '#'}
when it try to get inside link shown below
2023-01-25 16:33:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.niche.com/k12/search/best-schools/s/wisconsin/%5C\>
{'name': None, 'website': None, 'address': None}
Trying to get inside each school link and collect schoolname, address, telephone, tutuion fees, enrollment for particular link.

Answer 1

Not really a job for Scrapy, although it can certainly be accomplished with Scrapy.
Website is dynamic, pulling data from an API endpoint. I won't be setting up a Scrapy project just to answer your question, but I will demonstrate how you can get the data using Requests and pandas (code is ran in Jupyter notebook):
import requests
import pandas as pd
from tqdm.notebook import tqdm
pd.set_option('display.max_columns', None, 'display.max_colwidth', None)
headers = {
'accept-language': 'en-US,en;q=0.9',
'accept': 'application/json',
'referer': 'https://www.niche.com/k12/search/best-schools/s/wisconsin/',
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}
big_df = pd.DataFrame()
s = requests.Session()
s.headers.update(headers)
for x in tqdm(range(1, 5)):
r = s.get(f'https://www.niche.com/api/renaissance/results/?state=wisconsin&listURL=best-schools&page={x}&searchType=school')
df = pd.json_normalize(r.json(), record_path=['entities'])
big_df = pd.concat([big_df, df], axis=0, ignore_index=True)
display(big_df)
Result in terminal:
100%
4/4 [00:02<00:00, 2.12it/s]
guid ctas badge.display badge.ordinal badge.total badge.vanityURL badge.photoURLs.desktop badge.photoURLs.mobile content.centroid.lat content.centroid.lon content.entity.abbreviation content.entity.alternates.nces content.entity.character content.entity.claimed content.entity.displayable content.entity.genus content.entity.guid content.entity.isClaimed content.entity.isPremium content.entity.location content.entity.name content.entity.parentGUIDs.county content.entity.parentGUIDs.metroArea content.entity.parentGUIDs.state content.entity.parentGUIDs.town content.entity.parentGUIDs.zipCode content.entity.premium content.entity.published content.entity.shortName content.entity.tagline content.entity.type content.entity.url content.entity.variation content.facts content.featuredReview.author content.featuredReview.body content.featuredReview.categories content.featuredReview.created content.featuredReview.guid content.featuredReview.rating content.grades content.photos.default.crops.DesktopHeader content.photos.default.crops.MobileHeader content.photos.default.crops.Original content.photos.default.guid content.photos.default.licenseName content.photos.editorial.crops.Original content.photos.editorial.guid content.photos.editorial.licenseName content.photos.editorial.uploadTimestamp content.photos.mapbox_header.author content.photos.mapbox_header.crops.DesktopHeader content.photos.mapbox_header.crops.MobileHeader content.photos.mapbox_header.guid content.photos.mapbox_header.licenseName content.photos.mapbox_header.licenseUrl content.photos.mapbox_header.sourceUrl content.photos.spotlight.crops.Original content.photos.spotlight.crops.Spotlight content.photos.spotlight.guid content.photos.spotlight.licenseName content.photos.spotlight.uploadTimestamp content.reviewAverage.average content.reviewAverage.count content.virtualTour content.entity.alternates.ceeb content.photos.default.crops.Thumbnail content.photos.default.uploadTimestamp content.entity.parentGUIDs.parent content.entity.parentGUIDs.schoolDistrict content.entity.parentGUIDs.schoolNetwork content.entity.parentGUIDs.neighborhood content.photos.default.crops.Spotlight
0 d6574ad4-6add-45c3-a90a-9d24f58b040e [{'label': 'View Nearby Homes', 'type': 'realE... Best Private High Schools in Wisconsin 1 82 best-private-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.081737 -88.145195 Brookfield Academy 0 Private True True Private School d6574ad4-6add-45c3-a90a-9d24f58b040e True True BROOKFIELD, WI Brookfield Academy ba8709ae-856d-4583-83b7-4484b51ed4c2 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c cc01665b-5240-4885-b13d-a4ae0dd271fc b802227a-e061-45e5-9dfd-6c3ddaf8bebb True True Brookfield Academy [Private School, BROOKFIELD, WI, PK, K-12] School brookfield-academy-brookfield-wi 1041 [{'config': {'format': ['comma'], 'rounding': ... Parent When my kids started school something just did... [Overall Experience] 2022-07-28T18:37:49.017538Z e3bfc3ad-86eb-4bba-8c66-5eb95e4111f7 5.0 [{'description': 'Based on quality of academic... https://d13b2ieg84qqce.cloudfront.net/d1d42e87... https://d13b2ieg84qqce.cloudfront.net/c046f1e3... https://d13b2ieg84qqce.cloudfront.net/d1d42e87... a4125add-a984-4609-a879-ce1afa699db8 UNLICENSED https://d13b2ieg84qqce.cloudfront.net/352e79e1... 53acd2d2-1185-49bd-9928-6a1f1054fba0 UNLICENSED 2022-02-10T21:15:52.569792Z © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... f696705b-0766-48e5-97b5-72370788f0c6 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ https://d13b2ieg84qqce.cloudfront.net/2273b7a3... https://d13b2ieg84qqce.cloudfront.net/d512adbc... 2a0ddcf9-ae58-404f-9d91-13a5196c2217 UNLICENSED 2022-07-28T18:00:15.479792Z 4.333333 39 [{'label': 'Virtual Tour', 'value': 'https://w... NaN NaN NaN NaN NaN NaN NaN NaN
1 c5ce3267-c2ed-4785-a5d8-66c61fcf6063 [{'label': 'View Nearby Homes', 'type': 'realE... Best Private High Schools in Wisconsin 2 82 best-private-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.186400 -87.935800 USM 01512787 Private True True Private School c5ce3267-c2ed-4785-a5d8-66c61fcf6063 True True WI University School of Milwaukee 8b295479-c31f-47a9-83b8-94b2100e2832 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c 739d0594-0714-4d74-ad01-f07df19bc756 5d98fbca-9d9d-4219-8335-8dba54962ca7 True True University School [Private School, WI, PK, K-12] School university-school-of-milwaukee-river-hills-wi 1041 [{'config': {'format': ['comma'], 'rounding': ... Parent It is clear to see, in the short time we’ve be... [Overall Experience] 2022-10-28T07:41:07.70707Z a7a94913-bb20-4def-9553-761720f5cac8 5.0 [{'description': 'Based on quality of academic... https://d13b2ieg84qqce.cloudfront.net/184acaa6... https://d13b2ieg84qqce.cloudfront.net/d98566c3... https://d13b2ieg84qqce.cloudfront.net/c65ee0e3... be9334fb-56d4-4c0c-a4b9-b2de53c46b09 UNLICENSED https://d13b2ieg84qqce.cloudfront.net/887e0e98... cfb21e87-82a7-4fa5-8af6-bf33d199039a UNLICENSED 2022-02-10T21:12:07.916464Z © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... 733bf01a-d21c-4374-bb52-42175a61a2c2 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ https://d13b2ieg84qqce.cloudfront.net/e80f6114... https://d13b2ieg84qqce.cloudfront.net/e80f6114... c8cb35e1-b83c-47f1-a649-eb3766a53de7 UNLICENSED NaN 4.209524 105 [{'label': 'Virtual Tour'}] 501390 https://d13b2ieg84qqce.cloudfront.net/97d061e2... 2022-07-11T13:31:31.710239Z NaN NaN NaN NaN NaN
2 84ab245d-ad99-43c9-93d8-9e474a109434 [{'label': 'View Nearby Homes', 'type': 'realE... Best Private High Schools in Wisconsin 3 82 best-private-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.163916 -89.385004 MCDS A9904507 Private True True Private School 84ab245d-ad99-43c9-93d8-9e474a109434 True True WAUNAKEE, WI Madison Country Day School 4135e47a-62f6-4777-b514-d2e51894603f 1a1aaa73-65d0-490d-b3d3-d828716c5f6b 963a1085-efe7-45f5-81ee-d2bbf82a907c NaN 3bca1e55-0153-485a-a337-03448396568b True True MCDS [Private School, WAUNAKEE, WI, PK, K-12] School madison-country-day-school-waunakee-wi 1041 [{'config': {'format': ['comma'], 'rounding': ... Parent The MCDS faculty is truly exceptional -- they ... [Overall Experience] 2022-07-22T13:59:50.567397Z 6c714271-25ac-4206-9ef8-38d3ef1f92d6 5.0 [{'description': 'Based on quality of academic... https://d13b2ieg84qqce.cloudfront.net/68e0beb3... https://d13b2ieg84qqce.cloudfront.net/3a1cfdcf... https://d13b2ieg84qqce.cloudfront.net/b2d1416c... 86a7a6ce-2538-4bf1-8703-6b3b44fda5a4 UNLICENSED https://d13b2ieg84qqce.cloudfront.net/6cb7bdfd... 809aece8-55ce-4632-a3cf-d0a14417ffdc UNLICENSED 2022-02-09T21:25:15.513499Z © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... dc5b6bd7-5a5c-48ee-bdfd-5780de198bc9 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ https://d13b2ieg84qqce.cloudfront.net/66e8fd60... https://d13b2ieg84qqce.cloudfront.net/fb59b45f... 2e6d1bd0-7760-46ae-8ce2-02306508b864 UNLICENSED 2022-04-18T18:58:56.007652Z 3.882353 34 [{'label': 'Virtual Tour', 'value': 'https://w... 502396 https://d13b2ieg84qqce.cloudfront.net/6ea5d8cb... 2022-06-08T22:11:36.605259Z NaN NaN NaN NaN NaN
3 35ca6237-c994-4fe6-b5f9-f09142680d7b [{'label': 'View Nearby Homes', 'type': 'realE... Best Private High Schools in Wisconsin 4 82 best-private-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.457700 -88.827400 Wayland Academy 01514944 Private, Boarding True True Private School 35ca6237-c994-4fe6-b5f9-f09142680d7b True True BEAVER DAM, WI Wayland Academy 3c05ff22-e610-450d-8684-1b9f99edcd1f NaN 963a1085-efe7-45f5-81ee-d2bbf82a907c 1d49bb1b-d2a1-45e2-ac8e-c8d16ab29f3e f132a02a-1ead-4325-bf32-9079b435d74c True True Wayland [Private School, BEAVER DAM, WI, 9-12] School wayland-academy-beaver-dam-wi 1040 [{'config': {'format': ['comma'], 'rounding': ... Alum Though I only attended Wayland for two years (... [Overall Experience] 2022-08-14T20:05:05.231126Z a0bf7334-047c-4ee8-ab95-59c46dff42b3 5.0 [{'description': 'Based on quality of academic... https://d13b2ieg84qqce.cloudfront.net/7cc728a3... https://d13b2ieg84qqce.cloudfront.net/5e24f8a2... https://d13b2ieg84qqce.cloudfront.net/d7835cfd... 99230263-8332-4b03-b475-b948546402b7 UNLICENSED https://d13b2ieg84qqce.cloudfront.net/42561f2c... 697e0f82-7ccb-4651-877e-ffe881e188c5 UNLICENSED NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... c641160c-30c7-4b52-b336-e844ac8a059a © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ https://d13b2ieg84qqce.cloudfront.net/3aaf34d3... https://d13b2ieg84qqce.cloudfront.net/f197c0a9... 124661d4-71cd-4d13-bfcc-926f3e074ade UNLICENSED 2022-09-28T16:06:46.315837Z 3.833333 66 [{'label': 'Virtual Tour', 'value': 'https://y... 500170 https://d13b2ieg84qqce.cloudfront.net/9b54f4ea... 2022-07-26T17:26:09.050891Z NaN NaN NaN NaN NaN
4 9b394d9c-46a0-431d-8ae4-62b6142cd46b [{'label': 'View Nearby Homes', 'type': 'realE... Best Private High Schools in Wisconsin 5 82 best-private-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 42.773585 -87.774410 TPS 01513124 Private True True Private School 9b394d9c-46a0-431d-8ae4-62b6142cd46b True False WIND POINT, WI The Prairie School 5455e716-0063-4d63-a0e2-a07d199cdee1 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c 5ef4c7c2-c006-49ea-88e9-9f40a0da6ce6 0d949807-5d44-4fc8-8753-1ce81f4a5d67 False True Prairie [Private School, WIND POINT, WI, PK, K-12] School the-prairie-school-wind-point-wi 41 [{'config': {'format': ['comma'], 'rounding': ... Alum The teachers are awesome and so approachable! ... [Overall Experience] 2020-06-23T03:25:59.897153Z 2d7de44a-38a7-493c-ac87-a024ba85d42d 5.0 [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN https://d13b2ieg84qqce.cloudfront.net/608f2378... a69ad3c5-f274-4bbe-ab1b-f1977c79c6f9 UNLICENSED 2022-02-10T20:38:50.869965Z © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... 86b0f616-f1bf-4123-a2bc-f93255053083 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ https://d13b2ieg84qqce.cloudfront.net/a26a05f2... https://d13b2ieg84qqce.cloudfront.net/a26a05f2... ee7758c3-09ee-4d81-b4bd-c7f91c98652a UNLICENSED NaN 4.642857 70 [{'label': 'Virtual Tour', 'value': 'https://w... 501918 NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
95 8365632c-160e-4beb-b75c-dfafca1c2441 [{'label': 'View Nearby Homes', 'type': 'realE... Best Public Middle Schools in Wisconsin 24 594 best-public-middle-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 44.891018 -87.290723 Sevastopol Middle School 551350000496 Public False True Public School 8365632c-160e-4beb-b75c-dfafca1c2441 False False STURGEON BAY, WI Sevastopol Middle School caaa657e-9c5e-4740-b72f-bef5b2c75ac1 NaN 963a1085-efe7-45f5-81ee-d2bbf82a907c 65ab2591-75de-487d-8a82-bddd79e3d3bd 7ae55b50-154c-4e0e-aff7-ed2726f7ceb8 False True Sevastopol Middle School [Sevastopol School District, WI, 6-8] School sevastopol-middle-school-sturgeon-bay-wi 45 [{'config': {'format': ['comma'], 'rounding': ... NaN NaN NaN NaN NaN NaN [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN NaN NaN NaN NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... e750dc05-07ed-42b0-92e3-f24ab16f1b8b © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ NaN NaN NaN NaN NaN 0.000000 0 [{'label': 'Virtual Tour'}] NaN NaN NaN d4d24c63-d104-44cd-ad3f-0ded85522583 d4d24c63-d104-44cd-ad3f-0ded85522583 NaN NaN NaN
96 d46a53a4-62f4-4086-9a53-4c6f78f54915 [{'label': 'View Nearby Homes', 'type': 'realE... Best Public Elementary Schools in Wisconsin 36 1074 best-public-elementary-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 42.935063 -88.405594 Prairie View Elementary School 551006001321 Public True True Public School d46a53a4-62f4-4086-9a53-4c6f78f54915 True False NORTH PRAIRIE, WI Prairie View Elementary School ba8709ae-856d-4583-83b7-4484b51ed4c2 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c 1a756678-9c81-4d89-8620-604b8e10507c 8afa0d18-4b1f-4052-a09a-d9bfe3e67295 False True Prairie View Elementary School [Mukwonago Area School District, WI, PK, K-6] School prairie-view-elementary-school-north-prairie-wi 45 [{'config': {'format': ['comma'], 'rounding': ... NaN NaN NaN NaN NaN NaN [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN NaN NaN NaN NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... 51a79bc3-f67e-4b49-87b1-d36ef46e1145 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ NaN NaN NaN NaN NaN 0.000000 0 [{'label': 'Virtual Tour'}] NaN NaN NaN bda72d2a-3f49-4288-a9f2-d024898ca67b bda72d2a-3f49-4288-a9f2-d024898ca67b NaN NaN NaN
97 0ed32d0d-f062-4784-8dd4-57a9724209eb [{'label': 'View Nearby Homes', 'type': 'realE... Best Public Middle Schools in Wisconsin 25 594 best-public-middle-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.624047 -87.786299 Oostburg Middle School 551107001464 Public False True Public School 0ed32d0d-f062-4784-8dd4-57a9724209eb False False OOSTBURG, WI Oostburg Middle School 1db5c6d2-5b8f-44fa-87fc-f7471ee45443 NaN 963a1085-efe7-45f5-81ee-d2bbf82a907c b44a651d-bcfc-4d95-a6ce-f00c0c42671e d594fdef-3441-462c-93ad-981c8fd1f064 False True Oostburg Middle School [Oostburg School District, WI, 6-8] School oostburg-middle-school-oostburg-wi 45 [{'config': {'format': ['comma'], 'rounding': ... Niche User The middle school does a great job at preparin... [Academics] 2015-02-12T14:29:22Z 0c61a295-b7c7-44e9-ab7a-64993190796f 5.0 [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN NaN NaN NaN NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... 32fa503a-a377-44e8-bb10-f9a7fa0bb67c © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ NaN NaN NaN NaN NaN 4.800000 10 [{'label': 'Virtual Tour'}] NaN NaN NaN 3478a622-503f-47d5-93a0-c3207124cdd4 3478a622-503f-47d5-93a0-c3207124cdd4 NaN NaN NaN
98 17b8ac12-d893-4af4-bf79-3aa06bef648a [{'label': 'View Nearby Homes', 'type': 'realE... Best Public High Schools in Wisconsin 24 496 best-public-high-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 42.993816 -88.224033 WEPA 551578002688 Public, Charter True True Charter School 17b8ac12-d893-4af4-bf79-3aa06bef648a True False WAUKESHA, WI Waukesha Engineering Preparatory Academy ba8709ae-856d-4583-83b7-4484b51ed4c2 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c 5a94913e-87ac-4e4e-9b76-a2330bf1a635 b88c94da-24d3-4004-b43b-547d9da55e0d False True Waukesha Engineering Preparatory Academy [School District of Waukesha, WI, 9-12] School waukesha-engineering-preparatory-academy-wauke... 52 [{'config': {'format': ['comma'], 'rounding': ... Senior The Academy is well equipped and staffed, and ... [Overall Experience] 2021-10-13T21:07:42.049714Z 8d8241ae-fc49-4bab-a146-48d77f1e6391 4.0 [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN NaN NaN NaN NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... b1db421b-6290-4a08-a854-82dd9089e116 © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ NaN NaN NaN NaN NaN 3.681818 22 [{'label': 'Virtual Tour'}] 500331 NaN NaN a368f833-c451-45bb-a0f7-b656d02477f3 a368f833-c451-45bb-a0f7-b656d02477f3 NaN NaN NaN
99 c3b20454-cd71-45bb-ab33-0b3ea37527fb [{'label': 'View Nearby Homes', 'type': 'realE... Best Public Elementary Schools in Wisconsin 37 1074 best-public-elementary-schools/s/wisconsin https://d33a4decm84gsn.cloudfront.net/search/2... https://d33a4decm84gsn.cloudfront.net/search/2... 43.089194 -87.883770 Atwater Elementary School 551380001809 Public True True Public School c3b20454-cd71-45bb-ab33-0b3ea37527fb True False SHOREWOOD, WI Atwater Elementary School 8b295479-c31f-47a9-83b8-94b2100e2832 3940b781-a9f6-4333-b607-6a6367e6af44 963a1085-efe7-45f5-81ee-d2bbf82a907c 900b6b9c-206e-4c34-82a8-247fee552b49 542c1289-ad69-4fc3-afab-bf91c1a6110e False True Atwater Elementary School [Shorewood School District, WI, PK, K-6] School atwater-elementary-school-shorewood-wi 45 [{'config': {'format': ['comma'], 'rounding': ... NaN NaN NaN NaN NaN NaN [{'description': 'Based on quality of academic... NaN NaN NaN NaN NaN NaN NaN NaN NaN © Mapbox https://api.mapbox.com/styles/v1/niche-admin/c... https://api.mapbox.com/styles/v1/niche-admin/c... b523d19c-fdd1-497b-bd6d-ab394cde0dbf © OpenStreetMap http://www.openstreetmap.org/copyright https://www.mapbox.com/about/maps/ NaN NaN NaN NaN NaN 0.000000 0 [{'label': 'Virtual Tour', 'value': 'https://w... NaN NaN NaN 84c36616-1b72-4d85-998d-c9795aadb726 84c36616-1b72-4d85-998d-c9795aadb726 NaN NaN NaN
100 rows × 73 columns
​
You can get all data by adjusting the range (go for 123 for max records). Also, you may want to add some pause between requests, otherwise you'd be blocked.
You can also use Scrapy, if you wish.

Answer 2

You need to check carefully the HTML because you can find the url inside one div
import scrapy
class NicheschoolsSpider(scrapy.Spider):
name = 'nicheschools'
allowed_domains = ['www.niche.com']
start_urls = ['https://www.niche.com/k12/search/best-schools/s/wisconsin/']
def parse(self, response):
school_links = response.xpath("//div[#class='card ']/a/#href").extract()
for link in school_links:
yield response.follow(url=link, callback=self.parse_schools)
def parse_schools(self, response):
name = response.xpath("//h1[#class='postcard__title postcard__title--claimed']/text()").extract_first()
website = response.xpath("(//a[#class='profile__website__link']/#href)[1]").extract_first()
address = response.xpath("(//address[#class='profile__address--compact']/text())[1]").extract_first()
yield {
'name': name,
'link': response.url,
'website': website,
'address': address,
}
Result on json
{'name': 'Brookfield Academy', 'link': 'https://www.niche.com/k12/brookfield-academy-brookfield-wi/', 'website': 'https://www.brookfieldacademy.org', 'address': '3462 N BROOKFIELD RD'}
{'name': 'Wisconsin Lutheran High School', 'link': 'https://www.niche.com/k12/wisconsin-lutheran-high-school-milwaukee-wi/', 'website': 'https://www.wlhs.org', 'address': '330 N GLENVIEW AVE'}
{'name': 'Homestead High School', 'link': 'https://www.niche.com/k12/homestead-high-school-mequon-wi/', 'website': 'http://www.mtsd.k12.wi.us/homestead/', 'address': '5000 W MEQUON RD'}
{'name': 'Brookfield Central High School', 'link': 'https://www.niche.com/k12/brookfield-central-high-school-brookfield-wi/', 'website': 'https://www.elmbrookschools.org/brookfield-central-high-school', 'address': '16900 W GEBHARDT RD'}
{'name': 'Shorewood High School', 'link': 'https://www.niche.com/k12/shorewood-high-school-shorewood-wi/', 'website': 'https://www.shorewood.k12.wi.us/apps/pages/shs', 'address': '1701 E CAPITOL DR'}
{'name': 'School District of Waukesha', 'link': 'https://www.niche.com/k12/d/school-district-of-waukesha-wi/', 'website': 'https://sdw.waukesha.k12.wi.us', 'address': '222 MAPLE AVE'}
{'name': 'Pilgrim Park Middle School', 'link': 'https://www.niche.com/k12/pilgrim-park-middle-school-elm-grove-wi/', 'website': 'http://www.elmbrookschools.org/', 'address': '1500 PILGRIM PKWY'}
{'name': 'Marquette University High School', 'link': 'https://www.niche.com/k12/marquette-university-high-school-milwaukee-wi/', 'website': 'https://www.muhs.edu/', 'address': '3401 W WISCONSIN AVE'}
...
If you are new on web scraping you need to be careful with over hitting the site because they could block you and then you need to solve a captcha solution for enter the site.
Also If you want to expand your knowledge there are clusters of web scraping like Estela where you can run your spiders and also create cronjobs for do it everyday.