Scrape Data from Wikipedia - web-scraping

Question

I am trying to find or build a web scraper that is able to go through and find every state/national park in the US along with their GPS coordinates and land area. I have looked into some frameworks like Scrapy and then I see there are some sites that are specifically for Wikipedia such as http://wiki.dbpedia.org/About. Is there any specific advantage to either one of these or would either one work better to load the information into an online database?

Answer 1

Let's suppose you want to parse pages like this Wikipedia page. The following code should work.
var doc = new HtmlDocument();
doc = .. //Load the document here. See doc.Load(..), doc.LoadHtml(..), etc.
//We get all the rows from the table (except the header)
var rows = doc.DocumentNode.SelectNodes("//table[contains(#class, 'sortable')]//tr").Skip(1);
foreach (var row in rows) {
var name = HttpUtility.HtmlDecode(row.SelectSingleNode("./*[1]/a[#href and #title]").InnerText);
var loc = HttpUtility.HtmlDecode(row.SelectSingleNode(".//span[#class='geo-dec']").InnerText);
var areaNodes = row.SelectSingleNode("./*[5]").ChildNodes.Skip(1);
string area = "";
foreach (var a in areaNodes) {
area += HttpUtility.HtmlDecode(a.InnerText);
}
Console.WriteLine("{0,-30} {1,-20} {2,-10}", name, loc, area);
}
I tested it, and it produces the following output:
Acadia 44.35A°N 68.21A°W 47,389.67 acres (191.8 km2)
American Samoa 14.25A°S 170.68A°W 9,000.00 acres (36.4 km2)
Arches 38.68A°N 109.57A°W 76,518.98 acres (309.7 km2)
Badlands 43.75A°N 102.50A°W 242,755.94 acres (982.4 km2)
Big Bend 29.25A°N 103.25A°W 801,163.21 acres (3,242.2 km2)
Biscayne 25.65A°N 80.08A°W 172,924.07 acres (699.8 km2)
Black Canyon of the Gunnison 38.57A°N 107.72A°W 32,950.03 acres (133.3 km2)
Bryce Canyon 37.57A°N 112.18A°W 35,835.08 acres (145.0 km2)
Canyonlands 38.2A°N 109.93A°W 337,597.83 acres (1,366.2 km2)
Capitol Reef 38.20A°N 111.17A°W 241,904.26 acres (979.0 km2)
Carlsbad Caverns 32.17A°N 104.44A°W 46,766.45 acres (189.3 km2)
Channel Islands 34.01A°N 119.42A°W 249,561.00 acres (1,009.9 km2)
Congaree 33.78A°N 80.78A°W 26,545.86 acres (107.4 km2)
Crater Lake 42.94A°N 122.1A°W 183,224.05 acres (741.5 km2)
Cuyahoga Valley 41.24A°N 81.55A°W 32,860.73 acres (133.0 km2)
Death Valley 36.24A°N 116.82A°W 3,372,401.96 acres (13,647.6 km2)
Denali 63.33A°N 150.50A°W 4,740,911.72 acres (19,185.8 km2)
Dry Tortugas 24.63A°N 82.87A°W 64,701.22 acres (261.8 km2)
Everglades 25.32A°N 80.93A°W 1,508,537.90 acres (6,104.8 km2)
Gates of the Arctic 67.78A°N 153.30A°W 7,523,897.74 acres (30,448.1 km2)
Glacier 48.80A°N 114.00A°W 1,013,572.41 acres (4,101.8 km2)
(...)
I think that's a start. If some page fails, you have to see if the layout changes, etc.
Of course, you will also have to find a way of obtaining all the links you want to parse.
One important thing: Do you know if is permitted to scrape Wikipedia? I have no idea, but you should see if it is before doing it... ;)

Answer 2

Though the question is a little old, another alternative available right now is to avoid any scraping and get the raw data direct from protectedplanet.net - it contains data from the World Database of Protected Areas and the UN's List of Protected Areas. (Disclosure: I worked for UNEP-WCMC, the organisation that produced and maintains the database and the website.)
It's free for non-commercial use, but you'll need to register to download. For example, this page lets you download 22,600 protected areas in the USA as KMZ, CSV and SHP (contains lat, lng, boundaries, IUCN category and a bunch of other metadata).

Answer 3

I would conisder this not the best approach.
My idea would be to go to the API from openstreetmap.org (or any other GEO based API that you can query) and ask it for the data you want. National parks are likely to be found pretty easily. You can get the names from a source like Wikipedia and then ask ony of the GEO APIs to give you the information you want.
BTW, what'S wrong with Wikipedias List of National Parks?