Scraping a web table through multiple pages (some rows are missing) - r

Question

I'd like to scrape a table (containing information about 31,385 soldiers) from https://irelandsgreatwardead.ie/the-archive/ using rvest.
library(rvest)
library(dplyr)
page <- read_html(x = "https://irelandsgreatwardead.ie/the-archive/")
table <- page %>%
html_nodes("table") %>%
html_table(fill = TRUE) %>%
as.data.frame()
This works, but only for the first 10 soldiers. In the source code, I can only see the information for the first 10 soldiers either. Any help on how to obtain the rows with the other soldiers would be highly appreciated!
Thanks and have a great day!

Answer 1

Here is the RSelenium solution,
You can loop through page extracting table and joining to the previous table.
First launch the browser,
library(RSelenium)
driver = rsDriver(browser = c("firefox"))
remDr <- driver[["client"]]
remDr$navigate(url)
PART 1: Extracting table from first page and storing in df,
df = remDr$getPageSource()[[1]] %>%
read_html() %>%
html_table()
df = df[[1]]
#removing last row which is non-esstential
df = df[-nrow(df),]
PART 2: Loop through pages 2 to 5
for(i in 2:5){
#Building xpath for each page
xp = paste0('//*[#id="table_1_paginate"]/span/a[', i, ']')
cc <- remDr$findElement(using = 'xpath', value = xp)
cc$clickElement()
# Three second gap is given for the webpage to load
Sys.sleep(3)
df1 = remDr$getPageSource()[[1]] %>%
read_html() %>%
html_table()
df1 = df1[[1]]
df1 = df1[-nrow(df1),]
#Joining previous table `df` and present table `df1`
df = rbind(df, df1)
}
PART 3: Loop through rest of the pages 6 to 628
The xpath of remaining pages remains the same. Thus we have to repeat this code block 623 times to get table from remaining pages.
for (i in 1:623) {
x = i
cc <- remDr$findElement(using = 'xpath', value = '//*[#id="table_1_paginate"]/span/a[4]')
cc$clickElement()
Sys.sleep(3)
df1 = remDr$getPageSource()[[1]] %>%
read_html() %>%
html_table()
df1 = df1[[1]]
df1 = df1[-nrow(df1),]
df = rbind(df, df1)
}
Now we have df with info of all soldiers.

Answer 2

library(RSelenium)
driver = rsDriver(browser = c("firefox"))
remDr <- driver[["client"]]
url <- 'https://irelandsgreatwardead.ie/the-archive/'
remDr$navigate(url)
# Locate the next page link
webElem <- remDr$findElement(using = "css", value = "a[data-dt-idx='3'")
# Click that link
webElem$clickElement()
# Get that table
remDr$getPageSource()[[1]] %>%
read_html() %>%
html_table()
Your for loop needs to start at a value of 3 (thats the second page!). On the second page it becomes 4, etc. BUT it never goes over 5. because of the way it is 'designed' so you'd loop 3:5 then at 5 keep it at 5 each time.