R: Getting links from Google search results beyond the first page r - r

Question

I have this RSelenium setup (using selenium really shouldn't impact the answer to this question):
library(tidyverse)
library(rvest)
library(httr)
library(RSelenium) # running through docker
## RSelenium setup
remDr <- remoteDriver(port = 4445L, browserName = "chrome")
remDr$open()
## Navigate to Google Books
remDr$navigate("https://books.google.com/")
books <- remDr$findElement(using = "css", "[name = 'q']")
## Search for whatever, the Civil War, for example
books$sendKeysToElement(list("the civil war", key = "enter"))
## Getting Google web elements (10 per page)
bookElem <- remDr$findElements(using = "xpath", "//h3[#class = 'LC20lb']//parent::a")
## Click on each book link
links <- sapply(bookElem, function(bookElem){
bookElem$getElementAttribute("href")
})
This works great - and compiles all of the links from the first page of results (Google automatically limits it to 10 results, so ten links). What I would like is to have that same links vector compile every result link from the first, say, 12 pages (to keep it manageable). So:
goog_pgs <- seq(1:12) # to set the limit
Where I'm lost: how do I feed that into my links vector? The links from each page are too different and aren't simple enough to just feed the number to its end. I've tried inserting the following:
nextButton <- remDr$findElements("xpath", "//*[#id = 'pnnext']")
next_page <- sapply(nextButton, function(nextButton) {
next_elements$clickElement()
})
And that does not work. What's the solution here?

Answer 1

You can use the sequence 1:12 as something to iterate over, using a for loop, lapply, or other looping mechanism. I have a terrible time with the apply functions, so I swapped in with map. The steps that need to be done repeatedly are finding books, getting the href of each book, and clicking the "next" button. With some modification, you can use:
books_12 <- map(1:12, function(pg) {
bookElem <- remDr$findElements(using = "xpath", "//h3[#class = 'LC20lb']//parent::a")
links <- map_chr(bookElem, ~.$getElementAttribute("href")[[1]])
nextButton <- remDr$findElement("xpath", "//*[#id='pnnext']")
nextButton$clickElement()
links
})
Note that getElementAttribute returns a list; since each element only has one href, I kept the first (only) one with [[1]]. This yields a list of 12 vectors of 10 URLs each.