NA's for blanks in web scraping - r

Question

I want to scrape the below mentioned page, but there are some blanks in ".trans-section" node. '.trans-section' node is capturing 'title' as well as 'description'. In some table title will be there, but description is missing. I want the data to be filled with NA's when the description is blank. Since the node is same for both I am not getting any blank lines. Please help on this.
Weblink: https://patentscope.wipo.int/search/en/result.jsf?currentNavigationRow=5&prevCurrentNavigationRow=2&query=FP:(Gaming)&office=&sortOption=Pub%20Date%20Desc&prevFilter=&maxRec=39316&viewOption=All&listLengthOption=200
library(rvest)
library(httr)
library(XML)
FinalD <- data.frame()
for (i in 1:10) {
rm(Data)
## Creating web page
Webpage <- paste0('https://patentscope.wipo.int/search/en/result.jsf?currentNavigationRow=',i,'&prevCurrentNavigationRow=1&query=&office=&sortOption=Pub%20Date%20Desc&prevFilter=&maxRec=64653917&viewOption=All&listLengthOption=100')
Webpage <- read_html(Webpage)
#Getting Nodes
Node_Intclass <- html_nodes(Webpage,'.trans-section')
Intclass <- data.frame(html_text(Node_Intclass))
Intclass$sequence <- seq(1:2)
Node_Others <- html_nodes(Webpage,'.notranslate')
Others <- data.frame(html_text(Node_Others))
Others$sequence <- seq(1:9)
####Others
Data <- data.frame(subset(Others$html_text.Node_Others.,Others$sequence == 1))
Data$ID <- subset(Others$html_text.Node_Others.,Others$sequence == 2)
Data$Country <- subset(Others$html_text.Node_Others.,Others$sequence == 3)
Data$PubDate <- subset(Others$html_text.Node_Others.,Others$sequence == 4)
Data$IntClass <- subset(Others$html_text.Node_Others.,Others$sequence == 5)
Data$ApplINo <- subset(Others$html_text.Node_Others.,Others$sequence == 7)
Data$Applicant <- subset(Others$html_text.Node_Others.,Others$sequence == 8)
Data$Inventor <- subset(Others$html_text.Node_Others.,Others$sequence == 9)
###Content
ifelse ((nrow(Intclass) == 200),
((Data$Title <- subset(Intclass$html_text.Node_Intclass.,Intclass$sequence == 1))&
(Data$Content <- subset(Intclass$html_text.Node_Intclass.,Intclass$sequence == 2))),
((Data$Title <- 0 ) & (Data$Content = 0)))
#Final Data
FinalD <- rbind(FinalD,Data)
}
write.csv(FinalD,'FinalD.csv')

Answer 1

Well, I am not an expert of Web Scraping ( I have just tried a few times) but I have realized that it is a tiresome procedure with a lot of trials and errors.
Maybe you can use the RSelenium package as the page is dynamically generated.For me it works but it creates a kind of messy output ,maybe it is better though.
library(RSelenium)
library(rvest)
library(dplyr)
library(data.table)
library(stringr)
tables1 <- list()
for (i in 1:10) { # i <- 1; i
## Creating web page
url <- paste0('https://patentscope.wipo.int/search/en/result.jsf?currentNavigationRow=',i,'&prevCurrentNavigationRow=1&query=&office=&sortOption=Pub%20Date%20Desc&prevFilter=&maxRec=64653917&viewOption=All&listLengthOption=100')
rD <- rsDriver( browser="chrome")
remDr <- rD$client
remDr$navigate(url)
page <- remDr$getPageSource()
remDr$close()
table <- page[[1]] %>%
read_html() %>%
html_nodes(xpath='//table[#id="resultTable"]') %>% # specify table as there is a div with same id
html_table(fill = T)
table <- table[[1]]
tables1[[url]] <- table %>% as.data.table()
rm(rD)
gc()}
I would also suggest you to create the list of pages that you want to read, outside the loop, and create an index so as if the connection fails you continue from the page you were left.
In addition, if the connection fails, run the
rm(rD)
gc()
lines to avoid an error that says that the port is already in use.
I hope it helped.

Answer 2

(Not tested)
Can you try to add the option:
read_html(Webpage, options = c("NOBLANKS"))