Scraping table in RVest when there are multiple rows that span columns - r

Question

I am trying to scrape the following webpage: https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=031901
Initially, I tried this code:
library(rvest)
library(tidyverse)
webpage = read_html("https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=101912")
tables <- html_nodes(webpage, "table") %>%
html_table()
budget = tables[[2]]
Then I realized that the results were messy, because html_table() cannot read tables where rows span multiple columns. I could clean the dataframe up after scraping it, but I'm wondering if perhaps there's a way to scrape it that avoids the issue in the first place.
I read the answers to similar questions, but they all involved a situation where only the header rows spanned multiple columns, or only the first column spanned multiple rows. In this case, this table is made up of multiple tables squished together, so it's like there are headers all throughout the table. Is there a solution that can deal with rows spanning columns throughout the table?

Answer 1

One way could be as follows:
Decide on your final format. I have selected on a flat file format which entails splitting some header values, those with merged cells, into separate new columns, and repeating these values later
Gather and loop all the rows of data
In the loop process the row according to a set of rules
Generate your output row in the loop and write to a sized list
Drop empty entries list
Convert to DataFrame and add any additional info wanted
The rules I chose to apply where based on looking at the first column of each row as follows (pseudo-code):
if (first column className contains "linecontent") {
if (if first column has br and span child elements) {
split colmn text, after trimming, on line break
assign split to red and measure columns (re-use as filldown)
} else {
assign measure the trimmed column text
}
} else {
we are on a non-header row so grab the $ and % values by matching on class "data"
}
R:
library(tidyverse)
library(rvest)
library(httr2)
page <- request("https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=031901") %>%
req_headers(
"user-agent" = "Mozilla/5.0",
"accept" = "text/html",
"connection" = "keep-alive"
) %>%
req_perform() %>%
resp_body_html()
rows <- page %>% html_elements(".table tbody tr")
row_data <- vector("list", length(rows))
row <- 1
for (row_node in rows) {
first_column_node <- row_node %>% html_element("td")
first_column_node_classname <- html_attr(first_column_node, "class")
is_new_column <- if_else(grepl("linecontent", first_column_node_classname), 1, 0)
column_needs_split <- if_else((length(html_elements(first_column_node, "br")) > 0 &
!is.na(html_element(first_column_node, "span"))), 1, 0)
node_text <- first_column_node %>% html_text2()
data <- NULL
if (is_new_column) {
if (column_needs_split) {
new_values <- trimws(str_split_1(trimws(node_text), "\\n"))
red <- new_values[[1]]
measure <- new_values[[2]]
} else {
measure <- trimws(node_text)
}
} else {
data <- row_node %>%
html_elements(".data") %>%
html_text2()
}
if (!is.null(data)) {
row_data[[row]] <- c(c(red, measure), data)
# print(c(c(red, measure), data))
row <- row + 1
}
}
row_data <- discard(row_data, is.null)
df <- do.call(rbind, row_data) %>% as.data.frame()
colnames(df) <- c(
"red", "category", "measure",
"gen_fund", "gen_fund_perc", "gen_fund_per_student",
"all_fund", "all_fund_perc", "all_fund_per_student"
)
summary_info <- page %>% html_element('.c.systemtitle') %>% html_text(trim = T)
additional_info <- str_match_all(
gsub("\\n", "", summary_info),
"(\\d{4}\\s-\\s\\d{4}).*Totals for (.*?)\\sISD.*?\\((\\d{6})\\)"
)
df$year <- additional_info[[1]][, 2]
df$district <- additional_info[[1]][, 3]
df$isd_code <- additional_info[[1]][, 4]
head(df)
Sample output:
> head(df)
red category measure gen_fund gen_fund_perc
1 Revenues Operating Revenue Local Property Tax from M&O (excluding recapture) $70,019,020 15.03%
2 Revenues Operating Revenue State Operating Funds $333,999,269 71.68%
3 Revenues Operating Revenue Federal Funds $59,326,937 12.73%
4 Revenues Operating Revenue Other Local $2,644,317 0.57%
5 Revenues Operating Revenue Total Operating Revenue $465,989,543 100.00%
6 Revenues Other Revenue Local Property Tax from I&S $0 0.00%
gen_fund_per_student all_fund all_fund_perc all_fund_per_student year district isd_code
1 $1,823 $70,019,020 15.03% $1,823 2021 - 2022 BROWNSVILLE 031901
2 $8,695 $333,999,269 71.68% $8,695 2021 - 2022 BROWNSVILLE 031901
3 $1,544 $59,326,937 12.73% $1,544 2021 - 2022 BROWNSVILLE 031901
4 $69 $2,644,317 0.57% $69 2021 - 2022 BROWNSVILLE 031901
5 $12,131 $465,989,543 100.00% $12,131 2021 - 2022 BROWNSVILLE 031901
6 $0 $9,212,992 17.23% $240 2021 - 2022 BROWNSVILLE 031901