Problem with webscraping selector gadget and rvest - r

Question

I'm trying to scrape data from https://3g.dxy.cn/newh5/view/pneumonia , using SelectorGadget and rvest
I successfully scrape some text in the page with the following code.
library(rvest)
url <- 'https://3g.dxy.cn/newh5/view/pneumonia'
webpage <- read_html(url)
TEXT_html <- html_nodes(webpage,'.descText___Ui3tV')
TEXT <- html_text(TEXT_html)
But when I try to select the table where the most important data (the number of people infected in the table) see selection using the following code
TABLE_html <- html_nodes(webpage,'.areaBlock1___3V3UU p')
TABLE <- html_text(TABLE_html)
the output is "character 0"
I guess it's because the data in the table can't be seen because they are refreshed via API, but I don't really know how to solve this
Anybody has an idea? thank you very much

Answer 1

On this page, the data isn't retrieved seperately from an API. It is actually present in the html page you downloaded, but it is in JSON format inside a script tag and the reason why rvest can't read it is that the data is only added to the DOM by Javascript after the page loads. To get at the data you need to extract and parse the JSON:
library(rvest)
library(tibble)
library(jsonlite)
data <- 'https://3g.dxy.cn/newh5/view/pneumonia' %>%
read_html() %>%
html_node('#getAreaStat') %>% # This is the tag containing the JSON
html_text() %>% # Get the javascript from the node
strsplit("(getAreaStat = )|(}catch)") %>% # Carve out the JSON
unlist() %>%
`[`(2) %>% # Unlist and extract the JSON
fromJSON() # Parse the JSON
Now data is a data frame containing all the information from the JSON. However, the last column of data is actually a list of city-level data frames. Since these all have the same column names they can be bound together with rbind. The final column can then be removed from data so you have a data frame of province-level data and another one of city-level data.
city_data <- as_tibble(do.call(rbind, data$cities))
province_data <- as_tibble(data[, -8])
So province_data looks like a bit like this (the Chinese symbols haven't copied over but appear in the R console)
province_data
#> # A tibble: 33 x 7
#> provinceName provinceShortNa~ confirmedCount suspectedCount curedCount deadCount
#> <chr> <chr> <int> <int> <int> <int>
#> 1 ??? ?? 2714 0 52 100
#> 2 ??? ?? 207 0 4 0
#> 3 ??? ?? 173 0 3 0
#> 4 ??? ?? 168 0 0 1
#> 5 ??? ?? 143 0 0 0
#> 6 ??? ?? 132 0 0 0
#> 7 ??? ?? 106 0 0 0
#> 8 ??? ?? 95 0 0 0
#> 9 ??? ?? 91 0 2 1
#> 10 ??? ?? 90 0 0 0
#> # ... with 23 more rows, and 1 more variable: comment <chr>
and city_data looks like this (again, with cityName printed properly in the console).
#> # A tibble: 329 x 5
#> cityName confirmedCount suspectedCount curedCount deadCount
#> <chr> <int> <int> <int> <int>
#> 1 ?? 1590 0 47 85
#> 2 ?? 213 0 2 4
#> 3 ?? 173 0 0 1
#> 4 ?? 114 0 0 3
#> 5 ?? 91 0 0 0
#> 6 ?? 71 0 1 2
#> 7 ?? 70 0 0 0
#> 8 ?? 70 0 0 0
#> 9 ?? 65 0 0 0
#> 10 ?? 57 0 0 0
#> # ... with 319 more rows