Are there any website content monitoring packages in R? - r

Question

I know there are free website content monitoring programs that send email alerts when the content of a website is changed, but is there a package (or any way to hard code) in R which can do this? It would be helpful to integrate this in one work flow.

Answer 1

R is a general purpose programming language so you can do anything with it.
Core idiom for what you are trying to do is:
Identify target site
Pull content & content metadata
Cache ^^ (you need to figure this out; RDBMS tables? NoSQL tables? Files?)
Let n time-periods pass (you need to figure this out: cron? launchd? Amazon lambda?)
Pull content & content metadata
Compare ^^ against cached versions; NOTE this works best if you know the structure of the target site vs use an overly generic framework)
If difference is "significant", notify via whatever means you want (you need to figure this out: email? SMS? Twitter?)
For content, you may not be aware that httr::GET() returns a rich, complex data object full of metadata. I did not do a str(res) below to encourage you to do so on your own.
library(httr)
library(rvest)
library(splashr)
library(hgr) # devtools::install_github("hrbrmstr/hgr")
library(tlsh) # devtools::install_github("hrbrmstr/tlsh")
target_url <- "https://www.whitehouse.gov/briefings-statements/"
Get it like a browser would
httr::GET(
url = target_url,
httr::user_agent(splashr::ua_macos_safari)
) -> res
Cache page size and use a substantial difference to signal notification
(page_size <- res$headers['content-length'])
## $`content-length`
## [1] "12783"
Calculate & cache local sensitify hash value use tlsh_simple_diff() to see if there are "substantial" hash changes and use that as a signal to notify:
doc_text <- httr::content(res, as = "text")
(doc_hash <- tlsh_simple_hash(doc_text))
## [1] "563386E33C44683E060B739261ADF20CB2D38563EE151C88A3F95169999FF97A1F385D"
This site uses structured <div>'s so cache and use more/fewer/different ones to signal notification:
doc <- httr::content(res)
news_items <- html_nodes(doc, "div.briefing-statement__content")
(total_news_items <- length(news_items))
## [1] 10
(headlines <- gsub("[[:space:]]+", " ", html_text(news_items, trim=TRUE)))
## [1] "News Clips CNBC: “Job Openings Hit Record 7.136 Million in August” Economy & Jobs Oct 16, 2018"
## [2] "Fact Sheets Congressional Democrats Want to Take Away Your Doctor, Outlaw Your Private Insurance, and Put Bureaucrats In Charge of Your Healthcare Healthcare Oct 16, 2018"
## [3] "Remarks Remarks by President Trump in Briefing on Hurricane Michael Land & Agriculture Oct 15, 2018"
## [4] "Remarks Remarks by President Trump and Governor Scott at FEMA Aid Distribution Center | Lynn Haven, FL Land & Agriculture Oct 15, 2018"
## [5] "Remarks Remarks by President Trump During Tour of Lynn Haven Community | Lynn Haven, FL Land & Agriculture Oct 15, 2018"
## [6] "Remarks Remarks by President Trump and Governor Scott Upon Arrival in Florida Land & Agriculture Oct 15, 2018"
## [7] "Remarks Remarks by President Trump Before Marine One Departure Foreign Policy Oct 15, 2018"
## [8] "Statements & Releases White House Appoints 2018-2019 Class of White House Fellows Oct 15, 2018"
## [9] "Statements & Releases President Donald J. Trump Approves Georgia Disaster Declaration Land & Agriculture Oct 14, 2018"
## [10] "Statements & Releases President Donald J. Trump Amends Florida Disaster Declaration Land & Agriculture Oct 14, 2018"
Use a "readability" tool to turn the contents into plaintext cache & compare with one of the many "text diff/string diff" R packages:
content_meta <- hgr::just_the_facts(target_url)
str(content_meta)
## List of 11
## $ title : chr "Briefings & Statements"
## $ content : chr "<p class=\"body-overflow\"> <header class=\"header\"> </header>\n<main id=\"main-content\"> <div class=\"page-r"| __truncated__
## $ lead_image_url: chr "https://www.whitehouse.gov/wp-content/uploads/2017/12/wh.gov-share-img_03-1024x538.png"
## $ next_page_url : chr "https://www.whitehouse.gov/briefings-statements/page/2"
## $ url : chr "https://www.whitehouse.gov/briefings-statements/"
## $ domain : chr "www.whitehouse.gov"
## $ excerpt : chr "Get official White House briefings, statements, and remarks from President Donald J. Trump and members of his Administration."
## $ word_count : int 22
## $ direction : chr "ltr"
## $ total_pages : int 2
## $ pages_rendered: int 2
## - attr(*, "row.names")= int 1
## - attr(*, "class")= chr "hgr"
Unfortunately, you asked a general purpose computing-ish question and, as such, it is likely to get closed.