Text extraction from PDF with search criteria - r

Question

I need to extract text from a PDF, I have a list of keywords which tell me what text part I need to extract.
PDF looks something like this:
Schema element: Keyword1 This is my keyword
Fontsize: 14 I dont need this
Guide to complete schema element: Text text. This is the text I need and it can between 2 and 3 lines long. And even contain multiple sentences.
Schema element: Keyword2 This is my keyword
Fontsize: 18 I dont need this
Guide to complete schema element: Text text, this is the text I need and it can between 2 and 3 lines long. And even contain multiple sentences. This text is different from the text above.
This is my code so far:
library(pdftools)
library(pdfsearch)
library(tidyverse)
pdf <- pdf_text(dir(pattern = "*.pdf")) %>%
read_lines()
Keyword_list <- c("swDisproportionateCost", `"swDisproportionateCostOtherEULegislation", "swExemptionsTransboundary","swDisproportionateCostAlternativeFinancing","swDisproportionateCostAnalysis","swDisproportionateCostScale")`
Then I tried using keyword_search but it only told me which line the keyword was in.
I would like to extract the text in cursive into a new column in my keyword_list. I think it can be done with regex using the keyword and the text in bold as start and stops.
Here is a link to the pdf.
https://www.dropbox.com/s/kyyzr5wnh8z87if/FINAL%20Draft4_WFD_Reporting_Guidance_2022_resource_page.pdf?dl=0

Answer 1

This is just a rather pedestrian text extraction job. There are many ways to do it, and I'm sure there are more elegant ways to do it than this, but this one does the job:
library(pdftools)
library(dplyr)
keywords <- pdf_text("mypdf.pdf") %>%
strsplit("Schema element:") %>%
lapply(function(x) x[-1]) %>%
lapply(function(x) sapply(strsplit(x, "\r\n"), `[`, 1)) %>%
unlist %>%
trimws()
text <- pdf_text("mypdf.pdf") %>%
strsplit("Guidance on completion of schema element:") %>%
lapply(function(x) x[-1]) %>%
lapply(function(x) sapply(strsplit(x, ":"), `[`, 1)) %>%
lapply(function(x) sapply(strsplit(x, "\r\n"),
function(y) paste(y[-length(y)], collapse = ""))) %>%
unlist() %>%
{gsub(" ", " ", .)} %>%
trimws() %>%
strsplit("Guidance on contents") %>%
sapply(`[`, 1)
df <- tibble(keywords, text)
So the result looks like this:
df
#> # A tibble: 15 x 2
#> keywords text
#> <chr> <chr>
#> 1 swExemption44Driver "Required. Select from the enumeration list the driver~
#> 2 swExemption45Impact "Required. Select from the enumeration list the impact~
#> 3 swExemption45Driver "Required. Select from the enumeration list the driver~
#> 4 swDisproportionateCost "Required. Indicate if disproportionate costs have bee~
#> 5 swDisproportionateCostScale "Conditional. Select from the enumeration list the sc~
#> 6 swDisproportionateCostAnalysis "Conditional. Select from the enumeration list the an~
#> 7 swDisproportionateCostAlterna~ "Conditional. Select from the enumeration list the al~
#> 8 swDisproportionateCostOtherEU~ "Conditional. Indicate whether the costs of basic mea~
#> 9 swTechnicalInfeasibility "Required. Report how ‘technical infeasibility’ has be~
#> 10 swNaturalConditions "Required. Select from the enumeration list the eleme~
#> 11 swExemption46 "Required. Select from the enumeration list the reason~
#> 12 swExemption47 "Required. Select from the enumeration list the modif~
#> 13 swExemptionsTransboundary "Required. Indicate whether the application of exempt~
#> 14 swExemptionsReference "Required. Provide references or hyperlinks to the re~
#> 15 driversSWExemptionsReference "Required. Provide references or hyperlinks to the re~