Storing data from a for loop in a data frame - r

Question

I am trying to create a function that interacts with the pubmed api to retrieve xml files associated with 100 publications. I then want to parse the xml files individually to retrieve the title of each publication and the abstract of each publication. I am using the Rentrez package to interact with the api, and have successfully retrieved the necessary xml files. I am using the xml package to parse the xml files, and have verified that the Xpath expressions retrieve the data that I want. In truth, I am looking to take data from other fields (journal title, Mesh Terms, etc., but I am stuck at this step here)
However, I have not been able to create a proper for loop to move this data into a data frame. I receive the following error from running my code:
error in $<-.data.frame(*tmp*, "Abstract", value = list("text of abstract"):
replacement has 1 row, data has 0
When I test the function to receive title information (by removing the expression to retrieve abstract information), I receive an empty data frame with no information about the titles that I want. But there is no error message then.
If I execute pubmed_parsed("Kandel+Eric", n=2), my goal is to receive a data frame with the character vectors from two titles in the column "ATitle" (titles: "Roles for small noncoding RNAs in silencing of retrotransposons in the mammalian brain" and "ApCPEB4, a non-prion domain containing homolog of ApCPEB, is involved in the initiation of long-term facilitation"). And the character vectors from the two abstracts to correspondingly appear in the column "Abstract" (portions of abstracts: "Piwi-interacting RNAs (piRNAs), long thought to be restricted to gremlin...", "Two pharmacologically distinct types of local protein synthesis are required for synapse- specific...").
library(xml)
library(rentrez)
pubmed_parsed <- function(term, n=100){
df <- data.frame(ATitle = character(), JTitle = character(), MeshTerms = character(), Abstract = character(), FAuthor = character(), LAuthor = character(), stringsAsFactors = FALSE)
IdList <- entrez_search(db = "pubmed", term = term, retmode = "xml", retmax = n)
for (i in 1:n){
XmlFile <- entrez_fetch(db = "pubmed", id=IdList$ids[i], rettype = "xml", retmode = "xml", parsed=TRUE)
Parsed <- xmlRoot(XmlFile)
df$ATitle[i] <- xpathSApply(Parsed, "/PubmedArticleSet/PubmedArticle/MedlineCitation/Article/Title", xmlValue, simplify = FALSE)
df$Abstract[i] <- xpathSApply(Parsed, "/PubmedArticleSet/PubmedArticle/MedlineCitation/Article/Title", xmlValue, simplify = FALSE)
}
df
}

Answer 1

Here's one way to get a table and a few suggestions. First, I would use the Web history option and download all results together instead of looping through downloads.
ids <- entrez_search(db = "pubmed", term = "Kandel ER", use_history = TRUE)
ids
Entrez search result with 502 hits (object contains 20 IDs and a web_history object)
Search term (as translated): Kandel ER[Author]
doc <- entrez_fetch(db="pubmed", web_history=ids$web_history, rettype="xml", retmax = 3, parsed=TRUE)
Next, get the articles into a node set and query that to handle all your missing and multiple tags.
articles <- getNodeSet( doc, "//PubmedArticle")
length(articles)
[1] 3
articles[[1]]
<PubmedArticle>
<MedlineCitation Status="Publisher" Owner="NLM">
<PMID Version="1">27791114</PMID>
<DateCreated>
...
I usually create a function to add NAs if tags are missing and join multiple tags using a comma.
xpath2 <-function(x, path, fun = xmlValue, ...){
y <- xpathSApply(x, path, fun, ...)
ifelse(length(y) == 0, NA,
ifelse(length(y) > 1, paste(unlist(y), collapse=", "), y))
}
Then just apply that function to the nodes (with the leading dot in xpath so it's relative to that node). This will combine multiple keywords into a comma-separated list and include NA for article 3 with missing keywords.
sapply(articles, xpath2, ".//Keyword")
[1] "DNA methylation, behavior, endogenous siRNA, piwi-interacting RNA, transposon"
[2] "Aplysia, CPEB, CPEB4, Long-term facilitation"
[3] NA
Most xpath should work
sapply(articles, xpath2, ".//PubDate/Year")
[1] "2016" "2016" "2016"
sapply(articles, xpath2, ".//ArticleId[#IdType='pmc']")
[1] "PMC5111663" "PMC5075418" NA
You can also use xmlGetAttr if needed
sapply(articles, xpath2, ".//Article", xmlGetAttr, "PubModel")
[1] "Print-Electronic" "Electronic" "Electronic"
Finally, create a data.frame
data.frame(
ATitle = sapply(articles, xpath2, ".//ArticleTitle"),
JTitle = sapply(articles, xpath2, ".//Journal/Title"),
Keywords = sapply(articles, xpath2, ".//Keyword"),
Authors = sapply(articles, xpath2, ".//Author/LastName"),
Abstract = sapply(articles, xpath2, ".//AbstractText"))
I'm not sure what happened to MeSH terms, but I only see Keywords in the few examples I downloaded. Also, there are probably a few ways to get first and last authors. You could get both last name and initials (assuming both are always present) and replace the comma before the initials to get an Author string. Then split that to get first and last author or even print the first three below.
au <- sapply(articles, xpath2, ".//Author/LastName|.//Author/Initials")
au <- gsub(",( [A-Z]+,?)", "\\1", au)
authors_etal <- function(x, authors=3, split=", *"){
y <- strsplit(x, split)
sapply(y, function(x){
if(length(x) > (authors + 1)) x <- c(x[1:authors], "et al.")
paste(x, collapse=", ")
})
}
authors_etal(au)
[1] "Nandi S, Chandramohan D, Fioriti L, et al."
[2] "Lee SH, Shim J, Cheong YH, et al."
[3] "Si K, Kandel ER"