R xpathSApply: get node text without getting text from its children - r

Question

I am trying to extract text from part of a website. The div node which contains the text also contains several children each with their own text or other content. However, I only want the text from the top node not from its children!
This is how the relevant page section looks like:
<div class="body-text">
<div id="other" class="other"></div>
<div id="other2" class="other2"></div>
<div id="other3" class="other3">
<span>irrelevant text</span>
</div>
<h2>heading2</h2>
-Text which I want to get. There are also text parts which are linked.
</div>
This is my code which gets me the "messy" text. I tried /text() but this will truncate my text whenever part of it is linked. So I cannot use it. I also tried something with /div/node()[not(self::div)] but have not managed to get it to work. Could anybody help?
webpage = getURL(url)
webpage <- readLines(tc <- textConnection(webpage)); close(tc)
pagetree <- htmlTreeParse(webpage, useInternalNodes = TRUE, encoding='UTF-8')
body <- xpathSApply(pagetree, "//div[#class='body-text']", xmlValue)

Answer 1

1) Posted example
Try searching for nodes of text() or a/text() within the body-text division, removing any trivial nodes that only contain white space:
## input
Text <- '<div class="body-text">
<div id="other" class="other"></div>
<div id="other2" class="other2"></div>
<div id="other3" class="other3">
<span>irrelevant text</span>
</div>
<h2>heading2</h2>
-Text which I want to get. There are also text parts which are linked.
</div>'
library(XML)
pagetree <- htmlTreeParse(Text, asText = TRUE, useInternalNodes = TRUE)
## process it - xpth is the Xpath expression and xpathSApply() runs it
trim <- function(x) gsub("^\\s+|\\s+$", "", x) # trim whitespace from start & end
xpth <- "( //div[#class='body-text']/text() |
//div[#class='body-text']/a/text() ) [ normalize-space() != '' ]"
txt <- trim(xpathSApply(pagetree, xpth, xmlValue))
The result is the following:
> txt
[1] "-Text which I want to get. There are also text parts which are linked."
2) Example provided by poster in comments. Using this as Text
Text <- '<div class="body-text"> text starts here
<a class="footnote" href="link"> text continues here <sup>1</sup> </a>
and continues here</div>'
and repeating the above code we get:
> txt
[1] "text starts here" "text continues here" "and continues here"
EDIT: Have modified the above based on comments by poster. Main change was the xpath expression, xpth and final point which illustrates the same code with the example provided by the poster in the comments.
EDIT: Have moved the filtering out of whitespace-only nodes from R to Xpath. This lengthens the Xpath expression a bit but eliminates the R Filter() step. Also simplified and reduced the presentation slightly.

Answer 2

There are a few possible solutions to this problem, but, first, it's necessary to clarify which nodes you want to select. You say:
I only want the text from the top node not from its children!
But this is not true! All of the element nodes found in the article text (e.g. a, em,, etc) are themselves children of the body-text div. What you really want to do is select all of the text found within a certain section of the div. Conveniently, your source document (linked in the comments above) contains comment nodes that mark the start and end of the article. They look like this:
<!-- inizio TESTO -->article text<!-- fine TESTO -->
In fact, you really only need the start marker, because there is no additional content after it.
Selecting text after the start marker
The following expression selects the desired nodes:
//div[#class='body-text']/comment()[.=' inizio TESTO ']/following::text()
Testing on the following stripped-down document:
<div class="body-text">
<div class="fb-like-button" id="fb-like-head"></div>
<h2><!-- inizio OCCHIELLO -->IRAN<!-- fine OCCHIELLO --></h2>
<h1><!-- title -->"A Isfahan colpito sito nucleare"<br/>Londra annuncia azioni dure<!-- fine TITOLO --></h1>
<h3><!-- summary -->Secondo il<em>Times</em>, fonti di intelligence...<br/><strong><br/></strong><!-- fine SOMMARIO --></h3>
<div class="sidebar">Sidebar text...</div>
<!-- inizio TESTO --><strong>TEHERAN</strong> - L'esplosione avvenuta
lunedì scorso in Iran a Isfahan <sup>1</sup> avrebbe colpito un
sito nucleare. Lo hanno riferito fonti dell'intelligence israeliana al quotidiano britannico <em>The Times</em>, secondo le
quali alcune immagini satellitari "mostrano chiaramente colonne di fumo e la distruzione" di una struttura nucleare di Isfahan.
Sale, intanto, la tensione con la Gran Bretagna: dopo <a href="http://www.repubblica.it" class="footnote">l'assalto all'
ambasciata britannica <sup>2</sup></a> ieri...<!-- fine TESTO -->
</div>
Returns the following text nodes:
[#text: TEHERAN]
[#text: - L'esplosione avvenuta
]
[#text: lunedì scorso in Iran a Isfahan ]
[#text: 1]
[#text: avrebbe colpito un
sito nucleare. Lo hanno riferito fonti dell'intelligence israeliana al quotidiano britannico ]
[#text: The Times]
[#text: , secondo le
quali alcune immagini satellitari "mostrano chiaramente colonne di fumo e la distruzione" di una struttura nucleare di Isfahan.
Sale, intanto, la tensione con la Gran Bretagna: dopo ]
[#text: l'assalto all'
ambasciata britannica ]
[#text: 2]
[#text: ieri...]
[#text:
]
This is a node-set, which you can iterate, etc. I do not know R, so I cannot provide those details.
Selecting text between the start and end markers
If there could be content after the end marker that should be excluded -- there isn't in the provided example -- then use the following expression:
//div[#class='body-text']//text()[preceding::comment()[.=' inizio TESTO '] and
following::comment()[.=' fine TESTO ']]
Selecting text between the start and end markers (Kayessian Formula)
Note that the previous expression can be expressed more directly as the intersection of two node-sets: 1) all text nodes after the start marker and; 2) all text node before the end marker. There is a general formula for performing intersection in XPath 1.0:
$set1[count(.|$set2)=count($set2)]
The general idea here, in English, is that if you add an element from $set1 into $set2 and the size of $set2 does not change, then that node must have already been in $set2. The set of all nodes from $set1 for which this is the case is the intersection of $set1 and $set2.
In your specific case:
$set1 = //div[#class='body-text']/comment()[.=' inizio TESTO ']/following::text()
$set2 = //div[#class='body-text']/comment()[.=' fine TESTO ']/preceding::text()
Putting it all together:
//div[#class='body-text']/comment()[.=' inizio TESTO ']/following::text()[
count(.|//div[#class='body-text']/comment()[.=' fine TESTO ']/preceding::text())
=
count(//div[#class='body-text']/comment()[.=' fine TESTO ']/preceding::text())]