Extracting text fragment from a HTML body (in .NET) - asp.net

I have an HTML content which is entered by user via a richtext editor so it can be almost anything (less those not supposed to be outside the body tag, no worries about "head" or doctype etc).
An example of this content:
<h1>Header 1</h1>
<p>Some text here</p><p>Some more text here</p>
<div align=right>A link here</div><hr />
<h1>Header 2</h1>
<p>Some text here</p><p>Some more text here</p>
<div align=right>A link here</div><hr />
The trick is, I need to extract first 100 characters of the text only (HTML tags stripped). I also need to retain the line breaks and not break any word.
So the output for the above will be something like:
Header 1
Some text here
Some more text here
A link here
Header 2
Some text here
Some
It has 98 characters and line breaks are retained. What I can achieve so far is to strip the all HTML tags using Regex:
Regex.Replace(htmlStr, "<[^>]*>", "")
Then trim the length using Regex as well with:
Regex.Match(textStr, #"^.{1,100}\b").Value
My problem is, how to retaining the line break?. I get an output like:
Header 1
Some text hereSome more text here
A link here
Header 2
Some text hereSome more text
Notice the joining sentences? Perhaps someone can show me some other ways of solving this problem. Thanks!
Additional Info: My purpose is to generate plain text synopsis from a bunch of HTML content. Guess this will help clarify the this problem.

I think how I would solve this is to look at it as though it were a simple browser. Create a base Tag class, make it abstract with maybe an InnerHTML property and a virtual method PrintElement.
Next, create classes for each HTML tag that you care about and inherit from your base class. Judging from your example, the tags you care most about are h1, p, a, and hr. Implement the PrintElement method such that it returns a string that prints out the element properly based on the InnerHTML (such as the p class' PrintElement would return "\n[InnerHTML]\n").
Next, build a parser that will parse through your HTML and determine which object to create and then add those objects to a queue (a tree would be better, but doesn't look like it's necessary for your purposes).
Finally, go through your queue calling the PrintElement method for each element.
May be more work than you had planned, but it's a far more robust solution than simply using regex and should you decided to change your mind in the future and want to show simple styling it's just a matter of going back and modifying your PrintElement methods.

For info, stripping html with a regex is... full of subtle problems. The HTML Agility Pack may be more robust, but still suffers from the words bleeding together:
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
string text = doc.DocumentNode.InnerText;

One way could be to strip html in three steps:
Regex.Replace(htmlStr, "<[^/>]*>", "") // don't strip </.*>
Regex.Replace(htmlStr, "</p>", "\r\n") // all paragraph ends are replaced w/ new line
Regex.Replace(htmlStr, "<[^>]*>", "") // replace remaining </.*>

Well, I need to close this though not having the ideal solution. Since the HTML tags used in my app are very common ones (no tables, list etc) with little or no nesting, what I did is to preformat the HTML fragments before I save them after user input.
Remove all line breaks
Add a line break prefix to all block tags (e.g. div, p, hr, h1/2/3/4 etc)
Before I extract them out to be displayed as plain-text, use regex to remove the html tag and retain the line-break. Hardly any rocket science but works for me.

Related

How to avoid broken thematic sections (eg. div) in HTML?

I am trying to transfer a text from a printed book into HTML5, but meanwhile I am trying to keep its thematic and page/paragraph/lines layout structure exactly as it is. For example, every page of the printed book is divided as a <div> section eg. <div class=page id=55> so that it emulates/represents exactly the page unit of the printed book, and also facilitate referencing. I don't care much how the text will be rendered on the browser, this is something that I can think about later. I just want the HTML and the browser to "know" the original pagination and layout of the printed book.
The problem is that in the printed book, some paragraphs or even boxes, tables etc span over to the next page. If I translate it to HTML, I do it like this:
<div class=page id=1>
<p>Once upon a time...</p>
...
<p>...and so the bold knight
</div>
<div class=page id=2>
slew the evil dragon.</p>
<p>Text...</p>
...
This is illegal in HTML, as we have a <p> tag being interrupted by a </div> tag, and then a new div element beginning with a plain text, which is closed by a </p> tag.
HTML would expect me to close the first part of the broken paragraph with a </p>, and continue with a new <p> tag after the div, but I am not doing this because it doesn't correspond to the pagnation of the original book, and would result in half-paragraphs being understood are 2 proper paragraphs.
So, how to use legal HTML while maintaining the theoretical page/paragraph/broken paragraph/page break structure and information, or at least making the brower "know" the original pagination? Is there a more appropriate tag or method to emulate the page break while keeping the page number id?
Perhaps something like
<p>...and so the brave knight<some tag(s) that show page 2 begins here>killed the dragon</p>
How about instead of encapsulating each page within a div you include a tag at the start of each page designating the page number. An aside tag seems appropriate for this.
<aside class="page-number" data-page="1">Page 1</aside>
<p>Once upon a time...</p>
<p>...and so the bold knight</p>
<aside class="page-number" data-page="2">Page 2</aside>
<p class="continued">slew the evil dragon.</p>
<p>Text...</p>
If you need to continue a paragraph then you'll have to break into multiple elements, but perhaps you can specify when a paragraph is a continuation of a previous one. For instance using the continued class as shown above.
If you really don't want to break the p tag then you could put a span within it that is only used for semantic reasons. Something like this;
<p>...and so the bold knight
<span class="page-marker" aria-hidden="true" data-page="1"></span>
slew the evil dragon.</p>
But this kind of makes less semantic sense than the previous solution.
Try adding display: inline; to either the CSS style of the class page or the style attribute of each page div.

xpath/css href not printing

I am trying to print the href of a html doc, however I am not able to do so.
newurl = 'http://www.heroesfire.com/hots/guide/the-many-ways-of-abathur-1194'
buildpage = Nokogiri::HTML(open(newurl))
#puts buildpage
thistext = buildpage.css("div#wrap div#site-content.self-clear div#guide.view-guide div.col-l div.tab-contents.box div.guide-tab div.chapter-text div.text table.bbcode_columns tbody tr td.bbcode_column a").each do |href|
puts href['href']
end
I am expecting to see '/hots/wiki/talents/pressurized-glands'
I was able to get something similar to work earlier in my script, but I am having zero luck with this.
Invariably, the longer the Node selector, the less likely it will work correctly, especially if you're dealing with HTML you don't control.
Reduce it to find way-points, places that help you drill down instead of trying to define each step.
You're also relying on tbody in the selector. When we see that, the odds are good that it's not in the original HTML source but instead was injected by your browser. Selectors like that smell of using a browser and an inspector to locate a particular item in the page, but the resulting path won't work if the HTML doesn't actually contain tbody. Browsers do a lot of fix-up in an attempt to present something useful, including adding tags. So be careful when you see tbody and confirm it actually exists. In your case, it does, but the concern still exists when navigating through a document.
A simple example of simplifying the path is:
require 'nokogiri'
doc = Nokogiri::HTML(<<EOT)
<html>
<body>
<div id="foo">
<div id="bar">
<p>text1</p>
</div>
<div id="baz">
<p>text2</p>
</div>
</div>
</body>
</html>
EOT
doc.at('body div#foo div#bar p').text # => "text1"
Can be written more easily, while still accomplishing the same thing, using:
doc.at('#bar p').text # => "text1"
or perhaps one of these:
doc.at('#foo div p').text # => "text1"
doc.search('#foo div p').first.text # => "text1"
All scraping requires at least some advance knowledge of the target page's structure, so, while you're nosing around, take note of the important layout tags. id parameters are especially useful, followed by class and/or unique patterns of tags not replicated elsewhere in the document. Those make it easy to reduce the selector. Sometimes we have to step into the document incrementally like I did using first or one of the "sibling" methods after locating a particular node, but using a long selector rarely is needed.

How do I select only a certain character of text and turn it into a break?

I will have users input text in a textbox to set as their identifier, however, they can only enter 1 line of text. I have no way of changing that.
I would like to add CSS that takes the string of text and edits a | character and changes it to a <br>
The string of text they will type will be something like this: 1234-5678-1234 | Jim
I want it to show up like this:
1234-5678-1234
Jim
I'm guessing the code might look like this:
p:contains('|') {code for an enter and float right}
I would be posting this as comment but I need 50 rep :)
Just this: What you are trying to do needs JS. You should give RegExp a try. There's not a way to do that using pure css.
It is not possible to select an element on the basis of its textual content, except for the special case of empty content. There was once (in 2001) a draft suggesting a :contains(...) selector, but this feature was removed as the draft progressed (to eventually become Selectors Level 3 recommendation).
Still less is there a way to select something inside an element based on its content.
Besides, adding <br> would not be possible. You cannot add tags or elements with CSS, only textual content via pseudo-elements.
Moreover, if the input is read in an input element, you cannot make its content displayed in two lines. If the user input is actually echoed an in different element, like p element, then it is programmatically copied there, so the question is why the change is made there. You can modify the content with JavaScript, and it would be rather simple to replace any | by <br> in the content of a p element.
You should give a try using <div contenteditable="true"></div>. You would be able to solve your issue, using JS by wrapping and adding tags as necessary. Also, textbox wont support multiline & formatting.
A good read for contenteditable attribute on MDN: https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Content_Editable
I did some testing and created the following RegExp: Example Here
<input type="text" id="demo" value="123-456-7890 | John Doe" size="35">
<button onclick="myFunction()">Try it</button>
<p style="text-align:left;" id="number"></p>
<p style="text-align:right;" id="name"></p>
<script>
function myFunction() {
var str = document.getElementById("demo").value;
strnum = str.indexOf('|');
var name = str.slice(strnum+1);
var number = str.slice(0,strnum);
document.getElementById("number").innerHTML = number;
document.getElementById("name").innerHTML = name;
}
</script>
Is this what you want?

xquery- how to get content of a node which is immediately after a node with known text

I am trying to extract content from a XHTML document-- in this document, within a div, there are a number of 'b' elements, each followed by a link.
For eg--
<div id="main">
<b> Bold text 1</b>
some link 1
<b> Bold text 2</b>
some link 2
<b> ABRACADABRA</b>
abracadbralink
</div>
Now, I want to extract the link 'abracadabralink'-- the problems are that, I dont know how many and elements are there before this specific link-- in different documents there are a different number of such elements- sometimes there are many links immediately after a single element-- all I do know is that the text for the element that occurs just before the link that I want, is always fixed.
So the only fixed information is that I want the link immediately after the element with known text-- how do I get this link using XQuery?
If I get it right, you are interested in the value of the #href attribute? This can be done with standard XPath syntax:
doc('yourdoc.xml')//*[. = ' abracadbralink']/#href/string()
For more information on XPath, I’d advise you to check out some online tutorials, such as http://www.w3schools.com/xpath/default.asp
I guess the following should work for you:
$yournode/b[. = ' ABRACADABRA']/following-sibling::a/#href/string()

How can i find extra closing div in html string

I have some html stored in database.
I dont know that html stored in databse has extra closing div like </div> or not.
I want to find extra closing div in html string.
I have tried to find using HTML Agility pack but not find the way to achieve this.
Example:
<div class="readers">
A total of 218 users are reading this article.
</div>
</div>
</div>
How can i find these two extra closing div and extract fully valid html.
Use this pure javascript parser before rendering the html: http://ejohn.org/blog/pure-javascript-html-parser/
You can check out by pasting your code here,
http://ejohn.org/apps/htmlparser/
it removes the extra </div>s.
You just need to pass your html to the HTMLtoXML function as:
HTMLtoXML(your_html);
and it would remove the extra closing tags. Infact what it does is that it converts it into xml format, but since you are dealing with html strigs & all tags are expected to be valid in html, you can be safe to use this.
EDIT: You can easily call javascript functions from a C# file. See this question for more details.
Click here to find both unclosed (hanging) as well as extra div tags: tormus

Resources