Solr : Autolink on body from words dictionnary - dictionary

Question

I'm looking for to generate auto link in body result in solr. Words on link must be in a dictionnary.
For example :
a doc :
<doc>
[...]
<str name="title">Il faut, quand on gouverne, voir les hommes tels qu’ils sont, et les choses telles qu’elles devraient être.</str>
<str name="path">citation/faut-gouverne-voir-hommes-tels-choses-telles-devraient-etre-15.php</str>
<str name="ss_field_citation_keywords">#faut#gouverne#voir#hommes#tels#choses#telles#devraient#etre#</str>
[...]
</doc>
Body from title to display :
Il faut, quand on gouverne, voir les hommes tels qu’ils sont, et les choses telles qu’elles devraient être.
Links from ss_field_citation_keywords :
#faut#gouverne#voir#hommes#tels#choses#telles#devraient#etre#
Body must be display like this :
Il faut, quand on gouverne, voir les hommes tels qu’ils sont, et les choses telles qu’elles devraient être.
Il faut, quand on gouverne, voir les hommes tels qu’ils sont, et les choses telles qu’elles devraient être
Do you have any idea?

Answer 1

You have two phases here:
Identify the keywords. For this you want to build your analyzer chain properly. Whitespace tokenizer, lowercase filter and - that's the key part - KeepWordFilterFactory . This will make Solr keep only your keywords with offsets in the text.
Get those offsets. There is might be several ways, but one of them is to reuse Field Analyzer which you can experiment with in admin WebUI of latest (4+) Solr. Make sure to check the Verbose box. That uses /analysis/field end point and you can use it too (with verbose flag). The result is probably too verbose for you but good enough to start. Then you can look for better implementation or copy/reduce the one currently done.

Answer 2

a proposal for internal processing with velocity and a java class
public class autoLinkCitationDirective extends Directive{
public String getName() {
return "autolinkcitation";
}
public int getType() {
return LINE;
}
public boolean render(InternalContextAdapter context, Writer writer, Node node)
throws IOException, ResourceNotFoundException, ParseErrorException, MethodInvocationException {
String CitationMe = null;
String KeyWords = null;
String SchemaUrl = null;
//params
if (node.jjtGetChild(0) != null) {
CitationMe = String.valueOf(node.jjtGetChild(0).value(context));
}
if (node.jjtGetChild(1) != null) {
KeyWords = String.valueOf(node.jjtGetChild(1).value(context));
}
//schema url
if (node.jjtGetChild(2) != null) {
SchemaUrl = String.valueOf(node.jjtGetChild(2).value(context));
}
writer.write(autoLinkCitation(CitationMe, KeyWords, SchemaUrl));
return true;
}
public String autoLinkCitation(String CitationMe, String KeyWords, String SchemaUrl) {
if (CitationMe == null) {
return null;
}
List<String> tokens = new ArrayList<String>();
StringTokenizer stkKeyWords = new StringTokenizer(KeyWords, "#");
while ( stkKeyWords.hasMoreTokens() ) {
tokens.add(stkKeyWords.nextToken());
}
String patternString = "\\b(" + StringUtils.join(tokens, "|") + ")\\b";
Pattern pattern = Pattern.compile(patternString);
String strippedHtml = CitationMe.replaceAll("<(.|\n)*?>", "");
StringTokenizer st = new StringTokenizer(strippedHtml, ".,! ()[]");
while (st.hasMoreTokens())
{
String token = st.nextToken().trim();
if (token.length() > 3)
{
Matcher matcher = pattern.matcher(cleanString(token));
while (matcher.find()) {
if(CitationMe.indexOf( SchemaUrl + cleanString(token) + "'") == -1)
{
String tmpStringreplacement = "<a href='" + SchemaUrl + cleanString(token) + "'>"+token+"</a>";
CitationMe = CitationMe.replaceAll("\\b"+token+"\\b(?!/)",tmpStringreplacement);
}
}
}
}
return CitationMe;
}
public String cleanString(String CleanStringMe) {
if (CleanStringMe == null) {
return null;
}
CleanStringMe = Normalizer.normalize(CleanStringMe, Normalizer.Form.NFD).replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
CleanStringMe = CleanStringMe.toLowerCase();
CleanStringMe = CleanStringMe.replaceAll("[^A-Za-z0-9]", "-");
return CleanStringMe;
}
}
and to display:
#autolinkcitation($doc.getFieldValue('body'),$doc.getFieldValue('ss_field_citation_keywords'), '/citations/mot.php?mot=' )