Adding one to many data to Lucene Index - asp.net

Question

I am new to Lucene. I am trying to create an index of records. So far, I have just been adding one-to-one data to my index, and this seems fine. But, I have scenarios where I need to add one to many relationship data, and I am not sure what is the best way to handle this. I have tried adding each of the individual relationship, concacting feilds into CSV values, adding the field multiple times, but nothing seems to work. Here is my code for when the data is indexed:
Private Shared Sub _addToLuceneIndex(ByVal sampleData As LuceneSearchData, ByVal writer As IndexWriter)
Dim searchQuery = New TermQuery(New Term("Id", sampleData.Id.ToString()))
writer.DeleteDocuments(searchQuery)
Dim doc = New Document()
doc.Add(New Field("Id", sampleData.Id.ToString(), Field.Store.YES, Field.Index.NOT_ANALYZED))
doc.Add(New Field("Name", sampleData.Name, Field.Store.YES, Field.Index.ANALYZED))
doc.Add(New Field("Description", sampleData.Description, Field.Store.YES, Field.Index.ANALYZED))
For Each item As Integer In sampleData.HomeStates
doc.Add(New Field("Home_State", item, Field.Store.YES, Field.Index.ANALYZED))
Next
'i have also tried
'doc.Add(New Field("HomeStates ", String.Join(",", sampleData.HomeStates ), Field.Store.YES, Field.Index.ANALYZED))
writer.AddDocument(doc)
End Sub
While the above code seems to index the one-to-one data, it does not for the HomeStates list of integers. Do I have to add the same document for every item in the list of integers? If so, how do you best manage this? I have several "one-to-many" relationships I need to include. I can see this getting unwieldy quickly. Or, is there a better way?
EDIT
I updated to add the one to may value as a field like this:
doc.Add(New Field("Geo_Locations", String.Join(" ", sampleData.Geo_Location), Field.Store.YES, Field.Index.ANALYZED))
This is how I search for that field:
Private Shared Function _search(ByVal searchQuery As String, ByVal Optional searchField As String = "") As IEnumerable(Of LuceneSearchData)
If String.IsNullOrEmpty(searchQuery.Replace("*", "").Replace("?", "")) Then Return New List(Of LuceneSearchData)()
Using searcher = New IndexSearcher(_directory, False)
Dim hits_limit = 1000
Dim analyzer = New StandardAnalyzer(Version.LUCENE_30)
If Not String.IsNullOrEmpty(searchField) Then
Dim parser = New QueryParser(Version.LUCENE_30, searchField, analyzer)
Dim query = parseQuery(searchQuery, parser)
Dim hits = searcher.Search(query, hits_limit).ScoreDocs
Dim results = _mapLuceneToDataList(hits, searcher)
analyzer.Close()
searcher.Dispose()
Return results
End If
End Using
End Function
Private Shared Function _mapLuceneToDataList(ByVal hits As IEnumerable(Of ScoreDoc), ByVal searcher As IndexSearcher) As IEnumerable(Of LuceneSearchData)
Dim listOfResults As List(Of LuceneSearchData)
Try
listOfResults = hits.[Select](Function(hit) _mapLuceneDocumentToData(searcher.Doc(hit.Doc))).ToList()
Catch ex As Exception
Return Nothing
End Try
Return listOfResults
End Function
Private Shared Function _mapLuceneDocumentToData(ByVal doc As Document) As LuceneSearchData
Return New LuceneSearchData With {
.Id = Convert.ToInt32(doc.[Get]("Id")),
.Mechanism_Name = doc.[Get]("Name"),
.Mechanism_Purpose = doc.[Get]("Description"),
.Geo_Location = doc.[Get]("Home_State")
}
End Function
I then call the search bu:
LuceneData = LuceneSearch.Search("5451", "HomeStates")

Answer 1

Creating a new document for each item will result in duplicates when searching for other fields.
You should design your documents & fields according to your needs.
If you don't need to search for these fields, store them as you wish.
If you need to search these multiple valued fields then create a field that can be searched for each item.
For integer lists you can merge them with spaces.
For string lists you should replace white spaces with some custom character while indexing and searching to avoid matching substrings (eg "stack overflow" => stack_overflow)

Answer 2

One to many using a term
As you have figured out, integers can be concatenated using a variety of separators including spaces, resulting in something like:
"234 12342 345 5476456 234"
The StandardAnalyzer will tokenise the above string into separate tokens just as it would any other string of words. So you can search them and it will give you the expected results.
One to many using a phrase
If you need to tokenise phrases, such as:
"Control Support Engineer|Technical Support Engineer|Maintenance Technician"
into individual phrases, not down to words (terms), then you'll need to inherit from a few classes. A tokenizer that inherits from CharTokenizer and an analyzer that inherits from Analyzer.
Tokenizer
public sealed class PipeTokenizer: CharTokenizer
{
public PipeTokenizer(LuceneVersion matchVersion, TextReader input) :
base(matchVersion, input)
{
}
public PipeTokenizer(LuceneVersion matchVersion, AttributeFactory factory, TextReader input) :
base(matchVersion, factory, input)
{
}
protected override bool IsTokenChar(int c)
{
return !((char)c).Equals('|'); //<-- the only line that matters
}
}
Pretty simple stuff so far.
Analyzer
public class PipeAnalyzer: Analyzer
{
protected override TokenStreamComponents CreateComponents(string fieldName, TextReader reader)
{
var tokenizer = new PipeTokenizer(LuceneVersion.LUCENE_48, reader);
var lowerCaseFilter = new LowerCaseFilter(LuceneVersion.LUCENE_48, tokenizer);
return new TokenStreamComponents(tokenizer, lowerCaseFilter);
}
}
This is based on what the KeywordAnalyzer does but with a twist. The KeywordAnalyzer is very simple. It takes the entire string, or phrase, and indexes it as is. No further breaking down into smaller tokens. The analyzer above uses the PipeTokenizer to break phrases down by the | separator, and converts to lower case.
Querying with any of the following phrases will return the doc:
"control support engineer"
"technical support engineer"
"maintenance technician"