Interessante eccezione Lucene.net

Question

Secondo this o this, ho utilizzato lo stesso indexsearcher per thread multipli. Ma quando sono passato da FsDirectory a MMapDirectory, ho ottenuto interessanti eccezioni.

Questo lavoro bene:

static void Main(string[] args) 
{ 
    DirectoryInfo directoryInfo = new DirectoryInfo(@"C:\Users\Tams\Desktop\new\"); 
    var directory = FSDirectory.Open(directoryInfo); 
    var indexSearcher = new IndexSearcher(directory); 

    const int times = 100; 
    const int concurrentTaskCount = 5; 
    var task = new Task[concurrentTaskCount]; 
    for (int i = 0; i < concurrentTaskCount; i++) 
    { 
     task[i] = new Task(() => Search(indexSearcher, times)); 
     task[i].Start(); 
    } 

    Task.WaitAll(task); 
} 

static void Search(IndexSearcher reader, int times) 
{ 
    List<Document> docs = new List<Document>(10000); 
    for (int i = 0; i < times; i++) 
    { 
     var q = new TermQuery(new Term("title", "volume")); 
     foreach (var scoreDoc in reader.Search(q, 100).ScoreDocs) 
     { 
      docs.Add(reader.Doc(scoreDoc.Doc)); 
     } 
    } 
} 

Ma con questo:

static void Main(string[] args) 
{ 
    DirectoryInfo directoryInfo = new DirectoryInfo(@"C:\Users\Tams\Desktop\new\"); 
    var directory = new MMapDirectory(directoryInfo); // CHANGED 
    var indexSearcher = new IndexSearcher(directory); 

    const int times = 100; 
    const int concurrentTaskCount = 5; 
    var task = new Task[concurrentTaskCount]; 
    for (int i = 0; i < concurrentTaskCount; i++) 
    { 
     task[i] = new Task(() => Search(indexSearcher, times)); 
     task[i].Start(); 
    } 

    Task.WaitAll(task); 
} 

static void Search(IndexSearcher reader, int times) 
{ 
    List<Document> docs = new List<Document>(10000); 
    for (int i = 0; i < times; i++) 
    { 
     var q = new TermQuery(new Term("title", "volume")); 
     foreach (var scoreDoc in reader.Search(q, 100).ScoreDocs) 
     { 
      docs.Add(reader.Doc(scoreDoc.Doc)); 
     } 
    } 
} 

ricevo diverse eccezioni come:

System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative 
            and less than the size of the collection. 
Parameter name: index 
at System.ThrowHelper.ThrowArgumentOutOfRangeException() 
at System.Collections.Generic.List`1.get_Item(Int32 index) 
at Lucene.Net.Index.FieldInfos.FieldInfo(Int32 fieldNumber) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldInfos.cs:line 378 
at Lucene.Net.Index.FieldsReader.Doc(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldsReader.cs:line 234 
at Lucene.Net.Index.SegmentReader.Document(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 1193 
at Lucene.Net.Index.DirectoryReader.Document(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DirectoryReader.cs:line 686 
at Lucene.Net.Index.IndexReader.Document(Int32 n) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexReader.cs:line 732 
at Lucene.Net.Search.IndexSearcher.Doc(Int32 i) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 162 
at PerformanceTest.Program.Search(IndexSearcher reader, Int32 times) 
    in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 28 
at PerformanceTest.Program.<>c__DisplayClass2.<Main>b__0() 
    in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 43 
at System.Threading.Tasks.Task.InnerInvoke() 
at System.Threading.Tasks.Task.Execute() 

O

System.IO.IOException: read past EOF 
at Lucene.Net.Store.BufferedIndexInput.Refill() 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Store\BufferedIndexInput.cs:line 179 
at Lucene.Net.Store.BufferedIndexInput.ReadByte() 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Store\BufferedIndexInput.cs:line 41 
at Lucene.Net.Store.IndexInput.ReadVInt() 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Store\IndexInput.cs:line 88 
at Lucene.Net.Index.FieldsReader.Doc(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\FieldsReader.cs:line 230 
at Lucene.Net.Index.SegmentReader.Document(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\SegmentReader.cs:line 1193 
at Lucene.Net.Index.DirectoryReader.Document(Int32 n, FieldSelector fieldSelector) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\DirectoryReader.cs:line 686 
at Lucene.Net.Index.IndexReader.Document(Int32 n) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Index\IndexReader.cs:line 732 
at Lucene.Net.Search.IndexSearcher.Doc(Int32 i) 
    in d:\Lucene.Net\FullRepo\trunk\src\core\Search\IndexSearcher.cs:line 162 
at PerformanceTest.Program.Search(IndexSearcher reader, Int32 times) 
    in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 28 
at PerformanceTest.Program.<>c__DisplayClass2.<Main>b__0() 
    in c:\Users\Tams\Documents\Visual Studio 2012\Projects\BookCatalog\PerformanceTest\Program.cs:line 43 
at System.Threading.Tasks.Task.InnerInvoke() 
at System.Threading.Tasks.Task.Execute() 

L'ultimo codice funziona correttamente, con l'impostazione della variabile concurrentTaskCount su 1.

Mi manca qualcosa? Non riesco a capire cosa sia.

In realtà, io non sono il percorso

d: \ Lucene.Net \ FullRepo tronco \ src \ Core \ Store \ \ BufferedIndexInput.cs

non nemmeno un guidare con la lettera "d"

Answer 1

Il source for MMapDirectory indica che questa classe non utilizza memory-mapped files, come previsto. Carica tutti i file di indice in memoria utilizzando oggetti MemoryStream e suppongo che tali flussi siano la causa del problema quando i thread diversi cercano e leggono.

È possibile ottenere un indice basato su memoria caricandolo in una RAMDirectory. Questo supera il tuo test. (Ma fa quello MMapDirectory momento fa, non necessariamente quello che ci si aspetta che faccia ...)

var fsDirectory = FSDirectory.Open(directoryInfo); 
var directory = new RAMDirectory(fsDirectory);