Analysers

An analyser is the top-level pipeline: tokenise, normalise, filter, output terms.

flowchart LR
    I[Input text] --> C[Character filters]
    C --> T[Tokeniser]
    T --> N[Lowercase and normalisation]
    N --> S[Stop words and selection]
    S --> Y[Synonyms or shingles]
    Y --> M[Stemmer]
    M --> O[Indexed terms, positions, offsets, payloads]

The exact order is configured by the analyser. Changing it changes indexed terms and normally requires reindexing existing content.

Built-in analysers

Type	Pipeline	When
`StandardAnalyser`	Basic tokenise, lowercase, stop words	General-purpose text
`StemmedAnalyser`	`StandardAnalyser` + Porter stemming	English text, broader recall
`LanguageAnalyser`	Tokenise, lowercase, stop words, optional stemmer	Language-specific with custom tokeniser/stemmer
`IcuAnalyser`	`IcuTokeniser`, lowercase, stop words, optional Thai	Unicode-heavy text
`WhitespaceAnalyser`	`WhitespaceTokeniser` only	Punctuation and case should stay
`KeywordAnalyser`	`KeywordTokeniser` only	Whole field as one token
`SimpleAnalyser`	Letter-only tokenise, lowercase	Ignore digits and punctuation
`Analyser`	Your tokeniser and filters	Custom pipeline

Three starting points

using Rowles.LeanCorpus.Analysis;

var standard = new StandardAnalyser();
var stemmed  = new StemmedAnalyser();
var french   = AnalyserFactory.Create("fr");

AnalyserFactory languages

AnalyserFactory.Create(string) accepts BCP 47 codes. Region and script subtags are stripped (en-GB becomes en).

Supported: en, fr, de, es, it, pt, nl, ru, ar, zh, ja, ko, sk. Chinese uses lexicon-based longest-match segmentation, Japanese uses dictionary-backed Viterbi segmentation, and Korean keeps Hangul word runs intact. These analysers apply their language stop-word lists and skip stemming. AnalyserFactory.Create("en") uses EnglishStemmer.

Per-field override

Set the default on IndexWriterConfig.DefaultAnalyser. Override per-field by attaching an IAnalyser to a FieldMapping inside an IndexSchema.

Inspect tokens