ITokeniser

Namespace

Rowles.LeanLucene.Analysis.Tokenisers

Assembly

Rowles.LeanLucene.dll

Splits input text into raw tokens.

public interface ITokeniser

Methods

Tokenise(ReadOnlySpan<char>)

Splits the input text into a list of tokens at word boundaries.