ThaiTokeniser

Namespace

Rowles.LeanCorpus.Analysis.Tokenisers

Assembly

Rowles.LeanCorpus.dll

Lightweight Thai tokeniser using greedy longest-match segmentation with a user-supplied lexicon and grapheme-cluster fallback for unknown runs.

public sealed class ThaiTokeniser : ISpanTokeniser

ThaiTokeniser

Implements

ISpanTokeniser

Remarks

The lexicon must be provided via the constructor, FromFile(string), or FromStream(Stream). A starter lexicon is available in the repository under lexicons/thai-dict.txt.

ThaiTokeniser(IEnumerable<string>)

Initialises a new ThaiTokeniser with the supplied lexicon.

ThaiType

Token type emitted for Thai segments.

_lexicon

_maxWordLength

FromFile(string)

Loads a UTF-8 text lexicon from disk, using one word per line. Lines starting with # are ignored.

FromStream(Stream)

Loads a UTF-8 text lexicon from a stream, using one word per line. Lines starting with # are ignored. The stream is not disposed.

ReadThaiCluster(ReadOnlySpan<char>, int, int)

Tokenise(ReadOnlySpan<char>, ISpanTokenSink)

Splits the input text and emits tokens into the supplied sink.

TokeniseThaiRun(ReadOnlySpan<char>, int, int, ISpanTokenSink)

TryFindLongestLexiconMatch(ReadOnlySpan<char>, int, int)