Native tokenization

aner.token turns documents into reversible byte or learned byte BPE token sequences for native models. It preserves document boundaries, tokenizer identity and training data provenance. Use it with aner.gen for a complete native language model experiment, or use its IDs in your own model built with aner.nn and aner.tensor.

The from scratch guide explains the research process. The complete executable workflow checks tokenization, training, exact resume, causality and generation. These APIs require a native development runtime that includes aner.token and aner.gen; earlier development executables may not include them.

Train only on the training partition

Choose document, subject, experiment or time partitions before fitting a tokenizer. This example fits BPE only on the training documents, then applies the same codec to validation documents:

import aner.token

let documents = List("temperature α pressure β", "pressure β temperature α")
let fitting = token.corpus(token.byte_tokenizer(), documents, role: "training")
let codec = token.bpe(fitting, vocabulary: 280, min_frequency: 2)
let training = token.corpus(codec, documents, role: "training")
let validation = token.corpus(codec, List("new observation α β"), role: "validation")
let ids = codec.encode("scientific α β 🔬", bos: true, eos: true)
print(codec.decode(ids, skip_special: true))

Byte tokenization has no fitted vocabulary. BPE learns ordered byte pair merges; its original byte fallback remains available for previously unseen text. Neither algorithm normalizes Unicode or silently changes scientific symbols. One Unicode character can occupy several IDs.

API reference

Import the single module aner.token. Tokenizer algorithms are values and functions within it, rather than separate imports.

Call	Contract
`token.byte_tokenizer()`	Fixed byte codec: byte IDs `0..255`, BOS `256`, EOS `257`, PAD `258`.
`token.corpus(codec, documents, role)`	Snapshot a nonempty List of Strings. Insert one BOS/EOS per document; role is `"training"`, `"validation"` or `"test"`.
`token.files(codec, paths, role, max_bytes)`	Read each regular file as one document under an aggregate source byte budget. Files are not split on newline; symlinks are rejected.
`token.bpe(fitting, vocabulary, min_frequency)`	Fit deterministic byte BPE on a training role corpus. Vocabulary is an upper bound including byte/special IDs; fitting may stop earlier.
`codec.encode(text, bos, eos)`	Return token IDs with explicit boundary token choices.
`codec.decode(ids, skip_special)`	Decode to a String with strict `UTF-8` validation and an explicit special token policy.
`codec.encode_bytes(bytes, bos, eos)`	Encode arbitrary Bytes without interpreting them as text.
`codec.decode_bytes(ids, skip_special)`	Recover Bytes even when generated output is not complete `UTF-8`.
`codec.vocabulary()`, `codec.fingerprint()`	Inspect vocabulary size and exact algorithm/merge/provenance identity.
`corpus.role()`, `corpus.documents()`, `corpus.tokens()`, `corpus.summary()`	Inspect the role, document count, total stored IDs and bounded provenance summary.
`codec.save(path)`, `token.load(path)`	Publish/load a versioned tokenizer artifact. Saving requires a new path.
`token.fingerprint(codec_or_corpus)`	Inspect the tokenizer or ordered corpus fingerprint.

codec.encode does not interpret spellings such as <eos> as special tokens in native byte/BPE text. BOS/EOS insertion is controlled by arguments. Decoding a generated byte sequence can fail UTF-8 validation when generation stops inside a character; decode_bytes retains the exact result.

Identity, partitions and artifacts

BPE chooses the most frequent adjacent pair, breaks ties lexicographically and applies ordered merges left to right without overlap. A merge never crosses document boundaries or a special token. New merge IDs start at 259.

The codec stores its ordered merges, algorithm version and fitting document hashes. Constructing a validation/test corpus rejects exact document overlap with tokenizer fitting data. Model training and held out evaluation add their own lineage checks. These checks do not detect paraphrases, shared subjects or related experiments: choose the scientific partition policy before calling the API.

A corpus is an immutable in memory snapshot. Editing a source file later does not change that value; constructing a new corpus reads it again. Resume requires the same encoded corpus identity and order. Matching vocabulary size alone does not make different tokenizers compatible.

Tokenizer artifacts contain data and checksums. Checkpoints saved by aner.gen also retain the complete tokenizer; model.tokenizer() recovers it. Artifact publication does not replace an existing path.

Current research bounds

Native BPE fitting is bounded and in memory: requested vocabulary 259..8192, at most 4,000,000 raw corpus bytes including boundaries, 1,000,000 active pair types and a 100,000,000 operation fitting budget. Expanded decoded output is bounded to 4,000,000 bytes. The corpus limit is 100,000 documents. Exported language level ID Lists also have the existing 262,144 value heap bound; use file/document corpora rather than one giant ID List.

The supported GPT 2 importer carries its own compatible tokenizer conventions. Use the imported model's tokenizer; it is distinct from fitted native BPE. See the generative module.

Executable checks

The workflow example asserts exact Unicode BPE roundtrip before building its language model. The dedicated BPE example uses the fitted codec throughout native training and adaptation. Run either in a fresh writable output directory; the complete guide gives the commands and explains the expected output.