Build a language model from scratch

Aner can initialize and train a language model from new weights using native tokenization, embeddings, causal attention, differentiation and optimization. This guide connects those mechanisms to a reproducible research workflow: partition documents, fit a tokenizer, define a decoder, train and evaluate it, save/resume, generate and adapt its weights.

The built in decoder and the small executable below represent one way to use Aner. They are an inspectable starting point: researchers can compose richer supported models using ordinary functions and classes. The fixture's text and dimensions make verification quick; they do not define a ceiling on Aner programs or demonstrate useful natural language quality. The earlier academic miniLLM project is historical motivation, rather than an implementation dependency or a model quality target.

These APIs are available in the native development runtime that includes aner.token and aner.gen. Earlier development executables may not include them. Install a compatible runtime through the installation guide. Training and generation run inside Aner's native numerical runtime. The complete example needs no model service or downloaded weights.

The four modules

aner.token and aner.gen group the new generative capabilities. Individual tokenizer algorithms, attention heads, sampling policies and scientific generators are functions/types in these modules. aner.dev provides optional observations during development.

aner.llm has a separate purpose: connecting to a selected external inference service. A response from such a service does not train native Aner weights. This tutorial uses the native construction path.

Run the complete example

Download generative_research_workflow.aner. It is standalone and uses CPU Float64 with fixed seeds. Replace /absolute/path/to/ below with the directory where you saved the source, and run each command from a fresh writable output directory:

mkdir llm-research-prod
cd llm-research-prod
aner check /absolute/path/to/generative_research_workflow.aner
aner /absolute/path/to/generative_research_workflow.aner --prod

For development inspection, run the same source in a separate fresh directory:

mkdir llm-research-dev
cd llm-research-dev
aner /absolute/path/to/generative_research_workflow.aner --dev reports/workflow

Use independent directory paths when starting these two runs. The example publishes tokenizer and model files, and saving never overwrites an existing artifact. Development recording observes the same mathematical operations, masks and random streams. Production uses those operations without recording the inspection report.

The source prints the training and validation losses at updates 0, 30, 60, 90 and 120, checks checkpoint reload and exact split continuation, verifies causal prefix isolation and seeded generation, then exercises full fine tuning, freezing and LoRA. Every check_... label must be followed by true; a failed assertion exits through a checked Result.unwrap error. Successful completion prints workflow_checks_passed.

Partition the research documents

Choose the unit of independence before fitting anything. For papers, patients, instruments, simulations or time series, document identity alone may be insufficient; keep related subjects/experiments in one partition. Reserve validation for model selection and test for the final declared evaluation. Record the source, permission to use it, partition rule and immutable document order.

For actual text files, a native file based setup can be:

import aner.token
import aner.gen
import aner.nn

let training_paths = List("data/train-001.txt", "data/train-002.txt")
let fitting = token.files(token.byte_tokenizer(), training_paths,
    role: "training", max_bytes: 1000000)
let codec = token.bpe(fitting, vocabulary: 1024, min_frequency: 2)
let training = token.files(codec, training_paths,
    role: "training", max_bytes: 1000000)
let validation = token.files(codec, List("data/validation-001.txt"),
    role: "validation", max_bytes: 1000000)
let testing = token.files(codec, List("data/test-001.txt"),
    role: "test", max_bytes: 1000000)

Provide your own distinct files before running this setup. Each file is one document; newlines do not split it. Keep the source files unchanged between tokenizer fitting and corpus construction. Corpora retain immutable encoded snapshots, individual content hashes and ordered fingerprints. The runtime rejects exact fitting/training overlap in held out documents. It cannot detect paraphrases or shared subjects, so these checks complement the research partition policy.

Fit and preserve the tokenizer

Byte BPE starts from reversible byte IDs and learns ordered frequent pair merges using only fitting. Its requested vocabulary is an upper bound, not a guaranteed count. A different tokenizer changes the meaning of model IDs even when its vocabulary has the same size.

let symbols = "scientific α β 🔬"
let ids = codec.encode(symbols, bos: true, eos: true)
print(codec.decode(ids, skip_special: true))
print(codec.fingerprint())
codec.save("experiment.anertoken")

Native tokenization performs no Unicode normalization. Corpus construction adds BOS/EOS per document. Generated byte sequences can stop inside a UTF-8 character; use decode_bytes to retain their exact bytes when text decoding would be incomplete. The token reference defines special token policies, fitting bounds and serialization.

The complete executable additionally demonstrates BPE roundtrip, then uses a fixed byte tokenizer for its learning trace. This keeps the vocabulary and loss units unchanged across every plotted update. Do not compare byte token and BPE token perplexities as though they use the same unit.

Initialize the decoder and inspect the budget

nn.configure("cpu", "Float64", "preserve")
let initial = gen.decoder(codec, context: 64, width: 32,
    heads: 4, layers: 2, seed: 42, dropout: 0.1)
print(initial.summary())
print(initial.memory_summary(2, 64, true))
print(initial.loss(validation))

These are newly initialized weights. width / heads is the dimension of each attention head. Learned token embeddings and learned positions enter repeated pre normalized attention/feed-forward blocks. Residual connections carry the block input forward; the output projection shares the token embedding parameter.

For attention, native batched operations form queries, keys and values and compute softmax(Q Kᵀ / sqrt(head_width) + mask) V. The implementation applies masking inside the attention operation. A causal mask excludes future positions; a separate padding mask excludes padded keys. Integer cross entropy ignores padded targets. These policies apply to gradients as well as outputs.

The reference decoder fixes its feed forward width at four times model width, uses GELU and LayerNorm, and has learned positions. Choosing another block structure is a custom architecture experiment. More training steps do not expand the declared context.

Memory/work admission checks run before step graphs are allocated. An estimate's fits result checks the runtime's conservative numeric buffer, tensor/work and graph budgets; it does not report available system RAM or guarantee allocation.

Train and measure post update losses

Continuing the file based setup:

let trained = initial.train(training, steps: 200, batch: 2,
    rate: 0.001, seed: 99, clip_norm: 1.0,
    warmup_steps: 20, weight_decay: 0.01)
print(trained.loss(training))
print(trained.loss(validation))
print(trained.perplexity(validation))
print(trained.trained_tokens())

A step means one optimizer update. The trainer creates shifted input/target windows within documents, shuffles windows deterministically, and excludes padding from the objective. It scores each transition once per epoch. Later windows reset their learned position indices: the objective has truncated context, with no implicit cross window memory.

The extended optimizer policy supplies global gradient clipping, linear warmup and decoupled matrix weight decay. Training uses dropout; evaluation disables it. last_loss() describes the final batch before its update. For a learning curve or final comparison, call loss(corpus) on the returned model, as the executable does.

The recorded trace comes from the complete executable, with its actual seeds, source identity and checks. Training loss decreases while validation loss rises after update 60; the full trace remains visible. These related synthetic strings validate mechanics, not generalization to independent natural language tasks. Their uniform baseline is log(259) nats per byte/special-token target, rather than a competitive model baseline. A research study needs its own independent data, fixed selection rule and substantive baselines.

Save and resume the exact run

trained.save("experiment-step200.anergen")
let restored = gen.load("experiment-step200.anergen")
let continued = restored.resume(training, steps: 100)
continued.save("experiment-step300.anergen")

A full checkpoint contains the tokenizer, architecture, named weights, Adam moments, training policy, global update/data cursor, scored token count, lineage and execution identity. The complete example verifies that 30 + 30 updates followed by save/load + 30 + 30 updates have exactly the same model fingerprint as an uninterrupted 120 update run.

Exact continuation requires the same corpus/order, backend, precision and computational implementation. A changed dataset, rate or seed is a new experiment, handled explicitly with fine_tune. Artifact integrity checks detect corruption; the data only checkpoint format does not execute model code.

Generate with an explicit sampling policy

let prompt = codec.encode("Research", bos: true, eos: false)
let continuation = continued.generate(prompt, max_new_tokens: 16,
    temperature: 0.8, top_k: 40, top_p: 0.95, seed: 7)
print(codec.decode_bytes(continuation, skip_special: true).hex())

Generation returns new IDs only and stops on EOS or the requested limit. Temperature zero selects greedy output; positive temperature uses a call local random seed. Prompt plus requested output must fit the declared context. Generation currently recomputes attention at each token, so this implementation does not claim cached or streaming inference.

The complete example compares generated IDs element by element for resumed and uninterrupted models with the same sampling seed. Causality is checked separately by changing future input IDs and requiring the complete earlier vocabulary logits to remain exactly equal.

Adapt the native weights

Use an adaptation corpus with the same tokenizer and an explicit training role:

let adaptation = token.files(codec, List("data/adaptation-001.txt"),
    role: "training", max_bytes: 1000000)
let full = continued.fine_tune(adaptation, steps: 50, batch: 2,
    rate: 0.0005, seed: 71)
let selected = continued.freeze(List("embedding.token", "embedding.position"))
let adapters = continued.lora(List("block.0.qkv", "block.0.attention.output"),
    rank: 4, alpha: 8.0, seed: 23)
let tuned = adapters.fine_tune(adaptation, steps: 50, batch: 2,
    rate: 0.0005, seed: 72)
tuned.save("experiment-lora.anergen")

Full fine tuning trains all selected weights. Freezing excludes named weights from gradients, Adam state and decay. LoRA freezes the base and trains its added low rank factors; the base and activations still use memory. Each transition starts an explicit new optimizer run and retains cumulative training provenance. Validation/test isolation checks include earlier training and tokenizer fitting documents.

The complete adaptation example also publishes an adapter, checks its exact base identity and merges it for inference. A full checkpoint retains moments for exact resume; an adapter only artifact starts a new optimizer run when loaded.

Compose a different architecture

The built in reference family is one implementation. The custom decoder example constructs a multihead block directly as an Aner class with nn.layer_norm, nn.attention, nn.gelu, native tensor reshaping/batched products and named nn.update calls. Its embedding and output paths share one parameter handle and update it once. Researchers can change the forward composition, objectives and supported dimensions using these same operations.

The built in gen automatic trainer and checkpoint schema support the declared native decoder family and supported GPT 2 import. An arbitrary user class is not automatically serialized by gen.save. Custom architectures enumerate their own parameters, implement their update loop and define their own state/artifact contract. nn.backward replaces reachable gradients on each call; custom gradient accumulation needs an explicit design rather than assuming accumulation across calls.

Current complete sequence execution supports CPU Float64/Float32 and available Metal/MPS Float32. CUDA matrix support is separate from complete sequence support. The reference family currently admits context up to 512, width up to 256, layers up to 8 and at most 1,000,000 parameters, with stricter actual tensor/work/graph limits. These bounds define today's implementation. Larger or richer admitted compositions remain research choices; distributed language model training and large model scaling need additional runtime support.

Verification and scientific extensions

The executable provides reproducible end to end checks. Native regression tests additionally cover finite difference gradients of embeddings, attention Q/K/V and tensor composition; zero influence and zero gradients from future positions; fitted tokenizer partition isolation; frozen update/decay exclusion; adapter gradients; exact same backend resume with dropout and optimizer policies; and development/production equality. Independent scalar differentiation tests compare logits, parameter gradients and an optimizer update without becoming runtime dependencies.

Use these categories when validating a new architecture: numerical derivatives, mask behavior, state continuity, data isolation, sampling reproducibility and bounded resource lifetime. A decreasing training loss alone does not establish correctness or application quality. Evaluate the final selected model on untouched test data under the recorded tokenizer/window policy.

aner.gen also covers conditional scientific time series and small field diffusion. Those applications use numerical conditions, physical units/preprocessing and ensemble diagnostics beyond text token quality. Native variance calibration, optional mass constraints and PDE residual diagnostics have their own statistical and physical validation. They extend the same grouped generative module; the demonstrated field ensembles do not establish arbitrary scientific system fidelity.