Native generative research
Build and train a causal language model from scratch, evaluate and resume it, fine tune with LoRA, and explore conditional scientific generators with tested physical constraints.
Implemented in developmentaner.gen provides native decoder construction, training, evaluation, checkpoint/resume, generation, full fine tuning, parameter freezing and LoRA. It also groups conditional scientific time series generation and small field diffusion in the same module. It uses aner.token for language corpora and the existing aner.nn and aner.tensor numerical foundations.
The from scratch LLM guide walks through the complete research process with executable source and checks. These features require the native development runtime with aner.token and aner.gen; earlier development executables may not contain them.
A reference architecture researchers can change
The built in decoder starts with new seeded weights. It combines learned token/position embeddings, pre normalized causal multihead attention, residual connections, GELU feed forward blocks, a final LayerNorm and a tied vocabulary projection.
This is one working representation of Aner's capabilities. Its example dimensions are chosen for a quick reproducible run. Researchers can build different supported blocks and larger compositions with native functions and classes. The built in gen trainer/checkpoint schema handles its declared decoder families; custom class architectures own their parameter enumeration, training loop and serialization. See the custom decoder source and sequence primitives.
API reference
This minimal native program initializes new weights and performs one update. Use the complete guide for partitioning, validation and artifact handling.
import aner.token
import aner.gen
let codec = token.byte_tokenizer()
let training = token.corpus(codec, List("ababa", "babab"), role: "training")
let initial = gen.decoder(codec, context: 8, width: 8,
heads: 2, layers: 1, seed: 42, dropout: 0.0)
let trained = initial.train(training, steps: 1, batch: 2, rate: 0.001, seed: 99)
print(trained.loss(training))| Task | Native calls |
|---|---|
| Construct from new weights | gen.decoder(codec, context, width, heads, layers, seed, dropout) |
| Inspect and initialize | model.summary(), model.parameter_names(), model.parameter(name), model.with_parameter(name,value) before training |
| Train | model.train(training, steps, batch, rate, seed); extended form also supplies clip_norm, warmup_steps, weight_decay |
| Evaluate | model.loss(corpus), model.perplexity(corpus), gen.scored_tokens(corpus), model.baseline() |
| Checkpoint and continue | model.save(path), gen.load(path), model.resume(training, steps) |
| Begin a new training run | model.fine_tune(training, steps, batch, rate, seed); same extended optimizer arguments as train |
| Choose trainable weights | model.freeze(names), model.unfreeze(names), model.trainable_parameters() |
| Add adapters | model.lora(targets, rank, alpha, seed), model.save_adapter(path), gen.load_adapter(base,path), model.merge_lora() |
| Use a custom objective | model.logits(ids, batch, length, training) with nn.backward and nn.grad |
| Sample | model.generate(prompt, max_new_tokens, temperature, top_k, top_p, seed) |
| Inspect execution and admission | model.device(), model.dtype(), model.memory_estimate(batch,length,training), model.memory_summary(batch,length,training) |
| Change execution explicitly | gen.device_supported(device,dtype), gen.to(model,device,dtype) |
| Import the supported pretrained family | gen.import_gpt2(directory,context) |
Module functions and typed instance methods represent the same model operations. Names in freezing and adapter targets are exact registered parameter names, rather than patterns.
Training, evaluation and continuation
A step is one optimizer update. The trainer constructs shifted next token targets within each document, excludes padding from attention/loss and scores each document transition once per epoch. It shuffles windows with explicit deterministic seeds. The reference model uses Adam; the extended policy controls global clipping, linear warmup and decoupled matrix weight decay. Dropout is enabled only for training.
loss evaluates all actual targets without dropout and combines window losses by token count. last_loss() is the last batch's pre update objective; use loss for a measured post update value. Compare perplexities only with matching tokenizer and evaluation policies. A uniform vocabulary baseline checks correctness and is not a competitive language quality benchmark.
A full checkpoint retains tokenizer/configuration, named weights, Adam moments, training policy, counters, data lineage and execution identity. Exact resume requires the original corpus/order, backend, precision and runtime implementation. Changing data or training policy starts an explicit fine_tune run. Cross backend inference is supported subject to numerical differences; it does not establish exact cross backend continuation.
Artifact paths must be new. Checkpoints are bounded numeric data with integrity validation, not executable model code. A saved adapter contains its factors and exact base identity; use a full model checkpoint to retain optimizer state for resume.
Fine tuning and generation
Full fine tuning updates all currently trainable parameters. Freezing excludes selected parameters from gradients, optimizer moments and decay, while retaining their storage. Freezing the token embedding also freezes its tied output projection. LoRA freezes the base and learns low rank factors for selected block.i.qkv, block.i.attention.output, block.i.ff.in or block.i.ff.out matrices. Its factor rank is bounded to 1..64 and must fit the target dimensions.
Generation returns newly generated IDs, including EOS if selected. Temperature zero is greedy; positive temperature uses local seeded sampling with optional top-k/top-p filters. BOS and PAD are not generated. Prompt plus requested output must fit the declared context. The current implementation recomputes attention per token; a larger context or training horizon does not supply a KV cache or unlimited history.
The fine tuning example exercises BPE, full updates, freezing, LoRA, adapter publication and exact full checkpoint resume.
Pretrained import and execution
The supported import is a bounded GPT2LMHeadModel with local config.json, model.safetensors, vocab.json and merges.txt. It validates the expected architecture, tensor shapes, numerical conventions and tokenizer. The pretrained example expects a separately prepared compatible local directory; it performs no download. Full sized GPT 2 exceeds current parameter bounds. Other external architectures need their own compatibility implementation.
CPU Float64/Float32 and complete native Metal/MPS Float32 sequence execution are supported where available. CUDA's existing matrix operations do not provide the complete sequence backend. gen.to changes placement/precision as an explicit new run; matching placement preserves the current run. Exact same backend continuation is tested separately from tolerance based CPU/Metal comparisons.
The memory estimate counts conservative numeric buffers and checks tensor/work/graph limits. fits is admission to those limits, not a measurement of available RAM or VRAM. Actual attention workspace grows quadratically with sequence length.
Scientific generation in the same module
Conditional time series models use numeric data roles and explicit preprocessing; small field diffusion uses native noise schedules, learned denoising, optional velocity/Sobolev objectives, variance calibration and optional affine condition to mass constraints. Schedule aware inputs and optional EMA retain separate raw optimization and averaged inference weights. Calibration uses training data; physical and statistical diagnostics evaluate generated ensembles.
The scientific example demonstrates the native training/calibration/checkpoint process. The verification evidence records the declared study, including all 1,517 criteria and its remaining tighter band variance outlier. These are bounded conditional field ensembles, not a general PDE solver or universal physical fidelity guarantee.
Current scope
The reference decoder admits context 1..512, width 2..256, heads 1..64 dividing width, layers 1..8, at most 1,000,000 parameters, batches up to 64 and runs up to 100,000 updates. Stricter tensor, matrix work and differentiation graph budgets can reject configurations within those individual maxima. Corpora are bounded, in memory snapshots; streaming/distributed language model training and arbitrary pretrained architectures are future work.
These limits describe the current runtime and built in model family. They do not define the complexity of an Aner program or promise that merely increasing dimension limits makes large model execution practical. User compositions can explore richer supported architectures while retaining explicit numerical and resource checks.
aner.llm separately connects to external inference services. A service response does not train these native weights; see language model connections.