Neural networks

Aner Neural is the built in neural network library, imported as aner.nn. It provides rank two Float64/Float32 tensors, native CPU and optional Metal/MPS or CUDA operations, reverse mode automatic differentiation, and SGD or Adam parameter updates. Aner programs describe a model through the concise classifier API or an explicit training loop; Aner's native C++ runtime performs the tensor operations.

The current library supports small dense neural networks. CNNs, RNNs, Transformers, distributed training, streaming training over external datasets, and additional optimizer families are future work. Fitted classifier checkpoints and warm starts are implemented. The device backends share shape, differentiation, and explicit placement contracts; Tensor devices and performance documents requirements and matched measurements.

Dense model example

Follow the one time installation guide to put aner on your PATH. Open a terminal in the folder containing the binary bundle's examples directory, then check and run the XOR model:

aner check examples/neural_xor.aner
aner run examples/neural_xor.aner

Use the same commands on macOS, Linux, or Windows, including VS Code's terminal. No separate neural library installation is needed: aner.nn ships with Aner.

The XOR example trains on four synthetic observations, using two input features, eight hidden units with tanh, and one output with sigmoid. It computes mean squared error, runs 3,000 full batch updates at learning rate 0.5, and prints the initial loss, final loss, and four predictions. The target values are 0, 1, 1, 0, in that order. This demonstrates that gradients and parameter updates compose into a working training loop; it is not a medical data model or a benchmark for large datasets.

A recorded reference run produced these rounded results:

Your output should show the loss decreasing and predictions approaching the four targets. These reference values are not a guarantee of identical final decimals across platforms.

Concise dense classifier pipelines

The built in classifier API groups layer parameters and the repeated training loop into native values. It uses the same tensor kernels and autograd as the detailed examples.

import aner.dataset
import aner.nn
import aner.ml

let flowers = dataset.iris()
let split = flowers.split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 8)
    .tanh()
    .dense(inputs: 8, outputs: 3)
let pipeline = ml.classifier(network, standardize: true, loss: "cross_entropy", optimizer: "sgd")
let fitted = pipeline.fit(split.train(), epochs: 2000, rate: 0.1)
print(fitted.evaluate(split.test()).summary())

Run aner run examples/neural_iris_concise.aner. The Wine example uses 13 → 16 → 3 dimensions, 1,000 epochs, and rate 0.05. The Iris notebook separates data, topology, fitting, and evaluation into persistent cells. The older detailed Iris program retains its explicit forward/backward/update loop and visual capture calls.

Use all three imports above: aner.dataset for teaching data and partitions, aner.nn for topology, and aner.ml for fitted classification. The names are library APIs, not additional language keywords. Instance calls are resolved from the receiver's static type. Named arguments can make positional configuration clearer without changing type checking. A leading . on the next line continues a method chain.

The optional loss argument accepts exactly "cross_entropy" or "mse", with "cross_entropy" as the default for existing two argument calls. Cross entropy averages one hot cross entropy over rows from raw logits. MSE computes nn.mse(nn.softmax(logits), one_hot_targets): the sum of squared probability errors divided by rows × classes. The configured loss is used for every training update, initial and final training loss, and evaluation loss. Summaries identify the selected loss. Accuracy and predictions still use the highest scoring class.

To choose MSE, change only the classifier configuration:

let pipeline = ml.classifier(network, standardize: true, loss: "mse")
let fitted = pipeline.fit(split.train(), epochs: 2000, rate: 0.1)

Keep the network's final Dense layer as raw logits; the classifier applies softmax internally for MSE, so do not add a softmax layer. The MSE Iris program and MSE Iris notebook use this policy with the same split, topology, epochs, and rate as the concise cross entropy example. Different losses have different scales and gradients, so their loss values are not directly comparable and their convergence can differ at the same learning rate. The optional argument requires an updated native Aner executable; loss editing support was introduced in extension 0.1.18; the current extension 0.1.31 includes signatures and development inspection. Updating only the extension does not update the executable.

Configurations and fitted snapshots are separate immutable values. dense and activation methods return a new configuration; they do not mutate an existing let binding. Ordinary fit starts from the network initialization and learns standardization solely from the supplied training features. pipeline.warm_start(model) instead returns a plan that starts from that immutable fitted snapshot and reuses its scaler without refitting. Both forms allocate independent trainable parameters and leave earlier fitted values unchanged. Test features reuse exactly the fitted statistics. This protects that preprocessing boundary; it cannot determine whether the caller chose medically independent patients or a valid study partition.

The trainer uses the configured loss and SGD or Adam, through the legacy full batch overload or immutable ml.training mini batch settings. Plans capture a requested device, precision, and preserve/adapt/fast execution policy. CPU and CUDA support Float64/Float32; Metal/MPS supports native Float32. Preserve on Metal/MPS with Float64 uses reported CPU Float64 fallback. Numerical state is prepared once after CPU Float64 preprocessing. SGD remains the default; Adam uses beta1=0.9, beta2=0.999, epsilon=1e 8 and retains moments and update count across compatible warm starts. Default initialization uses seeded random weights with scale 0.5 and zero biases; the custom Dense overload accepts a finite weights/bias pair instead. Epoch count and finite positive learning rate are explicit. The optional optimizer string defaults to "sgd"; names other than "sgd" and "adam" raise R2701. A named optimizer: "sgd" may be supplied without a loss argument. Dense stages use consecutive seeds base_seed + dense_index, where the first Dense stage has index zero and activations do not increment the index. Seed overflow is rejected. Default trainable parameters are created when fit runs. Custom initialization tensors are copied and detached when the Dense configuration is created. The architecture must start with a Dense stage and end with Dense raw logits; its input width must match the dataset feature count and its output width must match the dataset class count. Copies of a fitted value share its immutable snapshot efficiently; fitting another model does not mutate that snapshot. The low level Tensor API still has shared gradient metadata, as documented below.

Summaries preserve topology and per layer seeds, training configuration and metrics, actual device/precision/backend information, fitted scaling statistics, source hashes, exact source row IDs, and dataset attribution. Evaluation checks feature names, units, column order, feature count, and class names/order against the fitted schema. Legacy full batch evaluation reports overlap with training rows for compatibility. Configured mini batch evaluation instead rejects overlap with retained fitting/preprocessing membership, and final test views cannot reuse earlier validation selection membership. Raw Tensor extraction discards Dataset identity, requiring explicit boundary audits in custom loops. Partition views retain their seed, fractions, roles, and source row fingerprints; keep the source with the output as well. A fixed seed supports repeatability in the same build and environment; it does not guarantee bitwise identity across compilers or future devices. Saved notebook output is not a serialized fitted model or checkpoint.

Training is bounded by 32 stages, 1,000,000 trainable parameters, 100,000 epochs, and 500,000,000 estimated training work units, in addition to the tensor/graph limits below. The work estimate is a conservative guard, not elapsed time, measured FLOPs, or a process memory quota. Raising the interpreter's step limit does not raise these library limits. Library failures use source located R2701 diagnostics. Configured mini batch training uses compact sources and lazy schedules, admits up to 100,000 updates and one trillion estimated work units, and retains the individual Tensor/graph bounds. Streaming notebook output during a running cell and automatic user defined layer registration remain future work. Development notebooks provide completed cell variables, captures, and loss views; see development and production.

Create a visual training report

aner run examples/neural_iris_concise.aner --debug-viz reports/iris-concise

Choose a new output directory. For variable/object observations as well, run aner examples/development_nn.aner --dev reports/nn-first and start at inspection.html. The CLI records the native training graph after backward and before each selected update: epochs 0, 1, 2, every max(100, ceil(epochs / 60)) epochs, and the final trained state. This keeps one fit within the recorder's 64 snapshot limit; the Iris example records every 100 epochs. Add --debug-values only when you want bounded raw tensor/gradient values in the report. Capture does not change the model's numerical updates. This is an offline HTML/JSON/text report, not a live notebook chart.

Multiple fits are recorded as separate series with local epoch steps and capture phases. The loss chart does not connect different fits into one continuous curve. All series share the bounded 64 snapshot budget; later fits may have fewer stored observations when the budget is exhausted. Manual viz.capture calls use their own strictly increasing step series. The detailed Iris example remains useful for choosing your own viz.capture points. Dataset attribution travels with recorded training reports.

Custom initialization and fitted checkpoints

let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 8,
           weights: nn.random(4, 8, seed: 7, scale: 0.25),
           bias: nn.zeros(1, 8))
    .tanh()
    .dense(inputs: 8, outputs: 3)
let pipeline = ml.classifier(network, standardize: true,
                             loss: "cross_entropy", optimizer: "sgd")
print(network.summary())
print(pipeline.summary())
let fitted = pipeline.fit(split.train(), epochs: 400, rate: 0.1)
fitted.save("iris.anermodel")
let restored = ml.load_classifier("iris.anermodel")
let resumed = pipeline.warm_start(restored)
    .fit(split.train(), epochs: 100, rate: 0.1)

This fragment follows the imports and split from the classifier example above. Save to a new file: checkpoints never overwrite an existing path or create missing parent directories. weights and bias are provided together, with finite values and exact shapes. A Tensor literal can supply manually chosen weights.

A warm start validates architecture, loss, standardization policy, feature/class schema, and parameter shapes. It reuses the prior training derived center/scale even when the new training rows differ; original scaler source identity and rows remain distinct from the new fit's rows. The returned model reports the new fit's epochs/rate rather than an invented cumulative history. SGD has no additional momentum state to restore.

The bounded, versioned binary checkpoint retains architecture, initialization policy, detached fitted parameters, scaling, schema, metrics, source metadata, and exact row IDs. It excludes raw feature/target rows and autograd graphs. Loading validates version, dimensions, finite values, positive scales, schema consistency, bounds, and trailing bytes. The payload limit is 64 MiB; failures use R2701.

For s stages, p copied or serialized parameters, r retained source row IDs, and m metadata bytes, custom Dense creation costs O(inputs × outputs + outputs + s) time and extra space; summaries cost O(s) time/output space; warm start configuration costs O(s) validation time and O(1) shared storage; weight/bias extraction costs O(selected elements) time and extra space; save/load costs O(p + r + m) time and space. These bounds exclude disk latency and the ordinary training graph. See the model state guide and the runnable checkpoint example.

Tensor operations and differentiation

Network parameters, features, predictions, and scalar objectives use native Tensor values. The Tensor reference defines construction, shape and dtype validation, matrix operations, activations, loss operations, reverse mode differentiation, parameter updates, device placement, and numerical limits. Network and fitted classifier APIs are defined on this page.

Built in modules and current scope

import aner.nn; selects a built in module and exposes the nn namespace. It does not load external packages. Third party native plugins, module aliases, package manifests, dependency resolution, version negotiation, and stable extension ABIs are not supported.

Use the built in sequential/classifier values for common dense classification, or ordinary Aner functions and explicit training loops for lower level experiments. The portable native CPU path is single threaded. A compiled CPU provider may manage internal worker threads; eligible macOS builds use Apple Accelerate for matrix products. Optional native CUDA builds support Float64/Float32 Tensor execution; Metal/MPS supports Float32, with precision policy controlling explicit CPU fallback or adaptation; distributed execution is not implemented.

Keep the data schema and partitions, preprocessing, model architecture, seeds, optimization settings, software version, and metrics with your experiment. Fitted classifier state can be saved, validated, loaded, and used for a compatible warm start. General Tensor/object checkpoints and KNN/K means serialization remain future work. The built in teaching catalogue and preprocessing are documented in Aner datasets; aner.data reads external CSV files with explicit table to tensor conversion.

Training observations

The visual training example imports aner.viz and captures the loss graph after backward at selected steps. Its offline report connects the model topology to recorded values, gradients, and loss history. See visual training reports for small neuron views, bounded summaries, and headless execution. Captures add overhead but do not alter numerical tensors, gradients, or parameter updates.

Multiclass classification

The Iris milestone adds nn.softmax(logits: Tensor) -> Tensor, nn.cross_entropy(logits: Tensor, one_hot_targets: Tensor) -> Tensor, nn.argmax(scores: Tensor) -> Tensor, and nn.accuracy(scores: Tensor, labels: Tensor) -> Float64.

Softmax normalizes each row and supports differentiation. Cross entropy accepts raw logits and matching constant, exact one hot targets; its scalar loss is averaged over rows. It combines normalization and logarithms stably rather than taking the log of rounded softmax probabilities. Logit gradients are (probabilities - targets) / rows. Differentiable targets and soft label distributions are rejected in this first version. Nonrepresentable results remain errors.

Argmax returns a detached rows × 1 Tensor of zero based class IDs in the input precision and device, choosing the first index on ties. Accuracy accepts one finite integral in range class ID per row and returns the correctly classified fraction; scores may be logits or probabilities. These metrics do not add autograd edges.

Run the Iris classifier, using the licensed dataset module, for a three class example with training only scaling, held out evaluation, and visual captures.

Related modules

Neural architecture, differentiable losses, and optimization remain in aner.nn. Classical fitted models and the new classifier pipeline live in aner.ml; common evaluation lives in aner.metrics. The Iris neural example now uses metrics.accuracy(nn.argmax(scores), labels), the same label contract as KNN. Existing nn.accuracy(scores, labels) is unchanged. metrics.mean_squared_error is a detached evaluation scalar, whereas nn.mse is a differentiable loss tensor.

The neutral aner.tensor module offers creation and inspection of the same native Tensor values. See classical ML APIs for KNN, K means, and shared evaluation metrics.

Training configuration and placement

Use nn.devices() to inspect actual runtime capabilities and tensor.dtype(), tensor.device(), and tensor.backend() to verify Tensor execution. nn.configure(device, precision, policy) selects a run/session profile, and .place() applies it before parameter creation. nn.execution() reports requested and actual settings. Strict .to("cuda") preserves Float64 or Float32; .astype("Float32").to("mps") selects native Metal/MPS Float32. .on(device, precision, policy) selects an immutable classifier profile. An unavailable explicitly requested accelerator fails. ROCm and cluster execution remain unavailable. Dense can select initialization: "xavier" or "he" in addition to the original default and custom tensors. fit(..., progress: true) prints measured CLI progress; fitted.training_seconds() reports duration. Version 4 checkpoints for nondefault execution profiles retain requested/effective precision and device policy, Adam state, and source device provenance; they load portably on CPU in their actual saved precision. Default CPU Float64 preserve saves remain version 2; versions 1 through 3 remain readable. See the full workflow and complexity reference for exact contracts, classes with custom forward/loss, Dev versus Prod, platform requirements, and runnable source/VS Code/Jupyter examples.

Mini batch and explicit optimizer APIs

nn.mlp(inputs, hidden: List, outputs, seed) builds a Xavier Dense network with hidden ReLU. ml.training(epochs, batch, rate, seed).validation(data).select_best(enabled) configures complete shuffled epochs and held out post epoch metrics; pass it as pipeline.fit(training, config: config, progress: true). nn.optimizer(name, rate) exposes keyed SGD/Adam updates to explicit Tensor loops. Version 5 fitted checkpoints retain mini batch epoch, exact source order, batch/shuffle settings, and cumulative fitting/selection provenance. The neural workflow defines API constraints, selected epoch continuation, phases, resource reporting, and complexity. The MNIST Dense benchmark supplies complete concise, functional, and class based programs and notebooks.