Neural learning, research, and devices

Neural network configuration provides two interfaces: immutable Sequential descriptions for Dense classifiers, and user defined classes with explicit Tensor operations for custom forward passes and objectives. Both interfaces use the same native differentiation and selected CPU or accelerator kernels. Classes organize model state; inheritance is not required to configure a Sequential network.

This guide describes implemented behavior in the current interpreter, with VS Code extension 0.1.31 or newer. Runnable downloads are the research classifier, Adam checkpoint continuation, a custom model class, the VS Code notebook, and the Jupyter notebook. VS Code notebooks use .anernb; Jupyter uses .ipynb with the Aner kernel.

Sequential model configuration

import aner.dataset
import aner.nn
import aner.ml

print(nn.devices())
let split = dataset.iris().split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 16, initialization: "he").relu()
    .dense(inputs: 16, outputs: 3, initialization: "xavier")
let plan = ml.classifier(network, standardize: true,
                         loss: "cross_entropy", optimizer: "adam")
    .on(device: "auto", precision: "Float64", policy: "preserve")
let fitted = plan.fit(split.train(), epochs: 400, rate: 0.02, progress: true)
print(fitted.evaluate(split.test()).summary())
print(fitted.predict(split.test()).device())

Iris is Fisher (1936), distributed by UCI under CC BY 4.0; Aner preserves its source identity and attribution. Split before fitting normalization. Standardization learns only from training rows and is reused for held out evaluation and prediction. The final Dense layer produces raw logits; the classifier applies softmax as needed. Add Dense, ReLU, Tanh, or Sigmoid stages with visible dimensions; every adjacent width is checked.

plan is an immutable specification. Each fit returns an independent fitted model; it does not silently mutate a model used for inference. A fitted model owns detached parameters and fitted preprocessing. predict and evaluate reuse fitted preprocessing and parameters on the selected device, so training, testing, and inference share one explicit placement contract.

Initialization and loss

Each Dense layer accepts one of these initialization policies, with zero biases and a reproducible seed equal to the network seed plus its zero based Dense index:

Initialization policies do not establish a model accuracy guarantee. Explicit weights and biases take precedence through the separate custom overload:

let custom = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 3,
           weights: nn.random(4, 3, seed: 7, scale: 0.1),
           bias: nn.zeros(1, 3))

Weights must be finite Tensor values with shape inputs × outputs; bias must be 1 × outputs. The builder detaches immutable values, so a separate parameter update cannot mutate the specification. The execution profile resolves precision before fitting. CPU and CUDA plans can retain Float64 or select Float32; Metal/MPS uses native Float32. Preserve on Metal/MPS with Float64 reports CPU Float64 fallback. Initialization, targets, parameters, gradients, and optimizer state use the same effective precision during training.

Classifier loss is "cross_entropy" or "mse". Cross entropy averages over rows using raw logits and one hot targets. Classifier MSE averages squared softmax probability error over rows and classes. A custom Tensor loop can instead define an objective from supported differentiable operations; a user function or class cannot currently be passed directly as a classifier loss callback.

Optimization and checkpoint continuation

"sgd" remains the default and updates each parameter by parameter - rate × gradient. "adam" retains first and second gradient moments, uses bias correction with the cumulative update count, and updates by rate × corrected_mean / (sqrt(corrected_second_moment) + epsilon). Adam uses fixed beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8, with no weight decay or AMSGrad. Mini batches and keyed optimizer objects are implemented. Momentum SGD, scheduling, clipping, and configurable Adam beta/epsilon remain future extensions.

The legacy fit(training, epochs, rate) overload uses one full batch forward pass, backward pass, and parameter update per epoch. rate must be finite and positive; epochs is an explicit positive fixed horizon. An optimizer is separate from the model's layer description and selected loss. The same configuration is used in Dev and Prod.

let partial = plan.fit(split.train(), epochs: 100, rate: 0.02)
partial.save("research.anermodel")
let restored = ml.load_classifier("research.anermodel")
let final = plan.warm_start(restored)
    .fit(split.train(), epochs: 100, rate: 0.02)

Saving creates a new file and refuses to overwrite an existing path. Version 2 checkpoints retain topology, initialization policies, detached parameters, normalization, schema, provenance, metrics, optimizer choice, Adam moments and update count, and measured duration. They do not contain executable source, raw training feature/target rows, or an autograd graph. Version 1 checkpoints remain readable as the original SGD model; their missing duration is reported as zero. Version 3 Float32 checkpoints remain readable; they retain actual dtype/source device but do not record the original requested precision or execution policy. Checkpoints with nondefault execution metadata use version 4 and retain requested device/precision/policy, effective precision, and source device provenance. Loading reconstructs CPU parameters and optimizer state in their saved effective precision, without requiring the source accelerator. Select .on(device, precision, policy).warm_start(restored) to continue under a deliberate profile. A precision change converts parameters and Adam moments consistently and is reported; casting a Float32 checkpoint to Float64 cannot restore precision lost before saving. The default CPU Float64 preserve profile retains version 2 saves. Version 4 checkpoints require this updated interpreter.

Warm starting requires matching topology, loss, scaling policy, and data schema. It retains the original normalization even when fitting compatible new rows. The same Adam optimizer restores its moments and update count, so two consecutive fits with identical rows, rate, precision, and native backend reproduce one uninterrupted fit. Changing device or precision can change rounding and reduction order. Deliberately choosing a different optimizer keeps the weights and scaler but resets optimizer state; the summary identifies that reset. Continuation restores only the state retained by the checkpoint format. Earlier checkpoint versions do not certify complete gradient history and cannot become validated mini batch continuations.

Version 5 additionally retains mini batch epoch boundary continuation, ordered source IDs, shuffle seed and schedule fingerprints, batch settings, cumulative fitting/preprocessing/validation membership, and selected epoch state. Mini batch snapshots use this format; full batch snapshots may also use it when expanded history must be retained. Exact continuation requires unchanged source order, group identities, batch size, shuffle settings, precision, and numerical backend. A best epoch snapshot retains the optimizer state corresponding to its selected parameters, rather than the optimizer state from a later epoch.

Mini batch training and validation

import aner.dataset
import aner.nn
import aner.ml

let parts = dataset.iris().partition(validation: 0.2, test: 0.2, seed: 42)
print(parts.audit())
let network = nn.mlp(inputs: 4, hidden: List(16, 8), outputs: 3, seed: 42)
let config = ml.training(epochs: 20, batch: 16, rate: 0.01, seed: 42)
    .validation(parts.validation()).select_best(true)
let fitted = ml.classifier(network, standardize: true, optimizer: "adam")
    .fit(parts.train(), config: config, progress: true)
print(fitted.evaluate(parts.test()).summary())

nn.mlp constructs Xavier initialized Dense layers with ReLU between hidden layers and raw output logits. Supply a List of hidden widths. Use the explicit Sequential builder for a different activation or custom initial parameters.

ml.training is immutable. Each epoch visits every training row once with shuffle seed seed + completedEpoch; the final partial batch is retained. .validation(data) performs detached held out evaluation after each epoch. .select_best(true) returns the earliest epoch with the lowest measured validation loss; validation is required. Its checkpoint corresponds to that selected epoch. This selection does not stop training before the fixed horizon. The benchmark uses select_best(false) to return the final epoch consistently.

All Dataset fits reject validation/test training roles. Configured mini batch fitting also checks cumulative row/group disjointness for fitting, preprocessing, validation, and subsequent evaluation; version 5 retains that audit history. Previously used validation selection rows cannot become a final test view. Missing required group metadata raises an error. The legacy full batch evaluation permits training row evaluation and reports the overlap for compatibility. Raw Tensor extraction discards Dataset identity; explicit custom loops retain original views and call require_training(), audit(), and dataset.require_disjoint before extraction. Dataset partitions defines roles and grouping constraints.

nn.optimizer(name: "adam", rate: 0.001) creates an optimizer for an explicit Tensor loop. Call optimizer.update(parameter, key: "unique_parameter_name") after backward and retain the returned parameter leaf. Each distinct parameter needs a stable unique key. Keys retain shape, dtype/device, moments, and update count; incompatible reuse raises an error. optimizer.reset() deliberately discards history. Custom objectives remain explicit differentiable Tensor functions or class methods.

Progress, measured timing, and Dev mode

progress: true prints bounded progress in the CLI and flushes each selected observation. The maximum is 64 lines per fit. Each line identifies completed epoch count, measured loss phase, elapsed seconds, approximate remaining seconds, actual placement, precision, and native kernel provider.

Legacy full batch intermediate losses are the objective immediately before the latest completed update; the final loss is measured after the last update. Mini batch epoch loss is the row weighted mean of batch losses before their individual updates. Validation loss and accuracy use held out rows after the epoch, and final training metrics use a separate detached pass. Online loss combines changing parameter states and must not be labelled as a single final model training loss. Dev metadata also records loss_epoch, so charts and tables can distinguish these phases. Missing epochs are not fabricated.

After at least five full batch epochs or two mini batch epochs, estimated remaining time is measured_loop_seconds / completed_epochs × remaining_epochs. Before that, progress says warming up; completion reports zero remaining. The estimate includes observed update and recording cost, but excludes preprocessing and future final evaluation, serialization, or report writing. It can change with thermal throttling, CPU contention, caching, or different observer cost. It is an estimate for a fixed epoch horizon, never a prediction of convergence or accuracy. Aner currently has no unbounded fit API; a user controlled indefinite loop receives no fabricated completion estimate.

fitted.training_seconds() measures this fit using a monotonic clock from setup and preprocessing through training, final metrics, observation callbacks, and parameter detachment. It is separate from pure kernel throughput and from saving/report writing. Repeating a fit can give different timings while returning the same model values on the same backend.

aner neural_research.aner --dev research-report
aner neural_research.aner --prod

Use a new report directory for each Dev run. Open research-report/inspection.html for bounded variables, actual Tensor dtype/device/backend metadata, loss curves, measured progress, and named captures; research-report/index.html adds topology and available gradient statistics. --debug-values opts into bounded individual values. Dev records real observations without changing the optimization algorithm. Prod disables automatic recording and the extra final backward needed for gradient inspection; optional progress is an independent choice. Remove progress: true when its output is not wanted.

In VS Code choose Dev and Inspector; the supplied .anernb opens in development mode. In Jupyter the supplied .ipynb uses %aner dev. Variables, loss curves, and measured progress appear after completed cells. The current notebook protocol buffers output until a cell returns, so it does not yet stream a running fit. %aner prod disables notebook recording. Disable broad variable capture with dev.automatic(false) when only explicit captures and epoch observations are wanted. This setting persists across notebook cells and resets with the session. Custom loops can record dev.progress(epoch, epochs, loss, accuracy, seconds) or add a sixth validation_loss argument; the loss is a finite row weighted online observation and accuracy is a fraction in [0, 1].

Notebook execution defaults to 120 seconds per cell. Use %aner timeout 600 in a separate Jupyter cell, or VS Code setting aner.notebook.timeoutMs: 600000, before a larger run. Valid limits are 1 to 86,400 seconds in Jupyter and 1 to 86,400,000 milliseconds in VS Code. Changing the limit preserves current variables and affects subsequent cells; an actual timeout or cancellation clears the native session. Pausing, stepping inside a native operation, editing live variables, and streaming notebook progress remain planned. Development mode explains the recording limits.

Dev/Prod describe observation policy, not neural training/inference semantics. If future Dropout or BatchNorm layers require different forward behavior, they need a separate explicit training/evaluation setting. Production also remains an interpreter running native C++ kernels; compiling Aner programs is a separate future feature.

Custom model classes

class DenseBlock {
    var weights: Tensor
    var bias: Tensor
    fn forward(input: Tensor) {
        return nn.tanh(nn.add_bias(nn.matmul(input, weights), bias))
    }
    fn update(rate: Float64) {
        weights = nn.sgd(weights, rate)
        bias = nn.sgd(bias, rate)
    }
}
let block = DenseBlock(
    weights: nn.parameter(nn.random(4, 8, seed: 42, scale: 0.25)),
    bias: nn.parameter(nn.zeros(1, 8)))

Import aner.nn before this snippet. Compose class methods and Tensor operations to build explicit forward logic, then call nn.backward(loss) on a scalar objective and update each parameter leaf. nn.grad(parameter) exposes its most recent gradient. Replace each parameter with the result of nn.sgd; intermediate graphs are released when references disappear. Aner does not automatically discover class fields as trainable parameters, serialize arbitrary user classes, or provide automatic parameter registration. The complete custom XOR class shows all steps and captures actual training states.

The architecture separates model description, parameter state, objective evaluation, and optimization. A future common Layer/Model interface requires parameter enumeration, shape inference, checkpoint state, and explicit training/evaluation behavior.

Devices, precision, and the actual backend

The default execution profile is CPU Float64 with the preserve policy. Set a profile once for the run or notebook session, then prepare data before creating trainable leaves:

import aner.nn

nn.configure(device: "auto", precision: "Float64", policy: "preserve")
print(nn.execution())
let input = nn.random(8, 4, seed: 2026, scale: 0.25).place()
var weights = nn.parameter(nn.random(4, 3, seed: 42, scale: 0.1).place())
print(weights.dtype())
print(weights.device())

CPU and compiled, available CUDA support Float64 and Float32. Metal/MPS provides native Float32. Preserve keeps requested precision, using reported CPU Float64 fallback for an available Metal/MPS Float64 request. Adapt retains precision when supported and permits Float32 only when native Float64 is unavailable. Fast intentionally selects Float32 for a performance experiment without promising faster execution. Auto considers available CUDA, then Metal/MPS, then CPU. An unavailable explicitly requested accelerator fails; a driver or missing backend error is not a precision limitation.

ml.classifier captures the current execution profile. Changing the global profile later does not mutate an existing plan or fitted model. CPU preprocessing fits and applies normalization in Float64. Before training, the classifier prepares features, targets, initialized or restored parameters, and Adam state with its resolved effective dtype/device. Numerical forward, backward, and parameter updates execute there. Adam moments remain resident between epochs. Evaluation and inference retain the fitted effective placement and precision; host metrics, requested progress, and Dev inspection can synchronize and read back data.

GPU execution requires a compiled native backend and a compatible local device/driver. The runtime creates its context and queue or stream. Source programs do not establish an external connection. Strict .to("cuda") preserves Float64 when available; strict .to("mps") rejects Float64. .place() makes an explicit execution policy decision before the numerical graph. No operation implicitly moves or promotes an operand, and a different dtype cannot be applied to an existing differentiation graph. NDArray remains CPU storage; its Float64 rank two to_tensor() creates a detached CPU Tensor.

Tensor devices and performance defines CLI overrides, warnings, numerical conversion limits, portable source/notebook experiments, hardware prerequisites, checkpoints, and synchronized benchmark methodology. Float64 provides greater arithmetic precision and range; it does not guarantee better predictive accuracy. Adaptation and intentional performance conversion remain visible in both Dev and Prod. CNN/RNN, AMD, and distributed execution require additional native operations and validation.

Native execution and complexity

Tensor values use row major native storage with explicit Float64 or Float32 precision. Forward and transposed backward matrix products share a cache blocked native kernel, avoiding materialized transpose copies. macOS builds can use Accelerate for eligible products; small or extreme magnitude cases retain checked native arithmetic. nn.backend() identifies the compiled provider, not a promise that every operation uses BLAS. Vendor libraries may manage their own threads, packing buffers, and caches. The portable native CPU path is single threaded; the reported logical CPU count is not a promise that every core is saturated. CUDA products use resident Float64/Float32 cuBLAS operations; Metal/MPS products use Float32. Elementwise operations, gradients, and optimizer updates use the matching native precision. CPU Float32 storage and matrix products coexist with Float64 intermediates in some scalar/reduction operations; execution reports distinguish storage from computation.

Let B be batch rows, layer widths be d0 through dL, P be the total weights and biases, A be retained activation and loss graph elements, G be reachable graph nodes, E be epochs, and S be stage count. For mini batches let N be training rows and V validation rows. For a matrix product use dimensions m × k and k × n.

Fresh mini batch fits sort provenance membership in O((N + V) log(N + V)); schedule copies, shuffling and hashes add O(EN). Every completed epoch retains loss/validation metrics and a schedule fingerprint, using O(E) fitted metadata independently of bounded Dev observations.

Each differentiable operation also validates its reachable graph before publication. This adds O(G) traversal time and temporary metadata to that operation; the training bound includes graph validation across stages. Detached inference avoids differentiation graph construction.

Dev graph/statistic captures add bounded traversal and optional value copying at selected epochs, plus a final backward pass when gradient inspection is requested. The CPU provider can change floating point reduction order; repeatability is scoped to the same settings and native backend, not bitwise identity across devices or vendors. No speedup factor is claimed without a matching benchmark.

Current limits remain 32 stages, one million parameter elements, one million elements per Tensor, four million retained graph elements, 100,000 epochs, and 500 million conservatively estimated work units per fit or prediction/evaluation. CPU Float64 Tensor products allow 50 million multiply add terms; CUDA Float64 and CPU/GPU Float32 Tensor products allow one billion. The legacy full batch classifier retains its separate 50 million term Dense stage admission bound. Configured mini batch fitting admits at most 100,000 updates and one trillion conservatively estimated work units; each batch retains Tensor and graph bounds and the actual native matrix product limit. Adam admission accounts for its extra update work. These limits are deterministic safeguards, not RAM quotas or automatic resource tuning. The canonical MNIST Dense benchmark uses compact sources and bounded mini batches within these limits; unrestricted workloads still require an explicit capacity plan.

Model extension requirements

Keep architecture, objective, optimizer state, fitted preprocessing, observation policy, device placement, and distributed strategy as separate concerns. CNN and RNN layers should enter the same visible layer composition and custom forward path once their native forward and gradients exist. General rank differentiation and native convolution kernels are still required for CNNs; CPU NDArray reshaping alone does not add a differentiable convolution backend. BatchNorm/Dropout need their own train/evaluate semantics. Distributed execution additionally needs coordination and sharding, through a distributed strategy distinct from selecting a local device.

The MNIST Dense benchmark uses matched data, topology, initial parameter bytes, batch size, loss, optimizer, precision, source order, and device. Report end to end time, warmed throughput, peak memory, and synchronization cost separately for Dev and Prod. The supplied larger Dense baseline separates source programming styles and observation costs. CNN/RNN benchmarks require implemented and validated operations; results from the Iris demonstration do not establish large model throughput. Machine learning architecture describes the longer term backend and model interface plan.