Recurrent neural networks

Implemented in development. These APIs require the recurrent enabled development runtime and matching editor. The published 0.0.3 downloads predate this feature. This guide publishes documentation and examples; it does not announce a new binary release.

Build a model, fit it and retain the returned snapshot:

import aner.nn

let model = nn.lstm(inputs: 4, hidden: 32, outputs: 1).on("auto")
let fitted = model.fit(x, y, epochs: 10, batch: 32, clip_norm: 1.0)
let prediction = fitted.predict(x)

Here x is a preprocessed Tensor [batch,time,4] and y is [batch,1]. See the complete runnable synthetic example. The same signatures work with nn.rnn and nn.gru. Each builder takes inputs, hidden, outputs: 1, layers: 1, bidirectional: false and seed: 42.

Choose the objective and layout

Select with .task("sequence_classification"). .logits(x) retains raw class scores; .predict(x) applies the declared inference contract. Sequence lengths are exact integers in 1..T, with right padding. Supply lengths: List(...) to fit, predict, logits or step; padded positions do not contribute to the loss or final state. Do not encode missing data as an unlabelled zero.

Use chronological train/validation/test splits for forecasting. Fit scalers and vocabulary on training data only, keep series identity explicit and avoid future information in covariates. The Tensor API does not infer temporal splits or attach protected Dataset roles to arbitrary tensors. Evaluate on held out time periods using the same transforms and inverse scale predictions before reporting errors.

Streaming and truncated training

let state = model.initial_state(batch: 8)
let first = model.step(chunk_a, state)
let second = model.step(chunk_b, first.state())
let values = second.values()

step returns RecurrentOutput, with task shaped values and detached RecurrentState. Each call ends the gradient graph. States belong to the model geometry, placement, dtype and batch ordering. Reset when a stream ends; never carry one series into another. Bidirectional models require future context and cannot be used as causal streaming models.

let fitted = model.fit_chunk(chunk_a, target_a, state, clip_norm: 1.0)
let continued = fitted.fit_chunk(chunk_b, target_b, fitted.state())

fit_chunk performs one update on one ordered batch and retains detached final state. This is an explicit truncated backpropagation boundary. Choose chunk lengths that fit the activation budget; it does not retain gradients into earlier chunks. See streaming example.

Advanced architectures

The implementation order is the standard cells, DeepAR, ConvLSTM, AWD LSTM, then xLSTM. These advanced profiles implement their declared mechanisms; they are not claims of matching published benchmark quality or reproducing a complete research training recipe.

DeepAR helpers construct causal features and masked targets:

let inputs = nn.deepar_inputs(values, observed)
let targets = nn.deepar_targets(values, observed)
let model = nn.deepar(inputs: 2, hidden: 32)
let fitted = model.fit(inputs, targets, epochs: 10)
let draws = fitted.forecast(inputs, last_target, horizon: 12, samples: 100)

values and binary observed are [B,T,O]; unobserved values are zero filled by the packer. With known covariates [B,T,C], use nn.deepar_inputs(values, observed, covariates) and declare inputs: 2*O+C. The current target never enters its own feature row. last_target is [B,O]; forecast trajectories are [B,samples,horizon,O]. For known future covariates, pass their [B,horizon,C] Tensor in place of horizon. Each trajectory feeds its own sampled value into the next step. Negative Gaussian NLL values can be valid. See forecast example.

Performance and parallel execution

Input projections for all standard cell timesteps are combined into one matrix multiplication per layer/direction. Gate matrices are fused. Recurrent steps retain their temporal dependency; adding GPUs does not remove that dependency.

.on("cpu", precision: "Float64"), .on("mps") and .on("cuda") select supported native backends. Metal/MPS uses Float32. CPU/CUDA support Float32 and Float64. Intel and AMD accelerator providers remain reserved, without a validated execution path; use CPU on those systems. Explicit unsupported placement or precision fails.

import aner.parallel

let fast = model.performance(amp: "fp16").on("cuda")
let fitted = fast.fit(x, y, batch: 64, execution: parallel.local())

CUDA profiles use FP32 master weights, moments, gradients and outputs with eligible GEMM operands in FP16/BF16. FP16 requires supported Tensor Core hardware; BF16 and TF32 require Ampere or newer. .performance(matmul: "tf32") explicitly permits TF32. FP16 dynamic loss scaling skips the whole update on overflow; inspect .attempts(), .skipped_updates() and .loss_scale(). Small or sequential workloads can be slower under AMP; measure the actual workload after warm up and synchronization.

fit(..., execution: context) reuses aner.parallel sessions for sample weighted gradient reduction. Local CPU replicas and configured MPI/NCCL contexts share the same optimizer contract. Configure one process per selected physical GPU for distributed GPU training. All ranks must agree on architecture, input data, objective, precision policy and collective order. Global batches can have uneven shards; losses and gradients use valid observation counts. No automatic server discovery or credential management occurs. See parallel computing and CNN compute and collective policy.

Global norm clipping defaults to clip_norm: 1.0; weight_decay: 0.0, optimizer: "adam", shuffle: true, rate: 0.001, batch: 32, epochs: 1 and seed: 42 are explicit fit defaults. Select "sgd" when needed. Weight decay is decoupled: each successful step subtracts rate * weight_decay * previous_weight from named weights and embeddings, excluding biases and normalization gains. Use .budget(tensor_elements, graph_elements) to bound allocations, including unrolled activation graphs. Parameter count does not estimate all training memory; sequence length, spatial geometry and xLSTM matrix memory matter.

Save, resume and inspect

fitted.save("sequence.anerrnn")
let restored = nn.load_recurrent("sequence.anerrnn").on("auto")
let resumed = restored.fit(x, y, epochs: 2)

Use a new checkpoint path. Checkpoints retain architecture, parameters, optimizer moments, update counters, random continuation and explicit retained state. Resume with compatible data, ordering and preprocessing. A checkpoint cannot restore unrecorded preprocessing or make floating point results bit identical across different accelerator architectures.

Development mode records bounded diagrams and training observations:

import aner.dev

dev.capture("Recurrent architecture", model)

Run aner examples/recurrent_basics.aner --dev reports/recurrent-first using a fresh report directory. Run the same source with --prod for normal execution. Production uses the same numerical kernels, shape checks, seeds and resource limits. .summary(), .diagram() and .performance_summary() inspect declarations without training. Captures retain immutable snapshots and do not replay training.

Primary architecture sources: LSTM, GRU, DeepAR, ConvLSTM, AWD LSTM, xLSTM. The canonical native reference documents equations, deviations and validation limits.

See the Neural networks module reference for the shared API.