Tensor devices and performance

Aner executes rank two Tensor operations on CPU or an explicitly selected, available accelerator. CPU and CUDA support Float64 and Float32. Metal/MPS supports native Float32. These backends provide matrix multiplication, elementwise and reduction kernels, reverse mode gradients, and resident parameter updates in their supported precision. Float64 is the default; a run or notebook session can preserve it, adapt only when necessary, or explicitly select Float32. Device placement is independent of Dev/Prod observation policy.

The complete Tensor device example, bounded matrix stress example, classifier comparison, VS Code notebook, and Jupyter notebook exercise CPU and whichever accelerator is available. Strict placement examples fail when an explicitly requested device is unavailable. The complete portable precision experiment, classifier precision experiment, precision VS Code notebook, and precision Jupyter notebook use a reported execution profile to keep the numerical source portable.

Runtime prerequisites

An ordinary CPU only Aner binary cannot use a GPU solely because the hardware or driver is installed. The native backend must be included in that binary. Creating the native device context, selecting a queue/stream, owning device buffers, dispatching kernels, and checking completion are responsibilities of the Aner runtime. No external connection or server login is needed from an Aner program to use a local GPU. nn.require_device validates initialization before the numerical workload.

Apple documents MPS matrix multiplication as a native Metal operation. The NVIDIA cuBLAS documentation describes GPU memory matrix operations, while the CUDA installation guide defines the supported host toolchain and driver requirements. Match the toolkit to the installed GPU architecture; support in a newer toolkit must not be assumed for an older GPU.

Native builds configure accelerator integration explicitly: Metal/MPS is enabled by the macOS build option, while CUDA is opt in and must target a compute capability supported by the selected toolkit. The build includes the native framework/runtime libraries; the program then selects an available device. Deployment packages must match both the host platform and their documented accelerator runtime requirements.

Precision and execution policy

Float64 retains more arithmetic precision and range than Float32; it does not by itself guarantee a better trained model. The numerical formats use eight bytes for Float64 and four for Float32 in a packed buffer. This comparison describes native device buffers, not total process memory. Both have finite range and rounding error. Native CUDA devices including V100 support Float64; Float64 throughput depends on the hardware architecture. A slower supported precision is a performance consideration, not a compatibility failure. Metal/MPS has no native Float64 execution path in this backend. The preserve policy provides explicitly reported CPU Float64 fallback for that request.

Choose precision once for the execution profile. Tensor literals, nn.zeros, and nn.random still construct CPU Float64 data. .place() prepares those values using the resolved profile, before creating trainable parameters. Numerical operations do not move, cast, or promote operands implicitly.

import aner.nn

nn.configure(device: "auto", precision: "Float64", policy: "preserve")
print(nn.execution())
let input = nn.random(rows: 32, cols: 64, seed: 2026, scale: 0.25).place()
var weights = nn.parameter(nn.random(rows: 64, cols: 16, seed: 42, scale: 0.1).place())
let output = nn.matmul(input, weights)
print(output.dtype())
print(output.device())
print(output.backend())
nn.synchronize(device: output.device())

All three nn.configure arguments are required. The unchanged default profile is "cpu", "Float64", "preserve". "auto" considers available CUDA, then Metal/MPS, then CPU. Resolution then applies the precision policy. Explicitly requesting an unavailable accelerator fails before training; a precision policy does not repair a missing native backend, device, runtime, or compatible driver.

CPU Float32 currently retains rounded values in Float64 host storage and a Float32 matrix cache, approximately twelve payload bytes per element before metadata, gradients, and optimizer state; CPU Float64 uses approximately eight. CPU gradients retain Float64 host storage. Consequently, CPU Float32 does not halve RAM and can use more host memory than CPU Float64. Native GPU buffers use four bytes per Float32 value or eight per CUDA Float64 value, but observing GPU values can add an eight byte per element Float64 host readback cache. Provider workspace and additional training buffers also contribute to total memory.

nn.execution() reports requested device, requested precision, policy, effective device, storage precision, compute precision, accumulation precision, and the resolution reason. The current CPU Float32 implementation stores Float32 values and uses Float32 matrix products; some checked scalar and reduction intermediates use Float64. CUDA Float64 buffers, arithmetic, gradients, and optimizer moments are Float64; CUDA Float32 uses Float32. Metal/MPS remains native Float32. Returning a host Float64 scalar from a Float32 result does not restore discarded precision.

The same source can be configured from the CLI:

aner examples/tensor_precision.aner --device auto --precision Float64 --precision-policy preserve
aner examples/tensor_precision.aner --device mps --precision Float64 --precision-policy adapt
aner examples/tensor_precision.aner --device auto --precision Float64 --precision-policy fast

An explicitly supplied CLI field overrides that field in a source nn.configure call throughout the run. Fields omitted from the CLI remain configurable in source. This supports a reusable source file with environment specific execution choices. In a persistent notebook, nn.configure applies to subsequent preparation and newly built classifier plans in that session; existing tensors and plans remain unchanged.

Warnings identify the requested and effective settings and the actual reason. Metal/MPS Float64 adaptation says that the native backend lacks Float64. Preserve fallback reports CPU Float64, never MPS Float64. Fast identifies an intentional precision choice. A CUDA driver error must not be described as a Float32 only GPU. Precision changes remain visible in production as well as development.

Float64 → Float32 conversion rounds values and can change numerical outcomes. For example, sequential Float32 arithmetic can produce zero for (100000000 + 1) - 100000000, whereas Float64 produces one. Casting back to Float64 cannot recover the lost information. Profile preparation rejects conversion overflow to nonfinite values and warns when finite nonzero values underflow to zero. A Float32 optimizer learning rate must remain finite and positive after conversion; a rate that rounds to zero or overflows is rejected before training. Finite conversion does not guarantee acceptable training error: compare losses, gradients, predictions, and tolerances on representative data before selecting the fast policy for a numerical application.

Strict precision and placement

The explicit primitives remain available when the program requires exact placement:

import aner.nn

nn.require_device(device: "cuda")
let input = nn.random(rows: 32, cols: 64, seed: 2026, scale: 0.25).to(device: "cuda")
var weights = nn.parameter(nn.random(rows: 64, cols: 16, seed: 42, scale: 0.1).to(device: "cuda"))
let output = nn.matmul(input, weights)
print(output.dtype())
print(output.device())

This example preserves Float64 on CUDA. For native Metal/MPS, convert detached values with .astype("Float32") before .to("mps"). .to changes memory placement while preserving precision and differentiation; it does not apply a profile or choose a fallback. .astype explicitly changes detached precision on the same device when supported; a different precision on an existing differentiation graph is rejected. Metal/MPS .astype("Float64") is rejected; transfer to CPU first when an explicit CPU conversion is intended.

Operands must have matching precision and device. Same device transfers share immutable storage. A host scalar read, inspection, or explicit .to("cpu") observes completed work and can synchronize or read back resident values. Keep these observations outside a measured resident loop when measuring throughput.

The zero argument nn.backend() describes the compiled CPU provider. Use the Tensor overload when reporting accelerator execution. Physical GPU presence alone does not establish where a Tensor operation executed.

Classifier placement

import aner.nn
import aner.ml
import aner.dataset

let split = dataset.iris().split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 16, initialization: "he").relu()
    .dense(inputs: 16, outputs: 3, initialization: "xavier")
let plan = ml.classifier(network, standardize: true,
    loss: "cross_entropy", optimizer: "adam")
    .on(device: "auto", precision: "Float64", policy: "preserve")
let fitted = plan.fit(split.train(), epochs: 20, rate: 0.02, progress: true)
print(fitted.weights(0).device())
print(fitted.weights(0).dtype())
print(fitted.evaluate(split.test()).accuracy())

CPU preprocessing fits and applies normalization in Float64. The classifier resolves the execution profile before preparing its numerical state. Preprocessed features, one hot targets, initialized or restored parameters, gradients, and optimizer moments then use the same effective precision and device. Forward, backward, SGD, and Adam execute there. Adam moments remain resident between epochs; checkpoint state is read back after fitting. Evaluation and inference reuse the fitted scaler and effective execution configuration. Metrics and requested progress observations can read results back to the host.

ml.classifier captures the current profile in an immutable plan. .on(device) changes that plan's requested device while retaining its precision and policy. .on(device, precision, policy) overrides all three for that plan. Later global configuration does not mutate an existing plan or fitted model.

Checkpoint versions 4 and 5 retain requested device, requested precision, policy, effective precision, and source device provenance. Configured mini batch fitting uses version 5, which also retains the training contract, exact schedules, and cumulative fitting/validation history. Legacy compatible full batch state can retain version 2 for default CPU Float64 preserve execution or version 4 for extended execution metadata; additional provenance requirements upgrade it to version 5. Loading reconstructs CPU parameters and optimizer state in their saved effective precision, without requiring the source accelerator. Versions 1 through 5 remain readable. Version 3 retains actual precision and source device provenance, but the original requested precision and execution policy were not recorded; loading does not invent that missing intent. Select the desired plan before .warm_start(restored) to continue under a deliberate profile; precision changes apply consistently to parameters and Adam moments and are reported. Exact continuation is scoped to the same effective precision, backend, rows, and optimizer settings; a device or precision change can alter reduction order and rounding.

Repeatable stress and performance measurements

Build the native aner_gpu_benchmark target alongside Aner, then run:

aner_gpu_benchmark --device all --dtype all --repetitions 3
aner_gpu_benchmark --device cpu --dtype Float32 --size 512 --repetitions 5
aner_gpu_benchmark --device mps --dtype Float32 --size 768 --repetitions 5
aner_gpu_benchmark --device cuda --dtype Float64 --size 768 --repetitions 5

Use the benchmark executable supplied with the matching native Aner build. Use --dtype Float32 or --dtype Float64 for a precision specific experiment; --dtype all measures supported precisions. Compare each GPU result with the CPU baseline of the same dtype and workload. The benchmark uses strict placement and never adapts precision or selects CPU fallback for a requested accelerator. A specifically requested unavailable accelerator fails. all records unavailable backends as skipped. Output is JSON Lines, suitable for retaining alongside the Aner source, version, compiler configuration, host and GPU inventory, and thermal/power conditions. The executable supports sizes 128, 256, 512, and 768 and 1 through 20 repetitions. Default execution is bounded; repeated stress should be selected deliberately.

Each matrix measurement separates dtype conversion, upload/placement, warmed synchronized resident operation, and readback. Resident timing includes output allocation and finite result validation; it is a runtime operation measurement rather than an isolated vendor library kernel time. Warm up uses the same shape and precision and is excluded. Float32 GPU matrix measurements compare every output cell with matched CPU Float32 results at each supported size. Float64 GPU measurements compare all output cells with CPU Float64 at sizes 128 and 256. At 512 and 768, the CPU Float64 matrix admission limit prevents the full product; the JSON records that omission explicitly, and no matched CPU Float64 matrix speedup is reported. An independent scalar Float64 reference checks all 128 × 128 cells, or 64 deterministic cells for larger products, including larger CUDA Float64 products. The Float64 tolerance is a scaled 2e 12; it is a correctness tolerance for these bounded experiments, not a universal numerical error guarantee.

The runner also measures a 32 row matrix forward pass, reverse differentiation of its mean squared output, and SGD update, checking resulting parameters and loss against the matching CPU precision. Incorrect placement, nonfinite results, excessive numerical discrepancy, or native errors fail the benchmark rather than producing a speedup claim.

For operation duration t, a square n × n matrix product reports approximately 2n³ / t floating point operations per second. Speedup is the same machine's matched CPU operation duration divided by its accelerator duration, using the same precision. Precision, shape, workload, synchronization, and warm up must match. Small workloads can be slower on GPU because context dispatch, allocation, validation, and synchronization dominate useful work. Classifier training_seconds() includes preprocessing, observers, and final metrics; it must not be treated as isolated matrix throughput.

Dev adds bounded statistics, values, gradient captures, and progress observations. Run Prod for performance measurements, then use Dev to inspect placement and training behavior. Notebook observations are available after the cell returns; the current notebook protocol does not stream an in progress fit.

Verified measurements: October 11, 2026

The following earlier baseline measurements use five warmed repetitions, Release optimized native code, and Float32 operands on both CPU and GPU. They time complete resident Tensor matrix operations, including output allocation, finite result checks, and synchronization. Conversion, upload, readback, and independent reference validation are measured separately.

The Mac uses an Apple M2 Max and the Accelerate CPU provider; its GPU is Metal/MPS. The Ubuntu test host uses a GTX 1060 with 3 GiB VRAM and a virtualized Xeon E5 to 2697 v2 with two available vCPUs. Its current CPU backend is the portable single thread cache blocked implementation, without an installed optimized BLAS provider. The large CUDA ratios describe this particular native CPU baseline and virtual machine; they do not predict a comparison against optimized CPU BLAS or another host.

Smaller workloads do not always benefit. On this Mac, 128 × 128 multiplication took about 0.047 ms on CPU and 0.248 ms on MPS; 256 × 256 took 0.155 ms and 0.226 ms respectively. The separate 32 row forward/backward/SGD workload at width 512 took about 0.949 ms per update on Mac CPU and 2.529 ms on MPS. On Ubuntu, the same width took 17.877 ms on its portable CPU backend and 0.659 ms on CUDA. These training step timings include graph admission and multiple synchronized tensor operations. They are not full classifier training or large model benchmarks.

All measured GPU outputs were compared against CPU Float32 results, and the independent double precision checks passed. Timings are observations of the stated workload and configuration, without confidence intervals or a guarantee of the same performance under different load, power, thermal, memory, or compiler conditions.

Native CUDA precision measurements

The following measurements use the updated precision aware native benchmark on the stated GPU, five measured repetitions after warm up, and synchronized resident matrix operations. CUDA runtime 12.4 and cuBLAS execute Float64 or strict Float32, without TF32. Times include result allocation and finite result validation. Input conversion, upload, readback, and reference checks are measured separately and excluded from these resident times. The ratio compares Float64 and Float32 on the same GPU and workload; it is not a CPU/GPU speedup.

Both GPUs executed native Float64; a larger Float64 cost on the GTX 1060 is not evidence that its hardware lacks double precision. These measurements support an explicit performance choice, without making Float32 the compatibility requirement on either GPU. The V100 ratios also vary with matrix size, showing why a fixed precision speedup should not be promised.

The 256 × 256 V100 Float64 result was compared with all 65,536 CPU Float64 result cells. At sizes 512 and 768, CPU Float64's existing matrix work limit prevents that full product; each GPU result instead passed the independent scalar Float64 check at 64 deterministic cells. At those two V100 sizes, the maximum observed scaled discrepancy among the sampled cells was respectively 8.70e 16 and 4.13e 16 for Float64, and 2.31e 7 and 4.95e 7 for Float32. Scaled discrepancy is abs(result - reference) / (1 + abs(reference)). Each precision's reference uses the corresponding input values; these comparisons do not include Float64 to Float32 input rounding. Sample maxima are not whole matrix error bounds or accuracy guarantees for another workload.

The separate 32 row forward/backward/SGD benchmarks compare every updated weight and the loss with the matching CPU precision at all supported widths. Native kernel and interpreter integration checks ran on both GPU architectures, and the CUDA memory checker reported zero errors on the GTX 1060. These establish bounded execution and numerical agreement; they do not measure large network convergence or peak hardware utilization. As with the earlier table, no confidence intervals were measured and timings can change with load, power, thermals, software, and operation shape.

Complexity and current limits

Let n be stored elements, G reachable graph nodes, m × k and k × p be matrix operands, and P trainable parameters. GPU parallelism changes constants and achievable throughput, not these total work bounds.

Differentiable publication also validates graph admission in O(G) time and temporary metadata. Each Tensor is limited to one million elements. CPU Float64 matrix products retain the 50 million multiply add bound; CUDA Float64 and CPU/GPU Float32 products allow one billion. Graph, architecture, and classifier admission bounds otherwise remain in force. These checks are deterministic workload limits, not a total RAM/VRAM quota. A device allocation failure remains an explicit error.

NDArray remains CPU typed storage, and GPU placement is currently a Tensor feature. This implementation targets rank two Dense workflows and explicit Tensor experiments. General rank autograd, CNN/RNN layers, mixed precision within a computation, multiple local GPUs, device memory accounting, distributed execution, and large model scheduling require separate implementations and validation. See Tensor, neural workflows, and development mode.