Distributed Dense training benchmark

This controlled experiment measures native Dense training on CPU, Metal/MPS and CUDA, separately and in explicitly configured groups. All 57 measured trials across 19 cases passed the fixed scientific and numerical checks. Nineteen additional warm ups are excluded from timing statistics. The fixture tests portable execution and learning quality; it does not establish real world task accuracy or peak hardware throughput.

Scientific contract

Every selected worker returned gradients in its declared native backend. Global row counts, source and partition fingerprints, fixed update counts and declared weight/tail allocations agree. Validation/test rows never supply gradients or normalization statistics. GPU replicas perform forward/backward; Dataset preparation, canonical updates and final fitted evaluation use CPU.

The native source contract is 43aaceb4725f52a5136643b824d790c26600f92378c6cf753287c0b069d5211a. Dataset source identity is 72bde3eff8b278c07be2746ea50d317557c7534244eee0452f1f7907e0f36562. The complete Aner exploration source and distributed neural guide expose the scientific and execution choices.

Platform configurations

These are configuration capacities, not measured process use or dedicated core guarantees. Atlas identities are opaque inventory numbers; configuration revisions preserve hardware/software policy changes. Earlier MNIST measurements and CPU task measurements retain their historical configurations and results.

Duration and scaling

Open the duration chart · Download PNG

The table gives median and observed minimum: maximum seconds over three repetitions. These ranges are descriptive, not confidence intervals. Complete command time includes source generation/partitioning, preprocessing, placement/process/SSH initialization, training, validation/final test and shutdown. All cases omit native checkpoint/report file I/O. Compilation and source staging occur outside the timing window. Native fit and complete command time have separate boundaries: standalone backend initialization may precede fit, while configured worker initialization can occur inside fit.

Float32 had a shorter median CPU fit on Atlas-01, while Float64 had a shorter median CUDA fit on Atlas-03. Both precisions retained the same held out test accuracy in this fixture. Precision alone does not determine elapsed time or all host storage peaks; the backend and complete update/staging path are part of the measurement.

0 of 8 groups had a lower median native fit time than their fastest selected standalone placement. Values below 1 indicate slower group execution; unfavorable outcomes remain in the report. Baseline selection is independent for fit and complete command time, and the JSON identifies both. This workload does not imply that two computers always improve training speed.

Learning and numerical agreement

Open the validation chart · Download PNG

The plots use actual detached post epoch validation observations. Pre update online batch loss is a different phase and is not presented as a final model loss snapshot. Final held out test measurements appear in the table; test rows remain outside model selection. Curves can overlap despite small numerical differences.

Each dtype uses its predeclared standalone CPU reference and first measured trial. Complete retained parameter vectors are compared before publication; hashes and independently recomputed errors are public, while full vectors stay internal. Cross precision quality is descriptive: Float32 and Float64 parameters do not use one shared tight tolerance. The unequal Float32 CUDA allocation case uses relative weights 1:7 but retains the same global rows and one update per batch.

Memory and phase observations

Open the memory chart · Download PNG

Coordinator RSS is a native high water query before final parameter JSON and shutdown. Worker RSS is queried during accepted gradient steps. Inline coordinator/rank share a process, and those query points differ. The largest rank is selected per trial. Tracked Tensor host/device capacities exclude raw Dataset source, allocator overhead, provider scratch and driver context. Peaks are not simultaneous totals. MPS shared buffers overlap host RAM and must not be added to it. External driver process VRAM remains N/A wherever no PID matched sample was obtained.

Native receipts separately report preparation/upload, forward/backward, readback, worker total duration, request writing, response waiting, payload bytes and coordinator exchange time. Response waiting includes worker computation and transfer; it is not pure network latency. Worker compute is measured with a host steady clock around native forward/backward and completion synchronization, including dispatch/backend work; it is not kernel only GPU event time. Upload/preparation also includes validation, casts and allocations. Worker total time excludes protocol decoding/encoding, checksums and reply delivery. Payload bytes exclude protocol/SSH framing. Request parameter/feature/target values use the selected Float32 or Float64 encoding; detached gradient sum replies use Float64. Metadata and decoded host exchange storage retain their own fixed scopes, so compute dtype alone does not halve all communicated or host staged bytes. Effective link bandwidth, physical host dedication, CPU/GPU utilization and pure network only latency were not measured and are unavailable.

Reproducibility and limits

Download all 57 sanitized trials and 19 excluded warm ups · Download 19 case summaries · Download checksummed public configuration facts

A separate source language correctness exercise used the protected Iris split (90 training, 30 validation and 30 test rows) across the three host execution context. The simple production, development and class based examples agreed: held out accuracy 96.6667% and cross entropy 0.17894501984119415. Development captures identify canonical CPU reference graphs explicitly. These checks verify the source API and modes; their accuracy and timings are not synthetic fixture benchmark results.

The records preserve every same dtype comparison, actual per rank counts/placement, source and partition fingerprints, measured phase/resource scopes, selected baselines and immutable configuration IDs. Private SSH routes, local paths, PIDs and complete parameter vectors are excluded.

This portable path rebuilds local native graphs and stages supported parameter/batch buffers each update. Every replica holds a complete model; memory is not pooled. Direct CUDA collectives, multi GPU per node selection, model/optimizer sharding, convolutional/recurrent models and durable distributed restart were not measured or implemented by this experiment. The synthetic teacher is a controlled numerical fixture. Its accuracy is not evidence about a separate real world task or the hardware performance ceiling.