LLM inference execution and optimization

This guide explains Aner's inference execution paths, measurements and limits. See native generative models for the generation APIs.

A fast LLM needs an efficient execution plan as well as a native host runtime. Useful changes reduce repeated device launches, avoid unnecessary tensors and transfers, and keep model state resident. The expected benefit depends on model dimensions, batch size, context, precision and hardware. The measurements here cover the declared reference workloads; other models and workloads require their own evaluation.

Keeping work on the device

An execution plan can reduce repeated dispatch, allocation and observation work. The useful choice depends on the shapes, sequence lengths and output policy:

A published inference study provides one example of these mechanisms. The evidence on this page comes from Aner measurements with separately stated workloads and timing boundaries.

Skipping unused cache regions can add work through smaller blocks or more dispatches. Compare variants over the intended context lengths and batch sizes rather than assuming that fewer cache reads always improve latency.

Three Aner execution paths

Choose the facility that matches the intended work. Their architecture and implementation boundaries differ.

A native application can use an external native engine or numerical library. These boundaries should remain visible when evaluating a performance claim. Native host code also pays GPU launch and synchronization costs. A host loop that waits after each primitive is different from one compiled device program.

The native reference decoder has a detached typed KV arena, fused cache append and attention, final row vocabulary projection, packed cached batching on CUDA and Metal, and optional CUDA graph replay. These existing features are documented in the generative guide. The native model loading guide and pretrained training guide describe their separate compatibility and lifecycle contracts.

Adaptive greedy selection

At temperature: 0.0, native reference generation selects the maximum allowed score without sorting the complete vocabulary. A generation specific helper applies the tokenizer's exclusions and returns one winning ID for each active row. CPU and small Metal workloads use a linear host scan; larger qualified accelerator workloads can reduce on the device before reading the selected IDs. This choice avoids adding a device dispatch when it costs more than the host work it replaces.

For native byte/BPE tokenizers, BOS and PAD remain excluded. The supported GPT2 shared BOS/EOS ID remains an eligible stop token. Equal allowed scores choose the lowest token index. Finite validation includes excluded scores. EOS accounting, prompt insertion, output budgets and cached/uncached behavior keep their existing meaning. Generic nn.argmax and public all logit APIs retain their contracts.

For an active batch of B rows and vocabulary V, the host scan uses linear selection work instead of sorting V candidates per row. Where the device reduction is selected, its selection readback changes from B * V score values to B IDs. The small output helper policy still reads scores, and the vocabulary projection and its score buffer still exist in all cases. That selector increment alone observes selected tokens on the host each step; resident cached Metal execution replaces that loop. It does not remove every synchronization or promise a proportional speedup.

Outside the resident cached Metal greedy path described below, the selector helper uses a host scan on Metal when the total active logit count B * V is less than 65,536. CUDA uses the host scan below 32,768 active logits. At or above each threshold, the provider reduction reads back only winning IDs. CPU always uses a linear scan. These practical cutoffs come from measured Apple M2 Max and V100 workloads; they are implementation choices, not universal hardware optima or new generation arguments.

The cached Metal plan keeps packed output rows on the device, removing the prior per row logit download and repacking before selection. The private primitive reference and uncached batch path retain that older flow. That whole forward increment alone retained host selection for small Metal outputs. Resident cached Metal greedy execution performs selection inside its decode commands and bypasses the helper threshold; sampled and uncached paths retain their existing behavior.

Positive temperature generation continues to use the established seeded sampler and filtering policy. It has a different numerical and random stream contract. The optimization applies to the temperature zero path rather than silently changing sampling behavior.

Validation should compare all generated IDs with the previous host selection, including masks, ties, EOS, empty prompts, independent calls and mixed row lengths. Device tests must execute on the requested provider and reject fallback. Numerical logit checks remain separate from exact token equality. A measured before/after result must state its model, vocabulary, active rows, dtype and timed region. A reduction in returned values alone is not an end to end speed measurement. Dispatch thresholds are implementation choices based on measured workloads and may change after further qualification.

Whole decoder execution on Metal

Cached native reference generation and supported GPT2 imports bind a reusable Metal Float32 decoder plan automatically. One command buffer contains the MPS matrix products and Metal compute stages for the complete forward, rather than waiting after each primitive. Both single and ragged batches retain activations, causal KV and selected final row vocabulary projection on the device. Packed batches skip inactive rows and avoid downloading, packing and reuploading each row's scores before selection.

The plan owns scratch and detached effective weights, including prepared LoRA projections. Every returned output Tensor owns independent immutable storage; later appends, workspace trimming/regrowth and plan destruction cannot change it. Accepted row positions commit together only after the complete numeric status check succeeds. Failed suffix writes remain outside the accepted history, and shorter or later retries overwrite them.

Private per plan diagnostics show one submitted command buffer and one completion/status read per admitted forward, including submitted numeric failures. Host admission errors leave those counts unchanged. The diagnostics count host submissions and encoded work, not device instructions or latency. KV growth copies, setup and token selection are separate: they can add work outside the counted plan forward. Known generation horizons are reserved once, while unknown horizon internal sessions can still grow their cache.

The sticky status flag checks the complete forward, including intermediate Float32 variance overflow that the prior primitive path could conceal through inverse zero and a later zero gain. LayerNorm preserves the previous affine expression and its rounding. A regression exposed a one ULP difference that changed a GPT2 greedy winner; restoring that expression fixed exact cached/full-prefix token parity without introducing approximate ties or weakening the assertion.

Qualification includes independent CPU Float64 full prefix logits using the exact placed Float32 parameter values, a separately selected legacy Metal primitive reference, native/GPT2/LoRA/merged families, odd and boundary widths, eight layers, 64 heads, ragged row horizons, zero logit mirrors during append, held outputs, early/late numeric errors, shorter retries and concurrent distinct plans. A wide prefill test verifies that a decode sizing hint releases at least 128 KiB of owned workspace without submitting work or changing KV/cursors, then verifies regrowth and immutable unread outputs against the CPU oracle. Buffer accounting excludes driver, allocator, command buffer and other untracked overhead.

This work changes the bounded aner.gen cached path. Generic Tensor/autodiff, optional LibTorch training and llama.cpp inference keep their separate implementations. The whole forward refinement and its preserved measurements below were qualified on the Mac. It supplies the execution stages for the subsequent resident Metal greedy request described next; sampled and uncached execution retain their host selection policy.

The Metal decoder report and benchmark source retain the new implementation's measurement protocol and scope. The earlier selector studies below are preserved independently and must not be substituted for whole decoder results.

Final resident greedy measurements on the Mac

The final gradual startup policy improves longer reference requests and regresses some short EOS cases. Eight row, sixty four output requests change from 24.751 to 15.017 ms (1.648x), while a controlled eight row EOS at five request changes from 4.124 to 4.989 ms (0.827x). The tables retain all 31 profiles, including controls and regressions. These comparisons use two Aner implementations on one M2 Max, with bounded reference fixtures and newly initialized weights. They measure execution behavior; pretrained model quality requires separate evaluation.

The baseline already has whole decoder Metal plans and packed outputs, with selection observed on the host each step. The final candidate adds resident greedy state, gradual decode chunks capped at eight, the small output one token bypass and specialized masked/unmasked Metal stages. Profiles use Float32, a 259 token byte vocabulary, four heads and context 96. Width/layers are 32/2 unless specified. Tiny and long profiles have eight token prompts and budgets of eight and sixty four. Ragged prompts have lengths 1,3,7,31,2,5,13,47 and budgets 32,1,16,2,24,3,8,4.

Each profile retains four matched pairs of fresh processes, ordered AB,BA,BA,AB, with rotating profile order. Each process excludes two warmups and records three complete public generation calls: twelve calls per implementation/profile and 744 timed calls in 248 fresh processes overall. Medians below use all twelve calls. The 95% intervals exhaustively enumerate all 256 ordered bootstrap draws of four matched process pairs with replacement. Every draw retains all three calls from each selected process and recomputes the same all call median ratio; percentile endpoints use linear interpolation. Inner calls and batch rows are not independent uncertainty units. Only four independent pairs support each interval, and slow observations remain in the data.

Timing includes validation, prepared weights, allocations, prefill, decode, selection, required output observation and release. Model construction/placement, startup, compilation, separate score/KV oracles and parameter serialization are outside timing. Both versions use matched Release builds, strict arithmetic settings and LTO disabled. Every timed profile repeats its own IDs and retains identical baseline/candidate IDs and actual row counts. Exact parameter serialization also matches.

Ratios are baseline time divided by final policy time; values below one indicate a slower candidate. All times are complete cohort wall times, not individual completion latency obtained by dividing by batch size.

Mixed EOS uses prompt lengths 1,57,58,59,60,61,63,65, budgets 64,31,31,31,31,31,31,31, and actual output counts 64,9,8,7,6,5,3,1. Mixed zero budgets use prompt lengths 0,1,8,3,8,0,31,5 and budgets/actual counts 0,1,0,8,64,1,0,16. These cases retain independent row positions and fixed positive budget decode lanes.

The controlled EOS profiles below use synthetic weights with zero attention/MLP projections and position directions that give a prescribed EOS margin. They have eight token prompts and declared budgets of sixty four, with full decoder stages still executed. They therefore exercise resident requests even when EOS arrives immediately. Both single and batch cases are retained where collected.

Fewer completion boundaries do not guarantee lower latency. EOS at output five schedules eight forwards under gradual startup, including three masked tails; EOS at output nine can schedule sixteen, including seven masked tails. The CPU still encodes those stages. A separate cost persists across boundaries when only some rows have ended: matrices retain every initially positive lane, with zero GEMM slots for ended rows. The active count read stops an all ended request; it does not compact a partially ended cohort. Forward counters count whole matrix stages, rather than useful live row forwards. Integer state/setup and selection kernels add further work. Immediate EOS single/batch, EOS at three single, EOS at five batch eight, EOS at six single/batch and EOS at nine batch eight intervals lie entirely below one in this capture. The final EOS at two intervals span one, while the initial fixed eight experiment remains separately retained. The final policy still has short request regressions.

The controls below use width 32, two layers and Float32. One token controls declare budget one and stay below the 65,536 logit threshold, so they use the existing public one step host scan. Zero budget controls perform admission without model forwarding. Sampled tiny requests use temperature 0.7, top k 8 and top p 0.95, with the established seeded sampler.

The final sampled, CPU, zero budget and one token control intervals span one, and all seeded IDs match. The preceding dynamic mask capture retained a slower batch eight sampling control: ratio 0.893 [0.736, 0.962]. Its baseline median also differs across captures, so the final control result alone does not establish a causal batch eight fix.

The final implementation specializes masked and unmasked Metal stages, avoiding unused mask bindings and dynamic mask branches in ordinary forwards. A separate comparison with the identical producer object and all other libraries held fixed supports a bounded warm single row sampling benefit: 1.091 [1.026, 1.357]. Batch eight sampling, 1.038 [0.957, 1.175], and long requests, 0.954 [0.815, 1.117], remain uncertain in that direct comparison. Seven additional pipeline states are created during initialization; this study excludes that setup and does not measure their cold start cost or opaque memory.

For the long batch eight case, tracked all buffer peak increment changes from 365,636 to 351,464 bytes, while the owned device peak increases from 349,060 to 351,464 bytes. The Tensor readback mirror peak changes from 16,576 to zero. Both versions return tracked storage to its starting level. These categories exclude ordinary returned C++ vectors, allocator/driver/command-buffer/BLAS overhead and RSS; Metal uses unified memory. The resident request retains its bounded prefill workspace and integer/output storage throughout the call, so this example establishes no general memory reduction.

The complete resident greedy report retains final observations, the initial fixed cap and intermediate dynamic mask experiments, the direct specialization comparison, exact protocol and source/build identities, score/KV checks, counters and qualification. The benchmark source declares workload and timing boundaries. New Metal measurements and separate V100 regression checks remain distinct from the earlier studies below.

Resident Metal greedy requests

Cached Metal Float32 single and batched generation at temperature: 0.0 use a resident request when the largest output budget is greater than one or a one token request has at least 65,536 packed logits across positive budget rows. Current IDs, device positions, live masks, output counts and produced IDs remain inside that request. Prefill selects the first output in its command buffer. The CPU then encodes decode commands of one step, two, four, and up to a maximum of eight, with one completion/status and active count observation at each successful boundary. An admitted remaining budget can shorten the final command. Public calls whose budgets are all at most one use the existing one step host scan only when positive budget rows times vocabulary is below 65,536. Larger one token outputs retain fused resident selection, avoiding a separate helper reduction command. An individual budget one row in a longer batch remains part of its resident request. This is a CPU encoded plan with device resident recurrence, not a GPU generated command graph.

Zero budget prompts are validated before all device work and do not receive an executing lane. Positive budget rows keep their initial fixed lanes for the entire request, including later chunk boundaries; ended lanes are masked before embeddings, affine biases, normalization, activation, attention and KV writes. No boundary compacts those lanes: one surviving row still has the original matrix row count with zeroed ended slots. EOS or budget exhaustion inside a chunk masks the remaining encoded steps. If all rows end, another command is avoided after that boundary; immediate EOS in prefill avoids every decode command. Selected EOS and final budget tokens are included in results without being forwarded. Caller row order, duplicate prompts, empty prompt BOS insertion, strict prompt/context bounds and the lowest allowed ID tie rule retain their meaning. Shared GPT2 BOS/EOS stays eligible; a separate BOS and PAD are excluded.

The first retained capture rejected a fixed eight startup policy. Controlled EOS at the second output changed from 1.926 to 4.540 ms for one row and 2.003 to 4.234 ms for eight rows. A public one token single request changed from 0.922 to 1.017 ms; its baseline/candidate ratio was 0.907 with a process paired 95% interval of [0.847, 0.941]. These observations motivated gradual startup and the public small output one token bypass; they do not justify extrapolating that bypass to larger device reduction workloads. They remain initial diagnostic results, separate from the final policy remeasurement in the resident greedy report.

The resident request retains its bounded prefill workspace for the entire call, together with private integer state and budget sized device ID output storage. It bypasses the automatic sizing hint used by the preceding stepwise public loop; explicit append plans still support checked trim/regrowth. Reduced submission or readback counts do not establish lower retained memory.

Only the successful final produced ID prefixes are copied to the host: 4 * emitted_tokens bytes of uint32 values, followed by ordinary caller owned integer vectors. Final counts/positions and chunk boundary status/active counts are separate metadata reads. There is no Tensor score mirror or per step host token observation. Metal's buffers use shared unified memory, so the ID byte counter does not measure physical device transfer traffic. The vocabulary score buffers and projection arithmetic still exist on the device.

Per plan diagnostics count actual command boundaries and separately count scheduled forwards, which can include masked tail steps after the last live row ends. A failed numeric boundary counts its submitted command and completed status check, then throws before reading an invalid active count. The maximum chunk size is a cap, rather than a fixed size: decode starts at one and doubles to that cap. With output budget 17 and cap eight, a request without EOS encodes prefill followed by decode lengths 1,2,4,8,1, for six commands and seventeen scheduled forwards. EOS at the second output ends after prefill plus one decode, with no larger command scheduled. The private qualification seam checks caps 1..32, startup/EOS outputs one through nine, and longer remainder cases. These are submission counts rather than GPU instruction counts or speed predictions.

Fresh request state commits only after the entire request succeeds. A numeric error after earlier chunks exposes no partial IDs or accepted cursor, and a retry overwrites uncommitted suffixes. Independent plans own their masks, status and output storage. The backend mutex is held for the complete resident request, so concurrent tests establish isolation rather than overlapping execution or fair interleaving. Tests compare exact IDs with the prior one step Metal loop and independently compare full prefix CPU Float64 logits from exact Float32 weights. Controlled fixtures additionally require the exact CPU winner and establish separate Float32 overflow witnesses for poisonous EOS/final-budget embeddings and otherwise unused ended lane bias/attention/GELU stages. The imported sinusoidal GPT2 fixture retains exact one step Metal ID parity and numerical CPU score checks because widening precision can change near tied winners.

Final gradual startup Mac Metal qualification passes 11,990,172 assertions in the dedicated resident request suite, including large one token GPT2 batches, with maximum observed scaled full prefix score error 5.78247e-8, using abs(actual - expected) / (1 + abs(expected)). The existing MPS single cache suite separately passes 2,392,539 assertions, and the generic GPU/direct-provider suite passes 267. These overlapping suites are not added into a count of independent experiments. The final Mac regression campaign executes 283 normal test groups: 274 pass, nine unavailable platform or live service groups skip, and none fail; six separate research campaigns are excluded from that normal run. AddressSanitizer/UndefinedBehaviorSanitizer pass the final resident, direct provider and CPU Float32 cache suites. The actual final resident Metal suite also passes with Metal API and shader validation confirmed enabled. Timings from these instrumented runs are excluded from performance claims. The initial fixed chunk campaign passed 11,143,784 resident assertions; that historical count belongs to its separately retained rejected policy experiment. The sealed evidence report retains the exact source/build identities and logs.

Separate V100 32 GB qualification passes 13 native CUDA Float64/Float32 regression groups and three Compute Sanitizer memcheck cases with zero reported errors; cache/batch producer smoke checks also pass in both precisions. The final follow up changes only the Metal file, which is not compiled in the Linux CUDA build, and every recorded compiled shared/CUDA source hash matches the accepted final source. These checks qualify CUDA correctness regressions, rather than Metal performance or a cross hardware speed comparison.

This refinement applies to the bounded native reference decoder and supported GPT2 import, including detached prepared LoRA weights. Positive temperature seeded sampling, uncached references, CPU and CUDA keep their existing execution paths. The GGUF inference engine and pretrained trainer retain their separate implementations. The resident greedy report and benchmark source distinguish new Mac Metal request measurements from separate V100 regressions and all earlier studies below. Model construction, source/build identities, timing boundaries, useful emitted work and masked scheduled work must be reported independently.

Earlier whole decoder measurements on the Mac

This preserved study compares the primitive implementation with the whole decoder Metal candidate, before resident multi step greedy, on one M2 Max. All profiles use Float32, a 259 token byte vocabulary, four attention heads, context 96 and new seeded weights. Width and layer count vary as shown. Uniform tiny requests have eight prompt tokens and request eight new tokens per row; uniform long requests request sixty four. The ragged prompts have lengths 1,3,7,31,2,5,13,47 and output budgets 32,1,16,2,24,3,8,4. These bounded reference decoders are not Gemma or pretrained language quality tests.

Each profile and phase retains four matched pairs of fresh processes, implementation order AB, BA, BA, AB, and rotating phase/profile order. Each process excludes two warmup calls and records three timed calls. The table reports complete natural generation medians across twelve calls per implementation, including preparation, allocation, prefill/decode, selection, required output readback and release. Model construction/placement, startup and the numerical oracle are outside timing. Both builds use the same Release compiler and strict arithmetic configuration with LTO disabled.

Ratios compare baseline time with candidate time. The 95% intervals resample whole matched process pairs: there are only four independent process clusters per profile. The wider single tiny interval retains the observed baseline variability; no slow observations were discarded. Batched ratios combine whole forward submission with genuinely shared packed computation, replacing the prior sequential Metal row execution. They are complete cohort times, not individual latency divided by batch size.

Every natural profile retains identical baseline/candidate output IDs and actual per row counts, and each runtime repeats its own IDs. Exact parameter serialization also agrees outside timing. The unchanged CPU control has an interval spanning one and establishes no CPU speedup. These ratios compare the two Aner implementations on these workloads; they are not a comparison with another framework or a prediction for larger models.

The primary equal work forced scans are retained separately in the report, with no EOS shortcut or timed score inspection. Their candidate diagnostics record one command buffer and status read per forward, and no append logit readback mirror. Legacy diagnostics are unavailable; their zero fields must not be interpreted as zero device work. Natural generation still performs its required final score readback under the small output selection policy.

Faster packed execution can retain wider device workspace. For the eight row, sixty four token profile, the reported natural owned peak increment changes from 348,832 to 365,636 bytes, while owned device peak changes from 307,200 to 349,060 bytes. Both versions release tracked buffers to their starting level. These counters exclude driver, allocator, command buffer and other untracked memory and do not establish a universal memory reduction.

The complete Metal report preserves natural and forced results, raw observations, counts, numerical checks, source/build identities and memory scope. The benchmark source declares the exact workload and timing boundaries. Qualification for this refinement is Mac only; the previous CUDA study below remains historical evidence.

Earlier greedy selection study on V100

This earlier study compares the preserved Aner baseline with the greedy selection candidate on one V100, before the whole decoder Metal refinement. Its recorded data and tables are unchanged. The model has new seeded weights, a 259 token byte vocabulary, width 32, four attention heads, two layers and context 96. Prompts have eight tokens, dropout is zero and generation is greedy with the cache enabled. These measurements describe a small reference decoder with newly initialized weights; pretrained language quality was not evaluated.

Each profile uses four matched pairs of fresh processes, with implementation order AB, BA, BA, AB and rotating profile order. Each process excludes two warmup calls, then records three timed calls: twelve timed calls per implementation and profile. Native synchronization brackets the complete generation call, including request preparation, inference, token selection, required output transfer and cache/workspace release. Model construction and placement, process startup and the two warmups are outside the measured calls.

The table reports median call times and the ratio of baseline to current time; a ratio above one favors the current implementation. Intervals are 95% bootstrap percentile intervals resampling whole matched process pairs. There are only four independent process pairs; the twelve inner calls are clustered observations and do not provide twelve independent process replicates.

Every recorded generated ID agrees between baseline and candidate. Within each precision, the independent parameter check produced byte identical serialization of the same twenty parameter tensors and 36,256 scalar values, with names, shapes and exact values checked outside timing. Each row produced its full stated token budget: eight or sixty four tokens per sequence. Graph requested identifies the public option; admission, capture and replay remain subject to the native graph policy.

All these generation cases have fewer than 32,768 active logits, so the current CUDA policy selects the linear host scan. Their gains measure removing unnecessary sorting in the complete native request; they do not establish the benefit of device argmax, fully resident generation or an absence of score readback. Larger vocabulary selection is measured separately from model forward work.

Read the measurement data and source identities, the complete generation producer and the selection stage producer. The retained data distinguishes complete request measurements from fresh resident logit selection experiments. These results describe these workloads on one device and do not promise the same ratios for larger models, different tokenizers or other hardware.

Earlier greedy selection study on the Mac

Before whole decoder Metal plans, the same reference decoder and full request timing boundary were measured on the M2 Max, using matched Release builds with Accelerate and primitive native Metal. Four pairs of fresh processes each excluded two warmups and retained three calls; all 144 timed calls and all token IDs are retained in the data. There are only four independent paired process clusters per profile.

CPU medians improve by about 5 to 8% in these small requests. The MPS complete request medians are approximately unchanged, and their intervals include one: this study establishes no whole request Metal speedup. That captured source used frequent synchronous primitive submissions and per row batch packing; it motivated the separate whole decoder Metal implementation described above.

In the isolated large selection case of eight rows and 32,768 scores per row, MPS selection changes from 15.826 ms for the old sort to 0.641 ms for the adaptive selector (24.68x); a linear host scan takes 1.247 ms (1.95x relative to device selection). This excludes model forward work and must not be presented as a whole LLM speedup. Small MPS outputs use a linear host scan to avoid an extra device launch.

Where further work belongs

The following directions are engineering priorities, not capabilities introduced by this guide:

1. Broader Metal plan qualification. Extend measured model/batch/context regimes, inspect profiles and retain the primitive reference. Whole decoder execution and device packed outputs are implemented for the declared bounded cached families; their presence alone does not establish a universal speedup.
2. Resident loop policy and serving boundaries. Qualify the implemented cached Metal greedy chunks across more vocabulary/context/batch regimes, including masked tails and fixed ended lane cost across boundaries. Device row compaction at boundaries, streaming, cancellation, dynamic request batching and an equivalent CUDA loop remain separate work.
3. Attention across longer contexts. Measure online softmax tiling for prefill and decode separately, and retain MQA/GQA head sharing without copying K/V into every query head.
4. Pretrained trainer attention. Qualify a memory efficient backend against gradients, deterministic profiles, adapter updates, loss and exact checkpoint continuation. The current dense attention path remains a correctness reference.
5. Bounded pretrained serving. Design scheduling, KV ownership, request admission and context reuse before comparing service throughput. Shared adapter weights require their existing synchronization until a proven replacement is available.
6. Distributed transformers. Persistent device replicas and GPU collectives need a separate design from today's portable host buffer Dense training protocol. More host tasks do not automatically make one GPU decode step faster.

The Mac backend uses Float32 neural kernels. The V100's Tensor Cores use FP16 matrix inputs with FP32 or FP16 accumulation, as described in the Volta tuning guide. Precision changes must be explicit and independently qualified. A device being available does not prove that a particular fused kernel or dtype ran.

How to assess a result

Keep loading, warmup/setup, prefill, decode, final transfer and text processing separately identifiable. Synchronize at declared measurement boundaries. Retain all repetitions, token counts, numerical checks, placement, memory scope and failed observations. Diagnostic profiles with extra synchronization should be separate from reported throughput.

A fair comparison uses the same architecture and weights, prompts/tokenizer, batch layout, context and cache capacity, dtype and accumulation policy, output length, EOS policy and timed output boundary. Compare with a suitably optimized baseline. Random weights can isolate execution mechanics, but do not establish language quality, correct real checkpoint mapping or general model compatibility.

Use neural sequence primitives, GPU placement and precision, parallel computing and the linked model guides to inspect the current contracts. Shape, graph, memory and work limits continue to apply.