Native local inference optimizer

The inference optimizer requires Aner 0.0.7 with the native LLM backend. The optional embedded engine uses the pinned llama.cpp revision in the native build configuration. See native local models for loading, compatibility, sessions, adapters, ownership, and native compilation, and the API guide for HTTP.

Aner owns model admission, typed handles, immutable execution profiles, cache isolation, scheduling, request limits, and measurement. The pinned engine executes the transformer and CPU, Metal, or CUDA kernels. The runtime does not download models, choose an external service, or change an explicit device request.

Choose a profile for the workload

Choose a latency profile with prefix reuse and optional prompt lookup, or a throughput profile with continuous batching. Combining those two profiles is not admitted in this increment. Unsupported combinations fail during configuration.

Configure immutable settings

import aner.llm

let base = llm.load("models/local-instruct-q4.gguf", context: 4096,
    device: "auto", kv_cache_type: "f16", flash_attention: "auto")
let tuned = base.with_optimizer(
    threads: 0, prefill_threads: 0,
    batch_tokens: 512, micro_batch_tokens: 128,
    prefix_cache_entries: 1, prefix_cache_tokens: 2048,
    speculative_tokens: 0)
base.close()
let response = tuned.generate("Explain this local inference profile.",
    max_tokens: 128, temperature: 0.0, seed: 7)
print(response.text())
print(response.timings())
print(tuned.memory_summary())
tuned.close()

llm.with_optimizer(model, ...) is the equivalent module form. The builder returns a new stateless lease sharing the same resident weights and effective adapter. It leaves the source unchanged, creates a separate compatible context pool, and preserves the model fingerprint. Configure before creating a session or a continuous scheduler. A persistent session rejects reconfiguration.

Optional arguments are trailing positional prefixes. Named arguments can be reordered within the selected overload, but setting a later option still requires all preceding options. For example, include the four thread/batch options when setting prefix_cache_entries; include all seven options for speculation.

A zero sized cache still requires a valid positive prefix_cache_tokens setting; this number does not enable caching by itself. Logical batch size one leaves no room for draft proposals and therefore performs ordinary single token decoding. The settings describe admitted engine limits; they do not guarantee that every kernel uses that many CPU threads or tokens.

For GPU execution, use a build with the corresponding backend. CPU thread settings still govern host work and any CPU kernels. device: "auto" can choose CPU; inspect model.device(). Explicit "metal"/"mps" and "cuda"/"cuda:0" requests fail when unavailable. This API selects one accelerator, with CPU execution available separately; it does not shard one model across several GPUs.

Understand shared prefix reuse

The shared cache belongs to one derived effective model/profile. Stateless leases created for its API requests share this cache. Independently built profiles, different adapters, and separately loaded models do not exchange KV. A session continues to own its own persistent context and does not borrow from this pool.

For each request the runtime renders the configured chat template and tokenizes the complete messages. It chooses an available retained context with the longest identical token prefix, removes divergent positions, and evaluates the remaining prompt. A fully cached prompt still replays its last token because logits are call local. cached_input_tokens excludes that replay. Template/BOS-only short matches do not displace a useful separate entry while capacity remains.

Only a successful request can return a context to the pool. Before retention the runtime removes every generated position and retains at most prefix_cache_tokens prompt tokens. Failed, cancelled, or invalid output contexts are discarded. Samplers and generated text are never reused. A zero output request validates input without allocating, reading, or changing cached KV.

Cache hits depend on the rendered token prefix, not on similarity, a repeated substring elsewhere in the prompt, or a response text cache. Put stable system instructions and source evidence before a changing user question when that ordering also fits the task. The application remains responsible for supplying the intended messages; the cache does not append hidden history.

Enable shared prefix reuse only for a trusted cache domain. Different application tenants can use independent derived profiles or disable shared retention. Prefix matching is not an authorization boundary. The default remains disabled.

with_context_pool(max_idle) retains its existing 0..4 idle allocation limit. On an optimized profile it preserves the configured prefix retention policy while creating a fresh pool; zero disables idle retention. Its bound covers idle contexts, while API worker/session admission governs active work. It is not a total memory reservation. Ordinary models retain empty contexts and report zero cached input tokens.

Use prompt lookup speculative decoding

let speculative = base.with_optimizer(
    threads: 0, prefill_threads: 0,
    batch_tokens: 512, micro_batch_tokens: 128,
    prefix_cache_entries: 0, prefix_cache_tokens: 1024,
    speculative_tokens: 6)
let answer = speculative.generate(
    "Summarize the following evidence using its original names and terms: ...",
    max_tokens: 192, temperature: 0.0, seed: 7)
print(answer.text())
print(answer.timings())

This example assumes base is still open. Prompt lookup searches bounded recent request history for matching 2..8 token suffixes and proposes a continuation from an earlier occurrence. It scans at most 4096 recent tokens. The target model processes the pending token and proposals together, compares each proposed token with the target's greedy choice, accepts matching tokens, and emits the target's correction at the first mismatch. Rejected and unconsumed KV positions are removed before the next step. Streaming receives only target verified tokens.

No second model, extra model file, learned draft head, or draft model vocabulary mapping is used. This is a concrete speculative technique appropriate to tight memory budgets. It is not draft model speculation or sampled speculative rejection sampling. Temperature above zero is rejected rather than approximating its output distribution. Target verification preserves greedy selection semantics; floating point kernel/batch changes can still affect near tied logits across configurations.

Observe speculative_proposed_tokens, speculative_accepted_tokens, and speculative_verification_steps. High acceptance can reduce forward calls; unsuccessful proposals still cost work. Compare repeated representative prompts with speculation disabled and enabled. A synthetic repeated token fixture proves verification and rollback, not application quality or universal acceleration.

Serve continuous batches through aner.api

import aner.api
import aner.llm

let base = llm.load("models/local-instruct-q4.gguf", context: 4096,
    device: "auto", kv_cache_type: "f16", flash_attention: "auto")
let tuned = base.with_optimizer(
    threads: 0, prefill_threads: 0,
    batch_tokens: 512, micro_batch_tokens: 128,
    prefix_cache_entries: 0, prefix_cache_tokens: 1024,
    speculative_tokens: 0)
let serving = tuned.with_batching(max_sequences: 2)
let app = api.app("Local batched inference").llm(path: "/chat", model: serving)
    .bind(host: "127.0.0.1", port: 11435)
    .limits(max_rows: 64, timeout_ms: 120000)
    .output_limit(max_tokens: 256)
app.serve()

llm.with_batching(model, max_sequences) is the equivalent module form. batch_tokens must be at least max_sequences. Configure adapters and optimizer settings before creating the scheduler. A scheduler rejects session/reset, context pool, adapter, or optimizer builders; it does not silently fall back to another policy. Use another lease to serve named conversation sessions.

curl --fail-with-body http://127.0.0.1:11435/chat \
  -H 'Content-Type: application/json' \
  --data '{"prompt":"Explain why local inference uses KV memory.","max_tokens":96,"temperature":0.0}'

The API supplies separate stateless worker leases that share the scheduler. Concurrent HTTP requests can therefore participate in the same target model call. Within a scheduler every sequence owns its token positions, sampler, output, stop condition, and cancellation state. Each step prioritizes active decode tokens and shares its remaining bounded token budget between prefills. Completed slots are cleared and can admit new work on the next step, before longer requests finish.

The scheduler queue is bounded at eight waiting requests. The HTTP service has its own bounded connection workers and queue. Capacity errors are reported as busy rather than allocating an unlimited backlog. Closing the source model does not invalidate an accepted request's resources. Slow streaming callbacks apply backpressure to this scheduler; recursively submitting work to the same scheduler from its callback fails promptly.

One logical model lease still admits one caller at a time. C++ embedders create native_stateless siblings for concurrent submissions. In Aner programs the API server supplies those siblings; successive ordinary generate calls remain successive requests. Batching does not make a serial caller concurrent.

Budget memory for an 8 GB GPU

Start with a small instruction tuned, supported quantized GGUF and bounded context and output. Use a model whose complete weights leave room for KV, graph/workspace, allocator and device driver storage, and other applications. Weight file size is only one part of the total. CPU execution uses system memory and should be tested separately. An 8 GB discrete card and a larger unified memory Mac are different hardware profiles.

For a dense decoder with equal per layer dimensions, a first order KV payload is:

KV bytes = context capacity × layers × KV heads
           × (bytes for one key head + bytes for one value head)

For F16, each head row uses head_dimension × 2 bytes. For GGML q8_0, each 32 element block occupies 34 bytes, including its scale. Key/value head dimensions must satisfy the backend block requirements. Real allocations can include padding. This estimate excludes graph/workspace, adapters, allocator, driver and weights.

Begin with one prefix entry or two batch slots, measure real peak memory, and increase only if the representative workload fits. Reducing context, microbatch, slots or retained entries are separate controls with different performance effects. Keep a CPU profile as a functional fallback selected explicitly by the application. The optimizer does not silently downgrade an explicit GPU request after failure.

CPU parallelism and cluster deployment

Tune threads and prefill_threads against the physical host and workload. Replicating several CPU processes while giving each every logical core can oversubscribe the host. Record process count and per process thread counts together. A larger prefill batch can improve arithmetic efficiency but raise transient workspace and delay cancellation boundaries. Generation and prefill have different bottlenecks, so measure both.

Cluster inference currently uses independent replicas: start an explicitly configured Aner API instance on each CPU/GPU host, then route requests through an existing trusted gateway or application client. Each replica loads and verifies its own GGUF. Keep a conversation on its owning session/replica if KV reuse is required. Prefix pools and scheduler queues are process local. Adding replicas increases aggregate request capacity; it does not combine several small cards to fit a model that exceeds one worker's memory.

Native LLMModel handles and KV contexts are not transferred through numerical aner.parallel/cluster tasks. Model inference remains an effectful operation with its own admission and native kernel parallelism. This increment does not implement tensor/pipeline parallelism, distributed KV, cross host batching, automatic replica provisioning, or a cluster load balancer. Follow the API deployment rules for authentication, TLS, ingress limits, and supervised service lifetime.

Measure the actual workload

1. Record aner --version, pinned backend, exact GGUF/adapter fingerprints, requested and resolved device, context, thread counts, batches and profile.
2. Measure cold load_summary() separately from warmed request timings. Both complete model content verification passes remain in place.
3. Use the same prompts, tokenizer, generation budget, temperature and seed for paired comparisons. Keep output hashes and actual token counts with timings.
4. For prefix tests compare cold and repeated identical prefixes, changed suffixes, and independent cache domains. Compare output against uncached inference.
5. For speculation record proposals, accepts and verification calls. Test both copied/repetitive text and workloads with little prompt/output overlap.
6. For batching run concurrent clients with mixed prompt/output lengths. Record arrival to completion latency and aggregate throughput as well as per request first token latency. A serial client is not a batching benchmark.
7. Measure process/device peak memory with host tools. A scoped KV estimate is not a complete measured memory ceiling or an 8 GB qualification result.
8. Report warmup policy, repeat count, medians and variability, model quality evaluation and any output changes. Preserve historical artifact/version labels.

response.timings() uses nanoseconds. cached_input_tokens describes actual prompt KV reuse. context_pool_reused describes allocation reuse and can be one when cached input is zero. prefix_cache_hit, retained entry/token counts, thread/batch settings and speculative counters make the execution path observable. context_clear_duration covers empty context clearing or successful prefix trimming.

Batched responses add queue_duration, batch_max_sequences, batch_peak_sequences, scheduler_steps, and batch_decoded_tokens. Shared step durations and token counts overlap across participating requests; do not sum them as independent device work. The batching KV estimate covers the whole shared scheduler context. first_token_duration and total_duration include request waiting. tokens_per_second() reports generated tokens over measured generation work and is unavailable when no timed decode occurred; aggregate serving throughput requires total completed tokens over external wall time. Missing measurements remain -1 rather than fabricated zeros.

model.supports(...) distinguishes supported operations for the effective lease. A batching lease reports continuous_batching and rejects session/prefix/speculative builders. Closed leases do not claim capabilities. Memory summaries identify configured policies and their accounting scope; the Flash Attention field reports the requested policy.

Verification and references

The native optimizer tests compare output token hashes with ordinary uncached attention, exercise changed prefixes, sampler isolation, budgets, cancellation, sessions, effective adapters, target verified accepted/rejected proposals, streaming, EOS, quantized weights and quantized KV. Batching tests use independent sequence oracles and concurrent callers, verify shared calls, sequence slot reuse, late short requests, bounded queues and recovery. Synthetic fixtures test execution semantics; real model quality and device performance require separate evidence.

• Pinned llama.cpp public API: contexts, batches, KV operations, sampler and precision controls.
• Original prompt lookup decoding implementation: repeated n gram continuations as a draft source. Aner's current implementation is greedy only.
• FlashAttention paper: IO aware exact tiled attention; backend implementation and hardware support remain explicit.
• PagedAttention paper: related serving/memory-management research. Aner's retained context prefix pool is not PagedAttention or shared block paging.

See the Language model connections module reference for the shared API.