Native local model inference

These APIs require the matching development native runtime and optional capabilities. Published Aner 0.0.5 runtime packages and the published editor do not supply these additions. This documentation publishes no runtime, editor package or model weights. See release status and Downloads for separately verified artifacts.

The matching development runtime adds llm.load alongside the existing Ollama and cloud connections. Native loading is an optional build capability. Service only builds continue to use llm.client and do not need llama.cpp or local model weights.

import aner.llm

let model = llm.load("models/research-model.gguf", context: 4096)
let answer = model.generate("Explain the experiment.", max_tokens: 256,
    temperature: 0.0)
print(answer.text())
model.close()

This is local execution inside Aner's native C++ runtime, without a model server, separate language runtime, automatic model download, or remote fallback. Obtain a compatible artifact separately. aner check verifies types and call shape without opening a model. Executing load validates the artifact and loads resident weights before returning. llm.native_available() reports the optional build capability without opening files or selecting an inference provider.

Compatibility and placement

The initial loader admits single file GGUF text models whose architecture is llama or qwen2, with a supported nonempty chat template and a context declared by the model. A .gguf extension does not establish compatibility with every architecture, quantization, tensor mapping, or model revision. Split GGUF models and multimodal input are outside this increment. Actual tensor and quantization compatibility is checked by the pinned engine. Hardware qualification is separate from a backend being compiled in.

Default device: "auto" resolves to an available supported accelerator or CPU; read model.device() to inspect the choice. Explicit "cpu", "metal"/"mps", and "cuda"/"cuda:0" requests do not silently fall back to another device. Metal and CUDA require a runtime containing the corresponding optional backend and a compatible host. Other CUDA device indices are not admitted by this initial API.

context defaults to 4096 and must be 1..65536, no greater than the artifact's declared context. A request's formatted prompt plus requested output must fit. Aner does not implicitly truncate history or extend a model's context.

Public API

Instance methods also have module forms. The first native generation/chat argument is named model; existing service forms retain their client label.

Native output budgets are 0..65536 tokens, temperatures finite in [0,2], and seeds 0..4294967295. Zero output validates the request and returns no generated tokens. Service budgets/defaults retain their existing contracts. Token usage belongs to the actual tokenizer; unsupported usage fields remain -1 rather than receiving fabricated values. An output limit may end an incomplete answer.

generate means one user message formatted with the supported chat template, consistent with the service interface. Raw base model continuation is not an implicit alternate meaning. Chat does not retain automatic history between calls. Prompts and messages retain the established byte/message bounds.

Ownership and resource lifetime

LLMModel is an opaque native handle. Assignment aliases one logical lease:

let alias = model
alias.close()
// A subsequent generation through model now fails because the lease is closed.

Closing a lease rejects new requests and releases its ownership of resident resources. Accepted work retains its resources until safe completion. Metadata accessors remain readable after closure. Automatic final owner release supplements explicit close, including notebook reset and failed execution cleanup.

Each request owns independent KV/sampler/workspace state. Only one active request is admitted per lease; another request fails as busy instead of creating an unbounded context allocation. There is no persistent chat session, hidden strong model cache, automatic deduplication across separate load calls, or sharing across independent processes. Reuse the loaded handle for repeated work.

Adapter derived leases share their base weight owner. Operations on that owner are serialized to preserve shared base inference and adapter state. Independent leases retain separate request state. This milestone does not promise concurrent throughput on a shared base.

Artifact hashing is streamed and checked again after resident loading to detect ordinary source changes during admission. These scans contribute to cold startup. Hashes identify bytes and do not prove authorship or training compatibility.

memory_summary reports resident tensor bytes, context capacity and the one active request policy. It excludes request KV/workspace, adapters, allocator, driver and other process allocations, and is not a total RAM/VRAM reservation or RSS limit. Resource estimates cannot guarantee that an allocation will succeed under external memory pressure. Request deadlines and general cooperative inference cancellation are not part of this initial public API.

LoRA deployment adapters

A compatible LoRA adapter creates a new effective model lease while preserving the source model. Its requests do not reuse another model configuration's KV state. Only one adapter is admitted per effective model in this increment.

let specialized = model.with_adapter("adapters/domain.manifest.json")
let answer = specialized.generate("Analyze the observation.", max_tokens: 128)
print(answer.text())
specialized.close()

The manifest has exactly these five fields. Replace both placeholder digests with the actual lowercase SHA256 values of the base GGUF and adapter GGUF:

{
  "schema": "aner.llm.adapter/1",
  "base_sha256": "BASE_GGUF_SHA256",
  "adapter_sha256": "ADAPTER_GGUF_SHA256",
  "architecture": "qwen2",
  "adapter_file": "domain-adapter.gguf"
}

adapter_file is a basename relative to the manifest directory. The runtime checks base bytes, adapter bytes, architecture, supported GGUF adapter metadata and target mapping. Matching a model name is insufficient. Checksums establish identity and integrity, not authorship or training quality. Activated LoRA and arbitrary adapter composition are not admitted. Conversion from another adapter representation must be performed explicitly and validated before this deployment manifest is written.

This inference loader does not turn Q4 deployment weights into a trainable model. Native reference model fine tuning, freezing, LoRA and exact resume remain in aner.gen. Training jobs wrap the bounded native LanguageModel workflow. The separate pretrained Llama workflow imports supported floating dense Llama GGUF for native CPU FP32 training; it does not train directly from Q4 deployment weights or implement QLoRA.

Compilation and compatibility

The standalone compiler admits the serial native loading/generation/accessor subset documented in Native compilation. It calls the native runtime directly; it does not invoke the Aner interpreter or a model service. Availability still depends on the matching optional engine and runtime libraries. Chat Lists, model handles in checked parallel tasks and training jobs are outside that initial compiled subset.

Existing LLMClient, credential references, Ollama/cloud adapters, request formats, provider defaults, and token accounting remain independent. A service only build can type check native model source; attempting execution reports the unavailable engine without selecting another provider.

Printing an LLMModel, storing it in a List, or development inspection shows an opaque type description. Explicit metadata/text accessors return ordinary Strings that may be printed or recorded. Native artifact paths, model weights, messages and adapter contents are not implicitly dumped through handle inspection.

Native pretrained training

The optional pretrained training workflow imports floating dense Llama GGUF, trains full weights or LoRA, and exports GGUF models/adapters for this inference API. llm.load retains its deployment role; training state, data preparation and exact resume belong to aner.gen. The first qualified training profile is CPU FP32. Ollama/cloud connection behavior is unchanged.

See the complete local inference source after obtaining the compatible local GGUF named in that source. Checking requires no model files; running requires the model and optional inference capability. Existing language model connections retain their provider and credential contracts.