Serve a native local chat model
Keep local GGUF weights resident behind bounded native chat routes with explicit cache precision, streaming and model telemetry.
// Requires an optional native-engine build and a compatible local GGUF file.
// Replace the artifact path before running. Checking does not load or serve it.
import aner.llm
import aner.api
let base = llm.load("models/research-model.gguf", context: 16384,
device: "metal", kv_cache_type: "q8_0", flash_attention: "on")
let model = base.with_context_pool(max_idle: 1)
base.close()
print(model.load_summary())
let app = api.app("Aner native chat").llm(path: "/chat", model: model)
.bind(host: "127.0.0.1", port: 11435)
.limits(max_rows: 64, timeout_ms: 120000)
.output_limit(max_tokens: 512)
.session_limit(max_sessions: 2)
app.serve()
model.close()
Make it your experiment.
Use the matching runtime. Save this program as examples/api_native_chat.aner inside a folder for your experiment. Open a terminal in that folder, then check and run the program.
aner check examples/api_native_chat.aner
aner run examples/api_native_chat.anerThe aner command must be on your PATH. Follow the installation guide if your terminal cannot find it.
Requires the matching optional native inference runtime and an explicit compatible local GGUF model. The shown Metal, Q8_0 KV and Flash Attention configuration requires a compatible build and hardware. Checking loads no model and opens no listener; running loads local weights and serves in the foreground. No Ollama process or provider credential is required.
aner check examples/api_native_chat.aner
aner examples/api_native_chat.anerBundled datasets retain their own source attribution and license notices. Example outcomes are not comparative benchmarks or evidence of clinical validity.
Dataset sources & attribution