Read the incident record. Find the relevant handbook passage. Compute the rule in Aner. Then ask a local language model to turn that evidence into a clear incident brief, with its sources beside it.
Local GGUF weights · 8 GB GPU planning target · CPU fallback
A small operations team has a temperature alert, a CSV of observations and several procedure pages. FieldDesk assembles an incident brief from that local evidence. The room, nonhazardous polymer sample experiments, observations and procedures are synthetic, so the full case can be shared and reproduced.
CASE-017 records a 24.2°C temperature, 72% relative humidity and a filter age of 820 hours. Aner compares each reading with the fixture’s declared limit and retrieves the three relevant handbook passages.
The decision is a deterministic result of the supplied rules and readings. Source IDs and calculated values travel with that result. The retained decision stays beside generated text, and the validation runner flags a missing decision label or an unknown source reference.
The model explains the evidence, states the declared action and cites the selected source IDs. Its generated wording remains reviewable beside the original observations and rule result. The explanation executes no physical action.
24.2°C observed − 22°C ceiling[FD-101]
72% RH observed − 65% ceiling[FD-102]
820 hours observed − 750-hour interval[FD-103]
Synthetic training procedure. The calibration room target is 18 to 22 degC. If temperature exceeds 22 degC pause new measurements and mark affected nonhazardous polymer sample coupons pending review. Record the observed temperature and notify the room lead. Do not assert that a sample is damaged without a separate assessment.
Synthetic training procedure. The calibration room relative humidity ceiling is 65 percent. Above this limit pause new measurements and record the humidity. Check the door and the dehumidifier indicator without dismantling equipment. The room lead decides when the experiment can resume.
Synthetic training procedure. The ventilation filter service interval is 750 operating hours. Record overdue hours and request authorized maintenance when the interval is exceeded. An overdue filter is a maintenance observation and does not prove the cause of a temperature or humidity excursion.
Decision: PAUSE_AND_ESCALATE - Temperature exceeded the ceiling by 2200 milli-degC ([FD-101]). - Relative humidity exceeded the ceiling by 700 basis points ([FD-102]). - Filter was 70 hours overdue, but its age does not establish the cause of the excursion ([FD-103]). Next steps: 1. Pause all new measurements. 2. Record the excursions and notify the room lead. 3. Check the dehumidifier and room door. 4. Request authorized maintenance for the filter. 5. Review the incident with the room lead.
Declared decision: PAUSE_AND_ESCALATE · Selected sources: FD-101, FD-102, FD-103
This saved answer preserves the computed decision and overages and states that filter age does not establish cause. A human still reviews each operational statement against the selected synthetic procedures; the program executes no equipment action.
CSV parsing, document selection, thresholds, ordering and the decision are ordinary application code. The case demonstrates lexical retrieval over a five-document curated collection. The small input makes the execution boundary easy to inspect; it does not establish a parallel speedup. No embedding service or vector database is required.
Local generation turns a compact evidence packet into useful prose. Checking source IDs catches unsupported references; a person still checks whether each statement follows from its source. A citation alone does not prove a claim.
The case concerns synthetic nonhazardous polymer sample experiments. Its procedures describe this demonstration only. The application executes no physical action and does not infer a cause, sample damage or approval from a language model answer.
The same prepared evidence can go directly to an in-process model or to a local service. Both routes retain the model artifact identity, response timing and sources used for the request.
Read the local fixtures, choose relevant passages, calculate the incident facts and write a bounded prompt. Put common instructions and stable source material first so repeated requests have a useful exact prefix.
The program loads a compatible local single-file GGUF, inspects its fingerprint and selects CPU, CUDA or Metal explicitly. “Auto” resolves to an available supported backend; the program records the resolved device.
Use a 4,096-token context and a bounded output. The complete formatted prompt plus maximum output must fit. The runtime rejects excess context instead of silently dropping the earliest evidence.
Save the rule result, source IDs, generated text, model fingerprint and native timings. Compare repeated and concurrent runs under the same input and sampling policy.
Embed generation in an Aner program, or keep a model resident behind aner.api and call it from another process. The companion programs share the evidence prepared by the same application; the source excerpts below show their native integration.
// Obtain this artifact separately. device auto also works on CPU-only builds.
// Edit the local model path and device constants below for your machine.
let base = llm.load("models/qwen2.5-7b-instruct-q4_k_m.gguf", context: 4096,
device: "auto", kv_cache_type: "f16", flash_attention: "auto")
let optimized = base.with_optimizer(threads: 0, prefill_threads: 0, batch_tokens: 512, micro_batch_tokens: 128, prefix_cache_entries: 0, prefix_cache_tokens: 1024, speculative_tokens: 0)
let model = optimized.session()
print("FIELD_DESK_LOAD " + base.load_summary())
print("FIELD_DESK_MEMORY " + model.memory_summary())
print("FIELD_DESK_MODEL " + quoted(base.fingerprint()))
print("FIELD_DESK_DEVICE " + quoted(base.device()))
let prepared = prepare_case("Temperature humidity filter: explain the calibration room incident and the next operator steps.", parallel.local(workers: 1))
print("FIELD_DESK_CASE " + prepared.record())
// Full intended messages are supplied every time. The exact common prefix is reusable.
let questions = List("Explain the immediate response.", "Explain the observations and avoid claiming a cause.", "Summarize a handover for the next operator.")
for iteration in Range(3) {
require(prepared.has_evidence, "retrieval must succeed before inference")
let messages = List(llm.message("system", prepared.system_prompt),
llm.message("user", prepared.user_prompt + "\nOUTPUT TASK: " + cast<String>(questions.get(iteration % questions.len()))))
let answer = model.chat(messages, max_tokens: 192, temperature: 0.0, seed: 2026)
print("FIELD_DESK_RESPONSE {\"iteration\":" + String(iteration) +
",\"text\":" + quoted(answer.text()) + ",\"finish_reason\":" + quoted(answer.finish_reason()) +
",\"input_tokens\":" + String(answer.input_tokens()) +
",\"output_tokens\":" + String(answer.output_tokens()) +
",\"cached_input_tokens\":" + String(answer.cached_input_tokens()) +
",\"tokens_per_second\":" + String(answer.tokens_per_second()) +
",\"timings\":" + answer.timings() + "}")
}
print("FIELD_DESK_FINAL_MEMORY " + model.memory_summary())
model.close()
optimized.close()
base.close()
// Standalone native server. Run from this directory; Ctrl+C stops it.
import aner.llm
import aner.api
let base = llm.load("models/qwen2.5-7b-instruct-q4_k_m.gguf", context: 4096,
device: "auto", kv_cache_type: "f16", flash_attention: "auto")
let optimized = base.with_optimizer(threads: 0, prefill_threads: 0, batch_tokens: 512, micro_batch_tokens: 128, prefix_cache_entries: 1, prefix_cache_tokens: 1024, speculative_tokens: 0)
let model = optimized
print(model.model())
print(model.fingerprint())
print(model.load_summary())
let app = api.app("FieldDesk local inference").llm("/api/chat", model)
.bind(host: "127.0.0.1", port: 11435)
.limits(max_rows: 16, timeout_ms: 300000)
.output_limit(max_tokens: 192).session_limit(max_sessions: 1)
app.serve()
model.close()
optimized.close()
base.close()
// Copy the fingerprint printed by server.aner into MODEL_ID_FROM_API_TAGS.
let model = llm.client("ollama", "MODEL_ID_FROM_API_TAGS", "http://127.0.0.1:11435", timeout_ms: 300000)
let prepared = prepare_case("Temperature humidity filter: explain the calibration room incident and the next operator steps.", parallel.local(workers: 1))
require(prepared.has_evidence, "retrieval must succeed before inference")
let answer = model.chat(List(llm.message("system", prepared.system_prompt),
llm.message("user", prepared.user_prompt + "\nOUTPUT TASK: Explain the immediate response.")),
max_tokens: 192, temperature: 0.0)
print("FIELD_DESK_CASE " + prepared.record())
print("FIELD_DESK_CLIENT {\"text\":" + quoted(answer.text()) +
",\"finish_reason\":" + quoted(answer.finish_reason()) + ",\"timings\":" + answer.timings() + "}")
llm.load admits local weights; generation executes in the same native process. Close the model when its work is complete. Use an explicit session when a conversation should retain exact-prefix KV.
The companion binds loopback at 127.0.0.1:11435 and exposes /api/chat. An Aner Ollama-style client can call that limited compatible chat contract. GET /api/tags discovers the configured model identity.
The API is a service-to-service interface with bounded requests, explicit route configuration and optional NDJSON streaming. Browser Origin requests are rejected by the current contract. For access from another machine, use the documented authentication and verified transport setup; the example listener stays local.
The reference model is Qwen2.5-7B-Instruct in Q4_K_M GGUF. Start with one active sequence and a 4,096-token context. Change the assumptions below to see why more users or longer prompts need deliberate memory planning.
≈ 4.36 GiB · Q4_K_M single-file GGUF
One sequence · F16 · calculated tensor storage
Optional smaller Qwen2.5 model for CPU and quick checks
Model sizes, quantizations and Apache 2.0 license: Qwen 7B model card and Qwen 1.5B model card. KV calculation uses the 7B architecture configuration: 28 layers, 4 KV heads and 128 dimensions per head. Approximate download sizes use decimal GB; this planner displays binary GiB.
6.08 / 8 GiB
Inside this planning budget. Verify actual free VRAM and peak allocation on your card.
Estimate = approximate GGUF file size + K/V tensor storage + a chosen 1.50 GiB reserve. The reserve is an assumption; compute workspace, adapters, backend alignment, display use and other processes can require more. Each retained independent prefix/session context adds its own cache. A speculative draft model would add its own weights and cache. Unified memory on Metal differs from dedicated GPU VRAM.
Teal: one slot. Amber: four slots. Dashed line: 8 GiB. These curves are arithmetic estimates, with the same 1.50 GiB reserve at every point.
| Context / sequence | 1 slot / GiB | 4 slots / GiB |
|---|---|---|
| 4,096 | 6.08 | 6.74 |
| 8,192 | 6.30 | 7.61 |
| 16,384 | 6.74 | 9.36 |
| 32,768 | 7.61 | 12.86 |
K/V bytes = 2 × layers × KV heads × head dimension × bytes per value × context × slots. Q8_0 uses 34/32 bytes per value, including block scales; F16 uses 2.
One active sequence, one resident model and a short output ceiling. Close unused leases and sessions.
Record free VRAM, resolved device and actual peak allocation. The runtime’s weight summary excludes some workspace and driver costs.
Compare longer context, extra slots and quantized KV independently. Preserve the same incident checks.
These mechanisms act at different stages of inference. The development optimizer exposes explicit controls and telemetry so a faster-looking configuration can be checked for actual support, correct output and useful end-to-end performance.
Reuse past tokens inside a request or an explicit session.
Autoregressive decoding retains keys and values. A session reuses an exact token prefix when the caller supplies the complete intended history.
Share repeated prompt work across independent requests.
An opt-in bounded cache retains compatible prompt contexts. Reuse requires an exact token prefix under the same model, adapter and execution configuration.
Reduce weights and, separately, KV storage.
The case uses prequantized Q4_K_M GGUF weights. F16 KV is the default; Q8_0 KV is an explicit compatible Flash Attention option.
Reduce attention intermediates and memory traffic.
The pinned native engine selects the supported attention implementation. Attention policy can be auto, on or off; unsupported explicit combinations fail.
Verify several candidate tokens in one target-model pass.
The development optimizer uses prompt lookup for greedy decoding: candidate continuations come from the supplied token history and the target model verifies them.
Keep several active sequences moving in one shared context.
An opt-in sequence scheduler combines work at token-step boundaries, releases finished slots and admits waiting requests under a one-to-four-sequence bound.
For repeated manuals and greedy answers, try a single retained prefix context and a short prompt-lookup verification window. The complete preceding arguments stay explicit. Zero thread values select the runtime’s bounded host default.
let latency = base.with_optimizer(
threads: 0, prefill_threads: 0,
batch_tokens: 512, micro_batch_tokens: 128,
prefix_cache_entries: 1, prefix_cache_tokens: 1024,
speculative_tokens: 4)
print(latency.memory_summary())For several simultaneous callers, choose a batched lease with two sequence slots. This alternative disables retained shared prefixes and prompt lookup. Use one profile at a time when testing a small device.
let throughput = base.with_optimizer(
threads: 0, prefill_threads: 0,
batch_tokens: 512, micro_batch_tokens: 128,
prefix_cache_entries: 0, prefix_cache_tokens: 1024,
speculative_tokens: 0)
.with_batching(max_sequences: 2)
print(throughput.memory_summary())| What you observe | First experiment | Keep visible |
|---|---|---|
| Repeated manuals make first token slow | Exact prefix reuse; stable content before the changing question | Cached vs newly evaluated input tokens; tenant cache ownership |
| A long context consumes the card | Shorter selected evidence; then compatible Q8_0 KV | Peak allocation, source coverage and answer checks |
| One CPU request is slow | Compare decode and prefill thread budgets, batch and microbatch sizes | Physical cores, memory bandwidth and competing workers |
| Repeated output needs many target passes | Greedy prompt-lookup speculative verification | Acceptance rate and equality with the target greedy reference |
| Several users leave the GPU underused | Bounded continuous batching with two slots, if memory permits | Queue delay, completion latency, total throughput and KV growth |
Keep CPU thread budgets across workers within the host’s resources. A bigger prefill batch can improve throughput while increasing workspace. Prefix caches, sessions and batched sequence slots have separate lifetimes and memory costs. Read the exact optimizer API, bounds and telemetry contract.
A single GPU accelerates model computation. CPU workers can prepare independent evidence packets. A cluster gives independent cases to machines that each own their model and memory budget.
Select device: "cpu". Tune prefill and decode threads independently. The optional 1.5B model shortens the experiment. CPU inference uses system RAM and is usually slower than a capable GPU for this workload.
Use an admitted CUDA or Metal backend. Prepare once, generate several answers and inspect warm timings. Start with one sequence, then test two-slot batching only after the device has enough headroom.
Use the supplied cluster preparation route with compatible enrolled runtimes. Each node computes its assigned independent cases. For inference services, each configured node owns its local model and bounded request admission.
The Aner programs distribute independent retrieval tasks. For inference replicas, run an Aner API server on each model owner and send each complete request to a chosen service. Each node owns a full model copy. Continuous batching is local to one model owner; this case does not implement cross-machine tensor parallelism.
Persistent session and prefix state belong to a process. Route a conversation consistently when using multiple service instances, or resend its complete history to a different node and accept fresh prefill. Enroll cluster workers explicitly and budget each node separately. Parallel computation · Cluster setup · GPU execution
The retained experiments record request telemetry and generated output. Compare cold and warm runs separately, keep exact model bytes identifiable and use a representative incident before increasing load.
Apple M2 Max with 64 GiB unified memory, using Metal. Same Qwen2.5 7B Q4_K_M artifact, F16 KV, 4,096-token context, greedy generation and a 192-token output ceiling. One process per profile; one warmup plus three timed requests. This is an exploratory application run, not an 8 GB discrete-GPU qualification.
Three timed requests per profile, after one warmup. Dots, median markers and ranges describe this exploratory run; they are not confidence intervals. Each profile ran in one process, so repeated processes and randomized ordering are needed for a broader performance comparison.
| Execution | Model | Device | Input / cached | Output | First text / ms | Total / ms | Decode / tokens·s⁻¹ |
|---|---|---|---|---|---|---|---|
| Stateless | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 510 / 0 | 144 | 824.2 | 3011.8 | 65.82 |
| Stateless | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 512 / 0 | 134 | 822.5 | 2927.6 | 63.65 |
| Stateless | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 506 / 0 | 157 | 821.1 | 3256.4 | 64.46 |
| Explicit session | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 510 / 498 | 144 | 57.9 | 2325.1 | 63.51 |
| Explicit session | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 512 / 496 | 134 | 44.4 | 2140.2 | 63.93 |
| Explicit session | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 506 / 496 | 157 | 41.8 | 2489.2 | 64.15 |
| Shared prefix | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 510 / 498 | 144 | 58.0 | 2412.8 | 61.15 |
| Shared prefix | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 512 / 496 | 134 | 49.3 | 2361.7 | 57.95 |
| Shared prefix | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 506 / 496 | 157 | 49.8 | 2775.3 | 57.60 |
| Prompt lookup | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 510 / 0 | 144 | 955.6 | 3290.2 | 61.68 |
| Prompt lookup | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 512 / 0 | 134 | 1003.8 | 3136.4 | 62.83 |
| Prompt lookup | Qwen2.5 7B Q4_K_M | Metal · M2 Max | 506 / 0 | 157 | 1032.4 | 3623.5 | 60.59 |
Each row is a retained trial, not a claim about every 8 GB card. Decode throughput and total request time have different timing boundaries. The raw output and model identity are retained with the application evidence. They accompany the published repository source.
Reference model artifact SHA-256: 1875fb29e8c91c86615c00e92d8b4114e56bc24359adb5a8db8b36452fae4a49. This identifies the merged local file; model weights are supplied separately.
Median native first-text latency was 822.5 ms for stateless requests, 44.4 ms for explicit sessions and 49.8 ms for shared prefixes. Prompt lookup accepted 109 of 404 proposals across the three timed requests and did not improve this workload: its median first-text latency was 1,003.8 ms and median decode throughput was 61.68 tokens/s, versus 64.46 tokens/s without lookup. Speculation remains opt-in. Mechanical checks do not establish full semantic accuracy: some other responses used imprecise equipment wording, so human source review remains necessary. Peak device memory, repeated-process uncertainty and 8 GB CUDA performance were not measured.
| Profile | Device | KV tensor estimate | First text / ms | Decode / tokens·s⁻¹ | Interpretation |
|---|---|---|---|---|---|
| CPU session | CPU · M2 Max | 224 MiB | 459.1 | 18.25 | Functional CPU fallback; output wording can differ from Metal despite greedy settings. |
| Q8 KV + Flash on | Metal · M2 Max | 119 MiB | 1615.1 | 42.96 | 119 MiB versus 224 MiB F16 KV payload; slower in this capture. Both KV precision and attention policy changed. |
Complete streamed HTTP replies were measured after one excluded Aner-client warmup. Aggregate throughput uses total returned output tokens divided by the external wall time for all timed requests; it differs from the native decode metric above.
Both two-caller runs returned 870 tokens, with identical corresponding text and token counts. Continuous batching reached two active sequence slots. This is one server process per setting, with six timed requests; repeated-process uncertainty and 8 GB device fit are unmeasured.
| Serving profile | Callers | Batch slots | Replies | Output tokens | Wall / s | Aggregate / tokens·s⁻¹ |
|---|---|---|---|---|---|---|
| Shared prefix API | 1 | Off | 3 | 435 | 7.181 | 60.57 |
| No batching | 2 | Off | 6 | 870 | 18.173 | 47.87 |
| Continuous · 2 slots | 2 | 2 | 6 | 870 | 16.194 | 53.73 |
The two-slot scheduler estimates 448 MiB of F16 K/V payload at 4,096 tokens per sequence, versus 224 MiB for one context. Workspace and driver memory are additional. API mechanical checks passed for every captured reply; a complete semantic evaluation and a live multi-machine deployment remain separate work.
Performance varies with hardware, prompt shape, output length, cache state and concurrency. A warm cache reduces some work; it does not make generated reasoning more accurate. A fixed seed alone does not guarantee identical sampled output across hardware and kernels.
The application’s Aner programs, synthetic fixtures, deterministic checks and retained measurements belong together in its source repository. Inspect the published code and README on GitHub, then run the case locally.
Use Aner 0.0.7 with the pinned native LLM dependency and the CPU, Metal or CUDA support needed for your machine. Verify the executable’s version before running the case.
Obtain both official Q4_K_M GGUF parts, verify their hashes and merge them with the pinned llama-gguf-split --merge tool. The repository guide documents exact commands and identities. Place the merged single-file artifact under models/, or set its local path in the Aner program.
Run the native checks and preparation program from the application directory. Inspect the deterministic rule result and retrieved sources, then run the direct or API companion.
Keep the same evidence, output budget and sampling policy. Retain raw records, compare correctness, and report the memory and latency costs alongside throughput.
aner --version
aner check verify.aner
aner verify.aner
aner check prepare.aner
aner prepare.aner
aner check direct.aner
aner direct.aner
# A separate terminal keeps local weights resident.
aner server.aner
# Copy the server fingerprint into MODEL_ID_FROM_API_TAGS in client.aner.
aner client.anerRead the repository README for the exact model path, fixture assumptions, CPU/GPU selection, validation steps and optional enrolled-cluster setup. Checking Aner source performs type checking without loading weights or starting a listener.
The published repository keeps the code, synthetic inputs and measurement records together.
GitHub repository: https://github.com/aner-lang/aner-fielddesk-local-llm
The application is original synthetic demonstration material. The model configuration, inference algorithms and backend APIs have their own primary references; retain those distinctions when adapting the case.
Official model artifact, available quantizations, model identity and Apache 2.0 license. The 1.5B variant is an optional smaller starting point.
Layer and grouped-query attention dimensions used in the transparent memory estimate. The reserve remains an application assumption.
IO-aware exact attention reduces transfers of intermediate attention data. Paper results describe their own models and hardware.
Draft-and-verify decoding provides the algorithmic context. This development implementation uses greedy prompt lookup, with its narrower contract.
KV memory management and serving throughput motivate bounded concurrency. This case does not claim that Aner implements vLLM’s PagedAttention.
Compatibility, model lifetime, device admission, explicit history, output budgets and telemetry. The companion API guide documents the served chat subset.