Model inference APIs

aner.api exposes a preconfigured Aner model through a native HTTP service. A fitted neural classifier accepts validated numerical rows and returns class predictions and probabilities. A language model route uses one configured LLMClient and returns its typed completion. Model construction, training, checkpoint selection, device placement, and credentials remain explicit program decisions.

Capability boundary

The current module serves fitted neural classifiers and configured language model clients. It provides immutable route configuration, strict JSON contracts, OpenAPI 3.1 description, bounded transport, and optional native TLS and bearer authentication. It does not execute arbitrary request code, accept uploaded checkpoints, train models in a request, expose general callback routes, stream results, or discover remote models. A request cannot choose another provider, endpoint, credential, or model.

A service starts in the foreground and remains running until stopped. The default listener is 127.0.0.1:8000, accessible only from the same host. A loopback listener has no authentication unless explicitly configured; local processes can contact it. Use authentication when the host is shared or a trusted reverse proxy exposes that listener.

Serve a neural classifier

The local example trains a small classifier once, before accepting requests:

import aner.api
import aner.dataset
import aner.nn
import aner.ml

let split = dataset.iris().split(test: 0.2, seed: 2026)
let network = nn.sequential(seed: 42)
    .dense(inputs: 4, outputs: 8).tanh()
    .dense(inputs: 8, outputs: 3)
let model = ml.classifier(network, standardize: true)
    .fit(split.train(), epochs: 200, rate: 0.1)
let app = api.app("Iris inference").model("/predict", model)
app.serve()

The fitted classifier retains preprocessing learned from training rows. Inference applies that same preprocessing to incoming feature rows; it does not refit normalization or use held out labels. The service shares the preloaded immutable fitted model across requests. A neural route preserves the fitted model's effective device and precision; serving does not silently retrain, move the model, or lower precision.

In a second terminal, send one feature row in the model's exact column order:

curl --fail-with-body http://127.0.0.1:8000/predict \
  -H 'Content-Type: application/json' \
  --data '{"inputs":[[5.1,3.5,1.4,0.2]]}'

The request object contains exactly inputs, a nonempty matrix of finite numbers. Every row must have the fitted feature width, four in this example. Rows use the units and column order documented by the training dataset. The default maximum is 256 rows per request; max_rows can be set from 1 through 4096, subject to the request byte and numerical element bounds. No labels, dataset paths, model paths, credentials, or additional fields are admitted.

The response has this structure; probability values depend on the trained checkpoint:

{"predictions":[0],"probabilities":[[0.97,0.02,0.01]]}

predictions contains one zero based class ID per input row. probabilities contains one probability row in the fitted class order per input row. The probabilities shown are illustrative, not a measured result or an accuracy guarantee. Retain the class mapping and feature schema with the model's deployment documentation.

Load once and organize the service

Training and serving can be separate programs. Run the checkpoint preparation example from a fresh working directory to create iris.anermodel. That example refuses to overwrite an existing checkpoint through the native model contract.

The checkpoint service example loads that checkpoint once and uses a class to retain the configured application:

import aner.api
import aner.ml

class InferenceService {
    let app: APIApp

    fn run() {
        app.serve()
    }
}

let model = ml.load_classifier("iris.anermodel")
let service = InferenceService(
    app: api.app("Iris inference").model("/predict", model))
service.run()

An APIApp is an opaque immutable configuration value. Builder methods return a new app; assign the result or chain calls. App construction and OpenAPI generation bind no socket and resolve no secret. A model route is registered before serving; request bodies cannot replace its checkpoint. aner check validates the source without loading a checkpoint, training a model, or opening a listener.

Authenticate and use HTTPS

Create a dedicated service credential with 32 through 2048 visible ASCII characters without spaces using hidden terminal input:

aner secrets set inference-api

Use a different credential from an upstream provider key. A server bearer token authorizes access to this service; an OpenAI credential authorizes upstream provider work. Keep both outside source. On a headless host, use an explicit injected environment reference or managed file reference described in Credentials.

The secure example configures an explicit bind address, token reference, and TLS files:

import aner.api
import aner.ml
import aner.secrets

let model = ml.load_classifier("iris.anermodel")
let app = api.app("Iris inference")
    .model("/predict", model)
    .auth(secrets.stored("inference-api"))
    .tls(certificate: "server.pem", private_key: "server-key.pem")
    .bind(host: "0.0.0.0", port: 8443)
    .hostname("models.example.org")
    .limits(max_rows: 64, timeout_ms: 10000)
app.serve()

A nonloopback bind requires both native TLS and an explicit credential reference. Wildcard binds such as 0.0.0.0 and :: additionally require an explicit permitted hostname. api.hostname adds at most eight exact hostname or IP aliases; URLs, ports, and wildcards are rejected. Clients use the hostname covered by the TLS certificate, models.example.org in this example. The native service validates the HTTP Host against the bind address or configured aliases and the actual listener port. External plain HTTP and anonymous external listeners fail before service admission. Host names other than localhost are not resolved for binding; supply an explicit IPv4 or IPv6 address. localhost binds IPv4 loopback. Native TLS uses a certificate chain and unencrypted private key prepared outside the program. Both TLS files must be readable regular PEM files without leaf symlinks, with 1 through 65536 bytes per file. Each file is opened once; type and private key permission checks and bounded reads apply to that exact open descriptor. The TLS context loads the validated bytes rather than reopening a path. The certificate chain and private key are loaded once at startup; restart the service after certificate renewal or key replacement. On POSIX, the private key must be owned by the current user with no group or other permissions, for example mode 0600 or 0400. api.tls_available() reports this engine's direct TLS build capability on the current platform without starting a listener or reading key files. On supported POSIX hosts, native TLS requires a compatible build with an OpenSSL 3 provider, requires TLS 1.2 or newer, and admits modern authenticated encryption suites. Native TLS serving is unavailable on Windows in this release until private key access control verification is implemented; use an authenticated loopback service behind a trusted HTTPS gateway. Encrypted private keys fail promptly without a passphrase prompt. The runtime rejects invalid files rather than disabling verification or generating a self signed certificate silently.

A TLS enabled engine linked against shared OpenSSL 3 libraries needs those compatible libraries at process startup, even when serving HTTP. The capability query runs after the engine starts and cannot repair a loader failure caused by missing libraries. Choose a compatible engine and runtime, or an engine without direct TLS that serves authenticated loopback HTTP behind a trusted HTTPS gateway.

Clients send Authorization: Bearer TOKEN over verified HTTPS. Supply that header through the client's protected credential mechanism, avoiding literal tokens in saved examples, command history, URLs, notebooks, and logs. Clients must verify the certificate chain and requested hostname. The server has no session cookie, password login, or token issuance endpoint. A configured token protects inference and metadata routes, including health and OpenAPI. It is resolved once at service startup and compared using the native bounded constant time comparison. The resolved token remains cached in the running listener. Changing or deleting its credential store entry does not revoke that listener's existing token; stop or restart the service to apply rotation or removal. A restarted service must resolve a valid configured credential before accepting requests.

For deployment behind an existing HTTPS reverse proxy, retain the native loopback listener and configure authentication. Terminate verified public TLS at the trusted same host proxy, forward only intended routes, and protect access to the loopback upstream. Set the upstream Host to the native listener address and port, for example 127.0.0.1:8000; retain the public host in external TLS and routing configuration. The server does not treat X-Forwarded-Proto, X-Forwarded-For, or other forwarded headers as evidence of encryption or caller identity. It does not automatically configure a proxy, certificates, firewall, DNS, or an operating system service. Restrict ingress, set proxy body and timeout limits consistent with Aner, and let the deployment supervisor manage restarts.

Serve a configured language model

A route can share the same API listener with a configured LLMClient:

import aner.api
import aner.llm

let model = llm.client("ollama", "llama3.2:3b")
let app = api.app("Local completion").llm("/generate", model)
    .output_limit(max_tokens: 256).rate_limit(requests_per_minute: 30)
app.serve()

The language model service example requires the named local model to be installed and already served. A request contains a nonempty prompt and optional max_tokens and temperature. The application owns an output token ceiling through api.output_limit, with default 512 and configurable range 1 through 16384. A request above that configured ceiling is rejected before provider work. Omitting a request budget uses the smaller of 512 and the application ceiling. The example limits output to 256 tokens per request and admits 30 requests per minute. These limits bound admitted work; they are not provider billing guarantees. Temperature must be finite and from 0.0 through 2.0. Stricter provider limits still apply:

{"prompt":"Explain validation data.","max_tokens":64,"temperature":0.2}

The response contains text, model, finish_reason, input_tokens, and output_tokens. Missing provider token accounting is represented as JSON null. Native LLMResponse token accessors use -1 for the same unavailable state; neither representation means zero usage. Completion limits and upstream transport requirements remain those of Language model connections. Upstream cloud requests can incur provider charges; configuring inbound service authentication does not replace the upstream credential. Application owners control data submitted to the preselected provider. Requests cannot supply arbitrary upstream URLs or credential labels.

OpenAPI and endpoint contracts

app.openapi() returns the OpenAPI description without starting the listener. GET /openapi.json serves the same configured route description. Inference bodies are represented with explicit schemas, required fields, rejected additional fields, and numerical shape limits. Authenticated applications advertise the bearer security requirement. TLS configuration changes the transport, not the JSON inference contract.

Route paths are fixed during application construction. Reserved metadata paths cannot be replaced. App titles contain 1 through 128 UTF8 bytes; at most 16 model routes may be registered. A route is a literal path with 2 through 128 ASCII characters drawn from letters, digits, slash, hyphen and underscore, with no repeated or trailing slash. Query strings, encoded route tricks, and alternative methods are rejected. JSON requests require Content-Type: application/json, optionally with UTF8 charset; compressed request bodies are unsupported. Browser requests containing Origin are rejected. CORS is disabled in this initial release, so the supported integration is service to service. A browser application should contact its own authenticated server integration rather than embed a shared service token in public client code.

Limits, reliability, and errors

Requests have a 1 MiB body limit and responses an 8 MiB limit. The parser accepts at most 32 headers with 4 KiB line bounds, and a 2 KiB request target. JSON nesting is bounded to 32 levels and 100000 structural nodes. A connection handles one request and then closes; keep alive, HTTP/2, streaming responses, and arbitrary transfer encodings are outside the current contract. Four socket workers admit at most eight additional queued connections. A full connection queue closes the new connection without pretending inference succeeded. One inference executes at a time, preserving a clear resource and device contract for native model evaluation.

The default global fixed minute rate limit is 120 parsed requests. api.rate_limit accepts 1 through 100000 requests per minute. Valid Host and authentication checks and the browser Origin exclusion run before this quota. Failed authentication and rejected Origin attempts do not consume the legitimate allowance. Admitted metadata calls, including health, readiness and OpenAPI, count toward it. An unauthenticated loopback app counts admitted local requests. A rate rejection returns 429 and Retry-After: 60; this is a conservative retry interval. The limit is neither per user nor shared between processes. Deployment ingress controls are still needed for a public service.

api.limits sets the admitted classifier row count and an absolute connection processing budget from 1 through 600000 ms. The default is 10000 ms, starting when a socket worker begins processing the connection. TLS handshake, headers, body reads, inference admission and response writes share that fixed budget; continuing to send small amounts of data does not reset it. The separate eight connection queue has a count bound; time spent there before a worker begins is outside this budget.

Expiry can close a connection without returning an HTTP 504 or JSON response. A running CPU or GPU operation cannot be forcibly preempted: it drains, and its late result is discarded. The budget is therefore not a hard wall clock bound for arbitrary model computation. Configure rows, model size, upstream timeout, client deadline and proxy deadlines together. No retry is performed automatically; a client timeout can occur after inference began, so retries may repeat upstream provider work.

Native application configuration and serving failures use source located R3301 diagnostics. HTTP failures return a sanitized envelope such as {"error":{"code":"invalid_input"}} and do not include authorization values, prompts, generated text, model weights, private keys, or internal source traces. This protects native error reporting; explicitly printing ordinary input or output Strings retains ordinary program visibility. The module does not create an access log containing inference content. Each parsed response receives a locally generated X-Request-ID for caller correlation; an incoming identifier does not replace it.

Stop with Ctrl+C or an operating system termination signal. Shutdown stops new admission, skips queued connections and interrupts pending network waits. Active CPU or GPU computation drains subject to the boundaries above. A supervisor should allow for completion of an active native operation. Persist checkpoints before starting the service; requests do not mutate or save the fitted classifier.

API reference

Instance methods omit the first app parameter, for example app.model(path, model), app.auth(key), and app.serve(). Immutable builders require retaining their return value. A printed APIApp remains opaque and does not reveal routes, credentials, or model content. OpenAPI is an explicit interface description, not a secret export.

Development and production

Develop a model with protected Dataset roles, device reports, and bounded development observations before exposing inference. Both execution modes run the same validated routes and use the same precision, checkpoint, credentials, and transport settings. Production disables automatic development recording; it does not weaken input validation or silently enable external access. Inspect model training separately from a long running API process.

Use standalone .aner programs for the foreground listener. Calling serve in a notebook cell blocks that cell and its interpreter session until stopped; notebook request cells should contact a separately supervised API service. Checking the complete examples opens no listener. VS Code extension 0.1.35 or newer supplies APIApp signatures, immutable builder methods, security parameter help, and the apilocal, apicheckpoint, apiauth, apitls, and apillm snippets. The engine must also support aner.api; an editor update alone does not install the native module.

Time and space complexity

Let s be route descriptor bytes, q request bytes, r response bytes, b admitted input rows, f input feature width, and c output class count. For a Dense classifier, let layer widths be d0 through dL with d0 = f and dL = c. These bounds describe native serving overhead and inference arithmetic; network waits and an external language model's computation are separate.

The service retains one configured model and its fitted preprocessing. Neural inference does not build a training graph or retain gradients. CPU or accelerator libraries may allocate device context, kernels, and workspace beyond input/output storage; report actual device memory separately when benchmarking. The fixed socket worker and queue counts bound admitted connection bookkeeping, not total process RAM. A larger model still requires its full parameter storage. External completion latency and memory belong to the selected provider; no universal inference Big O or completion estimate is asserted.

See Neural networks, Neural learning, research, and devices, Tensor devices and performance, Credentials, and Language model connections for the underlying model and credential contracts.

Interface standards

The interface description follows OpenAPI 3.1. Methods, status codes and authentication challenges use the semantics in HTTP Semantics, RFC 9110. These standards describe interoperability; the admitted Aner routes and limits remain those defined above.