SGD and Adam parameters

This reference targets Aner 0.0.5 or newer. Updating documentation or editor help alone does not upgrade an installed runtime; check Downloads for verified platform packages.

Neural networks · Model directory

An optimizer updates trainable parameters after nn.backward(loss). It does not choose a loss, preprocess data or create a split. The Dense classifier, CNN, recurrent and generative workflows have different fitting options and state contracts. The model API inventory lists the exact public signatures.

Which interface owns the state?

Workflow	Choice and default	State owner
Dense `ml.classifier(..., optimizer)`	`"sgd"` default; `"adam"` supported	Each independent fit, or explicit `warm_start` from a fitted snapshot.
Explicit Tensor loop	`nn.optimizer(name, rate)`, both required	Shared `Optimizer` handle; stable keys identify distinct parameter histories.
One stateless SGD update	`nn.sgd(parameter, rate)`	No momentum state. Returns a replacement parameter leaf.
CNN and recurrent `.fit`	`"adam"` default; `"sgd"` supported	Returned immutable family specific fitted model.
Native `gen.train`	Adam training implementation	Returned language/scientific model and its strict resume configuration. There is no optimizer name argument here.

Mathematical variables

For a parameter value w, its current gradient g, and learning rate rate, plain SGD computes w_next = w - rate*g. There is no hidden momentum, Nesterov correction or weight decay in Dense SGD or nn.sgd.

Adam retains first moment m, second moment v, and positive successful update count t:

m = beta1*m + (1-beta1)*g
v = beta2*v + (1-beta2)*g*g
m_hat = m / (1-beta1^t)
v_hat = v / (1-beta2^t)
w_next = w - rate*m_hat / (sqrt(v_hat) + epsilon)

Variable	Dense/keyed Adam value	Meaning and effect
`rate`	Caller supplied finite positive `Float64`	Scales the normalized update. Must remain positive/representable in effective Float32 when used there.
`beta1`	Fixed `0.9`	Decay of the gradient mean; retains a history of update directions.
`beta2`	Fixed `0.999`	Decay of squared gradients; controls the denominator's retained history.
`epsilon`	Fixed `1e-8`	Added after the corrected square root for numerical stability.
`m`, `v`	Initially zero per parameter	Detached moments with the parameter's exact logical shape, dtype and device. Second moments must be nonnegative.
`t`	Initially zero, advanced on successful update	Bias correction is per optimizer parameter key in explicit loops. It is not an epoch number when an epoch has multiple batches.

The three Adam constants are documented implementation choices, not configurable named arguments of ml.classifier or nn.optimizer. Dense/keyed Adam does not implement AMSGrad or AdamW. There is no public Aner nn.adam(first, second, ...) source call; use the registered keyed optimizer. A C++ helper exists internally with explicit moments.

Every explicit loop argument

import aner.nn

let optimizer = nn.optimizer(name: "adam", rate: 0.01)
var weight = nn.parameter([[1.0, -1.0]])
let objective = nn.mean(nn.multiply(weight, weight))
nn.backward(objective)
weight = nn.update(optimizer: optimizer, parameter: weight, key: "projection.weight")
print(optimizer.summary())

Public argument	Type / default	Contract
`name` in `nn.optimizer(name, rate)`	Required `String`	Exactly `"sgd"` or `"adam"`. Case sensitive; other strings fail.
`rate`	Required `Float64`	Finite and positive. Invalid rates fail before a parameter key is registered.
`optimizer` in `nn.update`	Required `Optimizer`	Retain the same handle across steps to retain moments. Constructing a fresh optimizer each step restarts Adam.
`parameter`	Required `Tensor`	Trainable leaf with a valid current gradient. Use `nn.parameter` to create it, and assign the returned update back.
`key`	Required `String`	Stable unique name, 1 to 1024 `UTF-8` bytes without NUL. Reusing it requires identical logical shape, dtype and device. One distinct key per distinct trainable parameter.
`parameter`, `rate` in `nn.sgd`	Both required	Same leaf/gradient/rate requirements; returns a detached replacement parameter leaf for the next graph.
`parameters` in `nn.clip_grad_norm(parameters, max_norm)`	Required `List` of parameter Tensors	Explicit parameters whose valid gradients are clipped together. Use after backward and before updates.
`max_norm`	Required finite positive `Float64`	Maximum global L2 gradient norm. Returns the original norm; the call modifies the listed distinct parameter leaves' gradients, not parameter values.

The optimizer's parameter registry is bounded to 4,096 keys, 1,000,000 total keyed elements and 100,000 successful updates per key. Failed validation or arithmetic does not register/advance a key or commit moments. Copying the handle shares its state; nn.reset(optimizer) discards all keys and moments through every alias. summary() discloses policy, rate and registered state. Arbitrary optimizer handles and user classes do not have a public generic checkpoint API.

nn.backward replaces reachable gradients rather than accumulating across independent calls. Build a fresh forward graph per step, update each distinct leaf once, and release historical loss/graph handles. Tied weights share one leaf and one optimizer key; assigning multiple keys to the same conceptual weight creates separate histories.

Fitting options are family specific

Setting	Dense classifier	CNN source API	Recurrent source API	`gen.train`
Learning rate	Required in fit/config	Default `0.001`	Default `0.001`	Required
Optimizer name	`sgd`/`adam`	`adam`/`sgd`	`adam`/`sgd`	No name parameter
Global clipping	Not a Dense fit option; explicit loops may call `clip_grad_norm`	Internal default `1`; not exposed by registered source `.fit`	`clip_norm`, default `1`; zero disables	Extended overload `clip_norm`, default `1`; zero disables
Decoupled weight decay	Not a Dense fit option	Internal default zero; not exposed by registered source `.fit`	`weight_decay`, default zero	Extended overload `weight_decay`, default zero
Warmup	Not a fit option	Not a source fit option	Not a source fit option	`warmup_steps`, default zero

Do not copy an option from one family's signature into another. CNN's native C++ FitOptions has fields beyond its registered Aner source overloads. Recurrent/generative weight decay is an explicit decoupled update with family specific excluded parameters; it does not change the meaning of Dense SGD/Adam. Read the family table before configuring it.

Continuation and traceability

Dense v7 checkpoints retain cumulative training updates for both SGD and Adam, plus Adam moments/step where applicable. Dropout uses the separate cumulative training identity; changing optimizer deliberately resets optimizer history while preserving the dropout position. Best epoch selection restores the moments and stochastic state corresponding to the selected parameters, not a later epoch's optimizer state.

CNN and recurrent models retain attempted, successful and skipped updates because FP16 overflow can skip an update. A skipped AMP update does not advance weights, moments or the successful Adam count; dropout schedules can still use attempted update identities. Generative resume additionally validates bound data/tokenization, options, runtime implementation and backend. None of these family specific guarantees imply bitwise equality across devices, numerical policies or software builds.

Use validation data to compare a declared rate and regularization design; do not choose a favorable seed using final test results. Retain model topology, each seed's purpose, rate, batch size, objective, precision/backend, checkpoint and actual metrics. The neural workflow explains online versus detached training losses and measured durations.