Evaluation metrics

aner.metrics evaluates classification IDs, numerical predictions and cluster partitions through four native functions. It consumes the shared Tensor type and returns detached evaluation results. It does not fit a model, select data partitions or update gradients.

Import aner.metrics for its functions, Tensor annotations and tensor literals. Import aner.tensor separately to inspect a confusion matrix or convert a score matrix to predicted IDs. See the Tensor reference for construction, precision and storage.

Run a complete example

The online metrics example includes runnable commands and downloadable source. Download the complete example, or save the program below as metrics_demo.aner. It uses small synthetic values and requires no database, credentials or downloaded dataset:

import aner.metrics;
import aner.tensor;

fn main() -> Unit {
    let predicted: Tensor = [[1.0], [1.0], [2.0], [0.0]];
    let actual: Tensor = [[0.0], [1.0], [1.0], [2.0]];
    print(metrics.accuracy(predicted: predicted, actual: actual));

    let counts = metrics.confusion_matrix(predicted, actual, classes: 3);
    print(tensor.value(counts, row: 0, column: 1));
    print(tensor.value(counts, row: 1, column: 2));

    let prediction: Tensor = [[1.0, 3.0], [4.0, 8.0]];
    let truth: Tensor = [[0.0, 1.0], [4.0, 4.0]];
    print(metrics.mean_squared_error(prediction, truth));

    let groups: Tensor = [[0.0], [0.0], [1.0], [1.0]];
    let same_partition: Tensor = [[9.0], [9.0], [8.0], [8.0]];
    let crossed_partition: Tensor = [[0.0], [1.0], [0.0], [1.0]];
    print(metrics.adjusted_rand_index(groups, same_partition));
    print(metrics.adjusted_rand_index(groups, crossed_partition));
}

Check and run it with an installed Aner executable:

aner check metrics_demo.aner
aner run metrics_demo.aner

If using the distributed example instead, run aner check examples/metrics_demo.aner and aner run examples/metrics_demo.aner from the folder containing examples. The aner.ml guide connects these evaluation functions to fitted models; the Tensor reference defines the shared input representation. This module has four functions and no importable submodules.

Expected output:

0.25
1.0
1.0
5.25
1.0
-0.5

The first comparison has one correct classification among four rows. Confusion matrix cells count rows using the actual class first and predicted class second. Mean squared error averages four squared errors. The last two comparisons demonstrate that matching cluster memberships can retain a score of one after label renaming, while crossed memberships can have a negative adjusted Rand index.

API reference

These are native signatures. Parameter names are also accepted as named arguments; the complete example uses predicted, actual and classes.

A Tensor has at most 1,000,000 cells. Label metrics require finite nonnegative integer values no larger than 9,007,199,254,740,991, represented in the Tensor's stored precision. Noncontiguous IDs are accepted by accuracy and adjusted Rand index. A confusion matrix instead indexes the explicit class range 0 through classes - 1.

Rows must represent the same observations in the same order. These functions compare positions and do not align source IDs, join tables, apply fitted preprocessing or check training/test membership. Keep row identities and target meanings with your data; Dataset partitions and FeatureLedger provide the corresponding contracts.

Classification and class IDs

metrics.accuracy divides the count of equal IDs by N. Its result lies between zero and one. The same numeric ID must mean the same class in both inputs: relabeling only one input changes classification accuracy.

metrics.confusion_matrix returns a detached CPU Float64 tensor of shape classes × classes. Rows represent actual classes; columns represent predictions. Cell (i, j) counts observations whose actual ID is i and predicted ID is j. An unobserved class retains a zero row and column. The sum of all cells is N. Every input ID must be below classes, including classes absent from the current batch.

A score or probability matrix has shape N × C, rather than N × 1 IDs. Convert it explicitly before using ID accuracy:

import aner.metrics;
import aner.tensor;

let scores: Tensor = [[0.0, 1.0, 1.0], [4.0, 0.0, 1.0]];
let actual: Tensor = [[1.0], [0.0]];
let predicted = tensor.argmax(scores);
print(metrics.accuracy(predicted, actual));

This prints 1.0. tensor.argmax uses the first maximum in each row, so the first row chooses column one. The resulting IDs are column indices: retain the model's class mapping if those indices differ from external class IDs. Existing nn.accuracy(scores, labels) performs argmax internally. The classical ML reference explains ml.classes and fitted KNN probability columns; the neural reference explains neural scores and labels.

Numerical prediction error

metrics.mean_squared_error computes the population mean of (predicted - actual)² over every cell. For shape N × D, the denominator is N × D. A 1 × 4 row and a 4 × 1 column do not match even though both contain four cells. Identical finite inputs return zero.

The native implementation scales errors before squaring and uses compensated accumulation. It can return a finite mean when an individual unnormalized square would overflow. It rejects an unrepresentable difference or final Float64 mean. The result has the squared unit of the target; changing a target's numerical scale changes the metric.

This result is an evaluation scalar detached from automatic differentiation. Use nn.mse to construct a differentiable neural training loss. Metrics preserve existing Tensor gradients and do not add a backward graph.

Cluster partition comparison

metrics.adjusted_rand_index compares which pairs of observations share a group in each partition, with chance correction. It ignores the numeric names of groups and accepts sparse, noncontiguous IDs. Identical partitions return one, including a single observation, one constant group or all singleton groups. A score near zero reflects the chance adjustment; negative scores are possible. It is not a percentage of correct class predictions.

The native implementation stores counts only for observed labels and observed label pairs. It does not allocate a dense matrix indexed by the largest ID. Pair counts use exact native integers before the final numerical calculation, so map iteration order does not change those counts.

K means cluster IDs represent centers and do not automatically mean class IDs. Use this partition metric for label independent comparison. The K means example and classical ML reference show the distinction between clustering and classification evaluation.

Execution, precision and resource limits

Metrics evaluate stored values on the CPU. Float64 and Float32 Tensor inputs retain their existing numerical precision; using Float64 arithmetic to evaluate a Float32 value does not restore information lost in a prior conversion. Keep large exact class IDs in a representation that preserves them.

An accelerator resident Tensor can require synchronization and a native download into a retained host cache. Include that transfer and cache in timing and memory measurements. A confusion matrix is a new detached CPU Float64 value. Numerical metrics return ordinary Float64 scalars. Inputs and their gradient state are retained without metric driven updates.

Let N be the number of label rows, M the number of cells in numerical tensors, C the confusion matrix class count, Kp and Ka the observed labels in each partition and J the observed label pairs. The following bounds exclude input storage and any host readback caches:

Adjusted Rand index uses native hash maps; pathological collisions can increase its worst case time to O(N²). Observed labels and pairs are bounded by N, and the Tensor cell limit bounds N to 1,000,000. Confusion matrix allocation separately enforces at most 1,000,000 output cells, which limits C to 1000. These bounds are not a total process memory quota.

Errors and evaluation workflow

Missing module imports and incompatible signatures fail during checking. Metric runtime failures use R2401, including mismatched shapes, invalid IDs, IDs outside the confusion matrix class range, an invalid class count and an unrepresentable numerical result. Tensor construction can fail earlier with its own diagnostic, such as R2001 for invalid literal storage.

Compute final evaluation on observations reserved for testing, after choosing settings with training and validation data. A metrics call does not prevent data leakage or enforce a partition policy. Use Dataset partitions to preserve dataset roles and fitted preprocessing contracts. The KNN example demonstrates held out classification; the neural reference documents fitted neural evaluation.

The current source API contains the four functions listed above. There are no sample weights, ignored missing cells, streaming accumulators or automatic class remapping. Prepare missing values and class mappings explicitly before evaluation.