NLP research workflows

Use aner.nlp for source aware lexical experiments and explicit neural input preparation. These complete cases use invented documents, run locally and need no credentials or downloaded models. They require Aner 0.0.5 or newer. The separate VS Code 0.1.48 candidate supplies matching editor help.

The POS and NER case studies add source aligned linguistic annotation, strict entity span scoring, per label error analysis and exact sequence tagger continuation.

Case study: classify support requests

The classification example separates password/account requests from billing/invoice requests. It declares six training documents and four validation documents with distinct source IDs. Labels 10 and 20 are stable external meanings rather than positions in a vocabulary.

The sequence is deliberate:

Declare document partitions before vocabulary or IDF fitting. In a real support dataset, keep each customer/conversation and copied ticket template in one partition; use time separation when predicting future requests.
Fit the word and bigram TF IDF encoder on the training corpus only. Record its canonical stop list, Unicode policy, frequency thresholds, sublinear weighting and row norm.
Transform both populations with that unchanged encoder. Inspect OOV term counts and empty rows before looking at classification metrics.
Admit a small dense matrix explicitly, fit a one neighbor baseline and retain the feature row IDs next to predictions.
Compare validation accuracy under a declared label mapping. Tune only on validation; obtain a separate final testing corpus before reporting a chosen model's quality.

aner check examples/nlp_classification.aner
aner run examples/nlp_classification.aner --prod

The program prints validation accuracy, original prediction IDs, OOV occurrences, empty rows, canonical configuration and the fitted representation fingerprint. Its tiny synthetic accuracy is a contract demonstration, not a support system benchmark.

The default L2 normalization makes Euclidean KNN a useful transparent baseline on these positive TF IDF rows. It does not turn KNN into a semantic language model. Review the KNN tie/class mapping contract before interpreting probabilities. A zero feature row carries no retained lexical evidence; decide whether to reject, label or route it before production use.

Case study: retrieve evidence before generation

The retrieval example indexes four fictional help documents. It searches for account/password recovery evidence, inspects score contributions, then assembles whole documents under 110 exact byte codec tokens.

aner check examples/nlp_retrieval.aner
aner run examples/nlp_retrieval.aner --prod

Run from a fresh writable directory containing the source, preserving its shown relative path. The program creates support-memory.anernlp exclusively. A second run in the same output directory fails at save rather than replacing evidence. It restores the index, repeats the search, and compares the complete fitted/search fingerprints.

TextIndex holds the evidence population and lexical statistics. TextMatches preserves score order, original IDs/text/positions and the unique query terms. Each contribution vector is aligned to those terms. Keep this evidence next to any later answer, rather than retaining only a concatenated prompt.

The exact budget includes separators and possible BPE boundary merges. Oversized candidates are skipped whole and later smaller candidates remain eligible. context.ids(), tokens() and skipped() expose what actually entered the final text. A short budget may select fewer documents than top_k; no hit or a document that cannot fit produces no fabricated evidence.

Before adding generation, define the complete model input: instructions, user question, evidence text, separators and any requested special tokens. Reserve room for generated output inside the model's context window. NLP's evidence budget covers only the assembled evidence, not that complete prompt. aner.gen remains responsible for the decoder's actual context admission and cached execution. The retrieved text supplies evidence, not application instructions.

BM25 relevance needs an evaluation set of queries and judged relevant document IDs. Measure retrieval recall/ranking and final evidence use separately from language fluency. This example demonstrates deterministic evidence handling; it supplies no relevance benchmark or factuality score.

Case study: a small differentiable text encoder

The neural example prepares three synthetic documents and performs one explicit Adam update. It is a complete composition of the existing neural stack and the new input/FFN/pooling bridges.

aner check examples/nlp_neural.aner
aner run examples/nlp_neural.aner --prod

The byte tokenizer is fixed and requires no fitting. The TokenBatch retains source IDs, exact integer IDs, original/retained lengths and truncation. A 16 position width admits each document with explicit BOS/EOS; truncation: "error" prevents silent information loss.

Stage	Input and result	Scientific contract
Embedding	Exact IDs to `[3,16,4]`	Byte IDs never pass through Float64. A seeded parameter table owns the learned vectors.
Layer normalization	Last axis of `[3,16,4]`	Learned gain/bias and explicit epsilon; per token population moments.
Self attention	`[3,16,4]` with key mask `[3,1,16]`	Padding keys are excluded. This example shares normalized vectors as Q/K/V and makes no pretrained representation claim.
FFN	Width 4 to 8 to 4	Explicit projections/biases and exact GELU. No hidden residual, normalization or random call.
Dropout and residual	`[3,16,4]`	Probability 0.1, training flag true and owned seed 7000 for declared update zero.
Masked mean	`[3,16,4]` and mask `[3,16]` to `[3,4]`	Pad queries may exist, but they contribute neither to pooling nor its gradient.
Head and loss	`[3,4]` to raw logits `[3,2]`	Stable sparse cross entropy with exact class IDs.
Clip and update	Every distinct parameter once	One global gradient clip followed by stable named Adam slots.

The program prints the pre update loss and provenance. It demonstrates gradients and state ownership, not a trained accuracy result. For a full experiment, add complete epochs, validation, independent testing, measured loss curves and an explicit dropout/mini-batch schedule.

With accelerators, transfer every parameter, activation and mask under a single declared dtype/device. The batch mask starts as CPU Float64; cast before strict Metal/MPS placement, which requires Float32. A changed backend can change floating point results even with unchanged seeds. See Tensor devices and sequence contracts.

Context, FFN and stabilization choices

Context length, positional information, neural memory, evidence memory and document grouping are separate design decisions. The example's attention has no learned positional embeddings and no persistent KV cache; it is an inspectable bag like baseline. Use aner.gen's managed decoder for learned positions, causal training, retained optimizer/data lineage and checkpoint continuation. Use explicit recurrent state only when the selected recurrent model contract fits the task.

Pre normalization, residual width matching, finite positive epsilon, stable log sum exp losses and padding masks address distinct numerical concerns. Gradient clipping limits one update's norm; it does not repair mislabeled data, invalid loss masking or a poor learning rate. FFN's hidden width and activation change capacity; they are settings to compare on validation data, not universal best values.

Do not silently rerun a fixed dropout seed at every training step. Define and record a versioned step/layer schedule. Turn training dropout off for validation. Stable optimizer keys must stay attached to the same parameter shapes and placements; tied weights must be updated once. Custom Tensor classes do not gain a checkpoint API automatically. If you need managed training continuation now, use the owning Dense or generative model APIs and their documented checkpoint formats.

Reproducibility record

Retain document source/IDs/order and partition policy, corpus role/fingerprint, all analyzer options, fitted vocabulary/IDF, vectorizer/index fingerprint, tokenizer identity, exact batch geometry and truncation, model architecture, numerical backend, seeds/schedules, optimizer settings and row aligned outputs. Preserve both raw evidence and accepted context IDs when evaluating retrieval plus generation.

Raw and policy normalized duplicate checks catch a defined subset of leakage. They do not replace customer, speaker, site or time grouping. Sparse export and token batch bounds make resource admission visible; they do not imply unbounded industrial workloads. The API reference lists every argument, output, deterministic tie rule, resource limit and currently unsupported capability.