POS and NER case studies

These complete synthetic examples demonstrate trainable linguistic tagging with source identities, held out evaluation and exact checkpoint continuation. They run in the native CPU runtime and require Aner 0.0.5 or newer. They use invented English sentences and annotations supplied in the source; no pretrained model, downloaded corpus, service or credential is needed.

Start with the tagging API reference for defaults, label validation, source offsets, resource bounds and continuation semantics. The NLP overview explains the common corpus roles and identities.

Case study: contextual POS annotation and error analysis

The complete POS program supplies 14 training sequences and four validation sequences. Each source row is one sequence. It tokenizes original source text, then attaches one exact Universal POS label per token. The training examples include proper/common nouns, verbs, determiners, adjectives, adverbs, numerals, pronouns, auxiliaries, conjunctions and punctuation.

aner check examples/nlp_pos.aner
aner run examples/nlp_pos.aner --prod

The validation sentences recombine familiar vocabulary and contexts into distinct sequences with independent source IDs. This makes the example useful for checking context learning and row alignment; it does not estimate performance on a natural corpus. A real experiment should split by original document, speaker, conversation or time before sentence extraction, annotate a declared target language/domain, select settings on validation and reserve independent final testing data.

The model fits eight epochs with seed 731, shuffling enabled, NFC feature normalization and full casefold. It learns lexical/context/shape/affix scores and label transitions with structured perceptron updates. Public predictions use averaged weights and exact Viterbi decoding. These settings are an example configuration, not universal best values.

The output retains several distinct observations:

• Token accuracy is correct labels divided by all gold tokens. It is not a sequence level exact match rate.
• Sparse confusion counts identify gold/predicted label pairs, including the correct diagonal. Their counts sum to the evaluated token count.
• Per row original tokens and predicted tags can be inspected next to their original source ID.
• The fitted label inventory shows which UPOS labels the training population can predict; admissible annotation labels absent from fitting do not become learned classes.
• Cumulative epochs, examples seen and updates distinguish the schedule, averaging clock and mistaken sequence corrections.
• The canonical configuration and @BOS weight vector expose the declared feature/training policy and label aligned score components.

The recorded run scores all 17 validation tokens correctly across the four invented POS sequences. It completes eight epochs, processes 112 nonempty sequences and performs 23 structured updates. These are measured outputs of this small declared example; they are not an estimate of general POS accuracy.

For error analysis, inspect off diagonal confusion pairs and the source sequences that produce them. Distinguish ambiguous annotation, tokenization mismatches and out of domain language from ordinary classification mistakes. A high aggregate accuracy can hide failures on rare tags; preserve per label support, precision/recall/F1 and representative examples.

Case study: source aligned entity extraction

The complete NER program annotates fictional collaboration briefs with PER, ORG and LOC types. Training includes multi token organizations and locations, adjacent nonentity words, punctuation and the Unicode name Zoë. Validation recombines names and contexts in three distinct sequences.

aner check examples/nlp_ner.aner
aner run examples/nlp_ner.aner --prod

The annotation uses strict BIO2: B-ORG starts an organization, I-ORG continues it and O marks outside tokens. The data boundary rejects invalid continuations. Decoding searches only legal BIO2 paths rather than applying an undocumented repair afterward. The model includes both B/I forms for each observed entity type, plus O.

The program fits ten epochs with seed 911 and the same explicit normalization/casefold policy. It reports strict full span entity counts and precision/recall/F1, metrics by entity type and token accuracy. A correct entity must match the source sequence, complete token interval and type. A partial Delta prediction for gold Delta Labs is a false prediction and a missed gold entity; token level agreement alone does not make it a correct entity.

The recorded NER run labels 17 of 18 tokens correctly and matches six of seven complete gold entities, with seven predicted entities. Entity micro precision, recall and F1 are each 6/7 (approximately 0.85714). The type breakdown exposes one missed PER and one extra LOC; aggregate token accuracy would obscure that entity error.

The first predicted sequence prints original token strings, BIO2 tags and entity Dictionaries. Each entity carries original ID/text and half open token, UTF-8 byte and Unicode scalar intervals. In the Zoë sentence, encoded byte starts differ from scalar starts after the name. Use scalar intervals with String.slice; use byte intervals for byte oriented annotation interchange. The model's normalized feature spelling must never be used as a replacement source offset.

The program also inspects zero based validation row 2, North Labs hired David ., printing gold and predicted tags and entities. In the recorded run, David is predicted as LOC instead of the gold PER. Its source span is correct while its type is wrong, so strict entity scoring records one false positive and one missed gold entity. This is an inspectable failure case, not a result to remove from the report.

These records are useful for an extraction audit: retain the input document ID, exact source span, type, model identity and annotation/evaluation policy alongside a downstream record. Review false positives and false negatives against original text. Entity tags express agreement with a declared annotation task; they do not establish identity, truth or safe automatic redaction.

Exact checkpoint continuation in both cases

Run each example from a fresh writable working directory with the shown source path available. The POS program exclusively creates pos-stage.anertagger; NER creates ner-stage.anertagger. Reusing an existing destination fails rather than replacing an earlier experiment.

The POS program compares an uninterrupted eight epoch model with three epochs, save/load and five resumed epochs. NER compares ten epochs with four, save/load and six resumed epochs. Both finish by comparing complete model and held out prediction fingerprints, printing true for each equality when continuation reproduces the uninterrupted run.

Continuation preserves the current training weights, lazy averaging totals/timestamps, cumulative epoch shuffle schedule and counters. It requires the exact annotated training fingerprint. Changing labels, source IDs/order, token spans or corpus role is a different experiment, even when the visible text looks similar. The resumed model's epochs() and canonical configuration are cumulative, so it has the same identity as the uninterrupted schedule.

Averaging processes every nonempty sequence, including correctly decoded sequences; updates() counts only structured mistakes. Restarting the epoch seed schedule or restoring only averaged inference weights would not reproduce this experiment. The checkpoints retain the continuation state needed for the stronger equality check.

Bring an independently annotated corpus

Choose and record a segmentation and annotation policy before fitting. A corpus row is a sequence, so a document containing multiple sentences is not automatically split. For an existing corpus with token offsets, use the explicit nlp.tokenize(corpus, starts, ends) overload and preserve its original byte boundaries. Supplied spans must cover every nonwhitespace scalar, remain ordered/nonoverlapping and avoid interior whitespace. Gold labels must align exactly to those tokens.

Split original sources before fitting or selecting any configuration. The validation/testing guard rejects fitting source IDs, raw text and identical normalized token sequences. It catches a defined class of exact reuse; related documents, paraphrases and repeated subjects still need a caller defined partition policy. Annotating a corpus as inference intentionally permits known source scoring and does not establish held out evaluation.

Use model.evaluate_tagger(gold) for model bound held out checks. Use nlp.evaluate_tags(prediction, gold) when evaluating already retained predictions; this path requires exact task/token identity but does not know how the model was trained. Preserve the role and source/model/gold/prediction fingerprints with the metrics.

All 17 UPOS labels are admissible, but a fitted POS model predicts its observed subset. NER models predict only entity types seen in fitting. Evaluate out of vocabulary names and rare labels deliberately. The small lexical/context model supplies an inspectable baseline, and its synthetic cases are not evidence for multilingual or domain wide accuracy.

Reproducibility record

Retain source IDs, original text/hashes/order, partition/group policy, tokenization protocol and exact spans, annotation task/schema and every gold label. Record the model feature/Unicode protocol, canonical normalization/casefold choices, fitted label/feature inventory, seed/shuffle schedule, cumulative epochs, examples/tokens seen and updates. Preserve checkpoint/model identity, prediction scores/labels/source spans and exact evaluation counts.

Tagger checkpoints omit whole original source documents but retain learned lexical/context/affix features, source IDs and hashes. A checkpoint can disclose source information. Its checksum establishes integrity and identity, not producer authentication or anonymity.