POS and named entity tagging

aner.nlp supports supervised part of speech (POS) and named entity recognition (NER) with source aligned tokens, a contextual averaged structured perceptron, exact Viterbi decoding and inspectable training state. The models learn from the annotations supplied by the caller. No pretrained linguistic model, model download or external service is implied.

These APIs require Aner 0.0.5 or newer; editor help and templates are included in the separate VS Code 0.1.48 publication candidate. The NLP overview describes shared corpus identities and roles. The complete tagging case studies cover held out evaluation and exact checkpoint continuation.

Keep source tokens separate from lexical features

The tagging tokenizer preserves original source spans. It does not reuse nlp.analyze, whose normalization, stopword removal and ngrams intentionally change the lexical sequence. A source aligned token snapshot retains the original corpus, document IDs, roles and raw text fingerprints.

Every corpus row is one explicit tagging sequence. The implementation neither detects sentence boundaries nor joins documents. Split sentences and assign stable source IDs before annotation when sentence level context is intended. Keep sentences from one original document, person or conversation together when choosing training and held out partitions.

nlp.tokenize(corpus) -> TextTokens groups Unicode 17 letter/number-led runs with their following combining marks. Unicode whitespace is skipped. Each remaining punctuation or symbol scalar becomes a separate token. Original UTF-8 is never normalized before source alignment. Continuous CJK letter runs remain unsplit. Emoji joiners and other scalars do not receive grapheme cluster semantics. This is a reproducible tokenization rule, not language specific linguistic segmentation.

nlp.tokenize(corpus, starts, ends) accepts caller defined tokenization as nested Lists of UTF-8 byte offsets, one List per corpus row. Offsets are zero based and half open. Both boundaries must lie on Unicode scalar boundaries; spans must be ordered, nonoverlapping and cover every nonwhitespace source scalar. Interior whitespace is rejected. The explicit policy supports evaluated contraction, punctuation or CJK token boundaries without silently retokenizing annotations.

Normalization can change encoded length. Offsets always refer to the original source, even when the model normalizes token text for its features. A multi token entity's source slice includes the original intervening whitespace.

Annotate POS or strict BIO2 NER

nlp.annotate(value, labels, task) -> TaggedCorpus takes value: TextTokens and requires one nested List of String labels per sequence and exactly one label per token. task is required and accepts "pos" or "ner". Empty rows require empty annotation Lists. The complete training population must contain at least one token.

POS labels use the 17 Universal POS spellings:

ADJ ADP ADV AUX CCONJ DET INTJ NOUN NUM PART PRON PROPN PUNCT SCONJ SYM VERB X

The label vocabulary defines admissible annotation, not an automatic linguistic analysis. See the Universal Dependencies POS definitions. A fitted POS model predicts the subset observed in its training annotations, ordered lexicographically.

NER uses strict BIO2:

• O is outside an entity.
• B-TYPE begins an entity, including a single token entity.
• I-TYPE continues the immediately preceding entity of the same type.

TYPE is an ASCII identifier matching [A-Za-z][A-Za-z0-9_]*, at most 64 bytes. An initial I-TYPE, an I-TYPE after O or a continuation of a different type is invalid. Annotation fails rather than repairing these errors. Separate adjacent entities of the same type with a new B-TYPE. This representation supports flat contiguous entities; nested and discontinuous annotations need a different schema.

The fitted NER inventory includes O and both B- and I- labels for each entity type observed during fitting. Labels have exact lexical order, which also defines deterministic decode tie order. An unseen entity type cannot appear at inference merely because it occurs in held out text.

tagged.tokens(), task(), rows(), token_count() and tags(row) inspect the immutable annotation. Annotation fingerprints include token alignment, task and every ordered gold label. A changed source span or label changes training identity.

Fit a contextual sequence model

nlp.fit_tagger(training, epochs, seed, shuffle, normalization, casefold) -> TextTagger fits only training role annotations.

The tagger is a first order linear sequence model. Its versioned features include bias, normalized current token, normalized context at offsets -2, -1, +1, +2 with BOS/EOS boundaries, scalar prefixes/suffixes of lengths 1 through 3, raw Unicode upper/lower/mixed/uncased category, initial uppercase, ASCII digit/hyphen flags and normalized scalar length clipped at 10, plus beginning/end and adjacent label transitions. It decodes a complete sequence with Viterbi. NER decoding admits only valid BIO2 paths; it never emits an invalid continuation and then repairs it. POS paths are restricted to the fitted inventory.

On a mistaken training sequence, the structured perceptron adds gold feature counts and subtracts predicted feature counts. Current integer weights train the next sequence. Prediction uses the accumulated averaged weights. Averaging state includes exact totals and timestamps for every learned cell; all of it is preserved for continuation.

Equal score complete paths choose the lexicographically smallest label index sequence under labels() order. This is a global path tie rule, not an arbitrary local argmax. No confidence or probability calibration is added. The algorithm follows the structured perceptron family described by Collins; Aner's feature, ordering, budget and artifact protocols are explicit implementation contracts.

Inspect training and decode evidence

Reserved transition feature names are @BOS, @EOS and @T:<previous label>. Weight vectors follow labels() order. model.tag_score(tokens, row, labels, averaged: true) inspects an explicitly supplied valid complete label path. It returns an uncalibrated linear sequence score. Scores depend on sequence length and feature counts; comparing them as probabilities or token confidence is incorrect.

model.tag(tokens) -> TextTagging checks fitted source IDs/raw/normalized token overlap for validation/testing roles, then predicts with averaged weights and retains original alignment, task, the exact model fingerprint and one finite complete sequence score per source row. Empty rows remain empty. prediction.tags(row), scores(), tokens() and model_fingerprint() inspect that evidence.

Entities and evaluation

entities(value, row) accepts NER gold annotations or predictions and returns source aligned entities. Each entity records source ID, sequence row, type, original text, half open token interval, original byte interval and original scalar interval. The source substring is preserved exactly. POS values do not acquire fabricated entity interpretations.

model.evaluate_tagger(gold) -> TaggingEvaluation predicts and evaluates a corpus. For validation/testing roles, it rejects reused fitting source IDs, exact raw documents and identical normalized token sequences. Training or inference evaluation permits intentional reuse and retains that role. These checks do not detect paraphrases or replace grouped/time-separated partition selection.

nlp.evaluate_tags(value, gold) takes value: TextTagging and evaluates that retained prediction only when task and exact token snapshot match. This lower level operation checks alignment; it cannot establish that the model was independent of the gold population. Use the model bound evaluation path for held out training overlap checks.

A correct entity requires matching source row, type and complete token span. Partial overlaps, correct type with wrong boundaries and wrong type with matching boundaries are errors. Undefined precision, recall or F1 denominators are reported as zero; retain the counts to distinguish no predictions from no annotated entities. Token accuracy can be high when O dominates, so report entity metrics for NER. Entity scoring measures annotation agreement, not factual correctness.

Checkpoint continuation

model.save(path) writes a new exclusive data only checkpoint. nlp.load_tagger(path) validates bounded schema, checksum, settings, feature/label inventory, exact integer weights, averaging totals/timestamps, provenance and counters. Existing files and symlink paths are rejected. No executable object format or remote code is loaded.

model.resume_tagger(training, epochs) continues current weights, cumulative epoch shuffle schedule, lazy averaging and counters. epochs is required and adds complete passes. The exact annotated training fingerprint must match; reordered source rows, changed tokens, labels, IDs or role fail. Resume does not refit with a new seed or silently reset averaging.

For a fixed runtime/protocol, an uninterrupted a+b epoch run and an a epoch run followed by save/load and b resumed epochs reproduce the same complete model and prediction fingerprints. This guarantee is stronger than merely obtaining equal final accuracy. Keep source/build identity with checkpoints; equal seeds alone do not promise identity after changing feature or Unicode protocols.

Tagger checkpoints do not retain raw training text. They do retain source IDs, hashes and learned lexical features, so they are not anonymous or privacy preserving artifacts. The checksum detects corruption and provides identity; it does not authenticate a producer.

Bounds, cost and capability scope

Feature extraction scales with retained token/context text. First order Viterbi has O(T × L²) transition work and bounded sequence/label storage for T tokens and L labels. Each fit/resume call builds training row features once; its epochs repeat exact decoding under the declared work budget. Per token label accuracy, full sequence score and entity span F1 answer different questions.

The implementation supplies transparent trainable POS/NER baselines. It does not include pretrained multilingual weights, dependency parsing, morphological feature annotation, nested/discontinuous NER or neural language model fine tuning through this tagger API. Evaluate the chosen tokenization, annotation policy and training population on the target language/domain before interpreting results from the synthetic examples.