Natural language processing

aner.nlp provides document corpora, fitted sparse text features, inspectable lexical retrieval, exact token budgets, padded neural input batches and supervised source aligned POS/NER tagging. Choose this module for a scientific text workflow whose source identities and transformations must remain visible. It runs natively and performs no implicit network requests.

These APIs require Aner 0.0.5 or newer. Download availability follows the verified platform packages on Downloads. Rich editor help and templates are included in the separate VS Code 0.1.48 publication candidate; a runtime update does not install the extension.

Choose the right owner

The name aner.nlp identifies the scientific domain. General text editing remains on String; tokenizer and decoder implementations retain their existing owners. This avoids competing APIs for the same operation.

A first evidence search

import aner.nlp
import aner.token

let documents = nlp.corpus(
    List("Reset an account password.", "An invoice records a payment."),
    List("password-guide", "invoice-guide"), role: "training")
let memory = nlp.index(documents, stop_words: List("an", "a"))
let hits = memory.search("password", top_k: 2)
let evidence = hits.context(token.byte_tokenizer(), max_tokens: 100)
print(hits.ids())
print(evidence.text())
print(evidence.tokens())

The complete retrieval example additionally inspects term contributions and verifies artifact restoration. The research workflow explains classification and neural composition.

Document identity and partition roles

nlp.corpus(texts: List, ids: List, role: String = "training") -> TextCorpus requires equally sized nonempty Lists containing only Strings. IDs must be unique, nonempty UTF-8, with at most 256 encoded bytes each. Empty document text is retained. At most 100,000 documents and 4,000,000 combined text and ID bytes are admitted.

Choose partitions by source document, person, conversation, organization or time before fitting text statistics. The accepted roles are "training", "validation", "testing" and "inference". NLP deliberately uses "testing"; token corpus roles in aner.token have their separately documented spellings.

Vectorizer and BM25 fitting require the training role. Applying a fitted vectorizer to validation/testing checks overlap with its fitting population. It rejects reused fitting IDs, matching raw content and matching content under the fitted normalization/casefolding policy. The audit is not a paraphrase, subject or near duplicate detector; choose group/time separation yourself. The inference role represents ordinary prediction and does not claim held out evaluation.

corpus.rows(), ids(), texts(), role() and document_hashes() inspect the ordered snapshot. Document hashes cover raw UTF-8 bytes and retain duplicate content hashes. IDs remain separate from zero based positions. Every native NLP value also supplies fingerprint() and summary().

Analyzer policy and Unicode

nlp.analyze(text, stop_words, analyzer, min_ngram, max_ngram, casefold, normalization) -> List exposes the lexical terms used by fitting and search. Only text is required; the following defaults apply consistently to vectorizers and indexes.

Word stop words are normalized and deduplicated, then removed before word ngrams are formed. A nonempty stop list with "char" fails. There is no hidden English stop list. Word ngrams join retained words with a single space. Character ngrams preserve normalized scalar order; a scalar is not a displayed grapheme cluster or a model token.

The word analyzer is a declared lexical rule, not full Unicode word segmentation, linguistic stemming or a language detector. A continuous CJK letter run remains one word. Character ngrams provide a transparent baseline for scripts without spaces; production segmentation requires an explicitly evaluated policy. Apostrophes and hyphens split words rather than receiving undocumented language heuristics.

For general text, text.normalize(form: "NFC") supports required uppercase forms "NFC", "NFD", "NFKC" and "NFKD". Other String operations still do not normalize automatically. Preserve original scientific text when compatibility distinctions or case carry meaning. The implementation follows the pinned Unicode 17 normalization specification.

Fit count and TF IDF features

nlp.vectorizer(training, stop_words, weighting, analyzer, min_ngram, max_ngram, min_df, max_df, max_features, casefold, sublinear_tf, norm, normalization) -> TextVectorizer fits an immutable representation. training is required; stop_words defaults to List(). Analyzer arguments use the defaults above; fitting arguments are:

The exact smoothed IDF is log((1 + N) / (1 + df)) + 1, where N includes all fitting documents and df counts each document once per term. This convention matches the documented smoothed TF IDF definition; analyzer, filtering, ordering and provenance remain Aner's explicit contracts.

Inspect encoder.configuration() for every canonical option, including the normalized stop list. vocabulary(), document_frequency() and idf() share fitted column order; fitted_documents() reports N, and training_hashes() retains raw fitting content hashes in source order. Fitting has no seed, random restart or hidden global state.

Sparse transformation and numerical boundaries

encoder.transform(corpus) -> TextFeatures applies the same analyzer, vocabulary and training IDF. Unknown or pruned terms do not acquire new columns. Their occurrence counts remain visible through features.oov().

Sparse native storage does not allocate a documents × vocabulary matrix. At most 2,000,000 nonzeros are retained. Exported Lists have the language's smaller 262,144 value limit. Scalar shape/readout calls and row level exports let a larger native sparse value remain usable without claiming that every full List export fits.

Review empty and OOV rows before classification. Converting to a Tensor intentionally drops typed document identity from that numerical value, so retain features.ids() beside predictions. Current aner.ml models do not accept TextFeatures directly; a huge sparse workload is not made scalable by increasing max_cells beyond the Tensor budget.

BM25 lexical memory and explanations

nlp.index(memory, stop_words, k1, b, analyzer, min_ngram, max_ngram, casefold, normalization) -> TextIndex fits lexical evidence statistics from a training role corpus. Analyzer defaults match the analyzer table.

memory.search(query: String, top_k: Int64 = 5) -> TextMatches analyzes the query under the fitted policy. top_k is from 1 through 1,000. Repeated query terms count once. A term contributes log(1 + (N - df + 0.5)/(df + 0.5)) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * length/average_length)). This uses the positive IDF convention documented by the BM25 reference implementation.

BM25 document length counts emitted analyzed term occurrences after the configured stop words and ngram policy. Average length and N include empty documents; they are not based on raw bytes or a second tokenizer.

Only positive score documents become hits. Unknown or empty queries yield zero hits, without fabricated nearest documents. Score ties use the original corpus row position. Scores are ranking values, not probabilities, confidence or semantic truth.

hits.ids(), texts(), positions() and scores() share ranked order. hits.terms() returns canonical unique query terms; contributions(hit) aligns a Float64 vector to those terms, including zero contributions. Its sum reconstructs the selected score within floating point arithmetic. rows() reports hit count. Index configuration(), training_hashes() and ids() preserve the fitted retrieval policy and evidence population.

Context and memory have separate meanings

hits.context(codec, max_tokens, separator: "\n\n") -> TextContext makes evidence text under an exact tokenizer budget. The required max_tokens is from 1 through 1,000,000; the separator is at most 4,096 UTF-8 bytes. Each candidate is the complete accepted text, separator and next document, encoded again with this codec. BPE merges can cross a separator boundary, so adding independently counted document token lengths would not be exact.

Candidates are considered in ranked order. An oversized document is skipped whole; a later smaller document may still fit. No evidence document is truncated. Context readouts are text(), accepted ids(), exact tokens(), skipped(), fingerprint() and summary(). No BOS/EOS or instructions are inserted. Zero hits or no fitting document produce empty text and zero tokens.

Lexical memory is the immutable TextIndex. Retrieved context is a selected evidence snapshot. A decoder's context window and KV cache remain aner.gen concepts. This module does not store conversation sessions, alter model parameters or create semantic long term memory. Retain source IDs outside the assembled text when adding citations or evaluating evidence use. Evidence is input data; the caller owns instructions, access controls and any generation step.

Token batches and explicit padding

nlp.batch(corpus, codec, max_length, truncation, bos, eos, pad_id) -> TokenBatch connects document identity to a neural model without encoding integer IDs through floating point tensors.

token_ids() exports row major exact Int64 IDs; token_ids(row) exports one complete padded row. mask() is a detached CPU Float64 [batch,time] binary Tensor. lengths() records retained real tokens, original_lengths() records encoded length before truncation, and truncated() records each overlength row. source_ids(), source_fingerprint(), batch_size(), sequence_length() and padding_id() expose identity and geometry.

If BOS/EOS are disabled, an empty document remains an all padding row. This is visible data, not a hidden synthetic token. Masked mean pooling rejects such a row because its mean is undefined. Before accelerator composition, explicitly cast and transfer the mask to the same dtype/device as activations. The flat ID List can exceed the language List budget even when the native batch is admitted; row export remains bounded by 4,096 entries.

FFN, normalization and stabilization

These differentiable operations stay in aner.nn, shared by NLP and other sequence domains.

FFN does not silently add normalization, residuals, dropout or a seed. This preserves the ability to test those choices separately. Layer normalization and position wise feed forward blocks are established components described in the original Layer Normalization and Transformer papers; the exact GELU architecture here is an explicit Aner choice.

Use finite inputs, matching numerical profiles, explicit padding/loss masks, seeded initialization and a distinct stochastic schedule. Update each tied parameter handle once with a stable optimizer key. See optimizer state and sequence primitives for graph, device and gradient contracts. The neural example performs one complete declared update and retains document provenance; it is not a pretrained classifier.

Artifacts, traceability and reproducibility

encoder.save(path) and memory.save(path) write exclusive new data artifacts. nlp.load_vectorizer(path) and nlp.load_index(path) validate canonical structure, resource bounds and checksum, rebuild the fitted statistics from the saved corpus/options and verify the fingerprint. Existing destinations and symlink paths are rejected. The binary layout is canonical, but the vectorizer fingerprint also covers computed Float64 IDF bits. Loading recomputes those numbers and requires exact identity; different platform math libraries may reject an artifact instead of silently accepting numeric drift. Same runtime restoration is the tested exact replay boundary.

Artifacts retain source text and IDs, not just a feature matrix or checksum. Treat them according to the source data's confidentiality. SHA256 detects changed bytes; it does not authenticate a producer or provide anonymization. Load does not execute serialized code.

Fingerprints distinguish source order, IDs, role, analyzer choices and fitted/derived state. Lexical fitting and ranking are deterministic and own no RNG. Neural initialization, dropout, shuffle and sampling keep their separately documented streams. Preserve the runtime source/build identity, policy configuration, original partitions, corpus/vectorizer/index/batch fingerprints, model seeds, output mapping and metrics together. Exact numerical identity across changed backends or software versions is not implied by equal seeds.

Bounds, complexity and current scope

Fitting and transformation scale with analyzed terms, vocabulary statistics and sparse nonzeros, not a silently allocated dense product. Sorting contributes logarithmic factors for vocabulary selection and per row CSR ordering. BM25 scores matching postings and orders positive candidates. Whole document context admission deliberately re encodes candidates, and fails at its explicit work bound rather than claiming constant time budgeting. Unicode normalization uses worst case O(n + r log r) time for r decomposed scalars and O(n + r) space.

Supervised POS and NER are available in the tagging reference and tagging case studies. This foundation does not include pretrained linguistic weights, stemming, translation, sentiment models, approximate vector indexes, sparse neural kernels, arbitrary model import, streaming industrial tokenizer fitting or automatic conversation orchestration. It addresses visible research failure modes through training only fitting, normalized duplicate checks, exact integer IDs, explicit sparse/dense boundaries, source linked explanations and measured token budgets. Task quality, multilingual validity and large scale throughput require their own datasets and evaluation; these mechanisms do not claim to solve every limitation of existing libraries.

See the complete callable index for the native module and method name catalogue.