Profiling validation and measured costs

This is development evidence from 23 October 2026. It does not establish a general performance advantage over other data libraries. The raw native measurements and fixture identity are in the machine readable record.

Every collected table now retains a basic overview: valid/missing counts, numeric mean and population standard deviation, exact integer and UTC timestamp extrema, and schema/receipt metadata. overview() and the returned report's summary() use that cache, including for disk backed tables. They do no source/table reads. Formatting still costs time proportional to the number of columns. The fixed accumulators consume managed memory and arithmetic during materialization; this is not zero cost analysis.

Rich profiling adds a bounded deterministic sample and category dictionaries. with_profile(...).collect(...) fuses this work into collection; an explicit scan_table(table).profile(...) traverses the retained table. CSV profiling validates every consumed source field, including unselected columns. Each operation remains subject to its own memory and work limits.

Warm cache native experiment

The deterministic CSV has 200,000 rows, four fields, 8,160,376 source bytes, large exact Int64 identifiers, two Float64 measurements, and 23 string groups. It contains no missing values. Four repetitions rotate execution modes; each mode starts in a fresh process. OS caches were not flushed. The build used Apple Clang 21, Release -O3 -DNDEBUG, macOS 26.7.1, arm64, with 12 logical CPUs. Hardware model and physical RAM could not be read and remain unavailable. There was no intentional overlapping build/test load during these measurements.

Median terminal times in milliseconds (basic caching is present in every load):

The fused profile added roughly 26 to 28% to collection time and 115,000 managed peak bytes in this fixture. It saved a second traversal when rich statistics were needed. The CSV twice case is deliberately explicit; it does not model a describe() call on an already loaded frame, which operates on its already loaded frame.

Explicit retained table profiling alone took a median 64.31 ms with one worker and 33.95 ms with four workers in the resident case. Including collection, the four worker workflow took 196.39 ms. These separate measurement blocks are not a randomized cross worker benchmark. Narrow, small, allocation heavy, or storage bound inputs can lose time to synchronization; one worker is the default. Parsing, validation and source hashing stay serial. Fused collection also stays serial. Native worker stacks and OS thread runtime memory are excluded from the managed PMR budget; requested workers are limited to 1 to 32 and capped by projected column count. GPU profiling is not implemented.

The timer includes native parsing, validation, hashes, accumulation, allocation, spill and finalization. It excludes schema/plan creation, cached access, formatting, plotting and teardown. Managed peak values are operation specific, not process RSS or combined peaks of simultaneously retained tables and reports. Source hashes, logical table hashes, complete row counts and selected profile statistics were checked before comparing times. The native probe is aner_data_profile_probe; it accepts source CSV, mode, memory bytes, batch rows, sample rows, category limit and optional CPU worker count. Its required header is i,x,y,label.

Correctness boundaries

Focused native/language tests cover cached access after source removal, exact Int64/DateTime extrema, missing/empty values, physical units and point spread, bounded samples/categories, prefix completeness, incomplete source ancestry, source validation before filtering, spill, memory admission, cancellation and failed output cleanup. Worker tests compare counts, extrema, category ordering, samples, quantiles and floating statistics across row batches and CPU counts. Plot tests verify paired observations, sampling disclosure, detached lineage, offline SVG metadata and size limits. Runnable synthetic and NOAA examples are checked separately. AddressSanitizer/UndefinedBehaviorSanitizer supplement these checks. The isolated worker/resource tests also passed ThreadSanitizer; that does not establish race freedom of every surrounding subsystem or platform.

Samples do not give exact full population quantiles once capped. Capped category dictionaries give exact counts only for retained keys and do not establish a global mode. Numeric means/spreads use Float64 and population ddof=0; large integer means can round. Default basic descriptions omit quartiles and category frequencies. GPU execution, distributed profiling, duplicate/correlation scans, weighted statistics and automatic domain specific validation are not provided. Saved hashes and lineage support reproduction; they do not establish that scientific assumptions, quality policies or conclusions are correct.

See the Native data processing module reference for the shared API.