Data profiling and native charts

aner.data always caches basic structure and descriptive statistics while it loads an immutable table. It can optionally accumulate a richer heterogeneous, bounded profile in the same pass. The profile connects descriptive statistics to the exact selected population, physical schema, source validation and execution receipt. aner.plot consumes that report or table directly by column name. There is no foreign runtime bridge, per row Aner callback, or need to copy columns into Lists to make these diagnostic charts.

Run the complete synthetic example with its twelve row fixture from the repository root:

aner check examples/data_profiling.aner
aner run examples/data_profiling.aner --display-dir build/profile-figures

build must exist and build/profile-figures must be a fresh destination. In the development VS Code extension, Run File displays the charts in its adjacent plot panel; notebooks attach each chart to its producing cell. The 0.1.48 editor candidate is prepared separately for publication.

Inspect basic information immediately

let table = source.collect(0, 8388608, 1024)
let basic = table.overview()
print(basic.summary())
print(basic.valid("signal"))

Every retained table has cached structure, missing/present counts and basic numeric moments/extrema. overview() exposes an immutable DataReport containing the cached structural and statistical information. Read its typed accessors or call report.summary() for a combined human readable String, including storage and execution coverage. Physical means/spreads keep their existing typed unit accessors. These accessors never trigger hidden source/table I/O. Basic capture uses O(columns) managed state and arithmetic during append; it has real CPU/memory cost, but retains no numeric sample or category dictionary. If rich capture was enabled, the same cached overview and its summary also contain those richer diagnostics.

An Int64 identifier can have a mean printed by a type based summary without that mean being scientifically meaningful. Inspect field roles and choose actual measurements for analysis; primitive data types do not infer experimental meaning. Run the actual NOAA climate workflow for basic inspection followed by an explicitly chosen deeper profile.

Profile during loading

let table = source.with_profile(sample_rows: 1024, max_categories: 64)
    .collect(max_rows: 0, max_bytes: 8388608, batch_rows: 1024)
let report = table.profile()
print(report.summary())
print(report.profile_json())
let figure = plot.missingness(report)
figure.show()

with_profile is an immutable lazy plan option. Collection updates the native accumulators as its selected batches are already being consumed. The cached DataReport describes exactly the retained rows and projected fields, after declared fills, unit conversions and filters. Calling table.profile() again returns that attached immutable report without reopening CSV, reading a disk backed table or aggregating the table again. The report is not recomputed when an upstream file changes.

Deep profiling is explicit because it uses additional CPU and managed memory. The fused path avoids a separate source pass and a separate traversal of the retained table; it is not free arithmetic. The standard deviation still uses the existing population convention. Existing lightweight summarize remains available when only basic streaming statistics are required.

A table exposes the structured report through overview(); formatting belongs to DataReport.summary(). Module calls such as data.profile(table) and data.with_profile(scan, 1024, 64) are also available. An uncaptured table can be profiled deliberately with table.scan_table().profile(...); this traverses the retained snapshot and may read its private disk backing. Explicit snapshot profiling may opt into native worker execution with .with_profile_workers(n); the default is one worker, and fused load time capture remains serial. Resource admission, supported source restrictions and measured costs are documented in profiling validation. A standalone CSV profile reopens its source as any other terminal does. Schema inference is a distinct inspection pass. Neither is silently hidden inside a cached accessor.

with_profile().preview(...) attaches a profile of the returned preview rows. A work bounded standalone profile can similarly return explicitly incomplete statistics. Check complete(), origin_complete() and stop_reason() before treating a result as a full acquisition profile. complete concerns the current terminal source; origin_complete additionally preserves whether the original source ancestry was fully consumed. Strict collect continues to reject an incomplete execution instead of publishing a truncated full result.

What is measured

All column accessors use the projected column name. report.cols(), column_name(index), column_type(index), column_unit(index) and column_nullable(index) expose the report's schema in projection order. Units describe current values; nullability remains the source declaration even if filtering removes every missing observation.

Inapplicable statistics fail instead of coercing strings, timestamps or booleans into numerical measurements. Min/max and quantiles require present values. JSON and the text summary indicate unavailable fields rather than inventing zero valued observations.

The ordinary min/max/mean/std numerical accessors remain Float64 statistics for primitive numerical fields; Int64 values beyond Float64 exactness may round there. Use minimum_text, maximum_text and quantile_text for exact Int64 extrema/order statistics. Exact storage and exact filtering do not make arbitrary floating point moments exact rational arithmetic. Numeric aggregation can fail when intermediate finite range arithmetic overflows.

Physical columns require typed statistics: unit_mean, unit_min, unit_max and unit_std take an explicit compatible target unit. A Point temperature mean/extremum can use degC, while its spread uses delta_degC. Offsets never shift a temperature spread. Primitive statistics reject physical columns instead of dropping their unit. The profile's text/JSON retains physical context, and diagnostic plots label the current unit.

DateTime extrema and quantiles order exact UTC nanoseconds. Input offsets have already been resolved during ingestion; the profile neither guesses named timezones nor reconstructs original offset spellings. Present nonfinite numeric inputs remain invalid source data rather than a second missing value convention. CSV missingness still follows the scan's explicit token: in the example NA is missing and an empty String in note is present.

Bounded quantiles and categories

For each numerical/DateTime field, the profiler retains at most sample_rows valid observations using deterministic bottom k priorities. Priorities are generated from the valid value ordinal by a fixed, versioned SplitMix64 policy. Sampling depends on source order and the selected population but not on batch size or RAM/disk backing. It is not an externally seeded randomized study design, and no distribution free quantile error bound is claimed.

Quantiles use the empirical nearest rank order statistic with no interpolation: for n retained values and 0 < q <= 1, select index ceil(q*n)-1 after ordering; q=0 selects the minimum. The endpoints q=0 and q=1 use exact full population extrema even when the sample is disabled. Interior quantiles require at least one retained sample. q must be finite and within [0,1]. quantiles_exact is true when every valid value was retained; otherwise interior quantiles describe the bounded sample. This differs from an interpolating quantile convention. Missing observations never occupy sample slots. Each field samples its own valid value ordinals independently; samples from different fields cannot be zipped into paired observations or used as an aligned correlation input. The table scatter adapter reads actual same row pairs instead.

Per column category dictionaries retain the first max_categories distinct present keys for every supported type. Int64/DateTime keys remain exact; Float64 keys use round trip representations with positive/negative zero treated as one value. Physical category labels retain the current unit. Counts for those tracked keys continue to update exactly through the entire selected population. If another distinct key occurs, distinct_complete becomes false; distinct_count is then the lower bound max_categories + 1, not an exact distinct estimate. Dropped keys are not accumulated into a guessed category count. Increasing a bound requires a new deliberate execution.

A complete dictionary supports exact top_value and top_frequency; ties use the stored textual key's byte order for deterministic results. Those top accessors reject an incomplete dictionary, because an unseen key could be more frequent. Indexed category access returns tracked buckets in descending frequency, then lexical key order, with exact counts for each tracked key. It does not turn an overflowed dictionary into a global ranking. No row level strings need to be retained merely to calculate counts.

Plot a report or table directly

plot.missingness(report).show()
plot.distribution(report, "signal", bins: 12).show()
plot.categories(report, "cohort").show()
let relationship = plot.scatter(table, x: "Distance_cm", y: "Temperature_DegC",
    max_points: 10000)
relationship.show()
print(relationship.provenance())

These adapters return ordinary immutable Chart values: reuse .title, .labels, .caption, .theme, .show, .svg, .html and existing exports. Structured chart.provenance() / plot.provenance(chart) records the adapter, source/report lineage, columns, coverage and adapter parameters. Native SVG metadata and offline HTML preserve that structured lineage. This supplements visible chart captions; a caption is not the only place the evidence is stored. Adapter provenance identifies construction inputs and policies, not a complete audit of every later manual chart edit or an analysis rationale. Figure data, category labels and embedded recipes may be sensitive: chart lineage includes source paths, schema names, fill literals and filter conditions as well as hashes and receipts. Review exported metadata before sharing it.

Scatter's row bound applies before missing pair omission, so it never scans indefinitely to find a requested number of complete pairs. Its ordered prefix may be biased by acquisition order. Report sampling and scatter prefixes are different policies and are labeled separately. A preview table and descendants retain incomplete acquisition ancestry in chart lineage, even when a later terminal completely traverses that preview snapshot. Units are not silently normalized for plotting; use the data plan's explicit conversion when a different coordinate unit is needed.

Missingness retains a bar for every selected field, but hundreds of column labels cannot remain readable within the renderer's 4096 pixel maximum width. The existing bar renderer shows at most 12 evenly spaced axis labels; larger missingness figures disclose sparse labeling in their caption and record the label stride in their lineage. Every bar retains its full column name in its tooltip. Select fewer fields before profiling to make focused figures, or explicitly resize a smaller chart with .size(width, height) within that bound. Long visible labels may be abbreviated. Adaptive sizing applies to missingness figures; ordinary bar chart defaults stay unchanged.

Numerical plot coordinates require an exact representable Int64 conversion within [-2^53, 2^53]; large observation IDs therefore remain usable for data predicates and exact profile text, but are rejected as rounded plot coordinates. Float64 and current unit physical magnitudes are supported. DateTime axes are rejected until a deliberate temporal coordinate/formatting policy exists. Empty or all missing numerical chart data are not fabricated into zero valued marks. Scatter additionally uses a separate 64 MiB native row workspace; a single decoded table row that exceeds it fails explicitly. Static SVG exports retain the existing 32 MiB cap, ordinary offline HTML has an 8 MiB cap, and attached structured adapter lineage has a 2 MiB cap. Charts remain bounded snapshots, not a live connection to a mutable source or a replacement for batched numerical/model output controls.

Traceability and scope

A profile carries the effective schema, projection, normalization/filter plan, profile options and algorithm policies. Scan recipes use aner.data.recipe.v4, including the profiling policy; condition recipes retain aner.data.condition.v1. profile_json includes the source/receipt evidence and typed statistics; provenance supplies the deterministic recipe and hash. Fused report lineage includes the retained logical table identity, independent of RAM/disk backing and batch size. Table source recipes preserve parent input kind/hash alongside the logical table hash; recursive frozen recipes retain the original supplied input digest without copying volatile ancestor timing receipts or retaining their data payloads. Receipt elapsed time and peak memory can vary by execution and are not deterministic statistical content.

A complete direct CSV execution validates every source field through EOF. complete() means the current terminal consumed its whole immediate source; origin_complete() records whether that result also descends from fully consumed source ancestry. For example, profiling every row of a four row preview snapshot returns complete == true and origin_complete == false: completing a snapshot does not validate the unread original acquisition tail. This flag propagates through derived scans, tables, reports and execution receipts, so repeated materialization cannot erase earlier truncation. A bounded report's digest/receipt describes work actually consumed; read ahead can include bytes beyond validated rows. A prefix source digest must not be presented as a complete file identity. A scan_table traversal instead uses the immutable parent snapshot's logical hash. The ancestry flag covers source history known to Aner. It cannot detect observations omitted by an upstream SQL LIMIT, export process or acquisition system before that source was supplied; archive those acquisition rules separately. Source bytes and the recipe must still be retained to reproduce a study; a digest cannot recover a missing file.

Profiling occurs after normalization, filtering and projection. For example, fill_missing reduces observed missingness and drop_missing changes the population. A selected profile is evidence about that selected population, not an automatic profile of the raw acquisition. A cached profile never silently inherits a new filter applied later. Derived collection requires its own with_profile option or deliberate standalone profile of its snapshot.

See the data scientist workflow evaluation for held out separation, research integrity, comparisons with other tabular and plotting tools and an honest performance measurement protocol. This report does not assign experiment roles, detect leakage, validate consent, test hypotheses or establish causation.

Resource cost and limits

Profile state, sample storage, category keys and accumulator allocations participate in the managed execution memory budget. Retention is bounded per projected field, not by total dataset rows. More fields, larger samples and longer retained category strings need more workspace. A profile allocation can force retained table values to spill sooner; spill cannot remove required parser, batch or profile state. A source record and the required accumulator state must fit the configured RAM budget. Disk directory/quota/cleanup, cancellation, timeout, full source validation and predictable row order retain their normal contracts.

Caching avoids repeat aggregation; producing a human readable report or JSON still allocates output text, and plotting still allocates and renders bounded chart data. Managed peak bytes are not a whole process resident memory limit. Configuration metadata, interpreter state, rendering and detached exports have their existing separate accounting. A program can disable rich capture with without_profile, use zero samples/categories or call lightweight summarize to reduce unnecessary work. Basic table statistics remain cached. See measured costs and validation for scoped native evidence; the benchmark does not claim zero overhead or comparative performance.

Profiling currently accepts DataScan/DataTable populations, not AdaptiveFrame context views directly. AdaptiveFrame retains its separate 1,000,000-row/64-field adapter limits, and model/Array/Tensor exports retain their explicit materialization limits; a spillable table does not automatically make a fitted model external memory.

The first implementation does not compute pairwise correlations, general group summaries, duplicate row counts, inferred domain rules, exact external memory quantiles, weighted statistics, confidence intervals or arbitrary custom per row statistics. Category overflow is disclosed rather than approximated. Numeric samples are bounded and deterministic, and no automatic incremental model fitting or distributed execution is added. Use explicit training only scientific workflows and retain the data needed to repeat any analysis.

The structured profile also records the native runtime version/channel, development marker, compiler, CPU target/profile, build type, and a SHA-256 fingerprint of the profiler implementation sources (including uncommitted development content). This helps identify the code that produced a report; it is not a complete dependency lock, signed attestation, or substitute for archiving the analysis program and build configuration. Finished reports keep frozen lineage metadata and do not retain ancestor table payloads solely to describe their source.

See the Native data processing module reference for the shared API.