Unit aware CSV columns

aner.data can declare a registered physical unit for a whole Float64 CSV column, normalize that unit during ingestion and return checked Quantity or Point values. Import aner.data and aner.units. This is a specialized workflow within aner.data, with scalar units and packed unit arrays; there is no deeper module name to import.

These features require the updated native development runtime built from this source and matching editor support. Earlier packaged executables may reject the new calls. This guide does not announce a new binary release or performance benchmark. See installation and the data module reference for native CSV parsing, resource limits and execution controls.

Declare what source numbers mean

CSV fields contain plain numerical magnitudes. The schema states the source unit of every unit bearing column; the reader does not guess from names, headers, suffixes or observed values.

The beginner fixture contains six complete synthetic source records:

sensor_id,temperature_f,distance_cm,elapsed_s,eligible
0001,68.0,100.0,2.0,yes
0002,71.6,120.0,4.0,yes
0003,NA,140.0,5.0,yes
0004,75.2,NA,2.0,yes
0005,78.8,80.0,4.0,no
0006,82.4,180.0,3.0,yes

temperature_f is declared as degF, distance_cm as cm and elapsed_s as s. The IDs remain Strings so their leading zeros survive. Eligibility is a String field selected with the existing String equality filter. NA is the explicitly configured missing token, and the temperature/distance columns permit it.

A complete schema must include all source columns in header order, even when a later projection excludes some:

import aner.data
import aner.units

var schema = data.schema()
schema = data.column(schema, "sensor_id", "String", false)
schema = schema.unit_column(name: "temperature_f", unit: "degF", nullable: true)
schema = data.unit_column(schema, name: "distance_cm", unit: "cm", nullable: true)
schema = schema.unit_column(name: "elapsed_s", unit: "s", nullable: false)
schema = data.column(schema, "eligible", "String", false)
let source = data.scan_csv("examples/fixtures/units/measurements.csv", schema, missing: "NA")
let eligible = data.where_eq(source, "eligible", value: "yes")
let report = data.summarize(eligible, max_bytes: 1048576, batch_rows: 2)
print(data.scanned_rows(report))
print(data.rows(report))
print(data.valid(report, "temperature_f"))
print(data.missing(report, "temperature_f"))
print(type(report.unit_mean("temperature_f", target_unit: "degC")))

Run from the folder containing examples/fixtures/units/measurements.csv. The output is 6, 5, 4, 1 and Point<degC>. Planning opens no file; summarizing reads and validates every source record. The unit column methods return new immutable schema handles, so assigning the returned handle updates var schema.

The fixed 23 code registry determines kind: degF, degC and K are absolute temperature Points; delta_degF, delta_degC and delta_K are interval Quantities. Length, time, speed and other registered linear codes produce Quantities. There is no new dtype called Quantity to pass to data.column; use unit_column with an exact registered code.

API reference and target types

Schema and scan handles have opaque static types. Their unit_column and convert_unit unit arguments can be runtime Strings; the operation validates the registered code, dimension and kind when it executes. The three output accessors unit_value, unit_values and unit_mean require literal target unit Strings because their target determines the checked scalar/array return type. There is no additional generic call syntax or arbitrary unit expression parser.

Use the exact argument labels name, unit, nullable, column, target_unit, row and value for these calls. The existing data execution labels remain max_rows, max_bytes and batch_rows, as in data.collect(scan, max_rows: 100, max_bytes: 1048576, batch_rows: 2).

Normalize units during ingestion

Use convert_unit to define the units of the imported columns before analysis:

import aner.data
import aner.units

var schema = data.schema()
schema = data.column(schema, "sensor_id", "String", false)
schema = schema.unit_column("temperature_f", "degF", true)
schema = schema.unit_column("distance_cm", "cm", true)
schema = schema.unit_column("elapsed_s", "s", false)
schema = data.column(schema, "eligible", "String", false)
let original = data.scan_csv("examples/fixtures/units/measurements.csv", schema, "NA")
let normalized = original.convert_unit(column: "temperature_f", target_unit: "degC")
    .convert_unit(column: "distance_cm", target_unit: "m")
let complete = data.where_present(data.where_present(normalized, "temperature_f"), "distance_cm")
let table = data.collect(complete, max_rows: 100, max_bytes: 1048576, batch_rows: 2)
print(data.column_type(table, 1))
print(data.column_unit(table, 1))
print(data.column_type(table, 2))
print(data.column_unit(table, 2))

This collects four complete source rows, including the row with eligibility no, because this particular snippet has no eligibility filter. The output is Point<degC>, degC, Quantity<m> and m. Header suffixes remain descriptive source names: temperature_f now contains converted Celsius magnitudes in the collected table, and its metadata records degC.

A normalization plan keeps both the original source interpretation and the final target descriptor. Every present source value is parsed in its declared source unit and converted directly into that final target before filters or projection. A failing conversion in an excluded column or rejected row still fails the scan. Missing values retain their missing flags and are not converted into zero.

Repeated convert_unit calls replace the final target for that column; they do not introduce sequential conversions of already rounded intermediate values. Changing a target refreshes compatible threshold comparison kernels so a filter retains its physical unit meaning. The original plan, source schema and source file remain unchanged.

This contract is explicit normalization, not automatic detection of source units. A whole column has one source unit. Mixed units within a single CSV field/column, unit suffixes in numerical cells and unregistered composed units remain unsupported. Choose a separate schema/column or prepare the source explicitly when its unit varies by observation.

Missing cells and observed means

Unit columns retain existing strict CSV parsing and missing token rules. Unquoted NA is missing with this fixture's token; quoted "NA" is a present field that fails numerical parsing. The unit schema does not relax finite Float64 validation, nullability or complete header matching. Reading a missing cell through unit_value fails.

Report means use present selected values only. In the beginner fixture, five rows have eligibility yes. Four temperatures are present and one is missing; their source values are 68.0, 71.6, 75.2 and 82.4 degF. Their mean is approximately 74.3 degF, or 23.5 degC. A column with no present selected value has no mean and fails explicitly.

unit_values does not drop missing rows or fill them. Apply where_present explicitly before collection when the analysis needs a complete column. In the beginner example, requiring both temperature and distance selects IDs 0001, 0002 and 0006, in original source order. Column nullability metadata stays true: completeness of this particular selection does not rewrite the source schema.

All full source fields are validated before missing filters, physical thresholds or projection. Retaining String sensor/group identifiers alongside measurements preserves their text and row association. These tables do not automatically create immutable source hashes, protected experimental partitions, an uncertainty model or named coordinate domains.

Physical thresholds

The existing where_ge accepts a compatible unit bearing threshold for a unit column. A Fahrenheit Point column can be compared with 21.0@degC; a centimeter Quantity column can be compared with 1.0@m. A primitive Float64 threshold is rejected for a unit column even if its numerical value resembles the desired magnitude. A temperature interval cannot filter an absolute temperature Point column.

Thresholds retain the shared scalar unit comparison rules: compatible dimensions and matching kind, canonical physical comparison where required, finite/range checks and exact Float64 comparison without an implicit epsilon. Define a deliberate scientific threshold rather than relying on exact equality at a rounded decimal conversion boundary. Missing values never pass a comparison.

Filters combine with AND and successful results preserve source row order. Unit filters do not turn the retained IDs into features or make a mean independent of the chosen selection policy. The research source applies a quality rule, explicit presence requirements, a 21 degC temperature threshold, a 1 m distance threshold and a 1 s elapsed time threshold.

Checked arrays and explicit numerical export

Use converted columns directly in physical arithmetic:

let temperatures: Array<Point<degC>> = table.unit_values("temperature_f", "degC")
let distances: Array<Quantity<m>> = table.unit_values("distance_cm", "m")
let times: Array<Quantity<s>> = table.unit_values("elapsed_s", "s")
let speeds: Array<Quantity<"m/s">> = distances.div(times)

This fragment assumes a complete collected table with those columns. The unit array guide documents independent buffers, exact writes, Point means, interval standard deviations, Point subtraction and the existing population statistic definition. Array length/dimension limits and the fixed registry still apply: there is no square meter or squared temperature unit invented by a reduction.

Raw accessors cannot erase a unit: data.number, data.mean and data.to_tensor reject unit bearing columns; the native numeric_value adapter applies the same protection. The same protection applies when a downstream numerical table adapter would otherwise read raw magnitudes. A table can be projected to genuinely primitive columns before using primitive numerical APIs; converting physical inputs for a model requires an explicit unit aware read and export.

A deliberate example boundary is:

let export_unit = speeds.unit()
let magnitudes = speeds.values()
let model_input = magnitudes.to_tensor(rows: speeds.len(), cols: 1)

The unit array is a fresh converted buffer. .values() returns another fresh primitive Float64 array without the checked physical unit, and the Tensor bridge makes detached numerical storage. Keep export_unit, sensor IDs, row order and target meaning with the consuming model. Exporting magnitudes does not transfer a calibration/evidence model or infer a source aware covariance.

Database connectors continue to return their separate primitive SQL/BSON table representations; they do not infer physical source units. Unit declarations in this CSV workflow are not an automatic label applied to existing database tables or generic raw cell adapters.

Complete research fixture and checks

Download sensor_runs.csv and the research source. All eight sensor records and their quality/group labels are synthetic. The whole source has six present temperatures and two missing temperatures, seven present distances and one missing distance. Six records have quality accepted; five of their temperatures are present and one is missing.

After explicit presence rules and physical thresholds, IDs 0002 through 0005 remain. Their normalized temperatures are approximately [22, 24, 26, 28] degC, distances [1.2, 1.4, 1.6, 1.8] m and elapsed times [4, 5, 4, 3] s.

The source checks these consequences with an explicit Float64 tolerance:

• Temperature mean: 25 degC; population standard deviation: sqrt(5) delta_degC.
• Declared correction: 0.5 delta_degC; corrected mean: 25.5 degC; reference anomalies approximately [-3, -1, 1, 3] delta_degC.
• Per run speeds: approximately [0.3, 0.28, 0.4, 0.6] m/s; their unweighted mean is 0.395 m/s.
• Total traveled distance divided by total elapsed time: 6 m / 16 s = 0.375 m/s. This aggregate has a different weighting interpretation from the mean per run speed.
• Retained IDs/group text, target unit/type metadata, explicit Tensor shape and detached extraction.

The program prints 21 named checks, each followed by the Bool expected to be true. These predicates report the documented fixture results; they do not stop the interpreter like assertions. Observed temperature spread and a chosen correction are not a measurement uncertainty model. Use aner.science and EvidenceTensor when common calibration uncertainty and correlations must be declared.

Native execution, limits and diagnostics

A unit column retains one registered descriptor with native Float64 magnitudes and existing missing flags. Planning retains immutable schema/projection/filter metadata. CSV parsing, batch traversal, aggregation, collection and array conversion remain native operations; no external dataframe service or per row unit String parsing is introduced.

For n selected rows, scalar unit retrieval performs the existing column lookup plus bounded conversion work; full column unit_values performs O(n) conversion and allocates O(n) result storage. unit_mean converts the already aggregated observed mean in bounded work after the projected column lookup. Scanning still checks every source field, including conversions for unit bearing source columns before filtering. The data reference defines managed buffer accounting; unit arrays defines the independently allocated array output.

Existing schema, CSV field, batch, row and memory limits remain in force. max_bytes controls managed data workspace allocations rather than total process RSS. A column array and any numerical/Tensor exports allocate additional buffers with their own array/Tensor limits. Unit conversion follows native Float64 rounding and representable range checks; there is no exact decimal or cross platform bitwise promise.

Missing imports, wrong Aner argument types and a dynamic target for a typed output accessor fail checking. Schema dependent errors use the existing data diagnostic R2601: unsupported units, incompatible target dimensions/kind, wrong threshold kind, implicit raw access, missing cell extraction, malformed source values and exceeded scan/collection limits fail explicitly. A scalar factory used to construct a threshold can fail first with unit diagnostic R3701. Existing contextual schema accessors retain their own documented diagnostics.

Checking a program reads neither CSV fixture. Running resolves paths against the process working directory and executes each summary/collection as a fresh scan. The file can change between those calls; a lazy plan is not a source snapshot. No successful partial report/table is returned on a failure.

Run the examples

Keep the fixtures under the paths used in source, or change those paths deliberately in your local program:

aner check examples/data_units_basic.aner
aner run examples/data_units_basic.aner
aner check examples/data_units_research.aner
aner run examples/data_units_research.aner

The beginner source needs measurements.csv. The research source needs sensor_runs.csv. Both contain complete schema/analysis code and synthetic local inputs with no private endpoints, credentials or service dependency. Native verification and publication status are established separately from documentation source edits.

Bounded inspection and streaming statistics

Use the data execution guide to preview unit columns without a complete materialization, inspect validation receipts, and calculate typed minimum, maximum and population standard deviation. A temperature point retains its point unit for extrema and means; its standard deviation has a difference unit such as delta_degC. Bounded summaries describe only their validated prefix; check complete() before claiming full dataset statistics.