Filtering and derived data

aner.data builds native conditions from reusable column references. Apply a condition with .filter(...), choose columns with .select(List(names)), and execute with the existing bounded operations. Scans stay lazy, tables share immutable snapshots, and AdaptiveFrame filtering retains observation identity and scientific context. No per row Aner callback is involved.

Run the complete example from the project root with its included seven row synthetic fixture:

aner check examples/data_filtering.aner
aner run examples/data_filtering.aner

The example selects two observations with exact IDs 9007199254740993 and 9007199254740997, keeps the source's seven rows intact, and reports a selected mean temperature of 23.0 degrees Celsius. It also demonstrates missing value behavior and bounded preview. See the data reference, execution controls and automatic disk storage for terminal operations and resource accounting.

Build a condition once

let age = data.col("age")
let population = age.between(18, 65) &&
    (data.col("cohort").is_in(List("A", "B")) || (data.col("active") == true))
let improving = data.col("score") > data.col("reference")
let eligible = population && improving
let result = source.filter(eligible)
    .select(List("row_id", "score", "temperature"))
    .collect(max_rows: 0, max_bytes: 1048576, batch_rows: 256)

data.col(name: String) returns an opaque immutable DataColumn. Comparisons return DataCondition; bind either in variables or pass them to functions. Conditions do not capture a source or read rows. Applying one resolves its column names and validates compatible types and units against that source. The same condition can be reused against compatible scans, table snapshots or frames.

Use parentheses around comparisons when composing conditions so the intended grouping is clear. Boolean scalar logic keeps its existing behavior; a DataCondition cannot be used directly as the condition of if or while. .filter(a).filter(b) adds the same requirement as .filter(a && b).

Types, precision and physical meaning

Int64 and Float64 comparisons accept mixed numeric operands and compare their represented values exactly. Large integer identifiers are never rounded to Float64 to evaluate a predicate. Float64 still represents a binary floating point value; a comparison does not turn it into decimal arithmetic. Strings are compared as text and are never parsed as numbers. Bool supports equality, inequality and membership with Bool operands; Bool ordering is rejected. String ordering follows the stored text byte order, with no locale or normalization applied. Other incompatible type pairs fail when bound to a source.

Physical columns require compatible unit bearing operands. A Point temperature can be compared with 21.0@degC, or with another compatible Point column in Fahrenheit or Kelvin. A Quantity distance can be compared with 1.0@m or a compatible distance column. Raw numeric thresholds do not implicitly acquire the column's unit. Point and interval quantities remain distinct. Comparisons use the shared scalar unit rules, including exact Float64 boundaries, with no added tolerance:

let warm = data.col("temperature") >= 21.0@degC
let traveled = data.col("distance_cm") >= data.col("reference_m")
let qualified = source.filter(warm && traveled)

DateTime operands compare instants at exact UTC nanosecond precision. Fixed offsets that describe the same instant compare equal. A String that resembles a timestamp remains a String; provide a DateTime scalar or another DateTime column. The column representation remains bounded by signed Int64 UTC nanoseconds, approximately 1677 to 2262. Named timezones, daylight saving resolution and naive local time guesses are not added by filtering; see timestamp ingestion rules.

Range endpoints and membership elements must be compatible with the column, including its physical kind and units. Nonfinite numeric literals are rejected. No implicit String to number conversion is performed. A reversed range has no matching present values. An empty membership set is false for every present value and remains unknown for a missing value. Membership sets are sorted and deduplicated once when bound, then searched natively.

Missing values use three valued logic

A missing value compared with a scalar or another column produces unknown, including !=. Ranges and membership also produce unknown for a missing value. Presence checks produce a definite true or false. Filtering retains only true rows.

AND and OR are symmetric. NOT maps true to false, false to true and unknown to unknown. Consequently, !(data.col("cohort") == "A") excludes a missing cohort. To include it, say so:

let other_or_unknown = data.col("cohort").is_missing() ||
    !(data.col("cohort") == "A")
let complete = source.filter(data.col("score").is_present())

fill_missing and unit conversion remain declarative normalization steps executed before predicates, even if declared later in a chain. Filling does not relax a source's nonnullable schema or repair malformed present values. drop_missing() requires all fields projected when it is called to be present. Nullable metadata continues to describe the source declaration after a filter selects complete rows.

Derive a scan, table or frame

scan.select(columns: List) replaces the output projection with a nonempty list of unique existing names, in exactly the requested order. A condition can refer to source schema columns outside that projection. Filtering preserves source row order and neither deduplicates nor sorts rows.

table.filter(condition) and table.select(fields) return lazy DataScan values over the immutable table snapshot. They share its lifetime and any private disk backing; they do not reopen a CSV or eagerly copy values. Execute with collect, preview, summarize, write_csv or the native batch consumer. The derived output has its own receipt and provenance, linked to the logical parent snapshot. Scan recipes use aner.data.recipe.v4; condition recipes use aner.data.condition.v1, including the typed literals, Boolean structure and kleene-keep-true missing policy. The original table remains unchanged.

frame.filter(condition) returns an AdaptiveFrame view over the current frame's observations. It preserves the selected observation IDs, their order, population role, trusted experiment scope, field roles and fitted preprocessing dependencies. It resolves currently visible renamed/projected fields and current transformed values. Empty selections remain inspectable; fitting and numerical packing still reject empty populations. Native physical table units are adopted by frames, while standardized fields have unit "1". It does not create a new root experiment from scalar values. See AdaptiveFrame selection contracts.

Execution and limits

Condition construction and binding do not read a CSV. The engine compiles bound native comparisons once and evaluates them in the existing batched execution path. Row payloads remain subject to managed memory, batch, work, timeout, cancellation and disk controls. Collecting a result may spill transparently; filtering does not require materializing the full source in RAM. A single row still has to fit the execution workspace, and numerical exports retain their separate output budgets.

Every consumed source field is validated before row selection, including unprojected fields and rows rejected by a condition. Unit conversion and physical comparison errors cannot be hidden behind a false AND branch or a true OR branch. Only a complete execution validates the whole source. Bounded preview validates its consumed prefix and reports why it stopped. CSV plans reopen their source on each execution; use a table snapshot when the same immutable source is required.

This API replaces the former individual threshold/string/presence filters and projection append operation. No compatibility wrappers or deprecated aliases remain. The current scope includes scalar and column comparisons, ranges, finite membership and missing checks. It does not add arbitrary row functions, lambdas, arithmetic column expressions, joins, grouping, sorting, an index alignment system or a distributed query engine. Frame context capabilities and numerical output admission remain explicit. Frame roots retain their current 1,000,000-row/64-field adapter limits and 64 MiB scratch ceiling; those limits do not constrain the streaming DataScan/DataTable engine. Primitive Int64 frame predicates remain exact; a nontrivial physical unit annotation on a primitive Int64 field additionally requires an exact Float64 magnitude within ±2^53.