Bounded data execution

aner.data can limit source work, retain a small prefix preview, compute richer statistics in native batches and record what one execution actually validated. An execution controller provides a deadline and cooperative cancellation. These are APIs within aner.data, rather than a separate module or an asynchronous language runtime.

These features require the updated native development runtime built from this source and matching editor support. Earlier published packages may reject the new calls. This guide does not announce a released binary or report a new timing benchmark. Follow installation, then use the complete local fixtures and sources linked below.

Preview a bounded prefix first

A row limited result is not necessarily a bounded search: a rare filter can require many rejected records before finding a match. Set an explicit source bound as well as the desired output size.

The eight row fixture has its first rare observation at source row seven. The following complete program permits four validated source rows and at most two returned matches:

import aner.data
import aner.units

var schema = data.schema()
schema = data.column(schema, "row_id", "String", false)
schema = schema.unit_column("temperature_f", "degF", true)
schema = data.column(schema, "signal", "Float64", true)
schema = data.column(schema, "cohort", "String", false)
let source = data.scan_csv("examples/fixtures/data_execution/observations.csv", schema, "NA")
    .convert_unit("temperature_f", "degC")
let controller = data.execution(timeout_ms: 30000)
let rare = data.where_eq(source, "cohort", "rare")
let bounded = data.bounded(data.with_execution(rare, controller),
    max_scanned_rows: 4, max_read_bytes: 0, timeout_ms: 30000)
let preview = data.preview(bounded, max_rows: 2, max_bytes: 1048576, batch_rows: 2)
print(data.rows(preview))
print(data.scanned_rows(preview))
print(data.complete(preview))
print(data.stop_reason(preview))
print(data.returned_rows(preview))

Run from the directory containing the fixture. Expected output is 0, 4, false, scan_rows, 0. No rare record occurs in that prefix. This does not establish that the complete source contains no rare record. preview validates all fields of each included source record, but does not validate the unvisited tail.

A zero output limit (max_rows: 0) performs no file I/O and returns an empty, incomplete preview; it does not validate the header or source.

data.preview stops after the requested number of matching records without probing the tail for EOF. Even when that number happens to equal the entire file's record count, the output row stop is an incomplete preview. Use a complete execution receipt when whole source validation is required.

Source limits and execution controllers

Timeout, scanned record and read byte bounds accept nonnegative Int64 values. There is no smaller additional maximum for those bounds. The effective deadline is the minimum positive deadline of the scan and controller. A controller can be reused sequentially if it has not been cancelled; the latest run replaces its receipt. Concurrent runs sharing one controller are rejected by the single flight contract.

Argument/configuration failures that occur before run admission leave the controller's previous receipt unchanged. Rejected concurrent reuse likewise does not overwrite the active run's receipt. A prior receipt therefore describes its previously admitted run, rather than a new validation failure.

A preview whose scan has never been explicitly configured with bounded uses safe implicit work limits: 100,000 source rows, 16,777,216 input bytes and 1,000 milliseconds. Calling bounded replaces that policy with the explicit settings, including zero settings. Thus bounded(scan, 0, 0, 0) deliberately removes those work bounds and can search the complete source before finding a rare match. The examples always choose a positive source record bound and an explicit deadline.

The implicit defaults can also be read as exact configuration values:

Existing output/workspace limits remain: max_rows is from zero through 1,000,000, max_bytes is from 1 through 1,073,741,824 bytes and batch_rows is from 1 through 65,536. A small output cap does not override a source bound, a workspace quota or full validation of each record that is visited.

Read the receipt before interpreting the result

data.receipt(value) returns a JSON String describing one execution. Report/table receipts are retained snapshots. A controller exposes the latest run or its unused/cancelled state. The getters below accept a DataExecution, DataReport or DataTable:

The new getter arguments use label value, except existing scanned_rows and peak_bytes, which retain the first argument label report for compatibility. Positional calls accept the newly supported handle families. Existing data.rows(report) equals its selected aggregate count; data.rows(table) equals its returned row count.

Receipt stop reasons include eof, preview_rows, scan_rows, read_bytes, timeout, cancelled, error and the unused controller's not_started. Operation and status accompany that reason; status can be idle, running, completed, stopped, failed or cancelled. Native raw cell adapters use the separate operation from_cells and reason native_cells: those complete adapter tables have no CSV header validation or CSV read bytes, so their receipt is not evidence that a file was scanned.

header_validated records whether the CSV header was checked. Read ahead can make read_bytes larger than the bytes in the validated prefix, even as large as a small file's entire byte size. It never establishes validation of that tail. A work bound can stop midway through a record; that incomplete record contributes no scanned/selected count or statistics.

A receipt records the work performed under that run's limits. It does not turn a lazy path into an immutable source snapshot, infer experimental roles or provide a complete scientific provenance graph. Every summary/preview/collection reopens the source. Retain an appropriate source identity and application policy with research results.

Prefix statistics and full source statistics

The twelve row research fixture has rare records only at rows ten and eleven. Its source schema retains text row IDs and sites, Fahrenheit temperature Points, centimeter distance Quantities, second elapsed time Quantities and a nullable primitive signal.

The complete research program runs three different prefix operations before the full summary:

• A rare filter preview with a six source row cap returns zero rows, complete=false and stop_reason=scan_rows.
• A six row summary includes six selected records, with five present/one missing temperature and signal. Its observed temperature mean is 24 degC and population spread is sqrt(8) delta_degC. It remains explicitly incomplete.
• A routine preview with a two output row cap returns IDs 0001, 0002, validates only those two source rows and reports preview_rows, rather than claiming EOF.

A generous 100 source row bound then allows the full twelve row source to reach EOF. The full temperature column has ten present/two missing observations, with values approximately 20 through 38 degC in steps of two. Its mean is 29 degC and population spread is sqrt(33) delta_degC. The primitive signal has ten present/two missing values: 1, 2, 4, 5, 6, 7, 9, 10, 11 and 12. Its mean is 6.7 and population spread is sqrt(12.81).

A final full rare filter summary validates all twelve source records and selects the two late records. Their temperature mean is 35 degC and population spread is 1 delta_degC. The empty prefix did not establish their absence. Earlier table/report receipts remain unchanged as the same controller is reused for these later runs.

Richer streaming summary API

Statistics operate on present selected values of projected columns. Missing counts remain explicit; no missing observation becomes zero. Every visited source field is still validated before filters/projection, including excluded fields and unit normalization.

The existing mean, unit_mean, valid and missing calls retain their established contracts. Min/max/std/mean fail when no observed value is available. Population spread divides by the number of present observations; a singleton has zero spread. There is no sample statistic or missing value imputation parameter. Primitive numerical summaries are Float64, including Int64 extrema, so integers beyond exact Float64 precision can round in these summaries while original Int64 table storage remains exact.

For a temperature Point, unit_min and unit_max use Point targets such as degC. unit_std requires an interval target such as delta_degC, never degC; offsets do not shift a spread. A length Quantity can use m for its standard deviation. This does not require inventing squared units for variance. Primitive min, max, std and mean reject unit columns rather than silently discarding their physical unit.

Typed unit targets must be literal Strings because checking fixes the return type. No arbitrary unit expressions or new generic call syntax are introduced. See CSV units, scalar units and packed arrays for source interpretation, kind compatibility, deliberate numerical export and the fixed registry.

Observed spread is not measurement uncertainty. A standard deviation does not declare independent noise, shared calibration errors, covariance or a confidence distribution. Use aner.science and EvidenceTensor when the source relationships form part of the scientific model.

Cancellation without pretending source code is asynchronous

Aner source calls remain synchronous. It can construct and pre cancel a controller without starting a throwing terminal call:

import aner.data
let controller: DataExecution = data.execution(timeout_ms: 0)
let alias = controller
data.cancel(controller)
print(data.is_cancelled(controller))
print(data.is_cancelled(alias))
print(data.scanned_rows(controller))
print(data.returned_rows(controller))

Expected output is true, true, 0, 0. Aliases share one sticky cancellation request. A cancelled controller cannot be reset for another run; create a new controller. Attempting a terminal operation through a cancelled controller fails and returns no partial table/report.

After a terminal result commits, a later cancellation request preserves that completed/stopped/failed receipt and any returned immutable handle. is_cancelled still becomes true; the next attempt fails before I/O and records the cancellation.

Native hosts can request cancellation while an attached controller is running. The CSV engine checks cooperatively at native parsing/work checkpoints. A deadline is also a cooperative stop condition; it does not forcibly preempt blocked input calls, and existing native file reads can block between checkpoints. This is not a hard real time interrupt or an asynchronous language callback facility.

Cancellation and malformed input/numerical/allocation failures do not publish partial results. Inspect the controller receipt after a native/host failure when no result handle exists. A normal configured work bound stop can publish an explicitly incomplete preview or summary; strict collection instead rejects incomplete execution. Distinguish those outcomes before consuming any statistics.

Native host cancellation example

A C++20 host can run a synchronous CSV operation on its own worker thread and cancel through the shared native controller. The fictional local file below is a one column CSV headed signal, with finite numbers and optional unquoted NA fields. Replace that path with an appropriate local source and retain its schema.

#include "aner/data.hpp"
#include <atomic>
#include <exception>
#include <iostream>
#include <thread>

int main() {
    namespace data = aner::data;
    auto schema = data::column(data::schema(), "signal", "Float64", true);
    auto source = data::scan_csv("local-signals.csv", schema, "NA");
    auto controller = data::execution(30000);
    auto plan = data::with_execution(data::bounded(source, 0, 0, 30000), controller);
    std::atomic<bool> finished = false;

    std::jthread worker([&] {
        try {
            auto report = data::summarize(plan, 1048576, 1024);
            std::cout << data::receipt(report) << '\n';
        } catch (const std::exception&) {
            std::cerr << "CSV worker returned no report; inspect its controller receipt\n";
        }
        finished.store(true, std::memory_order_release);
    });

    // A real host can place this request in its UI/action handler.
    // Here the host observes admission before requesting in-flight cancellation.
    while (!finished.load(std::memory_order_acquire)) {
        if (data::receipt_info(controller).status == "running") {
            data::cancel(controller);
            break;
        }
        std::this_thread::yield();
    }
    worker.join();
    std::cout << data::receipt(controller) << '\n';
}

This uses the native header/library API, not Aner language callbacks. A short source can commit before the host observes running, or between observing it and requesting cancellation. In that case the committed report and receipt remain valid; a late request affects the next attempt. A cancellation observed before commit returns no report and leaves its receipt on the controller. No timing or cancellation response bound is fabricated by this example.

Native memory, performance boundaries and diagnostics

Preview retains only bounded selected rows; streaming summaries maintain native per column aggregates across bounded parser batches. They do not materialize the whole selected table. State still includes CSV parsing buffers, projected batches, aggregates and any retained preview rows. Work is proportional to the visited source records/fields and their parsing/conversion cost, not merely the returned row count.

max_bytes and receipt peak_bytes describe managed PMR data workspace allocations. They include the same parser/batch/aggregation/retained table accounting as the data module memory contract. They exclude total process RSS, allocator/operating system overhead, other Aner values and returned receipt text. Source read byte limits bound counted input work separately from managed memory; byte counts include header/read ahead and can stop inside a record. Array/Tensor exports have their separate allocation limits.

Rich summary arithmetic remains finite native Float64 computation with its documented representable range and rounding limitations. Receipts expose elapsed observations, not throughput benchmarks, exact timing reproducibility or a deadline response guarantee. Streaming aggregation does not add database streams, automatic out of core model fitting, user batch callbacks, general aggregate expressions, joins or approximate quantiles.

Wrong argument types or missing imports fail during checking. Invalid bounds, incompatible statistic/unit requests, malformed visited input, cancellation, forbidden concurrent controller reuse, strict collection work limit stops and workspace failures use the existing located R2601 data diagnostic. A scalar unit factory evaluated before a data call can retain its separate R3701 diagnostic. Runtime failures preserve no usable partial result handle; source paths are still resolved at execution time.

Run the complete examples

Both sources and fixtures are fake local experiments with no private server, credentials or service dependency:

• Execution basics with observations.csv: eight source records and 13 named expected true predicates.
• Prefix versus full research with sensor_trials.csv: twelve source records and 23 named expected true predicates.

aner check examples/data_execution_basic.aner
aner run examples/data_execution_basic.aner
aner check examples/data_execution_research.aner
aner run examples/data_execution_research.aner

Keep fixtures at the relative paths used in source or change those paths deliberately. The source predicates use explicit tolerances for floating statistics and check counts, completeness, stop reasons, metadata and receipt snapshots. They report Bool outcomes rather than stopping like assertions. Timing and byte read ahead are checked only through their accounting contracts, not against fabricated measurements. Native verification and website publication are established separately from these source edits.