AdaptiveFrame

AdaptiveFrame is a native immutable view over a collected typed table. It preserves observation identity, declared groups, field roles, ordered feature meaning and fitted preprocessing dependencies through the supported transformations. Import aner.data. The first implementation provides explicit conversion and population contracts; automatic storage adaptation is future work.

The practical starting point is building energy regression: several rows represent each building, preprocessing learns only from training buildings, and predictions retain the original observation IDs. A plain numerical matrix cannot express those population and field contracts by itself.

Construction and field roles

data.frame(table, row_id, group, target, units) adopts an existing immutable DataTable. The identity column is a unique nonmissing Int64, the optional grouping column is a nonmissing String, and the target is a numeric column. Empty identity or grouping names select the documented generated identity or no grouping behavior. Units are explicit declarations in table column order; they do not validate how the observations were measured.

Identity, grouping and target fields are excluded from numerical input features automatically. Rename preserves a field's identity and role. Selecting a target under a new name does not turn it into a training feature. A heterogeneous feature matrix retains an ordered feature schema; its original columns are not assigned one shared physical unit.

Selection and declared populations

Projection, slicing and gathering apply one selection to the values and their row identities. Native snapshots and their context are immutable; copies share retained state. Array position and stable source observation identity remain different concepts.

data.partition_groups accepts exact training, validation and test group lists. The manifest must cover the eligible groups once: missing, duplicated or multiply assigned groups fail. Training must be nonempty and at least one held out population must be declared. Validation or test can be empty; its view reports zero rows, and fitting or numerical packing of an empty view fails. Each partition creates its own trusted experiment scope. Matching group names or identical numeric values do not authorize reuse across a separately created scope.

Training only fitting is enforced at supported model and fitted preprocessing boundaries. Transformation preserves the input population role and retains the learned artifact's fit dependencies. These checks apply to declared, tracked observations; manually reconstructed scalar values cannot recover the original fit authorization.

Explicit numerical batches

data.batch(frame, max_output_bytes, copy) creates a packed native model batch with ordered inputs, target, original row IDs and preserved fit dependencies. Separate table columns require allocation for row major packing. copy: "forbid" therefore fails before packing instead of silently copying. The copy report identifies the packed numerical allocation and preserved context.

Missing required inputs or targets fail in this first adapter. No imputation, row dropping or masked loss rule is inferred. Batch reports distinguish copied_bytes, new_numeric_bytes, shared_row_selection_bytes and budgeted_payload_bytes; descriptor and allocator overhead and process RSS are excluded. Batch shuffle applies one permutation to inputs, targets and IDs together. Raw numerical export is an explicit boundary requiring acknowledgement of context loss; ordinary model input should keep the contextual batch.

Native API signatures

Qualified functions and the corresponding recognized receiver methods use the same native contracts. self denotes the receiver in the qualified signature. Lists of names or units contain Strings; index lists contain Int64 values. Wrong List element types fail instead of coercing values.

Time and space complexity

Let N be source rows, n selected rows, p input features, c source columns, k selected columns and g distinct declared groups. Let h represent the retained dependency descriptors and m gathered rows. String and identifier byte lengths add their own scan or copy cost.

The current frame admits at most 1,000,000 source rows and 64 fields. Numerical packing admits at most 1,000,000 cells and a checked 64 MiB managed payload. Semantic model input facades additionally admit at most 65,536 row coordinates; a large underlying batch does not bypass that semantic limit. Exact Int64 IDs are available through row_id(index) or a bounded managed List export.

The numerical payload budget is not a global process RSS quota. It does not release the source table, fitted artifacts or other live values. Resource bounds and conversion reports must be considered with the existing typed CSV and Tensor limits.

Dependencies, backends and limitations

The implementation is native C++ and needs no external service. CSV loading follows the existing strict schema and missing token rules. The complete example uses CPU Float64 batch preparation and regression; a compatible development engine and VS Code helper 0.1.38 or newer are required. A normal CPU build does not require accelerator libraries.

Automatic dense/sparse layout selection, joins, sparse storage, out of core spilling, streaming windows and graph views remain future functionality. Existing accelerator and parallel APIs do not make this frame's transformations distributed automatically. See FeatureLedger for fitting, artifact persistence and guarantee boundaries.