A traceable data science workflow

A useful profile answers three questions together: what the observations contain, which observations contributed, and how the result was produced. Aner connects a native DataReport to the scan recipe, execution receipt, physical schema and plot metadata. The profiling guide and runnable synthetic example show the implemented APIs. This is a workflow evaluation, not evidence that one language always gives better science or faster execution.

From an unfamiliar source to a result

1. Keep the acquisition evidence. Retain an immutable source copy, its license and acquisition time, and any SQL query, spreadsheet worksheet or upstream export manifest. infer_csv recommends a schema; review identifier columns, nullability, units and timestamp interpretation before accepting it. A CSV digest identifies bytes, not the process that generated them. origin_complete covers source ancestry known to Aner; it cannot discover records an upstream query/export omitted before the file was supplied. Schema inference is its own inspection pass and is not made free by fused profiling.
2. Inspect coverage before interpreting statistics. Use a bounded preview for quick feedback, then inspect complete, origin_complete, stop_reason, scanned and selected row counts. A prefix may miss a late category, malformed value or rare event. Full collection validates every source field, including fields excluded by projection and rows excluded by predicates. A successful prefix validates only the consumed portion. Completely traversing a table made from that prefix yields complete == true for the current snapshot but keeps origin_complete == false; it does not turn an unread acquisition tail into validated evidence.
3. Inspect basic information, then choose deeper diagnostics deliberately. Ordinary retained tables already provide a cached structured overview() report and its summary() text, with no hidden traversal. Basic state is O(columns) and adds modest arithmetic during loading; samples and category dictionaries are opt in. The actual NOAA climate workflow demonstrates this decision with pinned government observations. Choose the profile population deliberately. Enable with_profile on the plan that will be collected. Its report describes the returned projection after normalization and filtering. To investigate raw missingness, profile the raw plan before fill_missing or drop_missing; a cleaned profile cannot reconstruct erased missingness. Keep both recipes when comparing raw and cleaned populations. Multiple terminals on a CSV plan reopen its path; use an immutable collected table for repeated analysis of the same snapshot.
4. Use the profile to ask questions. Check missingness, ranges, exact integer extrema, category coverage and timestamp bounds. Inspect physical units instead of comparing bare numeric magnitudes. A large range or a rare category is a review signal, not an automatic error. Sampling limits and category overflow remain visible. Do not interpret an approximate tail quantile as a safety guarantee or a sampled histogram as a count of the full population.
5. Declare cleaning and selection. Reusable native conditions, explicit fill literals and explicit unit conversion make the plan inspectable. Keep an observation identifier rather than treating row position as scientific identity. Table derived scans leave the source snapshot unchanged; AdaptiveFrame views additionally retain observation identity, roles and preprocessing dependencies. Preserve the decision rationale separately: a recipe records what happened, not why a scientist judged it appropriate.
6. Separate exploration from held out evaluation. Split observations by the appropriate experimental unit before learning imputation, scaling, feature selection or model decisions. A profile of held out observations can influence those decisions even when it uses no target labels. Use training only profiles for those choices. AdaptiveFrame's trusted split and fitted preprocessing context help enforce supported operations, but creating a DataReport does not establish a training scope or authorize a new analysis.
7. Plot the same evidence. Pass the report or immutable table directly to aner.plot by column name. Unit labels, missing pair handling, sampling/coverage disclosures and source lineage travel with the native chart. Preserve the chart provenance with the rendered figure. A first rows scatter is an ordered prefix, not a random sample; an irregular time series can make such a view unrepresentative. Independently retained per column profile samples are not aligned pairs and must not be zipped for correlation.
8. Archive a replayable analysis. Keep the Aner source, runtime/build identity, schema, source copy and hash, recipes, receipts, report JSON, exported figures and chart provenance. Record model seeds, split rules and fitted preprocessing state when relevant. Re run from a fresh process and compare logical hashes, selected IDs and deterministic statistics. Timing and peak memory are observations of an execution environment and need not match bit for bit.

What improves in the native workflow

The distinction is the integration of execution evidence with scientific meaning. The same bounded native execution can validate data, retain a spillable immutable table and accumulate its profile. A cached report then serves diagnostics and charts without reopening the source or traversing the retained table again. Units distinguish a temperature point from a temperature interval, and charts consume the same named physical fields as data operations. Receipts distinguish current terminal completeness from original source ancestry completeness, so profiling every row of a truncated snapshot cannot silently present the original acquisition as fully validated.

The following comparison concerns built in behavior documented by the respective projects, reviewed on October 23, 2026. It does not include third party pandas profilers, unit libraries, provenance systems or alternative execution engines. Those can provide valuable capabilities, and pandas has a substantially broader mature transformation ecosystem.

Pandas generally computes the requested diagnostics when called. Its current numeric describe implementation invokes column count, mean, standard deviation, extrema and quantiles on the loaded values; these are not a profile automatically stored by CSV ingestion. pandas implementation DataFrame.info combines shape/type metadata with optional non null counts and memory estimates. Counts can require a data scan; show_counts and deep memory introspection change the work performed. Fast output alone therefore does not establish load time profiling. pandas info

Calling describe after pandas loads a DataFrame normally scans data already in memory; it does not inherently reopen the CSV. Aner's fused path saves a separate traversal/aggregation of the retained table, and can avoid a second source read compared with executing two separate CSV scan terminals. It still performs additional arithmetic, sample maintenance, category lookup and allocation. No zero cost or general speedup claim follows from using native code.

Verify performance without weakening integrity

Compare three separate workloads: plain collection, collection with an attached profile, and plain collection followed by a standalone profile of the same table snapshot. Hold schema, selected population, batch size, memory/disk policy, sample/category bounds, build mode and source bytes constant. Verify matching counts, extrema, profile options and population identity before comparing performance. Use a complete execution and distinguish logical table reads from CSV reads.

Measure multiple repetitions in alternating order; report median and range, source bytes read, elapsed time, managed peak bytes, spill bytes, compiler/build identity, machine and storage/cache conditions. Reusing an operating system cache can reduce wall time without removing a logical pass. Profiling may cause an otherwise resident table to spill under the same memory budget. Treat profile access, JSON serialization and chart rendering as distinct costs. A cached accessor performing no I/O does not mean rendering a chart costs nothing.

The profiling guide documents the actual resource contract. Any benchmark evidence recorded with this change is scoped to its tested workload and machine. Do not infer pandas performance from a benchmark that runs only Aner.

Remaining research responsibilities

A profile is descriptive evidence. It does not establish independent observations, adequate sample size, missing at random assumptions, unbiased sampling, absence of leakage, statistical significance or clinical validity. It does not detect duplicate observations merely because identifiers look repeated, infer all valid domain ranges, or prove that an acquisition source is trustworthy. Missing counts do not distinguish nonresponse from sensor outage unless that meaning is represented in the data.

Deterministic sampled quantiles have no advertised distribution free error bound. A complete bounded category dictionary supports exact counts; an overflowed dictionary is not an estimate of all categories or of a global most frequent category. Timestamps describe UTC instants after ingestion; normalization does not preserve every original timezone spelling or infer daylight saving intent. CSV snapshot hashes detect changed bytes, but a hash plus a recipe cannot reproduce a deleted source by itself.

Current profiling operates on scans and tables; it does not directly profile an AdaptiveFrame's transformed/contextual view. AdaptiveFrame retains its separate 1,000,000-row/64-field adapter limits, and current model inputs retain explicit in memory export requirements. A spillable source table therefore does not imply an external memory model pipeline.

Aner's current advantages here are native consistency, bounded execution and explicit scientific evidence. Broader joins/grouping, exact external memory quantiles, general correlations, duplicate diagnostics, richer temporal/categorical charts, distributed processing and automatic incremental model training remain separate work. The workflow should be judged by reproducible, scoped outcomes and by the decisions it makes easier to audit.

The structured profile also records the native runtime version/channel, development marker, compiler, CPU target/profile, build type, and a SHA-256 fingerprint of the profiler implementation sources (including uncommitted development content). This helps identify the code that produced a report; it is not a complete dependency lock, signed attestation, or substitute for archiving the analysis program and build configuration. Finished reports keep frozen lineage metadata and do not retain ancestor table payloads solely to describe their source.

See the Native data processing module reference for the shared API.