Language architecture

Working design, October 10, 2026.

Implementation language

Use C++ for the first implementation. Compared with C, standard containers, RAII resource management, variants, and typed abstractions reduce manual bookkeeping in parsers, syntax trees, and runtime values. Keep the code simple: explicit ownership, modest templates, compiler warnings, and a small dependency set. C remains appropriate for stable native interfaces later. This is an engineering recommendation, not a claim that C cannot implement Aner.

Aner defines its own source semantics, type system, and execution model. Its native runtime can use existing numerical libraries and compiler infrastructure while preserving those language contracts.

One language with two execution paths

.aner source from any editor
    -> lexer -> parser -> name resolution -> static type checking
    -> checked Aner syntax tree
         |-> interpreter -> Aner runtime -> results                 (current)
         |-> typed IR -> native lowering -> LLVM -> executable      (future)

Currently, aner run interprets checked scalar control flow and dispatches tensor, neural, classical ML, metrics, dataset catalogue, and external CSV operations into native C++ libraries. Later, aner build compiles it ahead of time. Both paths share numeric kernels, shape validation, error conventions, and tests. The native runtime must not depend on the interpreter to execute ordinary compiled instructions.

LLVM's frontend tutorial covers language frontends and object code generation. LLVM ORC supplies JIT infrastructure, but a JIT is optional and is not the first interpreter. Install a standalone LLVM SDK only when the compilation milestone needs it; Apple's installed Clang alone is not proof that the LLVM development libraries are available.

Keep source spans in every frontend representation so errors can identify the actual expression. The first interpreter executes the checked syntax tree, with resolved bindings and expression types populated by the checker. A lower level typed IR can follow when native lowering or numerical operations need it. Avoid designing a large custom IR before real operations need it.

Core collection storage and sorting

Import free List, Dictionary, and Set use the existing tracing object heap. List stores a growable Value array; Dictionary and Set use ordered hash nodes. Class fields and collection contents participate in the same iterative graph tracing, including cycles. Public signatures and runtime dispatch are declared in the native collection registry; the editor's checked signature catalog mirrors that registry and documents time and auxiliary/result space costs.

List.sort() validates an ordering family before mutation: Int64/Float64 together, homogeneous Strings, or homogeneous Bools. Numeric comparisons retain full Int64 precision and the original payload types; NaN and unsupported/incompatible values report collection error R2901. A stable bottom up merge sort works on indices, allocating O(n) auxiliary storage before moving Values into their final order. This gives O(n log n) comparisons without deep String copies. String comparison uses encoded byte lexicographic order and adds prefix length work. There are no custom comparator callbacks or descending parameters. The public rules and runnable examples are in Core collections.

Typed array storage

Array<T>, DynamicArray<T>, and NDArray<T> are implemented import free types, with native contiguous Int64, Float64, one byte Bool, or String descriptor buffers. ObjectRef identity and tracing own their buffers; typed scalar elements contain no nested heap references and do not occupy tagged collection slots. A separate logical 64 MiB budget includes reserved capacity and owned String text. Dtype aware native signatures are shared by checking/runtime and mirrored by editor parity tests.

Fixed logical size and dynamic capacity are separate contracts. Dynamic append growth doubles, while explicit reserve allocates its requested minimum. Exact adjacent inversion counts enable O(1) sorted status queries and O(log n) first match binary search without rescanning the array on each query. Sorting uses in place heapsort with O(1) auxiliary storage and O(n log n) worst case time; it is not stable. Numeric loops check integer overflow and finite floating results. NDArray adds bounded rank 1 to 8 shape/strides, shared fixed reshape, detached dynamic reshape, contiguous transpose/permutation, native broadcasting, and rank two matmul. Buffer owner accounting charges shared reshape storage once and releases it after the final descriptor is collected. Odometer traversal avoids an element by rank broadcast index loop. General strided views, axis reductions and arbitrary ranks remain deferred; the existing Tensor receives detached Float64 copies. See the one dimensional array contract and multidimensional design and costs.

Large datasets are a design requirement

Doctors should be able to ask understandable questions of data without manually coordinating many libraries or learning memory management tricks. Data scientists still need ordinary functions, precise errors, inspectable plans, and extensible computations.

The first external data milestone now loads an explicit CSV schema, selects columns, filters records, computes summaries, and produces a small result while processing native batches. Its exact current contract is in Aner Data. The current built in Dataset is an immutable view of a small numeric teaching dataset, documented in Aner datasets. External ingestion uses the separate aner.data API and its DataSchema, DataScan, DataReport, and DataTable types; it does not extend the teaching Dataset type. DataScan retains a plan, while DataTable retains an explicit collection. A counting native allocator bounds parser, batch, aggregation, and collection capacities, including reallocations; bounded plan metadata and other interpreter values remain outside that budget. Explicit collection also limits selected rows. Sorting and joins will need spill to disk or documented capacity limits; batching alone does not make every algorithm bounded memory.

Design contracts should cover missing values, date/time parsing, patient identifiers, duplicate keys, join cardinality, measurement units, and data quality summaries. For repeated observations, partitioning and summaries may need patient/group or temporal structure. These semantics require domain examples and runtime validation; a language cannot infer the medical meaning of arbitrary columns.

Consider Apache Arrow for future columnar interchange. Its language neutral format describes typed buffers and validity bitmaps. Its native interfaces could support columnar interchange in Aner; a complete database engine would require additional components. Database connectors and Parquet readers remain future integration choices.

Use synthetic fixtures for development. No patient data or database credentials are needed to establish the toolchain or first language examples. The language itself is not a clinical decision system.

Leave room for acceleration

Preserve vector, matrix, and dataset operations until lowering so execution can later select suitable native kernels. Put kernel selection behind a small runtime interface. Do not make a per cell interpreter callback the only representation of an array operation.

The portable CPU path uses cache blocked kernels; eligible macOS builds also use Accelerate. Optional Metal/MPS and CUDA backends now execute resident Float32 rank two Tensor operations and gradients. Device memory, transfer, synchronization, precision, and failure policies are explicit. Select compatible hardware backends and compare matched workloads after profiling. Clusters and supercomputers additionally require partitioning, communication, scheduling, checkpointing, and deployment formats.

Hardware acceleration is not automatic portability. A requested unavailable accelerator raises an error; a program may explicitly select CPU execution instead. Define numerical tolerances and reduction policies before asserting agreement across parallel engines. CPU only builds remain available without accelerator dependencies. Tensor devices and performance defines the current local device contract.

VS Code and other editors

The command line tools are the product interface. An editor saves .aner files and invokes those tools. Use VS Code's C/C++ and CMake Tools extensions to develop Aner itself.

The local extension in editors/vscode associates .aner files with Aner and supplies syntax highlighting, bracket handling, comments, and snippets. Standard syntax scopes let light and dark themes choose their own colors. It is declarative: no extension runtime, language server, or semantic completion is required for these features.

Workspace tasks build Aner, check or run the current file or the hello program, and run tests. A problem matcher sends CLI diagnostics to the Problems panel. A future language server can provide live diagnostics and semantic completion across editors; Aner source debugging is also future work. The VS Code language extension overview distinguishes declarative and programmatic features. Any future Node.js/TypeScript extension tooling remains separate from Aner program execution.

Development stages and completion criteria

Windows, macOS, and Ubuntu are supported platform goals from the start. Use portable paths, standard C++, and CMake; avoid platform specific shell commands in the core. Establish a three platform build/test matrix when a repository host is selected. Native CPU correctness has been exercised on macOS, Ubuntu, and Windows with documented per host scope. Metal/MPS and CUDA execution require separate actual hardware validation and matched measurements. GUI debugging and the future native Aner compiler remain separate work.

Neural network architecture proposal

The neural network execution proposal develops the next numerical stages into typed tensors, automatic differentiation, CPU kernels, Apple/NVIDIA GPU backends, reproducible experiments, and third party packages. It remains a broader design proposal. The first implemented subset is documented in Aner Neural: a built in module, rank two Float64/Float32 tensors, explicit CPU and optional Metal/MPS or CUDA execution, automatic differentiation, and SGD/Adam. Fitted classifier checkpoints and compatible warm starts are implemented; see development and model state. General third party packages, general Tensor/object checkpoints, and distributed execution remain future work. Tensor acceleration can be reached from the interpreter before all scalar code has an ahead of time compiler.

Portable training observation

The aner_viz C++ target observes the existing neural graph through bounded snapshots; the interpreter passes an optional recorder into execution. Recording is off by default. Numeric buffers and gradient state are copied for inspection, never replayed or updated by a renderer. Reports describe reachable executed tensor graphs, not every path through the source program.

aner.viz produces a versioned trace, terminal summary, and a standalone browser report. The HTML template is embedded at build time so copying the native executable does not break report generation. Runtime execution on a headless host uses Aner’s native runtime and needs no browser, HTTP server, or GUI toolkit. Larger tensors retain summaries and larger graphs report explicit omissions. A future hierarchy of model/layer/operation groups can improve those summaries without changing training semantics.

The proposed aner.plot library should share chart specifications and renderers while serving datasets and statistics independently of neural training. Keep source/model definitions authoritative; graphical model editing requires a specified source translation and reviewable diff. See the current capture contract and roadmap.

First shared ML boundary

The current builtin prototype includes aner_ml for immutable KNN/K means fitted state, aner_metrics for detached evaluation, and aner_tensor for neutral creation/inspection. The latter currently delegates to existing aner_nn tensor storage; the public separation precedes an internal backend extraction. The interpreter resolves the two ml.predict signatures by fitted model type before execution. General module loading, user defined fitted types, pipeline protocols, serialization, and algorithm specific packages remain design milestones. The ML architecture distinguishes those proposals from the implemented APIs.

The external data engine uses native column batches with per column validity bytes; missingness is separate from each stored value. Float64 Tensor remains a distinct dense model format with an explicit copy/conversion boundary. User callbacks per CSV cell are not part of the execution path. Future Arrow interchange is a design option, not a current dependency. CSV projection reduces retained columns while the strict full schema validator still parses and checks all source fields. Record/field limits and byte aware flushing supplement row count batch limits.