===================================== Format Evolution and Design Rationale ===================================== Purpose ------- This document records the design decisions behind the ``con`` file format version 2, the alternatives considered, and the reasoning for each choice. It serves as an institutional memory for contributors and a reference for implementers in other languages. Version 1 to version 2 feature matrix ------------------------------------- .. table:: +----------------------------+-------------------+------------------------------------------+ | Feature | v1 (original eOn) | v2 (readcon-core 0.6.0+) | +============================+===================+==========================================+ | Machine-readable metadata | No | JSON object on line 2 | +----------------------------+-------------------+------------------------------------------+ | Spec version in file | No | ``con_spec_version`` key | +----------------------------+-------------------+------------------------------------------+ | Column 5 semantics | Undefined | ``atom_id`` (pre-grouping index) | +----------------------------+-------------------+------------------------------------------+ | Per-direction constraints | No (single flag) | Bitmask column 4 (0-7) | +----------------------------+-------------------+------------------------------------------+ | Declared data sections | No | ``sections`` JSON key | +----------------------------+-------------------+------------------------------------------+ | Force data | No | ``Forces of Component`` blocks | +----------------------------+-------------------+------------------------------------------+ | Per-frame energy/potential | No | ``energy``, ``potential`` metadata keys | +----------------------------+-------------------+------------------------------------------+ | Convergence tracking | No | ``convergence_fmax``, ``converged`` keys | +----------------------------+-------------------+------------------------------------------+ | Unit declaration | No | ``units`` metadata object | +----------------------------+-------------------+------------------------------------------+ | Trajectory ordering | No | ``frame_index``, ``time`` keys | +----------------------------+-------------------+------------------------------------------+ | NEB image identity | No | ``neb_bead``, ``neb_band`` keys | +----------------------------+-------------------+------------------------------------------+ | Compression | No | Transparent gzip (.con.gz) | +----------------------------+-------------------+------------------------------------------+ Why JSON on line 2 (not elsewhere) ---------------------------------- Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **New header line**: adding a 10th header line breaks every existing parser that expects exactly 9 lines before atom data. 2. **Line 5 or 6 (postbox)**: these carry opaque simulation state in some eOn files ("0 0", "218 0 1"). Overwriting them risks losing data in round-trips through tools that preserve those values. 3. **Line 1**: the generator comment ("Generated by eOn") is the most visible line. Tools and humans use it to identify the file origin. 4. **Separate sidecar file**: a ``.con.meta`` JSON file avoids changing the format but introduces file-pairing problems (lost sidecar, out-of-sync data). Decision ~~~~~~~~ Line 2 was historically "Time" or empty in eOn files. No tool assigns it semantic meaning. The Python writer in eOn emits an empty string. The C++ writer round-trips whatever was there. Placing JSON on line 2 is invisible to old readers (they just see a different comment string) and detectable by new readers (starts with ``{``). Why bitmask for constraints (not 3 separate columns) ---------------------------------------------------- Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **Three separate 0/1 columns**: changes atom lines from 5 columns to 7, breaking all existing parsers. 2. **JSON metadata per-atom**: storing ``fixed_directions`` as a list of 3-tuples in the metadata line would work but makes per-atom access expensive and splits constraint info across two locations. 3. **Separate constraint section**: a "Constraints of Component" section adds complexity without benefit -- the constraint is a property of the atom, not a separate dataset. Decision ~~~~~~~~ A 3-bit bitmask in the existing column 4 preserves the 5-column line format. Spec-1 files used 1 for fully fixed; spec-2 files use 1 for x-only and 7 for all-fixed. The decoder takes ``spec_version`` so a legacy file and a spec-2 write of an x-only atom do not collide. Old readers that check ``!`` 0= for "is fixed" will treat any non-zero bitmask value as fixed, which is a safe degradation. Why JSON-declared sections (not positional) ------------------------------------------- Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **Fixed order**: coordinates, then velocities, then forces -- always in that order, detected by blank separators. Simple but rigid: new section types require all implementations to update their order. 2. **Tagged sections**: each section starts with a line like ``#SECTION:forces``. Adds a new syntactic element to the format. 3. **File extension encoding**: ``.convelforce`` for velocity+force files. Combinatorial explosion of extensions. Decision ~~~~~~~~ The ``sections`` key in the JSON metadata declares which sections exist and their order. The parser reads exactly those sections. Benefits: - New section types require no format-level changes -- just add a string to the array. - Section order is explicit, not implicit. - Legacy files without ``sections`` fall back to blank-separator velocity detection (backward compatible). - The writer auto-populates ``sections`` from the frame data, so users never need to set it manually. The optional ``validate=true`` metadata flag lets producers ask readers to verify that declared sections match coordinate blocks exactly: component symbols, labels, fixed masks, atom ids, numeric finiteness, metadata schema, and physical header invariants must all agree. This keeps the default reader permissive for existing files while giving v2 producers a strict interoperability contract. Why per-frame energy in metadata (not per-atom section) ------------------------------------------------------- Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **Per-atom energy section**: a "Energies of Component" section with one scalar per atom. Useful for ML potentials that provide local energy decomposition. 2. **Both**: per-frame in metadata, optional per-atom section. Decision ~~~~~~~~ Most potentials (EMT, EAM, DFT) produce a total energy, not per-atom decomposition. Storing per-frame energy in the JSON metadata (``energy`` key) is sufficient for the primary use case. Per-atom energies can be added as a future section type if ML potential adoption demands it. The ``potential`` metadata key provides structured provenance: ``{"type":"EMT","params":{"cutoff":6.0}}``. This makes energy and force values interpretable without external context. Why gzip (not zstd, lz4, or bzip2) ---------------------------------- Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **zstd**: better compression ratio and speed, but less ubiquitous. Not available in Python stdlib. Would require an additional dependency for every implementation. 2. **lz4**: fastest decompression, weakest compression. Not in Python stdlib. 3. **bzip2**: best compression ratio, slowest. In Python stdlib but rarely used for scientific data. Decision ~~~~~~~~ Gzip is available everywhere: Python stdlib, Rust flate2, C zlib, Fortran, Julia. Every Unix system has ``gzip`` and ``zcat``. The magic bytes (``0x1f 0x8b``) are universally recognized. For the ``con`` format's typical file sizes (KiB to low MiB), gzip's compression ratio is adequate (60-80% reduction). zstd support is available behind the optional ``zstd`` Cargo feature since v0.10.0, using the same magic-byte detection pattern (``28 b5 2f fd`` for zstd frames; ``1f 8b`` for gzip). ``.con.zst`` files are read and written through ``ConFrameWriter::from_path_zstd`` and the same ``read_all_frames`` entry point. Builds without the feature still detect zstd magic bytes and return a clear error pointing at the feature flag, so consumers never see a corrupt parse on a zstd file. Migration guide for existing tools ---------------------------------- Reading v2 files in a v1 reader ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A v1 reader will: - Parse line 2 as a comment string (harmless). - Parse column 4 values: 0 and 7 work (free and fixed). Values 2-6 are treated as "fixed" by any ``!`` 0= check. Value 1 still works. - Ignore the ``sections`` key (no JSON parsing). - Stop at the first blank line (velocity section), missing any force section that follows. Degradation is safe: the reader gets coordinates and constraints (with reduced per-direction granularity) but misses forces. Upgrading a v1 writer to v2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. Emit ``{"con_spec_version":2}`` on line 2. 2. Emit 7 (not 1) for fully-fixed atoms. 3. Store and preserve ``atom_id`` (column 5) through read-write cycles. 4. If writing forces, add ``,"sections":["forces"]`` to the JSON and append a force section after coordinates. Reference implementations ~~~~~~~~~~~~~~~~~~~~~~~~~ .. table:: +----------------+--------------------------------------------------+-------+----------+ | Implementation | Source | Lines | Language | +================+==================================================+=======+==========+ | eOn C++ | ``addl/referenceImpls/eon_cpp/`` (archived) | 592 | C++ | +----------------+--------------------------------------------------+-------+----------+ | eOn Python | ``pip install eon`` / eOn repo ``eon/fileio.py`` | 760 | Python | +----------------+--------------------------------------------------+-------+----------+ | ASE | ``pip install ase`` / ``ase.io.eon`` | 307 | Python | +----------------+--------------------------------------------------+-------+----------+ The C++ implementation is archived (March 2026 snapshot) because it requires the full eOn build system. The Python implementations are installable packages and do not need archiving. All reference implementations support v1 only (no JSON metadata, no sections, no bitmask constraints). Version 2 to version 3 ---------------------- What changed ~~~~~~~~~~~~ Version 3 keeps the nine-line header and type-grouped coordinates. Writers prefer emitting ``units`` inside the line-2 JSON object so quantities (length, energy, force, …) are not under-specified. Readers that implement v3 accept v2 files; strict mode rejects unknown ``con_spec_version`` values and missing v3 ``units`` when required by the library's write policy. Alternatives considered ~~~~~~~~~~~~~~~~~~~~~~~ 1. **Bump the entire grammar** (new extensions, binary framing): rejected—breaks human inspectability and eOn/LODE interoperability that motivated CON. 2. **Leave units forever optional**: rejected for **new** writes—downstream optimizers and ML tools mis-scale energies and forces silently. 3. **Encode units only in filenames or READMEs**: rejected—pairing and provenance loss under campaign stores. Compatibility expectations ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. table:: +---------------------------+--------------------------------------------------------------------+ | Writer / reader | Expectation | +===========================+====================================================================+ | Legacy (no JSON line 2) | Permissive read as implicit v1; writers should emit ≥ v2 JSON | +---------------------------+--------------------------------------------------------------------+ | v2 JSON without ``units`` | Readable by v3 implementations; new writes should add ``units`` | +---------------------------+--------------------------------------------------------------------+ | v3 with ``units`` | Preferred interchange for new tools | +---------------------------+--------------------------------------------------------------------+ | Unknown future version | Strict mode errors; permissive mode may warn and best-effort parse | +---------------------------+--------------------------------------------------------------------+ Optional sections beyond velocities / forces / energies ------------------------------------------------------- The declared-section mechanism already supports new optional section names without a format major. ``charges`` / ``spins`` (scalar, like ``energies``) and ``magmoms`` (3-vector, like ``velocities``) are reserved on the **current** v2/v3 surface. Files use ``con_spec_version`` 2 or 3; omit the sections when unused. Do not invent ``con_spec_version: 4`` solely for optional physics blocks. Follow-ups (not format majors) ------------------------------ Conformance corpus keyed to spec clauses ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The metadata schema (``schema/con-metadata.schema.json``) machine-checks line 2; whole-file conformance still rests on the reference test suite. A published corpus of minimal valid and invalid files, each annotated with the normative clause it exercises and the required outcome (parse / specific ``ParseError`` variant), lets an independent implementation claim conformance without reading readcon-core source. Vocabulary registry over ad hoc keys ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Reserved metadata keys currently live in one table in the spec. A registry (one file, one key per row: name, type, since-version, producer semantics) with an extension-key convention (``x_`` prefix) keeps third-party producers from colliding on bare names while unrecognized-key preservation stays mandatory.