Architecture ============ Ecosystem --------- +-------------+-------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+ | Layer | Crate | Role | +=============+===================================================================================================================+==============================================================================================+ | Interchange | `readcon-core `__ (`docs `__) | CON/convel parse/write, chemfiles→=ConFrame=, typed ``energy`` / ``sections``, hourglass ABI | +-------------+-------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+ | Corpus | **readcon-db** (this project) | LMDB/Heed store, secondary indexes, xxHash3 dedup, multi-reader selects | +-------------+-------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------+ Frame **blobs are CON text**; indexes are derived at ingest from the same ``ConFrame`` readcon-core already built—no second metadata schema. Environment layout ------------------ :: Environment (Heed / LMDB) ├── frames : FrameKey → CON text blob (source span at ingest) ├── traj_meta : traj_id → { n_frames, source, optional topo_* } ├── idx_natoms : (n_atoms BE, FrameKey) → () ├── idx_symbol : (symbol ‖ 0xFF ‖ FrameKey) → () ├── idx_energy : (ord(E) BE, FrameKey) → () # finite energy only ├── idx_fmax : (ord(fmax) BE, FrameKey) → () # forces present only ├── idx_mass : (ord(mass) BE, FrameKey) → () ├── idx_volume : (ord(V) BE, FrameKey) → () ├── idx_pbc : mask ‖ FrameKey # metadata pbc only ├── idx_meta : channel ‖ ord(value) ‖ FrameKey # time, NEB, charge, … ├── idx_elem_count : symbol ‖ 0xFF ‖ BE count ‖ FrameKey ├── idx_formula : `Cu:2|H:2` ‖ 0xFF ‖ FrameKey ├── idx_topo : topology-key utf-8 ‖ 0xFF ‖ FrameKey # optional; empty if missing on readonly ├── topo_by_frame : FrameKey → topology key # optional; authority for matching reindex ├── idx_flags : (flag_id ‖ FrameKey) → () # forces / velocities / has_energy ├── frame_by_hash : xxh3-128 → FrameKey (first wins) ├── frames_soa : FrameKey → RCSO cooked numerics (optional, derived) └── hash_by_frame : FrameKey → xxh3-128 ``idx_topo`` and ``topo_by_frame`` are created on write-open. A readonly open of an older corpus that lacks them treats the topology index as empty. ``TrajMeta`` may record ``topo_cutoff``, ``topo_graph``, ``topo_hops``, and ``topo_method`` after ``annotate_topology``. Those fields have serde defaults so old JSON still loads. Ingest and extend preserve them. **H5MD interchange:** ``collect_h5md`` / ``export_h5md`` emit one ``[T][N][3]`` trajectory (position, optional force and velocity). CON text stays authority. Dest units are Å / ps / kJ mol\ :sup:`-1` Å\ :sup:`-1`; velocity dest is ``Angstrom ps-1``. Velocities follow D10: v1, v2, and v3.0 use the natural unit whatever the time label says; v3.1 uses ``units.time``. A v3.1 clock with no ``units.time`` is an error. v1, v2, and v3.0 clocks with no ``units.time`` are read as ``fs``. **Drain/join:** node-local ``shard-ingest``, ``drain_to`` compact-snapshots ``data.mdb`` (refuse overwrite), then ``join-drained``. ``compact-join`` is the single-root join (``open_existing``). Ops: :doc:`/howto/campaign`. **Cooked SoA tier:** optional RCSO in ``frames_soa`` accelerates ``get_positions`` / ``get_forces`` / ``get_velocities`` without CON parse when valid. RCSO is non-authoritative: CON text in ``frames`` remains sole authority for hash/dedup/join/reindex. RCSO is not fully equivalent (no symbols/metadata/exact bytes). See ``docs/orgmode/cooked-soa.org``. ``FrameKey`` is 12 bytes: ``traj_id`` (BE u64) + ``frame_idx`` (BE u32) so lexicographic order matches numeric order. **Flag ids** (u8): ``1`` = has forces, ``2`` = has velocities, ``3`` = has finite energy. Energy / fmax use an order-preserving map of finite ``f64`` bits. Formula is sorted ``Sym:count`` joined by ``|``. Forces/velocities from declared ``sections`` or per-atom data. Ingest path ----------- #. Path / CON text: ``ConFrameIterator::next_with_raw_span`` when possible. #. **In-memory** ``append_trajectory_frames`` / ``extend_trajectory_frames`` for chemfiles → corpus. #. Parse once for indexes (natoms, symbols, composition, energy, fmax, flags). #. Store blob; compute **xxHash3-128**; update secondary B-trees. Reindex ------- ``ConCorpus::reindex`` / CLI ``readcon-db reindex`` clears secondary DBs and rebuilds from authoritative ``frames`` (schema evolution without delete+re-ingest). ``idx_topo`` is rebuilt from ``topo_by_frame`` when every annotated trajectory records the same cutoff/graph/hops/method. Mixed parameters error without a half-rebuild. An unannotated corpus leaves ``idx_topo`` empty. That rebuild does not need the engine. Selection and query costs ------------------------- Postings lists from secondary DBs are intersected in-process (``BTreeSet``): +------------------------------------------------------------------+--------------------+----------------------------------------+ | Predicate | Index | Cost sketch | +==================================================================+====================+========================================+ | ``exact_hash`` | ``frame_by_hash`` | Point lookup | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``require_symbol`` | ``idx_symbol`` | Prefix walk for that element | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``element_exact`` / ``element_min`` | ``idx_elem_count`` | Prefix + count filter | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``exact_composition`` | ``idx_formula`` | Prefix on canonical formula | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``topo_key`` | ``idx_topo`` | Prefix on topology HEX | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``natoms_range`` | ``idx_natoms`` | Ordered scan, early stop | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``energy_range`` | ``idx_energy`` | Ordered scan on finite energies | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``fmax_range`` | ``idx_fmax`` | Ordered scan (no forces ⇒ not indexed) | +------------------------------------------------------------------+--------------------+----------------------------------------+ | ``require_forces`` / ``require_velocities`` / ``require_energy`` | ``idx_flags`` | Prefix walk on flag id | +------------------------------------------------------------------+--------------------+----------------------------------------+ | traj filter alone | ``frames`` keys | Full key scan if no other index | +------------------------------------------------------------------+--------------------+----------------------------------------+ Decode (``get_frame``) is separate: one LMDB get + readcon-core parse. Keys-only select avoids decode. Topology keys and the kinetic ART catalogue test ------------------------------------------------ xxHash3 on CON bytes is exact. A permutation of atom order, or relaxation that moves any coordinate, is a different frame. Kinetic ART catalogues events by the **bonded topology up to relabelling** instead. Two minima (or a new saddle) that share the same graph and ring census are the same catalogue entry even when the coordinates differ. A saddle search uses that test to decide that a state has been visited. d-SEAMS (``seams fingerprint``) supplies that coarser key: per-atom rooted-neighbourhood certificates (nauty or Weisfeiler-Lehman) and a frame key from their sorted multiset plus the primitive ring census. This crate stores the key in ``idx_topo`` after an explicit annotate pass. Every annotated ``TrajMeta`` records cutoff, graph, hops, and method. Mixing methods in one corpus is refused. Reference: Trochet, A., et al., Phys. Rev. B 91, 224106 (2015). Bindings hourglass ------------------ :: Python / Fortran / C++ apps │ ▼ C ABI (rkrdb_*, incl. rkrdb_select_meta) ◄── cdylib / staticlib optional readcon-db-mpi.h (caller MPI_Comm only) │ ▼ Rust ConCorpus (Heed + readcon-core) Fortran uses ``bind(C)`` wrappers under ``fortran/ReadConDb/``. Prebuilt ``libreadcon_db`` is ``readcon-db-clib-$VERSION-$target.tar.gz`` on the GitHub Release (:doc:`/howto/install`).