Architecture#

Ecosystem#

Layer

Crate

Role

Interchange

readcon-core (docs)

CON/convel parse/write, chemfiles→ConFrame, typed energy / sections, hourglass ABI

Corpus

readcon-db (this project)

LMDB/Heed store, secondary indexes, xxHash3 dedup, multi-reader selects

Frame blobs are CON text; indexes are derived at ingest from the same ConFrame readcon-core already built—no second metadata schema.

Environment layout#

Environment (Heed / LMDB)
├── frames          : FrameKey → CON text blob (source span at ingest)
├── traj_meta       : traj_id → { n_frames, source, optional topo_* }
├── idx_natoms      : (n_atoms BE, FrameKey) → ()
├── idx_symbol      : (symbol ‖ 0xFF ‖ FrameKey) → ()
├── idx_energy      : (ord(E) BE, FrameKey) → ()   # finite energy only
├── idx_fmax        : (ord(fmax) BE, FrameKey) → ()  # forces present only
├── idx_mass        : (ord(mass) BE, FrameKey) → ()
├── idx_volume      : (ord(V) BE, FrameKey) → ()
├── idx_pbc         : mask ‖ FrameKey  # metadata pbc only
├── idx_meta        : channel ‖ ord(value) ‖ FrameKey  # time, NEB, charge, …
├── idx_elem_count  : symbol ‖ 0xFF ‖ BE count ‖ FrameKey
├── idx_formula     : `Cu:2|H:2` ‖ 0xFF ‖ FrameKey
├── idx_topo        : topology-key utf-8 ‖ 0xFF ‖ FrameKey  # optional; empty if missing on readonly
├── topo_by_frame   : FrameKey → topology key               # optional; authority for matching reindex
├── idx_flags       : (flag_id ‖ FrameKey) → ()    # forces / velocities / has_energy
├── frame_by_hash   : xxh3-128 → FrameKey (first wins)
├── frames_soa      : FrameKey → RCSO cooked numerics (optional, derived)
└── hash_by_frame   : FrameKey → xxh3-128

idx_topo and topo_by_frame are created on write-open. A readonly open of an older corpus that lacks them treats the topology index as empty.

TrajMeta may record topo_cutoff, topo_graph, topo_hops, and topo_method after annotate_topology. Those fields have serde defaults so old JSON still loads. Ingest and extend preserve them.

H5MD interchange: collect_h5md / export_h5md emit one [T][N][3] trajectory (position, optional force and velocity). CON text stays authority. Dest units are Å / ps / kJ mol^{-1} Å^{-1}; velocity dest is Angstrom ps-1.

Drain/join: node-local shard-ingest, drain_to compact-snapshots data.mdb (refuse overwrite), then join-drained. compact-join is the single-root join (open_existing). Ops: campaign.

Cooked SoA tier: optional RCSO in frames_soa accelerates get_positions / get_forces / get_velocities without CON parse when valid. RCSO is non-authoritative: CON text in frames remains sole authority for hash/dedup/join/reindex. RCSO is not fully equivalent (no symbols/metadata/exact bytes). See docs/orgmode/cooked-soa.org.

FrameKey is 12 bytes: traj_id (BE u64) + frame_idx (BE u32) so lexicographic order matches numeric order.

Flag ids (u8): 1 = has forces, 2 = has velocities, 3 = has finite energy. Energy / fmax use an order-preserving map of finite f64 bits. Formula is sorted Sym:count joined by |. Forces/velocities from declared sections or per-atom data.

Ingest path#

  1. Path / CON text: ConFrameIterator::next_with_raw_span when possible.

  2. In-memory append_trajectory_frames / extend_trajectory_frames for chemfiles → corpus.

  3. Parse once for indexes (natoms, symbols, composition, energy, fmax, flags).

  4. Store blob; compute xxHash3-128; update secondary B-trees.

Reindex#

ConCorpus::reindex / CLI readcon-db reindex clears secondary DBs and rebuilds from authoritative frames (schema evolution without delete+re-ingest). idx_topo is rebuilt from topo_by_frame when every annotated trajectory records the same cutoff/graph/hops/method. Mixed parameters error without a half-rebuild. An unannotated corpus leaves idx_topo empty. That rebuild does not need the engine.

Selection and query costs#

Postings lists from secondary DBs are intersected in-process (BTreeSet):

Predicate

Index

Cost sketch

exact_hash

frame_by_hash

Point lookup

require_symbol

idx_symbol

Prefix walk for that element

element_exact / element_min

idx_elem_count

Prefix + count filter

exact_composition

idx_formula

Prefix on canonical formula

topo_key

idx_topo

Prefix on topology HEX

natoms_range

idx_natoms

Ordered scan, early stop

energy_range

idx_energy

Ordered scan on finite energies

fmax_range

idx_fmax

Ordered scan (no forces ⇒ not indexed)

require_forces / require_velocities / require_energy

idx_flags

Prefix walk on flag id

traj filter alone

frames keys

Full key scan if no other index

Decode (get_frame) is separate: one LMDB get + readcon-core parse. Keys-only select avoids decode.

Topology keys and the kinetic ART catalogue test#

xxHash3 on CON bytes is exact. A permutation of atom order, or relaxation that moves any coordinate, is a different frame. Kinetic ART catalogues events by the bonded topology up to relabelling instead. Two minima (or a new saddle) that share the same graph and ring census are the same catalogue entry even when the coordinates differ. A saddle search uses that test to decide that a state has been visited.

d-SEAMS (seams fingerprint) supplies that coarser key: per-atom rooted-neighbourhood certificates (nauty or Weisfeiler-Lehman) and a frame key from their sorted multiset plus the primitive ring census. This crate stores the key in idx_topo after an explicit annotate pass. Every annotated TrajMeta records cutoff, graph, hops, and method. Mixing methods in one corpus is refused.

Reference: Trochet, A., et al., Phys. Rev. B 91, 224106 (2015).

Bindings hourglass#

  Python / Fortran / C++ apps
            │
            ▼
     C ABI (rkrdb_*, incl. rkrdb_select_meta)  ◄── cdylib / staticlib
     optional readcon-db-mpi.h (caller MPI_Comm only)
            │
            ▼
   Rust ConCorpus (Heed + readcon-core)

Fortran uses bind(C) wrappers under fortran/ReadConDb/. Prebuilt libreadcon_db is readcon-db-clib-$VERSION-$target.tar.gz on the GitHub Release (install).