readcon-db design notes ======================= Problem ------- ``readcon-core`` streams and validates CON/convel **files**. NEB and long-timescale campaigns produce **corpora**: (10\ :sup:`3)–(10`\ 6) frames across many trajectories. Loading every frame as a ``ConFrame`` exceeds RAM; scanning every file for “all frames with H and Cu and (N<200)” wastes I/O. Approach -------- Treat the corpus as an **embedded key-value database** with **hand-built secondary indexes**, backed by **LMDB via Heed** so the OS page cache supplies RAM residency without a custom buffer pool. Embedded multi-process SOTA patterns (what we implement) -------------------------------------------------------- These are standard patterns for **local** high-performance multi-process KV stores (LMDB / Kyoto Cabinet / RocksDB-class embedded engines; Gray & Reuter MVCC lineage)— **not** distributed cluster consensus (Raft/Paxos), which is out of scope for a workstation campaign corpus. +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | Pattern | Chemistry-store gap (ASE.db / SQLite) | readcon-db | +=============================================+=========================================+================================================================+ | **Mmap primary storage** | Row pages + BLOB deserialize | OS page cache holds CON text | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **MVCC multi-reader, single-writer** | Writers/readers contend on SQLite locks | LMDB COW pages; ``max_readers=512`` | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **Cross-process shared env** | One connection model in Python | Separate OS processes each ``open()`` same dir | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **Authoritative value + secondary indexes** | Pickled ``Atoms`` + ad hoc KV | CON blob authoritative; B-tree indexes rebuild via ``reindex`` | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **Selective index scan + set intersection** | Often full table / formula scan | Smallest-first ``BTreeSet`` intersect | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **Batch read txn for trajectory extract** | Per-row ``toatoms()`` | ``touch_trajectory_blobs`` / ``get_frame_texts`` one ``RoTxn`` | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ | **Exact content addressing** | Row id / UUID | xxHash3-128 on stored blob | +---------------------------------------------+-----------------------------------------+----------------------------------------------------------------+ Cluster / network patterns (**not** implemented): multi-node sharding, Raft leader election, gRPC query plane. Those do not match the NEB-on-one-host threat model. Why Heed/LMDB (not SQL, not SQLite) ----------------------------------- - Mmap-first; readers do not copy the whole DB into a heap arena. - Concurrent read transactions are first-class. - Predictable latency for point lookups and ordered scans. - We control indexes (natoms, symbols, energy, composition, fmax, section flags) for exact access patterns of MD post-processing. - SQL would invite ad hoc joins that encourage copies and planner variance. Secondary indexes ----------------- +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | DB name | Key layout | Select predicates | +=======================================+==============================================================+==================================================================+ | ``idx_natoms`` | BE ``u32`` n\ :sub:`atoms` ‖ ``FrameKey`` | ``natoms_range`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_symbol`` | symbol UTF-8 ‖ ``0xff`` ‖ ``FrameKey`` | ``require_symbol`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_elem_count`` | symbol ‖ ``0xff`` ‖ BE count ‖ ``FrameKey`` | ``element_exact`` / ``element_min`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_formula`` | canonical ``Sym:count\|...`` ‖ ``0xff`` ‖ ``FrameKey`` | ``exact_composition`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_topo`` / ``topo_by_frame`` | topology-key utf-8 ‖ ``0xff`` ‖ ``FrameKey``; FrameKey → key | ``topo_key``, ``find_by_topology_*`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_energy`` | order-preserving BE bits of finite energy ‖ ``FrameKey`` | ``energy_range`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_fmax`` | order-preserving max ( | F\ :sub:`i` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``idx_flags`` | ``flag_id`` (u8) ‖ ``FrameKey`` | ``require_forces`` / ``require_velocities`` / ``require_energy`` | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ | ``frame_by_hash`` / ``hash_by_frame`` | xxHash3-128 | ``exact_hash``, dedup | +---------------------------------------+--------------------------------------------------------------+------------------------------------------------------------------+ Formula encoding: sorted non-empty symbols, ``Sym:count`` joined by ``|`` (e.g. =Cu:2|H:2=). Finite energies and fmax only; frames without forces never satisfy a finite ``fmax_range``. **Query cost model:** each predicate materializes a ``BTreeSet`` from one index scan (or hash point lookup). Final result is set intersection, then sort + optional ``limit``. Cost tracks selective indexes—not full corpus decode. Full-table fallback only when no indexed predicate is set. Ingest paths ------------ #. Path / multi-frame CON text (``append_trajectory_path`` / ``append_trajectory_str``) with ``next_with_raw_span`` when possible. #. **In-memory frames** (``append_trajectory_frames`` / ``extend_trajectory_frames``) for chemfiles → ``ConFrame`` → corpus without a temp file. Python: ``append_trajectory_str`` / ``append_trajectory_frames``. C: ``rkrdb_append_trajectory_str`` / ``rkrdb_append_trajectory_frame`` (``RKRConFrame*``). #. Directory ingest (``ingest_directory``). Reindex ------- ``ConCorpus::reindex`` (CLI: ``readcon-db reindex ``) clears secondary DBs and rebuilds them from authoritative ``frames`` blobs—**schema upgrade path** after adding indexes. Safe to run twice (idempotent key sets). Frames and ``traj_meta`` are not deleted. ``idx_topo`` is rebuilt from ``topo_by_frame`` when every annotated trajectory records the same cutoff/graph/hops/method; mixed parameters error without a half-rebuild. Unannotated corpora leave ``idx_topo`` empty. Invariants ---------- #. **Frame blob is authoritative** for fidelity; indexes are derived and rebuildable. #. **Single writer** for ingest/reindex; analysis is read-only (``ConCorpus::open_readonly`` / ``MDB_RDONLY``). MPI ranks that all need the **same** frames should not each open the env: rank 0 of the **caller communicator** ``pack_frame`` (RCSO) / ``pack_frames`` (RCSB) and ``MPI_Bcast`` on that same handle (``include/readcon-db-mpi.h``, Python ``bcast_packed_frame`` / ``bcast_packed_frames``). The host already owns MPI (LAMMPS ``lmp->world`` / a sub-comm, mpi4py ``Comm``); the library never calls ``MPI_Init`` and never names the process-wide world communicator. Workers ``unpack`` with no handle. Shared mmap (``open_readonly`` on every rank) is the other legal path when ranks touch *different* keys. #. **Decode with ``readcon-core``** so CON semantics never fork. #. **Selection returns keys first**; callers decode lazily. Core contracts (``readcon-core::index_proj``) --------------------------------------------- Screening scalars and ingest rules live in **readcon-core** so this crate does not fork CON meaning: +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Contract | API | Role | +==================+================================================================+==================================================================================+ | Index projection | ``FrameIndexProjection::from_frame`` | natoms, formula, finite energy, fmax, mass, volume, sections mask, meta channels | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Formula encoding | ``composition_formula`` / ``frame_composition_formula`` | ``idx_formula`` keys (``Cu:2|H:2``) | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Finite policy | ``finite_energy``, mass/volume/fmax | Non-finite scalars omitted from ordered indexes | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Sections mask | ``sections_present_mask`` / ``SECTIONS_MASK_*`` | forces / velocities / energies flags | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Span ingest | ``ConFrameIterator::next_with_raw_span``, ``frame_byte_spans`` | Store exact multi-frame substrings; no hot-path re-serialize | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Canonical write | ``ConFrameWriter::canonical(true)`` | Opt-in stable JSON key order for materialize-from-frames | +------------------+----------------------------------------------------------------+----------------------------------------------------------------------------------+ ``frame_scalars`` and ``corpus`` prepare/reindex call into these APIs (thin wrappers / delegates). Ecosystem --------- +-------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | Crate | Responsibility | +=============================================================+==============================================================================================================+ | `readcon-core `__ | CON/convel interchange, chemfiles ingress, multi-language hourglass ABI. CPC manuscript main claim. | +-------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | **readcon-db** (this repo) | Companion campaign store: corpora, indexes, mmap multi-reader, exact dedup, reindex. Not a second CPC paper. | +-------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ ASE is calculator-only; not the campaign store. Fair ASE.db timings for a CPC appendix, if used, are the freeze in ``paper/cpc/freeze/``. Security / multi-tenant ----------------------- Single trusted user on local disk for v1. No network protocol in v1. Status ------ Shipped: frames, traj\ :sub:`meta`, composition/energy/fmax/flags/natoms/symbol/hash indexes, ``Select``, reindex, append frames, CLI/Python/C campaign select. ## Optional cooked SoA tier (``frames_soa``) **Shipped (derived, non-authoritative):** RCSO is not authority. Each ``FrameKey`` may have a binary payload in LMDB DB ``frames_soa`` (magic ``RCSO``, v1 LE POD header + f64 N×3 positions and optional forces/velocities). Encode/decode: ``cooked_soa::CookedSoa`` from a parsed ``ConFrame``. Why we still carry CON text (not a “fully equivalent blob”) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ RCSO is **not** a complete stand-in for a CON frame: it omits element symbols, masses, cell/angles, constraint bits, JSON metadata, section structure, and **exact UTF-8 bytes**. Those are required for xxHash3 content addressing, composition/symbol indexes, CON export, and join/split fidelity. The tier's purpose is to **avoid CON parse on numeric extract** (``get_positions`` / ``get_forces`` / ``get_velocities``) when a valid cooked blob exists—not to drop ``frames`` storage. Deleting CON text while keeping RCSO is unsupported and fails ``get_frame_text`` / hash; keeping CON and deleting RCSO always works. +-----------------+----------------------------------------------------------------------------------------------------------------+ | Rule | Behavior | +=================+================================================================================================================+ | Authority | UTF-8 CON text in \`frames\` only; xxHash3 / dedup / join-split / ``reindex`` ignore SoA | +-----------------+----------------------------------------------------------------------------------------------------------------+ | Opt-in | Default ingest does **not** cook; ``cook_frame`` / ``recook_all`` / ``append_trajectory_path_cook(..., true)`` | +-----------------+----------------------------------------------------------------------------------------------------------------+ | Hot path | Numeric getters prefer valid cooked (**no CON parse** on hit); corrupt/missing → parse CON | +-----------------+----------------------------------------------------------------------------------------------------------------+ | Discard | ``delete_cooked_soa``; reindex and select unaffected; CON text and hash unchanged | +-----------------+----------------------------------------------------------------------------------------------------------------+ | Non-equivalence | Symbols/metadata/exact bytes live only in CON; RCSO cannot substitute for missing ``frames`` | +-----------------+----------------------------------------------------------------------------------------------------------------+ | DLPack | Still **ephemeral** in-process views on ``ConFrame``—not an LMDB value format | +-----------------+----------------------------------------------------------------------------------------------------------------+ Roadmap (unchanged): parallel chunked reindex; optional multi-dtype SoA matrix. ASE.db column ↔ readcon-db (competitive screening set) ------------------------------------------------------ Speed only matters if filters users already have in ASE.db exist. Architecture stays **non-SQL** (secondary LMDB DBs + intersection); the **feature claim** is campaign-column parity for CON-derivable fields. +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ASE.db / common filter | CON source | readcon-db predicate / index | +===================================+===================================================+============================================================================+ | ``natoms`` | atom count | ``natoms_range`` / ``idx_natoms`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``formula`` / species | multiset | ``exact_composition``, ``element_*`` / ``idx_formula``, ``idx_elem_count`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | symbol presence | symbols | ``require_symbol`` / ``idx_symbol`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``energy`` | metadata ``energy`` | ``energy_range`` / ``idx_energy`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | forces present / ``fmax`` | forces section / ‖F‖ | ``require_forces``, ``fmax_range`` / ``idx_flags``, ``idx_fmax`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | velocities | section / data | ``require_velocities`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | total mass | ``masses_per_type`` × counts | ``mass_range`` / ``idx_mass`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | cell volume | ``lattice_vectors`` or ``boxl=+=angles`` | ``volume_range`` / ``idx_volume`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``pbc`` | metadata ``pbc`` (explicit only; missing ≠ match) | ``pbc([x,y,z])`` / ``idx_pbc`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``time``, ``timestep`` | reserved metadata | ``time_range``, ``timestep_range`` / ``idx_meta`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``frame_index`` | reserved metadata | ``frame_index_range`` / ``idx_meta`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | NEB bead/band | ``neb_bead``, ``neb_band`` | ``neb_bead_range``, ``neb_band_range`` / ``idx_meta`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``charge``, ``magmom`` | optional JSON numbers | ``charge_range``, ``magmom_range`` / ``idx_meta`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | exact structure id | CON blob | ``exact_hash`` / xxHash3 | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | topology up to relabelling | ``seams fingerprint`` key | ``topo_key`` / ``idx_topo`` | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``id`` / row id | — | ``FrameKey`` (traj\ :sub:`id`, frame\ :sub:`idx`) | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``unique_id`` UUID | — | **N/A (ASE bookkeeping)**; use content hash | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``ctime`` / ``mtime`` / ``age`` | — | **N/A (ASE bookkeeping)** | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | ``user`` / ``calculator`` | — | **N/A** unless stored in CON metadata (not competitive set) | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | arbitrary ``key_value_pairs`` DSL | — | **N/A (no SQL DSL)**; reserved + charge/magmom cover screening | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ | SQL ``SELECT`` | — | **N/A (architecture)**; use ``Select`` / CLI / Python | +-----------------------------------+---------------------------------------------------+----------------------------------------------------------------------------+ ``reindex`` rebuilds **all** secondary indexes including mass/volume/pbc/meta. Writer concurrency ------------------ CPU-bound prepare (parse CON spans / serialize ConFrames) runs **outside** the exclusive LMDB ``write_txn``. Concurrent threads may prepare in parallel; only commits serialize at the engine (single active write txn). FFI handle-table locks do not cover ingest. HPC multi-writer (millions of ranks) ------------------------------------ A **single** LMDB environment cannot run concurrent =write\ :sub:`txn`\ =s. For site-scale ingest (many SLURM tasks uploading CON), use **``ShardedConCorpus``**: #. ``readcon-db shard-init /scratch/campaign --shards 256`` once on the shared FS. #. Each rank opens **only its shard**. One writer owns each ``shard_id`` across the job (``traj_id % n_shards``). If many ranks share a shard id, each node keeps a private tree, ``drain=s to a unique dest, then =join-drained``. #. Writers on **different shards never share a write lock** — up to ``n_shards`` parallel commits on one filesystem (bounded by FS, not one LMDB mutex). #. Global queries: ``shard-select`` / ``ShardedConCorpus::select`` fans out read-only across shards (multi-reader MVCC per shard). Assign trajectory IDs so ``traj_id % n_shards == shard_id`` (CLI ``shard-ingest`` advances start-id accordingly). This is **partitioned embedded writers**, not Raft multi-master — the right pattern for campaign uploads on Lustre/GPFS. H5MD interchange ---------------- ``collect_h5md`` / ``export_h5md`` emit one cooked ``[T][N][3]`` trajectory (position, optional force and velocity). CON text stays the corpus authority. Dest units are Å / ps / kJ mol\ :sup:`-1` Å\ :sup:`-1` / Å ps\ :sup:`-1` (``Angstrom ps-1`` for MDA 2.10). Velocities follow D10: v1, v2, and v3.0 use ``(amu*angstrom^2/eV)^0.5`` whatever the time label says; v3.1 uses ``units.time`` and fails when velocities have no ``units.time``. An explicit velocity time override exists for a file that breaks that rule. The same override supplies the clock unit when a v3.1 frame has ``time`` or ``timestep`` and no ``units.time``. v1, v2, and v3.0 clocks with no ``units.time`` are read as ``fs``. Node-local ingest then ``drain_to`` + ``join-drained`` is the campaign write path (see above). LMDB model decision (HPC) ------------------------- **KEEP sharded LMDB (Heed).** Single-env multi-writer contradicts LMDB SWMR; HPC-scale uploads use **independent envs per shard** (MDHIM-style local backends / industry sharded-LMDB practice). Full rationale, citations (LMDB docs, MDHIM HotStorage'15, PapyrusKV, RocksDB contrast, HDF5/Zarr/ADIOS2/DAOS contrast), risks, and falsifiers: see the architecture note on sharded LMDB. Do **not** interpret “one write\ :sub:`txn` per env” as “LMDB cannot do multi-process writes”—only that **partitioning is mandatory** for concurrent commits. Compaction: reversible join / split for analysis ------------------------------------------------ +---------------------+----------------------------------------+-----------------------------------------------+ | Mode | CLI | Use | +=====================+========================================+===============================================+ | **sharded-lmdb** | ``shard-init`` / ``compact-split`` | HPC multi-writer; parallel rank ingest | +---------------------+----------------------------------------+-----------------------------------------------+ | **single-env-lmdb** | ``compact-join`` / ordinary ``ingest`` | Laptop analysis; ``ConCorpus::open_readonly`` | +---------------------+----------------------------------------+-----------------------------------------------+ | **extxyz** | ``compact-export-extxyz`` [–sharded] | External ML tools (non-LMDB) | +---------------------+----------------------------------------+-----------------------------------------------+ Join copies CON blobs by traj (ids preserved; duplicate traj across shards errors). Split routes ``traj_id % n_shards`` into a new manifest root. Membership is reversible under that routing (indexes rebuilt via normal prepare/commit ingest).