readcon-db design notes

Problem

readcon-core streams and validates CON/convel files. NEB and long-timescale campaigns produce corpora: (10^3)–(106) frames across many trajectories. Loading every frame as a ConFrame exceeds RAM; scanning every file for “all frames with H and Cu and (N<200)” wastes I/O.

Approach

Treat the corpus as an embedded key-value database with hand-built secondary indexes, backed by LMDB via Heed so the OS page cache supplies RAM residency without a custom buffer pool.

Embedded multi-process SOTA patterns (what we implement)

These are standard patterns for local high-performance multi-process KV stores (LMDB / Kyoto Cabinet / RocksDB-class embedded engines; Gray & Reuter MVCC lineage)— not distributed cluster consensus (Raft/Paxos), which is out of scope for a workstation campaign corpus.

Pattern	Chemistry-store gap (ASE.db / SQLite)	readcon-db
Mmap primary storage	Row pages + BLOB deserialize	OS page cache holds CON text
MVCC multi-reader, single-writer	Writers/readers contend on SQLite locks	LMDB COW pages; max_readers=512
Cross-process shared env	One connection model in Python	Separate OS processes each open() same dir
Authoritative value + secondary indexes	Pickled Atoms + ad hoc KV	CON blob authoritative; B-tree indexes rebuild via reindex
Selective index scan + set intersection	Often full table / formula scan	Smallest-first BTreeSet intersect
Batch read txn for trajectory extract	Per-row toatoms()	touch_trajectory_blobs / get_frame_texts one RoTxn
Exact content addressing	Row id / UUID	xxHash3-128 on stored blob

Cluster / network patterns (not implemented): multi-node sharding, Raft leader election, gRPC query plane. Those do not match the NEB-on-one-host threat model.

Why Heed/LMDB (not SQL, not SQLite)

• Mmap-first; readers do not copy the whole DB into a heap arena.

• Concurrent read transactions are first-class.

• Predictable latency for point lookups and ordered scans.

• We control indexes (natoms, symbols, energy, composition, fmax, section flags) for exact access patterns of MD post-processing.

• SQL would invite ad hoc joins that encourage copies and planner variance.

Secondary indexes

DB name	Key layout	Select predicates
idx_natoms	BE u32 natoms ‖ FrameKey	natoms_range
idx_symbol	symbol UTF-8 ‖ 0xff ‖ FrameKey	require_symbol
idx_elem_count	symbol ‖ 0xff ‖ BE count ‖ FrameKey	element_exact / element_min
idx_formula	canonical Sym:count\|... ‖ 0xff ‖ FrameKey	exact_composition
idx_topo / topo_by_frame	topology-key utf-8 ‖ 0xff ‖ FrameKey; FrameKey → key	topo_key, find_by_topology_*
idx_energy	order-preserving BE bits of finite energy ‖ FrameKey	energy_range
idx_fmax	order-preserving max (	Fi
idx_flags	flag_id (u8) ‖ FrameKey	require_forces / require_velocities / require_energy
frame_by_hash / hash_by_frame	xxHash3-128	exact_hash, dedup

Formula encoding: sorted non-empty symbols, Sym:count joined by | (e.g. =Cu:2|H:2=). Finite energies and fmax only; frames without forces never satisfy a finite fmax_range.

Query cost model: each predicate materializes a BTreeSet<FrameKey> from one index scan (or hash point lookup). Final result is set intersection, then sort + optional limit. Cost tracks selective indexes—not full corpus decode. Full-table fallback only when no indexed predicate is set.

Ingest paths

1. Path / multi-frame CON text (append_trajectory_path / append_trajectory_str) with next_with_raw_span when possible.

2. In-memory frames (append_trajectory_frames / extend_trajectory_frames) for chemfiles → ConFrame → corpus without a temp file. Python: append_trajectory_str / append_trajectory_frames. C: rkrdb_append_trajectory_str / rkrdb_append_trajectory_frame (RKRConFrame*).

3. Directory ingest (ingest_directory).

Reindex

ConCorpus::reindex (CLI: readcon-db reindex <corpus_dir>) clears secondary DBs and rebuilds them from authoritative frames blobs—schema upgrade path after adding indexes. Safe to run twice (idempotent key sets). Frames and traj_meta are not deleted. idx_topo is rebuilt from topo_by_frame when every annotated trajectory records the same cutoff/graph/hops/method; mixed parameters error without a half-rebuild. Unannotated corpora leave idx_topo empty.

Invariants

1. Frame blob is authoritative for fidelity; indexes are derived and rebuildable.

2. Single writer for ingest/reindex; analysis is read-only (ConCorpus::open_readonly / MDB_RDONLY). MPI ranks that all need the same frames should not each open the env: rank 0 of the caller communicator pack_frame (RCSO) / pack_frames (RCSB) and MPI_Bcast on that same handle (include/readcon-db-mpi.h, Python bcast_packed_frame / bcast_packed_frames). The host already owns MPI (LAMMPS lmp->world / a sub-comm, mpi4py Comm); the library never calls MPI_Init and never names the process-wide world communicator. Workers unpack with no handle. Shared mmap (open_readonly on every rank) is the other legal path when ranks touch different keys.

3. Decode with ``readcon-core`` so CON semantics never fork.

4. Selection returns keys first; callers decode lazily.

Core contracts (readcon-core::index_proj)

Screening scalars and ingest rules live in readcon-core so this crate does not fork CON meaning:

Contract	API	Role
Index projection	FrameIndexProjection::from_frame	natoms, formula, finite energy, fmax, mass, volume, sections mask, meta channels
Formula encoding	composition_formula / frame_composition_formula	idx_formula keys (Cu:2|H:2)
Finite policy	finite_energy, mass/volume/fmax	Non-finite scalars omitted from ordered indexes
Sections mask	sections_present_mask / SECTIONS_MASK_*	forces / velocities / energies flags
Span ingest	ConFrameIterator::next_with_raw_span, frame_byte_spans	Store exact multi-frame substrings; no hot-path re-serialize
Canonical write	ConFrameWriter::canonical(true)	Opt-in stable JSON key order for materialize-from-frames

frame_scalars and corpus prepare/reindex call into these APIs (thin wrappers / delegates).

Ecosystem

Crate	Responsibility
readcon-core	CON/convel interchange, chemfiles ingress, multi-language hourglass ABI. CPC manuscript main claim.
readcon-db (this repo)	Companion campaign store: corpora, indexes, mmap multi-reader, exact dedup, reindex. Not a second CPC paper.

ASE is calculator-only; not the campaign store. Fair ASE.db timings for a CPC appendix, if used, are the freeze in paper/cpc/freeze/.

Security / multi-tenant

Single trusted user on local disk for v1. No network protocol in v1.

Status

Shipped: frames, traj_meta, composition/energy/fmax/flags/natoms/symbol/hash indexes, Select, reindex, append frames, CLI/Python/C campaign select. ## Optional cooked SoA tier (frames_soa)

Shipped (derived, non-authoritative): RCSO is not authority. Each FrameKey may have a binary payload in LMDB DB frames_soa (magic RCSO, v1 LE POD header + f64 N×3 positions and optional forces/velocities). Encode/decode: cooked_soa::CookedSoa from a parsed ConFrame.

Why we still carry CON text (not a “fully equivalent blob”)

RCSO is not a complete stand-in for a CON frame: it omits element symbols, masses, cell/angles, constraint bits, JSON metadata, section structure, and exact UTF-8 bytes. Those are required for xxHash3 content addressing, composition/symbol indexes, CON export, and join/split fidelity. The tier’s purpose is to avoid CON parse on numeric extract (get_positions / get_forces / get_velocities) when a valid cooked blob exists—not to drop frames storage. Deleting CON text while keeping RCSO is unsupported and fails get_frame_text / hash; keeping CON and deleting RCSO always works.

Rule	Behavior
Authority	UTF-8 CON text in `frames` only; xxHash3 / dedup / join-split / reindex ignore SoA
Opt-in	Default ingest does not cook; cook_frame / recook_all / append_trajectory_path_cook(..., true)
Hot path	Numeric getters prefer valid cooked (no CON parse on hit); corrupt/missing → parse CON
Discard	delete_cooked_soa; reindex and select unaffected; CON text and hash unchanged
Non-equivalence	Symbols/metadata/exact bytes live only in CON; RCSO cannot substitute for missing frames
DLPack	Still ephemeral in-process views on ConFrame—not an LMDB value format

Roadmap (unchanged): parallel chunked reindex; optional multi-dtype SoA matrix.

ASE.db column ↔ readcon-db (competitive screening set)

Speed only matters if filters users already have in ASE.db exist. Architecture stays non-SQL (secondary LMDB DBs + intersection); the feature claim is campaign-column parity for CON-derivable fields.

ASE.db / common filter	CON source	readcon-db predicate / index
natoms	atom count	natoms_range / idx_natoms
formula / species	multiset	exact_composition, element_* / idx_formula, idx_elem_count
symbol presence	symbols	require_symbol / idx_symbol
energy	metadata energy	energy_range / idx_energy
forces present / fmax	forces section / ‖F‖	require_forces, fmax_range / idx_flags, idx_fmax
velocities	section / data	require_velocities
total mass	masses_per_type × counts	mass_range / idx_mass
cell volume	lattice_vectors or boxl=+=angles	volume_range / idx_volume
pbc	metadata pbc (explicit only; missing ≠ match)	pbc([x,y,z]) / idx_pbc
time, timestep	reserved metadata	time_range, timestep_range / idx_meta
frame_index	reserved metadata	frame_index_range / idx_meta
NEB bead/band	neb_bead, neb_band	neb_bead_range, neb_band_range / idx_meta
charge, magmom	optional JSON numbers	charge_range, magmom_range / idx_meta
exact structure id	CON blob	exact_hash / xxHash3
topology up to relabelling	seams fingerprint key	topo_key / idx_topo
id / row id	—	FrameKey (trajid, frameidx)
unique_id UUID	—	N/A (ASE bookkeeping); use content hash
ctime / mtime / age	—	N/A (ASE bookkeeping)
user / calculator	—	N/A unless stored in CON metadata (not competitive set)
arbitrary key_value_pairs DSL	—	N/A (no SQL DSL); reserved + charge/magmom cover screening
SQL SELECT	—	N/A (architecture); use Select / CLI / Python

reindex rebuilds all secondary indexes including mass/volume/pbc/meta.

Writer concurrency

CPU-bound prepare (parse CON spans / serialize ConFrames) runs outside the exclusive LMDB write_txn. Concurrent threads may prepare in parallel; only commits serialize at the engine (single active write txn). FFI handle-table locks do not cover ingest.

HPC multi-writer (millions of ranks)

A single LMDB environment cannot run concurrent =write_txn=s. For site-scale ingest (many SLURM tasks uploading CON), use ``ShardedConCorpus``:

1. readcon-db shard-init /scratch/campaign --shards 256 once on the shared FS.

2. Each rank opens only its shard. One writer owns each shard_id across the job (traj_id % n_shards). If many ranks share a shard id, each node keeps a private tree, drain=s to a unique dest, then =join-drained.

3. Writers on different shards never share a write lock — up to n_shards parallel commits on one filesystem (bounded by FS, not one LMDB mutex).

4. Global queries: shard-select / ShardedConCorpus::select fans out read-only across shards (multi-reader MVCC per shard).

Assign trajectory IDs so traj_id % n_shards == shard_id (CLI shard-ingest advances start-id accordingly). This is partitioned embedded writers, not Raft multi-master — the right pattern for campaign uploads on Lustre/GPFS.

H5MD interchange

collect_h5md / export_h5md emit one cooked [T][N][3] trajectory (position, optional force and velocity). CON text stays the corpus authority. Dest units are Å / ps / kJ mol^-1 Å^-1 / Å ps^-1 (Angstrom ps-1 for MDA 2.10). Velocities follow D10: v1, v2, and v3.0 use (amu*angstrom^2/eV)^0.5 whatever the time label says; v3.1 uses units.time and fails when velocities have no units.time. An explicit velocity time override exists for a file that breaks that rule. The same override supplies the clock unit when a v3.1 frame has time or timestep and no units.time. v1, v2, and v3.0 clocks with no units.time are read as fs. Node-local ingest then drain_to + join-drained is the campaign write path (see above).

LMDB model decision (HPC)

KEEP sharded LMDB (Heed). Single-env multi-writer contradicts LMDB SWMR; HPC-scale uploads use independent envs per shard (MDHIM-style local backends / industry sharded-LMDB practice). Full rationale, citations (LMDB docs, MDHIM HotStorage’15, PapyrusKV, RocksDB contrast, HDF5/Zarr/ADIOS2/DAOS contrast), risks, and falsifiers: see the architecture note on sharded LMDB. Do not interpret “one write_txn per env” as “LMDB cannot do multi-process writes”—only that partitioning is mandatory for concurrent commits.

Compaction: reversible join / split for analysis

Mode	CLI	Use
sharded-lmdb	shard-init / compact-split	HPC multi-writer; parallel rank ingest
single-env-lmdb	compact-join / ordinary ingest	Laptop analysis; ConCorpus::open_readonly
extxyz	compact-export-extxyz [–sharded]	External ML tools (non-LMDB)

Join copies CON blobs by traj (ids preserved; duplicate traj across shards errors). Split routes traj_id % n_shards into a new manifest root. Membership is reversible under that routing (indexes rebuilt via normal prepare/commit ingest).