Campaign ops runbook#

HPC ingest is partitioned writers, not multi-writer inside one LMDB env. Each shard_XXXX/ is an independent ConCorpus. Route traj_id % n_shards to that shard. One process owns each shard_id across the job. If more writers share a shard id, each node keeps a private tree, drains to a unique dest, then join-drained.

Default shard count is 64 (DEFAULT_N_SHARDS). Power-of-two counts keep routing simple.

This page is the campaign ops runbook. Multi-shard writers are covered by src/shard.rs tests (parallel_writers_different_shards, join_drained_roots_*) and tests/cli_refuse.rs (mint/overwrite refuses).

Layout#

<root>/
  shards.json          # { "n_shards": 64, "version": 1 }
  shard_0000/data.mdb
  shard_0001/data.mdb
  ...

shards.json is the manifest. shard-init writes it. shard-ingest, drain, join-drained, compact-join, and shard-select require it. Commands that only read (shard-select, compact-join) use open_existing and do not mint a missing root.

One writer per shard (pattern A)#

Use this when the job can assign shard ids uniquely (SLURM_PROCID % n_shards, or one rank owns shard (S)).

# once, on a shared or node-local root
readcon-db shard-init /scratch/$USER/campaign --shards 64

# each writer: --start-id must satisfy start-id ≡ shard (mod n_shards)
readcon-db shard-ingest /scratch/$USER/campaign --shard $S --start-id $T \
    --units '{"length":"angstrom","energy":"eV","time":"fs"}' run.con

shard-ingest refuses a missing manifest (does not mint). It refuses a --start-id that does not route to --shard. After the first file it skips traj ids that would land on another shard so one writer never touches a foreign env.

Close every writer (drop the process, or rkrdb_close) before drain. LMDB does not allow a snapshot while a write txn is open.

If every shard id has exactly one writer, drain to one dest that does not already hold those data.mdb files:

readcon-db drain /scratch/$USER/campaign /lustre/proj/campaign/drained
readcon-db compact-join /lustre/proj/campaign/drained /lustre/proj/campaign_single

drain compact-snapshots data.mdb only (no lock.mdb) and refuses dest overwrite. compact-join is the single-root join (open_existing); dest must not exist.

Overlapping shard ids (pattern B)#

Use this when many nodes write the same shard ids (more writers than shards, or each node owns a full 0..N-1 set on burst-buffer).

Each node keeps a private tree, drains to a unique dest, then one join merges those dests:

# per node, on burst buffer (Frontier /mnt/bb, Aurora /tmp, ...)
readcon-db shard-init /mnt/bb/$USER/campaign --shards 64
readcon-db shard-ingest /mnt/bb/$USER/campaign --shard $S --start-id $T \
    --units '{"length":"angstrom","energy":"eV","time":"fs"}' run.con

# close writers, then drain to a dest no other node uses
readcon-db drain /mnt/bb/$USER/campaign \
    /lustre/orion/proj/campaign/node_$SLURM_NODEID

# once, after every node drained
readcon-db join-drained /lustre/orion/proj/campaign_single \
    /lustre/orion/proj/campaign/node_*

join-drained requires shards.json on every source, refuses an existing dest, and refuses traj-id collisions before creating dest. Give writers disjoint traj id ranges (or disjoint shard sets) so join does not collide.

drain to a dest that already has shard_XXXX/data.mdb is an error: two node-local trees that share a shard id must not last-writer-wins.

Select and compact#

# fan-out select; does not mint a missing root
readcon-db shard-select /lustre/orion/proj/campaign/drained --symbol Cu

# one sharded root -> new single-env corpus (dest must not exist)
readcon-db compact-join /lustre/orion/proj/campaign/drained \
    /lustre/orion/proj/campaign_single

# reverse: single-env -> new sharded root
readcon-db compact-split /lustre/orion/proj/campaign_single \
    /lustre/orion/proj/campaign_reshard --shards 64

# extXYZ only when an external tool demands XYZ on disk
readcon-db compact-export-extxyz /lustre/orion/proj/campaign/drained \
    subset.xyz --sharded --symbol Cu

compact-export-extxyz on a sharded root needs --sharded. Without the flag it refuses (does not open the root as a single-env corpus). Dest must not exist.

H5MD interchange is on the single-env dest after join:

# Python, dest must not exist
python -c 'from readcon_db import ConCorpus
ConCorpus("/lustre/orion/proj/campaign_single", readonly=True).export_h5md(1, "traj.h5")'

Units#

Stamp units on ingest (--units JSON, or C rkrdb_append_trajectory_units / Fortran db_append_units). Aliases canonicalize (A -> angstrom). Missing units.time is CON fs. set_units converts stored numbers; append/extend stamp incoming values. CON line-2 units is the authority.

C / Fortran writers#

The C ABI and Fortran module open one corpus directory (rkrdb_open / db_open). They do not implement shard routing. Campaign partition stays on the CLI / Rust ShardedConCorpus.

A C or Fortran rank that owns shard (S) opens the shard directory after shard-init:

/* traj_id % n_shards == 3 */
rkrdb_open("/scratch/campaign/shard_0003", &id);
rkrdb_append_trajectory_units(id, 3, "run.con",
    "{\"length\":\"angstrom\",\"energy\":\"eV\",\"time\":\"fs\"}", &n);
rkrdb_close(id);

Then the same drain / join-drained / compact-join sequence as above.

Same-frame MPI: do not open the env on every rank. Rank 0 of the caller communicator packs RCSO / RCSB and MPI_Bcasts on that handle (include/readcon-db-mpi.h, Fortran INTEGER + rkrdb_bcast_packed_frame_f). Shared open_readonly is the other legal path when ranks touch different keys. See workflows.

Refuses (do not fight them)#

Command	Refuse
`shard-ingest`	missing `shards.json`; `--start-id` not congruent to `--shard`
`shard-select` / `compact-join`	missing root / missing `shards.json` (no mint)
`drain`	missing source `shards.json`; dest `shard_XXXX/data.mdb` already exists; dest `n_shards` mismatch
`join-drained`	dest exists; a source lacks `shards.json`; traj id appears in more than one source
`compact-join` / `compact-split`	dest exists; source missing
`compact-export-extxyz`	sharded root without `--sharded`; dest exists

On refuse, dest is not created (or a partial dest is removed). Tests: tests/cli_refuse.rs, src/shard.rs.

CLI from a prebuilt clib prefix#

Unpack readcon-db-clib-$VERSION-$target.tar.gz (see install). bin/readcon-db is in that archive when the Release build produced it.

export PATH="$PREFIX/bin:$PATH"
export LD_LIBRARY_PATH="$PREFIX/lib:${LD_LIBRARY_PATH:-}"
readcon-db shard-init /scratch/$USER/campaign --shards 64