Developer workflow and release guidelines

Requirement words

The words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY on this page follow RFC 2119.

Development setup

Prerequisites

  • Rust >= 1.88 (edition 2024)

  • pixi for environment management

  • cocogitto (cog) for conventional commits and the git changelog

  • cogapp (python -m cogapp) to stamp generated copies of the package version

  • towncrier for news fragments under changelog.d/

  • Pins for those two tools: requirements-version.txt

  • prek for git hooks (prek.toml)

  • lychee for docs link checks (via pixi r -e docs linkcheck)

Getting started

git clone https://github.com/lode-org/readcon-core.git
cd readcon-core
pixi install

# Run tests
pixi r test

# Run with all features (needs capnproto)
pixi r test-all

# Hooks (prek) and docs
pixi r prek
pixi r -e docs docbld
pixi r -e docs linkcheck

# Build release
pixi r build

Environment-specific workflows

# Python bindings
pixi r -e python python-build
pixi r -e python python-test

# Julia bindings
pixi r -e julia julia-test

# Documentation
pixi r -e docs docbld
# Docs output in docs/build/

Commit conventions

The project uses conventional commits enforced by cocogitto. The CI lint check rejects non-conforming commit messages on the PR range. A push whose previous tip is not in the clone, including a rewritten branch, is checked from the merge-base with origin/main, the same as a new branch. Some historical commits use docs+bench: or untyped merges; cog check --from-latest-tag is the local equivalent of the PR gate. scripts/release-prep.sh calls scripts/bump-version.sh, so towncrier writes the ## vX.Y.Z heading from the fragments in changelog.d/.

Recognized commit types (from cog.toml):

Type

Changelog section

Use for

feat

Features

New user-facing functionality

enh

Enhancements

Improvements to existing features

fix

Bugfixes

Bug corrections

bug

Bugfixes

Alias for fix

doc

Documentation

Documentation-only changes

tst

Tests

Test additions or modifications

bld

Buildsystem

Build system, CI, dependency changes

maint

Maintenance

Refactoring, cleanup, internal changes

bench

Benchmarks

Benchmark additions or modifications

gen

Generated

Auto-generated file updates

dat

Data

Test data or resource changes

Examples:

feat: add convel format support with optional velocity fields
enh: integrate fast-float2 for f64 parsing hot path
fix: handle empty velocity section in multi-frame files
doc: update tutorials with Julia convel example
bld: add CMakeLists.txt for cmake subproject use
tst: add roundtrip test for convel writer

Branching and workflow

  • main is the release branch. All PRs target main.

  • Feature branches: feat/<description>

  • Bugfix branches: fix/<description>

  • Keep commits logical and atomic. Soft-reset and rebase as needed. Contributors MUST NOT soft-reset or rebase main.

  • The CI runs on every PR: Rust tests, lint, coverage, Fortran fpm tests, Python (ci_python.yml: maturin develop + pytest tests/python/ for lean and chemfiles features), and benchmark regression (one-runner ASV via Snakemake, memray peaks, asv-perch).

Testing

# Core Rust tests
cargo test

# All features (parallel, rpc, python)
cargo test --all-features

# Meson build with valgrind leak checking
meson setup bbdir -Dwith_tests=True -Dwith_examples=True
meson test -C bbdir

# Benchmarks
cargo bench
# or: pixi r bench

Test data lives in resources/test/. Use the test_case! macro in integration tests to locate test files.

Profile-guided optimisation (PGO)

The release and dist profiles in Cargo.toml already enable lto = "thin" / lto = "fat" and codegen-units = 1. PGO can give another 5–15% on parser hot paths by feeding the optimiser a real profile from the Criterion benchmarks. It is opt-in because the build takes 2–3x longer.

Three-step workflow (Linux, rustc 1.88+):

# 1. Build instrumented binary; pass --features for everything you
#    want profile data on.
mkdir -p /tmp/readcon-pgo
RUSTFLAGS="-Cprofile-generate=/tmp/readcon-pgo" \
  cargo bench --features parallel,zstd,rpc --no-run

# 2. Run the benches to collect .profraw files.
RUSTFLAGS="-Cprofile-generate=/tmp/readcon-pgo" \
  cargo bench --features parallel,zstd,rpc

# 3. Merge profraws then build with --use to consume the merged profile.
$(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata \
  merge -o /tmp/readcon-pgo/merged.profdata /tmp/readcon-pgo
RUSTFLAGS="-Cprofile-use=/tmp/readcon-pgo/merged.profdata" \
  cargo build --release --features parallel,zstd,rpc

The same flow works for cargo cinstall when you want PGO-optimised libreadcon_core.so for distribution; substitute cargo cinstall for cargo build --release in step 3.

Continuous integration

Workflows

Workflow

File

Trigger

Purpose

Prek

ci_prek.yml

push, PR

prek run -a (whitespace, ruff, taplo, codespell)

Documentation

ci_docs.yml

push, PR

pixi r -e docs docbld then lychee on HTML

C/C++ dist

ci_cxx.yml

push, PR

CMake/Meson/pkg-config without cbindgen

Python wheels

python_wheels.yml

v* tag, PR

Build wheels for 5 platforms, publish to PyPI

Benchmark PR

ci_benchmark.yml

PR to main

ASV base vs head on one runner (Snakemake)

Comment benchmark results

ci_bench_commenter.yml

benchmark complete

Post ASV table as PR comment

Coverage

coverage.yml

push, PR

Code coverage via tarpaulin

Lint

lint.yml

PR

Conventional commit check + large file audit

Release

release.yml

PR plan, v* tag

cargo-dist plan on PRs; GitHub Release on tag

crates.io

crates_publish.yml

v* tag

cargo publish --locked

cxx tarball

cxx_tarball.yml

v* tag, workflow_dispatch tag

Attach slim + vendor C/C++ source tarballs

clib tarball

c_lib_tarball.yml

v* tag, workflow_dispatch tag

Attach prebuilt C ABI (readcon-core-clib-$VER-$target)

Benchmark regression detection

Two-workflow pattern for safe PR comments:

  1. ci_benchmark.yml is one job. It runs snakemake -s benchmarks/workflow/Snakefile. asv builds and installs both commits. Main-vs-main repetitions on that runner set a per-benchmark noise band. For timing, one bound is the wider of the linear 99th percentile and median + 3*median absolute deviation (MAD) of the odd-round versus even-round ratios. A second bound is a Student t prediction limit for a ratio of two side medians from those repetitions. The timing upper bound is the wider of the two. The timing lower bound uses the reciprocal when it is smaller. The false-slowdown budget is 0.01, split across the timing rows in this comparison. Memory keeps the within-run bound, widened by one allocator granule over the median base peak. Base vs head wall time is asv continuous --interleave-rounds (no --quick). Memory peaks are ray_* benchmarks (asv_bench_memray). Tool versions are pinned in benchmarks/workflow/requirements.txt and asv.conf.json. ci_benchmark.yml also pins rustc 1.88.0, Actions commits, uv 0.13.6, Python 3.12.16, and exact cmake/gcc/g++ packages. The job writes results/comparison.txt and results/row-bounds.txt (benchmark, within upper, between upper, upper).

  2. ci_bench_commenter.yml triggers on workflow_run completion when the Benchmark PR conclusion is success or failure, downloads benchmark-results, and posts the table with asv-perch. A failed check does not meet the success condition. The comment table shows each row’s own upper bound, the sample count per side, the base median, the head median, and the interquartile ranges. A row is a slowdown when its ratio is above that bound. The comment step passes a threshold of 1. Every row upper is at least 1, so a slowed row still drafts. A blank mark in the table is not significant. The job refuses identical base and head commits and exits non-zero when a row is a significant slowdown or is missing samples.

The workflow_run split is required for fork PRs to have write access for posting comments (GitHub security model). Suite and local reproduce: Performance Benchmarks (benchmarks/, asv.conf.json, benchmarks/workflow/Snakefile).

Executable chemfiles pages

The chemfiles notebook is the executable Org Babel source of the chemfiles tutorial, for Python only. Continuous integration (CI) runs scripts/run-chemfiles-notebook.sh from ci_python.yml. The script tangles the notebook into docs/notebooks/chemfiles_ingress.py, fails if the committed file differs from the new tangle, and runs it with python3. The Rust blocks and the blocks in other languages on the chemfiles tutorial page are prose only, and CI does not run them. Contributors MUST NOT edit the tangled .py file by hand. Change the notebook Org source instead, and tangle it again with READCON_TANGLE_UPDATE=1 scripts/run-chemfiles-notebook.sh.

Release process

This section is the only guide a new contributor needs to cut a release without reading CI YAML. Three layers work together. Use all of them. Maintainers MUST NOT short-circuit with a local-only cargo publish unless CI secrets are broken (fallback is documented below).

Mental model (new contributor)

scripts/release-prep.sh X.Y.Z
      │  prek, pixi docbld + lychee, version bump, towncrier CHANGELOG,
      │  cxx-dist gate; cbindgen optional
      ▼
commit: maint: bump to vX.Y.Z   ──►  open Pull Request to main
      │                                    │
      │                         cargo-dist workflow "Release"
      │                         runs `dist plan` on the PR (validate)
      ▼                                    ▼
merge PR to main  ──►  git tag -s vX.Y.Z && git push origin vX.Y.Z
      │
      ├─► workflow "Release" (cargo-dist): CLI archives + GitHub Release
      ├─► workflow "Publish to crates.io": cargo publish --locked
      │         needs repo secret CARGO_REGISTRY_TOKEN
      ├─► workflow "Python wheels": maturin matrix → PyPI (OIDC env `pypi`)
      ├─► workflow "cxx source tarball": v* tag, upload after the Release exists
      └─► workflow "C ABI library tarball": v* tag (or dispatch tag later)

Layer

What

Workflow / tool

Prep

Hooks + docs + versions + changelog

prek, pixi r -e docs docbld, lychee, scripts/release-prep.sh, towncrier build

Release PR

Validate dist plan before a tag exists

.github/workflows/release.yml (cargo-dist; pr-run-mode = "plan" in dist-workspace.toml)

Tag publish

crates.io

.github/workflows/crates_publish.yml

Tag publish

PyPI wheels

.github/workflows/python_wheels.yml

Tag publish

GitHub Release + CLI tarballs

same cargo-dist Release workflow on the tag

Tag publish

C/C++ source tarballs

.github/workflows/cxx_tarball.yml (v* tag; upload after the Release exists)

Tag publish

Prebuilt C ABI tarballs

.github/workflows/c_lib_tarball.yml (v* tag or workflow_dispatch tag)

Attach prebuilt C ABI to an existing tag

.github/workflows/c_lib_tarball.yml matches cxx_tarball.yml. Both start on a v* tag push, with crates.io, cargo-dist, and wheels. They wait until the GitHub Release exists, then upload archives that are not already on the release with gh release upload --clobber. They do not create the release and do not move the tag. workflow_dispatch input tag backfills an existing release the same way and does not publish crates or wheels. Use that to put readcon-core-clib-$VER-$target.tar.gz on a tag that predates the workflow (for example v0.14.7):

gh workflow run cxx_tarball.yml --repo lode-org/readcon-core --ref main -f tag=v0.14.7
gh workflow run c_lib_tarball.yml --repo lode-org/readcon-core --ref main -f tag=v0.14.7
  1. Open Actions → C ABI library tarball.

  2. Run the workflow from a branch that has scripts/package-clib.sh (usually main after this lands).

  3. Set tag to the existing Git tag (v0.14.7).

The packager comes from the selected workflow ref. Sources and the GitHub Release come from inputs.tag. Windows + chemfiles is an explicit skip (Python wheel path only).

The uploader skips an asset whose sha256 already matches. It does not replace an asset whose bytes differ. On a v* tag push that mismatch is an error and the published file stays. On a workflow_dispatch backfill the same mismatch is a ::warning:: and the published file stays, so the job can still attach an asset that is missing.

Windows clib builds pass /Brepro to the MSVC linker and remap the crate root and the cargo home out of the binary. The Windows job builds the tarball twice from a clean target directory and fails when the sha256 differs. Linux and macOS pack those same libraries a second time and fail the same check. scripts/reproducible-tar.sh dates members with SOURCE_DATE_EPOCH, owns them as uid 0, sorts the names, and runs gzip -n. PDB files are not shipped.

cargo-dist release-PR path

  1. Do not push a version tag from a laptop until the version-bump commit is on main (via PR). That PR is the release PR: it SHOULD contain only the maint: bump to vX.Y.Z (and any intentional release-note / header sync) produced by scripts/release-prep.sh or the manual checklist below.

  2. On every PR, GitHub Actions runs the workflow named Release (file release.yml). With pr-run-mode = "plan" it runs dist plan only: it checks that cargo-dist can announce this version and list artifacts. It does not publish crates or create a GitHub Release on the PR.

  3. If the plan job is red, fix dist-workspace.toml / version alignment and push. Maintainers MUST NOT tag yet.

  4. After the PR is merged, on a clean main:

    git checkout main && git pull origin main
    git tag -s vX.Y.Z -m "vX.Y.Z"
    git push origin vX.Y.Z
    
  5. On the tag push, the same Release workflow switches to publishing mode: builds platform archives for the readcon-core CLI binary and creates (or updates) the GitHub Release with those artifacts and notes derived from the changelog.

Regenerate the workflow after editing dist-workspace.toml. Maintainers MUST NOT hand-edit release.yml as the long-term copy:

# cargo-dist 0.28 from the sha256-pinned archive (scripts/install-cargo-dist-ci.sh)
bash scripts/install-cargo-dist-ci.sh
dist generate --mode=ci
# Re-apply action SHA pins and the checksummed install steps. dist generate
# rewrites release.yml to mutable tags and curl|sh.
git add .github/workflows/release.yml dist-workspace.toml scripts/install-cargo-dist-ci.sh

crates.io publish workflow and CARGO_REGISTRY_TOKEN

Workflow file: .github/workflows/crates_publish.yml (name: Publish to crates.io).

  • Trigger: push of tags matching v*, or workflow_dispatch on a branch (uses Cargo.toml package version; on a v* tag the tag MUST match that version or the job fails fast).

  • Action: cargo publish --locked for the readcon-core crate.

  • Idempotent skip: if https://crates.io/api/v1/crates/readcon-core/<version> already returns 200, the job logs “already published” and does not call cargo publish (avoids “crate already exists” on re-runs).

  • Secret: repository secret exactly named CARGO_REGISTRY_TOKEN. Value is a crates.io API token for an account allowed to publish readcon-core (create at https://crates.io/settings/tokens ). Without it, the publish step fails with “please provide a non-empty token” / our explicit empty-secret guard.

Set or rotate the secret (example using pass and gh; run as a repo admin):

# Store offline (example path used by maintainers)
pass insert -m crates-io/readcon-core   # paste the crates.io token

# Push into GitHub Actions secrets for lode-org/readcon-core
printf '%s' "$(pass show crates-io/readcon-core)" \
  | gh secret set CARGO_REGISTRY_TOKEN --repo lode-org/readcon-core

# Confirm the secret exists (value is never shown)
gh secret list --repo lode-org/readcon-core

Local fallback only if CI is red and users are blocked:

CARGO_REGISTRY_TOKEN="$(pass show crates-io/readcon-core)" cargo publish --locked

Prefer fixing the secret and re-running the workflow over making local publish the default path.

PyPI (wheels) on the same tag

python_wheels.yml builds sdist + manylinux/macOS/Windows wheels with maturin (--features python, not chemfiles by default) and publishes with pypa/gh-action-pypi-publish using OIDC trusted publishing (environment pypi). No password secret is required once the trusted publisher is configured (see Initial PyPI setup below). Transient crates.io network errors are mitigated with CARGO_HTTP_MULTIPLEXING=false and a one-shot maturin retry; tag runs are not cancelled mid-flight by concurrency.

Version files

The package version is the single line in VERSION at the repository root. That is the only hand-written copy.

Bump with scripts/bump-version.sh X.Y.Z (or cog bump). scripts/stamp-version.sh only restamps cogapp copies. It does not write the towncrier heading or refresh the exported pins.

scripts/bump-version.sh X.Y.Z
# scripts/stamp-version.sh
# restamp only, after VERSION and the heading exist

scripts/bump-version.sh X.Y.Z (also scripts/set-version.sh) writes VERSION, runs python -m towncrier build --yes --version X.Y.Z, then scripts/stamp-version.sh. Towncrier is the only writer of ## vX.Y.Z headings. cog.toml sets disable_changelog = true and its pre_bump_hooks call bump-version.sh, so a cog bump commit already contains the heading and the compare links. CMake reads VERSION and sets CMAKE_CONFIGURE_DEPENDS. Meson passes files('VERSION') to project() (https://mesonbuild.com/Reference-manual_functions.html#project). Sphinx (docs/source/conf.py) reads VERSION at configure time. The C and Fortran strings, the Python and Julia and Fortran package files, citation metadata, and the changelog compare links are cogapp output. The Fortran parameter does not repeat public, because the module already lists readcon_version. Python __version__ follows the Cargo package version, which cogapp stamps from VERSION. A Windows checkout can store a carriage return before the newline in VERSION. build.rs strips that carriage return. Python reads VERSION in text mode, which already drops it. Tool versions are pinned in requirements-version.txt.

Published pages stay on the org-to-rst export (emacs --batch -l docs/export.el or pixi r -e docs orgbld). Org sources mark each install pin with an at-sign placeholder named READCON_VERSION. The exporter reads VERSION and does not take a version argument. Authors MUST NOT type the version into docs/source/*.rst.

News for a release is a fragment under changelog.d/, collected by towncrier. The towncrier config imports readcon_meta.__version__, which reads VERSION. Authors MUST NOT hand-edit the generated release section of CHANGELOG.md. Cocogitto still checks commit subjects and can render the git changelog; it is a separate program from cogapp, so version stamps call python -m cogapp.

scripts/check_version_sot.sh greps every copy of the version string and compares it to VERSION. CI runs python -m cogapp --check on the stamp file list and that script. The cog executable is cocogitto, so the drift job does not call it. scripts/release-prep.sh X.Y.Z is the release path: it calls scripts/bump-version.sh and then the org export. scripts/set-version.sh is only an alias for bump-version.sh.

Complete release checklist

Fast path (preferred):

prek run -a
pixi r -e docs docbld
pixi r -e docs linkcheck
scripts/release-prep.sh X.Y.Z
# review staged files, then:
git commit -m "maint: bump to vX.Y.Z"
# push a branch, open PR → cargo-dist Release workflow runs dist plan
# merge PR to main, then:
git checkout main && git pull
git tag -s vX.Y.Z -m "vX.Y.Z"
git push origin vX.Y.Z
# wait for Release + Publish to crates.io + Python wheels + cxx tarball

Manual equivalent of the script:

  1. Ensure all tests pass and the branch is clean:

    cargo test --locked
    cargo test --locked --features chemfiles   # optional; needs libchemfiles/cmake policy
    pixi r -e python python-test
    prek run -a
    pixi r -e docs docbld
    pixi r -e docs linkcheck
    
  2. Run scripts/bump-version.sh X.Y.Z. That writes VERSION, lets towncrier write the release heading, then runs cogapp. Re-export the docs with the org-to-rst pipeline so the published pages match.

  3. Check the lock with cargo test --locked. scripts/bump-version.sh already rewrites the readcon-core entry in Cargo.lock.

  4. Regenerate the README from readme_src.org when the user-facing docs changed (optional for patch releases that only touch the crate version). Use the project script so docs/orgmode/*.org links stay .org (plain ox-md rewrites them to missing .md paths):

    ./scripts/export-readme.sh
    
  5. The release heading comes from towncrier, which bump-version.sh already ran. Authors MUST NOT rebuild CHANGELOG.md with cog changelog. That drops the towncrier marker. Add a fragment under changelog.d/ before the bump. Authors MUST NOT hand-edit the generated release section.

  6. If the FFI surface changed, run scripts/regen-capi-headers.sh and commit include/readcon-core.h (CI build_test.yml runs --check).

  7. Commit the release preparation (maint: bump to vX.Y.Z) and open the release PR (cargo-dist plans on PRs). After merge:

    git tag -s vX.Y.Z -m "vX.Y.Z"
    git push origin vX.Y.Z
    
  8. Maintainers MUST NOT rely on a local cargo publish unless CI is red. Prefer crates_publish.yml with CARGO_REGISTRY_TOKEN set. Local fallback:

    CARGO_REGISTRY_TOKEN="$(pass show crates-io/readcon-core)" cargo publish --locked
    
  9. Tag CI also runs python_wheels.yml (PyPI) and cargo-dist Release (CLI tarballs + GitHub Release body from the changelog).

  10. Verify all distribution channels:

    • crates.io shows the new version

    • PyPI shows the new version (wait for CI)

    • GitHub Releases has cargo-dist archives (and any prior C ABI tarballs if still attached)

Optional local C ABI tarball for consumers that do not use cargo-dist CLI archives (headers + libreadcon_core.{a,so} via cargo build --release or cargo cinstall) can still be attached with gh release upload if needed.

Initial PyPI setup (first release only)

For the very first release, PyPI trusted publisher is not yet configured. Publish manually:

# Build wheels
maturin build --release --features python

# Upload (first time, creates the project on PyPI)
uvx twine upload target/wheels/*

Python ships two distributions from python_wheels.yml:

PyPI name

Features

Import

readcon

python only (stubs for chemfiles APIs)

import readcon

readcon-chemfiles

python,chemfiles (libchemfiles linked)

import readcon (has_chemfiles_support() True)

Configure a trusted publisher on each PyPI project (same workflow python_wheels.yml, environment pypi). Users MUST NOT install both wheels in one venv (both export the readcon module). Users SHOULD run pip install readcon-chemfiles when selection or import are required. Use lean readcon for CON I/O only. pip install 'readcon[chemfiles]' depends on readcon-chemfiles==X.Y.Z (pin updated by scripts/release-prep.sh).

Then configure trusted publisher on PyPI:

  1. Go to https://pypi.org/manage/project/readcon/settings/publishing/

  2. Add a new pending publisher:

    • Owner: lode-org

    • Repository: readcon-core

    • Workflow: python_wheels.yml

    • Environment: pypi

After this, all subsequent tagged releases auto-publish via OIDC.

Adding new features

Feature gates

Optional functionality is gated behind Cargo features:

Feature

Dependencies

Purpose

parallel

rayon

Multi-frame parallel parse

rpc

capnp, capnp-rpc, tokio, etc.

Cap’n Proto RPC serving

python

pyo3

Python bindings

Add new optional features in Cargo.toml under [features].

Adding a new binding

  1. Add the binding source under src/ (e.g., src/python.rs) or as a separate package (e.g., julia/ReadCon/).

  2. Gate behind a feature flag if it adds dependencies.

  3. Add tests under tests/ or the binding’s own test directory.

  4. Document in docs/orgmode/bindings.org and docs/orgmode/howto.org.

  5. Add a pixi environment and tasks if applicable.

Updating the C header

The C header include/readcon-core.h is the shipped ABI, the same model as metatensor/metatomic. CMake, Meson, cargo-c, CI, and the cxx tarball MUST NOT run cbindgen. Edit the header (or optionally draft with scripts/regen-capi-headers.sh) and commit it with the FFI change. CI compiles C/C++ against that file and checks symbols in the built library.

ci_cxx.yml installs no cbindgen and MUST stay green.

The C++ header include/readcon-core.hpp is hand-maintained and MUST be updated manually to match any changes to CAtom, CFrame, or the FFI function signatures.