Developer workflow and release guidelines¶
Requirement words¶
The words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY on this page follow RFC 2119.
Development setup¶
Prerequisites¶
Rust >= 1.88 (edition 2024)
pixi for environment management
cocogitto (
cog) for conventional commits and the git changelogcogapp (
python -m cogapp) to stamp generated copies of the package versiontowncrier for news fragments under
changelog.d/Pins for those two tools:
requirements-version.txtprek for git hooks (
prek.toml)lychee for docs link checks (via
pixi r -e docs linkcheck)
Getting started¶
git clone https://github.com/lode-org/readcon-core.git
cd readcon-core
pixi install
# Run tests
pixi r test
# Run with all features (needs capnproto)
pixi r test-all
# Hooks (prek) and docs
pixi r prek
pixi r -e docs docbld
pixi r -e docs linkcheck
# Build release
pixi r build
Environment-specific workflows¶
# Python bindings
pixi r -e python python-build
pixi r -e python python-test
# Julia bindings
pixi r -e julia julia-test
# Documentation
pixi r -e docs docbld
# Docs output in docs/build/
Commit conventions¶
The project uses conventional commits enforced by cocogitto.
The CI lint check rejects non-conforming commit messages on the PR
range. A push whose previous tip is not in the clone, including a
rewritten branch, is checked from the merge-base with origin/main,
the same as a new branch. Some historical commits use docs+bench:
or untyped merges;
cog check --from-latest-tag is the local equivalent of the PR gate.
scripts/release-prep.sh calls scripts/bump-version.sh, so towncrier
writes the ## vX.Y.Z heading from the fragments in changelog.d/.
Recognized commit types (from cog.toml):
Type |
Changelog section |
Use for |
|---|---|---|
feat |
Features |
New user-facing functionality |
enh |
Enhancements |
Improvements to existing features |
fix |
Bugfixes |
Bug corrections |
bug |
Bugfixes |
Alias for fix |
doc |
Documentation |
Documentation-only changes |
tst |
Tests |
Test additions or modifications |
bld |
Buildsystem |
Build system, CI, dependency changes |
maint |
Maintenance |
Refactoring, cleanup, internal changes |
bench |
Benchmarks |
Benchmark additions or modifications |
gen |
Generated |
Auto-generated file updates |
dat |
Data |
Test data or resource changes |
Examples:
feat: add convel format support with optional velocity fields
enh: integrate fast-float2 for f64 parsing hot path
fix: handle empty velocity section in multi-frame files
doc: update tutorials with Julia convel example
bld: add CMakeLists.txt for cmake subproject use
tst: add roundtrip test for convel writer
Branching and workflow¶
mainis the release branch. All PRs targetmain.Feature branches:
feat/<description>Bugfix branches:
fix/<description>Keep commits logical and atomic. Soft-reset and rebase as needed. Contributors MUST NOT soft-reset or rebase
main.The CI runs on every PR: Rust tests, lint, coverage, Fortran fpm tests, Python (
ci_python.yml:maturin develop+pytest tests/python/for lean and chemfiles features), and benchmark regression (one-runner ASV via Snakemake, memray peaks, asv-perch).
Testing¶
# Core Rust tests
cargo test
# All features (parallel, rpc, python)
cargo test --all-features
# Meson build with valgrind leak checking
meson setup bbdir -Dwith_tests=True -Dwith_examples=True
meson test -C bbdir
# Benchmarks
cargo bench
# or: pixi r bench
Test data lives in resources/test/. Use the test_case! macro in
integration tests to locate test files.
Profile-guided optimisation (PGO)¶
The release and dist profiles in Cargo.toml already enable
lto = "thin" / lto = "fat" and codegen-units = 1. PGO can give
another 5–15% on parser hot paths by feeding the optimiser a real
profile from the Criterion benchmarks. It is opt-in because the build
takes 2–3x longer.
Three-step workflow (Linux, rustc 1.88+):
# 1. Build instrumented binary; pass --features for everything you
# want profile data on.
mkdir -p /tmp/readcon-pgo
RUSTFLAGS="-Cprofile-generate=/tmp/readcon-pgo" \
cargo bench --features parallel,zstd,rpc --no-run
# 2. Run the benches to collect .profraw files.
RUSTFLAGS="-Cprofile-generate=/tmp/readcon-pgo" \
cargo bench --features parallel,zstd,rpc
# 3. Merge profraws then build with --use to consume the merged profile.
$(rustc --print sysroot)/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata \
merge -o /tmp/readcon-pgo/merged.profdata /tmp/readcon-pgo
RUSTFLAGS="-Cprofile-use=/tmp/readcon-pgo/merged.profdata" \
cargo build --release --features parallel,zstd,rpc
The same flow works for cargo cinstall when you want PGO-optimised
libreadcon_core.so for distribution; substitute cargo cinstall
for cargo build --release in step 3.
Continuous integration¶
Workflows¶
Workflow |
File |
Trigger |
Purpose |
|---|---|---|---|
Prek |
|
push, PR |
|
Documentation |
|
push, PR |
|
C/C++ dist |
|
push, PR |
CMake/Meson/pkg-config without cbindgen |
Python wheels |
|
|
Build wheels for 5 platforms, publish to PyPI |
Benchmark PR |
|
PR to main |
ASV base vs head on one runner (Snakemake) |
Comment benchmark results |
|
benchmark complete |
Post ASV table as PR comment |
Coverage |
|
push, PR |
Code coverage via tarpaulin |
Lint |
|
PR |
Conventional commit check + large file audit |
Release |
|
PR plan, |
cargo-dist plan on PRs; GitHub Release on tag |
crates.io |
|
|
|
cxx tarball |
|
|
Attach slim + vendor C/C++ source tarballs |
clib tarball |
|
|
Attach prebuilt C ABI ( |
Benchmark regression detection¶
Two-workflow pattern for safe PR comments:
ci_benchmark.ymlis one job. It runssnakemake -s benchmarks/workflow/Snakefile. asv builds and installs both commits. Main-vs-main repetitions on that runner set a per-benchmark noise band. For timing, one bound is the wider of the linear 99th percentile and median + 3*median absolute deviation (MAD) of the odd-round versus even-round ratios. A second bound is a Student t prediction limit for a ratio of two side medians from those repetitions. The timing upper bound is the wider of the two. The timing lower bound uses the reciprocal when it is smaller. The false-slowdown budget is 0.01, split across the timing rows in this comparison. Memory keeps the within-run bound, widened by one allocator granule over the median base peak. Base vs head wall time isasv continuous --interleave-rounds(no--quick). Memory peaks areray_*benchmarks (asv_bench_memray). Tool versions are pinned inbenchmarks/workflow/requirements.txtandasv.conf.json.ci_benchmark.ymlalso pins rustc 1.88.0, Actions commits, uv 0.13.6, Python 3.12.16, and exact cmake/gcc/g++ packages. The job writesresults/comparison.txtandresults/row-bounds.txt(benchmark, within upper, between upper, upper).ci_bench_commenter.ymltriggers onworkflow_runcompletion when the Benchmark PR conclusion issuccessorfailure, downloadsbenchmark-results, and posts the table with asv-perch. A failed check does not meet the success condition. The comment table shows each row’s own upper bound, the sample count per side, the base median, the head median, and the interquartile ranges. A row is a slowdown when its ratio is above that bound. The comment step passes a threshold of 1. Every row upper is at least 1, so a slowed row still drafts. A blank mark in the table is not significant. The job refuses identical base and head commits and exits non-zero when a row is a significant slowdown or is missing samples.
The workflow_run split is required for fork PRs to have write access for
posting comments (GitHub security model). Suite and local reproduce:
Performance Benchmarks (benchmarks/, asv.conf.json,
benchmarks/workflow/Snakefile).
Executable chemfiles pages¶
The chemfiles notebook is the executable Org Babel source of the chemfiles tutorial, for Python only.
Continuous integration (CI) runs scripts/run-chemfiles-notebook.sh from ci_python.yml.
The script tangles the notebook into docs/notebooks/chemfiles_ingress.py, fails if the committed file differs from the new tangle, and runs it with python3.
The Rust blocks and the blocks in other languages on the chemfiles tutorial page are prose only, and CI does not run them.
Contributors MUST NOT edit the tangled .py file by hand.
Change the notebook Org source instead, and tangle it again with READCON_TANGLE_UPDATE=1 scripts/run-chemfiles-notebook.sh.
Release process¶
This section is the only guide a new contributor needs to cut a release
without reading CI YAML. Three layers work together. Use all of them.
Maintainers MUST NOT short-circuit with a local-only cargo publish unless
CI secrets are broken (fallback is documented below).
Mental model (new contributor)¶
scripts/release-prep.sh X.Y.Z
│ prek, pixi docbld + lychee, version bump, towncrier CHANGELOG,
│ cxx-dist gate; cbindgen optional
▼
commit: maint: bump to vX.Y.Z ──► open Pull Request to main
│ │
│ cargo-dist workflow "Release"
│ runs `dist plan` on the PR (validate)
▼ ▼
merge PR to main ──► git tag -s vX.Y.Z && git push origin vX.Y.Z
│
├─► workflow "Release" (cargo-dist): CLI archives + GitHub Release
├─► workflow "Publish to crates.io": cargo publish --locked
│ needs repo secret CARGO_REGISTRY_TOKEN
├─► workflow "Python wheels": maturin matrix → PyPI (OIDC env `pypi`)
├─► workflow "cxx source tarball": v* tag, upload after the Release exists
└─► workflow "C ABI library tarball": v* tag (or dispatch tag later)
Layer |
What |
Workflow / tool |
|---|---|---|
Prep |
Hooks + docs + versions + changelog |
|
Release PR |
Validate dist plan before a tag exists |
|
Tag publish |
crates.io |
|
Tag publish |
PyPI wheels |
|
Tag publish |
GitHub Release + CLI tarballs |
same cargo-dist |
Tag publish |
C/C++ source tarballs |
|
Tag publish |
Prebuilt C ABI tarballs |
|
Attach prebuilt C ABI to an existing tag¶
.github/workflows/c_lib_tarball.yml matches cxx_tarball.yml. Both
start on a v* tag push, with crates.io, cargo-dist, and wheels. They
wait until the GitHub Release exists, then upload archives that are not
already on the release with gh release upload --clobber. They do not
create the release and do not move the tag. workflow_dispatch input
tag backfills an existing release the same way and does not publish
crates or wheels. Use that to put readcon-core-clib-$VER-$target.tar.gz
on a tag that predates the workflow (for example v0.14.7):
gh workflow run cxx_tarball.yml --repo lode-org/readcon-core --ref main -f tag=v0.14.7
gh workflow run c_lib_tarball.yml --repo lode-org/readcon-core --ref main -f tag=v0.14.7
Open Actions → C ABI library tarball.
Run the workflow from a branch that has
scripts/package-clib.sh(usuallymainafter this lands).Set
tagto the existing Git tag (v0.14.7).
The packager comes from the selected workflow ref. Sources and the
GitHub Release come from inputs.tag. Windows + chemfiles is an
explicit skip (Python wheel path only).
The uploader skips an asset whose sha256 already matches. It does not
replace an asset whose bytes differ. On a v* tag push that mismatch is
an error and the published file stays. On a workflow_dispatch backfill
the same mismatch is a ::warning:: and the published file stays, so the
job can still attach an asset that is missing.
Windows clib builds pass /Brepro to the MSVC linker and remap the crate
root and the cargo home out of the binary. The Windows job builds the
tarball twice from a clean target directory and fails when the sha256
differs. Linux and macOS pack those same libraries a second time and fail
the same check. scripts/reproducible-tar.sh dates members with
SOURCE_DATE_EPOCH, owns them as uid 0, sorts the names, and runs
gzip -n. PDB files are not shipped.
cargo-dist release-PR path¶
Do not push a version tag from a laptop until the version-bump commit is on
main(via PR). That PR is the release PR: it SHOULD contain only themaint: bump to vX.Y.Z(and any intentional release-note / header sync) produced byscripts/release-prep.shor the manual checklist below.On every PR, GitHub Actions runs the workflow named Release (file
release.yml). Withpr-run-mode = "plan"it runsdist planonly: it checks that cargo-dist can announce this version and list artifacts. It does not publish crates or create a GitHub Release on the PR.If the plan job is red, fix
dist-workspace.toml/ version alignment and push. Maintainers MUST NOT tag yet.After the PR is merged, on a clean
main:git checkout main && git pull origin main git tag -s vX.Y.Z -m "vX.Y.Z" git push origin vX.Y.Z
On the tag push, the same
Releaseworkflow switches to publishing mode: builds platform archives for thereadcon-coreCLI binary and creates (or updates) the GitHub Release with those artifacts and notes derived from the changelog.
Regenerate the workflow after editing dist-workspace.toml.
Maintainers MUST NOT hand-edit release.yml as the long-term copy:
# cargo-dist 0.28 from the sha256-pinned archive (scripts/install-cargo-dist-ci.sh)
bash scripts/install-cargo-dist-ci.sh
dist generate --mode=ci
# Re-apply action SHA pins and the checksummed install steps. dist generate
# rewrites release.yml to mutable tags and curl|sh.
git add .github/workflows/release.yml dist-workspace.toml scripts/install-cargo-dist-ci.sh
crates.io publish workflow and CARGO_REGISTRY_TOKEN¶
Workflow file: .github/workflows/crates_publish.yml (name: Publish to crates.io).
Trigger: push of tags matching
v*, orworkflow_dispatchon a branch (usesCargo.tomlpackage version; on av*tag the tag MUST match that version or the job fails fast).Action:
cargo publish --lockedfor thereadcon-corecrate.Idempotent skip: if
https://crates.io/api/v1/crates/readcon-core/<version>already returns 200, the job logs “already published” and does not callcargo publish(avoids “crate already exists” on re-runs).Secret: repository secret exactly named
CARGO_REGISTRY_TOKEN. Value is a crates.io API token for an account allowed to publishreadcon-core(create at https://crates.io/settings/tokens ). Without it, the publish step fails with “please provide a non-empty token” / our explicit empty-secret guard.
Set or rotate the secret (example using pass and gh; run as a repo admin):
# Store offline (example path used by maintainers)
pass insert -m crates-io/readcon-core # paste the crates.io token
# Push into GitHub Actions secrets for lode-org/readcon-core
printf '%s' "$(pass show crates-io/readcon-core)" \
| gh secret set CARGO_REGISTRY_TOKEN --repo lode-org/readcon-core
# Confirm the secret exists (value is never shown)
gh secret list --repo lode-org/readcon-core
Local fallback only if CI is red and users are blocked:
CARGO_REGISTRY_TOKEN="$(pass show crates-io/readcon-core)" cargo publish --locked
Prefer fixing the secret and re-running the workflow over making local publish the default path.
PyPI (wheels) on the same tag¶
python_wheels.yml builds sdist + manylinux/macOS/Windows wheels with maturin
(--features python, not chemfiles by default) and publishes with
pypa/gh-action-pypi-publish using OIDC trusted publishing (environment
pypi). No password secret is required once the trusted publisher is
configured (see Initial PyPI setup below). Transient crates.io network
errors are mitigated with CARGO_HTTP_MULTIPLEXING=false and a one-shot
maturin retry; tag runs are not cancelled mid-flight by concurrency.
Version files¶
The package version is the single line in VERSION at the repository root.
That is the only hand-written copy.
Bump with scripts/bump-version.sh X.Y.Z (or cog bump).
scripts/stamp-version.sh only restamps cogapp copies. It does not write
the towncrier heading or refresh the exported pins.
scripts/bump-version.sh X.Y.Z
# scripts/stamp-version.sh
# restamp only, after VERSION and the heading exist
scripts/bump-version.sh X.Y.Z (also scripts/set-version.sh) writes
VERSION, runs python -m towncrier build --yes --version X.Y.Z, then
scripts/stamp-version.sh. Towncrier is the only writer of ## vX.Y.Z
headings. cog.toml sets disable_changelog = true and its
pre_bump_hooks call bump-version.sh, so a cog bump commit already
contains the heading and the compare links.
CMake reads VERSION and sets CMAKE_CONFIGURE_DEPENDS. Meson passes
files('VERSION') to project()
(https://mesonbuild.com/Reference-manual_functions.html#project).
Sphinx (docs/source/conf.py) reads VERSION at configure time.
The C and Fortran strings, the Python and Julia and Fortran package
files, citation metadata, and the changelog compare links are cogapp
output. The Fortran parameter does not repeat public, because the
module already lists readcon_version. Python __version__ follows
the Cargo package version, which cogapp stamps from VERSION. A
Windows checkout can store a carriage return before the newline in
VERSION. build.rs strips that carriage return. Python reads
VERSION in text mode, which already drops it.
Tool versions are pinned in requirements-version.txt.
Published pages stay on the org-to-rst export (emacs --batch -l docs/export.el
or pixi r -e docs orgbld). Org sources mark each install pin with an
at-sign placeholder named READCON_VERSION. The exporter reads VERSION
and does not take a version argument. Authors MUST NOT type the version into
docs/source/*.rst.
News for a release is a fragment under changelog.d/, collected by
towncrier. The towncrier
config imports readcon_meta.__version__, which reads VERSION.
Authors MUST NOT hand-edit the generated release section of CHANGELOG.md.
Cocogitto still checks commit subjects and can render the git changelog;
it is a separate program from cogapp, so version stamps call
python -m cogapp.
scripts/check_version_sot.sh greps every copy of the version string and
compares it to VERSION. CI runs python -m cogapp --check on the stamp
file list and that script. The cog executable is cocogitto, so the drift
job does not call it.
scripts/release-prep.sh X.Y.Z is the release path: it calls
scripts/bump-version.sh and then the org export.
scripts/set-version.sh is only an alias for bump-version.sh.
Complete release checklist¶
Fast path (preferred):
prek run -a
pixi r -e docs docbld
pixi r -e docs linkcheck
scripts/release-prep.sh X.Y.Z
# review staged files, then:
git commit -m "maint: bump to vX.Y.Z"
# push a branch, open PR → cargo-dist Release workflow runs dist plan
# merge PR to main, then:
git checkout main && git pull
git tag -s vX.Y.Z -m "vX.Y.Z"
git push origin vX.Y.Z
# wait for Release + Publish to crates.io + Python wheels + cxx tarball
Manual equivalent of the script:
Ensure all tests pass and the branch is clean:
cargo test --locked cargo test --locked --features chemfiles # optional; needs libchemfiles/cmake policy pixi r -e python python-test prek run -a pixi r -e docs docbld pixi r -e docs linkcheck
Run
scripts/bump-version.sh X.Y.Z. That writesVERSION, lets towncrier write the release heading, then runs cogapp. Re-export the docs with the org-to-rst pipeline so the published pages match.Check the lock with
cargo test --locked.scripts/bump-version.shalready rewrites thereadcon-coreentry inCargo.lock.Regenerate the README from
readme_src.orgwhen the user-facing docs changed (optional for patch releases that only touch the crate version). Use the project script sodocs/orgmode/*.orglinks stay.org(plainox-mdrewrites them to missing.mdpaths):./scripts/export-readme.shThe release heading comes from towncrier, which
bump-version.shalready ran. Authors MUST NOT rebuildCHANGELOG.mdwithcog changelog. That drops the towncrier marker. Add a fragment underchangelog.d/before the bump. Authors MUST NOT hand-edit the generated release section.If the FFI surface changed, run
scripts/regen-capi-headers.shand commitinclude/readcon-core.h(CIbuild_test.ymlruns--check).Commit the release preparation (
maint: bump to vX.Y.Z) and open the release PR (cargo-dist plans on PRs). After merge:git tag -s vX.Y.Z -m "vX.Y.Z" git push origin vX.Y.Z
Maintainers MUST NOT rely on a local
cargo publishunless CI is red. Prefercrates_publish.ymlwithCARGO_REGISTRY_TOKENset. Local fallback:CARGO_REGISTRY_TOKEN="$(pass show crates-io/readcon-core)" cargo publish --lockedTag CI also runs
python_wheels.yml(PyPI) and cargo-distRelease(CLI tarballs + GitHub Release body from the changelog).Verify all distribution channels:
crates.io shows the new version
PyPI shows the new version (wait for CI)
GitHub Releases has cargo-dist archives (and any prior C ABI tarballs if still attached)
Optional local C ABI tarball for consumers that do not use cargo-dist CLI
archives (headers + libreadcon_core.{a,so} via cargo build --release or
cargo cinstall) can still be attached with gh release upload if needed.
Initial PyPI setup (first release only)¶
For the very first release, PyPI trusted publisher is not yet configured. Publish manually:
# Build wheels
maturin build --release --features python
# Upload (first time, creates the project on PyPI)
uvx twine upload target/wheels/*
Python ships two distributions from python_wheels.yml:
PyPI name |
Features |
Import |
|---|---|---|
|
|
|
|
|
|
Configure a trusted publisher on each PyPI project (same workflow
python_wheels.yml, environment pypi). Users MUST NOT install both wheels in one
venv (both export the readcon module). Users SHOULD run pip install readcon-chemfiles
when selection or import are required. Use lean readcon for CON I/O only.
pip install 'readcon[chemfiles]' depends on readcon-chemfiles==X.Y.Z (pin
updated by scripts/release-prep.sh).
Then configure trusted publisher on PyPI:
Go to https://pypi.org/manage/project/readcon/settings/publishing/
Add a new pending publisher:
Owner:
lode-orgRepository:
readcon-coreWorkflow:
python_wheels.ymlEnvironment:
pypi
After this, all subsequent tagged releases auto-publish via OIDC.
Adding new features¶
Feature gates¶
Optional functionality is gated behind Cargo features:
Feature |
Dependencies |
Purpose |
|---|---|---|
parallel |
rayon |
Multi-frame parallel parse |
rpc |
capnp, capnp-rpc, tokio, etc. |
Cap’n Proto RPC serving |
python |
pyo3 |
Python bindings |
Add new optional features in Cargo.toml under [features].
Adding a new binding¶
Add the binding source under
src/(e.g.,src/python.rs) or as a separate package (e.g.,julia/ReadCon/).Gate behind a feature flag if it adds dependencies.
Add tests under
tests/or the binding’s own test directory.Document in
docs/orgmode/bindings.organddocs/orgmode/howto.org.Add a pixi environment and tasks if applicable.
Updating the C header¶
The C header include/readcon-core.h is the shipped ABI, the same
model as metatensor/metatomic. CMake, Meson, cargo-c, CI, and the
cxx tarball MUST NOT run cbindgen. Edit the header (or optionally
draft with scripts/regen-capi-headers.sh) and commit it with the
FFI change. CI compiles C/C++ against that file and checks symbols
in the built library.
ci_cxx.yml installs no cbindgen and MUST stay green.
The C++ header include/readcon-core.hpp is hand-maintained and MUST
be updated manually to match any changes to CAtom, CFrame, or
the FFI function signatures.