Streaming append and random reads ================================= .. note:: Upgrading from 0.2.0: :doc:`/howto/migration`. ``readcon-db`` stores CON text. ``readcon`` (the ``readcon-core`` Python package) parses and writes that text. ASE is a different store. This page is the path from a trajectory file to one stored frame. Create a corpus --------------- A corpus is a directory. Creating ``ConCorpus`` creates it. The directory holds ``data.mdb`` and ``lock.mdb``. One process writes. Many processes may open the same directory with ``readonly=True``. .. code:: python from readcon_db import ConCorpus db = ConCorpus("/tmp/corpus") ro = ConCorpus("/tmp/corpus", readonly=True) Rust is ``ConCorpus::open`` and ``ConCorpus::open_readonly``. The command line is ``readcon-db ingest``. Stream an append ---------------- ``append_trajectory_frames`` commits 64 frames at a time. The first group is a new trajectory. Each later group extends it. A list returned by ``read_con`` is written from the frames the parser already stored, so the append does not build a Python atom for every site. Rust ``ConCorpus::append_trajectory_frames`` still commits the slice you pass in one transaction. The grouping is the Python binding. Random access ------------- ``get_frame(traj_id, frame_idx)`` returns a ``StoredFrame``. ============== ======================================================= Attribute Contents ============== ======================================================= ``xyz`` one ``(x, y, z)`` row per atom ``atoms`` a ``StoredAtoms`` list ``atoms[i]`` one ``StoredAtom`` with ``symbol``, ``x``, ``y``, ``z`` ``cell`` three box lengths ``angles`` three box angles, in degrees ``len(frame)`` the atom count ============== ======================================================= Indexing ``atoms`` builds that atom and no others. ``get_frame_text`` returns the CON text. ``readcon.read_con_string`` on that text returns a ``readcon.ConFrame`` when you need the full object. Compressed inputs ----------------- Python ``append_trajectory`` and Rust ``append_trajectory_path`` read the file as UTF-8. They do not decompress gzip or zstd. Read the compressed file with ``readcon.read_con``, which selects gzip or zstd from the extension, then append the frames: .. code:: python frames = readcon.read_con("traj.con.gz") # or traj.con.zst db.append_trajectory_frames(1, frames, source="traj.con.gz") A ``.con.zst`` file requires a ``readcon`` build with the ``zstd`` feature. Plain ``.con`` files can go straight to ``append_trajectory`` (Python) or ``append_trajectory_path`` (Rust). readcon-core ------------ ``readcon-core`` is the CON parser and writer. The corpus stores the text that parser produces and decodes a hit with the same library. Foreign formats (XYZ, PDB, GRO) enter through ``readcon.read_chemfiles`` on a chemfiles build, then ``append_trajectory_frames``. They do not enter through ASE. ASE --- ``readcon-db`` does not register an ``ase.ioformats`` entry point, and neither does ``readcon``. ASE reads a CON file with its built-in ``eon`` format: .. code:: python import ase.io ase.io.read("traj.con", format="eon") That call does not open a corpus and does not use ``readcon``. ``ase.db`` and ``ase.io.Trajectory`` are the stores a benchmark compares against. They are not how this corpus stores frames. A calculator that needs an ``ase.Atoms`` object can be built from a ``StoredFrame``. ``readcon.ConFrame.to_ase`` applies to a frame parsed by ``readcon``, including one recovered with ``read_con_string(db.get_frame_text(...))``. Example ------- This block is executed by ``python/tests/test_streaming_guide.py``. .. code:: python # runnable import gzip import importlib.metadata as metadata from pathlib import Path import readcon from readcon_db import ConCorpus root = Path("guide") root.mkdir() sites = [readcon.Atom("Cu", float(i), 0.0, 1.0, atom_id=i) for i in range(4)] frame = readcon.ConFrame((12.0, 12.0, 12.0), (90.0, 90.0, 90.0), sites) plain = root / "traj.con" readcon.write_con(str(plain), [frame] * 70) gz = root / "traj.con.gz" gz.write_bytes(gzip.compress(plain.read_bytes())) db = ConCorpus(str(root / "db")) rejected = False try: db.append_trajectory(1, str(gz)) except RuntimeError: rejected = True assert rejected loaded = readcon.read_con(str(gz)) n = db.append_trajectory_frames(1, loaded, source=gz.name) assert n == 70 got = db.get_frame(1, 64) assert type(got).__name__ == "StoredFrame" assert type(got.atoms).__name__ == "StoredAtoms" assert got.atoms[0].symbol == "Cu" assert got.atoms[-1].x == 3.0 assert tuple(got.cell) == (12.0, 12.0, 12.0) text = db.get_frame_text(1, 0) full = readcon.read_con_string(text)[0] assert full.atoms[0].symbol == got.atoms[0].symbol from ase import Atoms atoms = Atoms( symbols=[atom.symbol for atom in got.atoms], positions=got.xyz, cell=list(got.cell), pbc=True, ) assert len(atoms) == len(got) eps = metadata.entry_points() group = eps.select(group="ase.ioformats") if hasattr(eps, "select") else eps.get("ase.ioformats", []) registered = [ep for ep in group if "readcon_db" in ep.value] assert registered == []