Commit Graph

1 Commits

Author SHA1 Message Date
cba7291676 data: manifest-based incremental refresh of the parquet cache
The cache now tracks every file in the data dir (mtime_ns + size) in
.cache/manifest.json. On load, a directory scan is compared against the
manifest:
  - changed/added files are re-read and merged into the parquet panels
    (one read + one concat + one write per touched panel; new values
    win where present, old values kept where the new file is short)
  - removed files drop their symbols (and names)
  - an up-to-date cache is a ~30 ms memo hit

Measured on the real 4k-symbol set: full build 54 s, refresh of
5 modified + 1 added + 1 removed files 3.4 s. No scan TTL (a scan is
a few ms); a previous 5 s scan cache masked data updates.

Tests: tests/test_data.py (11 checks) added as step 1 of run_tests.sh.
2026-08-24 17:28:20 -04:00