Yahoo's dividend endpoint returns the fund's TOTAL per-share distribution
(dividend + capital gain) for many share classes while the capitalGains
endpoint returns the cap-gain portion separately; summing both double-counted
cap gains and broke pre/post-tax comparability (after-tax engine could beat
the adj-based pre-tax return: impossible).
Add _fix_total_distributions to the bundle-assembly invariants
(layout-agnostic, survives re-downloads): on same-date div>capg events, the
exact Yahoo-adj implied distribution (d = P - Q*A_{t-1}/A_t) must match the
div-file amount (not the sum) on >=3 dates before rewriting div -= capg.
Self-validating: genuine separate same-date distributions are untouched.
Rewrote 17,900 cells on 1,957 symbols.
Pool check (scripts/check_adj_consistency.py): all 14 symbols now
TR(close+events) == TR(adj) to 0.01pt and post < pre with plausible drags.
Tests: rewrite case, no-rewrite case, idempotency (tests/test_data.py).
JLPSX showed a bogus +29.3%/-22.9% 3-day wiggle in Dec 2020: Yahoo dated
the 6.824 year-end cap-gain distribution on the 12-11 record date but the
market went ex-div on 12-14 (close 30.10 -> 23.22), so the raw Adj Close
column pre-applied the adjustment 3 days before the price actually fell.
New 'history' correction op ({date: {col: value}}) patches individual OHLC
cells at bundle assembly (full build, incremental, and correction-changed
recompute paths); check_corrections validates dates/columns; 4 new tests.
scripts/scan_adj_misalign.py finds the artifact set-wide: 481 hits on 278
symbols, overwhelmingly December year-end distributions of value funds
(JLPSX's class of fund). scripts/fix_adj_misalign.py repairs it
arithmetic-only (rescale adj in [event, ex-div) by (1-f); cumulative
returns unchanged, cross-checked implied dist vs the price drop). Applied
to the only curated-fund hit (JLPSX) and its sister class JLPYX (implied
dist 6.824 both = official amount; ex-div 2020-12-14). The remaining ~276
symbols are reported in reports/adj_misalign/scan.md for a bulk run.
Yahoo's 2026 event-feed change (capitalGain events dropped for some
funds; no events at all for terminated tickers) can wipe good event
history on re-download. Defense in depth:
- overrides/event-backup/: last-known-good copy of every dividend/
capitalGain file (8,192 files, 58 MB); refresh with
scripts/backup_events.py after each dump update
- data.py event_file(): frozen > data-root (while populated) > backup;
used by panel reads, verify_official, and the double-listing scanner
- README: capital-gain files are mutual-fund-only in this dump; Yahoo
has no LT/ST split (tax.py taxes capg at lt_rate; the per-fund split
would come from fund-company annual tax statements or commercial feeds)
- tests: exact Timestamp .loc keys (pandas 3.x string matching returns a
Series on large DatetimeIndex)
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
230 tickers whose Yahoo chart responses now come back without a timestamp
array (terminated/merged funds): goget overwrites the .json on every pass
while ohlc.Conv skips the write, leaving the old CSVs as the last known
series. Snapshot them into overrides/frozen/ (git-tracked, audited in
reports/stale-funds.md) and make data.py prefer the frozen copies and
ignore any future data-root rewrite/delete for those symbols, so the
final series survives future goget runs. The cache manifest now covers
the overrides dir too; incremental refresh skips data-root files of
frozen symbols.
- data.py: non-blocking load_bundle(); background watcher thread refreshes
the parquet cache (5s scan, 30s min rebuild cadence); refresh()/
up_to_date()/generation()
- statistics tab: one table per benchmark (vs <label>), plain column names
(beta/alpha/return/vol...), selectable+reorderable stat list in
settings.json
- correlation tab: per-portfolio components-vs-benchmarks +
all-portfolios-vs-benchmarks; numbered columns
- global date range (window radio + start/end boxes) applied to all tabs;
metrics.xcorr(); equity window radio gains YTD/3M/1M
The cache now tracks every file in the data dir (mtime_ns + size) in
.cache/manifest.json. On load, a directory scan is compared against the
manifest:
- changed/added files are re-read and merged into the parquet panels
(one read + one concat + one write per touched panel; new values
win where present, old values kept where the new file is short)
- removed files drop their symbols (and names)
- an up-to-date cache is a ~30 ms memo hit
Measured on the real 4k-symbol set: full build 54 s, refresh of
5 modified + 1 added + 1 removed files 3.4 s. No scan TTL (a scan is
a few ms); a previous 5 s scan cache masked data updates.
Tests: tests/test_data.py (11 checks) added as step 1 of run_tests.sh.
