Yahoo's 2026 event-feed change (capitalGain events dropped for some
funds; no events at all for terminated tickers) can wipe good event
history on re-download. Defense in depth:
- overrides/event-backup/: last-known-good copy of every dividend/
capitalGain file (8,192 files, 58 MB); refresh with
scripts/backup_events.py after each dump update
- data.py event_file(): frozen > data-root (while populated) > backup;
used by panel reads, verify_official, and the double-listing scanner
- README: capital-gain files are mutual-fund-only in this dump; Yahoo
has no LT/ST split (tax.py taxes capg at lt_rate; the per-fund split
would come from fund-company annual tax statements or commercial feeds)
- tests: exact Timestamp .loc keys (pandas 3.x string matching returns a
Series on large DatetimeIndex)
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
- EFTS queries now include 497/497K (many fund families publish their
per-fund highlights there, not in the consolidated N-CSR) and re-rank
hits by registrant name match (ticker/brand words), newest first,
capped at 2 filings per CIK
- new parse_per_share_blocks for the JPMorgan-style 'Per share operating
performance' table (per-class value blocks; dashes = zero)
- parse_highlights now tolerates row labels split across table cells
(modernized N-CSRS format, e.g. Calamos 2026)
- region finders: word-flexible name patterns (US vs U.S., class letters),
self-validating per-share regions (a candidate block must match the
local series, so a name mention in notes doesn't attribute another
fund's tables in a combined 58 MB report)
- main() keeps the best result across candidate filings (N-CSRS vs 497
can round differently) and stops early on 'ok'
- local_series applies the corrections overlay so corrected funds verify
against their filing
Results: JLPSX and CVSIX now 'ok' (all bounded fiscal years agree with
the official filings); CVSIX also gets a 2023-12-21 0.510 capital-gain
correction. bnd/pmaix still ok (no regression).
scripts/verify_official.py locates each fund's latest N-CSR/N-CSRS/10-K/10-Q
via EDGAR full-text search (full fund-name phrase first, then ticker +
name words, then bare ticker), extracts the fund's Financial Highlights
tables (both Vanguard-style and Victory-style layouts, calendar and
non-calendar fiscal years, M/D/YY and month-name headers), matches the
share class by per-share distribution series + NAV magnitude, and
compares per period window against the local Yahoo CSVs (frozen copies
for stale tickers). Per-symbol JSONs + SUMMARY.md land in
reports/xcheck_official/; results are cached per fund.
Run on the 29 curated funds: 8 ok (exact to 3dp, e.g. VTSAX 2021-2026H1),
1 mismatch (CVSIX FY2009: local 1.104 vs official 0.81 - the Yahoo
2008-12-18 row of 0.292 looks spurious), 3 weak-match (uncovered doc
formats, e.g. Leuthold), 17 not-found (mostly ETF families whose
10-K layouts aren't covered yet).
