Greg Pomerantz
e20b2e30cb
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
46 lines
1.4 KiB
JSON
46 lines
1.4 KiB
JSON
{
|
|
"symbol": "cvsix",
|
|
"note": "Same Yahoo double-listing as JLPSX, expressed as layout-agnostic invariants (a 2026-08 re-download shows Yahoo no longer returns capitalGain events for this fund; the dividend stream is unchanged): 'dedup' entries are same-amount same-date pairs (keep the capitalGain copy when both files carry it); 'drop_capg_copy' dates carry a spurious capitalGain row next to the true dividend row (keep the dividend amount). 2008-12-18 and 2023-12-21 are official-verified: N-CSRS accession 0001104659-26-080185 (period 2026-04-30) - FY2009 = 0.81 = 0.641+0.076+0.050+0.045 (2008) and FY2024-10 = 0.79 = 0.691+0.017+0.042+0.042 (2023); the remaining years follow the same pattern (mechanism-inferred).",
|
|
"verified": [
|
|
"N-CSRS accession 0001104659-26-080185 (CIK 0000826732): FY2025-10 0.25, FY2024-10 0.79, FY2023-10 0.20, FY2022-10 0.03 all match after corrections; NAV 14.63 @ 2021-10-29 exact"
|
|
],
|
|
"dedup": [
|
|
[
|
|
"1997-12-08",
|
|
0.043
|
|
],
|
|
[
|
|
"2000-11-08",
|
|
0.002
|
|
],
|
|
[
|
|
"2001-11-07",
|
|
0.019
|
|
],
|
|
[
|
|
"2002-11-06",
|
|
0.002
|
|
],
|
|
[
|
|
"2003-11-13",
|
|
0.312
|
|
],
|
|
[
|
|
"2004-11-17",
|
|
0.48
|
|
]
|
|
],
|
|
"drop_capg_copy": [
|
|
"1998-12-02",
|
|
"1999-12-29",
|
|
"2008-12-18",
|
|
"2013-12-19",
|
|
"2014-12-18",
|
|
"2015-12-17",
|
|
"2016-12-15",
|
|
"2017-12-14",
|
|
"2018-12-20",
|
|
"2023-12-21"
|
|
]
|
|
}
|