Greg Pomerantz
e20b2e30cb
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
75 lines
1.3 KiB
JSON
75 lines
1.3 KiB
JSON
{
|
|
"symbol": "fcnkx",
|
|
"dedup": [
|
|
[
|
|
"2010-12-17",
|
|
0.0394
|
|
],
|
|
[
|
|
"2011-02-04",
|
|
0.0085
|
|
],
|
|
[
|
|
"2013-02-08",
|
|
0.0745
|
|
],
|
|
[
|
|
"2014-02-07",
|
|
0.097
|
|
],
|
|
[
|
|
"2015-02-06",
|
|
0.095
|
|
],
|
|
[
|
|
"2018-02-09",
|
|
0.1858
|
|
],
|
|
[
|
|
"2018-12-07",
|
|
0.821
|
|
],
|
|
[
|
|
"2019-02-08",
|
|
0.106
|
|
],
|
|
[
|
|
"2019-12-13",
|
|
0.464
|
|
],
|
|
[
|
|
"2020-02-07",
|
|
0.097
|
|
],
|
|
[
|
|
"2020-12-11",
|
|
1.246
|
|
],
|
|
[
|
|
"2021-02-12",
|
|
0.4
|
|
],
|
|
[
|
|
"2021-12-10",
|
|
1.625
|
|
],
|
|
[
|
|
"2022-02-04",
|
|
0.215
|
|
],
|
|
[
|
|
"2023-02-10",
|
|
0.186
|
|
],
|
|
[
|
|
"2025-02-07",
|
|
0.175
|
|
],
|
|
[
|
|
"2025-12-05",
|
|
1.09
|
|
]
|
|
],
|
|
"note": "Bulk correction from the double-listing scan (scripts/scan_double_listing.py): Yahoo listed this distribution in both the dividend and capitalGains event files (same date, same amount); one copy is spurious. The 'dedup' invariant keeps at most one copy, layout-agnostically. Mechanism verified against official filings for JLPSX/GDEUX/GSOUX/FAEVX/CVSIX/FZAGX; mechanism-inferred for this symbol (no per-fund official check). Revert the entry if a future official cross-check shows both rows were real."
|
|
}
|