Greg Pomerantz
e20b2e30cb
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
67 lines
2.6 KiB
Python
67 lines
2.6 KiB
Python
#!/usr/bin/env python3
|
|
"""Bulk-apply 'dedup' corrections for the same-date/same-amount cross-file
|
|
double-listings found by scripts/scan_double_listing.py.
|
|
|
|
Mechanism (verified against official filings for JLPSX, GDEUX, GSOUX,
|
|
FAEVX, CVSIX, FZAGX): Yahoo lists the (usually year-end) distribution in
|
|
both the dividend and capitalGains event maps; one copy is spurious. The
|
|
'dedup' invariant keeps at most one copy, layout-agnostically.
|
|
|
|
Skips symbols that already have a correction file (their pairs are handled
|
|
by explicit, officially-verified ops). Read the scan report first
|
|
(reports/double_listing/scan.md) — this applies the mechanism-inferred
|
|
correction to every symbol with at least one same-amount pair.
|
|
|
|
Usage: python3 scripts/apply_dedup_corrections.py [--dry-run]
|
|
"""
|
|
import argparse
|
|
import json
|
|
import subprocess
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
ROOT = Path(__file__).resolve().parent.parent
|
|
CORR = ROOT / "overrides" / "corrections"
|
|
SCAN = ROOT / "reports" / "double_listing" / "scan.json"
|
|
|
|
NOTE = ("Bulk correction from the double-listing scan "
|
|
"(scripts/scan_double_listing.py): Yahoo listed this distribution in "
|
|
"both the dividend and capitalGains event files (same date, same "
|
|
"amount); one copy is spurious. The 'dedup' invariant keeps at most "
|
|
"one copy, layout-agnostically. Mechanism verified against official "
|
|
"filings for JLPSX/GDEUX/GSOUX/FAEVX/CVSIX/FZAGX; mechanism-inferred "
|
|
"for this symbol (no per-fund official check). Revert the entry if a "
|
|
"future official cross-check shows both rows were real.")
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--dry-run", action="store_true")
|
|
args = ap.parse_args()
|
|
|
|
scan = json.loads(SCAN.read_text())
|
|
written = skipped = 0
|
|
for sym, r in sorted(scan["symbols"].items()):
|
|
pairs = r.get("same_amt") or []
|
|
if not pairs:
|
|
continue
|
|
f = CORR / f"{sym.upper()}.json"
|
|
if f.exists():
|
|
skipped += 1
|
|
continue
|
|
entry = {"symbol": sym, "dedup": [[x["date"], x["amount"]] for x in pairs],
|
|
"note": NOTE}
|
|
if not args.dry_run:
|
|
f.write_text(json.dumps(entry, indent=2) + "\n")
|
|
written += 1
|
|
print(f"{'would write' if args.dry_run else 'wrote'} {written} correction files, "
|
|
f"skipped {skipped} (existing corrections)")
|
|
if not args.dry_run:
|
|
subprocess.run([sys.executable, str(ROOT / "scripts" / "check_corrections.py")],
|
|
check=False)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|