The old model valued holdings in ADJ (total-return index) units but
computed distribution flows as raw per-share dollars — so every
distribution, and its tax, was overstated by the raw/adj ratio
(JLPSX: 30.10/11.25 = 2.7x; the Dec-2020 cap-gain tax showed as 12.1%
of the account instead of the true 4.5%). The wiggle in the after-tax
curve was this bug, not a convention issue.
tax.py now:
- values holdings in RAW share units on close prices (bundle.close);
- receives the per-share distribution on its event date, pays the tax
(recorded in TaxResult.taxes), and reinvests the after-tax remainder
at the same day's raw close — the tax's effect lives in the (smaller)
reinvested units and is NOT also deducted from cash (double-count
caught and fixed in review);
- recomputes market value after the reinvestment so equity[t] is the
post-event liquidation value.
With the fix, the 'as-if-liquidated' equity on JLPSX's ex-div day drops
by exactly the true tax cost (4.72% vs 12.1% before); the -22.9% price
drop is offset by the distribution kept.
Also:
- app.py passes bundle.close to the after-tax model (pre-tax
portfolio_returns still uses adj);
- JLPSX/JLPYX: the 2020-12-11 6.824 capital-gain distribution is moved
to the true ex-div date 2020-12-14 (remove/add correction ops), so
the reinvestment prices at the post-drop close;
- tests/test_tax.py: 8 synthetic regression tests (tax magnitude,
reinvestment MV, no double-count, ex-div equity step, per-component
rates); run_tests.sh now runs it.
JLPSX showed a bogus +29.3%/-22.9% 3-day wiggle in Dec 2020: Yahoo dated
the 6.824 year-end cap-gain distribution on the 12-11 record date but the
market went ex-div on 12-14 (close 30.10 -> 23.22), so the raw Adj Close
column pre-applied the adjustment 3 days before the price actually fell.
New 'history' correction op ({date: {col: value}}) patches individual OHLC
cells at bundle assembly (full build, incremental, and correction-changed
recompute paths); check_corrections validates dates/columns; 4 new tests.
scripts/scan_adj_misalign.py finds the artifact set-wide: 481 hits on 278
symbols, overwhelmingly December year-end distributions of value funds
(JLPSX's class of fund). scripts/fix_adj_misalign.py repairs it
arithmetic-only (rescale adj in [event, ex-div) by (1-f); cumulative
returns unchanged, cross-checked implied dist vs the price drop). Applied
to the only curated-fund hit (JLPSX) and its sister class JLPYX (implied
dist 6.824 both = official amount; ex-div 2020-12-14). The remaining ~276
symbols are reported in reports/adj_misalign/scan.md for a bulk run.
Some dump files carry a trailing 'ctime' column: an older pipeline
appended a FULL re-download of the symbol per download (up to ~28
vintages per date, 2019-2022). The consumer keeps the last row per date,
so it silently used stale (~2021) vintages where Yahoo later revised
values. scripts/clean_multivintage.py keeps, per date, the rows from the
newest ctime (event files keep distinct same-day amounts; history/split
keep the single newest row), drops the ctime column, sorts by date.
Applied to the data root and overrides/frozen (which carried the same
artifact): 1,065 files, 2,202,712 stale rows dropped.
Also: stripped 3 UTF-8 BOMs (teg/lo/krft), refreshed the event backup
with the cleaned files, and re-ran the double-listing scan: category C
(83 symbols, 18,636 repeated within-file rows) is fully explained by the
multi-vintage artifact and is now gone; A (6,589 pairs, corrected) and
B (12,366 pairs, open review list) are unchanged. Whole-set structural
QC after cleaning: 0 duplicate dates, 3 syms/15 rows of OHLC invariant
violations, 15 syms/2,897 rows of non-positive prices (mostly
long-delisted tickers), 8 unsorted files (reader sorts).
Yahoo's 2026 event-feed change (capitalGain events dropped for some
funds; no events at all for terminated tickers) can wipe good event
history on re-download. Defense in depth:
- overrides/event-backup/: last-known-good copy of every dividend/
capitalGain file (8,192 files, 58 MB); refresh with
scripts/backup_events.py after each dump update
- data.py event_file(): frozen > data-root (while populated) > backup;
used by panel reads, verify_official, and the double-listing scanner
- README: capital-gain files are mutual-fund-only in this dump; Yahoo
has no LT/ST split (tax.py taxes capg at lt_rate; the per-fund split
would come from fund-company annual tax statements or commercial feeds)
- tests: exact Timestamp .loc keys (pandas 3.x string matching returns a
Series on large DatetimeIndex)
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
Scanning the verified funds for same-date div/capg pairs showed the Yahoo
double-listing mechanism in every one of the 7 'mismatch' stale funds, and
the official fiscal-year totals pin down which amount is true:
- identical-amount pairs (JLPSX, GDEUX, GSOUX, FAEVX, CVSIX, FZAGX): the
year-end distribution is in both files; keep the capitalGain copy,
remove the dividend copy (GDEUX: FY2021-08 1.73 = 1.625+0.115 and
FY2022-08 0.39 = 0.348+0.036 after; JLPSX/GSOUX/FAEVX/CVSIX same)
- differing-amount pairs (OTCRX, SHXIX, FZAGX, FGIZX, CVSIX): the
dividend-file amount is the true one in every officially-verified case
(OTCRX FY2023 0.85 = 0.852488, SHXIX FY2023 1.01 = 1.0078, FZAGX all
four FYs exact); the capitalGain row is the spurious copy
After corrections GDEUX/SHXIX/CVSIX/JLPSX verify 'ok' against their
filings. The same pattern is extended to these funds' pre-2021 history
(marked mechanism-inferred in the correction notes). Remaining residuals:
FAEVX and FGIZX are also missing regular quarterly dividend rows in the
Yahoo dump (official FY totals exceed local even after dedup) - needs the
funds' per-date distribution archives; FIKAX's official extraction is
ambiguous (systematic ~0.12/yr offset = wrong class table in the
500-fund consolidated Fidelity N-CSRS).
- EFTS queries now include 497/497K (many fund families publish their
per-fund highlights there, not in the consolidated N-CSR) and re-rank
hits by registrant name match (ticker/brand words), newest first,
capped at 2 filings per CIK
- new parse_per_share_blocks for the JPMorgan-style 'Per share operating
performance' table (per-class value blocks; dashes = zero)
- parse_highlights now tolerates row labels split across table cells
(modernized N-CSRS format, e.g. Calamos 2026)
- region finders: word-flexible name patterns (US vs U.S., class letters),
self-validating per-share regions (a candidate block must match the
local series, so a name mention in notes doesn't attribute another
fund's tables in a combined 58 MB report)
- main() keeps the best result across candidate filings (N-CSRS vs 497
can round differently) and stops early on 'ok'
- local_series applies the corrections overlay so corrected funds verify
against their filing
Results: JLPSX and CVSIX now 'ok' (all bounded fiscal years agree with
the official filings); CVSIX also gets a 2023-12-21 0.510 capital-gain
correction. bnd/pmaix still ok (no regression).
overrides/corrections/{SYM}.json (git-tracked, with as_of/source/note)
holds remove/replace/add ops on the dividend and capital-gain series.
data.py applies them on top of whatever the data root (or the frozen
snapshot) provides, in both the full build and the incremental refresh
path, and the corrections dir joins the cache manifest so a change
invalidates the cache. A goget re-download of the base CSV can never
clobber a confirmed correction. Format and usage documented in data.py.
230 tickers whose Yahoo chart responses now come back without a timestamp
array (terminated/merged funds): goget overwrites the .json on every pass
while ohlc.Conv skips the write, leaving the old CSVs as the last known
series. Snapshot them into overrides/frozen/ (git-tracked, audited in
reports/stale-funds.md) and make data.py prefer the frozen copies and
ignore any future data-root rewrite/delete for those symbols, so the
final series survives future goget runs. The cache manifest now covers
the overrides dir too; incremental refresh skips data-root files of
frozen symbols.
