- No fixed risk-free rate: the statistics page now nets Sharpe, Sortino
and CAPM alpha against BIL (SPDR 1-3 Month T-Bill) daily total
returns from the data bundle — the same reference the Fund Lab uses
for all its alphas (pre-2007 dates fill 0, as in fundlab.decompose).
- metrics: sharpe/sortino/beta_alpha/summary accept a daily rf SERIES
(or scalar annual rate as before) via a shared excess() helper.
- Also fixes a latent double-count: the old scalar-rf alpha subtracted
rf twice (once in the returns, once in the intercept term); alpha is
now mean(excess fund) - beta * mean(excess bench), the standard CAPM
intercept on excess returns.
- Page caption states the T-bill reference (or warns if BIL is absent).
- Tests: series-rf identities (sharpe/sortino/beta/alpha), benchmark
self-row beta 1 / alpha 0, caption check.
- Pass the benchmark's DAILY price series to metrics.summary() instead of
a month-end-resampled one; summary() derives daily returns and
annualizes with 252d, so the old resample made beta/alpha regress the
fund's month-end daily returns against whole-month benchmark returns
(and skewed ann_return_bench).
- New sidebar setting 'Risk-free rate %' (default 4%, persisted) passed
through to summary(), so Sharpe, Sortino and CAPM alpha are computed
in excess of rf; noted in the page caption and stats help.
The full rate stack (federal + NIIT + state) was used for the recorded
tax columns and for realized gains at rebalances, but the equity path
itself deducted only the FEDERAL rate in two places:
1. distribution reinvestment kept (1 - div_rate)/(1 - lt_rate) instead of
(1 - div_rate - niit - sl_rate);
2. the per-date liquidation tax ('sell everything today') passed bare
(lt_rate, st_rate) to _liquidation_tax.
Symptom: FLCSX 10Y showed a 1.4pt after-tax drag instead of the true
~2.8pt. Fix: d_keep/c_keep factors and stacked liquidation rates.
Regression tests pin the equity path: reinvested net with NIIT+state,
ST and LT liquidation tax at the full stack (all fail on the old code).
Corrected 2015-2026 NYC after-tax: SPY 13.82->11.23, IVV 13.81->11.22,
JLPSX 13.68->9.18 (previously reported 12.51/12.50/11.31, superseded).
Yahoo's dividend endpoint returns the fund's TOTAL per-share distribution
(dividend + capital gain) for many share classes while the capitalGains
endpoint returns the cap-gain portion separately; summing both double-counted
cap gains and broke pre/post-tax comparability (after-tax engine could beat
the adj-based pre-tax return: impossible).
Add _fix_total_distributions to the bundle-assembly invariants
(layout-agnostic, survives re-downloads): on same-date div>capg events, the
exact Yahoo-adj implied distribution (d = P - Q*A_{t-1}/A_t) must match the
div-file amount (not the sum) on >=3 dates before rewriting div -= capg.
Self-validating: genuine separate same-date distributions are untouched.
Rewrote 17,900 cells on 1,957 symbols.
Pool check (scripts/check_adj_consistency.py): all 14 symbols now
TR(close+events) == TR(adj) to 0.01pt and post < pre with plausible drags.
Tests: rewrite case, no-rewrite case, idempotency (tests/test_data.py).
NYC residents pay, on a capital-gain distribution, roughly 20% federal
+ 3.8% NIIT + 6.85-9.65% NY + 3.876% NYC = ~34-37% — the old model's
flat 20% understated the real after-tax drag of high-distribution
funds for this user by ~15 points on exactly the flows that matter.
tax.py: new niit + sl_rate params (decimals). sl_rate is the
state+local marginal rate applied at ORDINARY rates to every flow —
state and local have NO preferential cap-gain rate, so the composite
is lt_rate+niit+sl_rate on cap-gain dists, div_rate+niit+sl_rate on
dividends, and (st/lt_rate)+niit+sl_rate on realized gains.
app.py: two new sidebar fields (persisted in settings.json), wired
through _compute_portfolio's cache key.
tests/test_tax.py: 3 new cases (capg and div composite rates,
net-taxed realized ST gains/losses at st+NIIT+SL).
README: NYC rate note with the 2025 IT-201 schedule values (NYC 3.876%
over $50k; NY 6.85% at $215,400-$1.077M, 9.65% at $1.077M-$5M,
single filer) and the composite example.
The old model valued holdings in ADJ (total-return index) units but
computed distribution flows as raw per-share dollars — so every
distribution, and its tax, was overstated by the raw/adj ratio
(JLPSX: 30.10/11.25 = 2.7x; the Dec-2020 cap-gain tax showed as 12.1%
of the account instead of the true 4.5%). The wiggle in the after-tax
curve was this bug, not a convention issue.
tax.py now:
- values holdings in RAW share units on close prices (bundle.close);
- receives the per-share distribution on its event date, pays the tax
(recorded in TaxResult.taxes), and reinvests the after-tax remainder
at the same day's raw close — the tax's effect lives in the (smaller)
reinvested units and is NOT also deducted from cash (double-count
caught and fixed in review);
- recomputes market value after the reinvestment so equity[t] is the
post-event liquidation value.
With the fix, the 'as-if-liquidated' equity on JLPSX's ex-div day drops
by exactly the true tax cost (4.72% vs 12.1% before); the -22.9% price
drop is offset by the distribution kept.
Also:
- app.py passes bundle.close to the after-tax model (pre-tax
portfolio_returns still uses adj);
- JLPSX/JLPYX: the 2020-12-11 6.824 capital-gain distribution is moved
to the true ex-div date 2020-12-14 (remove/add correction ops), so
the reinvestment prices at the post-drop close;
- tests/test_tax.py: 8 synthetic regression tests (tax magnitude,
reinvestment MV, no double-count, ex-div equity step, per-component
rates); run_tests.sh now runs it.
JLPSX showed a bogus +29.3%/-22.9% 3-day wiggle in Dec 2020: Yahoo dated
the 6.824 year-end cap-gain distribution on the 12-11 record date but the
market went ex-div on 12-14 (close 30.10 -> 23.22), so the raw Adj Close
column pre-applied the adjustment 3 days before the price actually fell.
New 'history' correction op ({date: {col: value}}) patches individual OHLC
cells at bundle assembly (full build, incremental, and correction-changed
recompute paths); check_corrections validates dates/columns; 4 new tests.
scripts/scan_adj_misalign.py finds the artifact set-wide: 481 hits on 278
symbols, overwhelmingly December year-end distributions of value funds
(JLPSX's class of fund). scripts/fix_adj_misalign.py repairs it
arithmetic-only (rescale adj in [event, ex-div) by (1-f); cumulative
returns unchanged, cross-checked implied dist vs the price drop). Applied
to the only curated-fund hit (JLPSX) and its sister class JLPYX (implied
dist 6.824 both = official amount; ex-div 2020-12-14). The remaining ~276
symbols are reported in reports/adj_misalign/scan.md for a bulk run.
Yahoo's 2026 event-feed change (capitalGain events dropped for some
funds; no events at all for terminated tickers) can wipe good event
history on re-download. Defense in depth:
- overrides/event-backup/: last-known-good copy of every dividend/
capitalGain file (8,192 files, 58 MB); refresh with
scripts/backup_events.py after each dump update
- data.py event_file(): frozen > data-root (while populated) > backup;
used by panel reads, verify_official, and the double-listing scanner
- README: capital-gain files are mutual-fund-only in this dump; Yahoo
has no LT/ST split (tax.py taxes capg at lt_rate; the per-fund split
would come from fund-company annual tax statements or commercial feeds)
- tests: exact Timestamp .loc keys (pandas 3.x string matching returns a
Series on large DatetimeIndex)
Yahoo changed the shape of its event feed between downloads: a 2026-08
re-download of CVIX/JLPSX shows it no longer returns capitalGain events at
all (the dividend stream still carries the year-end rows), while stale
tickers now return no events at all. File-specific remove ops therefore
break silently on the next re-download, so the double-listing fix is now
expressed as layout-agnostic invariants applied at bundle assembly
(idempotent, hold for full and incremental builds):
dedup: [[date, amount]] keep at most one copy of a same-date/
same-amount cross-file pair (the capitalGain copy when both present)
drop_capg_copy: [date] the capitalGain row on that date is the
spurious copy of the dividend row
- data.py: apply_invariants() at bundle assembly + pure dedupe_event_rows()
shared with verify_official and tests (8 new test cases, 22 passing)
- JLPSX/CVSIX corrections rewritten with the invariants (2019-08-08 now
keeps the dividend amount per the verified same-date pattern)
- scripts/scan_double_listing.py: whole-dump sweep -> reports/double_listing/
6,421 symbols scanned: 1,218 with same-amount pairs (6,589), 1,631 with
differing-amount pairs (12,366, reported only - not distinguishable from
legitimate same-day div+capg without per-fund official data), 83 with
repeated within-file rows (ingest keep-last already collapses them)
- scripts/apply_dedup_corrections.py: bulk 'dedup' corrections for the
1,218 same-amount symbols (1,212 new files; the 6 verified funds keep
their explicit, official-verified corrections)
- scripts/check_corrections.py: integrity check for every correction op
against the actual (frozen) files - caught a mis-filed CVSIX entry
230 tickers whose Yahoo chart responses now come back without a timestamp
array (terminated/merged funds): goget overwrites the .json on every pass
while ohlc.Conv skips the write, leaving the old CSVs as the last known
series. Snapshot them into overrides/frozen/ (git-tracked, audited in
reports/stale-funds.md) and make data.py prefer the frozen copies and
ignore any future data-root rewrite/delete for those symbols, so the
final series survives future goget runs. The cache manifest now covers
the overrides dir too; incremental refresh skips data-root files of
frozen symbols.
Joint backtracking placement for CEF tables that drop zero columns;
NAV-identity fallback when the printed total row is short; 4 fund-
family layouts verified (Franklin/classic/abrdn/Korea). 16/50 of the
CEF shortlist now have a verified div/gains/ROC split + FY-end
discount series. 17 new parser unit tests (123/123 green).
appreciation - a taxable-account feature, not an IRA one
Correction after user pushback. The earlier note "ROC does not help
the taxable case - the deferral replicates the IRA" was WRONG: a
traditional IRA defers to the ORDINARY rate at withdrawal, while ROC
in a taxable account defers to the LTCG rate on a >1y sale (the
distribution is basis-reducing and reappears inside the shareholder's
own capital gain). Under the premise LTCG rate < future ordinary
rate, ROC - like NAV appreciation - favors the taxable account.
Contrast ordinary income (interest, ordinary divs, STCG): taxed at
the ordinary rate in BOTH accounts, so only the IRA's deferral wins.
taxplan.py:
- _deferred_share(): per fund, share of 5y total return that defers
to the investor = (NAV change + ROC) / total return, from the
parsed per-share N-CSR table, max'd with the taxsplit appreciation
share.
- >= 50% deferred -> location "TAXABLE (defers to LTCG)" (renamed
from "TAXABLE (accrues)"); 10-50% ROC in distributions -> note.
- Merger-arb cap 0.35 -> 0.50: HMEZX's per-share table (52% NII /
30% gains / 18% ROC over 5y) refutes "mostly STCG" - HMEZX/MERVX
are now MIXED (check 1099), not clean IRA.
- RESEARCH.md: corrected placement write-up, incl. the distinction
between the tax question and the fund-quality question (heavy ROC
can mean principal erosion - PGSIX NAV -34%/5y - which affects
selection, not the optimal account).
App: order dict + filter updated to the new location name.
Tests: merger-arb expectation updated to MIXED; 2 new checks for
the ROC upgrade/note logic. 99/99 fundlab + 32/32 app.
fundlab/taxplan.py categorizes the 16-fund shortlist, the 22 N-PORT
cross-checked candidates, and all 250 screened candidates by the
expected CHARACTER of their distributions, given the user's premise
that the current LTCG rate < the post-retirement ordinary rate:
qualified div + LTCG -> TAXABLE (score >= 0.60)
tax-exempt (munis) -> TAXABLE
ordinary / STCG / REIT -> IRA (score <= 0.35)
in between -> MIXED (pull the 1099-DIV)
cash -> FLEXIBLE
score = estimated share of distributions that are tax-favorable,
from three tiers of ground truth: N-PORT keyword buckets (16), SEC
assetCat/issuerCat buckets (22), sleeve loadings (250), with a
sleeve fallback when the keyword parser left >50% of a book
unclassified, and a manual override for the Leuthold wrappers
(91.7% Leuthold Core ETF, no return history yet).
Key findings:
- shortlist: TAXABLE = ATESX, JLPSX, LAMHX, LCORX, LCRIX (equity);
IRA = ATRFX, COSIX, CVSIX, PMORX, SVARX, EAGMX/EGRSX;
MIXED = MBXIX, QSPNX, PMAIX/PMFKX (same fund, two classes)
- cross-checked: 4 munis -> TAXABLE; HMEZX + MERVX are the merger-
arb trap - equity-looking books whose distributions are mostly
SHORT-TERM gains -> IRA
- candidates: 109 munis TAXABLE, 127 IRA, 6 equity TAXABLE, 7 MIXED
App: Fund Lab "Tax location" expander. Output:
fundlab/taxplan_results.json. Tests: test_taxplan() (9 checks).
97/32 suites green.
fundlab/drawdown.py detects the severe equity drawdown scenarios from
the index (IVV) rather than hard-coding them: one worst peak->trough
per calendar year since 2022, min depth 8% (a 10% floor would silently
drop the 2023 rate shock at -9.9% and the 2024 Aug-5 dip at -8.4%).
Detected: 2022 bear mkt (-24.5%), 2023 rate shock (-9.9%), 2024 vol
spike (-8.4%), 2025 tariff crash (-18.8%), 2026 Q1 drawdown (-8.9%).
For each of the 2,384 screened funds it computes that fund's own-NAV
return over each peak->trough window (first print after the peak to
the last print on/before the trough) and ranks the 250 CANDIDATEs by
# scenarios positive.
Key finding: positive in all 5 scenarios = only 7 funds, all
ultra-short/cash (BILS, QCMMRX, PULS, FHCOX, FHMIX, SAFEX, COIAX).
Drawdown resilience at the top tier is a duration property, not alpha.
The interesting tier is 4/5 WITH real 5y alpha: HMEZX merger arb
(+1.5% 2022, +3.1% 2023, t5 +7.1), MERVX, CBHCX market-neutral, SCFZX
securitized credit (t5 +8.4), ENIAX (t5 +10.1), WMNUX (t5 +6.9), RCTIX.
App: Fund Lab "Drawdown resilience" expander (scenario table +
candidate table). Output: fundlab/drawdown_results.json.
Tests: test_drawdown() added (4 checks). 88/32 suites green.
fundlab/xcheck.py - for each screen candidate, resolve the fund's OWN
registrant CIK (browse-edgar; the 497-cover CIK is the family/trust),
get the exact series name for the ticker (the only reliable
disambiguator between sibling funds), walk the 4 most recent NPORT-P
filing dates, and parse holdings from the interactive NPORT XML
(primary_doc.xml at the accession root - NOT the XSL-rendered view the
submissions API points at). Exact seriesName match > best htm exhibit
parse. Buckets from the authoritative assetCat+issuerCat codes (ABS-O,
ABS-CBDO, DBT+UST/CORP/MUN/NUSS, LON, STIV, RA, EC+RF=fund, ...), not
position-name keywords. Resumable; raw filings cached under
nport_cache/raw/ (gitignored).
nport.py - _SECTION gains the "INVESTMENT PORTFOLIO (unaudited)"
variant (NPORT-EX Sch-F files); find_section/build gain a frac
token-tolerance param (Yahoo names drift from filing names); CMBS/ABS
bucket gains CLO/CBDO/DAC terms.
app Fund Lab - "N-PORT cross-check" expander: per-candidate table
(as-of, n, t5, top code-bucket, #1 position) + per-fund holdings
detail.
RESEARCH.md - cross-check verdicts. 21/22 resolved (qcmmrx is an MMF,
no holdings). The screen's top names are REAL:
- hmezx/mervx = genuine merger arb (equity in deal targets + escrow)
- egrix = 100% wrapper in one macro managed portfolio (underlying not
NPORT-disclosed); etsix = fund of EV internal multi-strat accounts
- wmnux = discounted/zero-coupon corporate bonds + equity swaps (the
"equity names" are bond issuers/swap underlyings)
- scfzx/rctix/aflix = securitized credit/CLO/distressed/levered loans
- hicox/fhmix/usmsx/btmix (munis), aguax/femdx (EM sovereign), anglx
(agency MBS), lpxax (rotated out of prefs into bank/financial debt)
= genuine missing-factor exposures the 35-sleeve model lacks
- fhcox/dultx/safex = short-duration carry (a short-duration sleeve
would explain them)
tests/test_fundlab.py - test_xcheck (14 checks): parse_interactive,
code buckets, name-match normalization, series-name disambiguation.
Also: untrack fundlab/streamlit.log; gitignore raw/ + xcheck_run.log.
84 fundlab / 32 app / 14 data tests pass.
fundlab/RESEARCH.md - running research log: sources that work/die
(full-index = discovery workhorse; browse-edgar JS-dead;
investment-company-tickers.json nonexistent; company_tickers.json
useless for OTC; Yahoo crumb throttled but chart API fine), 13
hard-won learnings (OTC funds report exchange 'Nasdaq' -> use
instrumentType; 497 SGML cover uses UNCLOSED line-based tags ->
parse before tag-stripping; full-index columns drift -> regex the
line; one quarter != universe -> 4-qtr union; accession paths
relative to /Archives/ not /Archives/edgar/data/; family CIKs
repeat -> dedupe by series name; portfolio is 50% MN so MN alpha
funds are 'correlated', not diversifying).
fundlab/overnight.py - resumable all-stage pipeline (kill/restart safe):
verify (Yahoo chart per non-local ticker, 4-thread, 429 backoff,
local tickers measured from CSV row counts) -> select (pure
select_rows: MUTUALFUND, >=5y, one longest-history class per series
name, alpha_name as TAG not filter) -> download (goget in 200-sym
batches) -> screen (streamed, skip-already-done) -> finalize
(verdict counts + candidates the v1 name-filter would have missed).
Universe: 10,372 class tickers -> 10,260 verified -> 2,384 funds
(407 local, 1,977 external; only 54 match the alpha name pattern -
the v2 point is to screen the other 2,330).
app: alpha table now dedupes by sym with search_all.json winning
(comprehensive superset).
tests: select_rows unit tests (ETF drop, short-history drop, class
collapse, name tagging). 70/70 fundlab.
Answer to 'find other alpha-driven funds that complement the portfolio':
- fundlab/search.py: complementarity screen - each fund's daily total
returns vs the same 21 broad sleeve axes (BIC forward selection,
|t|>2), full + 5y; alpha (intercept t), R2, rolling 6m alpha
persistence, correlation vs the current qspnx/pmaix portfolio and the
spy/agg/tlt benchmark mix. Verdict tiers: CANDIDATE (alpha +
persistent + portfolio-corr<0.3) / semi-alpha / alpha-but-correlated /
sleeve mix / weak.
- fundlab/dbmine.py: the actual search universe - the local stocks DB
already holds ~100 US open-end alternatives (AQR, PIMCO, JPM,
Principal, Calamos, GMO, Franklin K2, ...). Name-pattern miner with
share-class family dedupe (keeps the longest-history class).
- fundlab/tickers.py + searchlist.py: external longlist resolution
(chart-API name gate + EDGAR 497 cover tickers). Finding: the famous
multi-strategy/macro names (Millennium, Balyasny, Two Sigma, Winton,
Marshall Wace, Brevan Howard, AQR Event-Driven) are private/offshore
or terminated - not US open-end accessible. Fidelity Multi-Asset
Income (FMSDX) resolved and screens as weak alpha.
- app Fund Lab: 'Alpha search - all screened funds, ranked' table
(80 funds: 13 shortlist + 59 mined + 1 external).
- results (ranked candidates, 5y alpha / t / portfolio-corr):
wmnix Westwood Alt Income +3.8% t6.5 c0.09 | pyaix Payden ARB +3.0%
t4.8 c0.13 | srdax Stone Ridge Div Alts +7.7% t4.2 c0.10 | padqx PGIM
ARB +2.3% t2.4 c0.27 | bxmdx Blackstone Alt MS +3.5% t2.4 c0.30 |
aqmix AQR Mngd Futures +8.0% t2.2 c0.21 | cmnix/gioix semi-alpha.
Key insight: AQR MN / L/S-equity / Vanguard MN show strong alpha but
corr 0.35-0.76 with the portfolio - it is already 50% market-neutral
(qspnx), so more MN is not diversifying.
- tests: 59/59 fundlab (resolver gates, query ladder, family dedupe,
ticker regex), 32/32 app, 14/14 data
- fundlab/nport.py: parse the fund's own category/percentage lines,
as-of date, net assets and dollar-valued positions from N-PORT
schedules of investments (handles per-fund and combined multi-fund
family filings; conservative keyword bucketing of positions)
- fundlab/nport_cache/<sym>.json: parsed snapshots for 15 of 16 funds
(raw SOI HTML kept locally, gitignored; source URLs + filing dates in
nport_manifest.json, md5-verified against EDGAR)
- atesx: no current SOI found (Anchor's recent filings cover the Income
fund) - listed with an honest note
- app.py: new 'Fund Lab' tab - pick a fund, see objective, reported
composition (bar + table), rough keyword buckets, top positions, and
the prospectus strategy excerpt
- tests: parser unit tests (section finding, category regex, buckets)
- extract_strategy(): finds the 'Principal Investment Strategies' /
'main investment strategies' section, scores candidates (strategy prose
+3, Q&A heading +2, TOC -5, risk subheading -5, stop-heading -2),
truncates at the next section heading; falls back to the prose after
the objective sentence when no heading exists
- fundinfo --strategy [--refresh]: populates the strategy field of
funds.json from each fund's EDGAR document
- funds.json now carries objective + strategy for 20 funds (the 9
curated index funds have no strategy: their objective is the strategy)
- edgar.py: SEC FTS + submissions API; strict cover-gate extraction
(name in title position or (TICKER) on the cover; underlying-reference
names like leveraged wrappers rejected); 4-pass fetch (ticker->CIK
filings, name search, annual reports, ticker search); keyword category
classifier. Returns None rather than a wrong fund's objective.
- fundinfo.py CLI: curated -> cached -> EDGAR resolution into funds.json
- funds_curated.json: human-verified objectives for 13 benchmark-pool
funds (iShares/Vanguard family-trust classes the scraper can't reach)
- tests: 26 checks incl. live EDGAR fetch of VTSAX
- data.py: non-blocking load_bundle(); background watcher thread refreshes
the parquet cache (5s scan, 30s min rebuild cadence); refresh()/
up_to_date()/generation()
- statistics tab: one table per benchmark (vs <label>), plain column names
(beta/alpha/return/vol...), selectable+reorderable stat list in
settings.json
- correlation tab: per-portfolio components-vs-benchmarks +
all-portfolios-vs-benchmarks; numbered columns
- global date range (window radio + start/end boxes) applied to all tabs;
metrics.xcorr(); equity window radio gains YTD/3M/1M
The cache now tracks every file in the data dir (mtime_ns + size) in
.cache/manifest.json. On load, a directory scan is compared against the
manifest:
- changed/added files are re-read and merged into the parquet panels
(one read + one concat + one write per touched panel; new values
win where present, old values kept where the new file is short)
- removed files drop their symbols (and names)
- an up-to-date cache is a ~30 ms memo hit
Measured on the real 4k-symbol set: full build 54 s, refresh of
5 modified + 1 added + 1 removed files 3.4 s. No scan TTL (a scan is
a few ms); a previous 5 s scan cache masked data updates.
Tests: tests/test_data.py (11 checks) added as step 1 of run_tests.sh.
- single spec grammar for symbol and benchmark fields: commas join one
portfolio (MSFT:0.6,V:0.4), spaces separate distinct symbols/portfolios;
both fields accept one or many entries
- benchmarks simulated with the same scheme/cost/tax rules; per-benchmark
beta/alpha columns; after-tax benchmark curves
- global Curve mode (pre/after/both) above the tabs; clean names in
single-curve mode
- live updates: field commits on Enter/blur, page recomputes per rerun;
portfolio+tax sims cached (st.cache_data); plotly.js from CDN (4.6MB ->
browser-cached) with F_INLINE_PLOTLY=1 offline fallback
- chart: legend underneath, solid lines, pan sticks to data edges
(width-preserving), zoom edge-clamped
- inputs persist in settings.json across reloads/restarts/devices
- tests: tests/test_app.py (AppTest) + tests/test_e2e_browser.py
(Playwright) via ./run_tests.sh
