Greg Pomerantz
1f5720d7db
- EFTS queries now include 497/497K (many fund families publish their per-fund highlights there, not in the consolidated N-CSR) and re-rank hits by registrant name match (ticker/brand words), newest first, capped at 2 filings per CIK - new parse_per_share_blocks for the JPMorgan-style 'Per share operating performance' table (per-class value blocks; dashes = zero) - parse_highlights now tolerates row labels split across table cells (modernized N-CSRS format, e.g. Calamos 2026) - region finders: word-flexible name patterns (US vs U.S., class letters), self-validating per-share regions (a candidate block must match the local series, so a name mention in notes doesn't attribute another fund's tables in a combined 58 MB report) - main() keeps the best result across candidate filings (N-CSRS vs 497 can round differently) and stops early on 'ok' - local_series applies the corrections overlay so corrected funds verify against their filing Results: JLPSX and CVSIX now 'ok' (all bounded fiscal years agree with the official filings); CVSIX also gets a 2023-12-21 0.510 capital-gain correction. bnd/pmaix still ok (no regression).
110 lines
6.9 KiB
Markdown
110 lines
6.9 KiB
Markdown
# Stock & Portfolio Analyzer
|
||
|
||
Interactive tool for analyzing individual securities and portfolios
|
||
against local Yahoo Finance dumps (`~/prog/fin/stocks`, ~4k symbols).
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
python3 -m venv .venv
|
||
.venv/bin/pip install -r requirements.txt
|
||
./run.sh # serves the UI on the fixed port 8599 (http://localhost:8599)
|
||
```
|
||
|
||
First run builds a parquet cache in `.cache/` (~1 min for 4k symbols);
|
||
later runs load in well under a second. The cache tracks the data dir
|
||
per-file (mtime + size in `.cache/manifest.json`), so when the download
|
||
is updated, only the changed/added/removed symbols are re-read — a
|
||
partial refresh takes seconds instead of a full ~1 min rebuild.
|
||
|
||
## Modules
|
||
|
||
| Module | Purpose |
|
||
|-----------------|---------|
|
||
| `data.py` | Ingest `{sym}-history/dividend/capitalGain.csv` -> cached parquet panels (date x symbol), with manifest-based incremental refresh when the data dir changes. `Adj Close` already includes distributions, so it drives pre-tax total returns. |
|
||
| `metrics.py` | Total/annualized return, vol, Sharpe, Sortino, max drawdown, Calmar, CAPM beta/alpha. Pure pandas, all transparent. |
|
||
| `portfolio.py` | Weighted portfolios with drift and periodic rebalancing to target weights (`1W/1ME/QE/YE`), one-way cost in bps. Spec grammar: commas join the elements of ONE portfolio (`SYM` or `SYM:w`, bare = equal weight), spaces separate DISTINCT symbols/portfolios (`parse_items`). |
|
||
| `tax.py` | Simplified DAS after-tax engine: FIFO lots, 365-day long/short split, separate LT/ST/dividend rates. Headline curve = what you keep if you **sell everything today** (unrealized gains taxed daily by lot age). |
|
||
| `chart_widget.py` | Self-contained plotly.js chart in an iframe: mouse zoom/pan, x clamped to the data, view edges snapped to first/last data points with day-precise labels, y tight-fit, every line re-based to 1.0 at the left edge. |
|
||
| `portfolios.py` | Saved portfolio definitions in `portfolios.json` (name, spec, scheme, cost). |
|
||
| `settings.json` | Persisted UI inputs (symbol/benchmark specs, scheme, costs, tax rates, period, curve/window mode) — restored on every page load and server restart; delete to reset. |
|
||
| `app.py` | Streamlit UI: single "symbol or portfolio" spec field (page updates as soon as the input is valid; unknown symbols get click-to-fix "did you mean" suggestions) + a benchmark box with the same grammar (one benchmark per line; a line is a single symbol or a comma-joined portfolio, simulated with the same scheme/cost/tax rules — pre- and after-tax curves, first one drives beta/alpha), scheme/costs/tax rates, save + load/compare/delete portfolios (overlaid pre/after-tax curves), curve toggle (both / pre-tax only / after-tax only), stats table, allocation, per-year tax detail. |
|
||
|
||
## Data verification (Tier 3: against official filings)
|
||
|
||
`~/prog/fin/stocks` is a Yahoo dump and gets re-downloaded (overwritten),
|
||
so fixes must live outside it. Pipeline:
|
||
|
||
1. `scripts/audit_stale.py` — finds tickers whose latest history row is
|
||
old and whose fresh Yahoo download is empty (delisted/merged funds,
|
||
tickers Yahoo no longer serves). Snapshots their final
|
||
`{history,dividend,capitalGain}.csv` + `longName` into
|
||
`overrides/frozen/{SYM}.{ext|json}`. `data.py` shadows the data root
|
||
with the frozen copies, so a re-download can't clobber them.
|
||
2. `scripts/verify_official.py [syms | --stale]` — finds the fund's
|
||
shareholder report (EDGAR EFTS for `"TICKER"`, forms N-CSR/N-CSRS/
|
||
N-14/N-2/497) and parses the per-class *Financial Highlights* (or
|
||
JPMorgan-style *Per share operating performance*) tables: period-by-
|
||
period distribution totals compared against local
|
||
dividends + capital-gains over the same windows, plus a spot check of
|
||
the NAV-per-share row against the local close. Verdicts per symbol in
|
||
`reports/xcheck_official/{sym}.json`: **ok** (all bounded fiscal
|
||
years agree), **mismatch** (class matched, some year off — the
|
||
report tells you which), **weak-match** (best class fit too poor to
|
||
trust), **not-found** (fund not in any candidate filing). The local
|
||
side applies the corrections overlay, so a corrected fund verifies
|
||
against its filing. When several filings parse (e.g. the 497 annual
|
||
and the N-CSRS, which can round differently), the best match wins.
|
||
3. Confirmed findings go into `overrides/corrections/{SYM}.json` as
|
||
auditable deltas (`remove`/`replace`/`add` of distribution rows, each
|
||
entry dated and valued — amounts are what Yahoo reports, not the
|
||
official filing's). `data.py` applies them on cache build. Examples:
|
||
- **CVSIX** — 2008-12-18 0.292 duplicate of the same day's 0.641
|
||
(official FY09 = 0.81 balances without it); 2023-12-21 0.510
|
||
capital-gain row duplicated next to the day's 0.691 dividend
|
||
(official FY2024-10 = 0.79 balances without it).
|
||
- **JLPSX** — 13 year-end capital-gain distributions duplicated into
|
||
the dividend file (FY2021–2025 all match the JPMorgan 497 after
|
||
removal).
|
||
|
||
Known uncorrected Yahoo discrepancies (no per-fund official per-date
|
||
schedule to pick between same-date conflicting amounts — left for
|
||
manual review): FAEVX, FGIZX, FZAGX, FIKAX, GDEUX, GSOUX, OTCRX, SHXIX
|
||
(details in `reports/xcheck_official/*.json`).
|
||
|
||
## Development
|
||
|
||
- **Run**: `./run.sh` → http://localhost:8599 (fixed port; no-ops if a
|
||
server is already running). The chart loads plotly.js from a CDN; for
|
||
fully offline use set `F_INLINE_PLOTLY=1` in `run.sh`.
|
||
- **Test**: `./run_tests.sh`
|
||
1. `tests/test_app.py` — app-level tests via Streamlit AppTest (no
|
||
browser). Memory: one data bundle is ~2.3 GB, so this process keeps
|
||
at most ONE AppTest alive (see its header comment).
|
||
2. `tests/test_e2e_browser.py` — Playwright + headless Chromium driving
|
||
the real page with real keystrokes; needs the server running on 8599.
|
||
One-time setup: `.venv/bin/pip install playwright` and
|
||
`.venv/bin/python -m playwright install chromium`.
|
||
- **Gotchas**
|
||
- Streamlit caches imported modules per process: **restart the server**
|
||
after editing any `.py` (kill the old one first — `run.sh` refuses to
|
||
double-start).
|
||
- `st.cache_data` caches the portfolio + tax simulations: they recompute
|
||
only when symbols/scheme/cost/tax rates change, not on window or curve
|
||
toggles.
|
||
- `settings.json` (gitignored) persists UI inputs across reloads and
|
||
restarts; delete it to reset. Saved portfolios live in `portfolios.json`.
|
||
- Data cache: `.cache/*.parquet`; rebuild via the sidebar checkbox
|
||
(first build ~1 min for ~4k symbols).
|
||
|
||
## Known simplifications (roadmap)
|
||
|
||
- No loss carryover or carryforward across years; no wash-sale rules.
|
||
- Distributed capital gains taxed entirely at the long-term rate.
|
||
- Single (federal-like) tax bracket; no state taxes, no AMT.
|
||
- Equal treatment of benchmark for beta/alpha (CAPM, rf = 0 by default).
|
||
|
||
Ideas: vectorbt sweeps over rebalance schemes, NiceGUI/Textual frontend,
|
||
empyrical-reloaded metrics, monthly (not yearly) loss netting, tax-loss
|
||
harvesting simulation.
|