Greg Pomerantz
cba7291676
The cache now tracks every file in the data dir (mtime_ns + size) in
.cache/manifest.json. On load, a directory scan is compared against the
manifest:
- changed/added files are re-read and merged into the parquet panels
(one read + one concat + one write per touched panel; new values
win where present, old values kept where the new file is short)
- removed files drop their symbols (and names)
- an up-to-date cache is a ~30 ms memo hit
Measured on the real 4k-symbol set: full build 54 s, refresh of
5 modified + 1 added + 1 removed files 3.4 s. No scan TTL (a scan is
a few ms); a previous 5 s scan cache masked data updates.
Tests: tests/test_data.py (11 checks) added as step 1 of run_tests.sh.
68 lines
4.5 KiB
Markdown
68 lines
4.5 KiB
Markdown
# Stock & Portfolio Analyzer
|
|
|
|
Interactive tool for analyzing individual securities and portfolios
|
|
against local Yahoo Finance dumps (`~/prog/fin/stocks`, ~4k symbols).
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
python3 -m venv .venv
|
|
.venv/bin/pip install -r requirements.txt
|
|
./run.sh # serves the UI on the fixed port 8599 (http://localhost:8599)
|
|
```
|
|
|
|
First run builds a parquet cache in `.cache/` (~1 min for 4k symbols);
|
|
later runs load in well under a second. The cache tracks the data dir
|
|
per-file (mtime + size in `.cache/manifest.json`), so when the download
|
|
is updated, only the changed/added/removed symbols are re-read — a
|
|
partial refresh takes seconds instead of a full ~1 min rebuild.
|
|
|
|
## Modules
|
|
|
|
| Module | Purpose |
|
|
|-----------------|---------|
|
|
| `data.py` | Ingest `{sym}-history/dividend/capitalGain.csv` -> cached parquet panels (date x symbol), with manifest-based incremental refresh when the data dir changes. `Adj Close` already includes distributions, so it drives pre-tax total returns. |
|
|
| `metrics.py` | Total/annualized return, vol, Sharpe, Sortino, max drawdown, Calmar, CAPM beta/alpha. Pure pandas, all transparent. |
|
|
| `portfolio.py` | Weighted portfolios with drift and periodic rebalancing to target weights (`1W/1ME/QE/YE`), one-way cost in bps. Spec grammar: commas join the elements of ONE portfolio (`SYM` or `SYM:w`, bare = equal weight), spaces separate DISTINCT symbols/portfolios (`parse_items`). |
|
|
| `tax.py` | Simplified DAS after-tax engine: FIFO lots, 365-day long/short split, separate LT/ST/dividend rates. Headline curve = what you keep if you **sell everything today** (unrealized gains taxed daily by lot age). |
|
|
| `chart_widget.py` | Self-contained plotly.js chart in an iframe: mouse zoom/pan, x clamped to the data, view edges snapped to first/last data points with day-precise labels, y tight-fit, every line re-based to 1.0 at the left edge. |
|
|
| `portfolios.py` | Saved portfolio definitions in `portfolios.json` (name, spec, scheme, cost). |
|
|
| `settings.json` | Persisted UI inputs (symbol/benchmark specs, scheme, costs, tax rates, period, curve/window mode) — restored on every page load and server restart; delete to reset. |
|
|
| `app.py` | Streamlit UI: single "symbol or portfolio" spec field (page updates as soon as the input is valid; unknown symbols get click-to-fix "did you mean" suggestions) + a benchmark box with the same grammar (one benchmark per line; a line is a single symbol or a comma-joined portfolio, simulated with the same scheme/cost/tax rules — pre- and after-tax curves, first one drives beta/alpha), scheme/costs/tax rates, save + load/compare/delete portfolios (overlaid pre/after-tax curves), curve toggle (both / pre-tax only / after-tax only), stats table, allocation, per-year tax detail. |
|
|
|
|
## Development
|
|
|
|
- **Run**: `./run.sh` → http://localhost:8599 (fixed port; no-ops if a
|
|
server is already running). The chart loads plotly.js from a CDN; for
|
|
fully offline use set `F_INLINE_PLOTLY=1` in `run.sh`.
|
|
- **Test**: `./run_tests.sh`
|
|
1. `tests/test_app.py` — app-level tests via Streamlit AppTest (no
|
|
browser). Memory: one data bundle is ~2.3 GB, so this process keeps
|
|
at most ONE AppTest alive (see its header comment).
|
|
2. `tests/test_e2e_browser.py` — Playwright + headless Chromium driving
|
|
the real page with real keystrokes; needs the server running on 8599.
|
|
One-time setup: `.venv/bin/pip install playwright` and
|
|
`.venv/bin/python -m playwright install chromium`.
|
|
- **Gotchas**
|
|
- Streamlit caches imported modules per process: **restart the server**
|
|
after editing any `.py` (kill the old one first — `run.sh` refuses to
|
|
double-start).
|
|
- `st.cache_data` caches the portfolio + tax simulations: they recompute
|
|
only when symbols/scheme/cost/tax rates change, not on window or curve
|
|
toggles.
|
|
- `settings.json` (gitignored) persists UI inputs across reloads and
|
|
restarts; delete it to reset. Saved portfolios live in `portfolios.json`.
|
|
- Data cache: `.cache/*.parquet`; rebuild via the sidebar checkbox
|
|
(first build ~1 min for ~4k symbols).
|
|
|
|
## Known simplifications (roadmap)
|
|
|
|
- No loss carryover or carryforward across years; no wash-sale rules.
|
|
- Distributed capital gains taxed entirely at the long-term rate.
|
|
- Single (federal-like) tax bracket; no state taxes, no AMT.
|
|
- Equal treatment of benchmark for beta/alpha (CAPM, rf = 0 by default).
|
|
|
|
Ideas: vectorbt sweeps over rebalance schemes, NiceGUI/Textual frontend,
|
|
empyrical-reloaded metrics, monthly (not yearly) loss netting, tax-loss
|
|
harvesting simulation.
|