# Fund discovery research log Running log of attempts, dead ends, and learnings for finding candidate funds (alpha-driven, portfolio-complementing) with the `fundlab` pipeline. Newest entries at the bottom of each section. Update as you go — this file IS the knowledge base. ## Sources & what works | Source | Status | Notes | |---|---|---| | Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern | | SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers | | SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `` (UNCLOSED tag) + `` per class; often several series per filing | | Yahoo chart API (`query1.../v8/finance/chart/`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length | | `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym | | EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only | | SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML | | SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) | | SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent | | Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected | | stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued | ## Learnings (hard-won) 1. **Famous multi-strategy/macro funds are private/offshore** — Millennium, Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace, Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497, no Yahoo OTC ticker. Structural, not a search failure. 2. **Precision over recall for name→ticker resolution**: a guessed ticker that "looks right" is worse than no answer. Chart-API name gate + 2/3 token overlap rejected 23/24 memory-based guesses. 3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC mutual funds report `fullExchangeName="Nasdaq"`. Use `instrumentType` (MUTUALFUND vs ETF). 4. **497 SGML cover uses UNCLOSED tags, one per line** — `Foo Fund\nTNMAX\n`. Closed-tag regexes find nothing; `to_text()` (tag stripping) destroys the data. Parse line-based, BEFORE any tag stripping. 5. **Full-index columns drift** — don't trust fixed widths; the header line and data rows don't align. Regex the whole line. 6. **One quarter ≠ the universe**: each fund's annual base-497 re-filing lands in a random quarter; union of 4 consecutive quarters is the full active universe (1,209 in Q2 alone → 1,668 union). 7. **Accession paths in the index are relative to `/Archives/`**, not `/Archives/edgar/data/` — doubling the prefix 404s. 8. **Amendments (497A/497VPU) may lack the series cover** — base 497 / 497K carry it. (497A inclusion pending — see below.) 9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs, same series listed twice) — dedupe by series name, not CIK. 10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) — the |t|>2 gate on added regressors is essential (decompose.py). 11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity funds show the strongest alpha on screen (+12–17%/yr) but corr 0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU". 12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis candidate sets (one rep per sleeve family) or the betas split arbitrarily between ivv/vti/vt. 13. **Wrong-fund objectives are worse than none** — all EDGAR resolution stages gate on name match before accepting. ## Pipeline stages (current) ``` full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread) → parse_cover (line-based SGML) → [name filter — REMOVED in v2, kept as a tag] → Yahoo chart verify (instrumentType, 20y length) → share-class dedupe (longest history) → goget missing (batched) → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence, corr vs portfolio & benchmark) [0.3s/fund] → search_*.json → app Fund Lab "Alpha search" table ``` ## Overnight comprehensive screen (v2, started 2026-08-26) Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate. ### Notes while running (live - the log() below appends automatically) - [x] survey: 10,372 class tickers in covers cache; 1,223 already local, 9,149 to verify on Yahoo; screen costs 0.3s/fund. - [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also filed a base in window) - no need to include. - [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min; 112 dead/never-listed tickers dropped). Cached in universe_cache/yahoo_meta.json. - [x] select: **2,384 funds** (407 local, 1,977 external). Only 54 match the alpha name pattern - the other 2,330 are exactly what the v1 name-filter would have missed. Worklist cached in universe_cache/selected.json. - [ ] download: 1,960 missing symbols via goget, 200/batch, ~3.3s/sym (ETA ~1:20 AM). Logs to fundlab/overnight.log + this file. - [ ] screen: ~2,384 x 0.3s ≈ 25 min -> search_all.json. - [ ] finalize: verdict counts + "candidates v1 would have missed". Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the project (fundlab/overnight.log, fundlab/universe_cache/), and use `setsid nohup ... < /dev/null &` so a closed shell can't kill the job. ### Watchdog incident (2026-08-26 ~22:00) - First watchdog used `pgrep -f "fundlab.overnight"` for liveness. **Failed test**: a monitoring one-liner whose command line merely CONTAINED the string made pgrep match the monitor's own bash -c wrapper -> watchdog reported "alive" while the real run was dead. Lesson: `pgrep -f` on a distinctive-but-quotable string is fragile; any shell quoting the string is a false positive. Use a PID file the process writes itself (`fundlab/overnight.pid`), pattern only as fallback for pre-pidfile launches. - Fixed watchdog restarted; it detected the dead run on its first check, relaunched, and the run resumed at "1823 missing" (137 symbols already cached - exactly the expected resume). - Also hardened stage_download: a hung goget batch (TimeoutExpired) no longer kills the whole run - caught per batch. - 2026-08-26 16:01 === stage verify === - 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo - 2026-08-26 16:01 verify: 500/9149 - 2026-08-26 16:02 verify: 1000/9149 - 2026-08-26 16:03 verify: 1500/9149 - 2026-08-26 16:03 verify: 2000/9149 - 2026-08-26 16:04 verify: 2500/9149 - 2026-08-26 16:04 verify: 3000/9149 - 2026-08-26 16:05 verify: 3500/9149 - 2026-08-26 16:05 verify: 4000/9149 - 2026-08-26 16:06 verify: 4500/9149 - 2026-08-26 16:06 verify: 5000/9149 - 2026-08-26 16:07 verify: 5500/9149 - 2026-08-26 16:07 verify: 6000/9149 - 2026-08-26 16:08 verify: 6500/9149 - 2026-08-26 16:09 verify: 7000/9149 - 2026-08-26 16:09 verify: 7500/9149 - 2026-08-26 16:10 verify: 8000/9149 - 2026-08-26 16:10 verify: 8500/9149 - 2026-08-26 16:11 verify: 9000/9149 - 2026-08-26 16:11 verify done: 10260/10372 with data - 2026-08-26 16:11 === overnight run finished in 0.2h === - 2026-08-26 21:31 === stage select === - 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external) - 2026-08-26 21:31 === overnight run finished in 0.0h === - 2026-08-26 21:31 === stage download === - 2026-08-26 21:31 download: 1960 missing symbols - 2026-08-26 21:54:22 watchdog: watchdog started (interval 300s) - 2026-08-26 21:54:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 21:59:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:04:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:08:01 watchdog: watchdog started (interval 300s) - 2026-08-26 22:08:01 watchdog: process DEAD with 2384 funds unscreened - relaunching - 2026-08-26 22:08 === stage download === - 2026-08-26 22:08 download: 1823 missing symbols - 2026-08-26 22:09:01 watchdog: relaunch confirmed alive