# Fund discovery research log Running log of attempts, dead ends, and learnings for finding candidate funds (alpha-driven, portfolio-complementing) with the `fundlab` pipeline. Newest entries at the bottom of each section. Update as you go — this file IS the knowledge base. ## Sources & what works | Source | Status | Notes | |---|---|---| | Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern | | SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers | | SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `` (UNCLOSED tag) + `` per class; often several series per filing | | Yahoo chart API (`query1.../v8/finance/chart/`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length | | `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym | | EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only | | SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML | | SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) | | SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent | | Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected | | stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued | ## Learnings (hard-won) 1. **Famous multi-strategy/macro funds are private/offshore** — Millennium, Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace, Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497, no Yahoo OTC ticker. Structural, not a search failure. 2. **Precision over recall for name→ticker resolution**: a guessed ticker that "looks right" is worse than no answer. Chart-API name gate + 2/3 token overlap rejected 23/24 memory-based guesses. 3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC mutual funds report `fullExchangeName="Nasdaq"`. Use `instrumentType` (MUTUALFUND vs ETF). 4. **497 SGML cover uses UNCLOSED tags, one per line** — `Foo Fund\nTNMAX\n`. Closed-tag regexes find nothing; `to_text()` (tag stripping) destroys the data. Parse line-based, BEFORE any tag stripping. 5. **Full-index columns drift** — don't trust fixed widths; the header line and data rows don't align. Regex the whole line. 6. **One quarter ≠ the universe**: each fund's annual base-497 re-filing lands in a random quarter; union of 4 consecutive quarters is the full active universe (1,209 in Q2 alone → 1,668 union). 7. **Accession paths in the index are relative to `/Archives/`**, not `/Archives/edgar/data/` — doubling the prefix 404s. 8. **Amendments (497A/497VPU) may lack the series cover** — base 497 / 497K carry it. (497A inclusion pending — see below.) 9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs, same series listed twice) — dedupe by series name, not CIK. 10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) — the |t|>2 gate on added regressors is essential (decompose.py). 11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity funds show the strongest alpha on screen (+12–17%/yr) but corr 0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU". 12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis candidate sets (one rep per sleeve family) or the betas split arbitrarily between ivv/vti/vt. 13. **Wrong-fund objectives are worse than none** — all EDGAR resolution stages gate on name match before accepting. ## Pipeline stages (current) ``` full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread) → parse_cover (line-based SGML) → [name filter — REMOVED in v2, kept as a tag] → Yahoo chart verify (instrumentType, 20y length) → share-class dedupe (longest history) → goget missing (batched) → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence, corr vs portfolio & benchmark) [0.3s/fund] → search_*.json → app Fund Lab "Alpha search" table ``` ## Overnight comprehensive screen (v2, started 2026-08-26) Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate. ### Notes while running (live - the log() below appends automatically) - [x] survey: 10,372 class tickers in covers cache; 1,223 already local, 9,149 to verify on Yahoo; screen costs 0.3s/fund. - [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also filed a base in window) - no need to include. - [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min; 112 dead/never-listed tickers dropped). Cached in universe_cache/yahoo_meta.json. - [x] select: **2,384 funds** (407 local, 1,977 external). Only 54 match the alpha name pattern - the other 2,330 are exactly what the v1 name-filter would have missed. Worklist cached in universe_cache/selected.json. - [x] download: 1,960 symbols (resumed at 1,823 after the watchdog kill-test). Logs to fundlab/overnight.log + this file. - [x] screen: 2,384 funds -> search_all.json (1.2MB). - [x] finalize: verdict counts + "candidates v1 would have missed". ### RESULTS (run finished 2026-08-27 03:23, 5.2h, watchdog exited clean) - 2,384 screened: 250 CANDIDATE, 1336 sleeve mix, 707 weak, 42 alpha-but-correlated, 32 not-persistent, 17 no-5y-window. - Of the 250: ~37 are short-duration/floating-rate/money-market CARRY (model artifact - the 21-sleeve set has no short-duration axis, so carry shows up as a positive intercept). The rest mix: a) GENUINE idiosyncratic alpha: hmezx NexPoint Merger Arb (t5 7.1, tF 4.6, corr 0.14), mervx The Merger Fund, egrix Eaton Vance Global Macro (R2 0.07, +7.9%), wmnux Westwood Alt Income, dmszx Destinations Multi-Strat, anglx Angel Oak Multi-Strat, aguax Am Beacon Dev World Income (+6.5%, t5 3.7), femdx Franklin EM Debt Opps (+6.0%), rctix River Canyon TR Bond (t5 5.6), kmdnx Kinetics Multi-Disciplinary. b) MISSING-FACTOR exposures (sector not in the 21 sleeves, so sector exposure reads as "alpha"): munis (hicox, fhmix), preferreds (lpxax, dpiax), EM debt (femdx), securitized credit (scfzx), TIPS/real-return. - ~200 candidates have "boring" names the v1 alpha-name filter would have rejected - direct quantification of the v1 blind spot. - eniax SIIT Opportunistic Income: t5 10.1 but tF 1.44 -> alpha is RECENT (last 5y), not full-history. Good thing the screen keeps both stats. ### Next iterations (not done) 1. Add missing sleeves to the model (short-duration/floating-rate, muni, preferreds, EM debt, merger-arb proxy) to split true alpha from missing-factor exposure; re-screen (0.3s/fund, ~12 min). 2. N-PORT holdings cross-check on the top ~15 candidates (the v1 16-fund pipeline: edgar NPORT fetch + category buckets) to confirm what the alpha funds actually hold. 3. CEF universe (485/N-2 filers) - separate pass; CEFs have premium/discount dynamics the NAV screen can't see. Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the project (fundlab/overnight.log, fundlab/universe_cache/), and use `setsid nohup ... < /dev/null &` so a closed shell can't kill the job. ### Watchdog incident (2026-08-26 ~22:00) - First watchdog used `pgrep -f "fundlab.overnight"` for liveness. **Failed test**: a monitoring one-liner whose command line merely CONTAINED the string made pgrep match the monitor's own bash -c wrapper -> watchdog reported "alive" while the real run was dead. Lesson: `pgrep -f` on a distinctive-but-quotable string is fragile; any shell quoting the string is a false positive. Use a PID file the process writes itself (`fundlab/overnight.pid`), pattern only as fallback for pre-pidfile launches. - Fixed watchdog restarted; it detected the dead run on its first check, relaunched, and the run resumed at "1823 missing" (137 symbols already cached - exactly the expected resume). - Also hardened stage_download: a hung goget batch (TimeoutExpired) no longer kills the whole run - caught per batch. - 2026-08-26 16:01 === stage verify === - 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo - 2026-08-26 16:01 verify: 500/9149 - 2026-08-26 16:02 verify: 1000/9149 - 2026-08-26 16:03 verify: 1500/9149 - 2026-08-26 16:03 verify: 2000/9149 - 2026-08-26 16:04 verify: 2500/9149 - 2026-08-26 16:04 verify: 3000/9149 - 2026-08-26 16:05 verify: 3500/9149 - 2026-08-26 16:05 verify: 4000/9149 - 2026-08-26 16:06 verify: 4500/9149 - 2026-08-26 16:06 verify: 5000/9149 - 2026-08-26 16:07 verify: 5500/9149 - 2026-08-26 16:07 verify: 6000/9149 - 2026-08-26 16:08 verify: 6500/9149 - 2026-08-26 16:09 verify: 7000/9149 - 2026-08-26 16:09 verify: 7500/9149 - 2026-08-26 16:10 verify: 8000/9149 - 2026-08-26 16:10 verify: 8500/9149 - 2026-08-26 16:11 verify: 9000/9149 - 2026-08-26 16:11 verify done: 10260/10372 with data - 2026-08-26 16:11 === overnight run finished in 0.2h === - 2026-08-26 21:31 === stage select === - 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external) - 2026-08-26 21:31 === overnight run finished in 0.0h === - 2026-08-26 21:31 === stage download === - 2026-08-26 21:31 download: 1960 missing symbols - 2026-08-26 21:54:22 watchdog: watchdog started (interval 300s) - 2026-08-26 21:54:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 21:59:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:04:22 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:08:01 watchdog: watchdog started (interval 300s) - 2026-08-26 22:08:01 watchdog: process DEAD with 2384 funds unscreened - relaunching - 2026-08-26 22:08 === stage download === - 2026-08-26 22:08 download: 1823 missing symbols - 2026-08-26 22:09:01 watchdog: relaunch confirmed alive - 2026-08-26 22:14:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:19:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:24:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:29:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:34:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:39:01 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:41 download: batch 1 -> 200/200 - 2026-08-26 22:44:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:49:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:54:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 22:59:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:04:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:09:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:14:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:14 download: batch 2 -> 200/200 - 2026-08-26 23:19:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:24:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:29:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:34:02 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:39:03 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:44:03 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:47 download: batch 3 -> 200/200 - 2026-08-26 23:49:03 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:54:03 watchdog: overnight alive, still_missing=2384 - 2026-08-26 23:59:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:04:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:09:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:14:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:19:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:20 download: batch 4 -> 200/200 - 2026-08-27 00:24:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:29:03 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:34:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:39:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:44:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:49:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:54 download: batch 5 -> 200/200 - 2026-08-27 00:54:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 00:59:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:04:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:09:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:14:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:19:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:24:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:27 download: batch 6 -> 200/200 - 2026-08-27 01:29:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:34:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:39:04 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:44:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:49:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:54:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 01:59:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:00 download: batch 7 -> 200/200 - 2026-08-27 02:04:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:09:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:14:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:19:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:24:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:29:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:33 download: batch 8 -> 200/200 - 2026-08-27 02:34:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:39:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:44:05 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:49:06 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:54:06 watchdog: overnight alive, still_missing=2384 - 2026-08-27 02:59:06 watchdog: overnight alive, still_missing=2384 - 2026-08-27 03:04:06 watchdog: overnight alive, still_missing=2384 - 2026-08-27 03:06 download: batch 9 -> 200/200 - 2026-08-27 03:09:06 watchdog: overnight alive, still_missing=2384 - 2026-08-27 03:10 download: batch 10 -> 23/23 - 2026-08-27 03:10 download done: 0 still missing (no Yahoo data?) - 2026-08-27 03:10 === stage screen === - 2026-08-27 03:10 screen: 2384 funds, 2384 to do - 2026-08-27 03:11 screen: 100/2384 - 2026-08-27 03:11 screen: 200/2384 - 2026-08-27 03:12 screen: 300/2384 - 2026-08-27 03:12 screen: 400/2384 - 2026-08-27 03:13 screen: 500/2384 - 2026-08-27 03:13 screen: 600/2384 - 2026-08-27 03:14:06 watchdog: overnight alive, still_missing=1784 - 2026-08-27 03:14 screen: 700/2384 - 2026-08-27 03:14 screen: 800/2384 - 2026-08-27 03:15 screen: 900/2384 - 2026-08-27 03:15 screen: 1000/2384 - 2026-08-27 03:16 screen: 1100/2384 - 2026-08-27 03:16 screen: 1200/2384 - 2026-08-27 03:17 screen: 1300/2384 - 2026-08-27 03:17 screen: 1400/2384 - 2026-08-27 03:18 screen: 1500/2384 - 2026-08-27 03:18 screen: 1600/2384 - 2026-08-27 03:19:06 watchdog: overnight alive, still_missing=784 - 2026-08-27 03:19 screen: 1700/2384 - 2026-08-27 03:19 screen: 1800/2384 - 2026-08-27 03:20 screen: 1900/2384 - 2026-08-27 03:20 screen: 2000/2384 - 2026-08-27 03:21 screen: 2100/2384 - 2026-08-27 03:22 screen: 2200/2384 - 2026-08-27 03:22 screen: 2300/2384 - 2026-08-27 03:23 screen done: 2384 funds - 2026-08-27 03:23 === stage finalize === - 2026-08-27 03:23 finalize: 2384 funds screened - 2026-08-27 03:23 1336 sleeve mix - 2026-08-27 03:23 707 weak/unstable alpha - 2026-08-27 03:23 250 CANDIDATE - 2026-08-27 03:23 42 alpha, but correlated with current portfolio - 2026-08-27 03:23 32 alpha in 5y window, but not persistent - 2026-08-27 03:23 17 no 5y window - 2026-08-27 03:23 candidates v1 name-filter would have missed: ['abhyx', 'acfix', 'acitx', 'acthx', 'aflix', 'aftex', 'agovx', 'aguax', 'aidax', 'alcax', 'algrx', 'alnyx', 'althx', 'amaax', 'amhix', 'annpx', 'asdvx', 'atfax', 'avaax', 'bchyx', 'bcitx', 'bils', 'bmbix', 'bsnix', 'btmix', 'cfmox', 'cfnlx', 'cfrax', 'coiax', 'coltx', 'cpxax', 'csdax', 'cshix', 'cxhyx', 'deffx', 'delix', 'dflex', 'ditex', 'dmusx', 'dnmzx', 'dpiax', 'drcax', 'drnjx', 'dsibx', 'dultx', 'dvmhx', 'dvtax', 'eablx', 'eacax', 'eanax', 'eascx', 'ebsax', 'eniax', 'etazx', 'etgax', 'ethyx', 'etmdx', 'etmnx', 'etmox', 'etncx', 'etnjx', 'etorx', 'etpax', 'etsix', 'etvax', 'evfax', 'exmax', 'exnax', 'fafrx', 'faztx', 'fcamx', 'fcfix', 'fcotx', 'fcstx', 'fcvsx', 'fdmmx', 'femdx', 'ffrsx', 'fhaix', 'fhcox', 'fhmix', 'ficnx', 'fiscx', 'fkcix', 'fklax', 'fknix', 'fkstx', 'fktfx', 'fktix', 'fkutx', 'fkytx', 'flrn', 'fltdx', 'fltmx', 'fltr', 'fmdtx', 'fmoax', 'fmotx', 'fnmtx', 'fnytx', 'fpntx', 'fralx', 'frcox', 'frmox', 'frnjx', 'frorx', 'frpax', 'frstx', 'fsazx', 'ftazx', 'ftfmx', 'ftgax', 'ftlax', 'ftnyx', 'ftorx', 'fwiax', 'fxncx', 'gbonx', 'giyix', 'gtfbx', 'hicox', 'hmezx', 'hubax', 'icifx', 'ihiax', 'intax', 'kctax', 'kmdnx', 'kntax', 'lansx', 'lmsfx', 'lpxax', 'lsbdx', 'lsyux', 'ltebx', 'ltnyx', 'lubax', 'masax', 'mcoax', 'mdxbx', 'mervx', 'mfalx', 'mfarx', 'mfiax', 'mfpax', 'mfscx', 'mfsmx', 'mfssx', 'mfwvx', 'mgfox', 'missx', 'mmgax', 'mmhyx', 'mmufx', 'msncx', 'msnyx', 'msvax', 'mtbax', 'mtlfx', 'nefzx', 'nhmax', 'nmbax', 'nmdax', 'nmuix', 'nnjax', 'nnyax', 'nsmmx', 'nthex', 'olcax', 'omifx', 'onjcx', 'opcax', 'opnyx', 'orncx', 'pabax', 'pamfx', 'pbaix', 'pbcax', 'pgsix', 'phizx', 'phstx', 'ppnax', 'prfhx', 'prfrx', 'prfsx', 'prinx', 'prmdx', 'prnyx', 'prsmx', 'prtax', 'pruax', 'prvax', 'prvbx', 'prxcx', 'psnyx', 'ptebx', 'puls', 'pysix', 'qcmmrx', 'rctix', 'rlvsx', 'rmuyx', 'rpidx', 'rpifx', 'safex', 'scfzx', 'sdsax', 'sefix', 'seimx', 'shtix', 'sitex', 'smlax', 'sngvx', 'szmax', 'taftx', 'tisix', 'trbux', 'trhyx', 'tsdjx', 'twtix', 'ulst', 'usiax', 'usmsx', 'vcaix', 'vcitx', 'vctfx', 'vidax', 'vklmx', 'vkmmx', 'vmatx', 'vmltx', 'vnjtx', 'vnytx', 'vohix', 'vpaix', 'vtipx', 'vubfx', 'vusb', 'vwahx', 'vweax', 'vwitx', 'vwltx', 'wisex'] - 2026-08-27 03:23 === overnight run finished in 5.2h === - 2026-08-27 03:24:06 watchdog: process gone but screen COMPLETE (0 missing) - watchdog exiting - 2026-08-27 03:24:06 watchdog: watchdog exit