gmp/f
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
eac61683f6
f/fundlab/RESEARCH.md

7.0 KiB
Raw Blame History

Fund discovery research log

Running log of attempts, dead ends, and learnings for finding candidate funds (alpha-driven, portfolio-complementing) with the fundlab pipeline. Newest entries at the bottom of each section. Update as you go — this file IS the knowledge base.

Sources & what works

Source Status Notes
Local stocks DB (~/prog/fin/stocks/, 8,250 syms) ✅ primary ~100 open-end alt families already present; dbmine.py mines by name pattern
SEC full-index Archives/edgar/full-index/YYYY/QTRn/company.gz ✅ discovery workhorse lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers
SEC full-submission .txt (per accession) ✅ ~10–50KB; line-based SGML cover with <SERIES-NAME> (UNCLOSED tag) + <CLASS-CONTRACT-TICKER-SYMBOL> per class; often several series per filing
Yahoo chart API (query1.../v8/finance/chart/<T>) ✅ no crumb needed; meta has instrumentType/exchange/longName; range=20y gives history length
goget (~/go/bin/goget) ✅ batch downloader, idempotent, ~2–5s/sym
EDGAR FTS (efts.sec.gov/LATEST/search-index) ⚠️ fragile phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only
SEC browse-edgar company listing ❌ dead now JS-rendered, no data in HTML
SEC /files/investment-company-tickers.json ❌ doesn't exist 404 (misremembered)
SEC company_tickers.json ❌ for our purpose only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent
Yahoo search/crumb API ❌ throttled IP-level "Too Many Requests" on fc.yahoo.com/getcrumb; chart API unaffected
stockanalysis.com/funds/ ❌ 404 path guessed wrong, not pursued

Learnings (hard-won)

  1. Famous multi-strategy/macro funds are private/offshore — Millennium, Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace, Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497, no Yahoo OTC ticker. Structural, not a search failure.
  2. Precision over recall for name→ticker resolution: a guessed ticker that "looks right" is worse than no answer. Chart-API name gate + 2/3 token overlap rejected 23/24 memory-based guesses.
  3. Yahoo exchange name is a USELESS fund/ETF discriminator: OTC mutual funds report fullExchangeName="Nasdaq". Use instrumentType (MUTUALFUND vs ETF).
  4. 497 SGML cover uses UNCLOSED tags, one per line — <SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n. Closed-tag regexes find nothing; to_text() (tag stripping) destroys the data. Parse line-based, BEFORE any tag stripping.
  5. Full-index columns drift — don't trust fixed widths; the header line and data rows don't align. Regex the whole line.
  6. One quarter ≠ the universe: each fund's annual base-497 re-filing lands in a random quarter; union of 4 consecutive quarters is the full active universe (1,209 in Q2 alone → 1,668 union).
  7. Accession paths in the index are relative to /Archives/, not /Archives/edgar/data/ — doubling the prefix 404s.
  8. Amendments (497A/497VPU) may lack the series cover — base 497 / 497K carry it. (497A inclusion pending — see below.)
  9. Family CIKs repeat across the index (e.g. AB under 2 CIKs, same series listed twice) — dedupe by series name, not CIK.
  10. Large-n BIC is knife-edge (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) — the |t|>2 gate on added regressors is essential (decompose.py).
  11. The portfolio is 50% market-neutral (qspnx) — MN/L-S-equity funds show the strongest alpha on screen (+12–17%/yr) but corr 0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
  12. Near-duplicate sleeves make OLS knife-edge — distinct-axis candidate sets (one rep per sleeve family) or the betas split arbitrarily between ivv/vti/vt.
  13. Wrong-fund objectives are worse than none — all EDGAR resolution stages gate on name match before accepting.

Pipeline stages (current)

full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
  → parse_cover (line-based SGML)
  → [name filter — REMOVED in v2, kept as a tag]
  → Yahoo chart verify (instrumentType, 20y length)
  → share-class dedupe (longest history)
  → goget missing (batched)
  → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
     corr vs portfolio & benchmark)  [0.3s/fund]
  → search_*.json → app Fund Lab "Alpha search" table

Overnight comprehensive screen (v2, started 2026-08-26)

Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate.

Notes while running (live - the log() below appends automatically)

  • survey: 10,372 class tickers in covers cache; 1,223 already local, 9,149 to verify on Yahoo; screen costs 0.3s/fund.
  • 497A adds ZERO new CIKs over 497/497K (every 497A filer also filed a base in window) - no need to include.
  • verify: 10,260/10,372 tickers with data (4-thread, ~10 min; 112 dead/never-listed tickers dropped). Cached in universe_cache/yahoo_meta.json.
  • select: 2,384 funds (407 local, 1,977 external). Only 54 match the alpha name pattern - the other 2,330 are exactly what the v1 name-filter would have missed. Worklist cached in universe_cache/selected.json.
  • download: 1,960 missing symbols via goget, 200/batch, ~3.3s/sym (ETA ~1:20 AM). Logs to fundlab/overnight.log + this file.
  • screen: ~2,384 x 0.3s ≈ 25 min -> search_all.json.
  • finalize: verdict counts + "candidates v1 would have missed".

Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the project (fundlab/overnight.log, fundlab/universe_cache/), and use setsid nohup ... < /dev/null & so a closed shell can't kill the job.

  • 2026-08-26 16:01 === stage verify ===
  • 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
  • 2026-08-26 16:01 verify: 500/9149
  • 2026-08-26 16:02 verify: 1000/9149
  • 2026-08-26 16:03 verify: 1500/9149
  • 2026-08-26 16:03 verify: 2000/9149
  • 2026-08-26 16:04 verify: 2500/9149
  • 2026-08-26 16:04 verify: 3000/9149
  • 2026-08-26 16:05 verify: 3500/9149
  • 2026-08-26 16:05 verify: 4000/9149
  • 2026-08-26 16:06 verify: 4500/9149
  • 2026-08-26 16:06 verify: 5000/9149
  • 2026-08-26 16:07 verify: 5500/9149
  • 2026-08-26 16:07 verify: 6000/9149
  • 2026-08-26 16:08 verify: 6500/9149
  • 2026-08-26 16:09 verify: 7000/9149
  • 2026-08-26 16:09 verify: 7500/9149
  • 2026-08-26 16:10 verify: 8000/9149
  • 2026-08-26 16:10 verify: 8500/9149
  • 2026-08-26 16:11 verify: 9000/9149
  • 2026-08-26 16:11 verify done: 10260/10372 with data
  • 2026-08-26 16:11 === overnight run finished in 0.2h ===
  • 2026-08-26 21:31 === stage select ===
  • 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
  • 2026-08-26 21:31 === overnight run finished in 0.0h ===
  • 2026-08-26 21:31 === stage download ===
  • 2026-08-26 21:31 download: 1960 missing symbols