f/fundlab/RESEARCH.md

Fund discovery research log

Running log of attempts, dead ends, and learnings for finding candidate funds (alpha-driven, portfolio-complementing) with the fundlab pipeline. Newest entries at the bottom of each section. Update as you go — this file IS the knowledge base.

Sources & what works

Source	Status	Notes
Local stocks DB (~/prog/fin/stocks/, 8,250 syms)	✅ primary	~100 open-end alt families already present; dbmine.py mines by name pattern
SEC full-index Archives/edgar/full-index/YYYY/QTRn/company.gz	✅ discovery workhorse	lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers
SEC full-submission .txt (per accession)	✅	~10–50KB; line-based SGML cover with <SERIES-NAME> (UNCLOSED tag) + <CLASS-CONTRACT-TICKER-SYMBOL> per class; often several series per filing
Yahoo chart API (query1.../v8/finance/chart/<T>)	✅	no crumb needed; meta has instrumentType/exchange/longName; range=20y gives history length
goget (~/go/bin/goget)	✅	batch downloader, idempotent, ~2–5s/sym
EDGAR FTS (efts.sec.gov/LATEST/search-index)	⚠️ fragile	phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only
SEC browse-edgar company listing	❌ dead	now JS-rendered, no data in HTML
SEC /files/investment-company-tickers.json	❌ doesn't exist	404 (misremembered)
SEC company_tickers.json	❌ for our purpose	only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent
Yahoo search/crumb API	❌ throttled	IP-level "Too Many Requests" on fc.yahoo.com/getcrumb; chart API unaffected
stockanalysis.com/funds/	❌ 404	path guessed wrong, not pursued

Learnings (hard-won)

1. Famous multi-strategy/macro funds are private/offshore — Millennium, Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace, Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497, no Yahoo OTC ticker. Structural, not a search failure.
2. Precision over recall for name→ticker resolution: a guessed ticker that "looks right" is worse than no answer. Chart-API name gate + 2/3 token overlap rejected 23/24 memory-based guesses.
3. Yahoo exchange name is a USELESS fund/ETF discriminator: OTC mutual funds report fullExchangeName="Nasdaq". Use instrumentType (MUTUALFUND vs ETF).
4. 497 SGML cover uses UNCLOSED tags, one per line — <SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n. Closed-tag regexes find nothing; to_text() (tag stripping) destroys the data. Parse line-based, BEFORE any tag stripping.
5. Full-index columns drift — don't trust fixed widths; the header line and data rows don't align. Regex the whole line.
6. One quarter ≠ the universe: each fund's annual base-497 re-filing lands in a random quarter; union of 4 consecutive quarters is the full active universe (1,209 in Q2 alone → 1,668 union).
7. Accession paths in the index are relative to /Archives/, not /Archives/edgar/data/ — doubling the prefix 404s.
8. Amendments (497A/497VPU) may lack the series cover — base 497 / 497K carry it. (497A inclusion pending — see below.)
9. Family CIKs repeat across the index (e.g. AB under 2 CIKs, same series listed twice) — dedupe by series name, not CIK.
10. Large-n BIC is knife-edge (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) — the |t|>2 gate on added regressors is essential (decompose.py).
11. The portfolio is 50% market-neutral (qspnx) — MN/L-S-equity funds show the strongest alpha on screen (+12–17%/yr) but corr 0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
12. Near-duplicate sleeves make OLS knife-edge — distinct-axis candidate sets (one rep per sleeve family) or the betas split arbitrarily between ivv/vti/vt.
13. Wrong-fund objectives are worse than none — all EDGAR resolution stages gate on name match before accepting.

Pipeline stages (current)

full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
  → parse_cover (line-based SGML)
  → [name filter — REMOVED in v2, kept as a tag]
  → Yahoo chart verify (instrumentType, 20y length)
  → share-class dedupe (longest history)
  → goget missing (batched)
  → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
     corr vs portfolio & benchmark)  [0.3s/fund]
  → search_*.json → app Fund Lab "Alpha search" table

Overnight comprehensive screen (v2, started 2026-08-26)

Goal: screen EVERY OTC open-end fund with ≥5y history from the 497 universe — no name pre-filter. Name match becomes a tag/cross-check, not a gate.

Notes while running (live - the log() below appends automatically)

• survey: 10,372 class tickers in covers cache; 1,223 already local, 9,149 to verify on Yahoo; screen costs 0.3s/fund.
• 497A adds ZERO new CIKs over 497/497K (every 497A filer also filed a base in window) - no need to include.
• verify: 10,260/10,372 tickers with data (4-thread, ~10 min; 112 dead/never-listed tickers dropped). Cached in universe_cache/yahoo_meta.json.
• select: 2,384 funds (407 local, 1,977 external). Only 54 match the alpha name pattern - the other 2,330 are exactly what the v1 name-filter would have missed. Worklist cached in universe_cache/selected.json.
• download: 1,960 symbols (resumed at 1,823 after the watchdog kill-test). Logs to fundlab/overnight.log + this file.
• screen: 2,384 funds -> search_all.json (1.2MB).
• finalize: verdict counts + "candidates v1 would have missed".

RESULTS (run finished 2026-08-27 03:23, 5.2h, watchdog exited clean)

• 2,384 screened: 250 CANDIDATE, 1336 sleeve mix, 707 weak, 42 alpha-but-correlated, 32 not-persistent, 17 no-5y-window.
• Of the 250: ~37 are short-duration/floating-rate/money-market CARRY (model artifact - the 21-sleeve set has no short-duration axis, so carry shows up as a positive intercept). The rest mix: a) GENUINE idiosyncratic alpha: hmezx NexPoint Merger Arb (t5 7.1, tF 4.6, corr 0.14), mervx The Merger Fund, egrix Eaton Vance Global Macro (R2 0.07, +7.9%), wmnux Westwood Alt Income, dmszx Destinations Multi-Strat, anglx Angel Oak Multi-Strat, aguax Am Beacon Dev World Income (+6.5%, t5 3.7), femdx Franklin EM Debt Opps (+6.0%), rctix River Canyon TR Bond (t5 5.6), kmdnx Kinetics Multi-Disciplinary. b) MISSING-FACTOR exposures (sector not in the 21 sleeves, so sector exposure reads as "alpha"): munis (hicox, fhmix), preferreds (lpxax, dpiax), EM debt (femdx), securitized credit (scfzx), TIPS/real-return.
• ~200 candidates have "boring" names the v1 alpha-name filter would have rejected - direct quantification of the v1 blind spot.
• eniax SIIT Opportunistic Income: t5 10.1 but tF 1.44 -> alpha is RECENT (last 5y), not full-history. Good thing the screen keeps both stats.

Next iterations

1. Add missing factors + cluster by return driver (fundlab/factors.py + fundlab/cluster.py).
2. N-PORT holdings cross-check on the top candidates (fundlab/xcheck.py) - results below.
3. CEF universe (485/N-2 filers) - separate pass; CEFs have premium/discount dynamics the NAV screen can't see.

N-PORT cross-check (fundlab/xcheck.py, 2026-08-27)

21 of 22 top candidates resolved to their ACTUAL holdings (qcmmrx = money-market account, no holdings to parse).

Pipeline hard-won facts:

• The fund's NPORT is usually filed under the fund's OWN registrant (browse-edgar ticker->CIK), not the 497-cover family/trust CIK (EV: trust 1552324 vs fund 745463). covers.json of the own CIK gives the exact series name -> the ONLY reliable disambiguator between sibling funds ("...Absolute Return Fund" vs "...Advantage Fund" share 6 of 7 words; token-overlap alone can't tell them apart).
• Big trusts file dozens of NPORT-Ps per quarter (one per fund); a flat newest-30 window misses the fund's own filing. Take all filings on the 4 most recent distinct dates.
• The submissions API points at the XSL-RENDERED view (xslFormNPORT-P_X01/primary_doc.xml, a 5-20MB HTML page); the raw schema data (seriesName/invstOrSec/netAssets) sits at accession-root primary_doc.xml (5KB-1.5MB). Strip the xsl prefix.
• The raw file is malformed XHTML (CSS in