# Fund discovery research log

Running log of attempts, dead ends, and learnings for finding candidate
funds (alpha-driven, portfolio-complementing) with the `fundlab`
pipeline. Newest entries at the bottom of each section. Update as you
go — this file IS the knowledge base.

## Sources & what works

| Source | Status | Notes |
|---|---|---|
| Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern |
| SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers |
| SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `<SERIES-NAME>` (UNCLOSED tag) + `<CLASS-CONTRACT-TICKER-SYMBOL>` per class; often several series per filing |
| Yahoo chart API (`query1.../v8/finance/chart/<T>`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length |
| `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym |
| EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only |
| SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML |
| SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) |
| SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent |
| Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected |
| stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued |

## Learnings (hard-won)

1. **Famous multi-strategy/macro funds are private/offshore** — Millennium,
   Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace,
   Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497,
   no Yahoo OTC ticker. Structural, not a search failure.
2. **Precision over recall for name→ticker resolution**: a guessed
   ticker that "looks right" is worse than no answer. Chart-API name
   gate + 2/3 token overlap rejected 23/24 memory-based guesses.
3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC
   mutual funds report `fullExchangeName="Nasdaq"`. Use
   `instrumentType` (MUTUALFUND vs ETF).
4. **497 SGML cover uses UNCLOSED tags, one per line** —
   `<SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n`.
   Closed-tag regexes find nothing; `to_text()` (tag stripping)
   destroys the data. Parse line-based, BEFORE any tag stripping.
5. **Full-index columns drift** — don't trust fixed widths; the header
   line and data rows don't align. Regex the whole line.
6. **One quarter ≠ the universe**: each fund's annual base-497
   re-filing lands in a random quarter; union of 4 consecutive
   quarters is the full active universe (1,209 in Q2 alone → 1,668
   union).
7. **Accession paths in the index are relative to `/Archives/`**, not
   `/Archives/edgar/data/` — doubling the prefix 404s.
8. **Amendments (497A/497VPU) may lack the series cover** — base 497 /
   497K carry it. (497A inclusion pending — see below.)
9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs,
   same series listed twice) — dedupe by series name, not CIK.
10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) —
    the |t|>2 gate on added regressors is essential (decompose.py).
11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity
    funds show the strongest alpha on screen (+12–17%/yr) but corr
    0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis
    candidate sets (one rep per sleeve family) or the betas split
    arbitrarily between ivv/vti/vt.
13. **Wrong-fund objectives are worse than none** — all EDGAR
    resolution stages gate on name match before accepting.

## Pipeline stages (current)

```
full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
  → parse_cover (line-based SGML)
  → [name filter — REMOVED in v2, kept as a tag]
  → Yahoo chart verify (instrumentType, 20y length)
  → share-class dedupe (longest history)
  → goget missing (batched)
  → screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
     corr vs portfolio & benchmark)  [0.3s/fund]
  → search_*.json → app Fund Lab "Alpha search" table
```

## Overnight comprehensive screen (v2, started 2026-08-26)

Goal: screen EVERY OTC open-end fund with ≥5y history from the 497
universe — no name pre-filter. Name match becomes a tag/cross-check,
not a gate.

### Notes while running (live - the log() below appends automatically)
- [x] survey: 10,372 class tickers in covers cache; 1,223 already local,
      9,149 to verify on Yahoo; screen costs 0.3s/fund.
- [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also
      filed a base in window) - no need to include.
- [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min;
      112 dead/never-listed tickers dropped). Cached in
      universe_cache/yahoo_meta.json.
- [x] select: **2,384 funds** (407 local, 1,977 external). Only 54
      match the alpha name pattern - the other 2,330 are exactly what
      the v1 name-filter would have missed. Worklist cached in
      universe_cache/selected.json.
- [x] download: 1,960 symbols (resumed at 1,823 after the watchdog
      kill-test). Logs to fundlab/overnight.log + this file.
- [x] screen: 2,384 funds -> search_all.json (1.2MB).
- [x] finalize: verdict counts + "candidates v1 would have missed".

### RESULTS (run finished 2026-08-27 03:23, 5.2h, watchdog exited clean)
- 2,384 screened: 250 CANDIDATE, 1336 sleeve mix, 707 weak, 42
  alpha-but-correlated, 32 not-persistent, 17 no-5y-window.
- Of the 250: ~37 are short-duration/floating-rate/money-market CARRY
  (model artifact - the 21-sleeve set has no short-duration axis, so
  carry shows up as a positive intercept). The rest mix:
    a) GENUINE idiosyncratic alpha: hmezx NexPoint Merger Arb (t5 7.1,
       tF 4.6, corr 0.14), mervx The Merger Fund, egrix Eaton Vance
       Global Macro (R2 0.07, +7.9%), wmnux Westwood Alt Income,
       dmszx Destinations Multi-Strat, anglx Angel Oak Multi-Strat,
       aguax Am Beacon Dev World Income (+6.5%, t5 3.7), femdx
       Franklin EM Debt Opps (+6.0%), rctix River Canyon TR Bond
       (t5 5.6), kmdnx Kinetics Multi-Disciplinary.
    b) MISSING-FACTOR exposures (sector not in the 21 sleeves, so
       sector exposure reads as "alpha"): munis (hicox, fhmix),
       preferreds (lpxax, dpiax), EM debt (femdx), securitized credit
       (scfzx), TIPS/real-return.
- ~200 candidates have "boring" names the v1 alpha-name filter would
  have rejected - direct quantification of the v1 blind spot.
- eniax SIIT Opportunistic Income: t5 10.1 but tF 1.44 -> alpha is
  RECENT (last 5y), not full-history. Good thing the screen keeps both
  stats.

### Next iterations
1. [x] **Add missing factors + cluster by return driver**
   (fundlab/factors.py + fundlab/cluster.py).
2. [x] **N-PORT holdings cross-check on the top candidates**
   (fundlab/xcheck.py) - results below.
3. [x] **Drawdown-resilience screen** (fundlab/drawdown.py) -
   which candidates were positive when equities crashed.
4. CEF universe (485/N-2 filers) - separate pass; CEFs have
   premium/discount dynamics the NAV screen can't see.

### Price-appreciation vs. payout split (fundlab/taxsplit.py, 2026-08-27)
Follow-up to the tax-location plan: the user pointed out that fund
NAV appreciation is ALSO a capital gain (LTCG on a >1y sale) - the
taxplan score only measured DISTRIBUTION character. The fund price
files carry both series (Close = raw NAV with distributions paid
out; Adj Close = total return reinvested), so the split is
computable directly:
  price appreciation = f(Close)      -> investor's own gain on sale
  payout component   = f(Adj)-f(Close) -> taxed every year

FINDINGS (5y windows):
- ACCUMULATORS (>=50% of return is appreciation -> upgraded to
  "TAXABLE (accrues)" in taxplan): MBXIX 76% (0.0% payout 12m),
  ATESX 66%, LAMHX 62%, CVSIX 61%, and candidate PBAIX 60%
  (0.0% payout 12m). For these the taxable account's LTCG-on-sale
  benefit is real and large - the annual drag is tiny.
- PAY-OUT funds (return comes almost entirely as distributions):
  HMEZX (99% paid out - the STCG merger-arb case), MERVX, COSIX,
  PMORX, SVARX, SCFZX, DMSZX, all munis, EM/HY credit. IRA stands.
- DATA ARTIFACTS caught: JLPSX & QSPNX show one-time NAV gap events
  ~2022 (special distribution or share-class reorg) - the 5y payout
  average (13%/yr) is misleading; the 12m payout column (0-3%)
  reflects current behavior. QCMMRX (MMF) price data is a
  total-return index, not NAV - its "100% appreciation" is
  meaningless. Hence both the 5y and the 12m payout are shown.

CAPITAL-LOSS DISTRIBUTIONS: registered US RICs CANNOT distribute
net realized capital losses to shareholders (code) - losses stay in
the fund as a carryforward. The taxable-account benefit the user was
looking for exists as the fund's HARVEST RESERVE: a fund that
realized losses in down years (2022/2025) offsets FUTURE gain
distributions, so later appreciation converts to distributions
without triggering a gain distribution. Detectable only from the
annual report (N-CSR) / 1099 history: low-or-zero capital-gain
distributions after strong up-years. K-1 vehicles (MLPs, some
private credit) DO pass through losses, but they're ordinary-income
vehicles -> IRA anyway. NEXT: pull N-CSR per-share distribution
tables for the MIXED funds (PMAIX/PMFKX, QSPNX, MBXIX, EGRIX,
DMSZX, PBAIX, FCVSX/FISCX, MMUFX/PRUAX, PABAX, WISEX).

### Tax-location plan (fundlab/taxplan.py, 2026-08-27)
User's premise: current LTCG rate < future ordinary rate, so a fund's
account placement follows the CHARACTER of its distributions:
- qualified dividends + LTCG  -> TAXABLE (low LTCG rate is the benefit)
- tax-exempt (munis)          -> TAXABLE (wasted in an IRA)
- ordinary interest / STCG / REIT / K-1 -> IRA (deferral is the benefit)
- cash                        -> FLEXIBLE

No 1099-DIV characterizations on file for 2,400 funds, so this is a
STRUCTURAL estimate. `score` = estimated share of distributions that
are tax-favorable (QD + LTCG + tax-exempt), from three tiers of ground
truth:
1. 16-fund shortlist -> nport_cache buckets (keyword)
2. 22 cross-checked  -> xcheck_report buckets (SEC assetCat/issuerCat)
3. 250 candidates    -> factor sleeve loadings (return proxy)
Fallback: if the keyword parser left >50% of a book unclassified
("Other"), use the return sleeves. Manual override for the Leuthold
wrappers (no return history).

Location bands: score >= 0.60 TAXABLE, <= 0.35 IRA, else MIXED (check
the 1099). Name-based overrides: muni name -> TAXABLE (munis), money
market -> FLEXIBLE, and strategy caps (merger-arb/event-driven capped
at 0.35 because gains are mostly SHORT-TERM; market-neutral 0.35;
hedge 0.50; style-premia 0.50).

KEY FINDINGS:
- 16 shortlist: TAXABLE = ATESX, JLPSX, LAMHX, LCORX, LCRIX (all
  equity). IRA = ATRFX, COSIX, CVSIX, PMORX, SVARX, EAGMX/EGRSX
  (macro/market-neutral/income). MIXED (check 1099) = MBXIX (hedge),
  QSPNX (AQR factor), PMAIX/PMFKX (multi-asset income, same fund two
  classes).
- 22 cross-checked: the four MUNIS (BTMIX, FHMIX, HICOX, USMSX) ->
  TAXABLE, everything else IRA except the macro FOFs (EGRIX MIXED,
  ETSIX IRA) and DMSZX (MIXED, 38% equity + 36% CLO).
- MERGER-ARB TRAP: HMEZX + MERVX hold ~75% equity (looks tax-
  efficient) but their distributions are mostly SHORT-TERM gains
  (deals close <1 yr) -> IRA, not taxable. This is the one place the
  equity-looking book is misleading.
- 250 candidates: 109 munis (TAXABLE), 127 IRA (bonds/credit/HY/loans),
  6 equity TAXABLE (PHSTX, ANNPX, FKUTX, ALGRX, EBSAX, MCOAX), 7
  MIXED, 1 FLEXIBLE. The candidate pool is credit-heavy, so most are
  IRA.

App: Fund Lab -> "Tax location" expander (shortlist + cross-checked
+ 250-candidate tables, location filter). Output:
fundlab/taxplan_results.json. The last 1099-DIV is the final arbiter
for any MIXED fund.

### Drawdown-resilience screen (fundlab/drawdown.py, 2026-08-27)
Scenarios DETECTED from IVV (S&P 500) - one worst peak->trough per
calendar year since 2022, min depth 8% (2024's Aug-5 dip and 2023's
rate shock are just under 10%, so a 10% floor would silently drop
them):
- 2022 bear mkt     2022-01-03 -> 2022-10-12  -24.5%
- 2023 rate shock   2023-07-31 -> 2023-10-27  -9.9%
- 2024 vol spike    2024-07-16 -> 2024-08-05  -8.4%
- 2025 tariff crash 2025-02-19 -> 2025-04-08  -18.8%
- 2026 Q1 drawdown  2026-01-28 -> 2026-03-30  -8.9%

Fund return = its own NAV, first print after the peak to the last
print on/before the trough (per-fund dates, no reindexing). 2,384
funds screened; the 250 CANDIDATEs ranked by # scenarios positive.

FINDINGS:
- Positive in ALL 5: only 7 funds, and ALL are ultra-short/cash
  (BILS, QCMMRX, PULS, FHCOX, FHMIX, SAFEX, COIAX). Being positive
  through every equity drawdown is mostly a DURATION property, not
  alpha - the honest read of the 5/5 tier.
- The interesting tier is 4/5 WITH real 5y alpha:
  - HMEZX merger arb  +1.5% (2022) +3.1% (2023) +0.1% (2024)
    -0.4% (2025) +0.4% (2026), t5 +7.1, corr +0.14 - the standout:
    genuinely positive in the two biggest equity crashes.
  - MERVX merger arb  +0.2/+2.6/0.0/+0.5/+0.4, t5 +2.7, corr +0.19.
  - CBHCX market-neutral -5.4 (2022) but +3.1 (2023) +4.5 (2026),
    t5 +2.4 - a true equity hedge.
  - SCFZX securitized credit -2.6 (2022) then ~flat/small, t5 +8.4,
    corr +0.16.
  - ENIAX SIIT opportunistic t5 +10.1 (highest alpha in the set),
    only small 2025 dip.
  - WMNUX -2.6 (2022) then ~flat, t5 +6.9.
  - RCTIX -5.6 (2022, its one weak spot) then positive x4, t5 +5.6.
- EBSAX Campbell Systematic Macro: +35.9% in the 2022 bear market,
  +5.0% in 2026 Q1, but -4.1 (2024) -2.7 (2025) - a 2022/2026 macro
  winner, 3/5.

App: Fund Lab -> "Drawdown resilience" expander (scenario table +
candidate table sorted by # positive). Output:
fundlab/drawdown_results.json.

### N-PORT cross-check (fundlab/xcheck.py, 2026-08-27)
21 of 22 top candidates resolved to their ACTUAL holdings (qcmmrx =
money-market account, no holdings to parse).

Pipeline hard-won facts:
- The fund's NPORT is usually filed under the fund's OWN registrant
  (browse-edgar ticker->CIK), not the 497-cover family/trust CIK
  (EV: trust 1552324 vs fund 745463). covers.json of the own CIK gives
  the exact series name -> the ONLY reliable disambiguator between
  sibling funds ("...Absolute Return Fund" vs "...Advantage Fund" share
  6 of 7 words; token-overlap alone can't tell them apart).
- Big trusts file dozens of NPORT-Ps per quarter (one per fund); a
  flat newest-30 window misses the fund's own filing. Take all filings
  on the 4 most recent distinct dates.
- The submissions API points at the XSL-RENDERED view
  (xslFormNPORT-P_X01/primary_doc.xml, a 5-20MB HTML page); the raw
  schema data (seriesName/invstOrSec/netAssets) sits at
  accession-root primary_doc.xml (5KB-1.5MB). Strip the xsl prefix.
- The raw file is malformed XHTML (CSS in <style>) - strict ET fails;
  regex field extraction on the fixed schema works.
- nport._SECTION needed an "INVESTMENT PORTFOLIO (unaudited)" variant
  (NPORT-EX Sch-F files); find_section got a frac token-tolerance
  param (Yahoo names drift from filing names).
- EDGAR FTS lags ~1yr on recent NPORTs - don't use it for discovery.
- assetCat codes (ABS-O, ABS-CBDO, DBT, DIR, LON, STIV, RA, DE...) mix
  asset types within a code - bucket by position NAME keywords, show
  the raw code per row.

VERDICTS (screen t5 = 5y alpha t, full-sleeve R2):
Genuine idiosyncratic alpha (holdings confirm the returns story):
- hmezx t5+7.1: MERGER ARB - 73% equity in deal targets (Hologic,
  Clearwater, Sealed Air, OneStream, Air Lease, CSG, EA, Semrush,
  Masimo) + 22% escrow. Textbook.
- mervx t5+2.7: MERGER ARB - 83% equity targets + GS escrow.
- egrix t5+4.9: MACRO WRAPPER - 100% in "Global Macro Absolute Return
  Advantage Portfolio" (managed acct; underlying NOT disclosed in
  NPORT). Structure confirms pure macro; alpha is the SMA's.
- etsix t5+4.1: FOF of EV internal accounts - 76% "Global
  Opportunities Portfolio" (multi-strat), 13% macro, +HI/EM.
  Opaque but genuinely multi-strat.
- wmnux t5+6.9: ALTERNATIVE CREDIT - 67 discounted/zero-coupon
  corporate bonds (Centrus 0%, Datadog 0%, Ormat 2.5%, N. Oil 3.6%) +
  61 equity SWAPS (BNP counterparty: Tetra Tech, Akamai, AeroVironment,
  Synaptics, Etsy...) = synthetic equity overlay. The "equity names" in
  the book are bond issuers / swap underlyings, not stock holdings.
  Alpha = credit selection + synthetic equity, not any sleeve.
- scfzx t5+8.4: SECURITIZED CREDIT - 672 pos: CLOs (Aurium, TikeHau,
  Palmer Sq, Harot) + CMBS + ABS. Missing-factor sleeve.
- coiax t5+7.0: PRIVATE CREDIT / bank loans - 32% in 3 unlisted loan
  blocks (RA) + bank ABS. Missing factor.
- rctix t5+5.6: DISTRESSED/turnaround credit - DISH DBS, Puerto Rico
  GDB Debt Recovery, PR Commonwealth, Deutsche Bank, Cablevision,
  Avant credit-card ABS. Genuinely idio credit.
Missing-factor (real assets absent from the 35-sleeve set; alpha =
exposure, not skill):
- hicox t5+3.0 (Colorado munis 98%), fhmix t5+3.4 (muni microshort),
  usmsx t5+2.9 (ultrashort muni, 1092 pos), btmix t5+2.8 (short muni,
  1832 pos), aguax t5+3.7 (EM developing-world debt: Ghana, Argentina,
  Kenya, Angola + 46% EM equity), femdx t5+3.0 (EM sovereign: Brazil,
  Ethiopia, Turkiye, Mexico, Venezuela), lpxax t5+2.6 (now mostly
  bank/financial corporate debt - Truist, Citi, UBS, RBC, Enbridge,
  HSBC, TD, Goldman - only 17 prefs remain; rotated out of prefs),
  anglx t5+2.4 (agency MBS 63% + CMBS + senior loans),
  aflix t5+3.9 (levered loans: Energy Transfer, Caesars, Nissan,
  Aramark - semi-transparent), dmszx t5+3.5 (FOF + direct CLOs -
  semi-transparent).
Short-duration CARRY artifacts (alpha ~= carry; a short-duration sleeve
  would explain them - lower priority as complements):
- fhcox t5+6.3 (short bank notes: StanChart, MUFG, BNP, SG),
  dultx t5+2.9 (ultrashort bank debt: Santander, Telstra, HSBC),
  safex t5+2.7 (88% T-bills/Treas + MMF - pure cash carry).

CONCLUSION: the screen's top candidates are REAL - the highest t5 names
(hmezx, mervx, egrix, wmnux, scfzx, coiax, rctix) hold exactly what
their returns imply, and none is a data artifact. The "missing-factor"
funds (munis, EM debt, preferreds, agency MBS, securitized credit,
private credit) are genuine exposures the model lacks sleeves for.
The 3 carry funds are genuine but their excess is short-duration
carry, not alpha. Output: fundlab/xcheck_report.json +
fundlab/xcheck_run.log; app shows it under Fund Lab.

### Factor screen + clusters (iteration 2, 2026-08-27)
Design (per user: overinclusive, NO portfolio-corr screening - high
corr funds are REPLACEMENTS; group funds by return driver):
- fundlab/factors.py: full OLS of every fund's daily total returns on
  a 35-driver basis (was 21): +lqd (IG corp), +hyg, +pff (prefs),
  +emb (EM debt), +tip, +shy, +vtv (value), +8 sectors (xl*), +dbmf
  (CTA), +dbb (commodities). Dropped from the regression basis:
  vea/vug (>0.95 dupes of efa/qqq), **finux (TERMINATED 2017 - it
  silently zeroed the complete-case mask for every fund overlapping its
  lifetime; the v1 forward selector never hit this because sleeves
  were optional, full OLS needs a common sample)**, bil (its shv/bil
  pair let OLS express tiny net cash exposure as huge offsetting
  loadings shv+64/bil-54 - noise fits with R2~0.01, |t|<2).
  vblix residualized on [ivv, tlt] (tlt~vblix 0.963) -> pure vol axis.
  Small ridge (0.02, slopes only) stabilizes loadings; winsorize at
  +/-4 for clustering. Output: factor_results.json (per-fund
  full+5y loadings, R2, alpha, t + carried screen metadata).
- fundlab/cluster.py: (a) hierarchical avg-linkage tree (saved,
  4-8s) - but **fixed-k tree cuts are degenerate here**: most funds
  are multi-sleeve blends, so the tree lumps ~2250 of them into one
  blob and only peels pure single-sleeve clusters. (b) **k-means
  (k-means++ init, numpy, deterministic) is the useful grouping** -
  the app re-runs it live for any k (2384x35 is milliseconds).
  Labels = top |median loading| per cluster (>=0.30 gate, else
  "no dominant driver (balanced/idio)").
- App Fund Lab: "Return-driver clusters" expander - k slider
  (10-60, default 30), summary table (label, n, t5>=2 count, top
  fund by |t5|), selectbox -> member table sorted by alpha t desc
  (corr port as info column only).

Findings (k=30): the universe decomposes into readable driver groups:
US large blend (232), short T 1-3y (194), cash/ultra-short (185),
US small (128+70+47), US value (115), intl developed (109), growth-
tilted blend (79+27, loadings ivv+1.07/vtv-0.68), healthcare (15),
plus a 686-fund "no dominant driver (balanced/idio)" blob - the
largest single group, where the real candidate hunting happens (84
funds with t5>=2). The screen's 250 candidates now have a driver
address: e.g. hmezx (merger arb) and egrix (macro) sit in the
balanced/idio blob with no sleeve to blame; rctix/dflex sit in
short-duration; aguax/femdx in the EM group.

Caveats: loadings are FULL-history (5y fallback) and ridge-shrunk -
they describe the driver, not a tradable weight; the balanced/idio
blob is big by construction (most funds ARE mixes); Japan/China/India
single-country funds have no country sleeve and read as weak/idio -
a known coverage gap.

Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the
project (fundlab/overnight.log, fundlab/universe_cache/), and use
`setsid nohup ... < /dev/null &` so a closed shell can't kill the job.

### Watchdog incident (2026-08-26 ~22:00)
- First watchdog used `pgrep -f "fundlab.overnight"` for liveness.
  **Failed test**: a monitoring one-liner whose command line merely
  CONTAINED the string made pgrep match the monitor's own bash -c
  wrapper -> watchdog reported "alive" while the real run was dead.
  Lesson: `pgrep -f` on a distinctive-but-quotable string is fragile;
  any shell quoting the string is a false positive. Use a PID file the
  process writes itself (`fundlab/overnight.pid`), pattern only as
  fallback for pre-pidfile launches.
- Fixed watchdog restarted; it detected the dead run on its first
  check, relaunched, and the run resumed at "1823 missing" (137
  symbols already cached - exactly the expected resume).
- Also hardened stage_download: a hung goget batch (TimeoutExpired)
  no longer kills the whole run - caught per batch.
- 2026-08-26 16:01 === stage verify ===
- 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
- 2026-08-26 16:01 verify: 500/9149
- 2026-08-26 16:02 verify: 1000/9149
- 2026-08-26 16:03 verify: 1500/9149
- 2026-08-26 16:03 verify: 2000/9149
- 2026-08-26 16:04 verify: 2500/9149
- 2026-08-26 16:04 verify: 3000/9149
- 2026-08-26 16:05 verify: 3500/9149
- 2026-08-26 16:05 verify: 4000/9149
- 2026-08-26 16:06 verify: 4500/9149
- 2026-08-26 16:06 verify: 5000/9149
- 2026-08-26 16:07 verify: 5500/9149
- 2026-08-26 16:07 verify: 6000/9149
- 2026-08-26 16:08 verify: 6500/9149
- 2026-08-26 16:09 verify: 7000/9149
- 2026-08-26 16:09 verify: 7500/9149
- 2026-08-26 16:10 verify: 8000/9149
- 2026-08-26 16:10 verify: 8500/9149
- 2026-08-26 16:11 verify: 9000/9149
- 2026-08-26 16:11 verify done: 10260/10372 with data
- 2026-08-26 16:11 === overnight run finished in 0.2h ===
- 2026-08-26 21:31 === stage select ===
- 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
- 2026-08-26 21:31 === overnight run finished in 0.0h ===
- 2026-08-26 21:31 === stage download ===
- 2026-08-26 21:31 download: 1960 missing symbols
- 2026-08-26 21:54:22 watchdog: watchdog started (interval 300s)
- 2026-08-26 21:54:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 21:59:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:04:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:08:01 watchdog: watchdog started (interval 300s)
- 2026-08-26 22:08:01 watchdog: process DEAD with 2384 funds unscreened - relaunching
- 2026-08-26 22:08 === stage download ===
- 2026-08-26 22:08 download: 1823 missing symbols
- 2026-08-26 22:09:01 watchdog: relaunch confirmed alive
- 2026-08-26 22:14:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:19:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:24:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:29:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:34:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:39:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:41 download: batch 1 -> 200/200
- 2026-08-26 22:44:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:49:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:54:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:59:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:04:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:09:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:14:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:14 download: batch 2 -> 200/200
- 2026-08-26 23:19:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:24:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:29:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:34:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:39:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:44:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:47 download: batch 3 -> 200/200
- 2026-08-26 23:49:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:54:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:59:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:04:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:09:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:14:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:19:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:20 download: batch 4 -> 200/200
- 2026-08-27 00:24:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:29:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:34:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:39:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:44:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:49:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:54 download: batch 5 -> 200/200
- 2026-08-27 00:54:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:59:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:04:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:09:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:14:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:19:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:24:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:27 download: batch 6 -> 200/200
- 2026-08-27 01:29:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:34:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:39:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:44:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:49:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:54:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:59:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:00 download: batch 7 -> 200/200
- 2026-08-27 02:04:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:09:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:14:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:19:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:24:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:29:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:33 download: batch 8 -> 200/200
- 2026-08-27 02:34:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:39:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:44:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:49:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:54:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:59:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:04:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:06 download: batch 9 -> 200/200
- 2026-08-27 03:09:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:10 download: batch 10 -> 23/23
- 2026-08-27 03:10 download done: 0 still missing (no Yahoo data?)
- 2026-08-27 03:10 === stage screen ===
- 2026-08-27 03:10 screen: 2384 funds, 2384 to do
- 2026-08-27 03:11 screen: 100/2384
- 2026-08-27 03:11 screen: 200/2384
- 2026-08-27 03:12 screen: 300/2384
- 2026-08-27 03:12 screen: 400/2384
- 2026-08-27 03:13 screen: 500/2384
- 2026-08-27 03:13 screen: 600/2384
- 2026-08-27 03:14:06 watchdog: overnight alive, still_missing=1784
- 2026-08-27 03:14 screen: 700/2384
- 2026-08-27 03:14 screen: 800/2384
- 2026-08-27 03:15 screen: 900/2384
- 2026-08-27 03:15 screen: 1000/2384
- 2026-08-27 03:16 screen: 1100/2384
- 2026-08-27 03:16 screen: 1200/2384
- 2026-08-27 03:17 screen: 1300/2384
- 2026-08-27 03:17 screen: 1400/2384
- 2026-08-27 03:18 screen: 1500/2384
- 2026-08-27 03:18 screen: 1600/2384
- 2026-08-27 03:19:06 watchdog: overnight alive, still_missing=784
- 2026-08-27 03:19 screen: 1700/2384
- 2026-08-27 03:19 screen: 1800/2384
- 2026-08-27 03:20 screen: 1900/2384
- 2026-08-27 03:20 screen: 2000/2384
- 2026-08-27 03:21 screen: 2100/2384
- 2026-08-27 03:22 screen: 2200/2384
- 2026-08-27 03:22 screen: 2300/2384
- 2026-08-27 03:23 screen done: 2384 funds
- 2026-08-27 03:23 === stage finalize ===
- 2026-08-27 03:23 finalize: 2384 funds screened
- 2026-08-27 03:23    1336  sleeve mix
- 2026-08-27 03:23     707  weak/unstable alpha
- 2026-08-27 03:23     250  CANDIDATE
- 2026-08-27 03:23      42  alpha, but correlated with current portfolio
- 2026-08-27 03:23      32  alpha in 5y window, but not persistent
- 2026-08-27 03:23      17  no 5y window
- 2026-08-27 03:23   candidates v1 name-filter would have missed: ['abhyx', 'acfix', 'acitx', 'acthx', 'aflix', 'aftex', 'agovx', 'aguax', 'aidax', 'alcax', 'algrx', 'alnyx', 'althx', 'amaax', 'amhix', 'annpx', 'asdvx', 'atfax', 'avaax', 'bchyx', 'bcitx', 'bils', 'bmbix', 'bsnix', 'btmix', 'cfmox', 'cfnlx', 'cfrax', 'coiax', 'coltx', 'cpxax', 'csdax', 'cshix', 'cxhyx', 'deffx', 'delix', 'dflex', 'ditex', 'dmusx', 'dnmzx', 'dpiax', 'drcax', 'drnjx', 'dsibx', 'dultx', 'dvmhx', 'dvtax', 'eablx', 'eacax', 'eanax', 'eascx', 'ebsax', 'eniax', 'etazx', 'etgax', 'ethyx', 'etmdx', 'etmnx', 'etmox', 'etncx', 'etnjx', 'etorx', 'etpax', 'etsix', 'etvax', 'evfax', 'exmax', 'exnax', 'fafrx', 'faztx', 'fcamx', 'fcfix', 'fcotx', 'fcstx', 'fcvsx', 'fdmmx', 'femdx', 'ffrsx', 'fhaix', 'fhcox', 'fhmix', 'ficnx', 'fiscx', 'fkcix', 'fklax', 'fknix', 'fkstx', 'fktfx', 'fktix', 'fkutx', 'fkytx', 'flrn', 'fltdx', 'fltmx', 'fltr', 'fmdtx', 'fmoax', 'fmotx', 'fnmtx', 'fnytx', 'fpntx', 'fralx', 'frcox', 'frmox', 'frnjx', 'frorx', 'frpax', 'frstx', 'fsazx', 'ftazx', 'ftfmx', 'ftgax', 'ftlax', 'ftnyx', 'ftorx', 'fwiax', 'fxncx', 'gbonx', 'giyix', 'gtfbx', 'hicox', 'hmezx', 'hubax', 'icifx', 'ihiax', 'intax', 'kctax', 'kmdnx', 'kntax', 'lansx', 'lmsfx', 'lpxax', 'lsbdx', 'lsyux', 'ltebx', 'ltnyx', 'lubax', 'masax', 'mcoax', 'mdxbx', 'mervx', 'mfalx', 'mfarx', 'mfiax', 'mfpax', 'mfscx', 'mfsmx', 'mfssx', 'mfwvx', 'mgfox', 'missx', 'mmgax', 'mmhyx', 'mmufx', 'msncx', 'msnyx', 'msvax', 'mtbax', 'mtlfx', 'nefzx', 'nhmax', 'nmbax', 'nmdax', 'nmuix', 'nnjax', 'nnyax', 'nsmmx', 'nthex', 'olcax', 'omifx', 'onjcx', 'opcax', 'opnyx', 'orncx', 'pabax', 'pamfx', 'pbaix', 'pbcax', 'pgsix', 'phizx', 'phstx', 'ppnax', 'prfhx', 'prfrx', 'prfsx', 'prinx', 'prmdx', 'prnyx', 'prsmx', 'prtax', 'pruax', 'prvax', 'prvbx', 'prxcx', 'psnyx', 'ptebx', 'puls', 'pysix', 'qcmmrx', 'rctix', 'rlvsx', 'rmuyx', 'rpidx', 'rpifx', 'safex', 'scfzx', 'sdsax', 'sefix', 'seimx', 'shtix', 'sitex', 'smlax', 'sngvx', 'szmax', 'taftx', 'tisix', 'trbux', 'trhyx', 'tsdjx', 'twtix', 'ulst', 'usiax', 'usmsx', 'vcaix', 'vcitx', 'vctfx', 'vidax', 'vklmx', 'vkmmx', 'vmatx', 'vmltx', 'vnjtx', 'vnytx', 'vohix', 'vpaix', 'vtipx', 'vubfx', 'vusb', 'vwahx', 'vweax', 'vwitx', 'vwltx', 'wisex']
- 2026-08-27 03:23 === overnight run finished in 5.2h ===
- 2026-08-27 03:24:06 watchdog: process gone but screen COMPLETE (0 missing) - watchdog exiting
- 2026-08-27 03:24:06 watchdog: watchdog exit