gmp/f
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
8a9ca25750
f/fundlab/RESEARCH.md
Greg Pomerantz 8a9ca25750 Tax-location plan: taxable account vs IRA per fund 
fundlab/taxplan.py categorizes the 16-fund shortlist, the 22 N-PORT
cross-checked candidates, and all 250 screened candidates by the
expected CHARACTER of their distributions, given the user's premise
that the current LTCG rate < the post-retirement ordinary rate:

  qualified div + LTCG      -> TAXABLE  (score >= 0.60)
  tax-exempt (munis)        -> TAXABLE
  ordinary / STCG / REIT    -> IRA      (score <= 0.35)
  in between                -> MIXED (pull the 1099-DIV)
  cash                      -> FLEXIBLE

score = estimated share of distributions that are tax-favorable,
from three tiers of ground truth: N-PORT keyword buckets (16), SEC
assetCat/issuerCat buckets (22), sleeve loadings (250), with a
sleeve fallback when the keyword parser left >50% of a book
unclassified, and a manual override for the Leuthold wrappers
(91.7% Leuthold Core ETF, no return history yet).

Key findings:
- shortlist: TAXABLE = ATESX, JLPSX, LAMHX, LCORX, LCRIX (equity);
  IRA = ATRFX, COSIX, CVSIX, PMORX, SVARX, EAGMX/EGRSX;
  MIXED = MBXIX, QSPNX, PMAIX/PMFKX (same fund, two classes)
- cross-checked: 4 munis -> TAXABLE; HMEZX + MERVX are the merger-
  arb trap - equity-looking books whose distributions are mostly
  SHORT-TERM gains -> IRA
- candidates: 109 munis TAXABLE, 127 IRA, 6 equity TAXABLE, 7 MIXED

App: Fund Lab "Tax location" expander. Output:
fundlab/taxplan_results.json. Tests: test_taxplan() (9 checks).
97/32 suites green.
2026-08-27 14:08:15 -04:00

515 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Fund discovery research log
Running log of attempts, dead ends, and learnings for finding candidate
funds (alpha-driven, portfolio-complementing) with the `fundlab`
pipeline. Newest entries at the bottom of each section. Update as you
go — this file IS the knowledge base.
## Sources & what works
| Source | Status | Notes |
|---|---|---|
| Local stocks DB (`~/prog/fin/stocks/`, 8,250 syms) | ✅ primary | ~100 open-end alt families already present; `dbmine.py` mines by name pattern |
| SEC full-index `Archives/edgar/full-index/YYYY/QTRn/company.gz` | ✅ **discovery workhorse** | lists EVERY filing; 497/497K filers = all active open-end funds. 4-quarter union = 1,668 CIKs, 33,188 series, 10,372 class tickers |
| SEC full-submission `.txt` (per accession) | ✅ | ~10–50KB; line-based SGML cover with `<SERIES-NAME>` (UNCLOSED tag) + `<CLASS-CONTRACT-TICKER-SYMBOL>` per class; often several series per filing |
| Yahoo chart API (`query1.../v8/finance/chart/<T>`) | ✅ | no crumb needed; meta has instrumentType/exchange/longName; `range=20y` gives history length |
| `goget` (`~/go/bin/goget`) | ✅ | batch downloader, idempotent, ~2–5s/sym |
| EDGAR FTS (`efts.sec.gov/LATEST/search-index`) | ⚠️ fragile | phrase queries w/ hyphens fail; AND-semantics; 100-hit cap per query → incomplete for common phrases. OK for rare phrases only |
| SEC `browse-edgar` company listing | ❌ dead | now JS-rendered, no data in HTML |
| SEC `/files/investment-company-tickers.json` | ❌ doesn't exist | 404 (misremembered) |
| SEC `company_tickers.json` | ❌ for our purpose | only exchange-listed (ETFs/CEFs/stocks); open-end OTC funds absent |
| Yahoo search/crumb API | ❌ throttled | IP-level "Too Many Requests" on `fc.yahoo.com`/`getcrumb`; chart API unaffected |
| stockanalysis.com/funds/ | ❌ 404 | path guessed wrong, not pursued |
## Learnings (hard-won)
1. **Famous multi-strategy/macro funds are private/offshore** — Millennium,
Balyasny, Schonfeld, ExodusPoint, Two Sigma, Winton, Marshall Wace,
Brevan Howard, AQR Event-Driven: no US open-end class, no EDGAR 497,
no Yahoo OTC ticker. Structural, not a search failure.
2. **Precision over recall for name→ticker resolution**: a guessed
ticker that "looks right" is worse than no answer. Chart-API name
gate + 2/3 token overlap rejected 23/24 memory-based guesses.
3. **Yahoo exchange name is a USELESS fund/ETF discriminator**: OTC
mutual funds report `fullExchangeName="Nasdaq"`. Use
`instrumentType` (MUTUALFUND vs ETF).
4. **497 SGML cover uses UNCLOSED tags, one per line** —
`<SERIES-NAME>Foo Fund\n<CLASS-CONTRACT-TICKER-SYMBOL>TNMAX\n`.
Closed-tag regexes find nothing; `to_text()` (tag stripping)
destroys the data. Parse line-based, BEFORE any tag stripping.
5. **Full-index columns drift** — don't trust fixed widths; the header
line and data rows don't align. Regex the whole line.
6. **One quarter ≠ the universe**: each fund's annual base-497
re-filing lands in a random quarter; union of 4 consecutive
quarters is the full active universe (1,209 in Q2 alone → 1,668
union).
7. **Accession paths in the index are relative to `/Archives/`**, not
`/Archives/edgar/data/` — doubling the prefix 404s.
8. **Amendments (497A/497VPU) may lack the series cover** — base 497 /
497K carry it. (497A inclusion pending — see below.)
9. **Family CIKs repeat across the index** (e.g. AB under 2 CIKs,
same series listed twice) — dedupe by series name, not CIK.
10. **Large-n BIC is knife-edge** (ΔBIC=2 ≈ ΔR²=0.0008 at n=2500) —
the |t|>2 gate on added regressors is essential (decompose.py).
11. **The portfolio is 50% market-neutral (qspnx)** — MN/L-S-equity
funds show the strongest alpha on screen (+12–17%/yr) but corr
0.35–0.76 with the portfolio. "Alpha" ≠ "diversifying for YOU".
12. **Near-duplicate sleeves make OLS knife-edge** — distinct-axis
candidate sets (one rep per sleeve family) or the betas split
arbitrarily between ivv/vti/vt.
13. **Wrong-fund objectives are worse than none** — all EDGAR
resolution stages gate on name match before accepting.
## Pipeline stages (current)
```
full-index (4 qtrs) → per-CIK latest 497/497K .txt (cached, 4-thread)
→ parse_cover (line-based SGML)
→ [name filter — REMOVED in v2, kept as a tag]
→ Yahoo chart verify (instrumentType, 20y length)
→ share-class dedupe (longest history)
→ goget missing (batched)
→ screen_fund (sleeve OLS, BIC fwd-select, 5y alpha t, persistence,
corr vs portfolio & benchmark) [0.3s/fund]
→ search_*.json → app Fund Lab "Alpha search" table
```
## Overnight comprehensive screen (v2, started 2026-08-26)
Goal: screen EVERY OTC open-end fund with ≥5y history from the 497
universe — no name pre-filter. Name match becomes a tag/cross-check,
not a gate.
### Notes while running (live - the log() below appends automatically)
- [x] survey: 10,372 class tickers in covers cache; 1,223 already local,
9,149 to verify on Yahoo; screen costs 0.3s/fund.
- [x] 497A adds ZERO new CIKs over 497/497K (every 497A filer also
filed a base in window) - no need to include.
- [x] verify: 10,260/10,372 tickers with data (4-thread, ~10 min;
112 dead/never-listed tickers dropped). Cached in
universe_cache/yahoo_meta.json.
- [x] select: **2,384 funds** (407 local, 1,977 external). Only 54
match the alpha name pattern - the other 2,330 are exactly what
the v1 name-filter would have missed. Worklist cached in
universe_cache/selected.json.
- [x] download: 1,960 symbols (resumed at 1,823 after the watchdog
kill-test). Logs to fundlab/overnight.log + this file.
- [x] screen: 2,384 funds -> search_all.json (1.2MB).
- [x] finalize: verdict counts + "candidates v1 would have missed".
### RESULTS (run finished 2026-08-27 03:23, 5.2h, watchdog exited clean)
- 2,384 screened: 250 CANDIDATE, 1336 sleeve mix, 707 weak, 42
alpha-but-correlated, 32 not-persistent, 17 no-5y-window.
- Of the 250: ~37 are short-duration/floating-rate/money-market CARRY
(model artifact - the 21-sleeve set has no short-duration axis, so
carry shows up as a positive intercept). The rest mix:
a) GENUINE idiosyncratic alpha: hmezx NexPoint Merger Arb (t5 7.1,
tF 4.6, corr 0.14), mervx The Merger Fund, egrix Eaton Vance
Global Macro (R2 0.07, +7.9%), wmnux Westwood Alt Income,
dmszx Destinations Multi-Strat, anglx Angel Oak Multi-Strat,
aguax Am Beacon Dev World Income (+6.5%, t5 3.7), femdx
Franklin EM Debt Opps (+6.0%), rctix River Canyon TR Bond
(t5 5.6), kmdnx Kinetics Multi-Disciplinary.
b) MISSING-FACTOR exposures (sector not in the 21 sleeves, so
sector exposure reads as "alpha"): munis (hicox, fhmix),
preferreds (lpxax, dpiax), EM debt (femdx), securitized credit
(scfzx), TIPS/real-return.
- ~200 candidates have "boring" names the v1 alpha-name filter would
have rejected - direct quantification of the v1 blind spot.
- eniax SIIT Opportunistic Income: t5 10.1 but tF 1.44 -> alpha is
RECENT (last 5y), not full-history. Good thing the screen keeps both
stats.
### Next iterations
1. [x] **Add missing factors + cluster by return driver**
(fundlab/factors.py + fundlab/cluster.py).
2. [x] **N-PORT holdings cross-check on the top candidates**
(fundlab/xcheck.py) - results below.
3. [x] **Drawdown-resilience screen** (fundlab/drawdown.py) -
which candidates were positive when equities crashed.
4. CEF universe (485/N-2 filers) - separate pass; CEFs have
premium/discount dynamics the NAV screen can't see.
### Tax-location plan (fundlab/taxplan.py, 2026-08-27)
User's premise: current LTCG rate < future ordinary rate, so a fund's
account placement follows the CHARACTER of its distributions:
- qualified dividends + LTCG -> TAXABLE (low LTCG rate is the benefit)
- tax-exempt (munis) -> TAXABLE (wasted in an IRA)
- ordinary interest / STCG / REIT / K-1 -> IRA (deferral is the benefit)
- cash -> FLEXIBLE
No 1099-DIV characterizations on file for 2,400 funds, so this is a
STRUCTURAL estimate. `score` = estimated share of distributions that
are tax-favorable (QD + LTCG + tax-exempt), from three tiers of ground
truth:
1. 16-fund shortlist -> nport_cache buckets (keyword)
2. 22 cross-checked -> xcheck_report buckets (SEC assetCat/issuerCat)
3. 250 candidates -> factor sleeve loadings (return proxy)
Fallback: if the keyword parser left >50% of a book unclassified
("Other"), use the return sleeves. Manual override for the Leuthold
wrappers (no return history).
Location bands: score >= 0.60 TAXABLE, <= 0.35 IRA, else MIXED (check
the 1099). Name-based overrides: muni name -> TAXABLE (munis), money
market -> FLEXIBLE, and strategy caps (merger-arb/event-driven capped
at 0.35 because gains are mostly SHORT-TERM; market-neutral 0.35;
hedge 0.50; style-premia 0.50).
KEY FINDINGS:
- 16 shortlist: TAXABLE = ATESX, JLPSX, LAMHX, LCORX, LCRIX (all
equity). IRA = ATRFX, COSIX, CVSIX, PMORX, SVARX, EAGMX/EGRSX
(macro/market-neutral/income). MIXED (check 1099) = MBXIX (hedge),
QSPNX (AQR factor), PMAIX/PMFKX (multi-asset income, same fund two
classes).
- 22 cross-checked: the four MUNIS (BTMIX, FHMIX, HICOX, USMSX) ->
TAXABLE, everything else IRA except the macro FOFs (EGRIX MIXED,
ETSIX IRA) and DMSZX (MIXED, 38% equity + 36% CLO).
- MERGER-ARB TRAP: HMEZX + MERVX hold ~75% equity (looks tax-
efficient) but their distributions are mostly SHORT-TERM gains
(deals close <1 yr) -> IRA, not taxable. This is the one place the
equity-looking book is misleading.
- 250 candidates: 109 munis (TAXABLE), 127 IRA (bonds/credit/HY/loans),
6 equity TAXABLE (PHSTX, ANNPX, FKUTX, ALGRX, EBSAX, MCOAX), 7
MIXED, 1 FLEXIBLE. The candidate pool is credit-heavy, so most are
IRA.
App: Fund Lab -> "Tax location" expander (shortlist + cross-checked
+ 250-candidate tables, location filter). Output:
fundlab/taxplan_results.json. The last 1099-DIV is the final arbiter
for any MIXED fund.
### Drawdown-resilience screen (fundlab/drawdown.py, 2026-08-27)
Scenarios DETECTED from IVV (S&P 500) - one worst peak->trough per
calendar year since 2022, min depth 8% (2024's Aug-5 dip and 2023's
rate shock are just under 10%, so a 10% floor would silently drop
them):
- 2022 bear mkt 2022-01-03 -> 2022-10-12 -24.5%
- 2023 rate shock 2023-07-31 -> 2023-10-27 -9.9%
- 2024 vol spike 2024-07-16 -> 2024-08-05 -8.4%
- 2025 tariff crash 2025-02-19 -> 2025-04-08 -18.8%
- 2026 Q1 drawdown 2026-01-28 -> 2026-03-30 -8.9%
Fund return = its own NAV, first print after the peak to the last
print on/before the trough (per-fund dates, no reindexing). 2,384
funds screened; the 250 CANDIDATEs ranked by # scenarios positive.
FINDINGS:
- Positive in ALL 5: only 7 funds, and ALL are ultra-short/cash
(BILS, QCMMRX, PULS, FHCOX, FHMIX, SAFEX, COIAX). Being positive
through every equity drawdown is mostly a DURATION property, not
alpha - the honest read of the 5/5 tier.
- The interesting tier is 4/5 WITH real 5y alpha:
- HMEZX merger arb +1.5% (2022) +3.1% (2023) +0.1% (2024)
-0.4% (2025) +0.4% (2026), t5 +7.1, corr +0.14 - the standout:
genuinely positive in the two biggest equity crashes.
- MERVX merger arb +0.2/+2.6/0.0/+0.5/+0.4, t5 +2.7, corr +0.19.
- CBHCX market-neutral -5.4 (2022) but +3.1 (2023) +4.5 (2026),
t5 +2.4 - a true equity hedge.
- SCFZX securitized credit -2.6 (2022) then ~flat/small, t5 +8.4,
corr +0.16.
- ENIAX SIIT opportunistic t5 +10.1 (highest alpha in the set),
only small 2025 dip.
- WMNUX -2.6 (2022) then ~flat, t5 +6.9.
- RCTIX -5.6 (2022, its one weak spot) then positive x4, t5 +5.6.
- EBSAX Campbell Systematic Macro: +35.9% in the 2022 bear market,
+5.0% in 2026 Q1, but -4.1 (2024) -2.7 (2025) - a 2022/2026 macro
winner, 3/5.
App: Fund Lab -> "Drawdown resilience" expander (scenario table +
candidate table sorted by # positive). Output:
fundlab/drawdown_results.json.
### N-PORT cross-check (fundlab/xcheck.py, 2026-08-27)
21 of 22 top candidates resolved to their ACTUAL holdings (qcmmrx =
money-market account, no holdings to parse).
Pipeline hard-won facts:
- The fund's NPORT is usually filed under the fund's OWN registrant
(browse-edgar ticker->CIK), not the 497-cover family/trust CIK
(EV: trust 1552324 vs fund 745463). covers.json of the own CIK gives
the exact series name -> the ONLY reliable disambiguator between
sibling funds ("...Absolute Return Fund" vs "...Advantage Fund" share
6 of 7 words; token-overlap alone can't tell them apart).
- Big trusts file dozens of NPORT-Ps per quarter (one per fund); a
flat newest-30 window misses the fund's own filing. Take all filings
on the 4 most recent distinct dates.
- The submissions API points at the XSL-RENDERED view
(xslFormNPORT-P_X01/primary_doc.xml, a 5-20MB HTML page); the raw
schema data (seriesName/invstOrSec/netAssets) sits at
accession-root primary_doc.xml (5KB-1.5MB). Strip the xsl prefix.
- The raw file is malformed XHTML (CSS in <style>) - strict ET fails;
regex field extraction on the fixed schema works.
- nport._SECTION needed an "INVESTMENT PORTFOLIO (unaudited)" variant
(NPORT-EX Sch-F files); find_section got a frac token-tolerance
param (Yahoo names drift from filing names).
- EDGAR FTS lags ~1yr on recent NPORTs - don't use it for discovery.
- assetCat codes (ABS-O, ABS-CBDO, DBT, DIR, LON, STIV, RA, DE...) mix
asset types within a code - bucket by position NAME keywords, show
the raw code per row.
VERDICTS (screen t5 = 5y alpha t, full-sleeve R2):
Genuine idiosyncratic alpha (holdings confirm the returns story):
- hmezx t5+7.1: MERGER ARB - 73% equity in deal targets (Hologic,
Clearwater, Sealed Air, OneStream, Air Lease, CSG, EA, Semrush,
Masimo) + 22% escrow. Textbook.
- mervx t5+2.7: MERGER ARB - 83% equity targets + GS escrow.
- egrix t5+4.9: MACRO WRAPPER - 100% in "Global Macro Absolute Return
Advantage Portfolio" (managed acct; underlying NOT disclosed in
NPORT). Structure confirms pure macro; alpha is the SMA's.
- etsix t5+4.1: FOF of EV internal accounts - 76% "Global
Opportunities Portfolio" (multi-strat), 13% macro, +HI/EM.
Opaque but genuinely multi-strat.
- wmnux t5+6.9: ALTERNATIVE CREDIT - 67 discounted/zero-coupon
corporate bonds (Centrus 0%, Datadog 0%, Ormat 2.5%, N. Oil 3.6%) +
61 equity SWAPS (BNP counterparty: Tetra Tech, Akamai, AeroVironment,
Synaptics, Etsy...) = synthetic equity overlay. The "equity names" in
the book are bond issuers / swap underlyings, not stock holdings.
Alpha = credit selection + synthetic equity, not any sleeve.
- scfzx t5+8.4: SECURITIZED CREDIT - 672 pos: CLOs (Aurium, TikeHau,
Palmer Sq, Harot) + CMBS + ABS. Missing-factor sleeve.
- coiax t5+7.0: PRIVATE CREDIT / bank loans - 32% in 3 unlisted loan
blocks (RA) + bank ABS. Missing factor.
- rctix t5+5.6: DISTRESSED/turnaround credit - DISH DBS, Puerto Rico
GDB Debt Recovery, PR Commonwealth, Deutsche Bank, Cablevision,
Avant credit-card ABS. Genuinely idio credit.
Missing-factor (real assets absent from the 35-sleeve set; alpha =
exposure, not skill):
- hicox t5+3.0 (Colorado munis 98%), fhmix t5+3.4 (muni microshort),
usmsx t5+2.9 (ultrashort muni, 1092 pos), btmix t5+2.8 (short muni,
1832 pos), aguax t5+3.7 (EM developing-world debt: Ghana, Argentina,
Kenya, Angola + 46% EM equity), femdx t5+3.0 (EM sovereign: Brazil,
Ethiopia, Turkiye, Mexico, Venezuela), lpxax t5+2.6 (now mostly
bank/financial corporate debt - Truist, Citi, UBS, RBC, Enbridge,
HSBC, TD, Goldman - only 17 prefs remain; rotated out of prefs),
anglx t5+2.4 (agency MBS 63% + CMBS + senior loans),
aflix t5+3.9 (levered loans: Energy Transfer, Caesars, Nissan,
Aramark - semi-transparent), dmszx t5+3.5 (FOF + direct CLOs -
semi-transparent).
Short-duration CARRY artifacts (alpha ~= carry; a short-duration sleeve
would explain them - lower priority as complements):
- fhcox t5+6.3 (short bank notes: StanChart, MUFG, BNP, SG),
dultx t5+2.9 (ultrashort bank debt: Santander, Telstra, HSBC),
safex t5+2.7 (88% T-bills/Treas + MMF - pure cash carry).
CONCLUSION: the screen's top candidates are REAL - the highest t5 names
(hmezx, mervx, egrix, wmnux, scfzx, coiax, rctix) hold exactly what
their returns imply, and none is a data artifact. The "missing-factor"
funds (munis, EM debt, preferreds, agency MBS, securitized credit,
private credit) are genuine exposures the model lacks sleeves for.
The 3 carry funds are genuine but their excess is short-duration
carry, not alpha. Output: fundlab/xcheck_report.json +
fundlab/xcheck_run.log; app shows it under Fund Lab.
### Factor screen + clusters (iteration 2, 2026-08-27)
Design (per user: overinclusive, NO portfolio-corr screening - high
corr funds are REPLACEMENTS; group funds by return driver):
- fundlab/factors.py: full OLS of every fund's daily total returns on
a 35-driver basis (was 21): +lqd (IG corp), +hyg, +pff (prefs),
+emb (EM debt), +tip, +shy, +vtv (value), +8 sectors (xl*), +dbmf
(CTA), +dbb (commodities). Dropped from the regression basis:
vea/vug (>0.95 dupes of efa/qqq), **finux (TERMINATED 2017 - it
silently zeroed the complete-case mask for every fund overlapping its
lifetime; the v1 forward selector never hit this because sleeves
were optional, full OLS needs a common sample)**, bil (its shv/bil
pair let OLS express tiny net cash exposure as huge offsetting
loadings shv+64/bil-54 - noise fits with R2~0.01, |t|<2).
vblix residualized on [ivv, tlt] (tlt~vblix 0.963) -> pure vol axis.
Small ridge (0.02, slopes only) stabilizes loadings; winsorize at
+/-4 for clustering. Output: factor_results.json (per-fund
full+5y loadings, R2, alpha, t + carried screen metadata).
- fundlab/cluster.py: (a) hierarchical avg-linkage tree (saved,
4-8s) - but **fixed-k tree cuts are degenerate here**: most funds
are multi-sleeve blends, so the tree lumps ~2250 of them into one
blob and only peels pure single-sleeve clusters. (b) **k-means
(k-means++ init, numpy, deterministic) is the useful grouping** -
the app re-runs it live for any k (2384x35 is milliseconds).
Labels = top |median loading| per cluster (>=0.30 gate, else
"no dominant driver (balanced/idio)").
- App Fund Lab: "Return-driver clusters" expander - k slider
(10-60, default 30), summary table (label, n, t5>=2 count, top
fund by |t5|), selectbox -> member table sorted by alpha t desc
(corr port as info column only).
Findings (k=30): the universe decomposes into readable driver groups:
US large blend (232), short T 1-3y (194), cash/ultra-short (185),
US small (128+70+47), US value (115), intl developed (109), growth-
tilted blend (79+27, loadings ivv+1.07/vtv-0.68), healthcare (15),
plus a 686-fund "no dominant driver (balanced/idio)" blob - the
largest single group, where the real candidate hunting happens (84
funds with t5>=2). The screen's 250 candidates now have a driver
address: e.g. hmezx (merger arb) and egrix (macro) sit in the
balanced/idio blob with no sleeve to blame; rctix/dflex sit in
short-duration; aguax/femdx in the EM group.
Caveats: loadings are FULL-history (5y fallback) and ridge-shrunk -
they describe the driver, not a tradable weight; the balanced/idio
blob is big by construction (most funds ARE mixes); Japan/China/India
single-country funds have no country sleeve and read as weak/idio -
a known coverage gap.
Infra lesson: /tmp gets cleaned mid-run - keep logs + caches in the
project (fundlab/overnight.log, fundlab/universe_cache/), and use
`setsid nohup ... < /dev/null &` so a closed shell can't kill the job.
### Watchdog incident (2026-08-26 ~22:00)
- First watchdog used `pgrep -f "fundlab.overnight"` for liveness.
**Failed test**: a monitoring one-liner whose command line merely
CONTAINED the string made pgrep match the monitor's own bash -c
wrapper -> watchdog reported "alive" while the real run was dead.
Lesson: `pgrep -f` on a distinctive-but-quotable string is fragile;
any shell quoting the string is a false positive. Use a PID file the
process writes itself (`fundlab/overnight.pid`), pattern only as
fallback for pre-pidfile launches.
- Fixed watchdog restarted; it detected the dead run on its first
check, relaunched, and the run resumed at "1823 missing" (137
symbols already cached - exactly the expected resume).
- Also hardened stage_download: a hung goget batch (TimeoutExpired)
no longer kills the whole run - caught per batch.
- 2026-08-26 16:01 === stage verify ===
- 2026-08-26 16:01 verify: 10372 tickers, 9149 to hit Yahoo
- 2026-08-26 16:01 verify: 500/9149
- 2026-08-26 16:02 verify: 1000/9149
- 2026-08-26 16:03 verify: 1500/9149
- 2026-08-26 16:03 verify: 2000/9149
- 2026-08-26 16:04 verify: 2500/9149
- 2026-08-26 16:04 verify: 3000/9149
- 2026-08-26 16:05 verify: 3500/9149
- 2026-08-26 16:05 verify: 4000/9149
- 2026-08-26 16:06 verify: 4500/9149
- 2026-08-26 16:06 verify: 5000/9149
- 2026-08-26 16:07 verify: 5500/9149
- 2026-08-26 16:07 verify: 6000/9149
- 2026-08-26 16:08 verify: 6500/9149
- 2026-08-26 16:09 verify: 7000/9149
- 2026-08-26 16:09 verify: 7500/9149
- 2026-08-26 16:10 verify: 8000/9149
- 2026-08-26 16:10 verify: 8500/9149
- 2026-08-26 16:11 verify: 9000/9149
- 2026-08-26 16:11 verify done: 10260/10372 with data
- 2026-08-26 16:11 === overnight run finished in 0.2h ===
- 2026-08-26 21:31 === stage select ===
- 2026-08-26 21:31 select: 2384 funds (407 local, 1977 external)
- 2026-08-26 21:31 === overnight run finished in 0.0h ===
- 2026-08-26 21:31 === stage download ===
- 2026-08-26 21:31 download: 1960 missing symbols
- 2026-08-26 21:54:22 watchdog: watchdog started (interval 300s)
- 2026-08-26 21:54:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 21:59:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:04:22 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:08:01 watchdog: watchdog started (interval 300s)
- 2026-08-26 22:08:01 watchdog: process DEAD with 2384 funds unscreened - relaunching
- 2026-08-26 22:08 === stage download ===
- 2026-08-26 22:08 download: 1823 missing symbols
- 2026-08-26 22:09:01 watchdog: relaunch confirmed alive
- 2026-08-26 22:14:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:19:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:24:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:29:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:34:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:39:01 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:41 download: batch 1 -> 200/200
- 2026-08-26 22:44:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:49:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:54:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 22:59:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:04:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:09:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:14:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:14 download: batch 2 -> 200/200
- 2026-08-26 23:19:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:24:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:29:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:34:02 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:39:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:44:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:47 download: batch 3 -> 200/200
- 2026-08-26 23:49:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:54:03 watchdog: overnight alive, still_missing=2384
- 2026-08-26 23:59:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:04:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:09:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:14:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:19:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:20 download: batch 4 -> 200/200
- 2026-08-27 00:24:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:29:03 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:34:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:39:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:44:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:49:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:54 download: batch 5 -> 200/200
- 2026-08-27 00:54:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 00:59:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:04:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:09:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:14:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:19:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:24:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:27 download: batch 6 -> 200/200
- 2026-08-27 01:29:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:34:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:39:04 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:44:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:49:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:54:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 01:59:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:00 download: batch 7 -> 200/200
- 2026-08-27 02:04:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:09:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:14:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:19:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:24:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:29:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:33 download: batch 8 -> 200/200
- 2026-08-27 02:34:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:39:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:44:05 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:49:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:54:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 02:59:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:04:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:06 download: batch 9 -> 200/200
- 2026-08-27 03:09:06 watchdog: overnight alive, still_missing=2384
- 2026-08-27 03:10 download: batch 10 -> 23/23
- 2026-08-27 03:10 download done: 0 still missing (no Yahoo data?)
- 2026-08-27 03:10 === stage screen ===
- 2026-08-27 03:10 screen: 2384 funds, 2384 to do
- 2026-08-27 03:11 screen: 100/2384
- 2026-08-27 03:11 screen: 200/2384
- 2026-08-27 03:12 screen: 300/2384
- 2026-08-27 03:12 screen: 400/2384
- 2026-08-27 03:13 screen: 500/2384
- 2026-08-27 03:13 screen: 600/2384
- 2026-08-27 03:14:06 watchdog: overnight alive, still_missing=1784
- 2026-08-27 03:14 screen: 700/2384
- 2026-08-27 03:14 screen: 800/2384
- 2026-08-27 03:15 screen: 900/2384
- 2026-08-27 03:15 screen: 1000/2384
- 2026-08-27 03:16 screen: 1100/2384
- 2026-08-27 03:16 screen: 1200/2384
- 2026-08-27 03:17 screen: 1300/2384
- 2026-08-27 03:17 screen: 1400/2384
- 2026-08-27 03:18 screen: 1500/2384
- 2026-08-27 03:18 screen: 1600/2384
- 2026-08-27 03:19:06 watchdog: overnight alive, still_missing=784
- 2026-08-27 03:19 screen: 1700/2384
- 2026-08-27 03:19 screen: 1800/2384
- 2026-08-27 03:20 screen: 1900/2384
- 2026-08-27 03:20 screen: 2000/2384
- 2026-08-27 03:21 screen: 2100/2384
- 2026-08-27 03:22 screen: 2200/2384
- 2026-08-27 03:22 screen: 2300/2384
- 2026-08-27 03:23 screen done: 2384 funds
- 2026-08-27 03:23 === stage finalize ===
- 2026-08-27 03:23 finalize: 2384 funds screened
- 2026-08-27 03:23 1336 sleeve mix
- 2026-08-27 03:23 707 weak/unstable alpha
- 2026-08-27 03:23 250 CANDIDATE
- 2026-08-27 03:23 42 alpha, but correlated with current portfolio
- 2026-08-27 03:23 32 alpha in 5y window, but not persistent
- 2026-08-27 03:23 17 no 5y window
- 2026-08-27 03:23 candidates v1 name-filter would have missed: ['abhyx', 'acfix', 'acitx', 'acthx', 'aflix', 'aftex', 'agovx', 'aguax', 'aidax', 'alcax', 'algrx', 'alnyx', 'althx', 'amaax', 'amhix', 'annpx', 'asdvx', 'atfax', 'avaax', 'bchyx', 'bcitx', 'bils', 'bmbix', 'bsnix', 'btmix', 'cfmox', 'cfnlx', 'cfrax', 'coiax', 'coltx', 'cpxax', 'csdax', 'cshix', 'cxhyx', 'deffx', 'delix', 'dflex', 'ditex', 'dmusx', 'dnmzx', 'dpiax', 'drcax', 'drnjx', 'dsibx', 'dultx', 'dvmhx', 'dvtax', 'eablx', 'eacax', 'eanax', 'eascx', 'ebsax', 'eniax', 'etazx', 'etgax', 'ethyx', 'etmdx', 'etmnx', 'etmox', 'etncx', 'etnjx', 'etorx', 'etpax', 'etsix', 'etvax', 'evfax', 'exmax', 'exnax', 'fafrx', 'faztx', 'fcamx', 'fcfix', 'fcotx', 'fcstx', 'fcvsx', 'fdmmx', 'femdx', 'ffrsx', 'fhaix', 'fhcox', 'fhmix', 'ficnx', 'fiscx', 'fkcix', 'fklax', 'fknix', 'fkstx', 'fktfx', 'fktix', 'fkutx', 'fkytx', 'flrn', 'fltdx', 'fltmx', 'fltr', 'fmdtx', 'fmoax', 'fmotx', 'fnmtx', 'fnytx', 'fpntx', 'fralx', 'frcox', 'frmox', 'frnjx', 'frorx', 'frpax', 'frstx', 'fsazx', 'ftazx', 'ftfmx', 'ftgax', 'ftlax', 'ftnyx', 'ftorx', 'fwiax', 'fxncx', 'gbonx', 'giyix', 'gtfbx', 'hicox', 'hmezx', 'hubax', 'icifx', 'ihiax', 'intax', 'kctax', 'kmdnx', 'kntax', 'lansx', 'lmsfx', 'lpxax', 'lsbdx', 'lsyux', 'ltebx', 'ltnyx', 'lubax', 'masax', 'mcoax', 'mdxbx', 'mervx', 'mfalx', 'mfarx', 'mfiax', 'mfpax', 'mfscx', 'mfsmx', 'mfssx', 'mfwvx', 'mgfox', 'missx', 'mmgax', 'mmhyx', 'mmufx', 'msncx', 'msnyx', 'msvax', 'mtbax', 'mtlfx', 'nefzx', 'nhmax', 'nmbax', 'nmdax', 'nmuix', 'nnjax', 'nnyax', 'nsmmx', 'nthex', 'olcax', 'omifx', 'onjcx', 'opcax', 'opnyx', 'orncx', 'pabax', 'pamfx', 'pbaix', 'pbcax', 'pgsix', 'phizx', 'phstx', 'ppnax', 'prfhx', 'prfrx', 'prfsx', 'prinx', 'prmdx', 'prnyx', 'prsmx', 'prtax', 'pruax', 'prvax', 'prvbx', 'prxcx', 'psnyx', 'ptebx', 'puls', 'pysix', 'qcmmrx', 'rctix', 'rlvsx', 'rmuyx', 'rpidx', 'rpifx', 'safex', 'scfzx', 'sdsax', 'sefix', 'seimx', 'shtix', 'sitex', 'smlax', 'sngvx', 'szmax', 'taftx', 'tisix', 'trbux', 'trhyx', 'tsdjx', 'twtix', 'ulst', 'usiax', 'usmsx', 'vcaix', 'vcitx', 'vctfx', 'vidax', 'vklmx', 'vkmmx', 'vmatx', 'vmltx', 'vnjtx', 'vnytx', 'vohix', 'vpaix', 'vtipx', 'vubfx', 'vusb', 'vwahx', 'vweax', 'vwitx', 'vwltx', 'wisex']
- 2026-08-27 03:23 === overnight run finished in 5.2h ===
- 2026-08-27 03:24:06 watchdog: process gone but screen COMPLETE (0 missing) - watchdog exiting
- 2026-08-27 03:24:06 watchdog: watchdog exit
Reference in New Issue View Git Blame Copy Permalink