CEF pass: universe (SEC report, 973 -> 295 listed) + stage 1 screen + stage 2a character
fundlab/cef_universe.py: SEC 'Closed-End Fund Information' report (973 active CEFs) + company_tickers CIK join -> 295 listed common classes (preferreds and 6(c)-converted companies flagged). Supersedes the N-2 full-index approach (N-2/A annual updates + 404ing index paths). Prices via goget. fundlab/cef_screen.py: 290 screened - t5/t12, vol5, maxdd5, the 5 crash episodes, 12m payout proxy. Energy/midstream CEFs (EMO +305%, SRV +234%, NML +232%) top return AND crash resilience; Voya Dividend-Premium series (IGD maxDD -16%); EM CEFs volatile + 14-24% dist; long-dur munis -26..-42%. fundlab/cef_character.py: 50-fund shortlist, 35-sleeve character + crude tax_arb = character x (upside + 0.4 x vol). Tests: 6 new cef checks (105 total). RESEARCH.md: CEF form facts (N-2ASR, N-PX, N-23C-3A, BDC caveats) + remaining stage 2b work.
This commit is contained in:
parent
f8409fff7a
commit
d1e85026bf
2
adx-split.csv
Normal file
2
adx-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2000-10-20,3:2
|
||||||
|
2
aef-split.csv
Normal file
2
aef-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1995-07-17,2:1
|
||||||
|
2
asa-split.csv
Normal file
2
asa-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2010-05-04,3:1
|
||||||
|
2
brw-split.csv
Normal file
2
brw-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2022-05-20,1:2
|
||||||
|
3
bto-split.csv
Normal file
3
bto-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1997-12-02,4:1
|
||||||
|
2008-12-30,1:4
|
||||||
|
3
clm-split.csv
Normal file
3
clm-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2008-12-23,1:4
|
||||||
|
2014-12-29,1:4
|
||||||
|
3
crf-split.csv
Normal file
3
crf-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2008-12-23,1:2
|
||||||
|
2014-12-29,1:4
|
||||||
|
2
evg-split.csv
Normal file
2
evg-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2019-12-27,999649:1000000
|
||||||
|
|
|
@ -732,3 +732,61 @@ swing, so volatility is the right selection variable. Refinements:
|
||||||
identical (different holdings/NAV) - the standard harvest-and-swap
|
identical (different holdings/NAV) - the standard harvest-and-swap
|
||||||
is safe; the 250-candidate universe makes a like-for-like
|
is safe; the 250-candidate universe makes a like-for-like
|
||||||
substitute usually available.
|
substitute usually available.
|
||||||
|
|
||||||
|
### CEF pass (2026-08-27)
|
||||||
|
**Universe** (fundlab/cef_universe.py): the SEC's official
|
||||||
|
"Closed-End Fund Information" report (ALL active 1940-Act CEFs; 2026
|
||||||
|
file = 973; sec.gov/files/investment/data/other/closed-end-fund-
|
||||||
|
information/closed-end-investment-company-<YYYY>.csv - hyphenated
|
||||||
|
filenames 2024+, underscored before). CIK join with
|
||||||
|
company_tickers.json -> 295 listed common-class tickers (hyphenated
|
||||||
|
preferred series dropped; 668 small/dark/OTC funds have no listed
|
||||||
|
ticker; several 6(c)-converted "CEFs" are now operating companies -
|
||||||
|
Powerlaw, RoboStrategy, C1 Fund, Foxby - excluded). The N-2
|
||||||
|
full-index approach was abandoned: annual updates file as N-2/A, and
|
||||||
|
full-index paths 404 for newer filings; the SEC report is the whole
|
||||||
|
active population and is one download.
|
||||||
|
|
||||||
|
CEF form facts learned:
|
||||||
|
- CEFs file NPORT-P (same format as open-end - xcheck.py reuses)
|
||||||
|
- shareholder reports: N-CSR/N-CSRS AND/OR N-2ASR (rocdetect must try both)
|
||||||
|
- N-PX proxy = CEF signal (open-end funds have no boards)
|
||||||
|
- N-23C-3A/-2 = Rule 23c-3 repurchase-offer (tender) notifications - common
|
||||||
|
- BDCs also file N-2; their names don't contain "Business Development"
|
||||||
|
(Ares Capital Corp etc) - detect via form history (10-K vs N-2ASR)
|
||||||
|
- Yahoo instrumentType for CEFs is usually EQUITY - do NOT apply the
|
||||||
|
open-end MUTUALFUND filter
|
||||||
|
- CEF price = MARKET price (premium/discount on top of NAV); Adj Close
|
||||||
|
includes reinvested distributions, so the payout proxy works
|
||||||
|
|
||||||
|
**Stage 1** (fundlab/cef_screen.py): 290/295 screened (goget prices;
|
||||||
|
phd/unid/uniu 404). Stats: t5/t12, vol5, maxdd5, the 5 crash
|
||||||
|
episodes via drawdown.fund_windows, 12m payout proxy, n_pos_scen.
|
||||||
|
Findings:
|
||||||
|
- Energy/midstream infra CEFs dominate 5y return AND crash
|
||||||
|
resilience: EMO +305%, SRV +234%, NML +232%, KYN +206%, PEO +192%,
|
||||||
|
TYG +170% - all positive in 2022 (+10..25%) and 2026 Q1 (+10..17%),
|
||||||
|
dist 10-13%/yr.
|
||||||
|
- Voya "Dividend & Premium" series (IGD/IHD/IAE/EOD): +77..86%,
|
||||||
|
maxDD only -16..-30%, 11-13% dist - premium management keeps the
|
||||||
|
market price near NAV.
|
||||||
|
- EM/China CEFs (TWN +348%, KF +139%, EMF +98%, AEF, MXF): high vol,
|
||||||
|
-34..-47% in 2022 (harvestable), 14-24% dist.
|
||||||
|
- Long-duration muni CEFs (PCQ/PML/PNI/TDF) -26..-42% over 5y: the
|
||||||
|
2022-23 rate spike, duration not alpha.
|
||||||
|
- LTCFX (+1101%, 2023 +799%) and DXYZ (maxDD -90%) are
|
||||||
|
derivatives/meme vehicles - flag, don't rank.
|
||||||
|
|
||||||
|
**Stage 2a** (fundlab/cef_character.py): 50-fund shortlist (top 40
|
||||||
|
return + >=3 positive scenarios + top 15 payout) -> 35-sleeve
|
||||||
|
factor_screen -> taxplan.sleeve_score character -> crude
|
||||||
|
tax_arb = character x (upside + 0.4 x vol). Top: KF (char 0.63),
|
||||||
|
LENDX (0.69, maxDD -12%, all scenarios ~flat/positive), EMF/AEF/HQL/
|
||||||
|
HQH/IGD equity-character 0.55-0.75; EMO/NML/SRV/KYN/TYG char
|
||||||
|
0.29-0.39 (income-heavy -> IRA-tilted, ROC share pending the 1099).
|
||||||
|
|
||||||
|
**Remaining (stage 2b+):** per-fund N-2ASR/N-CSR distribution
|
||||||
|
character (the income/gains/ROC split is decisive for the EMO
|
||||||
|
family), NPORT NAV-per-share -> quarterly discount series, leverage
|
||||||
|
from NPORT financials, BDC flag via form history, tender-offer
|
||||||
|
status; app CEF tab.
|
||||||
|
|
|
||||||
1270
fundlab/cef_character.json
Normal file
1270
fundlab/cef_character.json
Normal file
File diff suppressed because it is too large
Load Diff
116
fundlab/cef_character.py
Normal file
116
fundlab/cef_character.py
Normal file
|
|
@ -0,0 +1,116 @@
|
||||||
|
"""CEF stage 2a: tax-character proxy for the shortlist.
|
||||||
|
|
||||||
|
Takes the cef_screen.json shortlist (top 5y return + crash-resilient +
|
||||||
|
high-payout CEFs), runs the 35-sleeve factor screen on each, and scores
|
||||||
|
the sleeve mix with taxplan.sleeve_score (same proxy the open-end
|
||||||
|
candidates use). Then ranks by a crude "tax-arb per dollar" heuristic
|
||||||
|
for a constrained taxable account:
|
||||||
|
|
||||||
|
tax_arb ~= character x (upside + 0.4 x vol)
|
||||||
|
|
||||||
|
character share of the return mix that is tax-favorable (0..1)
|
||||||
|
upside max(t5,0)/5 - the LTCG-deferral term
|
||||||
|
0.4 x vol5 ~ expected |annual loss| (normal approx) - the
|
||||||
|
harvest term (0.4 = 1/sqrt(2*pi))
|
||||||
|
|
||||||
|
Both terms are per dollar of balance; the heuristic is a SCREENER,
|
||||||
|
the EDGAR deep dive (N-2ASR/N-CSR distribution character + NPORT NAV
|
||||||
|
for the discount) is what decides.
|
||||||
|
|
||||||
|
Output: fundlab/cef_character.json
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from fundlab import factors, taxplan
|
||||||
|
|
||||||
|
HERE = Path(__file__).parent
|
||||||
|
SCREEN = HERE / "cef_screen.json"
|
||||||
|
OUT = HERE / "cef_character.json"
|
||||||
|
|
||||||
|
# 6(c)-converted / non-fund tickers that show up in the SEC CEF report
|
||||||
|
# (organized as CEFs, later converted to operating companies)
|
||||||
|
NON_FUNDS = {"cfnd", "pwrl", "bot", "fxby"}
|
||||||
|
SPECULATIVE = re.compile(r"corp\.?$|tech100", re.I)
|
||||||
|
|
||||||
|
|
||||||
|
def select(res: dict, n_ret: int = 40, n_pay: int = 15) -> list[str]:
|
||||||
|
rows = [v for v in res.values()
|
||||||
|
if v.get("t5") is not None and v["sym"] not in NON_FUNDS]
|
||||||
|
by_ret = sorted(rows, key=lambda x: -x["t5"])[:n_ret]
|
||||||
|
by_res = [v for v in rows if v.get("n_pos_scen", 0) >= 3]
|
||||||
|
by_pay = sorted(rows, key=lambda x: -(x.get("payout_12m") or 0))[:n_pay]
|
||||||
|
out: dict[str, dict] = {}
|
||||||
|
for v in by_ret + by_res + by_pay:
|
||||||
|
out.setdefault(v["sym"], v)
|
||||||
|
return list(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _betas(sym: str) -> dict:
|
||||||
|
fs = factors.factor_screen(sym)
|
||||||
|
if not fs:
|
||||||
|
return {}
|
||||||
|
return (fs.get("full") or fs.get("rec5") or {}).get("betas") or {}
|
||||||
|
|
||||||
|
|
||||||
|
def run() -> dict:
|
||||||
|
res = json.loads(SCREEN.read_text())
|
||||||
|
syms = select(res)
|
||||||
|
print(f"{len(syms)} shortlisted CEFs", flush=True)
|
||||||
|
out: dict = {}
|
||||||
|
for i, s in enumerate(syms, 1):
|
||||||
|
v = res[s.upper()]
|
||||||
|
comps = _betas(s.lower())
|
||||||
|
char = taxplan.sleeve_score(comps) if comps else None
|
||||||
|
t5 = v.get("t5") or 0.0
|
||||||
|
vol = v.get("vol5") or 0.0
|
||||||
|
upside = max(t5, 0.0) / 5.0
|
||||||
|
harvest = 0.4 * vol
|
||||||
|
arb = (char if char is not None else 0.3) * (upside + harvest)
|
||||||
|
out[s] = {
|
||||||
|
**{k: v.get(k) for k in ("name", "bdc", "days", "last")},
|
||||||
|
"sym": s, "t5": t5, "t12": v.get("t12"), "vol5": vol,
|
||||||
|
"maxdd5": v.get("maxdd5"),
|
||||||
|
"2022": v.get("2022 bear mkt"), "2023": v.get("2023 rate shock"),
|
||||||
|
"2025t": v.get("2025 tariff crash"),
|
||||||
|
"2026": v.get("2026 Q1 drawdown"),
|
||||||
|
"n_pos_scen": v.get("n_pos_scen"),
|
||||||
|
"payout_12m": v.get("payout_12m"),
|
||||||
|
"character": round(char, 2) if char is not None else None,
|
||||||
|
"sleeves": {k: round(b, 2) for k, b in comps.items()
|
||||||
|
if abs(b) > 0.15},
|
||||||
|
"tax_arb": round(arb, 4),
|
||||||
|
}
|
||||||
|
print(f"{i:2}/{len(syms)} {s:7} char={out[s]['character']} "
|
||||||
|
f"arb={out[s]['tax_arb']}", flush=True)
|
||||||
|
OUT.write_text(json.dumps(out, indent=1, default=str))
|
||||||
|
print(f"wrote {OUT}")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _pc(v: float | None) -> str:
|
||||||
|
return "" if v is None else f"{v:+.1%}"
|
||||||
|
|
||||||
|
|
||||||
|
def _print(out: dict, top: int = 35) -> None:
|
||||||
|
rows = sorted(out.values(), key=lambda x: -x["tax_arb"])
|
||||||
|
print(f"{'fund':7} {'name':40} {'char':>5} {'5yTR':>8} {'vol':>6}"
|
||||||
|
f" {'maxDD':>8} {'2022':>8} {'2026':>8} {'dist12':>8} {'tax_arb':>8}")
|
||||||
|
for v in rows[:top]:
|
||||||
|
c = v["character"]
|
||||||
|
print(f"{v['sym']:7} {v.get('name','')[:40]:40} "
|
||||||
|
f"{('' if c is None else f'{c:.2f}'):>5} "
|
||||||
|
f"{v['t5']:+8.1%} {v['vol5']:6.2f} "
|
||||||
|
f"{_pc(v.get('maxdd5')):>8} {_pc(v.get('2022')):>8} "
|
||||||
|
f"{_pc(v.get('2026')):>8} {_pc(v.get('payout_12m')):>8} "
|
||||||
|
f"{v['tax_arb']:8.3f}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
o = run()
|
||||||
|
_print(o)
|
||||||
6977
fundlab/cef_screen.json
Normal file
6977
fundlab/cef_screen.json
Normal file
File diff suppressed because it is too large
Load Diff
137
fundlab/cef_screen.py
Normal file
137
fundlab/cef_screen.py
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
"""CEF screen: stage 1 of the CEF pass.
|
||||||
|
|
||||||
|
Reads cef_universe.json (SEC active-CEF report + ticker map) and local
|
||||||
|
price files (goget-downloaded). Per ticker:
|
||||||
|
t5 / t12 total return (Adj Close)
|
||||||
|
vol5 annualized daily-return vol, 5y
|
||||||
|
maxdd5 deepest peak->trough drawdown, 5y
|
||||||
|
scen_* return over each of the 5 market-crash episodes
|
||||||
|
(drawdown.py episodes, fund_windows)
|
||||||
|
payout_12m distribution proxy = 12m adj return - 12m price
|
||||||
|
return (Close = market price, Adj Close = total
|
||||||
|
return incl. reinvested distributions)
|
||||||
|
p5y_share 5y version of the same
|
||||||
|
|
||||||
|
CEF-specific context for interpreting the output:
|
||||||
|
- price = MARKET price; premium/discount dynamics are on top of NAV
|
||||||
|
(stage 2 adds the NPORT NAV-per-share series)
|
||||||
|
- distributions are NOT the fund's discretion to skip - a CEF usually
|
||||||
|
keeps paying even in drawdowns (that's what the leverage is for),
|
||||||
|
so the payout proxy is more stable than for open-end funds
|
||||||
|
- BDCs in the set pay ordinary income + occasional ROC (stage 2
|
||||||
|
flags them from the EDGAR form history)
|
||||||
|
|
||||||
|
Output: fundlab/cef_screen.json
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from fundlab import drawdown
|
||||||
|
|
||||||
|
HERE = Path(__file__).parent
|
||||||
|
UNIVERSE = HERE / "cef_universe.json"
|
||||||
|
EPS = json.loads((HERE / "drawdown_results.json").read_text())["episodes"]
|
||||||
|
OUT = HERE / "cef_screen.json"
|
||||||
|
SCEN_KEYS = ["2022 bear mkt", "2023 rate shock", "2024 vol spike",
|
||||||
|
"2025 tariff crash", "2026 Q1 drawdown"]
|
||||||
|
|
||||||
|
|
||||||
|
def _load(sym: str) -> pd.DataFrame | None:
|
||||||
|
f = Path.home() / "prog/fin/stocks" / f"{sym.lower()}-history.csv"
|
||||||
|
if not f.exists():
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
d = pd.read_csv(f, parse_dates=["Date"])
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
d = d.set_index("Date").sort_index()
|
||||||
|
d = d[~d.index.duplicated(keep="last")]
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
def _perf(d: pd.DataFrame) -> dict:
|
||||||
|
adj = d["Adj Close"].dropna()
|
||||||
|
px = d["Close"].dropna()
|
||||||
|
out: dict = {}
|
||||||
|
for tag, n in (("5", 5 * 252), ("12", 252)):
|
||||||
|
a = adj.tail(n)
|
||||||
|
p = px.tail(n)
|
||||||
|
out[f"t{tag}"] = float(a.iloc[-1] / a.iloc[0] - 1) if len(a) > 20 else None
|
||||||
|
out[f"p{tag}"] = float(p.iloc[-1] / p.iloc[0] - 1) if len(p) > 20 else None
|
||||||
|
a5 = adj.tail(5 * 252)
|
||||||
|
if len(a5) > 20:
|
||||||
|
r = a5.pct_change().dropna()
|
||||||
|
out["vol5"] = float(r.std() * np.sqrt(252))
|
||||||
|
peak = np.maximum.accumulate(a5.to_numpy())
|
||||||
|
out["maxdd5"] = float(((a5.to_numpy() / peak) - 1).min())
|
||||||
|
else:
|
||||||
|
out["vol5"] = out["maxdd5"] = None
|
||||||
|
if out.get("t12") is not None and out.get("p12") is not None:
|
||||||
|
out["payout_12m"] = out["t12"] - out["p12"]
|
||||||
|
if out.get("t5") is not None and out.get("p5") is not None:
|
||||||
|
out["payout_5y_share"] = (out["t5"] - out["p5"]) / out["t5"] \
|
||||||
|
if abs(out["t5"]) > 1e-4 else None
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def screen_one(sym: str) -> dict:
|
||||||
|
s = sym.lower() # price files are lowercase
|
||||||
|
d = _load(s)
|
||||||
|
if d is None or len(d) < 30:
|
||||||
|
return {"error": "no data"}
|
||||||
|
out = _perf(d)
|
||||||
|
out.update(drawdown.fund_windows(s, EPS))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def run() -> dict:
|
||||||
|
uni = json.loads(UNIVERSE.read_text())
|
||||||
|
res: dict = {}
|
||||||
|
for t in sorted(uni):
|
||||||
|
info = dict(uni[t])
|
||||||
|
info["sym"] = t.lower()
|
||||||
|
s = screen_one(t)
|
||||||
|
info.update(s)
|
||||||
|
info["n_pos_scen"] = sum(
|
||||||
|
1 for k in SCEN_KEYS if info.get(k) is not None and info[k] > 0)
|
||||||
|
res[t] = info
|
||||||
|
n_ok = sum(1 for v in res.values() if "t5" in v)
|
||||||
|
print(f"{n_ok}/{len(res)} CEFs screened", flush=True)
|
||||||
|
OUT.write_text(json.dumps(res, indent=1, default=str))
|
||||||
|
print(f"wrote {OUT}")
|
||||||
|
return res
|
||||||
|
|
||||||
|
|
||||||
|
def _fmt(x, pct: bool = True, dec: int = 2) -> str:
|
||||||
|
if x is None:
|
||||||
|
return ""
|
||||||
|
if pct:
|
||||||
|
return f"{x:+.{dec}%}"
|
||||||
|
return f"{x:.{dec}f}"
|
||||||
|
|
||||||
|
|
||||||
|
def _print(res: dict, top: int = 30, sort: str = "t5") -> None:
|
||||||
|
rows = [v for v in res.values() if v.get("t5") is not None]
|
||||||
|
print(f"{'fund':6} {'name':42} {'5yTR':>8} {'vol':>6} {'maxDD':>8}"
|
||||||
|
f" {'2022':>8} {'2023':>8} {'2025t':>8} {'2026':>8}"
|
||||||
|
f" {'dist12':>8} {'n+':>3}")
|
||||||
|
for v in sorted(rows, key=lambda x: -x.get(sort, -9))[:top]:
|
||||||
|
print(f"{v['sym']:6} {v.get('name', '')[:42]:42}"
|
||||||
|
f" {_fmt(v.get('t5')):>8} {_fmt(v.get('vol5'), False):>6}"
|
||||||
|
f" {_fmt(v.get('maxdd5')):>8}"
|
||||||
|
f" {_fmt(v.get('2022 bear mkt')):>8}"
|
||||||
|
f" {_fmt(v.get('2023 rate shock')):>8}"
|
||||||
|
f" {_fmt(v.get('2025 tariff crash')):>8}"
|
||||||
|
f" {_fmt(v.get('2026 Q1 drawdown')):>8}"
|
||||||
|
f" {_fmt(v.get('payout_12m')):>8}"
|
||||||
|
f" {v.get('n_pos_scen', 0):>3}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
r = run()
|
||||||
|
_print(r)
|
||||||
2657
fundlab/cef_universe.json
Normal file
2657
fundlab/cef_universe.json
Normal file
File diff suppressed because it is too large
Load Diff
159
fundlab/cef_universe.py
Normal file
159
fundlab/cef_universe.py
Normal file
|
|
@ -0,0 +1,159 @@
|
||||||
|
"""CEF universe: the SEC's official "Closed-End Fund Information" report.
|
||||||
|
|
||||||
|
Primary source: https://www.sec.gov/files/investment/data/other/
|
||||||
|
closed-end-fund-information/closed-end-investment-company-<YYYY>.csv
|
||||||
|
(all ACTIVE 1940-Act closed-end companies: CIK, name, addresses,
|
||||||
|
filing status - 973 for 2026). This supersedes the N-2 full-index
|
||||||
|
approach (N-2/N-2-A filers are the CEFs that updated their
|
||||||
|
registration within the window - a biased subset; the report is the
|
||||||
|
whole active population).
|
||||||
|
|
||||||
|
Tickers: CIK join with the SEC company_tickers.json exchange map
|
||||||
|
(305 of 973 match - the rest are small/dark/OTC funds). Hyphenated
|
||||||
|
tickers (GAM-PB etc.) are PREFERRED series - dropped (their price
|
||||||
|
history is the preferred's near-flat NAV, useless for screening);
|
||||||
|
the common class is usually the fund's other listed security.
|
||||||
|
|
||||||
|
CEF notes for the rest of the pipeline:
|
||||||
|
- CEFs file NPORT-P (same format as open-end - xcheck.py works)
|
||||||
|
- shareholder reports are N-CSR/N-CSRS AND/OR N-2ASR (rocdetect must
|
||||||
|
try both)
|
||||||
|
- CEFs file N-PX proxies (open-end funds don't - a CEF signal)
|
||||||
|
- price = MARKET price (premium/discount to NAV is a free variable)
|
||||||
|
- Yahoo instrumentType is usually EQUITY - do NOT apply the open-end
|
||||||
|
MUTUALFUND filter
|
||||||
|
- BDCs are in the report too (name-flagged)
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import csv
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import urllib.request
|
||||||
|
from datetime import date, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
HERE = Path(__file__).parent
|
||||||
|
CACHE = HERE / "universe_cache"
|
||||||
|
CEF_CSV = CACHE / "cef_sec_2026.csv"
|
||||||
|
UNIVERSE = HERE / "cef_universe.json"
|
||||||
|
DATA = Path.home() / "prog/fin/stocks"
|
||||||
|
UA = {"User-Agent": "research test@example.com"}
|
||||||
|
MIN_DAYS = 1250 # >=5y of trading history (same bar as the 497 pass)
|
||||||
|
STALE_DAYS = 35 # last quote older than this = delisted
|
||||||
|
|
||||||
|
|
||||||
|
def _get(url: str) -> bytes:
|
||||||
|
req = urllib.request.Request(url, headers=UA)
|
||||||
|
return urllib.request.urlopen(req, timeout=120).read()
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_cef_report(year: int = 2026) -> list[dict]:
|
||||||
|
"""The SEC active-CEF report (CIK, name, ...). Cached locally."""
|
||||||
|
if CEF_CSV.exists():
|
||||||
|
raw = CEF_CSV.read_bytes()
|
||||||
|
else:
|
||||||
|
CACHE.mkdir(exist_ok=True)
|
||||||
|
# 2024+ uses hyphenated filenames, earlier underscored
|
||||||
|
for pat in (f"closed-end-investment-company-{year}.csv",
|
||||||
|
f"closed-end_investment_company_{year}.csv",
|
||||||
|
"closed-end_investment_company.csv"):
|
||||||
|
url = ("https://www.sec.gov/files/investment/data/other/"
|
||||||
|
f"closed-end-fund-information/{pat}")
|
||||||
|
try:
|
||||||
|
raw = _get(url)
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
raise RuntimeError("could not fetch the SEC CEF report")
|
||||||
|
CEF_CSV.write_bytes(raw)
|
||||||
|
rows = list(csv.DictReader(io.StringIO(
|
||||||
|
raw.decode("utf-8-sig"))))
|
||||||
|
return [{"cik": r["CIK"].lstrip("0") or "0",
|
||||||
|
"name": r["Registrant_Name"].strip()} for r in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def _company_tickers() -> dict:
|
||||||
|
"""cik(int) -> (ticker, title) from the SEC exchange-listed map."""
|
||||||
|
d = json.loads(_get("https://www.sec.gov/files/company_tickers.json"))
|
||||||
|
out: dict[int, tuple] = {}
|
||||||
|
for v in d.values():
|
||||||
|
cik = v.get("cik_str", v.get("cik"))
|
||||||
|
t = v.get("ticker", v.get("symbol"))
|
||||||
|
if cik is None or not t:
|
||||||
|
continue
|
||||||
|
out[int(cik)] = (t.upper(), v.get("title", ""))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _history(sym: str) -> tuple[int, str]:
|
||||||
|
"""(trading days, last date) from the local price file."""
|
||||||
|
p = DATA / f"{sym}-history.csv"
|
||||||
|
if not p.exists():
|
||||||
|
return 0, ""
|
||||||
|
last = ""
|
||||||
|
n = 0
|
||||||
|
with p.open() as f:
|
||||||
|
next(f, None) # header
|
||||||
|
for line in f:
|
||||||
|
n += 1
|
||||||
|
last = line.split(",")[0]
|
||||||
|
return n, last
|
||||||
|
|
||||||
|
|
||||||
|
def run() -> dict:
|
||||||
|
cefs = fetch_cef_report()
|
||||||
|
print(f"{len(cefs)} active CEFs in the SEC report", flush=True)
|
||||||
|
tick = _company_tickers()
|
||||||
|
|
||||||
|
# NOTE: do NOT exclude tickers present in the local DB - goget
|
||||||
|
# writes <sym>.json there, so after the first download every CEF
|
||||||
|
# would look "known". The SEC report already defines the set.
|
||||||
|
known = set()
|
||||||
|
for j in (Path(__file__).parent.parent / "funds.json",
|
||||||
|
HERE / "search_all.json", HERE / "search_external.json",
|
||||||
|
HERE / "xcheck_report.json"):
|
||||||
|
if j.exists():
|
||||||
|
d = json.loads(j.read_text())
|
||||||
|
if isinstance(d, dict):
|
||||||
|
known |= {k.upper() for k in d}
|
||||||
|
|
||||||
|
stale = (date.today() - timedelta(days=STALE_DAYS)).toordinal()
|
||||||
|
out: dict[str, dict] = {}
|
||||||
|
n_pref = n_unmatched = 0
|
||||||
|
for c in cefs:
|
||||||
|
hit = tick.get(int(c["cik"]))
|
||||||
|
if not hit:
|
||||||
|
n_unmatched += 1
|
||||||
|
continue
|
||||||
|
t, _yt = hit
|
||||||
|
if "-" in t: # preferred/series class
|
||||||
|
n_pref += 1
|
||||||
|
continue
|
||||||
|
if t in known:
|
||||||
|
continue
|
||||||
|
days, last = _history(t.lower())
|
||||||
|
live = False
|
||||||
|
try:
|
||||||
|
live = date.fromisoformat(last).toordinal() >= stale
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
out[t] = {"cik": c["cik"], "name": c["name"],
|
||||||
|
"bdc": bool(re.search(r"business development",
|
||||||
|
c["name"], re.I)),
|
||||||
|
"listed": days >= MIN_DAYS and live,
|
||||||
|
"days": days, "last": last, "known": False}
|
||||||
|
n_listed = sum(1 for v in out.values() if v["listed"])
|
||||||
|
n_bdc = sum(1 for v in out.values() if v["bdc"])
|
||||||
|
print(f"{len(out)} CEF tickers ({n_listed} live with >=5y history, "
|
||||||
|
f"{n_bdc} BDCs, {n_pref} preferred-series dropped, "
|
||||||
|
f"{n_unmatched} CIKs with no listed ticker)", flush=True)
|
||||||
|
UNIVERSE.write_text(json.dumps(out, indent=1))
|
||||||
|
print(f"wrote {UNIVERSE}")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
run()
|
||||||
137
fundlab/goget_cef.log
Normal file
137
fundlab/goget_cef.log
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
goget
|
||||||
|
|
||||||
|
Symbol = txemf
|
||||||
|
txemf: Bad format (timestamp)
|
||||||
|
Symbol = phd
|
||||||
|
Error fetching phd. Retries: 5: aborting
|
||||||
|
Get error (phd): 404 Not Found
|
||||||
|
Symbol = glq
|
||||||
|
Symbol = fmy
|
||||||
|
Symbol = pgp
|
||||||
|
Symbol = etv
|
||||||
|
Symbol = etw
|
||||||
|
Symbol = iga
|
||||||
|
Symbol = spxx
|
||||||
|
Symbol = ety
|
||||||
|
Symbol = bta
|
||||||
|
Symbol = glo
|
||||||
|
Symbol = agd
|
||||||
|
Symbol = caf
|
||||||
|
Symbol = fof
|
||||||
|
Symbol = exg
|
||||||
|
Symbol = aod
|
||||||
|
Symbol = nie
|
||||||
|
Symbol = jce
|
||||||
|
Symbol = iae
|
||||||
|
Symbol = eod
|
||||||
|
Symbol = edd
|
||||||
|
Symbol = awp
|
||||||
|
Symbol = etj
|
||||||
|
Symbol = chw
|
||||||
|
Symbol = srv
|
||||||
|
Symbol = ide
|
||||||
|
Symbol = nuw
|
||||||
|
Symbol = eot
|
||||||
|
Symbol = igi
|
||||||
|
Symbol = rcg
|
||||||
|
Symbol = jls
|
||||||
|
Symbol = gdo
|
||||||
|
Symbol = nbb
|
||||||
|
Symbol = bsl
|
||||||
|
Symbol = nhs
|
||||||
|
Symbol = bbn
|
||||||
|
Symbol = oxlcg
|
||||||
|
Symbol = gbab
|
||||||
|
Symbol = ihd
|
||||||
|
Symbol = ltcfx
|
||||||
|
Symbol = heq
|
||||||
|
Symbol = hyi
|
||||||
|
Symbol = psf
|
||||||
|
Symbol = edf
|
||||||
|
Symbol = bgx
|
||||||
|
Symbol = bwg
|
||||||
|
Symbol = nxg
|
||||||
|
Symbol = dpg
|
||||||
|
Symbol = ardc
|
||||||
|
Symbol = kio
|
||||||
|
Symbol = emo
|
||||||
|
Symbol = mmd
|
||||||
|
Symbol = ccid
|
||||||
|
Symbol = bgh
|
||||||
|
Symbol = dma
|
||||||
|
Symbol = vgi
|
||||||
|
Symbol = isd
|
||||||
|
Symbol = jri
|
||||||
|
Symbol = pdi
|
||||||
|
Symbol = dxr
|
||||||
|
Symbol = bgb
|
||||||
|
Symbol = ldp
|
||||||
|
Symbol = bpre
|
||||||
|
Symbol = ghy
|
||||||
|
Symbol = pgz
|
||||||
|
Symbol = nml
|
||||||
|
Symbol = etx
|
||||||
|
Symbol = dmb
|
||||||
|
Symbol = fpf
|
||||||
|
Symbol = banx
|
||||||
|
Symbol = thq
|
||||||
|
Symbol = nms
|
||||||
|
Symbol = qqqx
|
||||||
|
Symbol = eccv
|
||||||
|
Symbol = jgh
|
||||||
|
Symbol = bst
|
||||||
|
Symbol = thw
|
||||||
|
Symbol = acv
|
||||||
|
Symbol = rsf
|
||||||
|
Symbol = lendx
|
||||||
|
Symbol = nrsax
|
||||||
|
Symbol = ra
|
||||||
|
Symbol = xflt
|
||||||
|
Symbol = ascix
|
||||||
|
Symbol = cpz
|
||||||
|
Symbol = occim
|
||||||
|
Symbol = pmfax
|
||||||
|
Symbol = nichx
|
||||||
|
Symbol = fins
|
||||||
|
Symbol = rmi
|
||||||
|
Symbol = owscx
|
||||||
|
Symbol = larax
|
||||||
|
Symbol = eica
|
||||||
|
Symbol = pdx
|
||||||
|
Symbol = pdskx
|
||||||
|
Symbol = rmm
|
||||||
|
Symbol = aio
|
||||||
|
Symbol = rfm
|
||||||
|
Symbol = ndmo
|
||||||
|
Symbol = asgi
|
||||||
|
Symbol = pdo
|
||||||
|
Symbol = fthy
|
||||||
|
Symbol = vcrdx
|
||||||
|
Symbol = sdhy
|
||||||
|
Symbol = rfmz
|
||||||
|
Symbol = wdi
|
||||||
|
Symbol = npct
|
||||||
|
Symbol = nbxg
|
||||||
|
Symbol = megi
|
||||||
|
Symbol = nmai
|
||||||
|
Symbol = ecat
|
||||||
|
Symbol = gug
|
||||||
|
Symbol = npfd
|
||||||
|
Symbol = rmmz
|
||||||
|
Symbol = rlty
|
||||||
|
Symbol = paxs
|
||||||
|
Symbol = eiia
|
||||||
|
Symbol = dxyz
|
||||||
|
Symbol = spma
|
||||||
|
Symbol = pdpa
|
||||||
|
Symbol = cfnd
|
||||||
|
Symbol = unid
|
||||||
|
Error fetching unid. Retries: 5: aborting
|
||||||
|
Get error (unid): 404 Not Found
|
||||||
|
Symbol = uniu
|
||||||
|
Error fetching uniu. Retries: 5: aborting
|
||||||
|
Get error (uniu): 404 Not Found
|
||||||
|
Symbol = bot
|
||||||
|
Symbol = rvi
|
||||||
|
Symbol = pwrl
|
||||||
|
Failed downloads: [phd unid uniu]
|
||||||
2
fxby-split.csv
Normal file
2
fxby-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2021-05-07,1:5
|
||||||
|
2
grf-split.csv
Normal file
2
grf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2005-05-11,115:100
|
||||||
|
3
herz-split.csv
Normal file
3
herz-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2026-02-09,1:10
|
||||||
|
2021-11-18,1034:1000
|
||||||
|
2
iaf-split.csv
Normal file
2
iaf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2025-10-23,1:3
|
||||||
|
3
kf-split.csv
Normal file
3
kf-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1988-10-12,3:1
|
||||||
|
2008-12-22,1:10
|
||||||
|
3
mci-split.csv
Normal file
3
mci-split.csv
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1998-01-21,2:1
|
||||||
|
2011-02-22,2:1
|
||||||
|
2
mxf-split.csv
Normal file
2
mxf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1993-09-03,10547:10000
|
||||||
|
2
nro-split.csv
Normal file
2
nro-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2025-01-23,1048:1000
|
||||||
|
2
peo-split.csv
Normal file
2
peo-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2000-10-20,3:2
|
||||||
|
2
rvt-split.csv
Normal file
2
rvt-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2013-10-18,1084:1000
|
||||||
|
2
saba-split.csv
Normal file
2
saba-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2024-06-24,1:2
|
||||||
|
2
swz-split.csv
Normal file
2
swz-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
1998-10-16,2:1
|
||||||
|
|
|
@ -576,6 +576,40 @@ def test_drawdown() -> None:
|
||||||
check("fund_windows callable", callable(w), "")
|
check("fund_windows callable", callable(w), "")
|
||||||
|
|
||||||
|
|
||||||
|
def test_cef() -> None:
|
||||||
|
import numpy as np
|
||||||
|
import pandas as pd
|
||||||
|
from fundlab import cef_screen, cef_character
|
||||||
|
print("\ncef")
|
||||||
|
# _perf: flat market price, total-return line +30% over 3y -> the
|
||||||
|
# whole return is payout, maxDD ~ 0 (monotone up)
|
||||||
|
n = 3 * 252
|
||||||
|
idx = pd.bdate_range("2023-01-02", periods=n)
|
||||||
|
adj = 100.0 * (1 + np.arange(n) / 252 * 0.10)
|
||||||
|
p = cef_screen._perf(pd.DataFrame({"Close": 100.0,
|
||||||
|
"Adj Close": adj}, index=idx))
|
||||||
|
check("_perf t5 approx +30% (3y window)",
|
||||||
|
p["t5"] is not None and 0.29 < p["t5"] < 0.31, f"{p['t5']}")
|
||||||
|
check("_perf payout_12m == t12 - p12",
|
||||||
|
abs(p["payout_12m"] - (p["t12"] - p["p12"])) < 1e-12,
|
||||||
|
f"{p['payout_12m']}")
|
||||||
|
check("_perf vol5 positive", p["vol5"] is not None and p["vol5"] > 0)
|
||||||
|
check("_perf maxdd5 <= 0 (monotone up -> ~0)",
|
||||||
|
p["maxdd5"] is not None and p["maxdd5"] <= 1e-9,
|
||||||
|
f"{p['maxdd5']}")
|
||||||
|
# select(): top return + resilient + payout; NON_FUNDS excluded
|
||||||
|
scr = {
|
||||||
|
"AAA": {"sym": "aaa", "t5": 0.5, "n_pos_scen": 0, "payout_12m": 0.0},
|
||||||
|
"BBB": {"sym": "bbb", "t5": 0.4, "n_pos_scen": 4, "payout_12m": 0.0},
|
||||||
|
"CCC": {"sym": "ccc", "t5": -0.5, "n_pos_scen": 0, "payout_12m": 0.2},
|
||||||
|
"CFND": {"sym": "cfnd", "t5": 0.9, "n_pos_scen": 0, "payout_12m": 0.0},
|
||||||
|
}
|
||||||
|
sel = cef_character.select(scr)
|
||||||
|
check("select excludes NON_FUNDS", "cfnd" not in sel, str(sel))
|
||||||
|
check("select includes top return + resilient + payout",
|
||||||
|
{"aaa", "bbb", "ccc"} <= set(sel), str(sel))
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
def main() -> int:
|
||||||
test_pool()
|
test_pool()
|
||||||
test_text_and_objective()
|
test_text_and_objective()
|
||||||
|
|
@ -590,6 +624,7 @@ def main() -> int:
|
||||||
test_xcheck()
|
test_xcheck()
|
||||||
test_taxplan()
|
test_taxplan()
|
||||||
test_drawdown()
|
test_drawdown()
|
||||||
|
test_cef()
|
||||||
test_edgar_live()
|
test_edgar_live()
|
||||||
print(f"\n{PASS} passed, {FAIL} failed")
|
print(f"\n{PASS} passed, {FAIL} failed")
|
||||||
return 1 if FAIL else 0
|
return 1 if FAIL else 0
|
||||||
|
|
|
||||||
2
tyg-split.csv
Normal file
2
tyg-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2020-05-01,1:4
|
||||||
|
2
utf-split.csv
Normal file
2
utf-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2025-09-22,1017:1000
|
||||||
|
2
vlt-split.csv
Normal file
2
vlt-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2009-05-26,1:5
|
||||||
|
2
ztr-split.csv
Normal file
2
ztr-split.csv
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
Date,Stock Splits
|
||||||
|
2012-06-27,1:4
|
||||||
|
Loading…
Reference in New Issue
Block a user