f/lpt-dividend.csv at 3c5bee96ceb54e7c6d0fb87afbcec59e6b77a3f3

gmp/f

Greg Pomerantz 3c5bee96ce Clean multi-vintage 'ctime' files: 1,065 files, 2.2M stale rows

Some dump files carry a trailing 'ctime' column: an older pipeline
appended a FULL re-download of the symbol per download (up to ~28
vintages per date, 2019-2022). The consumer keeps the last row per date,
so it silently used stale (~2021) vintages where Yahoo later revised
values. scripts/clean_multivintage.py keeps, per date, the rows from the
newest ctime (event files keep distinct same-day amounts; history/split
keep the single newest row), drops the ctime column, sorts by date.
Applied to the data root and overrides/frozen (which carried the same
artifact): 1,065 files, 2,202,712 stale rows dropped.

Also: stripped 3 UTF-8 BOMs (teg/lo/krft), refreshed the event backup
with the cleaned files, and re-ran the double-listing scan: category C
(83 symbols, 18,636 repeated within-file rows) is fully explained by the
multi-vintage artifact and is now gone; A (6,589 pairs, corrected) and
B (12,366 pairs, open review list) are unchanged. Whole-set structural
QC after cleaning: 0 duplicate dates, 3 syms/15 rows of OHLC invariant
violations, 15 syms/2,897 rows of non-positive prices (mostly
long-delisted tickers), 8 unsorted files (reader sorts).

2026-08-31 22:25:02 -04:00

1.7 KiB

Raw Blame History

1	Date	Dividends
2	1994-10-26	0.43
3	1995-01-26	0.4
4	1995-04-25	0.4
5	1995-07-28	0.4
6	1995-10-30	0.4
7	1995-12-27	0.4
8	1996-03-28	0.4
9	1996-06-27	0.4
10	1996-09-27	0.41
11	1996-12-27	0.41
12	1997-03-27	0.41
13	1997-06-27	0.41
14	1997-09-29	0.42
15	1997-12-29	0.42
16	1998-03-30	0.42
17	1998-06-29	0.42
18	1998-09-29	0.45
19	1998-12-29	0.45
20	1999-03-30	0.45
21	1999-06-29	0.45
22	1999-09-29	0.52
23	1999-12-29	0.52
24	2000-03-29	0.52
25	2000-06-28	0.52
26	2000-09-27	0.57
27	2000-12-27	0.57
28	2001-03-28	0.57
29	2001-06-27	0.57
30	2001-09-27	0.59
31	2001-12-27	0.59
32	2002-03-27	0.59
33	2002-06-27	0.59
34	2002-09-27	0.6
35	2002-12-27	0.6
36	2003-03-28	0.6
37	2003-06-27	0.6
38	2003-09-29	0.605
39	2003-12-29	0.605
40	2004-03-30	0.605
41	2004-06-29	0.605
42	2004-09-29	0.61
43	2004-12-29	0.61
44	2005-03-30	0.61
45	2005-06-29	0.61
46	2005-09-28	0.615
47	2005-12-28	0.615
48	2006-03-29	0.615
49	2006-06-28	0.615
50	2006-09-27	0.62
51	2006-12-27	0.62
52	2007-03-28	0.62
53	2007-06-27	0.62
54	2007-09-27	0.625
55	2007-12-27	0.625
56	2008-03-28	0.625
57	2008-06-27	0.625
58	2008-09-29	0.625
59	2008-12-29	0.475
60	2009-03-30	0.475
61	2009-06-29	0.475
62	2009-09-29	0.475
63	2009-12-29	0.475
64	2010-03-30	0.475
65	2010-06-29	0.475
66	2010-09-29	0.475
67	2010-12-29	0.475
68	2011-03-30	0.475
69	2011-06-29	0.475
70	2011-09-28	0.475
71	2011-12-28	0.475
72	2012-03-29	0.475
73	2012-06-27	0.475
74	2012-09-27	0.475
75	2012-12-28	0.475
76	2013-03-27	0.475
77	2013-06-27	0.475
78	2013-09-27	0.475
79	2013-12-30	0.475
80	2014-03-28	0.475
81	2014-06-27	0.475
82	2014-09-29	0.475
83	2014-12-30	0.475
84	2015-03-30	0.475
85	2015-06-29	0.475
86	2015-09-29	0.475
87	2015-12-30	0.475
88	2015-12-31	0.475
89	2016-03-29	0.475
90	2016-03-30	0.475
91	2016-06-29	0.475
92	2016-09-29	0.475
93	2016-12-29	0.475
94	2017-03-29	0.4
95	2017-06-29	0.4
96	2017-09-29	0.4
97	2017-12-29	0.4
98	2018-03-29	0.4
99	2018-06-29	0.4
100	2018-09-28	0.4
101	2018-12-31	0.4
102	2019-03-29	0.41
103	2019-06-28	0.41
104	2019-09-30	0.41
105	2019-12-31	0.41

1.7 KiB Raw Blame History

1.7 KiB

Raw Blame History