PROOF_AND_SIMULATION_AUDIT_V09.md
Streaming Polyphase Factor Decomposition / SKAT
PROOF_AND_SIMULATION_AUDIT_V09.md
1 # Realistic SKAT simulation audit — v0.9
2
3 Date: 2026-09-23
4
5 ## Purpose
6
7 Validate the full streaming/deep-tail path on the project's established
8 rare-variant SKAT simulation machinery rather than only on abstract matrix
9 families.
10
11 The tested path is
12
13 raw genotype matrix
14 -> per-variant centering/scaling
15 -> covariate residualization
16 -> rare-variant weights
17 -> streamed sample kernel
18 -> eigenspectrum
19 -> K-COS deep-tail evaluation.
20
21 ## Historical copy-LD simulator defect
22
23 The historical strong-copy-LD benchmark allocated the genotype matrix using
24
25 `G = np.empty((n,m), dtype=np.float64)`
26
27 and then accumulated haplotypes with `+=`.
28
29 Because `np.empty` has unspecified initial contents, that implementation was
30 not a valid reproducible simulator.
31
32 v0.9 repairs the generator by using a zero-initialized matrix. Earlier
33 copy-LD benchmark numbers are therefore retained only as historical evidence.
34
35 The separate latent-Gaussian LD generator did not have this defect.
36
37 ## Scenarios
38
39 Four deterministic simulation regimes were tested:
40
41 1. latent-Gaussian moderate LD, n=600, m=3000;
42 2. latent-Gaussian strong LD, n=600, m=3000;
43 3. corrected strong copy-LD, n=600, m=3000;
44 4. corrected strong copy-LD, n=1000, m=5000.
45
46 Median adjacent within-block r^2 ranged from
47 0.0069
48 to
49 0.8298.
50
51 The large corrected copy-LD case had rank 996 and effective
52 spectral rank 353.027.
53
54 ## Raw-genotype streaming invariance
55
56 Resident block sizes 31, 127, and 509 were tested for every scenario.
57
58 Maximum relative discrepancy between the streamed kernel and the fully
59 materialized weighted/residualized factor reference:
60
61 6.611e-16.
62
63 ## Phenotype-space algebra
64
65 One hundred Gaussian-null residual draws per scenario were evaluated through
66 the factor, sample-kernel, and eigencoordinate forms.
67
68 Worst relative discrepancy:
69
70 8.318e-16.
71
72 ## Realistic deep tails
73
74 The ordinary scenario battery used tilt fractions 0.90, 0.97, and 0.99.
75
76 The n=1000, m=5000 corrected copy-LD case was additionally pushed through
77 0.995, 0.997, 0.999, and 0.9995.
78
79 Deepest high-resolution full-spectrum reference:
80
81 log10(p) = -457.973916.
82
83 Worst absolute K-COS/reference discrepancy across all realistic LD tail tests:
84
85 3.209e-12 decades.
86
87 ## Evidence label
88
89 - algebraic streaming equivalence: proved separately, numerically reproduced;
90 - realistic-LD behavior: finite synthetic validation;
91 - deep-tail accuracy: computationally verified on the stated spectra/queries;
92 - simulation code correction: confirmed implementation defect and repair.
93
94 This does not establish that these generators reproduce every feature of
95 population sequencing data. External cohort replication remains separate.
96 Finite-sample logistic/Bernoulli calibration also remains a separate
97 statistical problem.
98
99 ## Numerical warning observed during deepest adaptive refinement
100
101 At the deepest corrected copy-LD queries, NumPy emitted overflow warnings while
102 evaluating an intermediate conservative growth sum inside the adaptive
103 certificate planner. Those candidate bounds became infinite and were not
104 accepted as final certificates. The returned K-COS results retained finite
105 certified relative bounds between roughly 4e-9 and 9e-9 and agreed with the
106 higher-resolution exact-spectrum tilted reference. The warning is retained as
107 an implementation-hardening target rather than hidden.