EXPERIMENT_DESIGN.md
Arithmetic J-Space Training Lab
EXPERIMENT_DESIGN.md
1 # Preregistered candidate-space experiment v0.4
2
3 ## Status and estimand
4
5 The estimand is whether answer-only arithmetic training produces a small
6 residual-stream subspace whose target-blind local geometry predicts downstream
7 output-summary changes and whose coordinates have selective effects on free
8 generation under matched interventions.
9
10 The task domain is positive-operand addition/subtraction with nonnegative
11 results; it makes no inference about signed arithmetic, zero identities,
12 multiplication, or division.
13
14 The allowed conclusion is a **replicated candidate balanced causal arithmetic
15 subspace**. Version 0.4 cannot establish a J-space equivalent. Its machine-
16 readable claim ceiling is fixed to `JSPACE_CLAIM=NOT_ESTABLISHED` because formal
17 causal mediation, native shared multi-query access, and component-specific
18 upstream-writer rescue are not implemented.
19
20 ## Model and supervision
21
22 The serious profiles use the same conventional decoder architecture:
23
24 - 12 transformer blocks
25 - residual width 512
26 - 8 attention heads
27 - feed-forward width 2048
28 - approximately 38 million parameters
29 - fixed 260-token byte vocabulary
30 - maximum context 64
31
32 Weights start from scratch. Training minimizes ordinary causal next-token loss
33 on the answer and EOS; prompt tokens are masked. No intermediate arithmetic
34 trace, carry/borrow label, hidden-state target, auxiliary probe loss, bottleneck,
35 or workspace module is present.
36
37 ## Arms and pairing
38
39 | Arm | Training stream and target | Valid comparison |
40 |---|---|---|
41 | `correct` | Full structural stream; exact answer | Primary arithmetic behavior |
42 | `no_carry` | Restricted zero-transfer stream; exact answer | Local arithmetic without propagation |
43 | `pseudo_fst` | Same expressions/order as `correct`; seeded pseudo-valued FST digits with arithmetic-answer-matched orientation/length | Pseudo-valued sequential control with a matched arithmetic answer-length nuisance |
44 | `deranged` | Same expressions/order as `correct`; random deranged target | Random-label control |
45
46 For each seed, `correct`, `pseudo_fst`, and `deranged` have identical initial
47 model hashes and audited input-expression order. Targets differ. `no_carry`
48 uses an intentionally restricted stream and is not claimed to match the full
49 stream's prompts, token count, or answer-width distribution. The sealed
50 `no_transfer` panel is the exact shared task comparison between `correct` and
51 `no_carry`.
52
53 The `pseudo_fst` must pass its own target-task competence gate before its
54 geometry is used as an accuracy-matched specificity control. Comparing a
55 competent arithmetic model with an unlearned pseudo task is invalid.
56 Because its target width is taken from the true answer width, this arm is not
57 fully arithmetic-free and can support only a limited specificity contrast.
58
59 ## Derangement control
60
61 Every derangement window first attempts a random value derangement within
62 operator/output-width nuisance strata. If a stratum is underdetermined, the
63 construction deterministically coarsens it to output-width-only pooling. Exact
64 empirical target-multiset preservation remains required after either form of
65 pooling, and every assigned value must differ from its source value.
66
67 Only if an output-width-only stratum of at most 31 rows makes an
68 empirical-multiset value derangement mathematically impossible may the
69 construction deterministically synthesize canonical same-width decimal labels.
70 The limit of 31 applies separately to each impossible width stratum, not to the
71 window as a whole. Those labels still forbid all source/assigned value fixed
72 points, but the synthetic rows do not preserve the empirical target multiset.
73 The audit records the number and total-window fraction of synthetic rows,
74 rather than treating them as an exact derangement. That fraction has a strict
75 `0.001` (0.1%) cap. The audit records the configured cap and pass/fail result,
76 and the run fails closed if the cap is exceeded. Each constructed window that
77 uses synthetic rows emits exactly one `DERANGEMENT_SYNTHETIC_FALLBACK`
78 telemetry line. This is an explicit weakening of the random-label control,
79 accepted to prevent rare finite-window impossibility.
80
81 The construction therefore:
82
83 1. requires exact target-multiset preservation for every nonsynthetic stratum
84 2. forbids source/assigned value equality for both empirical and synthetic rows
85 3. never sorts answers by lexical or numeric value
86 4. records the target assignment, active nuisance strata, coarsening,
87 synthetic-row count and total-window fraction, its `0.001` cap, and cap
88 pass/fail
89 5. considers the synthetic fallback within a seeded candidate only after exact
90 operator/output-width and exact output-width-only constructions are
91 mathematically impossible; the up-to-eight seeded candidates test the
92 residual-correlation and string-agreement gates, not fallback eligibility
93 6. audits the initial window before training and every later window at
94 construction time; any configured correlation/string-agreement failure
95 or synthetic-fraction-cap failure stops the run
96
97 `derangement_max_abs_pearson` applies to the absolute row-weighted Pearson
98 correlation across eligible within-stratum correlations. Eligibility is frozen:
99 at least 32 rows, at least 10 distinct values, and output width of at least two.
100 All tiny and degenerate strata, including their correlations, remain in the
101 audit but do not enter that rejection gate. Prefix and suffix agreement are
102 tested as excess above the stratum chance baseline.
103
104 ## Counter-addressed data and splits
105
106 Training data are generated deterministically from `(seed, namespace, split,
107 counter, stage)`. Major train, validation, discovery, causal, IID, OOD, and final
108 classes use disjoint relation-family partitions. Addition commutation and
109 addition/subtraction inverse relations share a canonical family ID.
110
111 The counter stream is reproducible but does not guarantee global within-training
112 uniqueness; finite arithmetic collisions can occur. The pipeline does not
113 perform a global within-training collision audit, and no report may describe
114 all training rows as unique.
115
116 The fixed structural schedule is:
117
118 1. no carry/borrow
119 2. a single transfer event
120 3. contiguous transfer chains
121 4. a balanced mixture
122
123 Stages use fixed step boundaries, not observed-accuracy advancement, preserving
124 comparable schedules across the paired label arms. Structural evaluation panels
125 include IID trained widths, OOD longer widths, carry chains, borrow chains, and
126 the shared no-transfer panel.
127
128 ## Optimization and longitudinal records
129
130 Discovery uses 60,000 optimizer steps per arm and seed; locked replication uses
131 80,000. AdamW follows a warmup-stable-decay schedule. BF16 is used when the
132 installed CUDA stack supports it, otherwise scaled FP16 is used. Progress is
133 flushed every 100 optimizer steps, including stale-evaluation age. Light free-
134 generation evaluation occurs every 500 steps and full evaluation every 2,000.
135
136 Research checkpoints are specified in each JSON profile. The rolling resume
137 state includes model, optimizer, scaler, RNG, stream counter, stage, and runtime
138 compatibility information. Resume refuses incompatible resolved configuration
139 or runtime fingerprints.
140
141 ## Behavioral gates
142
143 Free generation is scored against each arm's trained completion, with the
144 ordinary arithmetic oracle also stored for controls. Reports retain exact
145 match, malformed rate, operation, digit width, and carry/borrow structure.
146
147 Candidate discovery is forbidden until the correct arm passes its
148 `probe_requires` IID, carry, borrow, and malformed thresholds. The stronger
149 `claim_requires` report additionally includes OOD exact match and the minimum
150 correct-minus-deranged arithmetic-oracle margin. The stronger report is retained
151 for scientific interpretation even though v0.4 cannot emit an established
152 claim.
153
154 `no_carry` and `pseudo_fst` have separate target-task competence gates. Transfer
155 panels for the `no_carry` model are scored on full arithmetic inputs and record
156 `evaluation_target_arm=correct`; its training restriction is not silently
157 applied to those evaluation rows.
158
159 ## Target-blind candidate discovery
160
161 Discovery accepts prefix token IDs only. Every prefix must end at ByteTokenizer
162 SEP, and EOS is forbidden, preventing teacher-forced answer leakage.
163
164 At upstream layer \(u\), sealed random residual perturbations are propagated to
165 candidate source layer \(\ell\). Their source-layer effects estimate
166
167 \[
168 G_c = \mathbb{E}[\Delta h_\ell\Delta h_\ell^\top] + \lambda I.
169 \]
170
171 Gradients at \(h_\ell\) estimate observability for exactly nine fixed,
172 answer-independent output consumers:
173
174 1. digit log-probability mass
175 2. EOS log probability
176 3. minus-sign log probability
177 4. output entropy
178 5. top log probability
179 6. four sealed random vocabulary log-probability projections
180
181 Thus
182
183 \[
184 G_o = \mathbb{E}[gg^\top] + \lambda I,
185 \qquad
186 B = G_c^{1/2}G_oG_c^{1/2}.
187 \]
188
189 The candidate basis is the leading mapped eigenspace of \(B\). Candidate rank
190 is the smallest registered rank reaching the heldout fraction of target-free
191 consumer-response variance; if none reaches it, the best registered rank is
192 retained with `threshold_not_met`. Discovery prefixes, rank-selection prefixes,
193 and sealed causal pairs are disjoint.
194
195 Layer/rank selection is exploratory in `carry_discovery`. A modal selection
196 across passing seeds is written to `workspace_lock.json`. Confirmation accepts
197 only that source layer, upstream layer, and rank.
198
199 The fitted rank-matched controls are:
200
201 - PCA of discovery source states
202 - one seeded random orthonormal subspace
203 - a complementary balanced-spectrum subspace
204
205 Version 0.4 does not implement a bootstrap stability analysis or a distribution
206 of many random subspace draws. The empirical upstream perturbation is a local
207 residual intervention, not an exhaustive head/MLP controllability estimate.
208
209 ## Implemented sealed causal assays
210
211 The provider constructs semantically matched clean/corrupt pairs differing at a
212 registered carry or borrow event column, with expected affected and unaffected
213 answer-digit positions. Answers are unavailable during candidate fitting and
214 selection. After the candidate is frozen, sealed answer values and widths are
215 used for pair construction, baseline eligibility, directional evaluation, and
216 free-generation scoring.
217
218 At the final prefix SEP position, the candidate, PCA, random, and complement
219 bases receive:
220
221 1. **Coordinate ablation:** remove the registered projection around the
222 discovery mean from a clean residual state.
223 2. **Clean-to-corrupt patch:** transplant clean donor coordinates into the
224 corrupt recipient state; audit predicted affected digits and preservation of
225 unaffected digits.
226 3. **Independent upstream residual transplant:** replace the clean upstream
227 residual with its matched corrupt donor residual and propagate to the source
228 layer.
229 4. **Coordinate rescue:** after that upstream transplant, restore only the
230 registered source-layer coordinates from the clean source state.
231
232 The causal assay regenerates autoregressively after intervention. Rank-matched
233 controls are energy-scaled to the candidate intervention norm for the same
234 state. Individual outputs and intervention norms are retained.
235
236 All primary causal estimands have registered denominators. Ablation counts only
237 pairs whose natural clean generation is a valid EOS-terminated signed decimal
238 and exactly correct. Patch requires both natural clean and corrupt generations
239 to be valid and correct with distinct targets. Rescue further requires the
240 unrescued upstream corrupt-donor transplant to generate the corrupt target
241 exactly. Discovery requires at least 128 baseline-qualified pairs and 32
242 directional rescue pairs; confirmation requires 512 and 128. Insufficient
243 qualified data fails the corresponding gate.
244
245 Coordinate rescue after an upstream residual transplant is an implemented
246 causal diagnostic. It is **not** formal mediation and does not identify which
247 attention head or MLP wrote the rescued state.
248
249 ## Arithmetic specificity
250
251 If both `correct` and `pseudo_fst` pass target-task competence and candidate
252 selection, the same geometry and causal battery runs in both models. The
253 arithmetic-specificity diagnostic requires:
254
255 - IID target-task accuracy matched within the registered tolerance
256 - correct-arm candidate-minus-strongest-control effect exceeding the analogous
257 pseudo-FST effect by the registered margin
258
259 This is a limited candidate specificity diagnostic. The answer-width nuisance
260 is shared deliberately, so this contrast does not isolate every arithmetic cue
261 and does not replace native shared access or mediation.
262
263 ## Locked replication statistics
264
265 Confirmation retains item-level predictions and directional patch records.
266 Seed is the replication unit. For ablation, patch, and rescue, the registered
267 effect is candidate minus the strongest available PCA/random/complement control.
268 The analysis uses:
269
270 - paired seed-level effect tests
271 - Holm correction across the three registered causal contrasts
272 - family-clustered hierarchical bootstrap for item-level directional patch
273 superiority
274
275 The configured confirmation has eight seeds and requires the registered seed
276 count. Passing statistics means that the implemented candidate effects
277 replicated. It cannot lift the release claim ceiling.
278
279 ## Mandatory assays not implemented in v0.4
280
281 The following are explicit fail-closed gates:
282
283 - formal causal mediation with a registered estimand and intervention chain
284 - native shared multi-query access from one held state
285 - component-specific upstream-writer perturbation and downstream coordinate
286 rescue
287
288 An unrelated-language preservation panel is also not bundled because the model
289 is arithmetic-only and has no meaningful language competence baseline.
290
291 No combination of loss, exact accuracy, balanced-space score, ablation, patch,
292 rescue, p-value, or bootstrap interval in this release can change
293 `JSPACE_CLAIM` from `NOT_ESTABLISHED`.