WORKSPACE_PROTOCOL.md
Arithmetic J-Space Training Lab
WORKSPACE_PROTOCOL.md
1 # Arithmetic workspace execution protocol v0.4
2
3 ## Release boundary
4
5 This package asks whether ordinary answer-only arithmetic training produces a
6 low-dimensional residual-stream coordinate system with target-blind predictive
7 geometry and selective causal effects. The model is standard and the candidate
8 is measured after training; no workspace is inserted or supervised.
9
10 The registered arithmetic domain is positive-operand addition and subtraction
11 with nonnegative results. Negative answers, zero identities, multiplication,
12 and division are outside this release's estimand.
13
14 Version 0.4 is candidate-only. The following mandatory assays are not
15 implemented:
16
17 - formal causal mediation
18 - native shared multi-query access
19 - component-specific upstream-writer rescue
20
21 Consequently, all profiles hard-disable an establishing claim and must report
22 `JSPACE_CLAIM=NOT_ESTABLISHED`. The strongest allowed description is
23 **candidate balanced causal arithmetic subspace**.
24
25 ## Recommended command
26
27 ```powershell
28 $d="$env:USERPROFILE\Downloads\Arithmetic_JSpace_Training_Lab_v0.4.0"; & "$d\Run_Arithmetic_Workspace_Experiment.ps1" -Profile carry_discovery -Python "$env:USERPROFILE\miniconda3\envs\komorebi-full\python.exe" -Device cuda -SkipInstall
29 ```
30
31 Rerun the identical command to resume. The default discovery output is
32 `Downloads\Arithmetic_Workspace_Runs\carry_discovery`.
33
34 ## Profiles
35
36 | Profile | Function | Model | Workload |
37 |---|---|---|---:|
38 | `workspace_smoke` | Wiring, checkpoint, and assay smoke | 3×128 | 1 seed × 4 arms × 20 steps |
39 | `carry_discovery` | Exploratory layer/rank selection | 12×512 (~38M) | 3 × 4 × 60,000 |
40 | `carry_confirmatory` | Locked candidate replication | 12×512 (~38M) | 8 × 4 × 80,000 |
41
42 Confirmation requires the frozen `workspace_lock.json` from discovery. With
43 default paths, the runner finds it automatically; otherwise pass `-LockFile`.
44 Confirmation cannot establish a J-space claim.
45
46 On an RTX 3070 Ti Laptop GPU, budget minutes for smoke, about 3–9 continuous
47 days for discovery, and about 12–35 days for confirmation. Geometry and causal
48 assays can add substantial time. The measured ETA after 500 steps is more
49 reliable than these planning ranges.
50
51 Plan for roughly 30 GB of free disk for discovery and 80 GB for locked
52 replication. Model-only research checkpoints are intentionally retained, and
53 the runner does not prune completed arms automatically.
54
55 ## Model and training contract
56
57 The serious model has 12 layers, width 512, 8 heads, feed-forward width 2048,
58 and a 64-token context. It uses the fixed 260-token byte vocabulary. It starts
59 from random weights and sees only equations and answer completions.
60
61 Training uses:
62
63 - causal loss on answer bytes and EOS; prompt tokens masked
64 - AdamW and a warmup-stable-decay schedule
65 - BF16 when supported, otherwise scaled FP16 on CUDA
66 - gradient clipping
67 - deterministic counter-addressed data
68 - atomic rolling resume state
69 - immutable research checkpoints at registered steps
70
71 There is no scratchpad, worked trace, carry label, hidden-state target,
72 representation loss, bottleneck, or probe loss.
73
74 The structural schedule is frozen: no-transfer examples, single events,
75 contiguous chains, then a balanced mixture. It does not adapt to observed
76 accuracy.
77
78 ## Training arms and valid pairing
79
80 | Arm | Training rule | Interpretation |
81 |---|---|---|
82 | `correct` | Full addition/subtraction stream, exact target | Arithmetic candidate |
83 | `no_carry` | Zero-carry/zero-borrow subset, exact target | Restricted local-computation control |
84 | `pseudo_fst` | Full shared expression stream, seeded FST target | Pseudo-valued sequential control, arithmetic-answer-length matched |
85 | `deranged` | Full shared expression stream, random deranged target | Random-label control |
86
87 For a seed, `correct`, `pseudo_fst`, and `deranged` have paired initial weights
88 and identical input-expression order. `no_carry` necessarily has a restricted
89 training stream. It is not globally prompt-, token-, or answer-width-matched.
90 The sealed `no_transfer` evaluation panel is the exact shared comparison with
91 the correct model.
92
93 The pseudo-FST produces deterministic stateful digit values with the same
94 orientation and output length as the arithmetic answer. It controls digit
95 values while retaining that coarse arithmetic-derived schema signal, so it is
96 a conservative, limited specificity control rather than a wholly
97 arithmetic-free task. It must meet its own target-task competence threshold
98 before its candidate is used for the limited specificity diagnostic.
99
100 ## Derangement audit
101
102 Each derangement window first uses random value permutations within
103 operator/output-width strata. An underdetermined bucket is deterministically
104 coarsened to output-width-only pooling when needed. Every nonsynthetic stratum,
105 including a coarsened one, must preserve its exact empirical target multiset,
106 reject every source/assigned value fixed point, and never sort answers before
107 assignment.
108
109 Only if a width-only stratum of at most 31 rows makes an empirical-multiset
110 value derangement mathematically impossible may the generator deterministically
111 synthesize canonical same-width decimal labels. The limit of 31 is per
112 impossible width stratum. These rows retain zero source/assigned value fixed
113 points but do not preserve the empirical target multiset. The audit explicitly
114 reports their count and total-window fraction, the strict `0.001` (0.1%) cap,
115 and cap pass/fail. The run fails closed if that cap is exceeded. Each
116 constructed window using the fallback emits one
117 `DERANGEMENT_SYNTHETIC_FALLBACK` telemetry line. This fallback weakens the
118 deranged control and must be disclosed in interpretation, but it prevents a
119 rare finite training window from being impossible to construct.
120
121 The initial window is audited before training; every later deterministic window
122 is audited when constructed and fails closed if it violates a gate. Audits
123 record fixed points, cycle lengths, nonsynthetic multiset preservation,
124 synthetic-row count and total-window fraction, the fraction cap and pass/fail,
125 numeric correlations, and prefix/suffix agreement above chance.
126 `derangement_max_abs_pearson` gates the absolute row-weighted Pearson
127 correlation across eligible within-stratum correlations. Eligibility is fixed
128 at at least 32 rows, 10 distinct values, and two output digits. Smaller or
129 degenerate strata remain fully reported but do not enter this gate. Up to eight
130 deterministic candidates are attempted for the residual-correlation and string
131 agreement gates. Within each candidate, synthetic labels are eligible only
132 after exact operator/output-width and exact width-only constructions are
133 mathematically impossible. After the narrowly eligible synthetic fallback, any
134 remaining audit failure stops the run.
135
136 ## Data integrity
137
138 Each example records split, counter, pair ID, relation-family ID, operands,
139 operation, answer, carry/borrow count, longest chain, event columns, and digit
140 positions affected by counterfactual events.
141
142 Major split classes use disjoint family partitions, including addition
143 commutation and addition/subtraction inverse relations. The stream is
144 counter-addressed and reproducible, but global within-training uniqueness is
145 not guaranteed or audited. No report may describe every training row as
146 unique.
147
148 ## Progress and evaluation
149
150 Every serious training arm emits unbuffered telemetry every 100 optimizer
151 steps. A progress record includes seed, arm, step, stage, rolling/current loss,
152 learning rate, gradient norm, throughput, ETA, peak VRAM, and the age
153 of the latest evaluation. A repeated validation value is therefore not mistaken
154 for a fresh evaluation.
155
156 Light free-generation evaluation occurs every 500 steps and full evaluation
157 every 2,000. Final behavior panels cover trained-width IID, longer-width OOD,
158 carry chains, borrow chains, and no transfer. Discovery and confirmation retain
159 item-level final predictions for paired and family-clustered analyses; smoke
160 does not.
161
162 Candidate probing is skipped unless the correct model passes the registered IID,
163 carry, borrow, and malformed-output gate. A second, stronger report includes
164 OOD and correct-minus-deranged arithmetic-oracle margin. Separate competence
165 reports apply to `no_carry` and `pseudo_fst`.
166
167 ## Implemented target-blind discovery
168
169 The discovery API accepts only tokenized prefixes ending at SEP. EOS and answer
170 continuations are rejected. Answers and correctness never enter candidate
171 fitting or layer/rank selection.
172
173 Random residual perturbations at an upstream layer are propagated to the source
174 layer to estimate empirical local controllability:
175
176 \[
177 G_c=\mathbb{E}[\Delta h\Delta h^\top]+\lambda I.
178 \]
179
180 Local observability uses gradients of exactly these answer-independent output
181 summaries:
182
183 1. digit log-probability mass
184 2. EOS log probability
185 3. minus-sign log probability
186 4. output entropy
187 5. top log probability
188 6. four sealed random vocabulary log-probability projections
189
190 With their gradient Gram matrix \(G_o\), the candidate comes from leading
191 directions of
192
193 \[
194 G_c^{1/2}G_oG_c^{1/2}.
195 \]
196
197 Rank is selected on separate prefixes by prediction of perturbation-induced
198 changes in the same output summaries. This is heldout rank selection, not a
199 bootstrap-stability assay.
200
201 The registered rank-matched basis controls are PCA, one seeded random subspace,
202 and a complementary balanced-spectrum subspace. Version 0.4 does not sample a
203 large random-subspace distribution.
204
205 Discovery runs independently in competent `correct` and `pseudo_fst` models.
206 The discovery lock stores the modal passing correct-arm source layer, upstream
207 layer, and rank across exploratory seeds. Confirmation forbids reselection.
208
209 ## Implemented causal battery
210
211 The sealed provider constructs matched clean/corrupt addition or subtraction
212 pairs differing at a registered carry/borrow event column. It records affected
213 and unaffected answer-digit positions.
214
215 At the final prefix SEP residual position, candidate, PCA, random, and
216 complement bases receive four free-generation assays:
217
218 1. **Coordinate ablation** around the discovery mean.
219 2. **Clean-to-corrupt coordinate patch**, including a directional audit of
220 affected digits and preservation of unaffected digits.
221 3. **Independent upstream residual transplant** from the matched corrupt donor
222 into the clean sequence.
223 4. **Coordinate rescue after transplant**, restoring only registered
224 source-layer coordinates from the clean state.
225
226 For control bases, interventions are scaled to the candidate intervention norm
227 for the same state. Individual generated outputs and norms are saved.
228
229 The primary denominators are fail-closed. Ablation requires a valid and correct
230 natural clean generation; patch requires valid and correct natural clean and
231 corrupt generations with distinct answers; rescue also requires the unrescued
232 upstream corrupt-donor transplant to produce the corrupt answer exactly.
233 Discovery requires 128 baseline-qualified and 32 directional-rescue pairs;
234 confirmation requires 512 and 128. Valid means EOS-terminated, free of illegal
235 special tokens, and a nonempty signed decimal.
236
237 These are genuine local interventions, but their scope is limited. The upstream
238 donor is an entire residual state, not a specific head or MLP writer. Restoring
239 coordinates after that transplant is not a formal mediation estimand. The
240 package has no native multi-query continuation assay and no unrelated-language
241 preservation assay.
242
243 ## Specificity and registered statistics
244
245 When both correct and pseudo-FST models are competent, the pipeline fits and
246 causally tests each candidate independently. It compares candidate-minus-
247 strongest-control effects while requiring target-task accuracy to be within the
248 registered tolerance. This is the implemented, limited math-specificity
249 diagnostic. Because pseudo-FST target length is arithmetic-answer matched, it
250 cannot by itself separate every arithmetic structural cue from sequential
251 computation.
252
253 In confirmation, seed is the replication unit for ablation, patch, and rescue.
254 The registered analysis applies paired seed-level tests and Holm correction
255 across those three contrasts. Retained directional patch rows support a
256 family-clustered hierarchical bootstrap. This bootstrap evaluates directional
257 patch superiority; it is not a subspace-stability bootstrap.
258
259 Even a complete pass establishes only replication of implemented candidate
260 effects. The release ceiling cannot be overridden by statistics.
261
262 ## Output contract
263
264 ```text
265 run/
266 resolved_experiment.json
267 runtime.json
268 progress.jsonl
269 training/
270 seed_<n>/
271 training_summary.json
272 <arm>/
273 progress.jsonl
274 metrics.jsonl
275 resume_a.pt
276 resume_b.pt
277 resume_pointer.json
278 checkpoints/
279 behavior/
280 seed_<n>/
281 control_audits.json
282 <arm>.json
283 geometry/
284 seed_<n>/
285 causal/
286 seed_<n>/
287 confirmation_statistics.json
288 workspace_lock.json
289 experiment_summary.json
290 ```
291
292 `workspace_lock.json` is produced only by discovery when a valid selection is
293 available. `experiment_summary.json` contains the candidate ladder,
294 confirmation statistics, unimplemented required assays, and final
295 `JSPACE_CLAIM` state. There is no separate `claim_status.json` in v0.4.
296
297 ## Resume rules
298
299 - Rerun the identical command and output path to resume.
300 - Resume validates resolved configuration, model, tokenizer, runtime, seed,
301 arm, optimizer, scaler, RNG, and stream state.
302 - A mismatch fails closed; use a new output path after any configuration edit.
303 - `-NoResume` disables resume and should be paired with a new output path.
304 - Confirmation requires the exact discovery lock and records its hash.
305 - Do not alter a lock or inspect confirmatory outcomes before changing a
306 preregistered threshold.
307
308 ## Interpretation
309
310 A loss curve, exact-match gain, readable direction, balanced eigenvalue,
311 ablation, patch, rescue, or significant replicated effect is not alone a
312 workspace. In v0.4, all of them together still yield only candidate evidence
313 because mediation, native shared access, and component-specific writer rescue
314 remain absent.