24. A growing operator grammar with coherent evidence
On this page · Permalink · Source line 1030
A large algebraic signal model should be able to add operator programs without changing the meaning of its existing evidence. This requires more than assigning a new softmax weight when a candidate is introduced.
Let \(\mathcal G\) be a countable declared program set, with one fixed probability mass function \(\pi(g)>0\) satisfying \(\sum_g\pi(g)=1\). Every program specifies a likelihood \(L_g(D)\) for the same observed data event or density relative to a common base measure. Assume
Let finite active sets \(\mathcal G_n\) increase to \(\mathcal G\). Their unnormalized evidence is \(Z_n=\sum_{g\in\mathcal G_n}\pi(g)L_g(D)\). Only stages with \(Z_n>0\) are considered.
Proposition 23: Exact accounting for an enumerated grammar
Define the full posterior \(P(g\mid D)=\pi(g)L_g(D)/Z\) and extend the active posterior \(P_n(g\mid D)=\pi(g)L_g(D)/Z_n\) by zero outside \(\mathcal G_n\). Then
For any common family of candidate predictive probability kernels, the corresponding predictive mixtures differ in total variation by at most \(\tau_n\). If the omitted unnormalized evidence has a bound
then
Read proof · Recorded Lean scope · Permalink · Source line 1042
Proof
On the active set, \(P=(Z_n/Z)P_n\), so the total mass of the difference there is \(\tau_n\). The mass outside is also \(\tau_n\). Half the sum of absolute differences is therefore \(\tau_n\). Applying the same probability kernel to two mixing measures cannot increase total variation: integrate any event probability, a function taking values in \([0,1]\), against their difference. For the final bound, write \(Z=Z_n+W_n\) with \(0\leq W_n\leq b_n\) and use that \(W_n/(Z_n+W_n)\) increases with \(W_n\). \(\square\)
Since the active sets exhaust the grammar and the evidence sum is finite, \(Z_n\uparrow Z\) and \(\tau_n\downarrow0\). This is posterior approximation inside the declared grammar. It does not establish that the real signal-generating mechanism belongs to it.
For a discrete data sequence, a probability likelihood is at most one, so the omitted prior mass is a valid \(b_n\). A continuous-data density need not be at most one. It requires a density envelope or another justified bound on omitted evidence. A small unnormalized tail can also carry substantial posterior mass when the active evidence is very small.
The fixed prior need not be uniform. For an explicit enumeration \(g_1,g_2,\ldots\), \(\pi(g_k)=2^{-k}\) is a normalized example, with omitted prior mass \(2^{-n}\) after \(n\) programs. It expresses a particular enumeration bias, not a canonical prior on mathematical meaning.
A program outside the enumerated prefix can be accounted for only when its prior and likelihood are defined or bounded. A genuinely outside-grammar component needs its own model. Assigning an unexplained residual probability is not a Bayesian treatment of missing mechanisms.
24.1 Refining a program without double counting it
On this page · Permalink · Source line 1073
A representation-rich language can contain many syntactic versions of one mechanism. It can also split one broad mechanism into more specific children. These operations must conserve prior mass when they are intended as a re-expression of the same model.
Proposition 24: Prior-preserving program refinement
Suppose a parent program \(g\) is replaced by children \(g_j\) with weights \(\alpha_j\geq0\), \(\sum_j\alpha_j=1\), and suppose its likelihood is the mixture
Give child \(j\) prior mass \(\pi(g)\alpha_j\) and leave all other programs unchanged. Then total evidence is unchanged and the sum of the children's posterior masses equals the former parent's posterior mass. Posterior predictive distributions are also unchanged when the same mixture relation holds for the joint likelihood of \((D,y)\) for every predicted event \(y\).
Read proof · Recorded Lean scope · Permalink · Source line 1077
Proof
The children's total unnormalized evidence is \(\sum_j\pi(g)\alpha_jL_{g_j}(D)=\pi(g)L_g(D)\). Thus both the total normalizer and every other program's posterior are unchanged. The same sum for joint evidence gives the predictive statement after normalization by the unchanged evidence of \(D\). \(\square\)
The mixture identity must hold for the whole data record. Averaging one-step likelihoods independently at each time generally describes a switching mechanism, not a fixed child selected once for the whole sequence.
If the children are only exact atlas recodings of the same operator and data model, their likelihoods are identical. Splitting the original prior mass leaves every prediction unchanged. Giving each duplicate a fresh full prior mass instead changes the model's prior preference. That may be an intentional new prior, but it is not invariance under representation.
The implementation's existing held-out predictive weights, described in Section 10, are not replaced by these identities merely by calling them posterior weights. The proposition supplies the probabilistic semantics that a future grammar-weighting implementation would need to enforce.