27. Actions and observations are different inputs
On this page · Permalink · Source line 1129
Let \(\mathcal A\) and \(\mathcal O\) be finite nonempty action and observation sets. A symbol \(s=(a,o)\) records an action followed by an observation. A word \(w=s_1\cdots s_k\) has temporal order from left to right. The empty word is \(\epsilon\).
Write
The process is reset to one specified initial distribution before this experiment. The actions in this definition are fixed externally. This is not, in general, the distribution obtained by conditioning a passive record on an adaptively selected sequence of actions. We assume the consistency laws
These laws specify causal probabilities on finite observation trees. A fixed history-dependent policy can be evaluated by selecting the action on each branch and multiplying the appropriate conditional probabilities. This does not establish optimality of that policy or its identification from insufficient data.
27.1 A finite hidden-state construction
On this page · Permalink · Source line 1149
A concrete realization has a belief \(p\in\mathbb R^n_+\) with \(\mathbf1^\top p=1\). For each \(s=(a,o)\), let \(M_s\in\mathbb R^{n\times n}\) be entrywise nonnegative, with
The matrix entry \(M_{a,o}(j,i)\) is the probability of next hidden state \(j\) and observation \(o\), given current hidden state \(i\) and chosen action \(a\). This is a discrete probability instrument. It includes an action-specific transition followed by an observation, but does not require that factorization.
Put \(M_w=M_{s_k}\cdots M_{s_1}\). Then
on branches of positive probability. An observation with zero probability has no conditional posterior under that model. It must not be handled by silently dividing by an arbitrary numerical floor.
Proposition 26: Finite instrument closure
Starting with \(W_0=\operatorname{span}\{\mathbf1^\top\}\), define
This stabilizes after at most \(n-1\) strict enlargements at
Choose a full row rank matrix \(R\) with row space \(W_*\) and first row \(\mathbf1^\top\). There are unique matrices \(B_s\) with
For \(z=Rp\), both the next-observation likelihood and the posterior predictive state close:
Every linear summary determining all word probabilities by linear readouts for every initial belief has row space containing \(W_*\).
Read proof · Recorded Lean scope · Permalink · Source line 1168
Proof
The nested-space argument from Proposition 8 applies to the finite family \(M_s\). Multiplication of a test row by \(M_s\) prepends the symbol \(s\) to that test, so the stable span is exactly the displayed word span. Invariance supplies the matrices \(B_s\), and full row rank makes them unique. Apply \(R\) to the unnormalized and normalized hidden-state updates. If all test probabilities are linear functions of another summary, then each row \(\mathbf1^\top M_w\) lies in its row space. Equality on all beliefs implies equality on their linear span, which is \(\mathbb R^n\). \(\square\)
This proposition uses a known hidden-state model. It is not yet an observable-data learner. It also concerns all initial beliefs, whereas a model initialized at one fixed \(p_0\) can have a smaller reachable predictive dimension. The next construction handles that distinction without requiring recovery of hidden states.