38. A complete two-action rational example
On this page · Permalink · Source line 1577
Use two hidden states only to generate a reference process. The learner does not receive these matrices. Let
The observation matrices are
The two transitions do not commute. Thus the process is genuinely action-dependent rather than one autonomous recurrence with an action label attached.
Choose tests \((\epsilon,(0,1))\) and histories \((\epsilon,(0,1))\). The observable tables give
The selected coordinate \(q\) is the probability of seeing observation one after choosing action zero. In the hidden realization,
The observable reconstruction \(B_s=G_sH^{-1}\) yields
The coefficients include negatives. Nevertheless, every operator preserves the cone \(R\mathbb R^2_+\), because \(B_sR=RM_s\). A sufficient normalized base is
This cone certificate uses the known generating model and is checked separately. It is not inferred merely from the finite probability table, and \(K_0\) is not claimed to be the maximal canonical base of Proposition 29.
The likelihoods and posterior maps can now be read from the matrices. For action zero and observation one,
Starting at \(q_0=13/24\), this gives probability \(13/24\) and \(q_1=15/26\). If action one then produces observation zero, its probability is
and the new predictive coordinate is
The joint probability of this two-step action/observation word is \(365/1152\). The reference checks compare these values and every word up to a declared finite test length against the hidden generator in exact rational arithmetic. The proposition, under its rank and coverage assumptions, is what extends the result to all word lengths.
The model's online state is not a sequence of newly measured future probabilities. It is a recursively maintained prediction initialized from learned experiment probabilities and updated after each actual action and observation. Uncertainty in those learned parameters is separate from the stochastic uncertainty in the next observation.