75. Actions need coverage, not only more observations
On this page · Permalink · Source line 3081
For intervention evidence use the information \(\mathcal H_{t-1}\) just before choosing the action. Let the action be randomized from a known history-dependent policy \(\pi_t\). The true next-observation kernel under action \(a\) is \(P_t(\cdot\mid\mathcal H_{t-1},a)\). Randomization uses no unrecorded future information. For a predictable test \(\phi_t\in[0,1]\), define
Proposition 67: Covered adaptive-action evidence
If the target action has \(\pi_t(a\mid\mathcal H_{t-1})\geq\pi_{\min}>0\) at every time, then
Consequently Proposition 66 gives a time-uniform certificate for \(\sum_{s\leq t}\mu_s^a\) from \(\sum_{s\leq t}W_s^a\), with its boundary multiplied by \(1/\pi_{\min}\). The target is the average action-conditioned quantity along the encountered contexts, not a reset-population table or the state distribution under a different long-term policy.
Read proof · Recorded Lean scope · Permalink · Source line 3090
Proof
Condition first on the action and then on its observation. Only action \(a\) contributes, with probability \(\pi_t(a\mid\mathcal H_{t-1})\), cancelling the denominator. The range bound follows directly. Apply the scaled conditional moment-generating-function argument to the resulting martingale differences. \(\square\)
This simple bound is conservative. Variance-aware weighting can improve it, but cannot remove the need for information about the action of interest. When \(\pi_{\min}=0\), two models can give exactly the same law under the observed policy and disagree on a never-selected action. For example, both can emit a Bernoulli observation with parameter \(1/2\) under action zero, while their action-one parameters are \(1/10\) and \(9/10\). Repeating action zero cannot distinguish them.
The action randomization and its recorded probabilities are part of the experimental assumption. An observational association with a hidden selection mechanism is not automatically an intervention estimate. Long-horizon off-policy predictions require additional occupancy or trajectory-coverage arguments that this one-step statement does not provide.