20 Correlated review
On this page · Permalink · Source line 1868
Several reviews are not equivalent to several independent reviews. Reviewers may share training, notation, examples, software, or an unnoticed premise.
Let \(X_i=\theta+\varepsilon_i\) be numerical estimates of a fixed quantity \(\theta\). Assume \(\mathbb E\varepsilon_i=0\), \(\operatorname{Var}(\varepsilon_i)=\sigma^2>0\), and common pairwise correlation \(\rho\). For \(n>1\), assume \(-1/(n-1)<\rho\leq1\), so the mean has positive variance. These variables model estimates, not proof truth values. The mean has variance
Theorem 20.1: Effective reviewer count
Under the equicorrelated model, define the variance-equivalent reviewer count by
so that \(\operatorname{Var}(\bar X)=\sigma^2/n_{\mathrm{eff}}\). This effective count is a positive real number and need not be an integer.
Proof
The mean of \(m\) independent reviewers of variance \(\sigma^2\) has variance \(\sigma^2/m\). Equating this with the correlated mean variance gives
Solving for \(m\) gives the formula.
For \(n=1\), the effective count is one. At the excluded boundary \(\rho=-1/(n-1)\), the mean has zero variance and there is no finite variance-equivalent count. For fixed \(0<\rho\leq1\), \(n_{\mathrm{eff}}\to1/\rho\) as \(n\to\infty\). Negative correlations can give \(n_{\mathrm{eff}}>n\), but a fixed negative correlation is not admissible for arbitrarily large \(n\).
For a general positive definite covariance matrix \(\Sigma\), consider unbiased linear aggregates \(w^\top X\) with \(\mathbf 1^\top w=1\).
Theorem 20.2: Minimum-variance covariance-aware weights
The unique minimum-variance unbiased linear aggregate has weights
Its variance is
Proof
Minimize \(w^\top\Sigma w\) subject to \(\mathbf 1^\top w=1\). The Lagrangian is
Stationarity gives \(\Sigma w=\lambda\mathbf 1\), hence \(w=\lambda\Sigma^{-1}\mathbf 1\). The constraint gives
Positive definiteness makes the objective strictly convex, so the solution is unique. Substitution gives the variance formula.
Warning 20.3: Bias is not covariance
Covariance-aware weighting does not remove a common bias that is absent from the covariance model. If every reviewer shares bias \(b\), the mean squared error contains the irreducible term \(b^2\).
Under the model, dependence rather than nominal count determines the variance reduction. Whether a particular review process has this covariance structure is an empirical question. The formula alone neither endorses nor discredits a community judgment.