This analysis of simulated performance ratings on a six-point scale by two independent raters is an attempt to account for nonsystematic error in performance ratings under restrictive assumptions of classical measurement theory. Results suggest that rater agreement or covariance is not always a dependable estimate of score reliability, and that the practice of seeking additional raters for the adjudication of discrepant ratings is not equally appropriate in every evaluation decision context or at every possible rating score step. And, although nonsystematic rating error is most common at the midpoints of the rating scale, most of this midpoint error disappears when rater scores are averaged, so that the need for adjudication is most critical at the rating scale extremes, whether or not raters are in agreement.