Standardized achievement tests continue to be used as the primary criterion measures in the evaluation of basic skills programs despite explicit recommendations against this practice by many testing and evaluation experts. A major issue behind this criticism is examined in this comparative study of three types of tests. These tests represent three points on a content validity continuum which references their correspondence to the instructional program being evaluated. Data are examined on these tests at two levels in eight groups of elementary school students receiving instruction on various basic skills. The results show why and how standardized tests provide different information from other tests that are more closely matched to program instruction. They also show why out-of-level standardized test results often correspond more closely to the results of instructionally referenced tests as compared to at-level tests. The implications of this research for basic skill program evaluations are discussed.