| dc.description |
In their response to my article, "Item Response Theory, Vertical Scaling, and Something's Awry in the State of Test Mark," Yen, Burket, and Fitzpatrick (this issue) question the validity of my field observations. I present evidence that validates those observations. They claim that my simulation was unrealistic. I present evidence (convincing, I believe) that they are simply misinformed. They argue that Thurstone scaling has several weaknesses. I present information that should enable them to understand the procedure better and that reveals that the supposed weaknesses do not, in fact, exist. They say they are very "up front" about not being able to measure students at the extremes accurately but claim the vast majority of students are assessed well, thus implying that my use of data for students at the 2nd and 98th percentiles led to conclusions that would not be found if other segments of the score distribution were examined. I duplicated the analyses at the 15th and 85th percentile points and demonstrated that they were wrong. Yen et al. seem to be convinced that the variance of performance decreases (they use the term "homogenization") as learning progresses. Using their published data for 7 on-grade tests administered at the beginning and end of each school year, when the same on-grade test form was used-thus eliminating any confounding introduced by scaling—I show that in 67 of 77 instances the variance increased. This should serve as convincing evidence to the most doubtful person that the variance of performance increases as learning progresses. Given that there is a serious problem, as clearly illustrated in Figure 2, I suggest some avenues that research could take to address it. |
