The goals of this study were threefold: (1) to provide a comparative analysis of five different item analysis indices using both IRT and non-IRT indices; (2) to describe the characteristics of flagged items; and (3) to investigate the appropriateness of logistic regression as an item analysis technique for further studies. The performance of five item analysis indices was examined in this study; p-value, point-biserial correlation, Rasch b parameter, Rasch χ² fit and χ² from the logistic regression model. A factor analysis indicated that the first factor was a difficulty factor and included the Rasch b and p-value indices, whereas the second factor represented the discrimination power of an item and consisted of the Rasch χ² fit, χ² from the logistic regression and the point-biserial correlation. There was a tendency for difficult items to exhibit more unusual response patterns. The χ² from the logistic regression fit was found to be an appropriate index when a researcher is attempting to detect items that function differentially and when the subject pool consists of varying ability groups.