Development of Statistically Sound Methods and User-friendly Software for Performing Data Forensics

Co-Principal Investigator: Yi-Hsuan Lee

Producers and consumers of student assessments are increasingly concerned about fraudulent behavior before and during the test. The United States Government Accountability Office noted that effective test security policies and procedures, when properly implemented, can help prevent and detect cheating and other irregularities that can undermine the validity and reliability of assessments. Statistical procedures for detection of testing irregularities, often referred to as data forensics analysis, are employed by virtually all testing organizations. Among the methods used in data forensics analysis, however, no method is universally accepted, several do not have satisfactory power to detect test fraud, and several cannot be performed using publicly available software packages. The goal of this project is to develop new statistics in each of four areas of data forensics analysis: erasure analysis; detection of person misfit in adaptive tests; detection of item pre-knowledge; and detection of unusual response similarity, including answer-copying. The findings and the user-friendly software developed as part of the project will be useful to practitioners in the assessment field, testing companies, state educational department officials, and school administrators.

The researchers will first conduct some theoretical work (i.e., derivations and math proofs) to develop the equations for the new statistics. They will then test the functioning of the new statistics via demonstrations using real data and through Monte Carlo simulations. The researchers will then develop a user-friendly software package for computing the new statistics. The research team will make the software available on a dedicated project website and inform potential users through conference presentations. The research team will also submit manuscripts to peer-reviewed journals.