Computerized Adaptive Testing
From Concept to Implementation
David J. Weiss and Alper Sahin
Request a free digital professor copy on VitalSource ?This title is part of the Methodology in the Social Sciences Series, edited by Todd D. Little, PhD.
“This comprehensive book on CAT is unique in being an authored work with a coherent style and perspective. It covers all aspects of using CAT, including the development of item banks and operational systems along with the technical details of item calibration, proficiency estimation, and practical requirements, such as exposure control and content balancing. Also included is a historical overview of the development of the concepts underlying CAT. Written at an accessible level, the book does not require a strong background in psychometrics. It is an excellent resource for anyone with administrative responsibility for a CAT system, or for courses in applied measurement.”
—Mark Reckase, PhD, University Distinguished Professor Emeritus, College of Education, Michigan State University
“This book is great—it is the most easy-to-understand material on CAT I have read. It covers a wide range of research, from the early days of CAT to today. I recommend this book to any student who is interested in CAT, as well as anyone trying to develop a CAT program.”
—T. Chris Oshima, PhD, Department of Educational Policy Studies, Georgia State University
“This book meets a need in the measurement field for a comprehensive introductory presentation of CAT. It covers the basics in an accessible way, while also making clear the range of applications and contexts in which the basic CAT approach can be extended, as well as areas in need of future research.”
—Daniel M. Bolt, PhD, Department of Educational Psychology, University of Wisconsin–Madison
“Weiss has been involved with CAT from the beginning—he even gave the process its name. Reading this incredibly detailed, practical guide is like having Weiss as a consultant. He offers step-by-step advice on the development and implementation of a CAT system, based largely on his experience, his own research, and that of his students. This book is an invaluable resource for testing and measurement professionals who are considering or implementing a CAT system, as well as graduate students in training.”
—David Thissen, PhD, Department of Psychology and Neuroscience (Emeritus), University of North Carolina at Chapel Hill
Table of Contents
I. The Why, What, and How of CAT
- What is CAT?
- Benefits of CAT
- Challenges of CAT
- When is CAT Feasible and When is it Not?
2. The Birth of Adaptive Testing and Its Current Status
- Binet’s Adaptive Test
- The Evolution of Adaptive Testing into CAT
- Early Approaches to CAT
- The Current Status of Applications of CAT
3. Item Response Theory Used in CAT
- Item Response Functions for Dichotomously Scored Items
- Option Response Functions for Polytomously Scored Items
- Information Functions
- Measuring People with IRT
- Estimating IRT Item Parameters
- Summary
4. Using CAT to Measure Individual Differences
- The Components of a Measurement CAT
- Putting it All Together: How CAT Works
- Some Properties of Fully Adaptive IRT-Based CATs
- Advanced Issues
5. CAT for Classification and Classification Plus Measurement
- Setting the Cutscore
- Types of Classification Applications
- Advanced Issues
II. Getting Your CAT Operational
6. Feasibility and Planning
- Developing a Business Case
- The Item Bank
- Costs of Test Delivery
- Potential Risks
- Non-Psychometric Risks
- Additional Considerations
- Conclusions
7. Developing a CAT Item Bank: Writing and Evaluating Items
- How Many Items Do I Need in My Bank?
- What Kinds of Items Work for CAT?
- Developing High-Quality Items and Item Banks
- Characteristics of a Good Item Bank
- Further Information
8. Pretesting and Linking Your Items, and Adding New Items to an Existing Bank
- Pretesting Your Items
- Creating Your Data Matrix
- IRT Item (and Person) Parameter Linking
- Additional Considerations in Item Bank Development
- Advanced Topic: Automated Item Generation
9. Using Simulations to Design a CAT
- Post-Hoc or Real-Data Simulation
- Designing a CAT from a CAT Item Bank
- Hybrid Simulations
- Conclusions
10. Implementing Your CAT
- CAT Delivery Software
- How Can I Get CAT Operational for My Organization?
III. Applications, Special Types of CATs, Special Issues, and a Potential Future for CAT
11. Examples of Operational CAT Programs
- CAT in Education
- CAT for Military Accessioning
- CAT for Pre-Employment Testing
- CAT for Certification and Licensure
- CAT in Healthcare
- Additional Applications of CAT
12. Other Types of CATs
- Multistage Tests
- CAT for Cognitive Diagnosis
- CAT for Measuring Multiple Variables
13. Special Issues
- Constrained CATs
- CAT Using Items with Multiple Questions Per Stimulus
- CAT Administration Issues
- The Examinee Experience
14. Adaptive Measurement of Change: Evaluating the Psychometric Significance of Measured Individual Change
- Identifying Significant Individual Change Using an Early Adaptive Measurement of Change (AMC) Method
- The Likelihood Ratio Index (LRI)
- Example AMC Applications
- Multivariate (Profile) Change
- AMC Research
- Discussion and Conclusions
15. Some Pathways Toward Future CATs and CAT Research
- Item Banking
- Test Delivery
- Supplementing and Improving Scores
- Integration of Testing and Instruction
- A Path for Future CAT Research: Does CAT Improve Validity?
- Leveraging Current and Future Technology to Improve Testing
Software Appendices
1. Software Packages for Estimating IRT Parameters
2. Software Packages for CAT Simulations
3. Software Platforms for CAT Delivery
References
Index
About the Authors
David J. Weiss, PhD, is Professor of Psychology at the University of Minnesota. Widely regarded as the father of computerized adaptive testing (CAT), he began his CAT research and founded the Quantitative and Psychometric Methods program at the University of Minnesota in 1970. His subsequent research provided the basis for the development of the CAT version of the Armed Services Vocational Aptitude Battery and a variety of other CAT implementations, including in education, nursing licensure, and personality measurement. Dr. Weiss has been active in CAT research continuously and has trained many leading CAT researchers and implementers. He was the founding editor of the journal Applied Psychological Measurement, and is co-founder and permanent President Emeritus of the International Association for Computerized Adaptive Testing (IACAT). He was the founding editor of IACAT’s official journal, the Journal of Computerized Adaptive Testing. Dr. Weiss also co-founded Assessment Systems Corporation and the Insurance Testing Corporation. He is a Fellow of the American Educational Research Association and the American Psychological Association.Alper Sahin, PhD, is Associate Professor of Educational Measurement and Evaluation in Türkiye. In the field of psychometrics, he is recognized for his seminal work on determining the minimum sample sizes necessary for the reliable estimation of item parameters in dichotomous item response theory models. He is the author of the State Test Anxiety Scale, which is based on a biopsychosocial model of test anxiety. From 2015 to 2023, Dr. Sahin served as Director of Education on the board of the International Association for Computerized Adaptive Testing. Currently, he is a board member of the Association for Measurement and Evaluation in Education and Psychology, and serves as Chairperson in the Department of Basic English at Atilim University.