Abstract
In automated test assembly (ATA), 0-1 linear programming (0-1 LP) methods are applied to select questions (items) from an item bank to assemble an optimal test. The objective in this 0-1 LP optimization problem is to assemble a test that measures, in as precise a way as possible, the ability of candidates. Item response theory (IRT) is commonly applied to model the relationship between the responses of candidates and their ability level. Parameters that describe the characteristics of each item, such as difficulty level and the extent to which an item differentiates between more and less able test takers (discrimination) are estimated in the application of the IRT model. Unfortunately, since all parameters in IRT models have to be estimated, they do have a level of uncertainty to them. Some of the other parameters in the test assembly model, such as average response times, have been estimated with uncertainty as well. General 0-1 LP methods do not take this uncertainty into account, and overestimate the predicted level of measurement precision. In this paper, alternative robust optimization methods are applied. It is demonstrated how the Bertsimas and Sim method can be applied to take this uncertainty into account in ATA. The impact of applying this method is illustrated in two numerical examples. Implications are discussed, and some directions for future research are presented.
This is a preview of subscription content, log in via an institution to check access.
References
Adema, J. J., Boekkooi-Timminga, E., & van der Linden, W. J. (1991). Achievement test construction using 0-1 linear programming. European Journal of Operations Research, 55, 103–111.
Alvarez, P. P., & Vera, J. R. (2011). Application of robust optimization to the sawmill planning problem. Annals of Operations Research. Advance online publication. doi:10.1007/s10479-011-1002-4.
Ariel, A., Veldkamp, B. P., & Breithaupt, K. (2006). Optimal testlet pool assembly for multi-stage testing designs. Applied Psychological Measurement, 30, 204–215.
Armstrong, R., Belov, D., & Weissman, A. (2005). Developing and assembling the law school admission test. Interfaces, 35, 140–151.
Atamtürk, A. (2006). Strong formulations of robust mixed 0-1 programming. Mathematical Programming, 108, 235–250.
Ben-Tal, A., El Ghaoui, L., & Nemirovski, A. (2009). Robust optimization. Princeton: Princeton University Press.
Ben-Tal, A., & Nemirovski, A. (2000). Robust solutions of linear programming problems contaminated with uncertain data. Mathematical Programming, 88, 411–424.
Bertsimas, D., & Sim, M. (2003). Robust discrete optimization and network flows. Mathematical Programming, 98, 49–71.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park: Sage.
Huitzing, H. A., Veldkamp, B. P., & Verschoor, A. J. (2005). Infeasibility in automatic test assembly models: a comparison study of different methods. Journal of Educational Measurement, 42, 223–243.
De Jong, M. G., Steenkamp, J. B. E. M., & Veldkamp, B. P. (2009). A model for the construction of country-specific, yet internationally comparable short-form marketing scales. Marketing Science, 28, 674–689.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Erlbaum.
Soyster, A. L. (1973). Convex programming with set-inclusive constraints and applications to inexact linear programming. Operations Research, 21, 1154–1157.
Theunissen, T. J. J. M. (1985). Binary programming and test design. Psychometrika, 50, 411–420.
van der Linden, W. J. (1998). Optimal assembly of psychological and educational test. Applied Psychological Measurement, 22, 195–211.
van der Linden, W. J. (2005). Linear models for optimal test design. New York: Springer.
van der Linden, W. J., & Boekkooi-Timminga, E. (1989). A maximin model for test design with practical constraints. Psychometrika, 54, 237–247.
van der Linden, W. J., & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer.
Veldkamp, B. P. (2002). Multidimensional constrained test assembly. Applied Psychological Measurement, 26, 133–146.
Veldkamp, B. P. (1999). Multi-objective test assembly problems. Journal of Educational Measurement, 36, 253–266.
Veldkamp, B. P., Matteucci, M., & de Jong, M. (2012, submitted). Uncertainties in the item parameter estimates and automated test assembly.
Acknowledgements
This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this report are those of the author and do not necessarily reflect the position or policy of LSAC.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Veldkamp, B.P. Application of robust optimization to automated test assembly. Ann Oper Res 206, 595–610 (2013). https://doi.org/10.1007/s10479-012-1218-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10479-012-1218-y