Psychometric Considerations for Learning Maps-Based Assessments

Learning map models are a type of cognitive model composed of multiple interconnected learning targets and other critical knowledge and skills. The Dynamic Learning Maps® (DLM®) Alternate Assessment System uses learning maps models as the basis for assessment for students with significant cognitive disabilities. The DLM maps and corresponding assessments provide access to multiple and alternate routes to achieving the learning targets, making it more inclusive for learners with various disabilities. However, the unique approach and design of the assessment system designed to maximize accessibility also poses unique psychometric challenges. In this presentation, we will discuss (1) the DLM assessment design; (2) the diagnostic classification model used to score the assessment; (3) approaches to empirically evaluating the map structures including future directions for data collection; and (4) results from research conducted on teachers’ interpretations of diagnostic assessment results.

Below are additional resources for those who are interested in getting more information about the topics discussed.

• DLM scoring model
  • Chapter 5 of the 2015–2016 Technical Manual Update—Integrated Model
  • Chapter 5 of the 2017–2018 Technical Manual Update—Integrated Model
  • Bartholomew, D., Knott, M., & Moustaki, I. (Eds.) (2011). Latent class models. In Latent Variable Models and Factor Analysis: A Unified Approach (3rd ed., pp. 157–189). West Sussex, United Kingdom: Wiley.
• Map Validation
  • Posterior predictive model checks
    • Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (Eds.) (2014). Model checking. In Bayesian Data Analysis (3rd ed., pp. 141–164). Boca Raton, FL: CRC Press.
    • Levy, R. & Mislevy, R. J. (Eds.) (2016). Model evaluation. In Bayesian Psychometric Modeling (pp. 231–252). Boca Raton, FL: CRC Press.
    • McElreath, R. (2016). Sampling the imaginary. In Statistical Rethinking: A Bayesian Course With Examples in R and Stan (pp. 49–70). Boca Raton, FL: CRC Press.
  • Model comparisons
    • loo R package
    • Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4.
    • Vehtari, A., Gelman, A., & Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646.