IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v86y2021i1d10.1007_s11336-020-09743-0.html
   My bibliography  Save this article

Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes

Author

Listed:
  • Esther Ulitzsch

    (IPN – Leibniz Institute for Science and Mathematics Education)

  • Qiwei He

    (Educational Testing Service)

  • Vincent Ulitzsch

    (Technische Universität Berlin)

  • Hendrik Molter

    (Technische Universität Berlin)

  • André Nichterlein

    (Technische Universität Berlin)

  • Rolf Niedermeier

    (Technische Universität Berlin)

  • Steffi Pohl

    (Freie Universität Berlin)

Abstract

Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees’ behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees’ behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.

Suggested Citation

  • Esther Ulitzsch & Qiwei He & Vincent Ulitzsch & Hendrik Molter & André Nichterlein & Rolf Niedermeier & Steffi Pohl, 2021. "Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 190-214, March.
  • Handle: RePEc:spr:psycho:v:86:y:2021:i:1:d:10.1007_s11336-020-09743-0
    DOI: 10.1007/s11336-020-09743-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-020-09743-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11336-020-09743-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Matthias von Davier & Lale Khorramdel & Qiwei He & Hyo Jeong Shin & Haiwen Chen, 2019. "Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 671-705, December.
    2. Qiwei He & Francesca Borgonovi & Marco Paccagnella, 2019. "Using process data to understand adults’ problem-solving behaviour in the Programme for the International Assessment of Adult Competencies (PIAAC): Identifying generalised patterns across multiple tas," OECD Education Working Papers 205, OECD Publishing.
    3. Trivitt, Julie & Cheng, Albert, 2016. "When you say nothing at all: The predictive power of student effort on surveysAuthor-Name: Hitt, Collin," Economics of Education Review, Elsevier, vol. 52(C), pages 105-119.
    4. Zamarro, Gema & Cheng, Albert & Shakeel, M. Danish & Hitt, Collin, 2018. "Comparing and validating measures of non-cognitive traits: Performance task measures and self-reports from a nationally representative internet panel," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 72(C), pages 51-60.
    5. Chen, Yunxiao & Li, Xiaoou & Liu, Jingchen & Ying, Zhiliang, 2019. "Statistical analysis of complex problem-solving process data: an event history analysis approach," LSE Research Online Documents on Economics 100871, London School of Economics and Political Science, LSE Library.
    6. Xueying Tang & Zhi Wang & Qiwei He & Jingchen Liu & Zhiliang Ying, 2020. "Latent Feature Extraction for Process Data via Multidimensional Scaling," Psychometrika, Springer;The Psychometric Society, vol. 85(2), pages 378-397, June.
    7. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    8. Oren E Livne & Lide Han & Gorka Alkorta-Aranburu & William Wentworth-Sheilds & Mark Abney & Carole Ober & Dan L Nicolae, 2015. "PRIMAL: Fast and Accurate Pedigree-based Imputation from Sequence Data in a Founder Population," PLOS Computational Biology, Public Library of Science, vol. 11(3), pages 1-14, March.
    9. Balart, Pau & Oosterveen, Matthijs & Webbink, Dinand, 2018. "Test scores, noncognitive skills and economic growth," Economics of Education Review, Elsevier, vol. 63(C), pages 134-153.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Esther Ulitzsch & Qiwei He & Steffi Pohl, 2022. "Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 3-35, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marianna Battaglia & Marisa Hidalgo-Hidalgo, 2020. "Non-Cognitive Skills and Remedial Education: Good News for Girls," Working Papers 20.10, Universidad Pablo de Olavide, Department of Economics.
    2. Marcenaro-Gutierrez, O.D. & Lopez-Agudo, L.A. & Henriques, C.O., 2021. "Are soft skills conditioned by conflicting factors? A multiobjective programming approach to explore the trade-offs," Economic Analysis and Policy, Elsevier, vol. 72(C), pages 18-40.
    3. Esther Ulitzsch & Qiwei He & Steffi Pohl, 2022. "Using Sequence Mining Techniques for Understanding Incorrect Behavioral Patterns on Interactive Tasks," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 3-35, February.
    4. Susu Zhang & Zhi Wang & Jitong Qi & Jingchen Liu & Zhiliang Ying, 2023. "Accurate Assessment via Process Data," Psychometrika, Springer;The Psychometric Society, vol. 88(1), pages 76-97, March.
    5. Yunxiao Chen, 2020. "A Continuous-Time Dynamic Choice Measurement Model for Problem-Solving Process Data," Psychometrika, Springer;The Psychometric Society, vol. 85(4), pages 1052-1075, December.
    6. Zhang, Yi & He, Jia, 2023. "Measuring non-cognitive skills exploiting log-files on online behaviour," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 103(C).
    7. Xueying Tang & Susu Zhang & Zhi Wang & Jingchen Liu & Zhiliang Ying, 2021. "ProcData: An R Package for Process Data Analysis," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 1058-1083, December.
    8. Cheng, Albert & Zamarro, Gema, 2018. "Measuring teacher non-cognitive skills and its impact on students: Insight from the Measures of Effective Teaching Longitudinal Database," Economics of Education Review, Elsevier, vol. 64(C), pages 251-260.
    9. Gema Zamarro & Malachi Nichols & Angela L Duckworth & Sidney K D’Mello, 2020. "Validation of survey effort measures of grit and self-control in a sample of high school students," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-16, July.
    10. Dorner, Zack & Lancsar, Emily, 2023. "Don’t pay the highly motivated too much," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 103(C).
    11. Fabian T C Schmidt & Clemens M Lechner & Daniel Danner, 2020. "New wine in an old bottle? A facet-level perspective on the added value of Grit over BFI–2 Conscientiousness," PLOS ONE, Public Library of Science, vol. 15(2), pages 1-25, February.
    12. Gabriel Heller-Sahlgren & Henrik Jordahl, 2024. "Test scores and economic growth: update and extension," Applied Economics Letters, Taylor & Francis Journals, vol. 31(11), pages 1024-1027, June.
    13. Hanushek, Eric A. & Kinne, Lavinia & Lergetporer, Philipp & Woessmann, Ludger, 2020. "Culture and Student Achievement: The Intertwined Roles of Patience and Risk-Taking," Rationality and Competition Discussion Paper Series 249, CRC TRR 190 Rationality and Competition.
    14. Annika B. Bergbauer & Eric A. Hanushek & Ludger Woessmann, 2024. "Testing," Journal of Human Resources, University of Wisconsin Press, vol. 59(2), pages 349-388.
    15. Anna Bussu & Dimitri Paolini & Manuela Pulina & Giuseppe Zanzurino, 2023. "From Choice to Performance in Secondary Schools: Evidence from a Disadvantaged Setting in Italy," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 9(2), pages 529-555, July.
    16. Rossi,Federico, 2018. "Human Capital and Macro-Economic Development : A Review of the Evidence," Policy Research Working Paper Series 8650, The World Bank.
    17. Giorgio Brunello & Angela Crema & Lorenzo Rocco, 2021. "Some Unpleasant Consequences of Testing at Length," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 83(4), pages 1002-1023, August.
    18. Gregor Gonza & Anže Burger, 2017. "Subjective Well-Being During the 2008 Economic Crisis: Identification of Mediating and Moderating Factors," Journal of Happiness Studies, Springer, vol. 18(6), pages 1763-1797, December.
    19. Anna Thum-Thysen & Rossella Cravetto & Jan Varchola, 2021. "Investing in People’s Competences A Cornerstone for Growth and Wellbeing in the EU," European Economy - Discussion Papers 139, Directorate General Economic and Financial Affairs (DG ECFIN), European Commission.
    20. Mark Reppell & John Novembre, 2018. "Using pseudoalignment and base quality to accurately quantify microbial community composition," PLOS Computational Biology, Public Library of Science, vol. 14(4), pages 1-23, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:86:y:2021:i:1:d:10.1007_s11336-020-09743-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.