IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0233977.html
   My bibliography  Save this article

Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application

Author

Listed:
  • Dirk Tempelaar
  • Bart Rienties
  • Quan Nguyen

Abstract

For decades, self-report measures based on questionnaires have been widely used in educational research to study implicit and complex constructs such as motivation, emotion, cognitive and metacognitive learning strategies. However, the existence of potential biases in such self-report instruments might cast doubts on the validity of the measured constructs. The emergence of trace data from digital learning environments has sparked a controversial debate on how we measure learning. On the one hand, trace data might be perceived as “objective” measures that are independent of any biases. On the other hand, there is mixed evidence of how trace data are compatible with existing learning constructs, which have traditionally been measured with self-reports. This study investigates the strengths and weaknesses of different types of data when designing predictive models of academic performance based on computer-generated trace data and survey data. We investigate two types of bias in self-report surveys: response styles (i.e., a tendency to use the rating scale in a certain systematic way that is unrelated to the content of the items) and overconfidence (i.e., the differences in predicted performance based on surveys’ responses and a prior knowledge test). We found that the response style bias accounts for a modest to a substantial amount of variation in the outcomes of the several self-report instruments, as well as in the course performance data. It is only the trace data, notably that of process type, that stand out in being independent of these response style patterns. The effect of overconfidence bias is limited. Given that empirical models in education typically aim to explain the outcomes of learning processes or the relationships between antecedents of these learning outcomes, our analyses suggest that the bias present in surveys adds predictive power in the explanation of performance data and other questionnaire data.

Suggested Citation

  • Dirk Tempelaar & Bart Rienties & Quan Nguyen, 2020. "Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-29, June.
  • Handle: RePEc:plo:pone00:0233977
    DOI: 10.1371/journal.pone.0233977
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0233977
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0233977&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0233977?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Weijters, Bert & Cabooter, Elke & Schillewaert, Niels, 2010. "The effect of rating scale format on response styles: The number of response categories and response category labels," International Journal of Research in Marketing, Elsevier, vol. 27(3), pages 236-247.
    2. Liesje Coertjens & Vincent Donche & Sven De Maeyer & Gert Vanthournout & Peter Van Petegem, 2013. "Modeling Change in Learning Strategies throughout Higher Education: A Multi-Indicator Latent Growth Perspective," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-12, July.
    3. B. Weijters & M. Geuens & N. Schillewaert, 2008. "The Stability of Individual Response Styles," Working Papers of Faculty of Economics and Business Administration, Ghent University, Belgium 08/547, Ghent University, Faculty of Economics and Business Administration.
    4. Pengsheng Ni & Molly Marino & Emily Dore & Lily Sonis & Colleen M Ryan & Jeffrey C Schneider & Alan M Jette & Lewis E Kazis, 2019. "Extreme response style bias in burn survivors," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-13, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jessica Catalano & Francesco Giffoni & Paolo Castelnovo, 2021. "The impact of space procurement on suppliers: Evidence from Italy," Working Papers 202102, CSIL Centre for Industrial Studies.
    2. Cha, Min-kyeong & Struthers, Cory L. & Brown, Marilyn A. & Kale, Snehal & Chapman, Oliver, 2024. "Toward residential decarbonization: Analyzing social-psychological drivers of household co-adoption of rooftop solar, electric vehicles, and efficient HVAC systems in Georgia, U.S," Renewable Energy, Elsevier, vol. 226(C).
    3. Kathleen M. Nokes & Dudu G. Sokhela & Penelope M. Orton & William Ellery Samuels & J. Craig Phillips & Kimberly Adams Tufts & Joseph D. Perazzo & Puangtip Chaiphibalsarisdi & Carmen Portillo & Rebecca, 2024. "Exploring the Interrelationships Between Physical Function, Functional Exercise Capacity, and Exercise Self-Efficacy in Persons Living with HIV," Clinical Nursing Research, , vol. 33(2-3), pages 165-175, March.
    4. Paolo Castelnovo & Gelsomina Catalano & Francesco Giffoni & Matteo Landoni, 2024. "The outcomes of public procurements: an empirical analysis of the Italian space industry," The Journal of Technology Transfer, Springer, vol. 49(1), pages 367-399, February.
    5. Sara Casagrande & Bruno Dallago, 2022. "To Be, or Not to Be: The Role of Self-Perception in European Countries’ Performance Assessment," Sustainability, MDPI, vol. 14(20), pages 1-23, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eline Moens & Louis Lippens & Philippe Sterkens & Johannes Weytjens & Stijn Baert, 2022. "The COVID-19 crisis and telework: a research survey on experiences, expectations and hopes," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 23(4), pages 729-753, June.
    2. Stylos, Nikolaos & Vassiliadis, Chris A. & Bellou, Victoria & Andronikidis, Andreas, 2016. "Destination images, holistic images and personal normative beliefs: Predictors of intention to revisit a destination," Tourism Management, Elsevier, vol. 53(C), pages 40-60.
    3. Elke Cabooter & Bert Weijters & Alain Beuckelaer & Eldad Davidov, 2017. "Is extreme response style domain specific? Findings from two studies in four countries," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(6), pages 2605-2622, November.
    4. Baumgartner, Hans & Weijters, Bert, 2019. "Measurement in Marketing," Foundations and Trends(R) in Marketing, now publishers, vol. 12(4), pages 278–400-2, December.
    5. Sandra Cortés & Soledad Burgos & Héctor Adaros & Boris Lucero & Lesliam Quirós-Alcalá, 2021. "Environmental Health Risk Perception: Adaptation of a Population-Based Questionnaire from Latin America," IJERPH, MDPI, vol. 18(16), pages 1-13, August.
    6. Stylos, Nikolaos & Bellou, Victoria & Andronikidis, Andreas & Vassiliadis, Chris A., 2017. "Linking the dots among destination images, place attachment, and revisit intentions: A study among British and Russian tourists," Tourism Management, Elsevier, vol. 60(C), pages 15-29.
    7. Marcella Corduas & Alfonso Piscitelli, 2017. "Modeling university student satisfaction: the case of the humanities and social studies degree programs," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(2), pages 617-628, March.
    8. Werner Bönte & Sandro Lombardo & Diemo Urbig, 2016. "Economics meets Psychology:Experimental and self-reported Measures of Individual Competitiveness," Schumpeter Discussion Papers SDP16006, Universitätsbibliothek Wuppertal, University Library.
    9. Paola Annoni & Nicholas Charron, 2019. "Measurement Assessment in Cross-Country Comparative Analysis: Rasch Modelling on a Measure of Institutional Quality," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 31-60, January.
    10. Gerhard Tutz & Moritz Berger, 2016. "Response Styles in Rating Scales," Journal of Educational and Behavioral Statistics, , vol. 41(3), pages 239-268, June.
    11. Tahrir Jaber & Elin M. Oftedal, 2020. "Legitimacy for Sustainability: A Case of A Strategy Change for An Oil and Gas Company," Sustainability, MDPI, vol. 12(2), pages 1-19, January.
    12. Loosveldt Geert & Beullens Koen, 2017. "Interviewer Effects on Non-Differentiation and Straightlining in the European Social Survey," Journal of Official Statistics, Sciendo, vol. 33(2), pages 409-426, June.
    13. Davide Giacomini & Anna Simonetto, 2020. "How Mayors Perceive the Influence of Social Media on the Policy Cycle," Public Organization Review, Springer, vol. 20(4), pages 735-752, December.
    14. Isabell Kuczynski & Martin Mädler & Yacine Taibi & Jessica Lang, 2020. "The Assessment of Psychosocial Work Conditions and Their Relationship to Well-Being: A Multi-Study Report," IJERPH, MDPI, vol. 17(5), pages 1-23, March.
    15. Cleemput, Irina & Devriese, Stephan & Kohn, Laurence & Westhovens, René, 2018. "A multi-criteria decision approach for ranking unmet needs in healthcare," Health Policy, Elsevier, vol. 122(8), pages 878-884.
    16. Belton, Ian & MacDonald, Alice & Wright, George & Hamlin, Iain, 2019. "Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process," Technological Forecasting and Social Change, Elsevier, vol. 147(C), pages 72-82.
    17. Exelmans, Liese & Van den Bulck, Jan, 2016. "Bedtime mobile phone use and sleep in adults," Social Science & Medicine, Elsevier, vol. 148(C), pages 93-101.
    18. Piotr Cichocki & Piotr Jabkowski, 2023. "Response scale overstretch: linear stretching of response scales does not ensure cross-project equivalence in harmonised data," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3729-3745, August.
    19. Cabooter, Elke & Weijters, Bert & Geuens, Maggie & Vermeir, Iris, 2016. "Scale format effects on response option interpretation and use," Journal of Business Research, Elsevier, vol. 69(7), pages 2574-2584.
    20. Mike Peters & Chung-Shing Chan & Anita Legerer, 2018. "Local Perception of Impact-Attitudes-Actions towards Tourism Development in the Urlaubsregion Murtal in Austria," Sustainability, MDPI, vol. 10(7), pages 1-28, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0233977. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.