IDEAS home Printed from https://ideas.repec.org/a/spr/reihed/v58y2017i8d10.1007_s11162-017-9452-4.html
   My bibliography  Save this article

The Causal Effect of Survey Mode on Students’ Evaluations of Teaching: Empirical Evidence from Three Field Experiments

Author

Listed:
  • Edgar Treischl

    (University of Mannheim)

  • Tobias Wolbring

    (University of Mannheim)

Abstract

In recent years many universities switched from paper- to online-based student evaluation of teaching (SET) without knowing the consequences for data quality. Based on a series of three consecutive field experiments—a split-half design, twin courses, and pre–post-measurements—this paper examines the effects of survey mode on SET. First, all three studies reveal marked differences in non-response between online- and paper-based SET and systematic, but small differences in the overall course ratings. On average, online SET reveal a slightly less optimistic picture of teaching quality in students’ perception. Similarly, a web survey mode does not impair the reliability of student ratings. Second, we highlight the importance of taking selection and class absenteeism into account when studying survey mode effects and also show that it is necessary and informative to survey the subgroup of no-shows when evaluating teaching. Third, we empirically demonstrate the need to account for contextual setting of the survey (in class vs. after class) and the specific type of the online survey mode (TAN vs. email). Previous research either confounded contextual setting with variation in survey mode or generalized results for a specific online mode to web surveys in general. Our findings suggest that higher response rates in email surveys can be achieved if students are given the opportunity and time to evaluate directly in class.

Suggested Citation

  • Edgar Treischl & Tobias Wolbring, 2017. "The Causal Effect of Survey Mode on Students’ Evaluations of Teaching: Empirical Evidence from Three Field Experiments," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(8), pages 904-921, December.
    • Handle: RePEc:spr:reihed:v:58:y:2017:i:8:d:10.1007_s11162-017-9452-4
    DOI: 10.1007/s11162-017-9452-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11162-017-9452-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11162-017-9452-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rosemary J. Avery & W. Keith Bryant & Alan Mathios & Hyojin Kang & Duncan Bell, 2006. "Electronic Course Evaluations: Does an Online Delivery System Influence Student Evaluations?," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(1), pages 21-37, January.
    2. David Romer, 1993. "Do Students Go to Class? Should They?," Journal of Economic Perspectives, American Economic Association, vol. 7(3), pages 167-174, Summer.
    3. Dolton, Peter & Marcenaro, Oscar D. & Navarro, Lucia, 2003. "The effective use of student time: a stochastic frontier production function case study," Economics of Education Review, Elsevier, vol. 22(6), pages 547-560, December.
    4. Samer Kherfi, 2011. "Whose Opinion Is It Anyway? Determinants of Participation in Student Evaluation of Teaching," The Journal of Economic Education, Taylor & Francis Journals, vol. 42(1), pages 19-30, January.
    5. Martine Selm & Nicholas Jankowski, 2006. "Conducting Online Surveys," Quality & Quantity: International Journal of Methodology, Springer, vol. 40(3), pages 435-456, June.
    6. Nowell, Clifford & Gale, Lewis R. & Kerkvliet, Joe, 2014. "Non-response bias in student evaluations of teaching," International Review of Economics Education, Elsevier, vol. 17(C), pages 30-38.
    7. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, October.
    8. Schmidt, Robert M, 1983. "Who Maximizes What? A Study in Student Time Allocation," American Economic Review, American Economic Association, vol. 73(2), pages 23-28, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:zbw:rwirep:0235 is not listed on IDEAS
    2. Dobkin, Carlos & Gil, Ricard & Marion, Justin, 2010. "Skipping class in college and exam performance: Evidence from a regression discontinuity classroom experiment," Economics of Education Review, Elsevier, vol. 29(4), pages 566-575, August.
    3. Massimiliano Bratti & Stefano Staffolani, 2013. "Student Time Allocation and Educational Production Functions," Annals of Economics and Statistics, GENES, issue 111-112, pages 103-140.
    4. Tsui-Fang Lin & Jennjou Chen, 2006. "Cumulative class attendance and exam performance," Applied Economics Letters, Taylor & Francis Journals, vol. 13(14), pages 937-942.
    5. Barbara S. Grave, 2010. "The Effect of Student Time Allocation on Academic Achievement," Ruhr Economic Papers 0235, Rheinisch-Westfälisches Institut für Wirtschaftsforschung, Ruhr-Universität Bochum, Universität Dortmund, Universität Duisburg-Essen.
    6. Barbara Grave, 2011. "The effect of student time allocation on academic achievement," Education Economics, Taylor & Francis Journals, vol. 19(3), pages 291-310.
    7. Mushay A. Ogundipe & Olufemi A. Falade, 2014. "Student-Teacher Perception of Time Allocation and Academic Achievement in Tai Solarin University of Education," International Journal of Learning and Development, Macrothink Institute, vol. 4(1), pages 65-70, March.
    8. Meng,Christoph & Heijke,Hans, 2005. "Student time allocation, the learning environment and the acquisition of competencies," ROA Research Memorandum 001, Maastricht University, Research Centre for Education and the Labour Market (ROA).
    9. Delaney, Liam & Harmon, Colm & Ryan, Martin, 2013. "The role of noncognitive traits in undergraduate study behaviours," Economics of Education Review, Elsevier, vol. 32(C), pages 181-195.
    10. Neckermann, Susanne & Turmunkh, Uyanga & van Dolder, Dennie & Wang, Tong V., 2022. "Nudging student participation in online evaluations of teaching: Evidence from a field experiment," European Economic Review, Elsevier, vol. 141(C).
    11. Meng, C.M. & Heijke, J.A.M., 2005. "Student time allocation, the learning environment and the acquisition of competencies," ROA Research Memorandum 1E, Maastricht University, Research Centre for Education and the Labour Market (ROA).
    12. Delavande, Adeline & Del Bono, Emilia & Holford, Angus, 2022. "Academic and non-academic investments at university: The role of expectations, preferences and constraints," Journal of Econometrics, Elsevier, vol. 231(1), pages 74-97.
    13. Göke, Michael, 2008. "Homo oeconomicus im Hörsaal: Die Rationalität studentischer Nebengespräche in Lehrveranstaltungen," Arbeitspapiere der FOM 9, FOM Hochschule für Oekonomie & Management.
    14. Afrogha, Olufolakemi & Oladele, Femi & Wright, Olatunde, 2020. "Influencing Factors On Academic Performance In Bowen University: The Perceptions Of Accounting Undergraduate Students," International Journal of Contemporary Accounting Issues-IJCAI (formerly International Journal of Accounting & Finance IJAF), The Institute of Chartered Accountants of Nigeria (ICAN), vol. 9(2), pages 120-140, September.
    15. Arulampalam, Wiji & Naylor, Robin A. & Smith, Jeremy, 2012. "Am I missing something? The effects of absence from class on student performance," Economics of Education Review, Elsevier, vol. 31(4), pages 363-375.
    16. Mause Karsten, 2008. "Ist Bildung eine Ware? Ein Klärungsversuch / Is Education a Market Good? An Attempt to Clarify," ORDO. Jahrbuch für die Ordnung von Wirtschaft und Gesellschaft, De Gruyter, vol. 59(1), pages 363-380, January.
    17. Cornéa van Walbeek, 2004. "Does Lecture Attendance Matter? Some Observations From A First‐Year Economics Course At The University Of Cape Town," South African Journal of Economics, Economic Society of South Africa, vol. 72(4), pages 861-883, September.
    18. Aurora A.C. Teixeira, 2013. "The impact of class absenteeism on undergraduates’ academic performance: evidence from an elite Economics school in Portugal," FEP Working Papers 503, Universidade do Porto, Faculdade de Economia do Porto.
    19. Tin-chun Lin, 2010. "Does a student's preference for a teacher's instructional style matter? An analysis of an economic approach," Economics Bulletin, AccessEcon, vol. 30(2), pages 1320-1332.
    20. Maarten Goos & Anna Salomons, 2017. "Measuring teaching quality in higher education: assessing selection bias in course evaluations," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(4), pages 341-364, June.
    21. Luca Stanca, 2006. "The Effects of Attendance on Academic Performance: Panel Data Evidence for Introductory Microeconomics," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(3), pages 251-266, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:reihed:v:58:y:2017:i:8:d:10.1007_s11162-017-9452-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.