IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v68y2018icp81-89.html
   My bibliography  Save this article

A counterfactual impact evaluation of a bilingual program on students’ grade point average at a spanish university

Author

Listed:
  • Arco-Tirado, J.L.
  • Fernández-Martín, F.
  • Ramos-García, A.M.
  • Littvay, L.
  • Villoria, J.
  • Naranjo, J.A.

Abstract

This observational study intends to estimate the causal effects of an English as a Medium of Instruction (EMI) program (as predictor) on students Grade Point Average (GPA) (as outcome) at a particular University in Spain by using a Counterfactual Impact Evaluation (CIE). The need to address the crucial question of causal inferences in EMI programs to produce credible evidences of successful interventions contrasts, however, with the absence of experimental or quasi-experimental research and evaluation designs in the field. CIE approach is emerging as a methodologically viable solution to bridge that gap. The program evaluated here consisted in delivering an EMI program in a Primary Education Teacher Training Degree group. After achieving balance on the observed covariates and recreating a situation that would have been expected in a randomized experiment, three matching approaches such as genetic matching, nearest neighbor matching and Coarsened Exact Matching were used to analyze observational data from a total of 1288 undergraduate students, including both treatment and control group. Results show unfavorable effects of the bilingual group treatment condition. Potential interpretations and recommendations are provided in order to strengthen future causal evidences of bilingual education programs’ effectiveness in Higher Education.

Suggested Citation

  • Arco-Tirado, J.L. & Fernández-Martín, F. & Ramos-García, A.M. & Littvay, L. & Villoria, J. & Naranjo, J.A., 2018. "A counterfactual impact evaluation of a bilingual program on students’ grade point average at a spanish university," Evaluation and Program Planning, Elsevier, vol. 68(C), pages 81-89.
    • Handle: RePEc:eee:epplan:v:68:y:2018:i:c:p:81-89
    DOI: 10.1016/j.evalprogplan.2018.02.013
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S014971891730335X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2018.02.013?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Card, David & Ibarrarán, Pablo & Villa, Juan Miguel, 2011. "Building in an Evaluation Component for Active Labor Market Programs: A Practitioner's Guide," IZA Discussion Papers 6085, Institute of Labor Economics (IZA).
    2. Vincent Tinto, 1997. "Classrooms as Communities," The Journal of Higher Education, Taylor & Francis Journals, vol. 68(6), pages 599-623, November.
    3. Brindusa Anghel & Antonio Cabrales & Jesus M. Carro, 2016. "Evaluating A Bilingual Education Program In Spain: The Impact Beyond Foreign Language Learning," Economic Inquiry, Western Economic Association International, vol. 54(2), pages 1202-1223, April.
    4. repec:idb:brikps:58518 is not listed on IDEAS
    5. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    6. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    7. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lucia Švábová & Katarína Kramárová & Marek Ďurica, 2021. "Evaluation of the Effects of the Graduate Practice in Slovakia: Comparison of Results of Counterfactual Methods," Central European Business Review, Prague University of Economics and Business, vol. 2021(4), pages 1-31.
    2. Teng Lin & Jun Lei, 2021. "English-Medium Instruction and Content Learning in Higher Education: Effects of Medium of Instruction, English Proficiency, and Academic Ability," SAGE Open, , vol. 11(4), pages 21582440211, November.
    3. Pelucha, Martin & Kveton, Viktor & Potluka, Oto, 2019. "Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 146-155.
    4. Lucia Svabova & Eva Nahalkova Tesarova & Marek Durica & Lenka Strakova, 2021. "Evaluation of the impacts of the COVID-19 pandemic on the development of the unemployment rate in Slovakia: counterfactual before-after comparison," Equilibrium. Quarterly Journal of Economics and Economic Policy, Institute of Economic Research, vol. 16(2), pages 261-284, June.
    5. del Campo, Cristina & Urquía-Grande, Elena & Pascual-Ezama, David, 2023. "Internationalizing the business school: A comparative analysis of English-medium and Spanish-medium instruction impact on student performance," Evaluation and Program Planning, Elsevier, vol. 98(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gerhard Krug, 2017. "Augmenting propensity score equations to avoid misspecification bias – Evidence from a Monte Carlo simulation [Erweiterung der Propensity Score Gleichung zur Vermeidung von Fehlspezifikationen? Ein," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 11(3), pages 205-231, December.
    2. Dudel Christian & Garbuszus Jan Marvin & Ott Notburga & Werding Martin, 2017. "Matching as Non-Parametric Preprocessing for the Estimation of Equivalence Scales," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 237(2), pages 115-141, April.
    3. Gary King & Christopher Lucas & Richard A. Nielsen, 2017. "The Balance‐Sample Size Frontier in Matching Methods for Causal Inference," American Journal of Political Science, John Wiley & Sons, vol. 61(2), pages 473-489, April.
    4. Jones, A.M.; & Pastore, C.; & Rice, N.;, 2018. "Tracking pupils into adulthood: selective schools and long-term well-being in the 1958 British cohort," Health, Econometrics and Data Group (HEDG) Working Papers 18/32, HEDG, c/o Department of Economics, University of York.
    5. Adeola Oyenubi & Martin Wittenberg, 2021. "Does the choice of balance-measure matter under genetic matching?," Empirical Economics, Springer, vol. 61(1), pages 489-502, July.
    6. Federico Biagi & Daniele Bondonio & Alberto Martini, 2015. "Counterfactual Impact Evaluation of Enterprise Support Programmes. Evidence from a Decade of Subsidies to Italian Firm," ERSA conference papers ersa15p1619, European Regional Science Association.
    7. Jian Jiu Chen & Sai Yin Ho & Wing Man Au & Man Ping Wang & Tai Hing Lam, 2015. "Family Smoking, Exposure to Secondhand Smoke at Home and Family Unhappiness in Children," IJERPH, MDPI, vol. 12(11), pages 1-14, November.
    8. Kube, Roland & von Graevenitz, Kathrine & Löschel, Andreas & Massier, Philipp, 2019. "Do voluntary environmental programs reduce emissions? EMAS in the German manufacturing sector," Energy Economics, Elsevier, vol. 84(S1).
    9. Blackman, Allen, 2013. "Evaluating forest conservation policies in developing countries using remote sensing data: An introduction and practical guide," Forest Policy and Economics, Elsevier, vol. 34(C), pages 1-16.
    10. K. Poehlmann & R. Helm & O. Mauroner & J. Auburger, 2021. "Corporate spin-offs’ success factors: management lessons from a comparative empirical analysis with research-based spin-offs," Review of Managerial Science, Springer, vol. 15(6), pages 1767-1796, August.
    11. Matilde Cappelletti & Leonardo M. Giuffrida, 2024. "Targeted Bidders in Government Tenders," CESifo Working Paper Series 11142, CESifo.
    12. Uz Akdogan, Idil, 2020. "Understanding the dynamics of foreign reserve management: The central bank intervention policy and the exchange rate fundamentals," International Economics, Elsevier, vol. 161(C), pages 41-55.
    13. Müller, Tobias & Schmid, Christian & Gerfin, Michael, 2023. "Rents for Pills: Financial incentives and physician behavior," Journal of Health Economics, Elsevier, vol. 87(C).
    14. Chen Liang & Yili Hong & Pei-Yu Chen & Benjamin B. M. Shao, 2022. "The Screening Role of Design Parameters for Service Procurement Auctions in Online Service Outsourcing Platforms," Information Systems Research, INFORMS, vol. 33(4), pages 1324-1343, December.
    15. Bensch, Gunther & Peters, Jörg & Sievert, Maximiliane, 2012. "Fear of the Dark? – How Access to Electric Lighting Affects Security Attitudes and Nighttime Activities in Rural Senegal," Ruhr Economic Papers 369, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    16. Deni Mazrekaj & Mirjam M. Fischer & Henny M. W. Bos, 2022. "Behavioral Outcomes of Children with Same-Sex Parents in The Netherlands," IJERPH, MDPI, vol. 19(10), pages 1-12, May.
    17. Robert L. Obenchain & S. Stanley Young, 2017. "Local Control Strategy: Simple Analyses of Air Pollution Data Can Reveal Heterogeneity in Longevity Outcomes," Risk Analysis, John Wiley & Sons, vol. 37(9), pages 1742-1753, September.
    18. Solomon Asfaw & Silvio Daidone & Benjamin Davis & Josh Dewbre & Alessandro Romeo & Paul Winters & Katia Covarrubias & Habiba Djebbari, 2012. "Analytical Framework for Evaluating the Productive Impact of Cash Transfer Programmes on Household Behaviour – Methodological Guidelines for the From Protection to Production Project," Working Papers 101, International Policy Centre for Inclusive Growth.
    19. Augusto Cerqua & Guido Pellegrini, 2013. "Beyond the SUTVA: how industrial policy evaluations change when we allow for interaction among firms," ERSA conference papers ersa13p340, European Regional Science Association.
    20. Petrov, Ivan & Ryan, Lisa, 2021. "The landlord-tenant problem and energy efficiency in the residential rental market," Energy Policy, Elsevier, vol. 157(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:68:y:2018:i:c:p:81-89. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.