IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v101y2014i1d10.1007_s11192-014-1397-1.html
   My bibliography  Save this article

A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program

Author

Listed:
  • Fernando A. B. Colugnati

    (University of Campinas
    Federal University of Juiz de Fora)

  • Sergio Firpo

    (São Paulo School of Economics/FGV)

  • Paula F. Drummond Castro

    (University of Campinas)

  • Juan E. Sepulveda

    (Institute of Economics/UNICAMP)

  • Sergio L. M. Salles-Filho

    (University of Campinas)

Abstract

Evaluation has become a regular practice in the management of science, technology and innovation (ST&I) programs. Several methods have been developed to identify the results and impacts of programs of this kind. Most evaluations that adopt such an approach conclude that the interventions concerned, in this case ST&I programs, had a positive impact compared with the baseline, but do not control for any effects that might have improved the indicators even in the absence of intervention, such as improvements in the socio-economic context. The quasi-experimental approach therefore arises as an appropriate way to identify the real contributions of a given intervention. This paper describes and discusses the utilization of propensity score (PS) in quasi-experiments as a methodology to evaluate the impact on scientific production of research programs, presenting a case study of the BIOTA Program run by FAPESP, the State of São Paulo Research Foundation (Brazil). Fundamentals of quasi-experiments and causal inference are presented, stressing the need to control for biases due to lack of randomization, also a brief introduction to the PS estimation and weighting technique used to correct for observed bias. The application of the PS methodology is compared to the traditional multivariate analysis usually employed.

Suggested Citation

  • Fernando A. B. Colugnati & Sergio Firpo & Paula F. Drummond Castro & Juan E. Sepulveda & Sergio L. M. Salles-Filho, 2014. "A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 85-107, October.
  • Handle: RePEc:spr:scient:v:101:y:2014:i:1:d:10.1007_s11192-014-1397-1
    DOI: 10.1007/s11192-014-1397-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-014-1397-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-014-1397-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2009. "Dealing with limited overlap in estimation of average treatment effects," Biometrika, Biometrika Trust, vol. 96(1), pages 187-199.
    2. Pierre Azoulay & Joshua S. Graff Zivin & Gustavo Manso, 2011. "Incentives and creativity: evidence from the academic life sciences," RAND Journal of Economics, RAND Corporation, vol. 42(3), pages 527-554, September.
    3. Lori Lynch & Wayne Gray & Jacqueline Geoghegan, 2007. "Are Farmland Preservation Program Easement Restrictions Capitalized into Farmland Prices? What Can a Propensity Score Matching Analysis Tell Us?," Review of Agricultural Economics, Agricultural and Applied Economics Association, vol. 29(3), pages 502-509.
    4. Stefan Hornbostel & Susan Böhmer & Bernd Klingsporn & Jörg Neufeld & Markus Ins, 2009. "Funding of young scientist and scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(1), pages 171-190, April.
    5. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    6. Ravallion, Martin, 2008. "Evaluating Anti-Poverty Programs," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 59, pages 3787-3846, Elsevier.
    7. Jalan, Jyotsna & Ravallion, Martin, 2003. "Estimating the Benefit Incidence of an Antipoverty Program by Propensity-Score Matching," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 19-30, January.
    8. Rüdiger Mutz & Hans-Dieter Daniel, 2012. "The generalized propensity score methodology for estimating unbiased journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 377-390, August.
    9. T. Paul Schultz & John A. Strauss (ed.), 2008. "Handbook of Development Economics," Handbook of Development Economics, Elsevier, edition 1, volume 4, number 5.
    10. Jacob, Brian A. & Lefgren, Lars, 2011. "The impact of NIH postdoctoral training grants on scientific productivity," Research Policy, Elsevier, vol. 40(6), pages 864-874, July.
    11. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    12. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    13. Susan Böhmer & Markus von Ins, 2009. "Different — not just by label: research-oriented academic careers in Germany," Research Evaluation, Oxford University Press, vol. 18(3), pages 177-184, September.
    14. Frondel, Manuel & Schmidt, Christoph M., 2005. "Evaluating environmental programs: The perspective of modern evaluation research," Ecological Economics, Elsevier, vol. 55(4), pages 515-526, December.
    15. Hoshino, Takahiro, 2008. "A Bayesian propensity score adjustment for latent variable modeling and MCMC algorithm," Computational Statistics & Data Analysis, Elsevier, vol. 52(3), pages 1413-1429, January.
    16. Sergio L M Salles-Filho & A Flavio Avila & Juan Ernesto O S Alonso & Fernando A B Colugnati, 2010. "Multidimensional assessment of technology and innovation programs: the impact evaluation of INCAGRO-Peru," Research Evaluation, Oxford University Press, vol. 19(5), pages 361-372, December.
    17. Roula Inglesi-Lotz & Anastassios Pouris, 2011. "Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 747-760, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Selcuk Besir Demir, 2018. "Pros and cons of the new financial support policy for Turkish researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2053-2068, September.
    2. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    3. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    4. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    5. Aboal, Diego & Tacsir, Ezequiel, 2016. "The impact of ex-ante subsidies to researchers on researcher's productivity: Evidence from a developing country," MERIT Working Papers 2016-019, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Adriana Bin & Sergio Salles-Filho & Luiza Maria Capanema & Fernando Antonio Basile Colugnati, 2015. "What difference does it make? Impact of peer-reviewed scholarships on scientific production," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1167-1188, February.
    2. Tohari, Achmad & Parsons, Christopher & Rammohan, Anu, 2019. "Targeting poverty under complementarities: Evidence from Indonesia's unified targeting system," Journal of Development Economics, Elsevier, vol. 140(C), pages 127-144.
    3. Volpe Martincus, Christian & Carballo, Jerónimo, 2010. "Beyond the average effects: The distributional impacts of export promotion programs in developing countries," Journal of Development Economics, Elsevier, vol. 92(2), pages 201-214, July.
    4. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    5. Chad D. Meyerhoefer & Muzhe Yang, 2011. "The Relationship between Food Assistance and Health: A Review of the Literature and Empirical Strategies for Identifying Program Effects," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 33(3), pages 304-344.
    6. Ferraro, Paul J. & Miranda, Juan José, 2014. "The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 344-365.
    7. Christian Volpe Martincus & Jerónimo Carballo & Pablo M. Garcia, 2012. "Public programmes to promote firms’ exports in developing countries: are there heterogeneous effects by size categories?," Applied Economics, Taylor & Francis Journals, vol. 44(4), pages 471-491, February.
    8. Krapf, Matthias & Ursprung, Heinrich W. & Zimmermann, Christian, 2017. "Parenthood and productivity of highly skilled labor: Evidence from the groves of academe," Journal of Economic Behavior & Organization, Elsevier, vol. 140(C), pages 147-175.
    9. Mazzutti, Caio Cícero Toledo Piza da Costa, 2016. "Three essays on the causal impacts of child labour laws in Brazil," Economics PhD Theses 0616, Department of Economics, University of Sussex Business School.
    10. Andrea ALBANESE & Bart COCKX, 2015. "Permanent Wage Cost Subsidies for Older Workers. An Effective Tool for Increasing Working Time and Postponing Early Retirement?," LIDAM Discussion Papers IRES 2015006, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    11. Xinwei Ma & Jingshen Wang, 2018. "Robust Inference Using Inverse Probability Weighting," Papers 1810.11397, arXiv.org, revised May 2019.
    12. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    13. Volpe Martincus, Christian & Carballo, Jerónimo, 2010. "Beyond the average effects: The distributional impacts of export promotion programs in developing countries," Journal of Development Economics, Elsevier, vol. 92(2), pages 201-214, July.
    14. Nicolas CARAYOL & Marianne LANOË, 2017. "The Impact of Project-Based Funding in Science: \r\nLessons from the ANR Experience," Cahiers du GREThA (2007-2019) 2017-04, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    15. Adriana Bin & Sergio Salles-Filho & Ana Carolina Spatti & Jesús Pascual Mena-Chalco & Fernando Antonio Basile Colugnati, 2022. "How much does a Ph.D. scholarship program impact an emerging economy research performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6935-6960, December.
    16. Callaway, Brantly & Li, Tong, 2023. "Policy evaluation during a pandemic," Journal of Econometrics, Elsevier, vol. 236(1).
    17. Tohari, Achmad & Parsons, Christopher & Rammohan, Anu, 2017. "Targeting Poverty under Complementarities: Evidence from Indonesia's Unified Targeting System," IZA Discussion Papers 10968, Institute of Labor Economics (IZA).
    18. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute of Labor Economics (IZA).
    19. van de Walle, Dominique & Mu, Ren, 2007. "Fungibility and the flypaper effect of project aid: Micro-evidence for Vietnam," Journal of Development Economics, Elsevier, vol. 84(2), pages 667-685, November.
    20. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:101:y:2014:i:1:d:10.1007_s11192-014-1397-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.