IDEAS home Printed from https://ideas.repec.org/a/pal/palcom/v4y2018i1d10.1057_s41599-018-0069-9.html
   My bibliography  Save this article

Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?

Author

Listed:
  • Margaret Dalziel

    (University of Waterloo)

Abstract

Based on the achievements of randomised controlled trials (RCTs) in medicine, and the need for effective government interventions in support of business, some have advocated for the use of RCTs in the evaluation of business support programmes (BSPs). Notwithstanding these recommendations, the use of RCTs in the evaluation of BSPs has been resisted by (almost) all. Policy makers and managers are correct in their reluctance to undertake RCT-based evaluations for four reasons. First, while RCTs require the random allocation of support, judicious programmes select firms on the basis of potential and amenability to support. Second, while RCTs require treatments that exhibit low variability, the most effective BSPs draw upon substantive knowledge to provide support that is customised. Third, BSPs aim to produce outliers—firms whose performance is exceptional. When outliers are present, very large samples will be required to produce reliable results. Finally, an RCT may not yield a meaningful contribution to knowledge. The strength of an RCT is its ability to estimate the magnitude of the treatment effect under controlled conditions. But where much depends on the nature of participants and circumstances, we seek evidence of what works, for whom, in which circumstances, and why.

Suggested Citation

  • Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
  • Handle: RePEc:pal:palcom:v:4:y:2018:i:1:d:10.1057_s41599-018-0069-9
    DOI: 10.1057/s41599-018-0069-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1057/s41599-018-0069-9
    File Function: Abstract
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1057/s41599-018-0069-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Raffaello Bronzini & Eleonora Iachini, 2014. "Are Incentives for R&D Effective? Evidence from a Regression Discontinuity Approach," American Economic Journal: Economic Policy, American Economic Association, vol. 6(4), pages 100-134, November.
    2. Robert W. Fairlie & Dean Karlan & Jonathan Zinman, 2015. "Behind the GATE Experiment: Evidence on Effects of and Rationales for Subsidized Entrepreneurship Training," American Economic Journal: Economic Policy, American Economic Association, vol. 7(2), pages 125-161, May.
    3. Howells, Jeremy, 2006. "Intermediation and the role of intermediaries in innovation," Research Policy, Elsevier, vol. 35(5), pages 715-728, June.
    4. Adam B. Jaffe & Manuel Trajtenberg & Rebecca Henderson, 1993. "Geographic Localization of Knowledge Spillovers as Evidenced by Patent Citations," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 108(3), pages 577-598.
    5. Feldman, Maryann P. & Kelley, Maryellen R., 2006. "The ex ante assessment of knowledge spillovers: Government R&D policy, economic incentives and private firm behavior," Research Policy, Elsevier, vol. 35(10), pages 1509-1521, December.
    6. Jaffe, Adam B. & Newell, Richard G. & Stavins, Robert N., 2005. "A tale of two market failures: Technology and environmental policy," Ecological Economics, Elsevier, vol. 54(2-3), pages 164-174, August.
    7. Fuchs, Erica R.H., 2010. "Rethinking the role of the state in technology development: DARPA and the case for embedded network governance," Research Policy, Elsevier, vol. 39(9), pages 1133-1147, November.
    8. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    9. Izushi, Hiro, 2003. "Impact of the length of relationships upon the use of research institutes by SMEs," Research Policy, Elsevier, vol. 32(5), pages 771-788, May.
    10. Warren Pearce & Sujatha Raman, 2014. "The new randomised controlled trials (RCT) movement in public policy: challenges of epistemic governance," Policy Sciences, Springer;Society of Policy Sciences, vol. 47(4), pages 387-402, December.
    11. Gustavo A. Crespi & Alessandro Maffioli & Pierre Mohnen & Gonzalo Vázquez, 2011. "Evaluating the Impact of Science, Technology and Innovation Programs: a Methodological Toolkit," SPD Working Papers 1104, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD).
    12. Sabrina T. Howell, 2017. "Financing Innovation: Evidence from R&D Grants," American Economic Review, American Economic Association, vol. 107(4), pages 1136-1164, April.
    13. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 123-150, January.
    14. Czarnitzki, Dirk & Hanel, Petr & Rosa, Julio Miguel, 2011. "Evaluating the impact of R&D tax credits on innovation: A microeconometric study on Canadian firms," Research Policy, Elsevier, vol. 40(2), pages 217-229, March.
    15. Gabriela Calderon & Jesse M. Cunha & Giacomo De Giorgi, 2020. "Business Literacy and Development: Evidence from a Randomized Controlled Trial in Rural Mexico," Economic Development and Cultural Change, University of Chicago Press, vol. 68(2), pages 507-540.
    16. Rajagopal, 2014. "Organizations and Innovation," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 3, pages 58-86, Palgrave Macmillan.
    17. repec:idb:brikps:62598 is not listed on IDEAS
    18. Hall, Bronwyn H. & Mairesse, Jacques & Mohnen, Pierre, 2010. "Measuring the Returns to R&D," Handbook of the Economics of Innovation, in: Bronwyn H. Hall & Nathan Rosenberg (ed.), Handbook of the Economics of Innovation, edition 1, volume 2, chapter 0, pages 1033-1082, Elsevier.
    19. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    20. Guido de Blasio & Davide Fantino & Guido Pellegrini, 2015. "Evaluating the impact of innovation incentives: evidence from an unexpected shortage of funds," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 24(6), pages 1285-1314.
    21. William Lazonick & Mariana Mazzucato, 2013. "The risk-reward nexus in the innovation-inequality relationship: who takes the risks? Who gets the rewards ?," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 22(4), pages 1093-1128, August.
    22. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    23. Yael V. Hochberg, 2016. "Accelerating Entrepreneurs and Ecosystems: The Seed Accelerator Model," Innovation Policy and the Economy, University of Chicago Press, vol. 16(1), pages 25-51.
    24. Mariana Mazzucato, 2015. "The Green Entrepreneurial State," SPRU Working Paper Series 2015-28, SPRU - Science Policy Research Unit, University of Sussex Business School.
    25. Victora, C.G. & Habicht, J.-P. & Bryce, J., 2004. "Evidence-Based Public Health: Moving Beyond Randomized Trials," American Journal of Public Health, American Public Health Association, vol. 94(3), pages 400-405.
    26. Abhijit Banerjee & Esther Duflo & Rachel Glennerster & Cynthia Kinnan, 2015. "The Miracle of Microfinance? Evidence from a Randomized Evaluation," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 22-53, January.
    27. Meuleman, Miguel & De Maeseneire, Wouter, 2012. "Do R&D subsidies affect SMEs’ access to external financing?," Research Policy, Elsevier, vol. 41(3), pages 580-591.
    28. Dirk Czarnitzki & Julie Delanote, 2013. "Young Innovative Companies: the new high-growth firms?," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 22(5), pages 1315-1340, October.
    29. Toole, Andrew A., 2012. "The impact of public basic research on industrial innovation: Evidence from the pharmaceutical industry," Research Policy, Elsevier, vol. 41(1), pages 1-12.
    30. Bonell, Chris & Fletcher, Adam & Morton, Matthew & Lorenc, Theo & Moore, Laurence, 2012. "Realist randomised controlled trials: A new approach to evaluating complex public health interventions," Social Science & Medicine, Elsevier, vol. 75(12), pages 2299-2306.
    31. Scott J. Wallsten, 2000. "The Effects of Government-Industry R&D Programs on Private R&D: The Case of the Small Business Innovation Research Program," RAND Journal of Economics, The RAND Corporation, vol. 31(1), pages 82-100, Spring.
    32. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    33. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    34. Martin Ravallion, 2015. "The World Bank: Why It Is Still Needed and Why It Still Disappoints - Working Paper 400," Working Papers 400, Center for Global Development.
    35. Angus Deaton & Nancy Cartwright, 2016. "Understanding and Misunderstanding Randomized Controlled Trials," Working Papers august_25.pdf, Princeton University, Woodrow Wilson School of Public and International Affairs, Research Program in Development Studies..
    36. Kevin Morrell, 2008. "The Narrative of ‘Evidence Based’ Management: A Polemic," Journal of Management Studies, Wiley Blackwell, vol. 45(3), pages 613-635, May.
    37. Dougherty, Deborah, 2016. "Taking Advantage of Emergence: Productively Innovating in Complex Innovation Systems," OUP Catalogue, Oxford University Press, number 9780198725299.
    38. Adam B. Jaffe, 2002. "Building Programme Evaluation into the Design of Public Research-Support Programmes," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 18(1), pages 22-34, Spring.
    39. David Greenberg & Mark Shroder & Matthew Onstott, 1999. "The Social Experiment Market," Journal of Economic Perspectives, American Economic Association, vol. 13(3), pages 157-172, Summer.
    40. Mark Freel, 2007. "Are Small Innovators Credit Rationed?," Small Business Economics, Springer, vol. 28(1), pages 23-35, January.
    41. Autio, Erkko & Rannikko, Heikki, 2016. "Retaining winners: Can policy boost high-growth entrepreneurship?," Research Policy, Elsevier, vol. 45(1), pages 42-55.
    42. Köhler, Christian & Larédo, Philippe & Rammer, Christian, 2012. "Fiscal incentives for business R&D: Compendium of evidence on the effectiveness of innovation policy intervention," ZEW Expertises, ZEW - Leibniz Centre for European Economic Research, number 111472.
    43. Gavin Murphy & Iulia Siedschlag & John McQuinn, 2017. "Employment protection and industry innovation," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 26(3), pages 379-398.
    44. Beise, Marian & Stahl, Harald, 1999. "Public research and industrial innovations in Germany," Research Policy, Elsevier, vol. 28(4), pages 397-422, April.
    45. Eom, Boo-Young & Lee, Keun, 2010. "Determinants of industry-academy linkages and, their impact on firm performance: The case of Korea as a latecomer in knowledge industrialization," Research Policy, Elsevier, vol. 39(5), pages 625-639, June.
    46. Pamela Palazzi, 2011. "Taxation and Innovation," OECD Taxation Working Papers 9, OECD Publishing.
    47. Autio, Erkko & Kanninen, Sami & Gustafsson, Robin, 2008. "First- and second-order additionality and learning outcomes in collaborative R&D programs," Research Policy, Elsevier, vol. 37(1), pages 59-76, February.
    48. Andrés Barge-Gil & Aurelia Modrego, 2011. "The impact of research and technology organizations on firm competitiveness. Measurement and determinants," The Journal of Technology Transfer, Springer, vol. 36(1), pages 61-83, February.
    49. Cumming, Douglas J. & Fischer, Eileen, 2012. "Publicly funded business advisory services and entrepreneurial outcomes," Research Policy, Elsevier, vol. 41(2), pages 467-481.
    50. Leighton Andrews, 2017. "How can we demonstrate the public value of evidence-based policy making when government ministers declare that the people ‘have had enough of experts’?," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-9, December.
    51. Bakhshi, Hasan & Edwards, John S. & Roper, Stephen & Scully, Judy & Shaw, Duncan & Morley, Lorraine & Rathbone, Nicola, 2015. "Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits," Research Policy, Elsevier, vol. 44(8), pages 1462-1472.
    52. Hottenrott, Hanna & Lopes-Bento, Cindy & Veugelers, Reinhilde, 2017. "Direct and cross scheme effects in a research and development," Research Policy, Elsevier, vol. 46(6), pages 1118-1132.
    53. Diana Epstein & Jacob Alex Klerman, 2012. "When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model," Evaluation Review, , vol. 36(5), pages 375-401, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Magnus Henrekson & Anders Kärnä & Tino Sanandaji, 2022. "Schumpeterian entrepreneurship: coveted by policymakers but impervious to top-down policymaking," Journal of Evolutionary Economics, Springer, vol. 32(3), pages 867-890, July.
    2. Fabrice Gilles & Yannick L'Horty & Ferhat Mihoubi, 2021. "The Effects of the Non-Financial Component of Business Accelerators," TEPP Working Paper 2021-06, TEPP.
    3. Priit Tinits & Carl F. Fey, 2022. "The Effects of Timing and Order of Government Support Mechanisms for SME Exports," Management International Review, Springer, vol. 62(2), pages 285-323, April.
    4. Albert Bravo-Biosca, 2020. "Experimental Innovation Policy," Innovation Policy and the Economy, University of Chicago Press, vol. 20(1), pages 191-232.
    5. Pietro Santoleri & Andrea Mina & Alberto Di Minin & Irene Martelli, 2024. "The Causal Effects of R&D Grants: Evidence from a Regression Discontinuity," The Review of Economics and Statistics, MIT Press, vol. 106(6), pages 1495-1510, November.
    6. Vladimír Baláž & Tomáš Jeck & Miroslav Balog, 2024. "Knowledge Transfers and Business Performance in Creative Networks," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(1), pages 4428-4454, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Caloffi, Annalisa & Colovic, Ana & Rizzoli, Valentina & Rossi, Federica, 2023. "Innovation intermediaries' types and functions: A computational analysis of the literature," Technological Forecasting and Social Change, Elsevier, vol. 189(C).
    2. Bianchi, Mattia & Murtinu, Samuele & Scalera, Vittoria G., 2019. "R&D Subsidies as Dual Signals in Technological Collaborations," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    3. Jason M. Rathje & Riitta Katila, 2021. "Enabling Technologies and the Role of Private Firms: A Machine Learning Matching Analysis," Strategy Science, INFORMS, vol. 6(1), pages 5-21, March.
    4. Pietro Santoleri & Andrea Mina & Alberto Di Minin & Irene Martelli, 2024. "The Causal Effects of R&D Grants: Evidence from a Regression Discontinuity," The Review of Economics and Statistics, MIT Press, vol. 106(6), pages 1495-1510, November.
    5. Enrico Vanino & Stephen Roper & Bettina Becker, 2020. "Knowledge to Money: Assessing the Business Performance Effects of Publicly Funded R&D Grants," ifo DICE Report, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 17(04), pages 20-24, January.
    6. Cameron Hepburn & Jacquelyn Pless & David Popp, 2018. "Policy Brief—Encouraging Innovation that Protects Environmental Systems: Five Policy Proposals," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 154-169.
    7. Aalto, Eero & Gustafsson, Robin, 2020. "Innovation Promotion Rationales and Impacts – A Review," ETLA Reports 99, The Research Institute of the Finnish Economy.
    8. Chiappini, Raphaël & Montmartin, Benjamin & Pommet, Sophie & Demaria, Samira, 2022. "Can direct innovation subsidies relax SMEs’ financial constraints?," Research Policy, Elsevier, vol. 51(5).
    9. Nadine Levratto & Aurelien Quignon, 2021. "Innovation Performance and the Signal Effect: Evidence from a European Program," Working Papers halshs-03466903, HAL.
    10. Chen, Jin & Heng, Cheng Suang & Tan, Bernard C.Y. & Lin, Zhijie, 2018. "The distinct signaling effects of R&D subsidy and non-R&D subsidy on IPO performance of IT entrepreneurial firms in China," Research Policy, Elsevier, vol. 47(1), pages 108-120.
    11. Guo, Di & Guo, Yan & Jiang, Kun, 2022. "Government R&D support and firms’ access to external financing: funding effects, certification effects, or both?," Technovation, Elsevier, vol. 115(C).
    12. Bettina Becker, 2013. "The Determinants of R&D Investment: A Survey of the Empirical Research," Discussion Paper Series 2013_09, Department of Economics, Loughborough University, revised Sep 2013.
    13. Wang, Yanbo & Li, Jizhen & Furman, Jeffrey L., 2017. "Firm performance and state innovation funding: Evidence from China’s innofund program," Research Policy, Elsevier, vol. 46(6), pages 1142-1161.
    14. Shuang Wang & Shukuan Zhao & Dong Shao & Hongyu Liu, 2020. "Impact of Government Subsidies on Manufacturing Innovation in China: The Moderating Role of Political Connections and Investor Attention," Sustainability, MDPI, vol. 12(18), pages 1-21, September.
    15. Andrea Bellucci & Luca Pennacchio & Alberto Zazzaro, 2019. "R&D Subsidies and Firms’ Debt Financing," CSEF Working Papers 527, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    16. Aurélien Quignon & Nadine Levratto, 2021. "Innovation Performance and the Signal Effect: Evidence from a European Program," EconomiX Working Papers 2021-34, University of Paris Nanterre, EconomiX.
    17. Casaburi, Gabriel & Crespi, Gustavo & De León, Ignacio L. & Figal Garone, Lucas & Grazzi, Matteo & Guaipatín, Carlos & Katz, Jorge & Llisterri, Juan J. & Maffioli, Alessandro & Navarro, Juan Carlos & , 2016. "La política de innovación en América Latina y el Caribe: Nuevos caminos," IDB Publications (Books), Inter-American Development Bank, number 7705, December.
    18. Gabriel Casaburi & Gustavo Crespi & Ignacio L. De León & José Fernández & Lucas Figal Garone & Matteo Grazzi & Carlos Guaipatín & Jorge Katz & Juan J. Llisterri & Alessandro Maffioli & Juan Carlos Nav, 2016. "La política de innovación en América Latina y el Caribe: Nuevos caminos," IDB Publications (Books), Inter-American Development Bank, number 94816 edited by Juan Carlos Navarro & Jocelyn Olivari, February.
    19. Munari, Federico & Toschi, Laura, 2021. "The impact of public funding on science valorisation: an analysis of the ERC Proof-of-Concept Programme," Research Policy, Elsevier, vol. 50(6).
    20. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pal:palcom:v:4:y:2018:i:1:d:10.1057_s41599-018-0069-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: https://www.nature.com/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.