IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v44y2015i8p1462-1472.html
   My bibliography  Save this article

Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits

Author

Listed:
  • Bakhshi, Hasan
  • Edwards, John S.
  • Roper, Stephen
  • Scully, Judy
  • Shaw, Duncan
  • Morley, Lorraine
  • Rathbone, Nicola

Abstract

Experimental methods of policy evaluation are well-established in social policy and development economics but are rare in industrial and innovation policy. In this paper, we consider the arguments for applying experimental methods to industrial policy measures, and propose an experimental policy evaluation approach (which we call RCT+). This approach combines the randomised assignment of firms to treatment and control groups with a longitudinal data collection strategy incorporating quantitative and qualitative data (so-called mixed methods). The RCT+ approach is designed to provide a causative rather than purely summative evaluation, i.e. to assess both ‘whether’ and ‘how’ programme outcomes are achieved. In this paper, we assess the RCT+ approach through an evaluation of Creative Credits – a UK business-to-business innovation voucher initiative intended to promote new innovation partnerships between SMEs and creative service providers. The results suggest the potential value of the RCT+ approach to industrial policy evaluation, and the benefits of mixed methods and longitudinal data collection.

Suggested Citation

  • Bakhshi, Hasan & Edwards, John S. & Roper, Stephen & Scully, Judy & Shaw, Duncan & Morley, Lorraine & Rathbone, Nicola, 2015. "Assessing an experimental approach to industrial policy evaluation: Applying RCT+ to the case of Creative Credits," Research Policy, Elsevier, vol. 44(8), pages 1462-1472.
    • Handle: RePEc:eee:respol:v:44:y:2015:i:8:p:1462-1472
    DOI: 10.1016/j.respol.2015.04.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733315000645
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2015.04.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. Hewitt-Dundas, Nola & Roper, Stephen, 2011. "Creating advantage in peripheral regions: The role of publicly funded R&D centres," Research Policy, Elsevier, vol. 40(6), pages 832-841, July.
    3. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    4. G. M.P. Swann, 2009. "The Economics of Innovation," Books, Edward Elgar Publishing, number 13211.
    5. Roper, Stephen & Arvanitis, Spyros, 2012. "From knowledge to added value: A comparative, panel-data analysis of the innovation value chain in Irish and Swiss manufacturing firms," Research Policy, Elsevier, vol. 41(6), pages 1093-1106.
    6. Garcia, Abraham & Mohnen, Pierre, 2010. "Impact of government support on R&D and innovation," MERIT Working Papers 2010-034, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    7. Donaldson, Stewart I. & Gooler, Laura E., 2003. "Theory-driven evaluation in action: lessons from a $20 million statewide Work and Health Initiative," Evaluation and Program Planning, Elsevier, vol. 26(4), pages 355-366, November.
    8. Nicholas Bloom & Benn Eifert & Aprajit Mahajan & David McKenzie & John Roberts, 2013. "Does Management Matter? Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(1), pages 1-51.
    9. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    10. Johan Lambrecht & Fabrice Pirnay, 2005. "An evaluation of public support measures for private external consultancies to SMEs in the Walloon Region of Belgium," Entrepreneurship & Regional Development, Taylor & Francis Journals, vol. 17(2), pages 89-108, March.
    11. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    12. Meuleman, Miguel & De Maeseneire, Wouter, 2012. "Do R&D subsidies affect SMEs’ access to external financing?," Research Policy, Elsevier, vol. 41(3), pages 580-591.
    13. Lerner, Josh, 1999. "The Government as Venture Capitalist: The Long-Run Impact of the SBIR Program," The Journal of Business, University of Chicago Press, vol. 72(3), pages 285-318, July.
    14. Espen Bratberg & Astrid Grasdal & Alf Erling Risa, 2002. "Evaluating Social Policy by Experimental and Nonexperimental Methods," Scandinavian Journal of Economics, Wiley Blackwell, vol. 104(1), pages 147-171, March.
    15. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    16. Trüby, Johannes & Rammer, Christian & Müller, Kathrin, 2008. "The Role of Creative Industries in Industrial Innovation," ZEW Discussion Papers 08-109, ZEW - Leibniz Centre for European Economic Research.
    17. Tuomas Takalo & Tanja Tanayama, 2010. "Adverse selection and financing of innovation: is there a need for R&D subsidies?," The Journal of Technology Transfer, Springer, vol. 35(1), pages 16-41, February.
    18. Paul Nightingale & Alex Coad, 2014. "Muppets and gazelles: political and methodological biases in entrepreneurship research," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 23(1), pages 113-143, February.
    19. repec:pri:rpdevs:deaton_instruments_randomization_learning_all_04april_2010 is not listed on IDEAS
    20. repec:bla:scandj:v:104:y:2002:i:1:p:147-71 is not listed on IDEAS
    21. Maarten Cornet & Björn Vroomen & Marc van der Steeg, 2006. "Do innovation vouchers help SMEs to cross the bridge towards science?," CPB Discussion Paper 58, CPB Netherlands Bureau for Economic Policy Analysis.
    22. Nola Hewitt-Dundas & Stephen Roper, 2009. "Output Additionality of Public Support for Innovation: Evidence for Irish Manufacturing Plants," European Planning Studies, Taylor & Francis Journals, vol. 18(1), pages 107-122, September.
    23. Rosenbusch, Nina & Brinckmann, Jan & Bausch, Andreas, 2011. "Is innovation always beneficial? A meta-analysis of the relationship between innovation and performance in SMEs," Journal of Business Venturing, Elsevier, vol. 26(4), pages 441-457, July.
    24. Forsman, Helena, 2011. "Innovation capacity and innovation development in small enterprises. A comparison between the manufacturing and service sectors," Research Policy, Elsevier, vol. 40(5), pages 739-750, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    2. Kleine, Marco & Heite, Jonas & Huber, Laura Rosendahl, 2022. "Subsidized R&D collaboration: The causal effect of innovation vouchers on innovation outcomes," Research Policy, Elsevier, vol. 51(6).
    3. Daunfeldt, Sven-Olov & Halvarsson, Daniel & Gustavsson Tingvall, Patrik & McKelvie, Alexander, 2021. "Do Targeted R&D Grants Towards Potential Highgrowth Firms Increase Employment and Demand for High Human Capital Workers?," HFI Working Papers 23, Institute of Retail Economics (Handelns Forskningsinstitut).
    4. Caloffi, Annalisa & Freo, Marzia & Ghinoi, Stefano & Mariani, Marco & Rossi, Federica, 2022. "Assessing the effects of a deliberate policy mix: The case of technology and innovation advisory services and innovation vouchers," Research Policy, Elsevier, vol. 51(6).
    5. Viktorie Klimova & Klaudia Glittova & Vladimir Zitek, 2023. "Innovation vouchers and cooperation: a different approach in two countries with a shared history," Eastern Journal of European Studies, Centre for European Studies, Alexandru Ioan Cuza University, vol. 14, pages 22-44, December.
    6. Chiara Dalle Nogare & Monika Murzyn-Kupisz, 2021. "Do museums foster innovation through engagement with the cultural and creative industries?," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 45(4), pages 671-704, December.
    7. Klaudia Bracio & Marek Szarucki, 2020. "Mixed Methods Utilisation in Innovation Management Research: A Systematic Literature Review and Meta-Summary," JRFM, MDPI, vol. 13(11), pages 1-27, October.
    8. Erik E. Lehmann & Matthias Menter, 2018. "Public cluster policy and performance," The Journal of Technology Transfer, Springer, vol. 43(3), pages 558-592, June.
    9. Vladimír Baláž & Tomáš Jeck & Miroslav Balog, 2024. "Knowledge Transfers and Business Performance in Creative Networks," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(1), pages 4428-4454, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hasan Bakhshi & John Edwards & Stephen Roper & Judy Scully & Duncan Shaw & Lorraine Morley & Nicola Rathbone, 2013. "An Experimental Approach to Industrial Policy Evaluation: The case of Creative Credits," Research Papers 0004, Enterprise Research Centre.
    2. Gani Aldashev & Georg Kirchsteiger & Alexander Sebald, 2017. "Assignment Procedure Biases in Randomised Policy Experiments," Economic Journal, Royal Economic Society, vol. 127(602), pages 873-895, June.
    3. Andrea Bellucci & Luca Pennacchio & Alberto Zazzaro, 2019. "R&D Subsidies and Firms’ Debt Financing," CSEF Working Papers 527, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    4. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    5. Brautzsch, Hans-Ulrich & Günther, Jutta & Loose, Brigitte & Ludwig, Udo & Nulsch, Nicole, 2015. "Can R&D subsidies counteract the economic crisis? – Macroeconomic effects in Germany," Research Policy, Elsevier, vol. 44(3), pages 623-633.
    6. Baldwin, Kate & Bhavnani, Rikhil R., 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series 024, World Institute for Development Economic Research (UNU-WIDER).
    7. Wu, Aihua, 2017. "The signal effect of Government R&D Subsidies in China: Does ownership matter?," Technological Forecasting and Social Change, Elsevier, vol. 117(C), pages 339-345.
    8. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    9. Demeulemeester, Sarah & Hottenrott, Hanna, 2015. "R&D subsidies and firms' cost of debt," DICE Discussion Papers 201, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    10. Kiman Kim & Sang Ok Choi & Sooyeon Lee, 2021. "The Effect of a Financial Support on Firm Innovation Collaboration and Output: Does Policy Work on the Diverse Nature of Firm Innovation?," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 12(2), pages 645-675, June.
    11. Albert Bravo-Biosca, 2020. "Experimental Innovation Policy," Innovation Policy and the Economy, University of Chicago Press, vol. 20(1), pages 191-232.
    12. Busom, Isabel & Corchuelo, Beatriz, 2011. "Tax incentives and direct support for R&D : what do firms use and why?," IC3JM - Estudios = Working Papers id-11-03, Instituto Mixto Carlos III - Juan March de Ciencias Sociales (IC3JM).
    13. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    14. Astrid Grasdal, 2001. "The performance of sample selection estimators to control for attrition bias," Health Economics, John Wiley & Sons, Ltd., vol. 10(5), pages 385-398, July.
    15. Busom, Isabel & Corchuelo, Beatriz & Martinez Ros, Ester, 2012. "Tax incentives or subsidies for R&D?," MERIT Working Papers 2012-056, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    16. Stjepan Srhoj & Bruno Škrinjarić & Sonja Radas, 2021. "Bidding against the odds? The impact evaluation of grants for young micro and small firms during the recession," Small Business Economics, Springer, vol. 56(1), pages 83-103, January.
    17. Kate Baldwin & Rikhil R. Bhavnani, 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series wp-2013-024, World Institute for Development Economic Research (UNU-WIDER).
    18. Alicia Mas-Tur & Domingo Ribeiro Soriano, 2014. "The level of innovation among young innovative companies: the impacts of knowledge-intensive services use, firm characteristics and the entrepreneur attributes," Service Business, Springer;Pan-Pacific Business Association, vol. 8(1), pages 51-63, March.
    19. Yan, Ziqiao & Li, Yue, 2018. "Signaling through government subsidy: Certification or endorsement," Finance Research Letters, Elsevier, vol. 25(C), pages 90-95.
    20. Bettina Becker, 2015. "Public R&D Policies And Private R&D Investment: A Survey Of The Empirical Evidence," Journal of Economic Surveys, Wiley Blackwell, vol. 29(5), pages 917-942, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:44:y:2015:i:8:p:1462-1472. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.