IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/26278.html
   My bibliography  Save this paper

Experimentation and Startup Performance: Evidence from A/B testing

Author

Listed:
  • Rembrand Koning
  • Sharique Hasan
  • Aaron Chatterji

Abstract

Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing business ideas. This paper provides the first evidence of how digital experimentation affects the performance of a large sample of high-technology startups using data that tracks their growth, technology use, and product launches. We find that, despite its prominence in the business press, relatively few firms have adopted A/B testing. However, among those that do, we find increased performance on several critical dimensions, including page views and new product features. Furthermore, A/B testing is positively related to tail outcomes, with younger ventures failing faster and older firms being more likely to scale. Firms with experienced managers also derive more benefits from A/B testing. Our results inform the emerging literature on entrepreneurial strategy and how digitization and data-driven decision-making are shaping strategy.

Suggested Citation

  • Rembrand Koning & Sharique Hasan & Aaron Chatterji, 2019. "Experimentation and Startup Performance: Evidence from A/B testing," NBER Working Papers 26278, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:26278
    Note: PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w26278.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Steven D. Levitt & John A. List & Chad Syverson, 2013. "Toward an Understanding of Learning by Doing: Evidence from an Automobile Assembly Plant," Journal of Political Economy, University of Chicago Press, vol. 121(4), pages 643-681.
    2. Ashish Arora & Anand Nandkumar, 2011. "Cash-Out or Flameout! Opportunity Cost and Entrepreneurial Strategy: Theory, and Evidence from the Information Security Industry," Management Science, INFORMS, vol. 57(10), pages 1844-1860, October.
    3. Steven N. Kaplan & Josh Lerner, 2016. "Venture Capital Data: Opportunities and Challenges," NBER Chapters, in: Measuring Entrepreneurial Businesses: Current Knowledge and Challenges, pages 413-431, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jinglong Zhao & Zijie Zhou, 2022. "Pigeonhole Design: Balancing Sequential Experiments from an Online Matching Perspective," Papers 2201.12936, arXiv.org, revised May 2024.
    2. Dushnitsky, Gary & Stroube, Bryan K., 2021. "Low-code entrepreneurship: Shopify and the alternative path to growth," Journal of Business Venturing Insights, Elsevier, vol. 16(C).
    3. Victor M. Bennett & Todd A. Hall, 2020. "Software availability and entry," Strategic Management Journal, Wiley Blackwell, vol. 41(5), pages 950-962, May.
    4. Guzman, Jorge & Li, Aishen, 2019. "Measuring Founding Strategy," SocArXiv 7cvge, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruneel, Johan & Clarysse, Bart & Bobelyn, Annelies & Wright, Mike, 2020. "Liquidity events and VC-backed academic spin-offs: The role of search alliances," Research Policy, Elsevier, vol. 49(10).
    2. Gordon L Clark & Ashby H B Monk, 2014. "The Geography of Investment Management Contracts: The UK, Europe, and the Global Financial Services Industry," Environment and Planning A, , vol. 46(3), pages 531-549, March.
    3. Braguinsky, Serguey & Honjo, Yuji & 本庄, 裕司 & Nagaoka, Sadao & 長岡, 貞男 & Nakamura, Kenta & 中村, 健太, 2010. "Science-Based Business : Knowledge Capital or Entrepreneurial Ability? : Theory and Evidence from a Survey of Biotechnology Start-ups," IIR Working Paper 10-05, Institute of Innovation Research, Hitotsubashi University.
    4. Bartels, Charlotte & Jäger, Simon & Obergruber, Natalie, 2020. "Long-Term Effects of Equal Sharing: Evidence from Inheritance Rules for Land," IZA Discussion Papers 13665, Institute of Labor Economics (IZA).
    5. Ben Klemens, 2021. "Attributing Value to Patents and Trademarks in Complex Production Chains," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 12(2), pages 842-875, June.
    6. Joern H. Block & Christian O. Fisch & Mirjam van Praag, 2017. "The Schumpeterian entrepreneur: a review of the empirical evidence on the antecedents, behaviour and consequences of innovative entrepreneurship," Industry and Innovation, Taylor & Francis Journals, vol. 24(1), pages 61-95, January.
    7. Kathrin Manthei & Dirk Sliwka & Timo Vogelsang, 2021. "Performance Pay and Prior Learning—Evidence from a Retail Chain," Management Science, INFORMS, vol. 67(11), pages 6998-7022, November.
    8. Chong Kyoon Lee & Griffin W. Cottle & Sharon A. Simmons & Johan Wiklund, 2021. "Fear not, want not: Untangling the effects of social cost of failure on high-growth entrepreneurship," Small Business Economics, Springer, vol. 57(1), pages 531-553, June.
    9. Ethan Ilzetzki, 2024. "Learning by Necessity: Government Demand, Capacity Constraints, and Productivity Growth," American Economic Review, American Economic Association, vol. 114(8), pages 2436-2471, August.
    10. Zunino, Diego & van Praag, Mirjam C. & Dushnitsky, Gary, 2017. "Badge of Honor or Scarlet Letter? Unpacking Investors' Judgment of Entrepreneurs' Past Failure," IZA Discussion Papers 11017, Institute of Labor Economics (IZA).
    11. Shanjun Li & Junji Xiao & Yimin Liu, 2015. "The Price Evolution in China's Automobile Market," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 24(4), pages 786-810, October.
    12. Carolyn D. Egelman & Dennis Epple & Linda Argote & Erica R.H. Fuchs, 2013. "Learning by Doing in a Multi-Product Manufacturing Environment: Product Variety, Customizations, and Overlapping Product Generations," NBER Working Papers 19674, National Bureau of Economic Research, Inc.
    13. Linda Argote & Sunkee Lee & Jisoo Park, 2021. "Organizational Learning Processes and Outcomes: Major Findings and Future Research Directions," Management Science, INFORMS, vol. 67(9), pages 5399-5429, September.
    14. Gurkan Calmasur & Meryem Emre Aysin, 2020. "Regional Technological Learning in Turkish Cement Industry," Eurasian Journal of Economics and Finance, Eurasian Publications, vol. 8(4), pages 204-216.
    15. Bian, Bo & Li, Yingxiang & Nigro, Casimiro A., 2022. "Conflicting fiduciary duties and fire sales of VC-backed start-ups," LawFin Working Paper Series 35, Goethe University, Center for Advanced Studies on the Foundations of Law and Finance (LawFin).
    16. Jaka Cepec & Peter Grajzl & Barbara Mörec, 2022. "Public cash and modes of firm exit," Journal of Evolutionary Economics, Springer, vol. 32(1), pages 247-298, January.
    17. Kraemer-Eis, Helmut & Botsari, Antonia & Gvetadze, Salome & Lang, Frank & Torfs, Wouter, 2019. "European Small Business Finance Outlook: December 2019," EIF Working Paper Series 2019/61, European Investment Fund (EIF).
    18. Avdic, Daniel & Lundborg, Petter & Vikström, Johan, 2014. "Learning-by-Doing in a Highly Skilled Profession when Stakes are High: Evidence from Advanced Cancer Surgery," Working Paper Series, Center for Labor Studies 2014:9, Uppsala University, Department of Economics.
    19. Wiltbank, Robert & Dew, Nicholas & Read, Stuart, 2015. "Investment and returns in successful entrepreneurial sell-outs," Journal of Business Venturing Insights, Elsevier, vol. 3(C), pages 16-23.
    20. Kaiser, Ulrich & Kuhn, Johan Moritz, 2019. "Who Founds? An Analysis of University and Corporate Startup Entrepreneurs Based on Danish Register Data," IZA Discussion Papers 12191, Institute of Labor Economics (IZA).

    More about this item

    JEL classification:

    • M13 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Business Administration - - - New Firms; Startups
    • M2 - Business Administration and Business Economics; Marketing; Accounting; Personnel Economics - - Business Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:26278. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.