IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v11y2014i1p11-16.html
   My bibliography  Save this article

One Swallow Doesn't Make a Summer: Reply to Kataria

Author

Listed:
  • Zacharias Maniadis
  • Fabio Tufano
  • John A. List

Abstract

In this paper we reply to Mitesh Kataria’s comment, which criticized the simulations of Maniadis, Tufano, and List (2014, Am. Econ. Rev. 104(1), 277–290). We view these simulations as a means to illustrating the fact that we economists are unaware of the value of key variables that determine the credibility of our own empirical findings. Such variables include priors (i.e., the pre-study probability that a tested phenomenon is true) and the statistical power of the empirical design. Economists should not hesitate to use Bayesian tools and meta-analysis in order to quantify what we know about these variables.

Suggested Citation

  • Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: Reply to Kataria," Econ Journal Watch, Econ Journal Watch, vol. 11(1), pages 11-16, January.
    • Handle: RePEc:ejw:journl:v:11:y:2014:i:1:p:11-16
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/793/ManiadisTufanoListJan2014.pdf?mimetype=pdf
    Download Restriction: no
    File URL: https://econjwatch.org/905
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Caplin, Andrew & Schotter, Andrew, 2008. "The Foundations of Positive and Normative Economics: A Handbook," OUP Catalogue, Oxford University Press, number 9780195328318.
    3. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    4. Daniele Fanelli, 2010. "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data," PLOS ONE, Public Library of Science, vol. 5(4), pages 1-7, April.
    5. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: Reply to Kataria," Econ Journal Watch, Econ Journal Watch, vol. 11(1), pages 11-16, January.
    6. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    7. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    8. Mitesh Kataria, 2014. "One Swallow Doesn't Make a Summer: A Comment on Zacharias Maniadis, Fabio Tufano, and John List," Econ Journal Watch, Econ Journal Watch, vol. 11(1), pages 4-10, January.
    9. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    10. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    11. Richard A. Bettis, 2012. "The search for asterisks: Compromised statistical tests and flawed theories," Strategic Management Journal, Wiley Blackwell, vol. 33(1), pages 108-113, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Greg Fischer & Dean Karlan & Margaret McConnell & Pia Raffler, 2014. "To Charge or Not to Charge: Evidence from a Health Products Experiment in Uganda," Working Papers 1041, Economic Growth Center, Yale University.
    2. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: Reply to Kataria," Econ Journal Watch, Econ Journal Watch, vol. 11(1), pages 11-16, January.
    3. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    4. Kathryn Graddy & Lara Loewenstein & Jianping Mei & Mike Moses & Rachel A. J. Pownall, 2023. "Empirical evidence of anchoring and loss aversion from art auctions," Journal of Cultural Economics, Springer;The Association for Cultural Economics International, vol. 47(2), pages 279-301, June.
    5. Eric Floyd & John A. List, 2016. "Using Field Experiments in Accounting and Finance," Journal of Accounting Research, Wiley Blackwell, vol. 54(2), pages 437-475, May.
    6. Carlos Alós-Ferrer & Georg D. Granic, 2023. "Does choice change preferences? An incentivized test of the mere choice effect," Experimental Economics, Springer;Economic Science Association, vol. 26(3), pages 499-521, July.
    7. Wu, Hanxiao & Qin, Botao, 2019. "An experimental test of the solemn oath in eliciting sincere preferences," MPRA Paper 95913, University Library of Munich, Germany.
    8. Strømland, Eirik & Torsvik, Gaute, 2019. "Intuitive Prosociality: Heterogeneous Treatment Effects or False Positive?," OSF Preprints hrx2y, Center for Open Science.
    9. Gächter, Simon & Kölle, Felix & Quercia, Simone, 2022. "Preferences and perceptions in Provision and Maintenance public goods," Games and Economic Behavior, Elsevier, vol. 135(C), pages 338-355.
    10. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    11. John List, 2020. "Non est Disputandum de Generalizability? A Glimpse into The External Validity Trial," Artefactual Field Experiments 00711, The Field Experiments Website.
    12. Kathryn Graddy & Lara Loewenstein & Jianping Mei & Mike Moses & Rachel A J Pownall, 2014. "Anchoring or Loss Aversion? Empirical Evidence from Art Auctions," ACEI Working Paper Series AWP-04-2014, Association for Cultural Economics International, revised Jun 2014.
    13. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    14. Brocas, Isabelle & Carrillo, Juan D., 2021. "Young children use commodities as an indirect medium of exchange," Games and Economic Behavior, Elsevier, vol. 125(C), pages 48-61.
    15. Krämer, Andreas, 2018. "Bounded Irrationality: Chancen und Grenzen beim verhaltensbasierten Pricing," Marketing Review St.Gallen, Universität St.Gallen, Institut für Marketing und Customer Insight, vol. 35(2), pages 102-110.
    16. Vincent Mak & Darryl A. Seale & Amnon Rapoport & Eyran J. Gisches, 2019. "Voting Rules in Sequential Search by Committees: Theory and Experiments," Management Science, INFORMS, vol. 65(9), pages 4349-4364, September.
    17. Vanessa Mertins & Christian Walter, 2021. "In absence of money: a field experiment on volunteer work motivation," Experimental Economics, Springer;Economic Science Association, vol. 24(3), pages 952-984, September.
    18. Jeworrek, Sabrina & Leisen, Bernd Josef & Mertins, Vanessa, 2021. "Gift-exchange in society and the social integration of refugees–Evidence from a survey, a laboratory, and a field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 192(C), pages 482-499.
    19. Karakaya, Etem & Alataş, Sedat & Yılmaz, Burcu, 2019. "Replication of Strazicich and List (2003): Are CO2 emission levels converging among industrial countries?," Energy Economics, Elsevier, vol. 82(C), pages 135-138.
    20. Despoina Alempaki & Emina Canic & Timothy L. Mullett & William J. Skylark & Chris Starmer & Neil Stewart & Fabio Tufano, 2019. "Reexamining How Utility and Weighting Functions Get Their Shapes: A Quasi-Adversarial Collaboration Providing a New Interpretation," Management Science, INFORMS, vol. 65(10), pages 4841-4862, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    4. Matteo M. Galizzi & Daniel Navarro-Martinez, 2019. "On the External Validity of Social Preference Games: A Systematic Lab-Field Study," Management Science, INFORMS, vol. 65(3), pages 976-1002, March.
    5. Brian Fabo & Martina Jancokova & Elisabeth Kempf & Lubos Pastor, 2020. "Fifty Shades of QE: Conflicts of Interest in Economic Research," Working Papers 2020-128, Becker Friedman Institute for Research In Economics.
    6. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    7. Jae H. Kim & Kamran Ahmed & Philip Inyeob Ji, 2018. "Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence," Abacus, Accounting Foundation, University of Sydney, vol. 54(4), pages 524-546, December.
    8. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    9. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    10. Strømland, Eirik & Torsvik, Gaute, 2019. "Intuitive Prosociality: Heterogeneous Treatment Effects or False Positive?," OSF Preprints hrx2y, Center for Open Science.
    11. Fabo, Brian & Jančoková, Martina & Kempf, Elisabeth & Pástor, Ľuboš, 2021. "Fifty shades of QE: Comparing findings of central bankers and academics," Journal of Monetary Economics, Elsevier, vol. 120(C), pages 1-20.
    12. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    13. Jakub Rybacki, 2020. "Are Central Banks’ Research Teams Fragile Because of Groupthink in the Area of Monetary Policy? – Evidence on Inflation Targeting," Gospodarka Narodowa. The Polish Journal of Economics, Warsaw School of Economics, issue 4, pages 81-103.
    14. Jakub Rybacki, 2019. "Are Central Banks' Research Teams Fragile Because of Groupthink?," KAE Working Papers 2019-045, Warsaw School of Economics, Collegium of Economic Analysis.
    15. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Papers 2006.04269, arXiv.org.
    16. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    17. Campbell R. Harvey & Yan Liu, 2020. "False (and Missed) Discoveries in Financial Economics," Journal of Finance, American Finance Association, vol. 75(5), pages 2503-2553, October.
    18. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    19. Carlos Alós-Ferrer & Georg D. Granic, 2023. "Does choice change preferences? An incentivized test of the mere choice effect," Experimental Economics, Springer;Economic Science Association, vol. 26(3), pages 499-521, July.
    20. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    Economic methodology; statistical inference;

    JEL classification:

    • B40 - Schools of Economic Thought and Methodology - - Economic Methodology - - - General
    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. One Swallow Doesn’t Make a Summer: New Evidence on Anchoring Effects (AER 2014) in ReplicationWiki
    2. One Swallow Doesn’t Make a Summer: A Comment on Zacharias Maniadis, Fabio Tufano, and John List (EJW 2014) in ReplicationWiki

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:11:y:2014:i:1:p:11-16. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jason Briggeman (email available below). General contact details of provider: https://edirc.repec.org/data/edgmuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.