IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i3p212-239.html
   My bibliography  Save this article

Raising the Bar

Author

Listed:
  • Neil Seftor

Abstract

Background: In 2002, the U.S. Department of Education’s Institute of Education Sciences (IES) established the What Works Clearinghouse (WWC) at the confluence of a push to improve education research quality, a shift toward evidence-based decision-making, and an expansion of systematic reviews. In addition to providing decision makers with evidence to inform their choices, a systematic review sets expectations regarding study quality and execution for research on program efficacy. In this article, we examine education research through the filter of a long running systematic review to assess research quality over time and the role of the systematic review in producing evidence. Method: Using the WWC’s database of reviewed studies, we explored the relationships between study characteristics and dispositions as well as the differences by topic area and changes over time. Conclusion: Through its design standards, the WWC has defined its requirements for a study to be considered causal evidence, which may have been one of the factors contributing to observed improvement in the quality of education research over the past 15 years. The levels and rates of studies meeting standards have been increasing over the life of the WWC. Additionally, the number and proportion of studies excluded due to ineligible design are decreasing. Thus, less research is ineligible due to design issues, and more eligible studies are meeting standards. As IES continues to conduct and fund studies designed to meet standards, and more decisions are directly tied to evidence, the body of rigorous education research may continue to grow.

Suggested Citation

  • Neil Seftor, 2017. "Raising the Bar," Evaluation Review, , vol. 41(3), pages 212-239, June.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:3:p:212-239
    DOI: 10.1177/0193841X16665023
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16665023
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X16665023?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    3. repec:mpr:mprres:7034 is not listed on IDEAS
    4. Sarah Avellar & M. Robin Dion & Andrew Clarkwest & Heather Zaveri & Subuhi Asheer & Kelley Borradaile & Megan Hague Angus & Timothy Novak & Julie Redline & Marykate Zukiewicz, 2011. "Catalog of Research: Programs for Low-Income Fathers," Mathematica Policy Research Reports d17bbc7a248948009778bbd86, Mathematica Policy Research.
    5. Diane Paulsell & Jaime Thomas & Shannon Monahan & Neil S. Seftor, "undated". "A Trusted Source of Information: How Systematic Reviews Can Support User Decisions About Adopting Evidence-Based Programs," Mathematica Policy Research Reports d637ad38ce7649ae84a7ddf06, Mathematica Policy Research.
    6. George Teeling Smith;Nick Wells, 1979. "Medicines for the year 2000," Monograph 000301, Office of Health Economics.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," The World Bank Economic Review, World Bank, vol. 31(3), pages 687-707.
    2. Czajkowski, Mikołaj & Zagórska, Katarzyna & Letki, Natalia & Tryjanowski, Piotr & Wąs, Adam, 2021. "Drivers of farmers’ willingness to adopt extensive farming practices in a globally important bird area," Land Use Policy, Elsevier, vol. 107(C).
    3. Michael J. Fell & Alexandra Schneiders & David Shipworth, 2019. "Consumer Demand for Blockchain-Enabled Peer-to-Peer Electricity Trading in the United Kingdom: An Online Survey Experiment," Energies, MDPI, vol. 12(20), pages 1-25, October.
    4. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    5. Su Keng Tan & Wai Keung Leung & Alexander Tin Hong Tang & Roger A Zwahlen, 2017. "Effects of mandibular setback with or without maxillary advancement osteotomies on pharyngeal airways: An overview of systematic reviews," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-20, October.
    6. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.
    7. Christopher Conlon & Julie Holland Mortimer, 2021. "Empirical properties of diversion ratios," RAND Journal of Economics, RAND Corporation, vol. 52(4), pages 693-726, December.
    8. Ángel Enrique & Juana Bretón-López & Guadalupe Molinari & Rosa M. Baños & Cristina Botella, 2018. "Efficacy of an adaptation of the Best Possible Self intervention implemented through positive technology: a randomized control trial," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 13(3), pages 671-689, September.
    9. Gerben ter Riet & Paula Chesley & Alan G Gross & Lara Siebeling & Patrick Muggensturm & Nadine Heller & Martin Umbehr & Daniela Vollenweider & Tsung Yu & Elie A Akl & Lizzy Brewster & Olaf M Dekkers &, 2013. "All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-6, November.
    10. Iranzu Mugueta-Aguinaga & Begonya Garcia-Zapirain, 2017. "FRED: Exergame to Prevent Dependence and Functional Deterioration Associated with Ageing. A Pilot Three-Week Randomized Controlled Clinical Trial," IJERPH, MDPI, vol. 14(12), pages 1-18, November.
    11. Spyridon N Papageorgiou & Georgios N Antonoglou & George K Sándor & Theodore Eliades, 2017. "Randomized clinical trials in orthodontics are rarely registered a priori and often published late or not at all," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-13, August.
    12. Milagros Molero-Zafra & María Teresa Mitjans-Lafont & María Jesús Hernández-Jiménez & Marián Pérez-Marín, 2022. "Psychological Intervention in Women Victims of Childhood Sexual Abuse: An Open Study—Protocol of a Randomized Controlled Clinical Trial Comparing EMDR Psychotherapy and Trauma-Based Cognitive Therapy," IJERPH, MDPI, vol. 19(12), pages 1-16, June.
    13. Eun-Hi Kong & Myoungsuk Kim & Seonho Kim, 2021. "Effects of a Web-Based Educational Program Regarding Physical Restraint Reduction in Long-Term Care Settings on Nursing Students: A Cluster Randomized Controlled Trial," IJERPH, MDPI, vol. 18(13), pages 1-10, June.
    14. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    15. Christos Genakos & Andreas Lamprinidis & James Walker, 2023. "Evaluating merger effects," CEP Discussion Papers dp1921, Centre for Economic Performance, LSE.
    16. Xiaoxuan Gong & Shaowen Tang & Jiangjin Li & Xiwen Zhang & Xiaoyi Tian & Shuren Ma, 2017. "Antithrombotic therapy strategies for atrial fibrillation patients undergoing percutaneous coronary intervention: A systematic review and network meta-analysis," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-13, October.
    17. Katherine Sawyer & Kathleen Gallagher Cunningham & William Reed, 2017. "The Role of External Support in Civil War Termination," Journal of Conflict Resolution, Peace Science Society (International), vol. 61(6), pages 1174-1202, July.
    18. Koki Arai, 2017. "Ex-post examination of mergers: effects on retail prices," Asia-Pacific Journal of Accounting & Economics, Taylor & Francis Journals, vol. 24(1-2), pages 145-162, April.
    19. Elliott Ash & W. Bentley MacLeod, 2015. "Intrinsic Motivation in Public Service: Theory and Evidence from State Supreme Courts," Journal of Law and Economics, University of Chicago Press, vol. 58(4).
    20. Stavros Petrou & Oliver Rivero-Arias & Helen Dakin & Louise Longworth & Mark Oppe & Robert Froud & Alastair Gray, 2015. "Preferred Reporting Items for Studies Mapping onto Preference-Based Outcome Measures: The MAPS Statement," Medical Decision Making, , vol. 35(6), pages 1-8, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:3:p:212-239. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.