IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i3p212-239.html
   My bibliography  Save this article

Raising the Bar

Author

Listed:
  • Neil Seftor

Abstract

Background: In 2002, the U.S. Department of Education’s Institute of Education Sciences (IES) established the What Works Clearinghouse (WWC) at the confluence of a push to improve education research quality, a shift toward evidence-based decision-making, and an expansion of systematic reviews. In addition to providing decision makers with evidence to inform their choices, a systematic review sets expectations regarding study quality and execution for research on program efficacy. In this article, we examine education research through the filter of a long running systematic review to assess research quality over time and the role of the systematic review in producing evidence. Method: Using the WWC’s database of reviewed studies, we explored the relationships between study characteristics and dispositions as well as the differences by topic area and changes over time. Conclusion: Through its design standards, the WWC has defined its requirements for a study to be considered causal evidence, which may have been one of the factors contributing to observed improvement in the quality of education research over the past 15 years. The levels and rates of studies meeting standards have been increasing over the life of the WWC. Additionally, the number and proportion of studies excluded due to ineligible design are decreasing. Thus, less research is ineligible due to design issues, and more eligible studies are meeting standards. As IES continues to conduct and fund studies designed to meet standards, and more decisions are directly tied to evidence, the body of rigorous education research may continue to grow.

Suggested Citation

  • Neil Seftor, 2017. "Raising the Bar," Evaluation Review, , vol. 41(3), pages 212-239, June.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:3:p:212-239
    DOI: 10.1177/0193841X16665023
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16665023
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X16665023?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    3. repec:mpr:mprres:7034 is not listed on IDEAS
    4. Sarah Avellar & M. Robin Dion & Andrew Clarkwest & Heather Zaveri & Subuhi Asheer & Kelley Borradaile & Megan Hague Angus & Timothy Novak & Julie Redline & Marykate Zukiewicz, 2011. "Catalog of Research: Programs for Low-Income Fathers," Mathematica Policy Research Reports d17bbc7a248948009778bbd86, Mathematica Policy Research.
    5. Diane Paulsell & Jaime Thomas & Shannon Monahan & Neil S. Seftor, "undated". "A Trusted Source of Information: How Systematic Reviews Can Support User Decisions About Adopting Evidence-Based Programs," Mathematica Policy Research Reports d637ad38ce7649ae84a7ddf06, Mathematica Policy Research.
    6. George Teeling Smith;Nick Wells, 1979. "Medicines for the year 2000," Monograph 000301, Office of Health Economics.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," The World Bank Economic Review, World Bank, vol. 31(3), pages 687-707.
    2. Czajkowski, Mikołaj & Zagórska, Katarzyna & Letki, Natalia & Tryjanowski, Piotr & Wąs, Adam, 2021. "Drivers of farmers’ willingness to adopt extensive farming practices in a globally important bird area," Land Use Policy, Elsevier, vol. 107(C).
    3. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    4. Ángel Enrique & Juana Bretón-López & Guadalupe Molinari & Rosa M. Baños & Cristina Botella, 2018. "Efficacy of an adaptation of the Best Possible Self intervention implemented through positive technology: a randomized control trial," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 13(3), pages 671-689, September.
    5. Gerben ter Riet & Paula Chesley & Alan G Gross & Lara Siebeling & Patrick Muggensturm & Nadine Heller & Martin Umbehr & Daniela Vollenweider & Tsung Yu & Elie A Akl & Lizzy Brewster & Olaf M Dekkers &, 2013. "All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-6, November.
    6. Spyridon N Papageorgiou & Georgios N Antonoglou & George K Sándor & Theodore Eliades, 2017. "Randomized clinical trials in orthodontics are rarely registered a priori and often published late or not at all," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-13, August.
    7. Eun-Hi Kong & Myoungsuk Kim & Seonho Kim, 2021. "Effects of a Web-Based Educational Program Regarding Physical Restraint Reduction in Long-Term Care Settings on Nursing Students: A Cluster Randomized Controlled Trial," IJERPH, MDPI, vol. 18(13), pages 1-10, June.
    8. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    9. Christos Genakos & Andreas Lamprinidis & James Walker, 2023. "Evaluating merger effects," CEP Discussion Papers dp1921, Centre for Economic Performance, LSE.
    10. Katherine Sawyer & Kathleen Gallagher Cunningham & William Reed, 2017. "The Role of External Support in Civil War Termination," Journal of Conflict Resolution, Peace Science Society (International), vol. 61(6), pages 1174-1202, July.
    11. Stavros Petrou & Oliver Rivero-Arias & Helen Dakin & Louise Longworth & Mark Oppe & Robert Froud & Alastair Gray, 2015. "Preferred Reporting Items for Studies Mapping onto Preference-Based Outcome Measures: The MAPS Statement," Medical Decision Making, , vol. 35(6), pages 1-8, August.
    12. Alexander P. L. Martindale & Carrie D. Llewellyn & Richard O. Visser & Benjamin Ng & Victoria Ngai & Aditya U. Kale & Lavinia Ferrante Ruffano & Robert M. Golub & Gary S. Collins & David Moher & Melis, 2024. "Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    13. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    14. Dossè Mawussi Djahini‐Afawoubo, 2024. "Understanding tax payment behaviour in the West African Economic and Monetary Union: The role of perceived detection capacity and honesty," Journal of International Development, John Wiley & Sons, Ltd., vol. 36(2), pages 795-823, March.
    15. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    16. Arthur Charpentier & Emmanuel Flachaire & Antoine Ly, 2017. "Econom\'etrie et Machine Learning," Papers 1708.06992, arXiv.org, revised Mar 2018.
    17. Deshpande, Ashwini & Desrochers, Alain & Ksoll, Christopher & Shonchoy, Abu S., 2017. "The Impact of a Computer-based Adult Literacy Program on Literacy and Numeracy: Evidence from India," World Development, Elsevier, vol. 96(C), pages 451-473.
    18. Maria Giné-Garriga & Carme Martin-Borràs & Anna Puig-Ribera & Carlos Martín-Cantera & Mercè Solà & Antonio Cuesta-Vargas & on behalf of the PPAF Group, 2013. "The Effect of a Physical Activity Program on the Total Number of Primary Care Visits in Inactive Patients: A 15-Month Randomized Controlled Trial," PLOS ONE, Public Library of Science, vol. 8(6), pages 1-8, June.
    19. Friberg, Richard & Romahn, André, 2015. "Divestiture requirements as a tool for competition policy: A case from the Swedish beer market," International Journal of Industrial Organization, Elsevier, vol. 42(C), pages 1-18.
    20. Nahid Norouzi-Gheidari & Alejandro Hernandez & Philippe S. Archambault & Johanne Higgins & Lise Poissant & Dahlia Kairy, 2019. "Feasibility, Safety and Efficacy of a Virtual Reality Exergame System to Supplement Upper Extremity Rehabilitation Post-Stroke: A Pilot Randomized Clinical Trial and Proof of Principle," IJERPH, MDPI, vol. 17(1), pages 1-11, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:3:p:212-239. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.