IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0265312.html
   My bibliography  Save this article

Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies

Author

Listed:
  • Nancy Elizabeth Doyle
  • Almuth McDowall
  • Raymond Randall
  • Kate Knight

Abstract

The evaluation of applied psychological interventions in the workplace or elsewhere is challenging. Randomisation and matching are difficult to achieve and this often results in substantial heterogeneity within intervention and control groups. As a result, traditional comparison of group means using null hypothesis significance testing may mask effects experienced by some participants. Using longitudinal studies of coaching interventions designed to provide support for dyslexic employees, this study describes and evaluates a different approach using a Meta-Impact score. We offer a conceptual rationale for our method, illustrate how this score is calculated and analysed, and show how it highlights person-specific variations in how participants react and respond to interventions. We argue that Meta-Impact is an incremental supplement to traditional variable-centric group-wise comparisons and can more accurately demonstrate in practice the extent to which an intervention worked. Such methods are needed for applied research, where personalized intervention protocols may require impact analysis for policy, legal and ethical purposes, despite modest sample sizes.

Suggested Citation

  • Nancy Elizabeth Doyle & Almuth McDowall & Raymond Randall & Kate Knight, 2022. "Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies," PLOS ONE, Public Library of Science, vol. 17(3), pages 1-21, March.
    • Handle: RePEc:plo:pone00:0265312
    DOI: 10.1371/journal.pone.0265312
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0265312
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0265312&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0265312?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hoenig J. M. & Heisey D. M., 2001. "The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis," The American Statistician, American Statistical Association, vol. 55, pages 19-24, February.
    2. Özbilgin, Mustafa F. & Beauregard, T. A. & Tatli, Ahu & Bell, Myrtle P., 2011. "Work-life, diversity and intersectionality: a critical review and research agenda," LSE Research Online Documents on Economics 36557, London School of Economics and Political Science, LSE Library.
    3. Briner, Rob B. & Walshe, Neil D., 2013. "The Causes and Consequences of a Scientific Literature We Cannot Trust: An Evidence-Based Practice Perspective," Industrial and Organizational Psychology, Cambridge University Press, vol. 6(3), pages 269-272, September.
    4. Santuzzi, Alecia M. & Waltz, Pamela R. & Finkelstein, Lisa M. & Rupp, Deborah E., 2014. "Invisible Disabilities: Unique Challenges for Employees and Organizations," Industrial and Organizational Psychology, Cambridge University Press, vol. 7(2), pages 204-219, June.
    5. Matthew C. Saleh & Susanne M. Bruyère, 2018. "Leveraging Employer Practices in Global Regulatory Frameworks to Improve Employment Outcomes for People with Disabilities," Social Inclusion, Cogitatio Press, vol. 6(1), pages 18-28.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nancy Elizabeth Doyle & Almuth McDowall, 2019. "Context matters: A review to formulate a conceptual framework for coaching as a disability accommodation," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-30, August.
    2. Lavanya Vijayasingham & Uma Jogulu & Pascale Allotey, 2018. "Enriching the Organizational Context of Chronic Illness Experience Through an Ethics of Care Perspective," Journal of Business Ethics, Springer, vol. 153(1), pages 29-40, November.
    3. Siti Khadijah Zainal, 2013. "Job Characteristics as the Antecedents of Work-to-family Enrichment: A Literature Review," Journal of Social and Development Sciences, AMH International, vol. 4(8), pages 394-401.
    4. Jeffrey C. Valentine & Therese D. Pigott & Hannah R. Rothstein, 2010. "How Many Studies Do You Need?," Journal of Educational and Behavioral Statistics, , vol. 35(2), pages 215-247, April.
    5. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    6. Fatima Malik, 2024. "UK Employed Women’s Experiences of Role-control Nuances (Decision Choice; Emotive Willingness) and Emotional-Experience Around Conflicting Work & Life," SAGE Open, , vol. 14(3), pages 21582440241, August.
    7. Samantha Evans & Madeleine Wyatt, 2023. "A Bridge over Troubled Borders: Social Class and the Interplay between Work and Life," Work, Employment & Society, British Sociological Association, vol. 37(1), pages 137-156, February.
    8. Nigel Golden & Kadambari Devarajan & Cathleen Balantic & Joseph Drake & Michael T Hallworth & Toni Lyn Morelli, 2021. "Ten simple rules for productive lab meetings," PLOS Computational Biology, Public Library of Science, vol. 17(5), pages 1-13, May.
    9. Asseburg, Julia, 2018. "Work-family conflict in the public sector: The impact of public service motivation and job crafting," EconStor Preprints 183185, ZBW - Leibniz Information Centre for Economics.
    10. repec:cup:judgdm:v:2:y:2007:i::p:317-325 is not listed on IDEAS
    11. Edna Rabenu & Aharon Tziner, 2016. "Selection of Employees with Disabilities – Has the Burden on the Employer Become too Heavy?," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 18(42), pages 423-423, May.
    12. Michal Ovádek, 2019. "The apolitical lawyer: experimental evidence of a framing effect," European Journal of Law and Economics, Springer, vol. 48(3), pages 385-415, December.
    13. Anna Carreri, 2020. "Control on the ‘Boundary-Work’ in Work-Life Articulation for Flexible Knowledge Workers. Insights into Gender Asymmetries," Social Sciences, MDPI, vol. 9(6), pages 1-15, June.
    14. Johanna Catherine Maclean & John Buckell, 2021. "Information and sin goods: Experimental evidence on cigarettes," Health Economics, John Wiley & Sons, Ltd., vol. 30(2), pages 289-310, February.
    15. Krystal Wilkinson & Jennifer Tomlinson & Jean Gardiner, 2017. "Exploring the work–life challenges and dilemmas faced by managers and professionals who live alone," Work, Employment & Society, British Sociological Association, vol. 31(4), pages 640-656, August.
    16. Tom Coupé & W. Robert Reed, 2021. "Do Negative Replications Affect Citations?," Working Papers in Economics 21/14, University of Canterbury, Department of Economics and Finance.
    17. Mary Wickenden & Philip Mader & Stephen Thompson & Jackie Shaw, 2022. "Mainstreaming disability inclusive employment in international development," Journal of International Development, John Wiley & Sons, Ltd., vol. 34(5), pages 933-941, July.
    18. Weili Ding, 2020. "Laboratory experiments can pre-design to address power and selection issues," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 6(2), pages 125-138, December.
    19. Jiarui Tian, 2021. "A Replication of “The effect of the conservation reserve program on rural economies: Deriving a statistical verdict from a null finding” (American Journal of Agricultural Economics, 2019)," Working Papers in Economics 21/12, University of Canterbury, Department of Economics and Finance.
    20. Thomas Aichner, 2021. "The economic argument for hiring people with disabilities," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    21. Jason W. Beckstead, 2007. "A note on determining the number of cues used in judgment analysis studies: The issue of type II error," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 2, pages 317-325, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0265312. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.