IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v41y2013icp19-30.html
   My bibliography  Save this article

Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care

Author

Listed:
  • Akin, Becci A.
  • Bryson, Stephanie A.
  • Testa, Mark F.
  • Blase, Karen A.
  • McDonald, Tom
  • Melz, Heidi

Abstract

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the project's initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.

Suggested Citation

  • Akin, Becci A. & Bryson, Stephanie A. & Testa, Mark F. & Blase, Karen A. & McDonald, Tom & Melz, Heidi, 2013. "Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care," Evaluation and Program Planning, Elsevier, vol. 41(C), pages 19-30.
    • Handle: RePEc:eee:epplan:v:41:y:2013:i:c:p:19-30
    DOI: 10.1016/j.evalprogplan.2013.06.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718913000578
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2013.06.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Dobson, Douglas & Cook, Thomas J., 1980. "Avoiding type III error in program evaluation : Results from a field experiment," Evaluation and Program Planning, Elsevier, vol. 3(4), pages 269-276, January.
    2. Schreiber, Jill C. & Fuller, Tamara & Paceley, Megan S., 2013. "Engagement in child protective services: Parent perceptions of worker skills," Children and Youth Services Review, Elsevier, vol. 35(4), pages 707-715.
    3. Courtney, Mark E. & Yin-Ling Irene Wong, 1996. "Comparing the timing of exits from substitute care," Children and Youth Services Review, Elsevier, vol. 18(4-5), pages 307-334.
    4. Connell, Christian M. & Katz, Karol H. & Saunders, Leon & Tebes, Jacob Kraemer, 2006. "Leaving foster care--the influence of child and case characteristics on foster care exit rates," Children and Youth Services Review, Elsevier, vol. 28(7), pages 780-798, July.
    5. W. Edwards Deming, 2000. "Out of the Crisis," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262541157, April.
    6. Akin, Becci A., 2011. "Predictors of foster care exits to permanency: A competing risks analysis of reunification, guardianship, and adoption," Children and Youth Services Review, Elsevier, vol. 33(6), pages 999-1011, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Akin, Becci A. & Byers, Kaela D. & Lloyd, Margaret H. & McDonald, Thomas P., 2015. "Joining formative evaluation with translational science to assess an EBI in foster care: Examining social–emotional well-being and placement stability," Children and Youth Services Review, Elsevier, vol. 58(C), pages 253-264.
    2. Akin, Becci A. & Lang, Kyle & Yan, Yueqi & McDonald, Thomas P., 2018. "Randomized trial of PMTO in foster care: 12-month child well-being, parenting, and caregiver functioning outcomes," Children and Youth Services Review, Elsevier, vol. 95(C), pages 49-63.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Davidson, Ryan D. & Tomlinson, Claire S. & Beck, Connie J. & Bowen, Anne M., 2019. "The revolving door of families in the child welfare system: Risk and protective factors associated with families returning," Children and Youth Services Review, Elsevier, vol. 100(C), pages 468-479.
    2. Shipe, Stacey L. & Shaw, Terry V. & Betsinger, Sara & Farrell, Jill L., 2017. "Expanding the conceptualization of re-entry: The inter-play between child welfare and juvenile services," Children and Youth Services Review, Elsevier, vol. 79(C), pages 256-262.
    3. Wulczyn, Fred & Gibbons, Robert & Snowden, Lonnie & Lery, Bridgette, 2013. "Poverty, social disadvantage, and the black/white placement gap," Children and Youth Services Review, Elsevier, vol. 35(1), pages 65-74.
    4. Lloyd, Margaret H. & Akin, Becci A. & Brook, Jody, 2017. "Parental drug use and permanency for young children in foster care: A competing risks analysis of reunification, guardianship, and adoption," Children and Youth Services Review, Elsevier, vol. 77(C), pages 177-187.
    5. Elizabeth Fernandez, 2014. "Child Protection and Vulnerable Families: Trends and Issues in the Australian Context," Social Sciences, MDPI, vol. 3(4), pages 1-24, October.
    6. Zinn, Andrew & Orlebeke, Britany, 2017. "Juvenile court judicial expertise and children's permanency outcomes," Children and Youth Services Review, Elsevier, vol. 77(C), pages 46-54.
    7. Foster, E. Michael & Hillemeier, Marianne M. & Bai, Yu, 2011. "Explaining the disparity in placement instability among African-American and white children in child welfare: A Blinder-Oaxaca decomposition," Children and Youth Services Review, Elsevier, vol. 33(1), pages 118-125, January.
    8. Connell, Christian M. & Bory, Christopher T. & Huang, Cindy Y. & Genovese, Maegan & Caron, Colleen & Tebes, Jacob Kraemer, 2019. "Caseworker assessment of child risk and functioning and their relation to service use in the child welfare system," Children and Youth Services Review, Elsevier, vol. 99(C), pages 81-86.
    9. Elgin, Dallas J., 2018. "Utilizing predictive modeling to enhance policy and practice through improved identification of at-risk clients: Predicting permanency for foster children," Children and Youth Services Review, Elsevier, vol. 91(C), pages 156-167.
    10. Malvaso, Catia G. & Delfabbro, Paul H., 2020. "Description and evaluation of a trial program aimed at reunifying adolescents in statutory long-term out-of-home care with their birth families: The adolescent reunification program," Children and Youth Services Review, Elsevier, vol. 119(C).
    11. Lin, Ching-Hsuan, 2014. "Evaluating Services for Kinship Care Families: A Systematic Review," Children and Youth Services Review, Elsevier, vol. 36(C), pages 32-41.
    12. Akin, Becci A., 2011. "Predictors of foster care exits to permanency: A competing risks analysis of reunification, guardianship, and adoption," Children and Youth Services Review, Elsevier, vol. 33(6), pages 999-1011, June.
    13. Zinn, Andrew & Cusick, Gretchen, 2014. "Juvenile court pathways to legal permanence for children in substitute care," Children and Youth Services Review, Elsevier, vol. 36(C), pages 90-100.
    14. Goemans, Anouk & Vanderfaeillie, Johan & Damen, Harm & Pijnenburg, Huub & Van Holen, Frank, 2016. "Reunification of foster children: Factors associated with reunification outcomes in Flanders and the Netherlands," Children and Youth Services Review, Elsevier, vol. 70(C), pages 284-292.
    15. Landers, Ashley L. & Danes, Sharon M. & Harstad, Jane & White Hawk, Sandy, 2017. "Finding their way home: Factors associated with reunification for American Indian and White adults," Children and Youth Services Review, Elsevier, vol. 82(C), pages 359-364.
    16. Haight, Wendy & Marshall, Jane & Woolman, Joanna, 2015. "The Child Protection Clinic: A mixed method evaluation of parent legal representation," Children and Youth Services Review, Elsevier, vol. 56(C), pages 7-17.
    17. Potter, Marina Haddock & Font, Sarah A., 2021. "State contexts and foster care adoption rates," Children and Youth Services Review, Elsevier, vol. 126(C).
    18. Landers, Ashley L. & Danes, Sharon M., 2016. "Forgotten children: A critical review of the reunification of American Indian children in the child welfare system," Children and Youth Services Review, Elsevier, vol. 71(C), pages 137-147.
    19. Lloyd, Margaret H. & Akin, Becci A., 2014. "The disparate impact of alcohol, methamphetamine, and other drugs on family reunification," Children and Youth Services Review, Elsevier, vol. 44(C), pages 72-81.
    20. Lindner, Abigail Rose & Hanlon, Ryan, 2024. "Outcomes of youth with foster care experiences based on permanency outcome – Adoption, aging out, long-term foster care, and reunification: A systematic review," Children and Youth Services Review, Elsevier, vol. 156(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:41:y:2013:i:c:p:19-30. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.