IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v36y2012i4p243-271.html
   My bibliography  Save this article

Measuring Stakeholder Participation in Evaluation

Author

Listed:
  • Pierre-Marc Daigneault
  • Steve Jacob
  • Joël Tremblay

Abstract

Background: Stakeholder participation is an important trend in the field of program evaluation. Although a few measurement instruments have been proposed, they either have not been empirically validated or do not cover the full content of the concept. Objectives: This study consists of a first empirical validation of a measurement instrument that fully covers the content of participation, namely the Participatory Evaluation Measurement Instrument (PEMI). It specifically examines (1) the intercoder reliability of scores derived by two research assistants on published evaluation cases; (2) the convergence between the scores of coders and those of key respondents (i.e., authors); and (3) the convergence between the authors’ scores on the PEMI and the Evaluation Involvement Scale (EIS). Sample: A purposive sample of 40 cases drawn from the evaluation literature was used to assess reliability. One author per case in this sample was then invited to participate in a survey; 25 fully usable questionnaires were received. Measures: Stakeholder participation was measured on nominal and ordinal scales. Cohen’s κ, the intraclass correlation coefficient, and Spearman’s Ï were used to assess reliability and convergence. Results: Reliability results ranged from fair to excellent. Convergence between coders’ and authors’ scores ranged from poor to good. Scores derived from the PEMI and the EIS were moderately associated. Conclusions: Evidence from this study is strong in the case of intercoder reliability and ranges from weak to strong in the case of convergent validation. Globally, this suggests that the PEMI can produce scores that are both reliable and valid.

Suggested Citation

  • Pierre-Marc Daigneault & Steve Jacob & Joël Tremblay, 2012. "Measuring Stakeholder Participation in Evaluation," Evaluation Review, , vol. 36(4), pages 243-271, August.
    • Handle: RePEc:sae:evarev:v:36:y:2012:i:4:p:243-271
    DOI: 10.1177/0193841X12458103
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X12458103
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X12458103?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jacob, Steve & Ouvrard, Laurence & Bélanger, Jean-François, 2011. "Participatory evaluation and process use within a social aid organization for at-risk families and youth," Evaluation and Program Planning, Elsevier, vol. 34(2), pages 113-123, May.
    2. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    3. Connors, Susan C. & Magilvy, Joan K., 2011. "Assessing vital signs: Applying two participatory evaluation frameworks to the evaluation of a college of nursing," Evaluation and Program Planning, Elsevier, vol. 34(2), pages 79-86, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schakel, Arjan Hille, 2009. "A Postfunctionalist Theory of Regional Government," MPRA Paper 21596, University Library of Munich, Germany.
    2. Zim Nwokora & Riccardo Pelizzo, 2017. "Measuring Party System Change: A Systems Perspective," Research Africa Network Working Papers 17/048, Research Africa Network (RAN).
    3. J. C. Sharman, 2007. "Rationalist and Constructivist Perspectives on Reputation," Political Studies, Political Studies Association, vol. 55(1), pages 20-37, March.
    4. Muhammad Nabeel Siddiqui, 2013. "Impact Of Work Life Conflict On Employee Performance," Far East Journal of Psychology and Business, Far East Research Centre, vol. 12(3), pages 26-40, September.
    5. Jose Antonio Alonso & Ana Luiza Cortez & Stephan Klasen, 2014. "LDC and other country groupings: How useful are current approaches to classify countries in a more hetergeneous developing world?," CDP Background Papers 021, United Nations, Department of Economics and Social Affairs.
    6. Willem E. Saris & André Pirralha & Diana Zavala-Rojas, 2018. "Testing the Comparability of Different Types of Social Indicators Across Groups," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 135(3), pages 927-939, February.
    7. Anete Veidemane & Frans Kaiser & Daniela Craciun, 2021. "Inclusive Higher Education Access for Underrepresented Groups: It Matters, But How Can Universities Measure It?," Social Inclusion, Cogitatio Press, vol. 9(3), pages 44-57.
    8. Maria J. Debre & Hylke Dijkstra, 2023. "Are international organisations in decline? An absolute and relative perspective on institutional change," Global Policy, London School of Economics and Political Science, vol. 14(1), pages 16-30, February.
    9. Todd Landman, 2018. "Democracy and Human Rights: Concepts, Measures, and Relationships," Politics and Governance, Cogitatio Press, vol. 6(1), pages 48-59.
    10. Pierre-Marc Daigneault & Dominic Duval & Louis M. Imbeau, 2018. "Supervised scaling of semi-structured interview transcripts to characterize the ideology of a social policy reform," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(5), pages 2151-2162, September.
    11. von Soest, Christian & Wahman, Michael, 2013. "Sanctions and Democratization in the Post-Cold War Era," GIGA Working Papers 212, GIGA German Institute of Global and Area Studies.
    12. Santiago Leal Paredes & Jaime O. Salomon & Jaime Rivera Camino, 2021. "Impact of Authentic Leadership on Work Engagement and Organizational Citizenship Behavior: The Meditating Role of Motivation for Work," International Journal of Economics & Business Administration (IJEBA), International Journal of Economics & Business Administration (IJEBA), vol. 0(3), pages 3-31.
    13. Achim Goerres & Katrin Prinzen, 2012. "Using mixed methods for the analysis of individuals: a review of necessary and sufficient conditions and an application to welfare state attitudes," Quality & Quantity: International Journal of Methodology, Springer, vol. 46(2), pages 415-450, February.
    14. Christensen, Robert K. & Szmer, John, 2012. "Examining the efficiency of the U.S. courts of appeals: Pathologies and prescriptions," International Review of Law and Economics, Elsevier, vol. 32(1), pages 30-37.
    15. Seraphine F. Maerz & Carsten Q. Schneider, 2020. "Comparing public communication in democracies and autocracies: automated text analyses of speeches by heads of government," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 517-545, April.
    16. von Soest, Christian & Wahman, Michael, 2013. "Are All Dictators Equal? The Selective Targeting of Democratic Sanctions against Authoritarian Regimes," GIGA Working Papers 230, GIGA German Institute of Global and Area Studies.
    17. Rachel M. Gisselquist, 2012. "Good Governance as a Concept, and Why This Matters for Development Policy," WIDER Working Paper Series wp-2012-030, World Institute for Development Economic Research (UNU-WIDER).
    18. Paolo Dardanelli, 2019. "Conceptualizing, Measuring, and Mapping State Structures—with an Application to Western Europe, 1950–2015," Publius: The Journal of Federalism, CSF Associates Inc., vol. 49(2), pages 271-298.
    19. Liesbet Hooghe & Gary Marks, 2012. "Beyond Federalism - Estimating and Explaining the Territorial Structure of Government," KFG Working Papers p0037, Free University Berlin.
    20. Christopher J Fariss & James Lo, 2020. "Innovations in concepts and measurement for the study of peace and conflict," Journal of Peace Research, Peace Research Institute Oslo, vol. 57(6), pages 669-678, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:36:y:2012:i:4:p:243-271. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.