IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0211884.html
   My bibliography  Save this article

Situational Judgment Tests as a method for measuring personality: Development and validity evidence for a test of Dependability

Author

Listed:
  • Gabriel Olaru
  • Jeremy Burrus
  • Carolyn MacCann
  • Franklin M Zaromb
  • Oliver Wilhelm
  • Richard D Roberts

Abstract

Situational Judgment Tests (SJTs) are criterion valid low fidelity measures that have gained much popularity as predictors of job performance. A broad variety of SJTs have been studied, but SJTs measuring personality are still rare. Personality traits such as Conscientiousness are valid predictors of many educational, work and life-related outcomes and SJTs are less prone to faking than classical self-report measurements. We developed an SJT measure of Dependability, a core facet of Conscientiousness, by gathering critical incidents in semi-structured interviews using the construct definition of Dependability as a prompt. We examined the psychometric properties of the newly developed SJTs across two studies (N = 546 general population; N = 440 sales professionals). The internal validity of the SJTs was examined by correlating the SJT scores with related self-report measures of Dependability and Conscientiousness, as well as testing the unidimensionality of the measure with CFA. Additionally, we specified a bi-factor model of SJT, self-report and behavioral checklist measures of Dependability accounting for common and specific measurement variance. External validity was examined by correlating the SJT scale and specific factor with work-related outcomes. The results show that the Dependability SJTs with an expert based scoring procedure were psychometrically sound and correlated moderately to highly with traditional self-report measures of Dependability and Conscientiousness. However, a large proportion of SJT variance cannot be accounted for by personality alone. This supports the notion that SJTs measure general domain knowledge about the effectiveness of personality-related behaviors. We conclude that SJT measures of personality can be a promising addition to classical self-report assessments and can be used in a wide variety of applications beyond measurement and selection, for instance as formative assessments of personality.

Suggested Citation

  • Gabriel Olaru & Jeremy Burrus & Carolyn MacCann & Franklin M Zaromb & Oliver Wilhelm & Richard D Roberts, 2019. "Situational Judgment Tests as a method for measuring personality: Development and validity evidence for a test of Dependability," PLOS ONE, Public Library of Science, vol. 14(2), pages 1-19, February.
    • Handle: RePEc:plo:pone00:0211884
    DOI: 10.1371/journal.pone.0211884
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0211884
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0211884&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0211884?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    2. Denny Borsboom, 2006. "The attack of the psychometricians," Psychometrika, Springer;The Psychometric Society, vol. 71(3), pages 425-440, September.
    3. Lievens, Filip & Motowidlo, Stephan J., 2016. "Situational Judgment Tests: From Measures of Situational Judgment to Measures of General Domain Knowledge," Industrial and Organizational Psychology, Cambridge University Press, vol. 9(1), pages 3-22, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Steven Andrew Culpepper & Herman Aguinis & Justin L. Kern & Roger Millsap, 2019. "High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 285-309, March.
    2. Omar N. Solinger & Woody van Olffen & Robert A. Roe & Joeri Hofmans, 2013. "On Becoming (Un)Committed: A Taxonomy and Test of Newcomer Onboarding Scenarios," Organization Science, INFORMS, vol. 24(6), pages 1640-1661, December.
    3. Denny Borsboom, 2022. "Possible Futures for Network Psychometrics," Psychometrika, Springer;The Psychometric Society, vol. 87(1), pages 253-265, March.
    4. Anthony Clairmont & Andrew Maul, 2019. "A Review of “Sociocognitive Foundations of Educational Measurement” by Robert Mislevy," Psychometrika, Springer;The Psychometric Society, vol. 84(4), pages 1097-1100, December.
    5. Lisa D. Wijsen & Denny Borsboom, 2021. "Perspectives on Psychometrics Interviews with 20 Past Psychometric Society Presidents," Psychometrika, Springer;The Psychometric Society, vol. 86(1), pages 327-343, March.
    6. Martin Biehler & Heinz Holling & Philipp Doebler, 2015. "Saddlepoint Approximations of the Distribution of the Person Parameter in the Two Parameter Logistic Model," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 665-688, September.
    7. Lisa D. Wijsen & Denny Borsboom & Tiago Cabaço & Willem J. Heiser, 2019. "An Academic Genealogy of Psychometric Society Presidents," Psychometrika, Springer;The Psychometric Society, vol. 84(2), pages 562-588, June.
    8. Willem Heiser, 2006. "Measurement without Copper Instruments and Experiment without Complete Control," Psychometrika, Springer;The Psychometric Society, vol. 71(3), pages 457-461, September.
    9. Mark Wilson, 2013. "Seeking a Balance Between the Statistical and Scientific Elements in Psychometrics," Psychometrika, Springer;The Psychometric Society, vol. 78(2), pages 211-236, April.
    10. Mair, Patrick & Hatzinger, Reinhold, 2007. "Extended Rasch Modeling: The eRm Package for the Application of IRT Models in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 20(i09).
    11. Klaas Sijtsma, 2006. "Psychometrics in Psychological Research: Role Model or Partner in Science?," Psychometrika, Springer;The Psychometric Society, vol. 71(3), pages 451-455, September.
    12. Michael Kane, 2006. "In Praise of Pluralism. A Comment on Borsboom," Psychometrika, Springer;The Psychometric Society, vol. 71(3), pages 441-445, September.
    13. Klaas Sijtsma, 2016. "Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 1-15, March.
    14. Cadogan, John W. & Lee, Nick, 2013. "Improper use of endogenous formative variables," Journal of Business Research, Elsevier, vol. 66(2), pages 233-241.
    15. Myszkowski, Nils & Storme, Martin, 2018. "A snapshot of g? Binary and polytomous item-response theory investigations of the last series of the Standard Progressive Matrices (SPM-LS)," Intelligence, Elsevier, vol. 68(C), pages 109-116.
    16. César Viloria-Núñez & Marcela Tovar & Anthony Constant Millán, 2023. "Digital archetypes: a tool for understanding personality characteristics in the digital culture," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-14, December.
    17. Michael Wintroub, 2020. "Sordid genealogies: a conjectural history of Cambridge Analytica’s eugenic roots," Palgrave Communications, Palgrave Macmillan, vol. 7(1), pages 1-16, December.
    18. Pascal Jordan & Meike C Shedden-Mora & Bernd Löwe, 2017. "Psychometric analysis of the Generalized Anxiety Disorder scale (GAD-7) in primary care using modern item response theory," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-14, August.
    19. Leann Schneider & Ulrich Schimmack, 2009. "Self-Informant Agreement in Well-Being Ratings: A Meta-Analysis," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(3), pages 363-376, December.
    20. repec:jss:jstsof:20:i09 is not listed on IDEAS
    21. Ulrich Schroeders & Oliver Wilhelm & Gabriel Olaru, 2016. "Meta-Heuristics in Short Scale Construction: Ant Colony Optimization and Genetic Algorithm," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-19, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0211884. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.