IDEAS home Printed from https://ideas.repec.org/p/diw/diwsop/diw_sp629.html
   My bibliography  Save this paper

Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior

Author

Listed:
  • Carsten Sauer
  • Katrin Auspurg
  • Thomas Hinz
  • Stefan Liebig
  • Jürgen Schupp

Abstract

This paper describes the implementation of a factorial survey within the SOEP-Pretest of 2008 and investigates (1) respondents' comments about the vignettes, (2) interviewers' assessments of respondents comprehension and willingness to answer, and (3) response behavior regarding response time, use of the answering scale, and consistency of evaluations by different age groups and educational background. The respondents evaluated 24 vignettes consisting of ten dimensions that described full-time employees and their gross earnings. The evaluation task was to assess whether the given earnings were just or unjust, and if they were rated as unjust, respondents had to specify the amount of injustice on an 100-point rating scale. In regard to respondents' comments, the critique mentioned most frequently by respondents referred to the content (unrealistic descriptions) and the number of the vignettes. The analysis of the interviewers' assessments of respondents' comprehension and willingness to answer revealed less comprehension and willingness to answer for older and less well educated respondents, although these differences were similar to those found for the complete questionnaire. The analysis of the response behavior revealed no di erences of response time between the groups. Analyses of response consistency show that one should consider hints for simplifying heuristics: such heuristics can lead to an artificially high response consistency. The implications of the findings are discussed.

Suggested Citation

  • Carsten Sauer & Katrin Auspurg & Thomas Hinz & Stefan Liebig & Jürgen Schupp, 2014. "Method Effects in Factorial Surveys: An Analysis of Respondents' Comments, Interviewers' Assessments, and Response Behavior," SOEPpapers on Multidisciplinary Panel Data Research 629, DIW Berlin, The German Socio-Economic Panel (SOEP).
    • Handle: RePEc:diw:diwsop:diw_sp629
    as

    Download full text from publisher

    File URL: https://www.diw.de/documents/publikationen/73/diw_01.c.438559.de/diw_sp0629.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Marcello Pagnini & Paola Rossi & Valerio Vacca, 2017. "Introduction," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 46(3), pages 409-410, November.
    2. Arie Kapteyn & James P. Smith & Arthur Van Soest, 2013. "Are A mericans Really Less Happy with Their Incomes?," Review of Income and Wealth, International Association for Research in Income and Wealth, vol. 59(1), pages 44-65, March.
    3. Guillermina Jasso, 2006. "Factorial Survey Methods for Studying Beliefs and Judgments," Sociological Methods & Research, , vol. 34(3), pages 334-423, February.
    4. Katrin Auspurg & Annette Jäckle, 2017. "First Equals Most Important? Order Effects in Vignette-Based Measurement," Sociological Methods & Research, , vol. 46(3), pages 490-539, August.
    5. Stefan Liebig & Jürgen Schupp, 2008. "Immer mehr Erwerbstätige empfinden ihr Einkommen als ungerecht," DIW Wochenbericht, DIW Berlin, German Institute for Economic Research, vol. 75(31), pages 434-440.
    6. Stefan Liebig & Jürgen Schupp, 2005. "Empfinden die Erwerbstätigen in Deutschland ihre Einkommen als gerecht?," DIW Wochenbericht, DIW Berlin, German Institute for Economic Research, vol. 72(48), pages 721-725.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Martin Kroczek & Jochen Späth, 2022. "The attractiveness of jobs in the German care sector: results of a factorial survey," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 23(9), pages 1547-1562, December.
    2. Katrin Auspurg & Annette Jäckle, 2017. "First Equals Most Important? Order Effects in Vignette-Based Measurement," Sociological Methods & Research, , vol. 46(3), pages 490-539, August.
    3. Blaine Robbins & Edgar Kiser, 2018. "Legitimate authorities and rational taxpayers: An investigation of voluntary compliance and method effects in a survey experiment of income tax evasion," Rationality and Society, , vol. 30(2), pages 247-301, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lulu P. Shi & Christian Imdorf & Robin Samuel & Stefan Sacchi, 2018. "How unemployment scarring affects skilled young workers: evidence from a factorial survey of Swiss recruiters," Journal for Labour Market Research, Springer;Institute for Employment Research/ Institut für Arbeitsmarkt- und Berufsforschung (IAB), vol. 52(1), pages 1-15, December.
    2. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    3. Fabian Hattke & Janne Kalucza, 2019. "What influences the willingness of citizens to coproduce public services? Results from a vignette experiment," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 2(1).
    4. Blaine Robbins & Edgar Kiser, 2018. "Legitimate authorities and rational taxpayers: An investigation of voluntary compliance and method effects in a survey experiment of income tax evasion," Rationality and Society, , vol. 30(2), pages 247-301, May.
    5. James A. Chalfant & Richard S. Gray & Kenneth J. White, 1991. "Evaluating Prior Beliefs in a Demand System: The Case of Meat Demand in Canada," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 73(2), pages 476-490.
    6. William H. Greene & Mark N. Harris & Rachel J. Knott & Nigel Rice, 2021. "Specification and testing of hierarchical ordered response models with anchoring vignettes," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(1), pages 31-64, January.
    7. Feldmann, Marcel, 2010. "A Factorial Survey on Fair Leadership Behavior and the Role of Superiors," MPRA Paper 26009, University Library of Munich, Germany.
    8. Wietzke, Frank-Borge, 2024. "Perceptions of social class in Africa. Results from a conjoint experiment," World Development, Elsevier, vol. 178(C).
    9. Knott, R. & Lorgelly, P. & Black, N. & Hollingsworth, B., 2016. "Differential item functioning in the EQ-5D: An exploratory analysis using anchoring vignettes," Health, Econometrics and Data Group (HEDG) Working Papers 16/14, HEDG, c/o Department of Economics, University of York.
    10. Frodermann, Corinna & Auspurg, Katrin & Hinz, Thomas & Bähr, Sebastian & Abraham, Martin & Gundert, Stefanie & Bethmann, Arne, 2013. "Das Faktorielle Survey-Modul zur Stellenannahmebereitschaft im PASS : 5. Erhebungswelle (2011)," FDZ Methodenreport 201305_de, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    11. van Dalen, H.P. & Henkens, K. & Koedijk, C.G. & Slager, A.M.H., 2010. "Decision Making in the Pension Fund Board Room : An Experiment with Dutch Pension Fund Trustees," Discussion Paper 2010-18, Tilburg University, Center for Economic Research.
    12. Guillermina Jasso, 2012. "Safeguarding Justice Research," Sociological Methods & Research, , vol. 41(1), pages 217-239, February.
    13. Kim, Sehoon & Connerton, Timothy Paul & Park, Cheongyeul, 2022. "Transforming the automotive retail: Drivers for customers' omnichannel BOPS (Buy Online & Pick up in Store) behavior," Journal of Business Research, Elsevier, vol. 139(C), pages 411-425.
    14. van Dalen, H.P. & Henkens, K., 2015. "Why Demotion of Older Workers is a No-Go Area for Managers," Other publications TiSEM cef69d5e-bcc2-4082-b9fa-0, Tilburg University, School of Economics and Management.
    15. Kapteyn, Arie & Wah, Saw Htay, 2016. "Challenges to small and medium-size businesses in Myanmar: What are they and how do we know?," Journal of Asian Economics, Elsevier, vol. 47(C), pages 1-22.
    16. Martin Abraham & Natascha Nisic, 2012. "A simple mobility game for couples’ migration decisions and some quasi-experimental evidence1," Rationality and Society, , vol. 24(2), pages 168-197, May.
    17. Niels Vermeer & Maarten Rooij & Daniel Vuuren, 2019. "Retirement Age Preferences: The Role of Social Interactions and Anchoring at the Statutory Retirement Age," De Economist, Springer, vol. 167(4), pages 307-345, December.
    18. O'Donnell, Gus & Oswald, Andrew J., 2015. "National well-being policy and a weighted approach to human feelings," Ecological Economics, Elsevier, vol. 120(C), pages 59-70.
    19. Di Falco, Salvatore & Sharma, Sindra, 2018. "Investing in Climate Change Adaptation: Motivations and Green Incentives in the Fiji Islands," Ecological Economics, Elsevier, vol. 154(C), pages 394-408.
    20. Valentine, Nicole & Verdes-Tennant, Emese & Bonsel, Gouke, 2015. "Health systems' responsiveness and reporting behaviour: Multilevel analysis of the influence of individual-level factors in 64 countries," Social Science & Medicine, Elsevier, vol. 138(C), pages 152-160.

    More about this item

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • D63 - Microeconomics - - Welfare Economics - - - Equity, Justice, Inequality, and Other Normative Criteria and Measurement
    • J31 - Labor and Demographic Economics - - Wages, Compensation, and Labor Costs - - - Wage Level and Structure; Wage Differentials

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:diw:diwsop:diw_sp629. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Bibliothek (email available below). General contact details of provider: https://edirc.repec.org/data/sodiwde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.