IDEAS home Printed from https://ideas.repec.org/p/osf/socarx/7ady6.html
   My bibliography  Save this paper

Estimators for Topic-Sampling Designs

Author

Listed:
  • Clifford, Scott
  • Rainey, Carlisle

Abstract

When researchers design an experiment, they must usually fix important details, or what we call the “topic” of the experiment. For example, researchers studying the impact of party cues on attitudes must inform respondents of the parties’ positions on a particular policy. In doing so, the researchers implement just one of many possible designs. Clifford, Leeper, and Rainey (2023) argue that researchers should implement many of the possible designs in parallel—what they call “topic sampling”—to generalize to a larger population of topics. We describe two estimators for topic-sampling designs. First, we describe a nonparametric estimator of the typical effect that is unbiased under the assumptions of the design. Second, we describe a hierarchical model that researchers can use to describe the heterogeneity. We suggest describing the variation in three ways: (1) the standard deviation in treatment effects across topics, (2) the treatment effects for particular topics, and (3) perhaps how the treatment effects for particular topics vary with topic-level predictors. We evaluate the performance of the hierarchical model using the Strengthening Democracy Challenge megastudy and show that the hierarchical model works well.

Suggested Citation

  • Clifford, Scott & Rainey, Carlisle, 2023. "Estimators for Topic-Sampling Designs," SocArXiv 7ady6, Center for Open Science.
  • Handle: RePEc:osf:socarx:7ady6
    DOI: 10.31219/osf.io/7ady6
    as

    Download full text from publisher

    File URL: https://osf.io/download/64d4c0c74f0a445c60994447/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/7ady6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ben M. Tappin & Adam J. Berinsky & David G. Rand, 2023. "Partisans’ receptivity to persuasive messaging is undiminished by countervailing party leader cues," Nature Human Behaviour, Nature, vol. 7(4), pages 568-582, April.
    2. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    3. Voelkel, Jan G. & Stagnaro, Michael & Chu, James & Pink, Sophia Lerner & Mernyk, Joseph S. & Redekopp, Chrystal & Ghezae, Isaias & Cashman, Matthew & Adjodah, Dhaval & Allen, Levi, 2023. "Megastudy identifying effective interventions to strengthen Americans’ democratic attitudes," OSF Preprints y79u5, Center for Open Science.
    4. Hainmueller, Jens & Hopkins, Daniel J. & Yamamoto, Teppei, 2014. "Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments," Political Analysis, Cambridge University Press, vol. 22(1), pages 1-30, January.
    5. Clifford, Scott & Sheagley, Geoffrey & Piston, Spencer, 2021. "Increasing Precision without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments," American Political Science Review, Cambridge University Press, vol. 115(3), pages 1048-1065, August.
    6. Yeojin Chung & Sophia Rabe-Hesketh & Vincent Dorie & Andrew Gelman & Jingchen Liu, 2013. "A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 685-709, October.
    7. Coppock, Alexander & Green, Donald P., 2015. "Assessing the Correspondence between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research," Political Science Research and Methods, Cambridge University Press, vol. 3(1), pages 113-131, January.
    8. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Felix Bader & Bastian Baumeister & Roger Berger & Marc Keuschnigg, 2021. "On the Transportability of Laboratory Results," Sociological Methods & Research, , vol. 50(3), pages 1452-1481, August.
    2. Aaron R Kaufman, 2020. "Implementing novel, flexible, and powerful survey designs in R Shiny," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-15, April.
    3. Michael J. Nelson & Christopher Witko, 2020. "Government reputational effects of COVID-19 public health actions: A job opportunity evaluation conjoint experiment," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 3(1).
    4. Lala Muradova & Ross James Gildea, 2021. "Oil wealth and US public support for war," Conflict Management and Peace Science, Peace Science Society (International), vol. 38(1), pages 3-19, January.
    5. Dustin Tingley, 2014. "Survey Research in International Political Economy: Motivations, Designs, Methods," International Interactions, Taylor & Francis Journals, vol. 40(3), pages 443-451, May.
    6. Tobia Spampatti & Ulf J. J. Hahnel & Evelina Trutnevyte & Tobias Brosch, 2024. "Psychological inoculation strategies to fight climate disinformation across 12 countries," Nature Human Behaviour, Nature, vol. 8(2), pages 380-398, February.
    7. Ozer, Adam & Sullivan, Brian & Van, Douglas, 2022. "Viewed from different Engels? Differences in reactions to “socialism” as a policy label," LSE Research Online Documents on Economics 111447, London School of Economics and Political Science, LSE Library.
    8. Raman, Shyam & Kriner, Douglas & Ziebarth, Nicolas & Simon, Kosali & Kreps, Sarah, 2022. "COVID-19 booster uptake among US adults: Assessing the impact of vaccine attributes, incentives, and context in a choice-based experiment," Social Science & Medicine, Elsevier, vol. 310(C).
    9. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    10. Matthew Amengual & Rita Mota & Alexander Rustler, 2023. "The ‘Court of Public Opinion:’ Public Perceptions of Business Involvement in Human Rights Violations," Journal of Business Ethics, Springer, vol. 185(1), pages 49-74, June.
    11. Lars Lenze & Claudia Klostermann & Markus Lamprecht & Siegfried Nagel, 2021. "Taking Up and Terminating Leisure-Time Physical Activity over the Life Course: The Role of Life Events in the Familial and Occupational Life Domains," IJERPH, MDPI, vol. 18(18), pages 1-15, September.
    12. Steffen Nestler & Edgar Erdfelder, 2023. "Random Effects Multinomial Processing Tree Models: A Maximum Likelihood Approach," Psychometrika, Springer;The Psychometric Society, vol. 88(3), pages 809-829, September.
    13. Pan, Jing Yu & Liu, Dahai, 2022. "Mask-wearing intentions on airplanes during COVID-19 – Application of theory of planned behavior model," Transport Policy, Elsevier, vol. 119(C), pages 32-44.
    14. JANSSENS, Jochen & DE CORTE, Annelies & SÖRENSEN, Kenneth, 2016. "Water distribution network design optimisation with respect to reliability," Working Papers 2016007, University of Antwerp, Faculty of Business and Economics.
    15. Chaikaew, Pasicha & Hodges, Alan W. & Grunwald, Sabine, 2017. "Estimating the value of ecosystem services in a mixed-use watershed: A choice experiment approach," Ecosystem Services, Elsevier, vol. 23(C), pages 228-237.
    16. Raymond Hernandez & Elizabeth A. Pyatak & Cheryl L. P. Vigen & Haomiao Jin & Stefan Schneider & Donna Spruijt-Metz & Shawn C. Roll, 2021. "Understanding Worker Well-Being Relative to High-Workload and Recovery Activities across a Whole Day: Pilot Testing an Ecological Momentary Assessment Technique," IJERPH, MDPI, vol. 18(19), pages 1-17, October.
    17. Christopher Hassall & Michael Nisbet & Evan Norcliffe & He Wang, 2024. "The Potential Health Benefits of Urban Tree Planting Suggested through Immersive Environments," Land, MDPI, vol. 13(3), pages 1-12, February.
    18. Michele Cantarella & Chiara Strozzi, 2021. "Workers in the crowd: the labor market impact of the online platform economy [An evaluation of instrumental variable strategies for estimating the effects of catholic schooling]," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 30(6), pages 1429-1458.
    19. Jie Zhao & Ji Chen & Damien Beillouin & Hans Lambers & Yadong Yang & Pete Smith & Zhaohai Zeng & Jørgen E. Olesen & Huadong Zang, 2022. "Global systematic review with meta-analysis reveals yield advantage of legume-based rotations and its drivers," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    20. Robbett, Andrea & Matthews, Peter Hans, 2018. "Partisan bias and expressive voting," Journal of Public Economics, Elsevier, vol. 157(C), pages 107-120.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:socarx:7ady6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://arabixiv.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.