IDEAS home Printed from https://ideas.repec.org/p/osf/socarx/cy3wj.html
   My bibliography  Save this paper

Meta-dominance analysis - A tool for the assessment of the quality of digital behavioural data

Author

Listed:
  • Schneck, Andreas
  • Przepiorka, Wojtek

    (Utrecht University)

Abstract

We propose a simple yet comprehensive conceptual framework for the identification of different sources of error in research with digital behavioral data. We use our framework to map potential sources of error in 25 years of research on reputation effects in peer-to-peer online market platforms. Using a meta-dataset comprising 346 effect sizes extracted from 109 articles, we apply meta-dominance analysis to quantify the relative importance of different error components. Our results indicate that 85% of explained effect size heterogeneity can be attributed to the measurement process, which comprises the choice of platform, data collection mode, construct operationalisation and variable transformation. Error components attributable to the sampling process or publication bias capture relatively small parts of the explained effect size heterogeneity. This approach reveals at which stages of the research process researcher decisions may affect data quality most. This approach can be used to identify potential sources of error in established strands of research beyond the literature of behavioral data from online platforms.

Suggested Citation

  • Schneck, Andreas & Przepiorka, Wojtek, 2023. "Meta-dominance analysis - A tool for the assessment of the quality of digital behavioural data," SocArXiv cy3wj, Center for Open Science.
  • Handle: RePEc:osf:socarx:cy3wj
    DOI: 10.31219/osf.io/cy3wj
    as

    Download full text from publisher

    File URL: https://osf.io/download/640b992f86049007ac08681b/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/cy3wj?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ruohuang Jiao & Wojtek Przepiorka & Vincent Buskens, 2022. "Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach," Journal of Computational Social Science, Springer, vol. 5(1), pages 1041-1067, May.
    2. Janet M. Box-Steffensmeier & Jean Burgess & Maurizio Corbetta & Kate Crawford & Esther Duflo & Laurel Fogarty & Alison Gopnik & Sari Hanafi & Mario Herrero & Ying-yi Hong & Yasuko Kameyama & Tatia M. , 2022. "The future of human behaviour research," Nature Human Behaviour, Nature, vol. 6(1), pages 15-24, January.
    3. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    4. Bajari, Patrick & Hortacsu, Ali, 2003. "The Winner's Curse, Reserve Prices, and Endogenous Entry: Empirical Insights from eBay Auctions," RAND Journal of Economics, The RAND Corporation, vol. 34(2), pages 329-355, Summer.
    5. Joseph N. Luchman, 2021. "Determining relative importance in Stata using dominance analysis: domin and domme," Stata Journal, StataCorp LP, vol. 21(2), pages 510-538, June.
    6. Meyer, Bruce D. & Mittag, Nikolas, 2021. "An empirical total survey error decomposition using data combination," Journal of Econometrics, Elsevier, vol. 224(2), pages 286-305.
    7. Inglehart, Ronald F. & Norris, Pippa, 2016. "Trump, Brexit, and the Rise of Populism: Economic Have-Nots and Cultural Backlash," Working Paper Series 16-026, Harvard University, John F. Kennedy School of Government.
    8. Przepiorka, Wojtek, 2013. "Buyers pay for and sellers invest in a good reputation: More evidence from eBay," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 42(C), pages 31-42.
    9. Steven Tadelis, 2016. "Reputation and Feedback Systems in Online Platform Markets," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 321-340, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ruohuang Jiao & Wojtek Przepiorka & Vincent Buskens, 2022. "Moderators of reputation effects in peer-to-peer online markets: a meta-analytic model selection approach," Journal of Computational Social Science, Springer, vol. 5(1), pages 1041-1067, May.
    2. Rense Corten & Judith Kas & Timm Teubner & Martijn Arets, 2023. "The role of contextual and contentual signals for online trust: Evidence from a crowd work experiment," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-17, December.
    3. März, Armin & Lachner, Michael & Heumann, Christian G. & Schumann, Jan H. & von Wangenheim, Florian, 2021. "How You Remind Me! The Influence of Mobile Push Notifications on Success Rates in Last-Minute Bidding," Journal of Interactive Marketing, Elsevier, vol. 54(C), pages 11-24.
    4. Dani Rodrik, 2018. "Populism and the economics of globalization," Journal of International Business Policy, Palgrave Macmillan, vol. 1(1), pages 12-33, June.
    5. Rosalie L Tung & Günter K Stahl, 2018. "The tortuous evolution of the role of culture in IB research: What we know, what we don’t know, and where we are headed," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 49(9), pages 1167-1189, December.
    6. Indra de Soysa & Synøve Almås, 2019. "Does Ethnolinguistic Diversity Preclude Good Governance? A Comparative Study with Alternative Data, 1990‐2015," Kyklos, Wiley Blackwell, vol. 72(4), pages 604-636, November.
    7. Obsa Urgessa Ayana & Jima Degaga, 2022. "Effects of rural electrification on household welfare: a meta-regression analysis," International Review of Economics, Springer;Happiness Economics and Interpersonal Relations (HEIRS), vol. 69(2), pages 209-261, June.
    8. Tausch, Arno, 2018. "The return of religious Antisemitism? The evidence from World Values Survey data," MPRA Paper 90093, University Library of Munich, Germany.
    9. Vigvári, Gábor, 2022. "Transzformáció és a populizmus a visegrádi országokban [Transformation and populism in the V4 countries]," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(3), pages 339-366.
    10. Matthew Gentry & Tong Li & Jingfeng Lu, 2015. "Identification and estimation in first-price auctions with risk-averse bidders and selective entry," CeMMAP working papers CWP16/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    11. Hui, Xiang & Klein, Tobias & Stahl, Konrad, 2022. "Learning from Online Ratings," CEPR Discussion Papers 17006, C.E.P.R. Discussion Papers.
    12. Rasmusen Eric Bennett, 2006. "Strategic Implications of Uncertainty over One's Own Private Value in Auctions," The B.E. Journal of Theoretical Economics, De Gruyter, vol. 6(1), pages 1-24, November.
    13. Fidrmuc, Jarko & Lind, Ronja, 2020. "Macroeconomic impact of Basel III: Evidence from a meta-analysis," Journal of Banking & Finance, Elsevier, vol. 112(C).
    14. Benedict E. DeDominicis, 2021. "Multinational Enterprises And Economic Nationalism: A Strategic Analysis Of Culture," Global Journal of Business Research, The Institute for Business and Finance Research, vol. 15(1), pages 19-66.
    15. Tito Boeri & Prachi Mishra & Chris Papageorgiou & Antonio Spilimbergo, 2018. "A Dialogue between a Populist and an Economist," AEA Papers and Proceedings, American Economic Association, vol. 108, pages 191-195, May.
    16. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    17. Grégory Jolivet & Bruno Jullien & Fabien Postel-Vinay, 2014. "Reputation and Pricing on the e-Market: Evidence from a Major French Platform," SciencePo Working papers Main hal-03460312, HAL.
    18. Ichiro IWASAKI & Satoshi MIZOBATA, 2018. "Post-Privatization Ownership And Firm Performance: A Large Meta-Analysis Of The Transition Literature," Annals of Public and Cooperative Economics, Wiley Blackwell, vol. 89(2), pages 263-322, June.
    19. Genti Kostandini & Elton Mykerezi & Eftila Tanellari & Nour Dib, 2011. "Does Buyer Experience Pay Off? Evidence from eBay," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 39(3), pages 253-265, November.
    20. Gary Bolton & Kevin Breuer & Ben Greiner & Axel Ockenfels, 2023. "Fixing feedback revision rules in online markets," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 32(2), pages 247-256, April.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:socarx:cy3wj. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://arabixiv.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.