IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v7y2013i1p138-144.html
   My bibliography  Save this article

First in, best dressed: The presence of order-effect bias in journal ranking surveys

Author

Listed:
  • Serenko, Alexander
  • Bontis, Nick

Abstract

The purpose of this study is to test for the presence of order-effect bias in journal ranking surveys. Data were obtained from 379 active knowledge management and intellectual capital researchers who rated 25 journals on a 7-point scale. Five different versions of the survey instrument were utilized. Consistent with the cognitive elaboration model, the satisficing theory, and the Gricean maxim of orderliness, order-effect bias was observed in journal ranking surveys. Journals that appear in the beginning of the ranking list delivered to survey respondents consistently receive higher scores than journals at the end of the list. Overall, the position of the journal in the list explains over 10% of its score. Therefore, authors of journal ranking studies are recommended to use multiple versions of the survey instrument with randomized journal orders.

Suggested Citation

  • Serenko, Alexander & Bontis, Nick, 2013. "First in, best dressed: The presence of order-effect bias in journal ranking surveys," Journal of Informetrics, Elsevier, vol. 7(1), pages 138-144.
    • Handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144
    DOI: 10.1016/j.joi.2012.10.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157712000879
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2012.10.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    2. Julian Diaz & Roy T. Black & Joseph Rabianski, 1996. "A Note on Ranking Real Estate Research Journals," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 24(4), pages 551-563, December.
    3. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    4. Clyde W. Holsapple, 2008. "A publication power approach for identifying premier information systems journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(2), pages 166-185, January.
    5. Michael E. Whitman & Anthony R. Hendrickson & Anthony M. Townsend, 1999. "Research Commentary. Academic Rewards for Teaching, Research, and Service: Data and Discourse," Information Systems Research, INFORMS, vol. 10(2), pages 99-109, June.
    6. Blaise Cronin & Lokman I. Meho, 2008. "Applying the author affiliation index to library and information science journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1861-1865, September.
    7. Serenko, Alexander & Bontis, Nick, 2011. "What's familiar is excellent: The impact of exposure effect on perceived journal quality," Journal of Informetrics, Elsevier, vol. 5(1), pages 219-223.
    8. L. Egghe, 2010. "The distribution of the uncitedness factor and its functional relation with the impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 689-695, June.
    9. Israel, Glenn D. & Taylor, C. L., 1990. "Can response order bias evaluations?," Evaluation and Program Planning, Elsevier, vol. 13(4), pages 365-371, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mieko Fujisawa & Kazuhisa Takemura & Yukihiko Funaki & Nobuyuki Uto & Ryo Takahashi, 2020. "An Experimental Study of the Effect of Energy Label Design on the Correct Evaluation of Buildings' Energy Performance and Promotion of Energy Saving," International Real Estate Review, Asian Real Estate Society, vol. 23(1), pages 733-762.
    2. Domingo Docampo & Vicente Safón, 2021. "Journal ratings: a paper affiliation methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8063-8090, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    2. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    3. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    4. Kishen Iyengar & Venugopal Balijepally, 2015. "Ranking journals using the dominance hierarchy procedure: an illustration with IS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 5-23, January.
    5. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    6. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    7. Tim Klopries, 2018. "Discussion of “Working from Home—What is the Effect on Employees’ Effort?”," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 70(1), pages 57-62, February.
    8. Denter, Philipp, 2020. "Campaign contests," European Economic Review, Elsevier, vol. 127(C).
    9. Daniel E. O'Leary, 2009. "Business School Research: Measuring Value Contribution Through Citations of Journals in Patents," Interfaces, INFORMS, vol. 39(6), pages 516-526, December.
    10. Antonia Ferrer-Sapena & Susana Díaz-Novillo & Enrique A. Sánchez-Pérez, 2017. "Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p -Variations," Publications, MDPI, vol. 5(3), pages 1-14, September.
    11. A. Abrizah & A. Noorhidawati & A. N. Zainab, 2015. "LIS journals categorization in the Journal Citation Report: a stated preference study," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1083-1099, February.
    12. Claudio Vitari & Marc Humbert & Jean-Philippe Rennard, 2012. "Les spécificités de la communauté francophone d'enseignant-chercheurs en Système d'information en termes de prestige des revues et de publications," Grenoble Ecole de Management (Post-Print) halshs-01923286, HAL.
    13. Wallace J. Hopp, 2004. "Fifty Years of Management Science," Management Science, INFORMS, vol. 50(1), pages 1-7, January.
    14. Hu, Zewen & Wu, Yishan, 2014. "Regularity in the time-dependent distribution of the percentage of never-cited papers: An empirical pilot study based on the six journals," Journal of Informetrics, Elsevier, vol. 8(1), pages 136-146.
    15. Zewen Hu & Yishan Wu & Jianjun Sun, 2018. "A quantitative analysis of determinants of non-citation using a panel data model," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 843-861, August.
    16. Holsapple, Clyde W. & Lee-Post, Anita, 2010. "Behavior-based analysis of knowledge dissemination channels in operations management," Omega, Elsevier, vol. 38(3-4), pages 167-178, June.
    17. Chien Hsiang Liao & Mu-Yen Chen, 2018. "Exploring knowledge patterns of library and information science journals within the field: a citation analysis from 2009 to 2016," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1991-2008, December.
    18. Diarmuid B. Verrier, 2012. "Evidence for the influence of the mere-exposure effect on voting in the Eurovision Song Contest," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 7(5), pages 639-643, September.
    19. Gaël Brulé & Ruut Veenhoven, 2017. "The ‘10 Excess’ Phenomenon in Responses to Survey Questions on Happiness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 131(2), pages 853-870, March.
    20. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.