IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v7y2013i1p138-144.html
   My bibliography  Save this article

First in, best dressed: The presence of order-effect bias in journal ranking surveys

Author

Listed:
  • Serenko, Alexander
  • Bontis, Nick

Abstract

The purpose of this study is to test for the presence of order-effect bias in journal ranking surveys. Data were obtained from 379 active knowledge management and intellectual capital researchers who rated 25 journals on a 7-point scale. Five different versions of the survey instrument were utilized. Consistent with the cognitive elaboration model, the satisficing theory, and the Gricean maxim of orderliness, order-effect bias was observed in journal ranking surveys. Journals that appear in the beginning of the ranking list delivered to survey respondents consistently receive higher scores than journals at the end of the list. Overall, the position of the journal in the list explains over 10% of its score. Therefore, authors of journal ranking studies are recommended to use multiple versions of the survey instrument with randomized journal orders.

Suggested Citation

  • Serenko, Alexander & Bontis, Nick, 2013. "First in, best dressed: The presence of order-effect bias in journal ranking surveys," Journal of Informetrics, Elsevier, vol. 7(1), pages 138-144.
    • Handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144
    DOI: 10.1016/j.joi.2012.10.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157712000879
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2012.10.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    2. Julian Diaz & Roy T. Black & Joseph Rabianski, 1996. "A Note on Ranking Real Estate Research Journals," Real Estate Economics, American Real Estate and Urban Economics Association, vol. 24(4), pages 551-563, December.
    3. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    4. Serenko, Alexander & Bontis, Nick, 2011. "What's familiar is excellent: The impact of exposure effect on perceived journal quality," Journal of Informetrics, Elsevier, vol. 5(1), pages 219-223.
    5. Israel, Glenn D. & Taylor, C. L., 1990. "Can response order bias evaluations?," Evaluation and Program Planning, Elsevier, vol. 13(4), pages 365-371, January.
    6. Clyde W. Holsapple, 2008. "A publication power approach for identifying premier information systems journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(2), pages 166-185, January.
    7. Michael E. Whitman & Anthony R. Hendrickson & Anthony M. Townsend, 1999. "Research Commentary. Academic Rewards for Teaching, Research, and Service: Data and Discourse," Information Systems Research, INFORMS, vol. 10(2), pages 99-109, June.
    8. Blaise Cronin & Lokman I. Meho, 2008. "Applying the author affiliation index to library and information science journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(11), pages 1861-1865, September.
    9. L. Egghe, 2010. "The distribution of the uncitedness factor and its functional relation with the impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 689-695, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mieko Fujisawa & Kazuhisa Takemura & Yukihiko Funaki & Nobuyuki Uto & Ryo Takahashi, 2020. "An Experimental Study of the Effect of Energy Label Design on the Correct Evaluation of Buildings' Energy Performance and Promotion of Energy Saving," International Real Estate Review, Asian Real Estate Society, vol. 23(1), pages 733-762.
    2. Domingo Docampo & Vicente Safón, 2021. "Journal ratings: a paper affiliation methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8063-8090, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    2. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    3. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    4. Kishen Iyengar & Venugopal Balijepally, 2015. "Ranking journals using the dominance hierarchy procedure: an illustration with IS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 5-23, January.
    5. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    6. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    7. Tim Klopries, 2018. "Discussion of “Working from Home—What is the Effect on Employees’ Effort?”," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 70(1), pages 57-62, February.
    8. A. Abrizah & A. Noorhidawati & A. N. Zainab, 2015. "LIS journals categorization in the Journal Citation Report: a stated preference study," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1083-1099, February.
    9. Hu, Zewen & Wu, Yishan, 2014. "Regularity in the time-dependent distribution of the percentage of never-cited papers: An empirical pilot study based on the six journals," Journal of Informetrics, Elsevier, vol. 8(1), pages 136-146.
    10. Holsapple, Clyde W. & Lee-Post, Anita, 2010. "Behavior-based analysis of knowledge dissemination channels in operations management," Omega, Elsevier, vol. 38(3-4), pages 167-178, June.
    11. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).
    12. Laura Vana & Ronald Hochreiter & Kurt Hornik, 2016. "Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 229-251, January.
    13. Anne E Winkler & Sharon G Levin & Paula E Stephan & Wolfgang Gl&aauml;nzel, 2014. "Publishing Trends in Economics across Colleges and Universities, 1991–2007," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 40(4), pages 560-582, September.
    14. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    15. Jianhua Hou & Jiantao Ye, 2020. "Are uncited papers necessarily all nonimpact papers? A quantitative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1631-1662, August.
    16. Tânia F. G. G. Cova & Alberto A. C. C. Pais & Sebastião J. Formosinho, 2013. "Iberian universities: a characterisation from ESI rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1239-1251, March.
    17. Thelwall, Mike & Fairclough, Ruth, 2015. "Geometric journal impact factors correcting for individual highly cited articles," Journal of Informetrics, Elsevier, vol. 9(2), pages 263-272.
    18. repec:cup:judgdm:v:7:y:2012:i:5:p:639-643 is not listed on IDEAS
    19. Leah Melani Christian & Nicholas L. Parsons & Don A. Dillman, 2009. "Designing Scalar Questions for Web Surveys," Sociological Methods & Research, , vol. 37(3), pages 393-425, February.
    20. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    21. Müller, Sune Dueholm & Mathiassen, Lars & Balshøj, Hans Henrik, 2008. "Organizational Change Perspectives on Software Process Improvement," Informatics Research Group Working Papers I-2008-02, University of Aarhus, Aarhus School of Business, Department of Business Studies.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:7:y:2013:i:1:p:138-144. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.