IDEAS home Printed from https://ideas.repec.org/a/bla/jamist/v64y2013i10p2024-2044.html
   My bibliography  Save this article

On the assessment of expertise profiles

Author

Listed:
  • Richard Berendsen
  • Maarten de Rijke
  • Krisztian Balog
  • Toine Bogers
  • Antal van den Bosch

Abstract

Expertise retrieval has attracted significant interest in the field of information retrieval. Expert finding has been studied extensively, with less attention going to the complementary task of expert profiling, that is, automatically identifying topics about which a person is knowledgeable. We describe a test collection for expert profiling in which expert users have self‐selected their knowledge areas. Motivated by the sparseness of this set of knowledge areas, we report on an assessment experiment in which academic experts judge a profile that has been automatically generated by state‐of‐the‐art expert‐profiling algorithms; optionally, experts can indicate a level of expertise for relevant areas. Experts may also give feedback on the quality of the system‐generated knowledge areas. We report on a content analysis of these comments and gain insights into what aspects of profiles matter to experts. We provide an error analysis of the system‐generated profiles, identifying factors that help explain why certain experts may be harder to profile than others. We also analyze the impact on evaluating expert‐profiling systems of using self‐selected versus judged system‐generated knowledge areas as ground truth; they rank systems somewhat differently but detect about the same amount of pairwise significant differences despite the fact that the judged system‐generated assessments are more sparse.

Suggested Citation

  • Richard Berendsen & Maarten de Rijke & Krisztian Balog & Toine Bogers & Antal van den Bosch, 2013. "On the assessment of expertise profiles," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(10), pages 2024-2044, October.
    • Handle: RePEc:bla:jamist:v:64:y:2013:i:10:p:2024-2044
    DOI: 10.1002/asi.22908
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/asi.22908
    Download Restriction: no
    File URL: https://libkey.io/10.1002/asi.22908?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xiancheng Li & Luca Verginer & Massimo Riccaboni & P. Panzarasa, 2022. "A network approach to expertise retrieval based on path similarity and credit allocation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 17(2), pages 501-533, April.
    2. Mark Bukowski & Sandra Geisler & Thomas Schmitz-Rode & Robert Farkas, 2020. "Feasibility of activity-based expert profiling using text mining of scientific publications and patents," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 579-620, May.
    3. A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jamist:v:64:y:2013:i:10:p:2024-2044. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.asis.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.