IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0121898.html
   My bibliography  Save this article

Rank Diversity of Languages: Generic Behavior in Computational Linguistics

Author

Listed:
  • Germinal Cocho
  • Jorge Flores
  • Carlos Gershenson
  • Carlos Pineda
  • Sergio Sánchez

Abstract

Statistical studies of languages have focused on the rank-frequency distribution of words. Instead, we introduce here a measure of how word ranks change in time and call this distribution rank diversity. We calculate this diversity for books published in six European languages since 1800, and find that it follows a universal lognormal distribution. Based on the mean and standard deviation associated with the lognormal distribution, we define three different word regimes of languages: “heads” consist of words which almost do not change their rank in time, “bodies” are words of general use, while “tails” are comprised by context-specific words and vary their rank considerably in time. The heads and bodies reflect the size of language cores identified by linguists for basic communication. We propose a Gaussian random walk model which reproduces the rank variation of words in time and thus the diversity. Rank diversity of words can be understood as the result of random variations in rank, where the size of the variation depends on the rank itself. We find that the core size is similar for all languages studied.

Suggested Citation

  • Germinal Cocho & Jorge Flores & Carlos Gershenson & Carlos Pineda & Sergio Sánchez, 2015. "Rank Diversity of Languages: Generic Behavior in Computational Linguistics," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-12, April.
  • Handle: RePEc:plo:pone00:0121898
    DOI: 10.1371/journal.pone.0121898
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0121898
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0121898&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0121898?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gerardo Iñiguez & Carlos Pineda & Carlos Gershenson & Albert-László Barabási, 2022. "Dynamics of ranking," Nature Communications, Nature, vol. 13(1), pages 1-7, December.
    2. Wołoszyn, Maciej & Kułakowski, Krzysztof, 2023. "Status achieved in an organization—Rank dynamics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 610(C).
    3. Sergio Sánchez & Germinal Cocho & Jorge Flores & Carlos Gershenson & Gerardo Iñiguez & Carlos Pineda, 2018. "Trajectory Stability in the Traveling Salesman Problem," Complexity, Hindawi, vol. 2018, pages 1-8, April.
    4. Hussain, Abid & Drekic, Steve & Cheema, Salman A., 2024. "A relative-rank measure for the rank transformation," Statistics & Probability Letters, Elsevier, vol. 204(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0121898. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.