IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v444y2016icp828-837.html
   My bibliography  Save this article

Randomness versus specifics for word-frequency distributions

Author

Listed:
  • Yan, Xiaoyong
  • Minnhagen, Petter

Abstract

The text-length-dependence of real word-frequency distributions can be connected to the general properties of a random book. It is pointed out that this finding has strong implications, when deciding between two conceptually different views on word-frequency distributions, i.e. the specific ‘Zipf’s-view’ and the non-specific ‘Randomness-view’, as is discussed. It is also noticed that the text-length transformation of a random book does have an exact scaling property precisely for the power-law index γ=1, as opposed to the Zipf’s exponent γ=2 and the implication of this exact scaling property is discussed. However a real text has γ>1 and as a consequence γ increases when shortening a real text. The connections to the predictions from the RGF (Random Group Formation) and to the infinite length-limit of a meta-book are also discussed. The difference between ‘curve-fitting’ and ‘predicting’ word-frequency distributions is stressed. It is pointed out that the question of randomness versus specifics for the distribution of outcomes in case of sufficiently complex systems has a much wider relevance than just the word-frequency example analyzed in the present work.

Suggested Citation

  • Yan, Xiaoyong & Minnhagen, Petter, 2016. "Randomness versus specifics for word-frequency distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 444(C), pages 828-837.
    • Handle: RePEc:eee:phsmap:v:444:y:2016:i:c:p:828-837
    DOI: 10.1016/j.physa.2015.10.082
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437115009413
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000
    File URL: https://libkey.io/10.1016/j.physa.2015.10.082?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Montemurro, Marcelo A., 2001. "Beyond the Zipf–Mandelbrot law in quantitative linguistics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 300(3), pages 567-578.
    2. Bernhardsson, Sebastian & da Rocha, Luis Enrique Correa & Minnhagen, Petter, 2010. "Size-dependent word frequencies and translational invariance of books," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 389(2), pages 330-341.
    3. Xiaoyong Yan & Petter Minnhagen, 2015. "Maximum Entropy, Word-Frequency, Chinese Characters, and Multiple Meanings," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-19, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yan, Xiaoyong & Yang, Seong-Gyu & Kim, Beom Jun & Minnhagen, Petter, 2018. "Benford’s law and first letter of words," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 512(C), pages 305-315.
    2. Yan, Xiaoyong & Minnhagen, Petter, 2018. "The dependence of frequency distributions on multiple meanings of words, codes and signs," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 490(C), pages 554-564.
    3. Yan, Xiaoyong & Minnhagen, Petter & Jensen, Henrik Jeldtoft, 2016. "The likely determines the unlikely," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 456(C), pages 112-119.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yan, Xiaoyong & Minnhagen, Petter, 2018. "The dependence of frequency distributions on multiple meanings of words, codes and signs," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 490(C), pages 554-564.
    2. Rotundo, Giulia, 2014. "Black–Scholes–Schrödinger–Zipf–Mandelbrot model framework for improving a study of the coauthor core score," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 404(C), pages 296-301.
    3. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    4. Tunnicliffe, Martin & Hunter, Gordon, 2022. "Random sampling of the Zipf–Mandelbrot distribution as a representation of vocabulary growth," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 608(P1).
    5. Ferrer i Cancho, Ramon, 2005. "Decoding least effort and scaling in signal frequency distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 345(1), pages 275-284.
    6. Yan, Xiaoyong & Yang, Seong-Gyu & Kim, Beom Jun & Minnhagen, Petter, 2018. "Benford’s law and first letter of words," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 512(C), pages 305-315.
    7. Yan, Xiaoyong & Minnhagen, Petter & Jensen, Henrik Jeldtoft, 2016. "The likely determines the unlikely," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 456(C), pages 112-119.
    8. Ghosh, Dipak & Chakraborty, Sayantan & Samanta, Shukla, 2019. "Study of translational effect in Tagore’s Gitanjali using Chaos based Multifractal analysis technique," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 523(C), pages 1343-1354.
    9. Young, D.S., 2013. "Approximate tolerance limits for Zipf–Mandelbrot distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 392(7), pages 1702-1711.
    10. Lambiotte, R. & Ausloos, M. & Thelwall, M., 2007. "Word statistics in Blogs and RSS feeds: Towards empirical universal evidence," Journal of Informetrics, Elsevier, vol. 1(4), pages 277-286.
    11. Miśkiewicz, Janusz & Ausloos, Marcel, 2008. "Correlation measure to detect time series distances, whence economy globalization," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(26), pages 6584-6594.
    12. Çavuşoğlu, Abdullah & Türker, İlker, 2014. "Patterns of collaboration in four scientific disciplines of the Turkish collaboration network," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 413(C), pages 220-229.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:444:y:2016:i:c:p:828-837. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.