IDEAS home Printed from https://ideas.repec.org/a/eee/phsmap/v553y2020ics0378437119321673.html
   My bibliography  Save this article

Universal and non-universal text statistics: Clustering coefficient for language identification

Author

Listed:
  • Espitia, Diego
  • Larralde, Hernán

Abstract

In this work we analyze statistical properties of 91 relatively small texts in 7 different languages (Spanish, English, French, German, Turkish, Russian, Icelandic) as well as texts with randomly inserted spaces. Despite the size (around 11260 different words), the well known universal statistical laws – namely Zipf and Herdan–Heap’s laws – are confirmed, and are in close agreement with results obtained elsewhere. We also construct a word co-occurrence network of each text. While the degree distribution is again universal, we note that the distribution of clustering coefficients, which depend strongly on the local structure of networks, can be used to differentiate between languages, as well as to distinguish natural languages from random texts.

Suggested Citation

  • Espitia, Diego & Larralde, Hernán, 2020. "Universal and non-universal text statistics: Clustering coefficient for language identification," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 553(C).
    • Handle: RePEc:eee:phsmap:v:553:y:2020:i:c:s0378437119321673
    DOI: 10.1016/j.physa.2019.123905
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378437119321673
    Download Restriction: Full text for ScienceDirect subscribers only. Journal offers the option of making the article available online on Science direct for a fee of $3,000
    File URL: https://libkey.io/10.1016/j.physa.2019.123905?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marcelo A Montemurro & Damián H Zanette, 2013. "Keywords and Co-Occurrence Patterns in the Voynich Manuscript: An Information-Theoretic Analysis," PLOS ONE, Public Library of Science, vol. 8(6), pages 1-9, June.
    2. Ricardo Baeza–Yates & Gonzalo Navarro, 2000. "Block addressing indices for approximate text retrieval," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 51(1), pages 69-82.
    3. Petroni, Filippo & Serva, Maurizio, 2010. "Measures of lexical distance between languages," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 389(11), pages 2280-2283.
    4. Ramon Ferrer-i-Cancho & Brita Elvevåg, 2010. "Random Texts Do Not Exhibit the Real Zipf's Law-Like Rank Distribution," PLOS ONE, Public Library of Science, vol. 5(3), pages 1-10, March.
    5. Camilo Akimushkin & Diego Raphael Amancio & Osvaldo Novais Oliveira Jr., 2017. "Text Authorship Identified Using the Dynamics of Word Co-Occurrence Networks," PLOS ONE, Public Library of Science, vol. 12(1), pages 1-15, January.
    6. Gamallo, Pablo & Pichel, José Ramom & Alegria, Iñaki, 2017. "From language identification to language distance," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 484(C), pages 152-162.
    7. Leo Egghe, 2007. "Untangling Herdan's law and Heaps' law: Mathematical and informetric arguments," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(5), pages 702-709, March.
    8. Naumis, G.G. & Cocho, G., 2008. "Tail universalities in rank distributions as an algebraic problem: The beta-like function," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(1), pages 84-96.
    9. Mark Newman, 2000. "The power of design," Nature, Nature, vol. 405(6785), pages 412-413, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mehri, Ali & Jamaati, Maryam, 2021. "Statistical metrics for languages classification: A case study of the Bible translations," Chaos, Solitons & Fractals, Elsevier, vol. 144(C).
    2. Li, Wentian, 2012. "Fitting Chinese syllable-to-character mapping spectrum by the beta rank function," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 391(4), pages 1515-1518.
    3. Ingo Eduard Isphording & Sebastian Otten, 2013. "The Costs of Babylon—Linguistic Distance in Applied Economics," Review of International Economics, Wiley Blackwell, vol. 21(2), pages 354-369, May.
    4. Reed, William J., 2001. "The Pareto, Zipf and other power laws," Economics Letters, Elsevier, vol. 74(1), pages 15-19, December.
    5. repec:zbw:rwirep:0337 is not listed on IDEAS
    6. Gamallo, Pablo & Pichel, José Ramom & Alegria, Iñaki, 2017. "From language identification to language distance," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 484(C), pages 152-162.
    7. Corrêa, Edilson A. & Marinho, Vanessa Q. & Amancio, Diego R., 2020. "Semantic flow in language networks discriminates texts by genre and publication date," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 557(C).
    8. repec:zbw:hohpro:352 is not listed on IDEAS
    9. Bertoli-Barsotti, Lucio & Lando, Tommaso, 2019. "How mean rank and mean size may determine the generalised Lorenz curve: With application to citation analysis," Journal of Informetrics, Elsevier, vol. 13(1), pages 387-396.
    10. Isphording, Ingo E. & Piopiunik, Marc & Rodríguez-Planas, Núria, 2016. "Speaking in numbers: The effect of reading performance on math performance among immigrants," Economics Letters, Elsevier, vol. 139(C), pages 52-56.
    11. Johan Fellman, 2021. "Aspects of Pareto distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 10(1), pages 1-4.
    12. Ibrahim Bousmah & Gilles Grenier & David M. Gray, 2021. "Linguistic Distance, Languages of Work and Wages of Immigrants in Montreal," Journal of Labor Research, Springer, vol. 42(1), pages 1-28, March.
    13. Shakibian, Hadi & Charkari, Nasrollah Moghadam, 2018. "Statistical similarity measures for link prediction in heterogeneous complex networks," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 501(C), pages 248-263.
    14. Anna Tietze & Philip Hofmann, 2019. "The h-index and multi-author hm-index for individual researchers in condensed matter physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 171-185, April.
    15. Valerio Ficcadenti & Roy Cerqueti & Ciro Hosseini Varde’i, 2023. "A rank-size approach to analyse soccer competitions and teams: the case of the Italian football league “Serie A"," Annals of Operations Research, Springer, vol. 325(1), pages 85-113, June.
    16. Choudhury, Nazim & Faisal, Fahim & Khushi, Matloob, 2020. "Mining Temporal Evolution of Knowledge Graphs and Genealogical Features for Literature-based Discovery Prediction," Journal of Informetrics, Elsevier, vol. 14(3).
    17. Katz, J. Sylvan, 2006. "Indicators for complex innovation systems," Research Policy, Elsevier, vol. 35(7), pages 893-909, September.
    18. Martinez-Saito, Mario, 2022. "Discrete scaling and criticality in a chain of adaptive excitable integrators," Chaos, Solitons & Fractals, Elsevier, vol. 163(C).
    19. Chen, Yanguang, 2012. "Zipf’s law, 1/f noise, and fractal hierarchy," Chaos, Solitons & Fractals, Elsevier, vol. 45(1), pages 63-73.
    20. Wei, Shelia X. & Tong, Tong & Rousseau, Ronald & Wang, Wanru & Ye, Fred Y., 2022. "Relations among the h-, g-, ψ-, and p-index and offset-ability," Journal of Informetrics, Elsevier, vol. 16(4).
    21. Zörnig, Peter, 2010. "Statistical simulation and the distribution of distances between identical elements in a random sequence," Computational Statistics & Data Analysis, Elsevier, vol. 54(10), pages 2317-2327, October.
    22. repec:old:wpaper:352 is not listed on IDEAS
    23. Ausloos, Marcel & Cerqueti, Roy & Lupi, Claudio, 2017. "Long-range properties and data validity for hydrogeological time series: The case of the Paglia river," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 470(C), pages 39-50.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:phsmap:v:553:y:2020:i:c:s0378437119321673. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/physica-a-statistical-mechpplications/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.