IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1002372.html
   My bibliography  Save this article

Efficient Sparse Coding in Early Sensory Processing: Lessons from Signal Recovery

Author

Listed:
  • András Lörincz
  • Zsolt Palotai
  • Gábor Szirtes

Abstract

Sensory representations are not only sparse, but often overcomplete: coding units significantly outnumber the input units. For models of neural coding this overcompleteness poses a computational challenge for shaping the signal processing channels as well as for using the large and sparse representations in an efficient way. We argue that higher level overcompleteness becomes computationally tractable by imposing sparsity on synaptic activity and we also show that such structural sparsity can be facilitated by statistics based decomposition of the stimuli into typical and atypical parts prior to sparse coding. Typical parts represent large-scale correlations, thus they can be significantly compressed. Atypical parts, on the other hand, represent local features and are the subjects of actual sparse coding. When applied on natural images, our decomposition based sparse coding model can efficiently form overcomplete codes and both center-surround and oriented filters are obtained similar to those observed in the retina and the primary visual cortex, respectively. Therefore we hypothesize that the proposed computational architecture can be seen as a coherent functional model of the first stages of sensory coding in early vision. Author Summary: Neural systems favor overcomplete sparse codes in which the number of potential output neurons may exceed the number of input neurons, but only a small subset of neurons become actually active. We argue that efficient use of such large dimensional overcomplete sparse codes requires structural sparsity by controlling the number of active synapses. Motivated by recent results in signal recovery, we introduce a particular signal decomposition as a pre-filtering stage prior to the actual sparse coding, which efficiently supports structural sparsity. In contrast to most models of sensory processing, we hypothesize that the observed transformations may actually realize parallel encoding of the stimuli into representations that describe typical and atypical parts. When trained on natural images, the resulting system can handle large, overcomplete representations and the learned transformations seem compatible with the various receptive fields characteristic to different stages of early vision. In particular, transformations realized by the prefiltering units can be approximated as ‘Difference-of-Gaussians’ filters, similar to the receptive fields of neurons in the retina and the LGN. In addition, sparse coding units have localized and oriented edge filters like the receptive fields of the simple cells in the primary visual cortex, V1.

Suggested Citation

  • András Lörincz & Zsolt Palotai & Gábor Szirtes, 2012. "Efficient Sparse Coding in Early Sensory Processing: Lessons from Signal Recovery," PLOS Computational Biology, Public Library of Science, vol. 8(3), pages 1-14, March.
  • Handle: RePEc:plo:pcbi00:1002372
    DOI: 10.1371/journal.pcbi.1002372
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002372
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1002372&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1002372?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Reuven Rubinstein, 1999. "The Cross-Entropy Method for Combinatorial and Continuous Optimization," Methodology and Computing in Applied Probability, Springer, vol. 1(2), pages 127-190, September.
    2. Li, Baibing & Martin, Elaine B. & Morris, A. Julian, 2002. "On principal component analysis in L1," Computational Statistics & Data Analysis, Elsevier, vol. 40(3), pages 471-474, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Plat, Richard, 2009. "Stochastic portfolio specific mortality and the quantification of mortality basis risk," Insurance: Mathematics and Economics, Elsevier, vol. 45(1), pages 123-132, August.
    2. Kondylis, Athanassios & Whittaker, Joe, 2008. "Spectral preconditioning of Krylov spaces: Combining PLS and PC regression," Computational Statistics & Data Analysis, Elsevier, vol. 52(5), pages 2588-2603, January.
    3. Ouyang, Yaofu & Li, Peng, 2018. "On the nexus of financial development, economic growth, and energy consumption in China: New perspective from a GMM panel VAR approach," Energy Economics, Elsevier, vol. 71(C), pages 238-252.
    4. Paschalis Arvanitidis & Athina Economou & Christos Kollias, 2016. "Terrorism’s effects on social capital in European countries," Public Choice, Springer, vol. 169(3), pages 231-250, December.
    5. Rizvi, Syed Kumail Abbas & Rahat, Birjees & Naqvi, Bushra & Umar, Muhammad, 2024. "Revolutionizing finance: The synergy of fintech, digital adoption, and innovation," Technological Forecasting and Social Change, Elsevier, vol. 200(C).
    6. Teerachai Amnuaylojaroen & Pavinee Chanvichit, 2024. "Historical Analysis of the Effects of Drought on Rice and Maize Yields in Southeast Asia," Resources, MDPI, vol. 13(3), pages 1-18, March.
    7. Weili Duan & Bin He & Daniel Nover & Guishan Yang & Wen Chen & Huifang Meng & Shan Zou & Chuanming Liu, 2016. "Water Quality Assessment and Pollution Source Identification of the Eastern Poyang Lake Basin Using Multivariate Statistical Methods," Sustainability, MDPI, vol. 8(2), pages 1-15, January.
    8. Adele Ravagnani & Fabrizio Lillo & Paola Deriu & Piero Mazzarisi & Francesca Medda & Antonio Russo, 2024. "Dimensionality reduction techniques to support insider trading detection," Papers 2403.00707, arXiv.org, revised May 2024.
    9. Catanzaro, Daniele & Pesenti, Raffaele & Ronco, Roberto, 2023. "Job scheduling under Time-of-Use energy tariffs for sustainable manufacturing: a survey," European Journal of Operational Research, Elsevier, vol. 308(3), pages 1091-1109.
    10. Cling, Jean-Pierre & Delecourt, Clément, 2022. "Interlinkages between the Sustainable Development Goals," World Development Perspectives, Elsevier, vol. 25(C).
    11. Hino, Hideitsu & Wakayama, Keigo & Murata, Noboru, 2013. "Entropy-based sliced inverse regression," Computational Statistics & Data Analysis, Elsevier, vol. 67(C), pages 105-114.
    12. Angelucci, Federica & Conforti, Piero, 2010. "Risk management and finance along value chains of Small Island Developing States. Evidence from the Caribbean and the Pacific," Food Policy, Elsevier, vol. 35(6), pages 565-575, December.
    13. Poskitt, D.S. & Sengarapillai, Arivalzahan, 2013. "Description length and dimensionality reduction in functional data analysis," Computational Statistics & Data Analysis, Elsevier, vol. 58(C), pages 98-113.
    14. Taner Akan & Tim Solle, 2022. "Do macroeconomic and financial governance matter? Evidence from Germany, 1950–2019," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 17(4), pages 993-1045, October.
    15. Paolo Rizzi & Paola Graziano & Antonio Dallara, 2018. "A capacity approach to territorial resilience: the case of European regions," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 60(2), pages 285-328, March.
    16. Pérez, Claudia & Claveria, Oscar, 2020. "Natural resources and human development: Evidence from mineral-dependent African countries using exploratory graphical analysis," Resources Policy, Elsevier, vol. 65(C).
    17. Zeynep Ozkok, 2015. "Financial openness and financial development: an analysis using indices," International Review of Applied Economics, Taylor & Francis Journals, vol. 29(5), pages 620-649, September.
    18. Asongu, Simplice A & Odhiambo, Nicholas M, 2019. "Governance,CO2 emissions and inclusive human development in Sub-Saharan Africa," Working Papers 25253, University of South Africa, Department of Economics.
    19. Anne M. Lausier & Shaleen Jain, 2018. "Diversity in global patterns of observed precipitation variability and change on river basin scales," Climatic Change, Springer, vol. 149(2), pages 261-275, July.
    20. Puppo, L. & Pedroni, N. & Maio, F. Di & Bersano, A. & Bertani, C. & Zio, E., 2021. "A Framework based on Finite Mixture Models and Adaptive Kriging for Characterizing Non-Smooth and Multimodal Failure Regions in a Nuclear Passive Safety System," Reliability Engineering and System Safety, Elsevier, vol. 216(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1002372. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.