Skip to main content Accessibility help

Crossmodal processing and sensory substitution: Is “seeing” with sound and touch a form of perception or cognition?

  • Tayfun Esenkaya (a1) (a2) and Michael J. Proulx (a2)


The brain has evolved in this multisensory context to perceive the world in an integrated fashion. Although there are good reasons to be skeptical of the influence of cognition on perception, here we argue that the study of sensory substitution devices might reveal that perception and cognition are not necessarily distinct, but rather continuous aspects of our information processing capacities.



Hide All
Bach-y-Rita, P. & Kercel, S. W. (2003) Sensory substitution and the human–machine interface. Trends in Cognitive Sciences 7(12):541–46. doi:S1364661303002900 [pii].
Ghazanfar, A. A. & Schroeder, C. E. (2006) Is neocortex essentially multisensory? Trends in Cognitive Sciences 10(6):278–85. Available at:
Klein, I., Dubois, J., Mangin, J.-F., Kherif, F., Flandin, G., Poline, J.-B. & Le Bihan, D. (2004) Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging. Cognitive Brain Research 22(1):2631.
Kosslyn, S. M., Pascual-Leone, A., Felician, O., Camposano, S., Keenan, J., Ganis, G. & Alpert, N. (1999) The role of area 17 in visual imagery: Convergent evidence from PET and rTMS. Science 284(5411):167–70.
Meijer, P. B. (1992) An experimental system for auditory image representations. IEEE Transactions on Biomedical Engineering 39(2):112–21. doi:10.1109/10.121642.
Meredith, M. A. & Stein, B. E. (1986) Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. Journal of Neurophysiology 56(3):640–62.
Morgan, M. J. (1977) Molyneux's question: Vision, touch and the philosophy of perception. Cambridge University Press.
Proulx, M. J. (2011) Consciousness: What, how, and why. Science 332(6033):1034–35. doi:10.1126/Science.1206704.
Proulx, M. J., Brown, D. J., Pasqualotto, A. & Meijer, P. (2014) Multisensory perceptual learning and sensory substitution. Neuroscience and Biobehavoral Reviews 41:1625. doi:10.1016/j.neubiorev.2012.11.017.
Stein, B. E., Huneycutt, W. S. & Meredith, M. A. (1988) Neurons and behavior: The same rules of multisensory integration apply. Brain Research 448(2):355–58.
von Helmholtz, H. (2005) Treatise on physiological optics, vol. 3. Courier Dover.
Ward, J. & Meijer, P. (2010) Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and Cognition 19(1):492500. doi:10.1016/j.concog.2009.10.006.
Yantis, S. (2013) Sensation and perception. Palgrave Macmillan.


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed