Sensory modalities Different approaches ORegan J K Noe

Sensory modalities Different approaches

O’Regan, J. K. , Noe, A. , A sensorimotor account of vision and visual consciousness Behavioral and Brain sciences (2001), 24: 5

Unsignificant internal architecture • There is no internal representations or architecture to explain the conscious quality of different sensations: – From the point of view of the brain there is nothing that in itself differentiates nervous influx coming from retinal, haptic, proprioceptive, auditory, and other senses • Even if the size, shape, firing patterns, places wqwhere neurons are localized in the cortex differ, this does not, in itself confer them with any particular visual, olfactory, motor or other perceptual qualities

Sensorimotor contingencies • What does explain the differences between sensations is the structure of the rules governing the sensory changes produced by various motor actions = sensorimotor contingencies – Within different sensory modalities, sensorimotor contingencies differ because they are subject to different invariance properties – Each sensory modality is defined by a particular structure of the rules governing sensoru changes produced by movement • Sensory modalities are modes of exploration of the world mediated by knowledge of sensorimotor contingencies

Classes of sensorimotor contingency • We can destinguish between 2 classes of sensorimotor contingency: – Sensorimotor contingencies determined by the character of the visual apparatus (the eye is a kind of sphere, etc. ) • They are the fundamental aspect of visual sensation (because they are independent of any interpretation or categorization of objects) – Sensorimotor contingencies determined by the visual attributes of the object, or geometric invariants (characteristics of the object that doesn’t change when the eye moves) • They are the basis of visual perception

Visual sensorimotor contingencies: visual percepts • Movements of the eyes: – When the eyes rotate, the sensory stimulation of the retina shifts and distorts in a very particular way, determined by the size of the eye movement, the shape of the retina, the nature of the ocular optics: contours shift and the curvature of the lines changes, the distribution of information on the retin changes (because of the difference of distribution of photoreceptors in central and perripheral vision) – When the body moves forwards, the flow pattern on the retina is an expanding one; when moving backwards a contracting one – When the eyes close during blinks, the stimulation changes drastically becoming uniform • Constraints of visual-type exploration/visual attributes of objects: – The retinal image of an object only provides a frontal view of the object – When one moves around the object parts of the object appear and disappear – We can only apprehend an object from a definite distance (retinal projection depends on distance) – Color and brightness of the objects change when one moves around the object or lighting conditions change

Perception of visual shape • Exemple: Patiens born blind and operated: • A patient cited by Helmholtz is surprised that a coin should so dramatically change its shape when rotated • A patient of Cheseldon was surprised by the change in extent of an object in fonction of the distance • « The visual quality of shape is precisely the set of all potential distortions that the shape undergoes when it is moved relative to us, or when we move relative to it. • Although it is an infinite set, the brain can abstract from this set a series of laws, and it is this set of laws which codes shape »

Other sensorimotor contingecies: auditory percepts • Auditory sensorimotor contingencies are not affected by eye movements or blinks, but they are affected in a special way by head movements: – rotations of the head change the temporal asynchrony between the ears – movements of the head in the direction of the sound source affect the amplitude but not the frequency of the input • Tactile exploration of an object obeys different sensorimotor contingencies: – One doesn’t touch an object from a point of view – The tactile aspect of an object doesn’t change with lighting conditions

Knowledge of sensorimotor laws Summary: « what differentiate the senses are not the neural pathways but the laws obeyed by the sensorimotor contingencies associates with these senses » – Sensorimotor contingencies are distinctive for each sensory modality – The laws of sensorimotor contingency are determined by the fact that the exploration is being made by a particular apparatus and exercised on specific characters of the object – The brain distinguishes between the senses on the basis of the different sensorimotor laws a percept respond to • Two conditions must be achieved to perceive in a certain modality: – Presence of the conditions for the two kind of sensorimotor contingency laws – Active master of the laws • Third condition for visual (sensory) awareness: – The master of the laws must be comprised in plannings, thougs, linguistic behavior

A practical form of knowledge • The mastery of the rules of sensorimotor contingency is not something we possess a propositional knowledge about – We are not able to describe in detail how it is distorted a seen surface when we rotate the eyes • The deviation from the laws that our brain normally extract causes an impression that something unusual is happening and let us see the distortions – When we put on a new pair of glasses we see distortions for a while because the movement of the eyes provokes displacements of unusual amplitude • The knowledge of the laws is a predictive or anticipatory knowledge: « Perceptual experience is a mode of activity involving practical knowledge about currently possible behaviors and associated sensory consequences » – When one sees or touch a bottle there is a knowledge in his nervous system about what he would feel or see if he move his hands or eyes upwards or downwards the bottle, and it is this anticipation that tells him that he is touching a bottle and not a pitcher

Some phenomena explained within sensorimotor contingency approach • Tactile visual sensory substitution – In experiences with TVSS it is only when the observer is allowed to move the videocamera that identification becomes possible and that he begins to « see » something as externally localized, in the space before him (also if the stimulus array is on his back) – If an experience in one sensory modality derives from the « rules that govern action-related changes in sensory input » visual experience could be obtainable by other sensory channels when the brain extracts the same invariants in the structure of the senorimotor contingencies – So subjects with TVSS see and not just « see » : the fact that the stimulus is provided by the skin is irrelevant, provided that the stimulation obeyes the required sensorimotor laws (of course the laws are not exactly the same as seeing with the eyes, as seeing in the dark is not the same thing that seein with full light)

Multimodal interactions • Synesthesia and other multimodal substitutions and interactions as ventriloquism effect, Mc. Gurk effect and others: – The experience associated with a modality exists only within the context of other senses available to the organism : senses are not really separated • Although sensory modalities have their own specificities dues to the particularities of sensors and of sensorimotor contingencies, when there are systematic correlations and common sensorimotor contingencies, interactions between the senses are to be expected • Anyway, the mentioned specificities define the particular experience associated with each sense

Binding problem • The binding problem is a pseudo problem • Neuroanatomists believe that the visual system is composed of numerous sub-systems or modules which are independent and often localized in different parts of the brain; so they raise the question of where and how the different features ultimately come together to produce a unified perception of reality as we experience; they suggest some solutions: • The grand-mother cell in which single cells at least combine information • The synchronized oscillation of the separate cortical areas which are analysing different components of the information • The binding problem can be analysed under two issues: – Temporal unit or coherence: – scientists think that, because we have the impression that we see simultaneously all the attributes when we identify an object, then the information about these attributes must be bound together synchronously. • It is a fallacy: physical synchrony is not necessary for having a synchronous experience because it is not the « internal representation » which counts for the coherence of the experience. « What counts for a temporally coherent experience is the fact is a thing we are doing, and we are doing it now » – Conceptual unit or coherence: • The fact that object attributes seem to be part of a single object doesn’t require them to be « represented » in a unified way, for example at a single location of the brain, or by a single process; and if they are represented in a spatially or temporally way this doesn’t explain the perceptual coherence. « What explains the conceptual unity of experience is the fact that experience is a thing we are doing and we are doing it with respect to a conceptually unified external world »

Qualia • Qualia doesn’exist: the differences in the qualitative character of perceptual experiences correspond to differences in the character of the relevant sensorimotor contingencies – The difference between driving a Porche and driving a tank is in the different things we do driving them, in the different practical knowledge about driving them – The difference between seein and smelling a red flower consists in the different patterns of sensorimotor contingencies that governs perceptual encounte with each: • To experience the redness of an object or to feel to drive a Porche is to know what would happen if the light changes or if the accelerator is pressed down

Some problems with multimodality and intermodality: is a sensorimotor account complete? • • The fact that the laws of sensorimotor contingency are different from a modality to another, doesn’t indicate the necessity of a kind of binding, even if not an internal binding, but a description of the way in which different sensorimotor contingencies become an object instead of two objects? Supposed that we accept there is no need of binding, what about the need for coherence: could a system act in a world that is not coherent? Don’t we need mechanism to warranty coherence? These mechanisms could act on information or on sensorimotor contingencies, in order to set an accord between contingencies? Are there laws that rule « multisensory contingency » ? Are these rules effective on perception? ie regarding the relation between vision and touch in prehension: complex rules for a motor act that define a sort of sensorimotor act? In tactile visual substitution, ie, how can we say that the laws of sensorimotor contingency are analogous with eyes or tactile substitution? Performances are they better if the device is placed on the front of the subject instead of on the back (in order to move the camera as he moves his head)? Visibly, not