Mid level vision neglected yet still important Ken

  • Slides: 57
Download presentation
Mid level vision, neglected yet still important Ken Nakayama Harvard University 1

Mid level vision, neglected yet still important Ken Nakayama Harvard University 1

21 st C challenge Existence and variation of occlusion and variable sources of illumination

21 st C challenge Existence and variation of occlusion and variable sources of illumination pose unsolved problems for vision 2

 • Object representation needs an intermediate level format • Low level vision alone

• Object representation needs an intermediate level format • Low level vision alone is not even explanatory for wide range of visual processes (motion, stereo, search) • Missing -- a satisfactory scientific description of surface level vision 3

1970 s Visual take-over of the whole brain visual 1980 s 1950 s Half

1970 s Visual take-over of the whole brain visual 1980 s 1950 s Half of primate brain and substantial fraction of human brain devoted to vision 4

Macaque monkey brain flattened Visual regions shown in color 5

Macaque monkey brain flattened Visual regions shown in color 5

Global division of the visual system dorsal (where, how) ventral what 6

Global division of the visual system dorsal (where, how) ventral what 6

action BYPASS? object recognition 7

action BYPASS? object recognition 7

action depth search motion attention object recognition 8

action depth search motion attention object recognition 8

Kanizsa Phenomenology, reviving the Gestalt approach Level: surfaces Method: phenomenology Practitioner: Gaetano Kanizsa new

Kanizsa Phenomenology, reviving the Gestalt approach Level: surfaces Method: phenomenology Practitioner: Gaetano Kanizsa new concepts: amodal and modal completion 9

Amodal competion (behind) modal competion (in front) 10

Amodal competion (behind) modal competion (in front) 10

Inferences, but at what level ? 11

Inferences, but at what level ? 11

12

12

Suggests that there is a completion process within the visual system Amodal completion trumps

Suggests that there is a completion process within the visual system Amodal completion trumps knowledge of horses 13

Amodal completion allows fragments to be grouped and thus recognized (strongest evidence) spot the

Amodal completion allows fragments to be grouped and thus recognized (strongest evidence) spot the 5 letter Bs letter B same fragments From Bregman, 1990 14

Occlusion and the problem of segmentation for object recognition What belongs together ? Border

Occlusion and the problem of segmentation for object recognition What belongs together ? Border ownership issues - for 3 -D scenes, borders cannot be shared. Border dispute needs resolution Rule - border belongs to the closest surface 15

Problem of segmentation Kanisza’s figure Normal or amputee ? 16

Problem of segmentation Kanisza’s figure Normal or amputee ? 16

Border ownership dictated by “lines” prevents modal and amodal completion 17

Border ownership dictated by “lines” prevents modal and amodal completion 17

New sources of evidence Stereoscopic disparity Surface in front “owns” the border. Thus face

New sources of evidence Stereoscopic disparity Surface in front “owns” the border. Thus face on right is broken up, on left is OK Nakayama et al. Perception ‘ 89 faces easier to recognize on left 18

Stereoscopic depth also determines border ownership between regions. Nearer surface will own the border

Stereoscopic depth also determines border ownership between regions. Nearer surface will own the border (for opaque surfaces) Nakayama & Shimojo stereo demonstrations 19

Image level can’t even explain much lower level vision Deployment of attention, motion perception,

Image level can’t even explain much lower level vision Deployment of attention, motion perception, texture, visual search 20

image where how dorsal pariet al what vent ral t emporal surf aces 21

image where how dorsal pariet al what vent ral t emporal surf aces 21

Surfaces needed for much lower visual function 22

Surfaces needed for much lower visual function 22

He and Nakayama search task Used stereo vision Nature (1992) 23

He and Nakayama search task Used stereo vision Nature (1992) 23

24

24

Random dot stereogram The correspondence problem: an image based problem unpaired points 25

Random dot stereogram The correspondence problem: an image based problem unpaired points 25

L. E. only R. E. only 26

L. E. only R. E. only 26

invisible to right eye occluding surfaces What gives rise to unpaired points? What would

invisible to right eye occluding surfaces What gives rise to unpaired points? What would happen if we presented unpaired points by themselves? 27

Da. Vinci stereopsis (Nakayama &Shimojo) 28

Da. Vinci stereopsis (Nakayama &Shimojo) 28

Scene depth from unpaired gaps a Gillam and Nakayama, 1999 29

Scene depth from unpaired gaps a Gillam and Nakayama, 1999 29

Forest vs plane A plane is a surface Which can occlude, a set of

Forest vs plane A plane is a surface Which can occlude, a set of random sticks cannot 30

Planes vs sticks Gillam and Nakayama, 200 31

Planes vs sticks Gillam and Nakayama, 200 31

Level of processing high or low level inference? Hypothesis - inferences learned via associative

Level of processing high or low level inference? Hypothesis - inferences learned via associative cortical learning 32

generic view principle when faced with more than one surface interpretation of an image,

generic view principle when faced with more than one surface interpretation of an image, the visual system assumes it is viewing the scene from a generic, not accidental, vantage point. Nakayama and Shimojo 33

Some counterintuitive observations folded wings? LE RE folded cards? Why don’t we interpolate depth

Some counterintuitive observations folded wings? LE RE folded cards? Why don’t we interpolate depth and see folded wings and cards? 34

35

35

Accidental vs generic vantage points 36

Accidental vs generic vantage points 36

 • accidental view • generic view 37

• accidental view • generic view 37

sur f aces images square ( surf ace) cube ( volume) viewing sphere 38

sur f aces images square ( surf ace) cube ( volume) viewing sphere 38

generic view principle when faced with more than one surface interpretation of an image,

generic view principle when faced with more than one surface interpretation of an image, the visual system assumes it is viewing the scene from a generic, not accidental, vantage point. 39

40

40

Some counterintuitive observations folded wings? LE RE folded cards? Why don’t we interpolate depth

Some counterintuitive observations folded wings? LE RE folded cards? Why don’t we interpolate depth and see folded wings and cards? 41

LE RE conclusion: this is a generic view of crossed bars not wings 42

LE RE conclusion: this is a generic view of crossed bars not wings 42

this is the generic view of transparent surface in front, not a folded card

this is the generic view of transparent surface in front, not a folded card 43

neural mechanisms of surface representation ? Cells in V 2 respond to subjective contours

neural mechanisms of surface representation ? Cells in V 2 respond to subjective contours Strategy: vary stimuli in ways that lead to Appearance and disappearance of subjective contours 44

Physiological correlates of illusory contours in single unit recordings Recordings from a single cell

Physiological correlates of illusory contours in single unit recordings Recordings from a single cell in area V 2 of monkey 45

Bakin, Nakayama, and Gilbert, 2000 46

Bakin, Nakayama, and Gilbert, 2000 46

Cells coding Border ownership? Edgar Rubin figure and ground Von der Heydt Et al.

Cells coding Border ownership? Edgar Rubin figure and ground Von der Heydt Et al. Edge labeling? contrast polarity vs edge labeling 47

+ +� -� + +� image based response -� 48

+ +� -� + +� image based response -� 48

Border ownership based response 49

Border ownership based response 49

Border ownership cells Von der Heydt and colleagues 50

Border ownership cells Von der Heydt and colleagues 50

Von der Heydt (1984) yes V 2 Bakin, Nakayama, Gilbert (2000) V 2 Da.

Von der Heydt (1984) yes V 2 Bakin, Nakayama, Gilbert (2000) V 2 Da. Vinci stereopsis Border ownership cells (V 2) Mechanistic account of surface representation? --> 51

21 st C challenge Existence and variation of occlusion and variable sources of illumination

21 st C challenge Existence and variation of occlusion and variable sources of illumination pose unsolved problems for vision 52

 • Object representation needs an intermediate level format • Low level vision alone

• Object representation needs an intermediate level format • Low level vision alone is not even explanatory for wide range of visual processes (motion, stereo, search) • Missing -- a satisfactory scientific description of surface level vision -- demos the importance for illumination for object recognition 53

importance of shadow processing Ted Adelson 54

importance of shadow processing Ted Adelson 54

Shadow processing in object recognition shadow face yes reduce contrast yes outline no 55

Shadow processing in object recognition shadow face yes reduce contrast yes outline no 55

reduce contrast add outline no yes Outline is very destructive to seeing regions as

reduce contrast add outline no yes Outline is very destructive to seeing regions as shaded. Line is interpreted as a bounding Contour of an object 56

57

57