Anaphoric dependencies A window into the architecture of

Anaphoric dependencies: A window into the architecture of the language system Eye tracking experiments Eric Reuland Frank Wijnen Arnout Koornneef

EYE-TRACKING BY ARNOUT KOORNNEEF

OVERVIEW OF LECTURE • Part 1: discussion of method - eye-tracking while reading • Part 2: discussion of current research - eye-tracking experiments

THE EYE-TRACKER

THE EYE-TRACKER

THE EYE-TRACKER • monitors eye-movements from millisecond to millisecond • provides information about where people look and for how long

TWO MAIN EYETRACKING PARADIGMS • eye-tracking while reading • eye-tracking while listening (not discussed)

EYE-TRACKING WHILE READING

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

SOME FACTS ABOUT READING • people do not read a text “smoothly”, but fixate a particular word (200 - 300 msec) and jump to the next • a jump (or saccade) covers 7 -9 letter spaces • during a saccade visual input is reduced • readers skip short words and words that are highly predictable (these words are identified in the parafoveal region) • readers regress (look back) • readers often undershoot on return sweeps (going from the end of a line to the next line) • the perceptual span is asymmetrical to the right (to the left for languages like Hebrew)

A TYPICAL READING EXPERIMENT Garden-path sentence Since Jay always jogs a mile seems like a short distance to him. Control sentence Since Jay always jogs a mile this seems like a short distance to him.

READING PATTERN (Garden-path sentence) Since Jay always jogs a mile seems like a short distance to him.

READING PATTERN (Garden-path sentence) Since Jay always jogs a mile seems like a short distance to him. = fixation after progressive saccade (first-pass) = fixation after regressive saccade = fixation after progressive saccade (second-pass)

READING PATTERN • If readers experience some sort of trouble they may fixate the difficult region longer and the may even regress to earlier parts of the sentence/text.

HOW DO WE INTERPRET THE READING PATTERNS?

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

DIFFERENT MEASURES • First fixation duration: duration of first fixation in a region • First-pass duration: time spent in a region before moving on or looking back • Regression path duration: time from first entering a region until moving the eyes beyond that region, includes regression time • Second-pass duration: duration of re-fixations • Total duration: the sum of all fixations in a region • Probability of a regression: the percentage of regressive eye-movements out of a region

EXPLANATION OF DIFFERENT MEASURES Bart annoyed Homer because… 1 2 5 3 4 6 Reading Times for word 3 (Homer) First fixation duration = 3 First-pass duration = 3 + 4 Regression Path duration = 3 + 4 + 5 Second-pass duration = 6 Total duration = 3 + 4 + 6 7

DIFFERENT FIRST-PASS MEASURES FIRST FIXATION DURATION FIRST-PASS DURATION REGRESSION PATH DURATION TIME

HOW DO WE INTERPRET THE READING TIMES?

THE LINKING PROBLEM eye mind

EYE-MIND ASSUMPTION (JUST & CARPENTER, 1980) • Readers retain fixation on a word until processing is completed • This includes processes like word recognition, syntactic parsing, semantic integration, referential integration

AN IDEAL WORLD (VAN BERKUM, 2004) Snowwhite W P S kissed a dwarf R W P S R TIME W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLD IS A REAL MESS (VAN BERKUM, 2004) Snowwhite W P S kissed a dwarf R W P W S P R S W R P S R TIME W = word recognition; P = parsing; S = semantic integration; R = referential integration

THE REAL WORLD IS A REAL MESS • In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc. ) • Thus, the effects of a manipulation are often visible more downstream (i. e. after the critical word or region). This is called spill-over.

THE REAL WORLD IS A REAL MESS • In the real world the processing of word X continues while fixating word X + 1 (and possibly while fixating word X + 2 etc. ) • Thus, the effects of a manipulation are often visible more downstream (i. e. after the critical word or region). This is called spill-over.

LINKING ASSUMPTION (BOLAND, 2004) • The eyes do not leave a word until it has been structurally integrated (tree building). Therefore, constraints that control structure-building influence first-pass reading time. other measures (e. g. , regression path duration) are sensitive for higher level processes (semantic integration, discourse processes)

SOLUTION LINKING PROBLEM? • Perhaps the different measures can provide information about what is happening? (this is an empirical question)

A DISADVANTAGE OF THE READING PARADIGM • You can only use skilled readers (no children or language-disordered populations). • This is possible in spoken language paradigms.

CAVEAT: • the eyes tell us that something is happening at a specific point in time, but not what that something is!

TO CONCLUDE • Eye-tracking (while reading and listening) excels in the “when” question. • Not really suited for “what” question (use EEG/MEG instead).

QUESTIONS?

PART 2: THEORY

CONSTRUCTING ANAPHORIC DEPENDENCIES: VARIABLE BINDING vs. CO-REFERENCE

BASIC ARCHITECTURE PROCESSING SYSTEM • distinct modules syntax, semantics, discourse • economy principle: cross-modular operations carry a cost (Reuland 2001) • syntax first in time-course-model of processing phases (Friederici 1995 -) syntax is cheaper than semantics is cheaper than discourse

CROSS-MODULAR OPERATIONS Discourse storage (values) C-I objects (variables) Syntactic objects (chains) Basic expressions a x 1 C 1 α Discourse storage (values) C-I objects (variables) Syntactic objects (chains) a x 1 C 1 Basic expressions α a x 2 C 2 β x 1 C 2 β C 1 β

TIME-COURSE SIMPLIFIED SYNTAX SEMANTICS DISCOURSE

SEMANTICS VS. DISCOURSE • variable binding (semantic dependency) Every clown thinks that he is not funny. • co-reference (discourse dependency) The clown has a big problem. He is not funny.

PREDICTIONS OF THEORY • variable binding is cheaper/faster than co-reference • in an ambiguous situation variable binding has precedence over co-reference

END OF INTRO