Invivo research on learning Charles Perfetti PSLC Summer

  • Slides: 25
Download presentation
In-vivo research on learning Charles Perfetti PSLC Summer School 2007 1

In-vivo research on learning Charles Perfetti PSLC Summer School 2007 1

When is a learning study invivo? n n Vitro, vivo “On-Line” course? An ITS?

When is a learning study invivo? n n Vitro, vivo “On-Line” course? An ITS? A real class; real students; an intervention that counts. PSLC summer school 2007 2

Why in-vivo is the gold standard n n n Noisy, uncontrolled environment Content of

Why in-vivo is the gold standard n n n Noisy, uncontrolled environment Content of instruction is validated Built in generalization to classroom learning PSLC summer school 2007 3

Problems faced by an in-vivo researcher n n Noisy, uncontrolled environment As for your

Problems faced by an in-vivo researcher n n Noisy, uncontrolled environment As for your experiment: n n Students have other things to do Instructors have other things to do PSLC summer school 2007 4

Examples of in-vivo studies n n Algebra, Physics, Chemistry, Geometry, French, Chinese, English Some

Examples of in-vivo studies n n Algebra, Physics, Chemistry, Geometry, French, Chinese, English Some with computer tutors in major role n n n ITS Practice tutors Some without tutors or tutors in minor role PSLC summer school 2007 5

Liu, Wang, Perfetti Chinese tone perception study n In-vivo study n n n Traditional

Liu, Wang, Perfetti Chinese tone perception study n In-vivo study n n n Traditional classroom (not online) Materials from students’ textbook New materials each week for 8 weeks of term 1 Term 2 continued this, and added novel syllables unfamiliar to the student 3 instructional conditions n n tone number + pin yin, contour + pin-yin; contour only Hint system (CTAT) Tutors presented materials in 3 different instructional interfaces, according to the 3 conditions Data shop logged individual student data PSLC summer school 2007 6

Illustration of 2 conditions from Liu et al shi PSLC summer school 2007 7

Illustration of 2 conditions from Liu et al shi PSLC summer school 2007 7

Learning Measures n n Across-session error rates (transfer to new items) Post-test tone judgments

Learning Measures n n Across-session error rates (transfer to new items) Post-test tone judgments presented by tutor n n Two successive syllables heard. Are they same or different in tone? (transfer to different task) Nature of syllable pairs n n Tone same, segments different /duan/3 /liang/3 Same onset and rime, shi 2 -- shi 3; Share rime only, e. g. dao 2 – kao 3; Share neither onset nor rime, e. g. , duo 2 -- gong 3. PSLC summer school 2007 8

Studies with major role for a computer tutor n n n Formative evaluation. How

Studies with major role for a computer tutor n n n Formative evaluation. How can the tutor be improved? Summative evaluation. Is the tutor effective? Both of these apply to all instructional interventions, whether tutor based or not PSLC summer school 2007 9

Formative Evaluation Examples n User interface testing n n Early, before the rest of

Formative Evaluation Examples n User interface testing n n Early, before the rest of the tutor is built Engage students and instructors Get detailed response from students viewing tutor with talk-aloud procedures Wizard of Oz n n n Human (the Wizard) in the next room watches a copy of screen Responds when student presses Hint button or makes an error User interface evaluation n Does the wizard have enough information? Can the wizard intervene early enough? Tutor tactics evaluation. What did the Wiz do when? PSLC summer school 2007 10

Formative Example 3: Snapshot critiques n Procedure: ITS log file n n n Examine

Formative Example 3: Snapshot critiques n Procedure: ITS log file n n n Examine match between help from experts and that from ITS. n n Select student help events from log file Experts examine context leading up to the help message noting the help they would provide Compare with match between two experts. Modify ITS help messages according to reliable expert input. PSLC summer school 2007 11

Summative evaluations n n Question: Is the tutor (or other instructional intervention) more effective

Summative evaluations n n Question: Is the tutor (or other instructional intervention) more effective than a control? Typical design n n Experimental group gets the instructional intervention (the tutor). Control group learns via the “traditional” or “current practice” method Pre & post tests Data analysis n Did the tutor group “do better” than the control? PSLC summer school 2007 12

Control conditions n Typical control conditions n n n Existing classroom instruction Textbook &

Control conditions n Typical control conditions n n n Existing classroom instruction Textbook & exercise problems (feedback? ) Another tutoring system Human tutoring Define a control condition early n n Study the existing instruction in detail Results of this study should influence the design of the tutor PSLC summer school 2007 13

Learning Assessments n n n Pre-test Immediate post-test (post-pre = Learning) Delayed post-test (Long-term

Learning Assessments n n n Pre-test Immediate post-test (post-pre = Learning) Delayed post-test (Long-term retention) n n n How long is long? Post-test using new dimension (content, presentation mode, response mode, etc/) (Transfer) Learning measures on new content (Accelerated future learning) PSLC summer school 2007 14

Data from Liu et al tone study Learning curves week-by-week 2 nd term transfer

Data from Liu et al tone study Learning curves week-by-week 2 nd term transfer items PSLC summer school 2007 15

Multiple kinds of transfer n Liu et al shows 2 kinds of materials transfer

Multiple kinds of transfer n Liu et al shows 2 kinds of materials transfer n n Within term 1, learning sessions, each syllable to be learned was different but familiar. So transfer of learning to familiar items At second term, there were unfamiliar syllables. So transfer of learning to unfamiliar items. (Not so good. ) PSLC summer school 2007 16

Example of acceleration of future learning (Min Chi & Van. Lehn) n First probability,

Example of acceleration of future learning (Min Chi & Van. Lehn) n First probability, then physics. During probability only, n Accelerated future learning Score n Half students taught an explicit strategy Half not taught a strategy (normal instruction) Ordinary transfer Pre Post Probability Training PSLC summer school 2007 Pre Post Physics Training 17

Composing a post-test n General strategy: n n Include some items from the pre-test

Composing a post-test n General strategy: n n Include some items from the pre-test n n Check for basic learning Some items similar to training items n n Guided by cognitive task analysis (pre-test as well) including learning goals and specific knowledge components Measures near-transfer Some problems dissimilar to training problems n Measures far-transfer PSLC summer school 2007 18

Mistakes to avoid in test design n Tests that are n n Notice variances

Mistakes to avoid in test design n Tests that are n n Notice variances Too difficult Too easy Too long Notice problems in test means Tests that n Fail to represent instructed content n n Missing content; over sampling from some content Depend too much on background knolwedge PSLC summer school 2007 19

Interpreting test results as learning n Post-test in relation to pre-test. 2 strategies: n

Interpreting test results as learning n Post-test in relation to pre-test. 2 strategies: n ANOVA on n gain scores n n First check pre-test equivalence Not recommended if pre-tests not equivalent Pre-test, post test as within-subjects variable (t-tests for non-independent samples) ANCOVA. Post-tests scores are dependent variable; pre-test scores are co-variate PSLC summer school 2007 20

Plot learning results n Bar graphs for instructional conditions n n Differences due to

Plot learning results n Bar graphs for instructional conditions n n Differences due to conditions Learning Curves n Growth over time/instruction PSLC summer school 2007 21

Bar graphs (with error bars!) PSLC summer school 2007 22

Bar graphs (with error bars!) PSLC summer school 2007 22

Learning Curves Error rate Weekly sessions over 2 terms PSLC summer school 2007 23

Learning Curves Error rate Weekly sessions over 2 terms PSLC summer school 2007 23

Learning curves can pin-point intervention effects B 1 -2: implicit B 3 -4: explicit

Learning curves can pin-point intervention effects B 1 -2: implicit B 3 -4: explicit EXPLICIT RULES (< B 3) 24

The end 25

The end 25