Using interviews to investigate implicit knowledge in computer

Using interviews to investigate implicit knowledge in computer programming Amber Settle Educational research seminar September 29, 2006

Topic of talk Rebecca Mancy and Norman Reid n “Using interviews to investigate implicit knowledge in computer programming” n Appeared in the International Conference of the Learning Sciences in July 2006 n http: //portal. acm. org/citation. cfm? id=1150034. 1150101&coll=&dl=ACM&CFID=1515 &CFTOKEN=6184618 n

ICLS n International Conference of the Learning Sciences http: //www. isls. org/ n http: //www. isls. org/icls 2006/ n n Focus of the conference is both K-12 and college-level education

Brief history of implicit learning n Term first surfaced in 1967 A. S. Reber, “Implicit learning of artificial grammars”, Journal of Verbal Learning and Verbal Behavior n M. Polanyi, The Tacit Dimension n Investigated in both philosophy and psychology n Mancy and Reid focus on psychological literature n n Interested in the processes by which it is gained

Reber n Artificial Grammar Learning experiments n Learning phase n Subjects see strings of letters generated by finite-state machines n They then carry out a task on them such as repeating them n Testing phase n Informed of existence of an underlying grammar n See letter strings and classify them as belonging or not OR n See pairs of letter strings, are told that one belongs and one does not, and must identify the one that does belong

Reber n Results n n n Participants are able to correctly classify strings at above chance level Participants cannot articulate the rules In short: n n n Act as though they possess abstract rules Rules are not declarative Participants claim no awareness of the rules Numerous subsequent studies have unambiguously demonstrated the same phenomenon Most studies concentrated on somewhat superficial forms of implicit learning

Implicit learning & education n Existence of two modes of learning With conscious awareness (explicit) n Without conscious awareness (implicit) n n Important considerations: Efficiency: Finding the best mode for a given topic n Strategies: Encouraging both modes of learning n Assessment: Testing implicit knowledge is difficult n

Implicit learning & education n n Little research on implicit learning and education Possible reasons: n n Assumption that explicit, articulate thinking is the most powerful form of cognition Narrow focus of research on implicit learning n n Rooted in experimental psychology Control as many parameters as possible Tasks that are far removed from complex, real-life learning situations Mancy and Reid: Develop new research methods that can be applied to real-life situations n Interviews

Problems with interviews Interview: Verbal report of some knowledge n Issues n n Verbal reports may not correspond to actual knowledge if not all memory is reported n Ericsson & Simon, 1980 n What is reported depends on a subject’s perception of their understanding n Nisbett & Wilson, 1977 n Offer explanations plausible to them n Report nothing or an alternative explanation

Problems with interviews n Issues (continued) n Conscious and unconscious knowledge may cooccur n Subject may offer conscious knowledge n Does not imply unconscious knowledge is not present n Explicit knowledge may be offered but not govern behavior when faced with a task using the skills n Potential n Verbal reports have been used in areas such as sensory processes, imagery and immediate memory

Mancy and Reid n Context First-year students n Introductory Ada course n University of Glasglow n 2003 -2004 n 12 (male) students recruited through requests during lecture and e-mails n Conducted near the end of the course n Paid for their assistance n

Mancy and Reid n Interview questions n n Decision points in programming Questions n n n Examples n n Asked to choose between possible language structures As a follow up, had they thought about this issue previously “How do you decide when to use a semicolon? ” “Can you explain how you use indentation, how you indent your code, when you write it? ” “How do you decide when to use an array? ” Topics n n Explicit rule: When to choose a for versus a while loop No explicit rule: When to choose an array

Mancy and Reid n Process n n Semi-structured interviews Hour in length Tape recorded Transcribed in full by investigators n n Attention paid to aspects such as hesitation Grounded-theory approach n n Identify statements that seem to indicate implicit knowledge Later categorize them according to indicators of the implicit status of that knowledge

Mancy and Reid n Indicators of implicit knowledge Little or no knowledge about how a skill is used n Claims of difficulty in articulating knowledge n Reliance on concrete examples n Use of language n Surprise n

Mancy and Reid n Little or no knowledge about how a skill is used n Interaction n Interviewer: “How did you decide when to use a semicolon? ” n Victor: “… I don’t, I just do it. Occasionally like … erm, I just … it’s when, it comes naturally when I’m typing something. ” n No conscious consideration or deliberation

Mancy and Reid n Reliance on concrete examples n Interaction n n Halsey (in response to a question about using arrays): “… actually when you use an array it’s. . useful for. . erm … lots. . of the same thing held like, if you’re wanting to … hold the number of cups of tea you drink in a day, you would have like you know the numbers would be day one to seven would be your day. . and inside each would be your number of cups of tea. ” Features n n n Much more confident when describing the example Example not considered in lecture Mathematics education: Concrete examples as prototypes (Tall & Pinto, 2002)

Mancy and Reid n Surprise n Interaction n n n Elwin (Asked to describe what might consitute a condition in a while loop): “… I suppose it could be any kind of expression sort of to do with … while E is equal to five, or … take it there’s gonna be a value to come back? [aha!] Perhaps, does it always have to come back with a Boolean value, then? ” Interviewer: “It does. ” Elwin: “Yeah, it does, doesn’t it! [Sounds surprised]” Interviewer: “You’d not thought about that before? ” Elwin: “No … no! [laughs]” Elwin demonstrates “noticing”, where implicit knowledge becomes explicit (Schmidt 1990; Truscott, 1998)

Mancy and Reid n Implicit knowledge lends itself to explanation using examples n is very loosely structured, with ideas linked associatively rather than logically n n Conclusions about process Subtle clues can be used to show implicit learning n Interviews are a richer source of information n

Mancy and Reid n Suggest a graduated scale of conscious status of knowledge No awareness of any knowledge n Awareness of knowledge, but no awareness of content n Some awareness of knowledge, but it is structured in a way that makes it difficult to be communicated n Structured declarative or explicit knowledge n

Mancy and Reid n Shortcoming Rate of occurrence of unambiguous evidence for implicit knowledge is low n Method is inefficient n n Future work Change type and wording of questions n Enable respondents to give pertinent answers n n Note: Interview questions not included in paper

Upcoming seminar events n Friday, October 27 th, time and location TBA n n n Early November: Distance learning panel n n Workshop on teaching programming Presenter: Keith Whittington of RIT (http: //www. rit. edu/) Dr. Whittington has an NSF grant to study the teaching of programming and will also be here on Thursday, October 26 th to meet with faculty on an individual basis Tell me TODAY if you want to meet with him on Oct. 26 th or 27 th Organized by Evelyn Lulis in conjunction with the Distance Learning Committee Topic: Excellence in Teaching Distance Learning Panelists: TBA Watch the Research Seminar page for details