Introducing Psychometric AI As exploration of this avenue

Introducing Psychometric AI As exploration of this avenue proceeds. Selmer Bringsjord & Bettina Schimanski & …? Department of Cognitive Science Department of Computer Science RPI Troy NY 12180

Roots of this R&D…

Seeking to Impact a # of Fields • This work weaves together relevant parts of: – Artificial Intelligence: Build machine agents to “crack” and create tests. – Psychology: Use experimental methods to uncover nature of human reasoning used to solve test items. – Philosophy: Address fundamental “big” questions, e. g. , What is intelligence? Would a machine able to excel on certain tests be brilliant? … – Education: Discover the nature of tests used to make decisions about how students are taught what, when. – Linguistics: Reduce reasoning in natural language to computation. Many applications!

The Primacy of Psychology of Reasoning There is consensus among the relevant luminaries in AI and theorem proving and psychology of reasoning and cognitive modeling that: machine reasoning stands to the best of human reasoning as a rodent stands to the likes of Kurt Godel. In the summer before Herb Simon died, in a presentation at CMU, he essentially acknowledged this fact -- and set out to change the situation by building a machine reasoner with the power of first-rate human reasoners (e. g. , professional logicians). Unfortunately, Simon passed away. Now, the only way to fight toward his dream (which of course many others before him expressed) is to affirm the primacy of psychology of reasoning. Otherwise we will end up building systems that are anemic. The fact is that first-rate human reasoners use techniques that haven't found their way into machine systems. E. g. , humans use extremely complicated, temporally extended mental images and associated emotions to reason. No machine, no theorem prover, no cognitive architecture, uses such a thing. The situation is different than chess -radically so. In chess, we knew that brute force could eventually beat humans. In reasoning, brute force shows no signs of exceeding human reasoning. Therefore, unlike the case of chess, in reasoning we are going to have to stay with the attempt to understand replicate in machine terms what the best human reasoners do. We submit that a machine able to prove that the key in an LR/RC problem is the key, and that the other options are incorrect, is an excellent point to aim for, perhaps the best that there is. As a starting place, we can turn to simpler tests. Multi-Agent Reasoning, modeled in “Chess Mental Metalogic, is the key is Too to reaching Simon’s Dream! Easy” Pilot experiment shows that groups of reasoners instantly surmount the errors known to plague individual reasoners! Come next Wed 12 n SA 3205

What is Psychometric AI?

An Answer to: What is AI? • Assume the ‘A’ part isn’t the problem: we know what an artifact is. • Psychometric AI offers a simple answer: – Some artificial agent is intelligent if and only if it excels at all established, validated tests of intelligence. • Don’t confuse this with: “Some human is intelligent…” • Psychologists don’t agree on what human intelligence is. – Two notorious conferences. See The g Factor. • But we can agree that one great success story of psychology is testing, and prediction on the basis of it. (The Big Test)

Some of the tests…

Intelligence Tests: Narrow vs. Broad Spearman’s view of intelligence Thurstone’s view of intelligence

Let’s look @ RPM (Sample 1)

RPM Sample 2

RPM Sample 3

Artificial Agent to Crack RPM -------- PROOF --------1 [] a 33!=a 31. 3 [] -R 3(x)| -T(x)|x=y| -R 3(y)| -T(y). 16 [] R 3(a 31). 24 [] T(a 31). 30 [] R 3(a 33). 31 [] T(a 33). 122 [hyper, 31, 3, 16, 24, 30, flip. 1] a 33=a 31. 124 [binary, 122. 1, 1. 1] $F. ------ end of proof ------------ times (seconds) -----user CPU time 0. 62 (0 hr, 0 min, 0 sec)

Artificial Agent to Crack RPM -------- PROOF --------1 [] a 33!=a 31. 7 [] -R 3(x)| -Striped. Bar(x)|x=y| -R 3(y)| Striped. Bar(y). 16 [] R 3(a 31). 25 [] Striped. Bar(a 31). 30 [] R 3(a 33). 32 [] Striped. Bar(a 33). 128 [hyper, 32, 7, 16, 25, 30, flip. 1] a 33=a 31. 130 [binary, 128. 1, 1. 1] $F. ------ end of proof ------------ times (seconds) -----user CPU time 0. 17 (0 hr, 0 min, 0 sec)

Artificial Agent to Crack RPM Correct! ====== start of search ====== given clause #1: (wt=2) 10 [] R 1(a 11). given clause #2: (wt=2) 11 [] R 1(a 12). given clause #3: (wt=2) 12 [] R 1(a 13). . given clause #4: (wt=2) 13 [] R 2(a 21). given clause #278: (wt=16) 287 [para_into, 64. 3. 1, 3. 3. 1] R 2(x)| -R 3(a 23)| -Empty. Bar(y)| -R 3(x)| -Empty. Bar(x)| -T(a 23)| R 3(y)| -T(y). given clause #279: (wt=16) 288 [para_into, 65. 3. 1, 8. 3. 1] R 2(x)| -R 3(a 23)| -Striped. Bar(y)| -R 3(x)| -Striped. Bar(x)| Empty. Bar(a 23)| -R 3(y)| -Empty. Bar(y). Search stopped by max_seconds option. ====== end of search ======

Possible Objection “If one were offered a machine purported to be intelligent, what would be an appropriate method of evaluating this claim? The most obvious approach might be to give the machine an IQ test … However, [good performance on tasks seen in IQ tests would not] be completely satisfactory because the machine would have to be specially prepared for any specific task that it was asked to perform. The task could not be described to the machine in a normal conversation (verbal or written) if the specific nature of the task was not already programmed into the machine. Such considerations led many people to believe that the ability to communicate freely using some form of natural language is an essential attribute of an intelligent entity. ” (Fischler & Firschein 1990, p. 12)

WAIS A Broad Intelligence Test…

Cube Assembly Basic Setup Problem: Solution:

Harder Cube Assembly Basic Setup Problem: Solution:

Picture Completion Cur rent ly u ntou cha ble AI - - bu t we sha ll se e.

And ETS’ tests…

“Blind Babies” Children born blind or deaf and blind begin social smiling on roughly the same schedule as most children, by about three months of age. The information above provides evidence to support which of the following hypotheses: correct A. For babies the survival advantage of smiling consists in bonding the care-giver to the infant. B. Babies do not smile when no one is present. C. The smiling response depends on an inborn trait determining a certain pattern of development. D. Smiling between people basically signals a mutual lack of aggressive intent. E. When a baby begins smiling, its care-givers begin responding to it as they would to a person in conversation.

“Blind Babies” in Prop. Calc. 1 2 3 4 5 5 b 6 7 8 Pilot protocol analysis SSBB SS-SCHBBNB experiment indicates that SSBB (1; elim) high-performers represent these items at the level of SSL SSI the propositional calculus. But that level not detailed (SSBB SSL) SEE-SOMEONE enough for. BB generating the Items. VPA experiment SEEBB planned for this semester. SEEBB SEE-SOMEONEBB (5, 5 b; elim) 6 b (SSBB SSL) (6, 4 modus tollens) 6 c SSBB SSL (6 b, demorgan’s) SSL (6 c, 2; disjunctive syllogism) SSI (3, 7 disj. Syll. )

The Now Time-Honored “Lobster” Lobsters usually develop one smaller, cutter claw and one larger, crusher claw. To show that exercise determines which claw becomes the crusher, researchers placed young lobsters in tanks and repeatedly prompted them to grab a probe with one claw – in each case always the same, randomly selected claw. In most of the lobsters the grabbing claw became the crusher. But in a second, similar experiment, when lobsters were prompted to use both claws equally for grabbing, most matured with two cutter claws, even though each claw was exercised as much as the grabbing claws had been in the first experiment. Which of the following is best supported by the information above? A B C D E Young lobsters usually exercise one claw more than the other. Most lobsters raised in captivity will not develop a crusher claw Exercise is not a determining factor in the development of crusher claws in lobsters. Cutter claws are more effective for grabbing than are crusher claws. Young lobsters that do not exercise either claw will nevertheless usually develop one crusher and one cutter claw.

sentences 2 & 3 in text not needed for proof of correct option (A) But they are needed for proof that option C is inconsistent with text!! Sample Part of D(LRE) A. For babies the survival advantage of smiling consists in bonding the care-giver to the infant. B. Babies do not smile when no one is present. C. The smiling response depends on an inborn trait determining a certain pattern of development. D. Smiling between people basically signals a mutual lack of aggressive intent. Whereas in “Blind Babies” the foils all involve predicates E. When a baby begins smiling, its care-givers begin responding to it as they would to a presumably outside of R(LRE) person in conversation. e. g. ,

Same Approach Used -------- PROOF --------1 [] -Lobster(x)|Cutter(r(x)). 3 [] -Lobster(x)| -Exercise(r(x))| -Exercise(l(x))|Cutter(l(x)). 4 [] -Lobster(x)| -Cutter(r(x))| -Cutter(l(x)). 5 [] Lobster($c 1). Therefore option A 6 [] Exercise(r($c 1)). Is correct! 7 [] Exercise(l($c 1)). 9 [hyper, 5, 1] Cutter(r($c 1)). 10 [hyper, 7, 3, 5, 6] Cutter(l($c 1)). 11 [hyper, 10, 4, 5, 9] $F. ------ end of proof ------------ times (seconds) -----user CPU time 0. 38 (0 hr, 0 min, 0 sec)

Underlying Math …

Additional Objections…

Psychometric AI in Context …

A Classic “Cognitive System” Setup Under Development Cognitive System Test Item Choice of correct option, and ruling out of others, and… “percept” “action” actions that involve physical manipulation of objects and locomotion.

Fits forthcoming Superminds book by Bringsjord & Zenzen… • “Weak” AI based on testing going back to Turing is implied for the practice of AI.

Fits “Complete” Cog. Sci…

Perception and Action High-level Low-level Perception Environment subdeclarative computation Cognitive System Action

Cognitive Modeling Low-level High-level Perception Cognitive System Environment Perception & Action ACT-R Long Term Memory subdeclarative computation Short Term Memory Action

Reasoning Low-level High-level Perception & Action Cognitive System Environment Long Term Memory subdeclarative computation Short Term Memory ACT-R Semantic Reasoning Mental Metalogic Syntactic Reasoning Action

Cognitive Human Factors: Engineering the Interface b/t Cognitive Systems and their Environments Low-level High-level Perception & Action Cognitive System Environment Long Term Memory subdeclarative computation Short Term Memory ACT-R Semantic Reasoning Mental Metalogic Syntactic Reasoning Action

Should we consider IGERT? The distinctive graduate education provided by RPI’s Department of Cognitive Science could be that we provide a truly integrated Cog. Sci education: We produce students able to deal with cognitive systems top-to-bottom. A number of particular applications anchor this distinctive pedagogical approach, viz. , Psychometric AI, Synthetic Characters, Cognitive Prostheses, etc. . These are applications which, by their very nature, call for top-to-bottom Cog. Sci.

Large Variation in Difficulty

Evan’s ANALOGY Program