PenCentric Interfaces and Palm v Xerox Lawsuit Charles

Pen-Centric Interfaces and Palm v. Xerox Lawsuit Charles C. Tappert Seidenberg School of CSIS, Pace University 1

Goldberg Patent, Xerox, filed 1993, granted 1997 (“unistroke symbols”) 2

Graffiti Alphabet Palm Computing, 1995 3

Palm-Xerox Patent Infringement Lawsuit n n n The nine-year battle (1997 -2006) between Palm and Xerox over handwriting recognition ends, see article. Palm pays Xerox $22. 5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements. 4

Agenda 1. 2. 3. 4. 5. Online (Pen-Centric) Handwriting Recognition Experimental and Commercial Systems Shorthand – Past and Present Palm-Xerox Patent Infringement Lawsuit Potential New Research 1. 2. Enhanced pen-centric shorthand interface Multi-touch tablet interface 5

Part I Online (Pen-Centric) Handwriting Recognition 6

Part 1: Online (Pen-Centric) Handwriting Recognition n n n n Written Languages and Handwriting Properties The Fundamental Property of Writing Handwriting Recognition Difficulties Online (Pen-Centric) Handwriting Recognition Online more accurate than Offline Recognition Online Info Can Complicate Recognition Process Design Tradeoffs / Design Decisions Computer Problems in English 7

Written Language and Handwriting Properties n Alphabet n n Letters, digits, punctuation, special symbols Writing is a time sequence of strokes n n n Stroke – writing from pen down to pen up Usually complete one character before beginning the next Spatial order – e. g. , in English left to right 8

Fundamental Property of Writing n n n Differences between different characters are more significant than differences between different drawings of the same character This makes handwritten communication possible Can there be exceptions – say, different characters written identically? 9

Fundamental Property of Writing in English n n n Property holds within subalphabets of uppercase, lowercase, and digits, but not across them “I”, “l”, and “ 1” written with single vertical stroke “O” and “ 0” written similarly with an oval 10

Handwriting Recognition Difficulties n n n Shape, size, and slant variation Similarly shaped characters – U and V Careless writing n n in the extreme, almost illegible writing Resolving difficult ambiguities requires sophisticated recognition algorithms, syntax/semantics 11

Handwriting Recognition Difficulties n Segmentation ambiguities n n character-within-character problem lowercase d might be recognized as a cl if drawn with two strokes that are somewhat separated from one another 12

Online (Pen-Centric) Handwriting Recognition n Electronic tablets invented in late 1950 s n n Digitizer and display in separate surfaces Pen Computers arrived in 1980 s n n Combined digitizer and display Brought input and output into one surface Immediate feedback via electronic ink Created paper-like interface 13

Tablet Digitizers – Dynamic Information n n Pen down – indication of inking X-Y coordinates as function of time n n Sampling rate: 100 points per second Resolution: 200 points per inch 14

Early Pen-Centric Interface n n Different surfaces for input and output Rand system, about 1959 15

Pen Computers n IBM vision n n Paper-like interface, 1992 Microsoft Tablet PC n Launched, 2001 16

Pen-Centric PDAs n n Early Palm Pilot Palm Tungsten T 3 and Sony Clié TH 55 17

Online (Pen-Centric) Handwriting Recognition n n Machine recognizes the writing as the user writes Digitizer equipment captures the dynamic information of the writing n n Stroke number, order, direction, speed A stroke is the writing from pen down to pen up 18

Online (Pen-Centric) more accurate than Offline (Static) Recognition n n Can use both dynamic and static information Can often distinguish between similarly shaped characters n E. g. , 5 versus S where the 5 is usually written with two strokes and the S with one stroke 19

Online Information Can Complicate Recognition Process n Large number of possible variations n n E can be written with one, two, three, or four strokes, and with various stroke orders and directions A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions) 20

Online Information Can Complicate Recognition Process n Other variations 21

English Writing Styles n Handprint n n n Uppercase – about 2 strokes per letter Lowercase- about 1 stroke per letter Cursive Script n n Usually less than 1 stroke per letter Delayed crossing and dotting strokes 22

Design Tradeoffs/Decisions n No constraints on the user n n Machine recognizes user's normal writing User severely constrained n n Must write in particular style such as handprint Must write strokes in particular order, direction, and graphical specification 23

Computer Problems in English n Constrained Handprint n n n Unconstrained Handprint n n n Printing one symbol per box – form filling Printing on lines – symbols can touch or overlap No lines and symbols can touch or overlap Cursive Script Mixed Printing and Cursive 24

Computer Problems in English 25

Part 2 Experimental and Commercial Systems (focus on systems relevant to Palm-Xerox lawsuit) 26

Tappert 1990 Journal Article Surveyed 44 Systems n n n 11 experimental systems for handprint 4 experimental systems for cursive script 16 commercial systems for opaque tablets 5 commercial systems for pen computers 8 experimental applications systems n Spreadsheets, flowcharts, etc. 27

Tappert 1990 Survey Symbol Segmentation Methods n n Pen lift Move to new tablet area n n n Or new box when filling out forms Pause – Time out User hits finish button 28

Tappert 1990 Survey English Handprint Systems n Early systems handled only specific variations n n Later systems handled all common variations n n E. g. , Groner experimental system E. g. , CIC and Pencept commercial products Recent systems (e. g. , IBM, Linus, etc. ) n n Trained to user Built-in coverage of common variations 29

Tappert 1990 Survey Three Categories of Systems University project systems – least robust Industrial project systems – more robust Commercial products – most robust 1. 2. 3. Fred Brooks’ Mythical Man Month n Program – programming systems product 30

Tappert 1990 Survey Lawsuit Relevant Systems n n n Rand Experimental System 1966 – Groner Pencept Commercial Product 1980 s ATT Experimental System 1983 – Burr IBM Runon System 1984 – Tappert Linus Commercial Product 1987 – Sklarew 31

Rand Experimental System 1966 n n Different surfaces for input and output System created about 1959 Handwriting system article published 1966 Used author’s small number of alphanumeric symbol variations 32

Pencept Commercial Product 1980 s (all common symbol variations, one per box) 33

Pencept Commercial Product 1980 s 34

Burr Article 1983 n n Proposed a curve matching classification technique To avoid segmentation problems, users must draw each letter with one stroke n E. g. , draw i, j without dots and t, x without crossings 35

Tappert Handprint System 1984 Runon Patent filed 1986 Part of Think. Write software in IBM’s penenabled Think. Pad product in early 1990 s 36

Sklarew Patent filed 1987 Linus Commercial Product 1987 37

1992 – IBM Pen-Enabled Think. Pad with Think. Write Handwriting Recognizer n n 1992 - IBM Think. Pad Trans. Note pen computer, the first IBM model named “Think. Pad” It included the “Think. Write” software for the recognition of handprinted characters 38

Part 3 Shorthand – Past and Present 39

Part 3 Shorthand – Past and Present n n n Famous Uses of Shorthand Historical Shorthand Alphabets Pen-Centric Word/Phrase Shorthand Allegro/Chatroom Shorthand System n M. S. thesis that could be extended into a DPS dissertation 40

Famous Uses of Shorthand n Famous writings throughout history were effectively written in a style of shorthand n n n Cicero’s orations Martin Luther’s sermons Shakespeare’s and George Bernard Shaw’s plays Samuel Pepys’ diary Sir Isaac Newton’s notebooks 41

Historical Shorthand Alphabets n n n We first review the history of shorthand systems prior to pen computing Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases” Shorthand can be traced back to the Greeks in 400 B. C. 42

Historical Shorthand Alphabets n n We focus on shorthand alphabets that might be appropriate for PDAs We review two types of shorthand n Geometric shorthand n n n Small number of basic shapes Shapes reused in multiple orientations Non-geometric shorthand 43

Historical Shorthand Alphabets n n n Ancient Greeks – 400 BC Tironian Alphabet – 63 BC John Willis’s Stenography – 1602 Gabelsberger Alphabet – 1834 Moon Alphabet – 1845 44

Tironian Alphabet, 63 B. C. Non-Geometric 45

Stenography Alphabet, 1602 46

Stenography Alphabet, 1602 n Basic Shapes and Orientations 47

Gabelsberger Cursive-Style, 1834 Non-Geometric Alphabet 48

Moon Geometric Alphabet, 1845 49

Other Historical Shorthand Systems n Phonetic alphabets n n n Pitman (1837), was popular in UK Gregg (1888), was popular in USA Systems for the blind n Braille (1821) 50

Pen-Centric Shorthand Alphabets n Some of the earliest were for CAD/CAM n n n symbols represent graphical items and commands Others developed for text input on small consumer devices like PDAs that have limited computing power We review geometric and non-geometric shorthands appropriate for small devices 51

Pen-Centric Shorthand Alphabets n Historical alphabets presented above could be used for machine recognition symbols drawn with a single stroke (except “K” in Tironian and “+” in Stenography) n n In addition to shape and orientation, online systems can use stroke direction to differentiate among symbols 52

Pen-Centric Shorthand Alphabets n Geometric Pen-Centric Shorthands n n Organek – 1991 Allen – filed 1991, patent 1993 Goldberg (Xerox) – filed 1993, patent 1997 Non-Geometric Pen-Centric Shorthands n n Graffiti (Palm Computing) – 1995 Allegro (Papyrus) – 1995 53

Organek Alphabet, 1991 54

Organic Alphabet, 1991 Basic Shapes and Orientations One shape 8 directions together with 3 lengths provides 24 symbols A second wheel provides additional symbols 55

Allen patent, filed 1991 56

Allen patent, filed 1991 Basic Shapes and Orientations only linear shapes 57

Goldberg patent, Xerox, filed 1993 (“unistroke symbols”) 58

Goldberg patent, filed 1993 Basic Shapes and Orientations 59

Goldberg patent, filed 1993 n 5 Basic shapes 4 Orientations 2 Stroked Directions 40 Possible Symbols Designed for Speed of Input and Maximum Symbol Separation 60

Shorthand Alphabet Design n n How would you design a shorthand alphabet? What would be the design criteria? 61

Design of Graffiti Alphabet for the Palm Pilot n Small alphabet n n n Uppercase, digits, special symbols One stroke per symbol to avoid segmentation difficulty Separate writing areas for letters and digits to avoid same-shape confusions 62

Graffiti Alphabet Palm Computing, 1995 63

Graffiti Mimics Keyboard Input n n Character by character input Mode shifts for n n n Uppercase Special characters Eyes can focus on application’s insertion point rather than on input area 64

Graffiti Alphabet Designed for Ease of Learning n n 21 letters match the Roman alphabet 5 remaining ones match partially (Mac. Kenzie, “The Immediate Usage of Graffiti, ” 1997) 65

Graffiti Alphabet: 11 of 26 characters have alternate variations 66

Frequently Confused Characters 67

Other Low Performance Characters 68

Symbol Overlap Comparison 69

Graffiti Alphabet Designed for Ease of Learning frequency of use (Mac. Kenzie, “The Immediate Usage of Graffiti, ” 1997) 70

Allegro Alphabet (Papyrus), 1995 (now Microsoft) 71

Simplified Design Tradeoffs/Decisions for Graffiti and Allegro PDA Alphabets n Small alphabet n n One stroke per character (character = stroke) n n preferably one Separate writing areas for letters and digits n n allows machine to recognize each character upon pen lift Small number of writing variations per letter n n one case rather than both upper and lowercase avoids confusion of similarly shaped letters and digits High correspondence to Roman alphabet for ease of learning n non-geometric, might not actually qualify as shorthand 72

Commercially Successful Shorthands n Similar to the Roman alphabet n Easy to learn Graffiti used in Palm OS devices n notably the Palm Pilot & Handspring models n Allegro used in Microsoft Windows devices n n Geometric alphabets not successful 73

Current Commercial Systems Company/System Writing Style Palm Computing/Graffiti* Special Shorthand Alphabet Microsoft/Papyrus Allegro Special Shorthand Alphabet CIC/Jot Relatively Unconstrained Handprint Microsoft Relatively Unconstrained Handprint and Cursive *A few years ago Palm switched from Graffiti to Graffiti 2, Graffiti 2 is basically Jot licensed from CIC. 74

Part 4 Palm-Xerox Patent Infringement Lawsuit 75

Goldberg Patent (Xerox), filed 1993 (“unistroke symbols”) n Goldberg (Xerox) Patent 5596656 n n n http: //patft. uspto. gov/ Claims: system, method, process Originally filed 1993, denied Re-filed 1995 Patent granted 1997 Palm v. Xerox Lawsuit 1997 -2006 n Patent Infringement – Graffiti alphabet 76

Goldberg Patent, Xerox, filed 1993, granted 1997 (“unistroke symbols”) 77

Graffiti Alphabet (Palm), 1995 78

Jeff Hawkins, innovator n n n 1979 BSEE Cornell, 1979 -1986 worked at Intel and GRi. D 1986 -1987 ABD Bio. Physics doctoral program, U. C. Berkeley 1987 - back at GRi. D he created GRi. DPAD, first pen computer 1992 formed Palm Computing, 1993 created first handwriting reco software product for a mobile handheld - Casio’s Zoomer 1995 Palm Computing bought by U. S. Robotics 1996 created Palm. Pilot, first PDA with Graffiti shorthand alphabet (over a million shipped in 18 months, a 66% market share, and the fastest growth of any computing product in history, faster than the TV and the VCR) 1997 U. S. Robotics bought by 3 Com (sued by Xerox for patent infringement) 1998 left Palm to form Handspring, 1999 launched the Visor handheld 2000 Palm Computing spun off by 3 Com 2002 created what is now the Redwood Center for Theoretical Neuroscience 2003 Handspring (with Hawkins, et al. ) acquired by Palm Computing 2005 Founded Numenta to build the ultimate brain-like machine 79

Goldberg versus Graffiti (Mac. Kenzie, “The Immediate Usage of Graffiti, ” 1997) 80

Palm’s Defense Approach n Invalidity based on prior art n n n Where claims interpreted to cover any alphabet of single, unbroken strokes Or where claims interpreted to cover an alphabet of the symbols shown in the patent Non-infringement n After the invalidity arguments limit the interpretation of the claims, the Graffiti alphabet does not infringe 81

Goldberg Patent Claims n 1. A system for interpreting handwritten text n User interface … pointer for writing n n n n mutually independent unistroke symbols some being linear and others arcuate each representing a predefined textual component delimiting operation distinguishing symbols from each other independent of their spatial relationship Sensor mechanism coupled to user interface Recognition unit Display & character generator to display output 82

Simplified Claim’s Chart for references against 656 patent Claim 656 Burr Allen IBM Y Y N Y mutually independent unistroke symbols Y Y Y * symbol independent delimiting operation Y Y Y * some unistroke symbols linear and others arcuate Y Y N Y each unistroke symbol is a predefined textual component N Y Y * alphabet of mutually independent unistroke symbols Y Y Y * stroke direction parameter Y Y some symbols differ only in stroke direction Y * N Y Y N 1. System: UI, tablet, pointer, x-y coord, reco unit, display 2. 1 + planar writing surface, etc. 9. 2 -6 + unistrokes well separated in sloppiness space 10. Method: 12. Process: unistrokes are natural language alphanumeric symbols 83

Simplified Claim’s Chart for references against 656 patent Claim 656 Burr IBM Linus Y Y mutually independent unistroke symbols Y Y * * symbol independent delimiting operation Y Y * * some unistroke symbols linear and others arcuate Y Y each unistroke symbol is a predefined textual component N Y * * alphabet of mutually independent unistroke symbols Y Y * * stroke direction parameter Y Y some symbols differ only in stroke direction Y * * * N Y N N 1. System: UI, tablet, pointer, x-y coord, reco unit, display 2. 1 + planar writing surface, etc. 9. 2 -6 + unistrokes well separated in sloppiness space 10. Method: 12. Process: unistrokes are natural language alphanumeric symbols 84

Simplified Claim’s Chart for references against 656 patent Claim 656 Tiro Allen Graf fiti Y N N Y mutually independent unistroke symbols Y * symbol independent delimiting operation Y * Y Y some unistroke symbols linear and others arcuate Y Y N Y each unistroke symbol is a predefined textual component N * Y * alphabet of mutually independent unistroke symbols Y * stroke direction parameter Y * Y Y some symbols differ only in stroke direction Y * N Y Y N 1. System: UI, tablet, pointer, x-y coord, reco unit, display 2. 1 + planar writing surface, etc. 9. 2 -6 + unistrokes well separated in sloppiness space 10. Method: 12. Process: unistrokes are natural language alphanumeric symbols 85

Goldberg-Graffiti Differences Design Criterion Goldberg Graffiti Geometric alphabet Designed for fast input Y Y N N Easy to learn Like Roman alphabet All single stroke symbols Highly separable symbols Alternate variations N N Y Y * N Y 86

Palm-Xerox Patent Infringement Lawsuit n n n The nine-year battle (1997 -2006) between Palm and Xerox over handwriting recognition ends, see article. Palm pays Xerox $22. 5 million for a fully paid-up license for Xerox patents covering its text input Unistrokes technology Xerox first sued Palm predecessor Palm Computing back in April 1997, claiming that the Graffiti text-entry system used in its PDAs infringed on patents for Unistrokes, which allows users to input letters and numbers into personal data units with basic, one stroke movements. 87

Conclusions Palm-Xerox Patent Infringement Lawsuit n Invalidity n n n Infringement n n Historical research showed that Goldberg alphabet not so unique Even though the patent was accepted as valid, these arguments narrowed the scope of the patent Analyses and comparisons of the Goldberg and Graffiti alphabets showed major differences Result was favorable settlement for Palm 88

Part 5 Potential New Research n n Enhanced pen-centric shorthand interface Multi-touch tablet interface 89

Enhanced Pen-Centric Shorthand Interfaces n n Can use word/phrase shorthand to speed text input Can provide critical infrastructure for many pen-centric applications Can enhance natural pen-centric interactions for many applications Will have greatest impact on the utility of applications running on small mobile devices, such as email 90

Chatroom Shorthand Examples CU CM @TEOTD ^5 2 nite LOL ASAP B/C or BC See you, or Cracking up Call me At the end of the day High five Tonight Laughing out loud As soon as possible Because 91

Allegro/Chatroom Shorthand System n Developed for M. S. dissertation n n Student was hearing impaired Developed as output component of communication system n n Handwriting to text to speech Two input writing areas n n One for Allegro (all-purpose) One for chatroom-like or user-defined words/phrases 92

Allegro/Chatroom Shorthand System 93

Allegro/Chatroom Shorthand System 94

Allegro/Chatroom Shorthand System M. S. Thesis Experimental Results n Allegro/Chatroom pen-centric shorthand input is faster than typing text and is comparable to typing text and chatroom shorthand characters 95

Multi-Touch Tablet Interface n To be explored 96

Conclusions Pen-Centric Shorthands n n Pen-centric interfaces should use shorthand, and especially word/phrase shorthand multi-touch, for fast text input Benefit of shorthand interfaces n n n Provides critical infrastructure for many pen-centric applications Enhances natural pen-centric interactions for many applications Has greatest impact on the utility of applications running on small mobile devices 97

Conclusions Handwriting Recognition n n Graffiti and Allegro greatly simplified the recognition problem Handprint problem not completely solved n n Even with IBM’s Think. Write, CIC’s Jot, and Microsoft products Cursive script problem clearly not solved 98

References n n n W. B. Huber, S. -H. Cha, C. C. Tappert, and V. L. Hanson, "Use of Chatroom Abbreviations and Shorthand Symbols in Pen Computing, " Proc. 9 th Int. Workshop on Frontiers in Handwriting Recognition , IWFHR 2004, Tokyo, Japan, October 2004. W. Huber, V. Hanson, S. Cha, and C. C. Tappert, "Common Chatroom Abbreviations Speed Pen Computing, " Proc. 11 th Int. Conf. Human. Computer Interaction, Las Vegas, NV, July 2005. C. C. Tappert and S. Cha, "Handwriting Recognition Interfaces, " Chapter 6, pp. 123 -137, in Text Entry Systems, Scott Mac. Kenzie and Kumiko Tanaka-Ishii (Eds. ), Morgan Kaufmann, 2007. C. C. Tappert, C. Y. Suen, and T. Wakahara, "The state-of-the-art in online handwriting recognition, " IEEE Trans. Pattern Analysis Machine Intelligence, Vol. PAMI-12, pp. 787 -808, August 1990. C. C. Tappert and J. R. Ward, "Pen-Centric Shorthand Handwriting Recognition Interfaces, " Proc. 1 st Int. Workshop on Pen-Based Learning Technologies, Catania, Italy, May 2007. 99