Nat Link A Python Macro System for Dragon

Nat. Link: A Python Macro System for Dragon Naturally. Speaking Joel Gould Director of Emerging Technologies Dragon Systems 1

Copyright Information 4 This is version 1. 1 of this presentation – Changes: look in corner of slides for V 1. 1 indication 4 This version of the presentation was given to the Voice Coder’s group on June 25, 2000 4 The contents of this presentation are © Copyright 1999 -2000 by Joel Gould 4 Permission is hereby given to freely distribute this presentation unmodified 4 Contact Joel Gould for more information joelg@alum. mit. edu 2

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 3

What is Naturally. Speaking? 4 World’s first and best large vocabulary continuous speech recognition system 4 Primarily designed for dictation by voice 4 Also contains fully functional continuous command recognition (based on SAPI 4) 4 Professional Edition includes simple basiclike language for writing simple macros 4

What is Python? 4 Interpreted, object-oriented pgm. language 4 Often compared to Perl, but more powerful 4 Free and open-source, runs on multiple OSs 4 Ideal as a macro language since it is interpreted and interfaces easily with C 4 Also used for web programming, numeric programming, rapid prototyping, etc. 5

What is Nat. Link? 4 A compatibility module (like Nat. Text): – Nat. Link allows you to write Nat. Speak command macros in Python 4 A Python language extension: – Nat. Link allows you to control Nat. Speak from Python 4 Works with all versions of Nat. Speak 4 Free and open-source, freely distributable* 6

*Licensing Restrictions 4 Nat. Link requires that you have a legally licensed copy of Dragon Naturally. Speaking 4 To use Nat. Link you must also agree to the license agreement for the Nat. Speak toolkit – Soon Natlink will require the Nat. Speak toolkit – The Nat. Speak toolkit is a free download from http: //www. dragonsys. com V 1. 1 7

Nat. Link is Better than Prof. Ed. 4 Grammars can include alternates, optionals, repeats and nested rules 4 Can restrict recognition to one grammar 4 Can change grammars at start of any recog. 4 Can have multiple macro files 4 Changes to macro files load immediately 4 Macros have access to all features of Python 8

Nat. Link is Harder to Use 4 Nat. Link is not a supported product Do not call Tech Support with questions 4 Nat. Link may not work with Nat. Speak > 5 – It will work fine with Nat. Speak 5. 0 V 1. 1 4 Documentation is not complete 4 No GUI or fancy user interface 4 Requires some knowledge of Python 4 More like real programming 9

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 10

What you Need to Install 4 Dragon Naturally. Speaking – Any edition, version 3. 0 or better 4 Python 1. 5. 2 for Windows: py 152. exe from http: //www. python. org/ – You do not need to install Tcl/Tk 4 Nat. Link: natlink. zip from http: //www. synapseadaptive. com/joel/default. htm 4 Win 32 extensions are optional: win 32 all. exe from http: //www. python. org/ 11

Setting up Nat. Link 4 Install Nat. Speak and Python 4 Unzip natlink. zip into c: Nat. Link 4 Run Nat. LinkMacro. SystemEnable. NL. exe – This sets the necessary registry variables – This also turns Nat. Link on or off 4 To run sample macros, copy macro files – From: Nat. LinkSample. Macros – To: Nat. LinkMacro. System 12

How to Create Macro Files 4 Macro files are Python source files 4 Use Wordpad or any other text editor – save files as text with. py extension 4 Global files should be named _xxx. py 4 App-specific files should be named with the application name (ex: wordpad_xxx. py) 4 Copy files to Nat. LinkMacro. System – Or to Nat. SpeakUsersusernameCurrent 13

Sample Example 1 4 File _sample 1. py contains one command 4 Say “demo sample one” and it types: Heard macro “sample one” 14

Source Code for _sample 1. py import natlink from natlinkutils import * class This. Grammar(Grammar. Base): This is the grammar. You can say: “demo sample one” gram. Spec = """ <start> exported = demo sample one; """ def got. Results_start(self, words, full. Results): natlink. play. String('Heard macro "sample one"{enter}') def initialize(self): self. load(self. gram. Spec) self. activate. All() this. Grammar = This. Grammar() this. Grammar. initialize() def unload(): global this. Grammar if this. Grammar: this. Grammar. unload() this. Grammar = None This is the action. We type text into the active window. Most of the rest of this file is boiler plate. 15

Sample Example 2 4 Add a second command with alternatives 4 Type (into application) the command alternative which was recognized 4 Nat. Link will tell you which rule was recognized by calling a named function – got. Results_first. Rule for <first. Rule> – got. Results_second. Rule for <second. Rule> 16

Extract from _sample 2. py #. . . class This. Grammar(Grammar. Base): This is the grammar. It has two rules. gram. Spec = """ <first. Rule> exported = demo sample two [ help ]; <second. Rule> exported = demo sample two ( red | blue | green | purple | black | white | yellow | orange | magenta | cyan | gray ); What we """ do when “first. Rule” is heard. def got. Results_first. Rule(self, words, full. Results): natlink. play. String('Say "demo sample two {ctrl+i}color{ctrl+i}"{enter}') def got. Results_second. Rule(self, words, full. Results): natlink. play. String('The color is "%s" {enter}'%words[3]) def initialize(self): self. load(self. gram. Spec) self. activate. All() #. . . What we do when “second. Rule” is heard. Words[3] is the 4 th word in 17 the result.

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 18

Strings and Things 4 String constants can use either single quote or double quotes 'This is a string' "This string has a single quote (') inside" 4 Use triple quotes for multiple line strings """line 1 of string line 2 of string""" 4 Plus will concatenate two strings 'one'+'two'='onetwo' 4 Percent sign allows sprintf-like functions 'I heard %d' % 13 = 'I heard 13' 'the %s costs $%1. 2 f' % ('book', 5) = 'the book costs $5. 00' 19

Comments and Blocks 4 Comments begin with pound sign # Comment from here until end of line print 'hello' # comment starts at pound sign 4 Blocks are delimited by indentation, the line which introduces a block ends in a colon if a==1 and b==2: print 'a is one' print 'b is two' else: print 'either a is not one or b is not two' x = 0 while x < 10: print x x = x + 1 print 'all done' 20

Lists and Loops 4 Lists are like arrays; they are sets of things 4 Uses brackets when defining a list my. List = [1, 2, 3] another = ['one', 2, my. List] 4 Use brackets to get or change a list element print my. List[1] print another[2] # prints 2 # prints [1, 2, 3] 4 The “for” statement can iterate over a list total = 0 for x in my. List: total = total + x print x # prints 6 (1+2+3) 21

Defining and Calling Functions 4 Use the “def” statement to define a function 4 List the arguments in parens after the name def global. Function(x, y): total = x + y print 'the total is', total 4 Example of a function call global. Function(4, 7) # this prints "the total is 11" 4 Return statement is optional def add. Numbers(x, y) return x + y print add. Numbers(4, 7) # this prints "11" 22

Modules and Classes 4 Call functions inside other modules by using the module name before the function import string print string. upper('word') 4 Define classes with “class” statement and class functions with “def” statement class My. Class: def local. Function(self, x): print 'value is x' object = My. Class # create instance of My. Class object. local. Function(10) # prints "value is 10" 23

Self and Class Inheritance 4 “Self” param passed to class functions points back to that instance class Parent. Class: def sample. Func(self, value): self. variable = value def parent. Func(self): self. sample. Func(10) return self. variable # returns 10 4 You can also use “self” to reference functions in parent classes (inherence) class Child. Class(Parent. Class): def child. Func(self): print self. parent. Func() print self. variable # prints "10" # also prints "10" 24

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 25

Introduction to Grammars 4 Nat. Link grammars are based on SAPI 4 Grammars include: rules, lists and words – distinguished by how they are spelled – <rule>, {list}, word, "word with space" 4 Grammar specification is a set of rules 4 A rule is combination of references to words, lists and other rules <my. Rule> = one <sub. Rule> and {number} ; <sub. Rule> = hundred | thousand ; 26

Specifying Rules 4 Nat. Link compiles a set of rules when a grammar is loaded def initialize(self): self. load(self. gram. Spec) self. activate. All() # this compiles and load rules 4 Rules should be defined in a Python string gram. Spec = "<my. Rule> = one two three; " gram. Spec 2 = """ <rule. One> = go to sleep; <rule. Two> = wake up; """ 4 Define rules as rule-name, equal-sign, expression; end rule with a semicolon 27

Basic Rule Expressions 4 Words in a sequence must spoken in order – <rule> = one two three; – Must say “one two three” 4 Use brackets for options expressions – <rule> = one [ two ] three; – Can say “one two three” or “one three” 4 Vertical bar for alternatives, parens to group – <rule> = one ( two | three four ) five; – Can say “one two five” or “one three four five” 28

Nested Rules and Repeats 4 Rules can refer to other rules – <rule> = one <sub. Rule> four; – <sub. Rule> = two | three; – Can say “one two four” or “one three four” 4 Use plus sign for repeats, one or more times – <rule> = one ( two )+ three – Can say “one two three”, “one two two three”, etc. 29

Exported and Imported Rules 4 You can only activate “exported” rules – <my. Rule> exported = one two three; 4 Exported rules can also be used by other grammars; define external rule as imported – <my. Rule> imported; – <rule> = number <my. Rule>; 4 Nat. Speak defines three importable rules: – <dgnwords> = set of all dictation words – <dgndictation> = repeated dictation words – <dgnletters> = repeated spelling letters 30

Dealing with (Grammar) Lists 4 Lists are sets of words defined later 4 Referencing a list causes it to be created – <rule> = number {my. List}; 4 Fill list with words using set. List function def initialize(self): self. load(self. gram. Spec) self. set. List('my. List', ['one', 'two', 'three']) self. activate. All() # fill the list – You can now say “number one”, “number two” or “number three” 31

What is a Word? 4 Words in Nat. Speak and Nat. Link are strings – Words can have embedded spaces – “hello”, “New York”, “: -)” 4 In Nat. Link grammars, use quotes around words if the word is not just text or numbers 4 Grammar lists are lists of words 4 For recognition, words from lists are returned just like words in rules 32

Special Word Spellings 4 Words with separate spoken form are spelled with backslash: “writtenspoken” 4 Punctuation is most common example – “. period” – “{open brace” 4 Letters are spelled with two backslashes – “a\l”, “b\l”, “c\l”, etc. V 1. 1 33

Grammar Syntax 4 Nat. Speak requires rules in binary format – Binary format is defined by SAPI and is documented in SAPI documentation 4 Gramparser. py converts text to binary 4 Rule syntax is described in gramparser. py 4 Nat. Speak also supports dictation grammars and “Select XYZ” grammars. These are covered in another talk. V 1. 1 34

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 35

Getting Results 4 When a rule is recognized, Nat. Link calls your function named “got. Results_xxx” – where “xxx” is the name of the rule 4 You get passed the sequential words recognized in that rule – got. Results(self, words, full. Results) 4 Function called for innermost rule only – consider the following example 36

Extract from _sample 3. py #. . . class This. Grammar(Grammar. Base): gram. Spec = """ <main. Rule> exported = <rule. One>; <rule. One> = demo <rule. Two> now please; <rule. Two> = sample three; """ def got. Results_main. Rule(self, words, full. Results): natlink. play. String('Saw <main. Rule> = %s{enter}' % repr(words)) def got. Results_rule. One(self, words, full. Results): natlink. play. String('Saw <rule. One> = %s{enter}' % repr(words)) def got. Results_rule. Two(self, words, full. Results): natlink. play. String('Saw <rule. Two> = %s{enter}' % repr(words)) def initialize(self): #. . . “repr(x)” formats “x” into a printable string. 37

Running Demo Sample 3 4 When you say “demo sample 3 now please”, resulting text sent to application is: Saw <rule. One> = ['demo'] Saw <rule. Two> = ['sample', 'three'] Saw <rule. One> = ['now', 'please'] 4 Rule “main. Rule” has no words so got. Results_main. Rule is never called 4 got. Results_rule. One is called twice, before and after got. Results_rule. Two is called 4 Each function only sees relevant words 38

Other got. Results Callbacks 4 If defined, “got. Results. Init” is called first 4 If defined, “got. Results” is called last – Both get passed all the words recognized 4 Called functions from previous example: got. Results. Init( ['demo', 'sample', 'three', 'now', 'please'] ) got. Results_rule. One( ['demo'] ) got. Results_rule. Two( ['sample', 'three'] ) got. Results_rule. One( ['now', 'please'] ) got. Results( ['demo', 'sample', 'three', 'now', 'please'] ) 39

Common Functions 4 natlink. play. String(keys) sends keystrokes – works just like “Send. Keys” in Nat. Speak Pro. – include special keystrokes in braces: “{enter}” 4 natlink. set. Mic. State(state) controls mic – where state is 'on', 'off' or 'sleeping' – natlink. get. Mic. State() returns current state 4 natlink. exec. Script(command) runs any built -in Nat. Speak scripting command – natlink. exec. Script('Send. Keys "{enter}"') 40

More Common Functions 4 natlink. recognition. Mimic(words) behaves as if passed words were “heard” natlink. recognition. Mimic(['Select', 'hello', 'there']) – works just like “Heard. Word” in Nat. Speak Pro. 4 natlink. play. Events(list) to control mouse – pass in a list of windows input events – natlinkutils. py has constants and button. Click() 4 natlink. get. Clipboard() returns clipboard text 41 – use this to get text from application

Mouse Movement _sample 4. py #. . . class This. Grammar(Grammar. Base): gram. Spec = """ <start> exported = demo sample four; Press control key """ def got. Results_start(self, words, full. Results): # execute a control-left drag down 30 pixels Press left button x, y = natlink. get. Cursor. Pos() natlink. play. Events( [ (wm_keydown, vk_control, 1), (wm_lbuttondown, x, y), Move mouse (wm_mousemove, x, y+30), (wm_lbuttonup, x, y+30), Get current (wm_keyup, vk_control, 1) ] ) mouse position def initialize(self): self. load(self. gram. Spec) self. activate. All() Release left button (at new position) #. . . Release control key 42

Clipboard Example _sample 5. py #. . . class This. Grammar(Grammar. Base): gram. Spec = """ <start> exported = demo sample five [ (1 | 2 | 3 | 4) words ]; """ If more than 3 words recognized, 4 th word will be word count. def got. Results_start(self, words, full. Results): # figure out how many words if len(words) > 3: count = int(words[3]) This selects previous else: “count” words count = 1 # select that many words natlink. play. String('{ctrl+right}{left}') natlink. play. String('{ctrl+shift+left %d}'%count) natlink. play. String('{ctrl+c}') Copy selected text to text = natlink. get. Clipboard() clipboard, then fetch it # reverse the text new. Text = reverse(text) natlink. play. String(new. Text) Reverse function 43 #. . . defined later in file

Debugging and using Print 4 If file is changed on disk, it is automatically reloads at start of utterance 4 Turning on mic also looks for new files 4 Python output is shown in popup window – Window automatically appears when necessary 4 Python errors cause tracebacks in window – Correct file, toggle microphone to reload 4 Use “print” statement to display debug info 44

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 45

Global vs App-Specific 4 Files whose name begins with underscore always loaded; ex: _mouse. py 4 Files whose name begins with a module name only load when that module is active – Ex: wordpad. py, excel_sample. py 4 Once a file is loaded it is always active 4 To restrict grammars: – test for active application at start of utterance – or, activate grammar for one specific window 46

Activating Rules 4 Any exported rule can be activated 4 Grammar. Base has functions to activate and deactivate rules or sets of rules – self. activate(rule) - makes name rule active – self. activate. All() - activates all exported rules 4 By default, activated rule is global – self. activate(rule, window=N) - activates a rule only when window N is active 4 You can (de)activate rules at any time 47

Start of Utterance Callback 4 If defined, “got. Begin” function is called at the start of every recognition – it gets passed the module information: module filename, window caption, window id 4 The “window id” can be passed to activate() 4 Use match. Window() to test window title if match. Window(module. Info, ’wordpad’, ’font’): self. activate(‘font. Rule’, no. Error=1) Prevents errors else: if rule is already self. deactivate(‘font. Rule’, no. Error=1) 48 (not) active.

Using Exclusive Grammars 4 If any grammar is “exclusive” then only exclusive grammars will be active 4 Allows you to restrict recognition – But you can not turn off dictation without also turning off all built-in command control 4 Use self. set. Exclusive(state), state is 0 or 1 – Can also call self. activate(rule, exclusive=1) 4 Any number of rules from any number of grammars can all be exclusive together 49

Activation Example _sample 6. py class This. Grammar(Grammar. Base): No activate. All() in initialize function ! gram. Spec = """ <main. Rule> exported = demo sample six [ main ]; <font. Rule> exported = demo sample six font; """ def initialize(self): self. load(self. gram. Spec) Link <main. Rule> to main window (has “Dragon” in title). def got. Begin(self, module. Info): window. Id = match. Window(module. Info, 'natspeak', 'Dragon') Turn on <font. Rule> if window. Id: exclusively when self. activate('main. Rule', window=window. Id, no. Error=1) window. Id = match. Window(module. Info, 'natspeak', 'Font') window title if window. Id: contains “Font” self. activate('font. Rule', exclusive=1, no. Error=1) else: Otherwise, turn off self. deactivate('font. Rule', no. Error=1) self. set. Exclusive(0) <font. Rule> and exclusiveness. 50

Activating Rules from a Table 4 This is from my own Lotus Notes macros: Activate nothing by default def got. Begin(self, module. Info): self. deactivate. All() captions = [ This table maps ( 'New Memo -', 'new. Memo' ), caption substring to ( 'New Reply -', 'new. Reply' ), rule-name to activate ( 'Inbox -', 'inbox' ), ( '- Lotus Notes', 'read. Memo' ), ] for caption, rule_name in captions: win. Handle = match. Window(module. Info, 'nlnotes', caption) if win. Handle: self. activate(rule_name, window=win. Handle) return V 1. 1 Loop over table to find first window caption which matches 51

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 52

Using OLE Automation 4 You can use OLE Automation from Python with the Python Win 32 extensions 4 Using excel_sample 7. py: – say “demo sample seven” 4 Any cells which contain the name of colors will change to match that color 53

Extract from excel_sample 7. py class This. Grammar(Grammar. Base): gram. Spec = """ <start> exported = demo sample seven; """ def initialize(self): self. load(self. gram. Spec) Activate grammar when we know window handle OLE Automation code just like using Visual Basic def got. Begin(self, module. Info): win. Handle=match. Window(module. Info, 'excel', 'Microsoft Excel') if win. Handle: self. activate. All(window=win. Handle) def got. Results_start(self, words, full. Results): application=win 32 com. client. Dispatch('Excel. Application') worksheet=application. Workbooks(1). Worksheets(1) for row in range(1, 50): “color. Map” maps for col in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ': name of color to value cell=worksheet. Range(col+str(row)) (defined earlier) if color. Map. has_key(cell. Value): cell. Font. Color=color. Map[cell. Value] cell. Borders. Weight = consts. xl. Thick #. . . 54

Mouse Control in Python 4 _mouse. py included in Nat. Link download 4 Control mouse and caret like in DDWin: – "mouse down … slower … left … button click" – "move down … faster … stop" 4 Uses exclusive mode to limit commands 4 Uses timer callback to move the mouse 55

Implementing “Repeat That” 1 #. . . last. Result = None This grammar is never recognized because list is empty class Catch. All. Grammar(Grammar. Base): gram. Spec = """ <start> exported = {empty. List}; """ But, all. Results flag means that got. Results. Object is called for every recognition def initialize(self): self. load(self. gram. Spec, all. Results=1) self. activate. All() def got. Results. Object(self, recog. Type, res. Obj): global last. Result if recog. Type == 'reject': last. Result = None else: last. Result = res. Obj. get. Words(0) #. . . V 1. 1 After every recognition, we remember what words were just recognized 56

Implementing “Repeat That” 2 class Repeat. Grammar(Grammar. Base): Notice that the count is optional gram. Spec = """ <start> exported = repeat that [ ( 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 20 | 30 | 40 | 50 | 100 ) times ]; """ def initialize(self): self. load(self. gram. Spec) self. activate. All() The 3 rd word in the result is the count def got. Results_start(self, words, full. Results): global last. Result if len(words) > 2: count = int(words[2]) else: count = 1 if last. Result: for i in range(count): natlink. recognition. Mimic(last. Result) #. . . V 1. 1 Use recognition. Mimic to simulate the recognition of the same words; Nat. Speak will test against active grammars or dictation as it the words were spoken. 57

Grammars with Dictation class This. Grammar(Grammar. Base): <dgndictation> is built-in rule for dictation. Optional word ”stop” is never recognized. gram. Spec = """ <dgndictation> imported; <rule. One> exported = demo sample eight <dgndictation> [ stop ]; <dgnletters> imported; <rule. Two> exported = demo sample eight spell <dgnletters> [ stop ]; """ def got. Results_dgndictation(self, words, full. Results): words. reverse() natlink. play. String(' ' + string. join(words)) def got. Results_dgnletters(self, words, full. Results): words = map(lambda x: x[: 1], words) natlink. play. String(' ' + string. join(words, '')) def initialize(self): self. load(self. gram. Spec) self. activate. All() #. . . V 1. 1 <dgnletters> is built-in rule for spelling. I had to add word “spell” or the spelling was confused with dictation in <rule. One> 58

Outline of Today’s Talk 4 Introduction 4 Getting started with Nat. Link 4 Basics of Python programming 4 Specifying Grammars 4 Handling Recognition Results 4 Controlling Active Grammars 4 Examples of advanced projects 4 Where to go for more help 59

Nat. Link Documentation 4 Nat. Link. SourceNat. Link. txt contains the documentation for calling the natlink module from Python 4 Example macro files are all heavily documented; in Nat. LinkSample. Macros 4 Grammar syntax defined in gramparser. py 4 Grammar. Base defined in natlinkutils. py – also defines utility functions and constants 60

Where to Get More Help 4 Joel’s Nat. Speak web site: http: //www. synapseadaptive. com/joel/default. htm 4 Python language web site: http: //www. python. org/ 4 Books on Python – See Joel’s Nat. Speak site for recommendations 4 Nat. Python mailing list: http: //harvee. billerica. ma. us/mailman/listinfo/natpython 4 Using COM from Python: Python Programming on Win 32 by Mark Hammond 61

Looking at the Source Code 4 Nat. Link source code included in download 4 Source code is well documented 4 Written in Microsoft Visual C++ 6. 0 4 Some features from Microsoft SAPI – get SAPI documentation from Microsoft 4 Dragon-specific extensions not documented 62

All Done “Microphone Off” 63