Design and Implementation of Online Braille Math Support

Design and Implementation of Online Braille Math Support in the Accessible Equation Editor Sam Dooley Pearson Assessment Dan Brown Pearson Assessment Susan Osterhaus Texas School for the Blind and Visually Impaired

Braille math is hard! 1 Ø Blind students need high-quality braille math Ø Advanced preparation is costly and takes time Ø Math teachers are often unfamiliar with braille Ø TVIs are often unfamiliar with math notation Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math software is harder! 2 Ø Software translation can be problematic Ø Forward translation takes many steps Ø Real-time back translation is non-existent Ø No support for online or classroom use Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math needs to be online! Blind students need: 3 Ø A level playing field for STEM instruction Ø To read and write online braille math Ø To interact with sighted instructors and peers Ø To participate in online activities Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math should be math! Ø Math concepts are independent of notation Ø Braille math codes capture all math notation Ø Math software can be independent of notation Blind students only have full access to math if their math is treated the same as printed math. 4 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Classroom challenges for blind math students 5 Ø They cannot read math the teacher can write Ø They cannot write math the teacher can read Ø Speech solutions incur high cognitive load Ø They need to read and write for themselves Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math translation Ø Visual math software is not truly accessible MS Word, Math. Type, Scientific Notebook Ø Braille math translation software Ø is not instantaneous Ø is not totally accurate La. Te. X, Duxbury, HIMS, Human. Ware 6 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Online Braille Math with the Accessible Equation Editor 7 Ø Sighted user can create math for a blind user Ø Blind user can create math for a sighted user Ø Real-time, two-way braille math translation Ø Instantaneous interactions with math content Ø Discoverable braille math encoding rules Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Nemeth Braille (1952) 8 Ø Abraham Nemeth (AFB, APH, BANA) Ø Tactile encoding for print math Ø Math for technical publications Ø Presentational math structures Ø Consistent with content markup Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Te. X/La. Te. X (1978/1985) 9 Ø Donald Knuth, Leslie Lamport Ø Typesetting language for print math Ø Math for technical publications Ø Presentational math structures Ø Fully programmable macro language Design and Implementation of the Accessible Equation Editor 10 Apr 2016

W 3 C Math. ML (1998) Ø World Wide Web Consortium (W 3 C) Ø XML element/attribute vocabulary Ø Typical K-12 and higher ed. math Ø Presentation and content markup 10 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Presentation v. Content Ø Presentation encodes signs/symbols Ø Content encodes functional structure Ø "x^2+1" v. "1346, 45, 23, 5, 346, 2" Ø "x^2+1" v. (plus (power x 2) 1) 11 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

QWERTY keyboard input Ø Math symbols on the keyboard Ø Other symbols on the buttons Ø Keyboard cursor navigation Ø Implicit and explicit selection Ø Backspace, delete, clear 12 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille terminal output Ø Math output on the screen Ø Braille output on the terminal Ø Screen reader support 13 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille terminal input Ø Each braille key is treated like a keyboard key Ø Sequences of braille keys create the math Ø Button tooltips with text and braille Ø Input cursor position and routing 14 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

QWERTY terminal output Ø Braille input creates math content Ø Math content becomes math output Ø Math output becomes braille output Ø The math markup is exactly the same 15 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math is math! Ø Blind students can read the same math Ø Blind students can create the same math Ø The math can be shared the same way Ø The math can be scored the same way Blind students can now have full access to math since their math is the same as printed math. 16 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Equation Editor Architecture Ø Syntax expression trees Ø Expression tree transformations Ø Configuration and customization Ø Universal design decision points 17 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Expression trees Ø An operator with its child arguments Ø The operator as the initial child argument Ø Parse trees, syntax expressions, XML elements Ø Often annotated with semantic attributes Ø Functional programming languages (Lisp) 18 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Content expressions Ø The functional structure of a math expression Ø Does not itself define any semantic behavior Ø The operators carry the semantic behaviors Ø Operator, child elements, child attributes Ø Direct translations to and from XML markup Ø Boxed reference to presentation expression 19 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Presentation expressions Ø The visual notation for a math expression Ø Does not itself define functional structure Ø The operators carry the notational schema Ø Operator, child elements, child attributes Ø Direct translations to and from XML markup Ø Boxed reference to content expression 20 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Expression references Ø Math document user interface elements Ø Content node matched with a presentation Ø Pointers to child expression references Ø References to the browser display nodes Ø Resolves from DOM to presentation to content 21 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Browser display nodes Ø Browser document user interface elements Ø Direct references into the browser DOM tree Ø Attached pointers into content, presentation Ø Listeners for math content change events Ø Wrap nodes not owned by the equation editor 22 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Expression trees are universal! Ø Generic encodings for math formulas Ø Separating notation from function Ø Separating interface from function Ø Enabling automated regression testing Ø Universal design for math behaviors 23 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Transformation architecture Ø Input transformations create expressions Ø Output transformations create XML markup Ø Layouts transform content to presentation Ø Templates create new content from old Ø Generic rewrite rules on expression trees Ø Enabling data-driven math behaviors 24 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Configuration architecture Ø External data defines conventional behaviors Ø Allows highly customizable math interactions Ø Fine control for solving math usability issues Ø Allows reconfiguration for new populations Ø Layouts, templates, buttons, palettes, keys 25 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Layout transformations Ø Transform structure into notation Ø Structure encoded as Content Math. ML Ø Notation encoded as. . . pick your target! Ø Layout rules encoded as custom XML Ø Source pattern as Content Math. ML Ø Output pattern as layout schemata 26 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Content to Presentation Ø Math operators determine layout rules Ø Minimal contextual dependencies Ø Presentation links back to content Ø (plus x 1) v. "x + 1" <layout name="plus"> <apply> <plus/> <i: args/> </apply> <i: infix> <i: op/> <i: args/> </i: infix> </layout> 27 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Content to Braille Ø Math operators determine braille symbols Ø Additional contextual dependencies Ø Flattening to linear output format Ø (plus x 1) v. "1346, 2" <layout name="plus"> <plus/> <mi>&#x 282 C; </mi> </layout> 28 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Template transformations Ø Create new structure from old structure Ø Template rules encoded as custom XML Ø Template names and key event names Ø Source pattern as Content Math. ML Ø Output pattern as Content Math. ML Ø Template variables control rewriting 29 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Keyboard to Content Ø Keys and button names select template rules Ø Minimal sequential dependencies Ø Presentation updated on each key Ø "x + 1" v. (plus x 1) <template name="plus" selector="%math %outfix"> <i: this/> <apply> <plus/> <i: this/> <i: box/> </apply> </template> 30 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Braille to Content Ø Braille cells indirectly select template rules Ø Non-trivial sequential dependencies Ø Presentation updated on each key Ø "1346, 2" v. (plus x 1) <template name="plus" selector="%math %outfix"> <i: this/> <apply> <plus/> <i: this/> <i: box/> </apply> </template> 31 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Braille key transformations Ø Transform braille cells to template names Ø Braille cells are received as ASCII braille Ø Each braille cell assigned a symbolic name Ø Symbolic names for braille cell sequences Ø Braille input mode transforms to key names Ø Previous template rules operate unchanged 32 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille cells to keys Ø Single cells that appear as keys <key prefix="" key="x" name="x"/> Ø Single cells that cause actions <key prefix="" key="> " name="osqrt"/> Ø Sequences that cause actions <key prefix="" key=". " name="_greek"/> <key prefix="_greek" key="p" name="constantpi"/> Ø Sequences that extend others <key prefix="" key='" k' name="lt"/> <key prefix="lt" key=": " name="leq"/> 33 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille finite state machine Ø Symbolic names for braille prefixes Ø Symbolic names identify input states Ø Braille cells identify state transitions Ø Accepting input states invoke templates Ø Subsequent states may replace inputs 34 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Universal design for online math Ø Expression trees provide universal data Ø Data-driven universal tree transformations Ø Flexible configuration customization rules Ø Universal presentation output notation Ø Universal input event transformations 35 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille Input Examples Ø Numeric indicator Ø Baseline indicator Ø Type form indicators Ø Shape indicators Ø Negated operators Ø Composed relations 36 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Braille Input Testing Ø Web browser input test page Ø Java. Script equation editor Ø Screen reader device drivers Ø Web accessibility APIs Ø Braille terminal input 37 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Research Studies Ø Two research studies in Fall 2015 (KY/AZ - Sep, TX - Oct) Ø Goal - to collect feedback from multiple populations on EE functionality Ø Populations - blind, low vision, regular print readers, learning disabled Ø Criteria - high school students who had completed Algebra I 38 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Research Outcomes Ø Students had limited knowledge of Nemeth Ø Erasing math content was unpredictable Ø Working with grouping symbols was difficult Ø Entering and closing fractions was unexpected Ø Ending trigonometric expressions was unusual 39 Content Math. ML Markup from Nemeth Braille Input 23 Mar 2016

Advanced preparation Ø Create math content in advance Ø Share with students by email Ø Include math content with slides Ø Students use the equation editor 40 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Extemporaneous content Ø Teacher uses the equation editor Ø Projects the screen to the class Ø Blind student uses braille terminal Ø Everyone can read the same content 41 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Shared presentations Ø Web. Ex, Google Hangouts, etc. Ø Shared screens for visual users Ø Individual work shown to the class Ø Blind students can show their work Ø Blind students viewing others work 42 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Online assessments Ø Common Core, PARCC, Test. Nav Ø Spring 2015 admin (15 m responses) Ø Machine scoring for constructed math Ø Blind students can use online forms Ø Blind students create online responses 43 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Nemeth Braille curriculum Ø Interactive Nemeth braille exercises Ø Immediate practice for lessons learned Ø Experimentation with braille concepts Ø Discoverability for unfamiliar encodings 44 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Homework assignments Ø Blind student uses the equation editor Ø Email homework files to sighted teacher Ø Teacher reads using the equation editor Ø Teacher marks up the math content Ø Returns marks to the blind student 45 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

TVI support Ø The role of the TVI remains essential Ø The technical work load can be reduced Ø The software can help assist the process Ø Discoverability for print math notations 46 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Math teachers Ø The role of the math teacher is vital Ø The technical work load can be reduced Ø Engaging directly with blind students Ø Discoverability for braille math encodings 47 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Group interactions Ø Effective group work with blind students Ø Each student has an opportunity to contribute Ø Each can learn from the work of others Ø The software can help assist the process 48 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Online Accessible Math! Blind students can have: Ø A level playing field for STEM instruction Ø Tools to read and write online braille math Ø Interaction with sighted instructors and peers Ø Active participation in online activities 49 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Expression trees are universal! Ø Generic encodings for math formulas Ø Separating notation from function Ø Separating interface from function Ø Enabling automated regression testing Ø Universal design for math behaviors 50 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math is math! Ø Blind students can read the same math Ø Blind students can create the same math Ø The math can be shared the same way Ø The math can be scored the same way Blind students can now have full access to math since their math is the same as printed math. 51 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Accessible Equation Editor Ø Try it out, give us feedback http: //accessibility. pearson. com/mathex-app/ Ø Please let us know how you would use it sam. dooley@pearson. com dan. brown@pearson. com osterhauss@tsbvi. edu 52 Design and Implementation of the Accessible Equation Editor 10 Apr 2016