Images Alternative Text and Artificial Intelligence SSB BART

Images, Alternative Text, and Artificial Intelligence SSB BART Group Silicon Valley (415) 975 -8000 sales@ssbbartgroup. com IT Accessibility Problem - Solved™ SSB BART Group Washington DC (703) 442 -5023 sales@ssbbartgroup. com

Agenda 4 About Us 4 About Me 4 The Project 4 What’s Next http: //amp. ssbbartgroup. com/public/research/Automatic_Image_Classification_090707. doc http: //amp. ssbbartgroup. com/public/research/SSB_BART_Group_Image_Alt_CSUN_2008. ppt Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Corporate Overview History 4 Approach Founded in 1997 by engineers with 4 4 Violation profiling across 5. 5 M human validated accessibility issues disabilities 4 750 commercial and government customers 4 4 Data driven and scalable 4 Scalable 1, 500 enterprise projects successfully 4 One completed 4 One to one million developers to one thousand production systems Pioneers of commercial accessibility validation tools Solutions 4 Fifty percent staffing mix of individuals with disabilities 4 Appropriately mixed automated, human and code level validation Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Supported Platforms Web Compiled Software 4 HTML 4 JFC and SWT Java Applications 4 XML 4 . Net Applications 4 Java. Script 4 MFC Windows Native Applications 4 CSS 4 Macintosh Applications 4 AJAX 4 BMC Remedy Applications 4 Adobe Flash and Flex Standalone Systems 4 Adobe Acrobat Documents 4 Telecommunications Hardware 4 Streaming Audio and Video 4 IVR Systems 4 Agent Systems 4 Digital Imaging Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Industry Solutions 4 Public Sector 4 Federal 4 Solutions 4 United Manufacturers States 4 European 4 Software Union 4 Hardware 4 Education 4 Web 4 K-12 and Local Mass Transit 4 Financial Services 4 Consumer 4 Government System Integrators 4 Healthcare 4 Primary Silicon Valley (415) 975 -8000 Banking 4 Insurance 4 Legal 4 Web Based Service Providers Care Providers 4 Insurance Based Service Providers 4 4 Universities 4 State Information Technology www. ssbbartgroup. com Washington DC (703) 637 -8955

Accessibility Management Platform Requirements Implementation Certification Baseline Audit Development Audit Maintenance Audit Standards Development Standards Maintenance VPAT Creation e. Learning Developer Support Certification In. Focus™ Suite AMP – SSB’s web based platform for managing all aspects of Accessibility process Benefits 4 Single point for tracking compliance over time 4 Scalable solutions from one to one million developers across multiple domestic markets 4 Support for all aspects of a successful accessibility initiative Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

About Me General Story 4 Accessibility Work Founder and Managing Director of SSB 4 BART Group 4 4 4 validation and education since 1999 Also Known As President and CEO 4 Architected and developed first Professional web site developer for 13 commercial accessibility testing and fixing years tool 4 In. Sight and In. Focus 1. x -> 4. x BS Computer Science Leland Stanford 4 Initial release in mid-200 Junior University (AKA Stanford) 4 Next release in a few months 4 4 Involved in Web Accessibility activities, Started in 1994 at the dawn of the Web Odds on Brad Pitt to 4 play me in the movie Architected and developed Accessibility Management Platform (AMP) 4 4 Current Version – 2008 R 1 Personal work with fifty enterprise class software vendors Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Project Overview Project Description 4 Create a decision tree to classify images into one of eight types 4 Image types are organized by alternative text requirements 4 Upon classification, alternative text validity can then be tested via straightforward heuristics Project Utility 4 Alternative text provides a textual description of an image 4 Alternative text validity 4 4 Ensures access to content for people with disabilities 4 Allows pages to be adapted effectively - low resolution, alternative browsers 4 Increases search engine relevance for pages Bottom Line – Good alternative text is good for society and good for profits 8 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Automated Testing Tools 4 A brief note on automated testing tools 4 First generation of automated testing tools, where we are now, can test about 25% of requirements accurately 4 Another 25% with so-so accuracy 4 And the rest need to be checked manually 4 We think the next generation of tools can double this efficacy through better AI, more complex page models and better leveraging of human judgment… 4 …but ultimately tools can only facilitate the process of human review they cannot replace it Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Image Types 4 Layout Element – The image is used solely to layout elements on the page 4 Decorative Picture – The image is a picture that is used solely for the purpose of making the page more visually appealing and it provides no information 4 Text – The image is used to stylize text on the page but is not used as an active element on the page 4 Picture – The image is a picture that contains information important to the use of the page 4 Hidden Link – The image provides a “hidden” link on a page for search engine optimization or screen reader users 4 Linked Text – The images is used to stylize text and provide a link to another page 4 Skip Link - The image is the root of an inner-document link that provides a means of skipping past page content that is not relevant 4 Linked Picture – The image is a picture that provides a link to another page 10 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Variables Width Height Edge Count The width of the image The height of the image The number of vertical and horizontal edges in the image Size The rectangular size of the image or width time height File Size Link The size of the file in bytes Whether or not the image is a link Inner-document Link Whether or not the image is a link within the current document Color Depth The number of unique colors that the image has 11 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Project Functionality Challenge 4 No database of relevant image classifications exists 4 Subject Matter Experts (SMEs) use experience to determine form of alternative text 4 Without a good data set the decision tree isn’t going to decide much Solution 4 Build a spider to crawl sites and gather sample data 4 Classify the images using a basic interface 4 Store the image classification and additional variables in a database 4 Build a decision tree from the database rather than a live site 4 Repeat using updated tree Result 4 Created an image database of 1000 images with about an hour of actual data entry 12 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Project Functionality Challenge 4 Build the decision tree 4 …which became build the decision tree before the end of time 4 …which became build the decision tree once and store it for later use Discussion 4 Building the tree is fairly straightforward and involves splitting on variables and analyzing remaining sets 4 Implementation uses Russell, Norvig algorithm 4 4 More on the tricky parts later The “catch” - a lot of the queries involve eliminating groups of images 4 SQL doesn’t have good concepts for handling unordered sets of keys so you enumerate out elements for queries… 13 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Project Functionality Discussion (Continued) 4 This results in lots of nasty queries and a fair amount of time to build the tree 4 This more or less grows exponentially as you add variables and quanta Solution 4 Build the tree once and persist to disk 4 Limit quanta for variables and require minimum information gain Result 4 Creation of the tree takes about forty minutes 4 Reading in the tree takes about forty milliseconds 4 Resolving against the tree takes about forty nanoseconds 14 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Project Functionality Challenge 4 Test the decision tree for accuracy 4 Avoid peeking at the data set Solution 4 Always test on new data [Tank!] 4 Don’t store the test set so we avoid any temptation to peek Name Hi 5 – www. hi 5. com Hillary Clinton for President - http: //www. hillaryclinton. com/ Department of Defense - http: //www. defenselink. mil/ Engadget – www. engadget. com Gamespot. com – www. gamespot. com Average Silicon Valley (415) 975 -8000 Accuracy 94. 7% 98. 6% 86. 84% 91. 45% 91. 57% 15 www. ssbbartgroup. com 92. 63% Washington DC (703) 637 -8955

The Tricky Parts Information Gain 4 Overfitting Successful classification provides 2. 391 bits of 4 Observe information 4 460, 800 4 Which means, what, exactly? 4 Technically – You have enough information 4 Sample Data Size – 1000 to answer 2. 391 yes/no questions 4 460, 800 >> 1000 4 Practically – You can order nodes to split on by information gain 4 Permutations of Variable Quanta - At each split choose node that provides highest Note - The amount of information provided by an attribute will change as you move Thus the risk of over fitting is significant Solution 4 information gain 4 4 Require that we gain at least. 05 bits to split – otherwise just return the modal value for the remaining set through the tree Solution 4 Calculate information gain for each split 4 This is where the nasty set queries occur Silicon Valley (415) 975 -8000 www. ssbbartgroup. com 16 Washington DC (703) 637 -8955

The Tricky Parts Variable Quantification 4 4 Edge Detection Strategy 4 4 Make everything an integer 4 Define ranges for all variables Used Sobel Edge detection and Java convolution application for images 4 Initially picked quanta based on guesses Count the number of edges in the image divisions 4 4 These turned out to be wildly inaccurate Solution 4 Count vertical and horizontal edges Picked variables based on image type 4 Turns out to be a great proxy for text in the image grouping and average 4 Lots of images have edges SQL AVG and COUNT make this easy 4 Accuracy goes from 78. 23% to 92. 63% with this types of edge detection Silicon Valley (415) 975 -8000 www. ssbbartgroup. com 17 Washington DC (703) 637 -8955

Future Features Second Order Variables 4 First order variables are primary data from images 4 Second order variables are derived from one or more primary variables 4 Specifically 4 edge_count, 4 height color_depth have much more relevance as ratios to size is more relevant as a ratio for width Classification Tightening 4 Current classifications have some overlap which could be refined out 4 Certain classifications evolved over the course of the project and the data set should be updated to reflect the final classification 18 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

Future Features Safe Failure 4 Okay to require alternative text when not necessary than not require text when necessary… 4 …or is it? ? Celebrity Endorsement 4 If K-Fed uses it wouldn’t you 19 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

In. Focus 5. 0 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955

For More Information Silicon Valley Washington DC Phone (415) 975 -8000 Phone (703) 637 -8955 E-mail sales@ssbbartgroup. com Fax (415) 624 -2708 (703) 734 -8381 300 Brannan Street 1489 Chain Bridge Road Suite 608 Suite 204 San Francisco, CA 94107 -1876 Mc. Lean, VA 22101 Silicon Valley (415) 975 -8000 www. ssbbartgroup. com Washington DC (703) 637 -8955