Programming By Voice Josh Pepperman University of Alabama

Programming By Voice Josh Pepperman University of Alabama, Department of Computer Science Advisor: Dr. Jeff Gray Introduction Over the past decade, there has been an increased interest in providing new environments for teaching children about computer programming. This has resulted in several environments and languages that offer a visual language that is much easier to learn than a traditional programming language. However, the visual interface of such environments requires use of a mouse and keyboard (with drag and drop usage), which limits the potential for adoption by children with a physical disability. The purpose of this Emerging Scholars project is to develop a voice interface to use along with visual programming languages so that individuals who cannot use a traditional keyboard and mouse can still learn to program. The tool support for this effort uses voice recognition software to understand what words are spoken, and then parses those words for understanding of what actions to take to mimic the equivalent mouse and keyboard events. The goal is to support the concept of “Programming by Voice” (PBV). The Myna project considers the topic of PBV for multiple environments. For this project, I investigated the application of PBV on a tool called Snap!, which is a web-based programming environment for children that was developed by the University of California at Berkeley. The main challenge is making the voice interface as versatile as the graphical interface. Simple tasks like scrolling and moving objects through voice commands are challenging because the voice interface has to know exactly where everything is with the programming tool, even when objects are moved or the screen is changed. We are investigating ways to implement solutions to these problems, and discovering that a voice interface, while offering its own challenges, is capable of providing the assistive needs to help children with special needs learn to program. Methods Next, the block can be dragged and deleted off to the left side: My primary goals for this project were to initiate the PBV concepts to handle the Snap! interface online, and to implement the delete function. In order to get the PBV concepts working with Snap!, I had to take screenshots (like the ones below) of each of the menus and record their pixel positions on the screen. Then I had to transfer all that data into the program’s code, and also check to make sure that all of it worked. As an example, the delete function works as if the user is using a mouse: drag the block to be deleted off into the menu on the left. If there is something under the to-be-deleted block, then move it down and proceed. Then, move the other block back up. Here is what that looks like in the Snap! environment: Here are three typical blocks. If we want to delete the block in the middle, the block below it must be moved beneath it, programmatically within Myna: Finally, the previous block can be dragged to reattach to the new sequence: Se This process seems simple at first, but when considering all the complications of not being able to use a mouse, it becomes a challenging problem. For example, if the screen is scrolled down, all the block positions are saved after they are moved. Thus, if the screen itself is moved, the internal positions representing each block must also be modified. Many complications arose when dealing with screen changes and trying to represent where everything is located in the internal representation. Other issues also arose in this investigation, such as: what if screen sizes are not consistent across different computers? How can we get the program to undo and redo numerous actions? How can we allow scrolling without using the mouse? Some blocks (like the ones pictured) have places to input text, so how can the user add text verbally, and how to handle the size of the block expanding? Conclusions Despite these issues with how to get certain functions of the program working, the rest of the team is making good progress on it. From what we have done so far, the project is certainly an achievable feat to accomplish, and once it is done it will be very helpful when teaching those with physical disabilities how to program. Some of the issues I listed can be fixed by forcing the program to run on a fixed resolution, so if we ask the user to set their screen at a certain resolution before running it, we could avoid certain issues related to screen size and positioning. Finding ways to make the program dynamic to its environment can be considered by scaling based on screen size. Also, changing the amount that the screen scrolls by a set rate, and then adding or subtracting that from the positions stored for the blocks as the screen moves are possible ideas for making the environment adapt to screen changes by the user. The ultimate goal of our efforts to bring the PBV concept to children is to support many different visual programming languages so that teachers are not restricted to just one when they are teaching children to program. A future idea for the project is to support adaptation to different environments instead of having to put in all the locations in for each environment element (as was done in this effort with Snap!). The motivation for this project is to provide a capability for children with disabilities to learn how to program a computer. Much of the experimentation was done by myself, but future efforts are planned to perform user-based testing with children identified by our collaborators at United Cerebral Palsy of Birmingham. This will let us see what works well with our project and what does not, and give us a better understanding of exactly how our Myna project will be used in a teaching environment. Seeing how teachers utilize it and how students use and react to it will help us as we continue to improve this project.