Project CodeTalk: How Microsoft India empowers visually impaired programmers

A team of researchers at Microsoft India has made the experience of coding much easier for visually impaired programmers. The researchers have added certain auditory features to Integrated Development Environment tools (IDEs) which help the visually impaired to write and test code using sound instead of visual cues. 

Integrated Development Environment (IDE) tools are made specifically to improve the efficiency of developers. While IDEs have proven to be extremely beneficial for sighted developers, they can still be challenging for visually impaired developers.

Much of the information from IDE systems such as Visual Studio and VS Code is given out visually and it becomes very challenging for the visually impaired developers relying on screen reader. On IDE systems, code is syntax highlighted in bright colors; errors are automatically highlighted using squiggles, and the debugger uses several windows to provide full context of a running program. These advanced features are generally not accessible to developers with visual impairments, because of which they cannot use IDEs to the fullest.

"The cues on an IDE are very subtle and mostly inaccessible to developers who are blind. The goal of CodeTalk is to bring all the functionalities of IDE and make them accessible to the visually impaired, so they are as productive as sighted programmers"

ct

Gopal Srinivasa, Senior RSDE, Project CodeTalk , Microsoft India

"IDE are the premiere tools for developers to build programs and they have a lot of functionalities which are exposed through visual cues. There are multiple windows that show different forms of  information and there are small changes that happen in these windows like for instance some color changes, like a red line is added to show errors or color changes when value of a variable is changed. "The cues on an IDE are very subtle, and mostly inaccessible to developers who are blind. The goal of CodeTalk is to bring all the functionalities of IDE and make them accessible to the visually impaired so they are as productive as sighted programmers," explains Gopal Srinivasa, Senior RSDE, Project CodeTalk at Microsoft India.

Challenges for the visually challenged

Visually impaired programmers use general purpose assistive software called screen readers which read the content on screen and this information does not effectively get transferred on audio channels. The features that IDEs provide in terms of making a developers job easier are not available to those using screen readers. Through a window of an IDE, sighted programmers can visually glance the structure of a code and with the help from colored indicators, various errors are highlighted like red squiggles and so on. For a visually impaired developer, to get a sense of the code, the only way is using  a screen reader going through code one line at a time, thus making it incredibly challenging for them when it comes to analyzing and debugging code

“The only way for a visually impaired developers to go about using an IDE is to go line by line and listen to the screen reader. Imagine going through thousands of lines of code like that.”

venk

Venkatesh Potluri, Research Fellow, Project CodeTalk, Microsoft

"The only way for a visually impaired developer to go about is to go line by line and imagine going through thousands of lines of code like that. So, there are glanceability issues, getting the complete glance of code, navigability issues which is quickly navigating through code, alertability issues where you have red squiggles showing up when there is an error and there are discoverability issues," says Venkatesh Potluri, Research Fellow, Project CodeTalk who is a visually impaired developer himself.

Using multiple audio channels 
Instead of using visual information, CodeTalk deploys audio through various channels, giving out different sounds for helping developers recognize the code and debugging errors. "We have multiple audio channels that we use. One is the regular screen reader so we filter out info from the code. For instance, you want to find the structure of a particular code, we extract and put it in a dialogue and the regular screen reader can show that. So, that's one audio channel.  We have another audio channel for things like alerts. So if there is an error on the screen, there is a red squiggle, and we issue a beep for that and that is a different audio channel," told Gopal Srinivasa.

Under Project CodeTalk, many of the GUI elements of Visual studio  are converted into a spatialized audio user interface (AUI).  This soundscape surrounding the user’s head utilize both speech and non-speech audio to complement the screen reader voice. It also includes  programmer introduced statements in the code that speak out information appropriately, known as Talk Points. Apart from that, there are techniques to present sketches and diagrams like UML diagrams as audio, which greatly help the visually challenged developers to improve the code.

Project CodeTalk is an open source project and can be downloaded from Github.