Evaluation and Design Work I Have Done

As founding principal of a usability research firm, I have spent a large portion of my professional activities over the past ten years developing and adapting methodologies for usability evaluation. I participate in many of the initial planning meetings with client organizations needing usability research, and I work with these clients to identify the research methods appropriate to achieve their goals. Often that process involves modifying existing methods and evolving new ones (see also my UPA 2000 paper, "Not Just a Hammer: When and How to Apply Multiple Methods in Usability Programs").

Although most of Tec-Ed's usability research projects are not focused primarily on accessibility, we are increasingly able to include an accessibility component, most often by including one or two disabled participants in usability tests. Occasionally an entire study addresses a particular accessibility issue, for example, a study of a new laptop screen where we recruited only participants who wore eyeglasses. Tec-Ed’s staff members are familiar with Section 508 and are comfortable working with clients who want to test for accessibility.

Need for Ethnographic and Longitudinal Data in Accessibility Research

The challenge for the usability community, as described by the workshop organizers, is to develop techniques for designing and testing products for accessibility--in particular, low-cost test processes to measure users' performance with the user interface (UI). One important approach for developing these techniques and processes is to consult ethnographic and longitudinal data.

Just as in usability research with non-disabled users, in accessibility research we often concentrate on measuring user efficiency, effectiveness, and satisfaction under controlled circumstances such as usability testing. A shortcoming of this approach is that, when we design the tasks and their context, we can't know what circumstances of actual use we are not observing. For this reason, a successful usability research program should include a combination of techniques: usability testing, inspection methods as appropriate, and ethnographic studies to inform the other techniques about the real-world context of use.

For us to be successful in creating techniques that both measure Section 508 compliance and improve product design, we need to include ethnographic studies to inform the development of standard tasks and performance methods. Ethnographic data, especially longitudinal data, will help us produce better evaluation methods both for user testing and for user-free evaluations.

Example of Ethnographic Accessibility Observations

Although Tec-Ed has not yet been commissioned by clients to perform longitudinal studies of accessibility, we work regularly with a technical consultant (a network software specialist) who has retinopathy and consequently extremely low vision. For the past two years, we've observed this person's computer use for several hours about twice a month, and we learned about several accessibility issues we probably would not have seen during usability testing.

Our subject sets all software to display text in a large type size and a readable type face, which he then reads one word at a time with a hand-held magnifier. He also occasionally uses Jaws screen-reading software, but the issues described here involve attempts to read text on-screen (an accessibility concern of increasing interest to the aging U.S. population).

His major concern during software use is not the primary functionality of an application, which usually displays in the large type size as instructed (although it does not always allow him to change displays to a more distinct and readable type face). Rather, whenever any problems occur, the error messages display in a default type size and type face that the user cannot adjust, even with the accessibility settings. This behavior occurs both for error messages from most applications and those from the Windows operating system.

Thus at unpredictable times, an otherwise competent computer user must ask for help from a physically present person with better vision. (Problems also occur when Jaws attempts to read error messages; I can describe these during the workshop if they're of interest.) Can the Help system provide assistance? Rarely, because most applications also have their Help messages programmed to display a fixed text size.

Usability problems occur with software applications that would otherwise be excellent accessibility tools, if a low-vision person could gain access to the tool. Our subject attempted to use IBM's dictation product, because dictation would be much faster than finding a letter at a time on the keyboard. To "train" the software to recognize the user's voice, one must read sample text out loud--and the sample text is displayed in a type face our subject can't read. (The product apparently offers the ability to change type sizes and type faces, but only after the start-up activities are completed.)

Similarly, a year or two ago I saw a demonstration of a new Adobe product that would enable users to change the type size of a pdf file, and the text would "rewrap" to create a readable page in the new type size. Unfortunately, the menus and icons to specify the larger type were themselves too small for a low-vision person to read--and I was told these could not be changed.

Some quite basic software bugs raise questions about QA of accessibility options. Windows 98 had an accessibility feature that magnified the whole screen by a specified percentage. About two years ago, our subject tried setting this feature on his new laptop to 200%, at which point only a quarter of the screen was visible (the top-left quarter). When he wanted to turn the feature off, or try a different magnification, all the controls he needed were on undisplayed parts of the screen. But the screen displayed no scroll bars (perhaps because this was a "setting," not a central part of the operating system). After many unsuccessful calls to the Microsoft Help desk, our subject returned the laptop to his vendor, who also couldn't solve the problem. Their solution was to replace the hard disk of the laptop with a new one!

Windows also offers a "hand magnifier," which you can move around to magnify selected portions of a screen. When our subject clicks on this tool, it works for about 15 minutes and then fails (with another unreadable error message). Probably most users of the magnifier don't try to use it for hours at a time. Again, the Help desk hasn't been able to help. Our subject pleads strongly for major software companies having accessibility specialists in tech support, so that Help desk questions about accessibility options can easily be escalated to someone knowledgeable about these features.

Issues Resulting from Ethnographic and Longitudinal Observations

The example above involves just one subject and only one disability condition (although we expect problems with low-vision users to become increasingly prevalent), yet it provided many insights. Our observations led to ideas, implications, and issues that we can usefully address as we think about developing tasks and performance measures for accessibility:

· Even inspection methods can improve accessibility far more than they currently do, simply by extending accessibility standards to the entire product—including the installation or start-up sequences (or wizards), the error messages, and the help system. Perhaps I’m wrong in saying “simply”; the technical challenge is straightforward, but the organizational challenges in requiring many development teams to cooperate may be greater. Still, user-free methods can be developed to ensure that every part of a user interface can be displayed in large, distinct type.

· The process of developing “standard” tasks for accessibility usability testing should be an iterative one, rather than performed once and never revisited. An ongoing ethnographic, longitudinal research program should be carried out with representatives of the major groups of people with disabilities. The information gleaned from this longitudinal research should be used to update any standardized tasks, at least yearly (probably more often).

· Although an ideal user interface would not require tech support, we won’t achieve such perfection in product design in our lifetimes (considering the diversity of user audiences and the trend toward more complex products). Given the requirement to provide user support for products, we can improve accessibility dramatically by providing specialized user support for accessibility features.

Note that, as Gregg Vanderheiden from TRACE R&D Center said during his CHI 2001 talk, most of these approaches to improving accessibility also result in more usable products for their wider audiences, providing an additional justification for investing in the required resources. For example, our low-vision subject commented that he could use Jaws more effectively if its help index included more “ordinary-language” entries rather than only the terminology used in the product; and usability studies of indexes show that adding “see also” entries supplied by users is an effective—and low-cost—way to improve product usability.