Position Paper on the Suitability to Task of Automated Utilities for Testing Web Accessibility Compliance
Bill Killam, User-Centered Design, Inc., Ashburn, VA 20147
bkillam@user-centereddesign.com
Bill Holland, Olympus Group, Inc. Alexandria, VA 22314
bill@olympusgroup.com
In
our opinion, the current discussions about accessibility have included two very separate issues that we believe
need to be addressed separately. The first
issue is the law and, specifically the requirements of sSection 508. As a law, these requirements of section 508 must be unambiguous or, as members of our industry is are more likely to think of
it, provide for near
perfect inter-rater reliability. We
believe that automated utilities that canvass individual web pages, web-based applications, or other software applications can improve the efficiency
of accessibility-related heuristic evaluations. Specifically, they are
valuable at identifying barriers to
accessibility, but they currently do not and potentially cannot address accessibility
itself.
However,
fFor
an encompassing evaluation of any web sitea product accessibility, it is ultimately a question
of the end user’s ability to use locate information or exercise
functionality that determines accessibility.
Given this distinction, we believe that the accessibility compliance issue is two fold problem: (1) removing the barriers
to accessibility, and (2) making the site accessible. human
Human
intervention is required to supplement a utility’s findings, as well as
conduct individual reviews of certain page elements and heuristics that
automated tools are unable to perform at this time.
1.Benefit:
Automated tools minimize the chances of missing
accessibility problems, and therefore
omission in the final report. For instance,
a human evaluator may have time to review only a representative
sample of images for ALT attribute inclusion,
whereas an automated tool can review all
images on a site.
1.Benefit:
Automated tools can operate much more quickly than
a human evaluator, therefore
conducting an evaluation in mere minutes, whereas a human evaluator
may take several hours to review the same accessibility criteria.
1.Risk:
Use of an automated tool may give a human evaluator
a false sense of security
when evaluating a user interface. Some tools are better than others in informing
evaluators when further review is necessary.
Human evaluators need to be aware of these shortcomings.
1.Risk:
Sole reliance on automated tools by web developers
as indicators of accessibility requirement
compliance gives a false sense of security to the developers.
In addition, it may inaccurately portray the site as being fully accessible to
persons with
disabilities, when in fact there may be glaring accessibility-related problems.
1. Benefit: Automated tools
minimize the chances of missing accessibility problems, and therefore omission
in the final report. For instance, a human evaluator may have time to review
only a representative sample of images for ALT attribute inclusion, whereas an
automated tool can review all images on a site.
2. Benefit: Automated tools can
operate much more quickly than a human evaluator, therefore conducting an
evaluation in mere minutes, whereas a human evaluator may take several hours to
review the same accessibility barrier criteria.
3. Risk: Use of an automated
tool may give a human evaluator a false sense of security when evaluating a
user interface. Some tools are better than others in informing evaluators when
further review is necessary. Human evaluators need to be aware of these
shortcomings.
4. Risk: Sole reliance on
automated tools by web developers as indicators of barrier requirement
compliance gives a false sense of security to the developers. In addition, it
may inaccurately portray the site as being fully accessible to persons with
disabilities, when in fact there may be glaring accessibility-related, and
therefore, usability, problems.
The accessibility of a web site is directly related to its
usability. And its usability is directly related to its accessibility. A site
may be in compliance with guidelines for persons with disabilities but still be
completely unusable to both able and disabled persons. Automated tools that
measure a site’s “usability”, despite the best intentions of their developers,
have not traditionally covered all aspects of a site adequately. Correspondingly,
no tool so far has demonstrated its ability to get beyond the accessible and
usable barrier.
Many recent tools
available provide feedback to evaluators for situations where the tool was unable to discern
compliance with accessibility standards. This can help identify where manual review by a human evaluator is necessary. Some examples:
o Color: Because of the
combination of style sheets for text, deprecated FONT tags, and graphic images
to render color, there are many combinations of web coding to consider. A
change in headings from one page to another can be signified not only by
non-color attributes, but color as well. In some cases however, color is the
sole indicator of change in an item. Automated tools are not currently equipped
to compare before-and-after views of web pages based on style sheets or
graphics. Therefore, human evaluation must be done to assure web pages are accessible
to color-blind persons.
o Images: The mere presence of an
ALT attribute for an image is not necessarily sufficient for true compliance
with current guidelines. For example, an image in the upper left-hand corner of
a page may be labeled as “logo” in its ALT attribute, and may even be a
hyperlink. If no TITLE attribute is set, the ALT attribute is the sole
indicator of what that link/image does. A visually impaired person would have
to guess as to the target of this particular link. An automated tool would pass
through this IMG tag, note the presence of an ALT attribute, and give that tag
a passing mark, even though the attribute is not fully descriptive of the image
or the target of the link. Here it may meet the accessibility barrier
criteria, but still be inaccessible.
o Tables: Most guidelines require column and row heading attributes be set for each table cell where there are two or more rows of table data. Because tables are used not only for numeric presentation, but also for page element positioning, the determination of each application is difficult to make without visually reviewing the table. For this reason, automated tools cannot adequately verify table-related guidelines.
The mere presence of an ALT attribute for an image
is not necessarily sufficient for true compliance with current guidelines. For
example, an image in the upper left-hand corner
of a page may be labeled as “logo” in its ALT attribute, and may even be a hyperlink.
If no TITLE attribute is set, the ALT attribute
is the sole indicator of what that link/image does. A visually impaired person
would have to guess as to the target of this particular
link. An automated tool would pass through this IMG tag, note the presence of
an ALT attribute, and assume all was well,
even though the attribute is not fully descriptive
of the image or the target of the link. Therefore,
it is not usable.
An automated tool can parse a style block, looking
for specifications for positioning and absolute size control of general page
elements. These tools can, for the most part, detect absolute positioning of
page elements and can thus raise an issue with it. Because of the disparity in
rendering between browsers
however, this is still an issue that should be reviewed manually.
Most guidelines require column and row heading
attributes be set for each table cell where there are two or more rows of table
data. Because tables are used not only for numeric presentation, but also for
page element positioning, the
determination of each application is difficult to make without visually
reviewing the table. For this reason, automated tools cannot adequately verify
table-related guidelines.
A meticulous code review is required to ensure
actions can be completed without the aid of client-side scripting. Unless all
possible actions are taken
on each page being reviewed, automated tools will not be able to determine if
they require scripts to execute or not. For instance, upon a form submission,
the INPUT tag may have an onClick event handler that provides for form element
verification. However, if scripting is turned off on the client side, the form
should still submit and the validation done on the server side. Automated tools
are currently not advanced
enough to determine the exact behavior
of client-side script commands and the repercussions of disabling them.
E.B. White, the author, once wrote the following words when
reviewsing a “Reading Ease
calculator,” an early version of automated tools such as the FOG index for determining
readability, wrote:
“There is, of course,
no such things
as the reading ease of written matter.
There is the ease with which matter can be read, but that is function of
the reader, not the matter.”
We believe that this applies to
accessibility in the same manner as as well as we
have always applied it to usability.
By our definition, once
the barriers to accessibility have been removed, we are now faced with the greater and more
difficult task of addressing actual accessibility. Since accessibility, in our definition is the ability of handicapped userspersons with
disabilities
to locate and use information or exercise the functionality of aan web-based or other software application, we are, in fact, talking about usability. As
such, the tools, techniques, and practices of usability now apply. Expert
reviews, heuristic evaluations, style guide compliance, and other
non-user-based techniques can be applied (assuming
experts, heuristics, and style guides exist). These techniques, though cost effective
and valuable, are not substitutes for user-based testing.
We are often compelled
to perform user-based testing is when the population of users
is a unique subset of the overall
population. Some of our clients and
potential clients have rejected this
notion since there is such a wide range of handicapped users with disabilities. But
as expert
reviewers, we docannot not claim to be able to eliminate our backgrounds,
experiences, and cognitive processes from the reviews, so we recognize the necessity of testing children,
older populationsadults, accountants, or other populations. And we accept that 5-8 participants, or a
minimum of 8, or even 20 (depending on to who’se position your subscribe) can provide
valuable insight into the issues of a population from a practical
(though not research) perspective. So why not for handicapped populations?
Even if we learned to
use the special adaptive tools of the handicapped community (e.g., refreshable
Braille, screen readers, a mouth stick, a sip-and-puff interface), can we really believe that we can close their eyes and be
blind? Can a person with “normal”
hearing ever really emulate the difference in language skills between
themselves and a congenitally deaf person? Can our infrequ3ent use of a mouth stick ever
compare with a person who is required to use one for their daily activities? The
unique nature of the handicapped population (which is, in fact, multiple populations) in terms of human factors
effecting usability, should be more likely to compel us to push for user-based
testing.
Automated tools can be very helpful to the
usability professional when conducting heuristic
evaluations of web sites. The speed at which these tools operate can
significantly improve the efficiency of such evaluations. It is important to
note that the exclusive use of such tools under the pretext of accessibility
compliance can be misleading at best, however. Web site operators that obtain
“certification” by such tools and whose sites are subsequently
shown to be inaccessible can incur significant
development costs to bring it into compliance
with W3C guidelines or even
the law, as the case may be. These development
costs could have been appreciably
reduced had manual human evaluation been done in addition
to that done by an automated tool, early in the development
lifecycle.
If we use automated
tools to triage the process of identifying the barriers to accessibility, we
have made our jobs significantly easier, more thorough, and more cost effective. If we can develop fully automated tools to
assess the issue of removing the barriers to accessibility, than we have accomplished no small feat in
terms of both technical accomplishment and value to the industry. But, if our assertion is correct, then we
have only performed the first necessary step in addressing accessibility—removing the barriers. We must now address the special condition of
usability related to handicapped users and accept that user-based evaluation is
the only true test of success, though there may be other tools that can serve
as reasonable substitutes. We must also
recognize that we are attempting to address two fundamentally different but interdependent
problems and must apply the right tool to the job and know when each job is
truly done.If
we can only use automated tools to triage the process of identifying the
barriers to accessibility, we have made our jobs significantly easier, more
thorough, and more cost effective. If we can develop fully automated tools to assess the
issue of removing the barriers to accessibility, than we have accomplish no
small feat in terms of both technical accomplishment and value to the industry. If
our assertion is correct, then we have performed the first necessary step in
addressing accessibility. We must now address the special condition of usability and accept that
user-based evaluation is ultimately the test of success, though there
may have other tools that can serve as reasonable substitutes. We must recognize that we are attempting to address
two fundamentally different but
interdependent problems and must
apply the right tool
to the job and know when each
job is truly done.