1997 Symposium Report
Usability Engineering 2: Measurement and Methods

( UE2 archived web site, 1996 symposium report)

Laura L. Downey, Chair
Sharon J. Laskowski
Elizabeth A. Buie
William E. Hefley
(Link Legend: Unvisited Link Visited Link)

NOTE: This report has been submitted to the SIGCHI Bulletin for publication in the October 1997 issue.
This report summarizes the activities of the Usability Engineering 2: Measurement and Methods (UE2) Symposium held March 3, 1997 at the National Institute of Standards and Technology (NIST) in Gaithersburg, Maryland, USA. The workshop was co-sponsored by NIST's Information Technology Laboratory (ITL) and by the Human- Computer Interaction Laboratory (HCIL) of the University of Maryland. Over 145 people attended a comprehensive technical program that included parallel tracks for managers, experts and novices.

UE2's purpose remained the same as the first symposium held in 1996: it once again brought together industry and government to exchange information and strategies for achieving effectiveness, efficiency, and satisfaction in computer-based government systems. New for 1997 was a special emphasis on measurement and methods along with a parallel track format to address the varying attendee profiles such as job function and level of expertise.

The symposium continues to focus on incorporating usability engineering into the government arena and extends the work from last year's symposium (see symposium report in SIGCHI Bulletin, October 1996, pp. 66-67, Vol. 28, Num. 4). The need to incorporate UE techniques into the government arena was identified and expanded upon during several previous gatherings of human-computer interaction (HCI) professionals. Past SIGCHI Bulletin issues (October, 1994, pp. 49-50 and October, 1995, pp. 35-37) contain reports on previous gatherings.

Usability Engineering and Its Necessity In Government
The National Performance Review has the explicit objective of improving the performance of US Government agencies. We believe every government agency's peformance depends on the usability of the computerized systems that support that agency's work. Usability engineering provides practical techniques for increasing productivity and decreasing costs - two primary concerns of those procuring and building government systems. The need to specify and verify usability requirements is critical in new government systems and just as applicable in the many current and planned legacy updates. The field of usability engineering also offers ways to evaluate the mandated integration of commercial- off-the-shelf (COTS) products and evaluation of the COTS products as stand-alone tools. Procurers and builders of government systems are faced with rapidly designing and developing systems that must meet the public demand for digitized information, industry's demand for critical operation information, and workers' demands for systems that enable them to disseminate and process various information requests. It is critical that procurers and builders of government systems employ techniques that enable the development of usable governmet systems in order to maximize the return on taxpayer's dollars that fund the design and development of these systems. Economics and productivity play major roles in the exploitation of technology. As Dr. H. Rex Hartson stated in the 1996 symposium executive overview:
"We expect technology to transform the American economy into the most productive in the world. The single most significant factor in manifesting this promised productivity from the technology is USABILITY.

The organizers consist of the authors of this report.

The symposium covered the following topics:
  • the bottom line, cost effectiveness in usability engineering
  • usability testing of web sites
  • government success stories
  • standards and guidelines
  • specifying usability in the RFP
  • re-engineering legacy systems
  • comparing usability evaluation techniques
  • evaluating collaborative tools
  • international standards and European legislation on usability
  • usability testing equipment
  • challenges in measurement and evaluation for usability
Presentations and Speakers
The UE2 archived web site contains abstracts of all the presentations.

"Towards User-Centered Software Engineering"
Steve Cross, Director, Software Engineering Institute

Afternoon Plenary:
"Designing Information Abundant Web Sites"
Ben Shneiderman, Head, Human-Computer Interaction Lab, University of Maryland

Success Stories:
  • "Usability Testing Web Sites at the Bureau of Labor Statistics (BLS)
    Michael Levi, BLS
  • "Re-Engineering User Interfaces for the Maryland Department of Juvenile Justice"
    Anne Rose, University of Maryland
Management Track:
  • "Executive Overview: Usability Engineering and the Bottom Line"
    Laura L. Downey, Vignette Corporation
  • "Making Sense of HCI Standards and Guidelines"
    Elizabeth Buie, Computer Sciences Corporation (CSC)
  • "Usability in the RFP: Specifying Product and Process"
    Elizabeth Buie, CSC and Bill Hefley, Carnegie Mellon University (CMU)
  • "International Standards and European Legislation"
    Nigel Bevan, National Physical Laboratory
  • "Ergonomics Legislation in Italy and Its Impact on Government Spending for Software Usability"
    Anna Giannetti, SOGEI
Novice Track:
  • "Achieving Usability Through User-Centered Design"
    Nigel Bevan, National Physical Laboratory
  • "Usability Considerations in the Re-Engineering of Legacy Systems"
    Charlie Kreitzberg, Cognetics Corporation
  • "Making Sense of HCI Standards and Guidelines"
    Elizabeth Buie, CSC
Expert Track:
  • "Development and Use of Guidelines at NASA's GSFC"
    Dana Uehling, NASA Goddard Space Flight Cente
  • "Increasing the Cost Effectiveness of Your User Interaction Development Process"
    Debby Hix, Virginia Polytechnic Institute and State University
  • "Comparing Usability Techniques: Expert Review vs. Usability Testing"
    Ginny Redish, Redish & Associates, Inc.
  • "Evaluating Collaborative Tools"
    Jane Mosier, MITRE
Birds of a Feather:
Meeting the Common Needs of Disabled Users and Nomadic Users
Susan A. Brummel, General Services Administration (GSA) and
Terri Youngblood, Department of Education

Usability Testing Equipment:
  • Dick Horst, UserWorks, Inc. and Norm Willcox Associates, Inc., Norm Wilcox
  • Ted Scostak, TRC
Closing Panel Discussion:
Challenges in Measurement and Evaluation
Sharon J. Laskowski, NIST (Chair) with
Donna Cuomo, MITRE; Nigel Bevan, NPL; Bill Hefley, CMU

The number of attendees rose 28% from last year (116 vs. 148). They included representatives from industry, government, academia, research institutes, and federally funded research and development corporations. We were especially pleased that several high-ranking government information technologists attended. One of our goals was to increase the number of decision-makers and that goal was met. All tracks were well attended including the managerial track. For comparison purposes, the attendee breakdown is listed for both the 1996 and 1997 symposiums:

1996 - 116 attendees 1997 - 148 attendees relative difference

(42%) industry
(44%) government
19 (16%) NIST
32 (28%) non-NIST gov.
(8%) academia
(7%) other

(26%) industry
(63%) government
31 (21%) NIST
62 (42%) non-NIST gov.
(6%) academia
(5%) other
(+ 5%)
(- 2%)
(- 2%)

Another important goal set after last year's symposium was to increase government participation especially that of non-NIST government personnel. That goal was also met as evidenced by a raw increase from 32 to 62 and an overall percentage increase from 28% of 1996 attendees to 42% of 1997 attendees. The one surprising demographic factor observed by the program committee was the decrease in industry attendance although we can attribute some of the decrease to the publicity efforts which heavily focused on attracting government personnel. Additionally, the CHI conference was early this year and several industry people informed us that they had to choose between the two meetings due to budget and time constraints.

Similar to last year, we received much positive feedback along with suggestions for improvements and direction. Feedback was gathered via email, conversations with speakers and organizers, and symposium feedback sheets (52/148 returned). A large majority of attendees indicated the need/desire for continuation of the symposium activities. General constructive feedback and suggestions for future symposia included:
  • more time for questions and discussion
  • more case study examples
  • construction of a UE cookbook/template
  • evaluation of prototyping tools
  • evaluation and comparison of different UE methods
  • more discussion and explanation of metrics
  • attract more project managers
  • address measurement in context (quality in use)
  • evaluation of COTS
  • more practical "how to" sessions
Future Activities
The organizers are currently putting together a post-symposium publication, which is scheduled for release sometime in Fall 1997. In order to reach a wide audience and to leverage the expertise of the Usability Professional's Association (UPA), NIST and UPA plan to hold a special "government day" in conjunction with the 1998 UPA annual meeting which will be held in Washington, D.C. The government day will immediately precede the UPA conference.

We invite and welcome participation in our activities. Anyone interested in focusing on usability in the government arena may send email to ldowney@vignette.com (+1-512-425-5495) or sharon.laskowski@nist.gov (+1-301-975-4535) for more information.

Author Information
Laura L. Downey, chair of the symposium, is a computer scientist at Vignette Corporation. Laura's research focus is on human-computer interaction with a special emphasis on usability engineering. Currently she is conducting a system evaluation of an information retrieval (IR) prototype, and investigating rapid, remote, and automatic methods for evaluating web site usability.

Laura L. Downey
Vignette Corporation
3410 Far West Blvd., Suite 300
Austin, TX 78731 USA

Dr. Sharon J. Laskowski is a computer scientist and group leader of the Visualization and Virtual Reality Group in the Information Technology Laboratory of the National Institute of Standards and Technology where she is currently investigating the application of visualization techniques to information retrieval and navigation. She has also participated in research and prototyping efforts focusing on usability and searching of World Wide Web-based virtual libraries. Previously, she conducted research and development in text analysis, information fusion, and plan recognition at the Artificial Intelligence Center of the MITRE Corporation. Dr. Laskowski received her PhD in computer science from Yale University.

Dr. Sharon J. Laskowski
National Institute of Standards and Technology
Information Access and User Interfaces Division
Building 225, Room A216
Gaithersburg, MD 20899 USA

Elizabeth Buie is a Senior Principal Engineer with Computer Sciences Corporation's System Sciences Division in Laurel, Maryland, where she focuses on HCI quality in spacecraft flight operations systems at NASA's Goddard Space Flight Center. She has worked on projects for a variety of civilian government agencies in the US and Europe. Elizabeth chairs CSC's company-wide SIG on HCI and is a member of the Human Factors and Ergonomics Society's HCI Standards Committee. She has a Master of Science in Mathematics from North Carolina State University and a Master of Arts in Human Development from the University of Maryland.

Elizabeth Buie
Computer Sciences Corporation
1100 West Street
Laurel, MD 20707 USA

Bill Hefley teaches in the Information and Decision Systems program in the Department of Social and Decision Sciences at Carnegie Mellon University. He is also a member of the Human-Computer Interaction Institute at Carnegie Mellon University and a Visiting Scientist at the Software Engineering Institute. His twenty-plus years of experience is in systems and software engineering, human-computer interaction, and project management. His research and consulting efforts currently focus on organizational and process improvement, focusing on the engineering workforce and both software and usability process improvement.

William E. Hefley, CCP, CDP
Social and Decision Sciences
Carnegie Mellon University
Pittsburgh, PA 15213 USA

Back to Government Day/UE3 Home Page

Last Updated: 4-28-98 Wednesday, 30-May-2001 12:46:18 MDT