This section provides background for the remainder of this document. The first subsection looks at related evaluation efforts from the HCI, CSCW and SLS communities. The second subsection defines our terminology, and the third subsection lays out the scenario-based approach in greater detail.
Evaluations of human-computer interaction have traditionally been done by a number of methods, including field studies, laboratory experiments, and inspections. Each method assesses different aspects of the interfaces and places different demands on the developer, user, and evaluator.
Evaluations of collaborative technology are done best through field evaluations because they can, among other things, be used to assess social-psychological and anthropological effects of the technology (Grudin, 1988). Field studies unfortunately require a robust system, an environment that can accommodate experimental technology, and substantial funding. These three conditions are difficult to satisfy, and so researchers turn to less onerous means, such as inspection methods and laboratory exercises.
System inspections, such as cognitive walk-through, heuristic evaluation, and expert inspection, employ a set of usability guidelines written by usability experts. There exists a fairly large set of guidelines for user interface design and single user applications, but few guidelines are available for multi-user applications or collaborative technology. Also, these methods are largely qualitative, not quantitative, and require HCI expertise which may not always be available.
This leaves laboratory experiments, or empirical testing, which as an
evaluation technique more closely relates to field studies than
inspection techniques. Experiments are very appealing for new and
rapidly evolving technology and are less expensive than field
studies. However, since they are conducted in controlled environments
with time restrictions they less accurately identify dynamic issues
such as embedding into existing environments, learning curves, and
acculturation. 
The process that we present here cannot presuppose a specific work context. Rather, we have chosen to develop scenarios and measures based on high-level collaborative system capabilities, to provide broad applicability across a range of applications. Ethnographic studies related to specific work contexts could provide a useful addition tool for validating the measures, because some measures may not be appropriate or of interest in certain contexts.
Scenarios are used in laboratory experiments to direct system usage. They are a versatile tool that can be used for many development activities including design, evaluation, demonstration, and testing. When used for evaluation, scenarios exercise a set of system features or capabilities.
We would like to define general, reusable scenarios for collaborative technologies. This is a challenge, requiring consideration of a large set of technologies, but we can build on earlier work in this area.
In 1995, researchers met at EHCI to develop scenarios that could be used to design, evaluate, illustrate, and compare CSCW systems (Bass, 1996). Their approach was to define generic tasks that would be used to construct technology specific scenarios. The tasks they described were mostly single user activities like joining a conference, setting up a new group, and integrating a system with a collaborative tool.
Our approach (see below) defines collaborative system capabilities or functional requirements that are used to decompose scenarios into a task/capability matrix. Our ``capabilities'' are defined at a higher level than the tasks defined by the EHCI researchers. The tasks we use to create the matrix are taken from the scenarios that we define which are much more general than those described by the EHCI group. Consequently, they are appropriate vehicles for cross-technology comparisons. Many of them can also be decomposed into subtasks if the whole scenario is too large to use for this purpose. Also, general scenarios can be instantiated for any system that supports the capabilities required by the scenario with little change to the scenario.
Nardi (1995), who has extensively studied the use of scenarios for interface design, has argued for a provision to create a library of reusable scenarios. We will begin to populate such a library with our capability-level scenarios. These scenarios are technology independent. Technology specific scenarios will be added when the scenarios are specialized for real systems.
In order for a scenarios to be reusable we must design them with ``salience.'' Potts (1995) emphasizes the importance of identifying goals and obstacles in developing salient scenarios. We are using these two attributes to characterize our scenarios. Each scenario will have a set of goals and obstacles demonstrated or exercised in a scenario. The evaluator using our scenario library will begin by identifying the goals and obstacles addressed by the system under evaluation. From that characterization the evaluator can select the appropriate scenario.
The sections that follow will clarify our methodology for defining scenarios for reusability and selecting scenarios for evaluation.
We find it necessary to pause at this point to define the terminology we will use in the remainder of this report.
This evaluation program is concerned with the three principle variables of people, collaborative environments and collaborative activities. We know that people are the human actors who engage in collaborative activities. But how can we classify the collaborative environments and activities in meaningful ways? What levels of abstraction are useful to our discussion of evaluation?
A collaborative environment is a system which supports users in performing tasks collaboratively. It may be a particular piece of software, such as Lotus Notes, or MITRE's Collaborative Virtual Workspace, or it may be a collection of simpler components used together to enable collaboration.
We are charged with the task of providing an evalutation methodology for collaborative environments, present and especially future. Part of our approach involves examining the types of things an environment allows one to do in support of collaboration. To describe them, we must define the terms capability, service, and technology.
Collaborative capabilities are relatively high-level requirements imposed upon a collaborative environment in order to support users in performing particular collaborative tasks. Examples are things like synchronous human communication, persistent shared object manipulation, archival of collaborative activity, etc.
The term service is used to describe the means by which one or more collaborative environments achieve a given capability, and technology is used to describe the the particular hardware and/or software implementation of a service.
So for example, for the collaborative capability of synchronous human communication, one service which may is used to achieve it in a variety of collaborative environments is audio conferencing. One technology for doing audio conferencing is LBL's Visual Audio Tool.
Examining which capabilities a collaborative environment supports, and the services and specific technologies it uses to do so is one way of generating a functional categorization of the collaborative environment, which can be used to determine more easily which types of activity it is best suited for.
Tasks or collaborative activities are the things which people do, or wish to do, collaboratively.
We use the term task synonymously with collaborative activity. This is a term that transcends the level of detail at which the activity is described: task may refer to anything from a real activity that people are engaging in to a highly scripted mock-up of such an activity intended strictly for evaluation purposes. We also talk about subtasks which are smaller chunks that a task may be broken down into.
We may utilize several different types of description of mocked-up activity in the course of our evaluations. At the highest level we have the scenario. This is a high-level description of an artificial collaborative task that subjects might be asked to engage in to test out collaborative systems. A fully specified scenario will include the background documentation, instructions, etc. required to actually have a subject or subjects enact the scenario. Scenarios are often broken down into their constituent tasks. In some cases it will be possible to do evaluations based upon only a subset of the tasks for a given scenario.
Some evaluation goals might be better met by scenarios which are precisely repeatable, perhaps even by automated actors. For this purpose we have scripts which are procedural descriptions of scenarios. Scripts can be written at levels corresponding to technologies, services, or capabilities.
A technology-level script specifies all the details of the enactment of the scenario, including how each step should be completed on a particular technology. This type of script would allow users unfamiliar with the collaborative environment being evaluated, or even automated participants (participatons) to play roles in the script.
A service-level script specifies the steps in the scenario in terms of the services to be used, without committing itself to a particular technological implementation of the service. It may, for example, state that a particular discussion is to take place via videoconferencing, without specifying exactly how that will be achieved in any particular system.
A capability-level script gives the detailed instructions in terms of the capabilities that the script uses. This script does not require the use of any particular services, so it can be run on any system that provides some means of instantiating the required capabilities. This last level makes it possible to define scripts which can even be used across platforms which provide radically different sets of services in support of the same basic capabilities.
Section 3 presents a framework of collaborative tasks, capabilities and services in greater detail.