Our goal is to describe a given collaborative system and to evaluate how well that system supports various kinds of collaborative work. The framework outlined in this chapter describes the design and implementation space for collaborative systems used in a work environment.
Groups should be able to use our framework to describe their requirements and to ascertain how well a given CSCW system supports their work. Developers of CSCW systems should be able to use our framework to describe their system and to determine the types of group work that could be easily accomplished using the system.
This framework builds on a framework devised by Pinsonneault and Kraemer (1989) to analyze the impact of technology on group process while controlling for the effect of other contextual variables.We have merged the work of McGrath(1984) into our expanded framework to enable us to classify task that groups perform.
In this section, we give an overview of the framework and we describe how the framework can be used to describe and evaluate CSCW systems. We also provide detailed descriptions of each level of the framework.
The framework is divided into four levels: requirement, capability, service, and technology.
The requirement level of the collaborative framework describes the requirements of the group with respect to the tasks being performed by the group and the support necessitated by the characteristics of the group. The tasks described in the framework include work tasks as well as transition tasks. Requirements for supporting different types of groups include support for the social interactions of the group as well as the requirements due to group size, location, computer platforms, etc. The requirement level is divided into four sections: work tasks, transition tasks, social protocol requirements, and group characteristics.
The capability level of the framework describes functionality that is needed to support the different requirements. The capabilities can be divided into subsections that correspond to the four subsections of the requirement level. The functionality described in capabilities can be obtained from different services. For example: the capability to have a side chat with another meeting participant during an electronic meeting could be accomplished by a text chat service or by telephone service. Certain capabilities may be necessary or recommended to support work and transition tasks, social protocols, and group characteristics in the requirement level.
The service level describes services such as e-mail, audio, video, application sharing, networking services, etc. that can be used to support the capabilities needed in CSCW systems. Different types of services can be used to provide the same capabilities needed to support specific requirements. Comparisons and tradeoffs of performance and cost can be made at this level.
The technology level describes specific implementations of services. This level could be considered as the set of all possible components needed to build a given CSCW system, including integration and user interface components. For example, all different e-mail systems would exist at this level, as would the numerous ways to implement floor control, and the various algorithms to control documentation locking and requesting, and the various networking services such as ATM. Specific implementations can be compared with respect to performance, cost, functionality, and usability.
Figure: The collaborative framework
The framework can be used for evaluation either in a top-down (requirement level to technology level) or bottom-up fashion. Table 1 gives an overview of the questions asked in each mode of evaluation.
The framework used in either mode of evaluation allows the evaluator to select a subset of the available systems or to determine if a single system will support the group adequately. Systems that do not meet or exceed acceptable levels for the measures available at a given level can be eliminated.
At the requirement level we can evaluate how well a CSCW system supports the work tasks, the transition tasks, the social protocols and characteristics of a specific group. We can also evaluate the artifacts produced as well as the process the group used in producing an artifact. The detailed framework description (Section 3.3) describes the measures for task outcome for each task type. It is important to note that the variables being measured differ depending on the type of task the group is doing, that is, different measures are needed to evaluate the outcome of a brainstorming task than to evaluate the outcome of a planning task.
Group process refers to the way the group works. Examples of measures of group process include time to reach a decision, efficiency of communication and equality of participation. Depending on the work task being perfomed, some measures are more relevant than others. (See Section 5 for details). For example, equal participation by group members is important in brainstorming tasks. Therefore, groups that work well in brainstorming should have a high rating for equality of participation. Equal participation is not as necessary in a lecture situation. Therefore, ratings on equality of participation from individuals involved in lecture types of collaboration would not necessarily be rated high even though the lecture was very successful.
At the capability level, we are interested in evaluating how well given capabilities support the work tasks, transition tasks, social protocols, and characteristics of the group. To obtain answers to these questions, measures such as time on task, awareness questions, amount of setup time for equipment and configuration, are used.
At the service level, we can use the functionality of various types of services to understand how a given capability would be supported using a particular type of service. This allows us to compare sevices to determine which service seems most appropriate given the characteristics of a group. Performance thresholds, including robustness, can be examined at this level. Tests, such as the ability to view an image with a certain amount of discrimination and the acceptability of audio or video, could be conducted at this level.
At the technology level, specific implementations exist. Therefore, actual usability measures can be examined for an implementation. The implementations of the services can be evaluated to make sure they meet threshold values. Performance comparisons can be made at this level between different implementations.
The previous section discussed the types of questions that should be answered in order to evaluate CSCW systems. In this section, we address the process of answering those questions. The best way to evaluate a system is to use it. In order to compare several sytems, we need to make sure that they are used in a similar fashion. This framework has been designed to support scenario based evaluations, thus allowing evaluations in a real-work context that are repeatable in order to make comparisons across systems. Section 4 gives a detailed description of using scenarios for evaluation.
Scenarios are constructed by putting together tasks based on the work task types described in the requirement level of the framework. A group selects the various task types they will be using the CSCW system for, the social protocol parameters appropriate for the group and the type of transition tasks needed. The work task types are generic and thus, to construct scenarios, each group will need to instantiate a generic task type with objects from the work domain of the group. Transition tasks will be more fully described in the requirement level detailed description (Section 3.3.1), but they can be viewed as overhead tasks: cleaning up from one meeting activity and moving on to the next, assigning responsibilities to group members, arranging meeting times, notifying members, etc.
For example, here is a scenario that could be representative of what a group might do:
"The human resource group meets at the usual group meeting time, decides upon the location of the Christmas party from the list suggested previously. They then decide upon the menu from the possibilities provided by the caterer and appoint Susan to call the caterer. Then they brainstorm on ideas for entertainment. The meeting ends after one hour. They decide to meet at the same time next week."
In the following descriptions T i refers to a type of transition task i. Task Type # refers to a work task type of that number. P j refers to a collection j of attributes for a social protocol. Using the collaborative framework to describe this session, we would say:
The group starts with a transition task to begin (T start). They perform a decision-making task with no known solution (Task Type 4) using an appropriate social protocol (P chair). They summarize the decision in the transition task (T summary) and move to the next task which is another instance of a decision making task with no known solution (Task Type 4) and use the same social protocol (P chair). The group transitions using a transition task to assign action items (T action item) to the brainstorming task (Task Type 2). A different social protocol (P no chair) is used for this task. An ending transition task which summarizes the plans and selects the time for the next meeting is then performed (T time plans).
More succinctly, we could write:
T start(Task Type 4, P chair)
In this section, we describe each level of the framework in detail.
We describe the requirement level based on the four subsections: work task types, transition tasks, social protocols, and group characteristics.
3.3.1.1 Work Task Types
The task descriptions are based mainly upon tasks from group research (McGrath, 1984). We have added an additional task type that is necessary to form a comprehensive collection of current work practices. We have noted in the description of task type nine that it is not from McGrath's original set.
The tasks described by McGrath should be thought of as a continuum, not discrete tasks. Along this continuum there are four higher level types: generate tasks, choose tasks, negotiate tasks, and execute tasks. Within these four task types, generic tasks are numbered [1-8, original McGrath] with successive tasks related to each other. The task descriptions and examples should enable users of the framework to select generic tasks that are comparable to the type of work they do.
In our detailed description, each task type has suggested measures of task outcome. In addition, for many task types, research has uncovered problems that may occur with groups performing this type of task. In some instances, computer-mediated processes or other group processes may be able to alleviate these problems. We have listed known problems so that comparison measures can be made to see if there is any effect. Where there is research about computer-mediated work or group interactions and the effect on the task outcome, we have included it under the heading "known research." An example of each task is also provided to help in understanding how the generic task maps to a real world task. Under "suggested capabilities" are listed those capabilities which the research suggests may be valuable in carrying out that particular generic task.
Type 1: Planning tasks (McGrath)
Group members are given a goal (previously chosen objective) and asked to develop a written plan for carrying out the steps needed to reach that goal. The written plan should include alternative paths or actions.
Specific measures for this type of task:
Known research:
Example: The group has to produce an advertising campaign for a new account by the end of the month. They have a meeting to plan the different tasks that each member will carry out, complete with time lines for doing so and different coordination points.
Suggested capabilities:
Type 2: Brainstorming and group creativity
Members of a group are given a particular topic area and asked to brainstorm on ideas.
Specific measures for this type of task:
Known research:
Example: The group has a goal to raise $200,000 to build a new community center. They generate ideas for funding raising events, people to ask for contributions and possibilities for loans or selling "shares" to the community members to raise this money.
Suggested capabilities:
Type 3: Intellective tasks
The group is asked to solve a problem for which there is a recognized solution. The group is asked to determine a concept, given instances of the concept. Alternatively, groups can be asked to generate an instance of a concept and are given feedback as to whether it is or is not the concept.
Specific measures for this type of task:
Known research:
Example: A logical reasoning problem such as the cannibal and missionary problem is an example of an intellective task with a demonstrable right answer.
The missionary and cannibal problem: Three cannibals and three missionaries are on a river bank. They need to cross to the other side. There is a row boat that can hold only two people. Only missionaries know how to row. At no time can there be more cannibals than mssionaries on either side of the river. What is the minimum number of trips that can be made to transport all six to the other side of the river?
Suggested capabilities:
Variant: The twenty-question task
A target object is selected. A group or individual is asked to determine that object by asking a series of questions that can be answered by "yes" or "no". The goal is to determine the target object with as few "no's" as possible with a maximum of 20 allowed.
Specific measures for this type of task:
Type 4: Decision making task
Group members are asked to develop consensus on issues that do not have correct answers.
Specific measures for this type of task:
How far and in what direction (if any), the group as a group and the individuals in the group shift. [Attitudes are measured before and after group discussion]
Example: The group is asked to decide which of three possible job candidates should be hired. All candidates are computer science graduates with specialties in computer engineering. However, their work experiences are quite different. The group must decide which candidate would be the best match for the job.
Suggested capabilities:
Type 5: Cognitive conflict tasks
Members of the group hold different viewpoints. The group is asked to make a series of decisions from available information that is imperfectly correlated with criterion. The group has to arrive at a decision.
Specific measures for this type of task:
Known research:
Example:
The group is hiring a designer for its User Interface Group. Three candidates are at the top of the list. One candidate is a computer scientist major with some interface design experience, another is a psychology and human factors major, and the third is a graphics art and industrial design major. The group is divided about the type of experience best suited to this position. The group is interdisciplinary and each discipline tends to favor hiring the candidate most closely aligned with their discipline.
Suggested capabilities:
Type 6: Mixed motive tasks
Mixed motive tasks present a range of tasks, differentiated by the degree to which a group member's outcome is affected by a combination of his own actions and the group's outcome. McGrath also includes dilemma tasks in this category. Dilemma tasks involves a conflict of interest where the choices made by individuals or groups are combined to jointly determine the outcome. However, the collaboration between the individuals or groups is not directly related to determining the choices of each. Therefore, we have not included dilemma tasks in our framework.
Type 6A: Negotiation task: The group is divided into x subgroups with a negotiator elected for subgroup. The different subgroups disagree on an issue but an outcome has to be reached. Tradeoffs have to be made in multiple dimensions. It is not necessarily a zero-sum problem.
Specific measures for this type of task:
Example:
Company A and Company B are negotiating the sale of supplies from A to B. Company A wants to sell more of the supplies at a lower price to company B, but this means that company B, while saving money on the sale, will have to arrange financing with company A.
Suggested capabilities:
Type 6B: Bargaining task: (suitable for two individuals)
A conflict of interest must be resolved among two individuals, but in this case whatever one individual gains results in a loss for the other individual. A tradeoff is made on a single dimension, such that what one party gains, the other party loses.
Specific measures for this type of task:
Known research:
Negotiators are more competitive when any of these conditions hold:
Negotiators who do not belong to the group feel freer in the negotiation process but are less supported by the group.
Example:
There has been a price set by Company A for a machine that company B wishes to purchase. Company B does not feel that the price is low enough. Company A is trying to maximize profit as the company is having cash flow problems, but loosing the sale will be a problem also.
Suggested capabilities:
Type 6C:Winning coalition tasks
Subsets of members make agreements. The subset that wins then allocates the resources among the group members. The two research questions are the formation of the coalition, and the allocation of the resources.
Specific measures for this type of task:
Example:
A variant of pachisi is played in which a player's piece is given a weight. Moves are based on product of the weight of the piece and the roll of a pair of dice. Players can play alone or can combine with one other player by adding their moves together. They must then agree on how to divide the payoff assuming they win.
Suggested capabilities:
Type 7: Competitive performances
Groups are competing against each other with no resolution of conflict expected. Rather the goal of each group is to win over the other group. Subgroups are paired against each other an equal number of times, under an equal pattern of situations.
In the original McGrath work, these performances are physical. In this framework, these types of tasks may be physical or nonphysical performances.
Specific measures for this type of task:
Known research:
Example:
A focal group competes with an opposing team that has a series of preplanned, semi-standardized patterns. The responses to the focal group's moves are based on pre-planned strategies. A reconnaissance patrol or defense of a position is an example of such an activity.
Suggested capabilities:
Type 8: Non -competitive contests - against standards
Groups perform some sort of complex group task. The plan for the task has already been decided upon. In this type of task, the group is merely executing the plan.
Specific measures for this type of task:
Known research:
Example:
A survival task or rescue task in which a group has to perform to achieve a goal: getting back to base or saving individuals.
Suggested capabilities:
Task Type 9: (non-McGrath)
Dissemination of information tasks
The task is to distribute information among members of the group. Information can be distributed by group members sharing information with each other or a superior sharing information with others in the group. There may or may not be a question and answer session.
Specific measures for this type of task:
Examples:
A corporate officer gives a briefing to a division about the new sales strategy.
A professor gives a lecture to a class.
Suggested capabilities:
3.3.1.2 Transition Tasks
Transition tasks are tasks used to move between work tasks. This may include summarizing the outcome of the last task, assigning action items to members of the group, and noting dates for expected completion of assignments. The beginning and ending of any group collaboration involve transition tasks such as taking role, requesting changes to an agenda, locating missing meeting participants. Transition tasks also apply to asynchronous collaborations. A group member may suggest that the e-mail discussions about a particular subject be ended and volunteer to summarize the discussion and distribute this to group members.
We have also defined special startup and shutdown transition tasks to account for the activities associated with getting all members connected to the collaborative session. If the collaborative session is asynchronous, then startup transition tasks will occur during the entire duration of the collaboration as individuals access the collaborative space. A shut down transition task describes the activities that an individual needs to do to exit from the collaborative space.
The beginning of a synchronous collaborative session may include: role taking, reading minutes from a previous session or reviewing the previous session, viewing the agenda and making sure everyone has the necessary documents for a session to begin. Ending a collaborative session may include planning for the next session, saving collaborative documents as needed and shutting down collaborative systems.
A transition task may occur formally or informally, depending on the social protocol that the group is using. Transitions to the next work task occur formally if the chair of the group either moves the group to the next agenda item or the group votes to move to the next item. Informal transitions to the next work task occur when the group moves the discussion rather naturally to another topic or starts another group activity.
Measures of transition tasks:
3.3.1.3 Social Protocols
Social protocols define the ways in which collaborative sessions are conducted. Collaborative sessions may vary from informal sessions to very formal sessions with a chair, an agenda that is strictly followed, and rules of order. Social protocol capabilities support communication between group members; awareness of group members, group activities, group objects; and basic meeting conduct. Table 3 lists some parameters that social protocol requirements take into consideration.
Table: Parameters for Social Protocol
Example measures of social protocol:
3.3.1.4 Group Characteristics
Groups have different requirements depending on the makeup of the group, the location of the group members and the time requirements for collaborative sessions. These dimensions are detailed in table 3. In addition to considering the current dimensions of the group, system requirements should also take into account anticipated changes in these dimensions. For example, all members of a task force might start by being co-located but with the knowledge that, in two weeks, half of the group will be remotely located.
The capability level defines basic capabilities that support different types of collaborative tasks. These "collaborative capabilities" provide a means of matching up tasks with services. Each service provides certain collaborative capabilities. Matching the tasks and services must be done taking into account how well the service supports the capabilities and whether the service is acceptable given the group characteristics.
The capability level can be divided into those capabilities that support work tasks, transition tasks, group characteristics and social protocols.
3.3.2.1 Capabilities for Work Tasks
Work tasks require objects to work on. CSCW systems can support the following object types:
For the object types supported, the following capabilities may or may not be supported:
3.3.2.2 Capabilities for Transition Tasks
3.3.2.3 Capabilities for Social Protocols
abstract future summary receiver groupness implicit viewer
At the capability level, measures of the various times spent using objects and object manipulations, as well as the various capabilities for transition tasks and social protocols can be taken. Collisions in turn taking and questions about awareness can also be measured. Errors, wrong paths, etc. can be used as indicators of how well the capabilities work.
The service level provides a list of the different types of services that can be used in developing CSCW systems. In the future, this level will be expanded to include pointers to threshold values a given service should meet to provide adequate support for various capabilities. A list of basic features for the various services will also be included at this level.
Service level evaluations are of two types: comparisons to thresholds and feature sets and comparisons between services (of the same type or of different types). Service level measures such as quality of imported image, (readability, discernability, contrast), audio quality (clarity, echo), video quality (contrast, color tone) are also taken and used to compare services.
All the various implementations of the services exist at the technology level along with a description of their performance. For example, a description of an image resolution application would include the transmission time under various network loads, the expected image quality, and the compression factor. Technology evaluations of performance and quality can be outside of a group work context. These measures can be compared to the threshold measures at the service level to ensure that this implementation meets or exceeds those values.
User interface components and the elements needed to integrate technology building blocks into a unified CSCW system also exist at this level. Usability evaluations can be done at the technology level. However, these evaluations will need to be conducted within group work tasks. That is, users can be given tasks to perform using a specific implementation of a service (i.e., send an e-mail message to another collaborator). The times, keystrokes, errors made by users in performing this task are factored into the usability of the system.
The system developer must supply the technology level descriptions. A template/ checklist can be provided at this level to facilitate this on a service by service basis.
The collaborative framework is intended to benefit both researchers and developers of collaborative systems as well as purchasers and users of collaborative systems. The framework is useful in identifying scenarios for a group to use in evaluating a system or in comparing several systems.
The framework can also serve as a collective memory for the CSCW community. Therefore, over time, it is expected that guidelines will appear at the various levels of the framework. These guidelines will suggest capabilities for work tasks and transition tasks, services best suited to certain capabilities, and the appropriate usability, performance, and features levels needed for specific services.
