Section 3
A Framework for Collaborative Systems
The framework outlined in this section provides a structured way of thinking about collaborative systems and the evaluation of those systems. The framework can aid the researcher in making some preliminary judgments about a system or its usefulness to a particular group; for example, the framework can identify systems that are likely or unlikely to support a group’s work, thus narrowing down the number of systems to be further investigated using the scenario-based portion of the methodology. The framework can also aid in choosing scenarios for a scenario-based evaluation of a system.
This framework builds on one devised by Pinsonneault and Kraemer (1989) to analyze the impact of technology on group process while controlling for the effect of other contextual variables. We have merged the work of McGrath (1984) into our expanded framework to enable us to classify tasks that groups perform.
This section includes an overview of the framework and a description of how the framework can be used to describe and evaluate CSCW systems. We also provide detailed descriptions of each level of the framework.
3.2 Overview of the Collaborative Framework
The framework is divided into four levels: requirement, capability, service, and technology.
The requirement level of the collaborative framework is composed of requirements generated from the tasks being performed by the group and the support necessitated by the characteristics of the group. Requirements for supporting different types of groups include support for the social interactions of the group as well as the requirements due to group size, location, computer platforms, etc. The requirement level includes both work tasks and transition tasks (described below).
The capability level of the framework describes functionality that is needed to support the different requirements. The functionality described in capabilities can be obtained from different services. For example, the capability to synchronously communicate with another meeting participant during an electronic meeting could be accomplished by a text chat service or by telephone service. Certain capabilities may be necessary or recommended to support work and transition tasks, social protocols, and group characteristics in the requirement level.
The service level describes services such as e-mail, audio, video, application sharing, and networking services, that can be used to support the capabilities needed in CSCW systems. Different types of services can be used to provide the same capabilities that support specific requirements. >For example, a need for synchronous, non-collocated communications could be satisfied by text chat or video teleconferences. Comparisons and tradeoffs of performance and cost can be made at this level.
The technology level describes specific implementations of services. This level could be considered to be the set of possible components needed to build a given CSCW system, as well as their integration and interfaces. For example, different e-mail systems would exist at this level, as would the numerous ways to implement floor control, the various algorithms to control documentation locking and requesting, and the various networking services such as ATM. Specific implementations can be compared with respect to performance, cost, functionality, and usability.
|
|
|
3.3 Using the Framework to Begin Evaluating a CSCW System
3.3.1 General Evaluation Approaches Using the Framework
The framework can be used for evaluation in a top-down (requirement to technology level) approach, a bottom-up (technology to requirement level) approach, or a “middle out” approach.
Depending on the approach taken, the framework may, for example, help an evaluator select a subset of the evaluation tools a group needs to chose from; systems that do not meet or exceed acceptable levels for the measures available at a given level can be eliminated from further consideration. Or the framework may help a researcher determine whether a particular system could support a particular group adequately, or understand whether and how to implement missing functionality in a system.
A top-down approach allows evaluators to match requirements of the group to the tools needed to support collaborative efforts. To perform a top-down evaluation, users would begin with a top-level requirement such as “we need to work together even though half our group is in Toronto and half is in Amsterdam.”The users would then decide which capabilities they need to support their group work, such as synchronous non-collocated meeting support and asynchronous document transfer. They would then look at services to provide these two types of capabilities, such as text chat and email. Finally, the users would evaluate at the technology level whether systems such as MITRE’s CVW or America On Line (AOL) Instant Messenger would provide the necessary text chat performance and features, and similarly evaluate systems such as Netscape Mail and Eudora for email needs.
A bottom-up approach allows evaluators to determine the types of collaboration requirements that a given system can support best. For example, a researcher may need to investigate a newly developed tool—which exists at the technology level—to find out whether it answers the need it was developed to meet, and whether it answers other requirements as well. The researcher would move up through the collaborative framework, first deciding what services the tool provides, then abstracting to the more general capabilities, and finally determining the top-level requirements that the tool satisfies.
A “middle-out” evaluation begins either at the capabilities or services level. Such an evaluation could be used whenever there are questions regarding whether missing functionality should be included in a system, or how some capabilities could be used. For example, researchers who would like to know the effectiveness of incorporating a new pointing feature in a shared whiteboard might use a middle-out evaluation. The “middle-up” portion of the evaluation would involve determining what requirements such a new feature would satisfy. The “middle-down” portion of the evaluation would result in guidance for how to implement such a feature at the detailed technology level.
3.3.2 Using the Framework at Each Level
At the requirement level we can evaluate how well a CSCW system supports the work tasks, the transition tasks, the social protocols, and the characteristics of a specific group in general. >We can also evaluate the artifacts produced as well as the process the group used to produce an artifact. The detailed framework description (Section 3.4) describes the measures for task outcome for each task type. It is important to note that the variables being measured differ depending on the type of task the group is doing. For example, different measures are needed to evaluate the outcome of a brainstorming task than to evaluate the outcome of a planning task. (See Section 5 for a more thorough discussion of measures.)
At the capability level, researchers and potential users of a system may evaluate the appropriateness of specific capabilities to support the work tasks, transition tasks, social protocols, and characteristics of the group. To obtain answers to these questions, measures such as time on task, awareness questions, amount of setup time for equipment and configuration are used.
At the service level, framework users will examine the functionality of various types of services to understand how a given capability would be supported using a particular type of service. This allows users to compare services in order to determine which service seems most appropriate for the requirements. Performance thresholds, including robustness, can be examined at this level. Tests such as the ability to view an image with a certain amount of discrimination and the acceptability of audio or video can be conducted at this level.
At the technology level, specific implementations exist. Therefore, actual usability measures can be examined for an implementation. The implementations of the services can be evaluated to make sure they meet threshold values. Performance comparisons can be made at this level between different implementations.
3.4 Detailed Framework Description
3.4.1 Requirement Level
The following requirement level description is based on its contents: work tasks, transition tasks, social protocols, and group characteristics.
3.4.1.1 Work Tasks
Work tasks are the heart of a collaboration—the work that people need to do to meet their collaborative goals.
Work tasks include activities such as solving a problem, developing a plan, disseminating information, negotiating, and reaching consensus.
3.4.1.2 Transition Tasks
Transition tasks are tasks used to move between work tasks. They may include summarizing the outcome of the last task, assigning action items to members of the group, and noting dates for expected completion of assignments. The beginning and ending of any group collaboration involve transition tasks such as taking roll, requesting changes to an agenda, and locating missing meeting participants. Transition tasks also apply to asynchronous collaborations. A group member may suggest that the e-mail discussions about a particular subject end and volunteer to summarize the discussion and distribute this to group members; or a new person may join the meeting and need to get “caught up.”
A transition task may occur formally or informally, depending on the social protocol that the group is using. Transitions to the next work task occur formally if the chair of the group either moves the group to the next agenda item or the group votes to move to the next item. Informal transitions to the next work task occur when the group moves the discussion rather naturally to another topic or starts another group activity.
3.4.1.3 Social Protocols
Social protocols define the ways in which collaborative sessions are conducted. Collaborative sessions may vary from informal sessions to very formal sessions with a chair, an agenda that is strictly followed, and rules of order. In the context of meeting support, for example, social protocols support role management, floor control, and other basic meeting conduct activities. Table 1 lists example parameters that social protocol requirements support for meetings.
Table 1 Example Parameters for Social
Protocols During Meetings
|
Meeting
Component |
|
| Chair | None, loose control, tight control |
| Agenda | None, modifiable, non-modifiable (strict) |
| Rules of order | Used, not used |
| Titles | Yes, no, anonymous |
| Floor control | Dictated by agenda, directed by chair, informal turn-taking, free-for-all |
| Hierarchy support | Voting, contributing-restricted, contributing-free access, observing only |
| Communication support | Private or public, 1-way or n-way |
| Security | From none to highly classified (e.g., top secret special compartmented information) |
Social protocols may also support awareness of other group members’ presence, activities, locations, temporality, and motivations. There are several different ways to organize awareness components; the approach used by Villegas and Williams (1997) is used in Table 2.
Table 2 Example Awareness
Components and Questions for Social Control
Awareness
Component Presence: Who? Who else is in the workspace? Can the user tell who
else is logged into the session? Can the user tell
whether anyone else is working on the collaborative task? Can the user tell the
identity of other people working on the collaborative task? Actions: What? What are other participants doing? Can the user tell what
tasks the other participants are working on? Can the user tell what
tools or objects the other participants are using or manipulating? Can the user tell what
changes the other participants are making to objects in the shared workspace? Can the user tell what
changes he/she and others are allowed to make? Can the user tell what
the relative activity levels of the other participants are? Can the user tell
whether the other participants are willing to be interrupted? Location: Where? Where are the other participants working? Can the user tell
where in the shared workspace the other participants are working? Can the user tell what
the other participants can see? Can the user tell
where the other participants are focused? Time: When? When do changes made by the other participants take place? Can changes made by
the other participants be shown to the user in real time? Can past elements be
replayed? Can the user find out
when a particular past event happened? Motivation/Intention:
Why? Why are the other participants doing what they are doing? Can the user tell what
the other participants’ immediate intentions are? Can the user tell what
the other participants’ goals are? Group characteristics are attributes that determine how a group can work together.
Groups have different requirements depending on the makeup of the group, the social
relations (peer-to-peer versus boss-employee), formality, the location of the group
members, and the time requirements for collaborative sessions. Examples of these
characteristics are included in Table 3. In addition to considering the current
dimensions of the group, system requirements should also take into account anticipated
changes in these dimensions. For example, all members of a task force might start by
being collocated but with the knowledge that, in two weeks, half of the group will be
remotely located. Group characteristics affect how all of the different types of tasks
are performed.
Table 3 Example Group
Characteristics
Category Characteristics Potential Values Group type Number of members Number Group location Same for all, various
locations Homogeneity Gender diversity,
peers only or multiple levels of corporate hierarchy, differences in computer
experience, cultural diversity Stage of development Newly formed to
well-established group Motivation of group
members Very low to very high Group’s time
constraints Duration of
collaboration sessions Number of hours to
number of days Synchronicity of
collaboration sessions Synchronous or
asynchronous Length of time over
which collaboration will take place Number of days to
indefinite Group’s Hardware, software
requirements Platforms, software
needed computer requirements Training expectations Walk-up-and-use to
formal classes Computer expertise of
group members Novice to expert The primary measures taken at the requirements level are time for task
completion, similarity of participants’ perceptions of the outcome, quality of
work produced, and satisfaction of users. The secondary measures include agreement
of participants about what will be done next, types of conflicts occurring in
turn-taking, and number of awareness breakdowns.
Collaborative capabilities provide a means of matching tasks with services.
This matching process must take into account how well the service supports the
capabilities and whether this level of support is acceptable given the high level
requirements.
The capability level can be divided into those capabilities that support the
different requirement drivers (i.e., work, transition, and social protocol). By
explicitly understanding which types of tasks a system supports well, potential
users can better weight an evaluation to choose the system that best supports
their highest priority types of tasks. Examples of the capabilities that support
work and transition tasks can be seen in Table 4.
Table 4. Examples of Capabilities that Support Work
and Transition Tasks Task Example Capabilities/Subcapabilities Work – Shared workspace –
Full access to all objects –
Restricted access –
Anonymous contributions – Communication –
Anonymous communication –
Side chats and private communication –
Message passing –
Message leaving –
N way communication –
1 way communication –
Gesturing, pointing, agreeing, disagreeing –
Feedback channel –
Private communication –
Secure communication –
Private workspace – Support for object types –
Object visualization –
Object manipulation –
Object management Transition – Collaboration coordination capabilities –
Summarization capabilities –
Playback facility –
Distribution of objects –
Translation of objects between modalities – Collaboration planning capabilities –
Agenda support –
Calendar support –
Meeting notification –
Voting – Locator capabilities –
Locate possible collaborators –
Locate group members –
Locate objects At the capability level, measures can be taken of the various times spent using
objects, broken down by modality of use. Times spent using capabilities for transition
tasks and social protocols can be noted as well. Collisions in turn taking and
questions about awareness can also be measured.
Part of the evaluation process could involve understanding whether competing systems
have the same set of capabilities.Table 5 shows a part of an example checklist that
includes a number of capabilities with corresponding boxes that are filled in if the
capability is provided, or provided partially, by the subject system, or left blank if
the system does not support that capability.Five sample systems are identified as “A”
through “E.”
Table 5. Sample Checklist of Collaborative Capabilities
The service level provides mechanisms to meet the user’s need for specific
capabilities. It includes different types of
services that can be used in developing CSCW systems. In the future, this level could
be expanded to include pointers to threshold values a given service should meet to
provide adequate support for various capabilities. A list of basic characteristics of
the various services could also be included at this level.
Examples of services include:
Service level evaluations are of two types: comparisons to thresholds and feature
sets and comparisons between services (of the same type or of different types).
Service level measures such as quality of imported image (readability, discernability,
contrast), audio quality (clarity, echo), and video quality (contrast, color tone)
could also be used to compare services.
A template/checklist can be provided to facilitate the research in determining
whether services are provided. An example of a partial checklist can be seen in Table 6
below. This table lists the services provided by sample systems, denoted system A
through E. Each cell in the table is filled in
if the service is present and left white if it is absent.
The technology level refers to the specific implementation of a system.
User interface components and the elements needed to integrate technology building
blocks into a unified CSCW system exist at this level. Usability evaluations are best
conducted at the technology level within the context of group work tasks, although
simple evaluations of performance and quality can be made outside of a group work
context. Performance and quality measures can be compared to threshold measures at the
service level to ensure that the implementation being evaluated meets or exceeds those
values.
Table 6. Sample Checklist of Collaborative Services
The types of tasks a group needs to accomplish can determine many of their specific
work and transition task requirements. A
listing of collaborative task types may be used as a starting point in analyzing a
group's requirements in a top-down evaluation.
In a bottom-up evaluation, systems can be analyzed to determine which tasks and
requirements they can support.
Our task descriptions are based mainly upon tasks described in McGrath’s group
research (1984). The tasks described by McGrath should be thought of as a continuum.
They are numbered from 1 to 8, with successive tasks related to each other. The tasks
are summarized in Table 7.
Note that a portion of McGrath task 6 is not included in Table 7 because it is not
applicable to collaborative work. >Slight differences in other tasks are also noted in
Table 7. In addition, we have added a task type
(task 9) for “information dissemination.” Information dissemination refers to
activities such as participating in classes or sharing news of corporate restructuring.
>Such tasks are not covered by the other McGrath tasks, and so the addition and
definition of a ninth task type was necessary to form a comprehensive collection of
current work practices. We refer to these nine types of tasks as “work tasks”
throughout the rest of this paper. Collectively, the work task descriptions and
examples should enable users of the methodology to identify the task types that reflect
the tasks performed by a particular group that is in need of collaborative computing
support.
In Table 7, each task type has suggested measures of task outcome. In addition, for
many task types, research has uncovered problems that may occur with groups performing
this type of task. In some instances, computer-mediated processes or other group
processes may be able to alleviate these problems. We have listed known problems so
that comparison measures can be made to see if there is any effect. Where there is
research about computer-mediated work or group interactions and the effect on the task
outcome, we have included it under the heading "known research." An example
of each task is also provided to help in understanding how the generic task maps to a
real world task. Under "suggested capabilities" are those capabilities that
the research suggests may be valuable in carrying out that particular generic task.
The collaborative framework is intended to benefit both researchers and developers
of collaborative systems as well as purchasers and users of collaborative systems. The
framework is useful in identifying scenarios for a group to use in evaluating a system
or in comparing several systems. Over time, we expect that guidelines will appear at
the various levels of the framework. These guidelines will suggest capabilities for
work tasks and transition tasks; services best suited to certain capabilities; and the
appropriate usability, performance, and features thresholds needed for specific
services.
Table 7. McGrath Tasks With Variations (“Work
Tasks”) Type Definition Specific measures Known Research Suggested Capabilities Example 1 Planning. Group members are given a goal and asked
to develop a written plan for carrying out the steps needed to reach that
goal. The written plan should include
alternative paths or actions. – Amount
of time per participant – Amount
of calendar time –
Practicality of plan (quality of task outcome) – Social
relations hinder task efforts – There can be a strong effect on the group
due to social influence and conformity – Groups often have trouble seeing
alternatives; tend to focus on only a few alternatives – Participation can be very unequal; this
increases as group size grows – Groups tend to avoid conflict and spend
more time on non-controversial issues.
Controversial issues tend to become personalized –
Calendar support – Text
object creation, editing, displaying, arranging, storing The group has to produce an
advertising campaign for a new account by the end of the month. They have a meeting to plan the different
tasks that each member will carry out, complete with time lines for doing so
and different coordination points. 2 Brainstorming and group
creativity. Members of a group are
given a particular topic area and asked to brainstorm on ideas. – Number
of ideas –
Originality of ideas –
Creativity of individuals is stifled by social influence of group –
Individuals are able to take advantage of creativity-enhancing forces in
group - social support, cross stimulation –
Anonymous communication –
Synchronous communication – N way
communication – Shared
workspace The group has a goal to raise
$200,000 to build a new community center.
They generate ideas for funding raising events, people to ask for
contributions and possibilities for loans or selling "shares" to
the community members to raise this money. 3 Intellective. The group is asked
to solve a problem for which there is a recognized solution. The group is asked to determine a concept,
given instances of the concept. – Number
of trials to solution –
Solution (quality of task outcome) – Errors –
Inferred strategies – Written
media is slower to arrive at a solution than voice media is. But voice media uses more messages than
written. – Audio
only does not differ significantly from face to face (and hence, probably
video) –
Interacting groups are almost always more accurate than their average member – Groups
seldom do as well as their best members – Shared
workspace –
Gesturing, pointing, agreeing, disagreeing – N way
communication – Private
group communications A logical reasoning problem such
as the cannibal and missionary problem.
Three cannibals and three
missionaries are on a riverbank. They
need to cross the river. There is a
rowboat that can hold only two people.
Only missionaries know how to row.
At no time can there be more cannibals than missionaries on either
side of the river. What is the
minimum number of trips that can be made to transport all six to the other
side? 4 Decision-making. Group members
are asked to develop consensus on issues that do not have correct answers. How far and in what direction
(if any), the group as a group and the individuals in the group shift. (Attitudes are measured before and after
group discussion.) – Groups may not use their collective
knowledge fully or efficiently – Some
members may have more influence than others; the influence may not be based
on competency – May
be pressure towards quick, rather than good, decisions – Diversity
of views and values may make reconciliation difficult – Shared
workspace – N way
communication – Side
chats The group must decide which of
three job candidates should be hired.
All candidates have the same degree and specialty type, but different
work experiences. The group must decide
which candidate to hire. 5 Cognitive conflict tasks.
Members of the group hold different viewpoints. The group is asked to make a series of decisions from available
information that is imperfectly correlated with criterion. –
Agreement among members –
Interpersonal trust – Changes
in member's views – Verbal
interactions can lead to clarification of why group members are consistently
using different policies. But if
policies are used inconsistently, this leads to a distrust of and a reduction
in understanding of the other. – Group
members may change policy to increase accuracy. – Shared
workspace – N way
communication The group is hiring a
designer. Three candidates are at the
top of the list. Each has a different
degree type. The group is divided about
the type of experience best suited to this position. The group is interdisciplinary and each
tends to favor hiring the candidate most closely aligned with their
discipline. 6 Mixed motive tasks. A range of tasks, differen-tiated by the
degree to which a group member's outcome is affected by a combination of his
own actions and the group's outcome.* Not applicable. Not applicable. *Note that McGrath also includes
dilemma tasks in this category.
However, since the dilemma decisions are made independently, no
collaboration occurs. Therefore, we
have not included dilemma tasks in our framework. Not applicable. Not applicable. 6A Negotiation task. The group is
divided into x subgroups with a negotiator elected for subgroup. The different subgroups disagree;
tradeoffs have to be made in multiple dimensions. It is not necessarily a zero-sum problem – Quality
of solution as evaluated by each subgroup – Time to
solution –
Attainment of solution (task completion) –
Evaluations of negotiators by group –
Interpersonal relations between group members –
Negotiators are more competitive when any of these conditions hold: – They
think constituents distrust them – They
were elected – They
are being observed – They
have a prior commitment – Their constituents belong to a highly
cohesive group –Negotiators
who do not belong to the group feel freer in the negotiation process but are
less supported by the group. – N way
communication – Group
private communication – Shared
workspace
Awareness: Top-Level and Subordinate
Questions
3.4.1.4 Group Characteristics
3.4.1.5 Requirements Level Measures
3.4.2 Capability Level
3.4.3 Service Level
3.4.4 Technology Level
3.5 Collaborative Tasks
3.6 Summary of Collaborative Framework