Introduction
The DARPA Intelligent Collaboration and Visualization program (ICV)
has the goal of developing the generation after next collaboration middleware
and tools to enable military components and joint staff groups to
enhance the effectiveness of collaborating problem solvers through:
-
Gathering appropriate problem solvers together across time and space for
rapid response in time critical situations; and
-
Bringing appropriate information resources together across time and space
within the context of a task.
The ICV program has funded a number of proposals for the development
of collaborative technologies and has devoted a portion of the funds towards
the establishment of evaluation metrics, methodologies and tools. Since
the technologies developed are likely to be diverse, it seems appropriate
to review the specific objectives of the program and use them as a basis
for establishing the direction which the investigation of evaluation
methods might take. The ICV program objectives are:
-
Enable access via diverse portals, from hand-held through room-sized;
-
Enable interoperability across diverse encoding formats, coordination
and consistency protocols, and real-time services;
-
Scale collaborations to 10 active contributors, 100 questioners, and
1000 observers;
-
Reduce by an order of magnitude the time needed to generate collaborative
applications;
-
Enable real-time discovery of relevant collaborators and information
within task context;
-
Reduce by an order of magnitude the time to establish collaborative
sessions across heterogeneous environments;
-
Reduce by an order of magnitude the time to review collaborative
sessions; and
-
Improve task-based performance of collaborators by two orders of
magnitude.
The effectiveness of each of the collaborative technologies is to be
evaluated with respect to each of these objectives, when it is appropriate,
and so ways of measuring the extent to which the objectives have been
achieved must be developed. Objectives 1, 2 and 5 seem to require
qualitative measures, whereas the remaining goals are amenable to quantitative
measures. It is the task of the Evaluation Group to develop the metrics
and evaluation methodology, and to develop, or guide the development of,
specific tests and tools for achieving effective and economical
evaluation.
The Evaluation Working Group and its Aims
The Evaluation Group comprises researchers from Carnegie Mellon University,
MITRE, NIMA and NIST, with diverse backgrounds whose aims include defining
and validating methods that any researcher in the collaborative computing
research community can use to evaluate their own or other research products.
This objective is further refined into a set of goals as follows:
-
To develop, evaluate and validate metrics and methodology for
evaluating collaborative tools.
-
To provide reusable evaluation technology, such that research
groups can assess their own progress.
-
To apply DOD-relevant criteria, e.g., evaluate using scenarios
drawn from:
-
Planning/design/analysis domains.
-
C2 environment to capture planning/reaction/replanning
cycle. To provide evaluation methods that are cheap
relative to the requirements.
-
To define a vision that will drive collaborative computing research.
EWG/ICV Home