Submitted by Craig Thompson on August 06, 1997 at 15:56:45
EWG,
Very well written paper. I especially liked
sections 2 and 3 of the framework.
The usefulness of scenarios for evaluation was well
enunciated but it might be valuable to provide one
extended example, maybe in an appendix, showing a
scenario broken into capabilities, services, ... with
metrics on each.
As one who builds componentware services,
I can relate to the capability-service-technology
hierarchy though there might be more explanation that
the divisions need to be recursive though you say this
about tasks and subtasks. EG an application task can
turn into many application subtasks that turn into many
capabilities and subcapabilities that etc.
I still need to read the scenarios database via the
URLs you referenced. Getting some scenarios that
fit the various GENOA, JFACC, DMIF, ALP, ... problems
would be useful to all of ICV. It would be great if
the DARPA application programs used the scenarios
database to communicate with the technology
providers by providing descriptions of scenarios,
possibly breaking them into requirements, capabilities
and services though I am sure that is a lot to ask.
But methodology-wise it would certainly be nice. Too
long I lived in a big company where product groups
could not enunciate their technology needs and
researchers were left to try to guess these. The food
chain can get discouraging. You might add this use
of scenarios (as negotiation vehicles between
producer-consumer communities) to your list of uses.
WRT requirements, I have just been trying an exercise
in matching our OBJS technologies to a list generated by
Trish Carbone
a concrete example of using mappings of requirements
to consumer ISO application development programs and to
ITO technology provider programs. I liked the
rather simple methodology for creating a marketplace
of producers and consumers and it provided ways of
locating technology wholes. It was a little simpler
than scenarios-driven, being just requirements driven.
[One puzzle was where the requirements came from since,
for instance, Trish's requirements did not match the
eight IC&V objectives listed in your paper's Intro
(and on the ICV homepage too) nor did her first pass
mapping really reflect what my project is doing - so
I am sending her feedback since a downside of
creating these kinds of mappings is that they may be
used in making programmatic decisions and that is
potentially harmful if they are inaccurate or
incomplete.]
As you say, the bias in your paper is in providing an
evaluation methodology geared more towards the tasks
humans do in groups and less is said about the evaluation
of collaborative infrastructure frameworks. So a lot
IC&V technology (like some of ours) might not be able
to be evaluated directly via this paper's collaboration
framework.
You also point out that much less is known about
collaborations across time and distance. Evaluating this
form of collaboration seems critical since many
many collaborations are across time and space. Almost
all activities in our OBJS virtual office are
collaborative across time and space. Same time
collaborations are limited to phone calls and
relatively rare meetings. But we do things together
daily. I'd expect the same in DoD where teams
performing tasks are often distributed as is task
expertise.
Also you point out in the Intro that bringing
appropriate info resources together in time and space
within the context of a task is the other main
objective. But most of the paper seems most relevant
to the human collaboration with the exception of
your "Task type 9 (Non McGrath) Dissemination of
Information" tasks. Since a lot of IC&V projects
have something to do with this, this one task
seems to be where many will contribute.
In later sections
there were long lists of criteria and evaluative
measures but I kept wondering whether those lists were
complete by any measure. I began to be overwhelmed
by the number of things that could be measured or
evaluated and the issue became what to measure. Some
words on tailoring the methodology might be useful.
I'm surprised there is not much
more literature than you site on this subject since
scenarios seem to be a widely used technique for
evaluating systems. We used them as a principal
vehicle in evaluating the NIIIP Consortium's
virtual enterprise technology.
Again, I found the paper very valuable. It was very
well written and well organized. And it provided
what appears to be (so far untested) a useful
evaluation framework.
Craig Thompson, 972-379-3320
PS I'd be interested in knowing a little more about
MITRE's CVW to see if it would be useful inside OBJS'
virtual office. One thing we can add to IC&V is a
testbed of tools that are supposed to be useful in
collaboration at a distance. Since that is how we
do all our work, we are always interested in trying
new tools and will give informal untrained usability
field feedback on how useful the tool seems.
PPS It is awkward to type long comments into little
windows. Maybe I should have said less.
Some specific comments:
3.3.1 in the task list, I
wasn't sure where you covered the case
of hierarchical decomposition of groups by task
hierarchy. EG, the org chart is a persistent form
of this, or any division of work divide-and-conquer
strategy, one of the most pervasive in collaboration.
Also, a lot of the tasks seems like they happened in
meetings where all participants were present. Most
of my collaborations are at a distance.
3.3.1 Sometimes I wondered if Transition Tasks were
also work tasks, that is, if the task categories
could overlap.
3.3.1.2 what about transition tasks that should have
occurred but did not, like the failure to prepare
an agenda or take minutes at a meeting.
3.3.1.4 also cost of a collaboration. Kinkos charges
quite a bit per hour for videoconferencing but its
still cheaper to interview that way in many
virtual office situations.
3.3.2.1 There seem to be an open ended additions to
some of these lists. Collecting, Fixing, Buying ...
The MITRE Multi-Model Logger - seemed like this should
be in a separate section or appendix on tools for
evaluation of collaboration. Is this the only one?
Aren't there other research tools at least that would
be useful. Why limit the toolkit to those from the
IC&V community?
Some analysis of the OBJS virtual office is at
http://www.objs.com/survey/survey.htm#Categories.
We use some of that to construct collaboration
scenarios for the services we are building. Also
we are starting to try to construct some DoD doctrine
scenarios. Maybe all this will fit your Scenarios
database?
Hope the above comments are useful.