Title Page

o

Report identified as a Common Industry Format (CIF) document

o

Name of product and version that was tested

o

Who led the test

o

When the test was conducted

o

The date the report was prepared

o

Who prepared the report

o

Contact information for questions about the test

Executive Summary

o

The identity and a description of the product

o

[Recommended] The reason and nature of the test

o

A summary of the method of the test

o

Results expressed as mean scores.

o

If differences between values or products are claimed, the probability that the difference did not occur by chance.

Full Product Description

o

[Recommended] A brief description of the environment in which it should be used.

o

The user population for which the product is intended

o

[Recommended] Any groups with special needs

o

[Recommended] The type of user work that is supported by the product

Test Objectives

o

All of the objectives for the test

o

If the tested product is a subset of the total product, the reason for focusing on the subset.

o

The functions and components of the product with which the user directly and indirectly interacted in this test.

Participants

o

Number of participants tested

o

The segmentation of user groups tested

o

The key characteristics and capabilities expected of the user groups being evaluated.

o

How participants were selected

o

Whether they had the essential characteristics and capabilities.

o

[Recommended] Whether the participant sample included representatives of groups with special needs

Tasks

o

The task scenarios for testing.

o

Why these tasks were selected

o

Describe the source of these tasks

o

Any task data given to the participants

o

Any completion or performance criteria established for each task.

Test Facility

o

Physical description

o

[Recommended] Setting

o

[Recommended] Factors that might affect test results

Participant's Computing Environment

o

Hardware model

o

The operating system versions

o

Any required libraries or settings

o

The make and version of web browser, if relevant

Display Devices [Recommended]

o

Screen size

o

Monitor resolution

o

Media size

o

Print resolution

o

Specify the size(s) used in the test.

Audio Devices [Recommended]

o

Relevant settings or values for the audio bits, volume, etc.

Manual Input Devices [Recommended]

o

Specify the make and model of devices used in the test.

Test Administrator Tools [Recommended]

o

Any hardware or software used to control the test or to record data

o

Standardized questionnaires

Procedure

o

[Recommended] The sequence of events, non-disclosure agreements

o

The details on form completion

o

Details on warm-ups

o

Details on pre-task training

o

[Recommended] Verification that the participants knew and understood their rights as human subjects.

o

[Recommended] The steps that the evaluation team followed to execute the test sessions and record data.

o

Operational definitions of measures and any presented

o

Independent variables or control variables.

o

Time limits on tasks

o

Any policies and procedures for training,

o

Any policies and procedures for coaching

o

Any policies and procedures for assistance,

o

Any policies and procedures for interventions

o

Any policies and procedures for responding to questions.

o

[Recommended] How many people interacted with the participant during the test sessions

Participant General Instructions

o

All instructions given to the participants

o

[Recommended] How participants were to interact with any other persons present

Participant Task Instructions

o

Summarize task instructions

o

Exact task instructions in report appendix

Effectiveness Metrics

o

The unassisted completion rate

o

The assisted completion rate

o

Errors

o

Assists

Efficiency Metrics

o

Task time

o

[Recommended] (Completion rate / mean time-on-task)

Acceptance metrics [Recommended]

o

Subjective satisfaction assessment

Data Scoring [Recommended]

o

The method by which the data collected were scored

o

How outliers were excluded

o

How error data was categorized

o

The criteria for scoring assisted or unassisted completion

Data Analysis [Recommended]

o

The method by which the data were analyzed

o

The statistical procedures

o

The statistical significance tests of differences between groups

o

The results of the significance tests

Data Reduction [Recommended]

o

The method by which the data were reduced

o

How data were collapsed across tasks or task categories

Graphical Format

o

Graphical description of key results