CIF Checklist

Title Page

o

Report identified as a Common Industry Format (CIF) document

 

o

Name of product and version that was tested

 

o

Who led the test

 

o

When the test was conducted

 

o

The date the report was prepared

 

o

Who prepared the report

 

o

Contact information for questions about the test

Executive Summary

o

The identity and a description of the product

 

o

[Recommended] The reason and nature of the test

 

o

A summary of the method of the test

 

o

Results expressed as mean scores.

 

o

If differences between values or products are claimed, the probability that the difference did not occur by chance.

Full Product Description

o

[Recommended] A brief description of the environment in which it should be used.

 

o

The user population for which the product is intended

 

o

[Recommended] Any groups with special needs

 

o

[Recommended] The type of user work that is supported by the product

Test Objectives

o

All of the objectives for the test

 

o

If the tested product is a subset of the total product, the reason for focusing on the subset.

 

o

The functions and components of the product with which the user directly and indirectly interacted in this test.


 

Participants

o

Number of participants tested

 

o

The segmentation of user groups tested

 

o

The key characteristics and capabilities expected of the user groups being evaluated.

 

o

How participants were selected

 

o

Whether they had the essential characteristics and capabilities.

 

o

[Recommended] Whether the participant sample included representatives of groups with special needs

Tasks

o

The task scenarios for testing.

 

o

Why these tasks were selected

 

o

Describe the source of these tasks

 

o

Any task data given to the participants

 

o

Any completion or performance criteria established for each task.

Test Facility

o

Physical description

 

o

[Recommended] Setting

 

o

[Recommended] Factors that might affect test results

Participant's Computing Environment

o

Hardware model

 

o

The operating system versions

 

o

Any required libraries or settings

 

o

The make and version of web browser, if relevant

Display Devices [Recommended]

o

Screen size

 

o

Monitor resolution

 

o

Media size

 

o

Print resolution

 

o

Specify the size(s) used in the test.

Audio Devices [Recommended]

o

Relevant settings or values for the audio bits, volume, etc.

Manual Input Devices [Recommended]

o

Specify the make and model of devices used in the test.

Test Administrator Tools [Recommended]

o

Any hardware or software used to control the test or to record data

 

o

Standardized questionnaires

Procedure

o

[Recommended] The sequence of events, non-disclosure agreements

 

o

The details on form completion

 

o

Details on warm-ups

 

o

Details on pre-task training

 

o

[Recommended] Verification that the participants knew and understood their rights as human subjects.

 

o

[Recommended] The steps that the evaluation team followed to execute the test sessions and record data.

 

o

Operational definitions of measures and any presented

 

o

Independent variables or control variables.

 

o

Time limits on tasks

 

o

Any policies and procedures for training,

 

o

Any policies and procedures for coaching

 

o

Any policies and procedures for assistance,

 

o

Any policies and procedures for interventions

 

o

Any policies and procedures for responding to questions.

 

o

[Recommended] How many people interacted with the participant during the test sessions

Participant General Instructions

o

All instructions given to the participants

 

o

[Recommended] How participants were to interact with any other persons present

Participant Task Instructions

o

Summarize task instructions

 

o

Exact task instructions in report appendix


 

Effectiveness Metrics

o

The unassisted completion rate

 

o

The assisted completion rate

 

o

Errors

 

o

Assists

Efficiency Metrics

o

Task time

 

o

[Recommended] (Completion rate / mean time-on-task)

Acceptance metrics [Recommended]

o

Subjective satisfaction assessment

Data Scoring [Recommended]

o

The method by which the data collected were scored

 

o

How outliers were excluded

 

o

How error data was categorized

 

o

The criteria for scoring assisted or unassisted completion

Data Analysis [Recommended]

o

The method by which the data were analyzed

 

o

The statistical procedures

 

o

The statistical significance tests of differences between groups

 

o

The results of the significance tests

Data Reduction [Recommended]

o

The method by which the data were reduced

 

o

How data were collapsed across tasks or task categories

Graphical Format

o

Graphical description of key results