Common Industry Format

DiaryMate is a computer version of a paper diary and address book. DiaryMate provides diary, contact and meetings management facilities for individuals and work groups. The test demonstrated the usability of DiaryMate installation, calendar and address book tasks for secretaries and managers.

Eight managers were provided with the distribution disk and user manual, and asked to install the product. Having spent some time familiarizing themselves with it, they were asked to add information for a new contact, and to schedule a meeting.

All participants installed the product successfully in a mean time of 5.6 minutes (although a minor subcomponent was missing from one installation). All participants successfully added the new contact information. The mean time to complete the task was 4.3 minutes.

Seven of the eight participants successfully scheduled a meeting in a mean time of 4.5 minutes. The overall performance data are summarized in the following table.

Combined Performance Results

Participant #	Unassisted Completion Rate (%) (all tasks)	Mean Goal achievement(%)	Total Task Time (min)	Completion Rate / Total Task Time	Total References to manual
1	67%	67%	9.7	7%	4.0
2	100%	98%	11.6	9%	2.0
3	100%	92%	16.4	6%	2.0
4	100%	93%	18.5	5%	4.0
5	100%	97%	11.7	9%	1.0
6	100%	87%	16.5	6%	1.0
7	100%	90%	13.7	7%	1.0
8	100%	93%	12.3	8%	2.0
Mean (#2-7)	100%	93%	14.4	7%	1.9
Standard error	0.0	1.5	1.0	0.5	0.4
Std Deviation	0.0	3.9	2.7	1.3	1.1
Min (#2-7)	100%	87%	11.6	5%	1.0
Max (#2-7)	100%	98%	18.5	9%	4.0

The overall score on the SUMI satisfaction questionnaire was 51. The target value of 50 (the industry average SUMI score) was within the 95% confidence limits for all scales.

Introduction

Full Product Description

DiaryMate is a computer version of a paper diary and address book. DiaryMate provides diary, contact and meetings management facilities for individuals and work groups. It is a commercial product which includes online help and a 50 page user manual.

The primary user group for DiaryMate is office workers, typically lower and middle level managers. DiaryMate requires Microsoft Windows 3 or higher, and is intended for users who have a basic knowledge of Windows. A full technical specification is provided on the SuperSoft web site: www.supersoft.com/diarymate.

Test Objectives

The aim of the evaluation was to validate the usability of the calendar and address book functions, which are the major features of DiaryMate. Representative users were asked to complete typical tasks, and measures were taken of effectiveness, efficiency and satisfaction.

It was expected that installation would take less than 10 minutes, and that all users could successfully fill in contact information in an average time of less than 5 minutes. All SUMI scores should be above the industry average of 50.

Method

Participants

Intended context of use: The key characteristics and capabilities expected of DiaryMate users are:

· Familiarity with a PC and a basic working knowledge of Microsoft Windows

· A command of the English language

· Familiarity with office tasks

· At least 10 minutes a day spent on tasks related to diary and contact information

Other characteristics of users which it is expected could influence the usability of DiaryMate are:

· amount of experience with Microsoft Windows

· amount of experience with any other diary applications

· attitude towards use of computer applications to support diary tasks

· job function and length of time in current job

Context used for the test: Eight junior or middle managers were selected who had the key characteristics and capabilities, but no previous experience of DiaryMate. The other characteristics of the participants that might influence usability were recorded, together with the age group and gender.

	Job	Time in job (years)	Windows experience(years)	Computer diary experience (years)	Attitude to computer diaries (1-7)^*	Gender	Age group
1	middle manager	5.5	3.5	0	6	F	20-35
2	junior manager	0.8	2.1	0.8	1	F	20-35
3	middle manager	2.1	2.5	2.1	3	M	20-35
4	junior manager	4.9	3.5	1.5	2	F	36-50
5	middle manager	0.7	0.7	0.7	2	M	20-35
6	junior manager	1.6	2.1	0	3	F	36-50
7	middle manager	4.3	1.4	0	4	M	36-50
8	junior manager	2.7	4.6	2.7	4	M	20-35

^*1=prefer to use a computer as much as possible, 7=prefer to use a computer as little as possible

Context of Product Use in the Test

Tasks

Intended context of use: Interviews with potential users suggested that installing the software was an important task. Having gained familiarity with the application, other key tasks would be adding information for a new contact, and scheduling a meeting.

Context used for the test: The tasks selected for the evaluation were:

1. The participant will be presented with a copy of the application on a disk together with the documentation and will be asked to perform the installation.

2. Following this each user will restart the program and spend some time familiarizing themselves with the diary and address book functions.

3. Each participant will then be asked to add details of a new contact using information supplied.

4. Each participant will then be asked to schedule a meeting using the diary facility.

Test Facility

Intended context of use: office environment.

Context used for the test: The evaluation was carried out in our usability laboratory in Hayden. The test room was configured to represent a closed office with a desk, chair and other office fittings. Participants worked alone without any interruptions, and were observed through a one way mirror, and by video cameras and a remote screen

Participant’s Computing Environment

Intended context of use: DiaryMate is intended for use on any pentium-based PC running Windows, with at least 8MB free memory.

Context used for the test: The PC used was a Netex PC-560/1 (Pentium 60, 32MB RAM) in standard configuration, with a Netex pro mouse and a 17" color monitor at 800x600 resolution. The operating system was Windows 95.

Test Administrator Tools

Tasks were timed using Hanks Usability Logger. Sessions were videotaped (a combined picture of the screen and a view of the participant), although information derived from the videotapes does not form part of this report. At the end of the sessions, participants completed a subjective ratings scale and the SUMI satisfaction questionnaire. SUMI scores have a mean of 50 and standard deviation is 10 (based on a standardization sample of 200 office-type systems tested in Europe and USA - for more information, see www.ucc.ie/hfrg/questionnaires/sumi/index.html).

Experimental Design

Eight junior and middle managers were tested.

The mean completion rate, mean goal achievement, mean task time, mean completion rate efficiency and mean goal achievement efficiency was calculated for three tasks:

· Install the product

· Add information for a new contact

· Schedule a meeting

Procedure

On arrival, participants were informed that the usability of DiaryMate was being tested, to find out whether it met the needs of users such as themselves. They were told that it was not a test of their abilities. Participants were shown the evaluation suite, including the control room, and informed that their interaction would be recorded. They were asked to sign a release form. They were then asked to confirm the information they had provided about themselves before participating: Job description, Time in job (years), Windows experience (years), Computer diary experience (years), and Age group. They also scored their attitude towards use of computer applications to support diary and contact management tasks, on a scale of 1 to 7, with anchors: prefer to use a computer as much as possible, prefer to use a computer as little as possible.

Participants were given introductory instructions. The evaluator reset the state of the computer before each task, and provided instructions for the next task. Participants were told the time allocated for each task, and asked to inform the evaluator (by telephone) when they had completed each task. Participants were told that no external assistance could be provided.

After the last task, participants were asked to complete a subjective ratings scale and the SUMI questionnaire.

The evaluator then asked them about any difficulties they had encountered (this information is not included in this report).

Finally they were given $75 for their participation.

Usability Metrics

Effectiveness

Completion Rate: Percentage of participants who completed each task correctly.

Mean goal achievement: Mean extent to which each task was completely and correctly achieved, scored as a percentage.

Errors: Errors were not measured.

Assists: The participants were given no assistance.

Efficiency

Task time: mean time taken to complete each task (for correctly completed tasks).

Completion rate efficiency: mean completion rate/mean task time.

Goal achievement efficiency: mean goal achievement/mean task time.

No of references to the manual: number of separate references made to the manual.

Satisfaction

Satisfaction was measured using a subjective ratings scale and the SUMI questionnaire, at the end of the session, giving scores for each participant’s perception of: overall satisfaction, efficiency, affect, controllability and learnability.

Results

Data Analysis

Data Scoring

Mean goal achievement

Mean extent to which each task was completely and correctly completed, scored as a percentage.

The business impact of potential diary and contact information errors was discussed with several potential customers, leading to the following scoring scheme for calculating mean goal achievement:

· Installation: all components successfully installed: 100%; for each necessary subcomponent omitted from the installation deduct 20%.

· New contact: all details entered correctly: 100%; for each missing item of information, deduct 50%; for each item of information in the wrong field, deduct 20%; for each typo deduct 5%.

· New meeting: all details entered correctly: 100%, incorrect time or date: 0%; for each item of information in the wrong field, deduct 20%; for each typo deduct 5%.

Combined deductions equaling or exceeding 100% would be as scored 0% goal achievement.

Data Reduction

In addition to data for each task, the combined results show the total task time and the mean results for effectiveness and efficiency metrics.

Participant #	Unassisted Task Completion Rate (%)	Goal Achievement (%)	Task Time (min)	Completion Rate / Task Time^*	References to manual
1	100%	100%	5.3	19%	1
2	100%	100%	3.9	26%	0
3	100%	100%	6.2	16%	1
4	100%	80%	9.5	11%	2
5	100%	100%	4.1	24%	0
6	100%	100%	5.9	17%	1
7	100%	100%	4.2	24%	0
8	100%	100%	5.5	18%	0
Mean	100%	98%	5.6

Executive Summary