The purpose of the report is to help purchasers decide if the product will be usable for their particular users, tasks and working environment. The design of the test should preferably be based on a simulation of how you expect your customers to use the product.
You should describe the test and the results in sufficient detail to enable purchasers to be able to judge the relevance of the results to the needs of their own users, tasks and working environments. The more closely the test simulates real world usage, the easier this will be. The more 'artificial' the test, the less useful it will be.
You should generally report the last usability test (or tests) carried out on the product. But some usability test procedures, which are effective for finding usability defects, are less useful as an overall report on the usability of the product. The following guidelines will help you check that your test procedure is as close to real world usage as possible:
1. Items to be included in the report format are either required or recommended (see the Checklist in Annex B). Elements that are recommended are marked as such; all other items are required implicitly.
2. The usability test report needs to make it clear which users, tasks and working environments the product is intended for, and the extent to which these characteristics were actually simulated in the usability test.
3. The task instructions should tell users what they need to achieve, without giving any clues about which product features to use.
4. To be representative of real world usage, the test situation should be as natural as possible. This may mean simulating distracters and other working conditions. The evaluators should be as unobtrusive as possible (preferably observing remotely in another room).
· During testing participants should not be asked to think aloud.
Thinking aloud may cause participants to behave differently, and makes task timing difficult to interpret.
· Participants should not be given any hints or assistance, other than by mechanisms available to real users (such as documentation or a telephone help desk).
If assists are given by the observer, it is difficult to know how usable the product would be if no assists were available.
5. Data should be obtained from sufficient users in each category for the sample of users to be representative of the intended user group. Given the typical variability of participants in a usability test, it has been found that for consistent results it is best to test at least 8 participants from each user group. Data from less than 8 participants may still give a good indication of usability, but the results will not be as reliable.
6. It should be possible for the recorded measures to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of equal value. For example, using an unweighted count of errors is usually inappropriate as the impact of errors usually differs.
7. There are three basic usability-related measures which are useful to the purchaser:
· Can users complete their tasks satisfactorily? [Effectiveness]
· How long do users take? [Efficiency]
· Are users satisfied? [Satisfaction]
The approach taken to reporting usability in the test report format is consistent with ISO 9241-11 [5]. If possible use industry standard measures of satisfaction. It is also useful to check whether major customers have preferred metrics for usability.
Although it may not always be possible to implement all of the ‘Recommended’ guidelines, the utility of the report will be enhanced by inclusion of as much information as possible.
The purpose of the report is to help purchasers decide if the product will be usable for their particular users, tasks and working environment. But some usability test procedures, which are effective for finding usability defects, are less useful as an overall report on the usability of the product. The more closely the test simulated real world usage, the more useful its results will be. The report format and checklist will help you decide how much reliance you can place on the results.
In addition you need to decide to what extent the participants, tasks and environment used for the test match your own users, tasks and environments. How big an impact could any differences have on the usability of the product in your context of use? In particular:
· How capable and experienced are your users?
· How much training will they have?
· Will they have the same task goals?
· Is their computer configuration as powerful?
· Where can they get assistance if they have problems?
[1] Brooke J (1996). SUS: A 'quick and dirty' usability scale. In: Usability Evaluation in Industry. Taylor and Francis. See http://www.redhatch.co.uk/sus.html
[2] Lewis, J.R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7, 57-78.
[3] Kirakowski, J. (1996). The software usability measurement inventory: background and usage. In: P. Jordan, B Thomas, & B Weerdmeester, Usability Evaluation in Industry. Taylor & Frances, UK. See also http://www.ucc.ie/hfrg/questionnaires/sumi/index.html
[4] Shneiderman, B. (1998). Designing the User Interface. Reading, MA, Addison-Wesley Publishing Co. See also: http://www.cs.umd.edu/hcil/quis/
[5] ISO 9241-11 (1998). Guidance on Usability. ISO (available in the USA from: American National Standards Institute, mailto: info@ansi.org, http://www.ansi.org/and outside USA from ISO member bodies: http://www.iso.ch/addresse/membodies.html)