Jack J. Yu, Prasad V. Prabhu, and Wayne C. Neale
jy@kodak.com,
pvprabhu@kodak.com,
neale@kodak.com
Eastman Kodak Company, Rochester, NY, USA
Abstract
In January 1998, Kodak introduced a new top-level structure and visual design for its Web site. This paper describes the user-centered approach utilized in the design process. We discovered that combining the knowledge gained from a variety of data collection methods was critical to understanding and defining Web site user requirements. We also found an on-line preview and survey to be a useful tool for assessing user acceptance of new designs. A sampling of results is provided to illustrate the process we used and to discuss its effectiveness.
Background
Like many corporate sites, the Kodak Web site (www.Kodak.com) is very large and diverse. With respect to quantity of resources, a recent file count revealed that Kodak.com contains over 25,000 pages and 74,000 files. With respect to traffic, Kodak.com averages around one million hits daily, including roughly a quarter of a million page views and accesses by some 24,000 unique hosts each day. In addition, Kodak.com serves a very diverse audience. Besides serving a broad consumer market for photographic products, Kodak develops, manufactures and delivers products and services for a variety of business, commercial, and work- related applications, and the content served on Kodak.com is appropriately varied.
Kodak.com maintained essentially the same home page (FIG. 1) and top-level site structure (see FIG. 2 for an example of a top-level page) from late 1995 to late 1997. This two-year period was one of explosive growth for the site. Kodak.com went from several thousands of pages to several tens of thousands. The nature of the content and the business objectives of the site evolved. New types of content and functionality (such as the on-line store and "Picture This" Web postcards) were added to the site. However, despite this tremendous growth, the top-level structure remained virtually unchanged. The objective of the design effort described in this paper was to revisit the design of the top-level site structure of Kodak.com, and modify it to address the tremendous growth of Kodak.com, focusing on the needs of the user and the usability of the site structure for information retrieval.
In order to achieve this task, we sought first to analyze and understand the needs and desires of Kodak.com users, as well as user behavior on our current site structure. We used our understanding of user needs and behavior to develop a set of user requirements, which we combined with Kodak's business objectives for the Web to guide the design of a new top-level structure for Kodak.com. We then subjected our design to several iterations of user testing and refinement to arrive at a finished new top-level structure for Kodak.com.
Understanding User Needs and Behavior
Before attempting to design a new top-level site structure for Kodak.com, we wanted to learn as much as possible about our users and their behavior on our existing top-level site structure. Analyzing the user population of a major corporate Web site is difficult, especially when the business of the corporation (and hence the audience for its Web site) is greatly varied. We examined information from a number of different sources to learn about the users of Kodak.com and to gather user requirements: Web server log files, search engine log files, comments left in the Kodak.com "guest book" (on-line feedback mechanism), feedback from focus groups, results from on-line user surveys, and results from previous usability tests. In addition, we interviewed representatives from several business units within Kodak that serve specific professional and commercial markets (such as medical imaging and filmmaking) to better understand the special needs of users within their markets.
Web Server Log Files
First, we analyzed traffic on our site as indicated by our Web server log files. Through Web server log file analysis we were able to identify the most frequently visited areas of our site, as well as the least frequently visited areas of our site, and the relative popularity of each. We were careful in the conclusions we drew based on Web server log files alone, recognizing that they indicate which pages people are accessing but neither their motivation for accessing them nor the value (if any) they gain from doing so [2]. Nonetheless, Web server log files were invaluable for examining general trends of user behavior and topics of interest on Kodak.com. Whenever possible, we complemented the knowledge we gained from our server log file analysis with knowledge gained from other sources (such as usability tests) to paint a more complete picture of the user.
As a specific example, the previous structure of Kodak.com included a link to "Kodak Online Services" from the top-level "Customer Support" page. This page contained basic information on how to access Kodak information through various Internet protocols and service providers (e.g., FTP, America Online, Compuserve). Although we learned from server log file analysis that this page was a highly visited page, we also learned from usability testing that many users mistakenly followed the "Kodak Online Services" link expecting to receive on-line service from Kodak, as from a live technical support representative. Thus, we concluded that the many hits received by this page likely contained a large proportion of mistaken hits, and that the server log files provided an inflated view of the popularity of the page. On the other hand, the popularity of "Picture This" multimedia postcards on Kodak.com, which received a tremendous amount of traffic, was corroborated by guest book comments and search log file analysis.
In addition, we employed a special analysis of server logs called "Click Balance". Click balance analysis looks at incoming and outgoing traffic for a given Web page (i.e., the pages visited prior to and following the visit to the page in question). Click balance analysis was useful for gaining a more complete understanding of user behavior for a specific page. This was knowledge that could not be gained simply by counting hits, since many pages on our site are linked from multiple areas, and user behaviors as bookmarking, searching, and accessing specific URLs effectively bypass the navigational structure of the site.
Search Log Files
Knowing that some users go to a search mechanism before attempting to navigate to information in any other way [3], we nonetheless assumed that the most popular search keywords on Kodak.com would reveal something about topics that were of high interest to users but hard to find by navigation on the site. Analyzing search log data from several months revealed that the most common search keywords by far had to do with our "Picture This" postcards. Despite the incredible popularity of "Picture This" postcards, there was no obvious navigational route to them from the home page. Other popular search topics were employment opportunities and sample pictures, neither of which were easily accessible from the home page. When designing the new site structure, we placed a high priority on linking to these topics either directly from the home page or in a clear logical path from the home page.
"Guest Book" Comments
On average, about 160 entries per day are submitted through the Kodak.com "Guest Book" (a form through which Kodak.com users can send questions, requests, comments, etc. to the company). Typically, the vast majority of these entries are requests for product support, requests for additional information, comments about the company and/or its products, etc., and less than two percent contain feedback about the Web site itself. However, these entries are the only regular means by which actual Kodak.com users can provide direct feedback about the site. Therefore, we analyzed guest book comments from several months as part of our effort to understand user needs and desires.
We found that the majority of guest book feedback pertaining to the site itself was too generic to be useful (e.g., "Great site!"), and that most specific problems identified (e.g., broken links, misspellings, inaccuracies) tended not to be helpful in identifying usability problems. However, there were a small number of comments that inspired our thinking about opportunities to improve Kodak.com. For example, a few comments brought up the issue that the essentially static appearance of the existing home page did not facilitate announcements of new product launches or the availability of popular applications (like "Picture This" postcards); also, it gave no indication that new information was being added or that the site was being kept up-to-date. This helped motivate our desire to make the new home page more dynamic and flexible than the existing home page.
Focus Groups
Focus groups were conducted in three different U.S. cities with both consumers and dealers of Kodak products. Participants discussed expectations and desires of a Kodak Web site, reviewing Kodak.com screenshots and providing feedback, and responding to proposals for Web- based services and offers. The content most often requested in these sessions, product information and photography tips, corresponded with highly visited areas on Kodak.com. The comments made by participants helped guide our thinking in the initial stages of developing a new site structure for the site by making us aware of what many users expect from a Kodak Web site.
Results from On-Line User Surveys
The results from a number of user surveys conducted on Kodak.com yielded valuable information about user characteristics and preferences. We gained insights into the reasons users visit (e.g., business or personal reasons), the frequency with which users visit, connection speeds, monitor settings, browsing habits, demographics, etc. These insights were extremely valuable during the development of user requirements for the new site structure.
Results from Previous Usability Tests
We also utilized relevant results from prior usability tests of specific portions of Kodak.com to guide us in making design decisions for the new site structure. For example, one test had assessed the usability of our product information page, which offered six different ways to search or browse through the product catalog. The usability test indicated that a few of the alternatives were difficult for users to understand or use. Consequently, for the new site structure, we simplified the product information page, leaving out problematic search methods and giving more prominence to methods with which users had more success in the usability test.
Interviews with Business Unit Representatives
Eastman Kodak Company consists of several business (operating) units, which focus on a specific set of products and services for specific markets. Examples of such business units include Kodak Professional, a unit that serves professional photographers and members of the printing and publishing industries, and Health Imaging, which markets medical imaging solutions to health care professionals, including radiologists, cardiologists, dentists, technicians, and hospital administrators. Each business unit has content developers and managers who handle the business unit's presence and content on Kodak.com. We interviewed representatives from each business unit to better understand:
This exercise helped us to better understand some of the specific markets and audiences Kodak is trying to reach and some of the specific goals Kodak is trying to achieve through the Web. This knowledge was invaluable for us as we considered how best to link all the content on Kodak.com together through a top-level site structure.
Defining User Requirements
Based on our understanding of user needs and behavior, we defined a set of user requirements that guided the design of a new top-level site structure. Our primary user requirement was that the new site structure be usable for information seeking. Some of the specifics of this requirement are described below:
Designing a New Top-Level Site Structure
In order to arrive at a first draft of a new site structure for Kodak.com, we organized the content on Kodak.com into categories, guided by the requirements we gathered from both users and business units. We worked with a multidisciplinary group of usability engineers, visual designers, and content developers to refine the categories and arrive at a first draft of a site structure.
In hindsight, with more time and resources we would have added another iteration of user involvement to the design process by having users recruited from outside the company perform a card sorting exercise to help organize information into an initial site structure. Although the initial site structure was not based only on usability (business objectives were important, too), it would nonetheless have been valuable to gain some user feedback at this point.
In creating a first draft of a site structure, we tried to adhere to the principles of link differentiability and predictability emphasized by Spool, et al. [5]. We evaluated not only each link, but also the context of each link, (i.e., whether the link would be clear and differentiable from the links around it). We also strived to speak the user's language; to that end, we replaced such corporate-centric link names as "Business Imaging Systems" and "Professional Motion Imaging" with the more user-centric terms "Business and Office Applications" and "Motion Picture Imaging".
We also loosened our previous limitation on numbers of links on each page, since we knew we wanted to flatten and broaden our site somewhat. However, we wanted to be careful of the increased cognitive load placed on the user by an increased number of options offered at the home page level. The ten links on the previous home page were presented in a single group. For the new home page, we wanted to present more links, but in logical groups, or "chunks", so that the user would be able to scan through a small number of chunks, rather than reading through every link on the page.
Usability Testing and Iterative Refinement
Usability Testing, Phase I
After developing a first draft of a new site structure, we subjected it to paper-and-pencil usability testing. Twenty participants were recruited for the study, which consisted of three stages.
In the first stage, we employed a methodology similar to the "category identification" activity described by Fuccella [1]. Each participant was presented with a simple listing, on paper, of the links we intended to have on the home page (FIG. 3). Links were presented in the major groupings ("chunks") we had agreed upon, but no other visual design was employed. Each participant was given the same set of 30 tasks (presented in random order). Tasks were designed to cover a broad range of content on Kodak.com, including information most frequently sought by users. For each task, the participant was asked to identify the home page link most likely to lead to information that would support the completion of that task, and to give a rating from 1-3 of his/her level of confidence that the targeted information would be found using that link. This first stage was intended to examine how well our proposed home page links were differentiated; that is, for a given set of representative tasks, whether the user knew which link to choose to find a particular piece of information. Link differentiability is one of the key attributes of links mentioned by Spool, et al. as contributing to the user's success at finding information on a Web site [5]. Table 1 shows the percentage of participants, for each task, who picked as their first choice the correct home page link that would lead to the completion of the task. This was the most useful stage of the test, as it provided a preview of what it would be like for a user to use the new home page.
In the second stage, we employed a methodology similar to the "category description" activity described by Fuccella [1]. We asked each participant to go through the entire list of links on the home page and describe briefly what s/he would expect to find "behind" each link. This second stage was intended to examine the predictability of the links we intended to offer on the home page. Predictability is one of the other key attributes of links mentioned by Spool, et al. as contributing to the user's success at finding information on a Web site [5]. Table 2 shows the percentage of participants making correct predictions for each link. We found this stage of testing to be helpful in that it identified unfamiliar or ambiguous terms, and terms for which the users' expectations differed from our intent. It also provided insight into possible misinterpretations of terms, and how pervasive these misinterpretations were likely to be. For example, we discovered that 80% of our test participants were unfamiliar with the term "Advanced Photo System", and did not realize that the term refers to an industry standard for general consumer photography. As another example, we constructed a link to "What's New at www.Kodak.com" hoping that users would realize that it linked to recent updates to the Web site, not necessarily "what's new" with Kodak. However, we found that 75% of test participants did not make this distinction, and for the final design we dropped the words "at www.Kodak.com" from this link.
In the third stage, we again went through the entire list of links on the home page with each participant and explained the content we intended to offer under that link. We asked the participant to give a rating on a seven-point scale of how well the link name we chose described the content we intended to offer "behind" it. In this stage, we were attempting to discover how accurate the participants found the names we devised. We discovered that this stage of the study was not particularly useful, since participants tended to give us high accuracy ratings for almost every link. We found that the fact that because a link name is accurate does not necessarily mean that it is predictable, or well differentiated from surrounding links, or otherwise useful to the user. As an illustration, a link marked "Information" may accurately describe what is found behind the link, but is not of much use to the user because it is not at all predictable or differentiable from other links. A specific example from our study is that we had a link marked "All About Photography" on our prototype home page. Five of the 30 tasks could be completed using this link, but for four of these five tasks, none of the 20 participants chose "All About Photography" as either their first or second choice links. Yet when the participants were told the content that would be offered through "All About Photography" (tips for taking great pictures in various situations, suggestions on how to do more with pictures, information about photography clubs, seminars and events, information about Kodak photographic products, and more) all participants gave the name a high rating in terms of accuracy. The problem was that although the title "All About Photography" was accurate, it was not well differentiated from its surrounding links (many of which had to do with photography, as well). In addition, because of the great diversity of information to which it pointed, it was difficult for users to predict what lay behind the link. As a result of this and other experiences, we concluded that this methodology was not useful for measuring the usability of the proposed site structure, or for identifying changes that would increase usability.
The results of this study were used to drive changes to our top level site structure.
Creating a Working Prototype
In contrast to our previous home page, which featured a large image map, we decided to make the new home page primarily textual. There were multiple reasons for this. The first reason was speed. Much existing research on Web usability points to greater speed and shorter download times as being of prime importance to users [4]. Because text downloads faster than images, we chose to use textual links to enable users to navigate our site more quickly and without having to wait for images to download. The second reason was flexibility. Because the previous home page was made up of a single large image map, it was difficult to make modifications or additions. Since the new home page was mostly textual, it could easily be modified to reflect the nature of the site. We could add, change, or remove links without much effort whenever necessary to accommodate new or changing content, address usability issues, and make special announcements.
Usability Testing, Phase II
In our second stage of usability testing, we sought to validate the changes we had made to the top-level site structure and test the home page in actual usage situations. Thirty-three participants were recruited to complete ten information-seeking tasks each using a working prototype of the new site structure. A home page prototype (FIG. 4), complete with visual design elements, was used. Top-level pages were prototyped as simple textual listings and groupings of links, without visual design elements. All links were functional and pointed to the appropriate content on Kodak.com. As in Phase I usability testing, tasks were designed to cover a broad range of content on Kodak.com, but this time with emphasis on information popularly sought by users as identified by corporate customer support staff. Each participant was given three minutes to successfully complete each task. Table 3 shows the percentage of participants who were able to complete the tasks in the time given.
In addition to the task completion data, the Phase II usability test identified several specific potential improvements (based on feedback from test participants) to the home page to increase its usability, such as: labeling the major "chunks" of information, reducing the number of links, adding descriptors to more of the links, and removing the images along the left-hand side. These improvements were incorporated into the next and final iteration of the home page design.
Public Response to the New Design
As we approached the finalization of the new site structure and visual design, we took one more opportunity to gather user feedback. We conducted a preview and survey of Kodak.com users to gauge their reaction to the new home page. From a link in the footer of every page on the site, Kodak.com users were invited to preview an HTML prototype of the new home page and to answer a brief survey describing their reaction to it. Each link on the prototype home page pointed to a single page describing the content that would eventually appear under that link once the site was live. The reason we did not make the prototype a working prototype was that we wanted users to focus their comments on the new home page and top-level site structure only. We wanted to keep users from wandering throughout the site, and either not returning to the survey and from giving feedback based on their opinion of the site as a whole rather than of the new home page in particular. We also wanted to use an HTML prototype (rather than a screen shot) so that they could see the new home page rendered on their browser/platform combination of choice, with their own font and appearance settings, consequently having an accurate preview.
Survey respondents were asked to rate the new home page both for overall appeal and in comparison to the current home page. In addition, respondents were given the opportunity to make any open-ended comments they had about the new home page design. Information about the respondent was collected, as well (e.g., gender, reason for and frequency of visiting Kodak.com), in order to compare the ratings for different populations.
In retrospect, we should have also asked the respondents to rate how well the information was organized and how easy they thought the new home page would be to use, since usability for information retrieval was our primary design objective. We left such questions out because we wanted to keep the survey short and the questions easy to answer, and overall appeal and comparison with the existing home page seemed to lead to relatively easy questions that would still reflect respondents' general reaction to the new design. However, when the survey responses came in, we found that a high proportion of respondents actually volunteered in their open-ended comments that they thought the information was well organized and that the new home page and site structure would be easier to use than the existing home page and site structure.
A first version of the new home page was posted for two days, and 94 survey responses were received. Based on comments received about the first version, some modifications to the visual design of the home page were made, and a second version (FIG. 5) was posted. The second version was posted for 19 days, and 779 survey responses were received. For both versions, feedback was overwhelmingly positive. Ratings for the second version are summarized in Tables 4 and 5.
A chi-squared statistical analysis indicated no significant differences in the ratings based on gender, frequency of visiting (whether first- time or repeat), or reason for visiting (whether for consumer information, for business related information, or just for fun).
Of the 630 open-ended comments that were submitted, a large number specifically praised the visual design, organization and the apparent ease at which it would be possible to get to information. There were criticisms about the black background, density of links and information and "cluttered" appearance, but these were far outnumbered by the praises of the visual design and apparent usability of the new home page. The top praises and criticisms are summarized in Tables 6 and 7.
The perceived usability of the new site structure was praised in such comments as "It looks as though you've done your homework on what people are looking for on your site." "The new home page is easier to understand when looking for a specific subject." and "Should make it easier to find the desired information. Nice logical layout."
From the results of the preview and survey, we concluded that the new home page and site structure would be well received by Kodak.com users. Top-level pages were developed using similar design principles and visual design elements as the home page (see FIG. 6 for an example), and the new home page and top-level pages were installed on Kodak.com at the end of 1997.
Four months after the "launch" of the new top-level site structure, we have observed mostly positive effects. As before, very few guest book comments address the design of the site, but such comments have been largely positive (much to our surprise, very few have specifically addressed the new site design). Site traffic continues to be strong. Overall search engine usage has decreased, with a particularly drastic reduction to searches for "Picture This" postcards and employment opportunities (which we brought up to the home page in the new structure).
Conclusions and Future Directions
We found the methods used to gather user requirements effective in that they enabled us to design a reasonable first draft of a top-level site structure. As we developed the design from a listing of links on paper to a set of working Web pages, we engaged users through usability testing at two different points in the development process, which helped ensure that the evolving design was usable. The preview and survey of the new home page prior to launch confirmed that user response to the new site structure would likely be positive.
While it is difficult to specify metrics for whether or not a major change to a large corporate site has been successful, we intend to continue to monitor user behavior and response to our new site structure and draw the best inferences possible (we have observed mostly positive effects so far). Since the usability testing in this project centered on the home page only, future research is needed to test the usability of other top-level pages.
References