There has been relatively little research on the development of auditory interfaces for the Internet. For example the recently published "Human Factors and Web Development" (Forsythe, Grose, & Ratner, 1998) contains only two short paragraphs on the implementation of audio. Typically the use sound has been reserved as a solution for visually impaired users, rather than as an enhancement for the sighted population, but there are several emerging issues which suggest the potential utility for sound in interface design, particularly non-speech delivery.

Constraints on Space

Advances in web browser interfaces can have significant effects on the relative salience of different types of information. As better graphics and video products emerge, the "geography" of displays is being dominated by web content, potentially at the expensive of more mundane browser interface elements that provide useful information about background processes (download rate, connection status, etc.). Much like television, which browsers are arguably attempting to emulate, there is a paucity of dynamic, continuously available information about background processes, beyond the content itself. This is adequate for television’s highly reliable, serial channel delivery, but may be lacking for multi-window browsing, FTPing and similar multitasking. The relatively high instability of web browsing coupled with the relatively low bandwidth dedicated to monitoring those background processes, makes it difficult for users to diagnosis, and even avoid, problems when they arise: Is the source of a communication lapse local or wide-area? Is this page still downloading or has my browser crashed?

In fact a hallmark of advanced technology is that people are often removed, both spatially and behaviorally, from the components of the systems that they are using. Successful interface designs generally enhance the performance of systems that are working properly, but this tends to distance users from what’s going on "behind the scenes". This separation can cause people to fall "out of the loop" when unexpected circumstances or problems arise (Norman, 1988). Users may become reliant on the features of the interface as true indications of the system state (e.g. content continuity), rather than the underlying causes. As richer primary content is delivered visually, there is less space available for secondary feedback. A potential solution is to transfer non-content information delivery off of the screen, to the auditory domain. But the utility of auditory display is not based solely on the limitations of screen sizes.

Constraints on Visual Attention

As information is delivered in more engaging formats such as video and interactive animation, in addition to text, users may fixate on a particular part of the display. When a user’s visual attention is focused on a particular area, he or she is unable to attend to visual information beyond the field of view. Much like immersion in any highly visual activity such as reading or watching television, the web can incite a "tunnel vision". A significant difference between web browsing and these more typical activities is the relative unreliability of the components: networks, operating systems, software, variability in user skill, etc. Given the dependency of information delivery on these subsystems, one might consider browsing as a simplified form of process monitoring. Traditionally, process monitoring has involved complex systems (air traffic control, power plant management) where operators must remain vigilant of system behavior. Similarly, web browsing can be viewed as being comprised of two subtasks: content acquisition (primary), and background process monitoring (secondary). Obviously, the danger in a nuclear power plant failing is not equivalent to a personal computer crash, but until reliability is achieved on par with telephony, monitoring remains as a secondary activity. Some of the lessons learned in the design of process monitoring systems can apply to web interface design. For example, peoples’ omnidirectional listening ability make auditory signals in particular, excellent for re-orienting attention towards a pressing matter. There are other characteristics that merit the use of sound as an important interface tool.

Properties of Sound

Mountford and Gaver (1990) eloquently suggested that "sound exists in time and over space, vision exists in space and over time." In other words, information delivered via sound tends to indicate changes over time, but can be picked-up over a wide range of spatial locations. Visual displays are usually less transitory, but can only be perceived at specific locations in space. This is an oversimplification of the characteristics of acoustically and optically conveyed information, but it points out that auditory displays differs from visual displays in several important ways. The relevance of this to web interface design can be readily seen from the previously described situations. Sound is omnidirectional and alerting, suggesting that it can be used to provide information over and above both visual display and visual attention limitations. This is not to suggest that sound be used in place of visual display, rather it should enrich the visual experience. Acoustical media can carry information that is distinct from optical media in both form and content, and an interface designer should chose the form and modality of a display to fit the context of the information and the user environment.

One particular case where this may be useful is in the presentation of background process information such as downloading, as sound is good at displaying changes over time. For example we are all familiar with the "modem-handshaking song" whose changes in pitch and timbre aurally confirm the status of a log-on. I propose an extension of this to other aspects of web browsing, providing continuous, but unobtrusive feedback , while allowing users to attend to visual displayed primary content. Furthermore, I see this as the basis for auditory development guidelines. Given that the use of sound is relatively limited, there is an excellent opportunity to consider the Human Factors issues in sound portrayal on the web, and particularly to focus on the design of sounds that are globally usable.

Two Ways of Listening

Experts in auditory display (Kramer, 1994) recognize a continuum of sounds ranging from audification to sonification. Audification, is similar to the concept of icons in visual displays, in that the acoustical structure of the sound parallels that of the represented event, such as a "You have mail!" indicating the receipt of a new e-mail message. That is, these are typically familiar, short-duration copies of real-world acoustical events. At the other end , sonification, refers to the use of sound as a representation aid to acoustically map information for events that do not typically have a relevant acoustic component. Examples of sonification use include sonar pings to convey distance information about objects in the environment. Here the sounds are the means of information conveyance, but are not meaningful in and of themselves. Rather it is the relations (in the case of sonar, temporal) among the sounds that provide information to a listener. Note that "realistic" sounds could be used for the purpose of sonification, but there again is the problem of loss of recognition with changes in data. In a nutshell: in audification the structure of the sound contains the information, and in sonification the relation among sounds conveys information. This means that, in the case of sonification, changing the specific sounds does not change the information, and even simple tones may be used.

An analysis of how people actively listen for non-speech information can play a large role in determining the acoustical characteristics that can be used to transform information into sound. Gaver’s work on everyday listening (1993a,b) provides a unique theoretical perspective on this issue. Gaver makes a categorical, but malleable, distinction between musical listening and everyday listening. The former describes attending to the acoustical components of an event, and the latter refers to perceiving events conveyed by the acoustical components. These are not definitive delineations; they refer to the kind of information that a listener is trying to obtain, and not specifically to the source of sounds he or she is hearing. People may attend to certain acoustical features for musical listening, and ignore those features while attending to others for everyday listening. This duality can easily be illustrated. Imagine sitting in a concert hall listening to a singer. If you attend to the relations between the sounds: patterns in the melody, changes in tempo, and such, this is considered musical listening. On the other hand if the sounds are used to obtain information about the event; i.e., perceiving that a harpsichord is being played at a certain distance, in a hall of a given size, etc., then that is everyday listening.

In everyday listening the sounds themselves are important - one kind of tone can convey distinct information from another about the listener’s world. In musical listening it is not the particular sound themselves which are essential, but the information that is conveyed by the spatio-temporal pattern and arrangement of sounds, and thereby not language or culturally specific. Spatio-temporal patterns are meaningful changes in the position of objects in space (real or virtual) over time. In other words the message conveyed by the choice and arrangement of the notes will remain invariant even though the piece could be played on different instruments, environments, etc. In a nutshell then, there is sound as information (everyday listening) and information as sound (musical listening).

The Trouble with Icons and Earcons

A key decision in designing an auditory display is determining the sounds which will be used to portray information. Developing for global usability means selecting sounds that are not culturally constrained. Hansen (1994) warned against the use of semantically-load icons in graphical displays. He is referring to components of the interface that resemble real-world objects and have some feature, or features, that are analogous to those objects, but do not afford all of the actions that one might expect. For example, the "Trash Can" icon on a pc desktop is generally recognizable as a place where unwanted items may be thrown away. In that respect it functions like one would expect a trash can to. A real trash can would have a myriad of other uses (affordances) such as holding things down, generating noise, impeding entrance, etc. On the desktop it does not have these uses, and even its operational function is context specific - you can thrown out a file, but dragging a disk to the trash ejects it (fortunately), rather than erasing it. Of course this all assumes that you have some reasonable understanding of what a trash can is, or trash for that matter. It is reasonable to suppose that the same kind of affordance expectancies exist with sounds. For example, what good would a foghorn warning signal be to someone who lives in the desert?

Empirical work on auditory display design has shown some of the problems with using sounds that were based on real acoustic events. The Arkola Simulation (Gaver, Smith, and O’Shea, 1991) was an evaluation of an auditory icon-based display for process/manufacturing monitoring and control. The authors discussed some of the difficulties that arose out of using semantically-loaded symbology:

"First, the urgency of the alarm sounds did not always appropriately reflect the urgency of the situations causing them. For example, the breaking bottle sound was so compelling semantically and acoustically that partners sometimes rushed to stop the sound without understanding its underlying cause or at the expense of ignoring other more serious problems."

Another related difficulty was that the use of realistic sounds meant that when a process had failed to operate it would become silent, rather than alerting, and operators would not be alerted to the system malfunction.

Difficulties in using metaphoric or iconic sounds become more apparent when considering the limitations of their use in representing change in dynamic processes, such as data transfer. The display of system processes requires the generation of sounds that correspond to those relations. Many of those processes involve complex /or abstract interactions, making it difficult, if not impossible, to generate realistic iconic sounds. Additionally, certain characteristics of these sounds will vary as a function of dynamic system states. For example changes in the pitch of an engine sound could be used to indicate changes in CPU use, but relatively small changes in pitch can easily make the engine noise unrecognizable as an engine (although potentially still useful as an indicator). Varying sonic parameters limits the identifying features of sounds. Changes in frequency and temporal rate can make sounds that were readily identifiable under normal operating conditions, cacophonous at system extrema, when they are most useful. Finally, the interactions of the sounds might also impede identification, and, instead of emergent sonic characteristics, would lead to confusion.

The solution to many of these problems is to use abstract or simple tones to convey information. Such sounds do not have particular cultural connotations, and this will avoid the problem of mistaken affordances, as sounds will not represent anything other than there designed function. Additionally, abstract sounds can be specifically tailored to achieve both acoustical and perceptual harmony for the particular system or function to be represented.

It is not a great leap to see the similarities between audification with everyday listening, and sonification with musical listening. Audification and sonification are descriptive of the display processes, while everyday and musical listening are perceptual level categories; describing what kind of information the listener is getting out of the sounds. One is capable of extracting either kind of information from either kind of display, but that information may not be optimal for the user’s task-specific goals. In the case of presenting information about background processes we want the listener to hear changes over time, and emergent patterns. This is a dynamic process best conveyed over time, rather than by static icons. Using sonification also avoids the problems of cultural literacy that can limit the usability of icons. Therefore, monitoring dynamic background processes, such as downloading, is akin to musical listening. And clearly we are able to perceive complex patterns among the relations of sounds even when the sounds themselves are different.

Developing a Dynamic Auditory Display

Given the research, an auditory display aimed at presenting information flow should emphasize musical listening. Sonification should be used to combine abstract sounds into informative displays. This will allow for the display of dynamic changes in information exchange, that are not constrained by culture-specific sounds. But what should be the form of such a display, and how can multiple sonic elements be arranged into a meaningful and perceptually salient presentation? A promising answer to these questions lies in work done on visually displays.

1The key point concerning configural displays is that the symbols themselves are meaningless - we could use lines, circles, cupcakes, etc. What does matter is the relations, in this case spatial, among the elements. Simple spatial relationships, such as symmetry and proximity are easily conveyed. Given the utility of visual configural displays and there potential for largely transcending linguistic and cultural differences, one might next consider the implementation of such displays in nonvisual domains.