I thought that this was an interesting study. It seems pretty supportive
of text!
David Andrews
---------- Forwarded message ----------
Date: Mon, 23 Jun 1997 10:33:48 -0500
From: "Jared M. Spool" <jspool@uie.com>
Reply-To: basr-l@trace.wisc.edu
To: Multiple recipients of list <basr-l@trace.wisc.edu>
Subject: Book Recommendation: Web Site Usability: A Designer's Guide
Here is chapter 1 of "Web Site Usability: A Designer's Guide" that
you requested. I've also included an annotated table of contents
and ordering information, including information on the Introductory
Offer that expires 7/15/97. Please feel free to forward this
message to anyone you think might find it of interest.
Please let me know if you have any questions.
Jared M. Spool, Founding Principal
User Interface Engineering
jspool@uie.com
- o - o - o -
WEB SITE USABILITY: A DESIGNER'S GUIDE
Chapter 1: Web Site Usability: The Big Picture
Contents:
What Is "Usability" on the Web?
The Sites We Tested
"Scavenger Hunt" Tests
The Results
The Rankings
Room for Improvement
Beyond the Rankings
The Major Implications
Implication 1: Graphic Design Neither Helps Nor Hurts
Implication 2: Text Links Are Vital
Implication 3: Navigation and Content Are Inseparable
Implication 4: Information Retrieval Is Different than Surfing
Implication 5: Web Sites Aren't Like Software
A year ago, we started wondering what made a web site usable. We had
heard the opinions of experienced designers about what they felt it
took to create a good site. We looked at books and magazines that
talked about how to make a "cool" site. But no matter where we looked,
we couldn't find any data -- based on real user experience -- about what
it takes to make a usable site.
This report is our attempt to start providing that data to web site
designers.
__________
What Is "Usability" on the Web?
We set out to study usability of web sites, but we first had to
determine what that means. The usability of a site depends on what
users are trying to accomplish. Are they surfing? Doing research?
Buying products? Downloading software? And it also depends on the
organization's goals for creating the web site. Is the site aimed at
marketing a service? Selling merchandise? Making information available
to employees, shareholders, and customers?
Whatever the goal, information is a central theme. For intranets
(internal web sites), information is the theme -- no one surfs the
online employee policy manual just for kicks. Because of this, we
focused our study on how successful sites are at providing people with
information so they can make decisions. The more a site helps people
find the information they are looking for, the more usable it is.
__________
The Sites We Tested
We picked a set of nine popular sites with content we thought would be
useful for a general audience:
C|net (http://www.cnet.com) Technology resources and
information, product reviews, listings of where to buy
products.
Disney (http://www.disney.com) Games, videos, merchandise,
Disneyland and Walt Disney World theme park information and
reservations.
Edmund's (http://www.edmunds.com) Car and truck prices,
specifications, reviews, and other resources for vehicle
buyers.
Fidelity (http://www.fidelity.com) Fidelity mutual funds,
personal and corporate investing opportunities.
Hewlett Packard (http://www.hp.com) Product information,
financial information, job opportunites.
Inc. (http://www.inc.com) Small business resources, book
reviews, articles, conferences, contact information for
organizations.
Olympics (no longer available) Schedules and results from the
1996 Olympic Summer games in Atlanta, merchandise, tickets.
Travelocity (http://www.travelocity.com) Airline tickets, car
and hotel reservations, guide to recreational activities
world-wide.
Websaver (http://www.websaver.com) Annuity sales and
information.
Two of the sites, Disney and C|net, were high-profile sites that had
been well-reviewed by the media. We expected that these sites would
do the best, and that we could learn from them about successful web
site design.
True to the ever-changing nature of the web, two of the sites were
updated in the middle of our study. The Disney site changed to a
frame-based design, and the layout of Inc.'s home page also changed.
We collected enough data to include both versions in our findings. To
our dismay, the Olympic site (which had done well in our study)
disappeared before we could obtain screen shots for our report.
__________
"Scavenger Hunt" Tests
All these sites, while obviously trying to sell products, also provide
information. Because we wanted to learn how easy it was for users to
answer questions on these sites, we set up a "scavenger hunt"
usability test.
We brought in users who were familiar with a web browser, sat them
down in front of a site and watched them try to answer several
questions. They had to hunt through the site to find the answer, even
if they already knew it. We wanted to see how they searched and what
factors of the site helped or hindered them.
__________
The Results
Searching for information on web sites is an intensely frustrating
experience. Throughout our study, we were amazed by the time and
effort it took users to answer even simple questions. And repeatedly,
users gave up without ever finding what they were looking for. Even in
the smaller web sites, we watched users get lost or wander off the
site without being aware of it.
>> The Rankings
After more than 50 tests, we have a good idea of how our sites compare
to each other. The results are startling. Disney and C|net -- the sites
we expected to do best -- fared poorly in our study. Edmund's, which
none of us would have bet on, came in first.
The relative ranking of the sites, from best to worst, were:
#1 Edmund's
#2 Olympic
#3 Hewlett Packard
#4 WebSaver
#5 Travelocity
#6 Inc. (New)
#7 C|Net
#8 Fidelity
#9 Inc. (Old)
#10 Disney (New)
#11 Disney (Old)
>> Room for Improvement
Comparing the sites to each other tells only part of the story. Even
Edmund's (with a score of approximately 44), the best site in our
study, fell far short of the highest possible score (100). Clearly,
when it comes to web site design, there is room for improvement.
For designers, this is actually good news. Many people we've talked to
know that their sites have usability problems, but don't know where to
look to begin fixing them. In the course of our research, we gained
some insight into which aspects of site design can help or hinder
users the most.
>> Beyond the Rankings
After looking at the site rankings, our ideas about successful web
site design were turned completely upside-down. To find out why, we
began scrutinizing the sites themselves, looking for similarities and
differences that might account for the users' success or lack thereof.
We used a statistical model to discover which factors were most
correlated with user success.
There were lots of surprises. When we found things that flew in the
face of common sense and the recommendations of other designers, we
did more tests to get more data. We even conducted some parts of the
analysis two or three different ways, and got the same results.
Other chapters describe the our findings in detail. But first, let's
look at some of the high-level implications of these results.
__________
The Major Implications
Some of results of our study are counter-intuitive, but what makes our
research both controversial and fascinating are its implications.
These results could dramatically change the way people develop web
pages.
>> Implication 1: Graphic Design Neither Helps Nor Hurts
As hard as we looked, we couldn't find any evidence that graphic
design helps users retrieve information from a site.
Consider the following:
* We measured all the graphic design elements we could think of,
and none of them had any significant correlation -- either
positive or negative -- with users' success (for more
information, see Chapter 7, Graphic Design on the Web).
* Several of the sites had very professional-looking designs. The
Hewlett Packard and Olympics sites did well, while C|net and
Disney -- also graphically intense -- scored at the lower end of
our ratings.
* Edmund's, the top-scoring site in our study, is mostly text.
* When users navigated, they often tried text links first,
ignoring nearby graphics.
Of course, graphics may be important in other ways, such as for
conveying marketing messages, making users more willing to return to
the site, or selling products. We didn't measure these, so we don't
know. But as far as we can tell, graphic design is completely
unrelated to success at finding information on web sites.
>> Implication 2: Text Links Are Vital
In watching users work with the sites, we couldn't help notice how
important text links are. Because of downloading delays, text links
are often the first things visible on the page. Most users examined
text links before considering image links.
There are many different types and styles of links, and some styles do
seem to work better than others (for more information, see Chapter 3,
Getting Around: Links). More important than style, however, is the
predictiveness of the link. The better users could predict where a
link would lead, the more successful they were in finding information.
The text link is the way users prefer to navigate sites. Yet, very
little of the design advice available talks about how to create
effective text links. This is clearly an area for further study.
>> Implication 3: Navigation and Content Are Inseparable
We've heard a lot about the shell strategy -- a technique that lets
developers design a navigational structure and hierarchy first, then
just plug the content into it. For example, one of our clients (a
large multinational bank) has one department working on the overall
look and feel of the site, including the home page, navigation bars,
style sheets, and templates for different types of interactions. Other
departments are responsible for creating the content. The developers
of the overall structure -- or shell -- don't know what the content will
be because it will be plugged in later.
When we were doing our research, we didn't talk with any of the site
developers, but saw a few shell sites nonetheless. (Our rule of
thumb for identifying shells is this: If you could remove more than
half of the site content without having to update the home page, it's
most likely a shell site.) The links in sites like Inc. and C|net are
so generic that users rarely got what they expected.
Based on our observations of Inc. and other shell sites, we have no
evidence to suggest that the shell strategy can succeed. The sites
that were most successful were those where content and navigation
were inextricably linked -- where you couldn't remove content without
updating all of the main navigation pages.
The problem with shells is that by definition they require lots of
generic links, which make it harder for users to predict what they
will find. This implication makes development of large web sites
significantly more difficult, because it suggests that the home page
and high-level links may need to change more frequently.
>> Implication 4: Information Retrieval Is Different than Surfing
Our study focused on one specific activity: information retrieval
within a large site. We didn't study surfing, the other primary use of
web sites. When users surf, they are just browsing, clicking whatever
looks most interesting or "cool," and content may not be the driving
force in coolness.
In the movie Wayne's World, there is a scene where Wayne opens a door
in a local donut shop to reveal a troupe of black-clad warriors
practicing martial arts. He watches them for a moment and then shuts
the door, proclaiming "I've always wanted to open a door and see a
bunch of guys Kung-Fu fighting." The scene is completely irrelevant to
the rest of the movie -- it's just cool. When users surf the web,
they're looking for the guys who are Kung-Fu fighting.
When looking for information, users are much more focused. They tend
to click on the link most likely to yield the information they're
hunting for. The kinds of things designers put on web sites to attract
surfing users proved to be distractions during information retrieval
tasks. For example, users saw advertisements as visual "noise," and
animation was so irritating that some users covered it up!
This implies that sites aimed at information retrieval need to be
designed differently from sites aimed at surfing. These are
conflicting goals. Unfortunately, we don't yet have all the answers
about where the differences lie.
>> Implication 5: Web Sites Aren't Like Software
Companies have been usability testing their software products for
years. Our firm alone has conducted thousands of usability tests on
hundreds of software and hardware products. We've learned a lot about
measuring usability. We assumed that the web sites would just be
another form of software, and could be tested similarly.
Boy, were we wrong! The web presents lots of problems that we've never
seen before, which make it hard to define what usability even is, let
alone measure it.
Consider user preference. When we do comparative testing of software
products, we ask users which product they like the most after they've
worked with all of them. Almost without fail, users choose the same
product that they were most successful with. For software, we've
found that user preference can be a reasonably good proxy for
measuring usability.
This is not true for web sites. We asked users to pick which site they
liked the best. While some people chose the site they were most
successful with, others did not. These users liked a site because of
its content, rather than the site's ability to help them find
information. They'd say things like "I liked Disney, it seemed more
interesting," even if they had gotten completely lost and failed to
complete any of the tasks.
For web sites, user preference obviously doesn't measure the same
things it does for software, so our tried-and-true proxy suddenly
doesn't work any more. And the more data we find, the more we realize
we don't really know what makes a web site usable.
The web is a whole new ball-game, and we're still learning how to
play. We don't yet know how to design for finding information. We
don't know how to design for comparisons so that users can find the
best house, car, or job -- things the press tells us the web excels
at. We don't know how to effectively use multiple media like graphics,
animation, interactive applets, and text to produce the best results.
To web site designers, this probably doesn't feel like good news. But
with this study we now have some insights we never had before, and we
hope that our results will inspire further research into all these
areas.
(c) Copyright 1997, User Interface Engineering
- o - o - o -
WEB SITE USABILITY: A DESIGNER'S GUIDE
Annotated Table of Contents
Web Site Usability: A Designer's Guide presents research findings
about what it takes to make a web site usable. It is based on more
than 50 usability tests of nine web sites. (Order information at the
end of this message.)
Part 1 Research Results
Everyone has opinions about what makes a good web site, but the
chapters in Part 1 present real data about what really makes a web
site usable.
Chapter 1 Web Site Usability: The Big Picture
When people use web sites, they are trying to get information. Given
that, our research focused on how successful sites are at providing
people with the information they need to make decisions. Some sites
fared much better than others, and Chapter 1 presents the site
rankings. It also discusses the high-level implications of this
research, including:
* Graphic design neither helps nor hurts
* Text links are vital
* Navigation and content are inseparable
* Information retrieval is different than surfing
* Web sites aren't like software
Chapter 2 Getting Around: Navigation
Like most aspects of usability, navigation is invisible when it's
working, but when there's a problem, users can get completely stuck.
Users had difficulty navigating when they lacked knowledge about the
domain of the site. They also had problems when the site structure
didn't meet their expectations. Chapter 2 also discusses some of the
traditional navigation devices -- frames, tables of contents, and so
on, and why some of these devices work better than others.
Chapter 3 Getting Around: Links
Links are closely tied to navigation, but are so critical to user
success that they merit their own chapter. The success of a link
depends on how well the user can:
* Predict where the link will lead.
* Differentiate one link from other, nearby links.
This chapter discusses those issues, and talks about how link layout
and where links lead can affect user success.
Chapter 4 Within-Site Searching
We didn't tell users to use the site's search mechanism to find
answers, but about half did. Users often didn't understand the scope
of the search. They also had trouble interpreting the search results.
Chapter 4 discusses these problems in detail.
Chapter 5 The Difficulty of Comparisons
Most web sites try to sell products, but do very little to help users
choose between different options. Chapter 5 looks at some of the
problems with comparisons (like pogo-stick structures) and presents
some possible solutions (like a catalog approach and questionnaires).
Chapter 6 Readability and Page Layout
Most users skim text quickly while they are looking for answers, so
pages that are designed to promote readability don't work so well.
Chapter 6 looks at how readability, white-space, and page layout
affect user success.
Chapter 7 Graphic Design on the Web
Although we studied every aspect of graphic design we could think of,
we found no evidence that graphic design helps users retrieve
information on a web site. Chapter 7 discusses what we looked at, and
highlights two graphic issues that did catch our attention: download
time and animation.
Chapter 8 User Preference
In software, we've found that the products people like best are
usually the ones they can use most easily. The same is not true for
web sites, however. Chapter 8 talks about why this is and what user
preference really measures.
Part 2 Site Scrapbook
Part 2 gives you a highly-graphical guided tour of the sites we
studied, beginning with the sites that scored best and continuing
down the rankings. The only exception is the Olympic site, which was
taken down before we got any pictures.
Chapter 9 The Edmund's Site
The Edmund's site finished first in our study. It is a resource for
people who want to research and purchase cars and trucks. Although it
is mostly text, with nearly no graphic design, users still did best
with it. We believe this is due in part to its long, descriptive
links.
Chapter 10 The Hewlett Packard Site
The Hewlett Packard site finished third in our study. Unlike
Edmund's, it does have a professionally designed look. Part of its
success may lie in how different its links are from each other.
Chapter 11 The WebSaver Site
WebSaver finished fourth in our study. It is a small site, designed
to sell annuities. The site's design does not facilitate comparison,
however. The Frequently Asked Questions (FAQ) page proved useful.
Chapter 12 The Travelocity Site
Travelocity lets users make travel reservations and research
recreational activities world-wide. It finished fifth in our study,
in part because it used some concepts that were unfamiliar to users.
The site also requires users to register, which was a turn-off for
many participants.
Chapter 13 The Inc. Site
The Inc. Site offers articles and resources for small businesses. It
was redesigned in the middle of our study. The new design finished
sixth, and the old one finished ninth. The home page suffers from too
many generic links. Watching users struggle with this site made us
doubt the effectiveness of the shell strategy.
Chapter 14 The C|net Site
C|net provides technology resources and information, and finished
seventh in our study. Like Inc., it is a shell site. Users often had
trouble deciding which link to follow.
Chapter 15 The Fidelity Site
The Fidelity site, ranking eighth in our study, has information about
the company's products and services. The pages are designed to
minimize scrolling, but that neither helped nor hurt. We did see some
confusion about frames. Users who found the Fidelity Site Map were
twice as successful as those who didn't.
Chapter 16 The Disney Site
The Disney site provides information about Disney's entire empire. We
saw problems caused by the structure of the site, and users had a lot
of trouble searching. The site was redesigned part way through our
study. The new site finished tenth, and the old site finished
eleventh. The new site used frames, and the site did slightly better,
but we can't say that frames made the difference.
Part 3 Testing Sites
Part 3 shares how we tested our sites. It's not intended to be an
overview of usability testing.
Chapter 17 How We Tested Web Sites
Chapter 17 describes the methods we used, including the types of
questions we asked, the data we collected, and how we calculated the
site rankings. This chapter is useful if you want to compare your site
to the ones we tested, or if you just want some insight into how to
test web sites.
(c) Copyright 1997, User Interface Engineering
All rights reserved.
__________
Ordering Information
Web Site Usability: A Designer's Guide is available from User
Interface Engineering. You can order by email, web, fax, or phone.
Introductory Offer: $39.95 (List price: $79.95. Offer expires 7/15/97)
Orders:
Email: web@uie.com
web: http://www.uie.com
fax: (508) 975-5353
phone: (800) 588-9855
Order Form (Send to web@uie.com)
Quantity: _____ x $39.95 = Total: $__________
Sales Tax (Mass. Only): Add 5% per order: $__________
US Shipping & Handling: $7.25 per copy: $ __________
Total Order: $__________
(Shipping and handling for international orders extra.)
Name: ____________________
Company: ____________________
Address: ____________________
City/State/Zip/Country: ____________________
Email: ____________________
Daytime Phone: ____________________
Payment Information:
___ Mastercard
___ Visa
___ American Express
___ Discover
Account #: _____________________
Expiration Date: _____________________
Allow 2-3 weeks for delivery.
==========================================================
Jared M. Spool User Interface Engineering
mailto:jspool@uie.com 800 Turnpike Street, Suite 101
(508) 975-4343 North Andover, MA 01845
fax: (508) 975-5353 USA
http://www.uie.com
If you send me your postal address, you'll get
the next issue of our newsletter, Eye For Design.
==========================================================
This archive was generated by hypermail 2b29 : Sun Dec 02 2012 - 01:30:04 PST