Cognitive walkthrough

Draft

Introduction

According to the NASA usability toolkit, retrieved 12:36, 18 March 2011 (CET), A cognitive walkthroug, is a “process of going step by step through a product or system design and getting reactions from key team players and typical users. One or two members of the design team can guide the walk through while one or more users will comment as the walk through proceeds.”

The Fluid project, retrieved 12:36, 18 March 2011 (CET), provides the following definition: “A cognitive walkthrough is a step-by-step exploration of a service to see how well a particular type of user, usually represented by a persona, is able to accomplish a particular objective or set of objectives. The objectives selected for testing are dictated by the persona that is chosen.”

James Hom defines cognitive walthrough as “review technique where expert evaluators construct task scenarios from a specification or early prototype and then role play the part of a user working with that interface--"walking through" the interface. They act as if the interface was actually built and they (in the role of a typical user) was working through the tasks. Each step the user would take is scrutinized: impasses where the interface blocks the "user" from completing the task indicate that the interface is missing something. Convoluted, circuitous paths through function sequences indicate that the interface needs a new function that simplifies the task and collapses the function sequence.”

In a recent state of the art study, Thomas Mahatody et al. (2010) identify eleven significant extensions of Cognitive Walkthrough (CW): Heuristic Walkthrough, The Norman Cognitive Walkthrough Method, Streamlined Cognitive Walkthrough, Cognitive Walkthrough for the Web, Groupware Walkthrough, Activity Walkthrough, Interaction Walkthrough, Cognitive Walkthrough with Users, Extended Cognitive Walkthrough, Distributed Cognitive Walkthrough and Enhanced Cognitive Walkthrough.

According to Mahatody et al, “at the beginning, CW was a method for assessing usability problems in "walk-up-and-use" systems (i.e., systems that can be used with little or no training). Due to its success, practitioners and researchers interested in this method tried to improve it or to adapt it for other specific types of application. In most cases, the modifications and extensions concern theoretical or conceptual aspects; however, some concern methodological aspects.”

Methods

Fluid project overview

The Fluid project roughly suggest the following procedure.

(1) Choose a user from whose perspective the walkthrough will be done. You may choose a persona for that.

(2) Define what the person wants to achieve

(3) Define the steps that this person should do in order to achive her/his goals

(4) Perform the task and take notes about the following kind of issues for each step:

• Will the user know what to do at this step?
• Is complex problem solving needed to figure out what to do?
• Will they know that they did the right thing (if they manage) and are making progress towards their goal?
• Is complex problem solving needed to interpret the feedback?

This project also defines a much more elaborated UX Walkthrough Process. In particular, it suggests to formalize the inspection process along three axis:

1. A protocol that clearly specifies what we are going to do, and what information we are going to capture along the way.
2. A predetermined clearly specified target that we are going to inspect: (product, version, instance, set of chunks) - the thing we're going to do it to.
3. A report template specifying the format, style and content of the report of the information we capture.

(retrieved 12:36, 18 March 2011 (CET))

Look at their Generic UX Walkthrough Report Template if you need an example

Wharton et al. 1994 (CW3)

According to Mahatody et al. (2010), this variant which they labelled "CW3" works like this: “the evaluator is invited to imagine a specific and credible scenario for each action that users must run to accomplish their task. To make the scenario credible, the evaluator must justify each action with respect to the user's background and knowledge and the feedback from the interface. To assess each action in the task, the evaluator must answer four questions related to various user thoughts and actions:”

1) What is the user thinking at the beginning of the action

Q1: Will the user try to achieve the right effect?

2) Is the user able to locate the command

Q2: Will the user notice that the correct action is available?

3) Is the user able to identify the command

Q3: Will the user associate the correct action with the effect that user is trying to achieve?

4) Is the user able to interpret the feedback

Q4: If the correct action is performed, will the user see that progress is being made toward solution of the task?.

Mahatody et al. then report that “most of the reviewing authors agreed that CW is tedious, despite being easy to learn and use.”

Streamlined CW - Spencer, 2000

Spencer (2000), presents the following model for a streamlined Cognitive Walkthrough process, adapted directly from Wharton, et al. (1994). Spencer argued that implementing a cognitive walkthrough procedure is not obvious and suggest that in many situations a stream-lined version would be more successful. He suggests to adopt the following ground rules for two reasons: (1) Ensure that developers will not consider the procedure as "lost time" and (2) make sure that they don't become defensive:

• Avoid design discussions (no designing, no defining of a design).
• No debating of cognitive theory
• The usability specialist is the leader of the session

Second, he argues in favor of simplication, in particular with respect to Warton:

• Collapse/simplify Warton's walk-through questions.
• Capturing Less Data

According to results of a study he conducted, {{quotation|Streamlining the walkthrough may trade-off granularity for coverage, but without that trade off, program managers and developers may perceive the walkthrough as being an inefficient use of time. Performing a streamlined CW is a good way to profile a user interface for potential problem areas, identify many steps that may be problematic for users, and accurately predict many usability problems. However the method will probably result in a few false positives.

Summary of the streamline cognitive walkthrough:

1. Define inputs to the walkthrough
   1. Identification of users
   2. Sample tasks for evaluation
   3. Action sequences for completing the tasks
   4. Description or implementation of interface
2. Convene the walkthrough
   1. Describe the goals of the walkthrough
   2. Describe what will be done during the CW
   3. Describe what will not be done during the
   4. lkthrough
   5. Explicitly defuse defensiveness
   6. Post ground rules in a visible place
   7. Assign roles
   8. Appeal for submission to leadership
3. Walkthrough the action sequences for each task
   1. Tell a credible story for these two questions:
      • Will the user know what to do at this step?
      • If the user does the right thing, will they know that they did the right thing, and are making progress towards their goal?
   2. Maintain control of the CW, enforce the ground rules (see below)
4. Record critical information
   1. Possible learnability problems
   2. Design ideas
   3. Design gaps
   4. Problems in the Task Analysis
5. Revise the interface to fix the problems

Cognitive Walkthrough for the Web (CWW) - Blackmon et al. 2006)

According to Mahatody (2010),

The questions for CWW evaluation are as follows: Q1: Will the correct action be made sufficiently evident to the user?

Q2: Will the user connect the correct subregion of the page with the goal using heading information and her understanding of the sites page layout conventions?

Q3: Will the user connect the goal with the correct widget in the attended to subregion of the page using link ,labels and other kinds of descriptive information ?

Q4: Will the user interpret the system's response to the chosen action correctly?" (Blackmon et al., 2002)

Links

Introductions

• Cognitive Walkthrough at Fluid. Added by Jonathan Hung, last edited by Allison Bloodworth on May 26, 2009.

• UX Walkthrough Process at Fluid Added by Colin Clark, last edited by Paul Zablosky on Feb 12, 2009

• Cognitive Walkthrough by James Hom.

Examples

• Fluid project makes available a series of well-documented examples (from real studies). Search for walkthrough in order to retrieve the impressive list of shared documents. Below are just 2 examples plus one of the templates:
  • uPortal Walkthrough - Ed (Student) Persona (Dogle)
  • Sakai - Justin's Walkthrough
  • Generic UX Walkthrough Report Template

Bibliography

• Blackmon, M., Polson, P., Kitajima, M. & Lewis, C. (2002). Cognitive Walkthrough for the Web, Proceeding of CHI, ACM Press, 463-470.

• John, B. E, & Packer, H. Learning and using the cognitive walkthrough method: A case study approach, in Proceedings of CHI '95 (Denver CO, May 1995), ACM Press, 429-436.

• Rowley, D. E., and Rhoades, D. G. The cognitive jogthrough: A fast-paced user interface evaluation procedure. Proceedings of CH1 '92 (May 1992), ACM Press, 389-395.

• Mahatody, Thomas / Sagar, Mouldi / Kolski, Christophe (2010). State of the Art on the Cognitive Walkthrough Method, Its Variants and Evolutions, International Journal of Human-Computer Interaction, 2, 8 741-785. , PDF preprint. This is a very good overview and discussion of various variants.

• Spencer, Rick. (2000). The streamlined cognitive walkthrough method" CHI 2000 Proceedings, 353-359. PDF at ACM, PDF at Fluid

• Wharton, Cathleen, et. al. (1994). "The Cognitive Walkthrough Method: A Practictioner's Guide." in Nielsen, Jakob, and Mack, R. (eds), Usability Inspection Methods, New York: John Wiley, ISBN 0-471-01877-5.