Category: Usability Testing

A Universal Usability Test, Take 1

In one of the darker corners of my mind I imagine a future where there are a set of laws and industry standards that dictate the acceptable usability of digital products and services, much like medical or engineering standards. I have to think that as we grow increasingly reliant on computer technology for our safety and well-being, minimum usability standards must follow. This kind of regulation has already happened for the food we eat, any electrical appliance, and our cars. It’s hard to imagine it won’t happen for software, hardware, and digital services. But it will need to be a different kind of regulation.

Here’s how it might happen: we first create this standard and find ways for people to voluntarily start using it. Perhaps a pro-consumer organization takes on the role of applying it, and consultancies provide testing services. Maybe that’s enough, or maybe industry organizations formally adopt it, and legislators make it mandatory in certain cases.

What does a universal usability test look like? Here’s a sketch:

  1. The basics of the process and the results are simple enough for the average consumer to understand, in the same way as the UL mark or the Consumer Reports harvey balls. As a standard, the results should simply indicate the product has met the standard or not.
  2. The standard is described in terms of the user’s experience:
    1. Time: there’s a maximum amount of time* designated to a task. Seven random people from the product’s user population are asked to complete the task and all must successfully do so in the time allotted.
    2. Emotion: each test participant rates their emotion using the product using a standard measure of feeling like the Wong-Baker Facial Grimace Scale. If the total score of the seven participants exceeds 25 the product fails the test.

Wong-Baker Facial Grimace Scale

* How do we determine the maximum allowable time? I haven’t figured that out yet.

The Ten Distinguishing Properties of Wicked Problems

You may have heard of Rittel and Webber’s wicked problems (problems that are messy, circular, and aggressive). I was interested to see their original paper (pdf) includes ten distinguishing properties “that planners had better be alert to” because “policy problems cannot be definitively described.”

  1. There is no definitive formulation of a wicked problem
  2. Wicked problems have no stopping rule
  3. Solutions to wicked problems are not true-or-false, but good-or-bad
  4. There is no immediate and no ultimate test of a solution to a wicked problem
  5. Every solution to a wicked problem is a “one-shot operation”; because there is no opportunity to learn by trial-and-error, every attempt counts significantly
  6. Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan
  7. Every wicked problem is essentially unique
  8. Every wicked problem can be considered to be a symptom of another problem
  9. The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution
  10. The planner has no right to be wrong

It makes me wonder if any politicians have tried to campaign on a process for solving wicked problems instead of prescriptive solutions.

Tests Well With Others

In this brave new world where user-centered designers meet old-school info technologists, we can all live, love, and test together. Here’s how my usability testing jives with their testing, as I understand it:

What types of testing do we do?


  • QA: technical check to make sure it works
  • UAT (user acceptance testing): ensure the system meets the business requirements
  • Usability: testing with end users

Who does it?


  • QA: technical staff, but not the people who programmed it
  • UAT: business staff. Originated in days when business people threw requirements over the wall to tech and needed to test what came back over the wall. In agency models it’s often rolled into design.
  • Usability: dedicated usability peeps or the design staff, but preferably not the people who designed it

The Goal of Usability Testing

What is the goal of usability testing? Let’s say it’s to improve the user’s experience.

The user’s experience will improve when designers have the skills and resources to design well.

The designers will improve when they have a better understanding of the users.

Usability testing offers us some understanding of users. The direct goal of usability testing is to yield this information, but the indirect goal of educating designers is ultimately more important (until someone devises a test that automatically tweaks the product without a designer, which won’t be anytime soon).

This struck me recently when I thought more about the various usability testing methods available to us and realized some offer us information about users without making us better designers, mostly because the information isn’t as rich, visceral, or immediate as with other methods.