Category: Process

• User Research

  1/5/2004

  ---

• One-Question Survey

  The One Number You Need to Grow by Frederick F. Reichheld is a great, short article on using one-question surveys that measure loyalty correlated with customer behavior. Highlights:

  
  

  ‘Every month, Enterprise polled its customers using just two simple questions, one about the quality of their rental experience and the other about the likelihood that they would rent from the company again. Because the process was so simple, it was fast. That allowed the company to publish ranked results for its 5,000 U.S. branches within days. …the company counted only the customers who gave the experience the highest possible rating…By concentrating solely on those most enthusiastic about their rental experience, the company could focus on a key driver of profitable growth: customers who not only return to rent again but also recommend Enterprise to their friends.’

  ‘In most of the industries that I studied, the percentage of customers who were enthusiastic enough to refer a friend or colleague — perhaps the strongest sign of customer loyalty — correlated directly with differences in growth rates among competitors.’

  ‘Companies have tended to focus on customer retention rates, but that measurement is merely the best of a mediocre lot…they basically track customer defections’

  ‘For a while, it seemed as though information technology would provide a means to accurately measure loyalty. Sophisticated customer-relationship-management systems promised to help firms track customer behavior in real time. But the successes thus far have been limited to select industries, such as credit cards or grocery stores, where purchases are so frequent that changes in customer loyalty can be quickly spotted and acted on. ‘ – Behavior is difficult to study and quantify, lots of data might help.

  ‘My personal bet for the top question (probably reflecting the focus of my research on employee loyalty in recent years) would have been "How strongly do you agree that [company X] deserves your loyalty?" Clearly, though, the abstract concept of loyalty was less compelling to customers than what may be the ultimate act of loyalty, a recommendation to a friend.’ – This question raises the same kind of emotion that occurs during real behavior, and might help explain why it’s a better indicator of behavior.

  
  

  I like how he used a 10-point scale then clustered the results into three types of customers, avoiding “grade inflation.”

  Also see the Loyalty Acid Test.

  1/5/2004

  ---

• How Designers Follow Constraints

  Notes on Web Site Designs: Influences of Designers Experience and Design Constraints (PDF) by Aline Chevalier and Melody Y. Ivory, which ‘demonstrates that the designers’ levels of expertise (novice and professional) as well as the design constraints that clients prescribe influences both the number and the nature of constraints designers articulate and respect in their web site designs.’ It’s part of the WebTango project at the iSchool, University of Washington.

  ‘We assert that understanding designers’ activities and identifying difficulties they encounter are essential to improving web site quality.’

  ‘we found studies showing that constraints are extremely important for understanding and for solving a design problem’

  ‘there is a wide gap between designers’ articulation of constraints and designers’ effective implementation of them.’

  Those pesky designers! Seriously, reading this I feel like we could do a better job making constraints explicit in our personas and scenarios. Most I see are filled with a lot of nice details on our fictional character meant to make them more real but doesn’t add anything to the design process.

  Also, I feel like we need something in-between the personas/scenarios and the design, an interaction model. More on this in a future post.

  Even the experienced designers could only satisfy 75% of the constraints they were given. While they achieved up to 95% of the client constraints, they couldn’t satisfy more than half of the user constraints.

  ‘professional designers in the condition without constraints were able to infer client constraints, because they had contextual knowledge acquired through experience (stored as mental schemata)’

  ‘Results from the first two studies show that professional and novice designers encounter difficulties in effectively considering users’ needs during the design process, even though they focus mainly on users’ needs during the evaluation process.’

  ‘we argue that heuristic evaluation with ergonomic criteria suggested by Nielsen (2000) has not been adapted for web site designers (who have no human factors knowledge), because the ergonomic criteria are both too abstract and too numerous. Our hypothesis is that it would be more effective to provide designers with a subset of ergonomic constraints that respect the users’ real needs.’

  Absolutely. There are simply too many guidelines to follow these days. There has to be a way of winnowing them down. Design patterns might help. A better design method might help.

  Results: ‘1. Help novice designers to consider both user and client constraints. 2. Help professional designers to focus more so on user than client constraints or at least help them to strike a balance between the two actors. 3. Help designers, regardless of their levels of expertise, to consider and implement ergonomic constraints in their sketches.’

  For the first two points, we suggest developing a knowledge-based system that fits the designer’s level of expertise (see Fischer et al., 1991). Specifically, the system should provide the following support:

  
  

  • The system should help novice designers to identify constraints that need to be respected in the web site design.
  • This system should also help novice designers to generate new constraints, through a design step oriented on the expectations of the client and the users. The system could help designers determine, based upon the current state of the design activity, additional information the designer may need to consider. For example, the system could propose questions for novice designers to ask the client.
  • The system should help professional designers deal with a client who has many expectations, in particular, to help the designer consider more user constraints. For example, the system could suggest relevant constraints that the designer did not consider.

  As solutions they suggest a focused questionnaire designers could use to evaluate designs through the design process, or an automated tool to evaluate the design. Both are probably helpful, but to truly advance I think we need to improve the method itself, not just devise better ways to find design flaws.

  Thank you Aline and Melody.

  11/11/2003

  ---

• Tests Well With Others

  In this brave new world where user-centered designers meet old-school info technologists, we can all live, love, and test together. Here’s how my usability testing jives with their testing, as I understand it:

  What types of testing do we do?

  
  

  • QA: technical check to make sure it works
  • UAT (user acceptance testing): ensure the system meets the business requirements
  • Usability: testing with end users

  Who does it?

  
  

  • QA: technical staff, but not the people who programmed it
  • UAT: business staff. Originated in days when business people threw requirements over the wall to tech and needed to test what came back over the wall. In agency models it’s often rolled into design.
  • Usability: dedicated usability peeps or the design staff, but preferably not the people who designed it

  11/5/2003

  ---

• IDEO’s Method Cards

  I received my IDEO Method Cards, and they’re big, twice as big as your usual playing cards. The writing is good; short and sweet with funky photos on the reverse. The content isn’t earth shattering – each features a user research technique – but the format is quite handy and sure to stir up some new ideas at work.

  

  A coup for IDEO is how – as in this Fast Company article – they position what could be considered an advertisement as a confident revelation of their methodology. Clever.

  10/6/2003

  ---

• Web Practices and Decentralized Companies

  Developing Best Practices for Distributed Networks of Sites: Heuristics, Design, and Politics (PDF) by Jeffrey Veen of Adaptive Path and Carolyn Gibson Smith of PBS sets a great example of improving web design and encouraging certain practices across a large, decentralized organization. One particular aspect I like is that by distributing templates and building examples they offer the affiliates carrots rather than attempt a futile effort of waving sticks, which often fall under the auspices of governance or compliance in contemporary corporate environments. Even when there is top-down authority to enforce these standards, it’s rarely done happily.

  The process used heuristic analysis, stakeholder interviews, and UCD to…

  • Examine the issues affiliates faced
  • Find internal best practices
  • Build prototype using best practices
  • Test prototype

  They delivered

  • Printed best practices report
  • Working prototype
  • Templates and code
  • Guide for conducting usability tests
  • Case studies
  • System-wide report card

  ‘We’ve heard from Web teams that they are using the recommendations not only to build their sites, but as justification for funding. The report is being used as ammunition for designers and developers as they seek to convince their internal stakeholders of the value of a strong Web presence.‘

  9/19/2003

  ---

• The Goal of Usability Testing

  What is the goal of usability testing? Let’s say it’s to improve the user’s experience.

  The user’s experience will improve when designers have the skills and resources to design well.

  The designers will improve when they have a better understanding of the users.

  Usability testing offers us some understanding of users. The direct goal of usability testing is to yield this information, but the indirect goal of educating designers is ultimately more important (until someone devises a test that automatically tweaks the product without a designer, which won’t be anytime soon).

  This struck me recently when I thought more about the various usability testing methods available to us and realized some offer us information about users without making us better designers, mostly because the information isn’t as rich, visceral, or immediate as with other methods.

  8/7/2003

  ---

• Usability Testing

  8/7/2003

  ---

• We’re All Out In The World

  In Tekka, Cathy Marshall addresses a topic that’s been stirring in my mind lately…

  ‘…do scenarios and personas actually help, or do they just create a warm illusion of user-centered design?‘

  She spends some time illustrating how products can drive personas…

  ‘Want to make personal security and encryption a necessary feature for [our persona’s] email application? Let’s just give her a secret life in which she’s rekindled a dormant college romance via email.’

  But we know that’s not a problem with personas, that’s just people creating personas improperly (personas, unlike guns, don’t shoot unsuspecting children, it requires a designer to do that). Research drives personas, and personas drive design, and design leads to products. Reversing that is incorrect practice, not a flaw in the practice.

  I’ve also seen a tendency to focus the scenario too much on how a persona uses the website to the exclusion of other activities. When Marshall writes, ‘Notice, for example, that people use other communication channels besides their computers‘ it reminds me that personas should capture how people arrived at the site, what other sites they’re looking at, and what they do when they leave.

  We know that to do this work properly one needs empathy with the users. And yet, people are, as George Bernard Shaw observed, bloody apes. People, some of them, can be nasty, messy complainers and it’s easier to just not interact with them. And this is why user-centered design is such a hard sell, because business people don’t want to deal with their customers. It’s easier to keep them at arms length – or further – with surveys and other market research. Personas and scenarios too help us keep people at arms length. Of course, they assume we’ve done our contextual inquiry yada yada beforehand, but that shouldn’t stop with the personas. In the tradition of the best industrial design it doesn’t stop, designers are always working with users. Researching and testing is ingrained in the process, not separate steps in the project plan. Marshall says,

  ‘Don’t just use scenarios and personas. Pay attention. Observe. Reflect. We’re all out in the world we design for.‘

  Which, to me, implies participatory design. It means working more closely with those bloody apes that drive us crazy. This is the hard work we have to do to arrive at a great product. It means I don’t believe personas can substitute for interacting with users during post-research project phases. I don’t know a lot about participatory design yet, but I think it’s time to learn.

  7/23/2003

  ---

• Personas & Scenarios

  7/23/2003

  ---

• RUP and UCD

  Dave Cronin’s new Cooper article, RUP & Goal-Directed Design: Toward a New Development Process is a great look at the intersection of conventional user-centered design practice and systems-focused processes. Although he focuses on the Rational Unified Process, the ideas can be applied to any big software development project, and is helpful in understanding and communicating with developers.

  He also hits one of the practices that currently drives me insane, gathering requirements…

  One problem in effectively defining high-level requirements up front is that many development organizations understand requirements as something to be “gathered.” This “gathering” activity often manifests itself as customers and users volleying “requirements” emails and phone calls to the help desk, salespeople, and engineers. Often, these are then translated into line items in requirements documents. There is little thought given to reconciling requirements from different sources, and this reconciliation is often done at construction time, which can have all sorts of dire consequences, ranging from confused developers (who rarely have enough information to make educated choices between two apparently reasonable requests) to a confused interface (that is not grounded in a coherent and cohesive product definition) to confused users (who wonder why a function they never use is front-and-center on the screen).

  Learning more about RUP recently, I realize I had some of it thrust on me before. Back at Razorfish a friend sneakily introduced aspects of RUP into our usual process while planning Tolerance.org. While activities like rating risks can be done anytime, we broke up the sections of the site into use cases. For many websites this approach wouldn’t work, as sites often need to work as a coherent whole, with changes in one section effecting other sections. In our case the sections differed significantly by subject matter and audience, so it wasn’t too harmful. I’m about to embark on another such situation, we’ll see what happens.

  7/16/2003

  ---

• Business, Design, and Time

  Revisiting one of Jesse’s elements diagrams I’m thinking about how the addition of a ‘length’ column (how long the layer requires) would generate some interesting outcomes. For example, significant business decisions can be made and ripple through an organization faster than upper layers, particularly Scope (Requirements, Specs) and Structure (IA, ID). A problem arises where a product can take longer to design and create than the span of the business environment the product was meant to serve. When business moves this fast, design must as well, limiting what can be built, or at least how.

  Incidentally, I disagree a bit with that use of the term ‘strategy,’ preferring Michael Porter’s clarification of strategy and business effectiveness.

  More incidentally, the summary on MSN Search for PeterMe.com says, ‘Web designer Peter Merholz offers some brain droppings for hungry designing surfers.’ Droppings? Hungry? Talk about your mixed metaphors. And what is a ‘designing surfer’ anyway?

  12/9/2002

  ---

• Mindset List

  You’ve probably seen the Mindset List, published each year by Beloit College. It goes like this…

  Most students entering college this fall were born in 1984.

  • A Southerner has always been President of the United States
  • Cars have always had eye-level rear stop lights, CD players, and air bags.
  • George Foreman has always been a barbecue grill salesman
  • …

  It’s always struck me as a nice little design document, grounding the instructors in the mindset of their students. What’s more, the items chosen appeal to people of different generations. For example, ‘Women have always been members of the Jaycees‘ didn’t resonate with me, but probably does for older Americans.

  I’m thinking about how to use this with my clients.

  Link courtesy Chris Pepper.

  10/8/2002

  ---

• Tire Swing Cartoon

  If a digital design team designed a tire swing. Excellent.

  Link courtesy of xblog.

  7/12/2002

  ---

• Design Early, Design Often

  In the latest issue of New Architect Alan Cooper espouses his usual philosophy: ‘Simply put, there is no downside to designing before coding.’

  In the same issue, a member of the Mozilla QA team advises, ‘Release early, release often, and let your customers bang on it.’

  Certainly two different approaches, but both valid I think. They could be seen as compatible in a couple ways:

  1. different approaches for different products: only some customers will tolerate being part of the QA process
  2. different stages of the same product: you could do all the design first and then take a ‘release early, release often’ approach to development.

  Although this last approach implies the dev team needs rapid feedback because sufficient cycles weren’t built into the design process. However, I could imagine a product that, even after a Cooper-style design stage, still needed to draw out tricky technology implementation issues.

  7/10/2002

  ---