Expanded bashing of WebCriteria (see original earlier in month):
I feel compelled to offer the contrarian view because this product scares me! Our usability tests are done with demographically targeted individuals, but WebCriteria uses one agent (“Max”) that falls short of even a generic human. Max doesn’t understand content or is influenced by it, he has a perfect memory, he doesn’t back track, and he doesn’t know how to perform a search.
WebCriteria’s objective measures, like load time, are certainly useful. But other measures like accessibility are too cookie cutter for my taste. They don’t take into consideration, for example, visual ways to direct the user’s attention.
Notice the subtle Jedi mind trick in their documentation:
“The WebCriteria Accessibility measurement does not consider all the factors that affect ease of navigation in a web site. Obviously the user will be affected by compelling content, clear wording and offers of free cars. Nevertheless Accessibility is an effective measure of the effect of the structure of the web site on ease of navigation.”
Also, their “freshness” measurement basically says new = good. What if we’re dealing with classic content that shouldn’t or couldn’t change? While we could provide disclaimers explaining the limited usefulness of WebCriteria’s results, I’m afraid clients will forget this, instead feeling justified in making decisions based on hard cold statistics.