in Evolve

Woulda, shoulda, coulda. Didn’t. (The Failure to Beta Test)

Monitor110 was a business/site that tried to filter information for institutional investors. This post mortem from a founder probably won’t reveal any new lessons, but it’s always powerful to see theory — in this case the value of the beta release — played out in the form of failure…

…By mid-2005 the system worked, but spam was becoming more prevalent and caused the matching results to deteriorate, e.g., too much junk clogging the output. Around the same time we started to dig into natural language processing and the statistical processing of text, thinking that this might be a better way to address the spam issue and to get more targeted, relevant results. This prompted us to not push version 1.0, instead wanting to see if we could come up with a more powerful release using NLP to mark the kick-off. In retrospect, this was a big mistake. Mistake #5, to be precise. We should have gotten it out there, been kicked in the head by tough customers, and iterated like crazy to address their needs. Woulda, shoulda, coulda. Didn’t.

We talked about “release early/release often,” but were scared of looking like idiots in front of major Wall Street and hedge fund clients.