Getting Unlost...

I've recently begun watching the first two seasons of Lost on DVD -- and have found some striking similarities between the TV show and launching an Internet site.  The TV show tells the story of the survivors of Oceanic flight 815 -- and the bizarre island they land on.  Gradually, over a period of weeks, the survivors learn about their environment and discover some of the secrets of the island and the mysterious Dharma initiative.  This knowledge is not gathered easily -- their trial-and-error results in a lot of false conclusions and costly mistakes.

Most post-launch startups are also lost.  They don't fully understand the environment they are operating in -- or how to improve/impact it.  Things don't behave as the founders initially expected or logic might dictate.  With all of the data exhaust a website can generate, it's very hard to differentiate the signal from the noise (ie, determine which metrics matter and which don't).  Big functionality deployments can have no real impact on the numbers, yet sometimes small actions like pressing (or not pressing) a button at a specified time, have a major effect. 

Munjal Shah, the CEO of Like.com (Riya), has resumed blogging about his startup's journey.  And his current episode (Number 23) does a wonderful job of describing the lack of knowledge a company initially has about their site -- and the process they must go through to gain understanding.  It brings back many vivid memories from my days at Half.com.  Trying to isolate cause and effect in an online environment is not as easy as it might sound.  It takes a lot of effort, testing and time to begin to understand the levers at your disposal -- and the impact each lever has on your key metrics.

Munjal and team have begun to build a core culture and competency around A/B testing (otherwise known as multivariate testing).    I'm impressed to see them get A/B religion so early in their development - it takes many companies much longer to adopt this approach.  I think that this type of disciplined, statistical approach towards site improvement is a critical component in metric improvement.  Munjal's post is definitely worth the read...

Comments

[Bogus Jones]

I really enjoyed the analogy of Lost to launching an Internet site. For your musings, as a fellow Lost fan, you may enjoy the humorous Lost Theory Generator @ http://www.lost-theory-generator.com. The Lost theory I'm currently latched onto is one in which the Island is not a far away place in terms of distance, but a far away place in terms of time (i.e. it's in the future).

I do wonder if your analogy could be thickened with commentary on the players. As one of a 3-member team who has launched an internet/mobile service, I was truly amused with your description of pressing buttons and launching features without really understanding the specific impact on numbers (traffic, retention, "catch and release" vs. "catch and hold", etc.) and separating the signal from the noise. However, the thickened analogy I am posing is the relationship between those of us who crash-land on the island of entrepreneurship and the 'others' who host this place and provide the funding. I often wonder if I'm dealing with Ben or Juliet when I'm speaking with a potential investor. Are the questions a trap?...are the promises real?...are we really gaining an understanding? Where are THEY headed and can they help ME? Seeing THAT forest beyond the trees can easily leave me hanging by the branches. Then, when I'm not dealing with the others I'm stumbling through the jungle running into the carnage and clues of past visitors (plane wrecks, hatches, etc.)...what lessons can I learn from those who failed to survive this island before me? Certainly I should take the hatches more seriously than the crashed biplanes...or should I?. Lastly, I'm in constant personal review of what works and what doesn't work when I communicate with the others. Should I be a friendly Hurley, a defiant Kate, or an aggressive Sawyer. Should I trade my principles to make-nice with a mysterious adversary (as Jack has done with Juliet)? How much of my past story really matters to how the others judge me. The past seems so important on this Island. The past has provided the others with their greatest power -- control. They have control not because they know something about the future that we don't. They have control because they have learned something about the past that we have not yet learned for ourselves...and they use it to manipulate us. The others are a mystery to us...a mystery we aim constantly to understand. I most often find myself standing by my own convictions steadily pursuing my own 'journey'...like Locke. However, while he has found adventure with the others, we just don't know if that's good or bad for him. What a great show! Really enjoyed your article.