The Science of Startups

“The Lean Startup,” Eric Ries’ thoughtful new book for entrepreneurs, is a worthy attempt to bring the scientific method to the often intuitive exploration of young companies.

What leads most startups astray is the lack of a disciplined, empirical procedure for making decisions, he says.

Ries, who also write on the blog Startup Lessons Learned ( and is a 2010-11 entrepreneur-in-residence at Harvard Business School, is by equal measure upbeat and cautionary. He sees a worldwide renaissance of entrepreneurialism, but worries about wasted, misguided efforts.

Venture investors take heart. He has an answer.

“The nice thing about relying on human judgment and using the scientific method is [we develop] a system for training judgment to get better over time,” he says. “We will eventually start to develop better entrepreneurial instincts.”

Instincts are certainly a part of the Ries’ methodology. But so is a mathematical precision used to test assumptions, learn from mistakes and change products in response.

It’s a process he calls “Build, Measure, Learn.”

So who will benefit from “The Lean Startup”? Entrepreneurs and venture investors, obviously. But Ries likes to tell the story of an LP at a university endowment who asked if there was a way to know if the hot startup a general partner was talking about as he laid out a pitch for a new fund was the real thing. The company is all over the press, but will it be dead in six months?

The answer might lie in a slim treatise called “The Lean Startup.”

VCJ had the chance to talk with Ries recently. Here is an edited transcript of the conversation:

Q: Are we more entrepreneurial than in the past?


My belief is there are more startups operating today than at anytime in history. There’s this worldwide entrepreneurial renaissance. But I think it’s not going to come to a good end unless we get serious about systematically improving our entrepreneurial practice. I think there is both greater opportunity and greater waste.

Q: “Build, Measure, Learn” is one of the key messages in “The Lean Startup.” Isn’t that what startups do today?


That is one of the things I think is missing in most of the startups I meet with today. When I meet with most entrepreneurial teams, I ask them a simple question: How do you know that you’re making progress? Most of them really can’t answer that question.

Q: If you could give these entrepreneurs one key piece of advice, what would it be?


I would say, as an entrepreneur everything you do, every action you take in product development, in marketing, every conversation you have, everything you do, is an experiment.

If you can conceptualize your work not as building features, not as launching campaigns, but as running experiments, you can get radically more done with less effort.

Q: So how should an entrepreneur get started?


What I encourage entrepreneurs to do is to try to put their assumptions to the test immediately rather than doing too much analysis.

Instead of doing a huge in-depth analysis of the market and customer focus groups and that kind of thing, let’s conduct a first experiment as quickly as possible.

Q: You caution in your book against relying on vanity metrics. What is a vanity metric?

A: Vanity metrics are the numbers you put in your press release to make your competitors feel bad. Millions of messages sent. The 3 million page views you had today. They actually don’t tell you very much about how the company is really doing.

If I have 100 million messages on my platform it could be 100 million people trying the product once and churning out. Or it could be one guy with a really active Web browser. How do I know which?

Q: So what metric should you monitor?


A metric is actionable if you look at it and can figure out what action you need to take to get more of [something] to happen. If I tell you we had 100,000 more website hits this month than last month, I didn’t really tell you what to do. Was it from the press release we put out, a better marketing campaign, or was it just a seasonal factor? You have to go down to further reports to figure out which is which.

Q: Give me an example of what you mean?


If I show you an A/B test … with 100 customers who saw a version of my product with a new feature and 100 customers who didn’t, and the 100 customers who saw this new feature generated more revenue per user for the company, that’s an actionable metric.

Q: You develop a concept in the book called “innovation accounting.” What is innovation accounting?


This is probably the most controversial idea in the whole book. My belief is that the current management tools that were developed in the 20th century are all really based in a very specific accounting paradigm. They have to do with forecasting plans. The idea is that you make forecasts about what is suppose to happen.

Q: And?


All that forecasting and accounting is based on having a long and stable operating history. Startups don’t have that operating history. They don’t have the data necessary to make good forecasts. That is at the heart of the reason why startups fail all the time.

Q: Give me an example of how to use innovation accounting. Let’s say I’m an Internet startup that has raised $1 million in venture capital and has a 1-year goal of obtaining 1 million customers.


What we want to do is find a quantitative basis to believe that we actually have learned something important. As we go along over the course of the year, instead of looking at the gross numbers, like how many millions of customers we have, we ask: what are the inputs to the business model? What’s the revenue per customers, what’s the per customer conversion rate, what’s the engagement rate of the early customers? Once we have that kind of base line, we can tell over time whether we are making the product better. You can see quantitatively that we’ve been able to change customer behavior for the better.

Q: Can you go a layer deeper than that?


Take a viral company. For a viral company, success is generally defined as viral coefficient greater than or equal to 1.0. This means that for every customer, on average, one more comes in. If we’re at 0.7, we’re not there yet. But if we’ve seen this really nice traction with the number going up each time we’ve change the product, that’s an indication we’re making progress.

Q: Another key concept in the book is the notion of pivoting, or changing direction. What is a pivot?


A pivot is a change in strategy without a change in vision.

Q: How do you know when or in what direction to pivot?


That really comes down to a matter of judgment, which is so frustrating. The truth is that for every team I’ve known that has had to go through a painful pivot, the hard part has been actually deciding to pivot. It’s not deciding what to pivot into. What happens is we get so attached to the thing we’ve already built. It blinds us to what is really going on.

Q: Startups have lots of assumptions to test. How do they know which one to focus on?


All you have to do is focus on the riskiest assumption first. The bad news is determining which assumption is riskiest is really a matter of judgment. There’s no formula.

In most situations the most risky assumption is just does anybody want this thing at all, does it have any value to anybody.