Abolishing point estimates

One of my major gripes in my classes is that students put too much reliance on point estimates. Of course, when they (or other business students) graduate, the error is propagated into business.

Saying the company will generate $3,422.13 in revenues for the first month of a new product is useless information. No one knows what will happen in the future: the economy, reception of a new product, the reaction of competitors. People make assumptions — which are only assumptions, i.e. a SWAG or maybe just a WAG. So when I see a point estimate — to four significant digits — then I know all subsequent calculations are GIGO.

Saying that we expect sales to be between $500 and $5000 is more useful. It would be nice to say there’s a 95% chance (two standard deviations) that the results will be in that range, but for most analyses, that’s GIGO.

A common solution is to offer three cases, e.g. when startups offer “best case,” “worst case” and the median or average case projections to VCs. This is certainly better than nothing, because it forces you (for example) to think about what the appropriate level of hiring or advertising spending for each one.

The formal answer to this problem is scenario planning. I was reminded of this in checking out Adam Hartung (who posted a recent comment to my blog). In a posting on his own blog last week, he discusses the importance of scenario planning. In particular, he points out that many businesses (or individuals) made assumptions about energy prices without considering other plausible scenarios:

Over the last year the price of energy was one such big theme that interested a lot of people. But most only explored one scenario ─ what if oil prices went to $200 or $250? Interesting, but not sufficient. While that scenario is worth investigating in great detail, it's also important to investigate other options ─ like oil at $150, or $100 or $65 or $35. All of those have different implications. What's important in scenario planning is to investigate them all.

I have sometimes taught scenario planning in MBA technology strategy class. Although the material doesn’t really fit with the other readings, managing high-tech companies is inherently about dealing with an uncertain future.

Since the existing materials are not very good, at some point I guess I’ll have to write my own. However, blogger/consultant Martin Börjesson has a starter list of scenario planning resources.

Interestingly, everyone (including Börjesson) who teaches scenario planning goes back to the use of scenario planning by Royal Dutch Shell. This does make some sense, beyond the fact that Shell is famous for its use of the technique. Oil companies face a highly uncertain future — in terms of the supply and demand for oil, as well as the degree of environmental regulation (or societal pressure to self-regulate). They also have to make capital investments that last for decades and are planned years in advance.

Perhaps we can interest some renewable energy firms in also doing scenario planning. When they’re going for grid parity, they need to consider what the existing energy prices will be — which might be oil at $50 a gallon or $200 a gallon.

Oddly, Shell CEO Jeroen van der Veer gave a speech in February where he made point predictions as to energy demand, the availability of conventional energy, and the cost of renewable sources. The scenario alternatives he presented (in February and earlier in January) were not about the economic or technical future, but about the policy choices made by energy-consuming nations.