The flaw of averages

The flaw of averages defined in a book of the same name by Sam L. Savage, states in effect, any plan based on average assumptions is wrong on average! http://www.flawofaverages.com/

However, every duration estimate, cost estimate, risk impact and other estimate our project plans are based on an ‘average’ or ‘expected value’ derived from past experience. And as naturalist Stephen Jay Gould commented, our culture encodes a strong bias either to neglect or ignore variation. We tend to focus instead on measures of central tendency, and as a result we make some terrible mistakes, often with considerable practical import.

The flaw of averages ensures plans based on a single average value that describes an uncertainty will be behind schedule and over budget! A typical example from the book looks at a stocking problem – the business is planning to import short shelf life exotic fruits with a high profit margin, the marketing team have analysed the market and developed a profile of likely sales. The boss looks at the distribution and demands a single figure. All the marketing team can do is take the ‘average’ expected sales and decide 500 cases per month are the most likely level of sales. Based on a profit of $100 per case the boss predicts a net profit of $50,000 per month. However, this is a very optimistic estimate, if less than 500 cases are sold, the fruits will spoil with losses of $50 per case, if more than 500 cases are required the cost of airfreighting extra cases is $150 per case resulting in a loss of $50 or the sales have to be foregone with a risk of losing the customer.

The highest possible monthly profit is $50,000 – if more or less are sold the profit reduces. On average each month more or less than 500 cases will be sold, resulting in returns lower than the estimated $50,000. The only time the predicted profit will be realised in the occasional month when exactly 500 cases are sold.

Even if the company decides not to airfreight additional cases on average the monthly profit will be less than $50,000. Without airfreight, for roughly half the time demand will exceed 500 cases but with no additional stock, profit is capped at $50,000. For the other months, sales will be less than 500 and there will be spoilage costs. Meaning on average, the monthly profit will be less than predicted!

The average is correct, the way the manger is using the average is the ‘flaw’. The same problem shown in the cartoon above, ‘on average’ the pond is only 1 meter (3ft) deep! But averages are rarely what is needed for prudent management.

To properly analyse the projected profits more in-depth analysis is needed, using techniques such as Monte Carlo analysis with the variability of sales being represented by the input probability distribution, the costs and income expected modelled in the tool and the resulting profits predicted in the output probability distribution.

The challenge is getting valid data to model. Projects are by definition ‘unique endeavours’ which means there is no pool of directly valid data; this problem is discussed in our paper The Meaning of Risk in an Uncertain World . When managing project uncertainties our basic data is uncertain!

Recognising this simple fact is a major step towards better project management. To quote George Box (Stamford University) ‘All models are wrong, some models are useful’. No model should be taken as correct, this includes schedules, cost plans, profit predictions, risk simulations and every other predictive model we use! They are never complete representations of exactly what will occur, but a successful model will tell you things you did not tell it to tell you (Jerry P. Brashear).

Building a successful model such as a useful schedule (useful schedules are useful because they are used) should go through the five stages defined by Donald Knuth:
1. Decide what you want the model to do
2. Decide how to build the model
3. Build the model
4. Debug the model
5. Trash stages 1 through 4 now you know what you really want.

And to get a large model to work, you must start with a small model that works, not a large model that does not work. If you want to understand flight what is more useful, a large highly detailed model of a Boeing Jumbo jet built out of Lego blocks that cannot fly or a simple paper aeroplane that does?

The complex Lego model may be visually impressive but is likely to be less useful in understanding a dynamic process such as flight.

The same is likely to be true for most dynamic project models. Edward Tufte says ‘Clear and precise seeing becomes as one with clear and precise thinking’, and John W. Tukey adds ‘It is far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made precise.’ It is dumb to be too smart!

These concepts are consistent with the PMBOK® Guide idea of ‘progressive elaboration’ and are embedded in the scheduling technique called ‘Schedule Density’ where the initial schedule is developed at ‘Low Density’ and additional detail added as needed (see more on Schedule Density).

The message from this blog is building a useful model is a skilled art, regardless of the subject being modelled (time, cost, risk). A good start is to keep the model simple, if you don’t understand how the model works how will you will be able to judge what it shows you? The model is never the truth; at best it is a useful! And its usefulness will be severely reduced if you rely on averages such as single point estimates without at least using some probability analysis. Melding the need for precision with probabilistic assessments are discussed in our paper Why Critical Path Scheduling (CPM) is Wildly Optimistic.

Whilst this post has focused on one dimension of uncertainty (time and schedule), the principles can be applied to any area of uncertainty.