The reference case for management reserves

Risk management and Earned Value practitioners, and a range of standards, advocate the inclusion of contingencies in the project baseline to compensate for defined risk events. The contingency may (should) include an appropriate allowance for variability in the estimates modelled using Monte Carlo or similar; these are the ‘known unknowns’.  They also advocate creating a management reserve that should be held outside of the project baseline, but within the overall budget to protect the performing organisation from the effects of ‘unknown unknowns’.  Following these guidelines, the components of a typical project budget are shown below.

PMBOK® Guide Figure 7-8

The calculations of contingency reserves should be incorporated into an effective estimating process to determine an appropriate cost estimate for the project[1]. The application of appropriate tools and techniques supported by skilled judgement can arrive at a predictable cost estimate which in turn becomes the cost baseline once the project is approved. The included contingencies are held within the project and are accessed by the project management team through normal risk management processes. In summary, good cost estimating[2] is a well understood (if not always well executed) practice, that combines art and science, and includes the calculation of appropriate contingencies. Setting an appropriate management reserve is an altogether different problem.

 

Setting a realistic management reserve

Management reserves are an amount of money held outside of the project baseline to ‘protect the performing organisation’ against unexpected cost overruns. The reserves should be designed to compensate for two primary factors.  The first are genuine ‘black swans’ the other is estimating errors (including underestimating the levels of contingency needed).

The definition of a ‘black swan’ event is a significant unpredicted and unpredictable event[3].  In his book of the same name, N.N. Taleb defines ‘Black Swans’ as having three distinct characteristics: they are unexpected and unpredictable outliers, they have extreme impacts, and they appear obvious after they have happened. The primary defence against ‘black swans’ is organisational resilience rather than budget allowances but there is nothing wrong with including an allowance for these impacts.

Estimating errors leading to a low-cost baseline, on the other hand, are both normal and predictable; there are several different drivers for this phenomenon most innate to the human condition. The factors leading to the routine underestimating of costs and delivery times, and the over estimating of benefits to be realised, can be explained in terms of optimism bias and strategic misrepresentation.  The resulting inaccurate estimates of project costs, benefits, and other impacts are major source of uncertainty in project management – the occurrence is predictable and normal, the degree of error is the unknown variable leading to risk.

The way to manage this component of the management reserves is through the application of reference class forecasting which enhances the accuracy of the budget estimates by basing forecasts on actual performance in a reference class of comparable projects. This approach bypasses both optimism bias and strategic misrepresentation.

Reference class forecasting is based on theories of decision-making in situations of uncertainty and promises more accuracy in forecasts by taking an ‘outside view’ of the projects being estimated. Conventional estimating takes an ‘inside view’ based on the elements of the project being estimated – the project team assesses the elements that make up the project and determine a cost. This ‘inside’ process is essential, but on its own insufficient to achieve a realistic budget. The ‘outside’ view adds to the base estimate based on knowledge about the actual performance of a reference class of comparable projects and resolves to a percentage markup to be added to the estimated price to arrive at a realistic budget.  This addition should be used to assess the value of the project (with a corresponding discounting of benefits) during the selection/investment decision making processes[4], and logically should be held in management reserves.

Overcoming bias by simply hoping for an improvement in the estimating practice is not an effective strategy!  Prof. Bent Flyvbjerg’s 2006 paper ‘From Nobel Prize to Project Management: Getting Risks Right[5]’ looked at 70 years of data.  He found: Forecasts of cost, demand, and other impacts of planned projects have remained constantly and remarkably inaccurate for decades. No improvement in forecasting accuracy seems to have taken place, despite all claims of improved forecasting models, better data, etc.  For transportation infrastructure projects, inaccuracy in cost forecasts in constant prices is on average 44.7% for rail, 33.8% for bridges and tunnels, and 20.4% for roads.

The consistency of the error and the bias towards significant underestimating of costs (and a corresponding overestimate of benefits) suggest the root causes of the inaccuracies are psychological and political rather than technical – technical errors should average towards ‘zero’ (plusses balancing out minuses) and should improve over time as industry becomes more capable, whereas there is no imperative for psychological or political factors to change:

  • Psychological explanations can account for inaccuracy in terms of optimism bias; that is, a cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience[6].
  • Political factors can explain inaccuracy in terms of strategic misrepresentation. When forecasting the outcomes of projects, managers deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that their project will gain approval and funding either ahead of competitors in a portfolio assessment process or by avoiding being perceived as ‘too expensive’ in a public forum – this tendency particularly affects mega-projects such as bids for hosting Olympic Games.

 

Optimism Bias

Reference class forecasting was originally developed to compensate for the type of cognitive bias that Kahneman and Tversky found in their work on decision-making under uncertainty, which won Kahneman the 2002 Nobel Prize in economics[7]. They demonstrated that:

  • Errors of judgment are often systematic and predictable rather than random.
  • Many errors of judgment are shared by experts and laypeople alike.
  • The errors remain compelling even when one is fully aware of their nature.

Because awareness of a perceptual or cognitive bias does not by itself produce a more accurate perception of reality, any corrective process needs to allow for this.

 

Strategic Misrepresentation

When strategic misrepresentation is the main cause of inaccuracy, differences between estimated and actual costs and benefits are created by political and organisational pressures, typically to have a business case approved or a project accepted. Reference class forecasting will still improve accuracy, but the managers and estimators may not be interested in this outcome because the inaccuracy is deliberate. Biased forecasts serve their strategic purpose and overrides their commitment to accuracy and truth; the application of reference class forecasting needs strong support from the organisation’s overall governance functions.

 

Applying Reference Class Forecasting

Reference class forecasting does not try to forecast specific uncertain events that will affect a particular project, but instead places the project in a statistical distribution of outcomes from the class of reference projects.  For any particular project it requires the following three steps:

  1. Identification of a relevant reference class of past, similar projects. The reference class must be broad enough to be statistically meaningful, but narrow enough to be truly comparable with the specific project – good data is essential.
  2. Establishing a probability distribution for the selected reference class. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.
  3. Comparing the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

The UK government (Dept. of Treasury) were early users of reference class forecasting and continue its practice.  A study in 2002 by Mott MacDonald for Treasury found over the previous 20 years on government projects the average works duration was underestimated by 17%, CAPEX was underestimated by 47%, and OPEX was underestimated by 41%.  There was also a small shortfall in benefits realised.

 

This study fed into the updating of the Treasury’s ‘Green Book’ in 2003, which is still the standard reference in this area. The Treasury’s Supplementary Green Book Guidance: Optimism Bias[8] provides the recommended range of markups with a requirement for the ‘upper bound’ to be used in the first instance by project or program assessors.

These are very large markups to shift from an estimate to a likely cost and are related to the UK government’s estimating (ie, the client’s view), not the final contractors’ estimates – errors of this size would bankrupt most contractors.  However, Gartner and most other authorities routinely state project and programs overrun costs and time estimates (particularly internal projects and programs) and the reported ‘failure rates’ and overruns have remained relatively stable over extended periods.

 

Conclusion

Organisations can choose to treat each of their project failures as a ‘unique one-off’ occurrence (another manifestation of optimism bias) or learn from the past and develop their own framework for reference class forecasting. The markups don’t need to be included in the cost baseline (the project’s estimates are their estimates and they should attempt to deliver as promised); but they should be included in assessment process for approving projects and the management reserves held outside of the baseline to protect the organisation from the effects of both optimism bias and strategic misrepresentation.  As systems, and particularly business cases, improve the reference class adjustments should reduce but they are never likely to reduce to zero, optimism is an innate characteristic of most people and political pressures are a normal part of business.

If this post has sparked your interest, I recommend exploring the UK information to develop a process that works in your organisation: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

______________________

[1] For more on risk assessment see: http://www.mosaicprojects.com.au/WhitePapers/WP1015_Risk_Assessment.pdf

[2] For more on cost estimating see: http://www.mosaicprojects.com.au/WhitePapers/WP1051_Cost_Estimating.pdf

[3] For more on ‘black swans’ see: https://mosaicprojects.wordpress.com/2011/02/11/black-swan-risks/

[4] For more on portfolio management see: http://www.mosaicprojects.com.au/WhitePapers/WP1017_Portfolios.pdf

[5] Project Management Journal, August 2006.

[6] For more on the effects of bias see: http://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

[7] Kahneman, D. (1994). New challenges to the rationality assumption. Journal of Institutional and Theoretical
Economics, 150, 18–36.

[8] Green Book documents can be downloaded from: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

Phronesis – A key attribute for project managers

Phronesis (Ancient Greek: φρόνησις, phronēsis) is a type of wisdom described by Aristotle in his classic book Nicomachean Ethics. Phronesis or practical wisdom[1] is focused on working out the right way to do the right thing in a particular circumstance. Aristotle understood ethics as being less about establishing moral rules and following them and more about performing a social practice well; being a good friend, a good manager or a good statesman. This requires the ability to discern how or why to act virtuously and the encouragement of practical virtue and excellence of character in others.  But in a post-truth world, the ability to use ‘practical wisdom’ to discern what is real and what is ‘spin’ in rapidly becoming a key social and business skill. So prevalent is this trend, the Oxford English Dictionary named ‘post-truth’ its 2016 word of the year.

This problem pre-dates Donald Trump and ‘Brexit’, but seems to be getting worse. How can a project manager work out the right way to do the right thing in the particular circumstance of her project when much of the information being received is likely to be ‘spun’ for a particular effect.  There may be a solution in the writings of Bent Flyvbjerg.

Professor Bent Flyvbjerg, Chair of Major Programme Management at the Saïd Business School, Oxford University, has a strong interest in both megaproject management and phronesis. A consistent theme in his work has been the lack of truthfulness associated with the promotion of mega projects of all types, worldwide and the consequences of this deception. To help with the challenge of cutting through ‘spin’, and based on his research, he has published the following eight propositions:

1. Truth is context dependent.
2. The context of truth is power.
3. Power blurs the dividing line between truth and lies.
4. Lies and spin presented as truth is a principal strategy of those in power.
5. The greater the power, the less the truth.
6. Power has deeper historical roots than truth, which weakens truth.
7. Today, no power can avoid the issue of ‘speaking the truth’, unless it imposes silence and servitude. Herein lies the power of truth.
8. Truth will not be silenced.

There is, of course, a book, Rationality and Power: Democracy in Practice[2] that goes into more detail but just thinking through the propositions can help you apply the practical ethics that underpin phronesis.  Being virtuous is never easy, but regardless of the power brought to bear, sooner or later the truth will be heard.

The problem is which ‘truth’, understanding and perception will influence what people see, hear and believe to be the truth. Nietzsche, a German counter-Enlightenment thinker of the late 19th century, suggests that objective truth does not really exist; that objective absolute truth is an impossibility. The challenge we all face is the practical one of understanding enough about ourselves and others (we are all biased[3]) to achieve a reasonable level of understanding and then do our best to make the right decisions (see more on decision making), and to do the right thing in the right way.

___________________

[1] From Practical Wisdom, the right way to do the right things by Barry Schwartz and Kenneth Sharp.  Riverhead Books, New York 2010.

[2] See: https://www.amazon.com/Rationality-Power-Democracy-Practice-Morality/dp/0226254518/

[3] See,  The innate effect of Biashttp://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

The problem with project controls….

Preparing two presentations, one to the PMI Virtual Scheduling Conference last month, the other for ProjectChat in Sydney this week has started me thinking….  Why do so few project use effective project controls as a core input to project management decision making?

Most projects that have project controls staff seem to use them for forensics, claims and to meet client imposed obligations rather than as ‘trusted advisers’ to the project manager and project decision making teams.  Many more, probably a significant majority, don’t employ project controls staff at all and either rely on part time external agents or expect the project manager to do everything!  There are many and varied reasons for this I’m sure, but developing these two presentations suggests to me that at least one of the root causes for this situation is the simple fact project controls processes are ‘bolt-on’ extras rather then core elements in the overall information flow used to run the project.

For example the project schedule currently resides in a specialist tool (Primavera, Microsoft Project, etc.) that needs a trained expert to operate; the scheduling tool is not part of the work authorisation and allocation processes (at best schedule reports are used to inform these decisions) and it is not part of the progress recording system; update information is gathered separately after the event and reported even later. The scheduling functions are separate activities that may assist management but can equally be ignored by management, or can be not done at all. A foreman does not need a schedule to tell a work crew where to go next, he just has to point to the next work area he thinks needs attention, the instruction may be based on a schedule report, based on intuition or experience, or just a whim.

However, the world is changing ‘big data’ and integrated systems are becoming mainstream and these trends are starting to affect project management.  One example (used in the presentations) is the evolution of BIM (Building Information Modelling), 5D BIM integrates a 3D model of the project with time (4D) and cost (5D) information.  If this type of model becomes ‘fully integrated’ not only will time and cost information be part of the model, but the tools that process schedule, cost and earned value information will be integral to the model – ‘built-in’ rather than ‘bolted-on’.  And importantly the work crews use PDAs to access the model to understand what to do next (what, who, how and when all in the same place), and to record progress in real time.

When you read these presentations, don’t believe he BIM concept is limited to building projects – with very minor changes the BIM concepts can be applied to any ‘three dimensional’ engineering project.  And whilst ‘soft projects’ don’t typically operate in 3D the same approach can be applied to any system that can be mapped as a functional architecture or as a work flow.

The PMI presentation Projects controls using integrated data – the opportunities and challenges! looks at these concepts from the perspective of schedule controls, download from: http://www.mosaicprojects.com.au/PDF_Papers/P200_Projects_controls_using_integrated_data.pdf

The ProjectChat presentation Earned Value Management – Past, Present & Future looks at this from an Earned Value Management perspective. Download from: http://www.mosaicprojects.com.au/PDF_Papers/P201_EVM_Past_Present_Future.pdf

These ideas (or something similar) are in all probability coming to a project near you sometime soon.  And when they do the challenges outlined in the presentations will present grand opportunities for those open to change.

Read the presentations and let us know what do you think?? – if you are lucky enough to be at ProjectChat I will be here for the duration!

Two exceptional workshops at PGCS 2017 – 1st May

PGCS 2017 is offering workshops by Dr. Lynda Bourne and Dr. Keith Joiner in Canberra on Monday 1st May. Both offer a unique international viewpoint on very different aspects of project management.

My (Lynda’s) workshop ‘Leading Successful Teams’ focuses on collaborative teams which are key to success in any business activity. The most effective teams consist of individuals who can work independently on their own tasks, but also recognise the need to work collaboratively with other team members toward the activity’s goal and the organization’s success.

The leader of the team contributes significantly to team success through inspiring all team members to work together to achieve this goal, but must also intervene to reduce conflict and to motivate team members to continue to work collaboratively.

This session will focus on the needs of first-time team managers and will consist some theory, and a little practice, on the following topics:

–  Motivation

–  Delegation

–  Giving feedback

–  Resolving conflict.

This full day workshop is based on my Master’s course I’ve been running at EAN University in South America for the last 5 years and offers exceptional value at $450 (catering and GST included)

Keith’s workshop ‘methods for test design and analysis prescribed in U.S. Industry & Defence’ will introduce and illustrate the new methods in test design and analysis are, and how they are used to:

– screen for significant design factors;

– model design factors;

– screen for operational factors;

– model operational factors; and

– where equipment is taken off-the-shelf, improve the efficiency of validating performance.

Participants will use an instructional toy system and study several example uses to reinforce how the methods work.

This half day workshop is great value at $330 (Afternoon tea and GST included)

Both workshops offer exceptional value and are open to everyone – you do not need to attend the PGCS symposium to enjoy these process…… For more information and bookings see:  http://www.pgcs.org.au/program/workshops/

Two ‘Not-to-be missed’ Conferences in May

The PGCS program for 2017 is now complete and offers two overseas speakers as well as Professor Peter Shergold – author of the landmark project management report to government “Learning From Failure”. To see what’s on at PGCS in Canberra between the 1st and 4th May go to:  http://www.pgcs.org.au/program/

In the USA you can attend the Annual Project Management College of Scheduling Conference in Atlanta from May 7th to 10th.  They have a terrific program, with speakers and panel discussions, prepared to give everyone a chance to participate.  In addition, we’re planning something special every evening.  Sunday night is the vendor reception, Monday night is our Gala Dinner and Tuesday night is a night to explore Atlanta.  For more details see: http://www.pmcos.org/

New Articles posted to the Web #58

BeaverWe have been busy beavers updating the PM Knowledge Index on our website with White Papers and Articles.   Some of the more interesting uploaded during the last couple of weeks include:

And we continue to tweet a free PMI style of exam question every day for PMP, CAPM, and PMI-SP candidates: See today’s question and then click through for the answer and the Q&As from last week.

You are welcome to download and use the information under our Creative Commons licence

PMI Exam Scoring Information – Improvements are on the way.

For many people, the information currently provided by PMI on their exam performance has been less than useless. Being told you are ‘not proficient’, ‘moderately proficient’ or ‘proficient’; with the added helpful advice these terms mean ‘below average’, ‘average’ or ‘above average’ tells you nothing.  No one outside of the PMI enclaves has any idea what average means or how wide the average band is.  All you really know is you have passed or failed the exam.

The good news is after years of complaint, PMI has listened and will be rolling out a vastly improved Exam Results Report over the next few months.  The passing score and your actual score remain confidential to PMI for exam security reasons, but with this limitation, the new report will provide candidates with a much better understanding of their performance in relation to the examination pass level.

The headline report shows your overall performance with the performance by domain also categorised into one of the four groupings.  On its own, this is a vast improvement on the old report!!

Click here to see a sample of the Exam Report

However, of even more value, backing up this summary will be a detailed report highlighting your performance against the various domains and topics, accessible from within your CCR portal on the PMI website. Below is a preview of this part of the report (provided by PMI):

You still won’t know the exact number of questions in each domain or how they are divided into each of the Tasks within the ‘domain’, but the report will tell you where improvement in your knowledge will be valuable and help you plan your continuing development as a project professional. This additional information will also help training design and deliver better courses based on feedback from our clients all round a win-win-win development.

For once PMI is to be highly commended for listening to their members and delivering a great initiative.

Scheduled roll-out

Anyone taking the PMI-PBA®, PfMP® or PgMP® certification exam on or after 28 April will receive the new report and explanation pages. The new report will then be rolled out to the remaining certifications over the next few months, ending with the PMP® on 28 August. Key dates for the launch of the reports for courses we teach:

PMI-SP – 25th May
(for more see:  http://www.planning-controls.com.au/ )

CAPM – 22nd August
(for more see: http://www.mosaicproject.com.au/index.php?cID=175 )

PMP – 28th August
(for more see: http://www.mosaicproject.com.au/index.php?cID=173 )