Tag Archives: Project Controls

New White Paper on the value of TCPI

The To Complete Performance Index is one of the least understood metrics available as part of an EVM system which in recent months seems to have been given prominence in both the PMP and PMI-SP examinations.

Download this White Paper for a straightforward explanation of this long-established (but little understood) metric and its value as an indicator of project outcomes from: http://www.mosaicprojects.com.au/WhitePapers/WP1097_TCPI_in_EVM.pdf

To access all of our WPs and other published papers available for free downloads, start at: http://www.mosaicprojects.com.au/PM-Knowledge_Index.html

The reference case for management reserves

Risk management and Earned Value practitioners, and a range of standards, advocate the inclusion of contingencies in the project baseline to compensate for defined risk events. The contingency may (should) include an appropriate allowance for variability in the estimates modelled using Monte Carlo or similar; these are the ‘known unknowns’.  They also advocate creating a management reserve that should be held outside of the project baseline, but within the overall budget to protect the performing organisation from the effects of ‘unknown unknowns’.  Following these guidelines, the components of a typical project budget are shown below.

PMBOK® Guide Figure 7-8

The calculations of contingency reserves should be incorporated into an effective estimating process to determine an appropriate cost estimate for the project[1]. The application of appropriate tools and techniques supported by skilled judgement can arrive at a predictable cost estimate which in turn becomes the cost baseline once the project is approved. The included contingencies are held within the project and are accessed by the project management team through normal risk management processes. In summary, good cost estimating[2] is a well understood (if not always well executed) practice, that combines art and science, and includes the calculation of appropriate contingencies. Setting an appropriate management reserve is an altogether different problem.

 

Setting a realistic management reserve

Management reserves are an amount of money held outside of the project baseline to ‘protect the performing organisation’ against unexpected cost overruns. The reserves should be designed to compensate for two primary factors.  The first are genuine ‘black swans’ the other is estimating errors (including underestimating the levels of contingency needed).

The definition of a ‘black swan’ event is a significant unpredicted and unpredictable event[3].  In his book of the same name, N.N. Taleb defines ‘Black Swans’ as having three distinct characteristics: they are unexpected and unpredictable outliers, they have extreme impacts, and they appear obvious after they have happened. The primary defence against ‘black swans’ is organisational resilience rather than budget allowances but there is nothing wrong with including an allowance for these impacts.

Estimating errors leading to a low-cost baseline, on the other hand, are both normal and predictable; there are several different drivers for this phenomenon most innate to the human condition. The factors leading to the routine underestimating of costs and delivery times, and the over estimating of benefits to be realised, can be explained in terms of optimism bias and strategic misrepresentation.  The resulting inaccurate estimates of project costs, benefits, and other impacts are major source of uncertainty in project management – the occurrence is predictable and normal, the degree of error is the unknown variable leading to risk.

The way to manage this component of the management reserves is through the application of reference class forecasting which enhances the accuracy of the budget estimates by basing forecasts on actual performance in a reference class of comparable projects. This approach bypasses both optimism bias and strategic misrepresentation.

Reference class forecasting is based on theories of decision-making in situations of uncertainty and promises more accuracy in forecasts by taking an ‘outside view’ of the projects being estimated. Conventional estimating takes an ‘inside view’ based on the elements of the project being estimated – the project team assesses the elements that make up the project and determine a cost. This ‘inside’ process is essential, but on its own insufficient to achieve a realistic budget. The ‘outside’ view adds to the base estimate based on knowledge about the actual performance of a reference class of comparable projects and resolves to a percentage markup to be added to the estimated price to arrive at a realistic budget.  This addition should be used to assess the value of the project (with a corresponding discounting of benefits) during the selection/investment decision making processes[4], and logically should be held in management reserves.

Overcoming bias by simply hoping for an improvement in the estimating practice is not an effective strategy!  Prof. Bent Flyvbjerg’s 2006 paper ‘From Nobel Prize to Project Management: Getting Risks Right[5]’ looked at 70 years of data.  He found: Forecasts of cost, demand, and other impacts of planned projects have remained constantly and remarkably inaccurate for decades. No improvement in forecasting accuracy seems to have taken place, despite all claims of improved forecasting models, better data, etc.  For transportation infrastructure projects, inaccuracy in cost forecasts in constant prices is on average 44.7% for rail, 33.8% for bridges and tunnels, and 20.4% for roads.

The consistency of the error and the bias towards significant underestimating of costs (and a corresponding overestimate of benefits) suggest the root causes of the inaccuracies are psychological and political rather than technical – technical errors should average towards ‘zero’ (plusses balancing out minuses) and should improve over time as industry becomes more capable, whereas there is no imperative for psychological or political factors to change:

  • Psychological explanations can account for inaccuracy in terms of optimism bias; that is, a cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience[6].
  • Political factors can explain inaccuracy in terms of strategic misrepresentation. When forecasting the outcomes of projects, managers deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that their project will gain approval and funding either ahead of competitors in a portfolio assessment process or by avoiding being perceived as ‘too expensive’ in a public forum – this tendency particularly affects mega-projects such as bids for hosting Olympic Games.

 

Optimism Bias

Reference class forecasting was originally developed to compensate for the type of cognitive bias that Kahneman and Tversky found in their work on decision-making under uncertainty, which won Kahneman the 2002 Nobel Prize in economics[7]. They demonstrated that:

  • Errors of judgment are often systematic and predictable rather than random.
  • Many errors of judgment are shared by experts and laypeople alike.
  • The errors remain compelling even when one is fully aware of their nature.

Because awareness of a perceptual or cognitive bias does not by itself produce a more accurate perception of reality, any corrective process needs to allow for this.

 

Strategic Misrepresentation

When strategic misrepresentation is the main cause of inaccuracy, differences between estimated and actual costs and benefits are created by political and organisational pressures, typically to have a business case approved, or a project accepted, or to get on top of issues in the 24-hour news cycle.  The Grattan Institute (Australia) has reported that in the last 15 years Australian governments had spent $28 billion more than taxpayers had been led to expect. A key ‘political driver’ for these cost overruns was announcing the project (to feed the 24-hour news cycle) before the project team had properly assessed its costs.  While ‘only’ 32% of the projects were announced early, these accounted for 74% of the value of the cost overruns.

The Grattan Institute (Australia) has reported that in the last 15 years Australian governments had spent $28 billion more than taxpayers had been led to expect on transport infrastructure projects. One of the key ‘political drivers’ for these cost overruns was announcing the project (to feed the 24-hour news cycle) before the project team had properly assessed its costs.  While ‘only’ 32% of the projects were announced early, these projects accounted for 74% of the value of the cost overruns.

Reference class forecasting will still improve accuracy in these circumstances, but the managers and estimators may not be interested in this outcome because the inaccuracy is deliberate. Biased forecasts serve their strategic purpose and overrides their commitment to accuracy and truth; consequently the application of reference class forecasting needs strong support from the organisation’s overall governance functions.

 

Applying Reference Class Forecasting

Reference class forecasting does not try to forecast specific uncertain events that will affect a particular project, but instead places the project in a statistical distribution of outcomes from the class of reference projects.  For any particular project it requires the following three steps:

  1. Identification of a relevant reference class of past, similar projects. The reference class must be broad enough to be statistically meaningful, but narrow enough to be truly comparable with the specific project – good data is essential.
  2. Establishing a probability distribution for the selected reference class. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.
  3. Comparing the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

The UK government (Dept. of Treasury) were early users of reference class forecasting and continue its practice.  A study in 2002 by Mott MacDonald for Treasury found over the previous 20 years on government projects the average works duration was underestimated by 17%, CAPEX was underestimated by 47%, and OPEX was underestimated by 41%.  There was also a small shortfall in benefits realised.

 

This study fed into the updating of the Treasury’s ‘Green Book’ in 2003, which is still the standard reference in this area. The Treasury’s Supplementary Green Book Guidance: Optimism Bias[8] provides the recommended range of markups with a requirement for the ‘upper bound’ to be used in the first instance by project or program assessors.

These are very large markups to shift from an estimate to a likely cost and are related to the UK government’s estimating (ie, the client’s view), not the final contractors’ estimates – errors of this size would bankrupt most contractors.  However, Gartner and most other authorities routinely state project and programs overrun costs and time estimates (particularly internal projects and programs) and the reported ‘failure rates’ and overruns have remained relatively stable over extended periods.

 

Conclusion

Organisations can choose to treat each of their project failures as a ‘unique one-off’ occurrence (another manifestation of optimism bias) or learn from the past and develop their own framework for reference class forecasting. The markups don’t need to be included in the cost baseline (the project’s estimates are their estimates and they should attempt to deliver as promised); but they should be included in assessment process for approving projects and the management reserves held outside of the baseline to protect the organisation from the effects of both optimism bias and strategic misrepresentation.  As systems, and particularly business cases, improve the reference class adjustments should reduce but they are never likely to reduce to zero, optimism is an innate characteristic of most people and political pressures are a normal part of business.

If this post has sparked your interest, I recommend exploring the UK information to develop a process that works in your organisation: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

______________________

[1] For more on risk assessment see: http://www.mosaicprojects.com.au/WhitePapers/WP1015_Risk_Assessment.pdf

[2] For more on cost estimating see: http://www.mosaicprojects.com.au/WhitePapers/WP1051_Cost_Estimating.pdf

[3] For more on ‘black swans’ see: https://mosaicprojects.wordpress.com/2011/02/11/black-swan-risks/

[4] For more on portfolio management see: http://www.mosaicprojects.com.au/WhitePapers/WP1017_Portfolios.pdf

[5] Project Management Journal, August 2006.

[6] For more on the effects of bias see: http://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

[7] Kahneman, D. (1994). New challenges to the rationality assumption. Journal of Institutional and Theoretical
Economics, 150, 18–36.

[8] Green Book documents can be downloaded from: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

Phronesis – A key attribute for project managers

Phronesis (Ancient Greek: φρόνησις, phronēsis) is a type of wisdom described by Aristotle in his classic book Nicomachean Ethics. Phronesis or practical wisdom[1] is focused on working out the right way to do the right thing in a particular circumstance. Aristotle understood ethics as being less about establishing moral rules and following them and more about performing a social practice well; being a good friend, a good manager or a good statesman. This requires the ability to discern how or why to act virtuously and the encouragement of practical virtue and excellence of character in others.  But in a post-truth world, the ability to use ‘practical wisdom’ to discern what is real and what is ‘spin’ in rapidly becoming a key social and business skill. So prevalent is this trend, the Oxford English Dictionary named ‘post-truth’ its 2016 word of the year.

This problem pre-dates Donald Trump and ‘Brexit’, but seems to be getting worse. How can a project manager work out the right way to do the right thing in the particular circumstance of her project when much of the information being received is likely to be ‘spun’ for a particular effect.  There may be a solution in the writings of Bent Flyvbjerg.

Professor Bent Flyvbjerg, Chair of Major Programme Management at the Saïd Business School, Oxford University, has a strong interest in both megaproject management and phronesis. A consistent theme in his work has been the lack of truthfulness associated with the promotion of mega projects of all types, worldwide and the consequences of this deception. To help with the challenge of cutting through ‘spin’, and based on his research, he has published the following eight propositions:

1. Truth is context dependent.
2. The context of truth is power.
3. Power blurs the dividing line between truth and lies.
4. Lies and spin presented as truth is a principal strategy of those in power.
5. The greater the power, the less the truth.
6. Power has deeper historical roots than truth, which weakens truth.
7. Today, no power can avoid the issue of ‘speaking the truth’, unless it imposes silence and servitude. Herein lies the power of truth.
8. Truth will not be silenced.

There is, of course, a book, Rationality and Power: Democracy in Practice[2] that goes into more detail but just thinking through the propositions can help you apply the practical ethics that underpin phronesis.  Being virtuous is never easy, but regardless of the power brought to bear, sooner or later the truth will be heard.

The problem is which ‘truth’, understanding and perception will influence what people see, hear and believe to be the truth. Nietzsche, a German counter-Enlightenment thinker of the late 19th century, suggests that objective truth does not really exist; that objective absolute truth is an impossibility. The challenge we all face is the practical one of understanding enough about ourselves and others (we are all biased[3]) to achieve a reasonable level of understanding and then do our best to make the right decisions (see more on decision making), and to do the right thing in the right way.

___________________

[1] From Practical Wisdom, the right way to do the right things by Barry Schwartz and Kenneth Sharp.  Riverhead Books, New York 2010.

[2] See: https://www.amazon.com/Rationality-Power-Democracy-Practice-Morality/dp/0226254518/

[3] See,  The innate effect of Biashttp://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

Project scheduling Update

1. A new paper looking at the origins of CPM has been uploaded to our PM-History page – http://www.mosaicprojects.com.au/Mag_Articles/P037_The_Origins_of_CPM.pdf looks at where the concepts that evolved into CPM and PERT originated. All of our papers can be found at: http://www.mosaicprojects.com.au/PM-History.html

2.  The PMI members’ only Scheduling Conference 2017 is going to be great! Over 17,000 people are registered already – I’m the last speaker for the day (which means I only have to get up at 6:00am Australian time to participate…..) More information see: https://www.projectmanagement.com/events/356123/PMI-Scheduling-Conference-2017  My topic looks at the effect of the data generated by BIM, drones and other technology on controls.

3.  PGCS Canberra is on in early May – too good to miss, see: http://www.pgcs.org.au/

Setting up a project controls system for success

A couple of hour’s hard thinking can make the difference between project success and failure!  Far too many projects are simply started without any real thought as to the best strategy for delivery and what control systems are really needed to support the management of that delivery – one size does not ‘fit-all’ and simply repeating past failures creates more failures.  Similarly, far too many control systems are implemented that simply generate useless paperwork (frequently to meet contractual requirements) when what’s needed is effective controls information.

Remembering that all project controls documents have to be used and maintained to be useful; the three key thinking processes needed to help build project success are:

  • First the big question – how are we going to do the work to maximise the opportunity of success and optimise risk??  This is a strategic question and affects procurement as much as anything – off-site assembly needs a very different approach to on-site assembly. This does not need a complicated document but the strategy does need to be agreed; see: www.mosaicprojects.com.au/WhitePapers/WP1038_Strategy.pdf
  • From the strategy, the project management team structure can be designed to best manage the work as it will be accomplished and these people (or at least the key people) can then contribute to the planning process. Pictures are as useful as anything to define the overall flow of the work; see: www.mosaicprojects.com.au/WhitePapers/WP1039_Project_Planning.pdf.
  • Once you know the way the work will be accomplished and the overall flow/sequence of the work you are now in a position to plan the project controls function aiming to apply the minimum amount of ‘controls’ necessary to be effective.  Excessive controls simply waste money and management time. My approach is always to do a bit less then I think may be needed because you can always add some additional features if the need eventuates – it Is nearly impossible to remove controls once they have been implemented.
  • Then you can develop the schedule and other control tools needed for effective management working within the framework outlined above.

This area is what PMI call Schedule strategy and Schedule planning and development. Getting this ‘front-end’ stuff right is the best foundation for a successful completion of a project; this is the reason these elements of project controls have a strong emphasis in the PMI-SP exam.

Conversely, stuffing up the strategy in particular, means the project is set up to fail and implementing control systems that do not support the management structures within the project simply mean the controls people are wasting their time and the time of everyone they engage with.

However, creating a project that is based on a sound strategy supported by a useful project controls system will require some cultural changes:

  • The project manager and project executive will need to take some time to look at strategic options and develop an effective delivery strategy.
  • The organisation and client will need to allow the project controls professionals to work through the challenges of developing a ‘light-but-effective’ controls system and then review/approve the system – this is more difficult than simply requiring every project to comply with some bloated standard controls process that no one uses (except for claims) but should deliver massive benefits.
  • The organisation will need skilled project controls professionals……….
  • And the project management team will need to be willing to work with and use the project controls.

The problem is easy to outline – fixing it to enhance the project success rate is a major challenge.

The origins of PERT and CPM – What came before the computers!

The development of PERT and CPM as Mainframe software systems starting in 1957 is well documented with contemporary accounts from the key people involved readily available.  What is less clear is how two systems developed contemporaneously, but in isolation, as well as a number of less well documented similar systems developed in the same timeframe in the UK and Europe came to have so many similar features.  These early tools used the ‘activity-on-arrow’ (AoA or ADM) notation which is a far from obvious model.  Later iterations of the concept of CPM used the ‘precedence’ notation which evolved from the way flow-charts were and are drawn.

stockpile

One obvious connection between the early developments was the community of interest around Operation (or Operational) Research (OR) a concept developed by the British at the beginning of WW2.  OR had developed to include the concept of linear programming by the mid-1950s which is the mathematical underpinning of CPM, but while this link explains some of the cross pollination of ideas and the mathematics it does not explain terms such as ‘float’ and the AoA notation (for more on the development of CPM as a computer based tool see http://www.mosaicprojects.com.au/PDF_Papers/P042_History%20of%20Scheduing.pdf).

A recent email from Chris Fostel, an Engineering Planning Analyst with Northrop Grumman Corporation (CFostel@rcn.com) appears to offer a rational explanation.  I’ve reproduced Chris’ email pretty much verbatim below – the challenge posed to you is to see if the oral history laid out below can be corroborated or validated.  I look forward to the responses.

Chris’ Oral History

quartermaster_corpsI was told this story in 1978 by a retired quartermaster who founded his own company after the War to utilize his global contacts and planning skills.  Unfortunately the individual who told me this story passed away quite a few years ago and I’m not sure any of his compatriots are still alive either.  Regardless, I thought I should pass this along before I join them in the next life.  I do not wish to minimize the work of Kelly and Walker. They introduced critical path scheduling to the world and formalized the algorithms.  They did not develop or invent the technique.

The origin of critical path scheduling was the planning of the US Pacific Island hopping campaign during World War II.  The Quartermaster Corps coordinated orders to dozens if not hundreds of warships, troop ships and supply ships for each assault on a new island.  If any ships arrived early it would alert the Japanese of an imminent attack.  Surprise was critical to the success of the island hopping campaign.  The US did not have enough warships to fight off the much larger Japanese fleet until late in the war. Alerting the Japanese high command would allow the Japanese fleet to intercept and destroy the slow moving US troop ships before they had a chance to launch an attack. 

Initially the quartermasters drew up their plans on maps of the pacific islands, including current location and travel times of each ship involved.  The travel times were drawn as arrows on the map.  Significant events, personnel or supplies that traveled by air were shown as dashed lines hopping over the ship’s arrows.  The quartermasters would then calculate shortest and longest travel times to the destination for all ships involved in the assault. The plans became very complicated.  Many ships made intermediate stops at various islands to refuel or transfer cargo and personnel.  The goal was to have all ships arrive at the same time.  It didn’t take the quartermasters long to realize that a photograph of the planning maps would be a devastating intelligence lapse.  They started drawing the islands as identical bubbles with identification codes and no particular geographical order on the bubble and arrow charts. These were the first activity on arrow critical path charts; circa 1942. 

The only validation I can offer you is that by now you should realize that activity on arrow diagrams were intuitive as was the term ‘float.’  Float was the amount of time a particular ship could float at anchor before getting underway for the rendezvous.  Later when the US quartermasters introduced the technique to the British for planning the D-Day invasion the British changed float to “Slack”, to broaden the term to include air force and army units which did not float, but could ‘slack off’ for the designated period of time. 

You will not find a written, dated, account of this story by a quartermaster corps veteran.  Critical path scheduling was a military secret until declassification in 1956.  In typical fashion, the veterans of WWII did not write about their experiences during the War.  No one broke the military secrecy.  After 1956 they were free to pass the method on to corporate planners such as Kelly and Walker.  A living WWII Quartermaster veteran, should be able to provide more than my intuitive confirmation.

This narrative makes sense to me from a historical perspective (military planning has involved drawing arrows on maps for at least 200 years) and a timing perspective.  Can we find any additional evidence to back this up??  Over to you!

The Yin and Yang of Integrated Data Systems

yin_yangIntegrated project management information systems (PMIS) are becoming more common and more sophisticated ranging from ‘web portals’ that hold project data through to the potential for fully integrated design and construction management using BIM[1].  The benefits derived from using these systems can be as much as 20% of the build price on complex construction projects using BIM.

pmisThe advantages of this type of information storage and retrieval system include:

  • Ready access to data when needed via PDAs and ‘tablets’ significantly reducing the need for ‘push’ communication and the existence of ‘redundant data’[2].
  • One place to look for information with indexing and cross-referencing to minimise the potential for missed information.
  • Audit trails and systems to ensure only the latest version of any document is available.
  • Cross-linking of data in different documents and formats to assist with configuration management, requirements traceability, and change control.
  • Controls on who can ‘see’ the data, access the data and edit the data.
  • Workflow functions to remind people of their next job, list open actions, record actual progress, etc[3].
  • A range of built-in functions to validate data and avoid ‘clashes’, including locking or ‘freezing’ parts of the data set when that information has been moved into ‘work’.

These benefits are significant and a well-designed system reduces errors and enhances productivity leading to reduced costs, but the ‘yin’ of well-designed PMIS comes with a ‘yang’!

People increasingly tend to believe information produced from a computer system, this is true of ‘Facebook’, Wikipedia and flows through to more sophisticated systems. There also seems to be a steady reduction in the ability of younger people in particular to critically analyse information; in short, if it comes from the computer many people will assume it is correct. Add to this the ability of many of the more sophisticated PMIS tools to transpose and transfer information between different parts of the systems automatically or semiautomatically and there is a potential for many of the benefits outlined above to be undermined by poor data. This issue has been identified for decades and has the acronym GIGO – garbage in, garbage out.

The question posed in this blog is how many projects and project support organisations (PMOs, etc.) consider or actively implement effective data traceability.  Failed audits, overruns from scope oversights, and uninformed or ill-informed decision-making are just a few of the consequences project teams suffer from if they do not have full traceability of their project management data. This issue exists in any information processing system from basic schedule updating, through monthly reporting to the most sophisticated, integrated PMIS. If you cannot rely on the source data, no amount of processing will improve the situation! And to be able to rely on data, you need to be able to trace it back to its source.

tracabilityTraceability is defined as ‘the ability to trace the location, history and use of each data element’. This sounds simple but in reality can be very challenging, and the results of poor visibility can be devastating to a project. Some of the key questions to ask are:

  • Where did this data or these actuals come from?
  • What is the authorizing document and when did it get signed/approved?
  • Has everyone approved the change request or action item?

Traceability does not happen by accident! Project management information systems have to be designed with traceability as a key element in each of its aspects.  As information comes into the system the author or the origin of the information has to be recorded (preferably automatically). Depending on the nature of the information it may need to be quarantined until appropriate checks have been carried out and/or approvals have been obtained and then there needs to be traceability of any subsequent changes. The foundation of traceability is the combination of processes (people) and data management.

Therefore, the ‘yang’ of a sophisticated integrated project management information systems is that as the systems become more integrated and sophisticated people will come to rely on the information provided and ‘trust it’ whilst the source and veracity of the data used becomes less obvious.

Resolving this is partly process and partly people. The Chartered Institute of Building (CIOB) has produced the Time and Cost Management Contract Suite 2015 focused on complex construction projects using BIM.  This contract defines a number of key support roles (largely independent of the parties) focused on managing the information flows into and out of the system to ensure its accuracy and validity. Similar roles and responsibilities are essential in any effective PMIS.

My latest post on the PMI ‘Voices blog’, From Data to Wisdom: Creating & Managing Knowledge highlights the importance of data as the underpinning of all reporting and communication.  So the question is, how much focus does your project team or PMO put on ensuring the data it is using is timely, complete, accurate and traceable?

____________________

[1] BIM = Building Information Modelling, see: http://www.mosaicprojects.com.au/WhitePapers/WP1082_BIM_Levels.pdf

[2] For more on planning project communication see: http://www.mosaicprojects.com.au/Mag_Articles/ESEI-09-communication-planning.pdf

[3] A discussion on how these capabilities can enhance project controls is at: https://mosaicprojects.wordpress.com/2016/11/26/the-future-of-project-controls/