Defining Project Success using Project Success Criteria

Everyone likes a successful project but the big question is what makes a project successful??  A good example is the Sydney Opera House; was the Sydney Opera House successful or not?

The project ran significantly over budget finished very late and was technically less than perfect; $millions are currently being spent rectifying many of the technical deficiencies in the building. But can anyone say Sydney Opera House is not one of the most recognised and therefore successful buildings in the world?[1]

Success is an ephemeral concept! Different people will have different perspectives and judge the success or failure project differently. Neither a project nor a program manager can control many of the factors that have made the Sydney Opera House worldwide icon but they can address the concept of success with their stakeholders and then work to deliver a successful outcome based on these discussions.

So what is success? There are probably three key elements, but these frequently create a paradox that requires a balanced approach to success. The three fundamental elements are:

• The Iron Triangle (Scope + Cost + Time)
• Benefits realised (or maximised)
• Satisfied stakeholders (but, when??)

One of the key paradox is a myopic focus on the Iron Triangle particularly time and cost can frequently destroy benefits and leave the stakeholders unhappy, but focusing on keeping stakeholders happy can frequently have detrimental effects on the Iron Triangle. There are no easy solutions to this problem[2].

In my view, the successful delivery of a project or program requires:

• Achieving the overall goal for the project;
• Delivering its objectives; and
• Meeting its success criteria.

But, to achieve success you need to define and agree the project goal, the project objectives, and the project success criteria with your key stakeholders with a view to achieving a combination of stakeholder satisfaction and value created. The goal and objectives frame the project’s work and direction. The success criteria frame how the objectives are achieved.

 

The Project Goal

Goals are high-level statements that provide the overall context defining what the project is trying to achieve. One project should have one goal (if there are multiple goals you are most likely looking at a program of work[3])!  For example:  Within 180 days, reduce the pollution in the rainwater runoff from a council tip by 98%.

The goal is a key statement in the Project Charter[4] and if the project is to be successful, all key stakeholders need to agree the goal.  The goal needs to be specific and should define the project in a way that focuses attention on the key outcomes required for overall success from a technical and strategic business perspective[5].

 

Project Objectives

The objectives are lower level statements that describe the specific, tangible products and deliverables that the project will create; each objective (and the overall goal) should be SMART[6]. For the runoff project the objectives may include:

• Develop wetlands to trap 99.8% of sediment
• Install channels to collect and direct the runoff
• Install screens remove floating debris
• Etc….. There will be a number of objectives……

Each objective requires defining and specifying with clear performance criteria so you know when it has been achieved. This may be done by the client or by the project team during the scope definition process. The performance criteria may be defined by a set of precise specifications that are specific and measurable or may be defined as a performance requirement with either:

• The external contractor to provide the specific details of how the objective will be achieved, or
• The internal project team to develop the details in consultation with the client

The defined objectives are the building blocks that facilitate the achievement of the goal and the creation of the benefits the organisation is expecting from the project[7]. The benefits need to be realised to create value.

 

Success criteria

Success criteria are different they measure what’s important to your stakeholders. Consequently, they are the standards by which the project will be judged at the end to decide whether or not it has been successful in the eyes of its stakeholders. As far as possible the stakeholders need to be satisfied; this includes having their expectations fulfilled and in general terms being pleased with both the journey and the outcome (in this respect scope, cost and/or time may be important).

Success criteria can be expressed in many different ways some examples include:

• Zero accidents / no environmental issues;
• No ‘bad press’ / good publicity received;
• Finalist in the project achievement awards;
• Plus the goal and all of the objectives achieved (yes – you still need to do the work).

For any project, the success criteria should be split between project management success criteria which of related to the professional aspects of running the project; plus project deliverable success criteria which are related to the performance and function of the deliverable.

Documenting the success criteria is important, it means you can get project stakeholders to sign up to them, and having them clearly recorded removes ambiguity about what you are setting out to do. The four basic steps to create useful success criteria are

1. Document and agree the criteria; each criteria should include:
   1. The name of success criteria,
   2. How it is going to be measured,
   3. How often it is going to be measured, and
   4. Who is responsible for the measurement.
2. Use continuous measurements where possible. For example, rather than ‘finish the project on time’ measure progress continually ‘no activity completes more than 5 days after its late finish date’.
3. Baseline today’s performance.
4. Track and report on your progress.

As with any performance indicators, the art is to select a few key measures that represent the wider picture if there are too many success criteria defined the impact will be severely reduced. For example, the effectiveness of meetings, communication and stakeholder attitude could be measured scientifically using the ‘Index Value’ in the Stakeholder Circle[8] or pragmatically by measuring the number of open issues against a target (eg, no more than 5 high priority open issues).

 

Summary

Goals and objectives are the building blocks required to allow the realisation value from the project’s outputs; they are essential ingredients in a successful project but are insufficient on their own.  The role of success criteria is to direct the way work at the project is accomplished so as to meet stakeholder expectations, and to craft a perception of success in the stakeholder’s minds.

Project success is an amalgam of value created for the organisation and your stakeholders being satisfied with the journey and the outcome.  This concept of success may seem subjective, but it does not have to be. Successful organisations work to take the guesswork out of this process by defining what success looks like and agreeing these definitions with the key stakeholders, so they all know when the project has achieved it.

This means the key to stakeholders perceiving your project as successful lays in understanding the criteria they will measure success by, incorporating those measures into your project success criteria, and then working to achieve the criteria. But even this is not enough, to engage your stakeholders you need to communicate the criteria, communicate your progress and communicate your success at the end. For more on effective communication see: http://www.mosaicprojects.com.au/PM-Knowledge_Index.html#PPM07

________________

 

[1] For more on the success or failure of the Sydney Opera House see Avoiding the Successful Failure!:  http://www.mosaicprojects.com.au/Resources_Papers_046.html

[2] For more on paradox see: https://www.projectmanagement.com/blog-post/30669/The-Problem-With-Paradox

[3] For more on differentiating projects and programs see: http://www.mosaicprojects.com.au/WhitePapers/WP1002_Programs.pdf

[4] For more on the project charter see: http://www.mosaicprojects.com.au/WhitePapers/WP1019_Charter.pdf

[5] For more on project success see: http://www.mosaicprojects.com.au/Mag_Articles/N001_Achieving_Real_Project_Success.pdf

[6] SMART = Specific, Measurable, Attainable, Relevant and Time-framed.

[7] For more on linking objectives and benefits see: http://www.mosaicprojects.com.au/WhitePapers/WP1042_Outputs_Outcomes_Benefits.pdf

[8] The Stakeholder Circle® index value see: http://202.146.213.160/help-files/stakeholder-engagement-profile/#engagement-index

New White Paper on the value of TCPI

The To Complete Performance Index is one of the least understood metrics available as part of an EVM system which in recent months seems to have been given prominence in both the PMP and PMI-SP examinations.

Download this White Paper for a straightforward explanation of this long-established (but little understood) metric and its value as an indicator of project outcomes from: http://www.mosaicprojects.com.au/WhitePapers/WP1097_TCPI_in_EVM.pdf

To access all of our WPs and other published papers available for free downloads, start at: http://www.mosaicprojects.com.au/PM-Knowledge_Index.html

The reference case for management reserves

Risk management and Earned Value practitioners, and a range of standards, advocate the inclusion of contingencies in the project baseline to compensate for defined risk events. The contingency may (should) include an appropriate allowance for variability in the estimates modelled using Monte Carlo or similar; these are the ‘known unknowns’.  They also advocate creating a management reserve that should be held outside of the project baseline, but within the overall budget to protect the performing organisation from the effects of ‘unknown unknowns’.  Following these guidelines, the components of a typical project budget are shown below.

PMBOK® Guide Figure 7-8

The calculations of contingency reserves should be incorporated into an effective estimating process to determine an appropriate cost estimate for the project[1]. The application of appropriate tools and techniques supported by skilled judgement can arrive at a predictable cost estimate which in turn becomes the cost baseline once the project is approved. The included contingencies are held within the project and are accessed by the project management team through normal risk management processes. In summary, good cost estimating[2] is a well understood (if not always well executed) practice, that combines art and science, and includes the calculation of appropriate contingencies. Setting an appropriate management reserve is an altogether different problem.

 

Setting a realistic management reserve

Management reserves are an amount of money held outside of the project baseline to ‘protect the performing organisation’ against unexpected cost overruns. The reserves should be designed to compensate for two primary factors.  The first are genuine ‘black swans’ the other is estimating errors (including underestimating the levels of contingency needed).

The definition of a ‘black swan’ event is a significant unpredicted and unpredictable event[3].  In his book of the same name, N.N. Taleb defines ‘Black Swans’ as having three distinct characteristics: they are unexpected and unpredictable outliers, they have extreme impacts, and they appear obvious after they have happened. The primary defence against ‘black swans’ is organisational resilience rather than budget allowances but there is nothing wrong with including an allowance for these impacts.

Estimating errors leading to a low-cost baseline, on the other hand, are both normal and predictable; there are several different drivers for this phenomenon most innate to the human condition. The factors leading to the routine underestimating of costs and delivery times, and the over estimating of benefits to be realised, can be explained in terms of optimism bias and strategic misrepresentation.  The resulting inaccurate estimates of project costs, benefits, and other impacts are major source of uncertainty in project management – the occurrence is predictable and normal, the degree of error is the unknown variable leading to risk.

The way to manage this component of the management reserves is through the application of reference class forecasting which enhances the accuracy of the budget estimates by basing forecasts on actual performance in a reference class of comparable projects. This approach bypasses both optimism bias and strategic misrepresentation.

Reference class forecasting is based on theories of decision-making in situations of uncertainty and promises more accuracy in forecasts by taking an ‘outside view’ of the projects being estimated. Conventional estimating takes an ‘inside view’ based on the elements of the project being estimated – the project team assesses the elements that make up the project and determine a cost. This ‘inside’ process is essential, but on its own insufficient to achieve a realistic budget. The ‘outside’ view adds to the base estimate based on knowledge about the actual performance of a reference class of comparable projects and resolves to a percentage markup to be added to the estimated price to arrive at a realistic budget.  This addition should be used to assess the value of the project (with a corresponding discounting of benefits) during the selection/investment decision making processes[4], and logically should be held in management reserves.

Overcoming bias by simply hoping for an improvement in the estimating practice is not an effective strategy!  Prof. Bent Flyvbjerg’s 2006 paper ‘From Nobel Prize to Project Management: Getting Risks Right[5]’ looked at 70 years of data.  He found: Forecasts of cost, demand, and other impacts of planned projects have remained constantly and remarkably inaccurate for decades. No improvement in forecasting accuracy seems to have taken place, despite all claims of improved forecasting models, better data, etc.  For transportation infrastructure projects, inaccuracy in cost forecasts in constant prices is on average 44.7% for rail, 33.8% for bridges and tunnels, and 20.4% for roads.

The consistency of the error and the bias towards significant underestimating of costs (and a corresponding overestimate of benefits) suggest the root causes of the inaccuracies are psychological and political rather than technical – technical errors should average towards ‘zero’ (plusses balancing out minuses) and should improve over time as industry becomes more capable, whereas there is no imperative for psychological or political factors to change:

• Psychological explanations can account for inaccuracy in terms of optimism bias; that is, a cognitive predisposition found with most people to judge future events in a more positive light than is warranted by actual experience[6].
• Political factors can explain inaccuracy in terms of strategic misrepresentation. When forecasting the outcomes of projects, managers deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that their project will gain approval and funding either ahead of competitors in a portfolio assessment process or by avoiding being perceived as ‘too expensive’ in a public forum – this tendency particularly affects mega-projects such as bids for hosting Olympic Games.

 

Optimism Bias

Reference class forecasting was originally developed to compensate for the type of cognitive bias that Kahneman and Tversky found in their work on decision-making under uncertainty, which won Kahneman the 2002 Nobel Prize in economics[7]. They demonstrated that:

• Errors of judgment are often systematic and predictable rather than random.
• Many errors of judgment are shared by experts and laypeople alike.
• The errors remain compelling even when one is fully aware of their nature.

Because awareness of a perceptual or cognitive bias does not by itself produce a more accurate perception of reality, any corrective process needs to allow for this.

 

Strategic Misrepresentation

When strategic misrepresentation is the main cause of inaccuracy, differences between estimated and actual costs and benefits are created by political and organisational pressures, typically to have a business case approved, or a project accepted, or to get on top of issues in the 24-hour news cycle.  The Grattan Institute (Australia) has reported that in the last 15 years Australian governments had spent $28 billion more than taxpayers had been led to expect. A key ‘political driver’ for these cost overruns was announcing the project (to feed the 24-hour news cycle) before the project team had properly assessed its costs.  While ‘only’ 32% of the projects were announced early, these accounted for 74% of the value of the cost overruns.

The Grattan Institute (Australia) has reported that in the last 15 years Australian governments had spent $28 billion more than taxpayers had been led to expect on transport infrastructure projects. One of the key ‘political drivers’ for these cost overruns was announcing the project (to feed the 24-hour news cycle) before the project team had properly assessed its costs.  While ‘only’ 32% of the projects were announced early, these projects accounted for 74% of the value of the cost overruns.

Reference class forecasting will still improve accuracy in these circumstances, but the managers and estimators may not be interested in this outcome because the inaccuracy is deliberate. Biased forecasts serve their strategic purpose and overrides their commitment to accuracy and truth; consequently the application of reference class forecasting needs strong support from the organisation’s overall governance functions.

 

Applying Reference Class Forecasting

Reference class forecasting does not try to forecast specific uncertain events that will affect a particular project, but instead places the project in a statistical distribution of outcomes from the class of reference projects.  For any particular project it requires the following three steps:

1. Identification of a relevant reference class of past, similar projects. The reference class must be broad enough to be statistically meaningful, but narrow enough to be truly comparable with the specific project – good data is essential.
2. Establishing a probability distribution for the selected reference class. This requires access to credible, empirical data for a sufficient number of projects within the reference class to make statistically meaningful conclusions.
3. Comparing the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

The UK government (Dept. of Treasury) were early users of reference class forecasting and continue its practice.  A study in 2002 by Mott MacDonald for Treasury found over the previous 20 years on government projects the average works duration was underestimated by 17%, CAPEX was underestimated by 47%, and OPEX was underestimated by 41%.  There was also a small shortfall in benefits realised.

 

This study fed into the updating of the Treasury’s ‘Green Book’ in 2003, which is still the standard reference in this area. The Treasury’s Supplementary Green Book Guidance: Optimism Bias[8] provides the recommended range of markups with a requirement for the ‘upper bound’ to be used in the first instance by project or program assessors.

These are very large markups to shift from an estimate to a likely cost and are related to the UK government’s estimating (ie, the client’s view), not the final contractors’ estimates – errors of this size would bankrupt most contractors.  However, Gartner and most other authorities routinely state project and programs overrun costs and time estimates (particularly internal projects and programs) and the reported ‘failure rates’ and overruns have remained relatively stable over extended periods.

 

Conclusion

Organisations can choose to treat each of their project failures as a ‘unique one-off’ occurrence (another manifestation of optimism bias) or learn from the past and develop their own framework for reference class forecasting. The markups don’t need to be included in the cost baseline (the project’s estimates are their estimates and they should attempt to deliver as promised); but they should be included in assessment process for approving projects and the management reserves held outside of the baseline to protect the organisation from the effects of both optimism bias and strategic misrepresentation.  As systems, and particularly business cases, improve the reference class adjustments should reduce but they are never likely to reduce to zero, optimism is an innate characteristic of most people and political pressures are a normal part of business.

If this post has sparked your interest, I recommend exploring the UK information to develop a process that works in your organisation: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

______________________

[1] For more on risk assessment see: http://www.mosaicprojects.com.au/WhitePapers/WP1015_Risk_Assessment.pdf

[2] For more on cost estimating see: http://www.mosaicprojects.com.au/WhitePapers/WP1051_Cost_Estimating.pdf

[3] For more on ‘black swans’ see: https://mosaicprojects.wordpress.com/2011/02/11/black-swan-risks/

[4] For more on portfolio management see: http://www.mosaicprojects.com.au/WhitePapers/WP1017_Portfolios.pdf

[5] Project Management Journal, August 2006.

[6] For more on the effects of bias see: http://www.mosaicprojects.com.au/WhitePapers/WP1069_Bias.pdf

[7] Kahneman, D. (1994). New challenges to the rationality assumption. Journal of Institutional and Theoretical
Economics, 150, 18–36.

[8] Green Book documents can be downloaded from: http://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-governent

Two ‘Not-to-be missed’ Conferences in May

The PGCS program for 2017 is now complete and offers two overseas speakers as well as Professor Peter Shergold – author of the landmark project management report to government “Learning From Failure”. To see what’s on at PGCS in Canberra between the 1^st and 4^th May go to:  http://www.pgcs.org.au/program/

In the USA you can attend the Annual Project Management College of Scheduling Conference in Atlanta from May 7th to 10th.  They have a terrific program, with speakers and panel discussions, prepared to give everyone a chance to participate.  In addition, we’re planning something special every evening.  Sunday night is the vendor reception, Monday night is our Gala Dinner and Tuesday night is a night to explore Atlanta.  For more details see: http://www.pmcos.org/

Project scheduling Update

1. A new paper looking at the origins of CPM has been uploaded to our PM-History page – http://www.mosaicprojects.com.au/Mag_Articles/P037_The_Origins_of_CPM.pdf looks at where the concepts that evolved into CPM and PERT originated. All of our papers can be found at: http://www.mosaicprojects.com.au/PM-History.html

2.  The PMI members’ only Scheduling Conference 2017 is going to be great! Over 17,000 people are registered already – I’m the last speaker for the day (which means I only have to get up at 6:00am Australian time to participate…..) More information see: https://www.projectmanagement.com/events/356123/PMI-Scheduling-Conference-2017  My topic looks at the effect of the data generated by BIM, drones and other technology on controls.

3.  PGCS Canberra is on in early May – too good to miss, see: http://www.pgcs.org.au/

Setting up a project controls system for success

A couple of hour’s hard thinking can make the difference between project success and failure!  Far too many projects are simply started without any real thought as to the best strategy for delivery and what control systems are really needed to support the management of that delivery – one size does not ‘fit-all’ and simply repeating past failures creates more failures.  Similarly, far too many control systems are implemented that simply generate useless paperwork (frequently to meet contractual requirements) when what’s needed is effective controls information.

Remembering that all project controls documents have to be used and maintained to be useful; the three key thinking processes needed to help build project success are:

• First the big question – how are we going to do the work to maximise the opportunity of success and optimise risk??  This is a strategic question and affects procurement as much as anything – off-site assembly needs a very different approach to on-site assembly. This does not need a complicated document but the strategy does need to be agreed; see: www.mosaicprojects.com.au/WhitePapers/WP1038_Strategy.pdf
• From the strategy, the project management team structure can be designed to best manage the work as it will be accomplished and these people (or at least the key people) can then contribute to the planning process. Pictures are as useful as anything to define the overall flow of the work; see: www.mosaicprojects.com.au/WhitePapers/WP1039_Project_Planning.pdf.
• Once you know the way the work will be accomplished and the overall flow/sequence of the work you are now in a position to plan the project controls function aiming to apply the minimum amount of ‘controls’ necessary to be effective.  Excessive controls simply waste money and management time. My approach is always to do a bit less then I think may be needed because you can always add some additional features if the need eventuates – it Is nearly impossible to remove controls once they have been implemented.
• Then you can develop the schedule and other control tools needed for effective management working within the framework outlined above.

This area is what PMI call Schedule strategy and Schedule planning and development. Getting this ‘front-end’ stuff right is the best foundation for a successful completion of a project; this is the reason these elements of project controls have a strong emphasis in the PMI-SP exam.

Conversely, stuffing up the strategy in particular, means the project is set up to fail and implementing control systems that do not support the management structures within the project simply mean the controls people are wasting their time and the time of everyone they engage with.

However, creating a project that is based on a sound strategy supported by a useful project controls system will require some cultural changes:

• The project manager and project executive will need to take some time to look at strategic options and develop an effective delivery strategy.
• The organisation and client will need to allow the project controls professionals to work through the challenges of developing a ‘light-but-effective’ controls system and then review/approve the system – this is more difficult than simply requiring every project to comply with some bloated standard controls process that no one uses (except for claims) but should deliver massive benefits.
• The organisation will need skilled project controls professionals……….
• And the project management team will need to be willing to work with and use the project controls.

The problem is easy to outline – fixing it to enhance the project success rate is a major challenge.

There’s a great Scheduling party in Atlanta this May!

Why not attend the Project Management College of Scheduling (PM-COS) annual conference either as a speaker or delegate and collect your ‘ticket to project success’?

Some of the reasons for joining us at The Ritz-Carlton, Atlanta include:

• CPM Scheduling has been around for more than fifty years.  How much is art?  How much is science?  Help us decide.
• Network with schedule professionals from around the globe including many of the top practitioners and experts in the field.
• Learn tips from experts such as the role that case law plays in schedule delay analysis.
• Help us address important issues facing the scheduling profession today and how to resolve them as we move forward.
• Attend presentations and panel discussions to learn the recent developments in the profession and how to implement them on your projects.

For more information and to register, visit us at www.pmcos.org!