Category Archives: Risk

The Schedule Compliance Risk Assessment Methodology (SCRAM)

SCRAM is an approach for identifying risks to compliance with the program schedule, it is the result of a collaborative effort between Adrian Pitman from the Australian Department of Defence, Angela Tuffley of RedBay Consulting in Australia, and Betsy Clark and Brad Clark of Software Metrics Inc. in the United States.

SCRAM focuses on schedule feasibility and root causes for slippage. It makes no judgment about whether or not a project is technically feasible. SCRAM can be used:

  • By organisations to construct a schedule that maximizes the likelihood of schedule compliance.
  • To ensure common risks are addressed before the project schedule is baselined at the commencement of a project.
  • To monitor project status, performed either ad hoc or to support appropriate milestone reviews
  • To evaluate challenged projects, to assess the likelihood of schedule compliance, root cause of schedule slippage and recommend remediation of project issues

Whilst the documentation is intensely bureaucratic, the concepts in SCRAM move beyond the concepts embedded in processes such as the DCMA 14 point checklist  to asking hard questions about the requirements of stakeholders and how effectively risk has been addressed before baselineing the schedule.

The SCRAM concept is freely available.  The SCRAM Process Reference Model (PRM) and a Process Assessment Model (PAM) documents are available for immediate download from: https://sites.google.com/site/scramsitenew/home

For more on schedule risk assessment and compliance assessment see: http://www.mosaicprojects.com.au/Planning.html#S-Risk

Be careful what you govern for!

Governance is an interesting and subtle process which is not helped by confusing governance with management or organisational maturity. A recent discussion in PM World Journal on the subject of governance and management highlighted an interesting issue that we have touched on in the past.

The Romans were undoubtedly good builders (see: The Roman Approach to Contract Risk Management). They also had effective governance and management processes, when a contractor was engaged to build something, they had a clear vision of what they wanted to accomplish; assigned responsibilities and accountability effectively; and failure had clearly understood, significant consequences.

Roman bridge builders were called pontiff. One of the quality control processes used to ensure the effective construction of bridges and other similar structures was to ensure the pontiffs were the first to cross their newly completed construction with their chariots to demonstrate that their product was safe.

An ancient Roman bridge

An ancient Roman bridge

This governance focus on safety and sanctions created very strong bridges some of which survive in use to the present day but this governance policy also stymied innovation. Roman architecture and engineering practice did not change significantly in the last 400 years of the empire!

No sensible pontiff would risk his life to test an innovative approach to bridge design or construction when the governance systems he operated under focus on avoiding failure. Or in more general terms; the management response to a governance regime focused on ‘no failure’ backed up by the application of sanctions is to implement rigid processes. The problem is rigid process prevents improvement.

To realise the significance of this consider the technology in use in the 17th century compared to the modern day – the vast majority of the innovations that have resulted in today’s improved living standards are the result of learning from failure (see: How to Suffer Successfully).

But the solution is not that simple, we know that well designed and implemented, processes are definitely advantageous. There is a significant body of research that shows implementing methodologies and processes using CMMI, OPM3, PRINCE2, P3M3 and other similar frameworks has a major impact on improving organisational performance and outcomes.

However, organisational maturity is a similar ‘two edged sword’ to rigid governance and management requirements. We know organisational maturity defined as the use of standardised processes and procedures creates significant benefits in terms of reduced error and increased effectiveness compared to laissez-faire / ad hoc systems with little or no standardisation. But these improvements can evolve to become an innovation-sapping straightjacket.

Too much standardisation creates processes paralysis and a focus on doing the process rather than achieving an outcome. In organisations that that have become fixated on ‘process’, it is common to see more and more process introduced to over come the problem of process paralysis which in turn consume more valuable time and resources until Cohn’s Law is proved: The more time you spend in reporting on what you are doing, the less time you have to do anything. Stability is achieved when you spend all your time doing nothing but reporting on the nothing you are doing.

Avoiding this type of paralysis before a review is forced by a major crisis is a subtle, but critical, governance challenge. The governing body sets the moral and ethical ‘tone’ for the organisation, determines strategy and decides what is important. Executive Management’s role is to implement the governing body’s intentions, which includes determining the organisation’s approach to process and methodology, and middle and lower level management’s role is to implement these directives (for more on this see: Governance Systems & Management Systems). The governance challenge is working out a way to implement efficient systems that also encourage an appropriate degree of innovation and experimentation. The ultimate level in CMMI and OPM3 is ‘continuous improvement’. But improvement means change and change requires research, experimentation and risk taking. As Albert Einstein once said, “If we knew what it was we were doing, it would not be called research, would it?”

To stay with the Roman theme of this post: Finis origine pendet (quoting 1st century AD Roman poet and astronomer Marcus Manilius: The end depends upon the beginning). The challenge of effective governance is to encourage flexibility and innovation where this is appropriate (ie, to encourage the taking of appropriate risks to change and improve the organisation) whilst ensuring due process is followed when this is important. The challenge is knowing when each is appropriate and then disseminating this understanding throughout the organisation.

Organisations that follow the Roman approach to governance and avoid taking any form risk are doomed to fade into oblivion sooner or later.

_______________

Note: According to the usual interpretation, the term pontifex literally means “bridge-builder” (pons + facere). The position of bridge-builder was an important one in Rome, where the major bridges were over the Tiber, the sacred river (and a deity). Only prestigious authorities with sacral functions could be allowed to ‘disturb’ it with mechanical additions.

However, the term was always understood in its symbolic sense as well: the pontifices were the ones who smoothed the ‘bridge’ between gods and men. In ancient Rome, the Pontifex Maximus (Latin, literally: greatest pontiff) was the high priest of the College of Pontiffs (Collegium Pontificum), the most important religious role in the republic. The word pontifex later became a term used for bishops in the early Catholic Church and the Bishop of Rome, the Pope, the highest of bridge-builders sumus pontiff.

What’s the Probability??

The solution to this question is simple but complex….

Probability2

There is a 1 in 10 chance the ‘Go Live’ date will be delayed by Project 1
There is a 1 in 10 chance the ‘Go Live’ date will be delayed by Project 2
There is a 2 in 10 chance the ‘Go Live’ date will be delayed by Project 3

What is the probability of going live on March 1st?

To understand this problem let’s look at the role of dice:

If role the dice and get a 1 the project is delayed, any other number it is on time or early.
If you role 1 dice, the probability is 1 in 6 it will land on 1 = 0.1666 or 16.66% therefore there is a 100 – 16.66 = 83.34% probability of success.

Similarly, if you roll 2 dice, there are 36 possible combinations, and the possibilities of losing are: 1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 6:1, 5:1, 4:1, 3:1, 2:1. (11 possibilities)

diceposs

The way this is calculated (in preference to using the graphic) is to take the number of ways a single die will NOT show a 1 when rolled (five) and multiply this by the number of ways the second die will NOT show a 1 when rolled. (Also five.) 5 x 5 = 25. Subtract this from the total number of ways two dice can appear (36) and we have our answer…eleven.
(source: http://www.edcollins.com/backgammon/diceprob.htm)

Therefore the probability of rolling a 1 and being late are 11/36 = 0.3055 or 30.55%, therefore the probability of success is 100 – 30.55 = 69.45% probability of being on time.

If we roll 3 dice we can extend the calculation above as follows:
The number of possible outcomes are 6 x 6 x 6 = 216
The number of ways not to show a 1 are 5 x 5 x 5 = 125

Meaning there are 216 combinations and there are 125 ways of NOT rolling a 1
leaving 216 – 125 = 91 possibilities of rolling a 1
(or you can do it the hard way: 1:1:1, 1:1:2, 1:1:3, etc.)

91/216 = 0.4213 or 42.13% probability of failure therefore there is a
100 – 42.13 = 57.87% probability of success.

So going back to the original problem:

Project 1 has a 1 in 10 chance of causing a delay
Project 2 has a 1 in 10 chance of causing a delay
Project 3 has a 1 in 5 chance of causing a delay

There are 10 x 10 x 5 = 500 possible outcomes and within this 9 x 9 x 4 = 324 ways of not being late. 500 – 324 leaves 176 ways of being late. 176/500 = 0.352 or a 35.2% probability of not making the ‘Go Live’ date.
Or a 100 – 35.2 = 64.8% probability of being on time.

The quicker way to calculate this is simply to multiply the probabilities together:

0.9 x 0.9 x 0.8 = 64.8%

These calculations have been added to our White Paper on Probability.

A Technical question for the risk experts??

Three schedule activities of 10 days duration each need to be complete before their outputs can be integrated.

Probability

Activity 1 & 2 both have a 90% probability of achieving the estimated duration of 10 days.

Activity 3 has an 80% probability of achieving the 10 days.

Scenario 1:

The three activities are in parallel with no cross dependencies, what is the probability of the integration activity starting on schedule?

Possible solution #1

There is a 10% probability of the start being delayed by Activity 1 overrunning.
There is a 10% probability of the start being delayed by Activity 2 overrunning.
There is a 20% probability of the start being delayed by Activity 3 overrunning.

Therefore in aggregate there is a 40% probability of the start being delayed meaning there is a 60% probability of the integration activity starting on time.

Possible solution #2

The three activities are in parallel and the start of the integration is dependent on all 3 activities achieving their target duration. The probability of a ‘fair coin toss’ landing on heads 3 times in a row is 0.5 x 0.5 x 0.5 = 0.125  (an independent series)

Therefore the probability of the three activities achieving ‘on time’ completion as opposed to ‘late’ completion should be 0.9 x 0.9 x 0.8 = 0.648 or a 64.8% probability of the integration activity starting on time.

Which of these probabilities are correct?

Scenario #2

The more usual project scheduling situation where activities 1, 2 and 3 are joined ‘Finish-to-Start’ in series (an interdependent series). Is there any way of determining the probability of activity 4 starting on time from the information provided or are range estimates needed to deal with the probability of the activities finishing early as well as late?

There is a correct answer and an explanation – see the next post
(its too long for a comment)

Value is created by embracing risk effectively

The latest briefing from the real ‘Risk Doctor’, Dr David Hillson #75: RESOLVING COBB’S PARADOX? starts with the proposition: When Martin Cobb was CIO for the Secretariat of the Treasury Board of Canada in 1995, he asked a question which has become known as Cobb’s Paradox: “We know why projects fail; we know how to prevent their failure – so why do they still fail?” Speaking at a recent UK conference, the UK Government’s adviser on efficiency Sir Peter Gershon laid down a challenge to the project management profession: “Projects and programmes should be delivered within cost, on time, delivering the anticipated benefits.” Taking up the Gershon Challenge, the UK Association for Project Management (APM) has defined its 2020 Vision as “A world in which all projects succeed.” The briefing then goes on to highlight basic flaw in these ambitions – the uncertainty associated with various types of risk. (Download the briefing from: http://www.risk-doctor.com/briefings)

Whilst agreeing with the concepts in David’s briefing, I don’t feel he has gone far enough! Fundamentally, the only way to achieve the APM objective of a “world in which all projects succeed” is to stop doing projects! We either stop doing projects – no projects – no risks – no failures. Or approximate ‘no risk’ by creating massive time and cost contingencies and taking every other precaution to remove any vestige of uncertainty; the inevitable consequence being to make projects massively time consuming and unnecessarily expensive resulting in massive reductions in the value created by the few projects that can be afforded.

The genesis of Cobb’s Paradox was a workshop focused on avoidable failures caused by the repetition of known errors – essentially management incompetence! No one argues this type of failure should be tolerated although bad management practices mainly at the middle and senior management levels in organisations and poor governance oversight from the organisation’s mean this type of failing is still all too common. (for more on the causes of failure see: Project or Management Failures )

However, assuming good project management practice, good middle and senior management support and good governance oversight, in an organisation focused on maximising the creation of value some level of project failure should be expected, in fact some failure is desirable!

In a well-crafted portfolio with well managed projects, the amount of contingency included within each project should only be sufficient to off-set risks that can be reasonably expected to occur including variability in estimates and known-unknowns that will probably occur. This keeps the cost and duration of the individual projects as low as possible, but, using the Gartner definitions of ‘failure’ guarantees some projects will fail by finishing late or over budget.

Whilst managing unknown-unknowns and low probability risks should remain as part of the normal project risk management processes, contingent allowances for this type of risk should be excluded from the individual projects. Consequently, when this type of risk eventuates, the project will fail. However, the effect of the ‘law of averages’ means the amount of additional contingency needed at the portfolio level to protect the organisation from these ‘expected failures’ is much lower than the aggregate ‘padding’ that would be needed to be added to each individual project to achieve the same probability of success/failure. (For more on this see: Averaging the Power of Portfolios)

Even after all of this there is still a probability of overall failure. If there is a 95% certainty the portfolio will be successful (which is ridiculously high), there is still a 5% probability of failure. Maximum value is likely to be achieved around the 80% probability of success meaning an inevitable 20% probability of failure.

Furthermore, a focus on maximising value also means if you have better project managers or better processes you set tighter objectives to optimise the overall portfolio outcome by accepting the same sensible level of risk. Both sporting and management coaches understand the value of ‘stretch assignments’ – people don’t know how good they are until they are stretched! The only problem with failure in these circumstances is failing to learn and failing to use the learning to improve next time. (For more on this see: How to Suffer Successfully)

The management challenge is firstly to eliminate unnecessary failures by improving the overall management and governance of projects within an organisation. Then rather than setting a totally unachievable and unrealistic objective that is guaranteed to fail, accept that risk is real and use pragmatic risk management that maximises value. As David points out in his briefing: “Projects should exist in a risk-balanced portfolio. The concept of risk efficiency should be built into the way a portfolio of projects is built, with a balance between risk and reward. This will normally include some high-risk/high-reward projects, and it would not be surprising if some of these fail to deliver the expected value.”

Creating the maximum possible value is helped by skilled managers, effective processes and all of the other facets of ‘good project management’ but not if these capabilities are wasted in a forlorn attempt to ‘remove all risk’ and avoid all failure. The skill of managing projects within an organisation’s overall portfolio is accepting sensible risks in proportion to the expected gains and being careful not to ‘bet the farm’ on any one outcome. Then by actively managing the accepted risks the probability of success and value creation are both maximised.

So in summary, failure is not necessary bad, provided you are failing for the ‘right reason’ – and I would suggest getting the balance right is the real art of effective project risk management in portfolios!

Stakeholders and Risk

Probably the biggest single challenge in stakeholder communication is dealing with risk – I have touched on this subject a few times recently because it is so important at all levels of communication.

Projects are by definition uncertain – you are trying to predict a future outcome and as the failure of economic forecasts and doomsday prophets routinely demonstrate (and bookmakers have always known), making predictions is easy; getting the prediction correct is very difficult.

Most future outcomes will become a definite fact; only one horse wins a race, the activity will only take one precise duration to complete. What is uncertain is what we know about the ‘winner’ or the duration in advance of the event. The future once it happens will be a precise set of historical facts, until that point there is always a degree of uncertainty, and this is where the communication challenge starts to get interesting……

The major anomaly is the way people deal with uncertainty. As Douglas Hubbard points out in his book the Failure of Risk Management: “He saw no fundamental irony in his position: Because he believed he did not have enough data to estimate a range, he had to estimate a point”. If someone asks you what a meal costs in your favourite restaurant, do you answer precisely $83.56 or do you say something like “usually between $70 and $100 depending on what you select”? An alternative answer would be ‘around $85’ but this is less useful than the range answer because your friend still needs to understand how much cash to take for the meal and this requires an appreciation of the range of uncertainties.

In social conversations most people are happy to provide useful information with range estimates and uncertainty included to make the conversation helpful to the person needing to plan their actions. In business the tendency is to expect the precisely wrong single value. Your estimate of $83.56 has a 1 in 3000 chance of actually occurring (assuming a uniform distribution of outcomes in a $30 range). The problem of precisely wrong data is discussed in Is what you heard what I meant?.

The next problem is in understanding how much you can reasonably expect to know about the future.

  • Some future outcomes such as the roll of a ‘true dice’ have a defined range (1 to 6) but previous rolls have absolutely no influence on subsequent rolls, any number can occur on any roll.
  • Some future outcomes can be understood better if you invest in appropriate research, the uncertainty cannot be removed, but the ‘range’ can be refined.

This ‘know-ability’ interacts with the type of uncertainty. Some future events (risks) simply will or won’t happen (eg, when you drop your china coffee mug onto the floor it will either break or not break – if it’s broken you bin the rubbish, if it’s not broken you wash the mug and in both situations you clean up the mess). Other uncertainties have a range of potential outcomes and the range may be capable of being influenced if you take appropriate measures.

The interaction of these two factors is demonstrated in the chart below, although it is important to recognise there are not absolute values most uncertainties tend towards one option or the other but apart from artificial events such as the roll of a dice, most natural uncertainties occur within the overall continuum.

Stakeholders and Risk - Risk Matrix

Putting the two together, to communicate risk effectively to stakeholders (typically clients or senior managers) your first challenge is to allow uncertainty into the discussion – this may require a significant effort if your manager wants the illusion of certainty so he/she can pretend the future is completely controllable and defined. This type of self-delusion is dangerous and it’s you who will be blamed when the illusion unravels so its worth making the effort to open up the discussion around uncertainty.

Then the second challenge is to recognise the type of uncertainty you are dealing with based on the matrix above and focus your efforts to reduce uncertainty on the factors where you can learn more and can have a beneficial effect on future outcomes. The options for managing the four quadrants above are quite different:

  • Aleatoric Incidents have to be avoided (ie, don’t drop the mug!)
  • Epistemic Incidents need allowances in your planning – you cannot control the weather but you can make appropriate allowances – determining what’s appropriate needs research.
  • Aleatoric Variables are best avoided but the cost of avoidance needs to be balanced against the cost of the event, the range of outcomes and your ability to vary the severity. You can avoid a car accident by not driving; most people accept the risk and buy insurance.
  • Epistemic Variables are usually the best options for understanding and improvement. Tools such as Monte Carlo analysis can help focus your efforts on the items within the overall project where you can get the best returns on your investments in improvement.

Based on this framework your communication with management can be used to help focus your efforts to reduce uncertainty within the project appropriately. You do not need to waste time studying the breakability of mugs when dropped; you need to focus on avoiding the accident in the first place. Conversely, understanding the interaction of variability and criticality on schedule activities to proactively managing those with the highest risk is likely to be valuable.

Now all you have to do is convince your senior stakeholders that this is a good idea; always assuming you have any after the 21st December!*

____________________

*The current ‘doomsday’ prophecy is based on the Mayan Calendar ending on 21st December 2012 but there may be other reasons for this:

Stakeholders and Risk Myan Prediction

Averaging the Power of Portfolios

The interaction between dependent or connected risk and independent risk is interesting and will significantly change the overall probability of success or failure of an endeavour or organisation.

As discussed in my last post on ‘The Flaw of Averages’  using a single average value for an uncertainty is a recipe for disaster. But there is a difference between averaging, connecting and combining uncertainties (or risk).

Adding risk

Where risk events are connected, the ability to model and appreciate the effect of the risk events interacting with each other is difficult. In ‘The Flaw of Averages’ Sam Shaw uses the simile of wobbling a step ladder to determine the uncertainty of how safe the ladder is to climb. You can test the stability of one ladder by giving it a good ‘wobble’. However, if you are trying to determine the stability of a plank between two stepladders doubling the information from wobbling just one is not a lot of help. Far more sophisticated modelling is needed and even then you cannot be certain the full set of potential interactions is correctly combined in the model. The more complex the interactions between uncertainties, the less accurate the predictive model.

However, when the risks or uncertainties are independent, combining the risks through the creation of a portfolio of uncertainties reduces the overall uncertainty quite dramatically.

The effect of portfolios

Consider a totally unbiased dice, any one throw can end up anywhere and every value between 1 & 6 has an equal probability of being achieved. The more throws, the more even the results for each possibility and consequently there is no possibility of determining the outcome!

The distribution after 10, 100 and 1000 throws.

As the number of throws increase, the early distortions apparent after 10 throws smooth out and after 1000 throws the probabilities are almost equal.

However, combine two dice and total the score results in a very different outcome. Whilst it is possible to throw any value between 2 & 12, the probability of achieving a number nearer the middle of the range is much higher than the probability of achieving a 2 or a 12. The potential range of outcomes starts to approximate a ‘normal distribution curve’ (or a bell curve). The reason for this is there is only one combination of numbers that will produce a 2 or a 12; there are significantly more combinations that can make 7.

The more dice you add to the ‘throw’, the closer the curve becomes to a ‘normal distribution’ (or bell curve), which is normally what you expect/get, which is the origin of the name!

The consequence of this phenomenon is to demonstrate that the creation of a portfolio of projects will have the effect of generating a normal distribution curve for the outcome of the overall portfolio, which makes the process of portfolio management a more certain undertaking than the management of the individual projects within the portfolio. The overall uncertainty is less than the individual uncertainties……

Each project carries its level of uncertainty and has a probability of succeeding off-set by a probability of failing (see Stakeholder Risk Tolerance) but as more projects are added the probability of the overall portfolio performing more or less as expected increases, provided each of the uncertainties are independent! This effect is known as the Central Limit Theorem.

One important effect of the Central Limit Theorem is the size if the contingency needed to achieve a desired level of safety for a portfolio of projects is much smaller than the sum of the contingencies needed to achieve the same level of ‘safety’ in each of the individual projects. Risk management is a project centric process; contingency management is better managed at the portfolio level. Not only is the overall uncertainty reduced, but the portfolio manager can offset losses in one project against gains in another.

Whist this theorem is statistically valuable, the nature of most organisations constrains the potential benefit. From a statistical perspective diversity is the key; this is why most conservative investment portfolios are diversified. However, project portfolios tend to be concentrated in the area of expertise of the organisation which removes some of the randomness needed for the Central Limit Theorem to have its full effect.

It is also important to remember that whilst creating a portfolio will reduce uncertainty, no portfolio can remove all uncertainty.

In addition to the residual risk of failure inherent in every project, there is always the possibility of a ‘black swan’ lurking in the future. Originally conceptualized by philosopher Karl Popper and refined by N. N. Taleb, a ‘black swan’ is a risk event that has never occurred before, if it did occur would have and extreme impact and is easy to explain after the event, but is culturally impossible to predict in advance (ie, the event could be foreseen if someone is asked to think about it but it is nearly impossible to think the thought for a compelling reason). For more on black swans see our blog post  and White Paper.

The Law of Averages

The Central Limit Theorem is closely aligned to The Law of Averages. The Law of Averages states that if you repeatedly take the average of the same type of uncertain number the average of the samples will converge to a single result, the true average of the uncertain number. However, as the ‘flaw of averages’ has demonstrated, this does not mean you can replace every uncertainty with an average value and some uncertain numbers never converge.

Summary

Both the Law of Averages and Central Limit Theorem are useful concepts; they are the statistical equivalent of the adage “don’t put all your eggs in one basket”. When you create a portfolio of projects, the average probability of any one project succeeding or failing remains the same as if the project was excluded from the portfolio, but the risk of portfolio suffering an overall failure becomes less as the number of projects included in the portfolio increases.

However, unlike physical laws such as gravity, these laws are not immutable – drop an apple within the earths gravitational pull and it will fall; create a portfolio and there is always a low probability that the results will not conform to normal expectations!

Certainly the probability of a portfolio of projects ‘failing’ is lower then the average probability of each project failing but a reduced level of risk still leaves a residual level of risk.