Understanding your PMP Score

– Updated –

PMP, CAPM  and PMI-SP scores are all based on how well you answer multi-choice questions in your exam.

Unfortunately, the information provided to candidates at the end of their PMI exam is minimal. You are told if you have passed or failed based on the number of questions correctly answered (but the passing score is a secret) and you are told if your score is below average, average or above average against a limited number of ‘domains’ or PMBOK® Guide chapters in the case of CAPM. But you are not told what average represents, how it relates to the passing score, or how wide the average band is. In reality the only option is to be happy if you are in the 70% to 80% who pass and talk to your training provider if you have failed. You cannot get anything additional out of PMI.

During your training the situation is (or should be) very different!

Almost every training organisation offers its PMI exam candidates sets of questions to help consolidate their learning – we build our courses around lots of regular tests.  If the tests are well designed they use the standard 4-option multi-choice question format used by PMI and provide feedback to the trainee on the number of questions correctly answered and detailed information about the answers to all of the questions. We do, and so does everyone else I talk to.

However, what many people don’t fully appreciate is the effect of the four options in the PMI multi-choice format and why a score of 55% is definitely in the ‘fail’ category! The cause is the fact that one of the options will always be correct, therefore a set of random guesses will score 25% on average, and the more questions you answer the more the 25% will come into play. The challenge is working out what proportion of the 55% can be attributed to ‘knowing the right answer’ and what proportion is likely to be consequence of the ‘lucky 25%’.

As a starting point, any score you achieve will have a certain number of questions answered correctly because you knew the correct answer and a certain number of questions that you answered incorrectly but fluked the right answer for the wrong reason (the ‘lucky 25%’ effect). One way to assess how many of each are included in your score is to assume a known number of answers are correct because you ‘know the right answer’ and then add 25% of the rest as likely to be correct based on the ‘lucky 25%’ factor. Therefore in a set of 100 questions, if you ‘know the right answers’ to 40 questions and add 25% of the remainder (60 / 4) = 15 you get a score of 55 (or 55%). So whilst superficially a score 55% may seem to be ‘not too bad’ knowing this represents a real score of around 40% gives a very different picture. And the reason increasing your score is quite hard work is because as you real score increases, the remaining number of questions subject to the ‘25% luck’ factor decreases

This chart is one way of assessing what you really know from any score achieved in a 4-option multi-choice question exam. If you don’t know the answer to any question and guess the lot, should still score around 25%. Conversely if you know the correct answer to every question, there are none left to be affected by the ‘25% luck’ factor.

PMI%

In the chart, your total score is represented by the Pale Blue line and the questions you can assume you ‘know the correct answer to’ is represented by the Dark Blue line.

Between these two lines, there are a steadily decreasing number of questions that you did not know the right answer to but got a correct answer to based on the ‘25% luck’ factor. This 25% factor is represented by the Green Line.

To use the chart

See what your score is and convert it to a percentage. This is your ‘Total Score’ on the vertical axis, and is represented by the Light Blue line. Look down the chart to see your real score on the horizontal axis, and the Dark Blue and Green lines will give you an approximate split. For example, if you scored 62.5% in total, you probably know the answer to around 50% of the questions and the random factor added another 12.5%.

Taking into account the possible distribution of results, the ‘luck factor’ is probably somewhere between 10% and 15% (at ± 1SD) meaning there is a 65% probability you actually scored somewhere between 47.5% and 52.5% (but the effect in any specific test may be greater).

Limitations & Observations

This chart explains why PMI has set a passing score above 60% in the past (when we knew this information) and seem to have maintained this policy into the present time. A score of around 65% would assure PMI the candidate probably achieved a score of 50% or better from knowledge. But there are a number of issues with the chart.

  • The first major problem is that on average there will be a ‘25% luck’ factor affecting the questions you don’t know the right answer to, but some people will be luckier than others.  Based on a series of Monte Carlo analyses by Jesus Carrillo we can assume 1 Standard Deviation is a bit less than 5 (at the starting point), therefore the ‘luck factor’ in around 65% of all tests will fall somewhere within the two dotted green lines (see comments below).
  • The number of questions subject to the 25% luck factor decreases as the number of questions you answer correctly increases and because we are dealing with whole numbers this introduced some distortions (skew) into the assessment, as does the 3:1 ratio between wrong and right answers.  The chart is a simplification of quite complex data which hopefully does not limit its usefulness.
  • The next major problem is very few questions will be answered randomly. Most trainers tell there candidates to make sure you answer every question, but the majority of questions will have a considered answer, some considered correctly and other considered incorrectly. This factor is impossible to graph.
  • There is no way of knowing which specific questions were answered correctly for the ‘right reason’ and which were in the lucky 25%. You need to check the answers from you training organisation to see if you were ‘right for the right reason’.
  • The last consideration is the potential to remove some variability by correctly deleting ‘obviously wrong’ answers. If you can remove one option the probability of your guess being correct rises to 33%, if you can knock out two options it is up to 50%. This shifts the odds into your favour but reduces the ‘known to be correct for the right reason’ value. Again impossible to graph.

Summary

Next time you test you take a practice test use this chart to asses your real level of knowledge but make sure you check all of the answers you got right so you can identify the questions you got right for the wrong reason, the ‘25% luck’ factor.

One of the reasons we set our trainees a benchmark score of 75% in our coursework is to help overcome the factors discussed in this post. By the time you are scoring around 75% the effect of the 25% random variability has reduced well below 10% and we know you know enough to score around 65% in the exam without the luck factor. This means most people will pass most of the time; score below 75% and you are starting to rely on the 25% luck factor and there is always a chance you may not have an average amount of luck on exam day.

To help you practice your PMPCAPM  and PMI-SP answering skills we Tweet a new question every day, for today’s see: http://www.mosaicprojects.com.au/Training-PMP-Q-Today.html

Advertisements

6 responses to “Understanding your PMP Score

  1. Pat
    Assuming this is correct – you don’t need a graph. Simply multiply your Total Score by 80% and that gives your ‘Knowledge-based’ Score.
    Colin

  2. Pat, your analysis assumes that because a person has a 25% chance at an answer, they will guess 25% correctly. This is simply not true. A Monte Carlo simulation is a better tool at seeing what that distribution would look like. Using the extreme case of guessing at all the questions (100 guesses with 100 questions), 25% probability, 1000 test takers, Barreto/Howland Monte Carlo add-in and Barreto/Howland random number generator my results were as follows:

    26.9% was the average number guessed right (no surprise)
    81% maximum (one person out of 1000 scored an 81 by guessing on the entire test!)
    9.2% minimum
    Right-tailed platykurtic distribution

    I did not repeat this for fewer than 100 guesses but sensitivity to streaks should inversely proportional to the number of guesses.

    Please provide feedback/thoughts.

    • Four comments:
      1. The CLT would suggest eventually the mean will revert to 25% – 26.9% is close enough
      2. The spread and distribution of purely random guesses should generate a Normal distribution the bias in your results suggests something other than a random selection was used. But again this may be the result of a limited number assessments
      3. Agree completely with your proposition (embedded in the graph) that the range and distribution outcome should scale in proportion to the number of questions guessed.
      4. As discussed in the section on ‘weaknesses’ the major flaws in the concept are firstly most answers in a test are not ‘random’ they are considered, this imparts distortion and second, without a significant number of measurements understanding the real distribution is not possible.

      The objective of developing this chart was to show out trainees that the real value of a low score in a multi-choice test is probably much worse that it appears and to demonstrate why improving a score is hard (the better you get, the less ‘help’ from the ‘random luck’ element).

      I would be interested in seeing the shape of your distribution – a couple of boundary lines at +/- 1 SD would be a useful addition.

  3. First, let me apologize. I analyzed streaks on the first Monte Carlo to arrive at the 81% which is incorrect. Here are my statistics for the case of 100 guesses/100 questions and 10 guesses/100 questions.

    100 guesses
    Mean: 25.0514
    Median/Mode: 25
    Standard Deviation: 4.339543894
    Kurtosis: 0.036662014
    Skewness: 0.11886126
    Range: 36
    Minimum: 9
    Maximum: 45
    38.21% guesses less than the median/mode (26)

    10 guesses
    Mean: 2.509486999
    Median/Mode: 2
    Standard Deviation: 1.376996669
    Kurtosis: -0.084293268
    Skewness: 0.325006541
    Range: 9
    Minimum: 0
    Maximum: 9
    24.16% guessed less than the median/mode
    27.95% guessed within +/- 10% of the median/mode (only 2)
    47.5% guessed greater than the median/mode

    Unfortunately, the forum will not allow me to post pictures/graphs so I cannot share them but would be willing to email them to you if you provide me an email

    The data suggest that as your guesses become fewer, you statistically have a better chance of guessing correctly as can be seen by the distribution becoming more positively skewed. This is purely random guessing. As you suggest, people don’t randomly guess but have “considered guessing”. I agree but this would skew the data even more. Third, as the your need to guess becomes less you are probably better at guessing based on your knowledge of the subject so I believe this would only increase the positive skew beyond the positive skew added by the two prior points.

    To show a basic flaw with your graph, if we follow your instructions for 25% score on the test. Following your instructions, this means they guessed at the entire test and knew 0% of the answers. That really should be a band from about from 16 to 33 (95% chance, 2 sigma spread) if they guessed at the entire test.

  4. It’s unfortunate that PMI won’t give you more information on where your weaknesses. Isn’t their mandate to certify project management professionals? So, shouldn’t they give you some indication on what you should work on if you fail?
    Of course, with proper training, you would be able to understand what your strengths and weaknesses are going in. Thanks for providing the chart for analyzing practice tests. Just gotta get over that 25% luck factor!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s