Warning: define(): Argument #3 ($case_insensitive) is ignored since declaration of case-insensitive constants is no longer supported in /home/u742613510/domains/strategy-at-risk.com/public_html/wp-content/plugins/wpmathpub/wpmathpub.php on line 65
Simulation modeling – Strategy @ Risk

Strategy @ Risk

Category: Simulation modeling

The Estimated Project Cost Distributions and the Final Project Cost
This entry is part 2 of 2 in the series The Norwegian Governmental Project Risk Assessment Scheme
Everybody believes in the exponential law of errors: the experimenters, because they think it can be proved by mathematics; and the mathematicians, because they believe it has been established by observation. (Whittaker& Robinson, 1967)

The growing use of Cost Risk Assessment models in public projects has raised some public concerns about its costs and the models ability to reduce cost overruns and correctly predict the projects final cost. We have in this article shown that the models are neither reliable nor valid, by calculating the probabilities of the projects final costs. The final cost and their probabilities indicate that the cost distributions do not adequately represent the actual cost distributions.

Introduction

In the previous post we found that the project cost distributions applied in the uncertainty analysis for 85 Norwegian public works projects were symmetric – and that they could be represented by normal distributions. Their P85/P50 ratios also suggested that they might come from the same normal distribution, since a normal distribution seemed to fit all the observed ratios. The quantile-quantile graph (q-q graph) below depicts this:

As the normality test shows, it is not exactly normal ((As the graph shows is the distribution slightly skewed to the right)), but near enough normal for all practical purposes ((The corresponding linear regression gives a value of 0.9540 for the coefficient of determination (R).)). This was not what we would have expected to find.

The question now is if the use of normal distributions representing the total project cost is a fruitful approach or not.

We will study this by looking at the S/P50 ratio that is the ratio between the final (actual) total project cost – S and the P50 cost estimate. But first we will take a look at the projects individual cost distributions.

The individual cost distributions

By using the fact that the individual project’s cost are normally distributed and by using the P50 and P85 percentiles we can estimate the mean and variance for all the projects’ the cost distributions (Cook, 2010).

In the graph below we have plotted the estimated relative cost distribution (cost/P50) for the projects with the smallest (light green) and the largest (dark green) variance. Between these curves lie the relative cost distributions for all the 85 projects.

Between the light green and the blue curve we find 72 (85%) of the projects. The area between the blue and the dark green curve contains 13 of the projects – the projects with the highest variance:

The differences between the individual relative cost distributions are therefore small. Average standard deviation for all 85 projects is 0.1035 with a coefficient of variation of 48%. For the 72 projects the average standard deviation are 0.0882 with a coefficient of variation of 36%. This is consistent with what we could see from the regression of P85 on P50.

It is bewildering that a portfolio of so diverse projects can end up with such a small range of normal distributed cost.

The S/P50 ratio

A frequency graph of the 85 observed ratios (S/P50) shows a pretty much symmetric distribution, with a pronounced peak. It is slightly positively skewed, with a mean of 1.05, a maximum value of 1.79, a minimum value of 0.41 and a coefficient of variation of 20.3%:

At first glance this seems as a reasonable result; even if the spread is large, given that the project’s total cost has normal distributions.

If the estimated cost distribution(s) gives a good representation of the underlying cost distribution, then – S – should also belong to that distribution. Have in mind that the only value we know with certainty to belong to the underlying cost distribution is – S, i.e. the final total project cost.

It is there for of interest to find out if the S/P50 ratio(s) are consistent with the estimated distributions. We will try to investigate this by different routes, first by calculating at what probability the deviation of S from P50 occurred.

What we need to find is, for each of the 85 projects, the probability of having had a final cost ratio (S/P50):

i. less or equal to the observed ratio for projects with S > P50 and
ii. Greater or equal to the observed ratio for projects with S < P50.

The graph below depicts this. The maroon circles give the final cost ratio (S/P50) and their probabilities:

A frequency graph of these probabilities should give a graph with a right tail, with most of the projects close to the 0.5 fractile (the median or P50 value), tapering off to the right as we move to higher fractiles.

We would thus anticipate that most projects have been finished at or close to the quality assurance schemes median value i.e., having had a probability of 0.5 for having had this or a lower (higher) value as final cost ratio, and that only a few would have significant deviations from this.

We will certainly not expect many of the final cost ratio probabilities above the 0.85 percentile (P85).

The final cost probability frequency graph will thus give us some of the completing information needed to assess the soundness of using methods and simulation techniques ending up with symmetric project cost distributions.

Final project cost ratio probability

The result is given in the graph below, where the red bars indicates projects that with probabilities of 85% or more should have had lower (or higher) final cost ratios:

The result is far from what we expected: the projects probabilities are not concentrated at or close to 0.5 and the frequency graph is not tapering off to the right. On the contrary, the frequency of projects increases as we move to higher probabilities for the S/P50 ratios, and the highest frequency is for projects that with high probability should have had a much less or a much higher final cost:
1. The final project cost ratio probabilities have a mean of 0.83, a median at 0.84 and a coefficient of variation of 21%.
2. Of the 85 projects, 51 % have final cost ratios that had a probability of 84% or less of being lower (or higher) and 49% have final cost ratios that had a probability of 85% or more of being lower (higher).
Almost fifty percent of the projects have thus been seriously under or over budgeted or have had large cost over- or underruns – according to the cost distributions established by the QA2 process.

The cumulative frequency distribution below gives a more detailed description:

It is difficult to say in what range the probability for the S/P85 ratio should have been for considering the estimated cost distributions to be “acceptable”. If the answer is “inside the quartile range”, then only 30% of the projects final cost forecasts can be regarded as acceptable.

The assumption of normally distributed total project costs

Based on the close relation between the P50 and P85 percentiles it is tempting to conclude that most if not all projects has had the same cost estimation validation process; using the same family of cost distributions, with the same shape parameter and assuming independent cost elements – ending up with a near normal or normal distribution for the projects total cost. I.e. all the P85/50 ratios belong to the same distribution.

If this is the case, then also the projects final costs ratios should also belong to the same distribution. In the q-q graph below, we have added the S/P50 ratios (red) to the P85/P50 ratios (green) from the first q-q graph. If both ratios are randomly drawn from the same distribution, they should all fall close onto the blue identity line:

The ratios are clearly not normaly distributed; the S/P50 ratios ends mostly up in both tails and the shape of the plotted ratios now indicates a distribution with heavy tails or may be with bimodality. The two ratios is hence most likely not from the same distribution.
A q-q graphn with only the S/P50 ratios shows however that they might be normaly distributed, but have been taken from a different distribution than the P85/P50 ratios:

The S/P50 ratios are clearly normaly distributed as they fall very close onto the identity line. The plotted ratios also indicates a little lighter tails than the corresponding theoretical distribution.

That the two sets of ratios so clearly are different is not surprising, since the S/P50 ratios have a coeficient of variation of 20% while the same metric is 4.6% for the P85/P50 ratios ((The S/P50 ratios have a mean of 1.0486 and a standard deviation of 0.2133. The same metrics for the P85/P50 ratios is 1.1069 and 0.0511.)) .

Since we want the S/P50 ratio to be as close to one as possible, we can regard the distribution of the S/P50 ratios as the QA2’s error distribution.This brings us to the question of the reliability and validity of the QA2 “certified” cost risk assessment model.

Reliability and Validity

The first that needs to be answered is then the certified model’s reliability in producing consistent results and second if the cost model really measures what we want to be measured.
1. We will try to answer this by using the S/P50 probabilities defined above to depict:
  The precision ((ISO 5725-Accuracy of Measurement Methods and Results.)) of the forecasted costs distributions by the variance of the S/P50 probabilities, and
2. The accuracy (or trueness) of the forecasts, or the closeness of the mean of the probabilities for the S/P50 ratio to the forecasts median value – 0.5.
The first will give us an answer about the model’s reliability and the second an answer about the model’s validity:
A visual inspection of the graph gives an impression of both low precision and low accuracy:
- the probabilities have a coefficient of variation of 21% and a very high density of final project costs ending up in the cost distributions tail ends, and
- the mean of the probabilities is 0.83 giving a very low accuracy of the forecasts.
The conclusion then must be that the cost models (s) are neither reliable nor valid:

Summary

We have in these two articles shown that the implementation of the QA2 scheme in Norway ends up with normally distributed project costs.

i.   The final cost ratios (S/P50) and their probabilities indicate that the cost distributions do not adequately represent the actual distributions.
ii.   The model (s) is neither reliable nor valid.
iii.   We believe that this is due to the choice of risk models and technique and not to the actual risk assessment work.
iv.   The only way to resolve this is to use proper Monte Carlo simulation models and techniques

Final Words

Our work reported in these two posts have been done out of pure curiosity after watching the program “Brennpunkt”. The data used have been taken from the program’s documentation. Based on the results, we feel that our work should be replicated by the Department of Finance and with data from the original sources, to weed out possible errors.

It should certainly be worth the effort:

i.   The 85 project here, amounts to NOK 221.005 million with
ii.   NOK 28.012 million in total deviation ((The sum of all deviations from the P50 values.)) from the P50 value
iii.   NOK 19.495 million have unnecessary been held in reserve ((The P85 amount less the final project cost > zero.)) and
iv.   The overruns ((The final project cost less the P50 amount > zero)) have been NOK 20.539 million
v.   That is, nearly every fifth “krone” of the projects budgets has been “miss” allocated
vi.   And there are many more projects to come.

References

Cook, J.D., (2010). Determining distribution parameters from quantiles.
http://biostats.bepress.com/mdandersonbiostat/paper55

Whittaker, E. T. and Robinson, G. (1967), Normal Frequency Distribution. Ch. 8 in The Calculus of Observations: A Treatise on Numerical Mathematics, 4th ed. New York: Dover, pp. 164-208, 1967. p. 179.
08/02/2015
The implementation of the Norwegian Governmental Project Risk Assessment scheme
This entry is part 1 of 2 in the series The Norwegian Governmental Project Risk Assessment Scheme
Introduction

In Norway all public investment projects with an expected budget exceeding NOK 750 million have to undergo quality assurance ((The hospital sector has its own QA scheme.)) . The oil and gas sector, and state-owned companies with responsibility for their own investments, are exempt.

The quality assurance scheme ((See, The Norwegian University of Science and Technology (NTNU): The Concept Research Programme)) consists of two parts: Quality assurance of the choice of concept – QA1 (Norwegian: KS1) ((The one page description for QA1 (Norwegian: KS1)have been taken from: NTNU’s Concept Research Programme)) and Quality assurance of the management base and cost estimates, including uncertainty analysis for the chosen project alternative – QA2 (Norwegian: KS2) ((The one page description for QA2 (Norwegian: KS2) have been taken from: NTNU’s Concept Research Programme))

This scheme is similar too many other countries’ efforts to create better cost estimates for public projects. One such example is Washington State Department of Transportations’ Cost Risk Assessment (CRA) and Cost Estimate Validation Process (CEVP®) (WSDOT, 2014).

One of the main purposes of QA2 is to set a cost frame for the project. This cost frame is to be approved by the government and is usually set to the 85% percentile (P85) of the estimated cost distribution. The cost frame for the responsible agency is usually set to the 50% percentile (P50). The difference between P50 and P85 is set aside as a contingency reserve for the project. This is reserves that ideally should remain unused.

The Norwegian TV program “Brennpunkt” an investigative program sponsored by the state television channel NRK put the light on the effects of this scheme ((The article also contains the data used here)):

The investigation concluded that the Ministry of Finance quality assurance scheme had not resulted in reduced project cost overruns and that the process as such had been very costly.

This conclusion has of course been challenged.

The total cost for doing the risk assessments of the 85 projects was estimated to approx. NOK 400 million or more that $60 million. In addition, in many cases, comes the cost of the quality assurance of choice of concept, a cost that probably is much higher.

The Data

The data was assembled during the investigation and consists of six setts where five have information giving the P50 and P85 percentiles. The last set gives data on 29 projects finished before the QA2 regime was implemented (the data used in this article can be found as an XLSX.file here):

The P85 and P50 percentiles

The first striking feature of the data is the close relation between the P85 and P50 percentiles:

In the graph above we have only used 83 of the 85 projects with known P50 and P85. The two that are omitted are large military projects. If they had been included, all the details in the graph had disappeared. We will treat these two projects separately later in the article.

A regression gives the relationship between P85 and P50 as:

P85 = (+/- 0.0113+1.1001)* P50, with R= 0.9970

The regression gives an exceptionally good fit. Even if the graph shows some projects deviating from the regression line, most falls on or close to the line.

With 83 projects this can’t be coincidental, even if the data represents a wide variety of government projects spanning from railway and roads to military hardware like tanks and missiles.

The Project Cost Distribution

There is not much else to be inferred about the type of cost distribution from the graph. We do not know whether those percentiles came from fitted distributions or from estimated Pdf’s. This close relationship however leads us to believe that the individual projects cost distributions are taken from the same family of distributions.

If this family of distributions is a two-parameter distribution, we can use the known P50 and P85 ((Most two-parameter families have sufficient flexibility to fit the P50 and P85 percentiles.)) percentiles to fit a number of distributions to the data.

This use of quantiles to estimate the parameters of an a priori distribution have been described as “quantile maximum probability estimation” (Heathcote & al., 2004). This can be done by fitting a number of different a priori distributions and then compare the sum log likelihoods of the resulting best fits for each distribution, to find the “best” family of distributions.

Using this we anticipate finding cost distributions with the following properties:
1. Nonsymmetrical, with a short left and a long right tail i.e. being positive skewed and looking something like the distribution below (taken from a real life project):
2. The left tail we would expect to be short after the project has been run through the full QA1 and QA2 process. After two such encompassing processes we would believe that most, even if not all, possible avenues for cost reduction and grounds for miscalculations have been researched and exhausted – leaving little room for cost reduction by chance.

3. The right tail we would expect to be long taking into account the possibility of adverse price movements, implementation problems, adverse events etc. and thus the possibility of higher costs. This is where the project risk lies and where budget overruns are born.

4. The middle part should be quite steep indicating low volatility around “most probable cost”.

Estimating the Projects Cost Distribution

To simplify we will assume that the above relation between P50 and P85 holds, and that it can be used to describe the resulting cost distribution from the projects QA2 risk assessment work. We will hence use the P85/P50 ratio ((If costs is normally distributed: C ∼ N (m, s²), then Z = C/m ∼ N (1, s²/ m2). If costs is gamma distributed: C ∼ Γ (a, λ) then Z = C/m ∼ Γ (1, λ).)) to study the cost distributions. This implies that we are looking for a family of distributions that have the probability of (X<1) =0.5 and the probability of (x<1.1) =0.85 and being positive skewed. This change of scale will not change the shape of the density function, but simply scale the graph horizontally.

Fortunately the MD Anderson Cancer Centre has a program – Parameter Solver ((The software can be downloaded from: https://biostatistics.mdanderson.org/SoftwareDownload/SingleSoftware.aspx?Software_Id=6 )) – that can solve for the distribution parameters given the P50 and P85 percentiles (Cook, 2010). We can then use this to find the distributions that can replicate the P50 and P85 percentiles.

We find that distributions from the Normal, Log Normal, Gamma, Inverse Gamma and Weibull families will fit to the percentiles. All the distributions however are close to being symmetric with the exception of the Weibull distribution that has a left tail. A left tail in a budgeted cost distribution usually indicates over budgeting with the aim of looking good after the project has been finished. We do not think that this would have passed the QA2 process – so we don’t think that it has been used.

We believe that it is most likely that the distributions used are of the Normal, Gamma or of the Gamma derivative Erlang ((The Erlang distribution is a Gamma distribution with integer shape parameter.)) type, due to their convolution properties . That is, sums of independent identically distributed variables having one of these particular distributions come from the same distribution family. This makes it possible to simplify risk models of the cost only variety by just summing up the parameters ((For the Normal. Gamma and Erlang distributions this implies summing up the shape parameters of the individual cost elements distributions: If X and Y are normally distributed: X ∼ N (a, b²) and Y∼ N (d, e²) and X is independent of Y, then Z=X + Y is N (a + d, b² + e²), and if k is a strictly positive constant then Z=k*X is N (k*a, k²* b²). If X and Y are gamma distributed: X ∼ Γ (a, λ) and Y∼ Γ (b, λ) and X is independent of Y, then X + Y is Γ (a +b, λ), and if k is a strictly positive constant then c*X is Γ (k*a, λ).)) of the cost elements to calculate the parameters of the total cost distribution.

This have the benefit of giving the closed form for the total cost distribution compared to Monte Carlo simulation where the closed form of the distribution, if it exists, only can be found thru the exercise we have done here.

This property can as well be a trap, as the adding up of cost items quickly gives the distribution of the sum symmetrical properties before it finally ends up as a Normal distribution ((The Central Limit Theorem gives the error in a normal approximation to the gamma distribution as n-1/2 as the shape parameter n grows large. For large k the gamma distribution X ∼ Γ (k, θ) converges to a normal distribution with mean µ = k*θ and variance s²= k*θ². In practice it will approach a normal distribution with the shape parameter > 10.)).

The figures in the graph below give the shapes for the Gamma and Normal distribution with the percentiles P50=1. and P85 = 1.1:

The Normal distribution is symmetric and the Gamma distribution is also for all practical purposes symmetric. We therefore can conclude that the distributions for total project cost used in the 83 projects have been symmetric or close to symmetric distributions.
This result is quite baffling; it is difficult to understand why the project cost distributions should be symmetric.

The only economic explanation have to be that the expected cost of the projects are estimated with such precision that any positive or negative deviations are mere flukes and chance outside foreseeability and thus not included in the risk calculations.

But is this possible?

The two Large Military Projects

The two projects omitted from the regression above: new fighter planes and frigates have values of the ratio P85/P50 as 1.19522 and 1.04543, compared to the regression estimate of 1.1001 for the 83 other projects. They are however not atypical, other among the 83 projects have both smaller (1.0310) and larger (1.3328) values for the P85/P50 ratio. Their sheer size however with a P85 of respective 68 and 18 milliard NOK, gives them a too high weight in a joint regression compared to the other projects.

Never the less, the same comments made above for the other 83 projects apply for these two projects. A regression with the projects included would have given the relationship between P85 and P50 as:

P85 = (+/- 0.0106+1.1751)* P50, with R= 0.9990.

And as shown in the graph below:

This graph again depicts the surprisingly low variation in all the projects P85/P50 ratios:

The ratios have in point of fact a coefficient of variation of only 4.7% and a standard deviation of 0.052 – for the all the 85 projects.

Conclusions

The Norwegian quality assurance scheme is obviously a large step in the direction of reduced budget overruns in public projects. (See: Public Works Projects)

Even if the final risk calculation somewhat misses the probable project cost distribution will the exercises described in the quality assurance scheme heighten both the risk awareness and the uncertainty knowingness. All, contributing to the common goal – reduced budget under- and overruns and reduced project cost.

It is nevertheless important that all elements in the quality assurance process catches the project uncertainties in a correct way, describing each projects specific uncertainty and its possible effects on project cost and implementation (See: Project Management under Uncertainty).

From what we have found: widespread use of symmetric cost distributions and possibly the same type of distributions across the projects, we are a little doubtful about the methods used for the risk calculations. The grounds for this are shown in the next two tables:

The skewness ((The skewness is equal to two divided by the square root of the shape parameter.)) given in the table above depends only on the shape parameter. The Gamma distribution will approach a normal distribution when the parameter larger than ten. In this case all projects’ cost distributions approach a normal distribution – that is a symmetric distribution with zero skewness.

To us, this indicates that the projects’ cost distribution reflects more the engineer’s normal calculation “errors” than the real risk for budget deviations due to implementation risk.

The kurtosis (excess kurtosis) indicates the form of the peak of the distribution. Normal distributions have zero kurtosis (mesocurtic) while distributions with a high peak have a positive kurtosis (leptokurtic).

It is stated in the QA2 that the uncertainty analysis shall have “special focus on … Event uncertainties represented by a binary probability distribution” If this part had been implemented we would have expected at least more flat-topped curves (platycurtic) with negative kurtosis or better not only unimodal distributions. It is hard to see traces of this in the material.

So, what can we so far deduct that the Norwegian government gets from the effort they spend on risk assessment of their projects?

First, since the cost distributions most probably are symmetric or near symmetric, expected cost will probably not differ significantly from the initial project cost estimate (the engineering estimate) adjusted for reserves and risk margins. We however need more data to substantiate this further.

Second, the P85 percentile could have been found by multiplying the P50 percentile by 1.1. Finding the probability distribution for the projects’ cost has for the purpose of establishing the P85 cost figures been unnecessary.

Third, the effect of event uncertainties seems to be missing.

Fourth, with such a variety of projects, it seems strange that the distributions for total project cost ends up being so similar. There have to be differences in project risk from building a road compared to a new Opera house.

Based on these findings it is pertinent to ask what went wrong in the implementation of QA2. The idea is sound, but the result is somewhat disappointing.

The reason for this can be that the risk calculations are done just by assigning probability distributions to the “aggregated and adjusted engineering “cost estimates and not by developing a proper simulation model for the project, taking into consideration uncertainties in all factors like quantities, prices, exchange rates, project implementation etc.

We will come back in a later post to the question if the risk assessment never the less reduces the budgets under- and overrun.

References

Cook, John D. (2010), Determining distribution parameters from quantiles. http://www.johndcook.com/quantiles_parameters.pdf

Heathcote, A., Brown, S.& Cousineau, D. (2004). QMPE: estimating Lognormal, Wald, and Weibull RT distributions with a parameter-dependent lower bound. Journal of Behavior Research Methods, Instruments, and Computers (36), p. 277-290.

Washington State Department of Transportation (WSDOT), (2014), Project Risk Management Guide, Nov 2014. http://www.wsdot.wa.gov/projects/projectmgmt/riskassessment

Endnotes
01/12/2014
Project Management under Uncertainty
You can’t manage what you can’t measure
You can’t measure what you can’t define
How do you define something that isn’t known?

DeMarco, 1982

1.     Introduction

By the term Project we usually understand a unique, one-time operation designed to accomplish a set of objectives in a limited time frame. This could be building a new production plant, designing a new product or develop new software for a specific purpose.

A project usually differ from normal operations by; being a onetime operation, having a limited time horizon and budget, having unique specifications and by working across organizational boundaries. A project can be divides into four phases: project definition, planning, implementation and project phase-out.

2.     Project Scheduling

The project planning phase, which we will touch upon in this paper, consists of braking down the project into tasks that must be accomplished for the project to be finished.

The objectives of the project scheduling are to determine the earliest start and finish of each task in the project. The aim is to be able to complete the project as early as possible and to calculate the likelihood that the project will be completed within a certain time frame.

The dependencies[i] between the tasks determine their predecessor(s) and successor(s) and thus their sequence (order of execution) in the project[1]. The aim is to list all tasks (project activities), their sequence and duration[2] (estimated activity time length). The figure[ii] below shows a simple project network diagram, and we will in the following use this as an example[iii].

This project thus consists of a linear flow of coordinated tasks where in fact time, cost and performance can vary randomly.

A convenient way of organizing this information is by using a Gantt[iv] chart. This gives a graphic representation of the project’s tasks, the expected time it takes to complete them, and the sequence in which they must be done.

There will usually be more than one path (sequence of tasks) from the first to the last task in a project. The path that takes the longest time to complete is the projects critical path. The objective of all this is to identify this path and the time it takes to complete it.

3.     Critical Path Analysis

The Critical Path (CP)[v] is defined as the sequence of tasks that, if delayed – regardless of whether the other project tasks are completed on or before time – would delay the entire project.

The critical path is hence based on the forecasted duration of each task in the project. These durations are given as single point estimates[3] implying that the project’s tasks duration contain no uncertainty (deterministic). This is obviously wrong and will often lead to unrealistic project estimates due to the inherent uncertainty in all project work.

Have in mind that: All plans are estimates and are only as good as the task estimates.

As a matter of fact many different types of uncertainty can be expected in most projects:
1. Ordinary uncertainty, where time, cost and performance can vary randomly, but inside predictable ranges. Variations in task durations will cause the projects critical path to shift, but this can be predicted and the variation in total project time can be calculated.
2. Foreseen uncertainty, where a few known factors (events) can affect the project but in an unpredictable way[4]. This is projects where tasks and events occur probabilistic and contain logical relationships of a more complicated nature. E.g. from a specific event some tasks are undertaken with certainty while others probabilistically (Elmaghraby, 1964) and (Pritsker, 1966). The distribution for total project time can still be calculated, but will include variation from the chance events.
3. Unforeseen uncertainty, where one or more factors (events) cannot be predicted. This will imply that decisions points about the projects implementation have to be included at one or more points in the projects execution.
As a remedy to the critical path analysis inadequacy to the existence of ordinary uncertainty, the Program Evaluation and Review Technique (PERT[vi]) analysis was developed. PERT is a variation on Critical Path Analysis that takes a slightly more skeptical view of the duration estimates made for each of the project tasks.

PERT uses a tree-point estimate,[vii] based on the forecast of the shortest possible task duration, the most likely task duration and the worst-case task duration. The tasks expected duration is then calculated as a weighted average of these tree estimates of the durations.

This is assumed to help to bias time estimates away from the unrealistically short time-scales that often is the case.

4. CP, PERT and Monte Carlo Simulation

The two most important questions we want answered are:
- How long will it take to do the project?
- How likely is the project to succeed within the allotted time frame?
- In this example the projects time frame is set to 67 weeks.
We will use the Critical Path method, PERT and Monte Carlo simulation to try to answer these questions, but first we need to make some assumptions on the variability of the estimated task durations. We will assume that the durations are triangular distributed and that the actual durations can be both higher and lower than their most likely value.

The distributions will probably have a right tail since underestimation is common when assessing time and cost (positively skewed), but sometime people deliberately overestimate to avoid being responsible for later project delay (negatively skewed). The assumptions of the tasks duration are given in the table below:

The corresponding paths, critical path and project durations is given in the table below. The critical path method finds path #1 (tasks: A,B,C,D,E) as the critical path and thus expected project duration to 65 weeks. The second question however cannot be answered by using this method. So, in regard to probable deviations from expected project duration, the project manager is left without any information.

By using PERT, calculating expected durations and their standard deviation as described in endnote vii, we find the same critical path and roughly the same expected project duration (65.5 weeks), but since we now can calculate the estimate’s standard deviation we can find the probability of the project being finished inside the projects time frame.

By assuming that the sum of task durations along the critical path is approximately normal distributed, we find that the probability of having the project finished inside the time frame of 67 weeks to 79%. Since this gives is a fairly high probability of project success the manager can rest contentedly – or can she?

If we repeat the exercise, but now using Monte Carlo simulation we find a different answer. We can no longer with certainty establish a critical path. The tasks variability can in fact give three different critical paths. The most likely is path #1 as before, but there is a close to 30% probability that path #4 (tasks: A,B,C,G,E) will be the critical path. It is also possible, even if the probability is small (<5%), that path #3 (tasks: A,F,G,E) is the critical path (see figure below).So, in this case we cannot use the critical path method, it will give wrong answers and misleading information to the project manager and. More important is the fact that the method cannot use all the information we have about the project’s tasks, that is to say their variability.

A better approach is to simulate project time to find the distribution for total project duration. This distribution will then include the duration of all critical paths that may arise during the project simulation, given by the red curve in figure below:

This figure gives the cumulative probability distribution for the possible critical paths duration (Path#: 1,3,4) as well as for total project duration. Since path #1 consistently have long duration times there are only in ‘extreme’ cases that path #4 is the critical path. Most strikingly is the large variation in path #3’s duration and the fact that it can end up in some of the simulation’s runs as critical path.

The only way to find the distribution for total project duration is for every run in the simulation to find the critical path and calculate its duration.

We now find the expected total project duration to be 67 weeks, one week more than what the CPM and PERT gave, but more important, we find that the probability of finishing the project inside the time frame is only 50%.

By neglecting the probability that the critical path might change due to task variability PERT is underestimating project variance and thus the probability that the project will not finish inside the expected time frame.

Monte Carlo models like this can be extended to include many types of uncertainty belonging to the classes of foreseen and unforeseen uncertainty. However, it will only be complete when all types of project costs and their variability are included.

5. Summary

Key findings in comparative studies show that using Monte Carlo along with project planning techniques allows better understanding of project uncertainty and its risk level as well as provides project team with the ability to grasp various possible courses of the project within one simulation procedure.

Notes

[1] This can be visualized in a Precedence Diagram also known as a Project Network Diagram.In a Network Diagram, the start of an activity must be linked to the end of another activity

[2] An event or a milestone is a point in time having no duration. A Precedence Diagram will always have a Start and an End event.

[3] As a “best guess” or “best estimate” of a fixed or random variable.

[4] E.g. repetition of tasks.

Endnotes

[i] There are four types of dependencies in a Precedence Diagram:
1. Finish-Start: A task cannot start before a previous task has ended.
2. Start-Start: There is a defined relationship between the start of tasks.
3. Finish-Finish: There is a defined relationship between the end dates of tasks.
4. Start-Finish: There is a defined relationship between the start of one task and the end date of a successor task.
[ii] Taken from the Wikipedia article: Critical path drag, http://en.wikipedia.org/wiki/Critical_path_drag

[iii] The Diagram contains more information than we will use. The diagram is mostly self-explaining, however Float (or Slack) and Drag is defined as the activity delay that the project can tolerate before the project comes in late and how much a task on the critical path is delaying project completion (Devaux,2012).

[iv] The Gantt chart was developed by Henry Laurence Gantt in the 1910s.

[v] The Critical Path Method (CPM) was developed in the late 1950s by Morgan R. Walker of DuPont and James E. Kelley, Jr. of Remington Rand.

[vi] The Program Evaluation and Review Technique (PERT) were developed by Booz Allen Hamilton and the U.S. Navy, at about the same time as the CPM. Key features of a PERT network are:
1. Events must take place in a logical order.
2. Activities represent the time and the work it takes to get from one event to another.
3. No event can be considered reached until ALL activities leading to the event are completed.
4. No activity may be begun until the event preceding it has been reached.
[vii] Assuming, that a process with a double-triangular distribution underlies the actual task durations, the tree estimated values (min, ml, max) can then be used to calculate expected value (E) and standard deviation (SD) as L-estimators, with: E = (min + 4m + max)/6 and SD = (max − min)/6.

E is thus a weighted average, taking into account both the most optimistic and most pessimistic estimates of the durations provided. SD measures the variability or uncertainty in the estimated durations.

References

Devaux, Stephen A.,(2012). “The Drag Efficient: The Missing Quantification of Time on the Critical Path” Defense AT&L magazine of the Defense Acquisition University. Retrieved from http://www.dau.mil/pubscats/ATL%20Docs/Jan_Feb_2012/Devaux.pdf

DeMarco, T, (1982), Controlling Software Projects, Prentice-Hall, Englewood Cliffs, N.J., 1982

Elmaghraby, S.E., (1964), An algebra for the Analyses of Generalized Activity Networks, Management Science, 10,3.

Pritsker, A. A. B. (1966). GERT: Graphical Evaluation and Review Technique (PDF). The RAND Corporation, RM-4973-NASA.
01/08/2014