Visualising Certainty in Project Estimates

When we are scoping out a project, we are asked to provide estimates for time and cost to certain confidence levels. At the start it might be +/-100%, then +/-30% and then +/-10% (or fixed price).

What we are doing is spending money and effort in order to learn more about what we are going to deliver before we get started on actually delivering it. Using a blanket level of confidence such as +/-30% could be introducing more risk into the project because most of the things that we work on actually have subset levels of certainty. Even at the very start of a project, when we are asked for +/-100% – some of the work will have +/-10% certainty, some +/-30% and the rest will be a bit of a wild guess (+/-100%). The risk that we are introducing by treating the whole estimate as a wild guess is that we will end up spending a lot more on investigation to get to the next levels of confidence than we really need to spend.

There is another way to express our levels of confidence. Imagine a stacked bar graph that could show the amount of work at the different confidence levels. At the start, and what we would call as a +/-100% level of confidence, it might actually be a very skinny layer at high confidence (+/-10 %), a bit more at medium confidence (+/-30%) and a large amount at wild guess level (+/-100%). Depending on the type of scope falling into the three different sections, we can then make better decisions about the remaining work to estimate the project and whether there is value in starting some of the higher confidence work earlier.

Fuzzy stacked bar graph Even when we have an overall estimate to +/-30%, we will still have subsets of scope sitting at the high certainty and wild guess levels also. By expressing the estimate as an overall +/-30% we are hiding crucial information about the project which could be used to make better decisions, be indicators of untapped opportunities, or risks.

For example, if we are looking at a project estimate that looks like the bar graph on the right in the above picture, we could have a good discussion about the amount of effort required to drive out the uncertainty from the wild guess components. It might be the case that we should start the other components first because that work will inform the wild guess components or completing some of that work will make it easier to determine some aspects of the wild guess work. Without this type of visualisation, we will treat the entire scope as still only having medium certainty and might start some work prematurely or delay useful work and potentially over-invest in effort to gain more perceived certainty.

Questions as Boundaries

How do we know when we are asking a good or a poor question? One sign of a poor question is when people are spending a lot of effort for seemingly little gain.

Question‘What should we do?’ can be both.

If it’s been done before and there is a predictable, best practice way to do it, then the obvious answer is ‘The next logical step.’

Obvious StepsHowever, if we are exploring complexity it can be a poor question, there are so many possible options that it can take a long time to decide what to do.

Complex SpaceIn order to explore something which has limited certainty, it is better to form a view or hypothesis and test it. This then creates a type of boundary around the thing we are trying to understand to help us make sense of it.

Questions as BoundariesQuestions like

  • Will ‘X’ happen if I do ‘Y’?
  • Is it feasible to achieve ‘X’ for ‘Y’ effort?
  • Will people buy our product?

For example, we often start a project by asking how much time and money it will take to make our idea a reality. This can drive a very large amount of effort to find out – when the project happens to be in the complex domain (as per the Cynefin Framework). If we change the question to ‘Is it feasible to make this idea a reality for $X and Y time-frame?’ it puts boundaries around the initial exploration stage and we can avoid large amounts of wasted and unfocussed effort. An example can be found in my previous post ‘Why do we Estimate?’

Questions on Cynefin FrameworkIn summary, check which type of system the question relates to according to the Cynefin Framework. Then check if the question is helping to drive useful logical work or useful exploration and if not, then change the question.

Why do we Estimate?

The first question we naturally ask when we have an idea is ‘How much will it cost?’

The main reason we do this is in order to make a decision about whether to proceed with the idea or not.

Asking the question this way will often result in a lot of work in many delivery teams, understanding the idea and then estimating how much it will cost to develop.  By the time we have done all this work it is as if we have started the project already – so we can find it very hard to stop.

It is more useful to ask ‘How much are we prepared to spend on this idea?’  A good way to do this is to use the IRACIS template and ask ourselves about the goals and drivers for this idea.  For each goal or driver, capture the item into the most appropriate box – does it Increase Revenue, Avoid Costs or Increase Service to our customers?

Next we look at the metrics and measures we would use to determine if each of the goals and drivers were met.  Would we count the number of sales? Would we look for a drop in complaints?  Could we reduce or delay investment? ….and so on…….

The next question is ‘With all of these benefits, how much should we be prepared to spend in order to meet these goals?’

This can be very hard to answer in a work context – but we do it all the time in our home lives.  For example, if I want to buy bananas, I already know how much I am prepared to spend.  If they are more than I have budgeted for, then I do not buy them.

Once we have the amount that we are prepared to spend, there are many light workshop techniques we can use to get an idea of the work involved in the solution and whether we think it is possible to build for the desired spend.

Back to the reason for estimation – to make a decision about whether to proceed or not with the idea.  If we estimate that we can build the solution for the desired spend, then let’s continue with the idea.

If not, then stop and spend no further effort on it.