An enterprise that funds only the half of its estimating loop that consults the database will consult the same database in ten years.
A programme finishes. The team is stood down over about six weeks, the good people are pulled onto the next thing before the last one is closed, a completion report is written by whoever is left, and it is filed. Eighteen months later a similar investment comes to the board, and the estimate in front of the directors was built the same way the last one was built — by experienced people, from first principles, informed by memory rather than by record.
Nobody in that sequence behaved badly. Every individual decision was defensible, including the decision to move the best people onto live work rather than onto a retrospective. And the compound effect is that the organisation's estimating capability is roughly where it was a decade ago.
This is not a local failure. In 2005, reporting research led by Bent Flyvbjerg of Aalborg University and published in the Journal of the American Planning Association, The Economist recorded a finding that ought to be more disturbing than it usually is: across 210 large rail and road projects in fourteen countries, the authors concluded that forecasts were "no more accurate now than they were 30 years ago". [SOURCE DETAILS REQUIRED] [FACT CHECK REQUIRED]
The Strategic Context
The scale of the error in that study is worth stating precisely, because it disciplines the argument. Rail passenger forecasts were, on average, 106 per cent higher than the outturn — with one in eight out by more than 400 per cent. Road forecasts were more modest, missing by over 20 per cent in more than half of cases. [FACT CHECK REQUIRED]
The same report set that against information technology performance, citing the Standish Group's annual evaluation: in 2004, on the research firm's own definition, 29 per cent of such projects succeeded, down from 34 per cent two years earlier, with cost over-runs averaging 56 per cent of original budgets and schedules running 84 per cent longer than planned. Those figures reach us twice removed — reported by a magazine, citing a commercial research firm, using that firm's own definition of success, a methodology publicly contested since. [FACT CHECK REQUIRED] They are cited here for the direction they indicate, not for their precision.
The report's own contemporary examples make the same point without any statistics at all. A £6 billion programme to put the medical records of 50 million people online was, at the time of writing, well over budget and postponed by several months; and a federal law-enforcement agency had abandoned a $170 million internal system in March 2005, two years after the first problems surfaced. [FACT CHECK REQUIRED] Both were large, well-resourced, closely watched, and staffed by people who had seen the previous generation of such programmes.
Two details in that report do more work than the headline numbers.
The first is historical. The article opens with a railway built between two English cities in the 1820s, which came in 45 per cent over budget and ran late crossing a bog. [FACT CHECK REQUIRED] The point is not that forecasting has always been hard. It is that the same error, in the same direction, at a similar magnitude, is still being made after two centuries of accumulated practice — which is not what a functioning feedback loop produces.
The second is the counter-example, and it matters because it prevents the argument becoming fatalism. The report notes that the previous year's Project of the Year award from the Project Management Institute went to a gas project in the Saudi desert, ten kilometres from the nearest road, which finished six months ahead of schedule and 27 per cent under budget on a three-year, two-billion-dollar build. [FACT CHECK REQUIRED] It is possible. It is simply not systematic.
What Organisations Believe They Have
Most enterprises believe they have a lessons-learned process, and most of them do, in the sense that a document exists. The gap is not between organisations that review and organisations that do not. It is between organisations whose next estimate is different because of a completed project, and organisations where it is not.
The useful specification of what would close that gap comes from Nick Lavingia, writing in Cost Engineering in 2003 out of a gated capital-project process. He describes four instruments, and the interesting property is where each one sits in time.
A pre-funding assessment, performed at the end of the third phase of the process, which rates the project against a database of past similar projects and recommends a cost contingency and a schedule.
Benchmarking, which he characterises as "a forward-looking quantitative approach based on the experience of thousands of past projects in the industry", used to work out what a competitor would spend and how long they would take, and to set demanding performance targets.
A post-project assessment, comparing end-of-project data against the figures approved at full funding, with the result used to update the database.
A business evaluation, conducted one to two years after completion, validating volumes, prices, margins, operating costs and economic indicators — for which, in his framing, "the project sponsor should be held accountable for the financial outcome". [SOURCE DETAILS REQUIRED]
A word on provenance. Lavingia's evidence for the benefit of these practices is a benchmarking firm's proprietary database, and his reference is the company rather than any study; the same firm is also the sole corroborator cited by Kul Uppal for a related claim in the same journal. He wrote as a project management consultant to a large oil company. None of that makes the specification wrong — it is a coherent description of a working loop — but the supporting evidence is a commercial vendor's assertion, and it is treated here as one. [SOURCE DETAILS REQUIRED]
Reframing the Issue: Which Half Gets Paid For
Read the four instruments again and notice where they sit relative to the funding decision.
The first two run before money is committed. Both consume the database. Both serve the investment immediately in front of the reader — the pre-funding assessment sharpens the number about to be approved, and benchmarking sets the target the approver will hold the team to. Both are therefore paid for by an investment that directly benefits from them, and both are comparatively easy to defend in a budget conversation.
The second two run after the work is finished. Both fill the database. Neither improves the project paying for them, because that project is over. Their beneficiary is a future investment that does not exist yet, has no sponsor, and cannot argue for the expenditure.
That asymmetry is the whole mechanism, and it explains the pattern without recourse to negligence. An enterprise that funds the consuming half and not the filling half is not being careless; it is behaving exactly as its funding structure instructs. Each project pays for what helps it and declines to pay for what helps its successor. The database is consulted by everyone and replenished by nobody, and the estimate produced from it in ten years will say what it says today.
There is a second-order effect worth naming. Because the front half still runs, the organisation experiences itself as evidence-led. The pre-funding assessment happens. The benchmark is quoted. The governance pack is thick with comparison. What is missing is invisible, because an empty database looks exactly like a full one from the front.
The Same Structure in Two Sectors
A metropolitan bus operator, hypothetically, replaces vehicles on a rolling cycle — a tranche every two or three years, each one a discrete funded programme. The depot conversion cost for the last electrified tranche came in well above estimate, because the grid connection took longer than assumed and the temporary arrangements ran for eleven months rather than four. That is precisely the kind of finding that would improve the next tranche's estimate. Whether it does depends entirely on whether anybody was funded to write it down in a form the next estimator will read — and the team that learned it was disbanded at practical completion.
A hospital's medical equipment replacement cycle, hypothetically, has the same shape and a harder version of the problem. Each replacement is capital, each is approved separately, and the recurring surprise is not the equipment cost but the works required to receive it — power, shielding, floor loading, clinical decant. Every project team discovers this. Almost none of them is still in existence when the next business case is written, and the clinical engineering staff who could carry the knowledge forward are not in the room where the estimate is prepared. [Related article: Whose Knowledge Does Your Governance System Actually Hear?]
Decision Framework
The instrument is a ledger, and it is diagnostic rather than procedural.
The loop ledger. For the last ten completed investments above the enterprise's capital threshold, four columns: whether an outturn comparison against the approved figures exists; who holds it; whether it changed any subsequent estimate; and whether anyone can name the change.
Most organisations can complete the first two columns for most rows and the third for almost none. That result is the finding. It converts an abstract concern about organisational learning into a table that a chief financial officer can read in ninety seconds, and it is difficult to argue with, because it is built entirely from the enterprise's own records.
Three tests then interpret it.
The named-change test. Can we point to one estimate that is materially different because of one completed project, and name both? A general claim that "we apply lessons learned" does not survive this. The test is deliberately specific, because specificity is what distinguishes a working loop from a documented one.
The funding test. Is the post-completion work paid for by the project that generates it, or from a standing line that survives it? Where it is funded from the project, it will be cut, and it will be cut by a competent manager making a defensible decision. This is the test that identifies the structural cause rather than the symptom.
The database test. What does the next estimate actually consume — a maintained record, a spreadsheet on someone's drive, or professional memory? And who last added to it? An enterprise whose answer is professional memory has an estimating capability that walks out of the building at retirement.
From Strategy to Execution
Immediate. Complete the loop ledger for ten investments. Do not commission a maturity assessment or a process review to do it; both will produce a rating, and a rating answers a different question. [Related article: Who Is Your Maturity Rating For?]
Medium-term. Move funding for post-completion work off the project and onto a standing line held by whoever owns estimating capability. The amount is small; the structural change is not. Then fix the two accountabilities the loop requires: someone who owns the record and is measured on whether it is used, and someone who owns the business evaluation at twelve to twenty-four months. The second is the harder appointment, because sponsors move on and the successor inherits a case they did not write. Deciding in advance who inherits it is cheaper than discovering later that nobody did.
Long-term. Treat the record as an asset with a value, and be honest about its half-life. Data about how long a grid connection took in a particular regulatory environment ages. A loop that fills a database nobody prunes produces a slower version of the same failure. The capability being built is not an archive; it is the practice of comparing what was approved with what happened, often enough that the comparison becomes unremarkable.
Signals to Monitor
- The proportion of completed investments with a documented outturn comparison, tracked as a rate rather than reported as a count.
- The elapsed time between practical completion and the comparison being written. Beyond about ninety days the people who know are unavailable.
- Whether estimating variance is narrowing across successive tranches of similar work. This is the only outcome measure that matters, and it takes years to read.
- Whether sponsors are still in post when their business case is evaluated, and what happens when they are not.
- Whether the pre-funding assessment cites internal data or only external benchmarks — reliance on the latter is a symptom of an empty database.
- Reviews being commissioned in response to a failure rather than run as a standing practice, which produces a record biased toward disasters.
Questions for the Leadership Team
- For our last ten completed investments, how many have an outturn comparison, and who holds them?
- Can anyone name one estimate that is different today because of one completed project?
- Who pays for the work that happens after a project ends, and would that funding survive a difficult quarter?
- When our sponsors move on, who inherits accountability for the business case they signed?
- Does our pre-funding analysis draw on our own outturn data, or only on external benchmarks?
- If our three most experienced estimators retired next year, what would remain?
Closing Perspective
The finding that forecasts were no more accurate after thirty years is usually read as a comment on the difficulty of forecasting. It is better read as a comment on feedback. Difficulty alone does not prevent improvement; the absence of a closed loop does, and the loop is open for a reason that is structural rather than cultural.
The choice in front of a leadership team is small and specific. It is not whether to value learning — every organisation says it does — but whether to fund the half of the loop that no current project benefits from. That is a capital allocation decision of trivial size and unusual leverage, and it is one of the few available where the enterprise can reasonably expect the return to compound. [Related article: From Producer to Orchestrator] [Related article: The Estimate Was Made by the People Who Needed to Win] [Related article: The Clock Starts Before the Project Does] [Related article: Measuring an Outcome You Cannot Predict] [Related article: Who Sits on the Board for the Benefits?] [Related article: From Estimate to Commitment] [Related article: Who Is Entitled to Say It Worked?]
About EraNorth Insights
EraNorth Insights publishes practical analysis on strategy, projects, operations, transformation and decision intelligence for professional and organisational use. About EraNorth.