Effective project reviews are about so much more than just the evidence. This extract from Graham Oakes’ book Project Reviews Assurance and Governance, explains the paramount importance of personal credibility and personal judgement. Essentially, if you are going to kill projects in which people have invested their time, effort and reputation, you need the trust and respect of those people, which is something you should deliberately cultivate. Jonathan Norman, Publisher, Taylor and Francis

“It’s hard to kill a project. It brings people face-to-face with the possibility that they aren’t as capable as they thought they were. For some people, it means that they have nothing to show for months or even years of effort. It may damage their reputation in the eyes of their peers and competitors. People may be angry about wasted investment, or scared by the possibility that they’ll lose their job. Even when intellectually we know that we’re managing a portfolio of risky investments, some of which won’t pay off, handling this emotional fallout can be tough.

As a consequence, many organisations are very bad at killing projects. It tends to be a lot easier to let a project drift, consuming budget and resources, than to call a halt. We defer the pain, hoping in the true spirit of Charles Dickens’s Mr Micawber that ‘something will turn up’ (Dickens, 2004[1850]) to rescue the project.

This was certainly my experience in the games industry. Intellectually, it was fairly clear that our projects fell into three classes:

  • Hits: Perhaps 10 percent of the portfolio, or six games out of the 60 under development at any time, would be outrageous successes. These would deliver returns on investment of 200 percent or more.
  • Break even: The next 80 percent of the portfolio would eventually deliver a credible but not especially noteworthy game. On average, such games might cover their development costs, give or take a little.
  • Disasters: The final six projects would never deliver anything credible. Even if they did actually complete the game, it’d be too dire to take to market.

In such an environment, there’s a clear path to business success: maximize your marketing investment in the hits, tweak the break-even projects to improve quality and reduce costs, and kill the disasters as quickly as possible.

As project reviewers, we could do little to help recognize the hits. (Fortunately, decisions on marketing investment could be deferred until well into development.) We could, however, recognize several signs of incipient disaster. Some projects would regularly defer deliverables, pushing milestones later and later into the schedule. They’d showcase the same artwork month after month. The project team might gradually lose all sense of urgency and purpose, or it might operate in a constant state of adrenaline and firefighting. Either way, their ability to take action to improve the project would be seriously impaired. Some projects eventually delivered a credible game despite all these signs, but many didn’t.

We found it very hard to deal with these projects. Their executive owners were in fierce competition with each other, so they tended to become very defensive whenever someone raised questions about one of ‘their’ projects. If we called attention to a project before we had clear evidence that it was in trouble, we created a standoff. The project owner would defend the project vigorously, building an entrenched position that meant we would ultimately need even more evidence before gaining their commitment to reshape or kill the project. In the meantime, dealing with such a combative project owner could be very stressful for the review team.

After we went round this loop a number of times, we learned to spend several months building a clear case before we called for action. This case might, for example, include independent quality reviews of the game’s code and artwork, or examples of three or more status reports all claiming the same deliverables as evidence of progress. It could be frustrating to see a project burning money while we painstakingly gathered evidence, but it was necessary.

Here’s what I learned from this experience:

  • Gathering evidence is hard work, and it takes time.
  • Many organisations have an asymmetric need for evidence. They may initiate projects on the basis of partial information and gut feel, but they will only reshape or kill projects when the evidence is very clear. (Initiation is a positive act, whereas killing a project runs into far more negative emotions.)
  • The amount of evidence you need to gather in order to trigger action will depend on your credibility and organizational context. As you build a reputation for making good calls, for example, the need for a full evidential chain may decrease.
  • Conversely, if you call for action before you have sufficient evidence, then you risk driving opposition into an entrenched position. This simply raises the amount of evidence you need to gather. And dealing with entrenched opposition can be very unpleasant.
  • If we could have triggered earlier action, we would certainly have saved our company significant sums of money. With earlier intervention, we would also have had more options, so we might even have reshaped some failed projects into at least partial successes. This creates a pressure to act before you have gathered all the evidence.
  • Many executive owners argued that killing a project would damage staff morale. In practice, morale often went up when a struggling project was cancelled. Most people knew in their hearts that their project was going nowhere, and no-one likes working on a pointless project.

Evidence is important for all decisions. It needs to be gathered carefully and used wisely. Many factors influence the amount of evidence you need: the credibility of the assurance team, organizational politics and power structures, your relationship with the various stakeholders. Reviewers who aren’t sensitive to all these factors will struggle to trigger effective action.”

Graham Oakes

Adapted from Project Reviews, Assurance and Governance by Graham Oakes, Gower, Farnham (2008). Read the full text of Graham’s book on GpmFirst.