Lord Coe, who did so much to make the Olympic games the success they were, has stepped down from his role in the Cabinet Office as Olympic legacy ambassador claiming that his moving on was crucial to ensuring legacy was delivered.

In itself, that sounds like a self-effacing gesture – the public service equivalent of Captain Oates leaving the tent. A less generous interpretation might be that Lord Coe has realised that a big name, however illustrious, cannot compensate for some underlying flaws in the way legacy is being approached in government.

The Institute for Government's report Making the Games focused on the delivery of the Games itself: and here it was clear that government did a lot of things right after the bid was won. It spent time and engaged the right expertise in scoping the project; it created organisations with clear roles and responsibilities and allowed them to employ people on terms and conditions that attracted global talent (a point to remember in this week's skirmishes over Olympic Delivery Authority pay-offs); and itprovided a substantial budget – eventually. It also created political space for success so that the project survived changes of administration both in London and nationally.

But although legacy was always part of the promise of the games – whether it was sporting legacy, regeneration legacy, volunteering legacy or economic legacy – legacy arrangements always came off second best, with no clear unifying vision and no single overarching legacy delivery body. The focus was – indeed had to be – on delivering the games on time to avoid making us the international laughing stock BBC2 comedy series TwentyTwelve predicted.

There are a number of reasons legacy was always going to be harder.

• First, it lacked the deadline that concentrated minds wonderfully as the games approached.

• Second, it was not a project to trump the silos of Whitehall. Where sport and school sport in particular has been a political football; volunteering has lacked a high profile Whitehall owner. The more tangible legacy – the regeneration of east London – has progressed, as former mayoral adviser Neale Coleman pointed out at an event held last January. An official we quoted in our report noted legacy suffered from not having "a nice collective agreement and mandate on legacy in 2005".

• Third, the level of project planning and coordination that made the games more than the sum of its parts was always lacking from legacy.

• Finally, there was no single obvious leader, accountable for legacy as a whole.

A year ago, Lord Coe's appointment to the Legacy Ambassador role was a belated attempt to compensate for those weaknesses. There were some warning signs at the time – as the Guardian reported, the role was "yet-to-be defined" and Lord Coe himself said: "the role, we will work that out". It is not really surprising that he has decided to opt out a year after taking it on.

Governments are keen to co-opt individuals to these sorts of high profile roles – but the record is mixed. In most cases, people are asked to do a review which may then be named in their honour. But a few are asked to perform more executive roles within government. We have had some successful tsars for specific NHS targets – but these were people with specific jobs and the authority to make things happen.

What is unreasonable is to expect a high profile outsider, sitting in the Cabinet Office, to be able to bring Whitehall together where those with long experience or political authority have failed. Lord Coe's departure reads like an admission that departments have won – and that legacy will fall or flourish by their efforts – and there is little the Cabinet Office can do to change that.

The games were exceptional. They showed a different way of government working, where the sum was greater, much greater, than the parts. Coe's announcement suggests that for legacy, we will be relying on business as usual.