Thoughts on Failure in Big Digital Projects

My take on most of the big digital projects is that the scope is far to ambitious at the start, because during initial planning the details are ignored and so it looks easy. As the project proceeds, a number of things happen:

• the details creep in and make the original vision significantly harder. Possibly the original vision was not well enough researched (e.g. by prototyping with real users, systemic testing), possibly something is discovered late on that makes some change impossible. However it happens, the original "simple" vision for the new system becomes "polluted" with lots of special cases, which then makes the project much harder to deliver, and can even make it unsustainable.
• management (whether political or not) requests changes which do not necessarily mesh with the original plan but which are "must have", so costs rise, possibly very significantly (a hundred-fold for a component is not unusual). Occasionally I have seen management types take positions akin to "I don't care if you think the earth is round, I need you to make it flat", which is supposed to encourage engineers to work around a problem but in fact just makes them frustrated. Usually, there is no extra resource at these times.
• costs increase... my cynical thoughts are that vendors (sometimes?) underbid and then return cap-in-hand once they are embedded. I am a strong believer in being an "informed customer" who knows how to break down the project and so what a reasonable charge is... I am unconvinced that this is a widely-held belief.

Overall, big digital projects are often under-planned,  over-scoped, and under-resourced. I am not sure why, but I can guess: "how much!!" / digital seen as "magic", so anything possible / "sound-bite" / "it's just software, you can change that!" / "shiny" /  "oxbridge classics".

My thoughts on doing these things better:

• build big things using small blocks: don't try to build a house from one brick;
• it is the interfaces between blocks that matter. Be very strict about how the blocks talk to each other, but let each block be built independently;
• test interfaces constantly: ideally, have independently created prototype implementations of each block that can replace the production version to ensure this, and then do it;
• don't build any block expecting that it will be the one true thing for ever and ever. It won't be;
• when designing interfaces, give interface users good tools to adapt to change. Change happens, always!
• never lose sight of the difference between the implementation and the interface. Peeking behind the covers always ends badly;
• for any one block, don't be afraid to start again!
• changing the scope during implementation costs 100 or 1000 times the cost of that change in the initial plan. Get the plan as right as you can!
• employ good engineers, and then trust them to do the right thing. No second guessing!
• private companies only exist to make money from you. Don't implicitly believe what they say, or even that they are part of the solution.
• special cases are bad. Always!
• experienced staff know your business better than youngsters. Value that experience appropriately.

This post was originally penned in the context of digital railway projects, so added some rail-related examples where the interface between blocks mattered:

• build a railway system across the country by building them between towns and villages, and using interchangeable locomotives and wagons.
• standardise track gauge at 4'8.5", loading gauge ..., etc.;
• run rolling stock interchanges between different lines and companies;
• make new locomotives when faults are found in the old; improve the way sleepers are tied down;
• make wagons longer when steel replaces wood to reduce handling;
• use bogie vehicles for better comfort and less track damage;
• provide gauge-testing facilities for new rolling stock;
• GWR steam locos were very, very, good, but only when fired with anthracite (Welsh steam coal) - in BR usage with northern coal they fared poorly. "Coal-fired" does not equal "Anthracite-fired".
• replace steam trains with electric trains without rebuilding the whole railway system;
• when you select 7' gauge and everyone else didn't, the cost of change is a pain!
• hire Mr I.K. Brunel or Mr N. Gresley, but do not hire Hudson's "Railway King"!
• Network rail brought the Bristol line in-house because they recognised they didn't know enough to manage the other lines well. Doing so enabled them to be an informed customer for the whole network;
• NR has to check each and every type of rolling stock for clearances on every mile of track because there are so many disjoint "standards" for stock and track. Check once - use many?
• NR had to re-invent OHL for the recent electrifications, despite using and renewing miles of pre-existing installations. Staff that knew what's-what had been "let go"...