“‘The creation of genuinely new software has far more in common with developing a new theory of physics than it does with producing cars or watches on an assembly line.” – Bollinger, T
One of the biggest problems in software projects is the inherent difficulty of planning and estimation. It is a problem as old as the industry and almost everybody in the life of a project regardless of their role will have a strong opinion on the matter.
Similarly every company will have its collection of war stories about projects that have slipped spectacularly or seemed to perennially overrun on their budgets and targets. Jobs may have been lost, careers ended and most importantly relationships destroyed.
For some companies this is such a common occurrence that the devastation wreaked by bad estimation may have become the norm instead of an exception. It could have become part of their culture to operate in a state of constant project flux. In environments such as these the trust between business and technology may be so eroded that any real attempt at even coming up with reasonable project estimates get abandoned regularly.
Instead it gets replaced with a constant barrage of unrealistic business demands that is framed as ‘critical-delivery’, typically based on no quantitative reality of effort but exclusively on what business would want to happen by a certain date for some particular reason. These ‘immovable’ business targets generally are manufactured to squeeze out as much delivery in a world that is hard to quantify. Where it gets ugly is that these targets are often arbitrarily tied to people’s KPIs even before the feasibility of the piece of work is explored. Typically the lucky recipients on these KPIs are people that have no experience in how to do proper software estimation, and who have no understanding of the differences between an estimate, target and goal (go read Software Estimation: Demystifying the Black Art by Steve McConnell).
At the heart of this problem lies the phenomenon of discovered complexity. For some reason people view software as a mechanical task similar to that of an assembly line in a factory. But as research has shown (even mathematically), ‘the creation of genuinely new software has far more in common with developing a new theory of physics than it does with producing cars or watches on an assembly line.’ [Bollinger, T]. It is a creative process that requires a lot of thinking about as of yet unknown problems that constantly present themselves. These accidentally discovered problems may in turn actually lead to more subtle and indirect problems that also require solving. This is called accidental discovered complexity.
Put simply at a particular point in time, you don’t know what you don’t know yet and there is no way to tell how long it will take you to figure it out when you discover it. In a world such as this how can you definitively estimate how long something is going to take? At best you can give a reasonable guess using the information you have at your disposal at that point in time. But even this is a problem since what you know about the problem domain develops and changes as the project progresses.
Given that it seems virtually impossible to estimate these creative tasks with reasonable accuracy. It therefore becomes imperative that you follow an iterative development methodology where there is a constant feedback loop on estimates. If you are not continually re-estimating how long a piece of work is going to take based on all the new information constantly learned, you are putting your project and more importantly your organizational goals and commitments at risk. Do that often enough and watch the carnage unfold.