One of the biggest challenges I have consistently faced as a developer and business owner is estimating the effort required in software development.
I'm by no means the first to face this. Various silver bullets have been proposed over the years and have consistently failed. This problem was well discussed in Fred Brooks's 1975 book "The Mythical Man Month", so this isn't new either.
The problem is that software development is not a production or manufacturing activity as it is sometimes described but a design activity.
So Why Can't Design Be Estimated?
Building software is about managing complexity. Depending on what you call a "part" it is not unreasonable to consider a medium-sized software project to be composed of 1000's of parts - far more than any developer can fit into their head.
So we have to use design approaches to manage this complexity.
Components
We group these parts into layers of components to be able to reason about the system at different levels of abstraction - in the same way you can talk about a car as being 4 wheels and an engine rather than starting at the nuts and bolts every time.
Sometimes adding a new part can be trivial - it has an obvious place in one of these components. Perhaps even an earlier designer saw the need for flexibility in that part of the design.
Other times it is obvious where the part should go but not how it should interact with other components. Do we already have connections between them, or do we need to design them in as well?
The worst case (for required effort) is when there is no clear place for the change because it crosses many existing components. Do you make a new component? Do you hammer it in and make it work? In this case, you almost always need to step back and start changing existing components to make space for the change - this is where the effort multiplies quickly! Especially if you have multiple developers who now need to co-ordinate on changes.
Sometimes when a request is made, it is quite obvious which one you are dealing with. Sometimes it escalates as you work and learn new things about the problem.
Learning and Iterative Design
The other aspect that makes estimating hard is that the solution is often not obvious either. So all of what I said in the previous section refers to the idea you know what change you want to make - but this is often the most challenging step.
There may be competing solutions, and you don't know which will be best. Or you need to balance the needs of different users or lack clarity on exactly what they need.
Either way, in software, we are constantly learning about the problems and solutions we are working on. This is one of the reasons software is never really finished - just working right now.
There is a clear and well-understood solution to manage this learning process: iterative design. You take small steps and get some feedback, adjusting as you go. The rate at which you gather feedback is often cited as one of the most significant predictors of software success (faster is better).
This is a big challenge when it comes to estimation, though. How many iterations will we need to go through? How many potential solutions exist?
Can't we make it "good enough"? a.k.a Acruing Technical Debt
Do we have to worry about all of this design stuff, though? Can't we just "move fast and break things"?
In software circles, we use a metaphor called technical debt. You can choose to take a shortcut now - but you have to pay back the time you saved and more in the future because you may need to:
- Eventually, implement the full solution.
- Until then, you have to work with a sub-par design which will delay other work. (the interest).
An example fresh in my mind is a project I'm currently working on with a customer. They are rewriting an application, but in the meantime, to get going, we took an existing application and built it on top of that.
The problem is that application was designed to be manually operated where we need to add layers of automation and new features.
Working on that code is like wading through treacle. I feel crap as the customer asks for seemingly simple changes, but I have to charge more than feels right to do them because we have to fight the existing code to get them in.
So this can work in the short term (It is probably still cheaper for that customer as we were able to get a headstart and fill a gap before the replacement is ready). But if you don't address the technical debt, it will cost time and money. And remember earlier, I said the faster the feedback, the better the software? This technical debt will now have second-order effects by slowing the feedback too.
So What's The Solution?
Working as a consultant, I really want to give my customers predictability which is hard with this backdrop! There are a few things that you can try:
Accept It
The first stage is acceptance. Some tasks will take half the time you think, and some will take 10x the time you think. You will not know in advance which.
If you want to read more on this, I suggest "Waltzing with Bears" as a read. This looked at understanding these risks and studied the software industry. They showed that, on average, projects overrun their estimates by 2-3x. (There is a little more nuance to how they describe it, but that is the headline!).
The way I manage this is to spread the risk across tasks. My current process is as follows:
- Estimate tasks approximately. I give them a 1, 2, 4 or 8-hour estimate. More precise is no use, so keeping to these sizes and rounding up works best. Bigger tasks are split down as smaller estimates are easier.
Some people may see this as a bit small - I think it is contextual. For the size of individual work packages that I take on it tends to work. - Apply a ratio to the whole group of tasks for overruns. I don't know which will overrun, but I know some will, so adding overrun time to the whole group attempts to manage this.
Sometimes this will still fail - you can never predict the biggest task going 10x over your estimate - that's life!
If I'm working in an environment where we can flex the scope rather than the time, I would tend to avoid estimates. This doesn't come up often for me, but in the past, I've used T-shirt sizing here. Each task is a small, medium, large or XL without specific time attachments. This lets product and business managers appreciate the scope of work without us getting tied in knots over specific estimates.
Manage the High-Risk Items
Certain tasks are just higher risk, either from a technical or a product perspective. This might be:
- A new technology you want to adopt.
- A new feature but without a clear solution to implement it.
- A user-facing feature where there may be a lot of strong or differing opinions. Or it is unclear if it will be adopted.
There are extra risk points if putting these into the existing application requires significant time and investment before you will know if they are worth doing.
In this case, I like to carve out a separate design & prototype work package. For this activity, the outcome is the answer to the question "Should we do this?" or "How should we do this?" rather than the final product.
This signals to everyone that this is a high-risk activity and that the approach will differ. It will be iterative and experimental and will cost significantly less than a full implementation to answer these questions.
We can quickly set a direction with a higher level of confidence. Any continued development packages will be lower risk and lower cost as a result.
Summary
So no silver bullet here. Estimates will always be wrong. But understanding this and accounting for it can offer a little more certainty in the process.
What I've outlined as my approach isn't perfect, and I'm constantly trying to adapt it and balance the business vs technical needs. But I hope it offers some inspiration or at least a bit of consolation that no one has this worked out!