He's sometimes right. There are lots of good reasons why you might need software schedule estimation. When you do, there is no point in throwing up your hands and saying, "You can't do that, everybody knows it." Instead get http://www.amazon.com/Software-Estimation-Demystifying-Pract... and teach yourself how to do it.
Why would software estimation matter? For many companies it matters a lot if you can announce something in time for Christmas. Or you might have to hit a deadline for compliance with a legal contract. Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation.
That doesn't mean that you always estimate. If the value proposition is clear enough, you're going to work on it until you're done and then you will be very happy. But the real world does not usually work like that.
"For many companies it matters a lot if you can announce something in time for Christmas."
It will be done by xmas or it won't. I don't recommend "no estimation", but intense estimating won't make you hit xmas. Not estimating may give you several weeks that you could be coding.
Either way you're takin a mighty big risk when you publicly commit to that. Welcome to hell, population: Dev Team. I've known of no major software project that has a public date that hasn't resulted in major delays, feature cuts, or massive overtime to try and hit that date.
"Or you might have to hit a deadline for compliance with a legal contract."
That is also a suitably terrible position to be in. I recommend you don't take such contracts. Again, your estimates will be wrong. Even if you spend months doing them. Then you're in the same spot.
"Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation."
Which delivers most business value? I find it farfetched that both are of equal importance.
I have just moved from B2B, where many "deadlines" were largely artificial and driven by sales targets rather than real business need, to online retail, where the deadline is often concrete, immovable, and external (trade show, Olympics, Christmas). In both environments I found it most useful to encourage my team to commit to completion of _the best solution they can manage_ by a business-meaningful date (picked by the team after suitable consideration, of course). In this model the team do not agree to a specific solution or scope, though to pick the target date at the beginning they usually have an initial solution in mind; as they work toward the date, they (working with the customer representatives in the team) have the freedom to cut features or components, or to add new ones, as they grow in their understanding of the feature and the limitations on delivery (legacy code, lack of experience, unclear business needs). I have generally found that in an environment where development teams are trusted (yes I know such environments are far too rare) this produces results that are as good or better than the recipients expected, and almost always on time or nearly so.
Features, budget, schedule. The development team has to have control of one of those three.
Generally schedule is fixed by external factors. Budget is fixed by the existing team size times the schedule. Therefore the corner that makes the most sense for the development team to control is features.
I should have said "sales quotas" rather than targets. Your target to sell £x million in June reflects a real revenue need for the business, but the deadline of 30 June this implies is artificial, and the business would (in most cases) do just as well if you delivered the feature and took the revenue on 1st or 10th July.
Estimating may be a way to figure out what is plausible to do by Christmas. Which may help you scale your desires up or down to have the best deliverable product that you can.
Also in any contract negotiations, someone has to take the risk. The more risk you take, usually the higher margins you can get. (I am currently working on a contract where my willingness to take on all of the risk of not delivering value has improved my likely profit margins 10-fold.) Thus if you're willing to do fixed schedule and/or fixed price contracts, on average you can charge more. But it would be insane to attempt that without being able to estimate your likely costs and schedules.
Sure, that estimation is an intrinsically hard problem. That is why you get to charge extra for having done it. But it is not trivially the wrong business decision to make.
As an extreme example, I offer you SpaceX. Which signs lots of fixed price bids on development tasks that make the average software project look like a cakewalk.
How does this apply in an agency context? I would like to see if this can be applied to my company, but I am a little fuzzy on how this theory can work with paying customers who want to get an idea of cost up front.
"your estimates will be wrong. Even if you spend months doing them. Then you're in the same spot"
If you spend months doing estimates you will never do "nothing else". At the very least, you will have a much learnt a lot about the problem domain, and thus you will have a better overview of the work to do. You likely also will have discovered a few pitfalls to avoid, and you already will have changed the requirements of the project at the moment when it is cheapest: before you have a pile of code.
The FSA (and other regulatory organisations) frequently give a set date when a feature (EG Souped up trade reporting or similar) needs to be in place by.
You claim to be talking about estimation, but I believe that you're actually talking about being given unrealistic, non-negotiable deadlines. (How did we know that the estimate was unrealistic, you ask? We did an estimate.)
I wonder what would happen if the rest of the world took this approach?
Customer: I was thinking of getting an extension built on my house
Builder: OK
Customer: How long would it take and what would it cost?
Builder; Sorry, but to tell you that would slow down the process of building your extension. It will be quicker if I just get on with it and tell you what you owe me at the end.
What would happen? The customer would know beforehand what is true, anyway.
Berlin Brandenburg Airport was estimated to be opened in October 2011, then 2012, and currently 2013. Costs were estimated with 630 million. The current estimate is 1.2 billion.
Not giving an estimate isn't the entire solution, either.
The problem with an airport is that every time a deadline arrives, all you can do is notice that you don't have an airport, yet. You must estimate again (and you're wrong again).
With software you can say: "We're going to give you a shippable version every week. You can cancel the project at any time and you keep what you've paid for so far. We can screw up completely and you still have last week's shippable product."
Personally, I'd love that pay-as-you-go approach for the rest of the world.
Why would software estimation matter? For many companies it matters a lot if you can announce something in time for Christmas. Or you might have to hit a deadline for compliance with a legal contract. Or you may be trying to prioritize two different possible products - without a sense of the effort involved you're lacking one of the parameters you need for such an estimation.
That doesn't mean that you always estimate. If the value proposition is clear enough, you're going to work on it until you're done and then you will be very happy. But the real world does not usually work like that.