You must mean "some software projects" and not "frequently."
In the 2000's and prior, it was common knowledge that the majority of software projects failed. Even if they succeeded on paper and shipped, they weren't actually used. By some estimates, it was something like 75% of software projects.
If you look at the contents of "ecosystems" like Steam and the the various app stores, you'll see much the same. Most of the software out there is a defacto failure, and much of it is due to the ignorance of the programmers resulting in substandard programs.
Do you have a citation supporting the claim that those failures were due to poor performance and not far more significant problems like failing to correctly model the actual business problem or handle changes? That era was dominated by waterfall development which is notoriously prone to failure due to the slow feedback loop.
This is highly relevant because one not uncommon problem with highly-tuned algorithms is the greater cost of writing that faster code and then having to change it when you realize the design needs to be different, especially if pride + sink cost leads to trying to duct tape the desired feature on top of the wrong foundation for awhile until it's obvious that the entire design is flawed. That failure mode is especially common in large waterfall projects where nobody wants to deal with the overhead of another round.
Very large numbers of shovelware apps in the iPhone app store had the problems with crashing.
This is highly relevant because one not uncommon problem with highly-tuned algorithms is the greater cost of writing that faster code and then having to change it when you realize the design needs to be different
Can you give me a specific example of this? I'd say, give me a specific example, and most likely, I'll give you a reason why the software architecture of that example is stupid.
> Do you have a citation supporting the claim that those failures were due to poor performance and not far more significant problems like failing to correctly model the actual business problem or handle changes?
"Poor performance" isn't the only negative result of using an insufficiently experienced developer, or a developer who doesn't have a full grounding in algorithms and data structures.
Someone without a full CS background (and without the ability to remember much of that background) is likely to know a few patterns and apply them all like a hammer to a screw. This leads to profoundly terrible designs, not only when you take performance into account, but finding the best model for the actual business problem, handling changes, permuting data in necessary ways, and other issues.
> That era was dominated by waterfall development which is notoriously prone to failure due to the slow feedback loop.
Extreme programming was the first formalized "agile" approach, and the very first agile project to utilize it was a failure. [1] A big problem was performance, in fact:
> "The plan was to roll out the system to different payroll 'populations' in stages, but C3 never managed to make another release despite two more years' development. The C3 system only paid 10,000 people. Performance was something of a problem; during development it looked like it would take 1000 hours to run the payroll, but profiling activities reduced this to around 40 hours; another month's effort reduced this to 18 hours and by the time the system was launched the figure was 12 hours. During the first year of production the performance was improved to 9 hours."
Nine hours to run payroll for 10,000 people. We're not talking about computers in the '70s with magnetic tapes. This was 1999 on minicomputers and/or mainframes. If that wasn't a key algorithmic and/or architectural problem, then I would be amazed.
When you're designing a system using agile, it often ends up with an ad hoc architecture. Anything complicated really needs BOTH agile and waterfall approaches to succeed. You need to have a good sense of the architecture and data flow to begin with, and you need to be able to change individual approaches or even the architecture in an agile manner as you come across new requirements that you didn't know up front.
> This is highly relevant because one not uncommon problem with highly-tuned algorithms is the greater cost of writing that faster code and then having to change it when you realize the design needs to be different
I'm going to say [citation needed] for this claim.
I gave an example in another thread of having improved the speed of a game development level compiler tool by a factor of about 1000, with no major architectural changes, and it took me about an hour.
At the same time I performed a minor refactor that made the code easier to read.
Bad design == Bad design. That's it. Good design can include an optimized algorithm. A good design tends to be easy to extend or modify.
Very rarely it makes sense to highly hand-tune an inner loop. The core LuaJIT interpreter is written in hand-tuned assembly for x86, x64, ARM, and maybe other targets. It's about 3x faster than the C interpreter in PUC Lua, without any JIT acceleration. That is a place where it makes sense to hand-tune.
> pride + [sunk] cost[s] leads to trying to duct tape the desired feature on top of the wrong foundation
That's another orthogonal error. It makes me sad sometimes to throw away code that's no longer useful, but I'll ditch a thousand lines of code if it makes sense in a project.
Every line of code I delete is a line of code I no longer need to maintain. I'm confident enough in writing new code that I don't feel any worry about deleting old code and writing new. This is as it should be. Sad for the wasted effort, yes. But I know that the new code will be better.
The C3 system only paid 10,000 people. Performance was something of a problem; during development it looked like it would take 1000 hours to run the payroll, but profiling activities reduced this to around 40 hours; another month's effort reduced this to 18 hours and by the time the system was launched the figure was 12 hours. During the first year of production the performance was improved to 9 hours.
And I happen to know for a fact that naive string concatenation was a big part of the performance problem! (Bringing it back to my original comment.)
In the 2000's and prior, it was common knowledge that the majority of software projects failed. Even if they succeeded on paper and shipped, they weren't actually used. By some estimates, it was something like 75% of software projects.
If you look at the contents of "ecosystems" like Steam and the the various app stores, you'll see much the same. Most of the software out there is a defacto failure, and much of it is due to the ignorance of the programmers resulting in substandard programs.