Over the course of fifteen years as a software developer, I've worked for nine different companies. They ranged in size from under fifty employees, to over a few thousand. They spanned many different industries including financial, medical, retail/e-commerce, and custom software development. While they each differed in many ways, one thing was constant; as software developers we were required to provide estimates for how long it would take to develop our software.
In particular, three of the companies that I worked for were software consulting shops. By that I mean we wrote custom software for our customers. In these cases getting software estimates wrong didn't just make the team look bad; they could be the difference between the company making or losing money on a project. If too many projects were underestimated, it could potentially result in financial hardship for the company and ultimately even layoffs.
The first software consultancy that I worked for was back in 1999. At that time agile wasn't something that was talked about. As far as I know Scrum didn't officially exist yet. A lot of small software shops were popping up, trying to take advantage of the dot com craze. In terms of the specific company that I worked for, they had issues with some of their earlier projects. Often customers asked for changes, but nobody was managing the scope. In most cases a developer took a direct request from the customer, and would simply make a change. There was no detailed, formalized requirements, so there were often disagreements in scope. The customers usually had expectations that were not in alignment to what was sold to them. In short, projects would often go over budget and customers were often unhappy.
Fortunately in those days, enough money was flowing in that our company was able to recover from several bad projects and managed to learn from those early mistakes. What came next was the implementation of processes that today we refer to as waterfall. All new projects had to have detailed requirement documents, that left nothing left to the imagination. This results in requirement documents that were over 100 pages in length, along with images showing exactly what the UI would look like. A change, no matter how small, had to go through a formal change request process. Project managers created gnat charts, that mapped out the details of the timeline. In short, the new process would fix all of the problems that we had before. Except for that they didn't.
Even with the details requirement documents, things would eventually come up that weren't covered. When it came to estimation, even though we tried to think of every possible detail, for projects that lasted several months this was simply not possible. Worse yet, a good chunk of the project budget would be eaten up during the requirement gathering phase, so by the time it got to development we were already late before we started. In the end, I'm not sure if things were any better than before. By 2001 the bubble hard burst, and this time people lost their jobs. When it was all done, the company lost approximately 50% of their workforce, although they did manage to survive.
I should mention that the company was run by intelligent people, with the best intentions. They were following what were generally considered best practices at the time. The post dot com crash was something that was out of their control. The project management practices weren't the sole factor in their fortunes. However, I do distinctly recall upper management making comments such as "We need to stop losing money on these projects". Even if these were exaggerations, clearly better practices could have resulted in more profitability and better overall financial health for the company.
Of course today we have the concept of agile software development, including Scrum. Lots of lessons were learned the hard way, and I'm sure there were many companies that went through these kind of pains. There's still some nonbelievers out there, but today they are a minority. However, despite all of these gains we still are expected to estimate our projects. Customers want to know, rightfully so, how much it's going to cost them for a custom piece of software. Despite this, proving an upfront number that represents the total cost of the project is not the answer.
In part 2 of this blog posting, we will review why estimating the entirety of an agile software project is not a good idea. Instead it will discuss how best to manage these projects in a way that will provide the most value to the customer, while being mindful of their budget. Ultimately the goal is to keep your customers happy, as a happy customer will most often result in more business.
Until next time, stay frosty.
Comments