A few months ago, I was having a conversation over coffee with an IT operations manager with a very large company about their lack of standards in managing deployments and dependency on manual processes. As these conversations usually go, what started as a very targeted discussion about optimization of a single aspect of software development turned into a fantastic philosophical journey across the entire spectrum of issues, from everything to managing executive expectations and what software development will look like ten years from now. Keep in mind, this is a multinational company with hundreds of programmers, ops, and IT support employees managing millions of lines of both completely custom software packages but also millions of lines of code related to customization of vendor software packages with software budgets in the 100’s of millions. That’s when the statement was dropped:
Much like Happy Gilmore would never admit he’s a golfer, but rather a hockey player, there are so many large companies out there having this identity crisis and refusing to acknowledge reality. If the product produced is not software, this does not exempt the operations from failing to adopt good processes and standards related to software. For example, if an employee of a widget manufacturing company was asked what the company did, the answer would be making widgets. In the same line, if you asked their CFO if the accounting department followed GAAP standards, the answer would certainly be “yes”. However, when you walk into the IT departments and ask the developers or software managers whether or not they do things like unit testing, regression testing or work towards automated security controls, you would be very surprised how often the answer is no. Why is there a perceived difference?
The main difference is visibility. Auditors don’t come in to make sure good software development standards are in place, but they do come in and pour through the ledgers to ensure accounting is happening according to standards. If standards aren’t in place for your accounting department, the pain can be immense, especially for public companies. However, the reality of following accounting standards is not simply because people will be checking. The standards are in place because of the collective experience in business of understanding that following standards is a good thing from a productivity standpoint. In the long term, standards create efficiencies and reduce costly errors. These same companies choose to implement rigid standards in one aspect of their business, yet leave other departments with hundreds of employees and millions of budgeted dollars free of standards.
Yet leaving the basic standards, like unit testing and automated builds, out of the software development processes will be visible, but just not in the same way. That production outage last week because of a last minute code change? That was because there weren’t any unit tests. That year long million dollar software upgrade that turned into a 3 million dollar two year effort? That’s because their weren’t any requirement standards, performance testing or architectural standards in place. The lack of standards in software development processes are very visible and are equally as painful as an accounting department not following GAAP, but the difference is the inability to draw the correct conclusions and root cause. The employees ultimately responsible will attribute the outage to the bad code or the poor requirements, but the answer is actually a lack of effective standards and best practices.
Just as GAAP standards won’t fix every accounting woe, great software development practices won’t catch every issue but it sure will stop a lot of them. So the next time there is a big production outage, ask a different set of questions:
- Can you provide me a list of all the unit tests that were run for this?
- Did the regression tests fail for this release?
- When the rollback scripts were run, why weren’t they effective?
The answers might surprise you.