Developers debate the value of static analysis, but the need for better software is real
Over at g2zero, the Web blog and community site run by New Rowley and devoted to improving software quality, a post on potential defects and security vulnerabilities found in the Firefox Web browser code base (version 1.5) generated numerous discussions on g2zero, as well as slashdot, digg.com, and other tech sites.
[Note: The Firefox code was tested by Klocwork's K7 static analysis tools. Klocwork occasionally delivers reports on open source projects it has tested to g2zero. While Klocwork sponsors the g2zero site, the site does not promote the vendor outside of the PR generated from posting testing results, Klocwork employee submitted posts, and talking about the topic of static analysis (SA). Any individual or organization is invited to submit posts on any topic related to software quality.]
What was fascinating about the debate is the varied opinions on the value of automated static analysis testing tools. Even though SA tools have been around for years and there are open (Splint) and closed (Klocwork, Coverity, Parasoft, etc.) source offerings, there is sharp debate in the developer community about the value of these tools. The crux of the matter is the issue of false positives, and whether the tools are delivering valuable defect insight for developers or simply burdening them with tracking down harmless bugs or security vulnerabilities that are unlikely to ever be exploited.
From the g2zero discussion, here are two competing thoughts on the issue:
While developers may balk at specific tools designed to improve code, there is no doubt that there is increasing pressure on organizations that rely on software to seek out and invest in quality software. The need to quickly respond to changing market conditions, reuse code as services, avoid security and privacy breaches, and meet increasingly tough compliance mandates is forcing non-techie business executives to demand better software.
The need for more predictable, reliable, and secure software is driving the concept of software assurance. The goal of this effort is to improve software quality and offer software users a way to gauge whether software, frameworks, components, and code snippets meet a certain acceptable criteria (see this g2zero post for additional information on software assurance).
As the concept of software assurance evolves and becomes more mainstream, we expect governments and industries to adopt specific measures for software quality that will in turn demand more rigorous testing of the underlying code. And in the absence of extra time and developers to manually review and test code, ever-improving automated solutions will find their place.
[Note: The Firefox code was tested by Klocwork's K7 static analysis tools. Klocwork occasionally delivers reports on open source projects it has tested to g2zero. While Klocwork sponsors the g2zero site, the site does not promote the vendor outside of the PR generated from posting testing results, Klocwork employee submitted posts, and talking about the topic of static analysis (SA). Any individual or organization is invited to submit posts on any topic related to software quality.]
What was fascinating about the debate is the varied opinions on the value of automated static analysis testing tools. Even though SA tools have been around for years and there are open (Splint) and closed (Klocwork, Coverity, Parasoft, etc.) source offerings, there is sharp debate in the developer community about the value of these tools. The crux of the matter is the issue of false positives, and whether the tools are delivering valuable defect insight for developers or simply burdening them with tracking down harmless bugs or security vulnerabilities that are unlikely to ever be exploited.
From the g2zero discussion, here are two competing thoughts on the issue:
"With most of these tools the signal:noise ratio is very high ... To spend smart developers' time going over long reports of machine-generated lint would be a waste."
"Like just about every other tool we software developers use, code checkers can provide significant value when used intelligently and can be a pain in the ass when used otherwise."
Despite varied opinions on SA tools, there is a need to improve code quality
Whether a developer has had a bad experience with an SA tool in the past -- meaning they were overwhelmed with a huge number of reported defects, many of which were false positives -- some sort of automated SA tool will be required to improve quality because code bases are becoming so large and code quality is often unknown. Problems arise due to a variety of now common circumstances, including:- Building off existing, often undocumented code bases where access to the original developers is minimal or not possible
- Patching, enhancing, or refactoring code and introducing or exposing dependencies or conditions that may have been previously benign
- Relying on code from third parties, where unknown quality components are introduced due to acquisition, licensing, or use of outsourcers (local or offshore)
- Exposing code to harmful conditions, particularly in the case of software that is accessible to the public or is being reused in ways the original developers never envisioned (e.g., as part of a service-oriented architecture service)
While developers may balk at specific tools designed to improve code, there is no doubt that there is increasing pressure on organizations that rely on software to seek out and invest in quality software. The need to quickly respond to changing market conditions, reuse code as services, avoid security and privacy breaches, and meet increasingly tough compliance mandates is forcing non-techie business executives to demand better software.
The need for more predictable, reliable, and secure software is driving the concept of software assurance. The goal of this effort is to improve software quality and offer software users a way to gauge whether software, frameworks, components, and code snippets meet a certain acceptable criteria (see this g2zero post for additional information on software assurance).
As the concept of software assurance evolves and becomes more mainstream, we expect governments and industries to adopt specific measures for software quality that will in turn demand more rigorous testing of the underlying code. And in the absence of extra time and developers to manually review and test code, ever-improving automated solutions will find their place.