The Quest for Perfect Software Testing and analyzing in the attempt to reduce software bugs. Written by James Yoder Dr. Deborah Hwang CS 390 October 22, 2008The Quest for Perfect Software Written by James Yoder Page 1 Perfect software is the dream of many programmers. Software tools and programming languages are getting better and making it easier to write complicated programs. Unfortunately, humans are fallible creatures. Whether it is critical software for the military, a commercial operating system sold throughout the world, or a freeware computer application, programmers make mistakes in their code. Unlike mistakes in other career fields however, these bugs often go undetected until they cause thousands of dollars in damage and require even more time and money to fix when they could have been detected and fixed early in development. Although they cannot solve communication organization issues, new tools and techniques such as automatic unit testing, static code analysis, and contract-based testing are helping to improve the quality of code and detect bugs and security holes early in development. As software grows larger and more complex, the industry is being buried in a deluge of bugs. Recent software faults have caused Toyota Priuses to stall at highway speeds, California’s 911 system to crash, airport radars to malfunction, and warning sirens at a New York nuclear plant to fail. [8] Software conflicts caused electronic voting machines to drop votes from 11 counties in Ohio, resulting in lawsuits from the state. [6] The average computer program is no better. According to Capers Jones of Software Productivity Research, there are on average about five faults per feature, and only 85% of these are fixed before the product is shipped. [1] Arden Bement, director of the National Institute of Standards and Technology (NIST) explained the breadth of the problem: “The impact of software errors is enormous because virtually every business in the United States now depends on software for the development, production, distribution, and after-sales support of products and services.” [7] Security holes are another problem with software. Although they are invisible to the average user, malicious people can use them to subvert the security of the computer. Security holes are harder to detect, and often go ignored until after they are exploited by a virus or computer criminal. Buggy software is prevalent and, sadly, is often considered to be normal by many people. But software faults are not just an inconvenience; they also cost the public dearly in time and money.The Quest for Perfect Software Written by James Yoder Page 2 Faults themselves have a high cost to both customers and the software makers themselves. According to a NIST study, software errors cost the U.S. economy around $59.5 billion dollars annually. [7] And the economic costs keep increasing. As programs get more complex and become more necessary in daily life, software errors have more of an impact and can cause more problems. The study also found that more than a third of the costs (around $22.2 billion) could have been prevented simply by using better testing techniques and discovering software faults earlier in the process. Discovering faults before shipping saves an enormous amount of time, money, and the fixes have fewer regression errors than attempting to deploy patches after the program has been shipped. Although this is taught in software engineering classes, some companies have to learn about the increasing cost of fixing bugs the hard way. Microsoft had to learn about the importance of software testing the hard way. After releasing Windows XP, they began to work on Windows Vista like they had always done; each programmer started working on their own section of code and wove it together every so often, which often took a week. [3] Unfortunately, they had focused on features and ignored existing and newly added bugs, leaving those to patch later. [3] Bugs were difficult to fix and were largely ignored until later because the programmers would rather write new features than fix elusive errors. After bringing in two brilliant consultants who were experienced in reviving stalling projects, they set up a new system. They set up automated testing to analyze the code before it was integrated into the code base. “If a feature had too many bugs, software "gates" rejected it from being used in Longhorn. If engineers had too many outstanding bugs they were tossed in "bug jail" and banned from writing new code.” [3] The new system prevented very buggy code from making its way into the project and penalized programmers who spent too much time writing new code and not enough time fixing the bugs that they caused. In the end, the time and effort spent paid off. The time to rebuild the operating system from the code was significantly shorted. The bug reports that should have numbered in the tens of thousands only numbered in the thousands. [3] Although they still have further to go, Microsoft is on their way to making a more quality operating system and provide another example of the need for automatic testing.The Quest for Perfect Software Written by James Yoder Page 3 Automatic unit testing is changing the way testing is done. Although software writers have written programs that use the computer’s power to make things simpler in other fields of work, only recently have they begun to use the computer to make testing easier on themselves. Programmers design test cases, often before they even begin to code the module, and have the computer run the tests and compare the results. These can be performed nightly to ensure that the code always works and that problems are fixed immediately instead of letting them slip into the finished product. Designing the tests before the module is built also helps prevent the module from drifting away from the specifications, the same problem that ultimately destroyed the NASA Mars Climate Orbiter. [4] In addition, automatic tests can perform regression testing, tests that make sure that fixing a fault does not break another section of code. When a problem is fixed, a test case can be made that fails if the problem is reintroduced. Automatic unit testing is becoming a key part of bug detection and is being incorporated into many programmers toolsets, but new tools are being developed that can detect buggy code before it is even run.