Test Automation and continuous integration

These days testing has an increasingly important role in software engineering. Organizations invest more time and resources to test products more comprehensively and efficiently. The testing budget can be as much as 25% to 50% of the total budget of the software project. The market demands complex and reliable products. The pressure to reduce time to market means less time to design, build and test new products.

The costs of software defects are increasing

Because almost every device has some kind of controlling software inside it, the overall cost of testing for software faults is increasing. According to Tassey (2002, The Economic Impacts of Inadequate Infrastructure for Software Testing) the incomplete test environment and onwards defects in the final products costs about 59,5 billion dollars per year in the Unites States. As an example, one of the most expensive software defect was realized when the Mars Climate Orbiter burnt in the atmosphere of Mars due to a mix-up between metric and Imperial units. The total cost of the project was 327,6 million dollars.

In this situation, testing has two, competing requirements to fulfill. It must find as many defects as possible, but it has to do so very efficiently (i.e. using less time and resources). Moreover, testing must support the design and implementation process, requiring different tests depending on the phase of the software project.

One solution to improve the coverage and efficiency of testing is automation. Proper use of automated testing has several significant benefits which makes the testing more efficient.

One of the most important benefits of test automation is the ability to efficiently repeat tests. This feature is very valuable for regression tests when a new version of a product is tested and most of the features are the same as in previous versions of the product. When test automation is integrated to the continuous integration process, the benefits are maximized.

Furthermore, automated testing enables new ways to test. For example, performance and stress tests using thousands or even millions of events is very difficult without test automation. Also the strict real time requirements of the embedded systems, whose time scale can be milliseconds or microseconds are impossible to test using manual tests. Test automation also enables and encourages testing product more precisely: Testing the same feature manually using all possible attribute values is very frustrating and the possibility of human error is significant. For instance, automated testing finds every corrupted pixel from the five mega pixel image every time, no matter how many times the test is executed. Manual implementation of such a test would be impractical. Furthermore, test automation forces the test execution to be exactly the same every time. This repeatability is a valuable feature when regression tests are executed.

All these benefits contribute to better time to market and increased product quality. Effective use of automated testing means more efficient use of valuable project resources.