When Agile Meets OO Testing – A Case Study Siegfried Goeschl Senior Software Engineer Spengergasse 25A/2/16 A-1050 Vienna, Austria +43-664-19420090 [email protected] Matthias Herp Senior Software Engineer Wernersdorf 2 A-3110 Wernersdorf/Neidling +43-676-3581488 [email protected] Clemens Wais Test Manager Schmalzhofgasse 10/8 A-1060 Vienna, Austria +43-664-3312838 [email protected] ABSTRACT This paper describes the testing approach for an object-oriented rich client application based on an agile software development process. The paper starts with an overview of project and the testing strategy being used. It then goes on to describe the test tools used in the project and the results achieved. The paper ends with a discussion of the discovered defects, their distribution and improvements for the testing process. Categories and Subject Descriptors D.2.9 [Software Engineering]: Management – pair programming, software process models, software quality assurance. General Terms Management, Measurement, Verification Keywords Agile Testing, Object-oriented Testing, Unit Testing, Integration Testing, Continuous Integration, Defect Rates, Technical Debts 1. INTRODUCTION When Scrum [1] was introduced at OOPSLA 1995, it did not only change the way to run a software development project but also had major impact on how to test it. Life under the waterfall required a project test plan, which called for a test phase of a few months and a separate test team. It would come with a detailed capacity planning for the testers, a fine-grained project plan, and a matrix with all the quality attributes and the effort we should take to test every attribute. This traditional approach to software testing looks less promising in the agile world where everything is centered on volatile requirements, sprints, standup meetings, features, retrospectives and shipping software after four weeks. Welcome to world of "agile testing". To shed some light on how "agile testing" works, the XYZ case study is introduced - a medium size customer information system used by 300 in-house employees. The XYZ front-end is written in Java 1.5 and based on Eclipse RCP (Rich Client Platform). The XYZ front-end communicates with the XYZ server and various other back-office systems using SOAP (Simple Access Object Protocol) over JMS (Java Message Service). From a testing point of view, it is a client/server system with some stringent requirements regarding availability and response time. The Scrum development approach is used for the project with sprint durations of four weeks and a strong focus on pair programming. The Scrum team is rather large consisting of 8-10 software developers and 2-3 testers all co-located in a single room to foster team communication. As of writing the project is now in its 25th iteration and has lasted for about two years. 2. THE TESTING STRATEGY Coming from a more traditional testing background, one might ask, "What is the difference between agile and chaotic testing"? The answer lies in the agile testing strategy being applied, so it is time to have a closer look at testing the XYZ project. 2.1 Focus on Individuals and Interactions Following the "Manifesto for Agile Software Development" [2] and its "individuals and interactions over processes and tools", extensive pair programming and close interaction with the testers are used. Once a software developer pair has finished its work, an informal and face-to-face review takes place together with a tester and/or business analyst to ensure that the required functionality has been implemented. Any problems found during this informal review are fixed immediately without creating a bug report and only then is the code declared ready for QA. Following these rules caters for continuous interaction, which in turn improves team productivity [3]. 2.2 Test Automation The test automation approach for the XYZ project follows the test automation pyramid [4] having three different layers of automated tests. These three levels have different characteristics regarding costs, execution speed and return of investment. The lowest tier consists of unit and component tests based on JUnit 4 and making extensive usage of Mockito, a mocking framework. The mock testing framework allows running component tests without depending on live back-office systems, thereby making the tests more stable and dependable. The JUnit tests are easily written and maintained by software developers due to the excellent refactoring support of current IDE's. As many tests as possible are pushed to this level of test automation since these tests provide quick feedback and good return of investment. The software developers are required to pass those tests before committing any changes to the source control system. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specificnd/or a fee. ECOOP ‘2010 Maribor, Slovenia EU Copyright 2010 ACM 0-89791-88-6/97/05 ...$10.00 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. ECOOP '2010 Maribor, Slovenia EU Copyright 2010 ACM 978-1-4503-0538-9/10/06 ...$10.00.The middle tier of the test automation pyramid consists of regression test that exercise the high-level business functionality and the back-office services. The business services and back-office services are tested using JUnit. These tests operate at the API level ("behind the GUI") thereby bypassing the presentation layer. Bypassing the presentation layer makes the tests less expensive to write and maintain than tests that depend directly on the user interface. Running the test of the middle tier is much slower because each test covers more ground than a unit test and accesses the database and/or back-office systems. The top tier operates on the presentation layer and represents what should be the smallest investment