10 October 2007: Agile user testing - results instead of reports


Agile user testing saves costs and helps reaching your project goals rapidly. It helps ensuring end user satisfaction and meeting the business goals set by the company.

    User-centered design aims at producing applications that are as useful and usable for end users as possible. An essential tool in achieving applications like this is user testing. Its key idea is to gather as much objective information as possible concerning the usability and experience it offers to its users. This can be achieved through interviews and surveys and, above all, by observing how successfully test users manage to use the user interface prototype that is built to communicate the application's design.

    The earlier the user testing is performed, the less costly it is to correct any identified problems, and the more likely it is that the corresponding improvements will be implemented in the technical end product. User testing a technically implemented system is often rather useless, because problems detected at a late stage are usually so slow and expensive to correct that, in practice, even essential changes are not made before the system is taken into use.

    This year, the premier conference in the field of usability, CHI (Computer/Human Interaction), arranged a separate workshop to address the minor impact that usability work has on application development work. Significant improvements in the quality of the user interface do certainly not always follow user testing. Why not?

    Are you achieving better solutions as a result of testing?
    User testing roughly consists of writing a testing plan, carrying out the test and analyzing the test results. The most significant outcome is a report, which lists the detected issues in a prioritised order. The report often includes suggested improvements with respect to at least the most critical problems, based on which the project team can draw conclusions concerning further development.

    Instead of reporting, the true purpose of user testing is, obviously, to improve the quality of the user interface from the end user's perspective, based on the reported issues and suggested improvements. However, transforming the list of identified problems into better solution in the final application is often a challenging problem for the project manager.

    The challenge lies in the fact that the project manager and implementation team must balance the additional design work and resources - implied by the reported problems - against the fixed budget and schedule. And when particular corrections would cause major additional costs and delays, it is the project manager's job to "sell" them to the project owners, who have committed to finishing the project within the agreed schedule and budget.

    The duration of the user testing may also cause problems. Typically, preparing and carrying out a test and creating the test report takes from two to three weeks. In projects with quick development cycles this can be too much, at least with respect to early stage testing.

    Due to these problems, user testing is performed too seldom or - in trying to achieve reliable results with better prototypes - too late. In the latter case, the intentions are good but the threshold for the reported issues to transform into an improved user interface may become too high.

    Bring agility into user testing
    Agile user testing - following the ideology of agile methods - favors economical and frequent testing instead of individual and exhaustive tests and exessive pondering over questions such as "have we surely discovered all of the bugs in the user interface yet?"

    The key points to characterise agile testing are: fewer testers, focus on making concrete improvements during the tests themselves and, more informal and less serious emphasis on reporting.

    All this aims at making testing lighter and faster so that its utilization in projects with quick development cycles is both possible and cost-effective. It also aims at reinforcing the justification of user testing through its promise of guaranteed productivity: the user interface is concretely improved as a result of the testing itself.

    In other words, the aim is to maximise the probability of any issues found during a test being corrected in the final implementation of the system. Where user testing normally requires five or six testers, agile user testing settles on three. Due to this and lighter reporting, an agile test can be planned, implemented and reported in its entirety in just three calendar days, which encourages utilization of the method at an early development phase, when making changes is cheap.

    In an agile test, testers and the customer's representatives analyze the results of each tester, and jointly agree on the necessary changes, which are immediately implemented in the user interface (where applicable) before the next tester arrives. Hence, changes are made immediately after an issue has been found and a solution suggested. This enables the immediate evaluation of the quality of the solutions devised for the problems, during the next user's test. Naturally, challenging problems require in-depth planning, and the fixes for them should not be rushed into the prototype. A standard reporting methodology is better for such cases.

    Keep your methods and tools up-to-date
    As a counterpoint to its benefits, agile testing places special requirements on the application of the methodology and project management, especially considering change management and documentation, which is particularly important when working on a non-agile project. Agile-style testing like this is, in fact, often used in practice when testing with paper prototypes. They, then again, are not suitable for a comprehensive user experience evaluation and greatly limit the content of the tests.

    Agile testing gets more difficult and the importance of prototyping tools increases, if you wish to utilise rich and interactive prototypes that run on actual devices to make it possible to measure the real user experience. In SYSOPENDIGIA's projects, we have utilised the Realizer by SYSOPENDIGIA product for this purpose, as it is specially designed for the fast creation and simulation of Web user interfaces. Basically any prototyping platform that enables sufficiently fast cycle changes is an alternative, assuming it can be efficiently utilised and that the prototype has been built to support agile ways of working.

    Jacob Nielsen introduced a lighter "Discount usability engineering" methodology already in 1994, but user testing ideology was formulated for the first time, as described above, in 2002, in Microsoft's paper "Using the RITE method to improve products; a definition and a case study". So the method is not new, as such, but general discussion on this issue is still surprisingly scarce.

    Are you aware of the end results of your project - early enough?
    Agile user testing as a method provides an interesting opportunity to develop usable systems: it enables a cost-effective way of ensuring guidelines for user interface development, integrates into the application development process better than the traditional model, and provides an improved user interface by default.

    On the other hand, sufficient understanding of the domain is required of the testers: the customer's and supplier's representatives must be capable of making joint decisions quickly between individual tests. Expertise is also required in drawing the right conclusions and taking the user interface in the right direction after only brief consideration. Agility does not mean "light thinking" here, either; good end results require knowledge of the users' problem areas, familiarity with the tools, usability know-how and social skills.

    It should also be mentioned that user testing - as it is traditionally interpreted - is still a valid method. It is often necessary to have a sufficient user volume for testing to provide statistical data on successful use. Agile testing does not support these goals.

    Whether agile or not, more and more companies and organizations seem to think that user-centered design performed before the actual projects are launched is the key to identifying the correct system requirements, and also a foundation for accurate budgeting and scheduling of the projects. Let us face the facts: according to studies, costs incurred by incorrect requirements generally constitute around 40 percent of the lifecycle costs of projects. While requirements documented on paper leave a lot of room for interpretation, a user interface prototype provides a genuine interface for dialog between the different interest groups involved in the project.

    Agile user testing provides an interesting window into the sanity and usefulness of conceptual design work in the early stages - from the end user's point of view. For whom, after all, do you develop or provide IT systems?

    For further information, please contact:
    Antti Martikainen, Senior Consultant, tel. +358 40 550 7174