Acceptance testing: acceptance testing kit, part 2: how to do it!

DEPARTMENT of ENERGY and RESOURCES "Acceptance Testing – How to do it!"
The Acceptance Testing Kit
Projects with an IT Enhancement
or Procurement Component
Project and Acceptance Test Managers Information Management Branch Corporate Services The Acceptance Testing Kit – Part 2 Acknowledgments
This Kit is the result of an initiative of the Information Management Branch of the
Department of Infrastructure, Energy and Resources.
Input was obtained from many individuals with experience in the field of Acceptance Testing, both internal and external to the Branch, and with particular knowledge gained from involvement with Projects with an IT enhancement or procurement component. In particular, the contribution of the following individuals in preparing this document is gratefully acknowledged: John Cuthbertson Information Systems Analyst Senior Consultant Allegra Huxtable Senior Business Analyst The Acceptance Testing Kit – Part 2 Table of Contents

ACKNOWLEDGMENTS . 2
INTRODUCTION . 6
FOREWORD.6 PURPOSE .7 HOW TO USE THIS KIT .7 PHASE 1 INITIATION AND PLANNING. 8
SMALL OR LARGE PROJECT?.8 CREATING AN ACCEPTANCE TEST PLAN.8 BEFORE YOU PREPARE THE PLAN .9 WHO DOES WHAT? .9 PHASE 2 PREPARING TESTS, TEST DATA, AND TRAINING .15
WHAT TEST DOCUMENTATION SHOULD WE MAKE?.15 HOW DO WE DEFINE ACCEPTANCE CRITERIA?.16 WHAT TEST PROCEDURES SHOULD WE FOLLOW? .17 WHAT TESTS CAN WE DO?.17 HOW DO WE DEVELOP ACCEPTANCE TESTS? .21 WHAT TEST DATA SHOULD WE USE?. 22 WHAT TRAINING IS NEEDED? . 23 WHERE IS THE BEST PLACE TO CONDUCT TESTING?. 23 APPROVING THE ACCEPTANCE TEST PLAN . 23 PREPARING TO CONDUCT THE ACCEPTANCE TEST. 25 PHASE 3 EXECUTION AND CONTROL .25
WHAT YOU NEED BEFORE YOU START TESTING . 25 MANAGING THE TEST ENVIRONMENT. 26 MANAGING TEST DATA. 26 RECORDING THE RESULTS. 29 FIXING PROBLEMS AND RETESTING . 30 SUSPENDING OR CANCELLING TESTING.31 PHASE 4 CLOSURE .32
FORMAL ACCEPTANCE. 32 TEST RECORDS MANAGEMENT AND RETENTION . 32 OTHER ACCEPTANCE TESTING ISSUES.33
THE TENDER PROCESS . 33 CONTRACT MANAGEMENT. 33 DOCUMENTATION . 34 SYSTEM DESIGN . 34 PACKAGE SYSTEMS . 34 ACCEPTANCE TESTING A LARGE SYSTEM . 35 ACCEPTANCE TESTING A SMALL SYSTEM. 36 SYSTEM ENHANCEMENTS AND MAINTENANCE. 37 SOME CAUTIONARY NOTES .38
The Acceptance Testing Kit – Part 2 TEST DATA CONFIDENTIALITY .38
INTERPRETATION OF BUSINESS REQUIREMENTS. 38 SYSTEM PROTOTYPING . 38 PARALLEL RUNNING. 38 MINOR SYSTEM DEFICIENCY DELAYS. 39 The Acceptance Testing Kit – Part 2 Introduction
Foreword
Acceptance Testing is a process of evaluating a new or revised system undertaken by
Business Unit Managers and end-users of the system to make sure it meets their
business needs.
With Acceptance Testing, the main objective to aim for is happy end-users and stakeholders. Whilst it does take time and resources to do Acceptance Testing, the rewards for the Department, end-users and stakeholders far outweigh the effort needed to undertake this activity. Acceptance Testing ensures that the new or improved system you want is the system you get, at the price you were prepared to pay. Acceptance Testing helps keep a Project budget under control by clearly stating and measuring the delivered product and outputs against the business requirements. The Service Provider also knows exactly what is required of them under the contract arrangements for the delivery of the product. The implications of overlooking or understating the importance of Acceptance Testing are considerable. Those accountable for the Project may find the system costing more than what was initially budgeted for because the ‘errors' were not properly identified and resolved before putting the product into production. Therefore the effects of lost productivity from a product that does not meet quality standards or functionality could have major long-term effects on Business Units. However, Acceptance Testing must be resourced effectively and appropriate time must be allocated to thoroughly test the product. While this Kit can help you achieve the above, it is not a step-by-step recipe for Acceptance Testing success. It is designed as a guide to alert and inform those involved with a Project, and the Acceptance Testing task, of the benefits, issues, and pitfalls that can come hand-in-hand with the Acceptance Testing process. Every project is unique and as such, this Kit will enable this unique nature to be successfully managed and accounted for. Above all, talk with others about the Acceptance Testing process and benefit from their knowledge and experience they may bring to the activity. In particular, discuss the process with stakeholders and end-users! In line with its Service Charter, IMB makes available this Acceptance Testing Kit to assist Departmental staff. The Acceptance Testing Kit – Part 2 Purpose The purpose of this Kit is to help Departmental staff plan, conduct, and manage the Acceptance Testing process for a business initiative. How to Use this Kit This Kit provides an overview of the essential components of Acceptance Testing. It is divided into two Parts. Part 1 Acceptance Testing – Why do it? introduces Acceptance Testing and outlines the reasons for making it an essential element of any Project. It provides the motivations for Project Managers and those involved in Testing to undertake the activity and the benefits that can be expected from conducting the Testing process thoroughly. There is also a brief outline of the steps required to execute Acceptance Testing from start to finish. Part 2 Acceptance Testing – How to do it! provides the person responsible for Acceptance Testing with a guide for moving through the process. Included are the types of individual tests that may be considered, how they can be managed, and a series of templates and checklists that minimise the need to ‘reinvent the wheel'. These resources should prove invaluable in assisting those involved in Testing. Use of this Kit enables a consistent approach to Acceptance Testing within different types of projects, for example from a small upgrade to an existing system, to implementation of a new, large, software package. By referring to the Kit, the person responsible for Acceptance Testing should be able to determine which components or key elements need to be included within their Acceptance Testing process. To err is human, but to really foul things up requires a computer. The Acceptance Testing Kit – Part 2 Phase 1 Initiation and Planning
Small or Large Project?
Whilst every Project has its own unique characteristics, they will often take on a
number of very similar features. As a general rule, the less expensive the Project, the
less resources will be required to Acceptance Test. This will not always be the case
but as a guide, a system developed for less than $50000 can be considered a small
system, and over $50000, a large system. If you have any uncertainty as to the nature
of the system you are involved with, consult with the Information Management
Branch in your Division.
Creating an Acceptance Test Plan The first step in Acceptance Testing is preparation of a Test Plan. Whenever Acceptance Testing is undertaken, there needs to be a clear plan and well-defined objectives. The planning stage includes the following activities: A good start to ► Agreement with management and stakeholders on preparing an Acceptance Test Plan the Table of Contents; is to first write a Table of Contents for review ► Documentation of the Acceptance Test operating and approval. environment, Acceptance Criteria, and testing responsibilities; ► Preparation of an Acceptance Test ► Definition of the Acceptance Test Where a formal contract exists with a Service Provi der, there may be clauses that require Acceptance Testing to be completed in an agreed timeframe. The Plan should be prepared according to these conditions. ► Determination and planning of training for Test participants; and ► Document arrangements for review and sign-off of the Acceptance Test Plan. The task of developing the Acceptance Test Plan is the responsibility of the Manager of the Business Unit that ‘owns' the system. However, the task Operating environment includes
may well be delegated, for example, to the Project the hardware, software, andnetwork within which the system will Manager. There is an Acceptance Test Plan run. It is important to define the template included as part of this Kit. operating environment because thesystem may work in oneenvironment but not in another. The Acceptance Testing Kit – Part 2 How to Prepare the Plan Gather together the following items, information, and documentation: ► The Acceptance Test Plan template and checklists; ► The requirements specification (either business or functional) for the system as this includes the requirements to be tested, Template – Acceptance Test Plan Checklist – Acceptance Testing and/or any additional Acceptance Criteria Checklist – Acceptance Test Planning that may exist in the Contract or Service Level Agreement; ► The environment the system will operate under as detailed in the Technical Environment – Hardware and Software document (obtainable from your Information Management Branch or on the intranet); and ► A list of the documentation to be delivered with the system including system release notes if this is a system upgrade. Once all this information has been obtained, work through the Acceptance Testing checklist and Acceptance Test Plan template, completing all the relevant sections. Who Does What? The roles and responsibilities for Acceptance Testing should be outlined in the Acceptance Test Plan. The roles required will depend upon the size of the Project, and consequently, the magnitude of the Testing task. There are four groups that need to be involved. Remember, one person may fill more than one role. Acceptance Testing Team Acceptance Test Manager
► Manage the Acceptance Test and coordinate Acceptance Testing activities; ► Prepare the Acceptance Test Plan and Acceptance Test procedures and request/obtain the necessary Acceptance Test resources; ► Advise on establishment and training of the Acceptance Test team; ► Assign Acceptance Test tasks; ► Coordinate compilation of, and access to, Acceptance Test data; ► Liaise with the Service Provider Project Manager; ► Oversee development of Test Cases/Scripts, based upon business requirements as detailed in the Contract, RFQ/RFT or other documentation upon which system development will be based; ► Formally report to the Project Sponsor and Business Unit Manager on the status of Acceptance Testing; The Acceptance Testing Kit – Part 2 ► Formally manage, record, and authorise modifications to the Acceptance Test system, the documentation, the Test data, and the Acceptance Test environment; ► Manage and liaise/communicate with stakeholders regarding change requests and Test problems; ► Ensure that Tests are completed to the agreed schedule; ► Review Test results; ► Administer Test problem reports and recommend priorities for resolution; ► Ensure Tests are repeated where necessary; ► In the event of serious problems, determine whether to recommend suspension or cancellation of Testing; and ► Recommend formal acceptance of the system to the Project Sponsor. Business System Administrator
It is desirable that the person who will serve as System Administrator once the system is in production, participates in Acceptance Testing as Business System Administrator. ► Administer and initialise the system configuration data for: • End-users and end-user system privileges; • Printers and stationery; • Work stations; • System codes and settings; • Access to the Acceptance Test data; and • Batch processing; ► Verify production support facilities; ► Test system administration functions; ► Coordinate generation of Test data; ► Test system documentation; ► Run routine batch processes and reports including audit trail reports; and ► Run data integrity checking routines and monitor data integrity. Senior User Representative
► Liaise with the Acceptance Test Manager; ► Assist in development of Test Cases/Scripts; ► Coordinate the testing activities of Business Unit officers; The Acceptance Testing Kit – Part 2 ► Coordinate the use of Test data; ► Undertake Tests as requested; ► Record Test Cases and conditions; and ► Record and report successful completion of Tests and document problems Business Unit Officers/End-Users
► Assist with the Acceptance Test Plan as assigned; ► Prepare and provide Test data, and write Test Cases/Scripts, with expected ► Undertake tests as requested; and ► Record and report successful completion of tests and document problems Service Provider Team Service Provider Project Manager
► Ensure that delivery responsibilities in relation to Acceptance Testing, as outlined in the contract, are met; ► Oversee establishment of the Acceptance Test environment in consultation ► Liaise with the Acceptance Test Manager and assist with Plan preparation and procedure definition as requested; ► Ensure Development Testing of the system is complete prior to hand-over of the system to the Acceptance Test Manager; and ► Supervise necessary amendments to the system and manage new system Service Provider Team Members
► Establish and maintain the Acceptance Test environment. ► Assist end-users with testing as required; ► Conduct all necessary technical Testing prior to system hand-over; ► Resolve system problems and update system versions as required; and ► Migrate programs, database changes, and data to the Acceptance Test The Acceptance Testing Kit – Part 2 IMB Representatives Database Administrator
► Migrate application to the appropriate test/pre-production environment; ► Initialise the database (and re-initialise as required). This may include the loading of appropriate security-related data; ► Ensure database integrity, both in content and security; ► Ensure establishment of the application environment parameters; ► Ensure ancillary and support software is loaded and configured; ► Provide database support services as required; ► Undertake any specialised Testing requirements; ► Ensure requested database modifications are appropriate; ► Coordinate database backup procedures as outlined in the Acceptance Test ► Establish all end-of-day and batch processes; and ► Support standard application release procedures and verify their operation. Infrastructure and Support Services
► Set up and maintain the application environment; ► Provide the required hardware components for the Testing environment; ► Provide and support desktop, communications, and server environments in accordance with the Acceptance Test Plan (which in turn will nominate the environments required for Testing, Training, and Pre-Production); and ► Assist with stress, performance, and integrity testing as required in the Test Business Analyst
► QA the Acceptance Test Plan. IMB Management
► Act as a contact point for any resource/staff requirements as outlined in the Acceptance Test Plan; and ► Oversee the hardware and environment requirements and budget accordingly. The Acceptance Testing Kit – Part 2 Management Representatives Project Steering Committee
► Provide policy and direction for Acceptance Testing; ► Review and verify the incorporation of business objectives and requirements into any Contract, RFQ/RFT etc.; ► Resolve contractual issues and issues relating to system scope; ► Review and approve or reject submitted change requests; and ► Endorse acceptance of the system and authorise its implementation. Project Sponsor/Business Unit Manager
► Develop business objectives and requirements and ensure they are clearly detailed in any Contract, RFQ/RFT etc. in order to base Acceptance Criteria upon them; ► Appoint and direct the Acceptance Test Manager; ► Approve the Acceptance Test Plan; ► Manage the provision of requested Acceptance Test resources, including Business Unit officers to participate in the Acceptance Tests; ► Review Test results; ► Accept each major sub-component of the system; ► Participate in, and report to the Project Steering Committee; and ► Formally accept the new system and recommend system implementation. The Acceptance Testing Kit – Part 2 Roles for Large Projects The recommendation for role assignments for a large project are: ► Acceptance Testing Team; ► Service Provider Team; ► IMB Representatives; and ► Management Representatives. Roles for Small Projects The recommendation for role assignments for a small project are: ► Acceptance Test Manager (this role may be undertaken by the Project Manager or Business System Administrator); ► Business Systems Administrator ► Service Provider Team ► IMB Representatives ► Business Unit Manager (or other appropriate Management representative) The Acceptance Testing Kit – Part 2 Phase 2 Preparing Tests, Test Data, and Training
Acceptance Testing is a project in itself, depending on the scale and scope of the
system or system changes. There is a defined body of work, a distinct start and end,
and a well-defined objective. As such, Acceptance Testing is best managed using
‘conventional' project management processes. There needs to be:
► An overall guiding plan; ► Well defined responsibilities and lines of Don't forget to report: • Overall progress; • Testing status; ► A detailed schedule of tasks with dates and • Problem status; • Change status. ► Well defined processes and tasks; ► A controlled environment, documentation, and product; ► Processes to capture and record details of progress and to monitor them against the plan; and ► Regular review meetings. What Test Documentation Should We Make? Depending on the scale of the Acceptance Test, for maximum benefit and minimum effort, establishing and maintaining the following records and documentation can be very helpful. The Acceptance Test Strategy is generally only necessary for a large project, whilst all other documentation is needed for both large and small projects, with documentation "scaled down" for the smaller projects. ► An Acceptance Test Strategy - outlining the approach and understandings for Testing activities during the Project. This is best formulated soon after Project inception and can be included in the Project Execution Plan (should one exist). A template can be found on the DPAC Project Management web site; ► The Acceptance Test Plan - outlining the Test approach, objectives, activities, responsibilities, and schedule. This includes: • Test Procedures - documenting the processes to be followed during Acceptance Testing. Procedures; • Authorisation requirements for commencement, suspension, and • The conduct of Tests and recording of results; • The recording and resolution of Test problems; • Signing-off and formal acceptance; The Acceptance Testing Kit – Part 2 • Authorisation requirements for changes to the Test system and/or environment; and • The release of a new version of the system; ► Test Cases - showing Tests to be performed, usually measured against business requirements (with any specific data or conditions), and the expected outcomes. There may be many Test Cases for a single business requirement. These include recording the outcome of each Test Case and/or Script as Test Results; ► Test Scripts - documenting individual testing processes as a series of steps, many of which will be series' of interrelated Test Cases and encompass an individual business requirement/process; and ► Problem Review Sheets and Problem Review Register - documenting problems encountered and indexing the status of each problem. In addition, maintain an: ► Acceptance Test Log - recording significant events and changes to the test environment and test data. Templates for these documents form part of this Kit. Samples and templates of these documents form part of this Kit. How Do We Define Acceptance Criteria? There needs to be clearly defined criteria on which to base system acceptance. Acceptance Criteria should be clearly documented so they are understood by all parties. Ideally, they Even though Acceptance Criteria are often specified in the relevant contract or should be decided "up front" at the Project agreement with the system supplier, they Planning stage, prior to system development. should still be included in the Acceptance Test Plan. They should then be incorporated into any Contract to be fulfilled by the Service Provider. Acceptance Criteria will then be based upon the requirements as specified in the Contract (or any other less formal agreement) with a Service Provider, or in the case of an internal system development, upon requirements as specified in the business requirements documentation. If specified in a Contract, these requirements will in turn, hopefully be based upon the business requirements as detailed in the relevant Business Systems Requirements Specification. It is not the responsibility of the Acceptance Test Manager to define or "invent" Acceptance Criteria at the Testing stage. This point is too late in the development process! If Acceptance Criteria are either sufficiently unclear or difficult to obtain, refer the issue to the Project Manager for direction and resolution. The Acceptance Testing Kit – Part 2 Often, Acceptance Criteria will also cover a number of "standard" items, that may include: ► system and user documentation; ► system useability; ► system performance and capacity; ► data integrity; ► system security; and/or ► system reliability. What Test Procedures Should We Follow? The Acceptance Test Plan details the overall process of initiating, conducting, and concluding Acceptance Testing. More Determine testing procedures at the specifically, there needs to be a clear outline of planning stage as they may require advice the actual testing process used for confirmation from outside the business unit. It may be necessary for IMB in your Division to assist of the Acceptance Criteria. Document these you with planning and execution. processes in the Acceptance Test Plan and translate them into Test Cases and Test Scripts that define the Testing steps. What Tests Can We Do? Template – Test Case If new business processes are intended and Template – Test Script Cover Sheet contracted to be introduced in conjunction with a Checklist – Test Case or Script new system, then it is necessary to prove these new processes during Acceptance Testing. Aspects of any new system relevant to business processes may prove unsatisfactory for end-user purposes. If this is the case, the Business needs to correlate the unacceptable system behaviour with the business requirements or specifications as detailed in the supply contract. If the number and degree of business process changes are significant, formal business change management processes can be applied in Responsibility for defining and proving new conjunction with system development and processes rests with the business unit, rather than the system supplier or developer. The Business Unit is obliged to accept the system ‘as is', should it satisfy all contractual obligations of the Service Provider. Therefore, it is imperative that business requirements are precisely identified at the Project Planning stage and included in the For each business or system requirement and for each Acceptance Criterion, there Contract. Should these contractual obligations should be at least one Test Case. be met, and the system still fails to meet one or The Acceptance Testing Kit – Part 2 more business requirements, steps may then be authorised (possibly at additional expense) to improve the system with alterations. However, system changes at such a late stage of the development life cycle are less than ideal. As well as basic function testing, Tests may also evaluate: ► Day, week, period, month, and year-end You may be able to use a stop watch to measure system response times. For more accurate readings and for internal processes (such ► System sociability; as database performance), system monitors and ► System performance and reliability in a range of recorders may be necessary. environments and under any special conditions; ► Stress capabilities; ► Backup and recovery procedures; ► System access and security; Checklist – Report Testing ► Concurrent updating; and Checklist – Web Page Usability Testing Checklist – User Interface Testing ► Disaster Recovery. There may also be specific tests reviewing the supplied system documentation (on-line help system, user guide, operations guide, maintenance documentation etc.). For many Tests there will be common features e.g. all screens need to be tested for ease of use and appearance. These generic tests are better managed with checklists or standard forms rather than Test Cases. For each screen utilised during the running of each Test, the Tester should review the screen using the generic User Interface checklist contained in this Kit. Similarly, for each report generated during the running of a Test, the tester should check the report using the Report checklist. An additional checklist for Web Page Useability is also included. Stress and Performance Testing This form of testing addresses both specific contractual criteria and general system performance. Stress testing is where specific
system functions (usually critical
By conducting both stress and performance testing in ones) are subjected to high loads to a controlled manner, you can be sure of accurate ensure that the system will still cope. Performance testing is where
response and performance measurements where specific system functions (usuallysystem loads are precisely known. These loads may critical ones) are timed undervarious system loads to ensure that include network traffic or database activity, as well the times meet contractualperformance acceptance criteria. as normal system functions. The Acceptance Testing Kit – Part 2 If requirements for system throughput are specified in the Contract, then stress testing is necessary, and Test Cases should be clearly defined. The functions to be tested may be either on-line or batch. Stress tests tend to focus on a particular aspect of the system's architecture such as database updating, for example, to ensure that the database server's capacity meets expectations. In performance testing, system loads may include network traffic, database activity as well as normal system functions. Controlling these Regression tests are most successful if the loads is not always straightforward and the test data, test environment, system dates, assistance of IT personnel is often needed with and any other test inputs are either kept constant or only changed in a controlled the setup and execution. The steps and tasks to manner so the impact of any changes can be undertaken should already be itemised in the be anticipated and/or explained. Acceptance Test Plan. Regression Testing Regression testing is an important method of Regression testing is where a
validating system changes that occur during series of tests (or indeed all tests) Acceptance Testing. It ensures, where a change or are systematically repeated andwhere expected results are known. changes have been made to the system, the test The results of the repeated test are environment, or the test data, that these changes compared with the results of theoriginal (successful) tests. do not impact the correct operation of functions not obviously or directly related to the changes. Regression tests are usually done manually and can be time consuming and tedious. By minimising the number of times that changes are made to the system, the Test environment, and the Test data, effort can be significantly reduced. It is most practical to regression test only critical functions and those known or suspected to be affected by changes. For any system change, the Acceptance Test Manager should determine which Test Cases or Test Scripts need to be repeated, and schedule these System crashes can logically or physically corrupt the database. To ensure this does not happen, Over time, and with the assistance of comprehensive database integrity checking routines experienced end-users of the system, a matrix of should be available. impact can be compiled i.e. for each system function there is a cross-referenced list of other functions that may be potentially affected by changes to the first function. While tools to support regression testing are available, their use is not always cost effective. However, using electronic means to compare the results of repeated Tests against the results of the original (successful) Tests is a possibility. The Acceptance Testing Kit – Part 2 Disaster Recovery Testing Disaster recovery testing needs to be conducted in a controlled manner to allow the system to be "crashed" at a precise point, with the status of active system functions accurately known, to test the system's disaster recovery process. Crashes are arranged to occur at strategic points such as in the middle of a multi-record database update. Recovery from simulated events such as complete loss of a database or server can also be tested. Two processes are used to determine the system's ability to recover from a disaster. Firstly, where the whole system has crashed, the database is restored from a previous backup, the system transaction log is applied to the restored database (rolled forward), and the logical and physical integrity, and the completeness of the database is checked. The second process is where multi-database update functions are crashed and the system is allowed to "roll back" i.e. the partial updates are undone automatically by the database software and the logical integrity and completeness of the database is checked (especially for the data involved in the crashed functions). Concurrent Update Testing If the system you are testing is an on-line, shared system, you will need to conduct concurrent Concurrent update tests can fail without any apparent impact on the user. However, the Concurrent update testing assesses a system's integrity of the database must remain intact and can be confirmed using integrity capability to deal with multiple system functions checking routines. simultaneously attempting to access or update the same database record(s). How the system handles these situations is based on the system's technical design. The system needs to cope with the situation where an update transaction tries to re-read records that were read earlier but that have now become locked or changed. Irrespective of the internal processes, from the end-user point of view the system should provide a meaningful message and terminate the transaction in The potential for problems in a controlled manner, allowing the user to restart the large batch processes is often high. If something goes wrong, it may not affect just one Case. The logical and physical integrity of the database should remain intact and this should be confirmed using the appropriate integrity checking routines. The Acceptance Testing Kit – Part 2 Large Reports, Batch Processes and Interfaces The amount of effort in preparing for tests of large reports, batch processes and interfaces is often underestimated. Most attention tends to be directed to the on-line system components. However, these items must be thoroughly tested as well. How Do We Develop Acceptance Test Scripts? It is important to always maintain the relationship between any Test Case and the requirement or acceptance criterion it is verifying. There are two steps to follow when developing Test Cases: ► Step 1 is to describe what is needed for each Test, what is to be done, and the
expected outcome. A Test Case template is included as part of this Kit. ► Step 2 is to document the required Test Data and conditions that enable the
Test to be undertaken. To help you in developing your Tests, a Test Case and Script checklist has been included as part of this Kit. For each Test, determine: ► The features to be tested; A Test Script is a documented
process involving a series of
► The source database(s) that will be required; steps, many of which will beinterrelated Tests. It may also ► The source of data required specifically for the describe a set of Test conditions. ► Any other resource requirements; ► The task that will be performed in the Test Case; A Test Case is a single record
or group of records used in aTest Script. ► The expected outcome. Once Test Cases are prepared, collate related Test Cases into Test Scripts. Each Test Script usually equates with a business process. The Script brings all related Tests together into a logical sequence. For example, in a financial system there may be a series of tests to record different types of on-line financial transactions and there may be other tests to create and post a batch of transactions to the Remember to write Test Cases for reports, offline processes, system and to print a batch report. It makes interfaces, and data migration. sense to relate the on-line transaction entry to the These are often overlooked yet most likely to cause headaches subsequent batch posting and printing. after the system has been implemented! If new business processes are involved, describe in the Test Script the overall approach required to test them, The Acceptance Testing Kit – Part 2 including the normal and alternative paths, and key exceptions. For each Test Script, identify resource requirements, including any resources outside the organisation. A Test Script template is available as part of this Kit. What Test Data Should We Use? Preparing Test Data can be a time consuming process. To To minimise data minimise effort, first validate Test Data and then preserve it contention between tests, discrete test for subsequent re-use should it become corrupt through data may be assigned to one or several (related) Test Scripts. It may be possible to create data via direct data entry or by electronic transfer (such as exporting from a spreadsheet and direct loading into the test database). Test Data may also be taken from standard input forms used within the Business Unit. In some instances where the data is not of great End-users familiar with the application area and system are best for creating test data. significance, the Tester will be required to "make some up". A useful source of data (especially where large volumes are required) is data migrated from an existing system. However, the data migration and load processes require thorough testing themselves and often migrated data may be incomplete and of questionable quality. A good way to organise Test Data is to match it with Test Scripts and maintain this link throughout the Testing process. Protecting Confidential Test Data By conducting Acceptance Testing in an environment and with data that closely emulates live operation, the chance of problems arising once the system is implemented is decreased considerably. Unfortunately, using "real" data often means a greater risk of accidental distribution of confidential information (for example, being printed, emailed, or transferred over a public or wide-area network). To minimise this risk, design security procedures into the Acceptance Test Plan and make sure all testing personnel are trained and alerted to the dangers. Other ways of maintaining data security include: ► Establishing a physically separate network with separate printers; ► Encrypting or modifying data such as names and addresses, so they are ► Creating server and workstation directories specifically for the use of Acceptance Testing; ► Establishing specific Acceptance Test email accounts; The Acceptance Testing Kit – Part 2 ► Regularly and routinely cleaning up Acceptance Test directories; ► Routinely shredding any printed reports; and ► Using special stationery that indicates the printed data is for test purposes What Training is Needed? Training needs should be outlined in the Acceptance Test Plan. This allows for the User Training is normally covered in the
scheduling of training and assignment of overall Project Plan. For end-userstrainers. This will usually be arranged by the participating in Acceptance Testing, training must occur prior to Testing commencement. Acceptance Test Manager. Participants in the Acceptance Test may also need training in the Test procedures for undertaking Tests and recording results. This may take the form of a walk-through or dry run of some Tests, as well as a formal training session. These sessions need to be allowed for in the Acceptance Test and Project Plans. Where is the Best Place to Conduct Testing? To help personnel focus on the job at hand, Acceptance Testing is best done in a separate work area where participants are not distracted by their normal work activities. A dedicated, Use a white board and notice board to laboratory-style facility with plenty of work space is ideal. communicate This allows informal communication between participants Acceptance Test "announcements" and to and the Acceptance Test Manager. display testing progress. A secure area also provides a good location for storing Acceptance Test records and documents and provides a focus for testing activities. Approving the Acceptance Test Plan It's important that management formally approves the Test Plan. The Manager of the Business Unit that is responsible for the system should sign-off the Plan. This helps ensure personnel and resources are committed to Testing when it occurs. The Acceptance Test Plan should be reviewed by: ► The Business Unit Manager; Remember, the Plan
► Those with key responsibilities in the Test Plan – the needs formal approval by the manager of the Acceptance Test participants should be offered the Business Unit opportunity to review and comment. It will help responsible for the them understand the nature of their involvement and allow them to voice any concerns; The Acceptance Testing Kit – Part 2 ► The Service Provider – this may be a contractual requirement (if it is not, allowing them to review the Plan will clarify any issues they may have). They should understand end-user expectations for the system and the support (if any) they are required to provide; ► Any other key stakeholders; and ► An independent quality inspector - they may pick up items that seem obvious to the author and participants, but may be open to misinterpretation. The Acceptance Testing Kit – Part 2 Phase 3 Execution and Control
Preparing to Conduct the Acceptance Test
If the Acceptance Test Manager takes on responsibility for scheduling and assigning
Testing tasks e.g. assigning individual Test Scripts to team members, workflow will
run more smoothly and efficiently. Try and avoid any individual member of the
Testing team shouldering too great a workload. It will also help in tracking work
progress as sometimes, when people from different Business Units are involved, the
same Test Script may require running by more than one Tester.
The Acceptance Test Manager should ensure that each participant understands their assignment, is aware of the Test procedures, and knows the timetable expectations for each task. By previewing the Test Script and individual Tests, a tester can clarify any concerns with the Acceptance Test Manger before undertaking a Test Script. What You Need Before You Start Testing Ideally, Acceptance Testing should run according to the Acceptance Test Plan from start to finish without disruption and without the need to change either the system or the Testing environment. In reality, things do go wrong, and continuity is often difficult to achieve. To minimise the likelihood of disruption, Acceptance Testing should only proceed if the following have occurred: 1. The Acceptance Test Plan, including all Acceptance Criteria, has been signed-off by the Business Unit Manager; 2. There has been formal management approval to proceed with Acceptance 3. Test Data has been made available; 4. The Acceptance Test environment has been established, stabilised, and protected against unauthorised or inadvertent change; 5. The system has been thoroughly system-tested by the suppliers or developers, the system test has been formally signed-off, and the system (including all documentation) formally handed over for Acceptance; and 6. Testing participants have received the necessary training for using the system and for following Test procedures. The Acceptance Testing Kit – Part 2 Managing the Test Environment Managing Changes Before you make changes to the Acceptance Test environment, make sure they are planned, authorised, controlled, recorded, and communicated to Test participants. Otherwise: ► even minor changes may impact the validity of tests already completed; and ► participants can become confused if the status and versions of the Test environment are uncertain. Keeping Track of Testing Progress Establish an Acceptance Test Log to record all significant events occurring within the Acceptance Test environment. The Log should be readily accessible and should be used (mainly by the Acceptance Test Manager and System Administrator) to log changes to: ► The system software; ► System documentation; A Test Log helps when problems arise with ► System parameters, code tables, and the test system or test data. If you know precisely what took place prior to the problem, the Log can be referenced to prevent the problem recurring. ► Baseline Test Data; ► Migrated Test Data; ► External systems and/or data; ► Database software, configuration and scripts; and/or ► Network, server, and workstation software, and configuration. The information you could log includes: ► The date and time of an event; Template – Acceptance Test Log ► The name of the person executing an event; ► Details of the event; ► The signature of the authorising person; and ► Any comments about the event and its outcome. Managing Test Data Initialising Test Data Initialisation usually involves loading or entering: ► The necessary system codes and system parameters; The Acceptance Testing Kit – Part 2 ► Either manual or automated Test Scripts and Test Data required for the Tests; ► Migrated data (usually used where large volumes of data are required, and often from a separate database). When testing an upgrade to an existing system, the Test database will normally be populated with data extracted Once you've prepared your or copied from the existing live database and there may test data, make a copy should the data become be little or no need to add additional data. corrupted in the future. It's useful for when further When testing a new system, the Test database will system changes are made probably be empty and need setting up with an initial Test and another round of testing is required. dataset. Some data may be loaded automatically from spreadsheets, text files, or other databases using database scripts. However, data may need to entered directly using the system's standard on-line functions. Backing up the Acceptance Test Database Regularly backing up the Acceptance Test database allows recovery of the database if the need arises to re-test, or to eliminate corrupt, spurious, or unnecessary data. Usually, backups (and any necessary restores) need to be authorised by either the Acceptance Test Manager or the System Administrator, and should be recorded in the Acceptance Test Log. Managing Changes to Test Data Another useful and important tool is a record of changes to the Test Data. These changes can affect the outcome of Backing up at strategic points will enable you to Tests and the effectiveness of regression tests. It may be more easily return to an historical data position in necessary at times to restore the Test Data from an earlier the testing process, with backup to eliminate unwanted or unsuccessful changes to less time and effort. Test Data. Record any important changes in the Acceptance Test Log. The Acceptance Test Manager or the System Administrator should coordinate the use of Test data between the various Testers and maintain a record of changes in the Acceptance Test Log. You may also find it useful if summaries of Test data entered during Acceptance Testing are logged as Tests are undertaken. Checking Database Integrity It is important that database integrity remains intact. Sometimes, Tests can affect database integrity, particularly by programmes that don't operate correctly. For example, Tests such as crashing the system to Test recovery processes can cause physical corruption of the database where whole records are lost or corrupted. Processes that check the integrity of the test database may be provided by the system The Acceptance Testing Kit – Part 2 supplier (they may be included in the system
requirements) or they may have to be provided by
database administrators in the form of database scripts. Database integrity is when the
When run, these routines scan the database tables and data retains internal consistency or
lack of corruption. Database
relationships and report any logical data integrity integrity checking routines are
problems. The Information Management Branch in processes (programs or scripts) that scan the database tables and your Division will be able to provide assistance with links and report any logical data integrity problems such a missingrelationships, undefined codes etc. The System Administrator routinely runs these scripts They may also check the physicalintegrity of the database during Acceptance Testing, either at the completion of each Test Script or at the end of each Test session. They should also be run following any significant event or activity that affects the database (such as the loading of migration data). Results should be reviewed to detect any problems generated by the Test undertaken since the previous (clean) integrity check report. Keep the results with the Acceptance Test Log for future reference. Checking Audit Trails When Tests are run (and if the system provides an audit trail of changes) then records written to the audit trail should confirm changes made by the Tests. The System Administrator should routinely run reports that allow the System's audit trails to be monitored. These will be run at the completion of each Test Script or at the end of a Test session, and also following any significant event or activity that affects the database. These reports should be reviewed with the Testers to ensure all significant updates have been correctly recorded in the audit trail. Results of these runs should be retained with Acceptance Test records. Migrated Data and the Migration Process Where data needs to be moved or relocated from an old system to a new system, it becomes necessary to test the migration process and the migrated data. If some Acceptance Testing steps are dependant on using migrated data (for example, if large volumes of migration data are to be used for stress and performance testing), then migration process testing may need to be completed first. Consider loading the migration data into a separate (full size) Test database. As migrated data is often incomplete and may include generated default values, it is necessary to repeat some functional Test Scripts using migrated data both with and without other Test data i.e. it is necessary to ensure that the system functions correctly with the migration data. The Acceptance Testing Kit – Part 2 Reconciliation of control totals for data extracted from existing systems and loaded into the new system is an important validation check. These steps should be included in the migration Test Script or Scripts. Recording the Results For every individual Test, the result will be either ‘Successful Completion' or ‘Failed'. The assigned Tester records the result of each Test Case within a Test Script in the Test result area on the Test Case sheet. The Tester then signs Remember, as the
Acceptance Test Manager, the Test Case sheet alongside each Test Case and also check and endorse the Test the Test Script sheet once the full Test Script has been Script sheets and file them in the Test records. completed. Testers must remember to verify the version Maintain a register of of the system tested, the date and time of the testing, and completed Tests and Test Scripts with their status. any Test data identifiers. The Test Script result (bearing in mind this covers several Test Cases) will then be either ‘Successful Completion', ‘Partial Completion', or ‘Failed'. How Do We Manage Problems? For each Testing issue encountered, the Tester should first seek assistance to confirm that the issue is a valid one and not a temporary configuration or data related problem. At least one experienced system user should be on hand. Ready access to the necessary technical support should also have been Template – Problem Review Sheet Template – Register of Problem Review pre-arranged. This may be either personnel from IMB or Service Provider staff. The Tester should complete an Acceptance Test Problem Review Sheet for any ‘Partial Completion' or ‘Failed' Test results and the Acceptance Test Manager should record these in the Problem Review Register. Also, remind Testing personnel to record any documentation and on-line help inconsistencies and errors, and any other issues, even if they are not directly related to the current Test Script. Record and Categorise Problems To keep track of Testing and Test problems, the Acceptance Test Manager needs to monitor and report on Acceptance Testing progress and the status of outstanding problems. Each recorded problem will reference the relevant Test Case, Test Script, and business requirement (or requirements as detailed in the Contract). Problems will be graded by priority (severity and impact) and categorised by type. The Acceptance Testing Kit – Part 2 Problems that can only be resolved by re-specifying business requirements (compared to those in the Contract) should still be recorded but categorised so as to be clearly identifiable as a possible future change request or Contract variation. Attempt Resolving Problems With the Supplier Management of any contractual issues arising from Acceptance Testing is best left to the Acceptance Test Manager and Project Manager who will try to resolve the problems with the Service Provider Project Manager. The outcome of any discussions needs to be recorded against individual problems as they appear in the Problem Review Register. The Acceptance Test Manager and the Service Provider Project Manager will endeavour to agree on the definition of the problem and a possible solution. If agreement can't be reached as to the cause of or responsibility for any problem, refer the matter to the Project Sponsor or Project Steering Committee. Fixing Problems and Retesting Change Control When Acceptance Testing occurs, it is in a defined and agreed environment, against an agreed Plan and timetable, with agreed Test data, against agreed Acceptance Criteria and with a specific delivered version of the system and documentation. Any changes to these items should be managed and assessed by the Project Team, the change impact defined, measured and costed, and due consideration given to the change before it is approved and applied. Where changes to the Test environment, the system, or the Test data are made, they may impact the integrity of Testing already completed. System Upgrades and Retesting The Acceptance Test Manager will need to manage and authorise any upgrades or changes to the Acceptance Test environment required to resolve Testing problems. All changes to the system and Acceptance Test environment should be logged. The system supplier should provide details of changes and the problems that have been resolved with each new Release. Where applicable, regression testing may need to be undertaken to confirm that new problems have not been inadvertently created by changes that resolve other issues. The number of new releases should be as few as possible and should be stipulated in the Contract. Ideally, there should be only one subsequent release of the system. Too many releases will result in extra work and delay the Acceptance Test schedule, bringing into question the overall quality of the delivered system. Where there is an The Acceptance Testing Kit – Part 2 external system supplier, the consequences of this should be determined by the contractual conditions. The Acceptance Test Manager and the Project Manager, with advice from the Service Provider Project Manager if required, will determine the degree and extent of retesting necessary. ‘Patching' The System While a "quick fix" may seem like a good idea at the time, don't rush into any process of making software or system "patches". It can often result in undocumented changes together with problems reconciling subsequent system releases with any patches and can lead to loss of control of the system and the status of the Test Environment. The Acceptance Test Manager can authorise patches, but only after considering all the options and preferably, allowing for the ability to reverse any changes. Remember to keep accurate documentation! Suspending or Cancelling Testing Where problems exist that inhibit the continuation of effective Testing, the Acceptance Test Manager (under advice from the Project Manager) may suspend Acceptance Testing until the relevant problem or problems are resolved. If no resolution is possible in the immediate future, there may be grounds for cancelling any further testing. The criteria for suspending or cancelling Acceptance Testing should be documented in the Acceptance Test Plan and in the Contract. The Acceptance Testing Kit – Part 2 Phase 4 Closure
Formal Acceptance
Once all Acceptance Criteria have been met, the Acceptance Test Manager is in a
position to recommend formal acceptance of the system. Upon this recommendation
to the Project Manager, the Project Sponsor or Project Steering Committee will
formally accept the system.
Acceptance is possible even if there are still specific (yet minor) outstanding problems, providing agreement has been reached on problem resolution following system implementation. The system may be accepted with the qualification that these non-critical items will be corrected to an agreed schedule under Warranty or Contract variation. In this way, acceptance can be achieved without unnecessary delay for minor items of non-compliance. The decision to allow a "qualified acceptance" and the items deemed "minor" is the prerogative of the Acceptance Test and Project Manager, and the Manager of the Business Unit that owns the system. Ideally, there should not be undue delay from completion of Acceptance Testing to system implementation. However, this may depend on overall Project and implementation planning. Test Records Management and Retention As far as is practicable, all test records should be retained at least for the duration of the Warranty period. Ideally the electronic components can be archived to off-line media. Test Cases, Test Scripts, Test Data, and expected results should be retained (as far is practicable) for testing subsequent releases of the system. These records should be formally handed over by the Acceptance Test Manager to the Business Unit Manager for safekeeping. You will usually find what corporate information management guidelines are relevant from IMB in your Division. The Acceptance Testing Kit – Part 2 Other Acceptance Testing Issues
The Tender Process
It is important to assess the ability of a Service Provider to satisfy system Acceptance
Criteria. Consequently, it is necessary to include these in any Request for Tender or
Quotation and to consider them in assessment of submitted proposals.
Possible Acceptance Criteria are outlined in the section How Do We Define Acceptance Criteria? on page 16. Contract Management Where a Contract is involved with delivery of a system, it is essential that Acceptance Criteria be included in the Contract. The standard GITC allows for the inclusion of Acceptance Test Criteria. These provide the Service Provider with a clear understanding of what is to be achieved to meet Acceptance. It is quite common for the business requirements to be a formal attachment to the Contract. Including Acceptance Criteria in the Contract also means that Acceptance and Acceptance Testing is more likely to be fully recognised at inception of the Project. As well as including Acceptance Criteria, the Contract should: ► Define the broad responsibilities for Acceptance Testing and for the provision of facilities and resources; ► Define the Operating Environment for the system (and by implication, the Acceptance Testing Environment); ► Define the duration (and possibly the schedule) for Acceptance Testing; ► Contain conditions to cater for delays in Acceptance Testing caused by either If there are clauses that require Acceptance Testing to be completed in an agreed time frame, then to ensure continuity and progress, it is important that personnel assigned to Acceptance Testing will be relieved of any conflicting responsibilities. This will allow them to be available in accordance with the Test Plan. Allowances should also be made for overruns and delays in Acceptance Testing. The timetable for Acceptance Testing in the contract and Test Plan should allow for a maximum of two sets of changes to the system (following the completion of the first pass of Acceptance Testing) and retesting for those corrections. The need for additional Releases may call in to question the quality of the system and the quality of Testing undertaken by the Service Provider. The Acceptance Testing Kit – Part 2 Documentation End-user and operational documentation for a newly delivered system also needs to be Acceptance Tested. Appropriate end-users should be assigned to Acceptance Testing the documentation by following it through and evaluating it for clarity and completeness. For each system function tested there should be a Test to check the documentation and any on-line or context-sensitive Help component. Those ultimately responsible for operating the system in production also need to review the documentation and follow through the operation instructions. This should be done in a Test environment that emulates the final production environment as closely as possible. Once again, documentation should be subject to systematic document management and change control procedures. System Design Where there are complexities in testing aspects of a system, it may be necessary to design features into a system that facilitate testing of the system e.g. for a voice-activated payment facility, the practicalities of testing this across a telephone network may require the system to have built-in features to support Testing. Package Systems Where a solution is to be based on a package, selection of the package is often made based on the package's perceived ability to satisfy the business requirements. In this case, it may be practical (or even contractually agreed) to test the package against its documented capabilities. Where a package is delivered with modifications, specifications for those modifications should be documented and agreed upon, and system documentation should be updated accordingly. These specifications can be used as the basis for Acceptance Tests. A second aspect of a package implementation is that a key factor in its selection will have been the cost-effectiveness of the solution i.e. as a rule of thumb, it's cheaper to implement a package if it meets 80% or more of the business requirements, than to custom-build a system. However, selection of a package will often involve compromise of at least some business requirements. Apart from these issues, Acceptance Testing a package system should not differ in approach to that for a custom-built system. The Acceptance Testing Kit – Part 2 Acceptance Testing a Large System Planning It is to be expected that the extent of a Test Plan for a large system will be greater and consequently, the time necessary to prepare the Plan will be more significant. Similarly, the number of stakeholders may be greater and more time may be needed to allow adequate review of the Plan. It may also be a contractual requirement that the system supplier also approves the Test Plan. If this is the case, some allowance should be made for negotiation. The Plan may imply certain functionality that the supplier believes is not part of the system requirements. Preparing Tests, Test Data, and Training As the number of Test Cases increases, so does the complexity of coordinating and ordering the Tests into Test Scripts. This is likely to involve more than one person to execute and will require suitable management. Similarly, the complexity of providing Test Data will increase and its preparation and maintenance will need to be managed. There is likely to a larger group of testers and the range of skills, knowledge, and capability will vary. Preparation of a Skills Transfer and Training Plan is needed to detail delivery and management of training needs. For the Acceptance Testing to be (contractually) valid, the Test environment needs to mirror the final operating environment, as defined in the contract. The Acceptance Test Manager will need to oversee this step and the system suppliers may request an audit of the Test environment. The degree of coordination and planning will be higher. There may be a wide range of resources and personnel to be confirmed. Execution and Control Its important to use the Register of Problem Reviews and plan several runs of Testing, in particular, regression testing. An allowance to accept the system subject to subsequent resolution of documented outstanding (minor) problems should be made. If this is the case, the Problem Review Register should be part of the Acceptance Document. The records for Acceptance Testing of a large system can be quite extensive. It's important that these are retained safely. The Acceptance Testing Kit – Part 2 Acceptance Testing a Small System A small project is defined as one where the external cost does not exceed $50,000. With a smaller project or system, the overall complexity of Acceptance Testing is reduced. There are fewer people involved, there are fewer types of data and the number of tests and problems is lower. Nevertheless, the basic steps listed still need to be executed. ► Prepare a Test Plan ► Review the Test Plan ► Arrange sign off Preparing Tests, Test Data, and Training ► Prepare the Test Cases ► Prepare the Test Scripts ► Prepare the Test Data • Conduct training • Set up test environment • Confirm people and resources Execution and Control ► Run Test Scripts and Test Cases ► Record the results ► Log and manage problems ► Fix problems and re-test The relatively small number of problems from testing a small system should mean a Problem Review Register is not required. ► Formal acceptance ► Hand over the system The Acceptance Testing Kit – Part 2 System Enhancements and Maintenance Usually changes to an existing production system are grouped together into a ‘Release'. The source of these changes may be requested functionality enhancements and/or resolution of reported problems. Existing DIER applications should already have processes established for system upgrades, maintenance, and enhancements. It may not be necessary to establish a formal Acceptance Test Plan, strategy, and procedures. However there is a minimum set of details that should be defined in order for the Acceptance Testing process to work successfully with established systems. There should be an agreed schedule and an understanding of who is conducting Tests. The responsibilities for providing a test version of the changed system also need to be understood. For systems with established support and testing processes it may be useful to go through a process of confirming any testing requirements and confirming agreement on the testing process. Sign-off need not be a significant undertaking. Preparing Tests, Test Data, and Training Apart from any new system functions, Test Cases and Scripts should already exist and these can be reused. A Test database may already be available. It may be necessary to ‘refresh' this by extracting data from the production system. Processes to do this should already exist. Training is probably unnecessary as experienced users should be available. There may be a need to ‘run through' any new functions with the developer. Given the minimal workload, it is quite likely that a single person can undertake all tasks. Upon formal acceptance of the revisions, it will be necessary to arrange installation of the new version of the system with the Information Management Branch in your Division. The Acceptance Testing Kit – Part 2 Some Cautionary Notes
Test Data Confidentiality
Acceptance Testing often involves the use of confidential data and the production of
real documents, real interface files, generated e-mails, etc.
It is important that during preparation of the Acceptance Test Plan, procedures to protect the security and confidentiality of data are included. Interpretation of Business Requirements Business system requirements may be open to misinterpretation. A system supplier (either internal or external) may contest the manner in which a system meets the business requirement. Therefore, the more accurately and specifically that business requirements are specified, the less open to misinterpretation they become. Resolution of these issues is a common part of negotiating acceptance of a system. If the system supplier accepts or endorses the Acceptance Test Plan, then these issues should arise early in the process, minimising any conflict at a later stage. System Prototyping System prototypes are sometimes built during system design and development as part of an iterative approach to system development. The prototype may be a working model of a system that will eventually be used and tested and it enables end-users to experience first-hand the proposed interaction with the system. The prototype is not a full working model and merely simulates the proposed system's behaviour. Any feedback and requested changes are incorporated into the system and the system trialed again. However, this iterative trialing of a system is strictly part of the development process of satisfying the system requirements and does not replace formal Acceptance Testing. Parallel Running Parallel running is normally conducted after implementation prior to a full cut-over to the new system. However, parallel running may also occur during Acceptance Testing. Parallel running is where an existing system continues to operate as normal, with the new system duplicating the processes and activities. The results of the old and new systems can then be compared. Caution should be taken when planning parallel running as it can place additional and excessive demands on end-users' time. It may be necessary to consider increased personnel numbers. The Acceptance Testing Kit – Part 2 Minor System Deficiency Delays Even minor system modifications, implemented at a late stage of development to overcome deficiencies in the business requirements specification, can cause final delivery of the system to be significantly delayed. Acceptance of the system, subject to any minor modifications being done after the system has been fully implemented, is at the discretion of the Project Manager and Steering Committee. Generally, these issues will be handled under change management or contract variation procedures.

Source: http://www.egovernment.tas.gov.au/__data/assets/pdf_file/0009/77949/Acceptance_test_kit_-_part_2.pdf