IT Today Catalog Auerbach Publications ITKnowledgebase IT Today Archives Book Proposal Guidelines IT Today Catalog Auerbach Publications ITKnowledgebase IT Today Archives Book Proposal Guidelines
IT Today is brought to you by Auerbach Publications

IT Performance Improvement



Networking and Telecommunications

Software Engineering

Project Management


Share This Article

Search the Site

Free Subscription to IT Today

Powered by VerticalResponse

Cloud Computing: Implementation, Management, and Security
The Green and Virtual Data Center
Best Practices in Business Technology Management
The Effective CIO: How to Achieve Outstanding Success through Strategic Alignment, Financial Management, and IT Governance
Enterprise Architecture A to Z: Frameworks, Business Process Modeling, SOA, and Infrastructure Technology

Software Testing Project Execution

By Ashfaque Ahmed

In an ideal world, project planning would be the main task and project execution would be like pressing a button to start and finish it. Alas, this is not the case. In many industries execution is still the king. This is because despite all the advances in automation and standardization of processes, executing any plan is still difficult. The road to execution is laden with unimaginable pitfalls and unavoidable circumstances that ensure that execution is a challenge and not a walk in the park.

Imagine you have made a good project plan, you have a good set of people in hand, you have good experience in handling testing projects, you have a good reputation with your customer, you are dedicated to the task, and you are determined to excel in the task.

So you think you have a good test project plan with most of the risks covered, good effort estimation done, and project strategies well defined. Well, only 5% of your job is done. Now you should be ready for the remaining 95% of the job. And this part of the job is more challenging, because you will face real problems, issues, challenging situations, and conflicts that will require your tact, patience, ability to make fast decisions in stressful situations, good reporting, and many more qualities.

If you are working on an iterative and incremental development model such as product development, then you will need to spend more time in understanding the architectural and functional model of the application as well as development framework so that for new releases you will be able to plan for testing without the need to have elaborate requirements and specifications for the new release, which are scarce commodities in such environments.

So if you are new to this environment, then you should try to get as much information on these topics as possible by going through product documentation, attending knowledge transfer sessions, going through the application by login with different roles in the application, and finding out about process flow, workflow, business logic, and so on. Once you are comfortable with the application, it will become easier for you to test it and make plans to test it.

For outsourced projects, it is very important that along the execution cycle you provide status reports from time to time. For this reason it is extremely useful to track your project using earned value management (EVM).

1. Earned Value Management

Whenever a project is initiated, there is a value associated with it against the costs that will be incurred in executing that project.

There is a big difficulty in implementing EVM in software projects. The reason is that in software projects, it is very difficult to make good estimates for efforts. That is why most software projects are executed on a time-and-material basis instead of on a fixed cost-fixed schedule basis. In such a scenario, using complex metrics like EVM is even more difficult (see Figure 1).

Figure 1. Project tracking without earned value.

However, in software test projects, it is still possible. Software projects that are offshored and where the service provider has many projects going on at any time are especially amenable to EVM implementation. This is because, due to extensive project execution knowledge, the service providers have a good repository of knowledge on which they can depend for new projects. The repository can provide critical information on which new projects can be estimated to a great accuracy. So EVM can definitely be applied for offshored software test projects (see Figures 2 to 4).

Figure 2. Schedule variance in project execution (variance by time and cost).

Figure 3. Cost variance in project execution.

Figure 4. Complete earned value chart for the project depicting actual cost, earned value, and planned value.

Now let us consider how we can implement EVM in software test projects.

1.1 Need for EVM
Any project is initiated to create value for the organization for which the project was conceived. If for some reason, at any stage of the project, things get delayed or resources are wasted, then it puts a question mark on the promised delivery of the project. If you track a project just in terms of percentage of work completed or percentage of budget spent, then it is difficult to know if the project is on track or ahead of schedule or is lagging behind. From this information it is also not clear as to what factors are causing delays or wasting of resources.

For this very reason, it is important to keep a clear charter of the project and its progress at any given time to know where the project is heading. By charting an ideal course of progress for the project from the start and comparing it with actual progress of the project, you can clearly see what factors, if any, are causing problems in the project and what appropriate actions can be taken to remedy this situation.

For instance, using EVM, it is possible to know which task in the project is getting delayed and what impact it can have on the entire project as well as on other tasks in the project. At the same time it will also give information as to how much impact it can have on the project.

EVM is also extremely useful in providing status reports to stakeholders of the project. It provides information such as when the project will finish given the current status, how much it will cost to finish the project, how the project performance fares against similar projects, how much it is costing to earn each unit of forecasted value, and so on.

EVM has good capability for reporting at program and portfolio levels. With many software service providers executing a lot of projects simultaneously, this becomes a handy tool to make reports at the program level for individual customers (having many projects being outsourced to the service provider) and make reports at the portfolio level for projects being executed at the departmental level.

1.2 EVM Implementation for Software Projects
Some of the assumptions that are needed before we move on to discussing implementing EVM for software test projects are as follows:

  • The project is not on a time-and-material basis: If this is the case, then required information necessary for calculating earned values will not be available for the project.
  • The project has relevant values available for baseline and actual information at any given time when a status report has to be made.

The primary requirement for implementing EVM is that you create a baseline schedule, budget, and resource requirement for your project. That means, at the very outset, you have complete information for your project including budget, schedule, and resource requirement. From this information you create baseline information for your project. From the baseline information you create an ideal progress path for the project. As the project progresses, you track your project by comparing actual values for cost and schedule to the baseline values.

To adopt the EVM methodology for software projects, certain modifications in the way software projects are tracked need to be made. One such requirement is that a gate should be introduced in the project before completion of any phase. The project will not progress to the next phase unless the quality standards are met for that phase.

To track projects at the portfolio or program level, a standard framework for stages and gates that will apply to all projects should also be implemented. The determination of the relative cost for completing a stage based on industry standards or past project performance should be made available. The definition of the work breakdown structure, outlining the work products/deliverables to be produced at each stage/phase of the project, should also be made available. The projects should have the capability to calculate total effort associated with producing each work product. The standard rate per hour for each project should be defined to avoid disclosing the salaries/costs associated with the team members. The number of resources assigned to each work product, which together with the estimated effort and standard cost determines the total cost for each work product, should be defined. A procedure to capture actual effort and costs associated with each work product should be defined in order to report deviations from the scheduled earned value determined by the model.

If these changes can be introduced in software project processes, then EVM can be implemented successfully.

1.3 Audit Trail
Nowadays government regulations compliance has become an important part of most activities performed by any business house. The Sarbanes-Oxley Act (Public Company Accounting Reform and Investor Protection Act of 2002) provides provision for information to be made available to government agencies. To comply with this act, software applications now include audit trail functionality so that any financial or accounting transactions can be tracked to know if any modifications were made in that transaction in the past.

For complying with the audit trail, EVM becomes a useful tool because in EVM the baseline and actual project information is kept and can be shared at any time.

2. Defect Tracking and Life Cycle

Once test cases start getting executed, the test team starts finding defects in the system. They have to report these defects in the proper manner so that the defects can be tracked, fixed, and closed after verification.

The life cycle of a defect includes open for assignment, assigned, open for verification, failed, reassigned, passed, or closed (see Figure 5).

Figure 5. Defect life cycle.

In the defect tracking system, the tester should open a new defect when he finds it. He should use a proper template for defect logging so that it will be easier for everybody in the loop who is responsible for assigning defects, fixing defects, reassigning defects, and closing defects.

Here is a template that can be used for reporting defects:

Name of tester:
Version or build: (Version or build of the product)
Module or component: (mention here the name of tested module or component)
Platform/operating system:
Type of error: (coding error/design error/suggestion/UI/documentation/text error/hardware error)
Assigned to:
Defect name:
Test data:
Steps to create defect:
Expected result:
Actual result:

3. Monitoring of Production Systems

Most of the production instances are tested nightly to check if the application is available or not available, if any functionality of the application has any problems, or if the application has any performance issues.

There are some issues involved in monitoring production instances. To run sanity tests, you need to have some testing data in the production instance database. These data have three issues. First, this test data should be completely isolated from production data that is critical to the business. This test data should never interfere with production data. Second, the test data should be such that it should not get deleted by any DBA or anybody responsible for maintaining the database.

Otherwise the test scripts cannot run. The best convention is that you should use some prefix or suffix with this data to identify that it is for testing purposes only. For instance, you can add a prefix like "qa_" with each test data. Third, the test scripts should never generate too much transaction data. Otherwise the database will be cluttered with unwanted data. So test scripts should be written such that if any transaction data is created, then the script should delete that data after the testing run is over.

The nightly sanity tests should be completely automated so that they can be executed fast, and no test engineer should do it manually. It should run automatically at a specified time so that no human intervention is needed to start them. Whenever a patch is applied on the production instance, the sanity test suite must be run to ensure that all application functionality is working fine.

4. Test Case Execution

One bottleneck during test execution is that many test cases may be dependent on each other. If the first test case fails, then the dependent test case will also fail. If the first test case has failed, then there is no point in executing the second test case. In software testing parlance this is called the showstopper defect. Unless the first test case passes, the dependent test cases cannot be executed. So apart from the usual categories of defects-that is, minor, medium, and severe defects-we have an additional category called showstopper. The development team must fix this defect as a priority so that test case execution does not get affected badly.

5. Checklist for Test Execution

  • Has test data been created and validated?
  • Has an inventory of test cases been created?
  • Have regression test cases been identified and created for this release of the application?
  • Have logical groups of similar test cases been created?
  • Have test cases and test data been reviewed and approved by customer?
  • Are all test support resources available?
  • Have expected results of each test case been defined?
  • Have automation scripts for all automated test cases been completed?
  • How do actual results compare with expected results for all test cases?
  • Have detected defects been fixed and verified? If not, is a work-around provided?
  • Has the final unfixed defect list after execution cycle completion been provided?

About the Author
Software Testing as a Service by EAshfaque Ahmed Chapter 7 from Software Testing as a Service by Ashfaque Ahmed, New York: Auerbach Publications, 2009.

© Copyright 2009-2010 Auerbach Publications