For more than 50 years, Auerbach Publications has been printing cutting-edge books on all topics IT.

Read archived articles or become a new subscriber to IT Today, a free newsletter.

This free newsetter offers strategies and insight to managers and hackers alike. Become a new subscriber today.


Partners




Contact

Interested in submitting an article? Want to comment about an article?

Contact John Wyzalek editor of IT Performance Improvement.

 

Using Measurement to Identify Improved IT Performance

David Garmus

Improved performance in software development can be achieved by investing in best software development practices. This article will discuss how one organization identified improvements in their performance by collecting a combination of quantitative measures and qualitative values. The organization then utilized the results to advance their process improvement programs and improve their development practices throughout.

The desire to achieve process improvement was driven by senior level management. Management wanted results that would directly impact stated business goals and objectives, which included: reduction of project costs (mostly effort), improvement in their delivery (project duration) of software, achievement of better product quality by minimizing defects delivered and improved overall organizational performance relative to industry benchmark data points.

Their organizational strategy was to achieve these goals by defining and leveraging best practices in software development. Senior management had a well defined vision of what they wanted to accomplish, and they agreed to dedicate the resources necessary to achieve the desired results. The ability to properly set management expectations and to gain their support was enhanced by the introduction of a measurement model that objectively and quantitatively displayed the results (both good and bad) of implementing best practices.

A Measurement Model

The key to successful performance management is measurement. The inclusion of performance measurement to manage and direct decisions is becoming more commonplace. Organizations typically recognize the necessity of establishing strategic goals and objectives; however, they frequently lack an appropriate set of measures that will provide quantitative evidence that those goals and objectives have been achieved.

A basic measurement model that was advanced by the Practical Software and Systems Measurement (PSM) program suggests that an organization follow these three steps:

  1. Identify the needs of the organization
  2. Select metrics appropriate to determine whether the needs have been met
  3. Integrate measurement into the software development process

The management of this organization identified the needs of their organization. The David Consulting Group (DCG) was requested to assist the organization in selecting appropriate metrics and in creating a measurement model that would result in the quantification of process performance levels and that would provide the ability to compare internal performance measures to industry benchmark levels of performance.

The basic measurement model (see Figure 1) used included the collection and analysis of both quantitative and qualitative elements. The quantitative elements included four basic metrics: size, effort, duration and defects. The qualitative elements included a variety of data points that were used to evaluate levels of competency regarding process, methods, skills, tools and management practices.


Figure 1. The Basic Measurement Model

Collected on a project by project basis, quantitative data can be displayed in a measured profile that indicates proactively how well a project is performing. Standard industry measures such as function points per effort month, defect density and project duration must be calculated. If function points are used to measure project size, there is an opportunity to make comparisons to industry data points that are also based on function points.

The qualitative data (again collected on a project by project basis) results in a matching capability profile. This profile data identifies the attributes that contribute to high or low yields of performance, such as those indicated through the Software Engineering Institute’s (SEI’s) Capability Maturity Model Integrated® (CMMI®).

These two elements (quantitative and qualitative) are used to determine what is commonly viewed as an organization’s baseline of performance. The baseline values are compiled from a selection of measured projects and represent the overall performance level of the organization.

Results can vary significantly. Some projects perform very well (i.e., they have low cost and high quality), while other projects do not perform well at all. Quantitative data provides senior management with an objective view of current performance levels. The qualitative data provides the opportunity to examine the project attributes to determine why certain projects outperformed others. Baseline measures on a sample set of representative projects can provide senior management with the information they need to make informed decisions. This analysis effort leads an organization to the identification of their best practices and opportunities for improvement.

The organization in this article wanted to determine the impact that the introduction of SEI CMMI® Level 3 processes had on their performance. In order to determine this improvement, the organization had to first compare its previous baseline of performance and establish a composite profile of contributing attributes.

Project data was then collected and analyzed. Averages for size (function points), productivity (function points per effort month), duration (calendar months) and effort (labor) were computed. Using a composite profile, a mapping of the previous project attributes for the organization was developed. In parallel, another model was developed for projects of a similar size with a mapping of attributes that matched CMMI® Level 3 characteristics.

The Findings

The impact of achieving CMMI® Level 3 for this organization was significant. For a similar size of enhancement projects (approximately 133 Function Points), productivity (Function Points / Effort Month) was projected to increase by 132%, project duration reduced by 50%, effort reduction by 50% and defect density reduced by 75%. This modeling technique helped this organization in evaluating the potential benefits of the CMMI® process best practices. Refer to Table 1.


Table 1. Summary of Findings

The potential impact indicated above may appear to be dramatic, but significant gains in productivity and reduction in defects should be expected over time as the organization matures. In fact, this particular organization exceeded those expectations within a two year period of time.

Summary

There are a variety of ways in which measurement data can be used to learn more about:

  • An organization’s level of performance
  • Key factors that contribute to high or low yields of productivity
  • The organization’s level of performance as compared to industry data points
  • The potential impact of strategic initiatives.

The utilization of a measurement model that includes a quantitative perspective as well as a qualitative perspective is very important. It is from this vantage-point that an organization can access the measured performance profiles along with an understanding of the process profile elements that contributed to the results. The process profiles have the added advantage of recommending a direction for future improvement strategies.

Similar outcomes can be achieved in your organization. A prudent action would be to take your own measures and create your own organizational performance baseline. Utilizing industry accepted measures, such as function points per effort month, will enable you to perform the necessary comparative analysis. The investment to perform a baseline study is relatively insignificant when compared to the value of the information gained and the potential return from process improvement practices.♦

Read more IT Process Improvement

About the Author

David Garmus was a founder of The David Consulting Group. He is an acknowledged authority in the sizing, measurement and estimation of software application development and maintenance. His engagements typically result in the forecast or evaluation of effort, size and schedules for development projects or application support. He is recognized as an expert court witness in U.S. Federal Courts on the sizing and valuation of software.

Mr. Garmus’ experience includes military systems, operating systems and management information, inventory control and financial management systems. Mr. Garmus has assisted numerous clients in identifying opportunities for improvements in productivity and software quality during engagements that have included a wide variety of challenges facing information systems executives.

Also by David Garmus

Certified Function Point Specialist Examination Guide

While there are a number of solid books on counting function points, this new book fills a gap in the function point literature by providing useful information on the specifics of becoming a certified function point counter. The authors are all qualified for the work at hand, and indeed have contributed to the function point counting methodology.
—Capers Jones, President, Capers Jones & Associates LLC