For more than 50 years, Auerbach Publications has been printing cutting-edge books on all topics IT.

Read archived articles or become a new subscriber to IT Today, a free newsletter.

This free newsetter offers strategies and insight to managers and hackers alike. Become a new subscriber today.


Partners




Contact

Interested in submitting an article? Want to comment about an article?

Contact John Wyzalek editor of IT Performance Improvement.

 

Achieving Business Objectives: Building a Software Metrics Support Structure

Jim Mayes

The business objectives of software projects should include balancing time, cost, and quality with the expected business value of the software produced. Software metrics are critical to this endeavor.

The benefits of a metrics program must be continuously justified in terms of business value and raison d'être. This is true whether we are trying to initiate a software metrics program, or trying to sustain one due to organizational changes, budget constraints, or other factors. Unfortunately, a metrics program is often perceived as an intrusive and unnecessary interference with getting "real work" done. The bottom line is that the metrics program must be directly aligned with business objectives at the project level in order to be successful and provide value. This chapter focuses on building a structural model to illustrate how this can be accomplished. Like building a house, this structure has four layers—foundation, floor, walls and roof—with each layer supporting the layers above it.

The primary component of this model is the establishment of Software Estimating Metrics Group (SEMG) within the foundation layer. Just as the CMU SEI-Capability Maturity Model (CMM) establishes a Software Engineering Process Group (SEPG) for managing software processes, a SEMG should be established for managing software metrics. The SEPG and SEMG functions complement each other, and have common goals in relation to achieving business objectives.

Background

In 1994, after 20 years as a software programmer and development manager, I read Ed Yourdon's book, The Decline and Fall of the American Programmer.[1] It was an epiphany for me, as if someone opened the curtains and turned on the lights for the first time. It was like Information Technology (IT) had been stuck in the dark ages, but now there was hope that the IT environment could be significantly improved. The evolution of process, metrics and quality initiatives over the years have certainly enhanced these prospects, but there remains a long way to go. Since reading Yourdon's book I immediately became engaged in process improvement initiatives as a SEPG Manager, SEMG Manager, and Metrics Consultant, primarily focusing on estimation.

Why did initially I focus on estimation? As a software development manager, one of my most difficult, yet critical tasks, was providing justifiable software project estimates. At that time, our basis for estimation, as one of my co-workers once put it, was more religious in nature, based on faith in our ability to deliver, rather than based on historical data; we provided hysterical estimates instead of historical estimates. Therefore, as I focused my energy on process improvement related to software estimation, I realized that software metrics are an integral part of estimation, continuous improvement, risk management, and project management. As to the importance of process improvement related to estimation, Putnam and Meyers put it this way: "An organization can accomplish its estimates only if it has a software development process it can repeat."[2]

The importance of project-level metrics (actual project results), cannot be overstated, because "if we don't know where we are, a map won't help". If our current capabilities aren't quantified, it would be difficult to determine what needs to be improved and what baseline to use for measuring improvement.

"So You Can Measure, So What!"

At every International Function Point Users Group (IFPUG) conference, Bill Hufschmidt handed out buttons with catchy phrases on them. At one conference, he gave out buttons with the phase: "So You Can Measure, So What!" I loved that button, and this is what it meant to me:

Merely measuring things or counting function points isn't what software metrics is all about. It's what you can do with the information, and the metrics derived from the data that matters. When metrics are only used for management reporting, the primary benefits are lost. In order to maximize the potential benefits, software metrics should be traceable to the project level, so they can be put to work, providing value to the overall software development process.

In this context, it is important to understand the definition of what is meant by measure, metric, and indicator (MMI). Basically, a measure is a standard unit of measurement, such as function points for size or hours for effort. A metric combines two or more measures into a meaningful derivation, such as using hours per function point as productivity metric. An indicator is something that draws attention to a particular situation, such as a flag or deviation outside predetermined tolerances or control limits. These MMI terms distinguish the difference between software metrics and project management status reporting. [3]

It is also important to understand these MMI terms in relation to how software metrics are perceived by the software organizations being measured. In his book Why Does Software Cost So Much? Tom DeMarco wrote, "I observe there are at least three different reasons we collect metrics, as follows:

  • to discover facts about our world
  • to steer our actions
  • to modify human behavior"[4]

Figure 1. DeMarco's Metrics Spectrum
Reprinted by permission of Dorset House Publishing from Tom DeMarco, Why Does Software Cost So Much? And Other Puzzles of the Information Age, pp. 16-17. Copyright © 1995.

DeMarco goes on to suggest a Metrics Spectrum, as shown in Figure 1, with Behavior Modification on the left, Steering in the middle, and Discovery to the right. The objective of the SEMG is to provide discovery metrics for continuous improvement, and steering metrics for risk management. Demarco suggests that the further to the right you are on the metrics spectrum, the more successful your use of metrics will be. As you move toward the left side of the spectrum (Behavior Modification), you must beware of the Hawthorne effect, where people respond differently just because they are being measured. An example of behavior modification that you don't want was illustrated in a Dilbert comic strip several years ago. In the first frame the pointy-haired boss said that he was going to pay a "bonus of ten dollars for every bug found and fixed" . A subsequent frame showed Wally shouting: "I'm going to write me a new minivan this afternoon." (Adams 1996) As this illustrates, a software metrics program should focus on the connection between steering and discovery, and stay away from behavior modification metrics of this type. It should be clear that the MMI's are related to continuous improvement, managing risk, and balancing time, cost and quality (TCQ). A balanced set of software metrics is important. To quote Karl Wiegers[5]:

"A risk with any metrics activity is dysfunctional measurement, in which participants alter their behavior to optimize something that is being measured, rather than focusing on the real organizational goal. For example, if we are measuring output productivity but not quality, expect some developers to change their coding style to expand the volume of material they produce, or to code quickly without regard for bugs. I can write code very fast if it doesn't actually have to run correctly. The balanced set of measurements helps prevent dysfunctional behavior by monitoring the group's performance in several complementary aspects of their work that lead to project success."

Blueprint for a Software Metrics Support Structure

A multi-layered metrics support structure is shown in Figure 2, with each of the four layers building onto the layers below it. Each layer is also dependent on the other layers for achieving business objectives, just as a building is dependent on its foundation, floor, walls, and roof in order for it to be structurally sound and to achieve its functional goal. There are many similarities in managing the construction of buildings and software. (Okay, so I have a degree in building construction from an engineering school, and This Old House is one of my favorite television programs.) How easy would it be to construct a building without any measures? It would be impossible, right? Because the construction mantra is: "Measure twice, cut once." Of course, software products can be constructed without measures, metrics, or processes; however, there are plenty of industry figures showing the high percentage of project failures each year. There are also an equally high percentage of projects that are completed, but fail to provide the appropriate business value due to cost or schedule overruns. It is important, therefore, to quantify project results and use empirical analysis to manage these risks, improve estimates, improve processes, and assess business value.

Figure 2. Software Metrics Support Structure

The environment needed to support this initiative consists of the following structural components (layers):

  • Foundation: Continuous Improvement
  • Floor: Risk Management
  • Walls: Project Management
  • Roof: Business Objectives

The SEMG provides support related to each of these components, as described in the paragraphs that follow.

Foundation: Continuous Improvement

A SEMG that is established in relation to a continuous improvement program and a process management group (SEPG), lays the foundation for achieving business objectives. The concept of establishing a group for managing CMM measurement practices is supported by CMU SEI-92-TR-25 Technical Report, "Software Measures and the Capability Maturity Model," [6], which states:

…many of the potential benefits that an organization can derive from a sound measurement program are often not achieved due to a half-hearted commitment to a measurement program. The commitment cannot be just a policy statement; it must be total commitment. The policy must be followed with the allocation of resources to a measurement program. This includes allocating staff as well as tools. It is important that an individual or group be assigned responsibility for the measurement program. This group identifies the measures to be collected, establishes training in measurement, monitors the consistency of the measures across the projects throughout the organization, and can help initiate measures. This group can also provide composites of historical data that managers can use in planning and monitoring projects. The human element in a measurement program should not be taken lightly. Success and failure are tied to people. All staff members need to see the benefits of the measurement program and understand the results will not be used against them.

Another Technical Report, CMU SEI-97-HB-003, "Practical Measurement for Process Management and Improvement," by William A. Florac, Robert E. Park, and Anita D. Carleton, "shows how well-established principles and methods for evaluating and controlling process performance can be applied to a software setting to achieve an organization's business and technical goals." The report focuses on "Process Management" related to operational software processes. Its primary focus is on the "enduring issues that enable organizations to improve not just today's performance, but long-term success and profitability of their business and technical endeavors."[7]

These CMU SEI reports describe different aspects of software measurement and metrics that should be managed by the SEMG for achieving business objectives. As stated in CMU SEI-97-HB-003, "Every organization asks the question, ‘Are we achieving the results we desire?' This gets couched in many ways, such as, Are our customers satisfied with our product and services? Are we earning a fair return on our investment? Can we reduce the cost of producing the product or service? How can we improve the response to our customers' needs or increase the functionality of our products? How can we improve our competitive position? Are we achieving the growth required for survival?" The vital SEMG role in this layer is shown in Figure 3, which illustrates the process management relationship between continuous improvement, the SEPG, and SEMG, in order to answer these questions. It is a chart provided in CMU SEI-97-HB-003 that I have adapted to show the SEPG functional boundary, the SEMG functional boundary, and where the boundaries overlap.

Figure 3. Continuous Improvement, SEPG, SEMG Relationship Model

The SEMG provides the measurement capability and expertise necessary to analyze process improvement opportunities using statistical analysis techniques for process control, root cause analysis, and solution analysis, which include:

  • Control Charts
  • Scatter Plots (Trending)
  • Run Charts
  • Cause-and-Effect Diagrams
  • Histograms
  • Bar Charts
  • Pareto Charts

Floor: Risk Management

"Risk Management is project management for adults."[8] This was the title of a keynote presentation that Tom DeMarco gave at the Quantitative Software Management (QSM), Inc. Users Conference. In the presentation, he stated that the software industry today is developing process sophistication and orderliness, via the CMM; however, the use of the term maturity in the CMM is deviant to the term that he would use to recognize project management adulthood. A great manager is one that deals with uncertainties, not certainties, and this is the fundamental reason that we are not mature. In order to effectively manage a software project, managers must be aware of the real world, and must have the capacity to deal with the real world. The SEMG can help quantify and identify these uncertainties, via discovery and steering metrics, so they can be managed.

If you go back to Figure 2, you see the Risk Management layer, which includes process and metrics, adds a floor on to the foundation layer supported by the functions of the SEPG, SEMG and Continuous Improvement. Of course, one of the primary reasons for implementing CMM processes is to reduce risks associated with software projects. Therefore, the SEPG's primary role in this layer is to reduce risk by insuring adherence to process, which includes a risk management process as well. The SEMG role is to provide statistically quantifiable data for managing risks related to project estimates and indicators for triggering alarms during a software project's lifecycle. DeMarco describes the Circular definition of Risk as follows:

"A Risk is a problem that has not occurred. A Problem is a risk that has occurred. A Transition is a problem going from risk to problem. This is illustrated in the Trucker's Maxim: behind every ball comes a running child (transition), apply brakes when you see the ball, and do not wait until you see the child."

The triggers and alarms (indicators) provided by the SEMG can identify transition, and help prevent a risk from becoming a problem. Project Managers will then be able to "apply brakes when they see the ball" (indicator). The SEMG functions that facilitate this ability include:

  • Validation of top-down versus bottom-up estimates to compare them to historical results and identify the risks related to estimates
  • Project monitoring of estimated versus actual data (e.g., size, schedule, effort, defect rates)
  • Predictive analysis as to probable project outcomes based on in-flight results
  • Providing indicators for managing risks associated with in-flight data variation outside of acceptable tolerances

Walls: Project Management

The triangular relationship in this layer includes Project Management for providing Customer Satisfaction and Business Value associated with the business objectives of the software project. The processes and metrics supported in the Continuous Improvement layer (the foundation) and Risk Management layer (the floor) carry forward to the Project Management layer of the structure. With the foundation and floor to build upon, the Project Management layer builds the walls, or software product, that will support the roof (Business Objectives) of the structural model. This layer operationalizes software development processes and project level metrics. It includes the following steps in the TCQ process for balancing time, cost and quality that are supported by the SEMG and software metrics:

  • Prioritizing business objectives and project constraints
  • Time-sensitive estimation for balancing TCQ
  • Providing alternative estimate solutions for partnering negotiation

Time-sensitive estimation is a critical component of the TCQ process. One of the primary reasons for projects failing to achieve business objectives is trying to deliver too much too fast. As schedules are compressed, project effort, cost, and defects exponentially increase. It doesn't take much compression to cause this effect. On the other hand, if the schedule is lengthened, effort, cost, and defects are exponentially decreased—balance is the key. One way or another the scale stays balanced as shown in Figure 4.

Figure 4. Balancing Time-Cost-Quality versus Business Objectives

The non-linear TCQ relationships associated with time-sensitive estimation are illustrated in Figures 5 and 6. This is not a theory, but is based upon the on empirical evidence as to how software projects actually behave, as proven by the work of Tom Demarco, Larry Putnam, Ware Myers, and others.[9,10] By applying the TCQ principles, the project schedule, cost, effort, and quality components can be prioritized and balanced in relation to business value and objectives. If the TCQ components are set in stone, then other alternatives may include de-scoping the project or assigning more highly skilled resources.

Figure 5. Effort Sensitivity to Schedule Compression

Software project estimates are not one-size-fits-all commodities. Alternative solutions should be provided for each project estimate, and the one that provides the most business value and has the most reasonable probability of success should be chosen. Signing up for an unachievable estimate serves the interest of no one. It may provide short-term gain in customer satisfaction with an estimate that satisfies the customer demand for an unrealistic completion date, but usually results in long-term pain for IT and the customer. Therefore, balancing TCQ with business value is only achievable when alternatives are provided and there is a long-term partnership between the client and IT organizations. Sharing information and trust are key elements to achieving customer satisfaction.

Figure 6. Quality Sensitivity to Schedule Compression

Roof: Business Objectives

Just as a roof completes the structure of a building, this layer is the culmination of the structural components below it in the metrics model. In this layer, the achievement of business objectives is realized via the time, cost and quality (TCQ) project results that drive business value. With partnering negotiation, the client organization determines the TCQ drivers related to the business, and the Information Technology (IT) organization determines the TCQ drivers related to the software. Joint ownership of the business objectives as a mutual goal, is the way to work through the issues objectively. A partnership between the client and the IT organization is required in order to achieve balance between software project estimates and business objectives. The layers below this level, Continuous Improvement, Risk Management, and Project Management are essential for successfully accomplishing this goal.

Also in this layer, the SEMG quantifies and analyzes the overall project results as they relate to achieving business objectives. This analysis leads to the identification of continuous improvement opportunities related to software processes, predictive analysis for risk management, and balancing TCQ. This is an iterative process with the experiences of previous projects being used to help new projects achieve greater success.

Conclusion

The benefits of a software metrics program must be justifiable in relation to business value and business objectives. The Software Metrics Support Structure and SEPG, SEMG, and TCQ initiatives illustrate how project-level software metrics provide value and are essential to achieving business objectives. Establishing a SEMG in this context provides the focus, continuity, experience, backup, and reinforcement of professional skills needed for sustaining and improving an organization's management capabilities for achieving business objectives. To provide these benefits, it must be a centralized function staffed by dedicated metrics and estimation specialists.

Without a metrics program supported by an SEMG, it would be difficult, if not impossible, to effectively achieve business objectives or to continuously improve project results. To quote Michael Mah: "Your history will help you better manage the future". The SEMG provides the corporate memory needed to learn from the past and build a better future.[11]◘

References

[1] Yourdon, Ed. 1992. Decline and Fall of the American Programmer. New Jersey: Prentice Hall.

[2] Putnam, Lawrence H. and Ware Myers. 1996. Controlling Software Development. Los Alamitos: IEEE Computer Society Press.

[3] Ragland, Bryce. 1995. Measure, Metric, or Indicator: What's the Difference? Crosstalk, March: www.stsc.hill.af.mill

[4] Demarco, Tom. 1995. Why Does Software Cost So Much? New York: Dorset House Publishing.

[5] Wiegers, Karl. 1999. A Software Metrics Primer. Software Development Vol 1.

[6] Baumert, John H., and Mark S. McWhinney. 1992. Software Measures and the Capability Maturity Model, CMU SEI-92-TR-25, ESD-TR-92-24. Carnegie Mellon University.

[7] Florac, William A., Robert E. Park, and Anita D. Carleton. 1997. Practical Software Measurement: Measuring for Process Management and Improvement. CMU SEI-97-HB-003. Carnegie Mellon University.

[8] DeMarco, Tom. 1997. Risk Management: Project Management for Adults. Presentation, QSM Users Conference, McLean, VA.

[9] Demarco, Tom. 1982. Controlling Software Projects. New York: Yourdon Press.

[10] Putnam, Lawrence H. and Ware Myers. 1992. Measures for Excellence: Reliable Software, On Time, Within Budget. Englewood Cliffs: P T R Prentice-Hall.

[11] Mah, Michael. 2000. Sizing Up Your Promises and Expectations, IT Metrics Strategies, September 2000, vol. VI, no. 12. Cutter Information Corp.

Read more IT Performance Improvement

This article is an excerpt from:

The International Function Point Users Group (IFPUG), a nonprofit and member-governed organization, has become the recognized leader in promoting the effective management of application software development and maintenance activities. Edited by IFPUG's Management and Reporting Committee, The IFPUG Guide to IT and Software Measurement brings together 52 leading software measurement experts from 13 different countries who share their insights and expertise. Covering measurement programs, function points in measurement, new technologies, and metrics analysis, the text is useful for IT project managers, process improvement specialists, measurement professionals, and business professionals who need to interact with IT professionals and participate in IT decision-making. It includes coverage of cloud computing, agile development, quantitative project management, process improvement, measurement as a tool in accountability, project ROI measurement, metrics for the CIO, value stream mapping, and benchmarking.

About Jim Mayes

Jim Mayes is a Senior Consultant with CGI Technologies and Solutions, Inc. in Atlanta, GA. He has over 35 years of IT experience in software development and lifecycle management in the corporate environment at BellSouth (AT&T), and as an industry consultant with CGI, QSM Associates, and Mayes Consulting LLC. Over the past 16 years, he has been directly involved in providing software metrics support for project estimation, project tracking, predictive analysis, data analysis and trending, benchmarking, process improvement, outsourcing, productivity measurement, SLIM estimation and metrics tools, and Function Point analysis. During his 28 years at Bellsouth, he gained Telecom IT experience as a Systems Analyst, Applications Development and Maintenance Manager, Software Engineering Process Group (SEPG) Manager, and Software Estimating Metrics Group (SEMG) Manager. This experience included both in-house software development and outsourced development environments for various disciplines such as Billing, Network Engineering, Provisioning, and Customer Support. Jim has a Bachelor of Science degree from the Georgia Institute of Technology and has been a Certified Function Point Specialist (CFPS) for 15 years. He has had numerous articles published, is a frequent speaker at conferences, and has been a member of the following organizations: Atlanta SPIN, AQAA, IEEE Computer Society, and the International Function Point Users Group (IFPUG).