For more than 50 years, Auerbach Publications has been printing cutting-edge books on all topics IT.

Read archived articles or become a new subscriber to IT Today, a free newsletter.

This free newsetter offers strategies and insight to managers and hackers alike. Become a new subscriber today.


Partners




Contact

Interested in submitting an article? Want to comment about an article?

Contact John Wyzalek editor of IT Performance Improvement.

 

A Framework for Measuring the Value of Software Development

Michael D. Harris and Thomas M. Cagley, Jr.

This article takes a broad view of "software development" to include all activities which result in a change to a piece of software for which the "users" are separate from the "developers." Hence, "software development" includes new development, enhancements, maintenance, architectural changes bug fixes and any of those other terms about which there are arguments regarding precise definitions.

The value of software development is an assessment, part objective and part subjective, of the user-perceived outputs from a process that converts inputs into a solution that the users (most often a business) experience. In a perfect world, software development value would be evaluated by a perfect market with perfect information to decide who does the work of delivering functionality with the cost of inputs and the efficiency of transformation measured against sales to (or use by) business users. Even if this were the case, there is no question that, like beauty, value will always be "in the eye of the beholder."

Value is perceived by those receiving the service so it tends to be specific to the project or service outputs. For the purposes of this article, if we are to measure value then it must be predictable and, in turn, there must be an assumption that there is a relationship between how the software is created and the value perceived. This article takes a systems view of the software development (see Figure 1) such that value is defined as a synthesis of how the software development team manages inputs and raw materials, the efficiency of the processes used in the transformation process (where value is typically added) and finally whether the output meets the users need and is fit for use.

Figure 1. Software Development Value System

A Systems View of the Value of Software Development

Measuring internal value requires gathering data about the inputs and raw materials, the processes used in transformation and the user’s perception of the output. A traditional view of a basic system is shown in Figure 1. Figure 1 is the framework which support the value measurement described in this article.

Any framework for measuring software development value must accommodate objective and subjective metrics. An important starting point is to develop a simple but broadly acceptable set of software development goals that are abstract enough to meet all varieties of software development yet substantive enough to allow metrics to be built using a simple goal-question-metric approach. Three words—effective, efficient and economical—taken together meet the requirement for a simple but embracing set of value goals for software development.

This article asserts that, while there may be arguments for other generic goals to be added, these three provide a broadly acceptable starting point.

How can our goals be mapped to our systems framework? There is room for discussion of more complex interpretations here but this article proposes a simple mapping when considering value metrics:

  • Effectiveness correlates very highly with system outputs
  • Efficiency correlates well with transformation and making best use of inputs
  • Economy relates to choosing the least costly inputs that will produce the desired results.

Hence, by choosing the right metrics for inputs, transformations and outputs and combining them in the right way, we should be able to produce a software development value metric. This approach has been applied in other areas. The book The Business Value of IT1 contains a number of approaches to combine weighted subjective and objective value metrics.

The approach in this article extends previous ideas by considering the prioritization of value multiplying activities in the transformation stage and the value of the inputs as potential limiters on the value of the outputs.

Identifying the Right Software Development Value Indicators for an Organization

While there are some well established value indicators associated with value for money, customer satisfaction and so on, it is important to try to capture some of the strategic and cultural values that are unique to every organization and are a mix of productive and unproductive judgements about where the organization has been before and where it thinks it needs to go in the future.

To elicit these values, we have used a questionnaire that is presented to the stakeholders, inside and outside the software development group, and asks them about successes/failures in the past year, how they personally measure these and what they would like to see change in one year’s time. This approach is very similar to a classic Goal-Question-Metric approach except that it looks backwards as well as forwards in time.

The answers to these questions are analyzed to extract value statements or judgements. The value statements are then categorized as being more relevant to input (economical use of resources), output (effective results) or transformation (efficiency of process). Metrics are developed for these then they are then mixed with a standard set of value metrics and presented back to the stakeholders for prioritization on the understanding that they cannot keep all of them! By way of example, the following tables 1 through 6 demonstrate the results of this process for one particular organization (note that different aggregations of the metrics have been suggested for reporting purposes):

Table 1. Possible Input (Economic) Metrics

In considering the input metrics when reviewing the stakeholder’s detailed comments in their questionnaire responses, it is common to see that stakeholders from within the software development organization pointing to the inadequacy of their inputs as being a major cause of any limitations in outputs.

For both transformation and output metrics at this organization, we are using function points (FP’s) as the solution for software size. Other software size metrics may be equally valid.

Table 2. Possible Transformation (Efficiency) Metrics
   1Function Point or other size measurement
   2DCUT = Development + Coding+ Unit Test

In considering input and transformation metrics, this organization needed to include measurement of outsourced software development vendors. All of the above metrics were applied but two additional metrics were felt to be necessary (see Table 3).

Table 3. Possible Additional Transformation (Efficiency) Metrics for Subcontractors

When considering output metrics, it has been necessary to differentiate between the value of a particular project (micro-level—Table 4) and the value of the software development group (macro-level—Table 5).

Table 4. Possible Output (Effectiveness) Metrics for Projects (Micro Level)




Table 5. Possible Output (Effectiveness) Metrics for Software Development (Macro Level)

Balancing the Software Development Portfolio

You will see that we have defined the concepts of "Value Neutral", "Value Add" and "Value Multiplier" for our software development transformation process. These refer to the transformation value of individual projects and are intended to help with the project prioritization process that is part of software development governance. Especially at budget planning time!

The following broad definitions are proposed:

  • A "Value neutral" project does not change perceived value of product or application e.g. some architectural changes, bug fixes, make work.
  • A "Value add" project clearly adds to perceived value but probably only for a subset of all the possible beneficiaries. Examples might be interfaces to 3rd party software, incremental functionality to existing component, new local capability.
  • A "Value multiplier" project increases value for all possible beneficiaries. Examples might be new front end, new component to enable new clients/market, new general capability, new language support.

Obviously, in an ideal world, all projects in a software development portfolio would be value multipliers. However, that is not realistic and could even be a high risk allocation of resources. It is also true that different stakeholders might have different views on how to categorize a particular project. For example, the architects might argue that an architectural change is not value neutral. This article takes the perspective that the end user is "king" in such stakeholder discussions but we accept that might not always be true. In practice, a typical portfolio will contain a mix of the three types of project.

Obviously, in an ideal world, all projects in a software development portfolio would be value multipliers. However, that is not realistic and could even be a high risk allocation of resources. It is also true that different stakeholders might have different views on how to categorize a particular project. For example, the architects might argue that an architectural change is not value neutral. This article takes the perspective that the end user is "king" in such stakeholder discussions but we accept that might not always be true. In practice, a typical portfolio will contain a mix of the three types of project.

So what is the right balance? It depends on the risk tolerance of the business, the available discretionary funds and the degree to which the business can drive the software development group to be flexible. However, one "rule of thumb" is to apportion your software development dollars somewhere between $1 - $2 - $1 and $1 - $2 - $3 for value neutral - value add - value multiplier. In other words, always try to spend twice as much on value add as value neutral then try to spend one to three times the value neutral amount of value multipliers.

A Software Development Value Index

Finally, we have developed the concept of a balanced Software Development Value Index (SDVI):

   ( x * I ) + ( y * T ) + ( z * O )
,
   where Priority Weights x + y + z = 100%

The answer to the annual or quarterly question, "Is the Software Development organization delivering value?" is "Yes" if the SDVI exceeds the annual target.

For example, for Year 20xx, SDVI priorities (or weights) are chosen by the Software Development (or IT) Governance Committee based on the strategic priorities for the year to be:

  • Effectiveness: z = 50 %
  • Efficiency: y = 20 %
  • Economic: x = 3 %

Let’s choose a target for value delivered of SDVI > 85%.

The Software Development Input, Output and Transformation scores are generated by combining and normalizing the scores for a set of metrics measured under each heading (e.g., input metrics might include budget and hours).

Let’s assume that for our example software development group in Q1, 20xx the normalized metrics are:

  • Effectiveness, O = 87%
  • Efficiency, T = 90%
  • Economic, I = 80%

Then our SDVI = (0.30 * 0.80) + (0.20 * 0.90)+ (0.50 * 0.87) = 0.855

At 85.5%, we can conclude that this software development group delivered value in the first quarter of 20xx!

Hence, we can present our final example value metrics in Table 6:

Table 6. Possible Overall Value Metrics

Conclusion

This article has presented some ideas for the simplification of metrics to allow the business to judge the value being delivered by IT. In compiling the final simple value metric, the business has the opportunity and responsibility of making explicit its value priorities for the year. This means prioritizing effectiveness, economy and efficiency. From experience, working with a real client using these metrics, this opportunity and responsibility may not be easily taken up by the business. However, the conversations about the definition of the proposed metrics and the resulting values, in and of themselves, increase the value of IT to the business.♦

Note
1Chapter 2 in
The Business Value of IT by Michael Harris, David Herron and Stasia Iwanicki.

Read more IT Process Improvement

About the Authors

Michael D. Harris is the president and owner of the David Consulting Group. Mr. Harris brings nearly 25 years of broad management experience in the computing field including periods in R&D, development, production, business and academia. The past ten years have been spent in the financial services technology area.

Thomas M. Cagley, Jr. leads the Software Measurement Consulting Practices, as well as the Software Process Improvement and IT Performance Improvement, at the David Consulting Group. The host of the Software Process and Measurement Cast, he has more than 20 years experience in the software industry in which he has been a consultant since 1997.

Also by Michael D. Harris

The Business Value of IT

An excellent reference for the CIO and for the line manager seeking to engage the business with the transparency into the investment and cost equation they demand to justify the cost of IT. —From the foreword by, Mike Antico, CTO, Wolters Kluwer, New York, USA