Why the first sketch of the SIG software measurement instrument is still relevant and in use more than 10 years on
Reading time: 4 minutes
In 2007, three authors from our company, Ilja Heitlager, Tobias Kuipers, and Joost Visser, published the article, A Practical Model for Measuring Maintainability – a preliminary report .
They provided a first sketch of the software measurement instrument that SIG still applies to this day, in a mature and calibrated form. We recently published a reflection spanning 10 years of this model’s measurement outcomes in our 2019 Benchmark Report .
Why is this instrument still relevant today? To answer that, some context needs to be understood first. Put briefly, SIG advises organizations in making decisions about their software systems. Should they be phased out or improved? Are they future proof? What is the risk that maintenance cost will increase? Typically, our clients face decisions about large software assets, representing millions of euros or dollars in value.
Decision making in business is often tricky due to the intangibility of many of the important variables. In this context, Douglas W. Hubbard wrote the book on how to make the most of measurements and reach decisions effectively . Many of our clients are aware of this approach, or similar methods, so let’s see how the SIG maintainability model fits in such a framework.
Hubbard lays out 5 steps:
- Define a decision problem and the relevant variables.
- Determine what you know.
- Pick a variable and compute the value of additional information.
- Apply a relevant measurement instrument to high value variables.
- Make a decision and act on it.
In practice, steps 1-3 have already been exercised by clients with mature decision-making processes. Let’s say the situation is that they have a core software system written in Delphi, written 30 years ago, that handles all their e.g. faulty product returns in e.g. Belgium. They know that it costs them roughly 300K euros per year to operate and maintain, far below its business value. Now, they intend to use the system in Germany as well, increasing their customer base both in numbers and complexity (e.g. adding German). Can it do this? Will it be possible to adapt the system to meet the demands of the German market? Will maintenance costs go through the roof? For our client, the “highest information value variable,” in Hubbard’s terms, has now become the technical quality of the software, because they know it can help predict risks for future development and maintenance.
Step 4 is where the SIG maintainability model comes into play. In the seminal paper of 2007, the model definition was based on the ISO standard 9126 for software product quality . It splits the concept of maintainability into aspects like analyzability, modifiability, stability, and testability, all which are aspects to measure. The standard, however, does not tell you explicitly how to do this. The SIG model approaches this issue by defining several proxy measures on source code to indicate the standard’s “-ilities”, and connects them to create an overall judgement of quality. For instance, the complexity of code units indicates how difficult future modifications will be, and the level of coupling has a bearing on stability, and together with the other properties, and overall measurement of maintainability is constructed.
To be practical, however, several other requirements need to be fulfilled:
– technology independence: software systems typically contain several
programming languages and frameworks. Currently the model supports
up to 275 programming languages and technologies.
– measures should be straightforward to define and compute so that
it will be easy to measure automatically.
– measures should be easy to explain to all stakeholders, including
management and non-technical staff.
– it should be possible to do root cause analysis so that improvements
can be defined.
The business success of the SIG model can be explained in large part by staying close to these requirements. The model concretely consists of a software tool, maintained and further developed by a dedicated team, and a large database of measurements to track the model’s outcomes as it is being applied. Nowadays, the model is used to measure several hundred software systems each month, providing clients with the highest value information variable and enabling them to decide.
One key aspect, however, which wasn’t yet foreseen in 2007, deserves special attention: maintainability models need maintenance themselves! Each year, SIG performs a re-calibration of the measures included in the model, sometimes amounting to definition of new measures and the removal of obsolete ones. In the early years, a feature was added that allows for the benchmarking of software versus its industry peers, an approach that was approved by TUViT as a certified measurement for trusted product maintainability. This feature indeed became a cornerstone to the model’s overall success. In the next edition of this blog, we will revisit those fundamentals.
 This standard was later superseded by the more detailed standard ISO 25010, with
some adjustments to the aspects.
Dr. Magiel Bruntink is Head of Research at Software Improvement Group (SIG). He has a scientific background in program analysis and empirical study. The majority of his work consists of mixed industry-academic projects positioned within the software engineering domain. Dr. Bruntink holds a PhD in Computer Science from Delft University of Technology.