3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

THE BALANCED SCORECARD (BSC) AS A SUPPORT TO

THE CMMI-DEV CONSTELLATION SCAMPI FOR THE

RECOGNITION OF THE MATURITY OF THE SOFTWARE

PROCESS

Oswaldo Alfaro Bernedo

National University Federico Villarreal, (Perú).

E-mail: oalfaro@unfv.edu.pe ORCID: https://orcid.org/0000-0002-9803-5986

Doris Esenarro

National University Federico Villarreal, (Perú).

E-mail: desenarro@unfv.edu.pe ORCID: https://orcid.org/0000-0002-7186-9614

Ciro Rodriguez

National University Mayor de San Marcos, (Perú).

E-mail: crodriguezro@unmsm.edu.pe ORCID: https://orcid.org/0000-0003-2112-1349

Maria Rene Alfaro

National University Federico Villarreal, (Perú).

E-mail: mralfaro@unfv.edu.pe ORCID: https://orcid.org/0000-0003-4601-6748

Recepción:

01/09/2020

Aceptación:

07/10/2020

Publicación:

13/11/2020

Citación sugerida Suggested citation

Alfaro, O., Esenarro, D., Rodriguez, C., y Rene, M. (2020). The balanced scorecard (BSC) as a support

to the CMMI-DEV constellation SCAMPI for the recognition of the maturity of the software process.

3C Tecnología. Glosas de innovación aplicadas a la pyme. Edición Especial, Noviembre 2020, 33-49. https://doi.

org/10.17993/3ctecno.2020.specialissue6.33-49

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

ABSTRACT

This research tries to establish the degree of inuence exerted by the design and use of

a Balanced Scorecard (BSC), as a support to the Standard CMMI Appraisal Method for

Process Improvement SCAMPI of the CMMI-DEV constellation, in recognition of the

level of maturity of the software process, due to the eciency and eectiveness provided

by its application in practical life. In addition, it uses the Systemic Approach to conceive

the problem comprehensively, under a holistic perspective, covering the relationships of

each element within the system and its relationships with external agents; the applied

methodology consists in the collection, tabulation, and analysis of the antecedents that

have been obtained for its direct validation during development, that is, the management of

cause-eect variables, where the independent or experimental variable is of interest to the

researcher because the variable that is hypothesized (X), is one of the causes that produce

the supposed eect. As a result of the eectiveness of the SCAMPI evaluation process,

going from 72.7% to 92.7%; This is equivalent to a 26.4% improvement in performance,

increasing from 10.9 correct evaluations to 13.9 for conducting such evaluations.

KEYWORDS

Balanced scorecard, Support, Constellation, SCAMPI, Maturity of the software.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

1. INTRODUCTION

In its research to help organizations develop and maintain quality products and services,

the Software Engineering Institute (SEI) has found several dimensions that an organization

can focus on to improve its business. Also, it illustrates the three fundamental dimensions

on which organizations usually focus: people, procedures and methods, and tools and

equipment. This virtuous circle involving these components is key to achieving a quality

product (2GC Active Management, 2019).

Software, understood in its duality of process and product, is inherently problematic;

such complexity is manifested in many aspects, the most relevant being: the fact of

being intangible, diculty in delimiting the scope or domain, diversity of languages to

express it in semantic and notational form, frequent changes in business rules that attempt

against its functional validity and, consequently, the predominance of accelerated product

obsolescence, etc.

The CMMI has two structures of Representations, the same ones that give rise to two dierent

types of evaluations. Figure 1 illustrates the structure of the Representations Continuous.

The dierences between the structures of Figure 1 and the Staged representation are subtle

but signicant. The staged picture uses maturity levels to characterize the overall state of the

organization’s processes concerning the model. In contrast, the continuous representation

uses capability levels to characterize the state of the organization’s processes concerning

an individual process area. This dimension (the capability/maturity dimension) of CMMI

is used for benchmarking and assessment activities, as well as to guide an organization’s

improvement eorts (Álvarez, 2016).

Figure 1. Continuous representation.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

As show in Figure 1, capacity levels refer to the achievement of process improvement in

an organization in individual process areas. These levels are a means of incrementally

improving the processes that correspond to a given process area.

Figure 2. The four levels of capability are numbered from 0 to 3.

Maturity levels, on the other hand, refer to the achievement of process improvement in an

organization in multiple process areas. These levels are a means of improving processes

for a given set of process areas (i.e., maturity level). As illustrated in the Figure 2, the ve

maturity levels are numbered from 1 to 5.

There is a dependency or causal relationship between Strategic Planning and the BSC, i.e.,

the formulation of the strategic plan is the primary input of the latter, so when designing and

operating the BSC, this condition should not be lost sight of. In this regard, the structure of

the strategic plan under the BSC approach has the following structure (De Flander, 2018):

a) Institutional: where the values, role, and scope of the institution are stated; the

philosophy of the institution; the expectations of related agents or interest groups.

b) Strategic Management: a segment in which the Mission, Vision, and General

Strategic Objectives and Specic Strategic Objectives for the established time

horizon are formally expressed.

c) Related strategies: the set of actions to be deployed to achieve the formulated

objectives.

d) Management indicators: sets of metrics to measure the institution’s functional

performance.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

The following table shows in a structured way the perspectives, general objectives, specic

objectives, and corresponding indicators that are part of a corporate strategic plan.

The BSC is congured in four hierarchically aligned perspectives under a causal alignment.

This structure responds to the logic of the so-called strategic map, which in turn is constituted

by the causal relationship of the strategic objectives derived from the institutional strategic

plan. The perspectives considered are nancial, commercial, internal processes, and

learning; each one of them, in turn, is made up of strategic objectives, indicators, and goals.

The BSC is used as a strategic management model, a communication tool, and, in its best

implementations, an organizational change tool. This management model is based on a

basic principle stated as “you can only manage what you can measure” (Lamé, Jouini &

Stal-Le Cardinal, 2019).

The BSC is a tool that translates strategy into action. BSC is a performance planning and

management model that places strategy at the center of the process. The applicability of the

BSC has no limits of any nature because being a comprehensive management measurement

system can be adapted to any organization, for tangible or intangible processes, for large or

small organizations, public or private.

There is a wide variety of software tools to implement and automate the operation of this

important strategic management tool. However, there is one that, taking the institutional

strategic plan as a reference base, allows move towards management by indicators. This

is the case of Sixtina BSC, whose exibility, robustness, and user-friendliness allow for

relatively easy incorporation of the designed strategic matrix into the strategic plan.

The software organizes the management indicators in a structured manner. The highest

hierarchy is General Compliance, then Critical Factor, then Indicator, and nally, Data.

This structure is causal, i.e., the sequencing starts from the Learning perspective, then

Internal processes, Customers, and Financial. The person responsible for designing the

BSC assigns the weighted weights to each of the management indicators.

The BSC dashboard shows in the second column, the performance status of each

indicator in the chromatic form: green means satisfactory, yellow is equivalent to alert,

and red indicates decient. The next column shows the actual value of each indicator; this

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

information is extracted from the database of the institution’s information system; then it

shows the measure of the expression of each indicator, corresponding points for general

compliance, critical factor, and indicator; instead, for data, it corresponds to the measure

corresponding to its nature. The columns follow it for the date, operation (maximize,

minimize or stabilize), perspectives already indicated, the person responsible for the results

achieved by each indicator; the weighted weight (%) in determining the formula for the

compound indicators, the trend, the target value dened, the deviation observed between

the actual value and the target, the origin of the data (formula, manual, database, Excel)

and nally the frequency of recording the information (daily, weekly, monthly or annual)

(Catchpole et al., 2017).

Figure 3. BSCS strategic Map.

Figure 3 shows the strategic map corresponding to the BSC scorecard, whose purpose

is to articulate in a systemic, causal, hierarchical, transversal, and weighted way all the

management indicators. This view is very important for the stakeholders because, in one

single view, the interaction of the indicator system can be understood.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

Figure 4. Homogenization of heterogeneous indicators.

The calculation and color representation of each of the indicators in the BSC control panel

is shown in Figure 4. The sequence is bottom-up, i.e., based on data of a heterogeneous

nature, quantity, orientation, and weighting, using the corresponding formula, the resulting

indicators of a homogeneous nature are determined in units of measurement (points)

and orientation (maximize). To determine the values of the critical factors and general

compliance, the same procedure is followed as above (Institute and Faculty of Actuaries,

2019).

Table 1. Parameterization and import of information in BSC Sixtina.

SERIES DETAIL DATA

GENERAL COMPLIANCE,

CRITICAL FACTOR OR

INDICATOR

REAL

Operation Maximize Minimize Stabilize Maximize

Capture excel import - databases

formula (rank/average index

children)

Magnitude original size score (0 to 10)

ALARM

Operation Maximize Minimize Stabilize Maximize

Capture excel - manual excel – manual

Magnitude original size score (0 to 10)

OBJECTIVE

Operation Maximize Minimize Stabilize Maximize

Capture excel - manual excel - manual

Magnitude original size score (0 to 10)

Maximize

Fees

0 Max

Deciency Alarm Target

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

Minimize

Fees

0 Max

Target Alarm Deciency

Stabilize

Fees

0 Max

Default default Lower alarm Target Higher alarm

Deciency due

to excess

Table 1 is a summary of the information processing for the construction of the BSC control

panel content. For each series (actual, alarm or target), depending on the type of indicator,

operations are performed (maximize, minimize or stabilize) as appropriate; information

capture is dened (import from Excel or database, Excel or manual); also, the magnitude

(points or original magnitude of the data) is established. Likewise, the dimensions for the

three operation options of the indicators and their corresponding color are dened.

Since the CMMI, SCAMPI, and BSC models have been characterized integrally, by the

research objective, it has been possible to establish an analogy between the SCAMPI

evaluation process and the BSC control panel. As shown in Table 2, it is possible to equate

the hierarchical levels of maturity assessment of SCAMPI and the performance of the

management indicators of the Sixtina BSC scorecard.

Table 2. Equivalences between SCAMPI model and Sistine BSC.

SCAMPI Maturity Assessment Model BSC control panel model

Maturity level General compliance

Process area Critical factor

Goals Indicator

Internship Fact

The convenience of applying the BSC as a tool to facilitate the SCAMPI evaluation of the

CMMI-DEV constellation, oriented to the recognition of the software process maturity

level, is supported by the following reasons.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

2. METHOD

This research work is based on the Scientic Research Method due to the eciency and

eectiveness provided by its application in practical life. Furthermore, it uses the Systemic

Approach to conceive the problem integrally, under a holistic perspective, covering the

relations of each element within the system and its relations with external agents.

Consequently, the project is following a proven method of collecting, tabulating, and

analyzing the background that has been obtained and proving its validity directly in the

eld in which the research fact is being presented (Borchert, Schirmeier & Spinczyk, 2017).

The design of the Experimental Research has been selected, that is to say, the handling of

variables of the cause-eect type, where the independent or experimental variable is of

interest to the researcher because the variable that is hypothesized (X) is one of the causes

that produce the supposed eect (Denicolai & Previtali, 2020).

The design is quasi experimental, allows the use of pre-tests and post-tests to analyze the

evolution of the eect of the “pilot” implementation before and after the experimental

treatment so that the subtype of Research design used is: “Post-test design and control

group”, whose general model is shown below:

RG1 X O1

RG2 – O2

Where:

RG1 = Experimental Group (formed randomly by software process evaluations for

recognition of maturity levels using the BSC)

RG2 = Control group (formed randomly by software process evaluations for recognition of

maturity levels in the absence of the stimulus)

X = Stimulation (El Balanced ScoreCard)

– = Absence of Stimulus

O1, O2 = Measurement of maturity level indicators

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

This design includes two groups, one receiving the experimental treatment (Experimental

Group) and the other not (Control Group). That is, the manipulation of the independent

variable reaches only two levels (presence and absence).

• Materials:

◦ CMMI-DEV Constellation

◦ SCAMPI model

◦ Software development processes

◦ Balanced Score Card Software

• Procedure:

1. Business processes that have been automated through software development

processes are identied.

2. The development constellation of the CMMI model is characterized.

3. Requests for evaluation of CMMI-DEV maturity levels are received.

4. The evaluation team is constituted, designating the leader.

5. The Balanced Score Card logic model applied to the SCAMPI model is built.

6. An evaluation or audit is carried out to recognize the level of maturity required.

7. The results of the evaluations are compared

8. The results are interpreted.

9. The hypotheses are veried.

3. RESULTS

By the objective of the research, the hypothesis, the design, and the procedure that has been

applied in this investigation, initially, a company has been characterized as a prototype,

which constitutes the control group to which the BSC was not applied in the SCAMPI

evaluation process, for the recognition of the level of maturity of the software process

implemented in that organization, that is, maintaining its original conditions of structure

and operation. Then, the prototype was made for a SCAMPI evaluation of maturity level

2, using the BSC, whose software corresponds to Sixtina BSC. Specically, the following

procedure has been followed. (Brocklesby, 2016).

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

1. The MWC-SCAMPI metamodel logic has been constructed to understand the scope

and reach of the entities included in this integral process. This model has been designed

with a view to the subsequent generation of the physical database (PIIDB), necessary to

import the information from the Sistine BSC.

Figure 5. Logical metamodel CMMI SCAMPI.

Figure 5 shows all the entities, relationships, and attributes that are most prominent in this

metamodel. Recursive relationships and sub-entities are also identied.

1) Advanced engineering has been carried out for the generation of the database

(PIIDB) corresponding to the entity-relationship model designed in the previous

point.

2) The equivalence between the components of the CMMI-SCAMPI model and the

BSC Sixtina control panel has been established, as shown in the following table. It

shows the relevance of establishing similar and, therefore, comparable hierarchies at

the corresponding levels.

This analogy constitutes the full support to the proposal of this research project to use the

Balanced Scorecard (BSC), as a support to the SCAMPI of the CMMI-DEV constellation,

in recognition of the maturity level of the software process.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

1) Using the software BSC Sixtina under the web environment, the SCAMPI evaluation

model of the CMMI development constellation (DEV) has been characterized.

2) In the same BSC board, the evaluation corresponding to maturity level 2 for the

Process and Product Quality Assurance (PPAQ) process area, its four goals, as well

as its 15 practices, has been carried out. I am applying for the SCAMPI protocol.

With the information recorded in the database, the information has been contrasted,

and the software has produced the color results observed, and, applying the decision

rule of the SCAMPI protocol, the requested maturity level has not been reached as

(Brocklesby, 2016) in an applied situation.

3) By way of simulation, the results of the research eciency indicator have been

measured. This metric expresses the time spent in carrying out a SCAMPI assessment,

for maturity level 2, both for the control group (in the conventional form) and for the

experimental group (using the BSC)

Table 3. Efciency for the Control Group.

EVALUATION

DURATION (DAYS)

EFFICIENCY

ESTIMATE REAL

1 12 15 80.0%

2 10 14 71.4%

3 9 12 75.0%

4 10 15 66.7%

5 12 14 85.7%

6 10 15 66.7%

7 8 12 66.7%

8 14 18 77.8%

9 12 16 75.0%

10 14 16 87.5%

AVERAGE

11.10

14.70

75.2%

DEVIATION 0.08

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

Table 4. Efciency for the Experimental Group.

EVALUATION

DURATION (DAYS)

EFFICIENCY

ESTIMATE REAL

1 12 12 100.0%

2 10 11 90.9%

3 9 9 100.0%

4 10 15 66.7%

5 12 11 109.1%

6 10 11 90.9%

7 8 8 100.0%

8 14 15 93.3%

9 12 13 92.3%

10 14 13 107.7%

AVERAGE 11.10 11.80 95.1%

DEVIATION 0.12

4) There is a substantial increase in the eciency of the SCAMPI evaluation process,

from 75.2% to 95.1%; this is equivalent to a 26.4% improvement in performance,

decreasing from 14.7 days to 11.8 days for such evaluation.

Table 5. Efciency for the Control Group.

EVALUATION

PRACTICES

EFFICACY

EVALUATED ACCEPTED

1 15 11 73.3%

2 15 10 66.7%

3 15 9 60.0%

4 15 13 86.7%

5 15 11 73.3%

6 15 11 73.3%

7 15 9 60.0%

8 15 12 80.0%

9 15 12 80.0%

10 15 11 73.3%

AVERAGE

15.00

10.90

72.7%

DEVIATION 0.09

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

5) Also, by way of simulation, the results of the research eectiveness indicator have

been measured. This metric expresses the degree of accuracy used in carrying

out a SCAMPI assessment, for maturity level 2, for both the control group (in a

conventional form) and the experimental group (using the BSC).

Table 6. Efciency for the Experimental Group.

EVALUATION

PRACTICES

EFFICACY

EVALUATED ACCEPTED

1 15 14 93.3%

2 15 14 93.3%

3 15 15 100.0%

4 15 13 86.7%

5 15 14 93.3%

6 15 13 86.7%

7 15 13 86.7%

8 15 14 93.3%

9 15 14 93.3%

10 15 15 100.0%

AVERAGE 15.00 13.90 92.7%

DEVIATION 0.05

There was a substantial increase in the eectiveness of the SCAMPI evaluation process

as we can compare in tables 5 and 6, from 72.7% to 92.7%; this is equivalent to a 26.4%

improvement in performance, increasing from 10.9 correct evaluations to 13.9 for

conducting such evaluations.

4. CONCLUSIONS

The BSC is a powerful tool insofar as it is applied to follow up on performance reports.

Using the software BSC Sixtina under the web environment, the SCAMPI evaluation

model of the CMMI development constellation (DEV) has been characterized.

As a result of the eectiveness of the SCAMPI evaluation process, going from 72.7% to

92.7%; This is equivalent to a 26.4% improvement in performance, increasing from 10.9

correct evaluations to 13.9 for conducting such evaluations.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

REFERENCES

2GC Active Management. (2019). Balanced Scorecard Usage Survey 2018. Retrieved on

August 30, 2019, from: https://2gc.eu/media/resource_les_survey_reports/2018_

Survey_Document_10_Final-compressed.pdf

Álvarez, I. (2016). FORBES. Retrieved on August 12, 2019, from: https://forbes.es/

listas/3865/las-diez-mejores-rmas-de-consultoria/

Borchert, C., Schirmeier, H., & Spinczyk, O. (2017). Generic soft-error detection and

correction for concurrent data structures. IEEE Transactions on Dependable and Secure

Computing, 14(1),22–36. https://ieeexplore.ieee.org/document/7097670

Brocklesby, J. (2016). The what, the why, and the how of operational behavioral research-

An invitation to potential skeptics. European Journal of Operational Research, 249(3), 796–

805. https://doi.org/10.1016/j.ejor.2015.09.034

Catchpole, K., Neyens, D.M., Abernathy, J., Allison, D., Joseph, A., & Reeves, S.T.

(2017). Framework for direct observation of performance and safety in healthcare. BMJ

Quality & Safety, 26(12), 1015–1021. https://doi.org/10.1136/bmjqs-2016-006407

De Flander, J. (2018). Strategy Execution - The Denitive Guide - Jeroen De Flander. Retrieved on

August 24, 2019, from: https://jeroen-de-ander.com/strategy-execution/

Denicolai, S., & Previtali, P. (2020) Precision Medicine: Implications for value chains

and business models in life sciences. Technological Forecasting and Social Change, 151, page

119767. https://doi.org/10.1016/j.techfore.2019.119767

Institute and Faculty of Actuaries. (2019, October 7). RSS and IFoA publish new ethical

guidance in data science. https://www.actuaries.org.uk/news-and-insights/news/rss-

and-ifoa-publish-new-ethical-guidance-data-science

Lamé, G., Jouini, O., & Stal-Le Cardinal, J. (2019) Combining Soft Systems

Methodology, ethnographic observation, and discrete-event simulation: A case study

in cancer care. Journal of the Operational Research Society, 71(10). https://doi.org/10.10

80/01605682.2019.1610339

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49

Ubalde, R., Rodriguez, C., Petrlik, I., Esenarro, D., Lezama, P., & Sotomayor, J.

(2020). Quality model for Peruvian microenterprises of a software product Factory.

Test Engineering and Management, 83, 13434.

3C Tecnología. Glosas de innovación aplicadas a la pyme. ISSN: 2254 – 4143 Edición Especial Special Issue Noviembre 2020

https://doi.org/10.17993/3ctecno.2020.specialissue6.33-49