Methodology
Maturity model
Defining the need
The Test Bed project is a cross-institutional and cross-sector project. It required an evaluation research design which could assure that both the strengths and weaknesses of the impacts of new technologies were recorded and understood. This would allow lessons learnt from the target institutions to be disseminated to the wider community.
The approach
The approach taken was 'Maturity Modelling', drawn from organisational research where it has been shown that it is possible to score organisations to reflect the level of maturity at which they operate. Maturity Model (MM) frameworks allow a rich description of an intervention over time. This allows us to ask whether the designated institutions have the resources and structures to deliver effective educational experiences using ICT.
The first goal of such models is to describe and assess the complex environments in which innovations are to function. We began building the Test Bed model by designing six sub-models, five of which were integral to the Test Bed evaluation. These resulted in the following sub-models: curriculum, workforce, leadership and management, and communications and linkage maturity. A technological MM was added by the research team.
Once the sub-models were defined we established framework descriptions of maturity along key features. For each feature, a set of levels or stages was constructed with the clear assumption that any institution would evolve through the stages in order. For example, one feature of technological maturity was the presence of action plans for the renewal and maintenance of the system year-on-year and also after the Test Bed funding.
The levels provide a scoring rubric that allows the evaluation team to record the progress of the institution on that feature. The score for each feature is designed to provide, along with the other features within the sub-models, a snapshot of the institution's progress on each key dimension sub-model. See figure 1 below for an exemplar dimension from the technological Maturity Models.
Figure 1: The Technological Maturity Model
Dimension: Extent of connectivity
| Dimension levels | Score |
|---|---|
| Most computers are standalone. External link by low speed connection. | 1 |
| There is a networked central resource or some clusters which are networked with low speed connection shared across the network. | 2 |
| Most computers are networked with a shared broadband institutional access. | 3 |
| All systems (management and curricula) are networked together allowing the sharing of resources and data. | 4 |
| All systems (management and curricula) are networked together allowing the sharing of resources and data. Differential internal and external access to the network. Awareness of need for security. | 5 |
| An empty sixth box to be used in cases where other levels have been surpassed. | 6 |
Building an effective Maturity Model
Building an effective MM is an iterative process. Initial work on constructing the models was completed by the evaluation team following in-depth interviews with 30 expert stakeholders. Once the initial model was created, an expert seminar was organised to critically assess the model to date. Over 30 experts attended drawn from a range of stakeholders, including government agencies, leading research and development groups in ICT in education, members of the Test Bed project core team and ICT leaders in Local Education Authorities and schools.
The models were also field trialed by several institutions known to be active ICT users. The purpose of these field trials was to check the usability of the models and also, crucially, to verify the scaling of the levels. As this is a medium-term project we needed to know that institutions were not already sitting at level 4 or 5 (the highest level) on too many features. There needed to be room for institutional growth in each of the models.
Using and administering the models
The models were developed primarily as an analytical tool with some data capture facilities. A wide range of data are used to evidence the models gathered from interviews, questionnaires and other sources which are then triangulated and used to provide a complete assessment of an institution. The models read from left to right horizontally, with a series of five boxes used to describe performance on any one dimension. For each dimension, one box out of the five is highlighted as most accurately reflecting an institution's position. In cases where institutions fall between boxes, they are marked down to the lower position. Box one reflects the lowest levels of maturity, whilst box five represents the highest level. A sixth box is also provided to encourage those institutions who feel that they have surpassed the highest given level to provide evidence of how they have progressed further than the model allows. The models are completed on an annual basis within the Test Bed project as a means of allowing us to track change over time and enable us to develop predictive models based on performance outcomes.
What the models can and cannot tell us
Each of the dimensions within the model may be treated as six point Likert scale with positive scoring; that is an institution that satisfies all the attributes within a given level of a dimension is assigned an appropriate score. The assessed scores for each of the dimensions can then be combined to create an overall model score for the institution and descriptive data obtained that will allow simple cross institutional and within institutional comparisons to be made.
The models can also be used a predictive and exploratory tool, for example to identify how the features within and across models relate to one another or to seek what predicts (contributes to) educational outcomes, that is to test predictions of causality. Since the development of the models is an ongoing and dynamic process we are unable at this stage to provide a comprehensive analysis of the limitations of the models. Whilst we have noted some of the strengths of this approach here, we are currently monitoring the use of the models and will only be able to provide a full assessment of this tool at the end of the project.
Whilst the models are primarily intended to be a research and evaluation tool, they do also provide institutions with an opportunity to self assess and reflect on their past, present and future goals. This is in comparison to the more diagnostic tools that have been developed by Becta and the National College for School Leadership such as the Strategic Leadership in ICT matrices.
Future development
The Maturity Models are ongoing and dynamic; the models presented here should not be viewed as a definitive version.
The development of the Maturity Models was funded by Becta/DfES and copyright of the models remains with Jean Underwood and Gayle Dillon. Reproduction of the models, or any part of them must be sought from the authors directly.