The most significant change (MSC) technique is a relatively new dialogue method for monitoring and evaluating complex interventions. Its main focus is on program improvement. It contributes to organisational change and targeting of services/activities towards the attainment of valued outcomes. It is highly participatory and has at its core the generation, analysis and use of stories. The technique is also known as ‘monitoring without indicators’ and ‘the story approach’.
Its main purpose is
to facilitate program improvement by focusing the direction of work towards explicitly valued directions and away from less valued directions. MSC can also make a contribution to summative evaluation through both its process and its outputs. The technique involves a form of continuous values inquiry whereby designated groups of stakeholders search for significant program outcomes and then deliberate on the value of these outcomes in a systematic and transparent manner. (Dart and Davies 2003:137)
The most significant change technique involves 10 steps (six or seven in earlier descriptions):
Starting and raising interest
Defining the domains of change
Defining the reporting period
Collecting significant change stories
Selecting the most significant of the stories
Feeding back the results of the selection process
Verification of stories
Quantification
Secondary analysis and meta-monitoring
Revising the system (Davies and Dart 2005).
A small number (three to five) of loosely defined domains within which the stories are located are selected—for example, ‘changes in the quality of life of the people affected by the program’. Stories are then generated by various stakeholders close to program implementation and knowledgeable about its outcomes within each domain using the question ‘During the last month, in your opinion, what was the most significant change that took place in the program?’. In one application to rural extension in Australia (Dart and Davies 2003), this was expanded by asking ‘What happened?’, ‘Why do you think this is a significant change?’ and ‘What difference has it made/will it make in the future?’. The respondents—the producers of the stories—allocate their stories to a domain. It is useful to include as one of the domains ‘lessons learned’ as this tends to reduce the bias towards stories illustrating positive outcomes of the program. The stories are typically one to two pages in length.
Those managing the most significant change process and the program managers closest to the program implementation level (these may be the same people) then select the most significant of the stories and pass them up the organisational hierarchy. At each level, they are read, discussed and the most significant are selected. Feedback is given to the lower levels, particularly as to the reasons why individual stories have been accepted for passing up, or set aside. The process reduces a large number of stories considered important at the local level to a smaller set that are most important at a higher level within the organisation. The refined set is discussed by senior management or funding bodies and taken into account in subsequent strategy development.
Various techniques exist for verifying the stories and attaching quantitative indicators to them in situations where this is feasible and useful.
As a method for research integration, the most significant change technique can be applied to monitor and evaluate research integration in a wide variety of complex interventions, particularly when information on outcomes and the value base of the interventions are important. It has been pointed out that
The types of programs that are not adequately catered for by orthodox approaches and [which] can gain considerable value from MSC include programs that are:
complex and produce diverse and emergent outcomes
large with numerous organisational layers
focused on social change
participatory in ethos
designed with repeated contact between field staff and participants
struggling with conventional monitoring systems
highly customised services to a small number of beneficiaries. (Davies and Dart 2005:12–13)
It is not as useful in situations in which the implementation processes and outcomes are straightforward and the causal paths connecting inputs and outcomes are clear. In these circumstances, more traditional quantitative indicators are often adequate.
One of the developers of the most significant change technique explains that ‘MSC can be conceived as a form of dynamic values inquiry whereby designated groups of stakeholders continuously search for significant program outcomes and then deliberate on the value of these outcomes. This process contributes to both program improvement and judgment’ (Dart and Davies 2003:140).
The most significant change technique is also useful at the following interfaces:
research and policy—for example, assisting senior decision makers to understand their programs’ outcomes and the values they reflect
research and professional practice—for example, assisting professionals responsible for designing and implementing complex interventions to understand how they have been implemented and with what outcomes
research and those affected by the research—for example, providing people’s own descriptions and analyses of program implementation, outcomes and attribution of causality.
The most significant change technique is a useful dialogue technique when integration is desired across powerful and weak players—for example, high-level program managers or program funders, on the one hand, and field staff and people intended to be program beneficiaries, on the other.
What was the context for the integration, what was the integration aiming to achieve and who was intended to benefit?
Target 10 is an extension dairy project that operates in Victoria under the aegis of the Department of Natural Resources and Environment (<http://dairyextension.com.au/>). Target 10 has been operating since 1992 and has diverse stakeholders, including farmers, university researchers, government extension officers and industry groups. In the late 1990s, the key stakeholders used the most significant change technique as a component of the project’s evaluation (Dart and Davies 2003).
Who did the integration and how was it undertaken?
The most significant change technique was applied in six steps:
pilot testing and familiarisation with the process
establishing the domains of change
establishing a reference group
establishing a method to collect and review the significant change stories
holding an annual round-table meeting for funders and others to review the stories
conducting a secondary analysis of all the stories generated by the project.
A simple form was developed to assist farmers and extension officers, in particular, to develop the stories. The Delphi technique was used with some 150 program stakeholders to develop the domains of change—namely, changes in on-farm practice, changes in profitability or productivity, changes in farmer decision-making skills and any other significant types of change. After the establishment of a reference group, the staff and others involved through program committees were asked to write stories on the forms provided for that purpose. Few were forthcoming so stories were also captured from verbal presentations at meetings and transcribed onto the forms.
Each regional committee reviewed the stories coming from the field and selected one from each of the four domains for consideration at the regular two-monthly or three-monthly state-level meetings. In addition, they documented why they had chosen those stories. The selection process involved the stories submitted being read out at regional meetings and voted on by the participants as to their usefulness. Since a great diversity of views occurred about which were most useful, the stories were discussed in detail with the aim of attaining group consensus, particularly about the value of the outcomes described.
There was a 12-month process of developing and discussing the stories: extension staff and other program people discussed with farmers their most significant change experiences and documented them, the regional committees discussed and selected the best stories and this process was repeated at the state level.
What was the outcome of the integration?
At the final stage, the funders and other key participants in the program met to share their reactions to the 24 stories that had filtered up to them from the regions. Those discussions revealed great diversity in key stakeholders’ reactions to the stories: the process demonstrated that they did not have a shared vision as to what Target 10 was intended to achieve for the dairy farmers or the dairy industry more broadly. After discussion, they were able to agree on one story that portrayed the types of outcomes that they all wished to see from the program and were happy about funding. This story was about a dairy farmer who attended a series of Target 10 courses, implemented what had been learned and, as a result, markedly increased the efficiency of use of farm inputs and gained confidence to move to managing a larger dairy farm.
What was being integrated?
At one level, the experiences of a diverse range of dairy farmers were being integrated. This occurred through them, and the extension officers who worked closely with them, documenting their stories and sharing them with their peers and people higher in the Target 10 hierarchy. More significantly, however, was the integrative work that occurred through discussing and judging the stories in terms of what they revealed about the most valuable outcomes of the program. This was a process of integration that occurred step by step at different levels in the program, until the winnowing process enabled the top level of managers to integrate their judgments to identify a single narrative that best captured the program’s intended outcomes.
This method integrated the experiences, and judgments of them, of a diverse range of dairy farmers. In this case, it was also instructive to examine how the method synthesised some of the key components of those judgments—namely, visions, values and interests.
The process revealed, at the regional and state levels, that the various stakeholders—including at the middle-management level and the top-level funding bodies—had widely diverging visions as to what Target 10 was established to achieve. They differed in the values they placed on the various program outcomes, and this had implications for the whole program’s direction. Through intense discussion of the stories, and voting on them, their separate visions as to what the program should attain were revealed, compared and tested. The process, grounded in the real-life stories of what people in the program considered to be the most significant changes, assisted the key stakeholders to examine their individual visions and their perceptions of what outcomes were most valued to produce a common, integrated vision for the program as a whole. Agreeing on the one particular story that encapsulated their shared perceptions was an important step.
Interests were also integrated through this application of the most significant change technique. The key stakeholders—funders, senior officers of the agriculture department, scientists and other university-based people—brought different interests to Target 10. The dialogue process of examining the stories of change helped make these interests explicit: able to be examined, compared, contrasted and weighed. The result was the attainment of a degree of consensus on desired program outcomes—in other words, a movement towards accommodating differing interests.
The core of this dialogue method is tapping the experiences and judgments of people involved at different levels of an intervention, with respect to its outcomes. The method seeks to identify the most significant changes that have occurred and to provide information on what are seen as the causal pathways that have produced the outcomes. In this manner, it integrates the evaluative judgments of various players, along with the underlying interests and values that feed into these judgments.
It applies democratic principles by giving voice to program participants at all levels, and providing feedback from the senior levels to lower levels about the program outcomes judged to be most worthy or desirable.
Our example of its use in the Target 10 dairy industry extension project illustrates researchers as one group of participants interacting through dialogue with others (for example, government funding body representatives, dairy farmers themselves, extension officers, and so on). It can operate, then, at a variety of interfaces between researchers and others, eliciting and judging the different perspectives of the various players.
While this is a strength as a component of a program evaluation, other evaluation techniques—not necessarily dialogic in nature—are also needed. These sometimes include quantitative performance indicators to supplement the most significant change technique’s narrative approach, and the use of the most significant change stories (along with other information sources) to develop the program logic. On the other hand, a strength of the method is that it can provide usable performance information in the absence of quantitative performance measures. This is particularly useful in complex, rapidly changing situations where the implementation of the intervention gives rise to emergent properties.
The Target 10 example is another illustration of how dialogue methods can be combined. In this case, the organisers used the Delphi technique with some 150 program stakeholders to clarify the domains of the program of change that would be explored in the most significant change project.
The most significant change technique was initially developed by Dr Rick Davies to contribute to the evaluation of a multifaceted social development project on Bangladesh (Davies 1996). It was developed further by Davies and Dr Jessica Dart (<www.clearhorizon.com.au/>). Dr Dart has applied the technique in a number of situations in Australia.
The method has its origins in evolutionary epistemology, ‘a branch of epistemology that applies the concepts of biological evolution to the growth of human knowledge’ (Wikipedia Contributors 2009). As Davies and Dart (2005:73) explain:
[I]n cultural evolution, the meaning of a given event…may be interpreted in a variety of ways by people. Some of those interpretations may have a better fit with the world view of the people concerned, and thus become more prevalent than other views held in the past. Within this newly dominant view, further variations of interpretations may emerge, and so on.
The MSC process…was an attempt to design a structured social process that embodied the three elements of the evolutionary algorithm: variation, selection and retention, reiterated through time.
Dart, J. and Davies, R. 2003, ‘A dialogical, story-based evaluation tool: the most significant change technique’, American Journal of Evaluation, vol. 24, no. 2, pp. 137–55.
Davies, R. 1996, An Evolutionary Approach to Facilitating Organisational Learning: An experiment by the Christian Commission for Development in Bangladesh, Centre for Development Studies, Swansea, Wales.
Davies, R. and Dart, J. 2005, The Most Significant Change (MSC) Technique: A guide to its use, Rick Davies and Jess Dart, Trumpington, Cambridge, United Kingdom, and Hastings, Victoria, Australia.
Clear Horizon, <http://www.clearhorizon.com.au/site/index.htm>
Most significant changes, <http://groups.yahoo.com/group/MostSignificantChanges/> (includes a repository of files illustrating the application of the technique in 10 countries)