An Overview
Laura Linnan and Allan Steckler
Improving and sustaining successful public health interventions relies increasingly on the ability to identify the key components of an intervention that are effective, to identify for whom the intervention is effective, and to identify under what conditions the intervention is effective. The published literature includes a plethora of reports about interventions that have successful outcomes. A limited number of studies, however, disentangle the factors that ensure successful outcomes, characterize the failure to achieve success, or attempt to document the steps involved in achieving successful implementation of an intervention. To truly advance science and our understanding of applied interventions, we must learn a great deal more about public health intervention successes and failures. Process evaluation efforts can assist in making these discoveries.
In the last decade, the literature on process evaluation related to public health interventions has grown considerably. In the late 1990s and in early 2000, there has been an explosion in the number of published studies that include extensive process evaluation components. There are several plausible explanations for this noticeable increase in the use of process evaluation. Social and behavioral interventions have become increasingly complex, making it important for researchers to know the extent to which all intervention components are actually implemented. This complexity stems from the fact that projects are often implemented at multiple locations, so that process evaluation becomes essential for ensuring that planned interventions are carried out equally at all sites. Complexity also results when interventions are implemented at multiple levels and with multiple audiences. Many contemporary studies use ecological approaches that intervene among individual, dyadic, group, organization, community, and population levels (McLeroy, Bibeau, Steckler, and Glanz, 1988). Accounting for the extent to which the intervention occurs at each level of influence (and among different subgroups) becomes increasingly complicated. As we attempt to eliminate inequalities in income and education level, we can, with thoughtful process evaluation efforts, obtain important clues about the influence of interventions among selected subgroups.
Another plausible explanation for why process evaluation efforts have proliferated is that we are looking for ways to explain why certain results were achieved. Specifically, when interventions lead to significant outcomes, it is important to understand which components of the intervention contributed to the success. Similarly, when large sums of taxpayer money are allocated to conduct multilevel community trials and the primary outcomes are insignificant (or the magnitude of the effect is small), there is an increased demand among researchers, funders, and members of the general public to understand why these interventions did not produce their intended effect (Fisher, 1995; Susser, 1995). Thus, process evaluation can help explain positive, modest, and insignificant results.
Process evaluation also provides important links to understanding and improving theory-informed interventions. Because more programs are developing theory-informed interventions, there is a greater need to understand which theoretical constructs make a difference (Glanz, Lewis, and Rimer, 1997). Process evaluation efforts can provide a link between theoretical constructs thought to be essential for intervention success and the final study outcomes. Understanding the mechanisms for how and why these constructs produce successful change (or fail to produce change) is key to refining theory and improving intervention effectiveness. Understanding under what conditions certain mediators are more (or less) influential in the achievement of certain study outcomes will inform the next and future generations of theory-informed interventions (Baranowski, Anderson, and Carmack, 1998; Baranowski and others, 1997).
Process evaluation efforts also help us understand the relationships among selected intervention or program components. In a comprehensive multimethod intervention, one might offer group classes, self-help programs, individual counseling sessions, and print materials. Process evaluation can help disentangle the effects of each method singly, and it can clarify the possible interactions that can occur to produce a synergistic effect. Few studies have attempted to reach into the "black box" of intervention effectiveness to explore which program components have been more or less effective, yet doing so is a powerful option within a comprehensive process evaluation effort (Harachi and others, 1999).
Assessing the quality and accuracy of the intervention delivered to program participants can also be achieved with process evaluation, which is another reason why these studies have proliferated. Increasingly, funders and program participants at all levels want assurances that the intervention being delivered is of a high quality and is highly accurate. In addition, attention to the cost-effectiveness of interventions is an increasingly important component of intervention planning and evaluation. Process evaluation efforts can assist with each of these requirements.
Finally, we contend that another reason for the rise in the use of process evaluation is the increasing recognition of the value of qualitative research methods (National Institutes of Health, 2001). Process evaluation frequently uses both quantitative and qualitative methods. Qualitative methods such as field site visits, structured observations of interventions, and open-ended interviews are often employed in conducting process evaluations. As the value attached to qualitative methods has increased, opportunities for publishing these findings and learning from their results have increased as well. Integrating different methods-such as qualitative and quantitative methods-yields rich detail about study outcomes that neither method could achieve alone (Tashakkori and Teddlie, 1998; Steckler and others, 1992).
Taken together, the recent increase in published literature on process evaluation results reflects the complexity of public health interventions today, and it reflects the many ways in which thoughtful, comprehensive process evaluation efforts can shed light on questions that will inform improvements in theory, intervention design, and methods in the future.
The remainder of this chapter presents a brief overview of the history of process evaluation as it relates to public health. After identifying current gaps in knowledge and clarifying barriers to effective process evaluation, the chapter presents definitions of key process evaluation terms and describes an approach that program planners, researchers, and evaluators may find helpful when planning and conducting process evaluation efforts.
Brief History and Review
Process evaluation is not a new concept. As early as during the 1960s, an explanation of process evaluation appeared in a widely used textbook on program evaluation (Suchman, 1967), although Suchman does not label it "process evaluation" per se. Suchman writes:
In the course of evaluating the success or failure of a program, a great deal can be learned about how and why a program works or does not work. Strictly speaking, this analysis of the process whereby a program produces the results it does is not an inherent part of evaluative research. An evaluation study may limit its data collection and analysis simply to determining whether or not a program is successful.... However, an analysis of process can have both administrative and scientific significance, particularly where the evaluation indicates that a program is not working as expected. Locating the cause of the failure may result in modifying the program so that it will work, instead of its being discarded as a complete failure [p. 66].
This early definition of process evaluation includes the basic framework that is still used today; however, as is discussed later in this chapter, the definitions of the components of process evaluation have been further developed and refined.
Few references to process evaluation were made in the literature during the 1970s. In evaluation research, the 1970s were devoted to the issues of improving evaluation designs and measuring program effects. For instance, Struening and Guttentag's Handbook of Evaluation Research (1975) does not contain any reference to process evaluation (see also Glass, 1976; Guttentag and Saar, 1977; Green, 1977).
In their influential book, Green, Kreuter, Deeds, and Partridge (1980) define process evaluation in a somewhat unusual way: "In a process evaluation, the object of interest is professional practice, and the standard of acceptability is appropriate practice. Quality is monitored by various means, including audit, peer review, accreditation, certification, and government or administrative surveillance of contracts and grants" (p. 134).
The emphasis on professional practice as the focus of process evaluation as suggested by Green, Kreuter, Deeds, and Partridge (1980) faded as attention returned to the idea of assessment of program implementation. By the mid-1980s, the definition of process evaluation had expanded. Windsor, Baranowski, Clark, and Cutter (1984) explain the purpose of process evaluation in the following way:
Process produces documentation on what is going on in a program and confirms the existence and availability of physical and structural elements of the program. It is part of a formative evaluation and assesses whether specific elements such as facilities, staff, space, or services are being provided or being established according to the given program plan.... Process evaluation involves documentation and description of specific program activities-how much of what, for whom, when, and by whom. It includes monitoring the frequency of participation by the target population and is used to confirm the frequency and extent of implementation of selected programs or program elements. Process evaluation derives evidence from staff, consumers, or outside evaluators on the quality of the implementation plan and on the appropriateness of content, methods, materials, media, and instruments [p. 3].
McGraw and others (1989) write that "'process' or 'implementation' evaluation is not a new concept.... Yet in the broad field of social experimentation, it is only recently-since the late 1970s, that this aspect of program evaluation is accorded more formal recognition" (p. 460). This historical assessment appears to be accurate. Starting in the mid- to late 1980s, we can see the beginnings of contemporary process evaluation theory and methods and their application to applied public health interventions.
One of the key publications in the mid-1980s that laid the groundwork for modern process evaluation was an article titled "Avoiding Type III Errors in Health Education Program Evaluations: A Case Study" (Basch and others, 1985). Researchers are familiar with a Type I error (for example, rejecting a "true" null hypothesis) or a Type II error (for example, failing to reject a "false" null hypothesis), but a Type III error ensues from "evaluating a program that has not been adequately implemented" (p. 316). Although the idea of the Type III error was not new in 1985 (Dobson and Cook, 1980; Rezmovic, 1982), our research suggests that this was the first time this idea appeared in the public health literature. The article argues that measuring program implementation is critical to avoiding a Type III error and thus drawing incorrect conclusions about the effectiveness of a given intervention. That is, in addition to answering the evaluation question Did the program work? evaluators must first answer the question Was the program actually carried out as planned?-and if it was not carried out as planned, they must answer the question How did the program vary from the original plan?
The community-based cardiovascular disease prevention (CVD) demonstration studies funded by the National Heart, Lung, and Blood Institute (NHLBI) in the early 1980s through the next decade represented an important step forward in planning, implementing, and evaluating community-based public health interventions. Investigators from the Stanford Five-City Program, the Pawtucket Heart Health Program, and the Minnesota Heart Health Program had the foresight to get together periodically and look for ways to collaborate on the development of research questions, data collection, interventions, and program evaluation. The program evaluation units in each of the studies were independent (Pirie and others, 1994), but as studies evolved, investigators began to realize the importance of developing a consistent approach to assessing the dose of intervention delivered by the program staff in each of the communities. As a result, the three demonstration studies combined key components of their process tracking systems so that they could compare the extent of their activities by targeted risk factors (smoking, nutrition, physical activity, blood pressure, and blood cholesterol levels) over time. It was a big effort to join components of the three tracking systems that were already in the field, but the benefits of this effort were significant. The intervention staff of each project obtained feedback on whether they were meeting expectations for program delivery. Corrective feedback was possible, based on the results.
In addition, these studies were among the first intervention studies to conceptualize
and measure the importance of the dose received by program participants.
Previously, program evaluators were content to measure the extent to which
the intervention was delivered as planned. However, investigators were aware that
many interventions were delivered that participants never received. For example,
programs were scheduled and offered, but no one attended them. It was
important to recognize that if programs were not received, important corrective
action needed to be taken to improve the intervention or marketing of these
programs. Moreover, when interpreting results, one must take into account the fact
that participants may not have received what was delivered. As a result, the CVD
demonstration studies advanced the field by collecting data from community members
(through surveys) to assess participation in recently sponsored programs
and exposure to various media messages. Pirie and others (1994) have published
a detailed descriptive paper on program evaluation strategies of the demonstration
studies which underscores the numerous contributions made by these investigators
to conceptual thinking and the operationalization of key program evaluation
components.
Continues...
Excerpted from Process Evaluation for Public Health Interventions and Research Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.