Some Recommendations for Evaluating and Supporting Collaborative Efforts to Promote Community Health

Stephen Fawcett and Jerry Schultz, Work Group for Community Health and Development, University of Kansas

A Context for the Evaluation of Collaborative Efforts to Promote Health:

Collaborative efforts to promote health—such as CDC’s REACH 2010, Healthy STEPS, or other community health initiatives—aim to create environments that reduce risk and promote healthy behaviors. Although significant differences exist among the aims, targets, and conditions of different community efforts, the context for evaluation remains similar. In community efforts to reduce risk for obesity, for instance, there are multiple and interrelated factors (e.g., opportunities for physical activity, access to healthy food choices, social support) that contribute to multiple and interrelated outcomes (e.g., prevalence of regular physical activity, daily intake of high fat foods). No single intervention – no one program or policy change – targeting one behavior is likely to improve population-level outcomes. Additionally, the long delay between targeted actions and the resulting widespread behavior change and improvement in population-level outcomes makes it difficult to assess whether the effort is bringing about change.

Accordingly, in dynamic systems, it is important to document those new programs, policies and practices—community and system changes—that form the comprehensive intervention that unfolds over time. We can analyze the contribution of documented instances of community and system change, an intermediate outcome, to subsequent behavior change and longer-term outcomes. Findings from this analysis can be used to improve the effort’s effectiveness along the way.

The Institute of Medicine’s (2003) framework for collaborative public health action in communities suggests a general model and four key evaluation questions.  Depicted below, it is applicable to the major phases of collaborative efforts to promote health:

 

Question 3

A framework for collaborative public health action in communities.

(Source: CDC, 2002; Fawcett et al., 2000; Institute of Medicine, National Academy of Sciences, 2003.)

 

Core Evaluation Questions and Related Measures:

To evaluate community efforts to promote health, we recommend four evaluation questions and related measures.  A brief description of each question and related methods follows:

  1. Is the collaborative effort serving as a catalyst for community/system change in order to promote health?

Measurement and Analysis:

To capture the unfolding accomplishments of the effort, we recommend documenting instances of community/system change; that is, all new or modified programs, policies and practices facilitated by the effort and related to the mission (e.g., promoting physical activity, promoting healthy diets). Any system of recording the effort’s actions and accomplishments should be highly accessible and easily shared among partners and community members (e.g., online documentation). Documented changes can subsequently be graphed, and participants can examine the rate (cumulative number) of community/system change over time.  Collaborative partners may review time series graphs to look for discontinuities (marked increases or decreases) in the rate of change for the distinct collaborative efforts.  See illustrative Figure 1 below.

          


Figure 1

 

  1. What factors or processes are associated with the rate of community/system change for promoting health?

Measurement and Analysis: 

Twice yearly or annually, we recommend gathering qualitative information on critical events (e.g., completed action plan; loss of leadership) in the distinct collaborative efforts.  These critical events can be overlaid on the time series graph(s) of the unfolding of community/system changes related to promoting health (e.g., promoting physical activity or healthy diets).  When particular events (e.g., action planning; changes in leadership) are associated with marked discontinuities (increases/decreases) in the rate of community change—and where these associations are replicated across contexts—they may suggest key processes or factors that can advance efforts to promote health.  See Figure 1 above.

  1. How are community/system changes contributing to the efforts to promote health?

Measurement and Analysis of Contribution: 

Twice annually, we recommend that key parties review an analysis of contribution and use it to make adjustments with the overall initiative and distinct collaborative/community efforts.  In order to more clearly understand what characteristics of community/system change influence rates of behavior change, we recommend that documenters record several relevant dimensions related to the conceptual framework (e.g., goal addressed, sector the change occurs within, intervention or behavior change strategy used, population targeted, duration of change) for each change.  This secondary scoring and review permits an analysis of contribution by relevant dimensions (e.g., amount of change by: goal, strategy, duration, population, sector, and place). See illustrative Figure 2 below.

 

Figure 2

 

  1. [For each place-based community effort] Are community/system changes associated with improvements in population-level outcomes?

Measurement and Analysis: 

Annually, examine whether (and under what conditions) the cumulative unfolding of community/system change is related to improvement in indicators of population-level health outcomes (e.g., decreasing prevalence of obesity; increasing self-reported levels of regular physical activity).  We recommend using the analysis of contribution to help discover the conditions under which associated improvements are noted (e.g., sufficient amount of change by goal; duration; intensity of strategy; penetration to targets, through sectors, in places).  Multiple case studies, in which data from discrete community efforts are compared, can help examine the generality of observed associations across contexts and conditions.  See illustrative Figure 3 below.

Figure 3

 

Summary and Potential Advantages of this Documentation and Evaluation Approach:

There are several advantages to the approach outlined above.  First, it can help systematically document the unfolding of the intervention in the dynamic and diverse contexts of collaborative efforts.  This helps to analyze the contribution of community/system changes intended to “tip” or improve population-level outcomes.  Second, it permits attention to several core evaluation questions for collaborative public health action in communities.  For instance, it can help address such key questions as:  “Is the initiative serving as a catalyst for community/system change?” and “Are community/system changes associated with improvements in population-level outcomes related to the mission?” Third, similar capabilities for documenting and analyzing the contribution of collaborative efforts are currently available.  For instance, the KU Work Group’s Online Documentation and Support System can be used for online data entry, graphing, data export, and reporting. Fourth, the community/system changes documented by collaborating partners will suggest promising approaches for intervention in other communities and contexts.  For example, by enabling collaborating partners to view documented changes, promising intervention elements can be adapted for other contexts. Finally, use of a cross-cutting measurement and support system will permit co-learning within and across community initiatives to promote health.

 

Selected References:

CDC (Centers for Disease Control and Prevention).  (2002).  Syndemics overview: What procedures are available for planning and evaluating initiatives to prevent syndemics?  The National Center for Chronic Disease Prevention and Health Promotion Syndemics Prevention Network.  Available online at www.cdc.gov/syndemics/overview-planeval.htm. Accessed April 7, 2002.

Fawcett, S. B., Francisco, V. T., Hyra, D., Paine-Andrews, A., Schultz, J. A., Russos, S., Fisher, J. L., and Evensen, P. (2000). Building healthy communities. In A. Tarlov and R. St. Peter (Eds.), The society and population health reader: A state and community perspective. (Pp. 75-93). New York: The New Press.

Fawcett, S.B., Francisco, V.T., and Schultz, J.A.  (2004).  Understanding and improving the work of community health and development.  In J. Burgos and E. Ribes (Eds.), Theory, basic and applied research, and technological applications in behavior science.  (Pp. 209-241). Guadalara, Mexico: Universidad de Guadalara.

Fawcett, S. B., Schultz, J. A., Carson, V. L., Renault, V. A., and Francisco, V. T. (2003). Using Internet-based tools to build capacity for community-based participatory research and other efforts to promote community health and development. In M. Minkler and N. Wallerstein (Eds.), Community-based participatory research for health. (Pp. 155-178). San Francisco: Jossey-Bass.

Fawcett, S. B., Sterling, T. D., Paine-Andrews, A., Harris, K. J., Francisco, V. T., Richter, K. P., Lewis, R. K., and Schmid, T. L. (1995). Evaluating community efforts to prevent cardiovascular diseases. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion.

Institute of Medicine. (2003). Chapter on “Community”. (Pp. 178-211). The future of the public’s health in the 21st century. Washington, D.C.: National Academies Press.

Paine-Andrews, A., Fisher, J. L., Berkley-Patton, J., Fawcett, S. B., Williams, E. L., Lewis, R. K., Harris, K. J. (2002). Analyzing the contribution of community change to population health outcomes in an adolescent pregnancy prevention initiative. Health Education and Behavior, 29(2), 183-193.

Roussos, S. T. and Fawcett, S. B. (2000). A review of collaborative partnerships as a strategy for improving community health.  Annual Review of Public Health, 21, 369-402.

U. S. Department of Health and Human Services.  (2000).  Healthy People 2010.  Washington, D.C.

 

Community Tool Box Resources -- Supporting Community Efforts for Health and Development

 

16 Core Competencies

Some “Best Processes” for Change and Improvement
  1. Creating and Maintaining Coalitions and Partnerships
  2. Assessing Community Needs and Resources
  3. Analyzing Problems and Goals
  4. Developing a Framework or Model of Change
  5. Developing Strategic and Action Plans
  6. Building Leadership
  7. Developing an Intervention
  8. Increasing Participation and Membership
  9. Enhancing Cultural Competence
  10. Advocating for Change
  11. Influencing Policy Development
  12. Evaluating the Initiative
  13. Implementing a Social Marketing Campaign
  14. Writing a Grant Application for Funding
  15. Improving Organizational Management and Development
  16. Sustaining the Project or Initiative
  • Analyzing Information About the Problem, Goals, and Factors Affecting Them
  • Establishing Your Group's Vision and Mission
  • Defining Organizational Structure and Operating Mechanisms
  • Developing a Framework or Model of Change
  • Developing and Using Strategic and Action Plans
  • Arranging for Community Mobilizers
  • Developing Leadership
  • Implementing Effective Interventions
  • Assuring Technical Assistance
  • Documenting Progress and using Feedback
  • Making Outcomes Matter
  • Sustaining the Work

 

Example WorkStation