Section 5. Developing an Evaluation Plan

Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts

Download PPT Slides >>

 

Table of Contents

General Evaluation Resources
Steps in Community Evaluation (and Related Resources)

1. Design Stakeholder Involvement
2. Identify Resources for Monitoring and Summative Evaluation 
3. Describe the Intervention’s Framework, Logic Model, or Theory of Change
4. Focus the Monitoring and Summative Evaluation  Plan
5. Plan for credible methods
6. Synthesize and Generalize
7. Systems and Evaluation

 

General Evaluation Resources

• Institute of Medicine (IOM):
  • ​Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010)
  • Measuring Progress in Obesity Prevention: Workshop Report (2012)
  • Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012)
  • Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013)
• U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
• Community Tool Box (CTB):
  • Toolkit: Evaluating the Initiative
  • A Framework for Program Evaluation: A Gateway to Tools
  • Evaluating Comprehensive Community Initiatives
  • Toubleshooting Guide: We don’t know how to evaluate our program or initiative

Steps in Community Evaluation (and related resources)

 

 

Back to Top

 

1. Design stakeholder involvement
   
   1. Identify stakeholders
   2. Consider the extent of stakeholder involvement
   3. Assess desired outcomes of monitoring and summative evaluation *
   4. Define stakeholder roles in the monitoring and summative evaluation *
   5. Resources:
      • Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Statement On Cultural Competence In Evaluation
      • Community Tool Box:
        • ​Identifying and Analyzing Stakeholders and Their Interests
        • Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
        • Participatory Evaluation
        • Gathering Information: Monitoring Your Progress

 

 

Back to Top

 

2. Identify resources for monitoring and summative evaluation
   
   1. Person-power resources
   2. Data collection resources
   3. Resources:
      • Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Identifying Community Assets and Resources
        • ​Choosing Evaluators

 

Back to Top

 

3. Describe the intervention’s framework, logic model, or theory of change
   
   1. Purpose or mission
   2. Context or conditions
   3. Inputs: resources and barriers
   4. Activities or interventions
   5. Outputs of activities
   6. Intended effects or outcomes
   7. Resources:
      • Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Toolkit: Developing a Framework or Model of Change
        • Developing a Logic Model or Theory of Change

 

 

Back to Top

 

4. Focus the monitoring and summative evaluation plan
   
   1. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
   2. Set priorities by end-user questions, resources, context
   3. What questions will the monitoring and summative evaluation answer?
   4. Ethical implications (benefit outweighs risk)
   5. Resources:
      • Institute of Medicine:
        1. Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        2. Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Guiding Principles For Evaluators
      • Community Tool Box:
        1. Choosing Questions and Planning the Evaluation
        2. Ethical Issues in Community Interventions
        3. Documenting Health Promotion Initiatives Using the PAHO Guide

 

 

Back to Top

 

5. Plan for credible methods
   
   1. Stakeholder agreement on methods
   2. Indicators of success
   3. Credibility of evidence
   4. Resources:
      • Institute of Medicine:
        • Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget)
        • Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • National Collaborative on Childhood Obesity Research (NCCOR):
        • Catalogue of Surveillance Systems
        • ​Measures Registry
      • Community Tool Box:
        • Data Collection: Designing an Observational System
        • Selecting an Appropriate Design for the Evaluation
        • Collecting and Analyzing Data
        • Collecting and Using Archival Data
        • Qualitative Methods to Assess Community Issues
        • Implementing Photovoice in Your Community
        • Community-Level Indicators: Some Examples
      • Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users

 

Back to Top

 

6. Synthesize and generalize
   
   1. Disseminating and compiling studies
   2. Learning more from implementation
   3. Ways to assist generalization
   4. Shared sense-making and cultural competence
   5. Disentangling effects of interventions
   6. Resources:
      • Institute of Medicine:
        • Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support
      • Community Tool Box:
        • Troubleshooting Guide: There is Not Enough Change in the Community or System
        • Troubleshooting Guide: There is Not Enough Improvement in Outcomes
        • Troubleshooting Guide: There are Unintended or Unwanted Outcomes
        • Toolkit: Enhancing Cultural Competence
        • Adapting Community Interventions for Different Cultures and Communities
        • Providing Feedback to Improve the Initiative
        • Communicating Information to Funders for Support and Accountability
        • Refining the Program Intervention Based on Research
        • The County Health Rankings & Roadmaps Take Action Cycle

 

 

Back to Top

 

7. Systems and Evaluation
   • Resources:
     • IOM:
       • Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
       • Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation
       • Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
     • Community Tool Box:
       • Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders
       • Best Change Processes
       • MAPP: Mobilizing for Action through Planning and Partnerships

 

Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box.

 

*Monitoring is the tracking of the implementation of interventions compared to standards of performance.
*Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions.

 

 

Tool 2: Table of Program Components and Elements

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery

 

Tool 3: Evaluation Questions and Methods

		Type of Data Collection *					Experimental Design **
Key Evaluation Questions	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured Interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- Posttest Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Assessing Attainment of Objectives	Outcome Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Participants
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Community
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.

Tool 4: Evaluation Timeline

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system
Train participants / staff  in data entry
Maintain system (data collection and reporting)
Critical events
Identify key people
Conduct interviews
Report results
Community-Level Indicators
Identify local indicators
Identify comparison site
Secure data
Report results
Other Indicators (e.g., behavioral surveys, data observations)
Identify
Secure data
Report results