Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts Download PPT Slides >> Table of Contents General Evaluation Resources Steps in Community Evaluation (and Related Resources) Design Stakeholder Involvement Identify Resources for Monitoring and Summative Evaluation Describe the Intervention’s Framework, Logic Model, or Theory of Change Focus the Monitoring and Summative Evaluation Plan Plan for credible methods Synthesize and Generalize Systems and Evaluation General Evaluation Resources Institute of Medicine (IOM): Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010) Measuring Progress in Obesity Prevention: Workshop Report (2012) Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012) Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013) U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide Community Tool Box (CTB): Toolkit: Evaluating the Initiative A Framework for Program Evaluation: A Gateway to Tools Evaluating Comprehensive Community Initiatives Toubleshooting Guide: We don’t know how to evaluate our program or initiative Steps in Community Evaluation (and related resources) Back to Top Design stakeholder involvement Identify stakeholders Consider the extent of stakeholder involvement Assess desired outcomes of monitoring and summative evaluation * Define stakeholder roles in the monitoring and summative evaluation * Resources: Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress American Evaluation Association: Statement On Cultural Competence In Evaluation Community Tool Box: Identifying and Analyzing Stakeholders and Their Interests Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests? Participatory Evaluation Gathering Information: Monitoring Your Progress Back to Top Identify resources for monitoring and summative evaluation Person-power resources Data collection resources Resources: Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Community Tool Box: Identifying Community Assets and Resources Choosing Evaluators Back to Top Describe the intervention’s framework, logic model, or theory of change Purpose or mission Context or conditions Inputs: resources and barriers Activities or interventions Outputs of activities Intended effects or outcomes Resources: Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Community Tool Box: Toolkit: Developing a Framework or Model of Change Developing a Logic Model or Theory of Change Back to Top Focus the monitoring and summative evaluation plan Purpose or uses: What does the monitoring and summative evaluation aim to accomplish? Set priorities by end-user questions, resources, context What questions will the monitoring and summative evaluation answer? Ethical implications (benefit outweighs risk) Resources: Institute of Medicine: Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress American Evaluation Association: Guiding Principles For Evaluators Community Tool Box: Choosing Questions and Planning the Evaluation Ethical Issues in Community Interventions Documenting Health Promotion Initiatives Using the PAHO Guide Back to Top Plan for credible methods Stakeholder agreement on methods Indicators of success Credibility of evidence Resources: Institute of Medicine: Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget) Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress National Collaborative on Childhood Obesity Research (NCCOR): Catalogue of Surveillance Systems Measures Registry Community Tool Box: Data Collection: Designing an Observational System Selecting an Appropriate Design for the Evaluation Collecting and Analyzing Data Collecting and Using Archival Data Qualitative Methods to Assess Community Issues Implementing Photovoice in Your Community Community-Level Indicators: Some Examples Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users Back to Top Synthesize and generalize Disseminating and compiling studies Learning more from implementation Ways to assist generalization Shared sense-making and cultural competence Disentangling effects of interventions Resources: Institute of Medicine: Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support Community Tool Box: Troubleshooting Guide: There is Not Enough Change in the Community or System Troubleshooting Guide: There is Not Enough Improvement in Outcomes Troubleshooting Guide: There are Unintended or Unwanted Outcomes Toolkit: Enhancing Cultural Competence Adapting Community Interventions for Different Cultures and Communities Providing Feedback to Improve the Initiative Communicating Information to Funders for Support and Accountability Refining the Program Intervention Based on Research The County Health Rankings & Roadmaps Take Action Cycle Back to Top Systems and Evaluation Resources: IOM: Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Community Tool Box: Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders Best Change Processes MAPP: Mobilizing for Action through Planning and Partnerships Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box. *Monitoring is the tracking of the implementation of interventions compared to standards of performance. *Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions. Tool 2: Table of Program Components and Elements Risk / Protective Factors Program Components (Strategies) Program Elements (Tactics) Targets and Agents of Change Mode / Context of Delivery Tool 3: Evaluation Questions and Methods Type of Data Collection * Experimental Design ** Key Evaluation Questions Type of Management Information and Evaluation Measure(s) Survey / Scale Structured Interview Self Report / Log Direct Observation Archival Records Case Study Pre- Posttest Control Group Time Series Planning and Implementation Issues Descriptive and Process Measures 1. 1. 2. 2. 3. 3. 4. 4. 5. 5. Assessing Attainment of Objectives Outcome Measures 1. 1. 2. 2. 3. 3. 4. 4. 5. 5. Impact on Participants 1. 1. 2. 2. 3. 3. 4. 4. 5. 5. Impact on Community 1. 1. 2. 2. 3. 3. 4. 4. 5. 5. Tool 4: Evaluation Timeline 2010 2011 2012 2013 Monitoring System J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D Establish system Train participants / staff in data entry Maintain system (data collection and reporting) Critical events Identify key people Conduct interviews Report results Community-Level Indicators Identify local indicators Identify comparison site Secure data Report results Other Indicators (e.g., behavioral surveys, data observations) Identify Secure data Report results
