Why should you have an evaluation plan? Evaluation helps you understand how an initiative is being implemented, what is changing, what is working well, what is not working as intended, and what should happen next. Evaluation should not be treated simply as a final judgment about whether a program “succeeded” or “failed.” A useful evaluation can help an organization: Understand how the initiative is being implemented Identify strengths and areas for improvement Understand intended and unintended effects Make decisions during implementation Assess progress toward goals Learn from community members and participants Meet reporting or accountability requirements Decide whether an initiative should be continued, changed, expanded, or ended There are many evaluation frameworks, tools, and methods available. Do not select a method simply because it appears in an evaluation handbook or because another program used it. Begin by asking what you actually need to learn and what decisions the information will help people make. Collecting large amounts of information is not useful if the information does not answer important questions or inform action. An evaluation plan can help you: Clarify why the evaluation is being conducted Identify who should participate in evaluation planning Determine what questions are most important Identify what evidence is needed Select appropriate evaluation methods Establish responsibilities for data collection, analysis, interpretation, and reporting Protect privacy and address ethical concerns Develop a realistic timeline and budget Decide how findings will be communicated and used Improve the initiative throughout its life cycle When should you develop an evaluation plan? Begin evaluation planning as early as practical. Ideally, evaluation is considered while the initiative itself is being designed. Early planning can help establish: Clear goals and intended outcomes Baseline information when needed Appropriate measures Data-collection systems Responsibilities Evaluation resources Plans for using findings If an initiative is already underway, it is not too late to begin evaluation. Determine what information is still available, what can reasonably be collected going forward, and what evaluation questions remain answerable. Evaluation should support learning throughout implementation rather than occur only after the initiative is finished. Who should be involved in evaluation planning? Evaluation can affect many people and organizations. Identify the people who: Are directly affected by the initiative Participate in the initiative Implement or manage it Provide funding or oversight Make decisions using the findings Have relevant community, professional, technical, or lived knowledge Different people may use evaluation findings for different purposes. Involving them early can help make the evaluation more relevant, useful, fair, and responsive to the context in which the initiative operates. People directly affected by the initiative People whose lives, services, neighborhoods, organizations, or communities are affected by an initiative may have important knowledge about: Whether the initiative addresses a real need Whether services are accessible How implementation is experienced What outcomes matter What unintended effects have occurred What changes would make the initiative more useful Do not refer to people primarily as “targets of change.” Whenever practical, involve people directly affected in shaping evaluation questions, interpreting findings, and deciding what actions should follow. Program staff, volunteers, and organizational leaders People involved in implementing the initiative can provide information about: Activities Resources Participation Implementation challenges Changes made during the program Organizational capacity Their involvement is important, but implementation staff should not be the only people interpreting whether the initiative is effective. Funders and oversight organizations Funders may have legitimate reporting, accountability, or learning requirements. Determine what information is required by: Grant agreements Contracts Government reporting requirements Organizational policies Funding requirements should be considered alongside the information needs of participants, community members, program staff, and other relevant groups. Do not design an evaluation solely around what the funder wants to know if doing so would leave important community or program questions unanswered. Evaluators, researchers, and people with technical expertise An initiative may involve: Internal evaluation staff Independent evaluators University researchers Data analysts Community-based researchers Subject-matter experts Involve them early enough to help determine what questions can reasonably be answered and which methods are appropriate. Researchers and evaluators should not automatically control the evaluation simply because they have technical expertise. Evaluation planning is stronger when technical knowledge is combined with community knowledge, lived experience, program expertise, and the needs of intended users. What do people need to know from the evaluation? Different people may need different information. Before selecting measures or methods, ask: Who will use the findings? What decisions do they need to make? What information would help them make those decisions? When will they need the information? What information would be useful to people directly affected by the initiative? How should findings be communicated so that people can understand and use them? Evaluation should support decisions Findings may inform decisions about: Program improvement Staffing Funding Program continuation Expansion or replication Modification of services Policy Community priorities Future evaluation or research Evaluation should produce honest and useful information. Do not design measures, analyses, or reporting practices to exaggerate success or minimize unfavorable results. Unexpected, mixed, or negative findings can provide important information for improvement. How much should evaluation cost? There is no single percentage of a program budget that is appropriate for every evaluation. Evaluation resources depend on factors such as: The questions being asked Program size and complexity Available data Required methods Evaluation duration Reporting requirements Staff capacity Participant burden Available funding Develop a realistic evaluation budget based on the work required rather than applying a fixed percentage automatically. Ask: What information is essential? What information would be useful but optional? What information is already available? What data can realistically be collected? What resources are required to analyze and use the information responsibly? How do you develop an evaluation plan? Evaluation planning is often iterative rather than strictly linear. A practical process includes: Assess the context. Describe the initiative and how it is expected to work. Identify who should participate in evaluation planning. Clarify the purpose of the evaluation and intended uses. Develop and prioritize evaluation questions. Select an evaluation design and appropriate methods. Gather credible evidence. Analyze and interpret findings. Communicate findings in useful forms. Act on what is learned. Describe the initiative and clarify its goals Before evaluating an initiative, make sure there is a shared understanding of: The problem or need being addressed Resources available Activities being implemented People or communities the initiative intends to reach Expected short-term outcomes Expected intermediate outcomes Expected long-term outcomes Important contextual factors A logic model, theory of change, program roadmap, or another visual representation may help clarify how activities are expected to contribute to outcomes. Do not assume that every observed change can automatically be attributed to the initiative. Develop evaluation questions Evaluation questions should reflect what people need to learn and what decisions the findings will inform. Common categories include: Context and need Possible questions include: What community need is the initiative intended to address? Has that need changed? What contextual factors may affect implementation or outcomes? What strengths and resources already exist in the community? Planning and implementation Possible questions include: Was the initiative implemented as intended? What changes were made during implementation and why? Who participated? Who encountered barriers to participation? Were activities accessible and appropriate for the people they were intended to reach? How did participants experience the initiative? What implementation challenges occurred? Progress toward objectives Possible questions include: Were planned activities completed? Were intended participants reached? Were services delivered at the expected level? Were short-term objectives achieved? Measures of participation and activity are important, but they should not automatically be treated as evidence that meaningful outcomes occurred. Outcomes for participants Possible questions include: What changed for participants? Did knowledge, skills, access, behavior, conditions, or well-being change? Were there unintended positive or negative effects? Did outcomes differ among groups or settings? How did participants describe the value or limitations of the initiative? Community or systems outcomes Possible questions include: Did relevant community conditions change? Were organizational practices changed? Did policies or systems change? Did access to services change? Were disparities reduced, maintained, or increased? Were there unintended community effects? Costs and value Depending on the evaluation purpose, you may also ask: What resources were used? What did implementation cost? How do costs compare with outcomes? Could similar outcomes be achieved another way? Select evaluation methods Choose methods according to the evaluation questions rather than starting with a preferred method and trying to make every question fit it. An evaluation may use quantitative methods, qualitative methods, mixed methods, participatory approaches, or a combination. Monitoring and performance information Monitoring can help track implementation while an initiative is underway. Measures may include: Activities completed Participation Services delivered Resources used Milestones Implementation changes Short-term outcomes Monitoring and evaluation are related but are not necessarily the same activity. Surveys Surveys can gather information about: Experiences Knowledge Attitudes Behaviors Satisfaction Perceived outcomes Barriers Design survey questions carefully and consider who may be excluded by the method used to administer them. Interviews and focus groups Interviews and group discussions can help explain: Why something happened How people experienced the initiative What contextual factors mattered Why implementation differed across settings What improvements participants recommend Include people because their knowledge or experience is relevant rather than referring only to “key” or high-status individuals. Administrative and program data Existing information may include: Participation records Service records Financial information Health records where lawful and appropriate Education data Public records Other administrative information Evaluate the quality, completeness, relevance, and privacy implications of existing data before using them. Community-level indicators Community indicators may help track changes in conditions over time. Select indicators that are: Relevant to the evaluation question Clearly defined Measured consistently Available at an appropriate geographic and population level Interpreted in context A change in a community indicator does not automatically prove that one initiative caused the change. Comparison and causal designs Some evaluations need to estimate whether observed outcomes can reasonably be attributed to the initiative. Depending on the question and context, designs may include: Experimental designs Quasi-experimental designs Comparison groups Interrupted time-series approaches Other appropriate causal or contribution approaches Not every evaluation requires an experimental design. Select the strongest feasible design that is appropriate to the question, ethical considerations, available data, timeline, and resources. Gather credible evidence ethically Evaluation planning should address: What data will be collected From whom Why those people or sources were selected How information will be collected How privacy and confidentiality will be protected Who will have access to identifiable information How data will be stored How long information will be retained How findings will be shared Whether additional ethical, legal, or institutional review requirements apply Collect only the information necessary to answer the evaluation questions. Analyze and interpret findings carefully Interpretation should consider: The evaluation questions Program context Data quality Alternative explanations Differences across groups or settings Intended and unintended outcomes Limitations of the evaluation design Involve relevant people in interpreting findings when practical. Different perspectives may reveal explanations or implications that would otherwise be missed. Do not interpret results only through the perspective of the funder, evaluation team, or program leadership. Develop an evaluation timeline Evaluation activities should be aligned with the life cycle of the initiative. The timeline may identify: Evaluation planning Baseline data collection Ongoing monitoring Interim analysis Feedback sessions Final data collection Analysis and interpretation Reporting Follow-up measurement Decisions based on findings Evaluation may continue after the formal project period Do not assume that evaluation must end shortly after program activities stop or when a final grant report is due. Some outcomes may take longer to appear. Follow-up evaluation may be useful when you need to understand: Whether outcomes were sustained Whether changes persisted after services ended Whether unintended effects emerged later Whether systems or policy changes were maintained The evaluation timeline should match the questions being asked. Provide feedback throughout the evaluation Do not wait until the final report to share everything that has been learned. Interim findings may help: Improve implementation Identify emerging problems Correct data-quality issues Adapt services Inform upcoming decisions Make clear when findings are preliminary. What products should come from an evaluation? The most useful product is not always a long final report. Different audiences may need different formats. Evaluation products may include: Full reports Executive summaries Briefs Presentations Community meetings Data dashboards Infographics Accessible digital summaries Technical appendices Internal learning sessions Recommendations or action plans Choose formats according to how people will actually use the information. Evaluation products should explain: Why the evaluation was conducted What questions were examined Who participated in the evaluation What methods were used What was found What limitations exist What differences appeared across groups or settings What unintended findings emerged What conclusions are supported by the evidence What actions should be considered next Report important unfavorable or unexpected findings as well as positive results. Use the findings Evaluation is most valuable when findings lead to learning and action. After reviewing the evidence, decision-makers may: Continue the initiative Modify it Expand it Reduce or discontinue ineffective components Improve accessibility Address unintended harms Change resource allocation Revise the theory of change Conduct additional evaluation or research Decide during evaluation planning how findings will reach the people able to act on them. What standards should guide an evaluation? Evaluation standards and professional guidance evolve over time. Current public-health evaluation guidance emphasizes several broad principles for high-quality evaluation. Relevance and utility Evaluation should answer questions that matter and provide information people can use. Rigor Methods should be appropriate to the questions, and conclusions should be supported by credible evidence. Limitations should be reported clearly. Independence and objectivity Evaluation processes should reduce inappropriate influence, conflicts of interest, and avoidable bias. Evaluators should reflect on how their own assumptions and roles may affect the work. Transparency Document: The purpose of the evaluation Evaluation questions Methods Important decisions made during the evaluation Changes to the original plan Limitations How conclusions were reached Ethics Protect the dignity, rights, safety, privacy, and confidentiality of people who contribute information. Consider who bears the burden of data collection, who benefits from the evaluation, and whether the evaluation could create unintended harm. Build collaboration, fairness, and use throughout the evaluation High-quality evaluation is not simply a technical exercise. Throughout the process: Engage relevant people collaboratively Consider equity and fairness in participation, methods, interpretation, and use Learn from findings as they emerge Use insights to improve the initiative and future decisions Ask who has been included in evaluation decisions and whose perspective may be missing. Pay particular attention to whether people most affected by the initiative have meaningful opportunities to contribute. In Summary An evaluation plan helps an initiative determine what it needs to learn, how it will gather credible evidence, who should participate, when evaluation activities should occur, and how findings will be used. Begin planning evaluation as early as practical, but do not abandon evaluation simply because an initiative is already underway. Start with the decisions and questions that matter rather than with a preferred method or a large list of available measures. Involve people directly affected, program staff, community partners, funders, decision-makers, evaluators, and other relevant participants according to the evaluation's context. Do not refer to people as “targets of change” or assume that technical experts are the only people who hold important evaluation knowledge. Develop a realistic budget based on the evaluation's actual questions, design, data needs, timeline, and available resources rather than assuming that every initiative should spend the same percentage of its budget on evaluation. Clarify how the initiative is expected to work and develop evaluation questions addressing implementation, outcomes, community or systems change, unintended effects, and other priorities. Select methods that fit the questions. Surveys, interviews, monitoring data, administrative information, community indicators, qualitative methods, and experimental or quasi-experimental designs can all be useful in the appropriate circumstances. Do not assume that every evaluation must prove that the initiative caused an observed outcome. Collect and use information ethically, protect privacy, explain limitations, and avoid designing an evaluation to produce a preferred result. Share learning throughout implementation when doing so can improve the initiative. Use reports, briefs, dashboards, presentations, community discussions, or other formats appropriate to the people who need the findings. Evaluation may continue after an initiative formally ends when important outcomes require longer-term follow-up. Finally, evaluation should not end with the production of a report. Its value comes from using credible evidence and multiple forms of knowledge to understand an initiative, improve decisions, address unintended effects, and determine what should happen next. Contributor Chris Hampton Resources Online Resource The Action Catalogue is an online decision support tool that is intended to enable researchers, policy-makers and others wanting to conduct inclusive research, to find the method best suited for their specific project needs. CDC Evaluation Resources provides an extensive list of resources for evaluation, as well as links to key professional associations and key journals. Developing an Evaluation Plan offers a sample evaluation plan provided by the U.S. Department of Housing and Urban Development. Developing an Effective Evaluation Plan is a workbook provided by the CDC. In addition to ample information on designing an evaluation plan, this book also provides worksheets as a step-by-step guide. Evaluating Your Community-Based Program [pdf] is a handbook designed by the American Academy of Pediatrics and includes extensive material on a variety of topics related to evaluation. GAO Designing Evaluations is a handbook provided by the U.S. Government Accountability Office. It contains information about evaluation designs, approaches, and standards. The Magenta Book - Guidance for Evaluation provides an in-depth look at evaluation. Part A is designed for policy makers. It sets out what evaluation is, and what the benefits of good evaluation are. It explains in simple terms the requirements for good evaluation, and some straightforward steps that policy makers can take to make a good evaluation of their intervention more feasible. Part B is more technical, and is aimed at analysts and interested policy makers. It discusses in more detail the key steps to follow when planning and undertaking an evaluation and how to answer evaluation research questions using different evaluation research designs. It also discusses approaches to the interpretation and assimilation of evaluation evidence. Plan an Evaluation is an extensive guide provided by MEERA aimed at providing detailed information on planning an evaluation. Using Data as an Equity Tool is an Urban Institute resource which provides strategies and key practices which place-based organizations can use to build local data capacity with their partners, improve service provision and day-to-day operations, and amplify community voices. Print Resources Argyris, C., Putnam, R., & Smith, D. (1990). Action Science, Chapter 2, pp. 36-79. San Francisco: Jossey-Bass. Fawcett, S., in collaboration with Francisco, V., Paine-Andrews, A., Lewis, R., Richter, K., Harris, K., Williams, E., Berkley, J., Schultz, J., Fisher, J., & Lopez, C. (1993). Work group evaluation handbook: Evaluating and supporting community initiatives for health and development. Lawrence, KS: Work Group on Health Promotion and Community Development, The University of Kansas. Fawcett, S., Sterling, T., Paine, A., Harris, K., Francisco, V., Richter, K., Lewis, R., & Schmid, T. (1995). Evaluating community efforts to prevent cardiovascular diseases. Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion. Francisco, V., Fawcett, S., & Paine, A. (1993). A method for monitoring and evaluating community coalitions. Health Education Research: Theory and Practice, 8(3), 403-416. Fetterman. (1996). Empowerment evaluation: An introduction to theory and practice. In D.M. Fetterman, S. J. Kaftarian, & A. Wandersman (eds.), Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability, (3-46). Green, L., & Kreuter, M. (1991). Evaluation and the accountable practitioner. Health promotion planning, (2nd Ed.), (pp. 215-260). Mountain View, CA: Mayfield Publishing Company. Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards. Evaluation Practice, 15, 334-336.
