Chapter 39. Using Evaluation to Understand and Improve the Initiative | Community Tool Box

Evaluation can help your initiative improve its methods, communicate with funders, support staff, and make better decisions. Its central purpose is to help you understand what is happening, what difference your work is making, and how to improve it.

Evaluations sometimes focus on activities and outputs, such as the number of people trained, without examining whether community conditions changed. Those measures are useful, but they do not tell the whole story. A broader evaluation asks whether intended changes occurred, how they developed, and what evidence connects them to the initiative.

Alongside assessing specific objectives, you can examine changes in programs, policies, practices, and community conditions. You can also investigate whether those changes contribute to longer-term outcomes. This section explains how to frame those questions, organize relevant information, and interpret the findings.

What does answering evaluation questions involve?

Evaluation questions identify what participants, community members, staff, partners, funders, and others need to know about an initiative. Useful questions connect to decisions that people will make. Examples include:

• How do participants experience the program, and what do they want improved? (Process and implementation)
• To what extent is the initiative meeting its stated objectives? (Attainment of objectives)
• What changed for participants, and what evidence connects those changes to the initiative? (Participant outcomes and the initiative's contribution)
• What changed across the community, for whom, and how did the initiative contribute? (Community outcomes and the initiative's contribution)

The key questions depend on your mission, objectives, and understanding of how change is expected to occur. A logic model describes the proposed connections among resources, activities, outputs, and shorter- and longer-term outcomes. It should also acknowledge assumptions and outside influences. Evaluation questions examine those connections: for example, "Did the initiative help change access to services?" and "Were changes in access followed by improvements in residents' experiences or health?"

The Center for Community Health and Development at the University of Kansas, which produces the Community Tool Box, developed the Community Check Box approach to documenting and evaluating community health and development efforts. Much of this section draws on the thinking behind that approach.

The examples show how a documentation system can organize evidence relevant to evaluation questions. A system can support analysis, but its records alone do not establish an initiative's effectiveness. The strength of the conclusions depends on the questions, data quality, evaluation design, and interpretation.

You can examine your work at several connected levels. One concerns specific objectives, such as adopting a policy, improving services, or changing an environmental condition. Another concerns the initiative's wider influence on organizations, systems, and community life. Both levels matter, and progress at one does not automatically establish progress at the other.

A comprehensive evaluation distinguishes processes, outputs, and outcomes. Processes describe how the work is carried out. Outputs are its immediate products, such as training sessions delivered or people reached. Outcomes are changes in conditions, behavior, access, or well-being. An impact evaluation goes further by investigating whether outcomes differ because of the initiative, using a design suited to that causal question.

Evaluation should also consider changes beyond the original objectives. An initiative may encourage partner organizations to start related programs, attract additional resources, or change how community members participate in decisions. For example, a youth violence prevention effort might expand opportunities for young people to shape local services. Even when an initiative does not meet an initial objective, it may produce useful learning or other changes worth documenting. Look for unintended harms as well as benefits, and avoid using broader achievements to obscure unmet commitments.

Two terms are useful throughout this discussion:

A community may be defined by place, such as a neighborhood, town, city, or rural area. It may also be defined by shared identities, interests, experiences, or relationships, including communities that connect online. This section often refers to communities of place, but the evaluation principles can apply more broadly. Be explicit about whom your initiative includes and avoid assuming that everyone within a community shares the same priorities.

A system includes the organizations, relationships, resources, rules, and practices that shape a function such as education, health care, housing, or public safety. Changes in a system can affect access to opportunities and services, how decisions are made, and the conditions in which people live.

At each level, ask what changed, who experienced the change, and what evidence explains how it happened. Examine both how the initiative influenced its setting and how changes in that setting affected the initiative.

Figure 1 presents a framework for collaborative action, along with evaluation questions about the proposed relationships between its stages.

The framework discussed by Fawcett and Schultz (2008) highlights four questions, restated here:

1. Is the initiative helping bring about community or system changes related to its mission? Look for new or modified programs, policies, and practices, and document the initiative's role alongside the contributions of partners and other influences.
2. What factors or processes are associated with community or system change? Examine events, relationships, resources, and conditions that may support or limit progress, including changes in the pace of documented activity.
3. How do those changes contribute to the initiative's goals? Consider their relevance, reach, quality, duration, and effects on different groups, including unintended consequences.
4. Are community or system changes associated with improvements in population-level outcomes? Examine whether changes in programs, policies, and practices align with outcomes such as fewer injuries, improved access to care, or reduced tobacco use. Choose measures that closely match the outcome rather than assuming that arrests or product sales directly measure behavior.

Figure 1. Framework for collaborative community action and related evaluation questions (Source: Fawcett and Schultz, 2008; Institute of Medicine, 2003).

The questions in Figure 1 examine proposed relationships along the framework's pathway. These relationships are assumptions to investigate, not conclusions established by the diagram. See Fawcett and Schultz (2008) in the print resources.

Source: Adapted from the Institute of Medicine's framework for collaborative public health action in communities (2003).

These questions place the initiative within its broader setting and help distinguish participant outcomes from community-wide changes. They may require several kinds of evidence and may not all be answerable with the same level of confidence. Later in this section, we also address a fifth question: whether the initiative caused improvements in population-level outcomes.

Why answer these questions?

Funding arrangements may emphasize short-term deliverables, while meaningful community changes can take longer. Evaluation can meet reporting needs while also examining the relationships and outcomes that matter to the community. Reasons to answer these questions include:

• To improve the work. Understanding how activities, community changes, and outcomes relate can help you identify what to strengthen, adapt, or discontinue.
• To understand what supports or limits progress. Examine how leadership, staffing, partnerships, planning, and other conditions affect implementation and the opportunities for change.
• To improve the pace and quality of progress. Findings may suggest ways to remove barriers or make better use of resources. Faster change is not always better; participation, quality, and sustainability also require attention.
• To understand the broader context. Economic hardship, unequal access to resources, policy changes, or community crises can affect both implementation and outcomes. Documenting these conditions helps explain results and inform future work.
• To understand how lasting improvements develop. Your goals may include preventing domestic violence, reducing pollution, improving health, or expanding educational opportunities. Evaluation can explore how specific changes in programs, practices, or policies contribute to those goals.
  
  Examine which changes appear useful, how they work, and under what conditions they might be sustained. Use the findings to refine your approach while remaining open to evidence that challenges the original plan.
• To manage organizational transitions. Leadership turnover, funding changes, or a shift between program phases may affect the work. Understanding these relationships can support continuity planning and help your group respond to disruptions.
• To communicate value and maintain accountability. Share achievements, challenges, and uncertainties with the community. A credible account of the work can support informed decisions about participation, partnerships, and resources.

Who should help answer these questions?

Involve people who experience, carry out, support, or make decisions about the initiative. Their perspectives can help identify useful questions and explain findings that records alone cannot capture. Participatory evaluation means sharing meaningful roles in decisions, interpretation, and use of findings; providing information alone is not the same as sharing decision-making. Agree on roles, address accessibility and language needs, and consider compensation for community members' time. People to involve may include:

• Participants and people directly affected by the initiative. They can identify priorities, describe benefits and barriers, and assess whether proposed changes meet their needs. Include people with mixed or negative experiences, not only those who remain actively involved.
• Residents of the area where the initiative works. Invite perspectives from across the area, including residents who do not use the program. Their experiences may reveal wider effects or needs that the initiative has overlooked.
• Staff and volunteers carrying out the work. They can explain implementation, resource constraints, adaptations, and practical opportunities for improvement. Make room for candid feedback without penalizing people for identifying problems.
• People whose work or relationships connect them to the issue. Depending on the initiative, these may include educators, health care providers, family members, caregivers, advocates, or other partners. Their perspectives complement, rather than replace, the views of people directly affected.
• Funders and public officials. Their involvement can help align evaluation questions with decisions about resources and policy. Establish arrangements that protect the integrity of the evaluation and prevent powerful participants from controlling findings or excluding other perspectives.

When should you establish and use an evaluation system?

Plan evaluation as early as possible, ideally alongside community assessment and initiative planning. Early planning helps identify baseline information, define expected changes, and document decisions that may matter later. If the initiative is already underway, begin with the information available and reconstruct earlier events where records support it. Be clear about gaps rather than treating a late start as a reason not to evaluate.

Collect and review information on a schedule that fits your decisions, resources, and expected timing of change. Record when activities and events occur so you can examine sequences and possible relationships. Some implementation information may need frequent review, while population outcomes may be available only annually or less often. The aim is timely learning and action, without collecting more information than the group can use responsibly.

How do you use the evaluation system to answer key questions?

With your evaluation team, decide what evidence each question requires, where it will come from, and how you will analyze it. Agree on definitions, responsibilities, privacy protections, and the decisions the findings will inform. Use both numerical information and accounts of people's experiences when they address different parts of the question.

Graphs can help reveal patterns and relationships. Figure 2 illustrates a cumulative count of documented community changes, such as new programs or policies related to an initiative's mission. Each recorded change is added to the previous total. A flat line means no additional changes were recorded during that period; a steeper line means more changes were recorded per unit of time. The total does not show whether changes remained in place or how important they were. Track reach, quality, and sustainability separately, and avoid counting the same change more than once.

Figure 2. Example cumulative record of community changes associated with the South Side Youth Development Initiative, 2008–2014. [Source: Example graph from the KU Center for Community Health and Development's Community Check Box]

Adding the dates of important inputs and events can help you explore possible influences on the pattern. Figure 3 illustrates this approach by placing milestones alongside a cumulative record of community and system changes.

The events identified in the example include:

• Formation of a planning team
• Appointment of a new director
• Implementation of a social marketing action plan
• Hiring of additional staff
• Action planning for the initiative
• A disruption in leadership

These events may have influenced the pace of documented change, but their timing alone cannot establish their effects. Review other records and participants' accounts to investigate possible explanations.

Figure 3. Historical example of cumulative community and system changes and associated events for the Lincoln Coalition for Healthy Living, 1999–2006.

The example invites questions about whether staffing and leadership changes were related to changes in the pace of activity. It also raises other possibilities: hiring staff might improve documentation as well as implementation, and leadership transitions might affect both. Examine these explanations before concluding that a particular event caused an increase or decrease. This analysis can help identify practical steps to strengthen the work.

You can also graph outcomes, compare patterns across neighborhoods or groups, or examine trends in relevant comparison communities. Choose measures based on the logic model and evaluation questions. Use clear labels and comparable time periods, and take care with graphs that use different scales, since their appearance can exaggerate relationships.

Decide which measures are most useful before collecting large amounts of data. Once the information is available, look for patterns, exceptions, and competing explanations. Keep the central purpose in view: what can the group learn that will help it achieve its goals more effectively and fairly?

The following five questions build on the framework in Figure 1. The first four examine documented changes and relationships; the fifth addresses the stronger question of whether the initiative caused improvements.

1. Is the initiative helping bring about community or system changes related to its mission? Identify specific changes in programs, practices, and policies. Document when they occurred, who contributed, what the initiative did, and what evidence supports the account. Relevant systems might include public health, education, public assistance, public safety, or environmental protection.

Types of changes to document may include:

• New or modified programs. Examples include a new youth development program, an expanded vaccination service, or a resident-led prevention initiative. Distinguish establishing or changing a program from repeatedly delivering its existing activities. Record the program's purpose, reach, start date, and the partners involved.
• New or modified practices. Examples include a clinic extending its hours or an organization changing its procedures to improve language access. Document whether the practice was implemented, who could benefit, and whether it continued.
• New or modified policies. Examples include changes in organizational rules, public regulations, or resource allocation policies. Distinguish policy adoption from implementation and enforcement. A new service location or store opening is a community change, but it should not automatically be classified as a policy change.

Depending on the initiative, you may also track organizational developments, media coverage, resources obtained, partnerships, and other important events. Keep these categories distinct so that fundraising, publicity, and implemented community changes are not treated as equivalent accomplishments.

Record both the timing of changes and the sequence of actions that preceded them. Meeting records, policy documents, implementation dates, and accounts from different participants can help clarify the initiative's contribution. A change occurring after an initiative's activity is a starting point for investigation, not sufficient evidence that the activity caused it.

Some changes take time to affect outcomes, and several changes may work together. Track whether they were implemented, whom they reached, and whether they remained in place. This can help explain why an outcome did or did not change within the evaluation period.

Look for patterns in the rate and type of documented change. Do they align with action planning, new partnerships, staffing, funding, or outside events? Check whether changes in documentation practices could explain part of the pattern. Use the findings to identify questions for follow-up and possible improvements.

An initiative might record relatively few changes while building relationships and planning, followed by more changes during implementation. Later, the focus may shift from starting new work to sustaining existing programs or policies. This is one possible pattern, not a required sequence or a universal standard of success.

If the cumulative count levels off, investigate why before trying to increase activity. The group may need more capacity, may have encountered barriers, or may be successfully maintaining important changes. The appropriate response depends on what the evidence shows.

Also ask whether the changes are substantial enough and sufficiently well implemented to affect the intended outcomes. Counting changes is useful, but understanding their relevance, reach, and quality is essential. Identify gaps in the records and consider how to obtain a more complete account without creating unnecessary reporting burdens.

2. What factors or processes are associated with community or system change? Monitor conditions that may influence implementation and outcomes, then investigate the relationships suggested by the information.

• Assessment, planning, implementation, and evaluation processes. Examine how action plans are developed and used, how community members influence decisions, and how documentation and feedback support the work. Consider whether responsibilities, resources, and timelines are realistic. Review the roles of staff, volunteers, leaders, and governing bodies without assuming that delays reflect individual failure. The Community Tool Box's "Promising Approaches" materials offer related ideas to explore.
• Participation, reach, and experiences. Examine who participated, who was missed, how much support people received, and whether it met their needs. Investigate why people stopped participating, including scheduling, accessibility, cost, or dissatisfaction. Distinguish being reached from benefiting. The "Solve a Problem" guide on insufficient community participation can support this discussion.
• Significant program and community events. These may include:
  • Leadership transitions and changes in decision-making responsibilities.
  • Staffing changes, vacancies, training, or changes in available skills.
  • Changes in methods or direction, including intentional adaptations to community priorities and unplanned changes caused by capacity or coordination problems.
  • Milestones such as adopting an action plan, starting or revising a program, completing training, or achieving an objective.
  • Increases, reductions, delays, or changes in the conditions of funding.
  • Unexpected operational challenges and responses, including safety concerns, financial pressures, changes in service hours or locations, and changes in eligibility or capacity.
  • Major changes in conditions, such as disasters, disease outbreaks, or disruptions to essential services.
  • Other community developments, such as:
    • Changes in public leadership or government administration.
    • Incidents of discrimination, community tensions, or efforts to address inequitable treatment.
    • New employers or investments, including their effects on employment, public resources, and affordability.
    • Employer closures, job losses, or other economic disruptions.
    • Community losses, acts of violence, celebrations, or recognition of local contributions.

Broader changes in government procedures, funder priorities, service systems, or public attitudes may also affect the initiative. Record relevant developments even when they are outside the group's control.

As Figure 3 illustrates, aligning events with patterns of documented change can suggest relationships to investigate. Use additional evidence to assess those explanations, then consider actions that could strengthen implementation or reduce harmful consequences.

3. How do community and system changes contribute to the initiative's goals? Examine the nature and distribution of changes, whether or not the initiative was their only source. Ask how they affect the conditions your group seeks to improve.

Look at where changes are concentrated and where they are missing. Are most changes occurring in schools, while other settings receive little attention? Are particular neighborhoods or groups benefiting more than others? Do the changes address all major objectives or mainly those that are easiest to achieve? Compare the pattern with community priorities and the logic model. Useful dimensions include:

• Goals and objectives. Group changes according to the outcomes they are intended to support. A heart health initiative might address physical activity, tobacco use, access to nutritious food, and access to appropriate preventive care.
• Intervention strategies. Distinguish changes involving information, services, social support, physical environments, organizational practices, or public policy.
• Risk and protective factors. Consider conditions that may increase or reduce the likelihood of an outcome. Examples include exposure to violence, supportive relationships, stable housing, and access to care. These factors describe probabilities and circumstances, not a person's destiny, and should not be used to label families or communities.
• Duration and sustainability. Distinguish a one-time event from an ongoing service or policy. Record how long a change remains in place and what resources or commitments are needed to sustain it.
• People reached and people benefiting. Identify the intended population and the people actually reached. Where evidence permits, examine who experienced benefits or harms. Do not assume that a policy's stated coverage establishes equal access or benefit.
• Community sectors. Changes may involve government, health care, education, labor, business, housing, faith communities, or other settings. Consider whether the sectors engaged match the initiative's objectives.
• Levels of influence. Examine whether changes focus on individuals, relationships, organizations, community conditions, or broader policy. Many initiatives work across several levels; for example, a physical activity effort may combine personal support with changes in public spaces and transportation policy.
• Place. Examine the distribution of changes across neighborhoods, municipalities, counties, rural areas, or other relevant locations.

Discuss what these patterns mean with community members and partners. They may reflect opportunities, unequal resources, access barriers, or the interests of particular organizations. Decide whether adjustments are needed based on the importance and reach of the changes, not simply whether every category has the same number.

Figure 4. Example distribution of documented community accomplishments (N = 125) by the primary sector in which each change occurred.

This chart can support discussion of which sectors are represented in the record. It does not, by itself, explain why changes were more common in one sector or demonstrate their relative value. If changes span several sectors, explain how the primary sector was assigned. The "Solve a Problem" troubleshooting guides offer additional support for considering adjustments.

4. Are community and system changes associated with improvements in population-level outcomes? Select indicators that reflect the intended outcomes for a defined population. Community-level indicators describe patterns across groups, but they are not automatic proof of success or failure. Potential data sources include:

• Census and other population data, such as educational attainment, housing conditions, and household income.
• Public health data, such as infant mortality rates and vaccination coverage among eligible children.
• Appropriately aggregated records from health and human service organizations, such as service reach, waiting times, or outcomes. Access may require permission or an agreement, and individual privacy must be protected.
• Statewide information and data from suitable comparison communities, using compatible definitions and time periods.
• Transportation, police, or court data relevant to the question, interpreted with attention to reporting practices and enforcement.
• Educational information, such as attendance, school completion, learning progress, school climate, and reported violence.
• Environmental information, such as pollutant levels, air quality, water quality, and access to open space.

In addition to archival data, you can use surveys, observations, interviews, and other methods. For example, repeated surveys may examine reported alcohol, tobacco, or other substance use, or specific sexual health practices. Structured observations may track use of public spaces. Service records may describe diagnoses or access to care, although they reflect who obtained services and was assessed. Keep definitions and collection methods consistent, and identify changes in participation or coverage.

Choose indicators that closely match the question. If you want to understand alcohol use among adolescents, a suitable population survey is more direct than a count of traffic violations. Alcohol-involved crashes may be relevant to alcohol-related harm, but nighttime single-vehicle crashes should not all be assumed to involve alcohol. Explain the limitations of any indirect measure.

Graphing outcomes alongside the timing of community changes can help identify patterns for further analysis. A similar upward or downward trend does not establish a causal relationship, particularly when one series is a cumulative count that can only increase.

Figure 5 provides a historical illustration involving community changes intended to prevent child sexual abuse and reported cases in Minnesota. The figure supports discussion of an observed association; it does not establish a reduction in the actual occurrence of abuse or identify the cause of the reported trend.

Figure 5. Historical illustration of an association between documented community changes and a decrease in reported cases of child sexual abuse in Minnesota.

The accompanying example describes a 28% decrease in reported cases over four years as documented community changes accumulated. Interpreting that pattern requires knowing how cases were defined, counted, and reported, and whether those practices changed. Fewer reports may reflect less abuse, reduced identification or reporting, or other factors. The pattern alone cannot establish that the initiative caused the decrease.

Reported cases may increase when people become better able to recognize abuse, seek help, or make reports. A decrease can also occur when access to trusted adults or reporting systems is disrupted. Examine reporting practices, service access, prevention implementation, and other evidence before interpreting either direction as success or failure. Continued tracking is useful, but it must be accompanied by careful investigation of alternative explanations.

5. Did the initiative cause improvements in population-level outcomes?

This question requires more than a graph showing that outcomes changed after implementation. The evaluation needs a credible way to assess what would likely have happened without the initiative. Plan for this question early and seek evaluation expertise appropriate to the setting and available data.

You may track several outcomes, such as traffic injury rates, experiences of violence, unintended pregnancies among adolescents, or vaccination coverage. Define each measure carefully and use comparable observations before and after implementation. Consider the expected delay between activities and outcomes, who was reached, and whether the initiative was implemented as intended.

Examine other plausible explanations, including existing trends, other programs, policy changes, population shifts, and changes in measurement. Where appropriate, experimental or quasi-experimental designs can strengthen causal conclusions. For example, an interrupted time-series design examines repeated observations before and after an intervention; adding a suitable comparison series can help address alternative explanations. These approaches require enough observations, appropriate analysis, and attention to their assumptions. If the evidence cannot establish causation, report what it supports about association or contribution and explain the remaining uncertainty.

Begin early and sustain evaluation for as long as it remains useful

Build evaluation into the initiative's regular work, with a realistic plan for collection, review, and action. Continue following important outcomes long enough to learn whether changes last, including after the initiative ends when that is feasible and appropriate. Review the burden and value of data collection periodically. The aim is a sustainable learning process that informs decisions, not indefinite collection without a clear purpose.

Using evaluation questions to guide improvement

Key evaluation questions help connect the initiative's activities and outputs with changes in programs, policies, practices, and longer-term outcomes. They also help clarify which conclusions the available evidence can support.

The five questions discussed in this section are:

• Is the initiative helping bring about community or system changes related to its mission?
• What factors or processes are associated with community or system change?
• How do community and system changes contribute to the initiative's goals?
• Are those changes associated with improvements in population-level outcomes?
• Did the initiative cause improvements in population-level outcomes?

Answering these questions requires clear definitions, useful data, careful interpretation, and meaningful community participation. Graphs can make patterns easier to see, while records, interviews, and appropriate study designs help explain them. Share findings and limitations openly, agree on practical changes, and review whether those changes improve the work.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: Community Tool Box.

Here you will find a checklist summarizing the important points of the section:

What do we mean by answering evaluation questions?

__ Evaluation questions refer to what stakeholders – the community and funders, for example – want to know about the functioning of the program or initiative.

Some illustrative evaluation questions:

__ Are participants satisfied with the program? (Process and Implementation Issue)

__ How well is the initiative meeting its stated objectives? (Attainment of Objectives)

__ How much and what kind of difference did it make for participants? (Impact on Participants)

__ How much and what kind of difference did it make on the community? (Impact on the Community)

Why answer the key questions?

__ To improve your work.     

__ To understand what affects the work in what ways.

__ To see how to accelerate results.

__ To understand specifically how broader actions, events, or conditions – e.g., a crisis or concentrated poverty – affect the work.

__ To understand what works to bring about community change, and adjust accordingly.

__ To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effects.

__ To show the community the value of your work.

Who should be involved in answering these questions?

__ This type of evaluation works best as a participatory effort.

Those who might be involved include:

__ Participants in or beneficiaries of the effort.

__ Residents of a geographic area you’re focused on.

__ Professionals and volunteers carrying out the work.

__ Those whose jobs or relationships bring them into contact and involvement with the population and/or issue you’re concerned with.

__ Funders and local officials.

When should you set up and use an evaluation system to answer key questions about the effort?

__ Start at the very beginning of the effort, so you can record the whole of your process – outreach, planning, implementation, and evaluation.

__ Continue gathering, recording, analyzing, and using data throughout the course of the effort.

How do you use an evaluation system to answer key questions about the effort?

__ Decide what information you need to answer each question.

__ Decide how to gather that information.

__ Devise a method for recording and setting up your data that makes it easy to analyze.

__ Graphing is particularly good because it allows you to easily compare different sets of data.

__ Consider each question separately:

Is the initiative serving as a catalyst for community/system change related to its mission?

Changes to look for include:

__ New or modified programs.

__ New or modified practices.

__ New or modified policies.

What factors or processes are associated with the rate of community or system change?

__ The processes by which you conduct the assessment, planning, implementation, and evaluation aspects of the effort.

__ People served or benefited.

__ Significant program events. These might include:

• Changes in leadership.
• Staffing changes.
• Changes in method or direction.
• Milestones of the effort.
• Increases or cuts in funding.
• Unforeseen circumstances.

__ Community events, such as:

• A change of local political leadership or government administration.
• Racial or ethnic conflict.
• An economic windfall.
• An economic downturn.
• A community tragedy or celebration.

__ Changes in broader conditions.

How are community/system changes contributing to efforts to promote community health and development?

__ Examine community changes in relation to various aspects of your work.

• Group goals or aims.
• The strategies of intervention you’re using.
• Risk and protective factors.
• The expected duration of change.
• The populations benefited.
• The sectors addressed.
• The ecological level addressed.
• Place.

Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort?

__ Community-level indicators of an issue are markers of success at the level of the community as a whole, rather than for particular individuals.

__ Population-level outcomes can be found or inferred by consulting publicly available statistics and records, including:

• Census data.
• Public health statistics.
• Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.
• Statewide data and data from other communities (for comparison purposes).
• Police and court files.
• Educational data – standardized test averages, truancy, dropouts, high school completion rates, incidence of school violence.
• Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc.

__ You can also collect other sources of data, using observation, surveys, and other methods.

__ Graphing and comparing sets of data on community-level indicators can show connections among them.

Did the effort lead to improvements in population-level outcomes?

__ The effort may have led to improvements by influencing changes in the community or system (intermediate outcome) and (more distant) community-level indicators of success. that brought them about.

__ Comparing data on population-level outcomes, community changes, and the timing of various phases and events of your effort can help you understand the connections among them.

__ Begin early and continue to gather, record, analyze, and use data indefinitely to understand how to adjust your effort for greatest effectiveness.

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What does providing evaluation feedback involve?

• Why provide feedback?

• When should you provide feedback?

• How should you provide feedback?

What does providing evaluation feedback involve?

Providing evaluation feedback means sharing findings with people involved in or affected by an initiative and discussing what those findings mean for the work. This includes participants, community members, staff, volunteers, leaders, and governing board members. Depending on the information and the audience, you might use tables, graphs, written summaries, examples of participant work, or audio and video. Share achievements, challenges, and uncertainties in formats people can understand and use.

Feedback is an exchange, not simply a report from evaluators or managers. People carrying out the work can use findings to understand what appears to be working and what needs attention. Their experiences can also help explain the results. Leaders and board members need to hear about organizational conditions that support or limit effective work.

Are staffing, supplies, time, and funding adequate? Do people have the training and support they need? Are participants welcomed and able to access services? Can staff, volunteers, and community members raise concerns without being dismissed or penalized? Discuss these questions alongside results so that feedback leads to appropriate changes in organizational support as well as practice.

Share feedback throughout the initiative, at times when people can act on it. A final evaluation report can contribute to learning, but it should not be the first opportunity to discuss important findings. Explain what decisions are open for discussion and how people can influence them.

This section focuses mainly on feedback from monitoring: the ongoing tracking of activities, implementation, and accomplishments that may contribute to longer-term outcomes. Monitoring can guide improvements, but activity counts alone do not establish the initiative's impact.

Why provide feedback?

Regular feedback can help your initiative:

• Assess progress toward shared goals with community members, participants, and organizational leaders
• Identify areas that need additional attention, resources, or support
• Recognize when activities are taking resources away from agreed priorities
• Recognize contributions and celebrate meaningful accomplishments
• Connect recent activities with longer-term progress, while considering the quality and reach of accomplishments
• Involve funders in discussions about resources and adjustments that support the initiative's mission and community priorities
• Give funders and partners a credible account of achievements, unmet needs, and opportunities for continued support

When should you provide feedback?

Set a schedule that matches the pace of the work, the availability of useful information, and the decisions your group needs to make. For example, monthly discussions may help during early implementation, with quarterly reviews once activities are more established. Review urgent findings sooner, particularly when they concern participant safety or serious barriers to access. At each session, revisit previous decisions and ask what happened after changes were made. Allow enough time for expected results to emerge before judging an adjustment.

How should you provide feedback?

Begin with a clear, balanced statement about how the initiative is doing

Summarize the most important findings, including both progress and concerns. Be specific about the evidence. For example, "We received 35 letters from parents describing positive experiences with the school safety committee this semester" is more precise than "Parents are pleased with the committee." It also avoids implying that the letters represent every parent's experience. Explain the period covered and any important gaps in the information.

Findings will often be mixed. Some activities may be working well while others need revision or more support. Discuss problems directly and respectfully, focusing on practices and conditions rather than labeling people. Participants may feel disappointed, concerned, or frustrated; allow space for those responses without minimizing the findings. The purpose is to understand the situation and decide what to do next.

Frame feedback as an opportunity for shared learning and action

Explain how the information can help the group make decisions. Recognize the effort people have contributed, while being honest about what the evidence shows. Invite questions about the findings and the methods used to obtain them. Maintain confidence in the group's ability to learn and respond without promising success or presenting every result as positive. Findings about harm, exclusion, or serious implementation problems should receive prompt attention rather than being softened to preserve morale.

Use graphs where they clarify the findings

One useful sequence connects implementation with outcomes. Label each graph clearly, explain its time period and data source, and provide an accessible text or table alternative:

• Process and output measures — How activities were carried out, who participated, participants' experiences, and the amount of work completed in relation to the initiative's objectives
• Intermediate outcomes — New or modified programs, policies, and practices, including whether they were implemented and whom they reached; discuss relevant contextual events alongside these measures
• Longer-term outcomes — Changes in behavior, access, conditions, or well-being among participants or across the community; distinguish observed changes from evidence that the initiative caused them

Share narratives and other forms of evidence

Interviews, participant comments, observations, and examples of work or learning can explain experiences that counts and graphs do not capture. Present these materials with enough context to make their meaning clear. Select examples that reflect the range of findings, including differing experiences, and obtain permission for identifiable quotations, images, or recordings.

These materials can inform questions about implementation, intermediate outcomes, and longer-term change. Analyze them systematically by identifying recurring themes, differences, and exceptions. Numerical data also require interpretation, and qualitative findings can be presented visually when useful. A trend is a pattern of change over time; it does not automatically indicate whether a goal has been achieved.

Discuss what the material helps explain. Which actions, services, or conditions do participants describe? What do examples of their work show about learning, and what do they leave uncertain? Which accounts reveal barriers or unintended effects? Avoid treating one compelling story as representative of everyone, while recognizing that an uncommon experience may still identify an important concern.

Much of this is qualitative data: descriptions, experiences, and interpretations that help explain how and why things happen. Some aspects can be counted, but counts should not replace the context and meaning that make the information useful.

Discuss and interpret the information together

Use these practices to support a productive discussion:

• Begin with the purpose of the discussion and a balanced overview of the most important findings.
• Ask how group members interpret the information. Explore differences in interpretation and identify what additional evidence could help; agreement is not always possible or necessary.
• Use a recent, specific example to connect the findings with day-to-day work.
• Identify strengths, challenges, and uncertainties supported by the evidence.
• Discuss patterns over time, differences among groups, and changes in data collection that might affect the results.
• Consider possible adjustments, including the resources and organizational support needed to carry them out.
• Review the examples behind the graphs and qualitative findings, and check whether they support the conclusions being discussed.
• Recognize demonstrated progress and contributions without overstating what they establish.
• Invite remaining questions and provide another way to contribute for people who prefer not to speak in the group.

If a measure does not show the expected progress:

• Check what the measure represents, the intended direction of change, and whether enough time has passed. A higher number is not always better, and a stable result is not automatically a failure.
• Review the longer-term pattern and relevant context. For example, funding receipts may cluster around award dates. Use documented patterns to explain variation rather than assuming every setback is temporary.
• Ask what may need to change and whether the issue involves the approach, implementation, resources, access barriers, or outside conditions.
• Check for missing or duplicate records and inconsistent definitions. Add previously unrecorded activities only when they can be verified, document corrections, and keep the same counting rules rather than changing the record to make results look better.

Close with shared conclusions and clear next steps

Summarize what the group learned, where interpretations differ, and what remains uncertain. Recognize strengths while stating concerns that require action. Review the evidence most relevant to the initiative's mission, then agree on specific steps, responsible people, needed resources, and a date to review progress. Share a brief record of decisions and explain how participants' feedback influenced them. People should leave understanding what happens next and what support is available.

Making feedback useful

Keep these principles in mind when preparing and discussing evaluation feedback:

• Share findings as information for learning, accountability, and decisions.
• Be respectful and encouraging while reporting strengths and problems honestly.
• Build confidence in the group's ability to respond without guaranteeing outcomes.
• Turn findings into specific adjustments, with responsibility, resources, and follow-up.
• Connect decisions to the initiative's mission and the priorities of people affected by its work.

Useful feedback combines clear evidence, thoughtful listening, and follow-through. It helps people understand what is happening and what they can change, including where leaders need to provide additional support. Return to earlier decisions, report what happened, and keep the conversation open as the initiative learns and develops.

What does it mean to provide feedback to improve your intiative?

___Providing feedback from your evaluation involves presenting the data on accomplishments that you're getting from your evaluation to those involved in the initiative - your staff and volunteers

___Most of this information is usually, but not always, presented in graph form

___Providing feedback should be done on an ongoing basis so that your staff and volunteers can be kept up-to-date on what they're doing well and what can stand improvement

___Providing feedback can also be done at the end of an evaluation (although not exclusively so)

___Your staff should get feedback at regular intervals, especially early in the initiative's development, so that they can continually adjust their efforts to improve the initiative's results

What are some reasons to consider providing feedback?

___To help community leadership assess progress towards meeting the initiative's goals

___To help see areas wherein the members of the initiative may want to put more energy

___To help detect when too much energy or effort is spent in areas less central to the mission

___To provide the opportunity to celebrate small accomplishments

___To help the initiative focus on the "big picture" by seeing cumulative accomplishments over time

___To provide funders the opportunity to help re-direct the initiative towards activities more directly related to the mission

___To provide funders the opportunity to see and reward the accomplishments of the initiative

How should you provide feedback?

___You began with an affirming and specific overall statement summing up how the initiative is doing

___You've presented the data to the group as a good thing, even if the feedback isn't all that positive

___You've tried to communicate the value of a group that really wants to know how it's doing and provided a shared vision of the initiative as a catalyst for change

___You've shown graphs and qualitative material in this order:

• Process Measures
• Intermediate Outcomes
• Ultimate Outcomes/Impact Data

___You’ve asked the group for its interpretation of the data

___You've introduced the information and provided a brief definition, example, and explanation of why it is important

___You've given a more detailed example from accomplishments that have happened recently - within the last few months, if possible

___You've pointed out what is positive about the data

___You've made note of any trends in the data, and what those trends mean

___You've discussed any suggested re-direction of efforts

___You've reviewed examples listed on the graph

___You've affirmed the initiative by noting evidence of progress

___You've asked for comments or questions from the group

How should you present the information If the measure hasn't shown recent growth?

___Ask the group for its ideas about why this is so

___Point out any previous times that growth occurred to keep people from getting discouraged

___If the measures are cyclical in nature and it seems appropriate, point this out

___When meeting with the initiative's leadership, ask if there were additional activities that were not recorded and remind leaders of the definition and list items that have been recorded over the life of the initiative

___Summarize the data by discussing strengths of the initiative

Example 1: Sharing Information through Participatory Evaluation—Enroll Wyandotte: A Community Mobilization Effort for the Affordable Care Act

Background

The passage of the Affordable Care Act (ACA) created for the first time the opportunity for Americans to receive health insurance that previously could not afford it. Healthcare insurance marketplaces were established to ensure healthcare access to those in need. State-level support for ACA marketplace enrollment varied considerably from the development of state-managed health exchanges to erecting barriers to impede marketplace access. In Kansas, local efforts were required to assure access to healthcare through community mobilization and partnerships. Wyandotte County (Kansas City), Kansas (pop. 160,384) is a low-income and culturally diverse area that ranks 96 out of 98 in the county health rankings. One in four residents are without health insurance, a rate twice that of the national average, with an equal number of residents below the national poverty level.

Context

Enroll Wyandotte, a local ACA exchange navigation effort, engaged multiple partnerships across a different sectors to support enrollment. Collaborative partnerships included government health organizations, public institutions, hospitals and health organizations, community and cultural organizations, community health coalitions, and universities.

Efforts of the Initiative

The initiative employed a community mobilization approach to assist Wyandotte County residents with obtaining health insurance through the ACA marketplace exchanges. This effort consisted of four broad components: a) Providing information and enhancing skills (eg, one-on-one education through the walk-in enrollment center, door-to-door education referral campaign to identify eligible community residents); b) Modifying access, barriers, exposures, and opportunities (eg, one-on-one enrollment assistance through the walk-in enrollment center, conducted community enrollment events in collaboration with community partners); c) Enhancing services and support (eg, hired a full-time navigator/project coordinator, trained a workforce of university volunteers as Certified Application Assistors); and d) Modifying policies and broader systems (eg, established collaborative partnerships with community organizations, provided testimony to senate committees on legislation that would have required expensive certification of volunteers trained as Certified Application Assistors).

Participatory Evaluation

A participatory evaluation involved the leadership of the Enroll Wyandotte effort and staff of the Center for Community Health and Development at the University of Kansas. The partners developed a monitoring and evaluation plan that identified some key indicators for success: a) Community/organizational changes (ie, new or modified programs, policies, or practices to facilitate enrollment in the ACA; eg, expanded enrollment hours at the health department; new sites at local public libraries, community organizations); b) Services provided (eg, media communications to promote awareness, direct assistance in enrollment sites); and c) Created accounts (the total number of family accounts created within the ACA website at three enrollment sites).

Measurement and Results

The KU Center for Community Health and Development designed an online documentation and support system (the Community Check Box) that included prompts and supports to: a) Capture instances of community/organizational changes and services provided (ie, what was done when and by whom toward what goal; b) Code by type of activity (ie, using definitions of community/organizational change and services provided); c) Characterize the activity (eg, by goal, strategy used, sector in which implemented); and d) Communicate progress (eg, graphs of the unfolding of community/organizational changes over time). The results showed that activities that created lasting changes in the environment (eg, programs, policies, and practices) and services provided were associated with increased enrollment. More than 500 new accounts were created as a result of the work of the initiative.

Providing Feedback through Participatory Sensemaking

Staff from the KU Center for Community Health and Development reviewed documented activities captured using the Community Check Box to assure data quality and to provide feedback on a monthly basis. The Check Box also supported participatory sensemaking by pairing graphs of activities/outcomes with reflection questions: a) What are we seeing? b) What does it mean? and c) What are the implications for adjustment? Enroll Wyandotte and KU Center for Community Health and Development staff met on two occasions to review the data and apply these questions. Participatory sensemaking provided an opportunity to evaluate data patterns to check for data completeness and identify key events (eg, when grants were received, when essential community/organizational changes occurred) that were essential to implementation of the marketplace navigation initiative. For example, the illustrating the unfolding of services provided over time showed a marked increase at the end of December, 2013. Enroll Wyandotte staff identified that this was associated with the hiring of a project coordinator made possible by community foundation funding and collaboration with community partners. Additionally, the use of sensemaking included the leveraging of additional funding from grantmakers and community partners through the use of data from the evaluation. In this initiative, providing feedback through participatory sensemaking helped guide systematic reflection of the data to make adjustments for continuous improvement and to communicate an objective story to community partners, grantmakers, and other stakeholders.

This example is based on the following article:

Fawcett, S. B., Sepers, C. E., Jr., Jones, J., Jones, L., & McKain, W. (Submitted for publication) Participatory evaluation of a community mobilization effort to enroll residents of Wyandotte County, Kansas through the Affordable Care Act.

A Community Health Center conducted an evaluation of a program designed to support physical activity and other health-promoting behaviors among people at increased risk for heart disease.

The evaluation showed mixed results.

About 15% of participants experienced substantial improvements. They increased their physical activity, made changes in nutrition, improved their fitness, lost weight, and experienced reductions in blood pressure and pulse rate. They also reported feeling more energetic and expressed high satisfaction with the program.

About 70% participated in physical activity, but less consistently than the program had anticipated. Health outcomes varied within this group. Some participants experienced modest reductions in blood pressure, while others maintained approximately the same levels they had when they entered the program.

The remaining 15% included people who left the program early and people who participated only minimally or irregularly. Most showed little measurable change in weight, blood pressure, or self-reported well-being, although a small number experienced positive outcomes.

What could the Community Health Center learn from these results?

The evaluation suggested that some participants benefited substantially, while others encountered barriers to participating consistently or did not experience the intended outcomes.

Rather than concluding simply that the intervention worked for people who “stuck with it,” the Center could examine how the program's design, accessibility, supports, implementation, and participants' circumstances contributed to these different experiences.

This chapter has discussed the elements of conducting a research-based evaluation. Evaluation, however, is not only about determining whether a program succeeded or failed.

It is a tool for understanding what happened, why results may have differed across participants or settings, and how an intervention can be improved.

In this section, we examine how evaluation findings can be used to strengthen programs and community interventions.

What do we mean by refining the intervention?

Evaluation findings can help you critically reflect on your work and identify opportunities for improvement.

Some useful reflection questions include:

• What are we seeing? What activities were implemented? Who participated? What changes, patterns, differences, or trends appear in the data?
• What might explain what we are seeing? What evidence suggests that the intervention, implementation process, community conditions, or other factors contributed to the results?
• What are the implications for improvement? What should be sustained, adapted, strengthened, discontinued, or examined further?

The questions you ask will depend on the nature of the intervention and the purpose of the evaluation.

Consider bringing together participants, staff, community partners, evaluators, and others with relevant experience to review findings.

Graphs, charts, qualitative findings, participant feedback, and other accessible formats can help people identify patterns and discuss what those findings may mean.

A facilitator may help the group consider different interpretations, identify unanswered questions, and develop recommendations.

Refining an intervention means using evaluation findings and community knowledge to make the work more effective, equitable, relevant, accessible, and sustainable.

Depending on what you learn, you might:

• Strengthen parts of the intervention that appear to be effective
• Adapt parts that are not working as intended
• Remove components that provide little benefit
• Add supports that address barriers to participation
• Adjust the intervention as community conditions, needs, or priorities change
• Examine whether different groups are experiencing different levels of access, participation, benefit, or harm

Meet with people who are involved in or affected by the intervention to review the findings, identify priorities for improvement, and determine what changes should be tested.

Evaluation results should inform decisions, but they should not be interpreted in isolation from participant experience, implementation conditions, and other relevant evidence.

In the Community Health Center example, staff, participants, and evaluation partners reviewed the findings together.

They noted that people who participated regularly often experienced improvements in health, but they also asked why regular participation was easier for some people than for others.

Their questions included:

• What features of the program helped participants engage consistently?
• What barriers made participation more difficult?
• Did transportation, work schedules, caregiving, cost, disability access, program timing, social support, or other conditions affect participation?
• Why did some participants experience improvements in blood pressure even when their physical activity changed only modestly?
• Should the program include stronger nutrition, stress-management, medication-support, social-support, or other components?
• Were particular groups experiencing different outcomes, and if so, what might explain those differences?

By asking what happened, what may explain the findings, and what should change, the Center can make more informed decisions about the next version of the program.

Why should you use evaluation findings to refine the intervention?

One of the most important purposes of evaluation is learning how to improve.

If an intervention is not producing the intended results, the choice is not simply between ending it and trying something else at random.

Evaluation can help identify:

• What is working
• What is not working
• For whom different parts of the intervention appear to work
• Which barriers are affecting participation or outcomes
• Whether the intervention was implemented as intended
• Whether the underlying theory or assumptions need to change
• Whether unintended benefits or harms have occurred

Using evaluation findings can help an organization make better-informed decisions about where to invest time, staff, funding, and other resources.

It can also strengthen accountability to participants, community members, funders, and partners by showing that the organization is willing to learn from evidence and change when necessary.

Most importantly, evaluation can help improve the likelihood that the intervention contributes to the outcomes that matter to the community.

When should you refine the intervention?

Monitoring, learning, and improvement should occur throughout the life of a program or initiative.

The appropriate review schedule depends on:

• The length of the intervention
• How quickly meaningful outcomes can reasonably be expected
• The level of risk associated with the intervention
• The availability of data
• Funding or reporting cycles
• Changes in community conditions

Some programs conduct a major review annually, but there is no universal schedule.

Smaller adjustments can often be made sooner when monitoring data, participant feedback, implementation experience, or emerging evidence identifies a clear need.

At the same time, avoid making major judgments before the intervention has had a reasonable opportunity to produce the outcomes it was designed to influence.

The challenge is to avoid both ending a promising effort too early and continuing an ineffective or harmful approach longer than the evidence supports.

Consider refinement when the intervention is not producing the intended results

If available evidence suggests that outcomes are not improving as expected, examine why.

Possible explanations may involve:

• Implementation
• Program design
• Participation barriers
• Insufficient intensity or duration
• Incorrect assumptions about what produces the desired outcome
• Changes in external conditions
• Measures that do not adequately capture important changes

When participation decreases or people leave the program earlier than expected

Do not assume that people leave because they are unmotivated.

Ask what may be affecting continued participation.

Possible factors include:

• Scheduling
• Transportation
• Cost
• Childcare or caregiving responsibilities
• Accessibility
• Work demands
• Language
• Safety
• Program relevance
• Relationships with staff or other participants
• Whether the program feels respectful and responsive
• Changes in participants' circumstances

Between cycles of a time-limited or recurring program

Programs that serve new groups over repeated cycles can use each round as an opportunity to learn.

Compare results across cycles, document changes that were made, and examine whether those changes improved access, implementation, participant experience, or outcomes.

When participants, community partners, staff, or funders raise concerns

Requests for change should be taken seriously.

Evaluation data can help determine whether a requested change is supported, whether additional information is needed, and what effects the change might have.

When funding or other resources change

Reduced resources may require difficult decisions.

Evaluation can help identify:

• Essential program components
• Components that could be redesigned more efficiently
• Changes likely to cause the least harm
• Services or supports whose loss could disproportionately affect participation or outcomes

When community conditions or priorities change

Economic conditions, public health needs, policies, housing conditions, transportation, community safety, technology, population changes, or other factors may change the context in which an intervention operates.

Evaluation and community input can help determine whether the program should adapt.

Who should be involved in refining the intervention?

Include people whose experience, knowledge, responsibilities, or decisions are relevant to the intervention.

This will vary by program, but important participants often include:

• People participating in or directly affected by the intervention. They can describe what participation is actually like, what supports or barriers they experience, what feels useful, and what changes may make the intervention more relevant or accessible.
• Staff members and volunteers. They can describe implementation challenges, workload, relationships, resource needs, and practical consequences of proposed changes.
• Community partners and others who support implementation. Depending on the intervention, these might include educators, public officials, health professionals, employers, funders, community organizations, or other partners.
• People involved in the evaluation. They can help interpret findings, explain limitations, identify unanswered questions, and distinguish what the evaluation can and cannot support.

Participation should be meaningful rather than symbolic.

People directly affected by the intervention should have opportunities to influence decisions about what changes are made, particularly when those changes affect their access, experience, or outcomes.

In the Community Health Center example, the review team included participants with different experiences of the program, health-care staff, organizational leadership, program staff, and an evaluation partner.

The group reviewed quantitative results, participant interviews, implementation records, and relevant research before recommending changes.

How do you refine an intervention based on evaluation findings?

A useful framework is to examine three related areas:

• Process: What did the program actually do, and how was it implemented?
• Impact: What shorter-term changes occurred in the behaviors, conditions, systems, environments, or other factors the intervention was intended to influence?
• Outcomes: What longer-term changes occurred in the health, well-being, development, systems, or community conditions that ultimately mattered?

These areas are related, but they should not be treated as a simple automatic chain.

A program can be implemented as planned and still fail to produce the intended impact.

It can also produce useful outcomes even when implementation differs from the original plan.

Evaluation should help determine what happened and which explanations are most consistent with the evidence.

Process

Process evaluation compares what you planned with what actually occurred.

It can examine both program operations and participants' experiences.

Areas to consider include:

• Community participation. Were the people and organizations you hoped to involve able to participate meaningfully? Who was not reached or included, and why?
• Community assessment. Did the assessment provide the information needed to understand community priorities, strengths, conditions, and barriers?
• Program planning. Was planning participatory? Was relevant research used? Were people directly affected involved in shaping the approach?
• Staff and volunteer recruitment. Did the organization recruit people with the skills, experience, capacity, and community knowledge needed?
• Training and support. Were staff and volunteers adequately prepared and supported throughout implementation?
• Outreach and engagement. Did outreach reach the people the intervention was intended to involve? Were there groups that experienced barriers to participation?
• Implementation. What activities, services, supports, and environmental changes actually occurred? How did participants experience them?
• Evaluation. Was data collection carried out as planned? Were the measures appropriate, accessible, and useful?
• Timelines and milestones. Were major activities completed when expected? If not, what affected the schedule?

Differences between the planned and actual intervention are not automatically failures.

Adaptations may be necessary because of changing conditions, participant feedback, new evidence, resource constraints, or implementation learning.

The important questions are:

• What changed?
• Why did it change?
• Who influenced the change?
• Did the change improve or reduce effectiveness, accessibility, equity, or feasibility?
• Should the adaptation be retained?

Possible reasons implementation differed from the plan

• Activities required more time or resources than expected
• Staff capacity or specialized expertise was insufficient
• Expected funding, partnerships, materials, or facilities were unavailable
• Participation was lower or higher than expected
• Participants or community partners identified a need for change
• New evidence suggested a better method
• Community conditions changed
• Accessibility or language barriers emerged
• Staffing or organizational circumstances changed
• Unexpected emergencies or disruptions occurred
• The original plan was not sufficiently realistic or responsive to community conditions

Understanding why implementation changed can help distinguish between a problem with the intervention itself and a problem with how it was implemented.

Impact

Impact refers to shorter- or intermediate-term changes that may contribute to the longer-term outcomes the intervention ultimately seeks.

Depending on the program, impacts might involve:

• Behaviors
• Knowledge or skills
• Access to services
• Organizational practices
• Policies
• Social or environmental conditions
• Protective factors
• Reduced exposure to risk

In the Community Health Center example, increased access to and participation in physical activity might be an intermediate impact.

Longer-term outcomes could include improved cardiovascular health, quality of life, or reduced risk of chronic disease.

If the intended impact did not occur, examine more than participant behavior.

Ask whether:

• The intervention was accessible
• The activities were relevant and acceptable
• Participants had enough opportunity and support to engage
• Structural or environmental barriers limited participation
• The intensity or duration was sufficient
• The program's assumptions were accurate
• The measures captured the changes that mattered

Examine differences in experience and outcomes

Aggregate results can hide important differences.

When appropriate and ethically collected, examine whether access, participation, experience, or outcomes differ across groups.

Relevant differences might involve:

• Age
• Disability
• Geography
• Language
• Income or economic conditions
• Race or ethnicity
• Gender
• Other characteristics relevant to the intervention

Differences should prompt further inquiry rather than assumptions about the people experiencing them.

Ask whether program design, institutional practices, access barriers, resource distribution, discrimination, environmental conditions, or other factors may contribute to the pattern.

Cultural responsiveness and community fit

If an intervention is not working well for particular communities, avoid describing the problem simply as “cultural influences” that participants failed to overcome.

Examine whether the intervention:

• Reflects participants' priorities and experiences
• Uses appropriate language and communication
• Is accessible within people's daily circumstances
• Respects different cultural practices and perspectives
• Was designed with meaningful community participation
• Addresses social, economic, organizational, and environmental conditions that influence participation

Lack of participation or limited outcomes may reflect a mismatch between the intervention and the community rather than a deficiency in the people the program is intended to serve.

Outcomes

Outcomes are the changes in health, well-being, development, systems, or community conditions that the intervention ultimately seeks to influence.

Some outcomes can be measured during or soon after an intervention.

Others may take months or years to become visible.

Evaluation should therefore distinguish between:

• Short-term changes
• Intermediate changes
• Longer-term outcomes

Avoid assuming that a lack of immediate long-term outcomes automatically means the intervention failed.

Consider whether:

• Enough time has passed
• The expected pathway from activities to outcomes is realistic
• External conditions influenced the result
• Measurement methods were sensitive enough to detect meaningful change
• Outcomes differed among groups
• Important unintended outcomes occurred

When the program produces the intended outcomes

Positive results are an opportunity to understand what should be maintained and strengthened.

Ask:

• Which components contributed most to the results?
• Were benefits experienced broadly or concentrated among particular participants?
• Are there evidence-informed ways to strengthen the intervention further?
• Can successful elements be sustained with available resources?
• Are there people who still face barriers to participating or benefiting?
• Can efficiency improve without reducing quality, accessibility, or equity?

Improvement does not always mean expansion.

Sometimes the most important next step is protecting what works, improving access, or making results more consistent.

When the program produces only some of the intended outcomes

Partial results may indicate that some parts of the approach are useful while others need refinement.

Possible explanations include:

• The intervention addressed only some of the factors influencing the outcome
• Program intensity or duration was insufficient
• Participation barriers limited exposure to the intervention
• Some components were more effective than others
• Implementation varied across sites or groups
• Important services or supports were missing
• Language, accessibility, cultural responsiveness, or program design created barriers
• External conditions affected the outcome
• More time may be needed for longer-term effects to appear

In the Community Health Center example, staff might compare the experiences of people with different levels of participation rather than assuming that motivation alone explains the differences.

They might learn that people who participated regularly had more flexible work schedules, transportation, social support, or access to convenient exercise options.

The Center could then test changes such as flexible scheduling, transportation assistance, virtual or home-based options, peer support, nutrition services, or other supports identified by participants.

These changes should be evaluated rather than assumed to be effective simply because they seem plausible.

When the program produces no intended outcomes

Lack of intended outcomes may require substantial changes, but first examine the evidence carefully.

Ask:

• Was the intervention implemented sufficiently to test the underlying approach?
• Were the measures appropriate?
• Was enough time available for change to occur?
• Were important external influences present?
• Were people able to access and participate in the intervention?
• Was the program theory itself incorrect or incomplete?

Organizations should be willing to redesign or discontinue an intervention when credible evidence shows that the approach is ineffective, harmful, or no longer appropriate.

Time and resources already invested should not be the main reason to continue an ineffective approach.

Examine unintended outcomes

Evaluation should look for effects that were not originally anticipated.

These may be:

• Positive
• Negative
• Mixed
• Different across groups

Do not immediately assume why an unintended outcome occurred.

Use additional data, participant input, and other evidence to understand the mechanism before deciding how to respond.

Use a “But why?” approach carefully

Working backward from outcomes can help identify where an intervention may need improvement.

Examine the outcomes

If the intervention produced the intended outcomes, identify what contributed to those results and what should be maintained.

Also examine whether benefits were distributed equitably and whether any unintended effects occurred.

Examine the impact

If outcomes fell short, examine whether the intermediate changes expected from the intervention actually occurred.

If they did occur but the longer-term outcomes did not, consider whether:

• The assumed pathway to outcomes was incomplete
• Additional factors influence the outcome
• The intervention needs more time
• Measurement did not capture meaningful change

Examine the process

If the expected impact did not occur, examine how the intervention was planned and implemented.

Determine whether the issue involved:

• Implementation quality
• Access
• Staffing
• Resources
• Community participation
• Program design
• Communication
• Cultural responsiveness
• External conditions

If the intervention was implemented well but still produced little or no impact, the underlying program theory or methods may need substantial revision.

Continue learning and adapting

Make the changes supported by the available evidence, document what you changed and why, and evaluate the revised intervention.

Improvement should be iterative.

At the same time, avoid changing programs simply for the sake of constant experimentation.

Preserve effective components, assess the consequences of adaptations, and give changes enough time to be meaningfully evaluated.

In Summary

Evaluation is valuable not only because it tells you whether an intervention produced the intended results, but because it helps explain what happened and what should happen next.

Refining an intervention means using evaluation findings, participant experience, implementation knowledge, and other relevant evidence to improve effectiveness, accessibility, equity, relevance, and sustainability.

Examine process, impact, and outcomes separately and together.

Process evaluation helps determine what was actually implemented and how people experienced it.

Impact evaluation examines shorter- and intermediate-term changes in the factors the intervention was intended to influence.

Outcome evaluation examines the longer-term changes the intervention ultimately seeks to produce.

Avoid interpreting limited participation or weak outcomes simply as a failure of participants to stay motivated or follow the program.

Examine program design, accessibility, implementation, structural conditions, social and economic barriers, and community fit.

Involve people directly affected by the intervention as meaningful partners in interpreting findings and deciding what should change.

Look for differences in access, experience, and outcomes across groups without assuming that those differences are caused by individual or cultural deficits.

Be willing to preserve what works, adapt what can be improved, investigate uncertainty, and discontinue approaches that credible evidence shows are ineffective or harmful.

Document changes and evaluate them so that improvement becomes a cycle of learning rather than a series of untested adjustments.

The goal is not simply to make participants conform more closely to a program.

It is to use evidence and community knowledge to make the intervention better able to serve the people and purposes for which it exists.

What do we mean by refining the intervention?

___Refining the intervention is the process of making your work more effective by using evaluation information

___Key reflection questions include:

• What are we seeing?
• What does it mean?
• What are the implications for improvement?

Why should you use your evaluation research to refine the intervention?

___Understanding and improving your work is one of the main purposes of evaluation

When should you refine the intervention?

___Throughout the life of an intervention or program

___When it’s clear that what you’re doing isn’t working

___When participants are dropping out at a high rate

___Between sessions of a time-limited or sequential program

___When funders or participants ask you to adjust some aspect(s) of your program

___When funding or other resources are reduced

___When the goal or issue changes character

Who should be involved in refining the intervention?

___Participants

___Staff members, paid or volunteer

___People who are directly or indirectly involved in implementing or supporting your work

___Those who led and participated in the evaluation

How do you refine an intervention based on research?

___Determine whether your process – the planning, set-up and implementation of the program – went as planned

___Determine whether your program had the desired impact on the behaviors and risk and protective factors you targeted

___If there was no impact, or only a partial impact (or a negative impact), that could affect the program’s outcomes

___Determine whether your program achieved its desired outcomes

___If the program achieved its desired outcomes, consider how you can expand and strengthen the areas that worked well, and make the program more efficient and effective

___If the program failed to achieve the desired outcomes, work backwards – “but why” – to find where the problems lie

___If the program failed to have the intended impact on the targeted behaviors and risk and protective factors, that could explain the fact that you didn’t meet your outcome goals

___If the program had the intended impact on the targeted behaviors and factors, but failed to achieve the desired outcomes, you may have targeted the wrong behaviors or factors

___If some or all of the process was different from what was planned, that might affect the program’s impact on the targeted factors

___If the planning process or reasoning was faulty, it could result in targeting the wrong factors, or in using ineffective methods that target the right factors

___Once you’ve determined where and how to make adjustments and done it, repeat the evaluation and refinement cycle for the life of the intervention

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• Why inform people about your evaluation findings?

• What are some key audiences for the data?

• How do you communicate evaluation findings?

Why inform people about your evaluation findings?

There are many reasons to inform people about your evaluation findings. Sharing findings can help community members understand the purpose of your coalition or initiative, the work you've been doing with and alongside the community, and what the evaluation has shown. Positive findings can demonstrate progress and accomplishments. Findings that are mixed or identify areas for improvement can help your group learn, adapt its strategies, and make the case for additional support or resources. Sharing evaluation findings can also encourage public interest, reflection, and discussion about the issues your initiative is addressing. The main reasons for making evaluation findings publicly available are to increase understanding of the issue, promote transparency and accountability, and support informed community action.

There are three main levels at which you can share evaluation findings with the public: local, regional/state, and national. Some reasons for sharing your evaluation results at each of these levels include:

Reasons for informing the public at the local level

• To help raise awareness about the issue
• To help attract volunteers, funding, and in-kind resources from community members and local organizations
• To promote awareness of the efforts of volunteers and collaborators
• To help advocate for local ordinances, policies, or program changes that address issues of concern
• To provide accountability to the community, trustees, and funders

Reasons for informing the public at the state level

• To increase recognition of your initiative across the state, which may strengthen opportunities for collaboration and access to resources
• To help establish a statewide network of people and organizations with similar goals
• To help advocate for legislative or policy changes that address issues of concern
• To help the initiative gain recognition and resources from across the state and region

Reasons for informing the public at the national level

• To increase recognition of the initiative nationally, which may strengthen opportunities to seek resources from state or federal agencies or large private foundations
• To help connect with nationwide networks of people and organizations with similar goals and diverse expertise
• To help the initiative gain recognition and resources from across the country
• To encourage community partnerships and collaborative action related to the issue

Three tips for making sure your findings aren't ignored

• Share your information with the people and organizations who can use it.
• Address issues that are relevant and meaningful to those audiences.
• Present the information in time for it to be useful and in a clear, accessible format.

What are some key audiences for the data?

Who should you share this data with? Staff, volunteers, community partners and supporters, and funders are among the groups that should be kept up to date on your initiative's efforts, accomplishments, and areas for continued improvement. You should also share evaluation findings with community members most directly affected by the issue or involved in the initiative. For example, if you work with a heart disease prevention program and your evaluation identifies a higher risk factor among members of a particular ethnic or cultural community, share those findings in a clear and respectful way with members of that community and, when possible, involve them in interpreting the findings and identifying appropriate next steps. Community members should have access to information about what the findings mean and how changes in programs, practices, environments, policies, or individual behaviors may support health. Finally, the broader public should be kept informed when the findings are relevant to community awareness or action.

Key groups to share your evaluation findings with

Local

• Civic organizations
• Business groups
• Grassroots and community-based organizations
• School boards
• Parent-teacher groups
• Faith-based organizations
• Local media
• Health organizations
• Elected and appointed local government officials
• Grantmakers

State/regional

• State and regional professional conferences
• Regional professional training workshops
• Grassroots and advocacy organizations
• Faith-based conferences and networks
• Grantmakers

National

• Professional conferences
• Professional training workshops
• Grassroots and advocacy organizations
• Faith-based conferences and networks
• Grantmakers

What about difficult audiences?

• Anticipate their questions, concerns, and objections. Think ahead about the concerns or perspectives a particular audience may have about your findings, and prepare calm, clear, evidence-informed responses. If possible, consult someone who understands the audience well to get feedback on questions they may raise and how your responses may be received. During the presentation, this preparation can help you respond thoughtfully rather than becoming defensive.
• Have a knowledgeable spokesperson or initiative leader present the findings. Someone who understands both the evaluation and the initiative can clearly explain the findings, answer questions, and provide appropriate context.
• Consider who is best positioned to communicate the information. With an audience that may be skeptical of or unfamiliar with the issue, or one your initiative does not normally have access to, it may be helpful to involve a trusted person from that audience or community in sharing or discussing the evaluation findings.
• Reinforce key findings consistently
• Stay calm and professional

Presenting evaluation results to the press

• Be honest with reporters. It is essential for any spokesperson for your coalition or group to maintain credibility with the media. Answer questions clearly and candidly, and if you cannot answer a question, explain why or indicate how you will follow up when appropriate.
• Write your own press releases. Give reporters a clear, concise release with the most important information and appropriate contact details in advance when possible. This can reduce misunderstandings and help reporters prepare for your presentation or announcement.
• Help reporters understand the data. If your evaluation findings include complex statistical or technical information, consider offering an informational session or one-on-one briefing for interested reporters. These sessions should be accurate, balanced, and broad enough that reporters can apply what they learn when covering similar issues. Topics might include:
  • What do percentile scores mean?
  • What are some common methods of gathering evaluation data?
  • What are community-level indicators?

How do you communicate evaluation findings?

Develop a general presentation format that can be lengthened or shortened depending on the amount of time available, including compelling descriptions and visuals of:

• The issue(s) of concern
• The initiative's goals, strategies, and methods for reaching those goals
• Data on activities (e.g., services provided or engagement activities)
• Data on accomplishments (e.g., community or organizational changes)
• Data on outcomes (e.g., behavioral measures, community-level indicators, and other relevant measures)

Keep your visuals simple and accessible to reduce confusion when interpreting data. The first and last visuals should reinforce your main message or primary findings—whatever you most want your audience to remember afterward.

Identify different avenues for sharing your evaluation results, such as:

• Word of mouth and community networks
• Presentations and community meetings
• Newspapers, newsletters, and other publications
• Radio, including public service announcements, local news, and call-in programs
• Television and video coverage
• Professional journals
• Websites, email, social media, and other digital communication channels

Why might you use different formats for presenting your evaluation results?

Different types of reports and communication formats work well for different audiences. Some audiences may not need detailed statistical information but may still want to understand the main findings and what they mean. Funders, grantmakers, advisory boards, program staff, and other partners may need more detailed information. Consider what information each audience needs, how they are most likely to access it, and how you can present the findings in a clear and accessible way.

Here are some ideas for matching types of reports and communication formats with different audiences:

• Technical reports: A detailed report on a specific issue or study. It can be shared at a staff meeting or included as part of a larger report.
  • Best for: Funding agencies, program administrators, advisory committees
• Executive summary: A brief section, usually at the beginning of a longer report or provided as a stand-alone document, that summarizes a study's major findings and recommendations.
  • Best for: Funding agencies, program administrators, board members and trustees, program staff, advisory committees, policymakers, service providers, and practitioners
• Technical professional paper: A detailed article that summarizes information for a scientific or technical audience. It usually describes what was done, how it was done, what worked and did not work, and why.
  • Best for: Program administrators, advisory committees, researchers, and organizations interested in program content
• General-audience article: An article written with the intended audience of the publication or platform in mind. It typically provides more information than a press release while focusing on a few central findings or messages.
  • Best for: Program administrators, board members and trustees, program staff, policymakers, community groups, current and prospective participants, service providers, and organizations interested in program content
• News release and/or press conference: A written announcement or gathering with the media for the purpose of releasing specific information.
  • Best for: Program administrators, the media, and broad distribution of concise information
• Public meeting: A gathering open to community members where evaluation findings are shared in a clear, accessible manner, with opportunities for questions, discussion, and feedback.
  • Best for: Community groups, program participants, community members, and the media
• Media appearance: A media interview, public event, or other appearance used to draw attention to evaluation findings—for example, a local author participating in a public reading connected to a study on adult literacy.
  • Best for: Program participants, community members, and the media
• Staff workshop: A more interactive working session for your group or coalition's staff, volunteers, and partners.
  • Best for: Program administrators, program staff, volunteers, partners, and service providers
• Brochures, posters, flyers, and infographics: Brief, clearly written print or digital materials that can be distributed through community organizations, events, websites, email, or social media. These formats usually work best when they focus on a small number of key messages.
  • Best for: Current and prospective participants, community members, and the broader public
• Memo: A short written communication circulated internally among program staff or partners.
  • Best for: Program administrators, program staff, and service providers
• Personal discussion: Meeting in person or virtually with an individual or small group to discuss evaluation findings.
  • Best for: Funding agencies, program administrators, program staff, partners, and service providers

Possible goals of your presentation

What do you hope the group you're presenting to will understand, contribute, or do after hearing the evaluation findings? How can you present the information in a way that supports those goals while encouraging informed discussion and participation?

A few things you might be hoping to gain from your audience include:

• Funding and in-kind resources for your initiative
• Volunteers or other forms of participation in project activities
• Support for changes in programs, policies, systems, or practices
• Input on how to make the initiative more responsive to community priorities and needs
• Greater understanding of concerns, questions, or resistance related to the initiative
• Ideas for making the initiative more effective and equitable

Steps in developing your presentation

• Understand your primary users and audiences. What information do they need, why do they need it, and how are they most likely to use it? Consider the audience's perspective, preferred communication methods, and accessibility needs, and share the report in time for it to be useful.
• Review the results of your evaluation with program staff, community partners, and other relevant participants before you finalize the evaluation report. This provides an opportunity to gather different perspectives on the meaning of the findings and discuss any ambiguous or unexpected data.
• Brief relevant policymakers, agency leaders, and other key partners before releasing your report publicly when the findings may have implications for policies, programs, or funding. This gives them an opportunity to understand the findings and prepare an appropriate response. For example, if an evaluation shows that a recreation program for families of children with developmental disabilities is effective but underfunded, local human service agencies may use the findings to consider whether additional funding or other support should be included in a future budget.
• Your final report can be a concise document summarizing the evaluation findings, with a technical appendix or links to additional information for people who want more detail.
• If you decide to give an oral presentation, use a manageable number of charts, tables, or other visuals that illustrate the most important findings. Prepare them in a format that works well for the presentation, such as digital slides or printed displays, and provide accessible copies or digital materials that audience members can refer to afterward. You may also wish to prepare a one-page summary of the overall results.
• Your report should begin with the reasons the evaluation was conducted, what questions were asked, and why those questions were chosen. Explain what your group, coalition, and community partners wanted to learn from the evaluation and what methods were used to conduct it.
• Depending on your audience, you may want to highlight the main results or provide greater detail about what you found. Explain what the findings may mean for your group, initiative, partners, and community. If the evaluation findings suggest changes to future strategies or activities, discuss those possibilities and, when appropriate, invite feedback from people affected by those decisions.

Why should you try to let others know about your initiative?

___To let the public to know you exist

___To let the public know some of what you've been doing to help your community

___To stir public interest

___To expose the issue and encourage the public to take action

Why should you inform the public at the local level?

___To help raise awareness about the issue

___To help attract volunteers, funding, and in-kind resources from local concerned citizens and agencies

___To promote awareness of the efforts of volunteers and collaborators

___To help lobby for local ordinances or program changes to address issues of concern

___To provide accountability to the community, trustees, and funders

Why should you inform the public at the state level?

___To create a "name" for your initiative in the state, which makes it more competitive when seeking state resources

___To help establish a statewide network of persons and agencies with similar goals

___To help lobby for legislative changes to address the issues of concern

___To help the initiative garner recognition and resources from the state and region

Why should you inform the public at the national level?

___To create a "name" for the initiative nationwide, which makes you more competitive when seeking resources from the state or federal government or from large private foundations

___To help tap into nationwide networks of persons and agencies with similar goals and wide expertise

___To help the initiative garner recognition and resources from across the country

___To encourage community partnerships to work on the problem or issue

How can you ensure that your findings aren't ignored?

___Give your information to the right people!

___Address issues which those people think are important

___Be sure the information has been presented in time to be useful and in a way that's clearly understood

What are some key audiences for sharing your data?

___Staff

___Volunteers

___Supporters in the community

___Funders

___Your target population

___The general public

What are some key local audiences for sharing your data?

___Civic organizations

___Business groups

___Grassroots organizations

___School boards

___Parent-teacher groups

___Church organizations

___The local press

___Health organizations

___Elected and appointed local government officials

___Grantmakers

What are some key state/regional audiences for sharing your data?

___State and regional professional conferences,

___Regional professional training workshops

___Grassroots and advocacy organizations

___Church conferences

___Grantmakers

What are some key national audiences for sharing your data?

___Professional conferences

___Professional training workshops

___Grassroots and advocacy organizations

___Church conferences

___Grantmakers

How should you deal with difficult audiences?

___Anticipate their questions, concerns, and objections

___If appropriate, have a primary figure in your initiative present the findings

___Have someone else give out the information

___Reinforce the data repeatedly

___Keep your cool

How should you present your evaluation findings to the press?

___Be honest with reporters

___Write your own press releases

___Train your reporters

What should you include in your presentation?

___The issue(s) of concern

___The initiative's goals, strategies, and methods for reaching those goals

___Data on activities (e.g., services provided)

___Data on accomplishments (e.g., community changes)

___Data on outcomes (i.e. behavioral measures and community-level indicators)

___Keep your visuals simple to cut down on problems interpreting data

What are some different avenues of getting the word out about your evaluation results?

___Word of mouth

___Presentations

___Newspapers and newsletters

___Radio - both public service announcements and local news or call-in shows

___Television coverage

___Professional journals

What are the different formats for presenting your evaluation results?

___Technical reports

___Executive summary

___Technical professional paper

___Popular article

___News release and/or press conference

___Public meeting

___Media appearance

___Staff workshop

___Brochures/posters

___Memo

___Personal discussion

What are possible goals of your presentation?

___Money and in-kind resources for your initiative

___Volunteers for project activities

___Influence in changing a program, policy, or practice

___Input on how to make the initiative more responsive

___Overcoming resistance to the initiative

___Ideas on how the initiative can become more effective

What are the steps for developing your presentation?

___Understand your primary users and audiences

___Review the results of your evaluation with program staff before writing up your evaluation report

___Take time to brief any important political figures before releasing your report to the public

___Know that your final report can just be a short document summarizing the evaluation findings with a technical appendix for those who are interested

___If you decide to do an oral presentation, make a small number of charts and tables illustrating the most important findings

___Begin your report with the reasons the evaluation was done, what questions were asked, and why those were the questions chosen

___Explain what your group or coalition wanted to learn from the evaluation and what methods were used to conduct the evaluation

___Explain what sort of implications the results have for your group or initiative

Tool: Evaluation Report Outline

This tool gives you an overview of the different parts of an evaluation report and what each part should include.

Front Cover

The front cover should include:

• Program title and location
• Name(s) of evaluator(s)
• Period covered by the report
• Date of the report

Lay out your front cover neatly and make it look nice - the front cover is the first thing your audience sees and it makes an important impression.

Section I - Summary (or Executive Summary)

This is a brief (two to three pages) overview of the evaluation outlining major findings and recommendations. Some folks are too busy to read any further than the summary, so make sure that this is as complete and clear as possible.

The summary should include:

• What was evaluated?
• Why was the evaluation done?
• What are the major findings and recommendations?

And, if space permits:

• What audience is the report aimed at?
• What decisions, if any, need to be made or have been made based on the results of the evaluation?
• Who else might find the report to be of interest or importance?

Section II - Background Information About the Program

Presumably, most of the people reading your evaluation report will at least be somewhat familiar with the program, but that's not necessarily the case. And even people who are familiar with the program may have some misconceptions, so take the time to make your goals, strategic plan, organizational structure, and other essential program elements clear.

Typically, this section will include:

• Origins of the program
• Program goals
• Clients involved with the program
• Administrative/organizational structure
• Program activities and services
• Materials used and produced by the program
• Program staff

Section III - Description of the Evaluation

This part explains why an evaluation was done and what you hoped to learn from it. It should also explain anything the evaluation was not intended to do.

Here are some of the questions that should be answered by this section:

• Who requested the evaluation?
• Was the evaluation meant to satisfy any particular audience and, if so, which one(s)?
• Were there any restrictions to the evaluation in terms of money, time, or other resources?
• Was any particular kind of evaluation design used and, if so, why?
• What was the timetable for collecting data?
• For each measure, what sort of data was collected?
• What sort of methods were used to gather data, and why were these particular methods chosen?
• How did the evaluators ensure accuracy?

Section IV - Results of the Evaluation

This part will explain what your findings were in detail.

This section may include:

• All data collected - analyzed, recorded, and organized in understandable forms (charts, tables, graphs, etc.)
• Excerpts from interviews
• Testimonials from participants and clients
• Questionnaire results
• Test scores
• Anecdotal evidence

Section V - Discussion of Results

Here is your chance to go into more detail --the why of your evaluation results.

This part should answer the following questions:

• How sure are you that your program or initiative caused these results?
• Were there any other factors that could have contributed to the results?
• How are the results different from what they would have been if your program didn't exist?
• What do the evaluators feel are the strengths and weaknesses of your program?

Section VI - Costs and Benefits

This part of the report is optional - if you choose to include it, it gives you a chance to justify your program's budget and financial choices.

If you include it, here's what we suggest you include:

• Costs associated with the initiative (not only financial costs, but costs in terms of resources, energy, results, and staff/volunteer hours)
• Methods used to come up with the budget
• Benefits from the program (both financial and non-financial)

Section VII - Conclusions

After writing all this stuff up, it may be tempting to dash off a quickie conclusion to this report, but avoid that temptation!

This is a very important piece of the big pie, because this is where you make your recommendations:

• What major conclusions about the initiative can be reached as a result of this evaluation?
• Is there anything you feel should not be judged at this time, and if so, why?
• Based on the evaluation results, what recommendations can you make for the program?
• If the evaluation gives you any idea of what the future holds for the initiative, what would that be?
• What worked well about the evaluation? What didn't work so well?
• What recommendations do you have for anyone doing future evaluations with the program?