Section 1. Using an Evaluation System to Answer Key Questions About Your Initiative

Evaluation can help your initiative improve its methods, communicate with funders, support staff, and make better decisions. Its central purpose is to help you understand what is happening, what difference your work is making, and how to improve it.

Evaluations sometimes focus on activities and outputs, such as the number of people trained, without examining whether community conditions changed. Those measures are useful, but they do not tell the whole story. A broader evaluation asks whether intended changes occurred, how they developed, and what evidence connects them to the initiative.

Alongside assessing specific objectives, you can examine changes in programs, policies, practices, and community conditions. You can also investigate whether those changes contribute to longer-term outcomes. This section explains how to frame those questions, organize relevant information, and interpret the findings.

What does answering evaluation questions involve?

Evaluation questions identify what participants, community members, staff, partners, funders, and others need to know about an initiative. Useful questions connect to decisions that people will make. Examples include:

• How do participants experience the program, and what do they want improved? (Process and implementation)
• To what extent is the initiative meeting its stated objectives? (Attainment of objectives)
• What changed for participants, and what evidence connects those changes to the initiative? (Participant outcomes and the initiative's contribution)
• What changed across the community, for whom, and how did the initiative contribute? (Community outcomes and the initiative's contribution)

The key questions depend on your mission, objectives, and understanding of how change is expected to occur. A logic model describes the proposed connections among resources, activities, outputs, and shorter- and longer-term outcomes. It should also acknowledge assumptions and outside influences. Evaluation questions examine those connections: for example, "Did the initiative help change access to services?" and "Were changes in access followed by improvements in residents' experiences or health?"

The Center for Community Health and Development at the University of Kansas, which produces the Community Tool Box, developed the Community Check Box approach to documenting and evaluating community health and development efforts. Much of this section draws on the thinking behind that approach.

The examples show how a documentation system can organize evidence relevant to evaluation questions. A system can support analysis, but its records alone do not establish an initiative's effectiveness. The strength of the conclusions depends on the questions, data quality, evaluation design, and interpretation.

You can examine your work at several connected levels. One concerns specific objectives, such as adopting a policy, improving services, or changing an environmental condition. Another concerns the initiative's wider influence on organizations, systems, and community life. Both levels matter, and progress at one does not automatically establish progress at the other.

A comprehensive evaluation distinguishes processes, outputs, and outcomes. Processes describe how the work is carried out. Outputs are its immediate products, such as training sessions delivered or people reached. Outcomes are changes in conditions, behavior, access, or well-being. An impact evaluation goes further by investigating whether outcomes differ because of the initiative, using a design suited to that causal question.

Evaluation should also consider changes beyond the original objectives. An initiative may encourage partner organizations to start related programs, attract additional resources, or change how community members participate in decisions. For example, a youth violence prevention effort might expand opportunities for young people to shape local services. Even when an initiative does not meet an initial objective, it may produce useful learning or other changes worth documenting. Look for unintended harms as well as benefits, and avoid using broader achievements to obscure unmet commitments.

Two terms are useful throughout this discussion:

A community may be defined by place, such as a neighborhood, town, city, or rural area. It may also be defined by shared identities, interests, experiences, or relationships, including communities that connect online. This section often refers to communities of place, but the evaluation principles can apply more broadly. Be explicit about whom your initiative includes and avoid assuming that everyone within a community shares the same priorities.

A system includes the organizations, relationships, resources, rules, and practices that shape a function such as education, health care, housing, or public safety. Changes in a system can affect access to opportunities and services, how decisions are made, and the conditions in which people live.

At each level, ask what changed, who experienced the change, and what evidence explains how it happened. Examine both how the initiative influenced its setting and how changes in that setting affected the initiative.

Figure 1 presents a framework for collaborative action, along with evaluation questions about the proposed relationships between its stages.

The framework discussed by Fawcett and Schultz (2008) highlights four questions, restated here:

1. Is the initiative helping bring about community or system changes related to its mission? Look for new or modified programs, policies, and practices, and document the initiative's role alongside the contributions of partners and other influences.
2. What factors or processes are associated with community or system change? Examine events, relationships, resources, and conditions that may support or limit progress, including changes in the pace of documented activity.
3. How do those changes contribute to the initiative's goals? Consider their relevance, reach, quality, duration, and effects on different groups, including unintended consequences.
4. Are community or system changes associated with improvements in population-level outcomes? Examine whether changes in programs, policies, and practices align with outcomes such as fewer injuries, improved access to care, or reduced tobacco use. Choose measures that closely match the outcome rather than assuming that arrests or product sales directly measure behavior.

Figure 1. Framework for collaborative community action and related evaluation questions (Source: Fawcett and Schultz, 2008; Institute of Medicine, 2003).

The questions in Figure 1 examine proposed relationships along the framework's pathway. These relationships are assumptions to investigate, not conclusions established by the diagram. See Fawcett and Schultz (2008) in the print resources.

Source: Adapted from the Institute of Medicine's framework for collaborative public health action in communities (2003).

These questions place the initiative within its broader setting and help distinguish participant outcomes from community-wide changes. They may require several kinds of evidence and may not all be answerable with the same level of confidence. Later in this section, we also address a fifth question: whether the initiative caused improvements in population-level outcomes.

Why answer these questions?

Funding arrangements may emphasize short-term deliverables, while meaningful community changes can take longer. Evaluation can meet reporting needs while also examining the relationships and outcomes that matter to the community. Reasons to answer these questions include:

• To improve the work. Understanding how activities, community changes, and outcomes relate can help you identify what to strengthen, adapt, or discontinue.
• To understand what supports or limits progress. Examine how leadership, staffing, partnerships, planning, and other conditions affect implementation and the opportunities for change.
• To improve the pace and quality of progress. Findings may suggest ways to remove barriers or make better use of resources. Faster change is not always better; participation, quality, and sustainability also require attention.
• To understand the broader context. Economic hardship, unequal access to resources, policy changes, or community crises can affect both implementation and outcomes. Documenting these conditions helps explain results and inform future work.
• To understand how lasting improvements develop. Your goals may include preventing domestic violence, reducing pollution, improving health, or expanding educational opportunities. Evaluation can explore how specific changes in programs, practices, or policies contribute to those goals.
  
  Examine which changes appear useful, how they work, and under what conditions they might be sustained. Use the findings to refine your approach while remaining open to evidence that challenges the original plan.
• To manage organizational transitions. Leadership turnover, funding changes, or a shift between program phases may affect the work. Understanding these relationships can support continuity planning and help your group respond to disruptions.
• To communicate value and maintain accountability. Share achievements, challenges, and uncertainties with the community. A credible account of the work can support informed decisions about participation, partnerships, and resources.

Who should help answer these questions?

Involve people who experience, carry out, support, or make decisions about the initiative. Their perspectives can help identify useful questions and explain findings that records alone cannot capture. Participatory evaluation means sharing meaningful roles in decisions, interpretation, and use of findings; providing information alone is not the same as sharing decision-making. Agree on roles, address accessibility and language needs, and consider compensation for community members' time. People to involve may include:

• Participants and people directly affected by the initiative. They can identify priorities, describe benefits and barriers, and assess whether proposed changes meet their needs. Include people with mixed or negative experiences, not only those who remain actively involved.
• Residents of the area where the initiative works. Invite perspectives from across the area, including residents who do not use the program. Their experiences may reveal wider effects or needs that the initiative has overlooked.
• Staff and volunteers carrying out the work. They can explain implementation, resource constraints, adaptations, and practical opportunities for improvement. Make room for candid feedback without penalizing people for identifying problems.
• People whose work or relationships connect them to the issue. Depending on the initiative, these may include educators, health care providers, family members, caregivers, advocates, or other partners. Their perspectives complement, rather than replace, the views of people directly affected.
• Funders and public officials. Their involvement can help align evaluation questions with decisions about resources and policy. Establish arrangements that protect the integrity of the evaluation and prevent powerful participants from controlling findings or excluding other perspectives.

When should you establish and use an evaluation system?

Plan evaluation as early as possible, ideally alongside community assessment and initiative planning. Early planning helps identify baseline information, define expected changes, and document decisions that may matter later. If the initiative is already underway, begin with the information available and reconstruct earlier events where records support it. Be clear about gaps rather than treating a late start as a reason not to evaluate.

Collect and review information on a schedule that fits your decisions, resources, and expected timing of change. Record when activities and events occur so you can examine sequences and possible relationships. Some implementation information may need frequent review, while population outcomes may be available only annually or less often. The aim is timely learning and action, without collecting more information than the group can use responsibly.

How do you use the evaluation system to answer key questions?

With your evaluation team, decide what evidence each question requires, where it will come from, and how you will analyze it. Agree on definitions, responsibilities, privacy protections, and the decisions the findings will inform. Use both numerical information and accounts of people's experiences when they address different parts of the question.

Graphs can help reveal patterns and relationships. Figure 2 illustrates a cumulative count of documented community changes, such as new programs or policies related to an initiative's mission. Each recorded change is added to the previous total. A flat line means no additional changes were recorded during that period; a steeper line means more changes were recorded per unit of time. The total does not show whether changes remained in place or how important they were. Track reach, quality, and sustainability separately, and avoid counting the same change more than once.

Figure 2. Example cumulative record of community changes associated with the South Side Youth Development Initiative, 2008–2014. [Source: Example graph from the KU Center for Community Health and Development's Community Check Box]

Adding the dates of important inputs and events can help you explore possible influences on the pattern. Figure 3 illustrates this approach by placing milestones alongside a cumulative record of community and system changes.

The events identified in the example include:

• Formation of a planning team
• Appointment of a new director
• Implementation of a social marketing action plan
• Hiring of additional staff
• Action planning for the initiative
• A disruption in leadership

These events may have influenced the pace of documented change, but their timing alone cannot establish their effects. Review other records and participants' accounts to investigate possible explanations.

Figure 3. Historical example of cumulative community and system changes and associated events for the Lincoln Coalition for Healthy Living, 1999–2006.

The example invites questions about whether staffing and leadership changes were related to changes in the pace of activity. It also raises other possibilities: hiring staff might improve documentation as well as implementation, and leadership transitions might affect both. Examine these explanations before concluding that a particular event caused an increase or decrease. This analysis can help identify practical steps to strengthen the work.

You can also graph outcomes, compare patterns across neighborhoods or groups, or examine trends in relevant comparison communities. Choose measures based on the logic model and evaluation questions. Use clear labels and comparable time periods, and take care with graphs that use different scales, since their appearance can exaggerate relationships.

Decide which measures are most useful before collecting large amounts of data. Once the information is available, look for patterns, exceptions, and competing explanations. Keep the central purpose in view: what can the group learn that will help it achieve its goals more effectively and fairly?

The following five questions build on the framework in Figure 1. The first four examine documented changes and relationships; the fifth addresses the stronger question of whether the initiative caused improvements.

1. Is the initiative helping bring about community or system changes related to its mission? Identify specific changes in programs, practices, and policies. Document when they occurred, who contributed, what the initiative did, and what evidence supports the account. Relevant systems might include public health, education, public assistance, public safety, or environmental protection.

Types of changes to document may include:

• New or modified programs. Examples include a new youth development program, an expanded vaccination service, or a resident-led prevention initiative. Distinguish establishing or changing a program from repeatedly delivering its existing activities. Record the program's purpose, reach, start date, and the partners involved.
• New or modified practices. Examples include a clinic extending its hours or an organization changing its procedures to improve language access. Document whether the practice was implemented, who could benefit, and whether it continued.
• New or modified policies. Examples include changes in organizational rules, public regulations, or resource allocation policies. Distinguish policy adoption from implementation and enforcement. A new service location or store opening is a community change, but it should not automatically be classified as a policy change.

Depending on the initiative, you may also track organizational developments, media coverage, resources obtained, partnerships, and other important events. Keep these categories distinct so that fundraising, publicity, and implemented community changes are not treated as equivalent accomplishments.

Record both the timing of changes and the sequence of actions that preceded them. Meeting records, policy documents, implementation dates, and accounts from different participants can help clarify the initiative's contribution. A change occurring after an initiative's activity is a starting point for investigation, not sufficient evidence that the activity caused it.

Some changes take time to affect outcomes, and several changes may work together. Track whether they were implemented, whom they reached, and whether they remained in place. This can help explain why an outcome did or did not change within the evaluation period.

Look for patterns in the rate and type of documented change. Do they align with action planning, new partnerships, staffing, funding, or outside events? Check whether changes in documentation practices could explain part of the pattern. Use the findings to identify questions for follow-up and possible improvements.

An initiative might record relatively few changes while building relationships and planning, followed by more changes during implementation. Later, the focus may shift from starting new work to sustaining existing programs or policies. This is one possible pattern, not a required sequence or a universal standard of success.

If the cumulative count levels off, investigate why before trying to increase activity. The group may need more capacity, may have encountered barriers, or may be successfully maintaining important changes. The appropriate response depends on what the evidence shows.

Also ask whether the changes are substantial enough and sufficiently well implemented to affect the intended outcomes. Counting changes is useful, but understanding their relevance, reach, and quality is essential. Identify gaps in the records and consider how to obtain a more complete account without creating unnecessary reporting burdens.

2. What factors or processes are associated with community or system change? Monitor conditions that may influence implementation and outcomes, then investigate the relationships suggested by the information.

• Assessment, planning, implementation, and evaluation processes. Examine how action plans are developed and used, how community members influence decisions, and how documentation and feedback support the work. Consider whether responsibilities, resources, and timelines are realistic. Review the roles of staff, volunteers, leaders, and governing bodies without assuming that delays reflect individual failure. The Community Tool Box's "Promising Approaches" materials offer related ideas to explore.
• Participation, reach, and experiences. Examine who participated, who was missed, how much support people received, and whether it met their needs. Investigate why people stopped participating, including scheduling, accessibility, cost, or dissatisfaction. Distinguish being reached from benefiting. The "Solve a Problem" guide on insufficient community participation can support this discussion.
• Significant program and community events. These may include:
  • Leadership transitions and changes in decision-making responsibilities.
  • Staffing changes, vacancies, training, or changes in available skills.
  • Changes in methods or direction, including intentional adaptations to community priorities and unplanned changes caused by capacity or coordination problems.
  • Milestones such as adopting an action plan, starting or revising a program, completing training, or achieving an objective.
  • Increases, reductions, delays, or changes in the conditions of funding.
  • Unexpected operational challenges and responses, including safety concerns, financial pressures, changes in service hours or locations, and changes in eligibility or capacity.
  • Major changes in conditions, such as disasters, disease outbreaks, or disruptions to essential services.
  • Other community developments, such as:
    • Changes in public leadership or government administration.
    • Incidents of discrimination, community tensions, or efforts to address inequitable treatment.
    • New employers or investments, including their effects on employment, public resources, and affordability.
    • Employer closures, job losses, or other economic disruptions.
    • Community losses, acts of violence, celebrations, or recognition of local contributions.

Broader changes in government procedures, funder priorities, service systems, or public attitudes may also affect the initiative. Record relevant developments even when they are outside the group's control.

As Figure 3 illustrates, aligning events with patterns of documented change can suggest relationships to investigate. Use additional evidence to assess those explanations, then consider actions that could strengthen implementation or reduce harmful consequences.

3. How do community and system changes contribute to the initiative's goals? Examine the nature and distribution of changes, whether or not the initiative was their only source. Ask how they affect the conditions your group seeks to improve.

Look at where changes are concentrated and where they are missing. Are most changes occurring in schools, while other settings receive little attention? Are particular neighborhoods or groups benefiting more than others? Do the changes address all major objectives or mainly those that are easiest to achieve? Compare the pattern with community priorities and the logic model. Useful dimensions include:

• Goals and objectives. Group changes according to the outcomes they are intended to support. A heart health initiative might address physical activity, tobacco use, access to nutritious food, and access to appropriate preventive care.
• Intervention strategies. Distinguish changes involving information, services, social support, physical environments, organizational practices, or public policy.
• Risk and protective factors. Consider conditions that may increase or reduce the likelihood of an outcome. Examples include exposure to violence, supportive relationships, stable housing, and access to care. These factors describe probabilities and circumstances, not a person's destiny, and should not be used to label families or communities.
• Duration and sustainability. Distinguish a one-time event from an ongoing service or policy. Record how long a change remains in place and what resources or commitments are needed to sustain it.
• People reached and people benefiting. Identify the intended population and the people actually reached. Where evidence permits, examine who experienced benefits or harms. Do not assume that a policy's stated coverage establishes equal access or benefit.
• Community sectors. Changes may involve government, health care, education, labor, business, housing, faith communities, or other settings. Consider whether the sectors engaged match the initiative's objectives.
• Levels of influence. Examine whether changes focus on individuals, relationships, organizations, community conditions, or broader policy. Many initiatives work across several levels; for example, a physical activity effort may combine personal support with changes in public spaces and transportation policy.
• Place. Examine the distribution of changes across neighborhoods, municipalities, counties, rural areas, or other relevant locations.

Discuss what these patterns mean with community members and partners. They may reflect opportunities, unequal resources, access barriers, or the interests of particular organizations. Decide whether adjustments are needed based on the importance and reach of the changes, not simply whether every category has the same number.

Figure 4. Example distribution of documented community accomplishments (N = 125) by the primary sector in which each change occurred.

This chart can support discussion of which sectors are represented in the record. It does not, by itself, explain why changes were more common in one sector or demonstrate their relative value. If changes span several sectors, explain how the primary sector was assigned. The "Solve a Problem" troubleshooting guides offer additional support for considering adjustments.

4. Are community and system changes associated with improvements in population-level outcomes? Select indicators that reflect the intended outcomes for a defined population. Community-level indicators describe patterns across groups, but they are not automatic proof of success or failure. Potential data sources include:

• Census and other population data, such as educational attainment, housing conditions, and household income.
• Public health data, such as infant mortality rates and vaccination coverage among eligible children.
• Appropriately aggregated records from health and human service organizations, such as service reach, waiting times, or outcomes. Access may require permission or an agreement, and individual privacy must be protected.
• Statewide information and data from suitable comparison communities, using compatible definitions and time periods.
• Transportation, police, or court data relevant to the question, interpreted with attention to reporting practices and enforcement.
• Educational information, such as attendance, school completion, learning progress, school climate, and reported violence.
• Environmental information, such as pollutant levels, air quality, water quality, and access to open space.

In addition to archival data, you can use surveys, observations, interviews, and other methods. For example, repeated surveys may examine reported alcohol, tobacco, or other substance use, or specific sexual health practices. Structured observations may track use of public spaces. Service records may describe diagnoses or access to care, although they reflect who obtained services and was assessed. Keep definitions and collection methods consistent, and identify changes in participation or coverage.

Choose indicators that closely match the question. If you want to understand alcohol use among adolescents, a suitable population survey is more direct than a count of traffic violations. Alcohol-involved crashes may be relevant to alcohol-related harm, but nighttime single-vehicle crashes should not all be assumed to involve alcohol. Explain the limitations of any indirect measure.

Graphing outcomes alongside the timing of community changes can help identify patterns for further analysis. A similar upward or downward trend does not establish a causal relationship, particularly when one series is a cumulative count that can only increase.

Figure 5 provides a historical illustration involving community changes intended to prevent child sexual abuse and reported cases in Minnesota. The figure supports discussion of an observed association; it does not establish a reduction in the actual occurrence of abuse or identify the cause of the reported trend.

Figure 5. Historical illustration of an association between documented community changes and a decrease in reported cases of child sexual abuse in Minnesota.

The accompanying example describes a 28% decrease in reported cases over four years as documented community changes accumulated. Interpreting that pattern requires knowing how cases were defined, counted, and reported, and whether those practices changed. Fewer reports may reflect less abuse, reduced identification or reporting, or other factors. The pattern alone cannot establish that the initiative caused the decrease.

Reported cases may increase when people become better able to recognize abuse, seek help, or make reports. A decrease can also occur when access to trusted adults or reporting systems is disrupted. Examine reporting practices, service access, prevention implementation, and other evidence before interpreting either direction as success or failure. Continued tracking is useful, but it must be accompanied by careful investigation of alternative explanations.

5. Did the initiative cause improvements in population-level outcomes?

This question requires more than a graph showing that outcomes changed after implementation. The evaluation needs a credible way to assess what would likely have happened without the initiative. Plan for this question early and seek evaluation expertise appropriate to the setting and available data.

You may track several outcomes, such as traffic injury rates, experiences of violence, unintended pregnancies among adolescents, or vaccination coverage. Define each measure carefully and use comparable observations before and after implementation. Consider the expected delay between activities and outcomes, who was reached, and whether the initiative was implemented as intended.

Examine other plausible explanations, including existing trends, other programs, policy changes, population shifts, and changes in measurement. Where appropriate, experimental or quasi-experimental designs can strengthen causal conclusions. For example, an interrupted time-series design examines repeated observations before and after an intervention; adding a suitable comparison series can help address alternative explanations. These approaches require enough observations, appropriate analysis, and attention to their assumptions. If the evidence cannot establish causation, report what it supports about association or contribution and explain the remaining uncertainty.

Begin early and sustain evaluation for as long as it remains useful

Build evaluation into the initiative's regular work, with a realistic plan for collection, review, and action. Continue following important outcomes long enough to learn whether changes last, including after the initiative ends when that is feasible and appropriate. Review the burden and value of data collection periodically. The aim is a sustainable learning process that informs decisions, not indefinite collection without a clear purpose.

Using evaluation questions to guide improvement

Key evaluation questions help connect the initiative's activities and outputs with changes in programs, policies, practices, and longer-term outcomes. They also help clarify which conclusions the available evidence can support.

The five questions discussed in this section are:

• Is the initiative helping bring about community or system changes related to its mission?
• What factors or processes are associated with community or system change?
• How do community and system changes contribute to the initiative's goals?
• Are those changes associated with improvements in population-level outcomes?
• Did the initiative cause improvements in population-level outcomes?

Answering these questions requires clear definitions, useful data, careful interpretation, and meaningful community participation. Graphs can make patterns easier to see, while records, interviews, and appropriate study designs help explain them. Share findings and limitations openly, agree on practical changes, and review whether those changes improve the work.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: Community Tool Box.