Section 3. Refining the Program or Intervention Based on Evaluation Research

A Community Health Center conducted an evaluation of a program designed to support physical activity and other health-promoting behaviors among people at increased risk for heart disease.

The evaluation showed mixed results.

About 15% of participants experienced substantial improvements. They increased their physical activity, made changes in nutrition, improved their fitness, lost weight, and experienced reductions in blood pressure and pulse rate. They also reported feeling more energetic and expressed high satisfaction with the program.

About 70% participated in physical activity, but less consistently than the program had anticipated. Health outcomes varied within this group. Some participants experienced modest reductions in blood pressure, while others maintained approximately the same levels they had when they entered the program.

The remaining 15% included people who left the program early and people who participated only minimally or irregularly. Most showed little measurable change in weight, blood pressure, or self-reported well-being, although a small number experienced positive outcomes.

What could the Community Health Center learn from these results?

The evaluation suggested that some participants benefited substantially, while others encountered barriers to participating consistently or did not experience the intended outcomes.

Rather than concluding simply that the intervention worked for people who “stuck with it,” the Center could examine how the program's design, accessibility, supports, implementation, and participants' circumstances contributed to these different experiences.

This chapter has discussed the elements of conducting a research-based evaluation. Evaluation, however, is not only about determining whether a program succeeded or failed.

It is a tool for understanding what happened, why results may have differed across participants or settings, and how an intervention can be improved.

In this section, we examine how evaluation findings can be used to strengthen programs and community interventions.

What do we mean by refining the intervention?

Evaluation findings can help you critically reflect on your work and identify opportunities for improvement.

Some useful reflection questions include:

• What are we seeing? What activities were implemented? Who participated? What changes, patterns, differences, or trends appear in the data?
• What might explain what we are seeing? What evidence suggests that the intervention, implementation process, community conditions, or other factors contributed to the results?
• What are the implications for improvement? What should be sustained, adapted, strengthened, discontinued, or examined further?

The questions you ask will depend on the nature of the intervention and the purpose of the evaluation.

Consider bringing together participants, staff, community partners, evaluators, and others with relevant experience to review findings.

Graphs, charts, qualitative findings, participant feedback, and other accessible formats can help people identify patterns and discuss what those findings may mean.

A facilitator may help the group consider different interpretations, identify unanswered questions, and develop recommendations.

Refining an intervention means using evaluation findings and community knowledge to make the work more effective, equitable, relevant, accessible, and sustainable.

Depending on what you learn, you might:

• Strengthen parts of the intervention that appear to be effective
• Adapt parts that are not working as intended
• Remove components that provide little benefit
• Add supports that address barriers to participation
• Adjust the intervention as community conditions, needs, or priorities change
• Examine whether different groups are experiencing different levels of access, participation, benefit, or harm

Meet with people who are involved in or affected by the intervention to review the findings, identify priorities for improvement, and determine what changes should be tested.

Evaluation results should inform decisions, but they should not be interpreted in isolation from participant experience, implementation conditions, and other relevant evidence.

In the Community Health Center example, staff, participants, and evaluation partners reviewed the findings together.

They noted that people who participated regularly often experienced improvements in health, but they also asked why regular participation was easier for some people than for others.

Their questions included:

• What features of the program helped participants engage consistently?
• What barriers made participation more difficult?
• Did transportation, work schedules, caregiving, cost, disability access, program timing, social support, or other conditions affect participation?
• Why did some participants experience improvements in blood pressure even when their physical activity changed only modestly?
• Should the program include stronger nutrition, stress-management, medication-support, social-support, or other components?
• Were particular groups experiencing different outcomes, and if so, what might explain those differences?

By asking what happened, what may explain the findings, and what should change, the Center can make more informed decisions about the next version of the program.

Why should you use evaluation findings to refine the intervention?

One of the most important purposes of evaluation is learning how to improve.

If an intervention is not producing the intended results, the choice is not simply between ending it and trying something else at random.

Evaluation can help identify:

• What is working
• What is not working
• For whom different parts of the intervention appear to work
• Which barriers are affecting participation or outcomes
• Whether the intervention was implemented as intended
• Whether the underlying theory or assumptions need to change
• Whether unintended benefits or harms have occurred

Using evaluation findings can help an organization make better-informed decisions about where to invest time, staff, funding, and other resources.

It can also strengthen accountability to participants, community members, funders, and partners by showing that the organization is willing to learn from evidence and change when necessary.

Most importantly, evaluation can help improve the likelihood that the intervention contributes to the outcomes that matter to the community.

When should you refine the intervention?

Monitoring, learning, and improvement should occur throughout the life of a program or initiative.

The appropriate review schedule depends on:

• The length of the intervention
• How quickly meaningful outcomes can reasonably be expected
• The level of risk associated with the intervention
• The availability of data
• Funding or reporting cycles
• Changes in community conditions

Some programs conduct a major review annually, but there is no universal schedule.

Smaller adjustments can often be made sooner when monitoring data, participant feedback, implementation experience, or emerging evidence identifies a clear need.

At the same time, avoid making major judgments before the intervention has had a reasonable opportunity to produce the outcomes it was designed to influence.

The challenge is to avoid both ending a promising effort too early and continuing an ineffective or harmful approach longer than the evidence supports.

Consider refinement when the intervention is not producing the intended results

If available evidence suggests that outcomes are not improving as expected, examine why.

Possible explanations may involve:

• Implementation
• Program design
• Participation barriers
• Insufficient intensity or duration
• Incorrect assumptions about what produces the desired outcome
• Changes in external conditions
• Measures that do not adequately capture important changes

When participation decreases or people leave the program earlier than expected

Do not assume that people leave because they are unmotivated.

Ask what may be affecting continued participation.

Possible factors include:

• Scheduling
• Transportation
• Cost
• Childcare or caregiving responsibilities
• Accessibility
• Work demands
• Language
• Safety
• Program relevance
• Relationships with staff or other participants
• Whether the program feels respectful and responsive
• Changes in participants' circumstances

Between cycles of a time-limited or recurring program

Programs that serve new groups over repeated cycles can use each round as an opportunity to learn.

Compare results across cycles, document changes that were made, and examine whether those changes improved access, implementation, participant experience, or outcomes.

When participants, community partners, staff, or funders raise concerns

Requests for change should be taken seriously.

Evaluation data can help determine whether a requested change is supported, whether additional information is needed, and what effects the change might have.

When funding or other resources change

Reduced resources may require difficult decisions.

Evaluation can help identify:

• Essential program components
• Components that could be redesigned more efficiently
• Changes likely to cause the least harm
• Services or supports whose loss could disproportionately affect participation or outcomes

When community conditions or priorities change

Economic conditions, public health needs, policies, housing conditions, transportation, community safety, technology, population changes, or other factors may change the context in which an intervention operates.

Evaluation and community input can help determine whether the program should adapt.

Who should be involved in refining the intervention?

Include people whose experience, knowledge, responsibilities, or decisions are relevant to the intervention.

This will vary by program, but important participants often include:

• People participating in or directly affected by the intervention. They can describe what participation is actually like, what supports or barriers they experience, what feels useful, and what changes may make the intervention more relevant or accessible.
• Staff members and volunteers. They can describe implementation challenges, workload, relationships, resource needs, and practical consequences of proposed changes.
• Community partners and others who support implementation. Depending on the intervention, these might include educators, public officials, health professionals, employers, funders, community organizations, or other partners.
• People involved in the evaluation. They can help interpret findings, explain limitations, identify unanswered questions, and distinguish what the evaluation can and cannot support.

Participation should be meaningful rather than symbolic.

People directly affected by the intervention should have opportunities to influence decisions about what changes are made, particularly when those changes affect their access, experience, or outcomes.

In the Community Health Center example, the review team included participants with different experiences of the program, health-care staff, organizational leadership, program staff, and an evaluation partner.

The group reviewed quantitative results, participant interviews, implementation records, and relevant research before recommending changes.

How do you refine an intervention based on evaluation findings?

A useful framework is to examine three related areas:

• Process: What did the program actually do, and how was it implemented?
• Impact: What shorter-term changes occurred in the behaviors, conditions, systems, environments, or other factors the intervention was intended to influence?
• Outcomes: What longer-term changes occurred in the health, well-being, development, systems, or community conditions that ultimately mattered?

These areas are related, but they should not be treated as a simple automatic chain.

A program can be implemented as planned and still fail to produce the intended impact.

It can also produce useful outcomes even when implementation differs from the original plan.

Evaluation should help determine what happened and which explanations are most consistent with the evidence.

Process

Process evaluation compares what you planned with what actually occurred.

It can examine both program operations and participants' experiences.

Areas to consider include:

• Community participation. Were the people and organizations you hoped to involve able to participate meaningfully? Who was not reached or included, and why?
• Community assessment. Did the assessment provide the information needed to understand community priorities, strengths, conditions, and barriers?
• Program planning. Was planning participatory? Was relevant research used? Were people directly affected involved in shaping the approach?
• Staff and volunteer recruitment. Did the organization recruit people with the skills, experience, capacity, and community knowledge needed?
• Training and support. Were staff and volunteers adequately prepared and supported throughout implementation?
• Outreach and engagement. Did outreach reach the people the intervention was intended to involve? Were there groups that experienced barriers to participation?
• Implementation. What activities, services, supports, and environmental changes actually occurred? How did participants experience them?
• Evaluation. Was data collection carried out as planned? Were the measures appropriate, accessible, and useful?
• Timelines and milestones. Were major activities completed when expected? If not, what affected the schedule?

Differences between the planned and actual intervention are not automatically failures.

Adaptations may be necessary because of changing conditions, participant feedback, new evidence, resource constraints, or implementation learning.

The important questions are:

• What changed?
• Why did it change?
• Who influenced the change?
• Did the change improve or reduce effectiveness, accessibility, equity, or feasibility?
• Should the adaptation be retained?

Possible reasons implementation differed from the plan

• Activities required more time or resources than expected
• Staff capacity or specialized expertise was insufficient
• Expected funding, partnerships, materials, or facilities were unavailable
• Participation was lower or higher than expected
• Participants or community partners identified a need for change
• New evidence suggested a better method
• Community conditions changed
• Accessibility or language barriers emerged
• Staffing or organizational circumstances changed
• Unexpected emergencies or disruptions occurred
• The original plan was not sufficiently realistic or responsive to community conditions

Understanding why implementation changed can help distinguish between a problem with the intervention itself and a problem with how it was implemented.

Impact

Impact refers to shorter- or intermediate-term changes that may contribute to the longer-term outcomes the intervention ultimately seeks.

Depending on the program, impacts might involve:

• Behaviors
• Knowledge or skills
• Access to services
• Organizational practices
• Policies
• Social or environmental conditions
• Protective factors
• Reduced exposure to risk

In the Community Health Center example, increased access to and participation in physical activity might be an intermediate impact.

Longer-term outcomes could include improved cardiovascular health, quality of life, or reduced risk of chronic disease.

If the intended impact did not occur, examine more than participant behavior.

Ask whether:

• The intervention was accessible
• The activities were relevant and acceptable
• Participants had enough opportunity and support to engage
• Structural or environmental barriers limited participation
• The intensity or duration was sufficient
• The program's assumptions were accurate
• The measures captured the changes that mattered

Examine differences in experience and outcomes

Aggregate results can hide important differences.

When appropriate and ethically collected, examine whether access, participation, experience, or outcomes differ across groups.

Relevant differences might involve:

• Age
• Disability
• Geography
• Language
• Income or economic conditions
• Race or ethnicity
• Gender
• Other characteristics relevant to the intervention

Differences should prompt further inquiry rather than assumptions about the people experiencing them.

Ask whether program design, institutional practices, access barriers, resource distribution, discrimination, environmental conditions, or other factors may contribute to the pattern.

Cultural responsiveness and community fit

If an intervention is not working well for particular communities, avoid describing the problem simply as “cultural influences” that participants failed to overcome.

Examine whether the intervention:

• Reflects participants' priorities and experiences
• Uses appropriate language and communication
• Is accessible within people's daily circumstances
• Respects different cultural practices and perspectives
• Was designed with meaningful community participation
• Addresses social, economic, organizational, and environmental conditions that influence participation

Lack of participation or limited outcomes may reflect a mismatch between the intervention and the community rather than a deficiency in the people the program is intended to serve.

Outcomes

Outcomes are the changes in health, well-being, development, systems, or community conditions that the intervention ultimately seeks to influence.

Some outcomes can be measured during or soon after an intervention.

Others may take months or years to become visible.

Evaluation should therefore distinguish between:

• Short-term changes
• Intermediate changes
• Longer-term outcomes

Avoid assuming that a lack of immediate long-term outcomes automatically means the intervention failed.

Consider whether:

• Enough time has passed
• The expected pathway from activities to outcomes is realistic
• External conditions influenced the result
• Measurement methods were sensitive enough to detect meaningful change
• Outcomes differed among groups
• Important unintended outcomes occurred

When the program produces the intended outcomes

Positive results are an opportunity to understand what should be maintained and strengthened.

Ask:

• Which components contributed most to the results?
• Were benefits experienced broadly or concentrated among particular participants?
• Are there evidence-informed ways to strengthen the intervention further?
• Can successful elements be sustained with available resources?
• Are there people who still face barriers to participating or benefiting?
• Can efficiency improve without reducing quality, accessibility, or equity?

Improvement does not always mean expansion.

Sometimes the most important next step is protecting what works, improving access, or making results more consistent.

When the program produces only some of the intended outcomes

Partial results may indicate that some parts of the approach are useful while others need refinement.

Possible explanations include:

• The intervention addressed only some of the factors influencing the outcome
• Program intensity or duration was insufficient
• Participation barriers limited exposure to the intervention
• Some components were more effective than others
• Implementation varied across sites or groups
• Important services or supports were missing
• Language, accessibility, cultural responsiveness, or program design created barriers
• External conditions affected the outcome
• More time may be needed for longer-term effects to appear

In the Community Health Center example, staff might compare the experiences of people with different levels of participation rather than assuming that motivation alone explains the differences.

They might learn that people who participated regularly had more flexible work schedules, transportation, social support, or access to convenient exercise options.

The Center could then test changes such as flexible scheduling, transportation assistance, virtual or home-based options, peer support, nutrition services, or other supports identified by participants.

These changes should be evaluated rather than assumed to be effective simply because they seem plausible.

When the program produces no intended outcomes

Lack of intended outcomes may require substantial changes, but first examine the evidence carefully.

Ask:

• Was the intervention implemented sufficiently to test the underlying approach?
• Were the measures appropriate?
• Was enough time available for change to occur?
• Were important external influences present?
• Were people able to access and participate in the intervention?
• Was the program theory itself incorrect or incomplete?

Organizations should be willing to redesign or discontinue an intervention when credible evidence shows that the approach is ineffective, harmful, or no longer appropriate.

Time and resources already invested should not be the main reason to continue an ineffective approach.

Examine unintended outcomes

Evaluation should look for effects that were not originally anticipated.

These may be:

• Positive
• Negative
• Mixed
• Different across groups

Do not immediately assume why an unintended outcome occurred.

Use additional data, participant input, and other evidence to understand the mechanism before deciding how to respond.

Use a “But why?” approach carefully

Working backward from outcomes can help identify where an intervention may need improvement.

Examine the outcomes

If the intervention produced the intended outcomes, identify what contributed to those results and what should be maintained.

Also examine whether benefits were distributed equitably and whether any unintended effects occurred.

Examine the impact

If outcomes fell short, examine whether the intermediate changes expected from the intervention actually occurred.

If they did occur but the longer-term outcomes did not, consider whether:

• The assumed pathway to outcomes was incomplete
• Additional factors influence the outcome
• The intervention needs more time
• Measurement did not capture meaningful change

Examine the process

If the expected impact did not occur, examine how the intervention was planned and implemented.

Determine whether the issue involved:

• Implementation quality
• Access
• Staffing
• Resources
• Community participation
• Program design
• Communication
• Cultural responsiveness
• External conditions

If the intervention was implemented well but still produced little or no impact, the underlying program theory or methods may need substantial revision.

Continue learning and adapting

Make the changes supported by the available evidence, document what you changed and why, and evaluate the revised intervention.

Improvement should be iterative.

At the same time, avoid changing programs simply for the sake of constant experimentation.

Preserve effective components, assess the consequences of adaptations, and give changes enough time to be meaningfully evaluated.

In Summary

Evaluation is valuable not only because it tells you whether an intervention produced the intended results, but because it helps explain what happened and what should happen next.

Refining an intervention means using evaluation findings, participant experience, implementation knowledge, and other relevant evidence to improve effectiveness, accessibility, equity, relevance, and sustainability.

Examine process, impact, and outcomes separately and together.

Process evaluation helps determine what was actually implemented and how people experienced it.

Impact evaluation examines shorter- and intermediate-term changes in the factors the intervention was intended to influence.

Outcome evaluation examines the longer-term changes the intervention ultimately seeks to produce.

Avoid interpreting limited participation or weak outcomes simply as a failure of participants to stay motivated or follow the program.

Examine program design, accessibility, implementation, structural conditions, social and economic barriers, and community fit.

Involve people directly affected by the intervention as meaningful partners in interpreting findings and deciding what should change.

Look for differences in access, experience, and outcomes across groups without assuming that those differences are caused by individual or cultural deficits.

Be willing to preserve what works, adapt what can be improved, investigate uncertainty, and discontinue approaches that credible evidence shows are ineffective or harmful.

Document changes and evaluate them so that improvement becomes a cycle of learning rather than a series of untested adjustments.

The goal is not simply to make participants conform more closely to a program.

It is to use evidence and community knowledge to make the intervention better able to serve the people and purposes for which it exists.