Section 4. Selecting an Appropriate Design for the Evaluation

When you hear the word “experiment,” you may picture people in laboratory coats working with specialized equipment. In a broader sense, however, an experiment simply involves trying something and systematically examining what happens. It can be as simple as testing a different ingredient in a recipe or as complex as developing and evaluating a comprehensive effort to improve child health outcomes across a city or state.

Researchers in public health and the social sciences conduct studies to understand how programs, policies, environments, and other conditions influence behavior and outcomes. Community programs and interventions can also be viewed as opportunities for systematic learning. They often begin with an assumption—or theory of change—about how particular activities are expected to contribute to desired outcomes. Those assumptions should be examined rather than taken for granted. Evaluation helps determine whether change occurred, how the program may have contributed to it, for whom it worked, under what conditions, and what might be improved.

In this section, we will examine several ways to structure an evaluation so that you can understand whether and how your program is contributing to change. These structures are known as experimental, quasi-experimental, or evaluation designs.

What do we mean by a design for the evaluation?

Every evaluation is a form of systematic inquiry. You may want to understand how effective your program or effort is overall, which components are working well and which should be adjusted, whether outcomes differ among groups, or how particular conditions influence results. To answer those questions meaningfully, the evaluation needs a structure that connects the questions you ask with the information you collect and the conclusions you draw. That structure is the evaluation design.

The design depends on the kinds of questions your evaluation is meant to answer.

Common evaluation questions include:

• Does a particular program, intervention, policy, or change in access and opportunity contribute to changes in behavior, physical or social conditions, health or development outcomes, or other indicators of success?
• Which components or elements of the program appear to contribute most strongly to the outcomes?
• What intended or unintended effects occurred, and how did they influence participants, communities, or outcomes?
• What happens when you introduce or adapt a new method, activity, or policy?
• Will an approach that worked in another setting also work in your community, with your participants, and under your local conditions?

To answer questions like these, you need a design that helps distinguish between changes associated with your program and changes that may have resulted from other factors. The stronger and more appropriate the design, the more confidence you can have in the conclusions you draw. Later in this section, we will discuss several ways to structure an evaluation.

Why should you choose a design for your evaluation?

Evaluation may seem straightforward: if conditions improve, the program appears to be working; if they do not, something should change. In practice, evaluation is more complicated. How will you define and measure progress? If outcomes do not improve, which parts of the program should change? If improvement occurs, how confident are you that the program contributed to it rather than another factor? And even when a program is working well, which components are essential and which could be strengthened, adapted, or removed? A thoughtful evaluation design helps answer questions like these.

Reasons to design your evaluation carefully include:

• To strengthen the reliability and usefulness of your evaluation. A well-designed evaluation can provide information you can use with greater confidence. It can help you understand whether change occurred and how strongly the evidence supports the conclusion that the program contributed to that change.
• To identify strengths and areas for improvement. A good design can help show which program components appear effective, where barriers exist, and where adjustments may increase effectiveness, accessibility, or equity.
• To strengthen the credibility of your findings. Clearly designed and transparently conducted evaluations make it easier for participants, community partners, funders, policymakers, and others to understand and assess the evidence.
• To identify factors outside the program that may influence outcomes. Participants' prior experiences, community events, policy changes, economic conditions, changes over time, and many other factors can affect outcomes. A strong design can help identify these influences and clarify how they should be considered when interpreting results.
• To identify unintended consequences. Programs may produce outcomes that were not anticipated, including benefits, harms, inequities, or changes in areas that were not originally measured. A thoughtful design makes it more likely that these effects will be recognized and addressed.
• To provide a clear plan for the evaluation. An appropriate design helps determine what information should be collected, when it should be collected, from whom, and how it should be analyzed. Planning this structure in advance can improve both efficiency and the quality of the resulting information.

When should you choose a design for your evaluation?

Once you have identified your evaluation questions and gathered information about the issue, community context, and possible approaches, the next step is to choose an evaluation design. Ideally, evaluation planning begins alongside program planning. Evaluation should be an integral part of your program rather than something added only after implementation has begun.

In practice, community organizations, public agencies, and other programs may begin formal evaluation months or even years after implementation because of limited staff time, funding, evaluation capacity, or other constraints.

Even when evaluation begins later, the same basic process applies: clarify your questions, gather relevant information, choose a feasible design, and follow the evaluation process systematically. If your program operates in cycles, it may be useful to begin evaluation at the start of a new cycle or at the beginning of participants' involvement so that baseline information can be collected.

Programs with rolling enrollment or continuous services may require a different approach. You might evaluate outcomes among newly enrolled participants, follow different groups over time, or begin examining the program as it currently operates. The design should reflect the structure of the program and the questions you are trying to answer.

Who should be involved in choosing a design?

The Community Tool Box generally recommends a participatory process that includes people affected by the program as well as staff, community partners, evaluators, researchers, and other stakeholders. Evaluation design benefits from this same participation, while also requiring appropriate methodological expertise.

Research and evaluation design can involve complex questions about measurement, bias, comparison groups, statistical analysis, ethics, and interpretation. For this reason, someone with relevant evaluation or research experience should usually help coordinate the technical aspects of the design. That person or team should work collaboratively with participants and community partners rather than making design decisions in isolation. Researchers can explain the strengths, limitations, resource requirements, and ethical considerations of different designs, while community partners can help determine which approaches are appropriate, feasible, and meaningful in the local context.

How do you choose a design for your evaluation?

How do you decide which research design will best serve the purposes of your evaluation?

Consider four areas:

• The nature of the evaluation or research questions you are trying to answer
• Potential challenges to drawing valid conclusions and ways to reduce those challenges
• The research designs that may be appropriate and what each one requires
• Whether a particular design fits your program structure, participant preferences and consent, ethical requirements, timeline, and available resources

We will begin by examining the concerns a research design should address, then consider several common designs and their strengths and limitations, and finally discuss how to select a design that is both feasible and capable of providing useful information.

Note: In this part of the section, we discuss evaluation as a type of research. For that reason, we sometimes use the term “research” even when the same principle could be described as “evaluation.”

Concerns research designs should address

One of the most important considerations in designing research or evaluation is whether the design will provide information that supports the conclusions you want to draw. A weak design may show that something changed without helping you understand why it changed. For evaluation, that can make it difficult to know what should be continued, adapted, or improved.

Consider a simple example. Suppose you measured children's height at age six, encouraged them to eat carrots regularly for three years, and measured them again at age nine. Most would probably be taller. Without an appropriate comparison or other evidence, however, you could not reasonably conclude that carrots caused the increase in height. Children ordinarily grow as they age, and the design would not distinguish that natural development from any effect of the intervention.

Two broad types of validity are especially important in research design: internal validity, which concerns whether the intervention actually contributed to the observed change, and external validity, which concerns how well findings may apply to other people, places, or circumstances.

Threats to internal validity

Threats to internal validity are alternative explanations for the changes observed during an evaluation. Other events, participant characteristics, measurement changes, or natural processes may occur at the same time as the program. If the evaluation cannot distinguish among these influences, it becomes harder to determine how much of the observed change can reasonably be attributed to the program.

Common threats to internal validity include:

• History. Events that occur before or during the evaluation may influence outcomes. These might include participants' prior experiences as well as external events such as policy changes, economic shifts, community conflict, natural disasters, public health emergencies, or other major developments.
• Maturation. People and communities change naturally over time. Children develop as they grow, people gain experience, physical conditions may change with age, and participants may become tired or more comfortable with activities as time passes. These changes can occur independently of the program.
• The effects of measurement or observation. Being surveyed, tested, recorded, observed, or repeatedly asked about a behavior can itself influence how people think or act.
• Changes in measurement. Equipment, assessment tools, data systems, or observers may change over time. Different instruments or observers may also produce different results. Consistent procedures and quality checks can reduce these problems.
• Regression toward the mean. Extremely high or low measurements often move closer to the average when measured again, even without an intervention. This can be particularly important when programs intentionally enroll people because they have unusually high or low values on a particular measure.
• Participant selection. People who participate in a program may differ in important ways from those who do not. Selection procedures may also unintentionally favor people who already have greater access to resources, fewer barriers, or characteristics associated with stronger outcomes. For example, an employment program rewarded only for job placements might be encouraged to enroll people who are already closest to employment while excluding people who face greater barriers. Self-selection can also matter because people who voluntarily enroll may already be particularly motivated to make a change.

Selection becomes especially important when two or more groups are compared. If the groups differ substantially before the intervention begins, differences observed later may reflect those initial differences rather than the program itself.

• Loss of data or participants. Missing information and participant attrition can affect results, particularly when people who leave a study or program differ from those who remain. When comparing groups, different rates or patterns of attrition can make the comparison less reliable.
• The nature and timing of change. Change is rarely perfectly steady. Outcomes may improve, decline, fluctuate, or take time to emerge. Measurements taken over too short a period or at poorly chosen times may not capture the actual pattern of change.
• Combinations of multiple factors. Several threats to validity may occur at the same time. For example, an educational program tested only among students who already have extensive experience learning multiple languages would involve both selection effects and differences in prior experience.

Threats to external validity

External validity concerns how well findings may apply beyond the people, setting, and circumstances included in the evaluation. An intervention that works well in one setting may not produce the same results elsewhere if the populations, resources, institutions, cultures, implementation conditions, or other contextual factors differ.

Threats to external validity, sometimes discussed in terms of generalizability or transferability, often arise from interactions between the intervention and the particular conditions under which it was implemented.

Examples include:

• Interaction between data collection and the intervention. Baseline testing, surveys, or observation may influence how participants respond to the program. If another setting does not use the same data-collection procedures, outcomes may differ.
• Interaction between participant selection and the intervention. A program may work particularly well for people who voluntarily enroll, are highly motivated, or meet specific eligibility criteria, but those findings may not apply equally to people with different experiences or circumstances.

For example, people who participate in a parenting program because of a court or child-welfare requirement may experience the program differently from people who enroll voluntarily. Differences in motivation, stress, trust, available supports, and other circumstances may influence outcomes and should be considered when comparing results.

• The effects of the research arrangements. People may respond differently because they know they are being observed or because of their relationship with particular researchers, facilitators, or staff members.

The broader lesson is that the act of observation can sometimes influence the behavior being observed. Evaluators should consider whether data-collection procedures themselves may be contributing to the outcomes they see.

• Exposure to multiple programs, services, or interventions. Participants may be involved in several programs or experience multiple changes at the same time. These combined influences can make it difficult to determine the effect of one intervention in isolation.

For example, an adult might participate in a high school completion program while also receiving substance use recovery services. A person managing diabetes might receive medication while participating in nutrition, physical activity, or peer-support programs. These overlapping services may reinforce one another, interfere with one another, or make it difficult to identify the contribution of any single component.

Common research designs

Research design is a large field, and this section provides only an introduction. Evaluation designs differ in several important ways, including how many times outcomes are measured, when measurements occur, whether one or multiple groups are included, and how participants or groups are selected.

Program evaluations often seek to answer three basic questions:

• Did meaningful change occur during the evaluation period?
• To what extent did the program, intervention, or effort contribute to that change?
• What program components, contextual conditions, or outside influences helped produce or prevent the change?

Outcomes may be influenced by the program, by outside factors, or by both. Participants' experiences, natural development, policy changes, economic conditions, organizational changes, community events, and other influences may all affect results. Understanding those influences can help you improve the program and interpret outcomes more accurately.

Measuring conditions only before and after a program can show whether something changed, but it may not explain why. For that reason, evaluation designs should consider potential threats to validity as well as practical and ethical constraints.

Research designs often differ according to the number and timing of measurements and whether one or multiple groups are included. We will begin with single-group designs and then consider designs involving multiple groups.

Before continuing, it is useful to define several basic research terms.

Researchers commonly refer to measurements or observations collected before an intervention begins as baseline measures or baseline observations. Baseline information establishes a starting point against which later observations can be compared.

Other useful research terms include:

• Independent variables are the program, intervention, policy, method, or condition whose effects you want to examine. An evaluation may include one independent variable or several.
• Dependent variables are the outcomes or conditions expected to change in response to the independent variable. These might include behaviors, health outcomes, community conditions, knowledge, attitudes, or other indicators.
• Measures are the tools or procedures used to assess those outcomes. They may include community assessments, administrative data, observations, surveys, interviews, validated scales, clinical measurements, or counts of particular events or conditions.
• Observations may involve formal measurement or systematic documentation of what occurs in a particular setting. In research, the term is sometimes used broadly to describe many forms of measurement and data collection.

Pre- and post- single-group design

One of the simplest designs is a pre- and post-measurement design. You measure an outcome before the intervention, implement the program, and measure the same outcome again afterward. Examples might include unemployment rates, environmental pollution levels, health indicators, or participant knowledge.

This design can show whether change occurred between the beginning and end of the evaluation period, but by itself it usually cannot establish why the change occurred. Other events, natural trends, changes in measurement, or participant characteristics may explain some or all of the difference. Even so, collecting information before and after an intervention generally provides more useful information than having no baseline at all.

Consider blood pressure in a cardiovascular health program. A lower measurement at the end of the program than at the beginning does not show what happened between those two points. Blood pressure may have fluctuated throughout the evaluation period, measurement conditions may have changed, or other health or life changes may have influenced the result. Repeated measurement can provide a clearer picture.

Interrupted time series design with a single group

An interrupted time series design uses repeated measurements before and after the introduction of an intervention. Establishing a pattern before implementation provides a baseline trend, while repeated measurements afterward help show whether that pattern changed when the intervention began.

This approach can provide substantially stronger evidence than a simple pre- and post-design because it shows how outcomes change over time. If the outcome begins changing before the intervention starts, for example, that suggests that another factor may be contributing to the trend.

Some time-series designs also examine the effects of introducing different program components at different times. When appropriate and ethically feasible, this can help evaluators understand which components appear associated with particular changes.

Many variations of interrupted time series designs are possible, including changes in measurement frequency, staggered implementation, or evaluating combinations of program components.

When using a time-series design, it is important to examine outcomes closely enough to identify meaningful patterns. An overall average can sometimes hide changes occurring in particular periods, locations, or groups. Disaggregating information when appropriate can reveal effects that would otherwise be missed.

Interrupted time series design with multiple groups

A multiple-group time series design applies similar repeated measurements across two or more groups, sites, or communities. The intervention may be introduced at different times across groups, creating multiple baselines that can strengthen confidence that observed changes are associated with the intervention rather than a single external event.

This type of design may allow you to:

• Examine the same method or program across two or more groups or sites
• Examine whether an approach produces similar or different results across populations or settings
• Vary the timing or intensity of an intervention when doing so is appropriate
• Compare different interventions or implementation approaches
• Examine whether the order or sequence of program components influences outcomes

Many other variations are possible, depending on the program and evaluation questions.

Control or comparison group design

Another common approach is to compare outcomes between a group receiving the intervention and another group that does not receive the same intervention during the same period, receives usual services, or receives a different approach. A well-designed comparison can help distinguish program effects from changes that would have occurred anyway.

A control group is generally part of a design intended to isolate the effect of the intervention by holding other influences as similar as possible. A comparison group may be selected because it resembles the intervention group on relevant characteristics, even when participants are not randomly assigned.

Randomized controlled designs are among the strongest methods for estimating causal effects when random assignment is feasible, ethical, and appropriate. In these designs, eligible participants or groups are assigned by chance to different study conditions. Randomization helps reduce systematic differences between groups before the intervention begins, although it does not eliminate every possible limitation or guarantee that findings will generalize to other settings.

For example, eligible participants might be randomly assigned to receive either a new program or an existing standard service. If implementation and measurement are otherwise comparable, differences in outcomes between groups can provide strong evidence about the effect of the different approaches.

Community organizations may not always be able—or ethically justified—to withhold a potentially beneficial program solely for evaluation purposes. Alternatives may include wait-list designs, phased implementation, usual-services comparisons, matched comparison groups, or other quasi-experimental approaches. The evaluation should not create avoidable harm or unfairly restrict access to needed services simply to strengthen the research design.

When a program has limited capacity and already uses a wait list, phased or randomized enrollment may sometimes create a comparison while preserving eventual access to services. Another option is to provide participants waiting for the full program with existing resources, alternative services, or another appropriate form of support rather than leaving them without assistance solely for research purposes.

When no appropriate control or wait-list group exists, organizations may use a comparison group from another program, community, or setting. Because those groups may differ in important ways, evaluators should examine baseline characteristics carefully and be cautious about attributing differences in outcomes solely to the intervention. In some situations, a strong interrupted time series design may provide more credible information than a weak comparison between substantially different groups.

Groups may appear similar while differing in ways that influence outcomes. For example, participants who voluntarily enter a substance use treatment program may differ from people who are required to participate through a court or institutional process. Motivation, trust, available supports, prior treatment experiences, and other circumstances may influence results independently of the treatment model.

Average results can also hide important differences within a single group. An intervention that appears only modestly effective overall may work particularly well—or poorly—for participants with particular circumstances, experiences, or barriers. Examining variation in outcomes can provide important information for improving equity and effectiveness.

Choosing a design

This section provides only an introduction to evaluation design. Research methods include many additional experimental, quasi-experimental, observational, qualitative, mixed-methods, implementation, and participatory approaches. Statistical methods can also help address some design limitations, although analysis cannot fully compensate for poor-quality or inappropriate data.

When possible, seek technical assistance from people with evaluation expertise. Universities, research centers, public agencies, community evaluation organizations, funders, and consultants may be able to provide methodological support. Whatever design you use, community members and program participants should remain involved in identifying meaningful questions, outcomes, ethical concerns, and interpretations.

Your goal is to obtain the most credible and useful information possible given your evaluation questions, program structure, ethical responsibilities, participant preferences and consent, timeline, and resources. A community evaluation does not need to resemble a laboratory experiment to be valuable. It does need to be designed carefully enough that you understand what the evidence can and cannot tell you.

Consider your evaluation questions

What do you need to know? If your question is simply whether a particular condition changed, a pre- and post-design may sometimes be sufficient. If you want to know whether the program contributed to that change, you will need a design that provides stronger evidence about alternative explanations.

Comparison groups can be useful, but they are not the only way to strengthen an evaluation. Repeated measurements, multiple forms of evidence, qualitative information, implementation data, and careful documentation of external events can provide a much fuller picture. For many community organizations, an interrupted time series design may offer a practical balance between rigor and feasibility.

If your program already includes multiple groups, sites, or classes working toward similar goals, you may be able to compare outcomes across them or stagger implementation in ways that provide additional information. Any such approach should be developed with attention to fairness, participant expectations, and whether differences in implementation are appropriate.

Sometimes the central question is not simply whether a program works, but whether it works better than another available approach. Employment training, parenting support, chronic disease management, conflict resolution, and many other programs may raise this question. Depending on the setting, comparisons may involve another program, usual services, published research, administrative information, or community-level indicators.

Some community efforts have outcomes that can be measured relatively directly. An initiative focused on reducing water pollution, for example, may use repeated water-quality measurements as a central outcome. Even in these situations, however, you may also want to understand which activities contributed to improvement, whether benefits were distributed equitably, and whether changes were sustained over time.

Consider the nature of your program

What does the program look like, and what is it intended to accomplish? Does it work with participants individually or in groups? Does it operate in fixed cycles, such as classes or workshops, or can people enter at different times? How much depends on staff implementation, participant activity, partnerships, organizational practices, or the surrounding environment? How important are relationships, physical setting, accessibility, language, organizational culture, and other contextual factors?

• If participants take part in distinct groups or sites, a multiple-group design may sometimes be practical. If participation is primarily individual, a single-group longitudinal or time-series design may be more appropriate.
• If your program is time-limited, choose a design that fits the program schedule and allows enough time to capture meaningful outcomes. If multiple program cycles occur, lessons from earlier cycles may inform adaptations to later ones, but those changes should be documented clearly.
• If your program has no clear beginning or end, you may need to follow participants from their individual starting points or organize analysis around different levels of exposure, participation, or baseline conditions.

Community programs operate in changing environments. People entering a program at different times may encounter different staff, participants, community events, resources, policies, or organizational conditions. These differences can influence experiences and outcomes even when the formal program model remains the same. Evaluators should document important contextual changes and consider them when interpreting results.

• If organizational culture, relationships, or program context are central to the intervention, include measures that capture those factors rather than treating them as background conditions. Comparisons across different organizational settings may also be useful when they are appropriate and sufficiently similar.

Many evaluation designs can be adapted to different situations. The important goal is to select an approach that fits both the evaluation questions and the realities of the program.

Consider what participants and staff will consent to

Evaluation methods should respect privacy, dignity, autonomy, safety, and informed participation. Extensive observation, recording, repeated testing, or collection of sensitive information can create burdens or risks for participants and staff. A technically strong design is not appropriate if it depends on methods that are unnecessarily intrusive or unacceptable to the people involved.

Privacy and safety concerns may be especially important when participation in a program involves sensitive health information, immigration status, substance use, interpersonal violence, legal issues, or other information that could create harm if disclosed. Participants should not be pressured to provide information that is not necessary for the evaluation.

Explain what information will be collected, why it is needed, how it will be used, who will have access to it, what protections will be in place, and what choices participants have. Include participants and staff in designing data-collection procedures whenever possible. Their concerns may reveal risks or burdens that evaluators would otherwise overlook. :chatgpt-content-reference{index="1"}

Consider your time constraints

Choose a design that can provide credible information within the time available. The goal is not necessarily to create the most complex design possible, but to select an approach that is appropriate to the questions and transparent about its limitations.

Common time constraints include:

• Program structure. Evaluation may need to align with a regular program cycle, school year, funding period, or implementation schedule.
• Funding. Grants and contracts may establish deadlines for reporting or evaluation. Pilot programs may also have limited time to demonstrate early outcomes, even when longer-term effects will take more time to appear.
• Participants' schedules and responsibilities. Work, school, caregiving, transportation, seasonal employment, cultural or religious commitments, and other responsibilities may affect when people can participate in program or evaluation activities.
• The urgency of the issue. When a program addresses serious risks to health or safety, evaluation timelines should support timely decisions rather than delaying necessary action solely for methodological reasons.
• The availability of evaluation support. Evaluators, analysts, community partners, and other people involved in the evaluation may only be available during particular periods.

Consider your resources

Resources include funding, staff time, community knowledge, partnerships, technology, space, equipment, data systems, and evaluation expertise. A lack of one resource can sometimes be addressed through another—for example, by partnering with a university, public agency, community organization, or evaluator who can provide technical assistance.

Begin by identifying the design that best fits your questions and ethical responsibilities, then determine how to make that design feasible. You may need to simplify or adapt it, but understanding the strongest realistic option can help you make intentional compromises rather than automatically choosing the least resource-intensive design.

In Summary

Evaluation design strongly influences how confidently you can interpret your findings and how useful those findings will be for improving a program or intervention. A strong design considers threats to internal validity—whether the intervention contributed to the observed change—as well as external validity or transferability—whether findings may apply in other settings or populations.

Designs such as pre- and post-measurement, interrupted time series, multiple-group designs, comparison groups, and randomized controlled designs each have different strengths, limitations, ethical considerations, and resource requirements. They can often be adapted or combined to fit the needs of community programs.

A useful evaluation design should address your questions while also reflecting the nature of the program, community context, participant and staff consent, ethical responsibilities, available time, and resources. The goal is not simply to create the most rigorous design possible, but to generate credible, relevant, and actionable information in a way that respects the people and communities involved.