| Community Tool Box

This section was originally adapted from the 1999 CDC report "Framework for Program Evaluation in Public Health," developed by Bobby Milstein, Scott Wetterhall, and the CDC Evaluation Working Group. It retains the original framework's organization while updating the discussion and identifying subsequent developments in evaluation practice.

Around the world, communities develop programs and initiatives to improve the conditions that shape people's lives. These efforts may prevent violence, expand access to safe and affordable housing, support student learning, or address other priorities identified by community members.

How do we know whether these programs are working, for whom, and under what conditions? How can we improve their effectiveness, accessibility, and fairness? And how can an organization decide which promising approaches are appropriate for its community?

Program evaluation provides a systematic way to explore these questions. By combining evidence with community knowledge, organizations can understand implementation, assess results, identify unintended consequences, and use what they learn to improve their work.

In 1997, the United States Centers for Disease Control and Prevention (CDC) convened an Evaluation Working Group to organize the essential elements of program evaluation. Its framework was published in 1999 and became the basis for this section. CDC updated the framework in 2024. The discussion below preserves the original six-step teaching structure and explains how it relates to the newer guidance.

Before beginning, it helps to define several terms used throughout this section.

By evaluation, we mean a systematic process for assessing the merit, value, or significance of an effort. Evaluation brings together questions, evidence, interpretation, and judgment to support learning and decisions. Its methods should fit the purpose, community context, and consequences of those decisions.

Throughout this section, the term program describes the effort being evaluated. It may include activities intended to improve outcomes across a community, within settings such as schools or workplaces, or among particular populations, such as young people, people affected by violence, or people living with HIV. This definition is intentionally broad.

Examples of different types of programs include:

• Direct service interventions, such as a program offering free school breakfasts to support children's access to nutritious food
• Community organizing efforts, such as a campaign led by farmworkers to improve wages and working conditions
• Research initiatives, such as a community partnership examining approaches to reducing racial inequities in access to health services
• Monitoring and surveillance systems, such as a system that tracks housing conditions to inform inspection and repair priorities
• Advocacy efforts, such as a campaign seeking policies that reduce exposure to secondhand smoke
• Social marketing campaigns, such as a locally developed campaign helping families find available nutrition and infant-feeding support
• Infrastructure and capacity-building projects, such as an initiative strengthening agencies' ability to support community-led development
• Training programs, such as a workforce development partnership that combines job training with support for transportation and other employment barriers
• Administrative systems, such as a revised appointment process intended to reduce waiting times and improve access to services

Program evaluation, the focus of this section, supports learning and accountability in community health and development work. It can examine a specific project, a set of related activities, or a broader initiative. The scope should be explicit so that the evaluation's questions and conclusions match the effort being assessed.

Community members and other interested or affected parties include people who participate in a program, people affected by its decisions, those who implement or fund it, and those who may support or question its approach. Earlier evaluation literature often uses the term stakeholders; newer CDC guidance uses interest holders. Useful questions include: Who is affected? Whose knowledge is needed? Who makes decisions? Whose perspectives have been excluded?

This section presents a practical framework for developing a shared understanding of evaluation. Its goal is to help people involved in community health and development assess their efforts, learn together, and use findings responsibly.

Why evaluate community health and development programs?

Evaluation can be closely connected to everyday program operations. Our emphasis is on practical, ongoing evaluation that involves program staff, community members, and other partners, with specialized expertise when needed. This approach can support both program improvement and accountability to the people a program is intended to serve.

For example, evaluation complements program management by:

• Clarifying program plans, assumptions, and intended results
• Improving communication and shared understanding among community members and partners
• Gathering evidence and feedback to improve effectiveness, address unequal outcomes, and account for the use of resources

People working to improve their communities already evaluate their work informally when they ask questions, consult partners, reflect on feedback, and adjust their approach. For low-stakes decisions, these practices may provide enough information. When decisions involve substantial resources, sensitive information, or significant consequences for people, a more systematic and documented evaluation can make the basis for those decisions clearer and more defensible.

How do you evaluate a specific program?

Before beginning an evaluation, work with the people involved to answer these questions:

• What will be evaluated, and how will the findings be used?
• What criteria will be used to assess program performance, and who will help choose them?
• What levels of performance would indicate success, including fair access and outcomes?
• What evidence is needed to assess performance against those criteria?
• What conclusions are justified by the evidence, and what uncertainties remain?

Consider Drive Smart, a hypothetical program designed to reduce alcohol-impaired driving. The following targets illustrate how an evaluation might be planned; they are not recommended benchmarks for every community. A real program would establish its targets using baseline information, available resources, community priorities, and a realistic timeline.

• What will be evaluated?
  • Drive Smart's public education and "Safe Rides" transportation service, including their reach, accessibility, implementation, and relationship to alcohol-impaired driving.
• What criteria will be used to assess program performance?
  • The percentage of community residents familiar with the program and its goals
  • The number of people using "Safe Rides," including whether the service reaches residents facing transportation barriers
  • The percentage of respondents reporting alcohol-impaired driving during a consistently defined period
  • The number of recorded crashes involving alcohol impairment, interpreted alongside changes in traffic volume, reporting, and other relevant conditions
• What levels of performance would indicate success?
  • 80% of community residents will know about the program and its goals after the first year.
  • The number of people using "Safe Rides" will increase by 20% relative to a defined baseline during the first year.
  • The percentage reporting alcohol-impaired driving will decrease by 20% relative to baseline during the first year. For example, a decline from 10% to 8% is a 20% relative decrease, or a decrease of 2 percentage points.
  • Recorded crashes involving alcohol impairment will decrease by 10% relative to a comparable baseline period during the first two years, with attention to uncertainty when counts are small.
• What evidence will indicate performance against these targets?
  • A community survey using a defensible sampling approach and accessible participation options will assess awareness and reported behavior at baseline and follow-up. The evaluation will examine nonresponse and the limitations of self-reported information.
  • "Safe Rides" records will document service use, distinguishing trips from unique users where this can be done without collecting unnecessary identifying information.
  • Crash records will provide information on documented alcohol involvement, with consistent definitions and attention to missing information or changes in reporting practices.
• What conclusions are justified by the available evidence?
  • What changed, for whom, and how plausibly did Drive Smart contribute? Could other policies, services, or community conditions explain the findings?
  • If intended changes have not occurred, should the program adjust its approach, address implementation barriers, allow more time, or gather better evidence?

The following framework provides an organized approach to answering these questions.

A framework for program evaluation

Program evaluation helps people understand and improve community health and development efforts. The original CDC framework presented here organizes evaluation around connected steps and standards for quality. It is a practical guide that can be adapted to a program's purpose, resources, and setting.

The original framework contains two related dimensions:

• Steps in evaluation practice
• Standards for evaluation quality

The illustration shows the original 1999 framework. Its six connected steps provide a useful teaching sequence, but evaluation is iterative: questions about evidence or use may require revisiting earlier decisions. Planning for how findings will be used should begin at the outset, and engagement should continue throughout the process.

CDC's 2024 revision uses six steps: assess context; describe the program; focus the evaluation questions and design; gather credible evidence; generate and support conclusions; and act on findings. Its accompanying guidance emphasizes collaboration, fair and just evaluation practices, and ongoing learning across every step. The original sequence below remains the organizing structure for this section, with those considerations incorporated throughout.

• Engage community members and other interested or affected parties
• Describe the program
• Focus the evaluation questions and design
• Gather credible evidence
• Develop and justify conclusions
• Use findings and share lessons learned

Use these steps as a connected learning process, adapting decisions as the program and its context change.

The original framework also drew on 30 evaluation standards organized into four categories. Those historical categories are retained below to explain the original model:

• Utility
• Feasibility
• Propriety
• Accuracy

These categories help assess whether an evaluation is useful, practical, ethical, and technically sound. CDC's 2024 framework uses five standards: relevance and utility; rigor; independence and objectivity; transparency; and ethics. When planning an evaluation, identify the applicable guidance and document how its standards will inform your decisions.

Engage Community Members and Other Interested or Affected Parties

People and organizations may be affected by both an evaluation's findings and the decisions made from them. Evaluation therefore requires attention to relationships, power, and differing perspectives. Community members, program participants, staff, partners, funders, and decision-makers may each contribute knowledge that others lack. Meaningful engagement gives people opportunities to shape questions, interpret findings, and influence decisions rather than simply provide data.

Participation can strengthen relevance, trust, and shared responsibility, but it does not guarantee agreement. People should be able to question methods, identify harms, or disagree with conclusions without pressure to endorse the program or the evaluation.

Although engagement appears first in the original framework, it belongs in every step. Begin by understanding the community context, identifying whose participation is needed, and agreeing on how their contributions will affect the work.

Three overlapping groups are important to involve:

• People or organizations involved in program operations may include community leaders, volunteers, collaborators, coalition partners, funders, administrators, managers, and staff. Their knowledge can clarify how the program actually operates and what resources it needs.
• People or organizations served or affected by the program may include participants, families, neighborhood organizations, advocacy groups, institutions, public officials, and other residents. Include people who face barriers to participation and those who question the program. Listening to different perspectives and engaging people who may oppose the program can identify blind spots and strengthen the evaluation.

Also consider people who could experience unintended consequences from decisions based on the evaluation. For example, changes to service locations, eligibility, hours, or funding may affect people who are not currently participating. Their perspectives can help identify consequences that program records alone will miss.

• Primary intended users of the evaluation are the specific people expected to act on its findings. They may include community representatives, participants, program staff, organizational leaders, and funders. They are not necessarily the same people as the program's intended participants. Identify these users early, clarify the decisions they face, and maintain communication without allowing their preferences to override evidence or the interests of affected communities.

The form and extent of community and partner involvement will vary. People may help govern the evaluation, develop questions, collect information, interpret results, or decide how findings are shared. Offer accessible participation options, language support, reasonable compensation where appropriate, and clear explanations of the time and responsibilities involved.

Agree on how decisions will be made, how power will be shared, and how the group will resolve conflicts. Document different views and create ways for people to raise concerns safely. These practices help prevent the priorities of the most influential participants from automatically determining the evaluation's direction.

Describe the Program

A program description explains the effort being evaluated: what it intends to accomplish, how its activities are expected to contribute, who is involved, and what resources and conditions shape its work. It should distinguish the program's planned approach from what is actually being implemented.

The description establishes the frame of reference for evaluation. For example, a program focused on enforcing alcohol sales restrictions has different activities and immediate outcomes from a program providing safe transportation for young people. A clear description helps the group choose appropriate questions and examine how particular activities may contribute to results.

People may also have different understandings of the program's purpose. In a youth development initiative, some may emphasize educational support, others access to employment, and others young people's leadership and decision-making. Discuss these expectations openly and include young people's own priorities when defining the program.

A shared description makes evaluation more useful, even when some differences remain. Document areas of agreement, unresolved questions, and assumptions about how change will happen. Developing this description can improve program planning before outcome data are available.

Include the following aspects when describing a program.

Statement of need and opportunity

A statement of need and opportunity describes the issue, goal, or possibility the program addresses, together with community strengths and existing efforts. Explain the nature of the issue, who is affected, its scale, and how it is changing. Identify relevant social, economic, and institutional conditions rather than treating a community's challenges as individual shortcomings.

Expectations

Expectations describe the program's intended results and what would count as meaningful progress. Organize them into short-term, intermediate, and longer-term outcomes, explaining how one is expected to contribute to another. A program's vision, mission, goals, and objectives express these expectations at different levels of detail. Check whether expectations are realistic and shared by the people affected.

Activities

Activities are the actions a program takes to contribute to change. Describe its components, their sequence, how they relate, and who is responsible for each. Distinguish activities delivered by program staff from those led by community members or partner organizations. Also identify external developments, such as changes in transportation services, employment opportunities, or public policy, that may influence results.

Resources

Resources include time, skills, relationships, facilities, equipment, information, funding, and other assets available to the program. Reviewing resources helps reveal whether planned activities are adequately supported. Include contributions from community members and partners, especially work that is unpaid or easily overlooked. Clear cost information is also needed when assessing efficiency, cost-effectiveness, or costs and benefits.

Stage of development

A program's stage of development reflects how established its activities and supporting systems are. Programs change over time, and their components may develop at different rates. A newly funded initiative may require different evaluation questions from a long-running program that is expanding or changing its approach.

Three useful phases are planning, implementation, and assessment of effects or outcomes. During planning, evaluation can test assumptions and refine the approach. During implementation, it can examine delivery, participation, accessibility, and adaptations. Once sufficient time has passed, it can assess intended and unintended outcomes. These phases may overlap, and planning for outcome assessment often needs to begin before implementation.

Context

A description of the program's context considers community history, geography, culture, politics, economic conditions, institutional practices, and related efforts. A responsive evaluation is sensitive to these influences and to differences in power and access to resources. Context helps explain findings and assess whether an approach may transfer elsewhere. For example, a housing initiative may need substantial adaptation across communities with different housing markets, transportation systems, or local regulations.

Logic model

A logic model summarizes how a program is expected to work. It connects resources and activities with outputs and intended outcomes, making assumptions about change visible. A flowchart, map, or table can show these relationships, including feedback loops or external influences when a simple sequence would be misleading.

Developing a logic model together can clarify program direction and identify questions for evaluation. The model expresses a theory about how change may occur; it does not establish that the program caused an outcome. If a model is used to estimate outcomes that were not directly measured, report those results as estimates and explain the evidence, assumptions, and uncertainty behind them.

The detail needed in a program description depends on the evaluation's purpose. Combine relevant documents, discussions with participants and staff, and observations of activities to create an accurate account. Check whether written plans match actual practice. Include contextual factors such as staff turnover, resource constraints, policy changes, community leadership, and partnerships that may affect implementation and results.

Focus the Evaluation Questions and Design

Focusing the evaluation means deciding what you most need to learn, why it matters, and how the findings will inform action. An evaluation cannot answer every possible question. Agreeing on priorities helps the group use its time and resources well while keeping community concerns visible.

Different questions require different designs and types of evidence. Plan before collecting information, including how you will analyze it and protect participants. Some design choices are difficult to change later, but the plan should allow justified adaptations when conditions change or important questions emerge.

Consider the following issues when focusing an evaluation:

Purpose

Purpose describes the overall reason for the evaluation. A clear purpose guides the questions, methods, level of detail, and intended use of findings. It also helps the group distinguish learning and improvement from other purposes, such as accountability or decisions about expansion.

There are at least four broad purposes for a community program evaluation:

• To gain insight. Evaluation can help a group explore a proposed approach, such as whether a resident-led neighborhood safety initiative fits local priorities and resources. Findings from similar efforts, together with local knowledge, can clarify assumptions, feasibility, and adaptations needed before implementation.
• To improve how work is carried out. Evaluation can examine what was delivered, who participated, what barriers arose, and how implementation differed from the plan. This information can guide improvements in quality, accessibility, coordination, and the use of resources.
• To assess program outcomes and contributions. Evaluation can examine changes associated with a program, such as improvements in school completion or access to stable housing. Assessing whether the program caused those changes requires a design that can address alternative explanations. Outcome findings can support accountability and decisions, provided the conclusions match the strength of the evidence.
• To support learning among people involved. Participating in evaluation can build skills, strengthen relationships, and help people reflect on their work. These benefits should arise through transparent and voluntary participation. They may:
  • Strengthen community decision-making, for example by giving participants meaningful authority in choosing questions and interpreting findings;
  • Support reflection on the program, for example through a follow-up conversation that helps participants identify useful experiences and unmet needs;
  • Build staff and community evaluation skills, including collecting, analyzing, and interpreting information; or
  • Contribute to organizational learning, for example by clarifying how activities align with the organization's mission and community priorities.

Users

Intended users are the people expected to apply the findings. Their information needs, decisions, and timelines should help shape the evaluation, alongside the interests of people affected by those decisions. Discuss tradeoffs openly: a smaller evaluation may answer a focused question well but provide less information about variation across settings or populations. Participation in choosing the focus can help users understand both the value and limits of the eventual findings.

Uses

Uses describe what people plan to do with what they learn. These might include revising activities, allocating resources, improving accessibility, deciding whether to expand a program, or strengthening community oversight. The following examples connect possible uses with the four broad purposes described above.

Some specific examples of evaluation uses

• To gain insight:

  • Assess community needs, priorities, and resources
  • Identify barriers to accessing and participating in the program
  • Learn how to describe and measure activities and outcomes meaningfully

• To improve how work is carried out:

  • Refine plans for introducing or adapting a practice
  • Determine how closely implementation followed the plan and why changes occurred
  • Improve the clarity and accessibility of educational materials
  • Strengthen cultural responsiveness and ongoing learning
  • Assess whether participants' rights, privacy, and well-being are protected
  • Set priorities for staff training and support
  • Make and document adjustments during implementation
  • Improve communication among participants, staff, and partners
  • Identify ways to improve participants' experiences and satisfaction
  • Compare costs with outcomes and benefits using an appropriate approach
  • Examine who benefits, who is underserved, and whether anyone experiences unintended harm
  • Inform community decisions about supporting and sustaining the program

• To assess program outcomes and contributions:

  • Assess participants' development and use of skills
  • Examine changes in behavior and conditions over time
  • Inform decisions about where to allocate resources
  • Document progress toward program objectives
  • Assess whether accountability commitments have been met
  • Combine findings from multiple evaluations to assess what may work in similar settings and where uncertainty remains

• To support learning among people involved:

  • Create opportunities to reflect on program experiences and information
  • Encourage dialogue and deepen understanding of community issues
  • Clarify shared goals and document differences among partners
  • Build evaluation skills among staff, participants, and community partners
  • Gather accounts of success, difficulty, and unexpected outcomes with appropriate permission
  • Support organizational learning, change, and improvement

Questions

An evaluation needs specific, answerable questions. Invite people involved in and affected by the program to identify what they need to know, then prioritize questions according to their importance, feasibility, and likely use. Ask whose experience each question captures and whether important differences or unintended consequences could be overlooked.

Methods

Evaluation methods draw on social research, community knowledge, and other fields. Common design families include experimental, quasi-experimental, and observational approaches, including case studies. Experimental designs use random assignment to estimate effects by comparing groups; randomization supports comparability but does not guarantee identical groups or eliminate every source of bias. Quasi-experimental designs use approaches such as comparison groups, interrupted time series, or differences in changes over time without random assignment. Observational and case study designs can describe experiences, implementation, context, and possible explanations using documents, interviews, observations, surveys, or other sources.

No single design is best for every question. Designs differ in their ability to support causal conclusions, describe experiences, explain implementation, or examine context. Select methods that fit the questions and ethical constraints, and state what they can and cannot establish. Combining qualitative and quantitative methods can strengthen understanding when the approaches are well designed and their findings are meaningfully integrated.

Methods may need to change as conditions or information needs change. For example, an evaluation focused on improving delivery may later need to inform expansion decisions. Document why changes were made, how they affect interpretation, and whether additional expertise or information is needed. Avoid changing methods simply to obtain preferred findings.

Agreements

Agreements describe the evaluation's purpose, questions, methods, intended users, deliverables, responsibilities, timeline, and budget. They should also address decision-making, data access and stewardship, privacy, reporting, conflicts of interest, and the process for revising the plan. Clarify who can review findings and how disagreements will be documented.

An agreement may be a contract, protocol, memorandum of understanding, or another written document appropriate to the work. Its purpose is to make expectations explicit and confirm shared understanding. The agreement should protect the integrity of the evaluation, including the reporting of unfavorable findings, while allowing justified changes through an agreed process.

Focusing the evaluation may involve discussions with supporters and critics, review of possible uses, and interviews with intended users about their decisions and timelines. Compare options for scope, cost, accessibility, and strength of evidence. A timely, focused evaluation can be useful, but explain the limitations of any compromises rather than treating speed or convenience as substitutes for credible methods.

Gather Credible Evidence

Credible evidence is information that is relevant, trustworthy, and adequate for answering the evaluation's questions. What is needed depends on the question and the claim being considered. Assessing a program's causal effect may require a strong comparison design, while understanding participants' experiences may call for carefully conducted interviews or observations. Community acceptance matters, but credibility also depends on transparent methods and attention to bias and uncertainty.

Community knowledge and technical expertise can complement each other. People with lived experience may identify questions, barriers, or interpretations that external evaluators would miss. Specialists can help address issues such as sampling, complex analysis, measurement, or sensitive data. Bring these perspectives together according to the evaluation's needs.

All evidence has limitations. Using multiple relevant sources and methods can help identify consistent patterns, explain differences, and reveal missing perspectives. Involving community members in gathering and interpreting information can improve relevance and understanding, provided they receive appropriate support and the process protects confidentiality and independent judgment.

The following features affect the credibility of evidence:

Indicators

Indicators translate concepts about a program, its implementation, and its expected outcomes into observable or measurable features. Define each indicator clearly, including what is counted or described, for whom, over what period, and using which source.

Examples of indicators include:

• The program's capacity to deliver services, such as staffing levels and appointment availability
• Participation rates, with a clearly defined eligible or intended population
• Participants' satisfaction, experiences, and reported barriers
• Program reach and intensity, including who participated, what they received, and for how long
• Changes in participants' knowledge, skills, opportunities, or behavior
• Changes in community conditions, relationships, or norms
• Changes in organizational or policy environments, such as new practices or improved service access
• Longer-term changes in community health or well-being, such as housing stability or reported access to needed care

Indicators should reflect the criteria that matter for assessing the program, including community-defined outcomes. Complex initiatives usually require several indicators. Where appropriate and feasible, examine differences across populations or locations while protecting privacy and avoiding misleading conclusions from small numbers.

One approach is a "balanced scorecard": a small, complementary set of indicators covering different aspects of the program. For example, it might describe service delivery, participants' experiences, observed outcomes, resource use, and changes in surrounding conditions. The aim is a balanced view rather than a large collection of measures with no clear purpose.

A logic model provides another way to choose indicators. Identify measures along the pathway from resources and activities to outputs and outcomes. Quantitative indicators can describe amounts or rates, while qualitative information can explain experiences, processes, and meanings that numbers alone may not capture.

Indicators need not focus only on long-term outcomes. They can also describe service quality, community capacity, trust, or relationships among organizations. Work with people familiar with the context to define observable signs of these concepts, and check whether those signs have the same meaning across participants and settings.

Indicators may need revision as understanding improves; document changes and their effects on comparisons over time. Remember that tracking program performance is one part of evaluation. A change in an indicator does not by itself explain why it happened. For example, rising unemployment among participants may reflect broader economic conditions, changes in who enrolled, program limitations, or a combination of factors.

Sources

Sources of evidence may include participants, residents, staff, administrative records, documents, observations, and other relevant information. Different sources provide different perspectives. Program records may describe services delivered, while participants and people unable to access the program can explain experiences those records do not capture. Examine discrepancies rather than assuming one source is always more reliable.

Explain how sources were selected, who or what was excluded, and how those choices may affect conclusions. Numerical and narrative information can complement each other when their purposes and methods are clear. For example, participation records may reveal a decline in attendance, while interviews help explain transportation difficulties or changes in scheduling.

Quality

Quality concerns whether information is sufficiently accurate, consistent, relevant, and complete for its intended use. It depends on clear definitions, appropriate tools, accessible data collection, training, sampling, coding, data management, and error checking. Pilot procedures when feasible and examine missing data and possible bias. Discuss tradeoffs, but do not lower evidence requirements simply because a preferred conclusion would be convenient.

Quantity

Quantity refers to how much information is needed. Plan the amount and range of evidence required for the questions, methods, and intended decisions. Quantitative studies may need sample-size planning; qualitative work requires attention to the depth and diversity of perspectives. More data do not automatically mean better evidence, especially if collection is biased or burdensome. Gather information with a clear purpose and establish a reasoned stopping point.

Logistics

By logistics, we mean the people, timing, settings, technology, and procedures needed to gather and manage information. Ask participants about language, communication, accessibility, and privacy preferences rather than assuming that everyone in a community shares the same needs. Explain participation clearly, establish appropriate consent procedures, and specify who can access data, how they will be protected, and how long they will be retained.

Develop and Justify Conclusions

Evidence does not interpret itself. Conclusions require analysis, attention to context, and a clear explanation of how findings relate to the evaluation questions and agreed criteria. Include different perspectives and consider alternative explanations. Agreement can support use, but consensus is not proof: a conclusion must remain grounded in evidence even when some people disagree with it.

The principal elements in developing and justifying conclusions are:

Performance standards

Performance standards describe the basis for judging the program, such as expected levels of access, service quality, or improvement. These differ from the standards used to assess the quality of the evaluation itself. Agree on performance criteria and thresholds early where possible, explain whose values they reflect, and document any revisions so that judgments are not adjusted simply to fit the results.

Analysis and synthesis

Analysis and synthesis help organize and interpret the evidence. Analysis examines patterns within information; synthesis brings findings together to develop a broader understanding. In an evaluation using multiple methods, analyze each source appropriately and then examine where findings agree, differ, or leave gaps. Explain how information was classified, compared, and combined, including decisions about missing data, uncertainty, and potentially conflicting evidence.

Interpretation

Interpretation explores what the findings mean in context. In a hypothetical community survey, 15% of respondents report witnessing violence during the previous year. That finding has a different meaning if a comparable earlier survey found 50% than if it found 7%. Before interpreting either pattern, check whether sampling, questions, reporting periods, and participation were comparable. Even a credible change over time does not establish that the program caused it. Community perspectives can help explain possible influences and the practical significance of the findings.

Judgments

Judgments assess the program's value or performance by comparing interpreted findings with explicit criteria. Different criteria can lead to different judgments. For example, managers may see a 10% increase in outreach as progress, while residents may point out that services still do not reach neighborhoods facing the greatest barriers. Both perspectives can inform the assessment. Explain the basis of each judgment and distinguish improvement from meeting an agreed standard.

Recommendations

Recommendations identify actions to consider in response to the findings. They require attention to evidence, feasibility, resources, alternatives, and the priorities of affected people. For example, evidence that a program expanded access to services for survivors of domestic violence may support continued investment, while also revealing needs for safer access, different hours, or stronger coordination. Decisions about changing or ending services should consider continuity and the consequences for people who rely on them.

Recommendations should make their reasoning explicit and remain proportionate to the evidence. Explain expected benefits, possible harms, practical requirements, and uncertainties. Where values or priorities differ, document those differences rather than presenting a contested choice as the only possible conclusion.

Three practices can improve the relevance and usefulness of recommendations:

• Share draft recommendations with people expected to use or be affected by them.
• Seek responses from people with different experiences, roles, and perspectives.
• Present feasible options and tradeoffs, identifying when the evidence supports a clear recommended action.

Strengthen conclusions by actively examining alternative explanations and evidence that challenges the initial interpretation. Do not dismiss an explanation merely because it is inconvenient. Where multiple interpretations remain plausible, describe them and the information that could help distinguish among them. Plan major analyses in advance when possible and identify additional exploratory analyses clearly.

Use Findings and Share Lessons Learned

Findings do not automatically lead to action. Plan for their use from the beginning by identifying decisions, responsible people, timelines, and the resources needed for follow-through. Continue communicating throughout the evaluation, and make space to discuss findings that are unexpected or difficult to act on.

The following elements can help people use evaluation findings:

Design

Design connects the evaluation's questions and methods with the decisions it is intended to inform. As discussed earlier, clarify who needs the findings, when they need them, and what actions are possible. Involving community members and intended users in these decisions can improve relevance while helping everyone understand the evaluation's boundaries.

Preparation

Preparation means building the understanding and capacity needed to act on findings. Discuss possible results before they are available, including what would count as sufficient evidence for a decision and how uncertainty will be handled. This can reveal unrealistic expectations and identify support that users will need.

For example, present intended users and community partners with hypothetical findings and ask what actions they would consider. If the information would not help them make a decision, revisit the questions or reporting plan. Explore favorable, unfavorable, and mixed findings without encouraging people to choose a preferred result in advance.

Feedback

Feedback is the continuing exchange of information among people involved in the evaluation. Provide accessible opportunities to comment on decisions, emerging findings, and draft interpretations. Explain how feedback influenced the work. Review can identify errors and missing context, but it should not give participants, staff, or funders authority to suppress credible findings.

Follow-up

Follow-up provides support as people interpret findings and decide what to do. Completing a report is a milestone; learning and action continue afterward. Assign responsibility for agreed actions, identify needed resources, set review dates, and track progress. Someone may coordinate this work while ensuring that the evidence and its limitations remain visible in decisions.

Supporting use also means guarding against misuse. Findings are shaped by the design, participants, setting, and period studied. A single case study may offer valuable insights, but it usually cannot establish that every site in a large program will have the same experience or outcome. Explain what can reasonably transfer and what requires further investigation.

Anyone involved may selectively emphasize favorable or unfavorable findings. Reduce this risk by reporting the full pattern of results, explaining uncertainty, and correcting misleading interpretations. New uses of existing findings may be reasonable, but they require a fresh assessment of whether the evidence supports the proposed decision.

Dissemination

Dissemination means sharing the evaluation's methods, findings, and lessons with relevant audiences in a timely and accessible way. Plan language, format, communication channels, and timing with intended users and affected communities. Use plain-language summaries and other appropriate formats alongside technical detail. Report findings fairly while protecting confidentiality and honoring legitimate restrictions on sensitive information.

Benefits can also arise from participating in the evaluation itself. These "process uses" include clearer thinking about assumptions, stronger skills, improved relationships, and a shared understanding of the program. Notice and assess these benefits rather than assuming that participation will always have a positive effect.

Evaluation can help staff and community partners clarify goals, examine how their work connects, and make decisions using evidence and explicit values. It can also reveal disagreements or organizational barriers that need attention. A learning-oriented approach treats those discoveries as information for improvement.

Additional process uses for evaluation include:

• Developing indicators together can clarify what different participants value and which outcomes need greater attention.
• Connecting findings to planning and resource decisions can make learning part of routine practice. For example, a funder might support adaptations identified through evaluation, while avoiding incentives that reward only favorable results or discourage serving people facing greater barriers.

Standards for Evaluation Quality

The Joint Committee on Standards for Educational Evaluation developed The Program Evaluation Standards to guide sound and fair evaluation. The discussion below adapts the 30 standards reflected in the original CDC framework, retaining their historical four-category organization. The later third edition of the Joint Committee's standards reorganizes the guidance into five categories, including evaluation accountability, and should be consulted when selecting standards for a new evaluation.

Standards help groups assess the quality of their evaluation and make reasoned choices when resources or conditions create tradeoffs. For example, technically accurate findings may have little value if they arrive after a decision or cannot be understood by intended users. A useful design also needs to be feasible and ethically acceptable.

Apply relevant standards during planning, implementation, reporting, and follow-up. The historical summaries below use updated explanatory language; they are not a verbatim reproduction of the current standards. Document how you apply the guidance and how you address tensions among practical requirements without compromising participants' rights or the integrity of findings.

The original 30 standards were grouped into four categories:

• Utility
• Feasibility
• Propriety
• Accuracy

The utility standards address the evaluation's relevance and usefulness:

• Identification of Interested and Affected Parties: Identify people involved in or affected by the program and its evaluation, including those whose perspectives are often overlooked, so that relevant priorities can shape the work.
• Evaluator Credibility: Those conducting the evaluation should bring appropriate skills, demonstrate trustworthiness, understand the context, and explain how they will manage potential conflicts of interest.
• Information Scope and Selection: Collect information that addresses the agreed questions and the needs of intended users and affected communities. Avoid gathering information that has no clear use.
• Values Identification: Explain the values, perspectives, criteria, and reasoning used to interpret findings and judge the program.
• Report Clarity: Describe the program, context, evaluation purpose, methods, findings, and limitations in language and formats that intended audiences can understand and use.
• Report Timeliness and Dissemination: Share relevant findings in time to inform decisions, clearly identifying preliminary results and protecting sensitive information.
• Evaluation Impact: Plan and conduct the evaluation in ways that support learning, responsible action, and follow-through, while monitoring possible unintended effects of the evaluation itself.

Feasibility Standards

Feasibility standards address whether the evaluation can be carried out effectively with the available resources and within its setting. They encourage practical choices without treating convenience as sufficient justification for weak or unfair procedures.

The feasibility standards address:

• Practical Procedures: Use procedures that gather needed information while minimizing unnecessary burden and disruption for participants, staff, and community partners.
• Political and Contextual Viability: Anticipate differing interests, power relationships, and conditions that may affect participation or use. Establish fair ways to address disagreement and protect the evaluation from inappropriate interference.
• Cost-Effectiveness: Match the evaluation's scope and resource use to the value of the information needed, including the time and contributions required from community members.

Propriety Standards

Propriety standards address ethical conduct, fairness, rights, and the well-being of people involved in or affected by the evaluation. The original framework included the following eight areas.

• Service Orientation: Design the evaluation to help the program respond effectively and fairly to the people it is intended to serve, including those experiencing barriers to access.
• Formal Agreements: Document responsibilities, procedures, timelines, reporting arrangements, and the process for changing agreements so that expectations are clear and accountable.
• Participants' Rights and Well-Being: Protect people's dignity, privacy, and ability to make informed choices about participation. Establish appropriate consent and review procedures for the evaluation's setting and activities.
• Respectful Interactions: Treat people with respect, provide accessible ways to participate, and address coercion, discrimination, distress, or other risks arising from the evaluation.
• Complete and Fair Assessment: Examine strengths, limitations, benefits, harms, and unintended consequences fairly, with attention to differences in experiences and outcomes.
• Disclosure of Findings: Make findings and limitations accessible to relevant audiences while protecting confidential information and meeting applicable disclosure obligations. Explain any restrictions on access.
• Conflicts of Interest: Identify, disclose, and manage actual or perceived conflicts that could influence the evaluation's design, conduct, interpretation, or reporting.
• Fiscal Responsibility: Use resources responsibly, maintain appropriate financial records, and account transparently for evaluation expenditures.

Accuracy Standards

Accuracy standards address whether the evaluation provides dependable information and defensible interpretations. They support confidence in the findings while requiring explicit acknowledgment of uncertainty and limitations.

The original framework included 12 accuracy standards:

• Program Documentation: Describe the program clearly and accurately, including what was planned, what was implemented, and any important changes.
• Context Analysis: Examine conditions that may shape implementation, outcomes, interpretation, and the transfer of findings to other settings.
• Described Purposes and Procedures: Document the evaluation's purposes and methods in enough detail for others to assess the work, including justified departures from the original plan.
• Defensible Information Sources: Explain how information sources were selected and why they are appropriate, including relevant exclusions and possible sources of bias.
• Valid Information: Use measures and collection procedures that support the intended interpretations, checking whether they are appropriate for the people, languages, and settings involved.
• Reliable Information: Use procedures that produce sufficiently consistent and dependable information for the evaluation's purposes, and report relevant measurement limitations.
• Systematic Information Management: Organize, review, protect, and verify information systematically. Correct errors through documented procedures and preserve a clear record of important changes.
• Analysis of Quantitative Information: Analyze numerical information using methods suited to the design and questions, accounting for missing data, uncertainty, and relevant assumptions.
• Analysis of Qualitative Information: Analyze interviews, observations, documents, and other descriptive information systematically, preserving context and considering interpretations that challenge emerging themes.
• Justified Conclusions: Show how conclusions follow from the evidence and criteria, and distinguish observed changes from claims about what caused them.
• Impartial Reporting: Report findings fairly and transparently, including unfavorable or mixed results. Explain how potential bias and competing interpretations were addressed.
• Metaevaluation: Assess the quality of the evaluation itself using appropriate standards, during the work and after completion, to identify strengths, limitations, and improvements.

Applying the Framework: Conducting Useful and Credible Evaluations

Evaluation can support community learning, program decisions, and accountability to participants, partners, and funders. The appropriate scope depends on what people need to know and the consequences of the decisions ahead. Useful starting questions include:

• What evaluation approach fits our questions, context, and resources?
• What are we learning, whose experiences are represented, and what remains uncertain?
• How will we use what we learn to improve effectiveness, accessibility, and fairness?

The framework helps connect these questions with practical steps and standards. Its value lies in making evaluation decisions deliberate, transparent, and responsive to the people affected by the program.

Applying the framework requires judgment and a suitable combination of community knowledge and evaluation skills. Relationships may be complex, resources limited, and conditions subject to change. Develop an approach that fits those realities while protecting ethical standards and the integrity of the evidence. Seek specialized support for questions or methods that exceed the team's experience.

Cost is a common concern, but an evaluation's budget should be considered in relation to its questions and intended decisions. A modest evaluation can provide useful information about implementation or participant experience. More demanding claims, such as estimating causal effects or detecting small differences across populations, may require additional resources and expertise.

Build evaluation into routine work where appropriate, using existing information only when it is suitable and can be used responsibly. Coordinate collection and reporting with decision timelines, and avoid repeatedly asking community members for information that is already available.

Technical complexity can also be a barrier. Begin with clear, useful questions and select methods the team can carry out well. Provide training and support, and be explicit about what the resulting evidence can establish. A focused, well-executed evaluation is more useful than an ambitious design that cannot be implemented credibly.

Staff and participants may reasonably worry that evaluation will be punitive, exclusionary, or used to justify decisions already made. Address those concerns directly. Explain the purpose, share decision-making where appropriate, protect candid feedback, and demonstrate how findings will support learning and accountability. Trust depends on how the evaluation is conducted and used, not simply on assurances that participation is welcome.

In Summary

Evaluation helps communities understand how programs operate, what changes occur, who benefits, and what needs improvement. It can inform decisions about resources, adaptation, expansion, or discontinuation while identifying unintended consequences. Its conclusions should reflect both the evidence and its limits.

The original CDC framework remains a useful teaching model when its historical basis is clear and newer guidance informs current practice. Use the steps and applicable standards to develop an evaluation that fits the program, involves affected people meaningfully, and connects findings with action. The Magenta Book - Guidance for Evaluation is an additional evaluation resource; this link points to an older edition, so consult the latest HM Treasury guidance when using it to plan an evaluation.

You understand the basic concepts and terms of program evaluation:

___You understand that evaluation is the systematic investigation of the merit, worth, or significance of an object or effort

___You understand that a program is used to describe the object or effort that is being evaluated

___You understand that program evaluation is a way to evaluate the specific projects and activities community groups may take part in

___You understand that stakeholders are those who care about the program or effort

You understand that program evaluation:

___Helps clarify program plans

___Improves communication among partners

___Gathers feedback needed to improve and be accountable for program effectiveness

You know how to evaluate a specific program:

___You know what will be evaluated

___You have decided what criteria will be used to judge program performance

___You have decided what standards of performance must be reached for the program to be considered successful

___You know what evidence will indicate performance

___You know what conclusions about program performance are justified based on the available evidence

___You have engaged stakeholders

___You have described the program

___You have focused the evaluation design

___You understand that possible focuses are: to gain insight, to improve how things get done, to determine what the effects of the program are, and to effect those who participate in it

___You have gathered credible evidence

___You have justified your conclusions

___You understand how to ensure the use of the information and share lessons learned

___You know that there are four categories of evaluation standards: utility, feasibility, propriety, and accuracy

___You understand the standards in each category

___You understand which standards are most applicable to each step of evaluation practice

When writing your evaluation report:

___Provide interim and final reports to intended users in time for intended uses

___Tailor the report content, format, and style for the audience(s) by involving audience members

___Include an executive summary

___Summarize the stakeholders and how they were engaged

___Describe essential features of the program (perhaps in appendices)

___Explain the focus of the evaluation and its limitations

___Include an adequate summary of the evaluation plan and procedures

___Provide all necessary technical information (perhaps in appendices)

___Specify the standards and criteria for evaluative judgments

___Explain the evaluative judgments and how they are supported by the evidence

___List both strengths and weaknesses of the evaluation

___Discuss recommendations for action along with their advantages and disadvantages

___Ensure protections for program clients and other stakeholders

___Anticipate how people or organizations may be affected by the findings

___Present minority opinions or rejoinders where necessary

___Verify that the report is accurate and unbiased

___Organize the report logically and include an appropriate level of detail

___Remove technical jargon

___Use examples, illustrations, graphics, and stories 

Tool 1: Tools Related to the Recommended Framework

See the following Community Tool Box sections for additional information.

 

Engage Stakeholders

To help you understand who stakeholders are:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

Identifying Targets and Agents of Change: Who Can Benefit and Who Can Help

Identifying Community Assets and Resources

To learn how to involve stakeholders:

Involving Key Influentials in the Initiative

Involving People Most Affected by the Problem

Conducting Focus Groups

To work together with a diverse group:

Training for Conflict Resolution

Encouraging Involvement of Potential Opponents as well as Allies

Describe the Program

To fully understand the need, problem, or goal that the program addresses:

Defining and Analyzing the Problem

Analyzing Community Problems

Conducting Needs Assessment Surveys

Conducting Concerns Surveys

Conducting Public Forums and Listening Sessions

To better explain the activities, components, and elements of the program:

Designing Community Interventions

Understanding Risk and Protective Factors: Their Use in Selecting Potential Targets and Promising Strategies for Interventions

Identifying Strategies and Tactics for Reducing Risks

To be able to describe resources and assets for the program:

Identifying Community Assets and Resources

For examples of logic models:

Our Model of Practice: Building Capacity for Community and System Change

Our Evaluation Model: Evaluating Comprehensive Community Initiatives

To look broadly at your program and its context:

An Overview of Strategic Planning or "VMOSA" (Vision, Mission, Objectives, Strategies, Action Plan)

Understanding and Describing the Community

To modify the program to fit the local context:

Adapting Community Interventions for Different Cultures and Communities

To learn to explain the program to others (and so that they see your point of view):

Reframing the Issue

Focus the Evaluation Design

To clarify the purpose:

Developing an Evaluation Plan

To identify potential users and uses:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

To clarify evaluation questions:

Developing an Evaluation Plan

For illustrative evaluation questions:

Our Model of Practice: Building Capacity for Community and System Change

For help identifying specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

For guidance about making agreements:

Identifying Action Steps in Bringing About Community and System Change

Gather Credible Evidence

For support in implementing specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

Justify Conclusions

To see an illustrative process for considering evidence:

Our Model of Practice: Building Capacity for Community and System Change

Ensure use and share lessons learned

To promote the use of what your organization has learned:

Providing Feedback to Improve the Initiative

Conducting a Social Marketing Campaign

Attracting Support for Specific Programs

To share what you have learned with diverse groups:

Making Community Presentations

Communications to Promote Interest

Arranging Celebrations

Tool 2: Evaluation Standards

Use this table to determine how well your evaluation met "good" standards for evaluation.

Standard	Did the evaluation meet this standard? (Yes or No)	Comments:
Utility Standards
1. Stakeholder Identification
2. Evaluator Credibility
3. Information Scope and Selection
4. Values Identification
5. Report Clarity
6. Report Timeliness and Dissemination
7. Evaluation Impact
Feasibility Standards
1. Practical Procedures
2. Political Viability
3. Cost Effectiveness
Propriety Standards
1. Service Orientation
2. Formal Agreements
3. Rights of Human Subjects
4. Human Interactions
5. Complete and Fair Assessment
6. Disclosure of Findings
7. Conflict of Interest
8. Fiscal Responsibility
Accuracy Standards
1. Program Documentation
2. Context Analysis
3. Described Purposes and Procedures
4. Defensible Information Sources
5. Valid Information
6. Reliable Information
7. Systematic Information
8. Analysis of Quantitative Information
9. Analysis of Qualitative Information
10. Justified Conclusions
11. Impartial Reporting
12. Metaevaluation

Tool 3: Steps in Evaluation Practice and the Most Relevant Standards

Codes following each standard designate the conceptual group and number of the standard.  U = Utility; F = Feasibility; P = Propriety; A = Accuracy.

Steps	Most Relevant Standards
Engage Stakeholders	Metaevaluation (A12) Stakeholder identification (U1) Evaluator Credibility (U2) Formal agreements (P2) Rights of human subjects (P3) Human Interactions (P4) Conflict of interest (P7)
Describe the program	Complete and fair assessment (P5) Program documentation (A1) Context analysis (A2)
Focus the evaluation design	Evaluation impact (U7) Practical procedures (F1) Political viability (F2) Cost effectiveness (F3) Service orientation (P1) Complete and fair assessment (P5) Fiscal responsibility (P8) Described purposes and procedures (A3)
Gather credible evidence	Information scope and selection (U3) Defensible information sources (A4) Valid information (A5) Reliable information (A6) Systematic information (A7)
Justify conclusions	Values identification (U4) Analysis of quantitative information (A8) Analysis of qualitative information (A9) Justified conclusions (A10)
Ensure use and share lessons learned	Evaluator credibility (U2) Report clarity (U5) Report timeliness and dissemination (U6) Evaluation impact (U7) Disclosure of findings (P6) Impartial reporting (A11)

 

In this hypothetical example, Della Roberts works as a nutritionist at Harperville Hospital. A Black resident of the city, she is concerned about unequal access to affordable, nutritious food in several predominantly Black neighborhoods. Families have described limited grocery options, transportation barriers, and rising food costs, but city officials have given the issue little attention. Della approaches neighborhood residents about researching these conditions together to inform community-led advocacy.

Della recognizes that residents bring knowledge, relationships, and priorities that hospital staff cannot supply on their own. She contacts local faith leaders, a physician, a community center director, and people in her own networks to invite interest in a research partnership. The group also reaches beyond these connections so that established leaders do not determine who participates. Residents of different ages, educational backgrounds, languages, and experiences join the team.

The team meets at accessible neighborhood locations to decide what it wants to learn and how members will share responsibility. Residents contribute local knowledge, while Della and other partners offer research training and practical support. Together, they develop interview questions, agree on consent and privacy procedures, and arrange options for recording information through writing, audio, or supported note-taking. They decide to ask residents about food access and shopping experiences and to document food availability, prices, store hours, and transportation options.

Their findings suggest that many residents want more affordable, nutritious options but face substantial barriers. Some must make long bus trips to reach a full-service grocery store. Others struggle to fit shopping around work and caregiving responsibilities. Participants also describe prices that exceed their food budgets and limited availability of foods that meet their cultural preferences or dietary needs.

Community researchers and hospital partners analyze the information together, examine gaps in participation, and jointly prepare a report. They present their findings and recommendations to residents and city officials. In this example, the report contributes to a city response that includes support for neighborhood food retailers and improved transportation access. The team continues tracking whether these changes make food more affordable and accessible.

The partnership illustrates community-based participatory research because residents help shape the questions, methods, interpretation, reporting, and action. Their role extends beyond collecting information for professionals. Community connections can support trust and understanding, but the team still needs clear methods, confidentiality protections, and opportunities for residents to disagree or decline participation.

This section explains community-based participatory research, its relationship to participatory action research, why it can be useful, who might participate, and how to organize and carry out a project.

What is community-based participatory research?

Community-based participatory research, or CBPR, is a partnership approach that meaningfully and equitably involves community members, organizations, and researchers throughout the research process. Partners contribute different forms of knowledge and share decisions about questions, methods, interpretation, and use of findings. The aim is to develop useful knowledge and connect it with action that addresses community priorities.

Community involvement in research can take different forms. In some studies, researchers collect information from residents through surveys, interviews, or other methods, while retaining control over the questions and analysis. This can produce useful research, but being a source of information does not by itself make someone a research partner or make a project CBPR.

In other projects, researchers recruit or hire community members to collect information because of their skills, relationships, or knowledge of the setting. These contributors may also help interpret findings. Such roles can be valuable, but the extent of participation depends on their actual influence over research decisions, working conditions, and use of the results.

A more collaborative approach involves community members and institutional researchers working as partners from the beginning. They jointly identify priorities, design the study, gather and analyze information, communicate findings, and decide what action to pursue. Responsibilities may differ, but partners establish meaningful ways to share authority, resources, recognition, and accountability.

Community members may also initiate and lead their own research. A group might investigate housing conditions, service access, or another shared concern, then invite technical partners when useful. Community-led work can be highly organized and rigorous, whether or not an academic institution participates.

This section focuses on partnerships and community-led approaches that involve affected people in research decisions and connect learning with action. CBPR overlaps with participatory action research, which emphasizes cycles of inquiry, reflection, and action with participants. The terms are related but are not interchangeable in every setting. Here, CBPR refers to the partnership approach, while action research describes the ongoing connection between inquiry and change.

CBPR can support program evaluation, research on community conditions, and efforts toward lasting change. Its value depends on the quality of the partnership and the research, including whether participation is accessible, adequately supported, and influential.

Community-based participatory research can strengthen understanding of an issue in several ways:

• Some residents may feel more comfortable speaking with researchers who understand their circumstances or have established relationships in the community. Others may prefer someone outside their social network, especially when discussing sensitive experiences.
• People directly affected by an issue or intervention may identify questions, experiences, and explanations that other researchers would overlook. Their knowledge can improve the relevance of interviews, surveys, observations, and other methods.
• Community researchers can help interpret local language, relationships, and context. An apparently minor comment may point to an important concern, although interpretations should be checked rather than assumed to represent everyone.
• Ongoing community connections can help researchers notice changes and refine their questions over time. Ordinary conversations and personal relationships should not automatically become research data; the team needs agreed boundaries and appropriate permission.
• Findings may be more relevant and credible to residents when community members have helped shape the research and can explain how conclusions were reached, including the limits of the evidence.

In an evaluation, these contributions can deepen understanding of how an intervention works, who benefits, and what needs improvement. In research on a community issue, they can clarify conditions, possible causes, and opportunities for action. CBPR does not guarantee complete or unbiased findings, but a well-designed partnership can combine community knowledge and research methods to support more informed decisions.

CBPR can also support learning and professional development. Community researchers bring existing skills and expertise, including knowledge gained through lived experience, employment, organizing, caregiving, and education. Participation can expand their research skills, networks, confidence, and opportunities to influence decisions. Partners should address barriers such as unpaid work, inaccessible meetings, limited transportation, or exclusion from institutional decision-making.

Two potential contributions of CBPR to community decision-making are:

• Community members can gain greater influence over the questions being studied and the policies, services, or investments that affect their lives.
• Institutions can become more accountable for recognizing community expertise and sharing authority. Residents do not need research credentials to deserve respect or a role in decisions. A well-supported partnership can create additional opportunities to use evidence, challenge assumptions, develop leadership, and pursue community priorities. Research and analytical skills may also support educational or employment goals, although these benefits should not be assumed or substituted for fair compensation.

Participatory action research has connections with traditions of critical education associated with Brazilian educator Paulo Freire. These traditions emphasize dialogue, reflection on social conditions, and collective action to address oppression. In practice, partners should examine how power operates within their own project as well as in the wider community. Different projects may emphasize learning, practical improvements, institutional accountability, or broader social change.

Participatory research can address issues such as access to public assistance, housing instability, education, employment, and environmental conditions. It can also support community priorities that focus on existing strengths and opportunities.

Examples include:

• Environmental conditions. Residents and technical partners might document recurring flooding, investigate possible sources, and use the findings to advocate for drainage improvements.
• Health and access to care. Community members and health researchers might examine barriers to appointments, map service availability, or study residents' experiences with a health program.
• Political and economic conditions. Residents might analyze public budgets, examine access to employment, or study how public investments are distributed across neighborhoods.

CBPR partnerships can begin in different ways. What matters is how partners establish the agenda, make decisions, share resources, and use the findings. A community group may seek technical assistance, or researchers may approach community organizations about a possible collaboration. An institution's initial involvement does not determine the partnership's quality.

Common starting points include:

• Researchers develop an initial study idea and hire community members to collect or analyze data. This becomes a CBPR partnership only if community partners also have meaningful influence over the research and its use.
• A researcher, service organization, or public agency identifies a concern and invites affected residents to assess its importance, refine the questions, and jointly design the work.
• A community-based organization brings residents together to define a priority, investigate it, or evaluate an intervention that affects them.
• Residents identify a shared concern, organize a research team, and decide whether outside expertise would help them investigate and address it.

Why would you use community-based participatory research?

CBPR can offer benefits for understanding community issues, evaluating programs, and deciding what to do next. The following reasons describe possibilities to work toward, rather than results that participation automatically guarantees.

CBPR can produce information that better reflects community experiences and priorities.

• Trusted relationships may support open discussion, especially when people have clear choices about who interviews them and how their information will be protected.
• Community researchers can contribute knowledge of local history, relationships, and institutions that helps place findings in context.
• People affected by an issue or intervention can identify outcomes that matter to them and help design questions that capture those experiences.

Sharing research decisions can make the work more responsive to community priorities.

Involving people directly affected by the issue can improve the questions asked, the interpretation of evidence, and the practicality of recommendations. Meaningful participation also gives residents opportunities to shape or challenge proposed responses. The goal is shared responsibility and informed decision-making, with room for disagreement, rather than securing endorsement of a plan already chosen.

Community participation can increase awareness of the research and opportunities to contribute.

Team members can explain the project through community networks, meetings, and other agreed communication channels. Public outreach should describe the research accurately while protecting participants' identities and avoiding pressure on friends, relatives, or neighbors to take part.

Transparent partnerships can strengthen the credibility of the research.

Residents may have greater confidence in a project when people they trust have meaningful roles and the team explains its methods, funding, findings, and limitations. Shared identity or familiarity alone does not establish trustworthiness. Researchers must demonstrate it through their conduct and accountability.

CBPR can build research capacity that communities use in other efforts.

Team members may apply skills in asking questions, gathering evidence, analyzing information, and presenting findings to other priorities. Community organizations can also develop systems, relationships, and resources that support future research. These longer-term benefits are more likely when learning is shared and the partnership plans for continuity.

Involvement in CBPR can expand opportunities for leadership and influence.

Community researchers may gain confidence, new connections, and greater access to decision-making. These opportunities can be especially valuable where institutions have excluded residents from research or policy discussions. The partnership should recognize existing leadership and address the barriers that limit participation, rather than treating people as lacking knowledge or initiative.

Research skills can be useful in other areas of team members' lives.

Interviewing, critical thinking, collaboration, data analysis, and public communication may support employment, education, organizing, and personal goals. Ask team members what they hope to gain and provide appropriate recognition, references, training records, or other support. Participation should not come with promises of benefits the project cannot deliver.

Participatory research can create opportunities to address racial, ethnic, economic, and institutional barriers.

Working together can help partners question assumptions and understand experiences different from their own. However, familiarity does not automatically remove discrimination or unequal power. Teams need fair decision-making, accessible participation, respectful discussion, and ways to address harm. They should also examine whose expertise receives recognition and who controls funding, data, and public communication.

A participant in the Changes Project, a CBPR effort examining changes to public assistance and their effects on adult literacy and English-language learners, described recognizing previously unexamined prejudices through conversations with other team members. The reflection illustrates how collaborative research can prompt personal learning, including among participants who initially believe they have little to reconsider.

Participatory research can deepen understanding of the conditions that shape people's lives.

Research and collective reflection can help a team examine how policies, economic conditions, institutional practices, and social relationships influence a community issue. That understanding can inform practical strategies for change while recognizing that many conditions cannot be changed by individuals acting alone.

Community-based participatory research can contribute to lasting social change.

Shared knowledge, stronger relationships, and greater community influence can support changes in programs, policies, and institutional practices. Lasting results require action, resources, and follow-through beyond the research itself. The team should assess whether changes benefit the people most affected and whether new problems or inequities emerge.

Who should be involved in community-based participatory research?

Participation should reflect the issue and its context, with meaningful leadership by the people most affected.

• People most affected by the issue or intervention. Their experiences and priorities should help shape the project from the beginning. Include people with different experiences of the issue, including those who have been excluded from services or public discussions. No individual should be expected to speak for an entire community.
• Other members of the affected community. Residents with relevant relationships, language skills, organizing experience, or other knowledge may contribute even when they do not personally experience the issue. Their involvement should complement the leadership of directly affected people.

For example, a Portuguese-speaking business owner might help a team studying access to services by contributing language skills, local relationships, and knowledge of area organizations. The team would still need direct participation from residents who use or cannot access those services, including people outside the business owner's networks.

• Decision-makers. Local officials, agency leaders, and others with authority may help the team understand policy processes and act on findings. Their involvement can take the form of partnership, consultation, or briefings. Consider whether their presence could discourage candid participation, and do not give them automatic access to identifiable information or control over conclusions.
• Academic and other research partners. Researchers may contribute subject knowledge, methodological skills, facilities, or access to resources. They also need to learn from community partners about local conditions, priorities, and appropriate ways of working. Technical expertise should support shared inquiry rather than establish a hierarchy of whose knowledge matters.

Treat all team members as colleagues and make equitable participation concrete. Agree on decision-making authority, budgets, compensation, data access, authorship, and conflict resolution. Differences in institutional responsibility or specialized skills may affect particular roles, but paying salaries or holding a grant does not justify excluding community partners from meaningful decisions. Be transparent about constraints and negotiate how to address them.

• Health, human service, and public agency staff and volunteers. These partners may contribute knowledge of service systems, implementation, and available resources. Residents can help them understand experiences that organizational records do not capture. Address possible conflicts between their research roles and their responsibilities to employers or service users.
• Other community members and organizations. Depending on the project, youth groups, disability organizations, faith communities, tenant associations, local businesses, and other groups may contribute useful perspectives. Offer participation through appropriate roles without allowing broader representation to displace the priorities of those most affected.

When should you use community-based participatory research?

Consider whether the partnership has the purpose, relationships, resources, and decision-making arrangements needed for meaningful collaboration.

Time is one consideration. Building relationships, agreeing on priorities, learning together, and sharing decisions can require substantial preparation. An established partnership may be able to respond quickly, while a new one may need more time. If a deadline makes meaningful collaboration impossible, be honest about the limits of the engagement rather than labeling a narrowly consultative project CBPR.

The required methods also matter, but CBPR is not limited to a particular type of research. Partners can use qualitative, quantitative, or combined methods, including technically complex approaches when the team has the necessary expertise and support. Community members can help govern and interpret a study without every partner needing to perform every technical task.

Qualitative research

Qualitative research examines experiences, meanings, processes, and context through methods such as interviews, group discussions, observations, and document analysis. Findings are developed through systematic interpretation and may be presented as themes, explanations, narratives, or descriptions supported by relevant evidence.

Some qualitative information can be counted, but counts alone may miss its meaning. For example, several interviewees might mention transportation, yet describe very different barriers involving cost, accessibility, safety, or scheduling. Researchers should explore those differences and avoid inferring feelings or intentions solely from body language or other ambiguous observations.

Quantitative research

Quantitative research examines numerical information, such as participation rates, travel times, service costs, survey responses, or changes in defined outcomes. Analysis can describe patterns, compare groups, and estimate relationships or effects when the research design supports those claims. Results require interpretation in light of measurement quality, sampling, missing information, uncertainty, and other possible explanations.

Numerical findings are not automatically more accurate or objective than other evidence. For example, a high program completion rate may conceal unequal access or differences in participants' experiences. Understanding how information was collected and what it represents is essential to interpreting the numbers.

Early in a policy change, interviews may reveal emerging experiences before administrative data show broader patterns. In a hypothetical study of changes to public assistance, adult learners might describe effects on class attendance, childcare, or work schedules. Those accounts can identify important questions and possible consequences, while later numerical information may help assess how widespread particular patterns are.

Choose methods according to the questions, context, and intended use of the findings. Combining qualitative and quantitative approaches can deepen understanding when the team has a clear plan for analyzing and integrating the information.

CBPR may be especially useful when:

• Partners have time and resources to establish shared expectations, build relationships, and provide the support needed for participation.
• Community knowledge is essential to defining the questions and interpreting findings, and the team can obtain the methodological expertise required.
• The work depends on an accountable relationship with the community, especially where previous research has damaged trust or provided little local benefit.
• Residents and organizations want to share decisions about research and the actions that may follow.
• Building research capacity, leadership opportunities, and institutional accountability is part of the purpose.
• Partners are prepared to connect research with sustained efforts toward community-defined change.

How do you organize and carry out community-based participatory research?

A CBPR project involves building a partnership, agreeing on priorities and responsibilities, preparing the team, choosing questions and methods, gathering and analyzing information, sharing findings, and deciding what action to take. Partners should also assess the quality of the research and the partnership itself. These activities often overlap and may be revisited as the team learns.

The steps below can be adapted to projects initiated by residents, community organizations, public agencies, or academic partners. No single organizational structure is ideal for every situation. Community-led teams may already have substantial research expertise, while institutional teams may need support in sharing power and working collaboratively.

Recruit a community research team

Recruitment should reflect the project's purpose and the range of people affected. An existing group may provide a starting point, but consider whose experiences are missing from its networks. Some projects need participation across several communities; others focus on a particular group, such as people using a service, residents who share a language, or people living with a specific condition.

Give directly affected people substantial representation and meaningful decision-making authority. Numbers alone do not establish equitable participation: a community majority can still be excluded from budget, data, or publication decisions. Agree on governance, compensation, meeting access, and support so that community partners can influence the work in practice.

Existing relationships with an organization or program can help with recruitment, but service users and employees must be able to decline without pressure or negative consequences. Recruiting more broadly also requires clear communication and deliberate efforts to reach people outside established networks.

• Use accessible language and formats. Explain the purpose, responsibilities, compensation, and choices in plain language. Provide translation, interpretation, and alternative formats according to participants' needs.
• Use communication channels people already rely on. These may include neighborhood organizations, faith communities, newsletters, messaging groups, local media, or personal invitations. Use several approaches when one channel would exclude part of the community.
• Develop messages with people familiar with the community. Ask what information matters and how it should be communicated. Avoid assuming that everyone sharing an identity has the same preferences or experiences.
• Make contact in accessible and appropriate settings. Share information where people live, work, gather, or connect online, with permission where needed. Offer ways to learn more without requiring travel or an immediate commitment.

A personal invitation from a trusted person can be helpful, but there is no single best recruitment method. Offer private ways to ask questions or decline, and review whether your approach is reaching the people you intend to include.

Orient and train the research team

Orientation and training may overlap, but they serve different purposes. Orientation helps partners understand the project, their roles, and how they will work together. Training develops the knowledge and skills needed for specific research activities.

Orientation might include:

• Introductions that allow members to share their interests, experiences, and preferred ways of participating
• An explanation of CBPR, the project's starting questions, and the decisions partners will make together
• Clear information about time commitments, compensation, on-site childcare or other caregiving support, transportation, accessibility, interpretation, and available equipment
• Opportunities to ask questions, revise expectations, raise concerns, and agree on decision-making and conflict-resolution procedures

Use orientation to establish a team of colleagues with complementary expertise. Formal education, professional titles, language fluency, or familiarity with meetings should not determine whose contributions count. Agree on practices that make discussion accessible, such as explaining technical terms, sharing materials in advance, and offering several ways to contribute. Institutional partners should also examine assumptions and practices that may limit others' participation.

Training should build on what members already know and address the tasks they will undertake. Different training approaches may include workshops, practice sessions, peer mentoring, demonstrations, and support during fieldwork.  Community members and technical partners can both teach and learn. Schedule training so that people can practice skills before using them and receive feedback throughout the project.

The following practices can make training more useful and accessible:

• Choose a comfortable, accessible space or online format that supports participation.
• Provide appropriate food, drinks, and other practical support when possible, asking about dietary and access needs.
• Plan regular breaks and flexible pacing. Ask participants what they need and allow people to step away without having to disclose personal information.
• Arrange the space or technology to support interaction. Consider audibility, visibility, mobility access, captions, interpretation, and ways to participate without speaking to the full group.
• Present and practice material in several ways. Combine clear explanations, accessible written or visual materials, discussion, demonstrations, and hands-on practice. Check understanding through supportive activities rather than assuming that one presentation works for everyone.
• Use training to strengthen the partnership. Make room for questions, shared problem-solving, feedback, and discussion of the principles guiding the work.

Training content will depend on the project, but may include:

• Research methods and ethical practice. Cover the relevant research and information-gathering methods, such as interviewing, surveying, observation, or reviewing records. Include consent, confidentiality, data security, appropriate boundaries, and how to respond if someone becomes distressed or wishes to stop.
• Community and program context. Share local history, existing evidence, institutional relationships, and differing perspectives on the issue, recognizing that no single person represents the whole community.
• Collaboration, facilitation, and negotiation. Practice listening, explaining reasoning, sharing decisions, raising disagreement, and preventing a few voices from dominating. Meeting practices should be negotiated by the team rather than treated as rules everyone must already know.
• Analysis and reporting. Help members interpret evidence, consider alternative explanations, and distinguish findings from recommendations. Reports may use written text, charts, presentations, video, audio, or oral testimony, with appropriate accessibility and privacy protections.
• Presenting findings. Practice communicating the research clearly, answering questions, explaining uncertainty, and connecting recommendations to evidence. Prepare for disagreement without implying that presentation skills alone determine whether decision-makers will act.

Determine the questions the research or evaluation is meant to answer

Develop research questions together, beginning with what community members want to understand and what decisions the findings could inform. Prioritize questions that are meaningful, answerable, and feasible, and record important questions that the project cannot yet address.

An evaluation may examine implementation: What was done, who participated, and how did delivery compare with the plan? It may examine outcomes: What changed, for whom, and were there unintended effects? It may also explore how implementation relates to outcomes and what needs adjustment. Claims that an intervention caused a change require a design and evidence capable of addressing alternative explanations.

Research on a community issue may ask whether a condition exists, how widespread it is, how people experience it, or which groups face particular barriers. The team may also compare possible responses. A commitment to improving conditions should guide the purpose of the work without predetermining what the findings must show.

Plan and structure your research activity

Develop a plan that fits the questions, timeline, resources, and team's capabilities. Agree on governance, compensation, data stewardship, and how findings will be used. Before collecting information, determine which institutional, community, or other ethical review procedures apply and obtain any required approvals.

Your plan should include:

• The methods, sources, and amount of information needed, including how participants will be selected, what consent procedures will be used, and how unnecessary data collection will be avoided
• Roles, responsibilities, decision-making authority, compensation, and support for each part of the work
• A realistic timeline and budget, including time for relationship-building, training, review, and adjustments
• How information will be stored, protected, analyzed, and interpreted, who will have access, and how long it will be retained
• The formats for reporting findings, the process for reviewing drafts, and agreements on authorship, acknowledgment, and use of quotations or images
• When, how, and with whom findings will be shared, including community access to results and responsibility for following up on recommendations

Anticipate challenges and agree on ways to respond

Research partnerships may encounter internal conflicts, changes in availability, funding delays, outside pressure, or problems with data collection. Establish clear guidelines for raising concerns, adjusting responsibilities, resolving disputes, and responding to privacy or safety incidents. If someone cannot complete a task, ask what barriers or support needs are involved before assuming a lack of commitment.

Implement your research plan

Pilot the methods, revise them as needed, and begin collecting information using the agreed procedures. Hold regular check-ins to address questions, review data quality, and identify missing perspectives. Analyze and interpret findings collaboratively, documenting changes to the plan and examining evidence that challenges the team's initial assumptions.

Prepare and present your report and recommendations

Develop the report together in formats suited to the intended audiences. Explain the questions, methods, findings, limitations, and reasoning behind recommendations. Different audiences may need different levels of detail, but the findings should remain consistent. Share results with the community, recognize contributions fairly, protect identifying information, and document significant disagreements rather than hiding them.

Possible ways to share findings include:

• A media briefing or press conference with agreed spokespersons and clear privacy boundaries
• An accessible community meeting, presentation, or discussion in relevant languages
• A local news article, newsletter, community website, or other communication channel
• A written or alternative-format report for community members, funders, agencies, and other interested audiences

Take or advocate for appropriate action

Action may include revising a program, improving service access, proposing a policy change, meeting with decision-makers, organizing a public demonstration, or starting a community initiative. Choose actions together based on the findings, community priorities, available resources, and likely consequences. Assign responsibilities and timelines, and identify what further research is needed where evidence remains incomplete.

Follow up

A project continues beyond the presentation of findings. Track responses to recommendations, assess the effects of any action, and provide opportunities for the team to reflect on what it learned. Plan how relationships, skills, and resources can remain available to the community after a particular grant or study ends.

• Evaluate the research and partnership process. Invite all partners to assess the quality of the methods, training, communication, compensation, and decision-making. Ask whose contributions influenced decisions, which barriers remained, and whether people felt able to disagree. Provide confidential ways to raise concerns where needed.
• Assess benefits, harms, and unintended effects. Examine what changed for the community, who benefited, and whether the project created burdens, conflict, exposure of sensitive information, or other problems. Distinguish observed results from hoped-for future benefits.
• Invite team members to reflect on their learning and goals. Areas may include research skills, public communication, collaboration, confidence, professional opportunities, and influence over decisions. Let participants define what matters to them without assuming that everyone should experience the same changes.
• Support continued involvement when members want it. Options may include another community-prioritized project, a paid research partnership, mentoring, consulting, or connections with other efforts. Plan for resources and leadership continuity while respecting people's choice to leave or take a break.

CBPR is one approach to research and evaluation, and its suitability depends on the purpose and conditions of the work. Use the term honestly: a project needs meaningful partnership, shared influence, and accountability to the community, not simply community recruitment or data collection.

When adequately supported, CBPR can produce useful knowledge, strengthen relationships and research capacity, and connect evidence with community-defined action. Its benefits depend on the quality of the methods, the distribution of power and resources, and whether partners follow through on their commitments.

In Summary

Community-based participatory research involves people most affected by an issue as partners in defining, conducting, interpreting, and using research. It combines different forms of expertise, supports shared learning, and connects findings with practical action. Meaningful participation includes influence over decisions and access to the knowledge the project produces.

CBPR requires time, resources, and sustained attention to relationships. Partners should build on existing strengths, provide fair compensation and accessible support, and address barriers to participation. Clear methods, ethical practices, shared accountability, and follow-through can help the research contribute to lasting improvements in community conditions.

What is community-based participatory research?

___You define community-based participatory research as research conducted by and for those most directly affected by the issue, condition, situation, or intervention being studied or evaluated

Why would you use community-based participatory research?

___It yields better and more nearly complete and accurate information from the community

___Involving the community in research makes a positive outcome that meets community needs more likely

­­___Action research trains citizen researchers who can turn their skills to other problems as well

___Involvement in CBPR changes people’s perceptions of themselves and of what they can do

___Skills learned in the course of action research carry over into other areas of researchers’ lives

___A participatory action research process can help to break down racial, ethnic, and class barriers

___Action research helps people better understand the forces that influence their lives

___Community-based participatory research can move communities toward positive social change

Who should be involved in community-based participatory research?

___People most affected by the issue or intervention under study

___Other members of the affected population

___Decision-makers

___Academics with an interest in the issue or intervention in question

___Health, human service, and public agency staff and volunteers

___Community members at large

When should you employ community-based participatory research?

___There’s time to properly train and acclimate community researchers

___The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)

___You need an entry to the community or group from whom the information is being gathered

___You’re concerned with buy-in and support from the community

___You want to have an effect on and empower the community researchers

___You want to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

___You recruit a community research team

___You orient and train the research team

___The team determines the questions the research or evaluation will try to answer

___The team plans and structures the research activity

___As a team, you carry out your plan

___The team prepares and presents a report and recommendations on its work

___You take, or try to bring about, appropriate action

___You evaluate the process

___You provide an opportunity for team members to reflect on and discuss their learning and the effects of the experience of being involved in a CBPR process

___You maintain gains by keeping team members involved

Example 1: The Changes Project: Holyoke Community College and SABES West

The Changes Project was a three-year participatory action research effort. Funded by a grant from the Office of Educational Research and Improvement, part of the U.S. Department of Education, its goal was to determine the effects on adult learners of the major changes in welfare regulations, immigration laws, and the workplace that took place in the 1990’s.

Background

The project was participatory in two senses:

• The 42 researchers (34 of whom were able to participate for the whole three-year span of the project) were themselves current and former learners and staffers in adult literacy, GED (high school equivalency), and ESOL (English as a Second or Other Language) programs
• All the effort’s data was obtained from 600 learners in the programs from which the researchers came

All the researchers were learners, graduates, or staff members from one of five Western Massachusetts adult education programs.  The Center for New Americans (CNA) in Greenfield, Amherst, and Northampton and the International Language Institute (ILI) of Northampton are independent organizations that work with ESOL learners.  The Mentor Program at Holyoke Community College provides services to adult learners who are making the transition between adult education and higher education.  Read/Write/Now (RWN) Adult and Family Learning Center, sponsored by the Springfield City Library, offers adult and family literacy and GED classes.  The University of Mass. Labor/Management Workplace Education Program (LMWEP) specifically serves University employees with a range of instructional offerings that encompass basic skills, technology, communication, problem-solving, and workplace-specific areas.

Research team members ranged in age from 22 to 55, and in education from adult literacy learners to holders of advanced degrees.  Among them, they spoke six languages; came from six countries on three continents; and claimed Black, Asian, European, and Hispanic heritage.  Twenty-six were parents (ten being single parents), and ten were current or former welfare recipients.  In general, the researchers comprised a group broadly representative of the community of Western Massachusetts adult learning center participants and staff.

Each of the five sites fielded a research team that worked both independently and in conjunction with the other four.  In addition to its other members, ranging in number from 3 to 13, each team included a site research facilitator (SRF) – a program staff member who was both a member and facilitator of the site’s research team.  SRF’s regularly met together with the program coordinator, and also received consultation from a research advisory group and a methodologist.

The research question the project examined was generated by the administrators and staff of 14 western Massachusetts adult learning programs, in response both to the voiced concerns of learners and to their own observations.

Goals

As stated in the Changes Project report, research goals were to learn more about:

• Effects of welfare reform, immigration reform, and the changing requirements of the workplace on adult learners at five adult literacy programs in Western Massachusetts
• Ways in which adult learners respond to and accommodate immigration reform, welfare reform, and the changing requirements of the workplace as they pursue their goals through education
• How practitioners and programs can alter educational services to be more effective, given welfare reform, immigration reform, and the changing requirements of the workplace
• Ways in which policies can be more responsive to the needs of adult learners as they deal with these three issues

The Changes Project began with a challenge: how to mold a large and diverse group of researchers into a team that could work together creatively and effectively.  Changes Project Coordinator Alex Risley Schroeder recalls that this challenge was made even greater by the fact that the project had not been 100% participatory from the beginning.

“We had to write the proposal, and, because of time and other constraints, we did that without the direct participation of learners.  It was something we struggled with throughout the life of the project.  We had teams of learner/researchers, and said to them, ‘Okay, we’ve promised the funders certain things.  Now what can we, as a group, do to make those things reflect what’s important to all of us?’”

There were only two occasions at the beginning of the project when the whole group, all five site teams, got together: an initial orientation/training and a project kick-off.  The real work of team-building actually began when the first set of data came in.

Because of the differing nature of the five adult education programs and their learners, not all teams researched all three areas of concern.  Most of the work on immigration reform, for instance, came out of the research of teams from ESOL programs. The one workplace education program team didn’t research the effects of welfare reform, because all learners in that program were, by definition, employed; it concentrated instead on changes in the workplace.

Data Collection and Information Synthesis

For all teams, information was collected in three phases from learners in their own programs:

• Individual interviews. Teams developed appropriate interview questions for the area they were researching, and conducted a first round of individual interviews.
• More interviews, focus groups, and surveys. Teams conducted a second round of interviews (including some follow-ups – almost all interviews were followed up with second interviews) with revised questions, and also conducted focus groups in each program and administered surveys project-wide.
• More interviews, more focus groups, and three separate surveys, each specifically aimed at learners with firsthand knowledge of one of the three areas of concern.

After each phase of data-gathering, teams struggled with ways to sort out and report what they had found, and with how to make sense of it. There was an enormous amount of material, most of it in the form of people’s stories about their experiences. Each team found its own ways to organize data.

Each phase ended with an Analysis Fest, a coming together of all the teams to try to integrate and analyze what they had found. The meetings were called Analysis Fests, according to Risley Schroeder, to capture both the excitement and the real joy to be found in gaining and making sense of knowledge.  It was in the Analysis Fests, she says, that the five teams really became a coherent unit.

“Not everyone knew the things that were obvious to some of the teams.  What women on welfare went through, for instance, or what it was like to not be able to speak English.  It was a very powerful learning time for all of us, and it gave everyone a chance to get to know others as teams and as individuals.  Really, to make this work, we all had to understand who we were as a large group, as teams, and as individuals.”

The group wrestled with organizing information and stories they had collected. The challenge was compounded by the fact that some researchers had difficulty reading and writing, or with English. It was made yet more difficult because the researchers, at the beginning, thought of themselves as having no expertise, and therefore no tools for the task.

The first Analysis Fest started out print-heavy. There were transcripts of all the interviews, many painstakingly translated into English from Spanish or other languages. The teams were constrained by what they thought “research” was.  In the subsequent Analysis Fests, as researchers relaxed and started to put the material from different teams together, they realized there were themes common to many of the interviews, regardless of the circumstances of the learners who had been interviewed. Those themes became both the core of the project’s ultimate findings and the key to organizing the data.

Those who had the most difficulty with print or with language were also the most creative in getting around that difficulty. They made up representative stories for each of the issues, used pictures and sound as well as – or instead of – print to show what they had found, and developed dramatic interactive role plays to demonstrate the kinds of difficulties learners faced in dealing with welfare and immigration reforms and workplace issues.

One of the most important findings of the project was that adult learners often depended on fragile networks of support of various kinds in order to be able to attend classes and learn successfully.

There were essentially four types of support necessary:

• Practical supports. These include the kinds of supports most people would understand to be necessary for successful completion of an adult education program: food, clothing, shelter, health care, child care, and transportation. But this category also includes information – about welfare recipients’, immigrants’, and workers’ rights, about the laws that govern those areas, about how to find and enroll in programs, about how to study, about how to find other academic and employment resources.
• Inner resources. Hopes, dreams, motivation, focus, determination, natural ability, prior knowledge, spiritual faith.
• Personal supports. These are both emotional and practical, and come not only from family and friends, but from professionals and the community at large. They include advocacy for individuals and programs, as well as respect, encouragement, and belief in people’s ability to change their lives.
• Institutional supports. These include not only the existence, but the responsiveness and appropriateness of programs, organizations, and institutions that provide help and services to those who need them; state and national policies that speak to the real needs of the people at whom they’re aimed; and regulations that accommodate the real needs and circumstances of those they’re meant to help.

The researchers conceived of these supports as threads in a web, all intertwined and interdependent. Cutting any one thread weakens the whole web, and makes an adult learner’s success less likely. The metaphor of the web made the concept of support easier for everyone to see and understand, and also made it clear how important it was that policy makers see that an adult learner’s success is dependent on far more than what goes on in a particular ABE or ESOL program.

Results

Ultimately, the Changes Project accomplished a number of purposes:  

It gathered a large amount of important, first-hand data about the effects – intended and unintended – of welfare and immigration reforms and changes in the workplace on adult learners and on their chances for successful pursuit of their educational goals.

It had a profound effect on the researchers themselves. Both adult learners and staff members found that the experience of participating in the project changed the way they saw themselves and the world.  More specifically:

• Team members increased their ability to communicate and developed job-related skills, and learners specifically increased their literacy and English language skills.
• Team members felt more self-confident and saw themselves as more competent and more able to contribute as a result of their work in the Changes Project.
• Many team members felt they had developed a voice, enabling them to speak up and be heard in both their private and public lives in ways that hadn’t been possible for them previously.
• Their Changes Project experience led many researchers to become active in their communities, and to change their perspective on political action in general and on their own ability to change things in particular.
• Some of the teachers among the team members found that their view of their jobs and of the learners they worked with had changed.  One explained “This project has given me that ability [to be a reflective teacher] because you really have to listen when you’re doing research.  It’s helped me to let go of...being in charge of how the research goes.  Learning to do that has changed me – it’s changed the way I think about how knowledge gets made.  It’s really different when you do it in a group. What you come up with is much richer.”
• The experience of building and belonging to a community that encompassed people of different cultures, ethnicities, and backgrounds stood out as enormously positive for all participants.  Many talked about the excitement of learning about how other cultures viewed the world, and the rewards of learning to get along with people with different perspectives.
• Team members gained valuable technical knowledge about conducting research, including  how to collect and analyze data (as well as how to engage in logical analysis in general), how to adjust research to respond to the culture of those being addressed, and how to develop the interpersonal skills needed to gain information from interviews and focus groups.

The project served as a model, not only for research, but for adult education. Many of the researchers, both teachers and learners, found that their participation had changed their views of learning.  hey realized that it could be fun, practical, and group-centered.  The skills were being learned, perhaps even more effectively than when instruction was focused on them, in the context of work that had a real-world goal.

The research identified peripheral issues that have great consequences for adult learners. The web of support described above is crucial to the success of a majority of learners; yet policy makers are ignorant of it, and often unwittingly unravel it, thereby making it more difficult for learners to succeed.

The Changes Project drafted a series of recommendations for the field, for learners, for workplaces, and for policymakers.

• Recommendations for the field center on how to make programs more effective by accommodating learner needs, including non-academic ones; by strengthening the web of support; by increasing learner participation in the organization’s decision-making; by providing help in transition to next steps – further education, career planning, employment; and by maintaining an organizational atmosphere of mutual respect and support
• The project recommends to learners that they take responsibility for getting all the information they need, for seeking support, and for becoming involved and active in their programs.
• Recommendations to workplaces include their providing support for workers’ occupational and professional development (including career ladders within the workplace, as well as education and training); maintaining an atmosphere of mutual respect and inclusion; recognizing and building on workers’ strengths, particularly those of immigrants; and being sensitive to the cultural diversity of the workforce.
• The Changes Project’s recommendations to policymakers and legislators, although they concern larger issues, are similar in some ways to those for others. They suggest mandating that welfare and immigration caseworkers develop respectful relationships with those they serve; that more effective policies be developed by soliciting the participation of those at whom those policies are aimed; that information and services be provided in whatever languages are necessary; that welfare policies be responsive to the actual needs of recipients; that education and training count toward the welfare work requirement; and that policies encourage fathers to take responsibility for their children.

Participation in the project led to some action on the part of the researchers. This was both perhaps the most promising and the most frustrating part of the effort. One of the greatest pieces of learning for researchers was that action research really does blur the distinctions among research, analysis, and action.  Each of the research teams initiated some action, not so much as a result of, but rather as an integral part of, its research and findings.

• The Changes Project’s research lit a fire under a writing group of parenting teens who had been interviewed by the team, and who wanted to write about their situation.  Their writings were incorporated into a group-composed “Mother’s Day Letter” (entitled “What’s Going to Happen to Us”) that was sent to legislators, the media, and other adult education programs.  As a result. the local paper published a story on the effects of welfare reform on teen parents, and included interviews with several of the writers.
• The CNA team made presentations of their recommendations to the staff and administrators of the program, and also participated in rallies and direct advocacy with legislators for immigrants’ rights at the State House.
• The Read/Write/Now team initiated a letter-writing campaign to their local, state, and national representatives, detailing the real effects of welfare reform on adult-learner welfare recipients.  They also participated in advocacy for economic justice, and, ultimately, self-published a widely-distributed (to legislators, other programs, and the public) book telling the personal stories of many of those learners.  They hope to dispel stereotypes of welfare recipients, and to help people understand the true bind that many adult learners find themselves in.
• The ILI team’s action was centered closer to home.  Struck by the stories of those they interviewed, team members undertook to provide information and assistance to fellow learners in the program.  As did the others, they also engaged in advocacy for immigrants’ rights by attending State House rallies and other events, and by making presentations about their work.

Finally, the UMass Workplace Education team responded to what they learned in two ways:

• Team members created a support and problem-solving group, called Making Work Work, in order to explore and affect workplace issues.This group has met with a group from the University’s Training and Development Unit to design new programs and involve more staff in planning.
• The second important action to grow out of LMWEP’s research is the development of a career ladder for University workers, a result of meetings between the union and University administration initiated as a result of the Changes Project.

While all of this action has helped those involved feel that they now have more of a voice in their own destinies, it is frustrating as well. Except for the LMWEP progress in the workplace, most of the actions taken had little or no effect on the situations they addressed.  Welfare and immigration reform have gone through as originally planned, with virtually no concessions to the needs of people who are being deprived of the education and training that could help them to become financially successful and better citizens.  Legislators and others have listened sympathetically, but the results have shown little sympathy to the poor and/or newly arrived.

All this can be taken as an excuse to stop trying to bring about social change, but it can also be seen as the beginning of something. A number of people who previously thought of themselves as powerless have seen that they can at least struggle for what they believe in or what they need. They have learned, and have passed on, ways of finding and examining information that give them more choices. And they seem to be willing to continue to practice what they’ve learned. It is in circumstances like those created by the Changes Project that the seeds of change take root. The name seems more than fitting.

Example 2: Community implements participatory action research for proposed business deal

When an out-of-state hazardous waste recycling company approached an economically-depressed New England mill town with an offer to build a plant in the community, the town leaders were overjoyed. The plant would mean a boost to the tax base, and the company promised that both the construction contracts and most jobs at the plant – jobs they described as well-paid – would go to local residents.  Without much discussion, the town tentatively agreed to grant permits, and to waive environmental regulations where necessary.

Some citizens were worried, however. The plant was scheduled to be built on land that drained into the aquifer that fed one of the town’s two wells. Although the company claimed that its process was foolproof and waste-free – all the waste they collected was to be recycled into other products, and there would be no chance of any pollution from their operation – it still seemed ill-considered to grant permission for something so dangerous-sounding without learning more about it.

The group of concerned citizens came from diverse backgrounds. One of the leaders was a teacher; another was a single mother on welfare. The group’s membership spanned the social structure of the town. Arrayed against it were town officials, concerned with growth in a period of economic stagnation, and many of the unemployed and underemployed citizens who hoped that the plant would mean jobs for them, and a better life for their families.

The town split down the middle, with officials refusing to budge, and neighbors and long-time friends ceasing to speak. The members of the concerned group realized they had to understand as much as they could about the company and its process, and set out to do research on both. They learned about hazardous waste recycling, and about both the waste the plant would accept, and the products it intended to turn that waste into.  They sought information on the company, discovering that it had operating plants in the Midwest.

Group members called officials in the Midwestern communities where the plants were located, and learned that there had, in fact, been some environmental problems. They also found out that the plant’s proposed methods were still largely untested, and that there were many scientists who believed that there were serious flaws in them. Armed with this information, the group notified the press and called a public meeting.

After a chaotic, but ultimately productive, evening, the group convinced enough people of potential problems that elected officials were pressured into appointing a commission to study the issue of the hazardous waste recycling plant.  Several members of the concerned citizens’ group were appointed to the commission, along with several other community members who favored the plant.

Over the course of several weeks, the commission met with company officials, and continued to do research on the science of hazardous waste recycling, receiving help from chemists and nuclear physicists at the nearby state university. Many of the commission members who had initially favored the plant became suspicious of the company’s motives, and questioned its truthfulness. Finally, the commission decided to send two of its members to one of the company’s other plants.

The commission delegation found a community that felt it had been lied to. There had been some serious leakage problems associated with the plant, and the company had not responded well.  Rather than admitting the problem and trying to fix it, company officials had tried to pretend there was nothing wrong, and accused the town of being alarmist.  When they were finally forced by a regulatory agency to clean up their mess, they had difficulty doing so, and many in town felt they still hadn’t succeeded.

Furthermore, the company had promised the community good jobs for local people, but the jobs hadn’t materialized. The well-paying jobs at the plant required graduate degrees in chemistry and other sciences, and the low-paying jobs local workers were offered were often part-time, and at wages no better than those available at Wal-Mart.

The New England town commission had heard enough. They returned and convinced the town to vote against the permits the plant needed.  Before that could happen, however, the company withdrew its offer, and looked for a site elsewhere in the country. Some people in town were still angry about what they saw as a loss of jobs, but most felt they had dodged a bullet.

A citizens’ group had turned itself into a participatory action research project, and had used the results of its research to take effective action on an issue of public importance.  Many of those involved in the effort are still active in the community. Others felt their moment of “fame” came at too great a cost, and have withdrawn from public participation.  In a small town, that is one of the risks of becoming involved in action research.

Example 3: True Collaboration for a Win-win in Community-engaged Research

“When we think about the word ‘partner,’ we can think about a significant other. If I were to say, ‘what do you want out of your partnership with your significant other?,’ you may say things like, ‘you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year.’ Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things. We want someone who is with us, who engages us from the very beginning of a process--so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this? I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But, as a partnership, we have to work through those solutions. Those are some of the things that we look for in a community-based research partnership, that flexibility and understanding. One of the things I noticed as I went from being a researcher to being in practice is that you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?” What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information. So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.”

Example 4: Compassion Fatigue in Evaluators

Evaluators working with trauma-exposed populations often develop compassion fatigue in the same way caregivers might. Compassion fatigue is the psychological strain one experiences due to sustained empathetic engagement. Dr. Stacy Huff's research analyzes and provides suggestions for managing evaluator compassion fatigue.

Compassion fatigue affects everyone differently. Many factors play a role in an evaluator's likelihood of developing compassion fatigue and their ability to overcome it. Things like the type of evaluation, personal identity, and organizational support all have an effect.

• Type of evaluation: Evaluators working with refugees, Indigenous populations, and those facing systemic oppression are more likely to experience compassion fatigue. They might also face ethical tension due to their proximity to trauma-exposed communities.
• Personal identity: Historically marginalized individuals, such as people of color and women, may face high mental strain from evaluation work. Evaluators who work with groups with whom they have a personal connection are more likely to experience compassion fatigue when hearing stories similar to their lived experiences.
• Organizational support: Many evaluators do not have a support system in place and are thus left to manage this large emotional weight by themselves.

Recommended resources for evaluators to recover from fatigue.

• Understanding your struggle: The Compassion Fatigue Awareness Project provides resources for understanding and preventing compassion fatigue within the workplace.
• Meditation: Huff specifically recommends The Guided Meditation for Compassion Fatigue to manage compassion fatigue.
• Overcoming burnout: The Reducing Compassion Fatigue, Secondary Traumatic Stress, and Burnout workbook is a great place to start!

To read Dr. Huff's blog post: https://aea365.org/blog/drg-tig-week-compassion-fatigue-in-evaluation-t…

Tool: Involving Community Members in Evaluation

When community members are involved in evaluation, all aspects of the project, from identification of the question to dissemination of results, are more likely to reflect the priorities, values, and experiences of the community. In any evaluation or research project, community involvement can be considered along a continuum, where community members have increasing levels of influence and responsibility. This planning framework, which can be used by organizations responsible for conducting or funding evaluation projects, can increase transparency, establish shared expectations, serve as a starting point to further clarify roles, and increase accountability. Read more from the Amherst H Wilder Foundation website.

Download the PDF for the tool here.

 

Tool: Data Walks and Narrative Methods

A Data Walk is a participatory process that engages stakeholders, including community members, in interpreting visual data such as maps, charts, photos, and statistics. Stakeholders should be engaged in the design of Data Walks to select which data to display and identify appropriate spaces to host the walk. This collaborative approach increases ownership, strengthens interpretation, and supports sustainability by positioning community members as co-creators of insight rather than passive recipients of information. 

Evaluators can consider using the Theory, Context, Characteristics, and Methodology (TCCM) framework to organize and reflect on what emerges from the walk. Using a structured reflection process ensures that community narratives are not only heard but thoughtfully integrated into context-sensitive evaluation and decision-making. Read more from the American Evaluation Association website. 

• Who are community leaders, evaluators, and funders?

• Why should you understand the interests of these groups?

• When should you understand the interests of these groups?

• What are the interests of these groups?

• How do you determine interests?

Your group wants to understand how its work is going. Community members want to know whether the effort addresses their priorities, and funders may want evidence of progress and responsible resource use. Evaluation can help answer these questions. Planning together makes it easier to identify what people need to learn, choose appropriate methods, and connect findings with decisions.

An important planning step is identifying the people involved in or affected by the evaluation and understanding their interests. This section focuses on community leaders, evaluators, and funders. Program participants, residents, staff, and others affected by the initiative also need meaningful opportunities to shape the evaluation, whether or not they hold a formal leadership role.

Designing the evaluation together can reveal shared priorities and differences that need discussion. One evaluation may serve several purposes, but it may not answer every question equally well. Agree on the most important questions, intended uses, and practical limits before collecting information, and make those choices transparent.

By needs and interests, we mean what people want to understand, the decisions they need to make, and the outcomes they consider important. These may include improving services, identifying unequal access, learning from implementation, or deciding how to allocate resources. People within the same group may have different priorities, so ask about their interests rather than assuming them.

Begin by considering why people want an evaluation.

Common reasons include:

• Being accountable to community members, participants, and those who provide resources
• Helping organizations receiving grants learn from and improve their work
• Improving funding decisions and support for community priorities
• Assessing program quality, accessibility, outcomes, and unintended effects
• Planning, adapting, and implementing programs
• Sharing useful approaches and the conditions needed to apply them elsewhere
• Building knowledge and strengthening the community's ability to use evidence

Any person or group may have several of these interests. Community leaders may want credible evidence about outcomes, evaluators may want findings that inform practical improvements, and funders may want to understand implementation challenges. Their interests can overlap. They may also differ in the decisions they face, the time they can contribute, and the formats in which they need information.

You do not need to be an evaluation specialist to help choose useful questions or identify community priorities. You do need a clear understanding of how findings will be used and what evidence is needed. Bring in technical support when the design, analysis, or interpretation requires skills the team does not yet have.

Who are community leaders, evaluators, and funders?

Community leaders

Community leaders may include residents, youth leaders, organizers, committee members, staff, administrators, public agency representatives, and members of governing boards. Leadership can be formal or informal. These individuals bring different experiences and levels of familiarity with evaluation, and they may want information to guide planning, improve services, or advocate for change. Their involvement should complement direct participation by affected residents rather than replace it.

Evaluators

Evaluators plan and carry out activities to answer questions about an initiative's implementation, outcomes, or value. They may be community researchers, program staff, independent consultants, or people working in universities and other organizations. Their roles should match their skills and the evaluation's requirements. Professional learning resources include the American Evaluation Association and its Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. Evaluators contribute methods for gathering and interpreting credible information while learning from community knowledge and experience.

Funders

Funders are individuals or organizations that provide financial support. They may include donors, foundations, government agencies, and other funding partners. Their evaluation expectations vary: some require particular measures or reports, while others help partners develop an approach together. Funders may want to understand progress, resource use, community outcomes, and what kinds of support would make the initiative more effective.

Why should you understand the interests of these groups?

Different people need different information to make decisions. A resident may want to know why a service is difficult to access, a program coordinator may need to understand staffing demands, and a funder may need evidence to inform continued support. Understanding these questions helps the team choose an appropriate scope, methods, and reporting plan.

People outside the initiative may have limited information about its activities and results. Those participating in or supporting the work may also see different parts of it. Evaluation can bring these perspectives together to examine what is working, what needs to change, and what remains uncertain. Include people who experience the program's effects as well as those who can provide funding or institutional support.

Information is useful when it helps answer a relevant question. For example, counting the number of people served can help describe program reach and staffing needs. It does not, by itself, show whether participants' circumstances improved or whether the program contributed to that improvement. Clarifying the intended use of information helps the team distinguish activities, outputs, and outcomes and avoid collecting data without a clear purpose.

Connect people's interests with evaluation questions and appropriate methods.

• Whose interests and experiences should the evaluation address?
• What decisions do they face, and what information would help?
• What evaluation questions will guide the inquiry?
• What methods and tools can provide credible answers within the available resources?

Meaningful involvement can strengthen collaboration and the ability to use findings. It can also reveal competing priorities before they disrupt the work. Understanding those priorities helps the team allocate resources, explain tradeoffs, and design an evaluation that supports learning without allowing the most influential group to determine every question.

When should you understand the interests of these groups?

Identify interested and affected people at the beginning of evaluation planning. Invite them to help define the purpose, questions, and intended uses before major design decisions are settled. Include people who may be affected by changes based on the findings, even if they are not currently involved in the initiative.

Revisit interests at agreed points and when circumstances change, such as a new funding requirement, a change in community priorities, or an unexpected finding. Adapt the plan when appropriate, documenting why changes were made and how they affect the evidence or comparisons. A new interest should prompt discussion rather than an automatic change that undermines the evaluation's purpose.

What are the interests of these groups?

How do you identify the people to involve?

Begin with a shared list of people and organizations involved in, affected by, or expected to use the evaluation. Ask community partners to help identify perspectives missing from the team's existing networks. Record initial ideas about people's interests as questions to explore, not as conclusions about what they must want.

Members of the initiative may reasonably want the evaluation to support their own learning and improvement. That is an important purpose. At the same time, participants, residents, partners, and funders may have questions about access, outcomes, resource use, or unintended effects. Consider these interests together so that the evaluation reflects both the initiative's needs and its responsibilities to the community.

Also consider differences in power, access, and ability to participate. People who rely on a service may hesitate to criticize it in front of staff, while residents who cannot attend meetings may be left out entirely.

Develop a broad initial list, then agree on appropriate roles. Some people may help make evaluation decisions, others may contribute information or review findings, and others may mainly need access to results. Explain how each form of participation can influence the work.

To identify people and groups to involve, ask:

• Who funds, supports, or makes decisions about the initiative, and how might they use the findings?
• Who will conduct the evaluation, and whose knowledge and skills are needed to do it well?
• Who participates in, collaborates with, or is affected by the initiative, including people who face barriers to access or have been left out of earlier discussions?

Once you have identified people to involve, ask about their priorities directly. The following descriptions offer starting points for discussion. They are not fixed expectations for everyone in a particular role.

Community leaders

Community leaders and residents may need information that is:

• Clear and accessible: Explain the purpose, methods, findings, and limitations in plain language. Offer relevant languages and formats, and make supporting detail available to people who want it.
• Focused and timely: Organize information around the decisions people face and provide it when they can use it. Respect their time while preserving the detail needed to interpret findings responsibly.
• Responsive: Address community-defined priorities and explain what the evidence can contribute to decisions about the initiative's future.
• Grounded in context: Describe accomplishments, challenges, resources, and surrounding conditions. Examine differences in experiences and outcomes rather than assuming the initiative works the same way for everyone.
• Useful for action: Identify practical options for improvement, explain the evidence behind them, and clarify what further information or resources may be needed.

Evaluators

Evaluators need a clear purpose, suitable evidence, and working arrangements that support credible inquiry.

• Shared understanding: Evaluators need input from participants, residents, staff, community leaders, and funders about the questions that matter, how the initiative operates, and how findings will be used.
• Appropriate and dependable information: Evaluators need data suited to the questions, with clear definitions and information about gaps, quality, and limitations. They also need realistic timelines and resources for collection and analysis.
• Clear responsibilities and respectful collaboration: Agree on access to information, privacy protections, participation procedures, and how findings will be reviewed and reported. Cooperation should support honest inquiry, including the reporting of unfavorable results.

Funders

Funders may need:

• Clear and timely reports: Provide information about implementation, progress, challenges, resource use, and support needs in time to inform funding decisions. Agree on reporting expectations early and coordinate requests to reduce unnecessary burden.
• Credible evidence of outcomes and contribution: Explain what changed, for whom, and what the evidence suggests about the initiative's contribution. Distinguish observed outcomes from demonstrated causal effects, and report limitations alongside accomplishments.

How do you determine interests?

Use conversations, interviews, surveys, or group discussions to learn what people want to understand and how they expect to use the evaluation. Ask about priorities, concerns, decision deadlines, and preferred ways to participate. Discuss differences early, and document which interests the evaluation will address, which fall outside its scope, and why.

Questions that can help identify evaluation interests include:

• What do you most need to learn about the initiative's strengths, limitations, or value?
• Which outcomes matter to you, and what would meaningful progress look like?
• What implementation challenges or barriers should the evaluation examine?
• What should we learn about participants' experiences, access to services, and interactions with staff?
• What unexpected benefits, harms, or differences in outcomes should we explore?
• What decisions will you use the findings to make, when do you need them, and how would you like to participate in the evaluation?

Review the answers for shared priorities and important differences. For example, a resident leader and an evaluator may both want to understand why participation drops after the first month. That shared question can guide further discussion about evidence and methods. Similar interests create opportunities to collaborate, but they do not mean everyone will agree on the interpretation or recommended action.

Ways to identify and discuss interests include:

• Interviews: Speak with people across relevant roles and experiences rather than relying on one person to represent an entire group. Use clear, open questions and ask for examples. Provide private opportunities to raise concerns that people may not feel comfortable discussing publicly.
• Surveys: Use a brief, accessible questionnaire to identify priorities and invite additional comments. Offer appropriate languages and participation formats. When interpreting results, consider who responded, who did not, and whether important perspectives are missing.
• Phone or remote conversations: Offer scheduled calls or other accessible options for people who cannot attend in person. Use consistent core questions while allowing relevant follow-up. Ask about communication and privacy preferences, and explain how responses will be recorded and used.
• Facilitated group discussions: Bring people together to explore possible questions, compare priorities, and discuss tradeoffs. Use facilitation that gives everyone opportunities to contribute, including written or small-group input. Record disagreements and unresolved questions as well as areas of agreement.

Check in during the evaluation and after findings are shared to learn whether the process and results met people's needs. A survey is one option; conversations, a reflection meeting, or feedback on a report may be more appropriate. Use what you learn to improve communication, follow-through, and future evaluations.

Decisions about improvement may involve small adjustments or substantial changes, depending on the findings and their consequences. Agree on who will act, what resources are needed, and when progress will be reviewed. Keep a record of how evidence and community priorities informed the decision.

With a documented set of priorities and intended uses, you can select an evaluation team and develop methods for gathering useful information. Confirm the scope, responsibilities, timeline, and reporting plan with the people involved. Return to these agreements as the work progresses so that the evaluation remains relevant and feasible.

In Summary

Understanding the interests of community leaders, evaluators, funders, and affected residents helps an evaluation ask useful questions and support informed decisions. Identify these interests early, discuss differences openly, and revisit them as conditions change. Shared planning works best when community participation has real influence and findings are reported honestly, including results that challenge expectations.

___You understand why it is important to determine the interests of the stakeholders

___You know who the stakeholders are

___You understand the community leaders' interests

___ You understand the funders' interests

___ You understand the evaluators' interests

___You know when you should understand those interests

___You know how to identify stakeholders

___You know the needs of each stakeholder

___You know how to determine interests

___You understand the interests involved in an evaluation

Example 1: Reasons to evaluate

Different organizations will have different reasons to implement an evaluation, and different interests will be at stake. For instance, some organization may want to focus on increasing the quality of its grants programs. Some other organization may be interested in becoming more accountable. Being aware of these interests in fundamental to understand how interests must be balanced. Here we offer a table with hypothetical examples of the kinds of interests that organizations may have.

Foundation / Organization	Interest
The Meyer Memorial Trust request-for-proposal program	Improve grantee programs, improve grantmaking
L.J. and Mary C. Skaggs Foundation program for arts organizations	Be accountable, improve grantmaking
Winthrop Rockfeller Foundation at-risk youth program	Improve grantee program, disseminate successful programs
Cities in Schools program	Increase the state of knowledge
Technology for Literacy project	Be accountable, improve grantmaking, assess quality or impact of program
William Penn Foundation career / vocational program	Be accountable, improve grantee programs, improve grantmaking
Gautreaux program	Increase the state of knowledge
W.K. Kellogg Foundation grants	Be accountable, improve grantee programs, improve grantmaking, assess quality or impact of grants / programs, plan new programs, disseminate successful programs

Example 2: Leading a community

Ultimately, the community is most affected by the results of an evaluation. That's why community leaders play a key role in the interest game of evaluations. Lisa Adkins knows enough about her community. She is the executive director of Youth Friends in Kansas City. Through Youth Friends, students are linked to volunteers with a shared interest, whether it's music or sports, computers or a special hobby, to create an environment in which every young person who needs a caring adult in his or her life has one.

Lisa talked to us about how her interests as a community leader interfere in the evaluation process. However, according to Lisa, conflict of interests has not been a problem in her evaluation process. She said that the work between evaluators, funders and herself as a community leader is usually of a cooperative nature, and everybody works together toward a common goal. She considers the evaluation process a positive experience for her organization, once they get a chance to review their programs and change what needs to be changed. So, evaluations are a constant element in her work

Her main interest, Lisa says, is to receive feedback on the programs Youth Friends provide, and evaluations are a great way of getting a response from the community. Youth Friends can adjust their services as the necessity arises from the community. Lisa points out that she is always the most interested in the evaluation's results since they could change her approach to a certain program and drastically improve it.

Example 3: Evaluating the evaluator

When it comes to figuring out interests in the evaluation game, the evaluator is a key character. Vince Francisco knows this game well. He's worked as an evaluator with diverse, community-based organizations doing hundreds of program evaluations. He has done evaluations of community policing programs, chronic and cardiovascular disease prevention programming, youth development, drug-abuse prevention, alcohol-use prevention, tobacco control initiatives, teen pregnancy prevention, and general youth development initiatives. Here Vince talks about how the evaluator's interests influence the score of the game:

"I try to find out immediately who the key players are and what they want to have answered. As an evaluator, what I bring to the table is a critical set of questions that I ask folks. Sometimes it's just sitting there watching folks in action, and other times it's asking key people very pointed questions about what they want to do, why they're doing it. I work with folks to take it to the next level and make it a little bit more formal in terms of coming up with some evaluation from very different points of view. If we don't understand their program, we can't go anywhere with their evaluation. I get a handle on their program (and I) will help them make their lives easier, help them run their program or initiative better, help them get more money, influence key players in the community to be on their side, the big issues. "

To find the other interests involved in an evaluation, Vince says that he talks to people until he has their roles figured out: "I look for the loudest voices, and then I look for the quietest voices those usually are the key players. The loudest voices usually speak for themselves, and not for the community. And the people who are the absolutely quietest, are the people who really have a voice in the community, but they're being drowned out by the folks who are loud. Very loud people usually get their way and they become key players so I want to know who they are and what their agenda is. And then I find whoever are the quietest people, and they're usually the people who sit in the background. I start talking to people and eventually there are some people that will talk back. I guess I just start talking to folks and (I work with) the ones that end up talking back to me. I combine a lot of talking."

Of course, Vince says, his own interests also influence the outcome of the intervention. To balance out this equation he thinks of the whole group or the whole community as his client. Talking to people, getting information, and shaping them, Vince produces something that works for his clients: "I outline and define the program : What's the initiative? What does it look like? Who are the key players? What are the program components and elements? Who are the targets of change? Who are the agents of change? Once I get a good handle on that, I connect that to some key questions. I collect evaluation data. What evaluation data would people believe as being evidence that these things are actually being implemented, and what would be key evidence that these things are successful? I combine that with appropriate components and elements in a systematically and very logical fashion. I look for ways for economizing procedures for collecting data. I get right to the chase. Don't do a whole lot of surveys that people have to spend time filling out."

Sometimes, however, interests conflict. Vince describes dealing with this kind of situation: "Usually, I'll find a few people that are really on board and very interested in evaluation. A lot of people just hate evaluators and think it's only a bunch of stupid surveys. My goal is to make it simple as possible. So my trick is to go in and sort of navigate the waters. Work with the people who want to work with me at first, and then convince everybody else that this is useful to them." Vince points out that, depending on how political the footfall of the program is, the evaluation may suffer.

Vince doesn't have a recipe to deal with all the interests that are involved in an evaluation: "All I know is that I juggle all the balls and keep them all in the air. And I look at all the different stakeholders. Who is the funding agent, and what is their interest in all this. I have to make sure their interest is represented. I try to give everybody the same exact information, so we're all moving forward together. That's the way I approach these things I try to change. I get everybody talking to each other in a real way and I don't get into the rumor way." Vince says that evaluators have a unique position because they can hold people accountable for their words: "I don't know if (the evaluator) is any more or less important than any other (interests). But without it, you don't do well. If we want to get lasting outcomes, we have to focus on behavior change. We have to focus on changing the systems that are controlling behavior."

Example 4: The funders role

Jeff Usher represents a very important interest in the evaluation process: the funder's. He is a program officer at the Kansas Health Foundation, a philanthropic organization that provides grants to improve the quality of health in Kansas. Here he talks about implementing evaluations and adjusting interests:

"We want to (see) change, we want to make a behavior change or community change. At the foundation, probably 10 to 20 percent of the overall budget will go to evaluation. The evaluation, of course, will be oftentimes both qualitative and quantitative in nature. We want to see the outcomes, whatever it might be; if it's teenage pregnancy, we'd like to see reductions of teenage pregnancies. So we like to see significant outcomes. We normally contract with people that have expertise in that particular area to conduct the evaluation."

Once the outcomes are in, Jeff says that it's time to decide what steps to take: "If it doesn't appear that the initiative is working, we clearly start losing interest in continuing the project in that manner. Oftentimes the evaluation portion of it is after the fact; the initiative is already over with, for the most part. If it comes back and it looks like the initiative was successful and the outcomes were going in the direction that we anticipated, then we may do another round of that initiative. For instance, the School Community Sexual Risk Reduction project, which is in its second phase now. Depending on the initiatives, if it appears that there is something else going on, we may still continue to work with those, or we may want to tweak that initiative a little bit to make it be a little more effective."

Jeff says that balancing interests among funders, community leaders and evaluators is not much of a problem for him: "We don't seem to have much of a problem when working with evaluators and as far as a private foundation, we don't hear a lot of problems." Jeff goes on to say that sometimes he experiences problems with some evaluators regarding the evaluation methods: "We may get some evaluations back that are measuring process rather than outcomes, and we clearly want to see outcomes. As far as balancing the grantee and the evaluator, normally, the evaluator is a grantee too, so in a sense there is even accountability that we want to see from the evaluator, and we would ask for that. Clearly there are certain things we want to see in that evaluation. For instance, we just did a memorandum agreement for an evaluation on a project that we are just going to be kicking off this next year. We want to see a change in the culture of campus, centered around binge drinking or heavy drinking. We outline those particular things in the memorandum when we start negotiating, so we expect to see that."

One of the funder's interests is to keep an eye on how the initiative conducts its business. Jeff says that in every memorandum agreement, they allow his foundation to perform site visits to have a third party come in, audit financial books, and see how the program is going: "Rarely do we follow up on those. The only time we would look at those is when we get a status report, and we ask for status reports from all of the grantees with what we want to see, and they haven't met those objectives. So, for almost all of our major initiatives, we'll have an evaluation component and have someone looking at it. We want to see behavior change, systems change, changes in culture centered around the health and well-being of communities, and individuals in communities."

The opening sections of this chapter introduce evaluation: why it matters, how it can be conducted, and whose interests it should address. This section and the next focus on selecting evaluators and developing an evaluation plan. These decisions are closely connected. Begin by clarifying the purpose with staff, community members, and people affected by the initiative, then select evaluators who can help refine the questions and design an appropriate approach.

Why pay attention to the selection of evaluators?

An evaluation should do more than declare that an initiative is doing well or needs improvement. It should examine the work from several perspectives, assess progress toward goals, identify strengths and limitations, and help people decide what to do next. A thoughtful plan needs evaluators with the skills, contextual understanding, and judgment to carry it out and interpret the findings responsibly.

Evaluation can support improvement, learning, and accountability. Useful evidence can help an organization build on strengths, address barriers, respond to changing community priorities, and inform advocacy or funding decisions. More information is not automatically better: the aim is credible information that answers meaningful questions and can guide action.

You may hire external evaluators, draw on staff and community expertise, or combine these approaches. You may also establish a planning team that helps select the evaluators. In each case, the people involved will shape the questions, methods, relationships, and interpretation of findings. Choose them for their ability to support a sound evaluation, rather than their likelihood of producing favorable results.

Organizations have different resources for evaluation. Options include staff-led work, community research teams, independent consultants, university partnerships, and combinations of these. Community members may be experienced evaluators, and professional evaluators may also be local residents. Consider what expertise already exists, what support is needed, and how contributors will be compensated.

A formal planning team is one option, not a requirement for every evaluation. A smaller group may be sufficient for a focused project. Match the scope and process to the available time and resources while maintaining credible methods and respect for participants. A manageable evaluation that answers a useful question can provide a foundation for more extensive work later.

All evaluators bring experiences, interests, assumptions, and relationships that can influence their work. Discuss these openly, whether you are considering consultants, university researchers, staff, students, or community members. Look for people who understand the agreed purpose, select methods that fit the questions, and can report findings honestly, including results the organization or funder may not welcome. Alignment on purpose should support independent judgment rather than require agreement with a preferred conclusion.

When should you choose evaluators?

Involve evaluation expertise early enough to inform planning and collect any baseline information you will need. Waiting until a project ends can limit the questions you can answer. For a new initiative, an initial planning group may help identify priorities and select evaluators. For an existing initiative, begin with its current decisions and available information, then plan opportunities to learn throughout the work.

For example, a new adult literacy program might include evaluation resources in its initial budget and arrange support from a supervised graduate researcher. The researcher could work with learners and staff to define useful questions, observe classes with appropriate permission, review progress information, and conduct interviews during the year. This approach could help the program understand both implementation and change over time. The budget should reflect the work required rather than a fixed percentage assumed to suit every program.

Before finalizing the selection, learn enough about the community, the initiative, and the intended use of findings to identify the expertise required. You do not need to resolve every question first. Evaluators can help assess context and refine the plan, provided community members and staff have meaningful roles in those decisions.

Context of the evaluation

The evaluation's context includes community history, geography, culture, economic conditions, politics, institutional relationships, and available resources. It also includes the initiative's stage of development and people's previous experiences with services or research. Understanding these conditions helps the team choose appropriate methods and avoid judging results without considering how they arose.

An example: Suppose a rural adult literacy program enrolls fewer learners than expected during its first year. Enrollment figures alone might suggest that the program is not meeting community needs. Conversations with residents, however, could reveal that previous programs closed quickly, making people uncertain about whether this one would last. Transportation, scheduling, and outreach may also be barriers. An evaluation that examines these conditions can help the team decide whether to strengthen relationships, adapt delivery, or revise expectations. Later evaluation can test whether those changes improve access and learning.

How should you choose among staff, community, and external evaluators?

Evaluation involves systematically gathering and interpreting information to support judgments and decisions. Relevant expertise can come from formal training, practical evaluation experience, community research, and knowledge of the initiative. The question is which combination of skills and perspectives your evaluation requires.

An external specialist may be useful for a complex design or independent review. Staff and community evaluators may contribute essential knowledge of implementation, relationships, and local priorities. A combined team can bring these strengths together, with clear responsibilities and support for collaboration.

Consider the following factors before deciding who will conduct the evaluation:

Money

Start with a realistic budget for the evaluation's purpose and scope. Include staff time, community participation, training, data collection, analysis, accessibility, and reporting. Ask funders whether evaluation support is available. If resources are limited, consider a smaller evaluation or targeted technical assistance rather than assuming that unpaid labor can cover all the work.

When specialized support is needed, possible sources include local colleges or universities, evaluation consultants, partner organizations, public agencies, and dedicated grants. Students may contribute through supervised coursework, internships, or research partnerships. Agree on the scope, supervision, timeline, data access, and community benefit before starting. Academic requirements should be compatible with the evaluation's purpose, and any additional use of information should be addressed explicitly.

For example, a graduate evaluation course could partner with community organizations on projects that fit a semester. Students and organizations might jointly define questions, develop methods, and produce practical reports, with faculty oversight. A shared learning event could highlight lessons and tools without disclosing sensitive information. Such arrangements work best when expectations are realistic and organizations have a plan for work that continues after the course ends.

Cost is one consideration. Also assess whether the proposed team has the capacity, availability, and working approach needed for the evaluation.

The complexity of the evaluation

Some evaluations address focused descriptive questions: How many residents participated? Did learners improve their reading skills? How did participants experience a recovery-support program? Other questions require more demanding designs: Did the initiative cause the observed change? Which aspects of delivery contributed? How did transportation, housing, or employment conditions influence participation and outcomes? Even a simple-looking question can require careful measurement and attention to missing information. Match the team's methods expertise to the claims you intend to make, and combine that expertise with community knowledge that can help explain the findings.

The type of information needed and how it will be analyzed

Quantitative information may require more than counts or averages. Depending on the questions, the team may need expertise in sampling, comparison designs, measurement, or statistical analysis. Statistical significance does not establish that the program caused a change or indicate the probability that a particular explanation is true. Look for evaluators who can explain the size and practical importance of findings, their uncertainty, and the assumptions behind the analysis. Qualitative information also requires systematic analysis rather than selecting a few memorable stories.

For example, a study of adults returning to education might initially focus on employment changes or access to classes. Systematic review of interviews could reveal that earlier caregiving responsibilities also shaped participants' educational opportunities. Survey information might help describe how common that experience was among respondents, while follow-up conversations could clarify its meaning. Combining methods can uncover overlooked patterns, but software or a statistical association cannot by itself establish what caused people's decisions.

How the evaluation will be used

Will the findings guide program improvement, inform community decisions, support accountability, or help determine future funding? The intended uses affect the questions, level of detail, timeline, and expertise needed. Consider whether participants would feel more comfortable speaking with a community member, a staff evaluator, or someone independent of the initiative. No role guarantees trust or neutrality. Choose arrangements that fit the circumstances and manage possible conflicts of interest.

How the evaluation will establish credibility

Credibility depends on the quality and transparency of the work, not simply on whether evaluators are external or community-based. Community participation can improve relevance and accountability, while independent review can help address particular conflicts or technical questions. Explain who will make decisions, how affected people will contribute, and how findings will be reported. These arrangements should reflect the initiative's mission and principles while protecting the integrity of the evaluation.

What should you look for when choosing evaluators?

Some qualities matter for every evaluator, while others depend on the responsibilities of the team.

Evaluations can use many combinations of expertise. A community researcher may have extensive professional experience, an external evaluator may provide limited technical assistance, and staff may carry out parts of the work with training and supervision. Paid status, credentials, and community membership describe different aspects of a person's role; none alone determines their ability to contribute.

An external evaluator might help a community team design the evaluation, conduct a specialized analysis, or work alongside partners throughout the process. A local team might develop the questions and collect information while a consultant supports interpretation and reporting. Whatever the arrangement, specify responsibilities, decision-making authority, communication, data access, and how knowledge will remain available to the organization and community.

For example, consider an evaluation of community HIV services. A planning group could include people living with HIV, service providers, community advocates, and an evaluator. The group would agree on its purpose and ensure that people using or seeking services have meaningful influence over the questions.

Together, members might decide whether a survey, interviews, service records, or a combination would best address their priorities. They could review questions, choose accessible participation methods, discuss privacy concerns, and agree on how information will be analyzed and shared. The evaluator's role would be to support informed choices and explain tradeoffs.

If the evaluator then conducts the technical analysis, the group can still help interpret findings, identify missing context, and develop recommendations. Participation in interpretation should not require access to identifying information or give any partner authority to suppress unfavorable results.

Characteristics to look for in all evaluators

Commitment to the agreed purpose and honest inquiry

Evaluators may have professional, academic, organizational, or personal interests in the work. A study might contribute to a dissertation, publication, funding decision, or advocacy effort. These interests are not necessarily incompatible with a useful evaluation, but they should be disclosed and discussed. Establish written agreements about scope, methods, deliverables, data use, authorship, and reporting. The agreement should protect the evaluation's purpose and the ability to report credible findings without pressure to reach a predetermined conclusion.

Apply the same expectations to staff, community members, funders, and external evaluators. Ask how they would handle evidence that challenges their views or the initiative's preferred approach. Select people who can listen, consider alternative explanations, and work through disagreement. Different perspectives can strengthen an evaluation when they are addressed openly rather than treated as reasons to exclude someone.

Ability to communicate with a broad range of people

Evaluators may work with people of different ages, languages, identities, beliefs, and experiences. They need to explain the work clearly, listen carefully, and create respectful opportunities for participation. Trust develops through reliable conduct, transparency, and responsiveness; it cannot be assumed from a title or shared background.

Have candidates worked with communities and organizations relevant to your initiative? Can they explain methods without unnecessary jargon and adapt their communication to different audiences? Do they ask about language and accessibility needs? Does the team include relevant language skills or a plan for qualified interpretation and translation?

Explore these questions through interviews, examples of previous work, and conversations with former partners. Ask candidates to explain a method or finding in plain language. The team should be able both to gather useful information respectfully and to present results in forms that community members and decision-makers can understand and use.

Cultural responsiveness and willingness to learn

Evaluators should learn about the community's history, relationships, and priorities without assuming that everyone sharing an identity has the same experiences. Ask how they examine their own assumptions, respond to unfamiliar perspectives, and adapt methods with community input. Knowledge of a particular setting is valuable, but curiosity, humility, and accountability remain important even for evaluators who belong to that community.

Respect for every participant and partner

Look for evaluators who treat people respectfully regardless of their role, education, income, or position in the organization. They should provide meaningful choices about participation, welcome questions and criticism, and respond appropriately to concerns. Their conduct affects both the quality of the evaluation and the organization's relationships with the community.

Commitment to privacy and clear confidentiality protections

Evaluators should explain how they will limit collection of identifying information, protect records, control access, and reduce the risk of identification in reports. Confidentiality concerns how information is protected and shared; anonymity means identities are not collected or linked to responses. Do not promise absolute protection that the evaluation cannot provide. Explain relevant limits in advance, and consider how disclosure could affect participants, especially when the work involves sensitive experiences such as domestic violence.

Informed consent is a communication process, not simply a signed form. Explain what participation involves, how information will be used, foreseeable risks, confidentiality protections and limits, and whom to contact with questions. Provide accessible opportunities to decide freely whether to participate. Determine the applicable review and consent requirements before collecting information; the appropriate documentation depends on the activity and its setting.

Commitment to a useful and credible evaluation

Evaluators should follow through on responsibilities, document important decisions, and communicate emerging issues promptly. The organization and community partners also need to make time for interpretation and action. Agree early on who will use the findings, what decisions they will inform, and how recommendations will be considered. Useful evaluation may confirm strengths, identify changes, or reveal that more information is needed.

Ask evaluators how their work can support future learning. This might include practical data-collection tools, staff and community training, clear documentation, or a schedule for reviewing selected indicators. Ongoing evaluation should fit the initiative's needs and capacity rather than require repeating a large study at every stage.

Community or staff planning and evaluation teams

A planning team helps define the evaluation's purpose, questions, scope, and intended uses. It may also help select evaluators. Its responsibilities can include identifying information sources, choosing methods with technical support, allocating tasks, and agreeing on how findings will inform decisions.

An evaluation team carries out the work, including gathering information, analyzing evidence, interpreting findings, and reporting results. It may include the same people as the planning team or bring in additional expertise. Members may be paid staff, compensated community researchers, volunteers, or external partners. Ensure that the team has appropriate skills and support for the methods it will use.

Assemble a team that can complete the work while including perspectives essential to the evaluation. Organizational position alone should not determine whose contributions carry weight. Team size will depend on the scope, workload, and participation arrangements. A smaller working group can consult more broadly, but affected people should have genuine influence over decisions. Consider the following qualities and arrangements:

Representation and meaningful participation

Depending on the initiative, a planning or evaluation team may include:

• Organizational representatives with different responsibilities, such as program staff, administrators, and governing board members
• People who participate in or are affected by the initiative, including those who have encountered barriers to access
• Community partners with relevant knowledge, such as residents, local businesses, service organizations, schools, and public agencies
• Young people when the initiative affects them, with appropriate support and meaningful roles
• People from relevant language communities and groups underrepresented in previous planning or evaluation efforts
• People with research and evaluation skills, whether developed through community work, employment, formal education, or other experience

Understanding of the evaluation's purpose

Team members should understand what the evaluation is intended to learn and how its questions, methods, and uses connect. Provide orientation and opportunities to revisit the purpose, especially when the team includes people new to evaluation or to the initiative.

Willingness to listen and learn

Every member should recognize both what they contribute and what they need to learn. For example, a school leader can learn from a young person who left school before graduating about barriers that school records do not capture. That young person may also want support with particular evaluation methods. Treat these as complementary contributions, and make room for people to question assumptions regardless of their formal position.

Ability to collaborate, with appropriate support

Familiarity with meetings, technical language, and decision-making processes varies among individuals and should not be inferred from income, occupation, education, or background. Agree on accessible meeting practices, explain unfamiliar terms, and offer several ways to contribute. Provide materials in advance, interpretation or other access support where needed, and fair opportunities to speak. Peer support or shared training can help, but it should respect people's preferences and avoid assigning some members a permanently subordinate role.

External evaluators and specialist teams

When considering a consultant, firm, or university team, assess the people who will actually do the work, including their availability, supervision, and responsibilities. A strong proposal should explain how their expertise fits the questions and how they will collaborate with the organization and community. Review both technical competence and the team's approach to relationships, learning, and accountability.

Knowledge of appropriate evaluation methods

Can candidates explain when quantitative, qualitative, or combined methods would be useful? Do they understand sampling, measurement, analysis, and the limits of different designs? Ask how they would adapt their approach to your questions and resources rather than assuming that their usual method is the best fit.

If you are seeking a university partner, consider departments or centers whose methodological skills and subject knowledge fit the initiative. Relevant contacts may include:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Relevant experience and quality of previous work

Ask about evaluations with similar questions, communities, or constraints. What methods did candidates use, what difficulties arose, and how were findings used? Request appropriate work samples and references, while recognizing that newer evaluators may demonstrate readiness through supervised work and relevant experience.

As when you hire someone for a position, use more than the interview or written proposal to assess fit. Speak with previous clients and community partners about reliability, responsiveness, collaboration, and how the evaluator handled disagreement or unexpected findings.

Review reports or other products that candidates are permitted to share. Are the methods and limitations clear? Are conclusions supported by the evidence? Can intended audiences understand the findings and use them? Ask how community members contributed to interpretation and how the evaluator communicated results to different audiences.

Experience working in partnership with communities

Ask for concrete examples of how candidates shared decisions, responded to community priorities, and addressed differences in power or access. How did they handle a suggestion that changed the design? How did they manage disagreement among partners? Look for a willingness to collaborate while maintaining sound methods and honest reporting.

Ability to listen carefully and check understanding

Evaluators should take others' knowledge seriously and explain where their own expertise can help. Careful listening includes asking follow-up questions, summarizing what they have heard, and checking interpretations rather than assuming they understand someone's meaning. This skill helps the team identify useful questions, gather information respectfully, and make sense of findings in context.

In Summary

The people selected to plan and conduct an evaluation shape its quality, usefulness, and relationships with the community. Choose a combination of skills and perspectives that fits the purpose, methods, resources, and setting. Assess relevant experience, communication, respect for participants, management of conflicts, and willingness to work collaboratively. Clear agreements and meaningful community involvement can help the team produce credible findings and support informed action.

Why should you evaluate?

___To see results of work

___To gain public opinion

___To make the proper adjustments if necessary

What should you consider when you select evaluators?

___Select evaluators carefully for reliability 

Possible evaluators:

___Volunteers

___Community being serviced

___Chosen evaluators by a team

Tool: Interview Questions for evaluation consultant selection

From EvalTalk, the discussion group of the American Evaluation Association (AEA).

• What are the main reasons why you think a program should conduct an evaluation?

• What are the main standards and guiding principles for conducting evaluations?

• How would you determine if a program is ready to undergo an evaluation?

• How do you estimate the resources needed to conduct an evaluation? That is, how do you compute the costs of an evaluation, both in money and time (yourself and others)?

• If you had results from an evaluation that were based on quantitative information that conflicted with results from qualitative information, how would you resolve the conflict?

• If you were to conduct a mailed-out survey to program participants, how would you achieve a representative sample and ensure that your results were generalizable to all participants?

• Frequently evaluations involve various and independent branches and agencies that all have different expectations for the evaluation. If you had a conflict among participants in a program evaluation that could threaten the evaluation, how would you resolve the conflict? What steps would you take to proactively avoid conflict?

• Read any interesting articles on evaluation lately? In what journals? What was interesting about it? What other journals do you see regularly?

• Under what circumstances, if any, would you blow the whistle on a client?

Why should you have an evaluation plan?

Evaluation helps you understand how an initiative is being implemented, what is changing, what is working well, what is not working as intended, and what should happen next.

Evaluation should not be treated simply as a final judgment about whether a program “succeeded” or “failed.”

A useful evaluation can help an organization:

• Understand how the initiative is being implemented
• Identify strengths and areas for improvement
• Understand intended and unintended effects
• Make decisions during implementation
• Assess progress toward goals
• Learn from community members and participants
• Meet reporting or accountability requirements
• Decide whether an initiative should be continued, changed, expanded, or ended

There are many evaluation frameworks, tools, and methods available.

Do not select a method simply because it appears in an evaluation handbook or because another program used it.

Begin by asking what you actually need to learn and what decisions the information will help people make.

Collecting large amounts of information is not useful if the information does not answer important questions or inform action.

An evaluation plan can help you:

• Clarify why the evaluation is being conducted
• Identify who should participate in evaluation planning
• Determine what questions are most important
• Identify what evidence is needed
• Select appropriate evaluation methods
• Establish responsibilities for data collection, analysis, interpretation, and reporting
• Protect privacy and address ethical concerns
• Develop a realistic timeline and budget
• Decide how findings will be communicated and used
• Improve the initiative throughout its life cycle

When should you develop an evaluation plan?

Begin evaluation planning as early as practical.

Ideally, evaluation is considered while the initiative itself is being designed.

Early planning can help establish:

• Clear goals and intended outcomes
• Baseline information when needed
• Appropriate measures
• Data-collection systems
• Responsibilities
• Evaluation resources
• Plans for using findings

If an initiative is already underway, it is not too late to begin evaluation.

Determine what information is still available, what can reasonably be collected going forward, and what evaluation questions remain answerable.

Evaluation should support learning throughout implementation rather than occur only after the initiative is finished.

Who should be involved in evaluation planning?

Evaluation can affect many people and organizations.

Identify the people who:

• Are directly affected by the initiative
• Participate in the initiative
• Implement or manage it
• Provide funding or oversight
• Make decisions using the findings
• Have relevant community, professional, technical, or lived knowledge

Different people may use evaluation findings for different purposes.

Involving them early can help make the evaluation more relevant, useful, fair, and responsive to the context in which the initiative operates.

People directly affected by the initiative

People whose lives, services, neighborhoods, organizations, or communities are affected by an initiative may have important knowledge about:

• Whether the initiative addresses a real need
• Whether services are accessible
• How implementation is experienced
• What outcomes matter
• What unintended effects have occurred
• What changes would make the initiative more useful

Do not refer to people primarily as “targets of change.”

Whenever practical, involve people directly affected in shaping evaluation questions, interpreting findings, and deciding what actions should follow.

Program staff, volunteers, and organizational leaders

People involved in implementing the initiative can provide information about:

• Activities
• Resources
• Participation
• Implementation challenges
• Changes made during the program
• Organizational capacity

Their involvement is important, but implementation staff should not be the only people interpreting whether the initiative is effective.

Funders and oversight organizations

Funders may have legitimate reporting, accountability, or learning requirements.

Determine what information is required by:

• Grant agreements
• Contracts
• Government reporting requirements
• Organizational policies

Funding requirements should be considered alongside the information needs of participants, community members, program staff, and other relevant groups.

Do not design an evaluation solely around what the funder wants to know if doing so would leave important community or program questions unanswered.

Evaluators, researchers, and people with technical expertise

An initiative may involve:

• Internal evaluation staff
• Independent evaluators
• University researchers
• Data analysts
• Community-based researchers
• Subject-matter experts

Involve them early enough to help determine what questions can reasonably be answered and which methods are appropriate.

Researchers and evaluators should not automatically control the evaluation simply because they have technical expertise.

Evaluation planning is stronger when technical knowledge is combined with community knowledge, lived experience, program expertise, and the needs of intended users.

What do people need to know from the evaluation?

Different people may need different information.

Before selecting measures or methods, ask:

• Who will use the findings?
• What decisions do they need to make?
• What information would help them make those decisions?
• When will they need the information?
• What information would be useful to people directly affected by the initiative?
• How should findings be communicated so that people can understand and use them?

Evaluation should support decisions

Findings may inform decisions about:

• Program improvement
• Staffing
• Funding
• Program continuation
• Expansion or replication
• Modification of services
• Policy
• Community priorities
• Future evaluation or research

Evaluation should produce honest and useful information.

Do not design measures, analyses, or reporting practices to exaggerate success or minimize unfavorable results.

Unexpected, mixed, or negative findings can provide important information for improvement.

How much should evaluation cost?

There is no single percentage of a program budget that is appropriate for every evaluation.

Evaluation resources depend on factors such as:

• The questions being asked
• Program size and complexity
• Available data
• Required methods
• Evaluation duration
• Reporting requirements
• Staff capacity
• Participant burden
• Available funding

Develop a realistic evaluation budget based on the work required rather than applying a fixed percentage automatically.

Ask:

• What information is essential?
• What information would be useful but optional?
• What information is already available?
• What data can realistically be collected?
• What resources are required to analyze and use the information responsibly?

How do you develop an evaluation plan?

Evaluation planning is often iterative rather than strictly linear.

A practical process includes:

1. Assess the context.
2. Describe the initiative and how it is expected to work.
3. Identify who should participate in evaluation planning.
4. Clarify the purpose of the evaluation and intended uses.
5. Develop and prioritize evaluation questions.
6. Select an evaluation design and appropriate methods.
7. Gather credible evidence.
8. Analyze and interpret findings.
9. Communicate findings in useful forms.
10. Act on what is learned.

Describe the initiative and clarify its goals

Before evaluating an initiative, make sure there is a shared understanding of:

• The problem or need being addressed
• Resources available
• Activities being implemented
• People or communities the initiative intends to reach
• Expected short-term outcomes
• Expected intermediate outcomes
• Expected long-term outcomes
• Important contextual factors

A logic model, theory of change, program roadmap, or another visual representation may help clarify how activities are expected to contribute to outcomes.

Do not assume that every observed change can automatically be attributed to the initiative.

Develop evaluation questions

Evaluation questions should reflect what people need to learn and what decisions the findings will inform.

Common categories include:

Context and need

Possible questions include:

• What community need is the initiative intended to address?
• Has that need changed?
• What contextual factors may affect implementation or outcomes?
• What strengths and resources already exist in the community?

Planning and implementation

Possible questions include:

• Was the initiative implemented as intended?
• What changes were made during implementation and why?
• Who participated?
• Who encountered barriers to participation?
• Were activities accessible and appropriate for the people they were intended to reach?
• How did participants experience the initiative?
• What implementation challenges occurred?

Progress toward objectives

Possible questions include:

• Were planned activities completed?
• Were intended participants reached?
• Were services delivered at the expected level?
• Were short-term objectives achieved?

Measures of participation and activity are important, but they should not automatically be treated as evidence that meaningful outcomes occurred.

Outcomes for participants

Possible questions include:

• What changed for participants?
• Did knowledge, skills, access, behavior, conditions, or well-being change?
• Were there unintended positive or negative effects?
• Did outcomes differ among groups or settings?
• How did participants describe the value or limitations of the initiative?

Community or systems outcomes

Possible questions include:

• Did relevant community conditions change?
• Were organizational practices changed?
• Did policies or systems change?
• Did access to services change?
• Were disparities reduced, maintained, or increased?
• Were there unintended community effects?

Costs and value

Depending on the evaluation purpose, you may also ask:

• What resources were used?
• What did implementation cost?
• How do costs compare with outcomes?
• Could similar outcomes be achieved another way?

Select evaluation methods

Choose methods according to the evaluation questions rather than starting with a preferred method and trying to make every question fit it.

An evaluation may use quantitative methods, qualitative methods, mixed methods, participatory approaches, or a combination.

Monitoring and performance information

Monitoring can help track implementation while an initiative is underway.

Measures may include:

• Activities completed
• Participation
• Services delivered
• Resources used
• Milestones
• Implementation changes
• Short-term outcomes

Monitoring and evaluation are related but are not necessarily the same activity.

Surveys

Surveys can gather information about:

• Experiences
• Knowledge
• Attitudes
• Behaviors
• Satisfaction
• Perceived outcomes
• Barriers

Design survey questions carefully and consider who may be excluded by the method used to administer them.

Interviews and focus groups

Interviews and group discussions can help explain:

• Why something happened
• How people experienced the initiative
• What contextual factors mattered
• Why implementation differed across settings
• What improvements participants recommend

Include people because their knowledge or experience is relevant rather than referring only to “key” or high-status individuals.

Administrative and program data

Existing information may include:

• Participation records
• Service records
• Financial information
• Health records where lawful and appropriate
• Education data
• Public records
• Other administrative information

Evaluate the quality, completeness, relevance, and privacy implications of existing data before using them.

Community-level indicators

Community indicators may help track changes in conditions over time.

Select indicators that are:

• Relevant to the evaluation question
• Clearly defined
• Measured consistently
• Available at an appropriate geographic and population level
• Interpreted in context

A change in a community indicator does not automatically prove that one initiative caused the change.

Comparison and causal designs

Some evaluations need to estimate whether observed outcomes can reasonably be attributed to the initiative.

Depending on the question and context, designs may include:

• Experimental designs
• Quasi-experimental designs
• Comparison groups
• Interrupted time-series approaches
• Other appropriate causal or contribution approaches

Not every evaluation requires an experimental design.

Select the strongest feasible design that is appropriate to the question, ethical considerations, available data, timeline, and resources.

Gather credible evidence ethically

Evaluation planning should address:

• What data will be collected
• From whom
• Why those people or sources were selected
• How information will be collected
• How privacy and confidentiality will be protected
• Who will have access to identifiable information
• How data will be stored
• How long information will be retained
• How findings will be shared
• Whether additional ethical, legal, or institutional review requirements apply

Collect only the information necessary to answer the evaluation questions.

Analyze and interpret findings carefully

Interpretation should consider:

• The evaluation questions
• Program context
• Data quality
• Alternative explanations
• Differences across groups or settings
• Intended and unintended outcomes
• Limitations of the evaluation design

Involve relevant people in interpreting findings when practical.

Different perspectives may reveal explanations or implications that would otherwise be missed.

Do not interpret results only through the perspective of the funder, evaluation team, or program leadership.

Develop an evaluation timeline

Evaluation activities should be aligned with the life cycle of the initiative.

The timeline may identify:

• Evaluation planning
• Baseline data collection
• Ongoing monitoring
• Interim analysis
• Feedback sessions
• Final data collection
• Analysis and interpretation
• Reporting
• Follow-up measurement
• Decisions based on findings

Evaluation may continue after the formal project period

Do not assume that evaluation must end shortly after program activities stop or when a final grant report is due.

Some outcomes may take longer to appear.

Follow-up evaluation may be useful when you need to understand:

• Whether outcomes were sustained
• Whether changes persisted after services ended
• Whether unintended effects emerged later
• Whether systems or policy changes were maintained

The evaluation timeline should match the questions being asked.

Provide feedback throughout the evaluation

Do not wait until the final report to share everything that has been learned.

Interim findings may help:

• Improve implementation
• Identify emerging problems
• Correct data-quality issues
• Adapt services
• Inform upcoming decisions

Make clear when findings are preliminary.

What products should come from an evaluation?

The most useful product is not always a long final report.

Different audiences may need different formats.

Evaluation products may include:

• Full reports
• Executive summaries
• Briefs
• Presentations
• Community meetings
• Data dashboards
• Infographics
• Accessible digital summaries
• Technical appendices
• Internal learning sessions
• Recommendations or action plans

Choose formats according to how people will actually use the information.

Evaluation products should explain:

• Why the evaluation was conducted
• What questions were examined
• Who participated in the evaluation
• What methods were used
• What was found
• What limitations exist
• What differences appeared across groups or settings
• What unintended findings emerged
• What conclusions are supported by the evidence
• What actions should be considered next

Report important unfavorable or unexpected findings as well as positive results.

Use the findings

Evaluation is most valuable when findings lead to learning and action.

After reviewing the evidence, decision-makers may:

• Continue the initiative
• Modify it
• Expand it
• Reduce or discontinue ineffective components
• Improve accessibility
• Address unintended harms
• Change resource allocation
• Revise the theory of change
• Conduct additional evaluation or research

Decide during evaluation planning how findings will reach the people able to act on them.

What standards should guide an evaluation?

Evaluation standards and professional guidance evolve over time.

Current public-health evaluation guidance emphasizes several broad principles for high-quality evaluation.

Relevance and utility

Evaluation should answer questions that matter and provide information people can use.

Rigor

Methods should be appropriate to the questions, and conclusions should be supported by credible evidence.

Limitations should be reported clearly.

Independence and objectivity

Evaluation processes should reduce inappropriate influence, conflicts of interest, and avoidable bias.

Evaluators should reflect on how their own assumptions and roles may affect the work.

Transparency

Document:

• The purpose of the evaluation
• Evaluation questions
• Methods
• Important decisions made during the evaluation
• Changes to the original plan
• Limitations
• How conclusions were reached

Ethics

Protect the dignity, rights, safety, privacy, and confidentiality of people who contribute information.

Consider who bears the burden of data collection, who benefits from the evaluation, and whether the evaluation could create unintended harm.

Build collaboration, fairness, and use throughout the evaluation

High-quality evaluation is not simply a technical exercise.

Throughout the process:

• Engage relevant people collaboratively
• Consider equity and fairness in participation, methods, interpretation, and use
• Learn from findings as they emerge
• Use insights to improve the initiative and future decisions

Ask who has been included in evaluation decisions and whose perspective may be missing.

Pay particular attention to whether people most affected by the initiative have meaningful opportunities to contribute.

In Summary

An evaluation plan helps an initiative determine what it needs to learn, how it will gather credible evidence, who should participate, when evaluation activities should occur, and how findings will be used.

Begin planning evaluation as early as practical, but do not abandon evaluation simply because an initiative is already underway.

Start with the decisions and questions that matter rather than with a preferred method or a large list of available measures.

Involve people directly affected, program staff, community partners, funders, decision-makers, evaluators, and other relevant participants according to the evaluation's context.

Do not refer to people as “targets of change” or assume that technical experts are the only people who hold important evaluation knowledge.

Develop a realistic budget based on the evaluation's actual questions, design, data needs, timeline, and available resources rather than assuming that every initiative should spend the same percentage of its budget on evaluation.

Clarify how the initiative is expected to work and develop evaluation questions addressing implementation, outcomes, community or systems change, unintended effects, and other priorities.

Select methods that fit the questions.

Surveys, interviews, monitoring data, administrative information, community indicators, qualitative methods, and experimental or quasi-experimental designs can all be useful in the appropriate circumstances.

Do not assume that every evaluation must prove that the initiative caused an observed outcome.

Collect and use information ethically, protect privacy, explain limitations, and avoid designing an evaluation to produce a preferred result.

Share learning throughout implementation when doing so can improve the initiative.

Use reports, briefs, dashboards, presentations, community discussions, or other formats appropriate to the people who need the findings.

Evaluation may continue after an initiative formally ends when important outcomes require longer-term follow-up.

Finally, evaluation should not end with the production of a report.

Its value comes from using credible evidence and multiple forms of knowledge to understand an initiative, improve decisions, address unintended effects, and determine what should happen next.

You understand that you should have an evaluation plan because:

___It guides you through each step of the process of evaluation

___It helps you decide what sort of information you and your stakeholders really need

___It keeps you from wasting time gathering information that isn't needed

___It helps you identify the best possible methods and strategies for getting the needed information

___It helps you come up with a reasonable and realistic timeline for evaluation

___Most importantly, it will help you improve your initiative!

You understand who your project or initiative's stakeholders are:

___Community groups

___Grantmakers and funders

___University-based researchers

You have taken into consideration:

___What stakeholders want to know about the evaluation

___What decisions stakeholders need to make

___How stakeholders would use the data to inform decisions

When considering how to balance costs and benefits, you have asked yourself the following questions:

___What do you need to know?

___What is required by the community?

___What is required by funding?

You understand these four main steps to developing an evaluation plan:

___Clarify program objectives and goals

___Develop evaluation questions

___Develop evaluation methods

___Set up a timeline for evaluation activities

Clarify the objectives and goals of your initiative:

___Make a table of program components and elements

Develop evaluation questions:

___You understand the four main categories of evaluation questions:

• Planning and implementation issues
• Assessing attainment of objectives
• Impact on participants
• Impact on the community

___You have considered the best possible methods to answer those questions

Develop evaluation methods to best address those questions:

___You understand how to use the monitoring and feedback system's three main elements (process measures, outcome measures, and observational system)

___You understand how to use member surveys about the initiative (member survey of goals, member survey of process, and member survey of outcomes)

___You understand how to use the goal attainment report

___You understand how to use behavioral surveys

___You understand how to use interviews with key participants

___You know how to use community-level indicators of impact

Setting up a timeline for evaluation activities:

___You understand that you should begin right now, or at least at the beginning of the initiative

___You've outlined questions for each stage of development of the initiative

___You've completed a table listing: key evaluation questions, type of evaluation measures to be used to answer them, type of data collection, and experimental design .

___You've determined when you feel it's appropriate to provide feedback and reports

___You will provide feedback and reports at the end of the evaluation

___You will also provide periodic feedback and reports throughout the duration of the project or initiative

___You've decided when the evaluation will end

___You've mapped out a proposed evaluation timeline

Expected evaluation products:

___You will make a report that you can share with everyone involved which includes effects expected by shareholders, differences in the behaviors of key individuals, and differences in conditions in the community

___You've decided whether to also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team

Evaluation standards:

___You've decided what standards you will use to ensure an accurate and useful evaluation

Example 1: Table of Program Components and Elements

Here are a few of the items from the table of program components and elements for a fictional teen pregnancy prevention program:

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery
Risk factor: Unsupervised activities by teens	Provide alternative activities	After-school activities (sports programs, hobby / craft classes, movies, etc.) Weekend "lock-ins", non-alcoholic teen dance club Daytime activities during school holidays	Targets:   Youth   Agents: Youth Teachers Parents	Group sessions Workshops Meetings
Risk factor: Lack of access to contraceptives	Increased access to health services	Extend hours at school health clinics Increase condom distribution by county health department Brown bag program at area pharmacies (teens wishing to buy contraceptives are able to call ahead and pick them up in a plain brown bag)	Targets:   Youth Agents: Health providers	Walk-in clinics Health Department Businesses
Protective factor: Involvement in the faith community	Encourage youth involvement in the faith community	Increase church-sponsored youth activities Institute church-based sexuality education Church-sponsored parent-child communication sessions	Targets: Youth Parents Agents: Youth Parents Religious leaders	Discussions Meetings Educational sessions Workshops

Example 2: Evaluation Questions and Methods

Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (Y.E.S.), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally, and interact with positive role models:

		Type of Data Collection*					Experimental Design**
Key Evaluation Question(s)	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- and Post-test Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1. Who participates? Is there diversity among the participants?	1. Basic demographic data (e.g., age, gender, disability, race, economic status, etc.) on participants.	X	X				X
2. Why do participants enter and leave the program?	2. Information on participants' reasons for entering and leaving the program.	X	X				X
3. Are there a variety of service and alternative activities generated?	3. Frequency of different types of service provided and alternative activities generated.			X	X	X	X
Assessing Attainment of Objectives	Outcome Measures
1. How many people participate?	1. Number of people participating.			X	X	X	X		X
2. How many hours are participants involved in service and activities?	2. Number of hours of participation by type of activity.			X	X	X	X		X
3. How many people are trained in cultural awareness and sensitivity?	3. Number of people trained in cultural awareness and sensitivity.			X	X	X	X		X
Impact on Participants
1. How do behavior and attitudes change as a result of participation in the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X		X	X
2. Does participation affect the incidence of problems?	2. Incidence of problems in living (e.g., substance use, gang involvement).	X	X			X		X
3. Are participants satisfied with the experience?	3. Satisfaction ratings from participants.	X					X
Impact on Community
1.  What resulted from the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X			X
2. Do the benefits of the program outweigh the costs?	2. Cost benefit data.		X	X		X	X
3. Are community members satisfied with the participants and the service they provided?	3. Satisfaction ratings from beneficiaries and community members.	X					X

* Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

• Survey/Scale: i.e., member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys
• Structured Interview: i.e., interview with key participants
• Staff Report/Log: i.e. monitoring system, goal attainment report
• Direct Observation: This refers to the direct observation of behavior by researchers .
• Archival Records: i.e., community-level indicators of impact

**Here are explanations of these terms:

• Case Study: an in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment--good for answering "how" or "why" questions
• Pre-/Posttest Control Group: monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made
• Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative

Example 3: Timeline for Evaluation Activities

This is the timeline for an evaluation being done by Kansas City's Promise on their initiative to promote community and personal caring for youth:

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system	[ - - - ]
Train participants / staff in data entry	[ - - - ]
Maintain system (data collection and reporting)	[- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - ]
Critical events
Identify key people	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Conduct interviews	[ - - ]	[ - - ]	[ - - ]	[ - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Community-Level Indicators
Identify local indicators	[ - - - - ]
Identify comparison site	[ - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Other Indicators (e.g., behavioral surveys, data observations)
Identify	[ - - - - - -	- - - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - -  - - - - -	- - - - - - - - - -]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]

 

Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts

Download PPT Slides >>

 

Table of Contents

General Evaluation Resources
Steps in Community Evaluation (and Related Resources)

1. Design Stakeholder Involvement
2. Identify Resources for Monitoring and Summative Evaluation
3. Describe the Intervention’s Framework, Logic Model, or Theory of Change
4. Focus the Monitoring and Summative Evaluation  Plan
5. Plan for credible methods
6. Synthesize and Generalize
7. Systems and Evaluation

 

General Evaluation Resources

• Institute of Medicine (IOM):
  • ​Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010)
  • Measuring Progress in Obesity Prevention: Workshop Report (2012)
  • Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012)
  • Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013)
• U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
• Community Tool Box (CTB):
  • Toolkit: Evaluating the Initiative
  • A Framework for Program Evaluation: A Gateway to Tools
  • Evaluating Comprehensive Community Initiatives
  • Toubleshooting Guide: We don’t know how to evaluate our program or initiative

Steps in Community Evaluation (and related resources)

 

 

Back to Top

 

1. Design stakeholder involvement
   
   1. Identify stakeholders
   2. Consider the extent of stakeholder involvement
   3. Assess desired outcomes of monitoring and summative evaluation *
   4. Define stakeholder roles in the monitoring and summative evaluation *
   5. Resources:
      • Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Statement On Cultural Competence In Evaluation
      • Community Tool Box:
        • ​Identifying and Analyzing Stakeholders and Their Interests
        • Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
        • Participatory Evaluation
        • Gathering Information: Monitoring Your Progress

 

 

Back to Top

 

2. Identify resources for monitoring and summative evaluation
   
   1. Person-power resources
   2. Data collection resources
   3. Resources:
      • Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Identifying Community Assets and Resources
        • ​Choosing Evaluators

 

Back to Top

 

3. Describe the intervention’s framework, logic model, or theory of change
   
   1. Purpose or mission
   2. Context or conditions
   3. Inputs: resources and barriers
   4. Activities or interventions
   5. Outputs of activities
   6. Intended effects or outcomes
   7. Resources:
      • Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Toolkit: Developing a Framework or Model of Change
        • Developing a Logic Model or Theory of Change

 

 

Back to Top

 

4. Focus the monitoring and summative evaluation plan
   
   1. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
   2. Set priorities by end-user questions, resources, context
   3. What questions will the monitoring and summative evaluation answer?
   4. Ethical implications (benefit outweighs risk)
   5. Resources:
      • Institute of Medicine:
        1. Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        2. Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Guiding Principles For Evaluators
      • Community Tool Box:
        1. Choosing Questions and Planning the Evaluation
        2. Ethical Issues in Community Interventions
        3. Documenting Health Promotion Initiatives Using the PAHO Guide

 

 

Back to Top

 

5. Plan for credible methods
   
   1. Stakeholder agreement on methods
   2. Indicators of success
   3. Credibility of evidence
   4. Resources:
      • Institute of Medicine:
        • Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget)
        • Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • National Collaborative on Childhood Obesity Research (NCCOR):
        • Catalogue of Surveillance Systems
        • ​Measures Registry
      • Community Tool Box:
        • Data Collection: Designing an Observational System
        • Selecting an Appropriate Design for the Evaluation
        • Collecting and Analyzing Data
        • Collecting and Using Archival Data
        • Qualitative Methods to Assess Community Issues
        • Implementing Photovoice in Your Community
        • Community-Level Indicators: Some Examples
      • Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users

 

Back to Top

 

6. Synthesize and generalize
   
   1. Disseminating and compiling studies
   2. Learning more from implementation
   3. Ways to assist generalization
   4. Shared sense-making and cultural competence
   5. Disentangling effects of interventions
   6. Resources:
      • Institute of Medicine:
        • Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support
      • Community Tool Box:
        • Troubleshooting Guide: There is Not Enough Change in the Community or System
        • Troubleshooting Guide: There is Not Enough Improvement in Outcomes
        • Troubleshooting Guide: There are Unintended or Unwanted Outcomes
        • Toolkit: Enhancing Cultural Competence
        • Adapting Community Interventions for Different Cultures and Communities
        • Providing Feedback to Improve the Initiative
        • Communicating Information to Funders for Support and Accountability
        • Refining the Program Intervention Based on Research
        • The County Health Rankings & Roadmaps Take Action Cycle

 

 

Back to Top

 

7. Systems and Evaluation
   • Resources:
     • IOM:
       • Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
       • Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation
       • Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
     • Community Tool Box:
       • Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders
       • Best Change Processes
       • MAPP: Mobilizing for Action through Planning and Partnerships

 

Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box.

 

*Monitoring is the tracking of the implementation of interventions compared to standards of performance.
*Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions.

 

 

Tool 2: Table of Program Components and Elements

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery

 

Tool 3: Evaluation Questions and Methods

		Type of Data Collection *					Experimental Design **
Key Evaluation Questions	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured Interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- Posttest Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Assessing Attainment of Objectives	Outcome Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Participants
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Community
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.

Tool 4: Evaluation Timeline

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system
Train participants / staff  in data entry
Maintain system (data collection and reporting)
Critical events
Identify key people
Conduct interviews
Report results
Community-Level Indicators
Identify local indicators
Identify comparison site
Secure data
Report results
Other Indicators (e.g., behavioral surveys, data observations)
Identify
Secure data
Report results

Experienced community builders know that involving stakeholders—the people directly connected to and affected by their projects—is essential. Their knowledge, lived experience, and perspectives can help shape, adjust, and improve the work so that it responds to community priorities and the needs of the people the initiative is intended to support. This is particularly important in evaluation.

As we have previously discussed, community-based participatory research can be used to describe a community, assess community issues, strengths, and needs, identify and select promising practices, and conduct evaluation. We consider participatory evaluation important enough to give it a section of its own and to show how it fits into the larger participatory research process.

It is a good idea to build meaningful stakeholder participation into a project from the beginning. One of the best ways to determine an appropriate direction for your work is to involve stakeholders in identifying community priorities, strengths, needs, and opportunities for meaningful impact. Likewise, one of the best ways to understand how a project affects people is to include those who participate in, experience, or are directly affected by its information, services, or advocacy in the evaluation process.

People directly involved in a project—including participants, staff, volunteers, community members, and partners—often have important insight into what is actually happening. Previously, we have discussed how you can involve community members in conducting research and choosing issues and priorities to address. This section focuses on involving stakeholders throughout the project, including in its evaluation, and on how that participation can strengthen the work and its outcomes.

What is participatory evaluation?

When many people think of evaluation, they imagine something that happens at the end of a project and determines whether it was successful. In practice, evaluation is most useful when it is integrated into a project from the beginning. Participatory evaluation involves stakeholders—particularly people directly affected by the project or involved in carrying it out—in developing an understanding of how the project is working and using that understanding to improve the work.

Participatory evaluation is more than simply asking stakeholders to take part. It creates opportunities for shared decision-making among community members, participants, staff, evaluators, and other partners throughout a project. People directly affected by the work contribute their lived experience, priorities, and expertise alongside professional and organizational knowledge. Together, participants can shape evaluation questions, interpret findings, and use what they learn to improve the project in ways that reflect community-defined needs and goals.

This approach to planning and evaluation depends on mutual trust, respect, transparency, and accountability. These relationships develop over time and are strengthened when organizations begin by learning about the community's history, culture, priorities, strengths, and existing knowledge. Respecting people's expertise and recognizing the value of different forms of knowledge can help build lasting trust and meaningful participation across a wide range of cultural, geographic, and socioeconomic contexts.

Another important part of a participatory process is making sure everyone involved has the information, preparation, resources, and support needed to participate fully. Some stakeholders may be unfamiliar with evaluation or research processes. Others may need support collaborating across different backgrounds or understanding how evaluation findings can be used. We will discuss stakeholder involvement, trust-building, accessible training, and other forms of support throughout this section.

The purpose of evaluation is not simply to determine what happened, but to use what is learned to strengthen the project and inform future action.

To accomplish this, evaluation should examine at least three areas:

• Process. The process of a project includes the planning, coordination, and logistical activities needed to establish and carry it out. Did we conduct an appropriate assessment beforehand to understand community priorities, strengths, and needs? Did we use what we learned to shape the project? Did we follow the intended timeline and organizational structure? Did we meaningfully involve the people we intended to involve? Did we have or obtain the necessary resources? Were staff, volunteers, and community partners adequately prepared and supported? Did we have the community partnerships and support we expected? Did we document our work accurately and in a timely way? Did we monitor and evaluate the work as planned?
• Implementation. Project implementation refers to the actual work of carrying out the project. Did we do what we intended to do? Did we reach and engage the people and communities we intended to reach? Did we use the approaches we planned to use? Was the level of activity what we expected—for example, did we provide the intended number of hours, events, services, or opportunities? What exactly did we provide or carry out? Did we make planned or unplanned changes, and what led to those changes?
• Outcomes. A project's outcomes are the changes or results associated with the work. Did the project contribute to the outcomes we hoped for? Were there other expected or unexpected effects? Who experienced benefits, challenges, or no meaningful change? Were there differences in outcomes among groups? What factors may have contributed to the results? What could we change to make the work more effective, equitable, or responsive?

Many people who write about participatory evaluation combine the first two areas into process evaluation and add another area—impact evaluation—alongside outcome evaluation. Impact evaluation considers longer-term changes associated with a project, including what happens after a particular program or initiative has ended.

For example, a community-based agricultural or public health initiative might focus on sharing skills and resources that local participants can continue using and adapting after the formal project ends. If an evaluation five or ten years later finds that the practices remain useful, have been adapted to local circumstances, and have spread through community networks, that may indicate sustained and meaningful impact.

To examine these areas effectively, evaluation should begin alongside assessment and planning rather than being added only after implementation.

In a participatory evaluation, stakeholders should be involved in:

• Naming and framing the issue, opportunity, or goal to be addressed
• Developing a theory of practice or logic model for how the desired change may occur
• Identifying the questions to ask about the project and the most appropriate ways to answer them

What is the goal, for example, of a program to improve school meals? It might be to increase access to fruits, vegetables, whole grains, and other nutrient-dense foods; improve students' satisfaction with meals; strengthen food literacy; increase participation in school meal programs; or support students' overall health and well-being.

The evaluation questions should reflect those goals. Measures might include what foods are offered, what students choose to eat, food waste, student feedback, meal participation, and relevant indicators of health and well-being. Looking at multiple outcomes provides a fuller understanding of the program's effects and avoids defining success primarily in terms of students' body weight.

• Collecting information about the project
• Interpreting and making sense of that information
• Deciding what to celebrate, continue, adjust, or change based on what is learned

Why would—and why wouldn't—you use participatory evaluation?

Why use participatory evaluation? The short answer is that it can be one of the most effective ways to understand what communities need and value, how a project is working, and what should change over time. It can also strengthen relationships, skills, shared decision-making, and community capacity in ways that more conventional evaluation approaches may not.

Some of the major advantages of participatory evaluation include:

• It can provide a fuller understanding of community priorities and a project's effects. When stakeholders are involved from the beginning in determining what should be evaluated, why it matters, and what the project should focus on, the work is more likely to reflect community priorities and the evaluation is more likely to capture outcomes that matter to the people affected.
• It can provide information that might otherwise be missed. When evaluation depends partly on community knowledge and experience, people may feel more comfortable sharing information with trusted peers, community partners, or evaluators who understand the local context.
• It helps explain what is and is not working from the perspective of people directly involved. Participants, staff, volunteers, and community partners can provide insight into which parts of a project are useful, which create barriers, and which may need to change.
• It can help explain why something does or does not work. People directly affected by a project can often identify practical, cultural, organizational, or structural reasons why a particular approach is effective for some people and less effective for others.
• It can contribute to a more effective and responsive project. Ongoing feedback from people directly involved helps projects identify emerging needs, recognize unintended effects, and make timely adjustments.
• It supports stakeholder agency and shared decision-making. Participatory evaluation creates opportunities for people who may historically have been excluded from formal decision-making processes to help determine how a project is understood, evaluated, and improved.
• It can center perspectives that institutions and decision-makers have not always included. Some communities experience barriers to participation in organizational, governmental, or research decision-making. Participatory evaluation can create more meaningful opportunities for people to contribute their knowledge, priorities, and perspectives and to influence decisions that affect them.
• It can strengthen skills that are useful in employment, education, civic participation, and other areas of life. Participatory evaluation may provide opportunities to develop research, communication, collaboration, critical-thinking, problem-solving, facilitation, data interpretation, and project-management skills.
• It can strengthen confidence while recognizing participants' existing knowledge and abilities. Being treated as a valued partner, contributing expertise, and seeing one's ideas influence real decisions can reinforce people's confidence in their ability to contribute to community change.
• It can strengthen people's ability to influence decisions that affect their lives and communities. Participating in a collaborative evaluation can provide experience navigating organizations, analyzing information, communicating recommendations, and taking collective action.
• It encourages shared ownership of the project. When people have meaningful opportunities to shape a project and its evaluation, they are more likely to see the work as a collaborative effort rather than something developed and imposed by others.
• It can encourage creativity. Bringing together people with different experiences, knowledge, and perspectives can generate new ideas and help challenge assumptions about familiar issues.
• It encourages collaboration. Participatory evaluation works best when participants recognize that different people bring different forms of knowledge, experience, and expertise. Collaboration of this kind can strengthen the evaluation while also building relationships that support future community efforts.
• It fits within a broader participatory approach. When assessment and project planning have already involved community members, participants, staff, and other partners, it makes sense for evaluation to follow the same collaborative approach. Planning for evaluation as part of the overall project can strengthen both the work and the evaluation process.

Participatory evaluation also involves challenges. Whether those challenges outweigh the benefits depends on the circumstances, resources, goals, timeline, and community context. Understanding potential barriers can help you decide whether a participatory approach is appropriate and how to design it effectively.

Potential challenges of participatory evaluation include:

• It can take more time than a conventional evaluation process. Meaningful participation requires time for relationship-building, discussion, collaborative decision-making, training, accessibility, and coordination. Bringing together people with different schedules, perspectives, and experiences can require more time than an evaluation conducted primarily by professional evaluators.
• It requires trust among participants. Historical inequities, prior negative experiences with institutions, differences in power, cultural differences, or unresolved conflicts can affect how comfortable people feel participating. Trust develops through transparency, accountability, follow-through, respect, and meaningful opportunities to influence decisions.
• You need to include a broad range of perspectives, not only established leaders. A participatory process can reproduce existing power structures if organizations consult only with people who already hold formal or informal leadership roles. Leaders may provide important perspectives, but they should not automatically be assumed to represent everyone in a community.

Involving only established leaders can exclude important experiences and perspectives, particularly those of people who have had fewer opportunities to participate in formal decision-making. Recruitment should therefore seek a broad range of participants, including people with different experiences, backgrounds, roles, and relationships to the project. This may require additional outreach and relationship-building, but it can make the evaluation more representative, credible, and useful.

• Participants may need accessible training, tools, and ongoing support. People come to participatory evaluation with different levels of experience with research, meetings, data collection, and analysis. Training and support should build on the knowledge participants already have and provide practical opportunities to develop additional skills.
• You need shared commitment from participants. Because participatory evaluators may take on significant responsibilities, people should understand the expectations, have meaningful input into their roles, and receive the resources and support needed to participate.
• Participants may face practical barriers that affect their ability to participate consistently. Work schedules, caregiving responsibilities, health needs, transportation, financial constraints, seasonal employment, and other circumstances can affect when and how people are able to participate. Participatory evaluation should anticipate these barriers rather than treating them as individual shortcomings. Flexible scheduling, transportation assistance, childcare, accessible meeting options, compensation for participants' time, and other forms of support can make participation more equitable and sustainable.
• You may need flexible and accessible ways to collect, document, and communicate information. Participants may have different literacy preferences, languages, communication styles, accessibility needs, or levels of familiarity with evaluation. Written reports may work well for some participants, while others may prefer verbal discussion, visual methods, audio, video, diagrams, mapping, storytelling, or other formats.
• Funders and policymakers may have different expectations about evaluation. Some may prefer external evaluations or particular research designs and may be unfamiliar with participatory approaches. Clear communication about the purpose, methods, rigor, benefits, and limitations of participatory evaluation can help build understanding and support.

Some of these challenges can also create opportunities. Training may strengthen transferable skills. Developing accessible ways to communicate information may improve the evaluation for everyone. Building trust can strengthen relationships beyond the evaluation itself. Helping funders and policymakers understand participatory approaches may also influence how future community work is supported. Even so, meaningful participation requires time and resources, and those realities should be considered when planning an evaluation.

When might you use participatory evaluation?

Participatory evaluation may be especially useful:

• When your project is already committed to a participatory approach. Evaluation can then be designed collaboratively as part of the broader project.
• When meaningful participation and useful results are more important than completing the evaluation as quickly as possible. Participatory evaluation requires time for collaboration, but that investment can lead to richer information and stronger community relationships.
• When funders and organizational partners support participatory approaches. Some funders may require external evaluation, while others may welcome participatory methods. Early communication can help clarify expectations and identify opportunities to combine different evaluation approaches when appropriate.
• When community members have important knowledge that external evaluators or project staff may not have. Historical, political, social, cultural, environmental, and interpersonal factors can influence how a project operates and how its outcomes should be interpreted. Community participants can help evaluators understand these contexts.
• When trust and relationships affect whether people are willing to share information. Community members may be more comfortable discussing experiences with people they know and trust, particularly when topics are sensitive or when institutions have previously failed to listen or follow through.
• When strengthening participant agency, leadership, or transferable skills is part of the project's goals. In these situations, the evaluation process itself can contribute to the broader purpose of the project.
• When you want to strengthen relationships and collaboration within a community. Participatory evaluation can create opportunities for people who may not regularly work together to exchange perspectives, build trust, and develop relationships that may support other community efforts.

Who should be involved in participatory evaluation?

We have referred throughout this section to stakeholders—people and groups who are affected by, involved in, responsible for, or able to influence the project being evaluated. Who those stakeholders are will vary depending on the project's focus, funding, activities, setting, and intended outcomes.

Groups that are often involved include:

• Participants and community members. These are people the project is intended to support or whose lives may be directly affected by it. They might include people living with a particular health condition, members of a particular community, recent immigrants, residents of a neighborhood, people participating in employment programs, or people affected by a community-wide initiative. Their experiences and perspectives are essential to understanding how the project is working.
• Frontline staff and volunteers. These are the people who carry out much of the project's day-to-day work. They may be professionals, people with specialized knowledge, community health workers, peer leaders, mentors, teachers, advocates, organizers, or volunteers. Because they interact closely with participants and implementation processes, they often have valuable insight into how the project functions in practice.
• Administrators and coordinators. These are people who oversee the project or particular aspects of it. They can provide insight into planning, operations, resources, partnerships, decision-making, and organizational constraints.
• External evaluators, when involved. External evaluators may provide methodological expertise, facilitate collaborative processes, support data analysis, or help organizations meet evaluation requirements.
• Community leaders, officials, and organizational partners. Their support may be important to the success of a project, and participation can create opportunities for greater understanding of community priorities and perspectives.
• Others whose lives or work are affected by the project. This group will vary substantially depending on the project and may include people whose employment, property, services, environment, access to resources, or daily experiences may be affected by the project's activities or outcomes.

For example, an environmental initiative or neighborhood plan might affect how local property owners, residents, businesses, community organizations, and other stakeholders use or experience a particular area.

How do you conduct a participatory evaluation?

Participatory evaluation can include aspects of project design as well as evaluation. What you evaluate depends on what the community and project partners want to understand and accomplish. Identifying evaluation questions helps determine what information will be collected and what aspects of the project will receive attention. When those questions emerge from an assessment process in which stakeholders have meaningfully participated, evaluation becomes part of a broader community-based participatory research process.

A participatory evaluation generally involves two interconnected stages. One involves identifying, recruiting, preparing, and supporting stakeholders who will participate in the evaluation. The other involves planning and carrying out the project and its evaluation. This second stage includes six steps:

• Naming and framing the issue, opportunity, or goal
• Developing a theory of practice or logic model
• Deciding what questions to ask and how to gather the information needed to answer them
• Collecting information
• Analyzing and interpreting the information
• Using what is learned to celebrate progress and adjust or improve the project

We will examine both stages in more detail.

Finding and preparing stakeholders to participate as evaluators

Recruiting participants for a participatory evaluation often requires intentional outreach and relationship-building. People may already have many demands on their time, may be unfamiliar with evaluation, or may be unsure whether their participation will genuinely influence decisions.

This is where trust-building becomes especially important. Community members may have good reasons to be cautious about outside organizations or institutions, particularly if previous engagement efforts were extractive, promises were not honored, or community input was collected but not acted upon. Building trust requires transparency, consistency, accountability, and follow-through. Organizations may need to demonstrate over time that participation will be meaningful and that community perspectives will influence decisions.

Once relationships have been established, participants may need different forms of preparation and support depending on their prior experience. Training should build on the skills, knowledge, and lived experience people already bring while providing any additional tools needed to participate confidently in the evaluation.

If the evaluation is part of a broader participatory effort, you may need to decide whether the same people who conducted assessments or helped plan the project should also participate in evaluation. There are advantages and disadvantages to either approach. An existing team may already understand the project, have experience working together, and be familiar with research methods.

At the same time, people who helped design or implement a project may have strong hopes for its success. That perspective can provide valuable insight, but it can also influence how findings are interpreted. Clear evaluation processes, reflection on potential bias, multiple perspectives, and transparent decision-making can help address this concern.

Capacity and workload are also important considerations. Evaluation may require additional time from people who have already contributed substantially to assessment and planning. Some may want to continue, while others may prefer to step back. Bringing in additional participants can distribute the workload and broaden the perspectives represented.

The best approach will depend on the people involved, the community context, available resources, recruitment capacity, and the commitments participants have already made.

Recruit participant evaluators

Recruitment can take many forms. In some situations, a broad invitation may be appropriate. In others, it may make sense to reach out to specific individuals, community organizations, participants, staff, or trusted community connectors. You might also ask community members to identify people whose experiences and perspectives should be represented.

Some basic guidelines for recruitment include:

• Use communication channels and approaches that reach the people you hope to involve
• Make the purpose and expectations as clear as possible
• Use clear, accessible language and provide information in the languages used by participants
• Share recruitment information in places and spaces the intended participants already use
• Reach out personally when appropriate, including through trusted community members and organizations
• Explain what people may gain from participation as well as how their participation may contribute to the project and community
• Make clear that participants are being invited because their knowledge, experiences, and perspectives are valuable
• Be transparent about the time, responsibilities, decision-making authority, and expectations involved
• Work with participants to identify roles and responsibilities that are realistic for them
• Provide or arrange practical supports—such as childcare, transportation, technology access, interpretation, accessibility accommodations, or compensation—when possible
• Ask participants to recommend others whose perspectives should be included

Potential participant evaluators—particularly community members whose participation is not part of their paid employment—should understand the expected commitment before agreeing to participate. The length of an evaluation will vary depending on the project, but some evaluations may continue for many months or longer. Planning should recognize that people's availability may change over time.

Discuss with participants what support they will need in order to participate meaningfully. This may include transportation, childcare, interpretation, accessible meeting formats, technology, flexible scheduling, or compensation for their time and expertise. Meetings should be scheduled in ways that reduce unnecessary barriers. When staff participation is required as part of their employment, evaluation responsibilities should be incorporated into their paid workload rather than treated as additional unpaid labor.

Aim to create an evaluation team that represents a range of stakeholder experiences and perspectives. Recruiting only established leaders may leave out people whose experiences differ from those of formal representatives. Broad participation can strengthen the quality of the evaluation while also creating opportunities for more people to develop leadership and decision-making experience.

Prepare and support participant evaluators

Participants may need different kinds of preparation depending on their experiences, roles, and familiarity with evaluation. Some may benefit from an introduction to participatory evaluation, meeting facilitation, research ethics, interviewing, observation, data interpretation, or other skills. Training should be accessible, practical, and responsive to participants' needs rather than based on assumptions about what they do or do not know.

Training can be organized in many ways. It might take place in short sessions over several weeks or months, during a concentrated workshop or retreat, through mentoring and coaching, or through a combination of approaches. The format should reflect participants' schedules, accessibility needs, learning preferences, and the demands of the project.

Different topics may also call for different learning methods. Facilitation and meeting skills may be strengthened through mentoring and practice. Interviewing may be learned through role-playing and feedback. Community engagement strategies may be best taught or co-developed by community participants themselves. Whenever possible, training should recognize that expertise exists throughout the group rather than flowing only from professional evaluators to community participants.

Some areas in which preparation or training may be helpful include:

• The participatory evaluation process. How participatory evaluation works, its goals, the roles participants may play, how decisions will be made, and what participants can expect.
• Meeting and collaboration skills. Listening, contributing to discussion, facilitating participation, navigating disagreement or conflict, sharing decision-making, and establishing group agreements.
• Interviewing. Building rapport, asking open-ended and follow-up questions, listening carefully, documenting responses accurately, being aware of body language and tone, and conducting individual or group interviews.
• Observation. Understanding different forms of observation, choosing appropriate settings and times, identifying relevant information, recognizing potential bias, and documenting observations accurately.
• Recording information and reporting it to the group. Documenting what participants say or do, the context in which information was collected, relevant environmental or situational factors, and other information needed to interpret findings responsibly.

Participants should have access to recording and reporting methods that work for them. Depending on people's preferences, languages, accessibility needs, and the setting, useful approaches may include written notes, drawings, maps, diagrams, photographs, audio recordings, video, storytelling, or other formats. When audio, photography, or video is used, participants should understand how the material will be used and provide informed permission before recording begins.

• Analyzing information. Critical thinking, interpreting quantitative and qualitative information, recognizing patterns and limitations, considering alternative explanations, and determining what findings may mean for the project.

Planning and implementing the project and its evaluation

A fully participatory approach assumes that stakeholders have meaningful opportunities to contribute not only to evaluation, but also to assessment, planning, implementation, and decision-making. The same people do not necessarily need to participate in every phase. However, when community members and other stakeholders have been involved from the beginning, they often have a deeper understanding of the project's goals, history, assumptions, and structure. That knowledge can strengthen the relevance and interpretation of the evaluation.

Naming and framing the issue or goal to be addressed

Identifying what you are evaluating helps define what the project is intended to address and accomplish. Community members, organizations, project participants, staff, and others with a stake in the work can collaborate to develop a shared understanding of priorities and desired outcomes. By examining community concerns alongside strengths, resources, relationships, and existing efforts, participants can identify where attention and action may be most useful.

Naming an issue or goal means identifying what the community wants to understand, prevent, improve, or change. Framing refers to how that issue is understood and what factors are considered relevant.

Consider community violence involving young people. If the issue is framed primarily as a law-enforcement problem, the responses considered may focus mostly on policing, prosecution, or punishment. A broader public health and community-safety framing may examine factors such as access to firearms, economic opportunity, housing stability, school climate, exposure to trauma, social connection, neighborhood conditions, access to recreation, mental health support, substance use, and relationships with community institutions.

That broader understanding may lead to a wider range of strategies, such as youth employment, mentoring, violence interruption, safe community spaces, behavioral health services, family and peer supports, community-led safety initiatives, firearm injury prevention, and changes to policies or systems that contribute to unequal exposure to violence. Including multiple perspectives can help communities understand both immediate risks and the conditions that shape them.

Developing a theory of practice to address the issue

How can a community effort contribute to meaningful and lasting change? Projects are often less effective when organizations launch disconnected activities or assume that a single strategy will address a complex issue. A stronger approach is to develop a clear plan for understanding the issue, coordinating action, implementing strategies, evaluating progress, adapting the work, and sustaining what is effective.

Whatever you call this plan—a theory of practice, theory of change, logic model, framework, or approach—it should explain how activities are expected to contribute to desired outcomes. It should also reflect community context, existing strengths, available resources, potential barriers, and the roles of different partners.

Once an issue has been identified, one possible theory of practice might include the following:

• Form a coalition of community members, organizations, agencies, and other partners connected to the issue.
• Recruit and prepare a participatory research and evaluation team that includes a range of stakeholder perspectives.
• Gather quantitative and qualitative information about the issue while also identifying community strengths, resources, relationships, and existing efforts that may contribute to solutions.
• Use what is learned to design strategies that reflect the complexity of the issue and the local context.

The result might be a single program or initiative or a coordinated community-wide effort involving several organizations, institutions, media partners, networks, and community members. When multiple partners are involved, coordination, communication, shared decision-making, resource development, and strategies for addressing organizational or systemic barriers should be included in the plan.

• Implement the agreed-upon strategies.
• Conduct monitoring and evaluation that provides ongoing information about implementation, outcomes, equity, and opportunities for improvement.
• Use evaluation findings to adapt and strengthen the work.
• Continue gathering information, evaluating progress, and adjusting strategies as community conditions and priorities change.

Deciding what evaluation questions to ask and how to answer them

Choosing evaluation questions helps determine what the project will pay attention to and what information will be collected. Important effects can be missed if no one asks questions designed to identify them. For this reason, evaluation questions should be selected carefully and developed with input from people who understand the project and the community from different perspectives.

Framing remains important at this stage. Participants should consider the issue in context, examine their assumptions, identify whose perspectives may be missing, and think about the outcomes they hope to see. They should also consider what social, environmental, organizational, economic, or structural conditions may influence those outcomes.

For example, if the goal is only to measure whether incidents of youth violence decrease, evaluation questions might focus primarily on reported incidents or injuries. If the broader goal is to strengthen long-term community safety and well-being, the evaluation may also examine young people's sense of safety and belonging, access to supportive relationships and opportunities, school engagement, exposure to violence, access to behavioral health services, neighborhood conditions, trust in institutions, disparities in outcomes, and other factors connected to community safety.

Broader goals generally require broader evaluation questions. They may also require combining multiple types of evidence and examining not only whether change occurred, but how, for whom, under what conditions, and with what unintended effects.

Collecting information

Collecting information is often one of the most time- and resource-intensive parts of an evaluation.

Depending on the questions being asked, evaluators may use one or more of the following approaches:

• Research using public records, administrative data, news archives, library collections, online sources, organizational records, and other available information
• Individual or group interviews
• Focus groups
• Community information-sharing and dialogue sessions
• Surveys
• Direct or participant observation

Evaluation methods should be adapted to participants' language, literacy preferences, accessibility needs, cultural context, and familiarity with research. Depending on the community and the evaluation questions, maps, drawings, photographs, storytelling, timelines, audio, video, participatory mapping, or other creative approaches may sometimes provide richer information than written questionnaires alone. These methods should be selected collaboratively rather than based on assumptions about what participants can or cannot do.

Analyzing the information you've collected

Once information has been collected, the next step is to make sense of it. What do the numbers suggest? What do people's experiences, observations, and stories reveal? Did the project unfold as planned? If changes were made along the way, what prompted them and what difference did they make?

Some questions may be relatively straightforward. If the project established clear participation or service goals, for example, you may be able to determine whether they were reached. If the goal was to engage 75 people and 82 participated, that tells you something about reach. If 50 participants were expected to complete a program and 61 completed it, that provides another piece of information.

Other findings require deeper interpretation. What if approximately half of participants report that a project was helpful while the other half report that it was not? That result should prompt further questions. Were people's experiences different depending on race, ethnicity, language, disability, geography, income, age, gender, program location, staff relationships, access barriers, or other relevant factors? Did different groups experience different parts of the project? Were the project methods equally accessible and responsive to everyone?

Collecting information often requires substantial effort, but analysis and interpretation are what turn that information into useful knowledge. Strong analysis can help identify ways to improve the project, understand differences in outcomes, recognize unintended effects, demonstrate accountability, and communicate the value and limitations of the work. Participatory analysis also helps ensure that conclusions are informed by people who understand the community and project from different perspectives.

In general, information gathering and analysis should address the three areas discussed earlier: process, implementation, and outcomes. The goal is both to support continuous improvement and to provide accountability to participants, communities, funders, organizational partners, and other stakeholders.

• Process. How well did planning, coordination, communication, participation, staffing, resource development, and decision-making work? Was the planning process meaningfully participatory? Were timelines realistic? Did the team have the resources and support it needed? Were meetings and participation accessible? Did planning and evaluation teams work effectively together? Were responsibilities clear? Did new leaders, partnerships, or opportunities emerge?
• Implementation. Did the project carry out the activities it intended to carry out? Did it reach and engage the people it hoped to reach? Were planned approaches actually used? Were services, activities, or opportunities provided at the intended level? What adaptations were made, and why? Implementation evaluation focuses on what was done and how it was carried out rather than on whether it produced the desired outcomes.
• Outcomes. What changes occurred? Were the expected outcomes observed? Who experienced those outcomes, and were there important differences among groups? What evidence suggests that the project contributed to those changes? What other factors may have influenced the results? Were there unexpected effects, whether beneficial, harmful, or neutral? What can be learned from those results?

Using the information to celebrate what worked and to adjust and improve the project

Accountability is important, but a major purpose of evaluation is also formative: to provide information that helps people strengthen the project as it develops. When reviewing process, implementation, and outcomes, useful questions include: What worked well? What created barriers? Who benefited and who may have been left out? What unintended effects occurred? What should continue? What should change?

Answering these questions may require additional analysis and discussion, but doing so can lead to meaningful improvements. Elements that are ineffective or create unnecessary barriers may need to be changed or discontinued. At the same time, successful strategies should also be examined carefully so they can be strengthened, sustained, and adapted when appropriate.

Remember to recognize progress and contributions. Celebrating accomplishments acknowledges the time, knowledge, effort, and relationships people contributed to the work. It can strengthen shared commitment, reinforce community relationships, and make visible what people have accomplished together.

Participatory research and evaluation are ongoing processes. Once a project has demonstrated positive results, continued learning and adaptation remain important. Community conditions, priorities, resources, and relationships change over time. Evaluation teams may also change, but continued participation from people with different stakeholder perspectives can help keep the work relevant and responsive.

Successful community work rarely means that no further effort is needed. Progress often depends on maintaining effective strategies, strengthening community capacity, monitoring changing conditions, addressing new challenges, and continuing to adapt. Participatory research and evaluation can help communities and organizations understand what is changing and determine how the work should evolve.

In Summary

Participatory evaluation is one component of participatory research. It involves people connected to and affected by a community project in determining what should be evaluated, identifying appropriate questions and methods, collecting and interpreting information, and using what is learned to strengthen the work.

Participatory approaches bring together multiple forms of knowledge, including lived experience, community history, cultural knowledge, professional expertise, organizational experience, and research skills. These perspectives can improve the relevance and usefulness of an evaluation. Meaningful participation can also strengthen relationships, skills, shared decision-making, community leadership, and confidence in collective action.

Conducting a participatory evaluation generally involves:

• Recruiting, preparing, and supporting a diverse stakeholder evaluation team
• Naming and framing the issue, opportunity, or goal
• Developing a theory of practice or logic model to guide the work
• Identifying meaningful evaluation questions and appropriate methods
• Collecting information
• Analyzing and interpreting information collaboratively
• Using what is learned to recognize progress, address challenges, and adjust or improve the work

As with many of the community-building strategies described in the Community Tool Box, the final step is to continue learning and adapting. Participatory research and evaluation should continue for as long as the work can benefit from understanding changing community priorities, conditions, strengths, and outcomes. Ongoing evaluation can help communities and organizations sustain progress, respond to new circumstances, strengthen effective approaches, and remain accountable to the people most affected by the work.

What is participatory evaluation?

___Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work

What are the advantages of using participatory evaluation?

___It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects

___It can get you information you wouldn't get otherwise

___It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff

___It can tell you why something does or doesn't work

___It results in a more effective project

___It empowers stakeholders

___It can provide a voice for those who are often not heard

___It teaches skills that can be used in employment and other areas of life

___It bolsters self-confidence and self-esteem in those who may have little of either

___It demonstrates to people ways in which they can take more control of their lives

___It encourages stakeholder ownership of the project

___It can spark creativity in everyone involved

___It encourages working collaboratively

___It fits into a larger participatory effort

What are the disadvantages of using participatory evaluation?

___It takes more time than conventional process

___It takes the establishment of trust among all participants in the process

___You have to make sure that everyone's involved, not just "leaders" of various groups

___You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills

___You have to get buy-in and commitment from participants

___People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation

___You may have to be creative about how you get, record, and report information

___Funders and policy makers may not understand or believe in participatory evaluation

When would you use participatory evaluation?

___When you're already committed to a participatory process for your project

___When you have the time, or when results are more important than time

___When you can convince funders that it's a good idea

___When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of

___When you need information that it will be difficult for anyone outside the community or population to get

___When part of the goal of the project is to empower participants and help them develop transferable skills

___When you want to bring the community or population together

Who should be involved in participatory evaluation?

___Participants or beneficiaries

___Project line staff and/or volunteers

___Administrators

___Outside evaluators, if they're involved

___Community officials

___Others whose lives are affected by the project

How do you conduct a participatory evaluation?

___Recruit stakeholders as participant evaluators

___Train evaluators

___Name and frame the issue

___Develop a theory of practice to address it

___Determine the evaluation questions

___Collect information

___Analyze the information

___Use your analysis to celebrate what worked and adjust the rest to improve the project

___Stick with it indefinitely

Tool 1: Team Data Collection - How to Do It Well

This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007. We have included her initial e-mail, and then added the summarized reponses to each of her questions.

From: Michelle Garred
Subject: Team data collection: how to do it well?

Dear colleagues,

I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines. Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.

Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well.

Among the many questions are:

• What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
• How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
• How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
• In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
• How would you cite team interviews in a Ph.D. thesis?
• What are the key resources I should review?

What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?

Advantages:

• Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.
• Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.

Pitfalls:

• More personalities and activities to coordinate means more opportunities for things to go wrong.
• Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.
• Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.

Pitfalls can be addressed through training:

• Need for consistent, systematic training of data collectors.
• Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.
• Conduct interviewing practice sessions before beginning actual interviews.
• During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.
• Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.
• Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?
• Demonstrated skills before graduation from training phase

And through regular team meetings:

• Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along.
• Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.
• Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.
• If not together on site, consider on-line conferencing.

How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?

Keep in mind that we will have a team of 5 people conducting semi-structured interviews and we will probably not use audio recording.

• Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.
• Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.
• Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked.  Here's one example of a card storming.
• If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.

How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?

Again, this can be addressed through team training and regular meetings . . .

• Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.
• Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.
• Revisit this issue during regular team meetings during the data collection phase.

And through wise design of the note-taking protocols:

• Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.
• Additionally, you might have team members write up a brief summary within 24 hours of completing the interview.
• Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.

In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?

I understand that team data collection is common in action research, but less common in a Ph.D. thesis. It would help me to hear some different opinions on this issue.

• For quantitative data, one might set a quota of random 10% of data reviewed for QA.
• I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.
• As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
• Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
  • Zuber-Skerritt, Ortrun, and Perry, Chad (2002)  Action research within organisations and university thesis writing. The Learning Organization, 9(4), 171-179.

How would you cite team interviews in a Ph.D. thesis?

• In table format within the document.
• I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
• I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
• When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
• Consider in advance whether you will use the participants' real names or pseudonyms.

Key resources

• See Madeline Church's doctoral thesis, Creating an Uncompromised place to belong - why do I find myself in networks? from the living theory section of Action Research.
• The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews. You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006).  "Finding common ground in team-based qualitative research using the convergent interviewing method." Qualitative Health Research, 16(8), 1145-1157.
  • There's a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006). Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.

• A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about. You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.

  Tool 2: Dabbling in Data

Dabbling in the Data from Public Profit is a practical guide that helps fill this gap. It shares easy-to-use, hands-on ways to explore and analyze data often found in mission-driven work. Whether you are just starting out or want to feel more confident with data, this guide gives you tools and ideas to make data exploration easier and more helpful.

The latest edition, released in late 2024, includes:

• 10 new activities
• More user-friendly tools
• Virtual adaptations
• Playlists for mixed methods analysis

Example 1: True Collaboration for a Win-win in Community-engaged Research

Dr. Marvia Jones, Director of the Kansas City Health Department, spoke on partnerships in community-engaged research at the Life Span Institute's 2024 Collaboratory on January 11th, 2024: 

When we think about the word "partner," we can think about a significant other. If I were to say, "What do you want out of your partnership with your significant other?", you may say things like, "you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year." Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things.

We want someone who is with us, who engages us from the very beginning of a process, so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this?

I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But as a partnership, through our partnership, we have to work through those solutions.

Those are some of the things that I think we look for in a community-based research partnership, having that flexibility and understanding. One of the things that I noticed, as I went from being a researcher to being in practice, is you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?”

What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we do need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information.

So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.

• What do we mean by choosing evaluation questions?

• Why is it necessary to choose evaluation questions carefully?

• When should you choose questions and plan the evaluation?

• Who should be involved in the process?

• How do you choose questions and plan the evaluation?

Chapters 36-39 of the Community Tool Box concern the evaluation of community programs.  We've chosen to devote so much space to evaluation because it's one of the most important parts of any effort to improve community life and bring about lasting social change.  It can help you to better understand how well the planning and preparations for your program went, whether you implemented it as you meant to, and what the consequences were.  It can tell you whether you met your original objectives and goals or not, and give you information about what you need to change to be more effective.

Chapter 36 explains how an evaluation works, and gives some guidance as to how to develop it. Chapter 38 will assist you in gathering and analyzing the information you want, and Chapter 39 deals with how to use that information to improve your program and garner community and funding support. Here, in Chapter 37, we look at evaluation from a research point of view - how to plan and structure an evaluation that can help you better understand and improve what you do.

In this section, we'll discuss the first, and perhaps most important, step in evaluation research: deciding exactly what to evaluate. Each of the rest of the sections in the chapter will deal in detail with one of the steps you'll need to take to design, implement, and use the evaluation. The goal of the chapter is to provide guidelines that are useful to grassroots or community-based organizations as well as students or academic researchers.

What do we mean by choosing questions, and why is it necessary?

Every evaluation, like any other research, starts with one or more questions. Sometimes, the questions are simple and easy to answer. (Will we serve something close to the 50 people we expect to?) Often, however, the questions can be complex, and the answers less easy to find. (Which, or which combination, of the three parts of our intervention will affect which of the two behavior changes we seek within participants?) The questions you ask will guide not only your evaluation, but your program as well. By your choice of questions, you're defining what it is you're trying to change.

You choose your evaluation questions by analyzing the community problem or issue you're addressing, and deciding how you want to affect it. Why do you want to ask this particular question in relation to your evaluation? What is it about the issue that is the most pressing to change? What indicators will tell you whether that change is taking place?  Is that all you're concerned with? The answer to each of these and other questions helps to define what it is you're trying to do, and, by extension, how you'll try to do it.

For example, what's the real goal of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss in kids who are overweight or obese.  It could be to educate them about healthy eating, and to persuade them to be more adventurous eaters.

The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure weight loss, for instance, then clearly that's not what you're aiming for. If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods? You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

Things to consider when choosing evaluation questions

What do you want to know?

Academics and other researchers may approach choosing research questions differently from those involved in community programs.  In addition to their practical and social applications, they may choose problems to research simply because they are interesting, or because they tie into other work that they or their colleagues are doing. Community service workers and others directly involved in programs, on the other hand, are concerned specifically with improving what they're doing so they can help to enhance the quality of life for the participants in their programs, and often for the community as a whole.  Since we assume that most people using this chapter of the Tool Box are likely to be practitioners in the community, let's look at some of the reasons they might pick a particular area to evaluate.

If you're running, or about to run, a program to affect a community issue or problem, you might want to know one or more of the following:

Is there a cause-and-effect relationship (i.e., does one action or condition directly cause another) between a particular action and a particular change?  Usually, you'll be concerned with this in terms of your program. (Does our smoking-cessation support group help members to quit smoking?) Sometimes, however, it might be important to look at it in terms of the community. (Does a smoking ban in public buildings, bars, and restaurants lead to a decrease in the number of community residents who smoke?)

If we try this new method, what will happen?

Will the program that worked in the next town, or the one that we read about in a professional journal, work with our population, or with our issue?

Why are you interested?

Some of the same differences between the concerns of researchers and the concerns of practitioners may hold here. Those interested primarily in research may simply be moved by curiosity or by the urge to solve a difficult problem. As a practitioner, on the other hand, you'll want to know the effects of what you're doing on the lives of participants or the community.

Your interest, therefore, might grow from:

• Your experience with an issue and its consequences in a particular population or community
• Your knowledge of promising interventions and their effects on similar issues
• The uniqueness of the issue to your particular community or population
• The similarity of the issue to other issues in your community, or the issue's interaction with other issues

Your interest as a community worker has to be considered in relation to your evaluation and the purpose of your program. Your basic intent is probably to improve things for the population or the community, but in what ways and by what means? Are you trying out some new things in the hope of making an already-successful program more successful? Are you importing a promising practice to see if it works with your population?  Are you trying to solve a particularly difficult professional problem?

A community mediation program found that it was having little success in cases involving adolescents.  After conferring with other similar programs - all of which were struggling with the same issue - mediators in the program devised a number of strategies to try to reach youth. The overall question they were concerned with - "Will these strategies make it possible to mediate successfully where teens are involved?" - was one with real consequences.

Is the issue you're addressing important to the community or to the society?

Media reports about or community attempts to address the issue are clear indicators that it is socially important. If it affects a particular group - violence in a given neighborhood, a high rate of heart disease among middle-aged Black males - it has an obvious impact on the community and society. If your program or intervention has the potential to help resolve the issue in other places, to be used by community workers in other fields, or to be applied in a number of ways, the importance of your analysis increases even further. If addressing the issue can lead to long-term positive social change, then the analysis is vitally important.

All of this affects your evaluation and the questions you ask. If the issue is one of social importance, then your evaluation of your work is socially important as well.  Are you addressing the aspects of your program or intervention that are of the greatest value to participants, the community, and society?  If not, how might you begin to do so?

How does the issue relate to the field?

The real question here is not whether the issue is important to the field - if it's important to the community, that's what matters. However, you should explore whether there's evidence from the field to apply to the issue.  Is what you're doing likely to be more effective than other approaches that have been tried?  If your approach isn't effective, are there other approaches out there that hold more promise? Can the published material about the issue help you understand it better, and give you better ideas about how to address it?

Is the issue general, rather than specific to your population or community?

Consider whether there is evidence that the issue occurs with a variety of populations and under a range of conditions. Also consider whether the observations or methods used to determine the issue's existence are accurate and whether they can be used in different situations and with different groups. Your evaluation may give you valuable information to pass on to practitioners in different fields or different circumstances.

Who might use the results of your evaluation?

If evaluation shows that your program or intervention is successful, that's obviously valuable information, especially if what you're evaluating is innovative and hasn't been tried before. Even if the evaluation turns up major problems with the intervention, that's still important information for others - it tells them what won't work, or what barriers have to be overcome in order to make it work.

Some of those who might use your results include individuals and groups affected by the issue; service providers and others who have to deal with the problem (in the case of youth violence, for instance, this last group might include police, school officials, small business owners, parents, and medical personnel, among others); advocates and community activists; and public officials and other policy makers.

Whose issue is it?

Who has to change in order to address the issue? The focus of the intervention will tell you whom the evaluation should focus on.

Some possibilities:

• Those directly affected by the problem
• Those in direct personal contact with those directly affected: parents, spouses and children, other relatives, friends, neighbors, coworkers
• Those who serve or otherwise deal with those directly affected: medical professionals, police, teachers, social workers, therapists, etc.
• Administrators and others who serve or deal with those indirectly affected: hospital or clinic directors, police chiefs, school principals, agency directors, etc.
• Appointed or elected officials and other policy makers

Why is it necessary to choose evaluation questions carefully?

You know why you're running your program.  Evaluating it should just be a matter of deciding whether things are better when you evaluate than they were before you started, right?  Well, actually...wrong.  It's not that simple. First of all, you need to determine what "things" you are actually looking at (remember the school lunch example?)  Second, you will need to consider how you will determine what you're doing right, and what you need to change. Here's a partial list of reasons why choosing questions beforehand is important.

• It helps you understand what effects different parts of your effort are having. By framing questions carefully, you can evaluate different parts of your effort. If you add an element after the start of the program, for instance, you may be able to see its effect separate from that of the rest of the program...if you focus on examining it.  By the same token, you can look at different possible effects of the program as a whole. (Do adult basic education learners read more as a result of being in a program?  Are they more likely to register to vote? Do their children improve their school performance?)
• It makes you clearly define what it is you're trying to do  What you decide to evaluate defines what you hope to accomplish. Choosing evaluation questions at the start of a program or effort makes clear what you're trying to change, and what you want your results to be.
• It shows you where you need to make changes. Carefully choosing questions and making them specific to your real objectives should tell you exactly where the program is doing well and where the program isn't having the intended effect.
• It highlights unintended consequences. When you find unusual answers to the questions you choose, it often means that your program has had some effects you didn't expect. Sometimes these effects are positive - not only did people in the heart-healthy exercise program gain in fitness, but a majority of them report changing their diet for the better and losing weight as well - sometimes negative - obese children in a healthy eating program actually gained weight, even though they were eating a healthier diet - and sometimes neither. Like the side effects of medication, the unintended consequences of a program can be as important as the program itself. (In the case of the exercise program, the changes in diet might do as much as or more than the exercise to maintain heart health, for instance, and may point toward changing the focus of the program in some way.)
• It guides your future choices. If you find that your program is particularly successful in certain ways and not in others, for example, you may decide to emphasize the successful areas more, or to completely change your approach in the unsuccessful areas. That, in turn, will change the emphasis of future evaluation as well.
• In participant evaluations, evaluation involves stakeholders in setting the course of the program, thus making it more likely that it will meet community needs.
• It provides focus for the evaluation and the program. Choosing evaluation questions carefully keeps you from becoming scattered and trying to do too many things at once, thereby diluting your effectiveness at all of them.
• It determines what needs to be recorded in order to gather data for evaluation. A clear choice of evaluation questions makes the actual gathering of data much easier, since it usually makes obvious what kinds of records must be kept and what areas need to be examined.

When should you choose questions and plan the evaluation?

Evaluation questions, since they help shape your work, should be chosen and the evaluation planned when planning the overall program or effort. That gives you time and room for a participatory process, and gives you the chance to use the evaluation as an integral part of the program. As the program unfolds, you might find yourself adjusting or adding questions to reflect the reality of what is happening, but unless your original questions were misguided (you were wrong about what behavior had to change in order to produce certain results, for instance), they should serve you well.

Now let's discuss reality for many community based and grassroots programs. They're often understaffed and underfunded. Staff members may be underpaid, and may often work many more hours a week than they're paid for, because of their dedication to social justice and social change. Most or all program staff may even be volunteers, with full-time jobs and family responsibilities aside from their work in the program.  Initial evaluation in these circumstances is often anecdotal - i.e., based on participants' comments and stories about their progress and staff members' personal, informal observations.  A formal evaluation will probably wait until there's funding for it, or until someone has the time to coordinate or take charge of it.

In that case, the "when" becomes "as soon as you can."  You may be dealing with a program that has just started, or with one that's been operating for a long time. You may know that changes need to be made, or it may seem that the program is in fact meeting its goals. Whatever the situation, evaluation questions need to be chosen, and an evaluation planned that will give you the information you need to improve your work. Even with a program that's been going on for a while, the questions can still help you define or redefine your work, and will certainly help you improve it over the long term.

Who should be involved in choosing questions and planning the evaluation?

If you've consulted other sections of the Tool Box concerned with evaluation, you probably know that we advocate that all stakeholders be involved in planning the evaluation.  We believe that the best evaluation is participatory. That means that there is representation of the views and knowledge of people affected by the issue to be addressed. The list of potential participants is essentially the same as that under "Whose problem is it?" in the first part of this section: those directly affected and their close contacts; those who work with those directly affected, or who deal directly or indirectly with them and the issue; and public officials. To these groups, we might add other concerned citizens, and those indirectly affected by the issue. (A shop owner may not be a victim of neighborhood violence, but fear of that violence might nonetheless keep customers away from his shop, for instance.)

Evaluations that involve all stakeholders have a number of advantages over those conducted in a vacuum by outside evaluators or agency or program staff. They're more likely to reflect the real needs of the community, and they bring to bear the community's knowledge of its own context - history, relationships, culture, etc. - without which a program and its evaluation can go astray.

Participation can range from simple consultation before the fact to complete involvement in every aspect of an evaluation - assessment, planning, data gathering, analysis, and passing on the information.  In general, the greater the involvement of stakeholders, the better, but in-depth involvement of the stakeholders may not always be possible. There are time disadvantages to participatory evaluation - it takes longer - and there are logistical concerns, as well. Participants may have nothing in their backgrounds to prepare them for research, so training in a number of areas may be necessary, requiring skill, careful planning, and yet more time. The level of participation your evaluation can sustain, therefore, relies to some extent on your time constraints and your capacity to train and support participants.

How do you choose questions and plan the evaluation?

Choosing questions

When you choose evaluation questions, you're really choosing a research problem - what you want to examine with your research. (Evaluation, whether formal or informal, is in fact research.) You have to analyze the issue and your program, consider various ways they can be looked at, and choose the one(s) that most nearly tell you what you want to know about what you're doing.  Are you just trying to determine whether you're reaching the right people in sufficient numbers with your program?  Do you want to know how well an intervention is working with specific populations? What kinds of behavior changes, if any, are taking place as a result?  What the actual outcomes are for the community? Each of these - as well as each of the many other things you might want to know - implies a different set of evaluation questions. To find the questions that best suit your evaluation, there is a series of steps you can follow.

Describe the issue or problem you're addressing

A problem is a difference between some ideal condition (all people 10 years of age or older should be able to read; people should be able to find a decent job) and some actual condition in the community or society (a 25% illiteracy rate among those attending a particular high school; 50% unemployment among minority youths in a particular city). This may mean the absence of some positive factor (qualified teachers and adequate educational facilities; entry-level jobs that are reachable from minority neighborhoods) or the presence of some negative factor (students' difficulty with English; discrimination against minority job applicants), or some combination of these.

To describe the issue or problem:

• Describe the ideal condition, including the positive factors present and the negative factors absent.  What should it look like if everything was as you'd want it to be?
• Describe the actual conditions that constitute the prolem of interest, including the negative conditions present and the positive conditions absent.  What are conditions really like?
• Describe the actual problem in terms of what you're hoping to change.  What positive factors do you want to produce and/or what negative factors do you want to eliminate?

Describe the importance of the problem

To be sure that this is a problem you really should be addressing, consider its importance to those affected and to the community.

• Is the discrepancy between ideal and actual conditions of the kind and size to be considered important?
• What are the consequences (positive and negative) of the problem?
• Who experiences these consequences (i.e. program participants; their families, friends, and peers; service providers, policymakers, and others)? How many people are affected?
• How often and for how long are they affected? What is the intensity of the effect?
• How much does the fact that the problem is experienced to this degree by these people matter to them?

You might also ask whether the effects of the problem matter to society, but in fact, that shouldn't make a difference.  If they matter to the people who experience them, they're important.  Society doesn't always consider a problem important if it's only a problem for a minority, or for a group that's generally ignored (the poor, the homeless).

In light of these factors, decide whether the problem is important to the evaluation.

Describe those who contribute to the problem

Whose behavior, by its presence or absence, contributes to the problem?  Are they in the program participants' personal environment (participants themselves, family, friends), service environment (teachers, police), or broader environment (policymakers, media, general public)? For each of them, consider the types of behavior that, by their presence or absence, contribute to the discrepancy that constitutes the problem.

Assess the importance and feasibility of changing those behaviors

How important is each of these behaviors to solving the problem?  What are the chances that your effort can have any effect on each of them?

Describe the change objective

Based on the above analysis, choose behavior changes to target in specific people. Where you can, specify the desired levels of change in targeted behaviors and outcomes (those changes in conditions that should occur if the problem were to be solved).

For example, a behavior change goal might be an increase in pre-employment capacity - self-presentation, job-seeking, interview skills, interpersonal competence, resume writing, basic skills, etc. - for minority job seekers aged 18-24. Or you might instead or in addition target policy makers, with the goal of having them offer tax incentives to businesses that locate in or close to minority communities.

This is a way of defining your work. If you're planning the evaluation as you plan the program - as you would in the ideal situation - then the questions you're asking the evaluation to examine reflect the problems you're trying to solve, and this kind of analysis is important.  If you're starting an evaluation of a program that has been in place for some time, then you're going to have to do some figuring after the fact about what consequences you think (hope) the program is having, and what they will lead to.  You may be talking about changes in specific participant behaviors, about behaviors that act as indicators of other changes, or about results of another sort (participants gaining employment, for instance, which may have a direct relationship to participant behavior or may have more to do with local economic conditions).

Make sure that the expected changes would constitute a solution or substantial contribution to the problem

If you conclude that they would not result in a substantial contribution, revise your choice of problem and/or your selection of targeted people and actions as necessary. If you think that what you're looking at in an evaluation doesn't address the problem, then you should be looking at something else. If the objectives you've chosen do constitute all or a substantial part of a solution, you've found your questions.

Setting

Now that you've chosen your questions, there may be other factors to consider, such as the settings in which the evaluation will be conducted. If your program is relatively small and/or has only one site, this wouldn't be an issue. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program.

 There are some situations in which the choice setting may be important:

• If your program is very large and/or has multiple sites
• If different sites provide different services, activities, or conditions, or use different methods

Multiple sites

Multiple sites

Can present a challenge for an evaluation, because, although every effort may be made to make the program at all sites exactly the same, it will seldom be so. If the program relies on human interaction - teacher/learner, counselor/counselee, trainer/trainee, doctor/patient, etc. - there will be differences from site to site depending on the people staffing each. (The exception is when the same people staff all sites, providing the same services at each site at different times or on different days.)  Even if all are equally competent, no two staff members or teams will do things in exactly the same way or relate to participants in exactly the same way, and the differences can be reflected in differences in outcomes.  If methods or other factors vary from site to site, that will further complicate the situation.

Furthermore, the physical character of a site can influence not only program effectiveness, but also the recruitment of participants and whether or not they remain in the program long enough for it to have some effect (often called "retention.")  The site's layout, comfort, apparent safety and security, and - often most important - how easy it is to get to, all affect whether participants enroll and stay in the program.

Where you do have the capacity to evaluate all sites, it will be helpful to build into the evaluation a method of comparing them. This will allow you to identify and adopt at all sites methods, conditions, or activities that seem to make one site particularly successful, and to identify and change at all sites methods, conditions, or activities that seem to create barriers to success at others.

If you can't evaluate each site separately, you'll have to decide which one(s) will give you the information that will most help in adjusting and improving your program. If you're most concerned with assessing your overall effectiveness, this may mean evaluating the site(s) closest to the program norm, in terms of methods, conditions, activities, goals, participant/staff interaction, etc. If, on the other hand, your chief consideration is learning whether a particular new or unusual method or situation is working, you may find yourself evaluating the site(s) least like the others.

If sites appear only minimally different, some other considerations that may come into play are:

• The number and character of participants at the site. Participants at a particular site may be experiencing the effects of the issue more severely, or may have a particular important characteristic, such as a language barrier.
• The ability and willingness of participants and staff to support the evaluation research. If staff at a particular site are unable or unwilling to record observations, attendance, and other key information, or if site participants are unable or unwilling to be interviewed or monitored, evaluation at that site might be difficult.
• The stability of the population at the site. If participants at a site come and go at a rapid rate - unless that's the program's intent - it can be difficult to gain information that contributes to an accurate evaluation.
• An exception, of course, occurs here if one point of the evaluation is to find out why participants stay for so short a time, and to try to develop methods or create conditions to assist them to remain in the program long enough to reach their goals.

Sites with different methods, conditions, activities, or services

Programs sometimes are organized so that different methods are used or different services provided at different sites. In other cases, conditions may vary from site to site because of the sites' geographical locations or the available space. The ideal situation is to evaluate all sites and compare the effects of the different methods, conditions, or services. When that's not possible, you'll have to decide what's most important to find out.

If the methods, services, or conditions at a particular site are new or innovative, you may want to evaluate them, rather than those that have a track record. There may be a particular method or service that you want to evaluate, in which case the decision about which site to choose is obvious. The decision should be based on what makes the most sense for your program, and what will give you the best information to improve its effectiveness.

When you have the capacity to choose more than one site to evaluate, it often makes sense to choose two or three sites that are different - especially if each is representative of other sites in the program or of program initiatives - so that you can compare their effectiveness. Even where sites are essentially similar, you'll get more information by evaluating as many as you can.

Participants

Another factor to consider is the participants whose behavior, activity, or circumstances will be evaluated. If your program is relatively small this might not be an issue - the participants will simply be all those in the program. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program, there are some situations in which the choice of participants may be important:

• If your program includes different groups of participants (groups that are in different stages of the program, or that are exposed to different methods or services).
• If groups of participants belong to populations with distinctly different cultures, stemming from race, ethnicity, class, religion, or other factors.

Multiple groups

There are a number of reasons why there might be multiple groups of participants in a program. You might start different groups at different times, either because the program has a rolling start schedule (when there are enough people for a class/training group, one will begin), or because the program is aimed at different groups (for example, 5 year-olds, 8-year-olds, and 14-year-olds). You might also be trying different strategies with different groups.

The Brookline Early Education Project (BEEP), a program aimed at school readiness for children aged pre-birth through 5, recruited pregnant families in three cohorts over the course of three years.  In addition, families in each cohort were assigned to one of three levels of service. Thus, there were actually nine different groups among BEEP participants, even though, by the third year, all were receiving services at the same time.

Once again, if there's no problem in evaluating the whole program, participants will simply include everyone. If that's not possible, there are a number of potential choices:

Evaluate your work with only one group, with the expectation that work with the others will be evaluated in the future. In this case, you'd probably want to choose the one for whom you consider the program most crucial. They might be at greater risk (of heart attack, of school failure, of homelessness, etc.) or might be experiencing the issue at a high level of intensity (daily shooting incidents in the neighborhood, high rates of teen pregnancy, massive unemployment).

Include a small number (2-4) of groups in your evaluation. You might want to choose groups with contrasting characteristics (different ages, for example, or addressed by different strategies). On the other hand, depending on the focus of your evaluation, you might want groups that are essentially similar, to see whether your work is consistent in its effects.

Choose a few participants from each group to focus your evaluation on.  While this won't give you a complete picture, it should give you enough information to tell where your program is accomplishing its goals and where it needs improvement. The differences in the ways participants in different groups respond to the program (assuming there are differences) can also give you ideas for ways to change what you're doing.

Participants from different populations and cultures

Cultural factors can have an enormous effect on participants' responses to a program. They can govern conceptions of social roles, family responsibilities, acceptable and unacceptable behavior, attitudes toward authority (and who constitutes authority), allowable topics of conversation, morality, the role of religion - the list goes on and on. In planning a program that involves members of different populations and cultures, you essentially have three choices:

• Plan your program and implement it in the same way for everyone. If the program involves groups - classes, support groups, etc. - participants' membership is determined not by population group but by when they sign up, what time of day they can attend, what they sign up for, or whatever other criteria make sense logistically.
• Plan your program to be as culturally sensitive as possible, and try to screen out anything that might be offensive to or difficult for any group. In this instance, you might be prepared to respond if participants from a particular population requested a group of their own.
• Divide participants by cultural group and plan different culturally sensitive approaches for each. Your overall approach might be the same for everyone, but the way you apply it might differ by culture.

In any of these instances, it would probably be important to understand how well your approach is working with members of the various populations.  If you can evaluate the whole program, make sure that you include enough members of each group so that you can compare results (and their opinions of the program) among them.  If your evaluation possibilities are limited, then your choices are similar to those for multiple groups of other kinds, and will depend on what exactly is most useful for you.

There are interactions between the choice of sites and the choice of participants here. You may be concerned about the effects of your program on a particular population, which may be largely concentrated at one site.  In that case, if you have limited resources, you may want to evaluate only that site, or that site and one other.

Regardless of other considerations, you may want to set some guidelines about whom you include in the evaluation. How long do people have to be in the program, for instance, before they're included? In other words, what constitutes participation? (This also sets a criterion for who should be counted as a drop-out: anyone who starts, but leaves before meeting the standard for participation.) What about those whose attendance is spotty - a few days here, a few days there, sometimes with weeks in between? Do they have to have attended a certain number of hours to be considered participants?

These issues can be more complex than they seem. People may start and drop out of a program numerous times, and then finally come back and complete it. Many others start programs numerous times, and never complete them. It's usually impossible to tell the difference until someone actually gets to the point of completion, whatever that means for the particular program.

In a reversal of the start-many-times-before-completing scenario, there can be a few people who stay in a program right up till the end and then drop out. This may have to do with the fear of having to cope with success and a change in self-image, or it may simply be a pattern the person has learned to follow, and will have to unlearn before being able to complete the program.

Should any or all of these people be included in or excluded from an evaluation, either before (because of their history in the program) or after the fact? That's a decision you'll have to make, based on what their inclusion or exclusion will tell you. Just be sure that your evaluation clearly describes the criteria that you decide to use for your participants.

If you're an outside evaluator or academic or other independent researcher

Up to this point, we've largely ignored the evaluation difficulties faced by evaluators not directly connected with the organization or institution running the program they're evaluating. If you've been hired or designated by the organization or a funder to evaluate the program, you have to establish trust, both with the organization and its staff and with participants, if you hope to get accurate information to work with. You also have to learn enough in a short period about the community, the organization, the program, and the participants to devise a good evaluation plan, and to analyze the data you and others gather.

If you're an independent researcher - a graduate student, an academic, a journalist - you face even greater obstacles. First, you have to find a place to conduct your research - a program to evaluate - that fits in with your research interests. Then, you have to convince the organization running that program to allow you to do the research.Once you've jumped that hurdle, you're still faced with all the same tasks as an outside evaluator: establishing trust, understanding the context, etc.

Let's look first at the process you as an independent researcher might follow in order to choose and gain access to a setting appropriate to your interests. Once you've gained that access, you've become an outside evaluator, so from that point on, the course of preparing for the evaluation will be the same for both.

Choose a setting

If you're an academic or student, you can probably find an appropriate program by asking colleagues, professors, and other researchers at your institution. If none of them knows of one offhand, someone can almost undoubtedly put you in touch with human service agencies and others who will. Other possible sources of information include the Internet, funders, professional associations, health and human service coalitions, and community organizations. Public funding information is often available on the web, in libraries, or in newspaper archives. The wider you spread your net, the more likely you are to find the program you're looking for.

The right program will obviously vary depending on your research interests, but some questions that will inform your choice include:

• Does the setting include people who are actually experiencing the problem that is of interest to you?
• Is the setting similar to others of this type? (If not, its program might not be useful to others dealing with the issue, even if it works well in its own context.)
• Does the setting provide support for the research? Will staff, participants, and others help with data gathering, be forthcoming about context questions, cooperate with you?
• Does the setting have the resources to maintain the program after your evaluation is done?
• Does the setting permit the changes in operation required by the research?  If the planning of the evaluation and choosing of questions point to doing things differently, can and will the program make the necessary changes?
• Is the setting accessible?
• Accessibility includes not only handicap accessibility, but whether a site is in a neighborhood that feels welcoming or safe to participants, whether it is easily reachable by public transportation or on foot from the areas from which participants are drawn, and whether it is in a building or institution that doesn't feel intimidating or strange (a university campus or building can seem as threatening as a fortress to someone who is insecure about his educational background, for example.) Accessibility can be the determining factor in whether participants consider a program, or whether they stay in it.
• Is the setting stable? Are the program and organization stable enough that you know they'll be able to support their work at the current level, at least until the evaluation is completed?

Once you've found an appropriate setting, you'll have to convince the organization to collaborate with you on an evaluation. The next three steps are directed toward that goal.

Learn as much as you can about the organization you've chosen

Just as you wouldn't go to a job interview without doing some research about the employer, you shouldn't try to gain the cooperation of an organization without knowing something about it - its mission, its goals, whom it serves, who the director and board members are, etc. If someone told you about the organization, she may have, or may know someone who has, much of the information you need. If the organization maintains a website, much of that information will be available there. If it's incorporated, the office of the Secretary of the state of incorporation and/or other state offices will have information about the officers (i.e., the Board of Directors) and other aspects of the organization. Funding agencies may also have information that's a matter of public record, including proposals.

Contact the appropriate person(s) and request an interview

Find out whom (by name as well as position) you should talk to about conducting a research project in the organization you've chosen.

Depending on the organization, this could be the board president, the executive director, or the program director (if the program you're interested in is only part of a larger organization).  In any case, it might be wise to involve the program director even if he's not the final decision-maker, since his cooperation will be crucial for the completion of your research.

• If you can, get a personal introduction. It's always best if you come recommended by someone familiar with the person you need to speak with.
• If you can't get a personal introduction, it's usually best to send a letter requesting a meeting and explaining why, and follow it up with a phone call.
• Before the meeting, send a proposal outlining what you want to do. This should be substantive enough to help the organization decide whether it wants to work with you, but not so specific that it doesn't allow for collaborative planning of the evaluation.

Plan and prepare for the initial meeting

There are several purposes for this meeting, besides the ultimate one of getting permission and support for your project (or at least an agreement to continue to discuss the possibility). They include:

• Establishing your credentials - the experience, educational background, and any other factors that equip you to conduct this evaluation. This might include references from colleagues, professors, or other organizations you've worked with.
• Learning more about the program and the organization
• Explaining what you want to do and why, what form the evaluation results are likely to take, what you'll do with them, who'll have access, etc. This explanation should also cover issues of confidentiality and permission of participants.
• Explaining what you need from the organization and/or program - participation of participants and staff, for instance, any logistical support, access to records, or access to program activities
• Explaining what you're offering in return - your services for a comprehensive formal evaluation, any stipends, equipment or materials, other support services, or whatever else you may have to offer
• Clarifying the organization's needs, and discussing how they fit with your own - and how both can be satisfied

Assuming that your presentation has been convincing, and you're now the program evaluator, the rest of the steps here apply to both independent researchers and outside evaluators.

Find out all you can about the context

This may play out differently for outside evaluators than it does for independent researchers, but it's equally important for both. It means finding out all you can about the community, the organization, the program, and the participants beforehand - the social structure of the community and where participants fit in it, the history of the issue in question, how the organization is viewed, relationships among groups and individuals, community politics, etc.

If you're an outside evaluator, you can pick the brains of program administrators, staff, and participants about the community, the organization, and the issue. Ask them to steer you to others - community leaders, officials, longtime residents, clergy, trusted members of particular groups - who can give you their perspectives as well. If possible, get to know the community physically: walk and/or drive around it, visit businesses, parks, restaurants, the library. Understanding how the issue plays out in the community, the nature of relationships among groups and individuals, and what life is like in the neighborhoods where participants live will help a great deal in analyzing the evaluation of the program.

If you're an independent researcher, learn as much about the context as you can before you contact the program. Websites (for the organization and/or the community) and libraries are two possible sources of information, as are community and organization literature and people who know the community.  Learning about the community, the organization, and the participants beforehand will both help you determine whether this program fits with your research and help you advocate for its cooperation with your project. Once you have that cooperation, you can follow the same path as an outside evaluator (since that's what you are) to learn as much about the context of the program as you can.

Establish trust with program administrators, staff, and participants

This can be the most difficult part of an evaluation for someone from outside the organization. There's no magic bullet or predictable timeline, but there are several things you can do:

• Be yourself.  Don't feel you have to act a certain way: deal with people in the program as you do with friends and acquaintances in other circumstances. People can tell when you're being false, and are unlikely to trust you if you are.
• Treat everyone with equal respect, as colleagues in a research project.
• Don't assume you know more than anyone else just because you're the professional.
• Share freely what you do know, but don't tie yourself to any one process or method, especially in response to an opposite stance from a key individual.
• Ask administrators, staff, and participants what they want from the evaluation, and discuss how the evaluation could provide it.
• Don't be afraid to say "I don't know, but I'll find out," and then do.
• Follow through on whatever you say you'll do. Don't promise anything you can't deliver on, and make deadlines reasonable, so you can meet them.

General tips for all evaluators

These steps apply to everyone, internal evaluators as well as external.

Aim for a participatory evaluation

We've discussed above the involvement of all stakeholders to the extent possible. Involving participants, program staff, and other stakeholders in participatory planning and research can often get you the most accurate data, and may give you entry to people and places you normally might not have. On the other hand, participatory planning and research, as we've explained, takes time and energy. If you have limited time, you may not be able to set up a fully participatory project.  You can, however, still consult with stakeholders, and involve them in ways that don't necessarily involve training or large amounts of your time. They can help you line up interviews with participants or other important informants, for instance, and/or act as informants themselves about community conditions and relationships.

At least the people in charge of the program, and probably those implementing it as well, will expect to be part of the planning of the evaluation. They are, after all, the ones who need to know whether their work is effective, and how to improve it. Involving participants as well, in roles ranging from informants about context to actual researchers, is likely to enrich the quantity and quality of the information you can obtain.

Plan the evaluation, in collaboration with stakeholders

That collaboration should be at the highest level of participation possible, given the nature of the program, the time available, and the capacity of those involved (if program participants are five-year-olds, they probably have relatively little to contribute to evaluation planning...but their parents might want to be involved.)

The actual planning involves ten different areas, each of which will be the subject of one of the remaining sections in this chapter:

• Information gathering and synthesis
• Designing an observational system
• Developing and testing a prototype intervention
• Selecting an appropriate experimental design
• Collecting and analyzing data
• Gathering and interpreting ethnographic information
• Collecting and using archival data
• Encouraging participation throughout the research
• Refining the intervention based on the evaluation
• Preparing the evaluation results for dissemination

Once the planning is done, it's time to get started on conducting the evaluation. And when you're finished - having analyzed the information and planned and made the changes that were needed - it's time to start the process again, so that you can determine whether those changes had the effects you intended.  Evaluation, like so much of community work, is a process that goes on as long as the work itself does.  It's absolutely essential to the continued improvement of your program.

In Summary

Choosing evaluation questions - the areas in your work you'll examine as part of your evaluation of your program - is key to defining exactly what it is you're trying to accomplish. For that reason, those questions should be chosen carefully as part of the planning process for the program itself, so that the questions can guide your work as well as your evaluation of it. The more that stakeholders can be involved in that choice and planning, the more likely you are to create a program that successfully meets its goals serving the community.

Choosing those questions well entails understanding the context of the program - the community, participants, the culture of any groups involved, the history of the issue and of the social structure of the community and the organization - and (if you're an outside evaluator without ties to the program) establishing trust with administrators, staff members, and participants. That trust will enable you to conduct a participatory evaluation that draws on the knowledge and talents of all stakeholders, and to plan an evaluation that fits the goals of the program and accurately analyzes its strengths and weaknesses. With that analysis in hand, you'll be able to make changes to improve the program. Then you're ready to start the whole process again, so you can evaluate the effects of the changes you've made.

What do we mean by choosing questions?

___Evaluation questions are the questions your evaluation is meant to answer about your work

___Evaluation questions help set the direction of the work, as well as assess its effectiveness

___Ideally, choosing evaluation questions is part of the planning of the overall program

Questions to ask yourself as you choose evaluation questions:

___What do you want to know?

___Why are you interested?

___Is the issue you’re addressing important to the community or to the society? 

___How does the issue relate to the field?

___Is the issue general, rather than specific to your population or community?

___Who might use the results of your evaluation? 

___Whose issue is it?

Why is it necessary to choose evaluation questions carefully?

___It helps you understand what effects different parts of your effort are having

___It makes you clearly define what it is you’re trying to do

___It shows you where you need to make changes

___It highlights unintended consequences

___It guides your future choices

___In participant evaluations, it involves stakeholders in setting the course of the program, thus making it more likely that it meets community needs

___It provides focus for the evaluation and the program

___It determines what needs to be recorded in order to gather data for evaluation

When should you choose questions and plan the evaluation?

___If possible, choosing questions and planning the evaluation should be an integral part of planning your program

___If your reality makes that impossible, choosing questions and planning the evaluation should take place as soon as possible after the program starts

Who should be involved in the process?

___To the extent possible, the process should involve all stakeholders, including program participants and beneficiaries

How do you choose questions?

___Describe the issue or problem you’re addressing

___Describe the importance of the problem

___Describe those who contribute to the problem

___Assess the importance and feasibility of changing those behaviors

___Describe the change objective

___Make sure that the expected changes would constitute a solution or substantial contribution to the problem

How do you plan the evaluation?

___Take into account the issues raised by multiple or very different settings

___Take into account the issues raised by participant groups that differ in culture, ability to complete the program, geographical location, and other factors

For outside evaluators, specifically:

___Choose a setting

___Learn as much as you can about the organization you’ve chosen

___Contact the appropriate person(s) and request an interview

___Plan and prepare for the initial meeting

For all evaluators:

___Find out all you can about the context

___Establish trust with program administrators, staff, and participants

___Aim for a participatory evaluation

___Plan the evaluation in collaboration with stakeholders

Consider all the elements of an evaluation in your planning:

___Information gathering and synthesis

___Designing an observational system

___Developing and testing a prototype intervention

___Selecting an appropriate experimental design

___Collecting and analyzing data

___Gathering and interpreting ethnographic information

___Collecting and using archival data

___Encouraging participation throughout the research

___Refining the intervention based on the evaluation

___Preparing the evaluation results for dissemination

Example 1: Using Miro Board for Collaborative Development of Evaluation Goals and Questions

 

Noah Goodman shares the process he used to prepare for and conduct a kick-off meeting to develop evaluation goals and questions for the Right Question Institute.

Integrating Technology in Evaluation TIG Week: Using Miro Boards to Engage Clients in Collaborative Development of Evaluation Goals and Questions by Noah Goodman

Suppose you wanted to design a house that used very little energy, required relatively few resources to build and maintain, and was affordable for many families. You might have some original ideas about how this could be done, but you would also want to learn from what others have tried. You might read about earth-bermed houses (houses built partly into a hillside or earth mound), solar panels or small wind turbines for producing electricity, energy-efficient windows and insulation, wastewater recycling, and lower-toxicity building materials that incorporate reclaimed wood or metal. You might talk with people who have built or lived in energy-efficient homes to learn about their experiences. You would also want to understand barriers to environmentally sustainable strategies and approaches communities have used to address those barriers. There is a great deal of information available, and gathering relevant evidence and experience can help you identify useful approaches, adapt them to your circumstances, and develop new ideas based on what has already been learned.

The same is true if you are designing an intervention or program to address a community health or other issue, or an evaluation of that program. Other communities and organizations have likely addressed similar issues, with varying results. Understanding what they did, how they did it, the context in which they worked, and what resulted can help you make informed decisions about your own effort. You may find methods or strategies from different sources that can be adapted and combined to fit the priorities, strengths, and conditions of your community. You may also learn that an approach you were considering has repeatedly produced limited or unintended results and should be reconsidered.

Gathering and using others' ideas does not mean giving up your own ideas or the opportunity to innovate. New approaches often develop by building on previous knowledge, experience, and experimentation. Understanding what has already been tried can help you identify opportunities to adapt, improve, or develop new approaches.

This section focuses on gathering information about a community issue and previous efforts to address it, then bringing that information together to inform program and evaluation design. Although this chapter focuses on evaluation, much of the material in this section also applies to planning the program or intervention itself. Program design and evaluation are closely connected.

An evaluation is a form of systematic inquiry: we are trying to understand what works, for whom, under what circumstances, and why. Many of the steps involved in designing an evaluation are also relevant to designing the program being evaluated. Evidence from previous efforts, community knowledge, and new ideas can help shape both an intervention and the questions used to evaluate it. Although this section discusses program design, the same principles apply to evaluation design.

What do we mean by information gathering and synthesis?

Information gathering refers to collecting relevant information about the issue you are addressing and about how other organizations, communities, researchers, and practitioners have responded to it. The more you understand about the issue, its context, and previous approaches, the better prepared you will be to design an effective and appropriate program or intervention.

Information can come from many sources, depending on what you need to learn. In general, you can consult existing published or documented sources and examine real-world examples of programs and interventions that have addressed similar issues. We will introduce both types here and discuss them in more detail later in the section.

• Existing sources. These include published and documented materials that can help you understand an issue or previous attempts to address it. Sources may include scholarly research, professional publications, government reports, community reports, journalism, organizational materials, and statistical or demographic data. Different sources provide different forms of evidence and perspective, so it is useful to consider their purpose, quality, methods, limitations, and relevance to your context.
• Real-world examples. These are programs, policies, practices, or interventions that communities and organizations have actually implemented. Studying them can help you understand what worked, what did not, under what conditions, and why. Information may come from people directly involved in these efforts, program documentation, evaluations, case studies, community reports, or research literature in fields such as community psychology and public health.

Synthesis means bringing together information from multiple sources to form a coherent understanding or approach.

In this section, synthesis refers to analyzing what you have learned through information gathering and combining relevant evidence, community knowledge, practical experience, and ideas into an approach that fits the priorities and context of the community you are working with.

Synthesis requires identifying the functional elements of the programs, policies, or ideas you examine. Functional elements are the core components—such as methods, frameworks, activities, strategies, relationships, policies, and implementation practices—that contribute to how an approach works. Once you understand these elements, you can determine which may be relevant to your community, which may need adaptation, and how they might fit with community knowledge and locally developed ideas.

Information gathering and synthesis are important both when developing a program and when designing an evaluation that can help strengthen it. Both begin with understanding the issue, identifying the people and communities involved, considering the conditions that influence it, and developing an approach for creating change. These decisions help define short- and long-term goals and, in turn, shape the evaluation questions. A meaningful evaluation depends on clearly understanding what the program is intended to accomplish and how it is expected to contribute to change.

Why gather and synthesize information?

If you are developing a program to address a community issue, such as violence prevention or early childhood education, you may already have substantial knowledge and experience related to that issue. Community members, practitioners, and other stakeholders may also have strong ideas about what could work. Why take additional time to review research, examine other programs, and speak with people involved in similar efforts? Because gathering and synthesizing information can help you build on existing knowledge, avoid preventable problems, understand your local context more fully, and strengthen the overall design of your program.

• It can help you avoid repeating avoidable mistakes. Other organizations and communities have likely addressed similar issues. Some efforts may have been effective, some may have produced mixed results, and some may not have worked as intended. Each can provide useful lessons. Understanding previous efforts can help you build on promising approaches while avoiding strategies that have repeatedly created problems or failed to achieve their goals.

Having an idea similar to one that has already been tried is not a problem. In fact, learning that an approach has worked elsewhere may strengthen the case for considering it. At the same time, discovering that a similar approach has repeatedly struggled can help you examine why and decide whether it should be adapted, replaced, or avoided. Learning from previous experience can save time and resources while improving the likelihood that your program will be useful and sustainable.

• It can help you develop a deeper understanding of the issue. Effective action starts with understanding what you are addressing. This may include immediate and underlying causes, historical and structural conditions, community strengths and resources, how people experience the issue, its consequences, and who or what has the ability to influence change.
• It gives you more tools for designing an effective program. Information is one of the most important resources for planning and implementation. Understanding what has worked, for whom, under what conditions, and why can help you select strategies that address multiple dimensions of an issue rather than relying on assumptions or a single approach.
• Most approaches are not one-size-fits-all. The more information you gather, the more approaches, methods, and frameworks you can consider. The goal is not simply to copy another program, but to identify elements that can be appropriately adapted to your community's context, priorities, strengths, and needs.
• It can help you design culturally responsive approaches. Information gathering can help you better understand the histories, cultures, values, languages, strengths, and priorities of the communities involved. It can also help identify approaches that have been used in similar contexts. This information should complement—not replace—direct engagement with community members and their knowledge about what is appropriate and meaningful in their own community.
• Learning from different settings and perspectives can generate new insights and ideas. New approaches often emerge from combining existing knowledge in new ways. Looking across fields, communities, cultures, and geographic settings can expose you to different ways of understanding an issue and expand the range of possible strategies. Those ideas can then be examined and adapted with community members to determine whether they are relevant locally.

When should you gather and synthesize information?

Information gathering and synthesis are important to both program effectiveness and meaningful evaluation. They should begin early in the planning process and continue throughout the life of the program. Ongoing learning can help you identify emerging evidence, changing community conditions, new opportunities, and possible improvements.

Major changes may be easier to assess when they occur after an evaluation cycle, when you have enough information to understand what was working and what was not. At the same time, smaller adjustments may be necessary during implementation to respond to participant feedback, accessibility concerns, emerging risks, or changing circumstances.

There can be tension between continuously adapting a program and determining whether a particular approach is effective. If a method or activity changes substantially during an evaluation period, it may become more difficult to determine which version of the approach contributed to the results.

How much adaptation is appropriate depends partly on the purpose of the program and evaluation. If the primary purpose is to test whether a specific approach works as designed, maintaining consistency may be particularly important. If the primary responsibility is to respond to the needs and well-being of current participants and communities, changes may need to be made as new information becomes available. When changes are made, documenting what changed and why can help evaluators interpret the results.

There can also be ethical issues involved. In research involving new medical treatments, for example, participants may be assigned to different study groups under an approved research protocol and informed-consent process. If evidence shows that an intervention is causing harm, researchers have an obligation to respond appropriately. Likewise, strong evidence that one approach provides substantial benefit may raise ethical questions about continuing to withhold it from other participants. Community programs may raise different ethical considerations, but potential benefits, harms, fairness, informed participation, and participant well-being should always be considered.

Who should gather and synthesize information?

Throughout this chapter, we assume that planning, design, implementation, and evaluation involve multiple stakeholders and forms of knowledge.

Stakeholders in a community program or intervention might include:

• Program participants and community members directly affected by the issue
• Program staff, volunteers, and administrators
• Others whose work or responsibilities intersect with the issue, such as health professionals, educators, public safety personnel, social service providers, or community-based organizations
• Researchers and subject-matter experts
• Local officials and policymakers
• Community advocates, organizers, and leaders

Information gathering

In a participatory process, information gathering can be strengthened by sharing responsibilities according to participants' knowledge, interests, skills, relationships, and opportunities to contribute. Researchers or others with experience reviewing academic literature may take the lead on some evidence reviews. Community members may bring important knowledge about the history of the issue, how it is currently experienced, and which sources or perspectives are most relevant locally. Program staff may have connections with organizations doing similar work. Participants with experience using databases, online research tools, policy analysis, or other methods may support those areas of the work. Roles should be determined collaboratively rather than assigned according to assumptions about people's backgrounds.

Training and support can also expand who is able to participate in different parts of the research process. People who are new to research can learn to interpret demographic information, review program materials, conduct interviews, contact organizations, analyze policy, or use research tools. The goal should be to identify what support, technology, accessibility accommodations, language access, or training participants need to contribute in ways that are meaningful to them.

Synthesis

It is especially important for people with different perspectives to participate in interpreting and synthesizing the information. Participants who bring lived experience or deep community knowledge may recognize patterns, assumptions, or implications that are less visible to professional researchers. Researchers and practitioners may contribute methodological or subject-matter expertise. Bringing these perspectives together can improve both interpretation and decision-making.

Information gathering and synthesis, like the rest of a participatory process, should reflect the priorities, knowledge, interests, strengths, and accessibility needs of stakeholders.

How do you gather and synthesize information?

Gathering and synthesizing information involves several steps. Many can be carried out collaboratively as part of a participatory process, while particular research tasks may be shared among individuals or smaller groups.

Decide what you need to know

The first step in gathering information is determining what you need to understand. Areas to explore may include:

• Details about the issue. These may include immediate and underlying causes; historical, social, economic, organizational, environmental, or policy conditions; effects on individuals and communities; how the issue has changed over time; community strengths and protective factors; and previous efforts to address it.
• How the issue has been addressed elsewhere. This might include evidence-informed or evidence-based practices, promising approaches, community-developed strategies, adaptations used in different settings, and approaches that produced limited or unintended results. Learning why an effort was unsuccessful can be just as valuable as learning why another succeeded.
• People and organizations who can contribute knowledge or support. This may include people with lived experience, community organizations, practitioners, researchers, evaluators, policymakers, advocates, and people involved in previous efforts to address the issue.
• Who is affected locally, and how. Consider which communities or groups experience the issue most directly and whether impacts differ by geography, race, ethnicity, culture, income, disability, age, gender, or other relevant characteristics. Also consider people who may be affected less visibly, including family members, caregivers, service providers, educators, employers, or others connected to those directly affected.
• The importance of the issue to the community. This includes at least two questions:
  • How important do community members consider the issue to be?
  • How and to what extent does the issue affect the community as a whole?
• Community priorities, strengths, and needs related to the issue. What resources, supports, opportunities, relationships, services, policies, or environmental changes could improve the situation? What existing community strengths can be built upon? Which approaches do community members consider acceptable, relevant, and feasible?
• Other contextual information. This may include community history, relationships among groups and organizations, prior experiences with institutions or programs, culture, language, economic conditions, transportation, geography, policy, and other factors relevant to the work.
• Who or what can influence change. Depending on the issue, this may include public officials, policymakers, businesses, property owners, government agencies, schools, employers, health systems, community organizations, funders, service providers, and people directly affected by the issue. It may also include policies, organizational practices, resource allocation, institutional systems, and environmental conditions.

Determine your likely information sources

As discussed above, information can come from published or documented sources as well as real-world experience. Published sources may include scholarly research, professional publications, government and community reports, journalism, organizational documents, and statistical information. Depending on your questions, you may use several types of sources to understand both the evidence and the community context.

The Internet provides access to a vast range of information, including academic research, government data, laws and regulations, program evaluations, organizational reports, community resources, statistical databases, professional guidance, and descriptions of promising practices. Libraries, government databases, university repositories, research organizations, and organizational websites can all be useful starting points.

Online information varies widely in quality. Consider who produced the information, what evidence it uses, when it was published or updated, what methods were used, whether potential conflicts of interest are disclosed, and whether other credible sources support its claims. General-reference websites, social media, discussion forums, and other informal sources can help identify questions, terminology, or additional sources, but important decisions should be supported by information whose credibility and relevance you can evaluate.

Existing sources

Scholarly and professional sources might include:

• Peer-reviewed academic and professional journals
• Academic and professional books
• Doctoral dissertations and master's theses available through universities, libraries, and research repositories
• Papers, posters, reports, and proceedings from academic and professional conferences
• Evidence reviews, professional publications, and reputable science and research journalism
• Newspaper and journalism archives
• Direct communication with researchers, evaluators, practitioners, and others with relevant expertise
• Professional networks, research communities, mailing lists, webinars, and other field-specific forums

Public-facing sources of information might include:

• Books written for general audiences
• Articles in reputable magazines, news outlets, and science or health publications
• Investigative reporting, explanatory journalism, podcasts, documentaries, and other media that provide useful context or identify additional sources

Statistical and demographic information may be available from:

• Census and other government data sources
• Community reports, community health assessments, community report cards, strategic plans, and needs or strengths assessments
• Organizational, agency, school, health system, and administrative data, when those data are available and appropriate to use

News and other media can also call attention to community issues, emerging research, or promising programs. Media coverage usually provides only part of the picture, so it is often most useful as a starting point for identifying additional sources, organizations, researchers, or questions to investigate.

Real-world examples

Potential sources of information about real-world programs and approaches include:

• Program directors, coordinators, and staff
• Colleagues and professional contacts
• Funders and public agencies
• Leaders and members of community coalitions, partnerships, and networks
• Officials or organizations coordinating community-wide efforts
• Community members directly affected by the issue
• Current or former program participants
• People whose work intersects with the program, such as educators, health professionals, social service providers, public safety personnel, or other community partners
• Community members with relevant knowledge, experience, or relationships
• Researchers, practitioners, evaluators, and other people with relevant expertise
• Your own experience and observations in the community

Look broadly when searching for useful models or new ideas. Approaches developed in another field, community, or geographic region may offer concepts that can be adapted to your work. Public health, education, social services, community development, urban planning, behavioral health, environmental health, and other fields often address overlapping social and community conditions. The important question is not whether an approach comes from the same field, but whether its underlying principles and components are relevant and adaptable to your context.

Devise a plan for collecting information

Consider the following questions:

• Who will gather what information? A strong information-gathering team includes different forms of knowledge, experience, relationships, and skills. Responsibilities can be shared according to participants' interests and strengths while also creating opportunities for people to learn new skills. Researchers may contribute experience reviewing academic literature, for example, while community members may have knowledge and relationships that are essential for understanding local history, priorities, and experience. These roles should remain flexible and should not be based on assumptions about who is capable of doing particular kinds of research.
• How will the information be gathered? Written and online sources may be reviewed through libraries, databases, organizational records, and websites. Information from people may be gathered through formal or informal interviews, observation, surveys, public meetings or community conversations, focus groups, participatory methods, or other approaches. Methods should reflect the questions being asked, the time and resources available, participant preferences, accessibility, privacy, and the depth of information needed.
• What support will participants need? People may have different levels of experience with research, technology, academic writing, data interpretation, or particular languages. The group may need interpretation or translation, accessible formats, assistive technology, transportation, childcare, technology access, research training, or support interpreting specialized materials. Identify these needs collaboratively and create ways for everyone to participate meaningfully.
• What is the timeline for information gathering? Learning should continue throughout the life of a project, but the initial information-gathering phase usually needs a clear timeframe so that planning does not indefinitely delay action. The timeline should reflect the urgency and complexity of the issue, community priorities, available resources, external deadlines, and the amount of information needed to make informed decisions.

Collect information

Once the plan is established, begin gathering information. Provide any training, tools, accessibility supports, or resources participants need, and make sure responsibilities are clear. Regular check-ins during the process can help participants share what they are learning, identify gaps, support one another, adjust the research plan, and begin interpreting emerging findings. These conversations can also make synthesis more manageable by allowing the group to process information throughout the research process rather than all at once.

Synthesize: Take it all apart

Synthesis begins by breaking information into useful components, examining how those pieces relate to one another, and identifying which ideas, strategies, and conditions are most relevant to your community.

Three major areas are usually important to consider:

• What is known about the issue itself. What individual, interpersonal, organizational, community, environmental, policy, and structural factors influence the issue? What are its immediate and underlying causes? What strengths or protective factors already exist? Which factors are within the scope of your effort, and which require broader partnerships or systems-level change? What are the potential consequences for individuals and the community if the issue is not addressed?
• The community context of the issue. Who is affected, and in what ways? Are some groups affected differently or disproportionately? How does the issue affect families, relationships, organizations, neighborhoods, and the wider community? What has the community's experience with the issue been? What previous efforts have been attempted? What local strengths and resources can support change? Are there policies, systems, practices, relationships, or environmental conditions that also need to change?
• Previous efforts to address the issue. Information may come from research literature, evaluations, community knowledge, case studies, program documentation, and people directly involved in those efforts. Examine the specific components of each approach. What strategies were used? What training or support was provided? What information and resources were available to participants? What organizational, environmental, or policy changes were involved? What assumptions or principles shaped the approach? Which aspects of the issue did it address? In what kinds of communities and contexts was it used? Who participated? Who benefited, and who may have been left out? What were the short- and long-term outcomes? What conditions contributed to success, limited effectiveness, or unintended consequences?

An unsuccessful program does not necessarily mean that every element of the program was ineffective. Some components may have been useful while others created barriers or were based on incorrect assumptions. A promising strategy may also fail because it was poorly implemented, lacked resources, was not culturally or contextually appropriate, excluded important stakeholders, or was introduced under conditions that made success unlikely. Understanding why an effort produced its results is often more useful than simply labeling it successful or unsuccessful.

It is also important to distinguish between identifying an approach that appears to work and understanding the conditions under which it works. Leadership, staffing, relationships, funding, community trust, organizational capacity, implementation quality, policy context, and other factors can strongly influence outcomes.

When examining programs for ideas, look beyond the program name or overall model. Identify the specific components, relationships, implementation practices, and contextual conditions that appear to contribute to its results.

Synthesize: Put it back together

Analyze what you have learned to determine which ideas and components are appropriate for your community and how they might fit together.

• What approaches have been used with communities or groups whose circumstances are similar to yours? Were successful approaches highly specific to their original setting, or do their underlying principles appear adaptable?
• Which components could be adapted for your context while preserving the elements that make them effective?
• What is missing from the approaches you reviewed? Are important community priorities, strengths, populations, barriers, or structural conditions not being addressed?
• Does the information confirm, challenge, or complicate what you previously believed about the issue?
• What features of your community or situation differ from the settings in which other approaches were developed? How should those differences affect adaptation and implementation?
• What does the information suggest about addressing underlying conditions such as economic inequity, social exclusion, unequal access to resources, or limited decision-making power?
• At what levels did successful efforts focus their strategies—individual, interpersonal, organizational, community, policy, or systems? Which combination is most relevant to your context?
• Perhaps most importantly, how does your community define success? Which approaches came closest to producing the outcomes that stakeholders value, and which components appear most relevant to the priorities and conditions you identified?

Answering these questions can help you identify components that may be appropriate for your community, determine what should be adapted, generate new ideas, and recognize approaches that may not fit your goals or context.

Synthesis is not simply a matter of combining unrelated program components. Begin with a clear framework informed by your vision, mission, community priorities, evidence, and understanding of how change is expected to occur. The components you select should reinforce one another and contribute to a coherent strategy. When elements conflict or are combined without understanding how they function, the resulting program may be difficult to implement or evaluate effectively.

Keep at it

Information gathering and knowledge synthesis should continue throughout the program. New evidence may emerge, community conditions may change, participants may identify new concerns or strengths, and additional approaches may become available. Even when major program changes are postponed until an evaluation cycle is complete, continuing to learn helps prepare the program for thoughtful adaptation. Ongoing reflection and learning can help keep the work responsive, relevant, and effective.

In Summary

Gathering existing information about a community issue and previous efforts to address it is an important part of planning both programs and evaluations. Bringing together research evidence, community knowledge, lived experience, lessons from previous efforts, and local context can help you design an approach that builds on what has been learned while creating room for local adaptation and innovation.

Information gathering and synthesis should continue throughout the life of the program. Ongoing learning can help communities and organizations understand changing conditions, strengthen promising approaches, recognize ineffective or harmful strategies, and adapt the work in ways that remain responsive to community priorities and evidence.

What do we mean by information gathering and synthesis?

___Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it

___You can gather information using both existing sources and natural examples

­­___Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something that meets the needs of the community and population you’re working with

___Synthesis involves extracting the functional elements of both the analysis of the issue and approaches to it

___Functional elements are those that are indispensable either to understanding the issue, or to implementing a particular program

Why gather and synthesize information?

___It will help you avoid reinventing the wheel

___It will help you to gain a deep understanding of the issue so that you can address it properly

___You need all the tools possible to create the best program you can

___It’s likely that most solutions aren’t one size fits all

___It can help ensure your program is culturally sensitive

___Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program

When should you gather and synthesize information?

___Information gathering and synthesis should continue throughout the life of the program

Who should gather and synthesize information?

___Information gathering and synthesis is often most effectively conducted by a multi-sectoral participatory group including all stakeholders in the issue

How do you gather and synthesize information?

___Decide what you need to know about the issue itself, successful and unsuccessful attempts to address it in various circumstances, and the local context

___Determine your likely sources for the various types of information you’re seeking

• Existing sources include scholarly, mass-market, and statistical/demographic published information
• Natural sources include some published information about programs, but can best be obtained by direct contact with those involved in planning, implementing, or participating in programs relevant to your issue
• It’s important to pay attention to both successful and unsuccessful attempts to address the issue, and to step outside your own field in search of solutions that work

___Devise a plan for gathering information

• Decide who will gather what information
• Decide how information will be gathered
• Decide what adjustments will be made for gaps in experience or skills
• Set a timeline for the initial information gathering

___Collect information

___Begin synthesis by taking it all apart – extract the functional elements of what you’ve learned

___Complete synthesis by putting the relevant pieces back together as a coherent program that speaks to your community’s needs

___Keep at it by continuing to gather and synthesize information throughout the life of the program

• What do we mean by an observational system?

• Why design an observational system?

• When should you design an observational system?

• Who should design an observational system?

• How do you design an observational system?

A local community health center was starting a program to support regular physical activity among people with high blood pressure. The program had one main objective: to help participants engage in about 45 minutes of moderate aerobic activity at least four times a week over the course of six months. The hope was that regular physical activity would help participants manage their blood pressure, improve cardiovascular health, and support their overall well-being. A related goal was for participants to identify forms of physical activity they enjoyed and could continue after the formal program ended.

The center recruited 50 people who wanted to participate. After appropriate health assessments, participants attended workshops about nutrition, high blood pressure, physical activity, injury prevention, and ways to incorporate movement into daily life. They also discussed different forms of activity—such as walking, bicycling, swimming, or other activities appropriate to their interests and abilities. Rather than requiring everyone to use the same exercise routine, the center encouraged participants to choose activities that worked for them while working toward the overall activity goal. Participants were asked to keep activity logs and to meet in small groups with health center staff once a month to review blood pressure, discuss progress and barriers, and receive support and guidance.

The center then had to decide how to evaluate the program. One important question was whether participants were able to maintain their physical activity routines over the six-month period. Because participants exercised independently and at different times and places, staff could not directly observe every activity session. The center also wanted to understand whether participants experienced changes in blood pressure, well-being, physical activity habits, and other relevant health indicators. How could the center design a system to gather accurate, respectful, and useful information about what participants were doing and what changes were occurring?

Once you have identified your evaluation questions and determined what information you need, you need a systematic way to collect that information. That is the purpose of an observational system. Like the center in the example, you may need several ways to examine behaviors, experiences, conditions, program activities, and changes over time. This section focuses on designing observational systems that provide useful information while respecting participants' privacy, dignity, and rights.

What do we mean by designing an observational system?

An observational system is a structured way of gathering information about a program—what is being implemented, how people are participating, what conditions are present, and what changes appear to be occurring. "Observation" may involve directly watching activities or conditions, but it can also include other systematic ways of documenting program processes and outcomes. The appropriate methods depend on the evaluation questions, the setting, the people involved, and ethical and privacy considerations.

• Direct observation. Direct observation involves systematically observing activities, behaviors, settings, or conditions firsthand. For example, if an initiative aims to increase use of a public park, observers might document how the park is used at different times of day, during different weather conditions, or during special events. Observation in public settings should still be planned carefully, with attention to privacy, research ethics, and whether consent or public notice is appropriate. In program settings, participants should generally understand when and why observation is taking place.
• Participant observation. A participant observer takes part in the setting or activity while also documenting what they experience and observe. In the park example, a neighborhood resident involved in the initiative might document patterns of park use while participating in everyday activities there. Participant observation can provide important contextual information because the observer experiences the setting from within it, but observers should be transparent about their role when appropriate and follow relevant ethical and privacy guidelines.
• Self-reports. Some experiences and behaviors cannot or should not be directly observed. People may be asked to describe their own experiences through interviews, journals, surveys, activity logs, questionnaires, or other forms of first-person reporting. Self-reports can provide information that is unavailable through external observation, including people's motivations, experiences, perceptions, and activities outside a program setting. Like all methods, self-reporting has limitations, so it may be useful to combine it with other appropriate sources of information when possible.
• Reports from others. An observational system may also include information from people who have relevant knowledge of the conditions or activities being evaluated. Depending on the program, these might include educators, health professionals, family members, community partners, service providers, or other people with direct and appropriate knowledge. Information should be collected in ways that protect confidentiality and respect participants' privacy and consent.
• Electronic or technology-assisted observation. Devices such as activity trackers, environmental sensors, cameras, audio recorders, or health-monitoring equipment can sometimes provide useful information. These tools should be used only when appropriate to the evaluation and with careful attention to informed consent, privacy, data security, accessibility, and how collected information will be stored and used. The fact that a technology can collect information does not necessarily mean that collecting it is appropriate.
• Tests and assessments. Depending on what is being evaluated, assessments may include academic measures, skills demonstrations, health screenings, validated questionnaires, or other tools. Assessments should be appropriate for the population and purpose, and their limitations should be considered when interpreting results.
• Public and administrative records. Census information, public health data, employment statistics, school or organizational records, and other administrative data may provide information about community-level conditions or outcomes. Evaluators should consider data quality, privacy requirements, who is represented or missing from the data, and whether the information is appropriate for the questions being asked.
• Products or results of behavior. Sometimes it is more practical or appropriate to examine the results of an activity rather than observing the activity itself. For example, an environmental initiative might measure changes in water quality, litter, or air pollution rather than attempting to observe every behavior that contributes to those conditions. A school-based health initiative might examine participation, student-reported well-being, access to healthy meals or physical activity, or other appropriate indicators rather than relying on body size as a primary measure of success.

In addition to deciding which methods to use, an observational system should specify when, where, how often, by whom, and under what circumstances information will be collected. These decisions should be guided by the evaluation questions and by ethical, practical, cultural, and accessibility considerations.

Consider whether you want to examine the process of the effort—how the program was planned and organized and whether implementation matched the intended approach. You may also want to examine what was actually delivered, who participated, how long activities lasted, what adaptations were made, and which parts participants found useful. Finally, consider which outcomes or changes you want to understand and how those outcomes can be measured responsibly.

Designing an observational system requires thinking carefully about what you need to know and selecting methods that can provide that information accurately, consistently, ethically, and feasibly. We will discuss this process in more detail below.

Why design an observational system?

If you're serious about evaluation, there are several reasons to design a strong observational system:

• It can help you collect reliable information. A system that clearly defines methods, timing, procedures, and responsibilities makes information collected across observers, settings, and time periods more consistent and useful.
• It can help you collect the information you actually need. A well-designed system focuses data collection on the evaluation questions rather than gathering large amounts of information without a clear purpose.
• It can make consistent data collection more likely. When the people collecting information understand and agree on the procedures, observations are more likely to happen when and how they are intended.
• It can make analysis easier. Consistent observations can support both quantitative analysis, which examines numerical patterns, and qualitative analysis, which examines experiences, meaning, context, and interpretation.
• It can help you avoid fragmented evaluation. A structured system helps connect the information you collect directly to the questions you are trying to answer.
• It can strengthen confidence in your findings. Clearly documented and consistently applied methods make it easier to explain how conclusions were reached and to identify the limitations of the evidence.
• It can strengthen accountability with communities, funders, and policymakers. A thoughtful evaluation based on systematically collected information can help stakeholders understand what the program accomplished, what remains uncertain, and what should happen next.
• It can help you share promising practices responsibly. Strong evidence can help you explain which program components appear effective, for whom, and under what conditions, rather than assuming that an approach that worked in one setting will automatically work everywhere.
• It can provide useful information about what is working, what is not, and what should be adjusted.

When should you design an observational system?

An observational system includes the methods used to examine and document program processes, activities, experiences, and outcomes. Because it is central to evaluation, the ideal time to design the system is before implementation begins. Doing so allows you to establish baseline information and monitor the program from the start.

In practice, many community organizations begin formal evaluation after a program is already underway because of limited time, staffing, funding, or evaluation capacity. Whenever evaluation begins, the observational system should be designed around the questions you are trying to answer and the information that can still be gathered accurately. It is worth taking the time to develop a system that is realistic, ethical, and useful.

Whenever possible, observations should cover a full program cycle from beginning to end. Some programs do not operate in fixed cycles, in which case observation may focus on individual participants, particular activities, implementation periods, or changing community conditions.

Records created before a formal evaluation begins—such as program notes, activity logs, meeting records, or participant feedback—may provide useful information. Before including them, determine whether they were collected consistently enough and for purposes compatible with the current evaluation. Information created for one purpose should not automatically be treated as equivalent to data collected under a defined observational protocol.

Beginning observation late may also mean that important early changes or implementation decisions have already occurred. When feasible, begin collecting relevant information early enough to establish a baseline and understand how the program develops over time.

Who should design an observational system?

The Community Tool Box emphasizes participatory research and evaluation. An observational system is generally stronger when the people who will collect information, the people whose experiences are being represented, and those who will use the findings have meaningful opportunities to contribute to its design.

In smaller community-based organizations, staff and volunteers may already have multiple responsibilities. The observational system should therefore be realistic enough to carry out with available time, staffing, technology, and other resources. Involving the people responsible for implementation can help identify unnecessary burdens and make the system more practical.

The design process should include people who will actually collect the information, along with people who understand the program and the communities involved. Researchers or evaluators can contribute methodological expertise, while program participants and community members can help ensure that the system reflects community priorities, cultural context, accessibility needs, privacy concerns, and the realities of participation. The design team may therefore include members of the broader evaluation group as well as people specifically recruited to help develop the observational system.

The group might include:

• Program staff, volunteers, and administrators
• Staff responsible for records, data management, technology, or administrative support
• External evaluators, researchers, or research consultants
• Program participants and community members affected by the work
• Community partners or trained volunteer observers

If the design group does not include anyone with evaluation or research experience, participants may benefit from training or technical assistance on observational methods, ethics, privacy, measurement, and the kinds of information different methods can provide. If experienced researchers are involved, they can share this knowledge as part of a collaborative design process rather than assuming that methodological expertise should determine the system without community input.

How do you design an observational system?

Once you have developed your evaluation questions and overall evaluation plan, the next step is deciding how you will gather the information needed to answer those questions.

Review your evaluation questions

Return to what you originally wanted to understand about the program. In the health center example, the program was designed to support regular physical activity among people with high blood pressure, with participants working toward approximately 45 minutes of moderate activity four times a week over six months.

The program hoped to contribute to several outcomes:

• Participants would experience improved blood pressure management
• Participants would increase or maintain regular physical activity in ways appropriate to their health, abilities, and preferences
• Participants would report improvements in overall well-being
• Participants would continue forms of physical activity they found sustainable after the six-month program ended

Some evaluation questions might therefore include:

• To what extent were participants able to engage in their planned physical activity during the program?
• How did participants' blood pressure change over the six-month period?
• What changes did participants report in physical functioning, well-being, or confidence in maintaining regular activity?

Other questions might include:

• How well attended were the workshops? Did participants find them useful, accessible, and relevant? Were particular program components associated with stronger engagement or outcomes?
• Did participants report changes in their overall well-being by the end of six months?
• Did participants continue engaging in regular physical activity after the formal program ended, and were improvements in blood pressure or well-being sustained?

These may represent only some of the information the center wants. Your own evaluation may also examine planning, implementation, participant experience, accessibility, equity, timelines, adaptations, and intermediate benchmarks. Each of these questions should inform the design of the observational system.

Decide what you need to observe to answer your questions

The kinds of information you need will depend on the program, community context, and evaluation questions.

Common areas include:

• Participants' behavior or activities. This might include participation in physical activity, demonstrated skills in an employment-training program, use of community resources, or interactions in a program setting. Behaviors should be defined carefully and observed in ways that respect participants' privacy and dignity.
• The behavior or practices of others involved. An evaluation might examine how staff interact with participants, how institutions implement a policy, or how other groups respond to an intervention. Examining organizational and staff practices can be particularly important when program outcomes depend on more than participant behavior alone.
• Conditions. An initiative may aim to change physical, social, environmental, organizational, or policy conditions—for example, improving affordable housing, restoring a polluted waterway, increasing accessibility, or changing institutional practices.
• Products or results of behavior or conditions. When an activity cannot or should not be observed directly, you may need to measure relevant outcomes or indicators. For example, a sexual health program might examine rates of sexually transmitted infections, access to preventive services, contraceptive use reported confidentially by participants, or other appropriate indicators rather than attempting to observe private behavior directly.

When you rely on indirect indicators, make sure they are meaningfully connected to the behavior or condition you are interested in. Consider other factors that could influence the same outcome, and avoid treating one indicator as definitive evidence when multiple explanations are possible.

• Participants' knowledge, perceptions, or attitudes. These might be assessed through interviews, surveys, validated scales, knowledge assessments, or other appropriate methods.
• The knowledge, perceptions, or attitudes of other groups. An advocacy initiative, for example, might examine changes in public understanding, decision-maker awareness, organizational practices, or community support for a policy.
• Goal attainment. Some efforts focus on a specific outcome, such as adoption of a policy, creation of a community resource, or completion of an infrastructure project. Even when the final goal is clear, it can still be useful to document intermediate progress, relationships, barriers, and lessons learned.
• Interactions. An evaluation may focus on how people communicate, collaborate, share decisions, respond to one another, or experience relationships within a program. For example, a family-support effort might examine patterns of communication and engagement rather than simply counting interactions.

These areas may relate either to program outcomes—what the program is intended to accomplish—or to process and implementation—how the work is planned and carried out.

Areas related specifically to program process and implementation may include:

• Planning. Who participated in planning? How were decisions made? Whose perspectives were included or missing? How well did the process reflect community priorities?
• Timeline. When did planning, implementation, evaluation, and follow-up begin? Were timelines realistic? If delays occurred, what contributed to them?
• Participation and engagement. How many people participated? How long did they remain involved? Were there patterns in who participated, who did not, or who stopped participating? What barriers or supports affected engagement?
• Methods. What strategies, activities, or methods were used? Were they implemented as intended, and what adaptations were made?
• Program implementation. What activities actually occurred? How often and for how long? Who participated? Where did activities take place? What resources were used? What changed during implementation, and why?

Whatever you choose to observe, define it carefully so that people collecting information understand what counts and what does not. Clear definitions, examples, and boundaries improve consistency and reduce the likelihood that different observers will interpret the same event in substantially different ways.

Returning to the health center example, the center might use activity logs or confidential self-reports to understand how often participants engaged in physical activity. Blood pressure could be measured using appropriate clinical procedures. Participants could report changes in well-being, energy, confidence, or barriers to activity through surveys or interviews. Follow-up interviews or activity logs could help determine whether participants continued physical activity after the program ended.

Decide how the observations will be conducted

Earlier, we introduced several methods of observation. Each has strengths, limitations, and ethical considerations.

• Direct observation. Direct observation may involve an evaluator, staff member, trained community observer, or other designated person observing defined activities or conditions. In program settings, participants should generally know that observation is occurring and understand its purpose. Observation of public settings may sometimes be conducted without identifying individual people, but privacy, ethics, local expectations, and applicable requirements should still guide the design.

Journals and activity logs can also support direct or participant observation. Observers may record what occurred, the context, relevant events, and their interpretations soon after an activity. When several people document the same period or activity, these records can provide multiple perspectives on how the program unfolded.

The format of journals or logs should fit the program and the people using them. Written journals may work well in some settings, while voice notes, structured forms, digital logs, or other accessible formats may work better in others.

• Participant observation. Participant observers take part in the setting or activity while documenting what they experience. They may be community members, program participants, staff, or evaluators who participate openly in activities. Their position can provide valuable insight into relationships, context, and experiences that may not be visible to an external observer.

For example, a small-grant program designed to support locally owned businesses might involve staff and community participants in workshops on budgeting, lending, business planning, and peer support. Staff and participants could reflect together on how the activities worked, which barriers emerged, and what participants found useful. In this setting, participant observation could contribute to learning while treating participants as partners in interpreting their own experiences rather than simply as subjects of observation.

• Self-reports. When relevant behaviors or experiences occur outside the program, participants may be the most appropriate people to describe them. Self-reports provide direct access to people's experiences, but responses can be influenced by memory, question wording, social expectations, concerns about privacy, or what participants believe the evaluator wants to hear. Clear questions, confidentiality protections, respectful data collection, and multiple sources of evidence can improve the usefulness of self-reported information.

Participants may benefit from clear guidance about what information is useful to record, especially when completing activity logs or journals. At the same time, evaluators should avoid coaching people toward preferred responses. Self-reporting is especially valuable because it allows participants to describe experiences, motivations, barriers, and changes that an outside observer may not be able to see.

Self-reports can be collected through individual and group interviews, focus groups, community meetings, surveys and questionnaires, journals, checklists, activity logs, or other ways for participants to share information.

• Reports from others. These may come from people who interact with participants or have relevant knowledge of the conditions being evaluated, such as educators, health professionals, service providers, family members, employers, or community partners. Consider whether the person has appropriate knowledge, whether participants have consented when necessary, and how the reporter's relationship, expectations, or biases may influence what they report.
• Electronic or technology-assisted observation. Technology can sometimes collect information that would be difficult to obtain otherwise. Examples include environmental sensors, activity monitors, clinical equipment, or traffic counters. Cameras, audio recording, location tracking, and other potentially intrusive technologies require especially careful consideration of informed consent, privacy, data minimization, data security, and who will have access to the information. Technology-generated data may appear objective, but equipment settings, algorithms, measurement error, and interpretation can still introduce limitations or bias.
• Tests or other assessment tools. Education, health, and human service organizations may use assessments to understand skills, knowledge, development, health status, or other outcomes. Whenever possible, use tools that are valid and appropriate for the population and purpose. Interpretation should also consider factors such as language, disability, culture, anxiety, fatigue, access to preparation, and testing conditions.
• Public and administrative records. Community-level indicators such as injury rates, health outcomes, school attendance, housing conditions, or transportation patterns may be available through government agencies, institutions, or other administrative systems. Consider the quality, timeliness, completeness, privacy protections, and limitations of these data before using them.

In the health center example, blood pressure could be measured at regular intervals using appropriate clinical equipment and procedures. Participants' physical activity could be documented through self-reported activity logs, interviews, or optional activity-tracking tools if participants choose to use them. Well-being could be assessed through participant reports using appropriate questions or validated measures.

Follow-up interviews, surveys, activity logs, or other appropriate methods could help determine whether participants continued regular physical activity after the program ended. Combining several types of information would provide a more complete picture than relying on any single measure.

Decide when you need to observe

Consider when information should first be collected and how often observations should occur. In many evaluations, collecting baseline information before or at the beginning of the program is important because it gives you a reference point for interpreting later changes.

Possibilities include:

• Before-and-after observation. Information is collected at the beginning and end of an evaluation period. This can show whether change occurred, but by itself may provide limited information about when change happened, why it happened, or which aspects of the program contributed to it.

For many evaluations, before-and-after measurement is most useful when combined with additional observations during implementation. Beginning with baseline information helps establish where participants, organizations, or community conditions started. Repeated measurement can then show whether change occurred gradually, rapidly, at particular stages, or only after the formal intervention ended.

Even when an effort has one highly visible goal—such as adoption of a policy or completion of a community project—evaluating only whether the final goal was reached leaves out valuable information. Documenting strategies, relationships, challenges, intermediate accomplishments, and adaptations can help explain what contributed to the result and what can be learned for future efforts.

• At regular intervals during the evaluation period. Observations might occur hourly, daily, weekly, monthly, or at another regular interval, depending on what is being measured. Regular schedules make it easier to compare information across time.
• At selected or randomly chosen intervals. Some observations may occur at varying times to capture a broader range of conditions or reduce the likelihood that the same circumstances are observed repeatedly. Random or strategically varied sampling can sometimes provide a more representative picture.
• At specific times or stages. You may want to examine conditions at particular stages of implementation or under different circumstances. For example, park use might be observed on weekdays and weekends, during different seasons, at different times of day, or during community events. A program evaluation might gather information during planning, outreach, implementation, evaluation, and follow-up.
• Ongoing observation. In some settings, staff, participants, sensors, logs, or other systems may provide ongoing information. Continuous collection should be used only when it is useful and proportionate to the evaluation need. Particularly when recording people or collecting digital data, consider whether continuous monitoring is necessary and how privacy will be protected.

At the health center, blood pressure might be measured at regular intervals during monthly meetings. Participants might document physical activity throughout the program using activity logs. Follow-up information could then be collected several months after the program to understand whether participants continued activities they found sustainable.

Define and describe the behaviors, products, conditions, and/or events that observers should be concerned with

Observers need clear definitions of what they are expected to document. The planning group—ideally including people who will conduct observations and people whose experiences are being represented—should establish shared definitions, examples, non-examples, and scoring or recording guidance for each element being observed. For example, an evaluation examining bullying or interpersonal aggression would need clear behavioral definitions rather than relying on assumptions about what those terms mean.

Design training for observers

Depending on their roles and previous experience, observers may need preparation in several areas:

• What should be recorded, and why. Observers should understand which contextual details matter, such as date, time, location, duration, environmental conditions, who was present, and other circumstances that might influence what occurred. Context can be as important as the event itself when interpreting observations.
• The definitions and descriptions of the behaviors, conditions, events, or situations being observed. Clear definitions improve consistency only if observers understand and practice using them.
• The effects of observation. People may behave differently when they know they are being observed. Observers should understand this possibility and use methods that reduce unnecessary influence while remaining transparent and respectful about the observation process.

Audio or video equipment can also affect behavior. Participants should understand what is being recorded, why it is being recorded, how the recordings will be used, who will have access, and how long they will be retained. Appropriate consent should be obtained before recording begins.

• Observer bias. Observers' relationships, expectations, experiences, identities, professional roles, and assumptions can influence what they notice and how they interpret it. This is especially important when staff members are evaluating programs in which they are invested. Training, reflection, structured observation tools, multiple observers, and opportunities to discuss differences in interpretation can help reduce the influence of bias. The goal is not to assume that observers can eliminate all perspective, but to recognize and manage how perspective can shape observation.
• Observer drift. Over time, observers may gradually interpret definitions differently from how they were originally established. Periodic review, refresher training, and comparison across observers can help maintain consistency.

Observational systems should include ways to identify and address observer effects, bias, inconsistent interpretation, or drift throughout the evaluation.

Devise checks for reliability and accuracy

Reliable information requires observers to apply definitions and measurement procedures consistently. Training is important, but it is also useful to periodically check whether different observers interpret the same behavior, condition, or event in similar ways. A participatory design process can strengthen this consistency because observers help define and understand what is being measured.

Ways to strengthen agreement and consistency include:

• Use a clearly defined standard. Observation criteria should describe the behaviors, conditions, or indicators being assessed and provide enough detail for different observers to apply them consistently. Checklists, coding guides, rating criteria, examples, and non-examples can support this process.

Research and evaluation teams often use standardized definitions and measurement procedures to improve consistency. Standards may include behavioral descriptions, clinical thresholds, environmental measurements, scoring rubrics, or other clearly defined indicators.

• Check inter-rater reliability. Inter-rater reliability refers to the degree to which different observers assess the same event or information consistently. Two or more observers may independently rate the same examples or observations and then compare results. The appropriate level of agreement depends on the type of measure and evaluation design; a single percentage threshold should not automatically be treated as adequate for every situation. If observers disagree, examine whether definitions, training, scoring procedures, or assumptions need clarification.
• Use periodic quality checks. A trained evaluator, supervisor, peer observer, or other designated person can periodically compare observations with those of regular data collectors. Repeated disagreement may signal that definitions, training, procedures, or measurement tools need to be reviewed.

Determine how to review and adjust your observational system for the next evaluation

Your observational system should itself be evaluated. Ask whether the system produced the information you needed, whether data collection was manageable, whether participants experienced the process as respectful, whether important perspectives or outcomes were missed, and whether any methods created unnecessary burden or privacy concerns. Use what you learn to improve the system for future evaluation cycles.

With thoughtful planning, appropriate training, clear ethical safeguards, and ongoing review, an observational system can provide useful information for understanding and improving your work.

In Summary

To evaluate a program or community effort effectively, you need a systematic way to collect useful information about what is being implemented, how people experience the work, what conditions influence it, and what outcomes occur. An observational system provides that structure. It may include direct or participant observation, self-reports, administrative records, assessments, technology-assisted measures, or other methods depending on the evaluation questions.

Observational systems should be feasible, participatory, ethically designed, and appropriate to the community and program context. Involving evaluators, data collectors, program staff, participants, and community members in design can improve the relevance of the information collected while helping identify concerns related to privacy, accessibility, interpretation, burden, and cultural context. Clear definitions, appropriate training, consistent procedures, and regular review can strengthen the reliability and usefulness of the resulting evaluation.

What do we mean by an observational system?

___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program

___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with

Why design an observational system?

___It can help you get reliable information

___It can help you find out exactly what you need to know, eliminating or reducing wasted effort

___It can ensure that observations are made

___It can make it easier to analyze your data

___It can help you avoid haphazard evaluation

___It will make it easier to justify your findings

___It can help you gain credibility with funders and policy makers

___It can let you pass on your practices with confidence

___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust

When should you design an observational system?

___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning

___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle

Who should design an observational system?

___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation

How do you design an observational system?

___Review your evaluation questions

___Decide what you need to observe to answer your questions:

• Participants’ behavior
• Someone else’s behavior
• Conditions
• Observations of results of behavior
• Participants’ knowledge or attitudes
• Someone else’s knowledge or attitudes
• Goal attainment
• Interactions
• Program process or implementation (e.g., number of participants)

___Decide how the observations will be conducted:

• Direct observation
• Participant observation
• Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
• Second-hand reports, including interviews, journals, surveys, etc.
• Electronic or mechanical observation
• Tests or other similar observation tools
• Public records and the like for community-level indicators

___Decide when you need to observe:

• Pre- and post- observation
• At regular intervals during the evaluation period
• At irregular intervals during the evaluation period
• At specific times during the evaluation period
• Continuously

___Define and describe the behaviors, conditions, and/or events that observers should be concerned with

___Train observers in:

• What it’s important to record, and why
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
• Effects of observation
• Observer bias
• Observer drift

___Devise checks for reliability and accuracy:

• Use an external standard
• Check for inter-rater reliability
• Use random third-party checks

___Adjust the system for the next evaluation

When you hear the word “experiment,” you may picture people in laboratory coats working with specialized equipment. In a broader sense, however, an experiment simply involves trying something and systematically examining what happens. It can be as simple as testing a different ingredient in a recipe or as complex as developing and evaluating a comprehensive effort to improve child health outcomes across a city or state.

Researchers in public health and the social sciences conduct studies to understand how programs, policies, environments, and other conditions influence behavior and outcomes. Community programs and interventions can also be viewed as opportunities for systematic learning. They often begin with an assumption—or theory of change—about how particular activities are expected to contribute to desired outcomes. Those assumptions should be examined rather than taken for granted. Evaluation helps determine whether change occurred, how the program may have contributed to it, for whom it worked, under what conditions, and what might be improved.

In this section, we will examine several ways to structure an evaluation so that you can understand whether and how your program is contributing to change. These structures are known as experimental, quasi-experimental, or evaluation designs.

What do we mean by a design for the evaluation?

Every evaluation is a form of systematic inquiry. You may want to understand how effective your program or effort is overall, which components are working well and which should be adjusted, whether outcomes differ among groups, or how particular conditions influence results. To answer those questions meaningfully, the evaluation needs a structure that connects the questions you ask with the information you collect and the conclusions you draw. That structure is the evaluation design.

The design depends on the kinds of questions your evaluation is meant to answer.

Common evaluation questions include:

• Does a particular program, intervention, policy, or change in access and opportunity contribute to changes in behavior, physical or social conditions, health or development outcomes, or other indicators of success?
• Which components or elements of the program appear to contribute most strongly to the outcomes?
• What intended or unintended effects occurred, and how did they influence participants, communities, or outcomes?
• What happens when you introduce or adapt a new method, activity, or policy?
• Will an approach that worked in another setting also work in your community, with your participants, and under your local conditions?

To answer questions like these, you need a design that helps distinguish between changes associated with your program and changes that may have resulted from other factors. The stronger and more appropriate the design, the more confidence you can have in the conclusions you draw. Later in this section, we will discuss several ways to structure an evaluation.

Why should you choose a design for your evaluation?

Evaluation may seem straightforward: if conditions improve, the program appears to be working; if they do not, something should change. In practice, evaluation is more complicated. How will you define and measure progress? If outcomes do not improve, which parts of the program should change? If improvement occurs, how confident are you that the program contributed to it rather than another factor? And even when a program is working well, which components are essential and which could be strengthened, adapted, or removed? A thoughtful evaluation design helps answer questions like these.

Reasons to design your evaluation carefully include:

• To strengthen the reliability and usefulness of your evaluation. A well-designed evaluation can provide information you can use with greater confidence. It can help you understand whether change occurred and how strongly the evidence supports the conclusion that the program contributed to that change.
• To identify strengths and areas for improvement. A good design can help show which program components appear effective, where barriers exist, and where adjustments may increase effectiveness, accessibility, or equity.
• To strengthen the credibility of your findings. Clearly designed and transparently conducted evaluations make it easier for participants, community partners, funders, policymakers, and others to understand and assess the evidence.
• To identify factors outside the program that may influence outcomes. Participants' prior experiences, community events, policy changes, economic conditions, changes over time, and many other factors can affect outcomes. A strong design can help identify these influences and clarify how they should be considered when interpreting results.
• To identify unintended consequences. Programs may produce outcomes that were not anticipated, including benefits, harms, inequities, or changes in areas that were not originally measured. A thoughtful design makes it more likely that these effects will be recognized and addressed.
• To provide a clear plan for the evaluation. An appropriate design helps determine what information should be collected, when it should be collected, from whom, and how it should be analyzed. Planning this structure in advance can improve both efficiency and the quality of the resulting information.

When should you choose a design for your evaluation?

Once you have identified your evaluation questions and gathered information about the issue, community context, and possible approaches, the next step is to choose an evaluation design. Ideally, evaluation planning begins alongside program planning. Evaluation should be an integral part of your program rather than something added only after implementation has begun.

In practice, community organizations, public agencies, and other programs may begin formal evaluation months or even years after implementation because of limited staff time, funding, evaluation capacity, or other constraints.

Even when evaluation begins later, the same basic process applies: clarify your questions, gather relevant information, choose a feasible design, and follow the evaluation process systematically. If your program operates in cycles, it may be useful to begin evaluation at the start of a new cycle or at the beginning of participants' involvement so that baseline information can be collected.

Programs with rolling enrollment or continuous services may require a different approach. You might evaluate outcomes among newly enrolled participants, follow different groups over time, or begin examining the program as it currently operates. The design should reflect the structure of the program and the questions you are trying to answer.

Who should be involved in choosing a design?

The Community Tool Box generally recommends a participatory process that includes people affected by the program as well as staff, community partners, evaluators, researchers, and other stakeholders. Evaluation design benefits from this same participation, while also requiring appropriate methodological expertise.

Research and evaluation design can involve complex questions about measurement, bias, comparison groups, statistical analysis, ethics, and interpretation. For this reason, someone with relevant evaluation or research experience should usually help coordinate the technical aspects of the design. That person or team should work collaboratively with participants and community partners rather than making design decisions in isolation. Researchers can explain the strengths, limitations, resource requirements, and ethical considerations of different designs, while community partners can help determine which approaches are appropriate, feasible, and meaningful in the local context.

How do you choose a design for your evaluation?

How do you decide which research design will best serve the purposes of your evaluation?

Consider four areas:

• The nature of the evaluation or research questions you are trying to answer
• Potential challenges to drawing valid conclusions and ways to reduce those challenges
• The research designs that may be appropriate and what each one requires
• Whether a particular design fits your program structure, participant preferences and consent, ethical requirements, timeline, and available resources

We will begin by examining the concerns a research design should address, then consider several common designs and their strengths and limitations, and finally discuss how to select a design that is both feasible and capable of providing useful information.

Note: In this part of the section, we discuss evaluation as a type of research. For that reason, we sometimes use the term “research” even when the same principle could be described as “evaluation.”

Concerns research designs should address

One of the most important considerations in designing research or evaluation is whether the design will provide information that supports the conclusions you want to draw. A weak design may show that something changed without helping you understand why it changed. For evaluation, that can make it difficult to know what should be continued, adapted, or improved.

Consider a simple example. Suppose you measured children's height at age six, encouraged them to eat carrots regularly for three years, and measured them again at age nine. Most would probably be taller. Without an appropriate comparison or other evidence, however, you could not reasonably conclude that carrots caused the increase in height. Children ordinarily grow as they age, and the design would not distinguish that natural development from any effect of the intervention.

Two broad types of validity are especially important in research design: internal validity, which concerns whether the intervention actually contributed to the observed change, and external validity, which concerns how well findings may apply to other people, places, or circumstances.

Threats to internal validity

Threats to internal validity are alternative explanations for the changes observed during an evaluation. Other events, participant characteristics, measurement changes, or natural processes may occur at the same time as the program. If the evaluation cannot distinguish among these influences, it becomes harder to determine how much of the observed change can reasonably be attributed to the program.

Common threats to internal validity include:

• History. Events that occur before or during the evaluation may influence outcomes. These might include participants' prior experiences as well as external events such as policy changes, economic shifts, community conflict, natural disasters, public health emergencies, or other major developments.
• Maturation. People and communities change naturally over time. Children develop as they grow, people gain experience, physical conditions may change with age, and participants may become tired or more comfortable with activities as time passes. These changes can occur independently of the program.
• The effects of measurement or observation. Being surveyed, tested, recorded, observed, or repeatedly asked about a behavior can itself influence how people think or act.
• Changes in measurement. Equipment, assessment tools, data systems, or observers may change over time. Different instruments or observers may also produce different results. Consistent procedures and quality checks can reduce these problems.
• Regression toward the mean. Extremely high or low measurements often move closer to the average when measured again, even without an intervention. This can be particularly important when programs intentionally enroll people because they have unusually high or low values on a particular measure.
• Participant selection. People who participate in a program may differ in important ways from those who do not. Selection procedures may also unintentionally favor people who already have greater access to resources, fewer barriers, or characteristics associated with stronger outcomes. For example, an employment program rewarded only for job placements might be encouraged to enroll people who are already closest to employment while excluding people who face greater barriers. Self-selection can also matter because people who voluntarily enroll may already be particularly motivated to make a change.

Selection becomes especially important when two or more groups are compared. If the groups differ substantially before the intervention begins, differences observed later may reflect those initial differences rather than the program itself.

• Loss of data or participants. Missing information and participant attrition can affect results, particularly when people who leave a study or program differ from those who remain. When comparing groups, different rates or patterns of attrition can make the comparison less reliable.
• The nature and timing of change. Change is rarely perfectly steady. Outcomes may improve, decline, fluctuate, or take time to emerge. Measurements taken over too short a period or at poorly chosen times may not capture the actual pattern of change.
• Combinations of multiple factors. Several threats to validity may occur at the same time. For example, an educational program tested only among students who already have extensive experience learning multiple languages would involve both selection effects and differences in prior experience.

Threats to external validity

External validity concerns how well findings may apply beyond the people, setting, and circumstances included in the evaluation. An intervention that works well in one setting may not produce the same results elsewhere if the populations, resources, institutions, cultures, implementation conditions, or other contextual factors differ.

Threats to external validity, sometimes discussed in terms of generalizability or transferability, often arise from interactions between the intervention and the particular conditions under which it was implemented.

Examples include:

• Interaction between data collection and the intervention. Baseline testing, surveys, or observation may influence how participants respond to the program. If another setting does not use the same data-collection procedures, outcomes may differ.
• Interaction between participant selection and the intervention. A program may work particularly well for people who voluntarily enroll, are highly motivated, or meet specific eligibility criteria, but those findings may not apply equally to people with different experiences or circumstances.

For example, people who participate in a parenting program because of a court or child-welfare requirement may experience the program differently from people who enroll voluntarily. Differences in motivation, stress, trust, available supports, and other circumstances may influence outcomes and should be considered when comparing results.

• The effects of the research arrangements. People may respond differently because they know they are being observed or because of their relationship with particular researchers, facilitators, or staff members.

The broader lesson is that the act of observation can sometimes influence the behavior being observed. Evaluators should consider whether data-collection procedures themselves may be contributing to the outcomes they see.

• Exposure to multiple programs, services, or interventions. Participants may be involved in several programs or experience multiple changes at the same time. These combined influences can make it difficult to determine the effect of one intervention in isolation.

For example, an adult might participate in a high school completion program while also receiving substance use recovery services. A person managing diabetes might receive medication while participating in nutrition, physical activity, or peer-support programs. These overlapping services may reinforce one another, interfere with one another, or make it difficult to identify the contribution of any single component.

Common research designs

Research design is a large field, and this section provides only an introduction. Evaluation designs differ in several important ways, including how many times outcomes are measured, when measurements occur, whether one or multiple groups are included, and how participants or groups are selected.

Program evaluations often seek to answer three basic questions:

• Did meaningful change occur during the evaluation period?
• To what extent did the program, intervention, or effort contribute to that change?
• What program components, contextual conditions, or outside influences helped produce or prevent the change?

Outcomes may be influenced by the program, by outside factors, or by both. Participants' experiences, natural development, policy changes, economic conditions, organizational changes, community events, and other influences may all affect results. Understanding those influences can help you improve the program and interpret outcomes more accurately.

Measuring conditions only before and after a program can show whether something changed, but it may not explain why. For that reason, evaluation designs should consider potential threats to validity as well as practical and ethical constraints.

Research designs often differ according to the number and timing of measurements and whether one or multiple groups are included. We will begin with single-group designs and then consider designs involving multiple groups.

Before continuing, it is useful to define several basic research terms.

Researchers commonly refer to measurements or observations collected before an intervention begins as baseline measures or baseline observations. Baseline information establishes a starting point against which later observations can be compared.

Other useful research terms include:

• Independent variables are the program, intervention, policy, method, or condition whose effects you want to examine. An evaluation may include one independent variable or several.
• Dependent variables are the outcomes or conditions expected to change in response to the independent variable. These might include behaviors, health outcomes, community conditions, knowledge, attitudes, or other indicators.
• Measures are the tools or procedures used to assess those outcomes. They may include community assessments, administrative data, observations, surveys, interviews, validated scales, clinical measurements, or counts of particular events or conditions.
• Observations may involve formal measurement or systematic documentation of what occurs in a particular setting. In research, the term is sometimes used broadly to describe many forms of measurement and data collection.

Pre- and post- single-group design

One of the simplest designs is a pre- and post-measurement design. You measure an outcome before the intervention, implement the program, and measure the same outcome again afterward. Examples might include unemployment rates, environmental pollution levels, health indicators, or participant knowledge.

This design can show whether change occurred between the beginning and end of the evaluation period, but by itself it usually cannot establish why the change occurred. Other events, natural trends, changes in measurement, or participant characteristics may explain some or all of the difference. Even so, collecting information before and after an intervention generally provides more useful information than having no baseline at all.

Consider blood pressure in a cardiovascular health program. A lower measurement at the end of the program than at the beginning does not show what happened between those two points. Blood pressure may have fluctuated throughout the evaluation period, measurement conditions may have changed, or other health or life changes may have influenced the result. Repeated measurement can provide a clearer picture.

Interrupted time series design with a single group

An interrupted time series design uses repeated measurements before and after the introduction of an intervention. Establishing a pattern before implementation provides a baseline trend, while repeated measurements afterward help show whether that pattern changed when the intervention began.

This approach can provide substantially stronger evidence than a simple pre- and post-design because it shows how outcomes change over time. If the outcome begins changing before the intervention starts, for example, that suggests that another factor may be contributing to the trend.

Some time-series designs also examine the effects of introducing different program components at different times. When appropriate and ethically feasible, this can help evaluators understand which components appear associated with particular changes.

Many variations of interrupted time series designs are possible, including changes in measurement frequency, staggered implementation, or evaluating combinations of program components.

When using a time-series design, it is important to examine outcomes closely enough to identify meaningful patterns. An overall average can sometimes hide changes occurring in particular periods, locations, or groups. Disaggregating information when appropriate can reveal effects that would otherwise be missed.

Interrupted time series design with multiple groups

A multiple-group time series design applies similar repeated measurements across two or more groups, sites, or communities. The intervention may be introduced at different times across groups, creating multiple baselines that can strengthen confidence that observed changes are associated with the intervention rather than a single external event.

This type of design may allow you to:

• Examine the same method or program across two or more groups or sites
• Examine whether an approach produces similar or different results across populations or settings
• Vary the timing or intensity of an intervention when doing so is appropriate
• Compare different interventions or implementation approaches
• Examine whether the order or sequence of program components influences outcomes

Many other variations are possible, depending on the program and evaluation questions.

Control or comparison group design

Another common approach is to compare outcomes between a group receiving the intervention and another group that does not receive the same intervention during the same period, receives usual services, or receives a different approach. A well-designed comparison can help distinguish program effects from changes that would have occurred anyway.

A control group is generally part of a design intended to isolate the effect of the intervention by holding other influences as similar as possible. A comparison group may be selected because it resembles the intervention group on relevant characteristics, even when participants are not randomly assigned.

Randomized controlled designs are among the strongest methods for estimating causal effects when random assignment is feasible, ethical, and appropriate. In these designs, eligible participants or groups are assigned by chance to different study conditions. Randomization helps reduce systematic differences between groups before the intervention begins, although it does not eliminate every possible limitation or guarantee that findings will generalize to other settings.

For example, eligible participants might be randomly assigned to receive either a new program or an existing standard service. If implementation and measurement are otherwise comparable, differences in outcomes between groups can provide strong evidence about the effect of the different approaches.

Community organizations may not always be able—or ethically justified—to withhold a potentially beneficial program solely for evaluation purposes. Alternatives may include wait-list designs, phased implementation, usual-services comparisons, matched comparison groups, or other quasi-experimental approaches. The evaluation should not create avoidable harm or unfairly restrict access to needed services simply to strengthen the research design.

When a program has limited capacity and already uses a wait list, phased or randomized enrollment may sometimes create a comparison while preserving eventual access to services. Another option is to provide participants waiting for the full program with existing resources, alternative services, or another appropriate form of support rather than leaving them without assistance solely for research purposes.

When no appropriate control or wait-list group exists, organizations may use a comparison group from another program, community, or setting. Because those groups may differ in important ways, evaluators should examine baseline characteristics carefully and be cautious about attributing differences in outcomes solely to the intervention. In some situations, a strong interrupted time series design may provide more credible information than a weak comparison between substantially different groups.

Groups may appear similar while differing in ways that influence outcomes. For example, participants who voluntarily enter a substance use treatment program may differ from people who are required to participate through a court or institutional process. Motivation, trust, available supports, prior treatment experiences, and other circumstances may influence results independently of the treatment model.

Average results can also hide important differences within a single group. An intervention that appears only modestly effective overall may work particularly well—or poorly—for participants with particular circumstances, experiences, or barriers. Examining variation in outcomes can provide important information for improving equity and effectiveness.

Choosing a design

This section provides only an introduction to evaluation design. Research methods include many additional experimental, quasi-experimental, observational, qualitative, mixed-methods, implementation, and participatory approaches. Statistical methods can also help address some design limitations, although analysis cannot fully compensate for poor-quality or inappropriate data.

When possible, seek technical assistance from people with evaluation expertise. Universities, research centers, public agencies, community evaluation organizations, funders, and consultants may be able to provide methodological support. Whatever design you use, community members and program participants should remain involved in identifying meaningful questions, outcomes, ethical concerns, and interpretations.

Your goal is to obtain the most credible and useful information possible given your evaluation questions, program structure, ethical responsibilities, participant preferences and consent, timeline, and resources. A community evaluation does not need to resemble a laboratory experiment to be valuable. It does need to be designed carefully enough that you understand what the evidence can and cannot tell you.

Consider your evaluation questions

What do you need to know? If your question is simply whether a particular condition changed, a pre- and post-design may sometimes be sufficient. If you want to know whether the program contributed to that change, you will need a design that provides stronger evidence about alternative explanations.

Comparison groups can be useful, but they are not the only way to strengthen an evaluation. Repeated measurements, multiple forms of evidence, qualitative information, implementation data, and careful documentation of external events can provide a much fuller picture. For many community organizations, an interrupted time series design may offer a practical balance between rigor and feasibility.

If your program already includes multiple groups, sites, or classes working toward similar goals, you may be able to compare outcomes across them or stagger implementation in ways that provide additional information. Any such approach should be developed with attention to fairness, participant expectations, and whether differences in implementation are appropriate.

Sometimes the central question is not simply whether a program works, but whether it works better than another available approach. Employment training, parenting support, chronic disease management, conflict resolution, and many other programs may raise this question. Depending on the setting, comparisons may involve another program, usual services, published research, administrative information, or community-level indicators.

Some community efforts have outcomes that can be measured relatively directly. An initiative focused on reducing water pollution, for example, may use repeated water-quality measurements as a central outcome. Even in these situations, however, you may also want to understand which activities contributed to improvement, whether benefits were distributed equitably, and whether changes were sustained over time.

Consider the nature of your program

What does the program look like, and what is it intended to accomplish? Does it work with participants individually or in groups? Does it operate in fixed cycles, such as classes or workshops, or can people enter at different times? How much depends on staff implementation, participant activity, partnerships, organizational practices, or the surrounding environment? How important are relationships, physical setting, accessibility, language, organizational culture, and other contextual factors?

• If participants take part in distinct groups or sites, a multiple-group design may sometimes be practical. If participation is primarily individual, a single-group longitudinal or time-series design may be more appropriate.
• If your program is time-limited, choose a design that fits the program schedule and allows enough time to capture meaningful outcomes. If multiple program cycles occur, lessons from earlier cycles may inform adaptations to later ones, but those changes should be documented clearly.
• If your program has no clear beginning or end, you may need to follow participants from their individual starting points or organize analysis around different levels of exposure, participation, or baseline conditions.

Community programs operate in changing environments. People entering a program at different times may encounter different staff, participants, community events, resources, policies, or organizational conditions. These differences can influence experiences and outcomes even when the formal program model remains the same. Evaluators should document important contextual changes and consider them when interpreting results.

• If organizational culture, relationships, or program context are central to the intervention, include measures that capture those factors rather than treating them as background conditions. Comparisons across different organizational settings may also be useful when they are appropriate and sufficiently similar.

Many evaluation designs can be adapted to different situations. The important goal is to select an approach that fits both the evaluation questions and the realities of the program.

Consider what participants and staff will consent to

Evaluation methods should respect privacy, dignity, autonomy, safety, and informed participation. Extensive observation, recording, repeated testing, or collection of sensitive information can create burdens or risks for participants and staff. A technically strong design is not appropriate if it depends on methods that are unnecessarily intrusive or unacceptable to the people involved.

Privacy and safety concerns may be especially important when participation in a program involves sensitive health information, immigration status, substance use, interpersonal violence, legal issues, or other information that could create harm if disclosed. Participants should not be pressured to provide information that is not necessary for the evaluation.

Explain what information will be collected, why it is needed, how it will be used, who will have access to it, what protections will be in place, and what choices participants have. Include participants and staff in designing data-collection procedures whenever possible. Their concerns may reveal risks or burdens that evaluators would otherwise overlook. :chatgpt-content-reference{index="1"}

Consider your time constraints

Choose a design that can provide credible information within the time available. The goal is not necessarily to create the most complex design possible, but to select an approach that is appropriate to the questions and transparent about its limitations.

Common time constraints include:

• Program structure. Evaluation may need to align with a regular program cycle, school year, funding period, or implementation schedule.
• Funding. Grants and contracts may establish deadlines for reporting or evaluation. Pilot programs may also have limited time to demonstrate early outcomes, even when longer-term effects will take more time to appear.
• Participants' schedules and responsibilities. Work, school, caregiving, transportation, seasonal employment, cultural or religious commitments, and other responsibilities may affect when people can participate in program or evaluation activities.
• The urgency of the issue. When a program addresses serious risks to health or safety, evaluation timelines should support timely decisions rather than delaying necessary action solely for methodological reasons.
• The availability of evaluation support. Evaluators, analysts, community partners, and other people involved in the evaluation may only be available during particular periods.

Consider your resources

Resources include funding, staff time, community knowledge, partnerships, technology, space, equipment, data systems, and evaluation expertise. A lack of one resource can sometimes be addressed through another—for example, by partnering with a university, public agency, community organization, or evaluator who can provide technical assistance.

Begin by identifying the design that best fits your questions and ethical responsibilities, then determine how to make that design feasible. You may need to simplify or adapt it, but understanding the strongest realistic option can help you make intentional compromises rather than automatically choosing the least resource-intensive design.

In Summary

Evaluation design strongly influences how confidently you can interpret your findings and how useful those findings will be for improving a program or intervention. A strong design considers threats to internal validity—whether the intervention contributed to the observed change—as well as external validity or transferability—whether findings may apply in other settings or populations.

Designs such as pre- and post-measurement, interrupted time series, multiple-group designs, comparison groups, and randomized controlled designs each have different strengths, limitations, ethical considerations, and resource requirements. They can often be adapted or combined to fit the needs of community programs.

A useful evaluation design should address your questions while also reflecting the nature of the program, community context, participant and staff consent, ethical responsibilities, available time, and resources. The goal is not simply to create the most rigorous design possible, but to generate credible, relevant, and actionable information in a way that respects the people and communities involved.

What do we mean by a design for the evaluation?

___The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know

___An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors

Why should you choose a design for your evaluation?

___So your evaluation will be reliable

___So you can pinpoint areas you need to work on, as well as those that are successful

___So your results are credible

___So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants

___So you can identify unintended consequences (both positive and negative) and correct for them

___So you’ll have a coherent plan and organizing structure for your evaluation

When should you choose a design for your evaluation?

___In the ideal, when you’re planning the program and evaluation, before you start any implementation

___In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters

Who should be involved in choosing a design for your evaluation?

___It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails

How do you select an appropriate design for your evaluation?

___Take into consideration the necessary research considerations

• Understand threats to internal validity ( whether the intervention produced the change)
  • History
  • Maturation
  • The effects of testing or observation on participants
  • Changes in measurement
  • Regression toward the mean
  • The selection of participants
  • The loss of data or participants
  • The nature of change
  • A combination of the effects of two or more of these
• Understand threats to external validity (generalizability of your findings)
  • Interaction of testing or data collection and the program or intervention
  • Interaction of selection procedures and the program or intervention
  • The effects of the research arrangements
  • The interference of multiple programs or interventions

___Understand common research designs

• Pre- and post- single group design
• Interrupted time series design with a single group (simple time series)
• Interrupted time series with multiple groups (multiple time series)
• Control or comparison group

___Choose a design

___Consider your evaluation questions

___Consider the nature of your program

___Consider what your participants and staff will consent to

___Consider your time constraints

___Consider your resources

In previous sections of this chapter, we discussed studying the issue, choosing a research design, and creating an observational system for gathering information for your evaluation. The next step is to collect and analyze your data—examining what the information shows and what it may mean for your work. This section focuses on how to carry out that process.

What do we mean by collecting data?

Collecting data means putting your information-gathering plan into practice. You have already decided how you will obtain information—through direct observation, interviews, surveys, assessments, administrative records, or other methods—and now you and others involved in the evaluation need to carry out that plan. Effective data collection also requires clear definitions, appropriate timing, consistent procedures, and careful documentation so that the information you collect is useful for answering your evaluation questions.

Recording and organizing information may take different forms depending on the type of data you are collecting. Your data-collection procedures should also reflect how you plan to analyze and use the information. Whenever possible, information should be recorded during data collection or soon afterward so that important details are not lost. Data should also be stored securely and in ways that protect participants' privacy and confidentiality.

Some of the steps involved in preparing collected information may include:

• Bringing together information from all relevant sources and observations
• Creating secure backups of forms, records, recordings, and other materials to reduce the risk of loss, accidental deletion, or damage
• Entering narratives, numbers, and other information into appropriate software, databases, spreadsheets, or data-management systems
• Preparing quantitative information for analysis. This may include entering observations into tables or spreadsheets and calculating descriptive statistics such as the mean, median, mode, percentages, rates, or other relevant measures
• Transcribing relevant portions of audio or video recordings when text versions are needed for analysis
• Coding qualitative information by identifying categories, concepts, themes, or other features that make the information easier to organize and analyze
• Organizing data in ways that support your evaluation questions. You might group information by outcome, participant or group, time period, activity, location, implementation stage, or another relevant category. Looking at data in more than one way can also help identify relationships and differences that might otherwise be missed.

Research often distinguishes between independent variables and dependent variables. An independent variable is the program, intervention, policy, method, or condition whose effects you want to examine. A dependent variable is an outcome or condition that may change in response to that intervention. For example, a smoking-cessation program may be treated as the independent variable, while changes in participants' smoking behavior may be one of the dependent variables.

What do we mean by analyzing data?

Analyzing information means examining it systematically to identify patterns, relationships, trends, differences, and other findings that can help answer your evaluation questions. Analysis may involve statistical procedures, comparisons across groups or time periods, visual inspection of trends, or careful interpretation of interviews, observations, documents, and other qualitative information. The purpose is to develop the most accurate and useful understanding possible of your program, its implementation, its outcomes, and the conditions that may have influenced those outcomes.

Evaluations commonly use two broad types of data, although not every evaluation will include both. Quantitative data are expressed as numbers and can be summarized or analyzed mathematically. Qualitative data include descriptions, experiences, observations, quotations, interpretations, and other information that helps explain meaning and context. Each provides different kinds of insight, and they often work especially well when used together.

Quantitative data

Quantitative data are often collected directly as numbers. Examples include:

• The frequency, rate, duration, or intensity of particular behaviors, events, or conditions
• Assessment or test scores related to knowledge, skills, health, or other outcomes
• Survey responses, such as ratings of satisfaction, stress, confidence, access, well-being, or reported behavior
• Counts, rates, or percentages describing a population—for example, the percentage of people diagnosed with a particular health condition, the unemployment rate, languages used in a community, age distribution, or levels of educational attainment

Information collected in other forms can sometimes be converted into quantitative data. Researchers might count how often particular themes or events appear in interviews or records, for example, or use a rating scale to describe the intensity or frequency of an observed condition. Community initiatives may also document the number and type of environmental, organizational, programmatic, or policy changes connected to their work. Whether converting information into numbers is useful depends on what you are trying to understand and whether doing so preserves the meaning of the original information.

Quantitative information may be summarized with descriptive statistics such as frequencies, percentages, rates, averages, medians, and ranges. More advanced statistical procedures can examine differences between groups, changes over time, associations among variables, and the strength of evidence for particular conclusions. Numerical analysis can be powerful, but numbers do not interpret themselves. The quality of the findings still depends on what was measured, how measurements were collected, whether the sample represents the people or conditions of interest, and whether the analysis is appropriate for the evaluation design.

Qualitative data

Qualitative information provides detail about people's experiences, meanings, perspectives, interactions, and the context in which events occur. It can help explain findings that numerical measures alone may not capture. For example, a test score may indicate how a student performed, while an interview or observation may help explain the student's experience, confidence, barriers, learning environment, or response to the result.

Qualitative information can sometimes be summarized numerically—for example, by counting how often themes appear or asking participants to rate dimensions such as satisfaction, importance, accessibility, or usefulness. However, converting qualitative information into numbers can reduce some of its context and meaning.

Rating scales illustrate this limitation. Two people may both select the same numerical rating while having very different reasons for doing so. A participant who rates a program a “3” might appreciate its content but find the schedule difficult, while another might like the schedule but feel the content does not meet their needs. The number alone cannot explain those differences.

Qualitative information can help explain why particular methods work well or create barriers, how participants experience a program, whether an approach fits the local context, and what participants consider most important. It can also reveal patterns or factors that were not anticipated when the evaluation was designed.

Whenever appropriate, collecting both quantitative and qualitative information can provide a more complete understanding of a program and its outcomes.

Neither quantitative nor qualitative analysis is completely free from human judgment. Quantitative analysis uses standardized mathematical procedures, but decisions about what to measure, which people or data to include, which statistical procedures to use, and how findings are interpreted still affect the results. Qualitative analysis involves interpretation more directly, making it especially important to use clear procedures, consider multiple perspectives, document how conclusions were reached, and reflect on how researchers' assumptions and experiences may influence interpretation.

Why should you collect and analyze data for your evaluation?

Not every organization has the resources to conduct a large or highly technical evaluation. Smaller community-based organizations may use simpler evaluation approaches that still provide valuable information for understanding and improving their work. Regardless of the level of complexity, systematic data collection and interpretation can strengthen decision-making and future planning.

• Data can show whether meaningful changes occurred in the outcomes you hoped to influence. Collecting information before, during, and after an intervention can help you determine whether conditions changed and whether the pattern of change is consistent with the goals of the program.

In statistical analysis, the term statistical significance has a specific meaning. Researchers often use a threshold such as p < .05 to indicate that the observed result would be relatively unlikely under a particular statistical assumption if there were actually no effect or difference. Statistical significance should not be confused with practical importance, the size of an effect, or the percentage of participants who benefited.

Statistical results should therefore be interpreted together with the evaluation design, effect size, confidence intervals when available, sample size, data quality, practical significance, and other evidence. A statistically significant result does not by itself prove that a program caused a particular outcome.

• Analysis can identify factors associated with differences in outcomes. For example, participants who have access to a peer-support group may show different outcomes from participants who do not. Findings like these can help identify implementation features that deserve closer examination.
• Analysis can identify relationships among different factors. Statistical procedures may reveal correlations or associations among variables. Two outcomes might increase together, decrease together, or move in opposite directions. These relationships can generate useful questions, but correlation alone does not establish that one variable causes another.
• Analysis can help explain why your work was more or less effective than expected. Combining quantitative and qualitative information can help identify implementation strengths, accessibility barriers, contextual conditions, participant experiences, or other factors that influence outcomes.
• Analysis can provide credible evidence for stakeholders. Participants, community members, governing boards, funders, and other partners may want to understand what was accomplished, what challenges remain, and what the organization has learned. Evidence about implementation, intermediate outcomes, and longer-term outcomes can support accountability and future planning.
• Regular use of data demonstrates a commitment to learning and improvement. Reviewing information about progress, barriers, and outcomes can help organizations act as responsible stewards of community resources.
• Sharing what you learn can strengthen work beyond your own program. Findings about effective strategies, implementation challenges, adaptations, and community context may help other organizations develop or improve related efforts.

When and by whom should data be collected and analyzed?

Whenever possible, data collection should begin before or when the program starts so that you can establish baseline information. Collecting information before implementation can also reveal trends that were already occurring. Data collection may continue throughout the program and, when longer-term outcomes matter, during an appropriate follow-up period after the formal intervention ends.

Data can be analyzed at different points depending on the purpose of the evaluation. A primarily summative evaluation may emphasize analysis after the main data-collection period so that findings can be considered together. A formative evaluation may examine information throughout implementation so that emerging findings can inform adjustments. Many evaluations combine these approaches by reviewing data periodically while also conducting a more comprehensive analysis at the end of an evaluation cycle.

Who collects and analyzes information depends on the methods being used, available expertise, resources, ethical requirements, and the degree of participation built into the evaluation.

Possible approaches include:

• Partnering with or hiring an external evaluator, researcher, university program, or other person with relevant expertise
• Conducting a less technical but systematic evaluation using descriptive statistics, participation records, interviews, feedback, and other information that can still provide useful insight into the program
• Building staff or community evaluation capacity through training, courses, technical assistance, or mentoring
• Collecting information internally and partnering with someone who has appropriate expertise to conduct or review more specialized analysis
• Using qualitative methods when they are appropriate to the evaluation questions. Qualitative information may be especially valuable for understanding participant experiences, implementation, accessibility, barriers, and context, even when it cannot by itself answer every question about program effectiveness
• Using an appropriate comparison strategy when your design and ethical circumstances support one. Comparisons can strengthen an evaluation, but conclusions should still take into account baseline differences, sample size, uncertainty, and other factors rather than relying only on raw percentages.

In a participatory evaluation, community members, program participants, staff, and other stakeholders can contribute to both data collection and interpretation. Some forms of measurement may require specialized training, while other information may be best collected or interpreted by people with lived experience or deep knowledge of the community. Even when statistical analysis is conducted by a specialist, community participants can play an important role in interpreting what the findings mean, identifying missing context, and deciding how the results should inform future action.

How do you collect and analyze data?

Whether your evaluation uses highly formal research methods or a simpler community-based approach, several basic steps can help you collect and analyze information systematically.

Implement your measurement system

We've previously discussed designing an observational system to gather information. The next step is to put that system into practice.

• Clearly define and describe the measurements or observations you need. Definitions should be specific enough that different data collectors understand what is being measured and record information consistently.
• Select and prepare data collectors. Depending on the evaluation, they may need training in observation, interviewing, confidentiality, data security, accessibility, informed consent, or inter-rater reliability.
• Collect information at appropriate times and for an appropriate period. This may include reviewing existing records, conducting interviews, administering surveys, facilitating focus groups, observing program activities, or gathering other forms of data.
• Record information using agreed-upon procedures. Depending on the setting, this may involve secure digital forms, databases, spreadsheets, written records, audio or video recordings, journals, activity logs, or other appropriate formats.

Organize the data you've collected

How you organize information depends on your evaluation questions and how you plan to analyze it.

• Enter and store information in an appropriate system. This might include a database, spreadsheet, qualitative analysis program, secure document system, or GIS (Geographic Information Systems) platform. Use appropriate privacy and security protections, particularly when data include personally identifiable or sensitive information.
• Transcribe audio or video recordings when needed. Transcription can make recorded information easier to search, code, compare, and analyze. Review transcripts for accuracy and protect identifying information appropriately.
• Score assessments or tests according to the procedures appropriate for the instrument and record the results accurately.
• Sort information in ways that support your questions. You might organize data by participant group, outcome, activity, location, time period, implementation stage, demographic characteristic, or another relevant category.
• When useful and appropriate, summarize qualitative information quantitatively. This might include counting how often particular themes, experiences, or barriers are mentioned. Preserve the original context where it is important for understanding what those counts mean.

Conduct graphing, visual inspection, statistical analysis, qualitative analysis, or other appropriate operations on the data

Statistical procedures can help examine whether outcomes changed, whether groups differed, whether trends emerged over time, and whether particular variables were associated with one another. The type of analysis should match the evaluation design, the quality and amount of available data, and the questions you are trying to answer.

Useful forms of analysis may include:

• Counting, graphing, and visually examining frequencies, rates, or other measures over time
• Looking for meaningful changes in trends, such as clear increases, decreases, or shifts following implementation of a program or policy
• Calculating descriptive statistics, such as the mean, median, mode, percentages, rates, or ranges
• Analyzing interviews, conversations, journals, and participant observations to understand experiences, implementation, and changes over time
• Identifying themes and patterns in qualitative information. When many participants describe similar barriers, supports, experiences, or concerns, those patterns may help explain how the program is functioning and where changes may be needed.
• Comparing results with previously established goals, benchmarks, or implementation expectations

Take note of significant, meaningful, or unexpected results

Findings may be statistically significant, practically important, unexpected, or especially meaningful to participants and communities. These are not necessarily the same thing. A statistically detectable difference may be too small to matter in practice, while an important community-level change may not be captured well by a conventional statistical test.

Types of findings you may want to examine include:

• Changes within individuals, groups, organizations, or communities over time. Repeated measurements can help show whether outcomes changed after an intervention was introduced. Staggered implementation across people, groups, or sites may strengthen confidence that observed changes are associated with the intervention.
• Differences between or among groups. When groups are appropriately comparable, differences in outcomes may provide evidence about program effects. Interpretation should consider how groups were formed, whether they differed at baseline, and the uncertainty around the estimates.
• Statistically significant changes. Statistical tests may indicate that an observed difference is unlikely under a particular null hypothesis. Whether that difference was caused by the intervention depends on the evaluation design and other evidence.
• Correlations and associations. Variables may change together in ways that suggest potentially important relationships. These findings can generate new questions and hypotheses, but should not automatically be interpreted as evidence that one variable causes another.

Correlation does not establish causation. Two variables may change together because one influences the other, because the relationship works in the opposite direction, because both are affected by a third factor, or because the association occurred for another reason.

For example, two outcomes may both increase as young people get older, even though neither causes the other. On the other hand, a strong and repeated association between a medication and an adverse health outcome could provide an important signal that deserves additional investigation. The appropriate response is to examine the relationship carefully rather than assume a causal explanation from correlation alone.

• Patterns. Both quantitative and qualitative data may reveal patterns by geography, access to resources, language, age, race, ethnicity, disability, income, program location, or other relevant characteristics. Such patterns should be explored carefully rather than interpreted as inherent characteristics of a group. They may reflect differences in exposure, opportunity, access, policy, environment, discrimination, or other contextual factors.
• Findings with clear practical importance. Some results may stand out even before advanced analysis. For example, participants in a health program might show meaningful improvements in blood pressure, physical activity, or well-being compared with an appropriate comparison group. Conversely, consistently unchanged outcomes may signal that the program needs adjustment, is not reaching people who could benefit, or is measuring the wrong outcomes.

Important findings do not always answer only the question of whether the program “worked.” Evaluation may reveal that a community-wide issue is concentrated in particular neighborhoods, that some groups experience greater barriers to services, or that outcomes differ according to access to resources or environmental conditions. Examining findings from multiple perspectives can help organizations improve their programs while also identifying broader conditions that may require attention.

Interpret the results

Once you have organized and analyzed the information, the next step is interpreting what it means. A common evaluation question is whether the program contributed to meaningful change. In research terms, this involves examining the relationship between the intervention and the outcomes it was intended to influence while considering other possible explanations.

Several broad patterns of findings are possible:

• The program contributed to the outcomes you hoped to achieve. Evidence may show meaningful improvements associated with implementation, particularly when findings are consistent across multiple measures or stronger than those seen in an appropriate comparison group.
• The program does not appear to have produced the intended outcomes. The evaluation may show little or no meaningful change in the outcomes being measured.
• The program may be associated with harmful or unintended outcomes. These findings require careful interpretation. For example, an increase in reported incidents of interpersonal violence after a program begins could indicate an actual increase, but it could also reflect improved screening, increased awareness, greater trust in reporting systems, or better access to support.
• The program may produce several different effects.
  • Some effects may be beneficial. A youth violence-prevention initiative, for example, might reduce violent incidents while also improving school engagement or participants' sense of safety and belonging.
  • Some effects may be neutral or unrelated to the program's central goals.
  • Some effects may be harmful or unintended. A strategy that improves one outcome might create new barriers, increase disparities, or have unintended effects in another area.
  • Effects may also be mixed, differing across outcomes, participants, settings, or time periods. Understanding who benefits, who does not, and under what conditions is often as important as calculating the average effect.

If the evidence suggests that the program is accomplishing its goals, continue examining how it might be strengthened, sustained, or adapted. Effective programs should continue learning rather than assuming that current results cannot be improved.

Strong community programs are dynamic. They use evaluation findings to continue learning and adapting rather than treating initial success as the end of the improvement process.

If the evaluation shows limited, uneven, or harmful outcomes, interpretation becomes especially important. Is the underlying approach inappropriate, or is a promising approach being implemented inconsistently? Are there organizational barriers, accessibility issues, resource constraints, policy conditions, discrimination, or other structural factors influencing outcomes? Are some groups benefiting while others face barriers? Should particular components be modified, expanded, removed, or redesigned?

Careful interpretation can help answer questions like these. Associations among variables may generate hypotheses about factors contributing to outcomes, but those relationships should be investigated rather than treated automatically as causal. Qualitative information can be especially valuable for understanding why participants experience a program differently and for identifying supports, barriers, or contextual conditions that are not apparent in numerical data.

For example, if participants experience ongoing health challenges, qualitative information might identify barriers related to food access, housing conditions, transportation, affordability of care, language access, previous experiences with health systems, work schedules, caregiving responsibilities, cultural preferences, or the availability of trusted and responsive providers. Interpretation should begin with participants' own experiences rather than assumptions about their knowledge, culture, or choices.

Once your information has been organized and analyzed, both numerical findings and qualitative evidence should be considered together using careful reasoning and critical thinking. Numbers may show that change occurred, while participant experiences, program records, and contextual information may help explain what contributed to it and why.

Analysis and interpretation bring the evaluation process back to where it began: learning how to improve the work. Use what you have learned to adjust the program, continue monitoring outcomes, evaluate those changes, and repeat the process as needed. Ongoing evaluation can help strengthen programs while also identifying organizational, environmental, policy, and systems-level changes that support healthier and more equitable communities.

Understanding an initiative requires attention to context, relationships, culture, history, systems, and participants' lived experiences. In that sense, evaluation often involves learning not only whether something changed, but how people understand and experience that change.

In Summary

The heart of evaluation is collecting meaningful information about a program or intervention and analyzing that information to understand implementation, outcomes, strengths, limitations, and opportunities for improvement.

Quantitative data—information expressed numerically—can be summarized through graphs, descriptive statistics, and more formal statistical analysis. These methods can help identify changes, differences, trends, and associations among variables. Their interpretation should always take the evaluation design, data quality, uncertainty, and practical significance into account.

Qualitative data—including interviews, observations, participant experiences, descriptions of conditions, documents, and other contextual information—can help explain how and why outcomes occurred, what participants experienced, what barriers or supports were present, and what may need to change.

Once you have learned from the information you collected, use those findings to strengthen the next cycle of planning, implementation, and evaluation. Continued data collection, reflection, and analysis can help keep community programs responsive, effective, equitable, and accountable to the people they are intended to serve.

What do we mean by collecting data?

___Collecting data means putting your design for a measurement system into operation

___Collecting data involves gathering information through observation, interviews, testing, surveys, and/or other means; recording it in appropriate ways; and organizing it so that it’s easier to work with

What do we mean by analyzing data?

___Analyzing data involves examining the information you’ve collected in ways that reveal the relationships, patterns, trends, etc. that can be found within it

___Data can be quantitative (collected as numbers) or qualitative (collected as narrative information, records, journal notes, etc.)

___Quantitative data is usually analyzed by subjecting it to one or more graphical displays or statistical operations that demonstrate the significance of relationships among variables

___Data are also usually analyzed logically, by looking for patterns and relationships revealed within them

___Qualitative data can sometimes be turned into quantitative data by, for instance, counting (e.g., the number of times a behavior occurs in various circumstances) or by rating on a number scale such dimensions as importance, satisfaction, or quality (e.g., the quality of housing or quality of life in neighborhoods)

___A combination of quantitative and qualitative data often yields the best overall picture

Why should you collect and analyze data for your evaluation?

___This can show whether or not there was actually any significant change in the dependent variable(s) you hoped to influence

___This can show connections between or among various factors that may have an effect on the results of your evaluation

___This can imply or show the reasons that your work was effective or ineffective

___This can provide you with credible evidence to show funders and the community that your program is successful, or that you’ve uncovered, and are fixing, the elements that are barriers to success

___This can show that you’re serious about evaluation and about improving your work

___This can show the field that what you’re doing works well, and thus pave the way for others to use similar methods and approaches as best practices

When and by whom should data be collected and analyzed?

___Data collection should start no later than when you begin your work and continue throughout

___If you want to see if any change is part of a long-term trend, you should collect data for some time before your program actually starts

___If you want to understand long-term effects, you should collect data on participants for some time after they leave your program

___Data should be collected and analyzed by people who are capable of doing so

___Data collection and analysis can be done by anyone from community members who have been trained to professionals with experience in conducting studies

How do you collect and analyze data?

___Implement the observational system you’ve planned

___Organize the data you’ve collected

___Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

___Take note of any significant or interesting results

___Interpret the results

Example #1: Santa Monica’s Use of Data to Lift All Boats: A Deeper Look

Four years ago, Santa Monica took a data-driven approach to become a true City of Wellbeing where everyone thrives. The city used a Wellbeing Index to produce a data-based guide for steering policy. The city used existing information from city departments and fresh input from surveys. The Wellbeing index sets a framework for priorities and promotes initiatives on health and wellness.

Contributed by Lia Thompson, University of Kansas, Community Tool Box Intern.

 

 

• What is ethnographic information?

• Why might ethnographic information be important to evaluation?

• When might you want to collect ethnographic information?

• Who should collect and interpret ethnographic information?

• How do you gather ethnographic information?

• How do you interpret ethnographic information?

Someone watching a baseball game or cricket match for the first time may be puzzled by a series of actions that experienced players and fans immediately recognize. A batter might adjust equipment, step out of position, look toward a coach, or repeat a familiar routine before each pitch. Some actions may help the player focus, some may relate to strategy or communication, and others may simply be personal habits. Without understanding the context, an outside observer could easily misinterpret what is happening.

Understanding why people act as they do can be important when trying to understand a community, organization, or social group. Ethnographic researchers often spend extended time in the settings they are studying, listening, observing, participating when appropriate, and learning how people themselves understand their experiences. This approach can help reveal not only what happens in everyday life, but also the meanings, relationships, histories, values, institutions, and assumptions that shape people's actions.

Depending on the issue you are evaluating, looking only at individual behavior may provide an incomplete picture. Social exclusion, violence, health inequities, institutional trust, access to services, and many other community issues are shaped by relationships, policies, history, economic conditions, social expectations, and other contextual factors. These cannot always be understood through tests, surveys, or counts alone. Ethnographic methods can help evaluators understand how people experience these conditions and what those experiences mean in the context of their everyday lives.

What is ethnographic information?

Ethnographic information is qualitative information gathered through sustained engagement with members of a community, organization, or social group, with particular attention to how people themselves understand their experiences and surroundings. Its purpose is to describe and interpret social practices, relationships, values, expectations, and meanings from the perspectives of the people involved rather than imposing assumptions from outside.

Depending on the evaluation, ethnographic inquiry may examine community norms, health and well-being, systems of power, religion and spirituality, economic conditions, relationships, identity, institutions, and broader ways of understanding the world.

The terms “culture,” “group,” and “community” are used broadly in this section. A group may be connected by geography, language, religion, ethnicity, profession, school, organization, shared experience, interest, or another form of affiliation. No community or cultural group is completely homogeneous. People within the same group may differ substantially by age, gender, socioeconomic position, disability, immigration experience, political perspective, religion, generation, neighborhood, and many other characteristics. Ethnographic research should therefore look for both shared patterns and important differences rather than assuming that all members of a group think or behave alike.

Community norms. These include expectations and practices that influence everyday behavior and relationships. Areas to explore may include:

• Gender expectations and roles. What expectations exist around gender, caregiving, employment, leadership, family responsibilities, dress, communication, and relationships? How consistent are these expectations across generations and individuals, and how are they changing?
• Social structure and status. How are influence, respect, opportunity, and social status distributed? What institutions or relationships shape access to resources and decision-making? Who is more likely to be included or excluded, and how easily can people move between social or economic positions?
• Behavior and interaction. What forms of communication and behavior are considered respectful, appropriate, or inappropriate in different settings? How do expectations vary according to age, role, relationship, setting, or other factors? What formal or informal consequences exist when community expectations are violated?

Social norms can sometimes support health, safety, belonging, and cooperation, while other norms may contribute to harm, exclusion, discrimination, or unequal treatment. Evaluators should avoid assuming that harmful practices are inherent to a particular culture. Instead, examine how history, policy, institutions, social conditions, and community perspectives interact to shape what is considered normal or acceptable.

Understanding these dynamics can help explain how people perceive a program or initiative and what changes may be needed to make the work more relevant, respectful, and effective.

Health and well-being. Ethnographic inquiry can help explain how people understand health, what resources they use, what barriers they encounter, and what practices support well-being. Areas to explore might include:

• Experiences and practices related to prenatal, infant, child, and family health
• Community experiences with alcohol, tobacco, other substances, substance use disorders, prevention, treatment, and recovery
• Awareness and experience of chronic health conditions, environmental risks, and protective factors
• Access to nutritious food, opportunities for physical activity, and the economic, environmental, and social conditions that influence those opportunities
• Access to and experiences with vaccinations, screenings, preventive care, and other health services
• Relationships with health care providers and levels of trust in health systems
• Access to affordable, culturally responsive, accessible, and timely health care

In communities with limited infrastructure or health resources, relevant questions might include: Do people have reliable access to safe drinking water, sanitation, medications, transportation, and health services? What barriers affect access to care? What local knowledge, practices, community health workers, mutual-aid networks, or other resources already support health and well-being?

Power and political realities. Who has formal and informal influence? Who participates in decisions, and whose perspectives are often excluded? How are laws, policies, and institutional practices experienced by different groups? Which organizations, leaders, networks, or community members are trusted? How do people advocate for change? These questions can help reveal the distribution of decision-making power and the ways policies and institutions affect daily life.

Religion and spirituality. Religion and spirituality may play a central role in some communities and a limited role in others. Relevant questions might include:

• What religious, spiritual, or nonreligious traditions are represented in the community?
• What roles do faith-based organizations or spiritual leaders play in social support, community life, advocacy, education, health, or civic engagement?
• How do community institutions respond to religious diversity and to people with no religious affiliation?
• Which values, traditions, practices, or ethical principles are especially important to community members?
• How do religious or spiritual beliefs influence people's decisions, relationships, health practices, or participation in community life?
• How do religious institutions interact with public institutions, government, schools, health systems, and other community organizations?

Economics. What economic conditions shape everyday life in the community?

• What are local patterns of employment, unemployment, underemployment, wages, and job security?
• Is the local economy heavily dependent on particular employers, industries, occupations, or seasonal work?
• Are there inequities or discrimination in hiring, pay, promotion, or access to economic opportunity?
• How accessible are education, training, transportation, childcare, technology, and other resources connected to economic opportunity?
• What kinds of work, caregiving, informal labor, and other contributions are valued or overlooked?
• What opportunities and barriers affect economic mobility?
• Is safe, stable, accessible, and affordable housing available?
• To what extent do households experience economic hardship, housing instability, food insecurity, or difficulty meeting basic needs?

What constitutes a basic need can vary according to geography, infrastructure, climate, household circumstances, local costs, and social conditions. Rather than assuming a universal standard based on one society's expectations, ask community members what resources they consider necessary for health, safety, dignity, participation, and well-being, and examine whether people have equitable access to those resources.

• How are income, wealth, resources, and economic opportunities distributed, and which groups experience the greatest barriers or disparities?

Worldview and shared understandings. This overlaps with many of the categories above and refers to the ways people understand themselves, their communities, institutions, relationships, and possibilities for change. Relevant dimensions may include:

• Views of authority and institutions. Which people or institutions are considered trustworthy or legitimate, and why? How have historical and current experiences with government, schools, law enforcement, employers, health systems, religious institutions, and community organizations influenced those views? People may trust different sources of authority depending on relationships, experience, accountability, expertise, and whether institutions are seen as acting fairly and in the community's interests.
• Sense of agency and influence. To what extent do people feel able to influence decisions affecting their lives and communities? What strengthens or limits that sense of agency? How do institutional practices, economic conditions, discrimination, legal status, age, access to resources, and community relationships shape people's opportunities to participate in change?

People may locate decision-making power in themselves, their families, community networks, employers, government institutions, legal systems, religious traditions, civic organizations, or other sources. These perceptions often reflect both personal experience and broader social conditions, and they may vary considerably within the same community.

• Relationships and trust. How is trust built and maintained? Which relationships provide support, belonging, caregiving, information, or resources? How are family relationships, friendships, partnerships, intergenerational relationships, and community networks understood? Where do conflict, exclusion, or unequal power appear?
• Responsibilities and rights. What responsibilities do people feel toward family, neighbors, community organizations, or the broader society? What rights and opportunities do people expect to have? Do community members experience those rights equally in practice?

Expectations about mutual support, individual responsibility, collective responsibility, and civic participation differ both within and across communities. These differences should be explored through people's experiences rather than inferred from whether a community is rural or urban, wealthy or economically constrained, or located in a particular part of the world.

Likewise, legal rights may exist formally while access to those rights differs in practice. Ethnographic inquiry can help reveal how people experience voting, property, employment, education, health care, public services, legal protections, and other rights in their everyday lives.

• Inclusion and belonging. Who is welcomed, included, or represented in community life? Who experiences exclusion, stigma, discrimination, or barriers to participation? How do institutions and informal community practices respond to differences in race, ethnicity, disability, language, gender, sexual orientation, immigration experience, socioeconomic position, religion, age, or other characteristics?
• Identity and affiliation. How do people describe themselves, and which relationships or communities are especially important to their sense of identity and belonging? These might include family, neighborhood, cultural or ethnic identity, profession, faith, generation, peer networks, civic organizations, sports, or other communities.

You may not need to study every aspect of a community. In most evaluations, ethnographic inquiry focuses on the specific cultural, institutional, or social factors most relevant to the issue. For example, an evaluation related to school engagement might explore how students experience school, what supports or barriers affect attendance and belonging, how families view the school system, and how institutional practices influence participation.

Several key principles characterize ethnographic information:

• Ethnographic information comes primarily from people who participate in or directly experience the community or setting being studied. Depending on the evaluation, this might include program participants, residents, students, workers, caregivers, community leaders, people from particular age groups, language communities, or others whose experiences are relevant.
• Ethnographic information is generally gathered in natural settings where people normally live, work, learn, gather, or participate. This helps evaluators understand experiences beyond the formal program environment.
• Ethnographic inquiry seeks to understand experiences from the perspectives of the people involved.
• Ethnographic description should aim to understand before judging. Researchers should document perspectives, practices, relationships, and context accurately while remaining attentive to power, harm, inequality, and the researcher's own assumptions. The goal is not to treat every practice as acceptable, but to understand the meanings and conditions surrounding it well enough to support thoughtful and effective action.

Why might ethnographic information be important to evaluation?

Ethnographic information can help explain why programs work differently across people, settings, and circumstances in ways that numerical information alone may not.

For example, ethnographic research has been used to understand why some households and communities are more likely to be missed in population counts. Interviews and observation may identify problems such as inaccurate address records, complex or changing household arrangements, concerns about how information will be used, limited language access, accessibility barriers, distrust based on previous experiences with institutions, or difficulty understanding forms and procedures.

Findings like these can lead agencies to improve outreach, simplify materials, provide information in multiple languages and accessible formats, work with trusted community organizations, and revise data-collection procedures so that more people are represented accurately.

Ethnographic information provides insight into how participants actually experience programs and initiatives. Staff may believe that a service is accessible and welcoming while participants experience long waits, confusing procedures, transportation barriers, disrespectful treatment, language barriers, or requirements that conflict with work and caregiving responsibilities. Ethnographic inquiry can reveal these differences.

Ethnographic information can help identify what should change for an approach to be more useful and responsive. Understanding participants' experiences can help organizations adjust communication, scheduling, staffing, services, policies, physical spaces, or other aspects of the program.

Ethnographic information can clarify how people understand an issue and how important it is in their lives. Community members may define an issue differently from program staff or researchers. Understanding those perspectives can improve how problems and goals are framed.

Ethnographic information can deepen your understanding of community context. This can strengthen future planning, relationships, advocacy, implementation, and evaluation while reducing the risk that programs rely on stereotypes or assumptions.

When might you want to collect ethnographic information?

Ethnographic research can provide valuable insight, but it also requires time, trust, skill, and sustained engagement. You may not need an extensive ethnographic study for every evaluation.

• When you are working with a community or social group whose experiences and context are unfamiliar to your organization. Ethnographic inquiry can help challenge assumptions and identify issues that outside observers might otherwise overlook.
• When you are working with a group that has developed shared norms or practices. A school classroom, faith community, workplace, neighborhood organization, sports team, peer network, or other group may develop its own expectations and relationships that influence how a program is experienced.
• When understanding community context is fundamental to the issue. Issues such as racism, discrimination, violence, institutional trust, health inequities, or social exclusion cannot be fully understood without examining history, relationships, policies, power, opportunities, and people's lived experiences.
• When your evaluation focuses on a particular population, setting, or experience. Understanding how people experience a job-training program, health initiative, school program, public service, or other intervention can help explain what is working, what creates barriers, and why.

Who should collect ethnographic information?

Who collects ethnographic information depends on the purpose of the evaluation, available time and resources, existing community relationships, and the level of trust required. Trained evaluators or researchers may conduct the work, but community members, peer researchers, program participants, or other people with relevant lived experience can also play central roles.

In many situations, involving community members as research partners can improve trust, interpretation, cultural responsiveness, and access to perspectives that might otherwise be missed. Community researchers should receive appropriate training, compensation, support, and decision-making authority rather than being used only as sources of access or information.

How do you gather ethnographic information?

Begin by deciding what you need to understand. Then identify appropriate methods, build relationships and trust, develop an ethical fieldwork plan, and carry out the research in collaboration with the people and communities involved.

1. Decide what kinds of information you need. You may be interested in a broad understanding of a community or in a specific question connected to your evaluation. Most evaluation-focused ethnographic research examines particular aspects of culture, institutions, relationships, or daily experience rather than attempting to describe an entire culture.

Define the topic clearly enough to guide observation and conversation. Useful questions might include: How do participants experience the program? Which parts feel relevant or useful? What creates barriers? Are organizational goals aligned with participants' goals? How do staff-participant relationships influence participation? Are language, accessibility, scheduling, institutional practices, or prior experiences affecting engagement?

Suppose, for example, you want to understand how students experience an English-language learning program. The focus would be the learning environment and students' experiences rather than assumptions about any particular ethnic or national culture.

Questions might include:

• What goals do students have for learning English, and how does the program support those goals?
• How do students experience relationships with instructors and other students?
• Which teaching and learning approaches help students participate, practice, and build confidence?
• What supports or barriers outside the classroom—such as work schedules, transportation, childcare, technology, or prior educational experiences—affect participation?
• Which topics, examples, learning activities, and teaching approaches do students find relevant and useful?
• What do students discuss among themselves about the class and their learning experiences?
• Are there language, identity, power, accessibility, or interpersonal dynamics that affect participation and belonging?

Findings might suggest changes to class schedules, instructional materials, language supports, teaching methods, group activities, staffing, technology access, childcare support, or other aspects of the learning environment. Adjustments should be based on what students report and experience rather than assumptions about their cultural backgrounds.

Knowing which questions you want to answer will help determine whom to engage, what settings to observe, how much time will be needed, and which methods are appropriate.

2. Determine what you have the resources to do. An evaluation-focused ethnographic study does not need to involve years of immersion. A community organization may gain more useful information by spending sustained time in participants' neighborhoods, workplaces, schools, community spaces, or program settings than by attempting a broad study of an entire national or ethnic culture.

3. Build trust and relationships with the community. Trust is not simply a means of obtaining information; it is an ethical foundation of community-based research. Explain who you are, why the research is being conducted, what will happen with the information, and how community members can influence the process.

Organizations with established community relationships may already have a foundation of trust. When that is not the case, working with trusted community organizations, peer researchers, neighborhood leaders, faith leaders, cultural organizations, advocates, or other respected community members may help establish relationships.

Avoid treating any individual as a spokesperson for an entire community. Trusted partners can help with introductions and context, but the research should still include a range of perspectives.

When engaging people in community settings, choose spaces and methods that are appropriate, accessible, and comfortable for participants. Respect people's privacy and avoid approaching individuals in ways that feel intrusive or coercive.

4. Plan your field study. Ethnographic fieldwork takes place in the settings where people normally live, work, learn, gather, or participate. Depending on the evaluation, this might include classrooms, community centers, workplaces, public spaces, meetings, homes when invited, online communities, or other relevant environments.

By this point, you have already completed some of the planning. Additional steps include:

• Decide on your questions. Ethnographic questions should help explain participants' experiences, relationships, and context in ways that inform the broader evaluation.
• Determine which methods you will use. Possible approaches include:
  • Individual or group interviews, ranging from informal conversation to structured or semi-structured interviews
  • Participant or non-participant observation
  • Informal conversations and relationship-building
  • Participation in community activities, meetings, celebrations, work, or other relevant settings when invited and appropriate
  • Open-ended surveys or questionnaires
  • Interviews with people who have relevant knowledge or experience
  • Documents and cultural materials, such as organizational communications, policies, websites, social media, artwork, meeting records, educational materials, media, or other sources relevant to the group
  • Demographic, administrative, and public data used carefully alongside community perspectives
  • For more detail on several of these approaches, see Chapter 3, Section 15, Qualitative Methods to Assess Community Issues.
• Decide whom you need to engage and how. Seek a range of perspectives rather than relying only on formal leaders or the people easiest to reach. Recruitment methods should respect privacy, voluntary participation, language access, accessibility, and people's right to decline.

Ethical considerations: Your first responsibility is to the people and communities participating in the research. Explain the purpose of the study, what participation involves, how information will be recorded and used, possible risks and benefits, and how privacy and confidentiality will be protected. Obtain appropriate informed consent and make clear that participation is voluntary.

Consider who owns or controls the information collected, who will have access to it, how long it will be retained, whether participants or community partners will be able to review findings, and how results will be shared back with the community. Researchers should avoid extractive practices in which communities provide knowledge but have little influence over interpretation or use of the findings.

Ethnographic research also requires reflexivity—ongoing attention to how researchers' identities, relationships, assumptions, institutional roles, and power can influence what they observe and how they interpret it. Accuracy does not come from pretending that researchers have no perspective, but from documenting and critically examining how that perspective may affect the work.

5. Carry out your fieldwork and keep fieldnotes. Fieldwork means spending time in relevant settings, developing relationships, observing, listening, participating appropriately, and gathering information using the methods you planned.

Your methods may change as you learn more. Ethnographic inquiry is often iterative: early observations lead to new questions, which guide later conversations and observations. Document those changes and why you made them.

Fieldnotes are records of observations, conversations, context, questions, interpretations, and researcher reflections. They should be written during or as soon as practical after fieldwork while details are still fresh. Audio, photographs, or video may also be useful when appropriate, but only with clear permission and suitable privacy protections.

Useful elements of fieldnotes may include:

• Date, time, setting, and relevant context
• Specific observations and details about what occurred
• Relevant sensory and environmental observations
• Reflections on how the researcher's presence may have influenced the setting
• Important words, phrases, terminology, and summaries of conversations
• Questions or issues for future investigation
• A consistent system for organizing notes and linking them to dates, settings, participants, or themes

Fieldnotes can include several distinct types of material:

• Jottings are brief notes made during fieldwork to help the researcher remember details that can be expanded later.
• Description records what happened, who was involved, relevant context, and other details of the setting or experience.
• Preliminary analysis records emerging patterns, questions, interpretations, and possible connections while making clear that these ideas may change as more information is gathered.
• Researcher reflection documents reactions, assumptions, uncertainties, relationships, comfort or discomfort, and ways the researcher's position may influence what they observe or how others respond.

Reflexive notes can help evaluators identify assumptions and recognize when a program may conflict with participants' priorities or experiences. The goal is not to label community perspectives as incorrect or deficient, but to understand where differences exist and how programs can respond respectfully while still addressing safety, equity, or other important goals.

Timely, detailed, accurate, and ethically maintained fieldnotes are central to strong ethnographic research.

How do you interpret ethnographic information?

Ethnographic information becomes useful when it helps explain the relationships, experiences, conditions, and meanings connected to your program and evaluation. Begin by organizing fieldnotes and other materials systematically.

Organize the final version of your notes. You might categorize information by theme, setting, time period, type of interaction, participant role, or other dimensions relevant to your questions. Demographic characteristics may also be useful when they are relevant, but avoid assuming that differences are caused by identity alone. Patterns may instead reflect differences in access, treatment, opportunity, exposure, institutional practices, or other contextual factors.

Develop a description of the setting or community from participants' perspectives. Depending on the evaluation, this may be broad or narrowly focused on a particular program, service, policy, or experience.

Look for connections among relationships, institutions, history, resources, identities, practices, and other contextual factors. The goal is not to reduce a community to a single “culture,” but to understand how different factors shape people's experiences and participation.

Consider an English-language learning program in which some students participate actively in certain activities but speak less in others. Rather than assuming that their behavior reflects a fixed cultural trait, the instructor could explore multiple explanations: comfort with the language, previous educational experiences, confidence, classroom power dynamics, fear of making mistakes, teaching methods, relationships among students, accessibility, or expectations about participation.

Students themselves might suggest solutions such as smaller discussion groups, additional practice time, peer support, different participation formats, more relevant lesson topics, or opportunities to contribute without being called on unexpectedly. Ethnographic inquiry helps the instructor understand the actual reasons behind participation patterns before deciding how to respond.

Reexamine your initial interpretations. What did you learn about participants' experiences, expectations, relationships, and context? Which early assumptions were supported, challenged, or complicated by later information? Revise your interpretations as your understanding develops.

Try to understand your original question from the perspectives of the people involved.

• Do community members define the issue in the same way you do?
• Are there contextual factors you were previously unaware of?
• Are participants interpreting program activities or conditions differently from what you expected?
• How important is the issue to participants, and what other priorities compete for their time and attention?
• Do participants and the organization share the same goals? If so, do they agree on how those goals should be pursued?
• Are program practices, institutional expectations, or community conditions creating barriers to participation or success?
• What community strengths, relationships, practices, or resources support participation and positive outcomes?
• Are there unintended or unexpected effects related to context, identity, relationships, or institutional practices?

In a language-learning class, for example, participation patterns might be influenced by differences in language proficiency, prior educational experience, age, confidence, classroom relationships, learning preferences, responsibilities outside class, or expectations about speaking in groups. Rather than assuming the reason, ask students and observe how these factors interact.

Translate what you have learned into information that helps answer your evaluation questions. Consider:

• How do community context, relationships, institutions, norms, and lived experiences influence program effectiveness?
• Are program methods culturally responsive, accessible, and appropriate to participants' goals and circumstances?
• Are any activities, policies, or requirements unintentionally creating barriers?
• What changes could make participation more accessible, respectful, useful, and rewarding?
• Which parts of the program appear most useful, and what does the ethnographic information suggest about why?

Continue gathering ethnographic information when it remains useful. Communities, institutions, programs, and relationships change over time. Continued listening and engagement can help organizations recognize those changes and adapt their work accordingly.

In Summary

Ethnography is an approach to understanding people's experiences, practices, relationships, and social context from their own perspectives. In evaluation, ethnographic information can help explain why participants experience programs differently, why particular strategies work well or create barriers, and how history, institutions, culture, relationships, and community conditions influence outcomes.

The group or setting being studied might be a classroom, neighborhood, workplace, organization, faith community, professional network, or broader social or cultural community. Whatever the setting, ethnographic inquiry requires time, relationship-building, careful observation, listening, ethical practice, and continued attention to diversity within groups.

Planning an ethnographic study involves identifying what you need to understand, determining what resources are available, building trust, choosing appropriate research questions and methods, engaging a range of participants, and documenting observations and conversations carefully. Ethical practice requires informed consent, privacy protection, transparency, attention to power, and meaningful community involvement in how information is interpreted and used.

Organizing and interpreting ethnographic information can deepen your understanding of evaluation findings and help you improve programs in ways that are more responsive to the people and communities involved. Continued engagement can help ensure that your understanding evolves as community priorities, relationships, and conditions change.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

What is ethnographic information?

__ Ethnographic information is information about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view.

__ Elements of ethnographic information include:

• Community norms
• Health conditions and knowledge
• Power and political realities
• Religion
• Economics
• World view

__ Ethnographic information is obtained directly from those who live it.

__ The gathering of ethnographic information takes place in the environment where members of the culture normally spend their time (natural setting).

__ Ethnographic information is meant to help you understand a culture from the point of view of its members.

­­__ Ethnographic information is simply descriptive, not judgmental.

Why might ethnographic information be important to evaluation?

__ In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

__ Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them.

__ Ethnographic information helps to clarify what needs to be addressed in order for participants to respond to your approach.

__ Ethnographic information can clarify the issue and its effect on and importance to the population of interest.

__ Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future.

When might you want to collect ethnographic information?

__ When you’re engaged with a population or cultural group that you’re not familiar with or part of.

__ When you’re working with a clearly-defined group that has had a chance to develop its own culture.

__ When an understanding of the context and culture of the community is fundamental to what you’re doing.

__ When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group.

Who should collect and interpret ethnographic information?

__ That depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc.

How do you gather ethnographic information?

__ Decide what kinds of information you need.

__ Determine what you have the resources to do.

__ Gain the trust of the group you’re concerned with

__ Plan your field study

• Decide on your questions.
• Decide on your methods of data collection.
• Decide whom you’ll need to contact and how.
• Work out ethical issues.

__ Carry out your field study, taking careful field notes, including jottings, descriptions, analyses, and personal reflections on what you learned.

How do you interpret ethnographic information?

__ Organize your data.

__ Write as complete a description as possible of the culture or the element(s) of the culture you’re concerned with from the perspective of its members.

__ Reexamine the analyses from your field notes.

__ Try to understand the answer to your original question from the point of view of those you’re concerned with.

__ Translate what you’ve learned about participants into answers to, or supporting information for answers to, your evaluation questions.

__ Continue to gather ethnographic information to guide your work.

Existing data can help you evaluate whether an intervention is working without requiring you to collect every piece of information yourself. When relevant, reliable data are already available, using them can save time and resources while strengthening your evaluation. It is important to have data to evaluate the effectiveness of an intervention.

To support an evaluation, existing data must be relevant to your population, issue, intervention, and evaluation questions. They also need to be available in a form you can analyze. You data may need to clean, combine, or otherwise prepare the data before use. This section explains how to find, assess, and use existing data thoughtfully.

WHAT IS EXISTING DATA?

Existing data is information that an organization, agency, researcher, or other source has already collected for reporting, program administration, research, or another purpose. Examples include administrative and program records, public datasets and dashboards, surveys, research studies, and community assessments. Some sources are updated regularly, while others provide a snapshot from a particular point in time.

 

COMMON SOURCES OF EXISTING DATA 

Some of the most common sources of data that may be useful for your evaluation include: 

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry 

Existing data is often available through online dashboards, open-data portals, reports, electronic record systems, downloadable files, or direct requests to the organization from which the data originated. Some information will still require a formal data request or data-sharing agreement, particularly when it contains confidential or sensitive information. 

TYPES OF EXISTING DATA YOU MIGHT LOOK FOR 

Depending on your evaluation questions, you might look for data about a specific population or geographic area, including: 

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population 

WHY COLLECT AND USE EXISTING DATA?  

There are sometimes good reasons to collect new data, including when the information you need is not available elsewhere. When you collect new data, you also have more control over who is included, what is measured, and how the information is gathered. However, primary data collection can be time and resource intensive. 

If the information you need, or something very close to it, already exists, there are several good reasons to find and use it. 

• Existing data can save substantial time and effort, especially when you need information about a large population, geographic area, or time period. They may also provide broader coverage or greater detail than your organization could collect on its own.
• Existing datasets may already have been cleaned, organized, analyzed, or documented by people with specialized expertise. This work can reduce the technical burden on your team, although you should still review the methods, definitions, and limitations before using the results.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The existing data you find may be more sweeping or more specific than what you’d be able to gather. They may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Existing data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for.  In cases like these, the use of data that already exist might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• The data may have been collected using established methods and systems. This can save time and strengthen your evaluation, but it does not guarantee that the data are free from bias, missing information, inconsistent definitions, or other limitations. You still need to understand how the data were collected.
• Existing data can help you examine change over time. Historical information can show whether an observed change is part of a longer trend and whether similar patterns appear in comparable communities, the state, or the nation. This perspective may be difficult to obtain through short-term data collection alone.
• Existing data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exist elsewhere. 

WHEN SHOULD YOU COLLECT AND USE EXISTING DATA? 

• When it’s available and accessible. This is the key question. If you know the data exist and can obtain appropriate access, they may be useful. If the data do not exist, are too difficult to locate, or cannot be shared because of privacy or other restrictions, you may need another approach.
• When it’s relevant. As with availability, the relevance of the data to what you’re trying to find out is a key issue. All the existing data in the world won’t do you any good if they don’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, existing data may be the only source of the information you need.
• When the data match your evaluation needs. Data should align with the population, place, time period, measures, and program elements you are studying. Select only sources that can help answer your evaluation questions; otherwise, the information may provide an incomplete or misleading picture. 

HOW DO YOU FIND AND USE EXISTING DATA? 

As you search for and use existing data, there are several questions you should ask. 

Are the data a good fit for your evaluation? 

Before using an existing source, look beyond whether it is easy to obtain. Ask whether it can answer your evaluation question and whether it can be meaningfully compared with any data you collect yourself. 

• Who and what do the data represent? Make sure the population, geographic area, and issue are relevant to your program.
• When were the data collected or updated? Older data can be useful for trends and context, but may not reflect current conditions.
• How were the data collected and defined? Review the data source’s methods, measures, and definitions. Two sources may use the same term—for example, “service received” or “graduation”—but mean different things.
• Are the data complete and sufficiently detailed? Consider missing data, small numbers, the level of detail available, and whether results can be broken down in ways that are useful and appropriate.
• What limitations or potential biases should you keep in mind? Administrative data reflect the people who use a service and the way staff record information; survey and research data may be affected by sampling, response rates, or measurement choices.
• Can you use the data ethically and securely? Follow data-sharing agreements and privacy requirements. Whenever possible, use aggregate or de-identified data rather than information that could identify individuals. 

WHAT INFORMATION DO YOU NEED, AND WHY? 

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants. 

Some possibilities include: 

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• Community-level indicators are specific, measurable signs of conditions or change across a community. Examples include program participation, services delivered, crime rates, or new cases of HIV. These indicators can help you track community trends and assess whether an initiative may be contributing to change. 

Community-level indicators describe trends for the community as a whole, rather than changes among individual participants. For example, an initiative to reduce alcohol use among young people might track weekend and nighttime single-vehicle crashes involving teenagers. Choose indicators because they are meaningful to the initiative and community—not only because data happen to be available. 

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include: 

• Demographics:
  • Demographic characteristics, such as age, race, ethnicity, gender identity, household composition, education, income, employment, housing, disability status, primary language, and geographic location
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level  
  • Other characteristics –primary language spoken, etc. 
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Alcohol and other substance use
  • Sexual behavior  
  • Various other behaviors – work habits, consumer patterns, etc. 
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Developmental and educational outcomes, such as completion of primary or secondary education and equitable participation by people with disabilities
  • General health and well-being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins) 

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes. 
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

Combining several years of data can produce more stable estimates when annual counts are small. Rolling averages—for example, comparing 1997–2000 with 1998–2001—can help show change over time while reducing year-to-year variation. The tradeoff is that multi-year estimates may make recent changes or program effects harder to detect.

Expanding the geographic area can also increase the number of observations and improve statistical stability. For example, you might use regional or state data when county-level counts are too small. However, broader estimates may not reflect local conditions and can hide meaningful differences among communities. 

Who is likely to have collected that information?

Public agencies, research organizations, universities, schools, health and human service providers, funders, advocacy groups, community organizations, and businesses may all maintain useful data. Start with the source most likely to collect information about your topic, population, or geographic area.

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

WHERE SHOULD YOU LOOK FOR EXISTING DATA? 

The question here is not only where to find existing information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble. 

Your own records and data 

Unless it’s brand new, your own organization should have administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information. 

Online data portals and websites 

Many public documents and datasets are available online. Start with the website or open-data portal of the government agency, organization, or institution most likely to have collected the data. In the U.S., federal, state, tribal, county, and municipal agencies often provide searchable reports, dashboards, or downloadable files. Similar sources are available in many other countries.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

Assess online data sources carefully. Favor information from the organization that originally collected or maintains the data, and look for clear documentation of methods, definitions, update schedules, and limitations. If a source is unfamiliar or a finding seems uncertain, compare it with other credible sources before using it. 

Go directly to the source

When data is not publicly available, contact the organization that maintains them. Explain what information you need, how you plan to use it, the level of detail required, and how you will protect privacy. Be prepared to discuss a data-sharing agreement, reasonable costs, or a mutually beneficial exchange of information or expertise. 

When requesting data from another organization for comparison, frame the conversation around shared learning rather than competition. Explain why the comparison is useful, how you will account for differences between programs or populations, and how findings will be interpreted and shared. An existing relationship built on trust and mutual respect can make this collaboration easier. 

Libraries

Libraries and librarians can help you identify credible datasets, government reports, scholarly research, archives, and specialized databases. They may also help you develop search terms, locate hard-to-find sources, and assess whether a source is authoritative. 

What are you planning to do with the data once you have it?

Plan how you will use the data before requesting or downloading them. Determine which variables, time periods, population groups, and level of detail you need. If you intend to compare existing data with information from your program, make sure the groups, definitions, measures, and collection periods are sufficiently similar to support a meaningful comparison.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the existing data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending. 

USING EXISTING DATA 

In many evaluations, you will combine existing data with new information collected from participants. Existing program or administrative records may also provide current information about participants, depending on what is recorded, how quickly the data are updated, and whether you have appropriate access. You may be able to find earlier data on participants to use as a baseline or data on a similar group to use as a comparison or control. 

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find existing data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both. 

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use existing data: 

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. A clearer picture of the community allows for a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved because of your work. 

In an evaluation, the intervention is the program, policy, service, or other action expected to contribute to change. The outcomes are the behaviors, conditions, or other results that may change. For example, a violence-prevention program is the intervention, while changes in violent behavior and related injuries are outcomes. 

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding. 

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group. 

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use existing data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change. The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find existing data that tells you whether the effects of your program continued, kept growing, or trailed off. 

In Summary

Existing data can help you understand community conditions, establish a baseline, identify trends, compare outcomes, and examine change over time. Useful sources include your own records as well as data maintained by public agencies, community organizations, health and human service providers, universities, funders, and other institutions. Select data that fit your evaluation questions, examine their quality and limitations, and follow all privacy and data-sharing requirements. Used thoughtfully, existing data can make evaluation more feasible and informative—especially for organizations with limited time and resources.

What are archival data?

___Archival data is data that already exists as a result of administrative procedures or past studies or evaluations

___Be aware of sources of archival data:

• Public records
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

___Be aware of the possible types of data available:

• Demographics
• Behavior
• Health and development outcomes
• Attitudes – racial, political, social, etc.
• Knowledge and awareness of issues
• Environmental conditions or factors affecting the population and/or your work

Why collect and use archival data?

___It’s easier and less time-consuming than collecting all the data yourself

___Archival data may have already been processed by people with more statistical expertise than you, making it easier to use in analysis

___Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources

___It’s quite possible that you can find more information than you’d be able to gather if you did it yourself

___Archival data could touch on important areas you might not have thought of, or identify patterns or relationships you wouldn’t have looked for

___It may eliminate the need to correct for such problems as lack of inter-rater reliability or observer bias

___Archival data allows the possibility of looking at the effects of your work over time

___Archival data can make it possible for small organizations with limited resources to nonetheless conduct thorough evaluation studies

When should you collect and use archival data?

___When it’s available

___When it’s relevant

___When you don’t have the time and/or resources to collect the data yourself

___When it can truly inform your evaluation

How do you collect and use archival data?

___Determine what information you’re looking for and why

Possible categories of information you’d be interested in:

___Data on past participants

___General information on the population and/or the community you’re working with

___Specific information on appropriate characteristics of the population you’re working with

___Cultural information

___Data on a similar group that can be used as a control or comparison

___Results of previous studies

___Determine who is likely to have collected that information:

• Government departments and agencies, the census, and other entities whose data are likely to be available in public records
• Research organizations
• Academia
• News media
• Foundations and other private funders
• Hospitals and other health care providers
• Mental health providers
• Human service and other community-based organizations
• Advocates and watchdog organizations
• Community activists
• Community economic development organizations
• Businesses and corporations

___Decide where you should look for archival data

• Your own archives
• The Internet
• The original source
• Libraries

___Decide what you plan to do with the data once you have it

Use archival data:

___To better understand the context of your evaluation

___To identify areas to address

___To establish a baseline against which to measure your results

___To identify already-existing trends that may affect the results of your evaluation study

___To establish a standard of comparison against which to measure your efforts

___To act as a control or comparison group

___To provide data for a longitudinal study

Tool 1: Common Secondary Data Sources Used to Address Gender-Based Violence

This tool from Strategic Prevention Solutions provides descriptions of common secondary data sources useful for addressing gender-based violence, along with common challenges with secondary sources.

Whenever you begin a new job or start a project, you will probably want to understand how well your work is going. You might ask yourself: "Is my work meeting its goals and the expectations of the people involved or affected?"

Community initiatives also benefit from evaluation throughout their work. Asking community members, participants, staff, and other stakeholders whether plans are working as intended is one important part of evaluation. Positive findings can help confirm that an initiative is moving in a useful direction, while other findings can identify where adjustments may be needed.

This chapter explains why evaluation is important and offers step-by-step approaches for evaluating your group's work. Evaluation can help you recognize progress, learn from challenges, and make informed decisions about what to continue, strengthen, or change.

What does it mean to evaluate a community initiative?

Evaluation is the systematic process of understanding the value, implementation, and effects of your work. After developing and carrying out an initiative, you need information about how well it is working, for whom, under what conditions, and where improvements may be needed.

In some areas of life, feedback is immediate. You press the brakes and the car slows down. You shoot a basketball and see whether the ball goes in. Community initiatives are more complex. Changes may take time, multiple factors may influence outcomes, and different people may experience the same initiative differently. Evaluation provides a structured way to understand what is happening rather than relying only on impressions or assumptions.

In this chapter, evaluation refers to gathering and interpreting information about your group, its activities, and the changes connected to its work. This information can help you determine which parts of your action plan are working well, which are creating barriers, and which should be adapted. Evaluation findings can then inform changes to goals, strategies, implementation, and future planning.

The sections in this chapter focus specifically on ways to gather useful information. The next chapter discusses ways to use that information to strengthen and improve your initiative. Organizations sometimes hesitate to evaluate their work, so it is helpful to recognize and address some common concerns.

Common concerns about evaluation

People may feel uncertain about evaluation because they are unfamiliar with the process, concerned about the time it requires, or worried about what the findings might show. These are understandable concerns, but they do not have to prevent useful evaluation.

I don't know how to conduct an evaluation.

Evaluation skills can be learned. This chapter walks through the basic steps involved in planning an evaluation, gathering information, interpreting findings, and using what you learn to improve your work.

I don't have the time.

Evaluation does require time and attention. However, identifying problems, barriers, or implementation challenges early can prevent larger difficulties later. Evaluation can also help organizations focus limited time and resources on approaches that are most useful.

Once useful information becomes part of routine program operations, evaluation may require less additional effort. Attendance records, participant feedback, implementation notes, community indicators, and other information can often be collected as part of regular work rather than through a separate process.

The findings may be disappointing or create challenges for us.

Evaluation may show that some parts of an initiative are not producing the outcomes you expected. That information can still be valuable. Identifying limitations gives you an opportunity to understand what is happening, make changes, and improve the quality and responsiveness of the initiative. Evaluation should be approached as a tool for learning and accountability rather than simply as a judgment of success or failure.

Why should you evaluate a community initiative?

Effective community work requires attention throughout planning, implementation, and follow-up. Consider a musician learning a new piece for a performance. Practice is essential, but feedback can help identify whether tempo, technique, interpretation, or another element needs adjustment. Without feedback, it may be difficult to recognize what is working well and what needs additional attention.

Community initiatives also need feedback throughout their work. An organization may devote substantial time, energy, and resources to pursuing its goals, but without evaluation it may be difficult to know whether the effort is reaching the people intended, responding to community priorities, being implemented as planned, or contributing to meaningful change. Evaluation helps identify progress and informs adjustments along the way.

There are many reasons evaluations are valuable. Some of the ways evaluation can benefit a community initiative include:

• Evaluation can document progress and strengthen support. Evidence of meaningful outcomes can help community members, partners, funders, and decision-makers understand the value of the work. For example, an initiative might document increased volunteer participation, improved access to preventive health services, stronger community partnerships, or other accomplishments. Clear evidence can support continued investment and help organizations communicate what they have achieved.
• Evaluation can turn challenges into opportunities for learning. When an initiative does not meet its goals, evaluation can help identify why. The issue may involve implementation, accessibility, resources, outreach, program design, changing community conditions, or another factor. Understanding these challenges provides a stronger basis for improvement than simply labeling the initiative unsuccessful.
• Evaluation can recognize accomplishments and sustain motivation. Seeing evidence of progress can reinforce the value of people's efforts. Findings that identify areas for improvement can also support motivation when they are treated as information for learning rather than as personal or organizational failure.
• Evaluation can strengthen future action. Once you understand what has worked, what has not, and why, you can refine your strategies and make better-informed decisions about future efforts. Sharing these lessons may also help attract new partners or strengthen community support.
• Evaluation can help you understand important aspects of the initiative. Suppose a free health screening is well attended while a monthly healthy-cooking program has low participation. Evaluation can help you explore why. Scheduling, transportation, location, childcare, cost, communication, cultural relevance, accessibility, or competing responsibilities might all influence participation. Evaluation can help explain not only what happened, but why.

When should you evaluate a community initiative?

Evaluation should take place throughout the life of an initiative.

When your plan is being implemented:

• Establish baseline information for the behaviors, conditions, experiences, or outcomes you hope to influence. Understanding where things began makes it easier to assess change over time.
• Examine the effects your work is having on participants and the broader community, including intended and unintended outcomes.
• Continue revising and updating action plans as new information becomes available.
• Keep the group focused on its goals while remaining responsive to community priorities. You may want to use a survey that assesses community goals and use the findings to refine priorities.

When some of your action plans are complete:

• Use evaluation findings to understand the initiative's outcomes, identify what should be sustained or adapted, and plan future work. For example, community-level indicators can help show whether broader community conditions are changing in ways related to the initiative.

How do you begin an evaluation?

Evaluation can seem complex when viewed as a single large task, but it can be broken into manageable steps. Begin by identifying what you most need to understand about your work.

Before moving into specific methods, keep one principle in mind: your evaluation should address questions that matter to community members, program participants, staff, partners, decision-makers, and funders. Time, funding, staffing, and available information may limit what you can examine, so focus on questions that will help your organization learn and make useful decisions. Every initiative is different, but the following questions are relevant to many community efforts:

• Who participated in the initiative, and whose perspectives may be missing?
• What programs, policies, practices, services, relationships, or community conditions changed?
• What changes occurred in people's experiences, behaviors, access to resources, or other relevant outcomes?
• What evidence suggests that your initiative contributed to those changes, and what other factors may also have influenced them?

Tips for implementing a successful evaluation

Build a commitment to evaluation.

Evaluation is most useful when the organization sees it as part of learning and improvement rather than simply as a requirement. Be clear about why you are evaluating the initiative, what you hope to learn, and how the findings will be used.

For example, if an initiative seeks to reduce tobacco use among young people, evaluation questions might examine changes in tobacco use, access to tobacco products, exposure to prevention efforts, enforcement of age-of-sale requirements, or other community and policy conditions that influence use.

Your evaluation should be grounded in your stated objectives or goals.
When you planned your initiative, you should have identified what you hoped to accomplish. Clearly defined objectives provide a foundation for deciding what information to collect and how progress will be assessed.
For example, if your objective is to reduce tobacco use among local high school students to 5% by May 2032, you can compare later estimates with that target while also examining trends, differences among groups, and other factors that may influence the result. Clear objectives make evaluation more focused, but they should not prevent you from noticing important outcomes that were not originally anticipated.

For each objective, identify indicators that provide meaningful and appropriate measures of progress.

Indicators should reflect what you are actually trying to change. Whenever possible, use multiple measures rather than relying on a single number to represent a complex issue.

For example, if your objective is to prevent violence involving young people in a local school or community, possible indicators might include violence-related injuries, reports of feeling safe at school, participation in conflict-resolution or mentoring programs, school attendance, access to supportive adults, reported incidents, and other measures of community safety and well-being. Arrest data may sometimes provide additional context, but they should be interpreted carefully because they also reflect reporting and enforcement practices.

If your objective is to increase access to heart-healthy foods, possible indicators might include the availability and affordability of fruits, vegetables, whole grains, and other nutritious foods; the number of stores or restaurants offering healthier options; participant-reported access; or changes in organizational food policies. Measures should reflect access and opportunity rather than assuming that individual choices alone determine health.

You need to collect information on each of these indicators.

Sometimes the information you need already exists. For example, library records might help evaluate efforts to increase borrowing, while transportation or public safety records might provide information about traffic conditions. In other cases, relevant information may not be available and you may need to collect it directly through observation, surveys, interviews, sensors, administrative tracking, or other appropriate methods.

Whenever possible, gather information from before and after implementation and, when useful, at several points in between. These data can help you determine whether meaningful change occurred. However, a simple before-and-after difference does not by itself prove that the initiative caused the change. Consider trends, outside events, differences among groups, and other factors when interpreting the findings.

Use the results to adjust the program or intervention as necessary.

Ask whether the initiative is moving toward its objectives and whether participants and communities are experiencing the intended benefits. If progress is limited, use the findings to understand why before deciding what to change. The issue may involve program content, outreach, accessibility, implementation quality, community conditions, or the objective itself.

For example, if a tobacco-prevention initiative is not producing the expected change among high school students, you might examine whether young people find the program relevant, whether cessation support is available, how social and commercial environments influence tobacco use, whether new nicotine products are affecting behavior, and whether policies and enforcement practices support prevention. Findings like these can guide changes that address both individual behavior and the broader conditions influencing it.

In Summary

Begin by asking, "What do we need to understand about our initiative?" Evaluation can then help you gather and interpret the information needed to answer those questions.

Evaluation should be an ongoing part of community work rather than something conducted only at the end. It can help you recognize progress, understand participant and community experiences, identify barriers and unintended effects, strengthen accountability, and make informed decisions about what to continue, adapt, expand, or discontinue.

The following sections in this chapter describe methods you can use to evaluate your initiative systematically and use what you learn to strengthen future work.

What is evaluation?

___You understand that evaluating means determining the value of your work

What are some objections to evaluation and how can you resolve those objections?

___"I don't know how to do an evaluation."

• Use a resource such as the Community Tool Box to find guide once the steps you should take to plan and implement the evaluation, and use the results.

___"I don't have time."

• Although it is true that evaluation takes time, it will save time in the long run, by pointing out potential problems before they become disasters

___"The results may be negative or hurt us."

• If you begin evaluating early, you should be able to prevent any major problems
• If you do find problems, you can use that to improve the quality of your program

Why should you evaluate your program?

___Success is reinforcing

___Failure is instructive

___Evaluation can make you feel good

___Evaluation raises the chances of further action

___Evaluation can help you understand important aspects of the initiative

___You should evaluate your initiative when your plan is in action, or when some of your plans have been completed

How should you do an evaluation?

___You want to evaluate

___You have evaluated in terms of objectives or goals

___You have identified criteria or indicators that will provide ways to measure your objectives

___You have collected data on each of these indicators

___You have used the results to adjust the program or intervention as necessary

 

 

 

What does it mean to monitor your progress?

Monitoring means regularly collecting and reviewing information about your initiative's activities, progress, and emerging outcomes. It helps you compare what is happening with your plans and community priorities. Monitoring can identify implementation challenges and changes that need further investigation, although it does not by itself establish what caused an outcome. A useful monitoring system can help you:

• Understand how the initiative operates and who it reaches
• Make informed decisions about activities, resources, and improvements
• Communicate accomplishments, challenges, and lessons learned
• Build community participation and support
• Inform funding decisions and account for the use of resources

Monitoring is already part of many everyday activities. Teachers review students' work to understand learning progress. A household might compare utility use across several months. A community organization might track attendance and ask participants about their experiences. For a community initiative, monitoring provides a consistent way to document activities, accomplishments, challenges, and changes over time.

The monitoring approach described here has three key parts:

• Clearly defined process and outcome measures
• A system for observing and documenting activities and changes
• Regular feedback and discussion to inform action

This section explains process and outcome measures and how to organize an observational monitoring system. Later sections discuss ways to share feedback with coalition members and the wider community. Plan for that use from the beginning by agreeing on who needs the information and which decisions it will inform.

Process measures

Process measures describe how an initiative is implemented, including its activities, participation, reach, and quality of delivery. Teams may track these alongside inputs, such as funding, and outputs, such as workshops delivered or plans completed. Keeping these distinctions clear helps you avoid treating activity counts as evidence of improved community outcomes.

Implementation monitoring can cover many aspects of your initiative, such as:

• Participation: The number of people involved, frequency of attendance, new and continuing participation, and barriers to involvement.
  • Example: Seventeen people attended the most recent coalition meeting. Ten had attended previously, and seven were participating for the first time.
• Planning products: Agreed objectives, action plans, bylaws, and other documents or structures that support the initiative.
  • Example: The coalition approved an action plan identifying priorities, responsibilities, and implementation dates.
• Media coverage and communication: Coverage through radio, television, print, online news, and other channels, distinguishing independent coverage from the initiative's own communications.
  • Example: A local radio station aired three interviews about the initiative. The team recorded the dates and topics, noting that broadcasts alone do not show how many people listened or acted on the information.
• Financial and other resources: Grants, donations, and in-kind contributions, such as meeting space, equipment, or donated services. These are resources supporting the work rather than outcomes of the intervention.
  • Example: A fundraising lunch generated $8,000 in gross revenue for local substance use prevention and recovery-support efforts. The team recorded event expenses separately to determine the net amount available.
• Services and activities delivered: Classes, workshops, publications, referrals, and other services or communications provided through the initiative.
  • Example: Childcare providers delivered four nutrition education workshops. Records documented attendance, languages offered, and participant feedback.
• Community actions: Activities intended to encourage changes in programs, policies, practices, or community conditions.
  • Example: Residents and coalition members met with local service providers to request evening appointments for people unable to attend during daytime hours.

Event logs can help document these activities and resources consistently. Select measures that relate to your goals and intended decisions rather than trying to record everything. An example of an event log appears later in this section.

Outcome measures

Outcome measures describe changes relevant to the initiative's goals, such as improved access, changes in practices, or changes in people's experiences and well-being. New programs, policies, and practices may be intermediate outcomes when the initiative aims to change community systems. Their adoption does not automatically demonstrate improved conditions or establish that the initiative caused the change.

Examples of intermediate community changes include:

• Changes in programs: New or adapted services that respond to community priorities.
  • Example: A community center introduced a parenting-support program developed with local families. Later monitoring examined who could access it and whether participants found it useful.
• Changes in policies: Adoption or revision of rules and commitments that guide organizational or community practices.
  • Example: Following discussions with residents and the coalition, a service agency adopted a policy requiring its appointment system to offer evening options. The team documented the policy's approval and planned to track implementation.
• Changes in practices: New or revised ways of carrying out work.
  • Example: Local service providers began offering evening appointments. The team then tracked availability, use, and whether the change reduced scheduling barriers.

Together, implementation and outcome measures help explain what the initiative is doing and what is changing. Interpret them alongside community feedback and information about other influences. A count of new policies, for example, becomes more useful when you also know whether they were implemented and whom they benefited.

Why do you need to monitor your progress?

Monitoring is useful when the information leads to discussion and decisions. Before collecting data, agree on what the team needs to understand, who will review the findings, and how often. A focused monitoring system can help in the following ways:

• Data can show where the initiative places its effort. For example, records might show extensive service delivery but little progress on a planned policy change. Both may be valuable; compare the pattern with the initiative's goals and community priorities to decide whether adjustments are needed.
• Data can help identify who is reached and where change is occurring. Is the initiative making progress in schools but encountering barriers in other settings? Who can access the activities, and whose experiences are missing? Examine differences where the information is adequate and can be reported without compromising privacy.
• Data can reveal which strategies are being implemented. For example, records may show that information sessions are taking place while planned peer support or service referrals have not yet started. This can prompt discussion about resources, responsibilities, and implementation barriers.
• Data can support learning, communication, and accountability. Staff and community partners can use findings to improve activities, explain accomplishments and challenges, invite participation, and inform funding decisions. Reports should represent the full picture rather than highlight only favorable results.

How do you monitor your progress?

Begin by agreeing on a manageable set of measures, clear definitions, and responsibilities for collecting and reviewing information. Decide how records will be stored, who can access them, and how summaries will be used. The following steps describe a practical system built around event logs.

Collect data by completing event logs and other forms regularly

Event logs are structured records of important activities and developments. They can document actions taken by the initiative and new or modified programs, policies, or practices relevant to its goals. Record the initiative's role and supporting evidence without assuming that every related change resulted from its work.

An event log might include:

• The date of the event, using a consistent format
• A description of the event, including:
  • How it relates to the initiative's goals and why it matters
  • What happened, what evidence supports the record, and what follow-up is needed
• Relevant details, including:
  • The roles or groups involved, avoiding unnecessary identifying information
  • Which organizations contributed staff time, funding, space, or other resources
  • The community setting, strategy, or objective addressed
  • Whether this was a new activity, a repeated activity, or an update to an earlier record

For example, consider a hypothetical youth well-being coalition working with students to expand their role in school decisions. A member records: "On September 15, student representatives and coalition members met with the school principal to propose a student-led action committee. The principal approved the committee's formation, and students agreed to develop its membership process and priorities. The approval creates an opportunity for student leadership; the committee's launch and influence on decisions will be tracked separately."

The event log records the date, participants' roles, purpose, decision, and next steps. Meeting notes provide supporting documentation. Separating approval from implementation helps the coalition avoid counting a planned change as a completed activity.

Involve staff and community members in completing the logs

People directly involved in an activity are often well placed to document it. Assign responsibility for each record so that information is not missed or counted more than once. Agree on a prompt reporting schedule, such as within a week when practical. Provide a simple template, examples, and accessible ways to contribute, then send completed records to the person or team coordinating monitoring.

Gather information carefully and accurately

The time required depends on the initiative's size, activity level, and reporting needs. Try the system on a small scale to estimate the workload and simplify unnecessary steps. Train contributors to use consistent definitions, distinguish observations from interpretations, and identify information they have not yet confirmed. Review records for missing details, duplicate entries, and errors.

Organize the data so that it can be used

Store records in a format the team can maintain, such as a shared spreadsheet, database, or organized paper system. Use consistent categories and a unique identifier for each event. Keep sensitive information in appropriately restricted records, and document corrections or changes in definitions. Clear organization makes it easier to compare reporting periods and answer questions without repeatedly reconstructing what happened.

Summarize the data and share findings with the group

Prepare regular summaries that connect activities and observed changes with the initiative's goals. Use counts, brief descriptions, or charts as appropriate, and explain gaps or limitations. Distinguish new developments from repeated activities and avoid treating every event as equally important. Discuss findings with staff and community partners, agree on any action needed, and identify who will follow up and when.

In Summary

Monitoring helps a community initiative understand its implementation, track progress, and respond to emerging information. A useful system combines clear measures, consistent records, and regular discussion of findings. Event logs can document activities and community changes, while surveys, interviews, and other evaluation methods help explain experiences and outcomes. Use what you learn to guide action, and keep the monitoring process practical enough to sustain.