Chapter 36. Introduction to Evaluation | Community Tool Box

This section was originally adapted from the 1999 CDC report "Framework for Program Evaluation in Public Health," developed by Bobby Milstein, Scott Wetterhall, and the CDC Evaluation Working Group. It retains the original framework's organization while updating the discussion and identifying subsequent developments in evaluation practice.

Around the world, communities develop programs and initiatives to improve the conditions that shape people's lives. These efforts may prevent violence, expand access to safe and affordable housing, support student learning, or address other priorities identified by community members.

How do we know whether these programs are working, for whom, and under what conditions? How can we improve their effectiveness, accessibility, and fairness? And how can an organization decide which promising approaches are appropriate for its community?

Program evaluation provides a systematic way to explore these questions. By combining evidence with community knowledge, organizations can understand implementation, assess results, identify unintended consequences, and use what they learn to improve their work.

In 1997, the United States Centers for Disease Control and Prevention (CDC) convened an Evaluation Working Group to organize the essential elements of program evaluation. Its framework was published in 1999 and became the basis for this section. CDC updated the framework in 2024. The discussion below preserves the original six-step teaching structure and explains how it relates to the newer guidance.

Before beginning, it helps to define several terms used throughout this section.

By evaluation, we mean a systematic process for assessing the merit, value, or significance of an effort. Evaluation brings together questions, evidence, interpretation, and judgment to support learning and decisions. Its methods should fit the purpose, community context, and consequences of those decisions.

Throughout this section, the term program describes the effort being evaluated. It may include activities intended to improve outcomes across a community, within settings such as schools or workplaces, or among particular populations, such as young people, people affected by violence, or people living with HIV. This definition is intentionally broad.

Examples of different types of programs include:

• Direct service interventions, such as a program offering free school breakfasts to support children's access to nutritious food
• Community organizing efforts, such as a campaign led by farmworkers to improve wages and working conditions
• Research initiatives, such as a community partnership examining approaches to reducing racial inequities in access to health services
• Monitoring and surveillance systems, such as a system that tracks housing conditions to inform inspection and repair priorities
• Advocacy efforts, such as a campaign seeking policies that reduce exposure to secondhand smoke
• Social marketing campaigns, such as a locally developed campaign helping families find available nutrition and infant-feeding support
• Infrastructure and capacity-building projects, such as an initiative strengthening agencies' ability to support community-led development
• Training programs, such as a workforce development partnership that combines job training with support for transportation and other employment barriers
• Administrative systems, such as a revised appointment process intended to reduce waiting times and improve access to services

Program evaluation, the focus of this section, supports learning and accountability in community health and development work. It can examine a specific project, a set of related activities, or a broader initiative. The scope should be explicit so that the evaluation's questions and conclusions match the effort being assessed.

Community members and other interested or affected parties include people who participate in a program, people affected by its decisions, those who implement or fund it, and those who may support or question its approach. Earlier evaluation literature often uses the term stakeholders; newer CDC guidance uses interest holders. Useful questions include: Who is affected? Whose knowledge is needed? Who makes decisions? Whose perspectives have been excluded?

This section presents a practical framework for developing a shared understanding of evaluation. Its goal is to help people involved in community health and development assess their efforts, learn together, and use findings responsibly.

Why evaluate community health and development programs?

Evaluation can be closely connected to everyday program operations. Our emphasis is on practical, ongoing evaluation that involves program staff, community members, and other partners, with specialized expertise when needed. This approach can support both program improvement and accountability to the people a program is intended to serve.

For example, evaluation complements program management by:

• Clarifying program plans, assumptions, and intended results
• Improving communication and shared understanding among community members and partners
• Gathering evidence and feedback to improve effectiveness, address unequal outcomes, and account for the use of resources

People working to improve their communities already evaluate their work informally when they ask questions, consult partners, reflect on feedback, and adjust their approach. For low-stakes decisions, these practices may provide enough information. When decisions involve substantial resources, sensitive information, or significant consequences for people, a more systematic and documented evaluation can make the basis for those decisions clearer and more defensible.

How do you evaluate a specific program?

Before beginning an evaluation, work with the people involved to answer these questions:

• What will be evaluated, and how will the findings be used?
• What criteria will be used to assess program performance, and who will help choose them?
• What levels of performance would indicate success, including fair access and outcomes?
• What evidence is needed to assess performance against those criteria?
• What conclusions are justified by the evidence, and what uncertainties remain?

Consider Drive Smart, a hypothetical program designed to reduce alcohol-impaired driving. The following targets illustrate how an evaluation might be planned; they are not recommended benchmarks for every community. A real program would establish its targets using baseline information, available resources, community priorities, and a realistic timeline.

• What will be evaluated?
  • Drive Smart's public education and "Safe Rides" transportation service, including their reach, accessibility, implementation, and relationship to alcohol-impaired driving.
• What criteria will be used to assess program performance?
  • The percentage of community residents familiar with the program and its goals
  • The number of people using "Safe Rides," including whether the service reaches residents facing transportation barriers
  • The percentage of respondents reporting alcohol-impaired driving during a consistently defined period
  • The number of recorded crashes involving alcohol impairment, interpreted alongside changes in traffic volume, reporting, and other relevant conditions
• What levels of performance would indicate success?
  • 80% of community residents will know about the program and its goals after the first year.
  • The number of people using "Safe Rides" will increase by 20% relative to a defined baseline during the first year.
  • The percentage reporting alcohol-impaired driving will decrease by 20% relative to baseline during the first year. For example, a decline from 10% to 8% is a 20% relative decrease, or a decrease of 2 percentage points.
  • Recorded crashes involving alcohol impairment will decrease by 10% relative to a comparable baseline period during the first two years, with attention to uncertainty when counts are small.
• What evidence will indicate performance against these targets?
  • A community survey using a defensible sampling approach and accessible participation options will assess awareness and reported behavior at baseline and follow-up. The evaluation will examine nonresponse and the limitations of self-reported information.
  • "Safe Rides" records will document service use, distinguishing trips from unique users where this can be done without collecting unnecessary identifying information.
  • Crash records will provide information on documented alcohol involvement, with consistent definitions and attention to missing information or changes in reporting practices.
• What conclusions are justified by the available evidence?
  • What changed, for whom, and how plausibly did Drive Smart contribute? Could other policies, services, or community conditions explain the findings?
  • If intended changes have not occurred, should the program adjust its approach, address implementation barriers, allow more time, or gather better evidence?

The following framework provides an organized approach to answering these questions.

A framework for program evaluation

Program evaluation helps people understand and improve community health and development efforts. The original CDC framework presented here organizes evaluation around connected steps and standards for quality. It is a practical guide that can be adapted to a program's purpose, resources, and setting.

The original framework contains two related dimensions:

• Steps in evaluation practice
• Standards for evaluation quality

The illustration shows the original 1999 framework. Its six connected steps provide a useful teaching sequence, but evaluation is iterative: questions about evidence or use may require revisiting earlier decisions. Planning for how findings will be used should begin at the outset, and engagement should continue throughout the process.

CDC's 2024 revision uses six steps: assess context; describe the program; focus the evaluation questions and design; gather credible evidence; generate and support conclusions; and act on findings. Its accompanying guidance emphasizes collaboration, fair and just evaluation practices, and ongoing learning across every step. The original sequence below remains the organizing structure for this section, with those considerations incorporated throughout.

• Engage community members and other interested or affected parties
• Describe the program
• Focus the evaluation questions and design
• Gather credible evidence
• Develop and justify conclusions
• Use findings and share lessons learned

Use these steps as a connected learning process, adapting decisions as the program and its context change.

The original framework also drew on 30 evaluation standards organized into four categories. Those historical categories are retained below to explain the original model:

• Utility
• Feasibility
• Propriety
• Accuracy

These categories help assess whether an evaluation is useful, practical, ethical, and technically sound. CDC's 2024 framework uses five standards: relevance and utility; rigor; independence and objectivity; transparency; and ethics. When planning an evaluation, identify the applicable guidance and document how its standards will inform your decisions.

Engage Community Members and Other Interested or Affected Parties

People and organizations may be affected by both an evaluation's findings and the decisions made from them. Evaluation therefore requires attention to relationships, power, and differing perspectives. Community members, program participants, staff, partners, funders, and decision-makers may each contribute knowledge that others lack. Meaningful engagement gives people opportunities to shape questions, interpret findings, and influence decisions rather than simply provide data.

Participation can strengthen relevance, trust, and shared responsibility, but it does not guarantee agreement. People should be able to question methods, identify harms, or disagree with conclusions without pressure to endorse the program or the evaluation.

Although engagement appears first in the original framework, it belongs in every step. Begin by understanding the community context, identifying whose participation is needed, and agreeing on how their contributions will affect the work.

Three overlapping groups are important to involve:

• People or organizations involved in program operations may include community leaders, volunteers, collaborators, coalition partners, funders, administrators, managers, and staff. Their knowledge can clarify how the program actually operates and what resources it needs.
• People or organizations served or affected by the program may include participants, families, neighborhood organizations, advocacy groups, institutions, public officials, and other residents. Include people who face barriers to participation and those who question the program. Listening to different perspectives and engaging people who may oppose the program can identify blind spots and strengthen the evaluation.

Also consider people who could experience unintended consequences from decisions based on the evaluation. For example, changes to service locations, eligibility, hours, or funding may affect people who are not currently participating. Their perspectives can help identify consequences that program records alone will miss.

• Primary intended users of the evaluation are the specific people expected to act on its findings. They may include community representatives, participants, program staff, organizational leaders, and funders. They are not necessarily the same people as the program's intended participants. Identify these users early, clarify the decisions they face, and maintain communication without allowing their preferences to override evidence or the interests of affected communities.

The form and extent of community and partner involvement will vary. People may help govern the evaluation, develop questions, collect information, interpret results, or decide how findings are shared. Offer accessible participation options, language support, reasonable compensation where appropriate, and clear explanations of the time and responsibilities involved.

Agree on how decisions will be made, how power will be shared, and how the group will resolve conflicts. Document different views and create ways for people to raise concerns safely. These practices help prevent the priorities of the most influential participants from automatically determining the evaluation's direction.

Describe the Program

A program description explains the effort being evaluated: what it intends to accomplish, how its activities are expected to contribute, who is involved, and what resources and conditions shape its work. It should distinguish the program's planned approach from what is actually being implemented.

The description establishes the frame of reference for evaluation. For example, a program focused on enforcing alcohol sales restrictions has different activities and immediate outcomes from a program providing safe transportation for young people. A clear description helps the group choose appropriate questions and examine how particular activities may contribute to results.

People may also have different understandings of the program's purpose. In a youth development initiative, some may emphasize educational support, others access to employment, and others young people's leadership and decision-making. Discuss these expectations openly and include young people's own priorities when defining the program.

A shared description makes evaluation more useful, even when some differences remain. Document areas of agreement, unresolved questions, and assumptions about how change will happen. Developing this description can improve program planning before outcome data are available.

Include the following aspects when describing a program.

Statement of need and opportunity

A statement of need and opportunity describes the issue, goal, or possibility the program addresses, together with community strengths and existing efforts. Explain the nature of the issue, who is affected, its scale, and how it is changing. Identify relevant social, economic, and institutional conditions rather than treating a community's challenges as individual shortcomings.

Expectations

Expectations describe the program's intended results and what would count as meaningful progress. Organize them into short-term, intermediate, and longer-term outcomes, explaining how one is expected to contribute to another. A program's vision, mission, goals, and objectives express these expectations at different levels of detail. Check whether expectations are realistic and shared by the people affected.

Activities

Activities are the actions a program takes to contribute to change. Describe its components, their sequence, how they relate, and who is responsible for each. Distinguish activities delivered by program staff from those led by community members or partner organizations. Also identify external developments, such as changes in transportation services, employment opportunities, or public policy, that may influence results.

Resources

Resources include time, skills, relationships, facilities, equipment, information, funding, and other assets available to the program. Reviewing resources helps reveal whether planned activities are adequately supported. Include contributions from community members and partners, especially work that is unpaid or easily overlooked. Clear cost information is also needed when assessing efficiency, cost-effectiveness, or costs and benefits.

Stage of development

A program's stage of development reflects how established its activities and supporting systems are. Programs change over time, and their components may develop at different rates. A newly funded initiative may require different evaluation questions from a long-running program that is expanding or changing its approach.

Three useful phases are planning, implementation, and assessment of effects or outcomes. During planning, evaluation can test assumptions and refine the approach. During implementation, it can examine delivery, participation, accessibility, and adaptations. Once sufficient time has passed, it can assess intended and unintended outcomes. These phases may overlap, and planning for outcome assessment often needs to begin before implementation.

Context

A description of the program's context considers community history, geography, culture, politics, economic conditions, institutional practices, and related efforts. A responsive evaluation is sensitive to these influences and to differences in power and access to resources. Context helps explain findings and assess whether an approach may transfer elsewhere. For example, a housing initiative may need substantial adaptation across communities with different housing markets, transportation systems, or local regulations.

Logic model

A logic model summarizes how a program is expected to work. It connects resources and activities with outputs and intended outcomes, making assumptions about change visible. A flowchart, map, or table can show these relationships, including feedback loops or external influences when a simple sequence would be misleading.

Developing a logic model together can clarify program direction and identify questions for evaluation. The model expresses a theory about how change may occur; it does not establish that the program caused an outcome. If a model is used to estimate outcomes that were not directly measured, report those results as estimates and explain the evidence, assumptions, and uncertainty behind them.

The detail needed in a program description depends on the evaluation's purpose. Combine relevant documents, discussions with participants and staff, and observations of activities to create an accurate account. Check whether written plans match actual practice. Include contextual factors such as staff turnover, resource constraints, policy changes, community leadership, and partnerships that may affect implementation and results.

Focus the Evaluation Questions and Design

Focusing the evaluation means deciding what you most need to learn, why it matters, and how the findings will inform action. An evaluation cannot answer every possible question. Agreeing on priorities helps the group use its time and resources well while keeping community concerns visible.

Different questions require different designs and types of evidence. Plan before collecting information, including how you will analyze it and protect participants. Some design choices are difficult to change later, but the plan should allow justified adaptations when conditions change or important questions emerge.

Consider the following issues when focusing an evaluation:

Purpose

Purpose describes the overall reason for the evaluation. A clear purpose guides the questions, methods, level of detail, and intended use of findings. It also helps the group distinguish learning and improvement from other purposes, such as accountability or decisions about expansion.

There are at least four broad purposes for a community program evaluation:

• To gain insight. Evaluation can help a group explore a proposed approach, such as whether a resident-led neighborhood safety initiative fits local priorities and resources. Findings from similar efforts, together with local knowledge, can clarify assumptions, feasibility, and adaptations needed before implementation.
• To improve how work is carried out. Evaluation can examine what was delivered, who participated, what barriers arose, and how implementation differed from the plan. This information can guide improvements in quality, accessibility, coordination, and the use of resources.
• To assess program outcomes and contributions. Evaluation can examine changes associated with a program, such as improvements in school completion or access to stable housing. Assessing whether the program caused those changes requires a design that can address alternative explanations. Outcome findings can support accountability and decisions, provided the conclusions match the strength of the evidence.
• To support learning among people involved. Participating in evaluation can build skills, strengthen relationships, and help people reflect on their work. These benefits should arise through transparent and voluntary participation. They may:
  • Strengthen community decision-making, for example by giving participants meaningful authority in choosing questions and interpreting findings;
  • Support reflection on the program, for example through a follow-up conversation that helps participants identify useful experiences and unmet needs;
  • Build staff and community evaluation skills, including collecting, analyzing, and interpreting information; or
  • Contribute to organizational learning, for example by clarifying how activities align with the organization's mission and community priorities.

Users

Intended users are the people expected to apply the findings. Their information needs, decisions, and timelines should help shape the evaluation, alongside the interests of people affected by those decisions. Discuss tradeoffs openly: a smaller evaluation may answer a focused question well but provide less information about variation across settings or populations. Participation in choosing the focus can help users understand both the value and limits of the eventual findings.

Uses

Uses describe what people plan to do with what they learn. These might include revising activities, allocating resources, improving accessibility, deciding whether to expand a program, or strengthening community oversight. The following examples connect possible uses with the four broad purposes described above.

Some specific examples of evaluation uses

• To gain insight:

  • Assess community needs, priorities, and resources
  • Identify barriers to accessing and participating in the program
  • Learn how to describe and measure activities and outcomes meaningfully

• To improve how work is carried out:

  • Refine plans for introducing or adapting a practice
  • Determine how closely implementation followed the plan and why changes occurred
  • Improve the clarity and accessibility of educational materials
  • Strengthen cultural responsiveness and ongoing learning
  • Assess whether participants' rights, privacy, and well-being are protected
  • Set priorities for staff training and support
  • Make and document adjustments during implementation
  • Improve communication among participants, staff, and partners
  • Identify ways to improve participants' experiences and satisfaction
  • Compare costs with outcomes and benefits using an appropriate approach
  • Examine who benefits, who is underserved, and whether anyone experiences unintended harm
  • Inform community decisions about supporting and sustaining the program

• To assess program outcomes and contributions:

  • Assess participants' development and use of skills
  • Examine changes in behavior and conditions over time
  • Inform decisions about where to allocate resources
  • Document progress toward program objectives
  • Assess whether accountability commitments have been met
  • Combine findings from multiple evaluations to assess what may work in similar settings and where uncertainty remains

• To support learning among people involved:

  • Create opportunities to reflect on program experiences and information
  • Encourage dialogue and deepen understanding of community issues
  • Clarify shared goals and document differences among partners
  • Build evaluation skills among staff, participants, and community partners
  • Gather accounts of success, difficulty, and unexpected outcomes with appropriate permission
  • Support organizational learning, change, and improvement

Questions

An evaluation needs specific, answerable questions. Invite people involved in and affected by the program to identify what they need to know, then prioritize questions according to their importance, feasibility, and likely use. Ask whose experience each question captures and whether important differences or unintended consequences could be overlooked.

Methods

Evaluation methods draw on social research, community knowledge, and other fields. Common design families include experimental, quasi-experimental, and observational approaches, including case studies. Experimental designs use random assignment to estimate effects by comparing groups; randomization supports comparability but does not guarantee identical groups or eliminate every source of bias. Quasi-experimental designs use approaches such as comparison groups, interrupted time series, or differences in changes over time without random assignment. Observational and case study designs can describe experiences, implementation, context, and possible explanations using documents, interviews, observations, surveys, or other sources.

No single design is best for every question. Designs differ in their ability to support causal conclusions, describe experiences, explain implementation, or examine context. Select methods that fit the questions and ethical constraints, and state what they can and cannot establish. Combining qualitative and quantitative methods can strengthen understanding when the approaches are well designed and their findings are meaningfully integrated.

Methods may need to change as conditions or information needs change. For example, an evaluation focused on improving delivery may later need to inform expansion decisions. Document why changes were made, how they affect interpretation, and whether additional expertise or information is needed. Avoid changing methods simply to obtain preferred findings.

Agreements

Agreements describe the evaluation's purpose, questions, methods, intended users, deliverables, responsibilities, timeline, and budget. They should also address decision-making, data access and stewardship, privacy, reporting, conflicts of interest, and the process for revising the plan. Clarify who can review findings and how disagreements will be documented.

An agreement may be a contract, protocol, memorandum of understanding, or another written document appropriate to the work. Its purpose is to make expectations explicit and confirm shared understanding. The agreement should protect the integrity of the evaluation, including the reporting of unfavorable findings, while allowing justified changes through an agreed process.

Focusing the evaluation may involve discussions with supporters and critics, review of possible uses, and interviews with intended users about their decisions and timelines. Compare options for scope, cost, accessibility, and strength of evidence. A timely, focused evaluation can be useful, but explain the limitations of any compromises rather than treating speed or convenience as substitutes for credible methods.

Gather Credible Evidence

Credible evidence is information that is relevant, trustworthy, and adequate for answering the evaluation's questions. What is needed depends on the question and the claim being considered. Assessing a program's causal effect may require a strong comparison design, while understanding participants' experiences may call for carefully conducted interviews or observations. Community acceptance matters, but credibility also depends on transparent methods and attention to bias and uncertainty.

Community knowledge and technical expertise can complement each other. People with lived experience may identify questions, barriers, or interpretations that external evaluators would miss. Specialists can help address issues such as sampling, complex analysis, measurement, or sensitive data. Bring these perspectives together according to the evaluation's needs.

All evidence has limitations. Using multiple relevant sources and methods can help identify consistent patterns, explain differences, and reveal missing perspectives. Involving community members in gathering and interpreting information can improve relevance and understanding, provided they receive appropriate support and the process protects confidentiality and independent judgment.

The following features affect the credibility of evidence:

Indicators

Indicators translate concepts about a program, its implementation, and its expected outcomes into observable or measurable features. Define each indicator clearly, including what is counted or described, for whom, over what period, and using which source.

Examples of indicators include:

• The program's capacity to deliver services, such as staffing levels and appointment availability
• Participation rates, with a clearly defined eligible or intended population
• Participants' satisfaction, experiences, and reported barriers
• Program reach and intensity, including who participated, what they received, and for how long
• Changes in participants' knowledge, skills, opportunities, or behavior
• Changes in community conditions, relationships, or norms
• Changes in organizational or policy environments, such as new practices or improved service access
• Longer-term changes in community health or well-being, such as housing stability or reported access to needed care

Indicators should reflect the criteria that matter for assessing the program, including community-defined outcomes. Complex initiatives usually require several indicators. Where appropriate and feasible, examine differences across populations or locations while protecting privacy and avoiding misleading conclusions from small numbers.

One approach is a "balanced scorecard": a small, complementary set of indicators covering different aspects of the program. For example, it might describe service delivery, participants' experiences, observed outcomes, resource use, and changes in surrounding conditions. The aim is a balanced view rather than a large collection of measures with no clear purpose.

A logic model provides another way to choose indicators. Identify measures along the pathway from resources and activities to outputs and outcomes. Quantitative indicators can describe amounts or rates, while qualitative information can explain experiences, processes, and meanings that numbers alone may not capture.

Indicators need not focus only on long-term outcomes. They can also describe service quality, community capacity, trust, or relationships among organizations. Work with people familiar with the context to define observable signs of these concepts, and check whether those signs have the same meaning across participants and settings.

Indicators may need revision as understanding improves; document changes and their effects on comparisons over time. Remember that tracking program performance is one part of evaluation. A change in an indicator does not by itself explain why it happened. For example, rising unemployment among participants may reflect broader economic conditions, changes in who enrolled, program limitations, or a combination of factors.

Sources

Sources of evidence may include participants, residents, staff, administrative records, documents, observations, and other relevant information. Different sources provide different perspectives. Program records may describe services delivered, while participants and people unable to access the program can explain experiences those records do not capture. Examine discrepancies rather than assuming one source is always more reliable.

Explain how sources were selected, who or what was excluded, and how those choices may affect conclusions. Numerical and narrative information can complement each other when their purposes and methods are clear. For example, participation records may reveal a decline in attendance, while interviews help explain transportation difficulties or changes in scheduling.

Quality

Quality concerns whether information is sufficiently accurate, consistent, relevant, and complete for its intended use. It depends on clear definitions, appropriate tools, accessible data collection, training, sampling, coding, data management, and error checking. Pilot procedures when feasible and examine missing data and possible bias. Discuss tradeoffs, but do not lower evidence requirements simply because a preferred conclusion would be convenient.

Quantity

Quantity refers to how much information is needed. Plan the amount and range of evidence required for the questions, methods, and intended decisions. Quantitative studies may need sample-size planning; qualitative work requires attention to the depth and diversity of perspectives. More data do not automatically mean better evidence, especially if collection is biased or burdensome. Gather information with a clear purpose and establish a reasoned stopping point.

Logistics

By logistics, we mean the people, timing, settings, technology, and procedures needed to gather and manage information. Ask participants about language, communication, accessibility, and privacy preferences rather than assuming that everyone in a community shares the same needs. Explain participation clearly, establish appropriate consent procedures, and specify who can access data, how they will be protected, and how long they will be retained.

Develop and Justify Conclusions

Evidence does not interpret itself. Conclusions require analysis, attention to context, and a clear explanation of how findings relate to the evaluation questions and agreed criteria. Include different perspectives and consider alternative explanations. Agreement can support use, but consensus is not proof: a conclusion must remain grounded in evidence even when some people disagree with it.

The principal elements in developing and justifying conclusions are:

Performance standards

Performance standards describe the basis for judging the program, such as expected levels of access, service quality, or improvement. These differ from the standards used to assess the quality of the evaluation itself. Agree on performance criteria and thresholds early where possible, explain whose values they reflect, and document any revisions so that judgments are not adjusted simply to fit the results.

Analysis and synthesis

Analysis and synthesis help organize and interpret the evidence. Analysis examines patterns within information; synthesis brings findings together to develop a broader understanding. In an evaluation using multiple methods, analyze each source appropriately and then examine where findings agree, differ, or leave gaps. Explain how information was classified, compared, and combined, including decisions about missing data, uncertainty, and potentially conflicting evidence.

Interpretation

Interpretation explores what the findings mean in context. In a hypothetical community survey, 15% of respondents report witnessing violence during the previous year. That finding has a different meaning if a comparable earlier survey found 50% than if it found 7%. Before interpreting either pattern, check whether sampling, questions, reporting periods, and participation were comparable. Even a credible change over time does not establish that the program caused it. Community perspectives can help explain possible influences and the practical significance of the findings.

Judgments

Judgments assess the program's value or performance by comparing interpreted findings with explicit criteria. Different criteria can lead to different judgments. For example, managers may see a 10% increase in outreach as progress, while residents may point out that services still do not reach neighborhoods facing the greatest barriers. Both perspectives can inform the assessment. Explain the basis of each judgment and distinguish improvement from meeting an agreed standard.

Recommendations

Recommendations identify actions to consider in response to the findings. They require attention to evidence, feasibility, resources, alternatives, and the priorities of affected people. For example, evidence that a program expanded access to services for survivors of domestic violence may support continued investment, while also revealing needs for safer access, different hours, or stronger coordination. Decisions about changing or ending services should consider continuity and the consequences for people who rely on them.

Recommendations should make their reasoning explicit and remain proportionate to the evidence. Explain expected benefits, possible harms, practical requirements, and uncertainties. Where values or priorities differ, document those differences rather than presenting a contested choice as the only possible conclusion.

Three practices can improve the relevance and usefulness of recommendations:

• Share draft recommendations with people expected to use or be affected by them.
• Seek responses from people with different experiences, roles, and perspectives.
• Present feasible options and tradeoffs, identifying when the evidence supports a clear recommended action.

Strengthen conclusions by actively examining alternative explanations and evidence that challenges the initial interpretation. Do not dismiss an explanation merely because it is inconvenient. Where multiple interpretations remain plausible, describe them and the information that could help distinguish among them. Plan major analyses in advance when possible and identify additional exploratory analyses clearly.

Use Findings and Share Lessons Learned

Findings do not automatically lead to action. Plan for their use from the beginning by identifying decisions, responsible people, timelines, and the resources needed for follow-through. Continue communicating throughout the evaluation, and make space to discuss findings that are unexpected or difficult to act on.

The following elements can help people use evaluation findings:

Design

Design connects the evaluation's questions and methods with the decisions it is intended to inform. As discussed earlier, clarify who needs the findings, when they need them, and what actions are possible. Involving community members and intended users in these decisions can improve relevance while helping everyone understand the evaluation's boundaries.

Preparation

Preparation means building the understanding and capacity needed to act on findings. Discuss possible results before they are available, including what would count as sufficient evidence for a decision and how uncertainty will be handled. This can reveal unrealistic expectations and identify support that users will need.

For example, present intended users and community partners with hypothetical findings and ask what actions they would consider. If the information would not help them make a decision, revisit the questions or reporting plan. Explore favorable, unfavorable, and mixed findings without encouraging people to choose a preferred result in advance.

Feedback

Feedback is the continuing exchange of information among people involved in the evaluation. Provide accessible opportunities to comment on decisions, emerging findings, and draft interpretations. Explain how feedback influenced the work. Review can identify errors and missing context, but it should not give participants, staff, or funders authority to suppress credible findings.

Follow-up

Follow-up provides support as people interpret findings and decide what to do. Completing a report is a milestone; learning and action continue afterward. Assign responsibility for agreed actions, identify needed resources, set review dates, and track progress. Someone may coordinate this work while ensuring that the evidence and its limitations remain visible in decisions.

Supporting use also means guarding against misuse. Findings are shaped by the design, participants, setting, and period studied. A single case study may offer valuable insights, but it usually cannot establish that every site in a large program will have the same experience or outcome. Explain what can reasonably transfer and what requires further investigation.

Anyone involved may selectively emphasize favorable or unfavorable findings. Reduce this risk by reporting the full pattern of results, explaining uncertainty, and correcting misleading interpretations. New uses of existing findings may be reasonable, but they require a fresh assessment of whether the evidence supports the proposed decision.

Dissemination

Dissemination means sharing the evaluation's methods, findings, and lessons with relevant audiences in a timely and accessible way. Plan language, format, communication channels, and timing with intended users and affected communities. Use plain-language summaries and other appropriate formats alongside technical detail. Report findings fairly while protecting confidentiality and honoring legitimate restrictions on sensitive information.

Benefits can also arise from participating in the evaluation itself. These "process uses" include clearer thinking about assumptions, stronger skills, improved relationships, and a shared understanding of the program. Notice and assess these benefits rather than assuming that participation will always have a positive effect.

Evaluation can help staff and community partners clarify goals, examine how their work connects, and make decisions using evidence and explicit values. It can also reveal disagreements or organizational barriers that need attention. A learning-oriented approach treats those discoveries as information for improvement.

Additional process uses for evaluation include:

• Developing indicators together can clarify what different participants value and which outcomes need greater attention.
• Connecting findings to planning and resource decisions can make learning part of routine practice. For example, a funder might support adaptations identified through evaluation, while avoiding incentives that reward only favorable results or discourage serving people facing greater barriers.

Standards for Evaluation Quality

The Joint Committee on Standards for Educational Evaluation developed The Program Evaluation Standards to guide sound and fair evaluation. The discussion below adapts the 30 standards reflected in the original CDC framework, retaining their historical four-category organization. The later third edition of the Joint Committee's standards reorganizes the guidance into five categories, including evaluation accountability, and should be consulted when selecting standards for a new evaluation.

Standards help groups assess the quality of their evaluation and make reasoned choices when resources or conditions create tradeoffs. For example, technically accurate findings may have little value if they arrive after a decision or cannot be understood by intended users. A useful design also needs to be feasible and ethically acceptable.

Apply relevant standards during planning, implementation, reporting, and follow-up. The historical summaries below use updated explanatory language; they are not a verbatim reproduction of the current standards. Document how you apply the guidance and how you address tensions among practical requirements without compromising participants' rights or the integrity of findings.

The original 30 standards were grouped into four categories:

• Utility
• Feasibility
• Propriety
• Accuracy

The utility standards address the evaluation's relevance and usefulness:

• Identification of Interested and Affected Parties: Identify people involved in or affected by the program and its evaluation, including those whose perspectives are often overlooked, so that relevant priorities can shape the work.
• Evaluator Credibility: Those conducting the evaluation should bring appropriate skills, demonstrate trustworthiness, understand the context, and explain how they will manage potential conflicts of interest.
• Information Scope and Selection: Collect information that addresses the agreed questions and the needs of intended users and affected communities. Avoid gathering information that has no clear use.
• Values Identification: Explain the values, perspectives, criteria, and reasoning used to interpret findings and judge the program.
• Report Clarity: Describe the program, context, evaluation purpose, methods, findings, and limitations in language and formats that intended audiences can understand and use.
• Report Timeliness and Dissemination: Share relevant findings in time to inform decisions, clearly identifying preliminary results and protecting sensitive information.
• Evaluation Impact: Plan and conduct the evaluation in ways that support learning, responsible action, and follow-through, while monitoring possible unintended effects of the evaluation itself.

Feasibility Standards

Feasibility standards address whether the evaluation can be carried out effectively with the available resources and within its setting. They encourage practical choices without treating convenience as sufficient justification for weak or unfair procedures.

The feasibility standards address:

• Practical Procedures: Use procedures that gather needed information while minimizing unnecessary burden and disruption for participants, staff, and community partners.
• Political and Contextual Viability: Anticipate differing interests, power relationships, and conditions that may affect participation or use. Establish fair ways to address disagreement and protect the evaluation from inappropriate interference.
• Cost-Effectiveness: Match the evaluation's scope and resource use to the value of the information needed, including the time and contributions required from community members.

Propriety Standards

Propriety standards address ethical conduct, fairness, rights, and the well-being of people involved in or affected by the evaluation. The original framework included the following eight areas.

• Service Orientation: Design the evaluation to help the program respond effectively and fairly to the people it is intended to serve, including those experiencing barriers to access.
• Formal Agreements: Document responsibilities, procedures, timelines, reporting arrangements, and the process for changing agreements so that expectations are clear and accountable.
• Participants' Rights and Well-Being: Protect people's dignity, privacy, and ability to make informed choices about participation. Establish appropriate consent and review procedures for the evaluation's setting and activities.
• Respectful Interactions: Treat people with respect, provide accessible ways to participate, and address coercion, discrimination, distress, or other risks arising from the evaluation.
• Complete and Fair Assessment: Examine strengths, limitations, benefits, harms, and unintended consequences fairly, with attention to differences in experiences and outcomes.
• Disclosure of Findings: Make findings and limitations accessible to relevant audiences while protecting confidential information and meeting applicable disclosure obligations. Explain any restrictions on access.
• Conflicts of Interest: Identify, disclose, and manage actual or perceived conflicts that could influence the evaluation's design, conduct, interpretation, or reporting.
• Fiscal Responsibility: Use resources responsibly, maintain appropriate financial records, and account transparently for evaluation expenditures.

Accuracy Standards

Accuracy standards address whether the evaluation provides dependable information and defensible interpretations. They support confidence in the findings while requiring explicit acknowledgment of uncertainty and limitations.

The original framework included 12 accuracy standards:

• Program Documentation: Describe the program clearly and accurately, including what was planned, what was implemented, and any important changes.
• Context Analysis: Examine conditions that may shape implementation, outcomes, interpretation, and the transfer of findings to other settings.
• Described Purposes and Procedures: Document the evaluation's purposes and methods in enough detail for others to assess the work, including justified departures from the original plan.
• Defensible Information Sources: Explain how information sources were selected and why they are appropriate, including relevant exclusions and possible sources of bias.
• Valid Information: Use measures and collection procedures that support the intended interpretations, checking whether they are appropriate for the people, languages, and settings involved.
• Reliable Information: Use procedures that produce sufficiently consistent and dependable information for the evaluation's purposes, and report relevant measurement limitations.
• Systematic Information Management: Organize, review, protect, and verify information systematically. Correct errors through documented procedures and preserve a clear record of important changes.
• Analysis of Quantitative Information: Analyze numerical information using methods suited to the design and questions, accounting for missing data, uncertainty, and relevant assumptions.
• Analysis of Qualitative Information: Analyze interviews, observations, documents, and other descriptive information systematically, preserving context and considering interpretations that challenge emerging themes.
• Justified Conclusions: Show how conclusions follow from the evidence and criteria, and distinguish observed changes from claims about what caused them.
• Impartial Reporting: Report findings fairly and transparently, including unfavorable or mixed results. Explain how potential bias and competing interpretations were addressed.
• Metaevaluation: Assess the quality of the evaluation itself using appropriate standards, during the work and after completion, to identify strengths, limitations, and improvements.

Applying the Framework: Conducting Useful and Credible Evaluations

Evaluation can support community learning, program decisions, and accountability to participants, partners, and funders. The appropriate scope depends on what people need to know and the consequences of the decisions ahead. Useful starting questions include:

• What evaluation approach fits our questions, context, and resources?
• What are we learning, whose experiences are represented, and what remains uncertain?
• How will we use what we learn to improve effectiveness, accessibility, and fairness?

The framework helps connect these questions with practical steps and standards. Its value lies in making evaluation decisions deliberate, transparent, and responsive to the people affected by the program.

Applying the framework requires judgment and a suitable combination of community knowledge and evaluation skills. Relationships may be complex, resources limited, and conditions subject to change. Develop an approach that fits those realities while protecting ethical standards and the integrity of the evidence. Seek specialized support for questions or methods that exceed the team's experience.

Cost is a common concern, but an evaluation's budget should be considered in relation to its questions and intended decisions. A modest evaluation can provide useful information about implementation or participant experience. More demanding claims, such as estimating causal effects or detecting small differences across populations, may require additional resources and expertise.

Build evaluation into routine work where appropriate, using existing information only when it is suitable and can be used responsibly. Coordinate collection and reporting with decision timelines, and avoid repeatedly asking community members for information that is already available.

Technical complexity can also be a barrier. Begin with clear, useful questions and select methods the team can carry out well. Provide training and support, and be explicit about what the resulting evidence can establish. A focused, well-executed evaluation is more useful than an ambitious design that cannot be implemented credibly.

Staff and participants may reasonably worry that evaluation will be punitive, exclusionary, or used to justify decisions already made. Address those concerns directly. Explain the purpose, share decision-making where appropriate, protect candid feedback, and demonstrate how findings will support learning and accountability. Trust depends on how the evaluation is conducted and used, not simply on assurances that participation is welcome.

In Summary

Evaluation helps communities understand how programs operate, what changes occur, who benefits, and what needs improvement. It can inform decisions about resources, adaptation, expansion, or discontinuation while identifying unintended consequences. Its conclusions should reflect both the evidence and its limits.

The original CDC framework remains a useful teaching model when its historical basis is clear and newer guidance informs current practice. Use the steps and applicable standards to develop an evaluation that fits the program, involves affected people meaningfully, and connects findings with action. The Magenta Book - Guidance for Evaluation is an additional evaluation resource; this link points to an older edition, so consult the latest HM Treasury guidance when using it to plan an evaluation.

You understand the basic concepts and terms of program evaluation:

___You understand that evaluation is the systematic investigation of the merit, worth, or significance of an object or effort

___You understand that a program is used to describe the object or effort that is being evaluated

___You understand that program evaluation is a way to evaluate the specific projects and activities community groups may take part in

___You understand that stakeholders are those who care about the program or effort

You understand that program evaluation:

___Helps clarify program plans

___Improves communication among partners

___Gathers feedback needed to improve and be accountable for program effectiveness

You know how to evaluate a specific program:

___You know what will be evaluated

___You have decided what criteria will be used to judge program performance

___You have decided what standards of performance must be reached for the program to be considered successful

___You know what evidence will indicate performance

___You know what conclusions about program performance are justified based on the available evidence

___You have engaged stakeholders

___You have described the program

___You have focused the evaluation design

___You understand that possible focuses are: to gain insight, to improve how things get done, to determine what the effects of the program are, and to effect those who participate in it

___You have gathered credible evidence

___You have justified your conclusions

___You understand how to ensure the use of the information and share lessons learned

___You know that there are four categories of evaluation standards: utility, feasibility, propriety, and accuracy

___You understand the standards in each category

___You understand which standards are most applicable to each step of evaluation practice

When writing your evaluation report:

___Provide interim and final reports to intended users in time for intended uses

___Tailor the report content, format, and style for the audience(s) by involving audience members

___Include an executive summary

___Summarize the stakeholders and how they were engaged

___Describe essential features of the program (perhaps in appendices)

___Explain the focus of the evaluation and its limitations

___Include an adequate summary of the evaluation plan and procedures

___Provide all necessary technical information (perhaps in appendices)

___Specify the standards and criteria for evaluative judgments

___Explain the evaluative judgments and how they are supported by the evidence

___List both strengths and weaknesses of the evaluation

___Discuss recommendations for action along with their advantages and disadvantages

___Ensure protections for program clients and other stakeholders

___Anticipate how people or organizations may be affected by the findings

___Present minority opinions or rejoinders where necessary

___Verify that the report is accurate and unbiased

___Organize the report logically and include an appropriate level of detail

___Remove technical jargon

___Use examples, illustrations, graphics, and stories 

Tool 1: Tools Related to the Recommended Framework

See the following Community Tool Box sections for additional information.

 

Engage Stakeholders

To help you understand who stakeholders are:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

Identifying Targets and Agents of Change: Who Can Benefit and Who Can Help

Identifying Community Assets and Resources

To learn how to involve stakeholders:

Involving Key Influentials in the Initiative

Involving People Most Affected by the Problem

Conducting Focus Groups

To work together with a diverse group:

Training for Conflict Resolution

Encouraging Involvement of Potential Opponents as well as Allies

Describe the Program

To fully understand the need, problem, or goal that the program addresses:

Defining and Analyzing the Problem

Analyzing Community Problems

Conducting Needs Assessment Surveys

Conducting Concerns Surveys

Conducting Public Forums and Listening Sessions

To better explain the activities, components, and elements of the program:

Designing Community Interventions

Understanding Risk and Protective Factors: Their Use in Selecting Potential Targets and Promising Strategies for Interventions

Identifying Strategies and Tactics for Reducing Risks

To be able to describe resources and assets for the program:

Identifying Community Assets and Resources

For examples of logic models:

Our Model of Practice: Building Capacity for Community and System Change

Our Evaluation Model: Evaluating Comprehensive Community Initiatives

To look broadly at your program and its context:

An Overview of Strategic Planning or "VMOSA" (Vision, Mission, Objectives, Strategies, Action Plan)

Understanding and Describing the Community

To modify the program to fit the local context:

Adapting Community Interventions for Different Cultures and Communities

To learn to explain the program to others (and so that they see your point of view):

Reframing the Issue

Focus the Evaluation Design

To clarify the purpose:

Developing an Evaluation Plan

To identify potential users and uses:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

To clarify evaluation questions:

Developing an Evaluation Plan

For illustrative evaluation questions:

Our Model of Practice: Building Capacity for Community and System Change

For help identifying specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

For guidance about making agreements:

Identifying Action Steps in Bringing About Community and System Change

Gather Credible Evidence

For support in implementing specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

Justify Conclusions

To see an illustrative process for considering evidence:

Our Model of Practice: Building Capacity for Community and System Change

Ensure use and share lessons learned

To promote the use of what your organization has learned:

Providing Feedback to Improve the Initiative

Conducting a Social Marketing Campaign

Attracting Support for Specific Programs

To share what you have learned with diverse groups:

Making Community Presentations

Communications to Promote Interest

Arranging Celebrations

Tool 2: Evaluation Standards

Use this table to determine how well your evaluation met "good" standards for evaluation.

Standard	Did the evaluation meet this standard? (Yes or No)	Comments:
Utility Standards
1. Stakeholder Identification
2. Evaluator Credibility
3. Information Scope and Selection
4. Values Identification
5. Report Clarity
6. Report Timeliness and Dissemination
7. Evaluation Impact
Feasibility Standards
1. Practical Procedures
2. Political Viability
3. Cost Effectiveness
Propriety Standards
1. Service Orientation
2. Formal Agreements
3. Rights of Human Subjects
4. Human Interactions
5. Complete and Fair Assessment
6. Disclosure of Findings
7. Conflict of Interest
8. Fiscal Responsibility
Accuracy Standards
1. Program Documentation
2. Context Analysis
3. Described Purposes and Procedures
4. Defensible Information Sources
5. Valid Information
6. Reliable Information
7. Systematic Information
8. Analysis of Quantitative Information
9. Analysis of Qualitative Information
10. Justified Conclusions
11. Impartial Reporting
12. Metaevaluation

Tool 3: Steps in Evaluation Practice and the Most Relevant Standards

Codes following each standard designate the conceptual group and number of the standard.  U = Utility; F = Feasibility; P = Propriety; A = Accuracy.

Steps	Most Relevant Standards
Engage Stakeholders	Metaevaluation (A12) Stakeholder identification (U1) Evaluator Credibility (U2) Formal agreements (P2) Rights of human subjects (P3) Human Interactions (P4) Conflict of interest (P7)
Describe the program	Complete and fair assessment (P5) Program documentation (A1) Context analysis (A2)
Focus the evaluation design	Evaluation impact (U7) Practical procedures (F1) Political viability (F2) Cost effectiveness (F3) Service orientation (P1) Complete and fair assessment (P5) Fiscal responsibility (P8) Described purposes and procedures (A3)
Gather credible evidence	Information scope and selection (U3) Defensible information sources (A4) Valid information (A5) Reliable information (A6) Systematic information (A7)
Justify conclusions	Values identification (U4) Analysis of quantitative information (A8) Analysis of qualitative information (A9) Justified conclusions (A10)
Ensure use and share lessons learned	Evaluator credibility (U2) Report clarity (U5) Report timeliness and dissemination (U6) Evaluation impact (U7) Disclosure of findings (P6) Impartial reporting (A11)

 

In this hypothetical example, Della Roberts works as a nutritionist at Harperville Hospital. A Black resident of the city, she is concerned about unequal access to affordable, nutritious food in several predominantly Black neighborhoods. Families have described limited grocery options, transportation barriers, and rising food costs, but city officials have given the issue little attention. Della approaches neighborhood residents about researching these conditions together to inform community-led advocacy.

Della recognizes that residents bring knowledge, relationships, and priorities that hospital staff cannot supply on their own. She contacts local faith leaders, a physician, a community center director, and people in her own networks to invite interest in a research partnership. The group also reaches beyond these connections so that established leaders do not determine who participates. Residents of different ages, educational backgrounds, languages, and experiences join the team.

The team meets at accessible neighborhood locations to decide what it wants to learn and how members will share responsibility. Residents contribute local knowledge, while Della and other partners offer research training and practical support. Together, they develop interview questions, agree on consent and privacy procedures, and arrange options for recording information through writing, audio, or supported note-taking. They decide to ask residents about food access and shopping experiences and to document food availability, prices, store hours, and transportation options.

Their findings suggest that many residents want more affordable, nutritious options but face substantial barriers. Some must make long bus trips to reach a full-service grocery store. Others struggle to fit shopping around work and caregiving responsibilities. Participants also describe prices that exceed their food budgets and limited availability of foods that meet their cultural preferences or dietary needs.

Community researchers and hospital partners analyze the information together, examine gaps in participation, and jointly prepare a report. They present their findings and recommendations to residents and city officials. In this example, the report contributes to a city response that includes support for neighborhood food retailers and improved transportation access. The team continues tracking whether these changes make food more affordable and accessible.

The partnership illustrates community-based participatory research because residents help shape the questions, methods, interpretation, reporting, and action. Their role extends beyond collecting information for professionals. Community connections can support trust and understanding, but the team still needs clear methods, confidentiality protections, and opportunities for residents to disagree or decline participation.

This section explains community-based participatory research, its relationship to participatory action research, why it can be useful, who might participate, and how to organize and carry out a project.

What is community-based participatory research?

Community-based participatory research, or CBPR, is a partnership approach that meaningfully and equitably involves community members, organizations, and researchers throughout the research process. Partners contribute different forms of knowledge and share decisions about questions, methods, interpretation, and use of findings. The aim is to develop useful knowledge and connect it with action that addresses community priorities.

Community involvement in research can take different forms. In some studies, researchers collect information from residents through surveys, interviews, or other methods, while retaining control over the questions and analysis. This can produce useful research, but being a source of information does not by itself make someone a research partner or make a project CBPR.

In other projects, researchers recruit or hire community members to collect information because of their skills, relationships, or knowledge of the setting. These contributors may also help interpret findings. Such roles can be valuable, but the extent of participation depends on their actual influence over research decisions, working conditions, and use of the results.

A more collaborative approach involves community members and institutional researchers working as partners from the beginning. They jointly identify priorities, design the study, gather and analyze information, communicate findings, and decide what action to pursue. Responsibilities may differ, but partners establish meaningful ways to share authority, resources, recognition, and accountability.

Community members may also initiate and lead their own research. A group might investigate housing conditions, service access, or another shared concern, then invite technical partners when useful. Community-led work can be highly organized and rigorous, whether or not an academic institution participates.

This section focuses on partnerships and community-led approaches that involve affected people in research decisions and connect learning with action. CBPR overlaps with participatory action research, which emphasizes cycles of inquiry, reflection, and action with participants. The terms are related but are not interchangeable in every setting. Here, CBPR refers to the partnership approach, while action research describes the ongoing connection between inquiry and change.

CBPR can support program evaluation, research on community conditions, and efforts toward lasting change. Its value depends on the quality of the partnership and the research, including whether participation is accessible, adequately supported, and influential.

Community-based participatory research can strengthen understanding of an issue in several ways:

• Some residents may feel more comfortable speaking with researchers who understand their circumstances or have established relationships in the community. Others may prefer someone outside their social network, especially when discussing sensitive experiences.
• People directly affected by an issue or intervention may identify questions, experiences, and explanations that other researchers would overlook. Their knowledge can improve the relevance of interviews, surveys, observations, and other methods.
• Community researchers can help interpret local language, relationships, and context. An apparently minor comment may point to an important concern, although interpretations should be checked rather than assumed to represent everyone.
• Ongoing community connections can help researchers notice changes and refine their questions over time. Ordinary conversations and personal relationships should not automatically become research data; the team needs agreed boundaries and appropriate permission.
• Findings may be more relevant and credible to residents when community members have helped shape the research and can explain how conclusions were reached, including the limits of the evidence.

In an evaluation, these contributions can deepen understanding of how an intervention works, who benefits, and what needs improvement. In research on a community issue, they can clarify conditions, possible causes, and opportunities for action. CBPR does not guarantee complete or unbiased findings, but a well-designed partnership can combine community knowledge and research methods to support more informed decisions.

CBPR can also support learning and professional development. Community researchers bring existing skills and expertise, including knowledge gained through lived experience, employment, organizing, caregiving, and education. Participation can expand their research skills, networks, confidence, and opportunities to influence decisions. Partners should address barriers such as unpaid work, inaccessible meetings, limited transportation, or exclusion from institutional decision-making.

Two potential contributions of CBPR to community decision-making are:

• Community members can gain greater influence over the questions being studied and the policies, services, or investments that affect their lives.
• Institutions can become more accountable for recognizing community expertise and sharing authority. Residents do not need research credentials to deserve respect or a role in decisions. A well-supported partnership can create additional opportunities to use evidence, challenge assumptions, develop leadership, and pursue community priorities. Research and analytical skills may also support educational or employment goals, although these benefits should not be assumed or substituted for fair compensation.

Participatory action research has connections with traditions of critical education associated with Brazilian educator Paulo Freire. These traditions emphasize dialogue, reflection on social conditions, and collective action to address oppression. In practice, partners should examine how power operates within their own project as well as in the wider community. Different projects may emphasize learning, practical improvements, institutional accountability, or broader social change.

Participatory research can address issues such as access to public assistance, housing instability, education, employment, and environmental conditions. It can also support community priorities that focus on existing strengths and opportunities.

Examples include:

• Environmental conditions. Residents and technical partners might document recurring flooding, investigate possible sources, and use the findings to advocate for drainage improvements.
• Health and access to care. Community members and health researchers might examine barriers to appointments, map service availability, or study residents' experiences with a health program.
• Political and economic conditions. Residents might analyze public budgets, examine access to employment, or study how public investments are distributed across neighborhoods.

CBPR partnerships can begin in different ways. What matters is how partners establish the agenda, make decisions, share resources, and use the findings. A community group may seek technical assistance, or researchers may approach community organizations about a possible collaboration. An institution's initial involvement does not determine the partnership's quality.

Common starting points include:

• Researchers develop an initial study idea and hire community members to collect or analyze data. This becomes a CBPR partnership only if community partners also have meaningful influence over the research and its use.
• A researcher, service organization, or public agency identifies a concern and invites affected residents to assess its importance, refine the questions, and jointly design the work.
• A community-based organization brings residents together to define a priority, investigate it, or evaluate an intervention that affects them.
• Residents identify a shared concern, organize a research team, and decide whether outside expertise would help them investigate and address it.

Why would you use community-based participatory research?

CBPR can offer benefits for understanding community issues, evaluating programs, and deciding what to do next. The following reasons describe possibilities to work toward, rather than results that participation automatically guarantees.

CBPR can produce information that better reflects community experiences and priorities.

• Trusted relationships may support open discussion, especially when people have clear choices about who interviews them and how their information will be protected.
• Community researchers can contribute knowledge of local history, relationships, and institutions that helps place findings in context.
• People affected by an issue or intervention can identify outcomes that matter to them and help design questions that capture those experiences.

Sharing research decisions can make the work more responsive to community priorities.

Involving people directly affected by the issue can improve the questions asked, the interpretation of evidence, and the practicality of recommendations. Meaningful participation also gives residents opportunities to shape or challenge proposed responses. The goal is shared responsibility and informed decision-making, with room for disagreement, rather than securing endorsement of a plan already chosen.

Community participation can increase awareness of the research and opportunities to contribute.

Team members can explain the project through community networks, meetings, and other agreed communication channels. Public outreach should describe the research accurately while protecting participants' identities and avoiding pressure on friends, relatives, or neighbors to take part.

Transparent partnerships can strengthen the credibility of the research.

Residents may have greater confidence in a project when people they trust have meaningful roles and the team explains its methods, funding, findings, and limitations. Shared identity or familiarity alone does not establish trustworthiness. Researchers must demonstrate it through their conduct and accountability.

CBPR can build research capacity that communities use in other efforts.

Team members may apply skills in asking questions, gathering evidence, analyzing information, and presenting findings to other priorities. Community organizations can also develop systems, relationships, and resources that support future research. These longer-term benefits are more likely when learning is shared and the partnership plans for continuity.

Involvement in CBPR can expand opportunities for leadership and influence.

Community researchers may gain confidence, new connections, and greater access to decision-making. These opportunities can be especially valuable where institutions have excluded residents from research or policy discussions. The partnership should recognize existing leadership and address the barriers that limit participation, rather than treating people as lacking knowledge or initiative.

Research skills can be useful in other areas of team members' lives.

Interviewing, critical thinking, collaboration, data analysis, and public communication may support employment, education, organizing, and personal goals. Ask team members what they hope to gain and provide appropriate recognition, references, training records, or other support. Participation should not come with promises of benefits the project cannot deliver.

Participatory research can create opportunities to address racial, ethnic, economic, and institutional barriers.

Working together can help partners question assumptions and understand experiences different from their own. However, familiarity does not automatically remove discrimination or unequal power. Teams need fair decision-making, accessible participation, respectful discussion, and ways to address harm. They should also examine whose expertise receives recognition and who controls funding, data, and public communication.

A participant in the Changes Project, a CBPR effort examining changes to public assistance and their effects on adult literacy and English-language learners, described recognizing previously unexamined prejudices through conversations with other team members. The reflection illustrates how collaborative research can prompt personal learning, including among participants who initially believe they have little to reconsider.

Participatory research can deepen understanding of the conditions that shape people's lives.

Research and collective reflection can help a team examine how policies, economic conditions, institutional practices, and social relationships influence a community issue. That understanding can inform practical strategies for change while recognizing that many conditions cannot be changed by individuals acting alone.

Community-based participatory research can contribute to lasting social change.

Shared knowledge, stronger relationships, and greater community influence can support changes in programs, policies, and institutional practices. Lasting results require action, resources, and follow-through beyond the research itself. The team should assess whether changes benefit the people most affected and whether new problems or inequities emerge.

Who should be involved in community-based participatory research?

Participation should reflect the issue and its context, with meaningful leadership by the people most affected.

• People most affected by the issue or intervention. Their experiences and priorities should help shape the project from the beginning. Include people with different experiences of the issue, including those who have been excluded from services or public discussions. No individual should be expected to speak for an entire community.
• Other members of the affected community. Residents with relevant relationships, language skills, organizing experience, or other knowledge may contribute even when they do not personally experience the issue. Their involvement should complement the leadership of directly affected people.

For example, a Portuguese-speaking business owner might help a team studying access to services by contributing language skills, local relationships, and knowledge of area organizations. The team would still need direct participation from residents who use or cannot access those services, including people outside the business owner's networks.

• Decision-makers. Local officials, agency leaders, and others with authority may help the team understand policy processes and act on findings. Their involvement can take the form of partnership, consultation, or briefings. Consider whether their presence could discourage candid participation, and do not give them automatic access to identifiable information or control over conclusions.
• Academic and other research partners. Researchers may contribute subject knowledge, methodological skills, facilities, or access to resources. They also need to learn from community partners about local conditions, priorities, and appropriate ways of working. Technical expertise should support shared inquiry rather than establish a hierarchy of whose knowledge matters.

Treat all team members as colleagues and make equitable participation concrete. Agree on decision-making authority, budgets, compensation, data access, authorship, and conflict resolution. Differences in institutional responsibility or specialized skills may affect particular roles, but paying salaries or holding a grant does not justify excluding community partners from meaningful decisions. Be transparent about constraints and negotiate how to address them.

• Health, human service, and public agency staff and volunteers. These partners may contribute knowledge of service systems, implementation, and available resources. Residents can help them understand experiences that organizational records do not capture. Address possible conflicts between their research roles and their responsibilities to employers or service users.
• Other community members and organizations. Depending on the project, youth groups, disability organizations, faith communities, tenant associations, local businesses, and other groups may contribute useful perspectives. Offer participation through appropriate roles without allowing broader representation to displace the priorities of those most affected.

When should you use community-based participatory research?

Consider whether the partnership has the purpose, relationships, resources, and decision-making arrangements needed for meaningful collaboration.

Time is one consideration. Building relationships, agreeing on priorities, learning together, and sharing decisions can require substantial preparation. An established partnership may be able to respond quickly, while a new one may need more time. If a deadline makes meaningful collaboration impossible, be honest about the limits of the engagement rather than labeling a narrowly consultative project CBPR.

The required methods also matter, but CBPR is not limited to a particular type of research. Partners can use qualitative, quantitative, or combined methods, including technically complex approaches when the team has the necessary expertise and support. Community members can help govern and interpret a study without every partner needing to perform every technical task.

Qualitative research

Qualitative research examines experiences, meanings, processes, and context through methods such as interviews, group discussions, observations, and document analysis. Findings are developed through systematic interpretation and may be presented as themes, explanations, narratives, or descriptions supported by relevant evidence.

Some qualitative information can be counted, but counts alone may miss its meaning. For example, several interviewees might mention transportation, yet describe very different barriers involving cost, accessibility, safety, or scheduling. Researchers should explore those differences and avoid inferring feelings or intentions solely from body language or other ambiguous observations.

Quantitative research

Quantitative research examines numerical information, such as participation rates, travel times, service costs, survey responses, or changes in defined outcomes. Analysis can describe patterns, compare groups, and estimate relationships or effects when the research design supports those claims. Results require interpretation in light of measurement quality, sampling, missing information, uncertainty, and other possible explanations.

Numerical findings are not automatically more accurate or objective than other evidence. For example, a high program completion rate may conceal unequal access or differences in participants' experiences. Understanding how information was collected and what it represents is essential to interpreting the numbers.

Early in a policy change, interviews may reveal emerging experiences before administrative data show broader patterns. In a hypothetical study of changes to public assistance, adult learners might describe effects on class attendance, childcare, or work schedules. Those accounts can identify important questions and possible consequences, while later numerical information may help assess how widespread particular patterns are.

Choose methods according to the questions, context, and intended use of the findings. Combining qualitative and quantitative approaches can deepen understanding when the team has a clear plan for analyzing and integrating the information.

CBPR may be especially useful when:

• Partners have time and resources to establish shared expectations, build relationships, and provide the support needed for participation.
• Community knowledge is essential to defining the questions and interpreting findings, and the team can obtain the methodological expertise required.
• The work depends on an accountable relationship with the community, especially where previous research has damaged trust or provided little local benefit.
• Residents and organizations want to share decisions about research and the actions that may follow.
• Building research capacity, leadership opportunities, and institutional accountability is part of the purpose.
• Partners are prepared to connect research with sustained efforts toward community-defined change.

How do you organize and carry out community-based participatory research?

A CBPR project involves building a partnership, agreeing on priorities and responsibilities, preparing the team, choosing questions and methods, gathering and analyzing information, sharing findings, and deciding what action to take. Partners should also assess the quality of the research and the partnership itself. These activities often overlap and may be revisited as the team learns.

The steps below can be adapted to projects initiated by residents, community organizations, public agencies, or academic partners. No single organizational structure is ideal for every situation. Community-led teams may already have substantial research expertise, while institutional teams may need support in sharing power and working collaboratively.

Recruit a community research team

Recruitment should reflect the project's purpose and the range of people affected. An existing group may provide a starting point, but consider whose experiences are missing from its networks. Some projects need participation across several communities; others focus on a particular group, such as people using a service, residents who share a language, or people living with a specific condition.

Give directly affected people substantial representation and meaningful decision-making authority. Numbers alone do not establish equitable participation: a community majority can still be excluded from budget, data, or publication decisions. Agree on governance, compensation, meeting access, and support so that community partners can influence the work in practice.

Existing relationships with an organization or program can help with recruitment, but service users and employees must be able to decline without pressure or negative consequences. Recruiting more broadly also requires clear communication and deliberate efforts to reach people outside established networks.

• Use accessible language and formats. Explain the purpose, responsibilities, compensation, and choices in plain language. Provide translation, interpretation, and alternative formats according to participants' needs.
• Use communication channels people already rely on. These may include neighborhood organizations, faith communities, newsletters, messaging groups, local media, or personal invitations. Use several approaches when one channel would exclude part of the community.
• Develop messages with people familiar with the community. Ask what information matters and how it should be communicated. Avoid assuming that everyone sharing an identity has the same preferences or experiences.
• Make contact in accessible and appropriate settings. Share information where people live, work, gather, or connect online, with permission where needed. Offer ways to learn more without requiring travel or an immediate commitment.

A personal invitation from a trusted person can be helpful, but there is no single best recruitment method. Offer private ways to ask questions or decline, and review whether your approach is reaching the people you intend to include.

Orient and train the research team

Orientation and training may overlap, but they serve different purposes. Orientation helps partners understand the project, their roles, and how they will work together. Training develops the knowledge and skills needed for specific research activities.

Orientation might include:

• Introductions that allow members to share their interests, experiences, and preferred ways of participating
• An explanation of CBPR, the project's starting questions, and the decisions partners will make together
• Clear information about time commitments, compensation, on-site childcare or other caregiving support, transportation, accessibility, interpretation, and available equipment
• Opportunities to ask questions, revise expectations, raise concerns, and agree on decision-making and conflict-resolution procedures

Use orientation to establish a team of colleagues with complementary expertise. Formal education, professional titles, language fluency, or familiarity with meetings should not determine whose contributions count. Agree on practices that make discussion accessible, such as explaining technical terms, sharing materials in advance, and offering several ways to contribute. Institutional partners should also examine assumptions and practices that may limit others' participation.

Training should build on what members already know and address the tasks they will undertake. Different training approaches may include workshops, practice sessions, peer mentoring, demonstrations, and support during fieldwork.  Community members and technical partners can both teach and learn. Schedule training so that people can practice skills before using them and receive feedback throughout the project.

The following practices can make training more useful and accessible:

• Choose a comfortable, accessible space or online format that supports participation.
• Provide appropriate food, drinks, and other practical support when possible, asking about dietary and access needs.
• Plan regular breaks and flexible pacing. Ask participants what they need and allow people to step away without having to disclose personal information.
• Arrange the space or technology to support interaction. Consider audibility, visibility, mobility access, captions, interpretation, and ways to participate without speaking to the full group.
• Present and practice material in several ways. Combine clear explanations, accessible written or visual materials, discussion, demonstrations, and hands-on practice. Check understanding through supportive activities rather than assuming that one presentation works for everyone.
• Use training to strengthen the partnership. Make room for questions, shared problem-solving, feedback, and discussion of the principles guiding the work.

Training content will depend on the project, but may include:

• Research methods and ethical practice. Cover the relevant research and information-gathering methods, such as interviewing, surveying, observation, or reviewing records. Include consent, confidentiality, data security, appropriate boundaries, and how to respond if someone becomes distressed or wishes to stop.
• Community and program context. Share local history, existing evidence, institutional relationships, and differing perspectives on the issue, recognizing that no single person represents the whole community.
• Collaboration, facilitation, and negotiation. Practice listening, explaining reasoning, sharing decisions, raising disagreement, and preventing a few voices from dominating. Meeting practices should be negotiated by the team rather than treated as rules everyone must already know.
• Analysis and reporting. Help members interpret evidence, consider alternative explanations, and distinguish findings from recommendations. Reports may use written text, charts, presentations, video, audio, or oral testimony, with appropriate accessibility and privacy protections.
• Presenting findings. Practice communicating the research clearly, answering questions, explaining uncertainty, and connecting recommendations to evidence. Prepare for disagreement without implying that presentation skills alone determine whether decision-makers will act.

Determine the questions the research or evaluation is meant to answer

Develop research questions together, beginning with what community members want to understand and what decisions the findings could inform. Prioritize questions that are meaningful, answerable, and feasible, and record important questions that the project cannot yet address.

An evaluation may examine implementation: What was done, who participated, and how did delivery compare with the plan? It may examine outcomes: What changed, for whom, and were there unintended effects? It may also explore how implementation relates to outcomes and what needs adjustment. Claims that an intervention caused a change require a design and evidence capable of addressing alternative explanations.

Research on a community issue may ask whether a condition exists, how widespread it is, how people experience it, or which groups face particular barriers. The team may also compare possible responses. A commitment to improving conditions should guide the purpose of the work without predetermining what the findings must show.

Plan and structure your research activity

Develop a plan that fits the questions, timeline, resources, and team's capabilities. Agree on governance, compensation, data stewardship, and how findings will be used. Before collecting information, determine which institutional, community, or other ethical review procedures apply and obtain any required approvals.

Your plan should include:

• The methods, sources, and amount of information needed, including how participants will be selected, what consent procedures will be used, and how unnecessary data collection will be avoided
• Roles, responsibilities, decision-making authority, compensation, and support for each part of the work
• A realistic timeline and budget, including time for relationship-building, training, review, and adjustments
• How information will be stored, protected, analyzed, and interpreted, who will have access, and how long it will be retained
• The formats for reporting findings, the process for reviewing drafts, and agreements on authorship, acknowledgment, and use of quotations or images
• When, how, and with whom findings will be shared, including community access to results and responsibility for following up on recommendations

Anticipate challenges and agree on ways to respond

Research partnerships may encounter internal conflicts, changes in availability, funding delays, outside pressure, or problems with data collection. Establish clear guidelines for raising concerns, adjusting responsibilities, resolving disputes, and responding to privacy or safety incidents. If someone cannot complete a task, ask what barriers or support needs are involved before assuming a lack of commitment.

Implement your research plan

Pilot the methods, revise them as needed, and begin collecting information using the agreed procedures. Hold regular check-ins to address questions, review data quality, and identify missing perspectives. Analyze and interpret findings collaboratively, documenting changes to the plan and examining evidence that challenges the team's initial assumptions.

Prepare and present your report and recommendations

Develop the report together in formats suited to the intended audiences. Explain the questions, methods, findings, limitations, and reasoning behind recommendations. Different audiences may need different levels of detail, but the findings should remain consistent. Share results with the community, recognize contributions fairly, protect identifying information, and document significant disagreements rather than hiding them.

Possible ways to share findings include:

• A media briefing or press conference with agreed spokespersons and clear privacy boundaries
• An accessible community meeting, presentation, or discussion in relevant languages
• A local news article, newsletter, community website, or other communication channel
• A written or alternative-format report for community members, funders, agencies, and other interested audiences

Take or advocate for appropriate action

Action may include revising a program, improving service access, proposing a policy change, meeting with decision-makers, organizing a public demonstration, or starting a community initiative. Choose actions together based on the findings, community priorities, available resources, and likely consequences. Assign responsibilities and timelines, and identify what further research is needed where evidence remains incomplete.

Follow up

A project continues beyond the presentation of findings. Track responses to recommendations, assess the effects of any action, and provide opportunities for the team to reflect on what it learned. Plan how relationships, skills, and resources can remain available to the community after a particular grant or study ends.

• Evaluate the research and partnership process. Invite all partners to assess the quality of the methods, training, communication, compensation, and decision-making. Ask whose contributions influenced decisions, which barriers remained, and whether people felt able to disagree. Provide confidential ways to raise concerns where needed.
• Assess benefits, harms, and unintended effects. Examine what changed for the community, who benefited, and whether the project created burdens, conflict, exposure of sensitive information, or other problems. Distinguish observed results from hoped-for future benefits.
• Invite team members to reflect on their learning and goals. Areas may include research skills, public communication, collaboration, confidence, professional opportunities, and influence over decisions. Let participants define what matters to them without assuming that everyone should experience the same changes.
• Support continued involvement when members want it. Options may include another community-prioritized project, a paid research partnership, mentoring, consulting, or connections with other efforts. Plan for resources and leadership continuity while respecting people's choice to leave or take a break.

CBPR is one approach to research and evaluation, and its suitability depends on the purpose and conditions of the work. Use the term honestly: a project needs meaningful partnership, shared influence, and accountability to the community, not simply community recruitment or data collection.

When adequately supported, CBPR can produce useful knowledge, strengthen relationships and research capacity, and connect evidence with community-defined action. Its benefits depend on the quality of the methods, the distribution of power and resources, and whether partners follow through on their commitments.

In Summary

Community-based participatory research involves people most affected by an issue as partners in defining, conducting, interpreting, and using research. It combines different forms of expertise, supports shared learning, and connects findings with practical action. Meaningful participation includes influence over decisions and access to the knowledge the project produces.

CBPR requires time, resources, and sustained attention to relationships. Partners should build on existing strengths, provide fair compensation and accessible support, and address barriers to participation. Clear methods, ethical practices, shared accountability, and follow-through can help the research contribute to lasting improvements in community conditions.

What is community-based participatory research?

___You define community-based participatory research as research conducted by and for those most directly affected by the issue, condition, situation, or intervention being studied or evaluated

Why would you use community-based participatory research?

___It yields better and more nearly complete and accurate information from the community

___Involving the community in research makes a positive outcome that meets community needs more likely

­­___Action research trains citizen researchers who can turn their skills to other problems as well

___Involvement in CBPR changes people’s perceptions of themselves and of what they can do

___Skills learned in the course of action research carry over into other areas of researchers’ lives

___A participatory action research process can help to break down racial, ethnic, and class barriers

___Action research helps people better understand the forces that influence their lives

___Community-based participatory research can move communities toward positive social change

Who should be involved in community-based participatory research?

___People most affected by the issue or intervention under study

___Other members of the affected population

___Decision-makers

___Academics with an interest in the issue or intervention in question

___Health, human service, and public agency staff and volunteers

___Community members at large

When should you employ community-based participatory research?

___There’s time to properly train and acclimate community researchers

___The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)

___You need an entry to the community or group from whom the information is being gathered

___You’re concerned with buy-in and support from the community

___You want to have an effect on and empower the community researchers

___You want to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

___You recruit a community research team

___You orient and train the research team

___The team determines the questions the research or evaluation will try to answer

___The team plans and structures the research activity

___As a team, you carry out your plan

___The team prepares and presents a report and recommendations on its work

___You take, or try to bring about, appropriate action

___You evaluate the process

___You provide an opportunity for team members to reflect on and discuss their learning and the effects of the experience of being involved in a CBPR process

___You maintain gains by keeping team members involved

Example 1: The Changes Project: Holyoke Community College and SABES West

The Changes Project was a three-year participatory action research effort. Funded by a grant from the Office of Educational Research and Improvement, part of the U.S. Department of Education, its goal was to determine the effects on adult learners of the major changes in welfare regulations, immigration laws, and the workplace that took place in the 1990’s.

Background

The project was participatory in two senses:

• The 42 researchers (34 of whom were able to participate for the whole three-year span of the project) were themselves current and former learners and staffers in adult literacy, GED (high school equivalency), and ESOL (English as a Second or Other Language) programs
• All the effort’s data was obtained from 600 learners in the programs from which the researchers came

All the researchers were learners, graduates, or staff members from one of five Western Massachusetts adult education programs.  The Center for New Americans (CNA) in Greenfield, Amherst, and Northampton and the International Language Institute (ILI) of Northampton are independent organizations that work with ESOL learners.  The Mentor Program at Holyoke Community College provides services to adult learners who are making the transition between adult education and higher education.  Read/Write/Now (RWN) Adult and Family Learning Center, sponsored by the Springfield City Library, offers adult and family literacy and GED classes.  The University of Mass. Labor/Management Workplace Education Program (LMWEP) specifically serves University employees with a range of instructional offerings that encompass basic skills, technology, communication, problem-solving, and workplace-specific areas.

Research team members ranged in age from 22 to 55, and in education from adult literacy learners to holders of advanced degrees.  Among them, they spoke six languages; came from six countries on three continents; and claimed Black, Asian, European, and Hispanic heritage.  Twenty-six were parents (ten being single parents), and ten were current or former welfare recipients.  In general, the researchers comprised a group broadly representative of the community of Western Massachusetts adult learning center participants and staff.

Each of the five sites fielded a research team that worked both independently and in conjunction with the other four.  In addition to its other members, ranging in number from 3 to 13, each team included a site research facilitator (SRF) – a program staff member who was both a member and facilitator of the site’s research team.  SRF’s regularly met together with the program coordinator, and also received consultation from a research advisory group and a methodologist.

The research question the project examined was generated by the administrators and staff of 14 western Massachusetts adult learning programs, in response both to the voiced concerns of learners and to their own observations.

Goals

As stated in the Changes Project report, research goals were to learn more about:

• Effects of welfare reform, immigration reform, and the changing requirements of the workplace on adult learners at five adult literacy programs in Western Massachusetts
• Ways in which adult learners respond to and accommodate immigration reform, welfare reform, and the changing requirements of the workplace as they pursue their goals through education
• How practitioners and programs can alter educational services to be more effective, given welfare reform, immigration reform, and the changing requirements of the workplace
• Ways in which policies can be more responsive to the needs of adult learners as they deal with these three issues

The Changes Project began with a challenge: how to mold a large and diverse group of researchers into a team that could work together creatively and effectively.  Changes Project Coordinator Alex Risley Schroeder recalls that this challenge was made even greater by the fact that the project had not been 100% participatory from the beginning.

“We had to write the proposal, and, because of time and other constraints, we did that without the direct participation of learners.  It was something we struggled with throughout the life of the project.  We had teams of learner/researchers, and said to them, ‘Okay, we’ve promised the funders certain things.  Now what can we, as a group, do to make those things reflect what’s important to all of us?’”

There were only two occasions at the beginning of the project when the whole group, all five site teams, got together: an initial orientation/training and a project kick-off.  The real work of team-building actually began when the first set of data came in.

Because of the differing nature of the five adult education programs and their learners, not all teams researched all three areas of concern.  Most of the work on immigration reform, for instance, came out of the research of teams from ESOL programs. The one workplace education program team didn’t research the effects of welfare reform, because all learners in that program were, by definition, employed; it concentrated instead on changes in the workplace.

Data Collection and Information Synthesis

For all teams, information was collected in three phases from learners in their own programs:

• Individual interviews. Teams developed appropriate interview questions for the area they were researching, and conducted a first round of individual interviews.
• More interviews, focus groups, and surveys. Teams conducted a second round of interviews (including some follow-ups – almost all interviews were followed up with second interviews) with revised questions, and also conducted focus groups in each program and administered surveys project-wide.
• More interviews, more focus groups, and three separate surveys, each specifically aimed at learners with firsthand knowledge of one of the three areas of concern.

After each phase of data-gathering, teams struggled with ways to sort out and report what they had found, and with how to make sense of it. There was an enormous amount of material, most of it in the form of people’s stories about their experiences. Each team found its own ways to organize data.

Each phase ended with an Analysis Fest, a coming together of all the teams to try to integrate and analyze what they had found. The meetings were called Analysis Fests, according to Risley Schroeder, to capture both the excitement and the real joy to be found in gaining and making sense of knowledge.  It was in the Analysis Fests, she says, that the five teams really became a coherent unit.

“Not everyone knew the things that were obvious to some of the teams.  What women on welfare went through, for instance, or what it was like to not be able to speak English.  It was a very powerful learning time for all of us, and it gave everyone a chance to get to know others as teams and as individuals.  Really, to make this work, we all had to understand who we were as a large group, as teams, and as individuals.”

The group wrestled with organizing information and stories they had collected. The challenge was compounded by the fact that some researchers had difficulty reading and writing, or with English. It was made yet more difficult because the researchers, at the beginning, thought of themselves as having no expertise, and therefore no tools for the task.

The first Analysis Fest started out print-heavy. There were transcripts of all the interviews, many painstakingly translated into English from Spanish or other languages. The teams were constrained by what they thought “research” was.  In the subsequent Analysis Fests, as researchers relaxed and started to put the material from different teams together, they realized there were themes common to many of the interviews, regardless of the circumstances of the learners who had been interviewed. Those themes became both the core of the project’s ultimate findings and the key to organizing the data.

Those who had the most difficulty with print or with language were also the most creative in getting around that difficulty. They made up representative stories for each of the issues, used pictures and sound as well as – or instead of – print to show what they had found, and developed dramatic interactive role plays to demonstrate the kinds of difficulties learners faced in dealing with welfare and immigration reforms and workplace issues.

One of the most important findings of the project was that adult learners often depended on fragile networks of support of various kinds in order to be able to attend classes and learn successfully.

There were essentially four types of support necessary:

• Practical supports. These include the kinds of supports most people would understand to be necessary for successful completion of an adult education program: food, clothing, shelter, health care, child care, and transportation. But this category also includes information – about welfare recipients’, immigrants’, and workers’ rights, about the laws that govern those areas, about how to find and enroll in programs, about how to study, about how to find other academic and employment resources.
• Inner resources. Hopes, dreams, motivation, focus, determination, natural ability, prior knowledge, spiritual faith.
• Personal supports. These are both emotional and practical, and come not only from family and friends, but from professionals and the community at large. They include advocacy for individuals and programs, as well as respect, encouragement, and belief in people’s ability to change their lives.
• Institutional supports. These include not only the existence, but the responsiveness and appropriateness of programs, organizations, and institutions that provide help and services to those who need them; state and national policies that speak to the real needs of the people at whom they’re aimed; and regulations that accommodate the real needs and circumstances of those they’re meant to help.

The researchers conceived of these supports as threads in a web, all intertwined and interdependent. Cutting any one thread weakens the whole web, and makes an adult learner’s success less likely. The metaphor of the web made the concept of support easier for everyone to see and understand, and also made it clear how important it was that policy makers see that an adult learner’s success is dependent on far more than what goes on in a particular ABE or ESOL program.

Results

Ultimately, the Changes Project accomplished a number of purposes:  

It gathered a large amount of important, first-hand data about the effects – intended and unintended – of welfare and immigration reforms and changes in the workplace on adult learners and on their chances for successful pursuit of their educational goals.

It had a profound effect on the researchers themselves. Both adult learners and staff members found that the experience of participating in the project changed the way they saw themselves and the world.  More specifically:

• Team members increased their ability to communicate and developed job-related skills, and learners specifically increased their literacy and English language skills.
• Team members felt more self-confident and saw themselves as more competent and more able to contribute as a result of their work in the Changes Project.
• Many team members felt they had developed a voice, enabling them to speak up and be heard in both their private and public lives in ways that hadn’t been possible for them previously.
• Their Changes Project experience led many researchers to become active in their communities, and to change their perspective on political action in general and on their own ability to change things in particular.
• Some of the teachers among the team members found that their view of their jobs and of the learners they worked with had changed.  One explained “This project has given me that ability [to be a reflective teacher] because you really have to listen when you’re doing research.  It’s helped me to let go of...being in charge of how the research goes.  Learning to do that has changed me – it’s changed the way I think about how knowledge gets made.  It’s really different when you do it in a group. What you come up with is much richer.”
• The experience of building and belonging to a community that encompassed people of different cultures, ethnicities, and backgrounds stood out as enormously positive for all participants.  Many talked about the excitement of learning about how other cultures viewed the world, and the rewards of learning to get along with people with different perspectives.
• Team members gained valuable technical knowledge about conducting research, including  how to collect and analyze data (as well as how to engage in logical analysis in general), how to adjust research to respond to the culture of those being addressed, and how to develop the interpersonal skills needed to gain information from interviews and focus groups.

The project served as a model, not only for research, but for adult education. Many of the researchers, both teachers and learners, found that their participation had changed their views of learning.  hey realized that it could be fun, practical, and group-centered.  The skills were being learned, perhaps even more effectively than when instruction was focused on them, in the context of work that had a real-world goal.

The research identified peripheral issues that have great consequences for adult learners. The web of support described above is crucial to the success of a majority of learners; yet policy makers are ignorant of it, and often unwittingly unravel it, thereby making it more difficult for learners to succeed.

The Changes Project drafted a series of recommendations for the field, for learners, for workplaces, and for policymakers.

• Recommendations for the field center on how to make programs more effective by accommodating learner needs, including non-academic ones; by strengthening the web of support; by increasing learner participation in the organization’s decision-making; by providing help in transition to next steps – further education, career planning, employment; and by maintaining an organizational atmosphere of mutual respect and support
• The project recommends to learners that they take responsibility for getting all the information they need, for seeking support, and for becoming involved and active in their programs.
• Recommendations to workplaces include their providing support for workers’ occupational and professional development (including career ladders within the workplace, as well as education and training); maintaining an atmosphere of mutual respect and inclusion; recognizing and building on workers’ strengths, particularly those of immigrants; and being sensitive to the cultural diversity of the workforce.
• The Changes Project’s recommendations to policymakers and legislators, although they concern larger issues, are similar in some ways to those for others. They suggest mandating that welfare and immigration caseworkers develop respectful relationships with those they serve; that more effective policies be developed by soliciting the participation of those at whom those policies are aimed; that information and services be provided in whatever languages are necessary; that welfare policies be responsive to the actual needs of recipients; that education and training count toward the welfare work requirement; and that policies encourage fathers to take responsibility for their children.

Participation in the project led to some action on the part of the researchers. This was both perhaps the most promising and the most frustrating part of the effort. One of the greatest pieces of learning for researchers was that action research really does blur the distinctions among research, analysis, and action.  Each of the research teams initiated some action, not so much as a result of, but rather as an integral part of, its research and findings.

• The Changes Project’s research lit a fire under a writing group of parenting teens who had been interviewed by the team, and who wanted to write about their situation.  Their writings were incorporated into a group-composed “Mother’s Day Letter” (entitled “What’s Going to Happen to Us”) that was sent to legislators, the media, and other adult education programs.  As a result. the local paper published a story on the effects of welfare reform on teen parents, and included interviews with several of the writers.
• The CNA team made presentations of their recommendations to the staff and administrators of the program, and also participated in rallies and direct advocacy with legislators for immigrants’ rights at the State House.
• The Read/Write/Now team initiated a letter-writing campaign to their local, state, and national representatives, detailing the real effects of welfare reform on adult-learner welfare recipients.  They also participated in advocacy for economic justice, and, ultimately, self-published a widely-distributed (to legislators, other programs, and the public) book telling the personal stories of many of those learners.  They hope to dispel stereotypes of welfare recipients, and to help people understand the true bind that many adult learners find themselves in.
• The ILI team’s action was centered closer to home.  Struck by the stories of those they interviewed, team members undertook to provide information and assistance to fellow learners in the program.  As did the others, they also engaged in advocacy for immigrants’ rights by attending State House rallies and other events, and by making presentations about their work.

Finally, the UMass Workplace Education team responded to what they learned in two ways:

• Team members created a support and problem-solving group, called Making Work Work, in order to explore and affect workplace issues.This group has met with a group from the University’s Training and Development Unit to design new programs and involve more staff in planning.
• The second important action to grow out of LMWEP’s research is the development of a career ladder for University workers, a result of meetings between the union and University administration initiated as a result of the Changes Project.

While all of this action has helped those involved feel that they now have more of a voice in their own destinies, it is frustrating as well. Except for the LMWEP progress in the workplace, most of the actions taken had little or no effect on the situations they addressed.  Welfare and immigration reform have gone through as originally planned, with virtually no concessions to the needs of people who are being deprived of the education and training that could help them to become financially successful and better citizens.  Legislators and others have listened sympathetically, but the results have shown little sympathy to the poor and/or newly arrived.

All this can be taken as an excuse to stop trying to bring about social change, but it can also be seen as the beginning of something. A number of people who previously thought of themselves as powerless have seen that they can at least struggle for what they believe in or what they need. They have learned, and have passed on, ways of finding and examining information that give them more choices. And they seem to be willing to continue to practice what they’ve learned. It is in circumstances like those created by the Changes Project that the seeds of change take root. The name seems more than fitting.

Example 2: Community implements participatory action research for proposed business deal

When an out-of-state hazardous waste recycling company approached an economically-depressed New England mill town with an offer to build a plant in the community, the town leaders were overjoyed. The plant would mean a boost to the tax base, and the company promised that both the construction contracts and most jobs at the plant – jobs they described as well-paid – would go to local residents.  Without much discussion, the town tentatively agreed to grant permits, and to waive environmental regulations where necessary.

Some citizens were worried, however. The plant was scheduled to be built on land that drained into the aquifer that fed one of the town’s two wells. Although the company claimed that its process was foolproof and waste-free – all the waste they collected was to be recycled into other products, and there would be no chance of any pollution from their operation – it still seemed ill-considered to grant permission for something so dangerous-sounding without learning more about it.

The group of concerned citizens came from diverse backgrounds. One of the leaders was a teacher; another was a single mother on welfare. The group’s membership spanned the social structure of the town. Arrayed against it were town officials, concerned with growth in a period of economic stagnation, and many of the unemployed and underemployed citizens who hoped that the plant would mean jobs for them, and a better life for their families.

The town split down the middle, with officials refusing to budge, and neighbors and long-time friends ceasing to speak. The members of the concerned group realized they had to understand as much as they could about the company and its process, and set out to do research on both. They learned about hazardous waste recycling, and about both the waste the plant would accept, and the products it intended to turn that waste into.  They sought information on the company, discovering that it had operating plants in the Midwest.

Group members called officials in the Midwestern communities where the plants were located, and learned that there had, in fact, been some environmental problems. They also found out that the plant’s proposed methods were still largely untested, and that there were many scientists who believed that there were serious flaws in them. Armed with this information, the group notified the press and called a public meeting.

After a chaotic, but ultimately productive, evening, the group convinced enough people of potential problems that elected officials were pressured into appointing a commission to study the issue of the hazardous waste recycling plant.  Several members of the concerned citizens’ group were appointed to the commission, along with several other community members who favored the plant.

Over the course of several weeks, the commission met with company officials, and continued to do research on the science of hazardous waste recycling, receiving help from chemists and nuclear physicists at the nearby state university. Many of the commission members who had initially favored the plant became suspicious of the company’s motives, and questioned its truthfulness. Finally, the commission decided to send two of its members to one of the company’s other plants.

The commission delegation found a community that felt it had been lied to. There had been some serious leakage problems associated with the plant, and the company had not responded well.  Rather than admitting the problem and trying to fix it, company officials had tried to pretend there was nothing wrong, and accused the town of being alarmist.  When they were finally forced by a regulatory agency to clean up their mess, they had difficulty doing so, and many in town felt they still hadn’t succeeded.

Furthermore, the company had promised the community good jobs for local people, but the jobs hadn’t materialized. The well-paying jobs at the plant required graduate degrees in chemistry and other sciences, and the low-paying jobs local workers were offered were often part-time, and at wages no better than those available at Wal-Mart.

The New England town commission had heard enough. They returned and convinced the town to vote against the permits the plant needed.  Before that could happen, however, the company withdrew its offer, and looked for a site elsewhere in the country. Some people in town were still angry about what they saw as a loss of jobs, but most felt they had dodged a bullet.

A citizens’ group had turned itself into a participatory action research project, and had used the results of its research to take effective action on an issue of public importance.  Many of those involved in the effort are still active in the community. Others felt their moment of “fame” came at too great a cost, and have withdrawn from public participation.  In a small town, that is one of the risks of becoming involved in action research.

Example 3: True Collaboration for a Win-win in Community-engaged Research

“When we think about the word ‘partner,’ we can think about a significant other. If I were to say, ‘what do you want out of your partnership with your significant other?,’ you may say things like, ‘you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year.’ Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things. We want someone who is with us, who engages us from the very beginning of a process--so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this? I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But, as a partnership, we have to work through those solutions. Those are some of the things that we look for in a community-based research partnership, that flexibility and understanding. One of the things I noticed as I went from being a researcher to being in practice is that you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?” What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information. So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.”

Example 4: Compassion Fatigue in Evaluators

Evaluators working with trauma-exposed populations often develop compassion fatigue in the same way caregivers might. Compassion fatigue is the psychological strain one experiences due to sustained empathetic engagement. Dr. Stacy Huff's research analyzes and provides suggestions for managing evaluator compassion fatigue.

Compassion fatigue affects everyone differently. Many factors play a role in an evaluator's likelihood of developing compassion fatigue and their ability to overcome it. Things like the type of evaluation, personal identity, and organizational support all have an effect.

• Type of evaluation: Evaluators working with refugees, Indigenous populations, and those facing systemic oppression are more likely to experience compassion fatigue. They might also face ethical tension due to their proximity to trauma-exposed communities.
• Personal identity: Historically marginalized individuals, such as people of color and women, may face high mental strain from evaluation work. Evaluators who work with groups with whom they have a personal connection are more likely to experience compassion fatigue when hearing stories similar to their lived experiences.
• Organizational support: Many evaluators do not have a support system in place and are thus left to manage this large emotional weight by themselves.

Recommended resources for evaluators to recover from fatigue.

• Understanding your struggle: The Compassion Fatigue Awareness Project provides resources for understanding and preventing compassion fatigue within the workplace.
• Meditation: Huff specifically recommends The Guided Meditation for Compassion Fatigue to manage compassion fatigue.
• Overcoming burnout: The Reducing Compassion Fatigue, Secondary Traumatic Stress, and Burnout workbook is a great place to start!

To read Dr. Huff's blog post: https://aea365.org/blog/drg-tig-week-compassion-fatigue-in-evaluation-t…

Tool: Involving Community Members in Evaluation

When community members are involved in evaluation, all aspects of the project, from identification of the question to dissemination of results, are more likely to reflect the priorities, values, and experiences of the community. In any evaluation or research project, community involvement can be considered along a continuum, where community members have increasing levels of influence and responsibility. This planning framework, which can be used by organizations responsible for conducting or funding evaluation projects, can increase transparency, establish shared expectations, serve as a starting point to further clarify roles, and increase accountability. Read more from the Amherst H Wilder Foundation website.

Download the PDF for the tool here.

 

Tool: Data Walks and Narrative Methods

A Data Walk is a participatory process that engages stakeholders, including community members, in interpreting visual data such as maps, charts, photos, and statistics. Stakeholders should be engaged in the design of Data Walks to select which data to display and identify appropriate spaces to host the walk. This collaborative approach increases ownership, strengthens interpretation, and supports sustainability by positioning community members as co-creators of insight rather than passive recipients of information. 

Evaluators can consider using the Theory, Context, Characteristics, and Methodology (TCCM) framework to organize and reflect on what emerges from the walk. Using a structured reflection process ensures that community narratives are not only heard but thoughtfully integrated into context-sensitive evaluation and decision-making. Read more from the American Evaluation Association website. 

• Who are community leaders, evaluators, and funders?

• Why should you understand the interests of these groups?

• When should you understand the interests of these groups?

• What are the interests of these groups?

• How do you determine interests?

Your group wants to understand how its work is going. Community members want to know whether the effort addresses their priorities, and funders may want evidence of progress and responsible resource use. Evaluation can help answer these questions. Planning together makes it easier to identify what people need to learn, choose appropriate methods, and connect findings with decisions.

An important planning step is identifying the people involved in or affected by the evaluation and understanding their interests. This section focuses on community leaders, evaluators, and funders. Program participants, residents, staff, and others affected by the initiative also need meaningful opportunities to shape the evaluation, whether or not they hold a formal leadership role.

Designing the evaluation together can reveal shared priorities and differences that need discussion. One evaluation may serve several purposes, but it may not answer every question equally well. Agree on the most important questions, intended uses, and practical limits before collecting information, and make those choices transparent.

By needs and interests, we mean what people want to understand, the decisions they need to make, and the outcomes they consider important. These may include improving services, identifying unequal access, learning from implementation, or deciding how to allocate resources. People within the same group may have different priorities, so ask about their interests rather than assuming them.

Begin by considering why people want an evaluation.

Common reasons include:

• Being accountable to community members, participants, and those who provide resources
• Helping organizations receiving grants learn from and improve their work
• Improving funding decisions and support for community priorities
• Assessing program quality, accessibility, outcomes, and unintended effects
• Planning, adapting, and implementing programs
• Sharing useful approaches and the conditions needed to apply them elsewhere
• Building knowledge and strengthening the community's ability to use evidence

Any person or group may have several of these interests. Community leaders may want credible evidence about outcomes, evaluators may want findings that inform practical improvements, and funders may want to understand implementation challenges. Their interests can overlap. They may also differ in the decisions they face, the time they can contribute, and the formats in which they need information.

You do not need to be an evaluation specialist to help choose useful questions or identify community priorities. You do need a clear understanding of how findings will be used and what evidence is needed. Bring in technical support when the design, analysis, or interpretation requires skills the team does not yet have.

Who are community leaders, evaluators, and funders?

Community leaders

Community leaders may include residents, youth leaders, organizers, committee members, staff, administrators, public agency representatives, and members of governing boards. Leadership can be formal or informal. These individuals bring different experiences and levels of familiarity with evaluation, and they may want information to guide planning, improve services, or advocate for change. Their involvement should complement direct participation by affected residents rather than replace it.

Evaluators

Evaluators plan and carry out activities to answer questions about an initiative's implementation, outcomes, or value. They may be community researchers, program staff, independent consultants, or people working in universities and other organizations. Their roles should match their skills and the evaluation's requirements. Professional learning resources include the American Evaluation Association and its Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. Evaluators contribute methods for gathering and interpreting credible information while learning from community knowledge and experience.

Funders

Funders are individuals or organizations that provide financial support. They may include donors, foundations, government agencies, and other funding partners. Their evaluation expectations vary: some require particular measures or reports, while others help partners develop an approach together. Funders may want to understand progress, resource use, community outcomes, and what kinds of support would make the initiative more effective.

Why should you understand the interests of these groups?

Different people need different information to make decisions. A resident may want to know why a service is difficult to access, a program coordinator may need to understand staffing demands, and a funder may need evidence to inform continued support. Understanding these questions helps the team choose an appropriate scope, methods, and reporting plan.

People outside the initiative may have limited information about its activities and results. Those participating in or supporting the work may also see different parts of it. Evaluation can bring these perspectives together to examine what is working, what needs to change, and what remains uncertain. Include people who experience the program's effects as well as those who can provide funding or institutional support.

Information is useful when it helps answer a relevant question. For example, counting the number of people served can help describe program reach and staffing needs. It does not, by itself, show whether participants' circumstances improved or whether the program contributed to that improvement. Clarifying the intended use of information helps the team distinguish activities, outputs, and outcomes and avoid collecting data without a clear purpose.

Connect people's interests with evaluation questions and appropriate methods.

• Whose interests and experiences should the evaluation address?
• What decisions do they face, and what information would help?
• What evaluation questions will guide the inquiry?
• What methods and tools can provide credible answers within the available resources?

Meaningful involvement can strengthen collaboration and the ability to use findings. It can also reveal competing priorities before they disrupt the work. Understanding those priorities helps the team allocate resources, explain tradeoffs, and design an evaluation that supports learning without allowing the most influential group to determine every question.

When should you understand the interests of these groups?

Identify interested and affected people at the beginning of evaluation planning. Invite them to help define the purpose, questions, and intended uses before major design decisions are settled. Include people who may be affected by changes based on the findings, even if they are not currently involved in the initiative.

Revisit interests at agreed points and when circumstances change, such as a new funding requirement, a change in community priorities, or an unexpected finding. Adapt the plan when appropriate, documenting why changes were made and how they affect the evidence or comparisons. A new interest should prompt discussion rather than an automatic change that undermines the evaluation's purpose.

What are the interests of these groups?

How do you identify the people to involve?

Begin with a shared list of people and organizations involved in, affected by, or expected to use the evaluation. Ask community partners to help identify perspectives missing from the team's existing networks. Record initial ideas about people's interests as questions to explore, not as conclusions about what they must want.

Members of the initiative may reasonably want the evaluation to support their own learning and improvement. That is an important purpose. At the same time, participants, residents, partners, and funders may have questions about access, outcomes, resource use, or unintended effects. Consider these interests together so that the evaluation reflects both the initiative's needs and its responsibilities to the community.

Also consider differences in power, access, and ability to participate. People who rely on a service may hesitate to criticize it in front of staff, while residents who cannot attend meetings may be left out entirely.

Develop a broad initial list, then agree on appropriate roles. Some people may help make evaluation decisions, others may contribute information or review findings, and others may mainly need access to results. Explain how each form of participation can influence the work.

To identify people and groups to involve, ask:

• Who funds, supports, or makes decisions about the initiative, and how might they use the findings?
• Who will conduct the evaluation, and whose knowledge and skills are needed to do it well?
• Who participates in, collaborates with, or is affected by the initiative, including people who face barriers to access or have been left out of earlier discussions?

Once you have identified people to involve, ask about their priorities directly. The following descriptions offer starting points for discussion. They are not fixed expectations for everyone in a particular role.

Community leaders

Community leaders and residents may need information that is:

• Clear and accessible: Explain the purpose, methods, findings, and limitations in plain language. Offer relevant languages and formats, and make supporting detail available to people who want it.
• Focused and timely: Organize information around the decisions people face and provide it when they can use it. Respect their time while preserving the detail needed to interpret findings responsibly.
• Responsive: Address community-defined priorities and explain what the evidence can contribute to decisions about the initiative's future.
• Grounded in context: Describe accomplishments, challenges, resources, and surrounding conditions. Examine differences in experiences and outcomes rather than assuming the initiative works the same way for everyone.
• Useful for action: Identify practical options for improvement, explain the evidence behind them, and clarify what further information or resources may be needed.

Evaluators

Evaluators need a clear purpose, suitable evidence, and working arrangements that support credible inquiry.

• Shared understanding: Evaluators need input from participants, residents, staff, community leaders, and funders about the questions that matter, how the initiative operates, and how findings will be used.
• Appropriate and dependable information: Evaluators need data suited to the questions, with clear definitions and information about gaps, quality, and limitations. They also need realistic timelines and resources for collection and analysis.
• Clear responsibilities and respectful collaboration: Agree on access to information, privacy protections, participation procedures, and how findings will be reviewed and reported. Cooperation should support honest inquiry, including the reporting of unfavorable results.

Funders

Funders may need:

• Clear and timely reports: Provide information about implementation, progress, challenges, resource use, and support needs in time to inform funding decisions. Agree on reporting expectations early and coordinate requests to reduce unnecessary burden.
• Credible evidence of outcomes and contribution: Explain what changed, for whom, and what the evidence suggests about the initiative's contribution. Distinguish observed outcomes from demonstrated causal effects, and report limitations alongside accomplishments.

How do you determine interests?

Use conversations, interviews, surveys, or group discussions to learn what people want to understand and how they expect to use the evaluation. Ask about priorities, concerns, decision deadlines, and preferred ways to participate. Discuss differences early, and document which interests the evaluation will address, which fall outside its scope, and why.

Questions that can help identify evaluation interests include:

• What do you most need to learn about the initiative's strengths, limitations, or value?
• Which outcomes matter to you, and what would meaningful progress look like?
• What implementation challenges or barriers should the evaluation examine?
• What should we learn about participants' experiences, access to services, and interactions with staff?
• What unexpected benefits, harms, or differences in outcomes should we explore?
• What decisions will you use the findings to make, when do you need them, and how would you like to participate in the evaluation?

Review the answers for shared priorities and important differences. For example, a resident leader and an evaluator may both want to understand why participation drops after the first month. That shared question can guide further discussion about evidence and methods. Similar interests create opportunities to collaborate, but they do not mean everyone will agree on the interpretation or recommended action.

Ways to identify and discuss interests include:

• Interviews: Speak with people across relevant roles and experiences rather than relying on one person to represent an entire group. Use clear, open questions and ask for examples. Provide private opportunities to raise concerns that people may not feel comfortable discussing publicly.
• Surveys: Use a brief, accessible questionnaire to identify priorities and invite additional comments. Offer appropriate languages and participation formats. When interpreting results, consider who responded, who did not, and whether important perspectives are missing.
• Phone or remote conversations: Offer scheduled calls or other accessible options for people who cannot attend in person. Use consistent core questions while allowing relevant follow-up. Ask about communication and privacy preferences, and explain how responses will be recorded and used.
• Facilitated group discussions: Bring people together to explore possible questions, compare priorities, and discuss tradeoffs. Use facilitation that gives everyone opportunities to contribute, including written or small-group input. Record disagreements and unresolved questions as well as areas of agreement.

Check in during the evaluation and after findings are shared to learn whether the process and results met people's needs. A survey is one option; conversations, a reflection meeting, or feedback on a report may be more appropriate. Use what you learn to improve communication, follow-through, and future evaluations.

Decisions about improvement may involve small adjustments or substantial changes, depending on the findings and their consequences. Agree on who will act, what resources are needed, and when progress will be reviewed. Keep a record of how evidence and community priorities informed the decision.

With a documented set of priorities and intended uses, you can select an evaluation team and develop methods for gathering useful information. Confirm the scope, responsibilities, timeline, and reporting plan with the people involved. Return to these agreements as the work progresses so that the evaluation remains relevant and feasible.

In Summary

Understanding the interests of community leaders, evaluators, funders, and affected residents helps an evaluation ask useful questions and support informed decisions. Identify these interests early, discuss differences openly, and revisit them as conditions change. Shared planning works best when community participation has real influence and findings are reported honestly, including results that challenge expectations.

___You understand why it is important to determine the interests of the stakeholders

___You know who the stakeholders are

___You understand the community leaders' interests

___ You understand the funders' interests

___ You understand the evaluators' interests

___You know when you should understand those interests

___You know how to identify stakeholders

___You know the needs of each stakeholder

___You know how to determine interests

___You understand the interests involved in an evaluation

Example 1: Reasons to evaluate

Different organizations will have different reasons to implement an evaluation, and different interests will be at stake. For instance, some organization may want to focus on increasing the quality of its grants programs. Some other organization may be interested in becoming more accountable. Being aware of these interests in fundamental to understand how interests must be balanced. Here we offer a table with hypothetical examples of the kinds of interests that organizations may have.

Foundation / Organization	Interest
The Meyer Memorial Trust request-for-proposal program	Improve grantee programs, improve grantmaking
L.J. and Mary C. Skaggs Foundation program for arts organizations	Be accountable, improve grantmaking
Winthrop Rockfeller Foundation at-risk youth program	Improve grantee program, disseminate successful programs
Cities in Schools program	Increase the state of knowledge
Technology for Literacy project	Be accountable, improve grantmaking, assess quality or impact of program
William Penn Foundation career / vocational program	Be accountable, improve grantee programs, improve grantmaking
Gautreaux program	Increase the state of knowledge
W.K. Kellogg Foundation grants	Be accountable, improve grantee programs, improve grantmaking, assess quality or impact of grants / programs, plan new programs, disseminate successful programs

Example 2: Leading a community

Ultimately, the community is most affected by the results of an evaluation. That's why community leaders play a key role in the interest game of evaluations. Lisa Adkins knows enough about her community. She is the executive director of Youth Friends in Kansas City. Through Youth Friends, students are linked to volunteers with a shared interest, whether it's music or sports, computers or a special hobby, to create an environment in which every young person who needs a caring adult in his or her life has one.

Lisa talked to us about how her interests as a community leader interfere in the evaluation process. However, according to Lisa, conflict of interests has not been a problem in her evaluation process. She said that the work between evaluators, funders and herself as a community leader is usually of a cooperative nature, and everybody works together toward a common goal. She considers the evaluation process a positive experience for her organization, once they get a chance to review their programs and change what needs to be changed. So, evaluations are a constant element in her work

Her main interest, Lisa says, is to receive feedback on the programs Youth Friends provide, and evaluations are a great way of getting a response from the community. Youth Friends can adjust their services as the necessity arises from the community. Lisa points out that she is always the most interested in the evaluation's results since they could change her approach to a certain program and drastically improve it.

Example 3: Evaluating the evaluator

When it comes to figuring out interests in the evaluation game, the evaluator is a key character. Vince Francisco knows this game well. He's worked as an evaluator with diverse, community-based organizations doing hundreds of program evaluations. He has done evaluations of community policing programs, chronic and cardiovascular disease prevention programming, youth development, drug-abuse prevention, alcohol-use prevention, tobacco control initiatives, teen pregnancy prevention, and general youth development initiatives. Here Vince talks about how the evaluator's interests influence the score of the game:

"I try to find out immediately who the key players are and what they want to have answered. As an evaluator, what I bring to the table is a critical set of questions that I ask folks. Sometimes it's just sitting there watching folks in action, and other times it's asking key people very pointed questions about what they want to do, why they're doing it. I work with folks to take it to the next level and make it a little bit more formal in terms of coming up with some evaluation from very different points of view. If we don't understand their program, we can't go anywhere with their evaluation. I get a handle on their program (and I) will help them make their lives easier, help them run their program or initiative better, help them get more money, influence key players in the community to be on their side, the big issues. "

To find the other interests involved in an evaluation, Vince says that he talks to people until he has their roles figured out: "I look for the loudest voices, and then I look for the quietest voices those usually are the key players. The loudest voices usually speak for themselves, and not for the community. And the people who are the absolutely quietest, are the people who really have a voice in the community, but they're being drowned out by the folks who are loud. Very loud people usually get their way and they become key players so I want to know who they are and what their agenda is. And then I find whoever are the quietest people, and they're usually the people who sit in the background. I start talking to people and eventually there are some people that will talk back. I guess I just start talking to folks and (I work with) the ones that end up talking back to me. I combine a lot of talking."

Of course, Vince says, his own interests also influence the outcome of the intervention. To balance out this equation he thinks of the whole group or the whole community as his client. Talking to people, getting information, and shaping them, Vince produces something that works for his clients: "I outline and define the program : What's the initiative? What does it look like? Who are the key players? What are the program components and elements? Who are the targets of change? Who are the agents of change? Once I get a good handle on that, I connect that to some key questions. I collect evaluation data. What evaluation data would people believe as being evidence that these things are actually being implemented, and what would be key evidence that these things are successful? I combine that with appropriate components and elements in a systematically and very logical fashion. I look for ways for economizing procedures for collecting data. I get right to the chase. Don't do a whole lot of surveys that people have to spend time filling out."

Sometimes, however, interests conflict. Vince describes dealing with this kind of situation: "Usually, I'll find a few people that are really on board and very interested in evaluation. A lot of people just hate evaluators and think it's only a bunch of stupid surveys. My goal is to make it simple as possible. So my trick is to go in and sort of navigate the waters. Work with the people who want to work with me at first, and then convince everybody else that this is useful to them." Vince points out that, depending on how political the footfall of the program is, the evaluation may suffer.

Vince doesn't have a recipe to deal with all the interests that are involved in an evaluation: "All I know is that I juggle all the balls and keep them all in the air. And I look at all the different stakeholders. Who is the funding agent, and what is their interest in all this. I have to make sure their interest is represented. I try to give everybody the same exact information, so we're all moving forward together. That's the way I approach these things I try to change. I get everybody talking to each other in a real way and I don't get into the rumor way." Vince says that evaluators have a unique position because they can hold people accountable for their words: "I don't know if (the evaluator) is any more or less important than any other (interests). But without it, you don't do well. If we want to get lasting outcomes, we have to focus on behavior change. We have to focus on changing the systems that are controlling behavior."

Example 4: The funders role

Jeff Usher represents a very important interest in the evaluation process: the funder's. He is a program officer at the Kansas Health Foundation, a philanthropic organization that provides grants to improve the quality of health in Kansas. Here he talks about implementing evaluations and adjusting interests:

"We want to (see) change, we want to make a behavior change or community change. At the foundation, probably 10 to 20 percent of the overall budget will go to evaluation. The evaluation, of course, will be oftentimes both qualitative and quantitative in nature. We want to see the outcomes, whatever it might be; if it's teenage pregnancy, we'd like to see reductions of teenage pregnancies. So we like to see significant outcomes. We normally contract with people that have expertise in that particular area to conduct the evaluation."

Once the outcomes are in, Jeff says that it's time to decide what steps to take: "If it doesn't appear that the initiative is working, we clearly start losing interest in continuing the project in that manner. Oftentimes the evaluation portion of it is after the fact; the initiative is already over with, for the most part. If it comes back and it looks like the initiative was successful and the outcomes were going in the direction that we anticipated, then we may do another round of that initiative. For instance, the School Community Sexual Risk Reduction project, which is in its second phase now. Depending on the initiatives, if it appears that there is something else going on, we may still continue to work with those, or we may want to tweak that initiative a little bit to make it be a little more effective."

Jeff says that balancing interests among funders, community leaders and evaluators is not much of a problem for him: "We don't seem to have much of a problem when working with evaluators and as far as a private foundation, we don't hear a lot of problems." Jeff goes on to say that sometimes he experiences problems with some evaluators regarding the evaluation methods: "We may get some evaluations back that are measuring process rather than outcomes, and we clearly want to see outcomes. As far as balancing the grantee and the evaluator, normally, the evaluator is a grantee too, so in a sense there is even accountability that we want to see from the evaluator, and we would ask for that. Clearly there are certain things we want to see in that evaluation. For instance, we just did a memorandum agreement for an evaluation on a project that we are just going to be kicking off this next year. We want to see a change in the culture of campus, centered around binge drinking or heavy drinking. We outline those particular things in the memorandum when we start negotiating, so we expect to see that."

One of the funder's interests is to keep an eye on how the initiative conducts its business. Jeff says that in every memorandum agreement, they allow his foundation to perform site visits to have a third party come in, audit financial books, and see how the program is going: "Rarely do we follow up on those. The only time we would look at those is when we get a status report, and we ask for status reports from all of the grantees with what we want to see, and they haven't met those objectives. So, for almost all of our major initiatives, we'll have an evaluation component and have someone looking at it. We want to see behavior change, systems change, changes in culture centered around the health and well-being of communities, and individuals in communities."

The opening sections of this chapter introduce evaluation: why it matters, how it can be conducted, and whose interests it should address. This section and the next focus on selecting evaluators and developing an evaluation plan. These decisions are closely connected. Begin by clarifying the purpose with staff, community members, and people affected by the initiative, then select evaluators who can help refine the questions and design an appropriate approach.

Why pay attention to the selection of evaluators?

An evaluation should do more than declare that an initiative is doing well or needs improvement. It should examine the work from several perspectives, assess progress toward goals, identify strengths and limitations, and help people decide what to do next. A thoughtful plan needs evaluators with the skills, contextual understanding, and judgment to carry it out and interpret the findings responsibly.

Evaluation can support improvement, learning, and accountability. Useful evidence can help an organization build on strengths, address barriers, respond to changing community priorities, and inform advocacy or funding decisions. More information is not automatically better: the aim is credible information that answers meaningful questions and can guide action.

You may hire external evaluators, draw on staff and community expertise, or combine these approaches. You may also establish a planning team that helps select the evaluators. In each case, the people involved will shape the questions, methods, relationships, and interpretation of findings. Choose them for their ability to support a sound evaluation, rather than their likelihood of producing favorable results.

Organizations have different resources for evaluation. Options include staff-led work, community research teams, independent consultants, university partnerships, and combinations of these. Community members may be experienced evaluators, and professional evaluators may also be local residents. Consider what expertise already exists, what support is needed, and how contributors will be compensated.

A formal planning team is one option, not a requirement for every evaluation. A smaller group may be sufficient for a focused project. Match the scope and process to the available time and resources while maintaining credible methods and respect for participants. A manageable evaluation that answers a useful question can provide a foundation for more extensive work later.

All evaluators bring experiences, interests, assumptions, and relationships that can influence their work. Discuss these openly, whether you are considering consultants, university researchers, staff, students, or community members. Look for people who understand the agreed purpose, select methods that fit the questions, and can report findings honestly, including results the organization or funder may not welcome. Alignment on purpose should support independent judgment rather than require agreement with a preferred conclusion.

When should you choose evaluators?

Involve evaluation expertise early enough to inform planning and collect any baseline information you will need. Waiting until a project ends can limit the questions you can answer. For a new initiative, an initial planning group may help identify priorities and select evaluators. For an existing initiative, begin with its current decisions and available information, then plan opportunities to learn throughout the work.

For example, a new adult literacy program might include evaluation resources in its initial budget and arrange support from a supervised graduate researcher. The researcher could work with learners and staff to define useful questions, observe classes with appropriate permission, review progress information, and conduct interviews during the year. This approach could help the program understand both implementation and change over time. The budget should reflect the work required rather than a fixed percentage assumed to suit every program.

Before finalizing the selection, learn enough about the community, the initiative, and the intended use of findings to identify the expertise required. You do not need to resolve every question first. Evaluators can help assess context and refine the plan, provided community members and staff have meaningful roles in those decisions.

Context of the evaluation

The evaluation's context includes community history, geography, culture, economic conditions, politics, institutional relationships, and available resources. It also includes the initiative's stage of development and people's previous experiences with services or research. Understanding these conditions helps the team choose appropriate methods and avoid judging results without considering how they arose.

An example: Suppose a rural adult literacy program enrolls fewer learners than expected during its first year. Enrollment figures alone might suggest that the program is not meeting community needs. Conversations with residents, however, could reveal that previous programs closed quickly, making people uncertain about whether this one would last. Transportation, scheduling, and outreach may also be barriers. An evaluation that examines these conditions can help the team decide whether to strengthen relationships, adapt delivery, or revise expectations. Later evaluation can test whether those changes improve access and learning.

How should you choose among staff, community, and external evaluators?

Evaluation involves systematically gathering and interpreting information to support judgments and decisions. Relevant expertise can come from formal training, practical evaluation experience, community research, and knowledge of the initiative. The question is which combination of skills and perspectives your evaluation requires.

An external specialist may be useful for a complex design or independent review. Staff and community evaluators may contribute essential knowledge of implementation, relationships, and local priorities. A combined team can bring these strengths together, with clear responsibilities and support for collaboration.

Consider the following factors before deciding who will conduct the evaluation:

Money

Start with a realistic budget for the evaluation's purpose and scope. Include staff time, community participation, training, data collection, analysis, accessibility, and reporting. Ask funders whether evaluation support is available. If resources are limited, consider a smaller evaluation or targeted technical assistance rather than assuming that unpaid labor can cover all the work.

When specialized support is needed, possible sources include local colleges or universities, evaluation consultants, partner organizations, public agencies, and dedicated grants. Students may contribute through supervised coursework, internships, or research partnerships. Agree on the scope, supervision, timeline, data access, and community benefit before starting. Academic requirements should be compatible with the evaluation's purpose, and any additional use of information should be addressed explicitly.

For example, a graduate evaluation course could partner with community organizations on projects that fit a semester. Students and organizations might jointly define questions, develop methods, and produce practical reports, with faculty oversight. A shared learning event could highlight lessons and tools without disclosing sensitive information. Such arrangements work best when expectations are realistic and organizations have a plan for work that continues after the course ends.

Cost is one consideration. Also assess whether the proposed team has the capacity, availability, and working approach needed for the evaluation.

The complexity of the evaluation

Some evaluations address focused descriptive questions: How many residents participated? Did learners improve their reading skills? How did participants experience a recovery-support program? Other questions require more demanding designs: Did the initiative cause the observed change? Which aspects of delivery contributed? How did transportation, housing, or employment conditions influence participation and outcomes? Even a simple-looking question can require careful measurement and attention to missing information. Match the team's methods expertise to the claims you intend to make, and combine that expertise with community knowledge that can help explain the findings.

The type of information needed and how it will be analyzed

Quantitative information may require more than counts or averages. Depending on the questions, the team may need expertise in sampling, comparison designs, measurement, or statistical analysis. Statistical significance does not establish that the program caused a change or indicate the probability that a particular explanation is true. Look for evaluators who can explain the size and practical importance of findings, their uncertainty, and the assumptions behind the analysis. Qualitative information also requires systematic analysis rather than selecting a few memorable stories.

For example, a study of adults returning to education might initially focus on employment changes or access to classes. Systematic review of interviews could reveal that earlier caregiving responsibilities also shaped participants' educational opportunities. Survey information might help describe how common that experience was among respondents, while follow-up conversations could clarify its meaning. Combining methods can uncover overlooked patterns, but software or a statistical association cannot by itself establish what caused people's decisions.

How the evaluation will be used

Will the findings guide program improvement, inform community decisions, support accountability, or help determine future funding? The intended uses affect the questions, level of detail, timeline, and expertise needed. Consider whether participants would feel more comfortable speaking with a community member, a staff evaluator, or someone independent of the initiative. No role guarantees trust or neutrality. Choose arrangements that fit the circumstances and manage possible conflicts of interest.

How the evaluation will establish credibility

Credibility depends on the quality and transparency of the work, not simply on whether evaluators are external or community-based. Community participation can improve relevance and accountability, while independent review can help address particular conflicts or technical questions. Explain who will make decisions, how affected people will contribute, and how findings will be reported. These arrangements should reflect the initiative's mission and principles while protecting the integrity of the evaluation.

What should you look for when choosing evaluators?

Some qualities matter for every evaluator, while others depend on the responsibilities of the team.

Evaluations can use many combinations of expertise. A community researcher may have extensive professional experience, an external evaluator may provide limited technical assistance, and staff may carry out parts of the work with training and supervision. Paid status, credentials, and community membership describe different aspects of a person's role; none alone determines their ability to contribute.

An external evaluator might help a community team design the evaluation, conduct a specialized analysis, or work alongside partners throughout the process. A local team might develop the questions and collect information while a consultant supports interpretation and reporting. Whatever the arrangement, specify responsibilities, decision-making authority, communication, data access, and how knowledge will remain available to the organization and community.

For example, consider an evaluation of community HIV services. A planning group could include people living with HIV, service providers, community advocates, and an evaluator. The group would agree on its purpose and ensure that people using or seeking services have meaningful influence over the questions.

Together, members might decide whether a survey, interviews, service records, or a combination would best address their priorities. They could review questions, choose accessible participation methods, discuss privacy concerns, and agree on how information will be analyzed and shared. The evaluator's role would be to support informed choices and explain tradeoffs.

If the evaluator then conducts the technical analysis, the group can still help interpret findings, identify missing context, and develop recommendations. Participation in interpretation should not require access to identifying information or give any partner authority to suppress unfavorable results.

Characteristics to look for in all evaluators

Commitment to the agreed purpose and honest inquiry

Evaluators may have professional, academic, organizational, or personal interests in the work. A study might contribute to a dissertation, publication, funding decision, or advocacy effort. These interests are not necessarily incompatible with a useful evaluation, but they should be disclosed and discussed. Establish written agreements about scope, methods, deliverables, data use, authorship, and reporting. The agreement should protect the evaluation's purpose and the ability to report credible findings without pressure to reach a predetermined conclusion.

Apply the same expectations to staff, community members, funders, and external evaluators. Ask how they would handle evidence that challenges their views or the initiative's preferred approach. Select people who can listen, consider alternative explanations, and work through disagreement. Different perspectives can strengthen an evaluation when they are addressed openly rather than treated as reasons to exclude someone.

Ability to communicate with a broad range of people

Evaluators may work with people of different ages, languages, identities, beliefs, and experiences. They need to explain the work clearly, listen carefully, and create respectful opportunities for participation. Trust develops through reliable conduct, transparency, and responsiveness; it cannot be assumed from a title or shared background.

Have candidates worked with communities and organizations relevant to your initiative? Can they explain methods without unnecessary jargon and adapt their communication to different audiences? Do they ask about language and accessibility needs? Does the team include relevant language skills or a plan for qualified interpretation and translation?

Explore these questions through interviews, examples of previous work, and conversations with former partners. Ask candidates to explain a method or finding in plain language. The team should be able both to gather useful information respectfully and to present results in forms that community members and decision-makers can understand and use.

Cultural responsiveness and willingness to learn

Evaluators should learn about the community's history, relationships, and priorities without assuming that everyone sharing an identity has the same experiences. Ask how they examine their own assumptions, respond to unfamiliar perspectives, and adapt methods with community input. Knowledge of a particular setting is valuable, but curiosity, humility, and accountability remain important even for evaluators who belong to that community.

Respect for every participant and partner

Look for evaluators who treat people respectfully regardless of their role, education, income, or position in the organization. They should provide meaningful choices about participation, welcome questions and criticism, and respond appropriately to concerns. Their conduct affects both the quality of the evaluation and the organization's relationships with the community.

Commitment to privacy and clear confidentiality protections

Evaluators should explain how they will limit collection of identifying information, protect records, control access, and reduce the risk of identification in reports. Confidentiality concerns how information is protected and shared; anonymity means identities are not collected or linked to responses. Do not promise absolute protection that the evaluation cannot provide. Explain relevant limits in advance, and consider how disclosure could affect participants, especially when the work involves sensitive experiences such as domestic violence.

Informed consent is a communication process, not simply a signed form. Explain what participation involves, how information will be used, foreseeable risks, confidentiality protections and limits, and whom to contact with questions. Provide accessible opportunities to decide freely whether to participate. Determine the applicable review and consent requirements before collecting information; the appropriate documentation depends on the activity and its setting.

Commitment to a useful and credible evaluation

Evaluators should follow through on responsibilities, document important decisions, and communicate emerging issues promptly. The organization and community partners also need to make time for interpretation and action. Agree early on who will use the findings, what decisions they will inform, and how recommendations will be considered. Useful evaluation may confirm strengths, identify changes, or reveal that more information is needed.

Ask evaluators how their work can support future learning. This might include practical data-collection tools, staff and community training, clear documentation, or a schedule for reviewing selected indicators. Ongoing evaluation should fit the initiative's needs and capacity rather than require repeating a large study at every stage.

Community or staff planning and evaluation teams

A planning team helps define the evaluation's purpose, questions, scope, and intended uses. It may also help select evaluators. Its responsibilities can include identifying information sources, choosing methods with technical support, allocating tasks, and agreeing on how findings will inform decisions.

An evaluation team carries out the work, including gathering information, analyzing evidence, interpreting findings, and reporting results. It may include the same people as the planning team or bring in additional expertise. Members may be paid staff, compensated community researchers, volunteers, or external partners. Ensure that the team has appropriate skills and support for the methods it will use.

Assemble a team that can complete the work while including perspectives essential to the evaluation. Organizational position alone should not determine whose contributions carry weight. Team size will depend on the scope, workload, and participation arrangements. A smaller working group can consult more broadly, but affected people should have genuine influence over decisions. Consider the following qualities and arrangements:

Representation and meaningful participation

Depending on the initiative, a planning or evaluation team may include:

• Organizational representatives with different responsibilities, such as program staff, administrators, and governing board members
• People who participate in or are affected by the initiative, including those who have encountered barriers to access
• Community partners with relevant knowledge, such as residents, local businesses, service organizations, schools, and public agencies
• Young people when the initiative affects them, with appropriate support and meaningful roles
• People from relevant language communities and groups underrepresented in previous planning or evaluation efforts
• People with research and evaluation skills, whether developed through community work, employment, formal education, or other experience

Understanding of the evaluation's purpose

Team members should understand what the evaluation is intended to learn and how its questions, methods, and uses connect. Provide orientation and opportunities to revisit the purpose, especially when the team includes people new to evaluation or to the initiative.

Willingness to listen and learn

Every member should recognize both what they contribute and what they need to learn. For example, a school leader can learn from a young person who left school before graduating about barriers that school records do not capture. That young person may also want support with particular evaluation methods. Treat these as complementary contributions, and make room for people to question assumptions regardless of their formal position.

Ability to collaborate, with appropriate support

Familiarity with meetings, technical language, and decision-making processes varies among individuals and should not be inferred from income, occupation, education, or background. Agree on accessible meeting practices, explain unfamiliar terms, and offer several ways to contribute. Provide materials in advance, interpretation or other access support where needed, and fair opportunities to speak. Peer support or shared training can help, but it should respect people's preferences and avoid assigning some members a permanently subordinate role.

External evaluators and specialist teams

When considering a consultant, firm, or university team, assess the people who will actually do the work, including their availability, supervision, and responsibilities. A strong proposal should explain how their expertise fits the questions and how they will collaborate with the organization and community. Review both technical competence and the team's approach to relationships, learning, and accountability.

Knowledge of appropriate evaluation methods

Can candidates explain when quantitative, qualitative, or combined methods would be useful? Do they understand sampling, measurement, analysis, and the limits of different designs? Ask how they would adapt their approach to your questions and resources rather than assuming that their usual method is the best fit.

If you are seeking a university partner, consider departments or centers whose methodological skills and subject knowledge fit the initiative. Relevant contacts may include:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Relevant experience and quality of previous work

Ask about evaluations with similar questions, communities, or constraints. What methods did candidates use, what difficulties arose, and how were findings used? Request appropriate work samples and references, while recognizing that newer evaluators may demonstrate readiness through supervised work and relevant experience.

As when you hire someone for a position, use more than the interview or written proposal to assess fit. Speak with previous clients and community partners about reliability, responsiveness, collaboration, and how the evaluator handled disagreement or unexpected findings.

Review reports or other products that candidates are permitted to share. Are the methods and limitations clear? Are conclusions supported by the evidence? Can intended audiences understand the findings and use them? Ask how community members contributed to interpretation and how the evaluator communicated results to different audiences.

Experience working in partnership with communities

Ask for concrete examples of how candidates shared decisions, responded to community priorities, and addressed differences in power or access. How did they handle a suggestion that changed the design? How did they manage disagreement among partners? Look for a willingness to collaborate while maintaining sound methods and honest reporting.

Ability to listen carefully and check understanding

Evaluators should take others' knowledge seriously and explain where their own expertise can help. Careful listening includes asking follow-up questions, summarizing what they have heard, and checking interpretations rather than assuming they understand someone's meaning. This skill helps the team identify useful questions, gather information respectfully, and make sense of findings in context.

In Summary

The people selected to plan and conduct an evaluation shape its quality, usefulness, and relationships with the community. Choose a combination of skills and perspectives that fits the purpose, methods, resources, and setting. Assess relevant experience, communication, respect for participants, management of conflicts, and willingness to work collaboratively. Clear agreements and meaningful community involvement can help the team produce credible findings and support informed action.

Why should you evaluate?

___To see results of work

___To gain public opinion

___To make the proper adjustments if necessary

What should you consider when you select evaluators?

___Select evaluators carefully for reliability 

Possible evaluators:

___Volunteers

___Community being serviced

___Chosen evaluators by a team

Tool: Interview Questions for evaluation consultant selection

From EvalTalk, the discussion group of the American Evaluation Association (AEA).

• What are the main reasons why you think a program should conduct an evaluation?

• What are the main standards and guiding principles for conducting evaluations?

• How would you determine if a program is ready to undergo an evaluation?

• How do you estimate the resources needed to conduct an evaluation? That is, how do you compute the costs of an evaluation, both in money and time (yourself and others)?

• If you had results from an evaluation that were based on quantitative information that conflicted with results from qualitative information, how would you resolve the conflict?

• If you were to conduct a mailed-out survey to program participants, how would you achieve a representative sample and ensure that your results were generalizable to all participants?

• Frequently evaluations involve various and independent branches and agencies that all have different expectations for the evaluation. If you had a conflict among participants in a program evaluation that could threaten the evaluation, how would you resolve the conflict? What steps would you take to proactively avoid conflict?

• Read any interesting articles on evaluation lately? In what journals? What was interesting about it? What other journals do you see regularly?

• Under what circumstances, if any, would you blow the whistle on a client?

Why should you have an evaluation plan?

Evaluation helps you understand how an initiative is being implemented, what is changing, what is working well, what is not working as intended, and what should happen next.

Evaluation should not be treated simply as a final judgment about whether a program “succeeded” or “failed.”

A useful evaluation can help an organization:

• Understand how the initiative is being implemented
• Identify strengths and areas for improvement
• Understand intended and unintended effects
• Make decisions during implementation
• Assess progress toward goals
• Learn from community members and participants
• Meet reporting or accountability requirements
• Decide whether an initiative should be continued, changed, expanded, or ended

There are many evaluation frameworks, tools, and methods available.

Do not select a method simply because it appears in an evaluation handbook or because another program used it.

Begin by asking what you actually need to learn and what decisions the information will help people make.

Collecting large amounts of information is not useful if the information does not answer important questions or inform action.

An evaluation plan can help you:

• Clarify why the evaluation is being conducted
• Identify who should participate in evaluation planning
• Determine what questions are most important
• Identify what evidence is needed
• Select appropriate evaluation methods
• Establish responsibilities for data collection, analysis, interpretation, and reporting
• Protect privacy and address ethical concerns
• Develop a realistic timeline and budget
• Decide how findings will be communicated and used
• Improve the initiative throughout its life cycle

When should you develop an evaluation plan?

Begin evaluation planning as early as practical.

Ideally, evaluation is considered while the initiative itself is being designed.

Early planning can help establish:

• Clear goals and intended outcomes
• Baseline information when needed
• Appropriate measures
• Data-collection systems
• Responsibilities
• Evaluation resources
• Plans for using findings

If an initiative is already underway, it is not too late to begin evaluation.

Determine what information is still available, what can reasonably be collected going forward, and what evaluation questions remain answerable.

Evaluation should support learning throughout implementation rather than occur only after the initiative is finished.

Who should be involved in evaluation planning?

Evaluation can affect many people and organizations.

Identify the people who:

• Are directly affected by the initiative
• Participate in the initiative
• Implement or manage it
• Provide funding or oversight
• Make decisions using the findings
• Have relevant community, professional, technical, or lived knowledge

Different people may use evaluation findings for different purposes.

Involving them early can help make the evaluation more relevant, useful, fair, and responsive to the context in which the initiative operates.

People directly affected by the initiative

People whose lives, services, neighborhoods, organizations, or communities are affected by an initiative may have important knowledge about:

• Whether the initiative addresses a real need
• Whether services are accessible
• How implementation is experienced
• What outcomes matter
• What unintended effects have occurred
• What changes would make the initiative more useful

Do not refer to people primarily as “targets of change.”

Whenever practical, involve people directly affected in shaping evaluation questions, interpreting findings, and deciding what actions should follow.

Program staff, volunteers, and organizational leaders

People involved in implementing the initiative can provide information about:

• Activities
• Resources
• Participation
• Implementation challenges
• Changes made during the program
• Organizational capacity

Their involvement is important, but implementation staff should not be the only people interpreting whether the initiative is effective.

Funders and oversight organizations

Funders may have legitimate reporting, accountability, or learning requirements.

Determine what information is required by:

• Grant agreements
• Contracts
• Government reporting requirements
• Organizational policies

Funding requirements should be considered alongside the information needs of participants, community members, program staff, and other relevant groups.

Do not design an evaluation solely around what the funder wants to know if doing so would leave important community or program questions unanswered.

Evaluators, researchers, and people with technical expertise

An initiative may involve:

• Internal evaluation staff
• Independent evaluators
• University researchers
• Data analysts
• Community-based researchers
• Subject-matter experts

Involve them early enough to help determine what questions can reasonably be answered and which methods are appropriate.

Researchers and evaluators should not automatically control the evaluation simply because they have technical expertise.

Evaluation planning is stronger when technical knowledge is combined with community knowledge, lived experience, program expertise, and the needs of intended users.

What do people need to know from the evaluation?

Different people may need different information.

Before selecting measures or methods, ask:

• Who will use the findings?
• What decisions do they need to make?
• What information would help them make those decisions?
• When will they need the information?
• What information would be useful to people directly affected by the initiative?
• How should findings be communicated so that people can understand and use them?

Evaluation should support decisions

Findings may inform decisions about:

• Program improvement
• Staffing
• Funding
• Program continuation
• Expansion or replication
• Modification of services
• Policy
• Community priorities
• Future evaluation or research

Evaluation should produce honest and useful information.

Do not design measures, analyses, or reporting practices to exaggerate success or minimize unfavorable results.

Unexpected, mixed, or negative findings can provide important information for improvement.

How much should evaluation cost?

There is no single percentage of a program budget that is appropriate for every evaluation.

Evaluation resources depend on factors such as:

• The questions being asked
• Program size and complexity
• Available data
• Required methods
• Evaluation duration
• Reporting requirements
• Staff capacity
• Participant burden
• Available funding

Develop a realistic evaluation budget based on the work required rather than applying a fixed percentage automatically.

Ask:

• What information is essential?
• What information would be useful but optional?
• What information is already available?
• What data can realistically be collected?
• What resources are required to analyze and use the information responsibly?

How do you develop an evaluation plan?

Evaluation planning is often iterative rather than strictly linear.

A practical process includes:

1. Assess the context.
2. Describe the initiative and how it is expected to work.
3. Identify who should participate in evaluation planning.
4. Clarify the purpose of the evaluation and intended uses.
5. Develop and prioritize evaluation questions.
6. Select an evaluation design and appropriate methods.
7. Gather credible evidence.
8. Analyze and interpret findings.
9. Communicate findings in useful forms.
10. Act on what is learned.

Describe the initiative and clarify its goals

Before evaluating an initiative, make sure there is a shared understanding of:

• The problem or need being addressed
• Resources available
• Activities being implemented
• People or communities the initiative intends to reach
• Expected short-term outcomes
• Expected intermediate outcomes
• Expected long-term outcomes
• Important contextual factors

A logic model, theory of change, program roadmap, or another visual representation may help clarify how activities are expected to contribute to outcomes.

Do not assume that every observed change can automatically be attributed to the initiative.

Develop evaluation questions

Evaluation questions should reflect what people need to learn and what decisions the findings will inform.

Common categories include:

Context and need

Possible questions include:

• What community need is the initiative intended to address?
• Has that need changed?
• What contextual factors may affect implementation or outcomes?
• What strengths and resources already exist in the community?

Planning and implementation

Possible questions include:

• Was the initiative implemented as intended?
• What changes were made during implementation and why?
• Who participated?
• Who encountered barriers to participation?
• Were activities accessible and appropriate for the people they were intended to reach?
• How did participants experience the initiative?
• What implementation challenges occurred?

Progress toward objectives

Possible questions include:

• Were planned activities completed?
• Were intended participants reached?
• Were services delivered at the expected level?
• Were short-term objectives achieved?

Measures of participation and activity are important, but they should not automatically be treated as evidence that meaningful outcomes occurred.

Outcomes for participants

Possible questions include:

• What changed for participants?
• Did knowledge, skills, access, behavior, conditions, or well-being change?
• Were there unintended positive or negative effects?
• Did outcomes differ among groups or settings?
• How did participants describe the value or limitations of the initiative?

Community or systems outcomes

Possible questions include:

• Did relevant community conditions change?
• Were organizational practices changed?
• Did policies or systems change?
• Did access to services change?
• Were disparities reduced, maintained, or increased?
• Were there unintended community effects?

Costs and value

Depending on the evaluation purpose, you may also ask:

• What resources were used?
• What did implementation cost?
• How do costs compare with outcomes?
• Could similar outcomes be achieved another way?

Select evaluation methods

Choose methods according to the evaluation questions rather than starting with a preferred method and trying to make every question fit it.

An evaluation may use quantitative methods, qualitative methods, mixed methods, participatory approaches, or a combination.

Monitoring and performance information

Monitoring can help track implementation while an initiative is underway.

Measures may include:

• Activities completed
• Participation
• Services delivered
• Resources used
• Milestones
• Implementation changes
• Short-term outcomes

Monitoring and evaluation are related but are not necessarily the same activity.

Surveys

Surveys can gather information about:

• Experiences
• Knowledge
• Attitudes
• Behaviors
• Satisfaction
• Perceived outcomes
• Barriers

Design survey questions carefully and consider who may be excluded by the method used to administer them.

Interviews and focus groups

Interviews and group discussions can help explain:

• Why something happened
• How people experienced the initiative
• What contextual factors mattered
• Why implementation differed across settings
• What improvements participants recommend

Include people because their knowledge or experience is relevant rather than referring only to “key” or high-status individuals.

Administrative and program data

Existing information may include:

• Participation records
• Service records
• Financial information
• Health records where lawful and appropriate
• Education data
• Public records
• Other administrative information

Evaluate the quality, completeness, relevance, and privacy implications of existing data before using them.

Community-level indicators

Community indicators may help track changes in conditions over time.

Select indicators that are:

• Relevant to the evaluation question
• Clearly defined
• Measured consistently
• Available at an appropriate geographic and population level
• Interpreted in context

A change in a community indicator does not automatically prove that one initiative caused the change.

Comparison and causal designs

Some evaluations need to estimate whether observed outcomes can reasonably be attributed to the initiative.

Depending on the question and context, designs may include:

• Experimental designs
• Quasi-experimental designs
• Comparison groups
• Interrupted time-series approaches
• Other appropriate causal or contribution approaches

Not every evaluation requires an experimental design.

Select the strongest feasible design that is appropriate to the question, ethical considerations, available data, timeline, and resources.

Gather credible evidence ethically

Evaluation planning should address:

• What data will be collected
• From whom
• Why those people or sources were selected
• How information will be collected
• How privacy and confidentiality will be protected
• Who will have access to identifiable information
• How data will be stored
• How long information will be retained
• How findings will be shared
• Whether additional ethical, legal, or institutional review requirements apply

Collect only the information necessary to answer the evaluation questions.

Analyze and interpret findings carefully

Interpretation should consider:

• The evaluation questions
• Program context
• Data quality
• Alternative explanations
• Differences across groups or settings
• Intended and unintended outcomes
• Limitations of the evaluation design

Involve relevant people in interpreting findings when practical.

Different perspectives may reveal explanations or implications that would otherwise be missed.

Do not interpret results only through the perspective of the funder, evaluation team, or program leadership.

Develop an evaluation timeline

Evaluation activities should be aligned with the life cycle of the initiative.

The timeline may identify:

• Evaluation planning
• Baseline data collection
• Ongoing monitoring
• Interim analysis
• Feedback sessions
• Final data collection
• Analysis and interpretation
• Reporting
• Follow-up measurement
• Decisions based on findings

Evaluation may continue after the formal project period

Do not assume that evaluation must end shortly after program activities stop or when a final grant report is due.

Some outcomes may take longer to appear.

Follow-up evaluation may be useful when you need to understand:

• Whether outcomes were sustained
• Whether changes persisted after services ended
• Whether unintended effects emerged later
• Whether systems or policy changes were maintained

The evaluation timeline should match the questions being asked.

Provide feedback throughout the evaluation

Do not wait until the final report to share everything that has been learned.

Interim findings may help:

• Improve implementation
• Identify emerging problems
• Correct data-quality issues
• Adapt services
• Inform upcoming decisions

Make clear when findings are preliminary.

What products should come from an evaluation?

The most useful product is not always a long final report.

Different audiences may need different formats.

Evaluation products may include:

• Full reports
• Executive summaries
• Briefs
• Presentations
• Community meetings
• Data dashboards
• Infographics
• Accessible digital summaries
• Technical appendices
• Internal learning sessions
• Recommendations or action plans

Choose formats according to how people will actually use the information.

Evaluation products should explain:

• Why the evaluation was conducted
• What questions were examined
• Who participated in the evaluation
• What methods were used
• What was found
• What limitations exist
• What differences appeared across groups or settings
• What unintended findings emerged
• What conclusions are supported by the evidence
• What actions should be considered next

Report important unfavorable or unexpected findings as well as positive results.

Use the findings

Evaluation is most valuable when findings lead to learning and action.

After reviewing the evidence, decision-makers may:

• Continue the initiative
• Modify it
• Expand it
• Reduce or discontinue ineffective components
• Improve accessibility
• Address unintended harms
• Change resource allocation
• Revise the theory of change
• Conduct additional evaluation or research

Decide during evaluation planning how findings will reach the people able to act on them.

What standards should guide an evaluation?

Evaluation standards and professional guidance evolve over time.

Current public-health evaluation guidance emphasizes several broad principles for high-quality evaluation.

Relevance and utility

Evaluation should answer questions that matter and provide information people can use.

Rigor

Methods should be appropriate to the questions, and conclusions should be supported by credible evidence.

Limitations should be reported clearly.

Independence and objectivity

Evaluation processes should reduce inappropriate influence, conflicts of interest, and avoidable bias.

Evaluators should reflect on how their own assumptions and roles may affect the work.

Transparency

Document:

• The purpose of the evaluation
• Evaluation questions
• Methods
• Important decisions made during the evaluation
• Changes to the original plan
• Limitations
• How conclusions were reached

Ethics

Protect the dignity, rights, safety, privacy, and confidentiality of people who contribute information.

Consider who bears the burden of data collection, who benefits from the evaluation, and whether the evaluation could create unintended harm.

Build collaboration, fairness, and use throughout the evaluation

High-quality evaluation is not simply a technical exercise.

Throughout the process:

• Engage relevant people collaboratively
• Consider equity and fairness in participation, methods, interpretation, and use
• Learn from findings as they emerge
• Use insights to improve the initiative and future decisions

Ask who has been included in evaluation decisions and whose perspective may be missing.

Pay particular attention to whether people most affected by the initiative have meaningful opportunities to contribute.

In Summary

An evaluation plan helps an initiative determine what it needs to learn, how it will gather credible evidence, who should participate, when evaluation activities should occur, and how findings will be used.

Begin planning evaluation as early as practical, but do not abandon evaluation simply because an initiative is already underway.

Start with the decisions and questions that matter rather than with a preferred method or a large list of available measures.

Involve people directly affected, program staff, community partners, funders, decision-makers, evaluators, and other relevant participants according to the evaluation's context.

Do not refer to people as “targets of change” or assume that technical experts are the only people who hold important evaluation knowledge.

Develop a realistic budget based on the evaluation's actual questions, design, data needs, timeline, and available resources rather than assuming that every initiative should spend the same percentage of its budget on evaluation.

Clarify how the initiative is expected to work and develop evaluation questions addressing implementation, outcomes, community or systems change, unintended effects, and other priorities.

Select methods that fit the questions.

Surveys, interviews, monitoring data, administrative information, community indicators, qualitative methods, and experimental or quasi-experimental designs can all be useful in the appropriate circumstances.

Do not assume that every evaluation must prove that the initiative caused an observed outcome.

Collect and use information ethically, protect privacy, explain limitations, and avoid designing an evaluation to produce a preferred result.

Share learning throughout implementation when doing so can improve the initiative.

Use reports, briefs, dashboards, presentations, community discussions, or other formats appropriate to the people who need the findings.

Evaluation may continue after an initiative formally ends when important outcomes require longer-term follow-up.

Finally, evaluation should not end with the production of a report.

Its value comes from using credible evidence and multiple forms of knowledge to understand an initiative, improve decisions, address unintended effects, and determine what should happen next.

You understand that you should have an evaluation plan because:

___It guides you through each step of the process of evaluation

___It helps you decide what sort of information you and your stakeholders really need

___It keeps you from wasting time gathering information that isn't needed

___It helps you identify the best possible methods and strategies for getting the needed information

___It helps you come up with a reasonable and realistic timeline for evaluation

___Most importantly, it will help you improve your initiative!

You understand who your project or initiative's stakeholders are:

___Community groups

___Grantmakers and funders

___University-based researchers

You have taken into consideration:

___What stakeholders want to know about the evaluation

___What decisions stakeholders need to make

___How stakeholders would use the data to inform decisions

When considering how to balance costs and benefits, you have asked yourself the following questions:

___What do you need to know?

___What is required by the community?

___What is required by funding?

You understand these four main steps to developing an evaluation plan:

___Clarify program objectives and goals

___Develop evaluation questions

___Develop evaluation methods

___Set up a timeline for evaluation activities

Clarify the objectives and goals of your initiative:

___Make a table of program components and elements

Develop evaluation questions:

___You understand the four main categories of evaluation questions:

• Planning and implementation issues
• Assessing attainment of objectives
• Impact on participants
• Impact on the community

___You have considered the best possible methods to answer those questions

Develop evaluation methods to best address those questions:

___You understand how to use the monitoring and feedback system's three main elements (process measures, outcome measures, and observational system)

___You understand how to use member surveys about the initiative (member survey of goals, member survey of process, and member survey of outcomes)

___You understand how to use the goal attainment report

___You understand how to use behavioral surveys

___You understand how to use interviews with key participants

___You know how to use community-level indicators of impact

Setting up a timeline for evaluation activities:

___You understand that you should begin right now, or at least at the beginning of the initiative

___You've outlined questions for each stage of development of the initiative

___You've completed a table listing: key evaluation questions, type of evaluation measures to be used to answer them, type of data collection, and experimental design .

___You've determined when you feel it's appropriate to provide feedback and reports

___You will provide feedback and reports at the end of the evaluation

___You will also provide periodic feedback and reports throughout the duration of the project or initiative

___You've decided when the evaluation will end

___You've mapped out a proposed evaluation timeline

Expected evaluation products:

___You will make a report that you can share with everyone involved which includes effects expected by shareholders, differences in the behaviors of key individuals, and differences in conditions in the community

___You've decided whether to also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team

Evaluation standards:

___You've decided what standards you will use to ensure an accurate and useful evaluation

Example 1: Table of Program Components and Elements

Here are a few of the items from the table of program components and elements for a fictional teen pregnancy prevention program:

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery
Risk factor: Unsupervised activities by teens	Provide alternative activities	After-school activities (sports programs, hobby / craft classes, movies, etc.) Weekend "lock-ins", non-alcoholic teen dance club Daytime activities during school holidays	Targets:   Youth   Agents: Youth Teachers Parents	Group sessions Workshops Meetings
Risk factor: Lack of access to contraceptives	Increased access to health services	Extend hours at school health clinics Increase condom distribution by county health department Brown bag program at area pharmacies (teens wishing to buy contraceptives are able to call ahead and pick them up in a plain brown bag)	Targets:   Youth Agents: Health providers	Walk-in clinics Health Department Businesses
Protective factor: Involvement in the faith community	Encourage youth involvement in the faith community	Increase church-sponsored youth activities Institute church-based sexuality education Church-sponsored parent-child communication sessions	Targets: Youth Parents Agents: Youth Parents Religious leaders	Discussions Meetings Educational sessions Workshops

Example 2: Evaluation Questions and Methods

Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (Y.E.S.), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally, and interact with positive role models:

		Type of Data Collection*					Experimental Design**
Key Evaluation Question(s)	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- and Post-test Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1. Who participates? Is there diversity among the participants?	1. Basic demographic data (e.g., age, gender, disability, race, economic status, etc.) on participants.	X	X				X
2. Why do participants enter and leave the program?	2. Information on participants' reasons for entering and leaving the program.	X	X				X
3. Are there a variety of service and alternative activities generated?	3. Frequency of different types of service provided and alternative activities generated.			X	X	X	X
Assessing Attainment of Objectives	Outcome Measures
1. How many people participate?	1. Number of people participating.			X	X	X	X		X
2. How many hours are participants involved in service and activities?	2. Number of hours of participation by type of activity.			X	X	X	X		X
3. How many people are trained in cultural awareness and sensitivity?	3. Number of people trained in cultural awareness and sensitivity.			X	X	X	X		X
Impact on Participants
1. How do behavior and attitudes change as a result of participation in the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X		X	X
2. Does participation affect the incidence of problems?	2. Incidence of problems in living (e.g., substance use, gang involvement).	X	X			X		X
3. Are participants satisfied with the experience?	3. Satisfaction ratings from participants.	X					X
Impact on Community
1.  What resulted from the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X			X
2. Do the benefits of the program outweigh the costs?	2. Cost benefit data.		X	X		X	X
3. Are community members satisfied with the participants and the service they provided?	3. Satisfaction ratings from beneficiaries and community members.	X					X

* Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

• Survey/Scale: i.e., member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys
• Structured Interview: i.e., interview with key participants
• Staff Report/Log: i.e. monitoring system, goal attainment report
• Direct Observation: This refers to the direct observation of behavior by researchers .
• Archival Records: i.e., community-level indicators of impact

**Here are explanations of these terms:

• Case Study: an in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment--good for answering "how" or "why" questions
• Pre-/Posttest Control Group: monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made
• Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative

Example 3: Timeline for Evaluation Activities

This is the timeline for an evaluation being done by Kansas City's Promise on their initiative to promote community and personal caring for youth:

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system	[ - - - ]
Train participants / staff in data entry	[ - - - ]
Maintain system (data collection and reporting)	[- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - ]
Critical events
Identify key people	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Conduct interviews	[ - - ]	[ - - ]	[ - - ]	[ - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Community-Level Indicators
Identify local indicators	[ - - - - ]
Identify comparison site	[ - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Other Indicators (e.g., behavioral surveys, data observations)
Identify	[ - - - - - -	- - - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - -  - - - - -	- - - - - - - - - -]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]

 

Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts

Download PPT Slides >>

 

Table of Contents

General Evaluation Resources
Steps in Community Evaluation (and Related Resources)

1. Design Stakeholder Involvement
2. Identify Resources for Monitoring and Summative Evaluation
3. Describe the Intervention’s Framework, Logic Model, or Theory of Change
4. Focus the Monitoring and Summative Evaluation  Plan
5. Plan for credible methods
6. Synthesize and Generalize
7. Systems and Evaluation

 

General Evaluation Resources

• Institute of Medicine (IOM):
  • ​Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010)
  • Measuring Progress in Obesity Prevention: Workshop Report (2012)
  • Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012)
  • Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013)
• U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
• Community Tool Box (CTB):
  • Toolkit: Evaluating the Initiative
  • A Framework for Program Evaluation: A Gateway to Tools
  • Evaluating Comprehensive Community Initiatives
  • Toubleshooting Guide: We don’t know how to evaluate our program or initiative

Steps in Community Evaluation (and related resources)

 

 

Back to Top

 

1. Design stakeholder involvement
   
   1. Identify stakeholders
   2. Consider the extent of stakeholder involvement
   3. Assess desired outcomes of monitoring and summative evaluation *
   4. Define stakeholder roles in the monitoring and summative evaluation *
   5. Resources:
      • Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Statement On Cultural Competence In Evaluation
      • Community Tool Box:
        • ​Identifying and Analyzing Stakeholders and Their Interests
        • Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
        • Participatory Evaluation
        • Gathering Information: Monitoring Your Progress

 

 

Back to Top

 

2. Identify resources for monitoring and summative evaluation
   
   1. Person-power resources
   2. Data collection resources
   3. Resources:
      • Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Identifying Community Assets and Resources
        • ​Choosing Evaluators

 

Back to Top

 

3. Describe the intervention’s framework, logic model, or theory of change
   
   1. Purpose or mission
   2. Context or conditions
   3. Inputs: resources and barriers
   4. Activities or interventions
   5. Outputs of activities
   6. Intended effects or outcomes
   7. Resources:
      • Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Toolkit: Developing a Framework or Model of Change
        • Developing a Logic Model or Theory of Change

 

 

Back to Top

 

4. Focus the monitoring and summative evaluation plan
   
   1. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
   2. Set priorities by end-user questions, resources, context
   3. What questions will the monitoring and summative evaluation answer?
   4. Ethical implications (benefit outweighs risk)
   5. Resources:
      • Institute of Medicine:
        1. Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        2. Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Guiding Principles For Evaluators
      • Community Tool Box:
        1. Choosing Questions and Planning the Evaluation
        2. Ethical Issues in Community Interventions
        3. Documenting Health Promotion Initiatives Using the PAHO Guide

 

 

Back to Top

 

5. Plan for credible methods
   
   1. Stakeholder agreement on methods
   2. Indicators of success
   3. Credibility of evidence
   4. Resources:
      • Institute of Medicine:
        • Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget)
        • Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • National Collaborative on Childhood Obesity Research (NCCOR):
        • Catalogue of Surveillance Systems
        • ​Measures Registry
      • Community Tool Box:
        • Data Collection: Designing an Observational System
        • Selecting an Appropriate Design for the Evaluation
        • Collecting and Analyzing Data
        • Collecting and Using Archival Data
        • Qualitative Methods to Assess Community Issues
        • Implementing Photovoice in Your Community
        • Community-Level Indicators: Some Examples
      • Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users

 

Back to Top

 

6. Synthesize and generalize
   
   1. Disseminating and compiling studies
   2. Learning more from implementation
   3. Ways to assist generalization
   4. Shared sense-making and cultural competence
   5. Disentangling effects of interventions
   6. Resources:
      • Institute of Medicine:
        • Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support
      • Community Tool Box:
        • Troubleshooting Guide: There is Not Enough Change in the Community or System
        • Troubleshooting Guide: There is Not Enough Improvement in Outcomes
        • Troubleshooting Guide: There are Unintended or Unwanted Outcomes
        • Toolkit: Enhancing Cultural Competence
        • Adapting Community Interventions for Different Cultures and Communities
        • Providing Feedback to Improve the Initiative
        • Communicating Information to Funders for Support and Accountability
        • Refining the Program Intervention Based on Research
        • The County Health Rankings & Roadmaps Take Action Cycle

 

 

Back to Top

 

7. Systems and Evaluation
   • Resources:
     • IOM:
       • Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
       • Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation
       • Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
     • Community Tool Box:
       • Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders
       • Best Change Processes
       • MAPP: Mobilizing for Action through Planning and Partnerships

 

Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box.

 

*Monitoring is the tracking of the implementation of interventions compared to standards of performance.
*Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions.

 

 

Tool 2: Table of Program Components and Elements

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery

 

Tool 3: Evaluation Questions and Methods

		Type of Data Collection *					Experimental Design **
Key Evaluation Questions	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured Interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- Posttest Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Assessing Attainment of Objectives	Outcome Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Participants
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Community
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.

Tool 4: Evaluation Timeline

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system
Train participants / staff  in data entry
Maintain system (data collection and reporting)
Critical events
Identify key people
Conduct interviews
Report results
Community-Level Indicators
Identify local indicators
Identify comparison site
Secure data
Report results
Other Indicators (e.g., behavioral surveys, data observations)
Identify
Secure data
Report results

Experienced community builders know that involving stakeholders—the people directly connected to and affected by their projects—is essential. Their knowledge, lived experience, and perspectives can help shape, adjust, and improve the work so that it responds to community priorities and the needs of the people the initiative is intended to support. This is particularly important in evaluation.

As we have previously discussed, community-based participatory research can be used to describe a community, assess community issues, strengths, and needs, identify and select promising practices, and conduct evaluation. We consider participatory evaluation important enough to give it a section of its own and to show how it fits into the larger participatory research process.

It is a good idea to build meaningful stakeholder participation into a project from the beginning. One of the best ways to determine an appropriate direction for your work is to involve stakeholders in identifying community priorities, strengths, needs, and opportunities for meaningful impact. Likewise, one of the best ways to understand how a project affects people is to include those who participate in, experience, or are directly affected by its information, services, or advocacy in the evaluation process.

People directly involved in a project—including participants, staff, volunteers, community members, and partners—often have important insight into what is actually happening. Previously, we have discussed how you can involve community members in conducting research and choosing issues and priorities to address. This section focuses on involving stakeholders throughout the project, including in its evaluation, and on how that participation can strengthen the work and its outcomes.

What is participatory evaluation?

When many people think of evaluation, they imagine something that happens at the end of a project and determines whether it was successful. In practice, evaluation is most useful when it is integrated into a project from the beginning. Participatory evaluation involves stakeholders—particularly people directly affected by the project or involved in carrying it out—in developing an understanding of how the project is working and using that understanding to improve the work.

Participatory evaluation is more than simply asking stakeholders to take part. It creates opportunities for shared decision-making among community members, participants, staff, evaluators, and other partners throughout a project. People directly affected by the work contribute their lived experience, priorities, and expertise alongside professional and organizational knowledge. Together, participants can shape evaluation questions, interpret findings, and use what they learn to improve the project in ways that reflect community-defined needs and goals.

This approach to planning and evaluation depends on mutual trust, respect, transparency, and accountability. These relationships develop over time and are strengthened when organizations begin by learning about the community's history, culture, priorities, strengths, and existing knowledge. Respecting people's expertise and recognizing the value of different forms of knowledge can help build lasting trust and meaningful participation across a wide range of cultural, geographic, and socioeconomic contexts.

Another important part of a participatory process is making sure everyone involved has the information, preparation, resources, and support needed to participate fully. Some stakeholders may be unfamiliar with evaluation or research processes. Others may need support collaborating across different backgrounds or understanding how evaluation findings can be used. We will discuss stakeholder involvement, trust-building, accessible training, and other forms of support throughout this section.

The purpose of evaluation is not simply to determine what happened, but to use what is learned to strengthen the project and inform future action.

To accomplish this, evaluation should examine at least three areas:

• Process. The process of a project includes the planning, coordination, and logistical activities needed to establish and carry it out. Did we conduct an appropriate assessment beforehand to understand community priorities, strengths, and needs? Did we use what we learned to shape the project? Did we follow the intended timeline and organizational structure? Did we meaningfully involve the people we intended to involve? Did we have or obtain the necessary resources? Were staff, volunteers, and community partners adequately prepared and supported? Did we have the community partnerships and support we expected? Did we document our work accurately and in a timely way? Did we monitor and evaluate the work as planned?
• Implementation. Project implementation refers to the actual work of carrying out the project. Did we do what we intended to do? Did we reach and engage the people and communities we intended to reach? Did we use the approaches we planned to use? Was the level of activity what we expected—for example, did we provide the intended number of hours, events, services, or opportunities? What exactly did we provide or carry out? Did we make planned or unplanned changes, and what led to those changes?
• Outcomes. A project's outcomes are the changes or results associated with the work. Did the project contribute to the outcomes we hoped for? Were there other expected or unexpected effects? Who experienced benefits, challenges, or no meaningful change? Were there differences in outcomes among groups? What factors may have contributed to the results? What could we change to make the work more effective, equitable, or responsive?

Many people who write about participatory evaluation combine the first two areas into process evaluation and add another area—impact evaluation—alongside outcome evaluation. Impact evaluation considers longer-term changes associated with a project, including what happens after a particular program or initiative has ended.

For example, a community-based agricultural or public health initiative might focus on sharing skills and resources that local participants can continue using and adapting after the formal project ends. If an evaluation five or ten years later finds that the practices remain useful, have been adapted to local circumstances, and have spread through community networks, that may indicate sustained and meaningful impact.

To examine these areas effectively, evaluation should begin alongside assessment and planning rather than being added only after implementation.

In a participatory evaluation, stakeholders should be involved in:

• Naming and framing the issue, opportunity, or goal to be addressed
• Developing a theory of practice or logic model for how the desired change may occur
• Identifying the questions to ask about the project and the most appropriate ways to answer them

What is the goal, for example, of a program to improve school meals? It might be to increase access to fruits, vegetables, whole grains, and other nutrient-dense foods; improve students' satisfaction with meals; strengthen food literacy; increase participation in school meal programs; or support students' overall health and well-being.

The evaluation questions should reflect those goals. Measures might include what foods are offered, what students choose to eat, food waste, student feedback, meal participation, and relevant indicators of health and well-being. Looking at multiple outcomes provides a fuller understanding of the program's effects and avoids defining success primarily in terms of students' body weight.

• Collecting information about the project
• Interpreting and making sense of that information
• Deciding what to celebrate, continue, adjust, or change based on what is learned

Why would—and why wouldn't—you use participatory evaluation?

Why use participatory evaluation? The short answer is that it can be one of the most effective ways to understand what communities need and value, how a project is working, and what should change over time. It can also strengthen relationships, skills, shared decision-making, and community capacity in ways that more conventional evaluation approaches may not.

Some of the major advantages of participatory evaluation include:

• It can provide a fuller understanding of community priorities and a project's effects. When stakeholders are involved from the beginning in determining what should be evaluated, why it matters, and what the project should focus on, the work is more likely to reflect community priorities and the evaluation is more likely to capture outcomes that matter to the people affected.
• It can provide information that might otherwise be missed. When evaluation depends partly on community knowledge and experience, people may feel more comfortable sharing information with trusted peers, community partners, or evaluators who understand the local context.
• It helps explain what is and is not working from the perspective of people directly involved. Participants, staff, volunteers, and community partners can provide insight into which parts of a project are useful, which create barriers, and which may need to change.
• It can help explain why something does or does not work. People directly affected by a project can often identify practical, cultural, organizational, or structural reasons why a particular approach is effective for some people and less effective for others.
• It can contribute to a more effective and responsive project. Ongoing feedback from people directly involved helps projects identify emerging needs, recognize unintended effects, and make timely adjustments.
• It supports stakeholder agency and shared decision-making. Participatory evaluation creates opportunities for people who may historically have been excluded from formal decision-making processes to help determine how a project is understood, evaluated, and improved.
• It can center perspectives that institutions and decision-makers have not always included. Some communities experience barriers to participation in organizational, governmental, or research decision-making. Participatory evaluation can create more meaningful opportunities for people to contribute their knowledge, priorities, and perspectives and to influence decisions that affect them.
• It can strengthen skills that are useful in employment, education, civic participation, and other areas of life. Participatory evaluation may provide opportunities to develop research, communication, collaboration, critical-thinking, problem-solving, facilitation, data interpretation, and project-management skills.
• It can strengthen confidence while recognizing participants' existing knowledge and abilities. Being treated as a valued partner, contributing expertise, and seeing one's ideas influence real decisions can reinforce people's confidence in their ability to contribute to community change.
• It can strengthen people's ability to influence decisions that affect their lives and communities. Participating in a collaborative evaluation can provide experience navigating organizations, analyzing information, communicating recommendations, and taking collective action.
• It encourages shared ownership of the project. When people have meaningful opportunities to shape a project and its evaluation, they are more likely to see the work as a collaborative effort rather than something developed and imposed by others.
• It can encourage creativity. Bringing together people with different experiences, knowledge, and perspectives can generate new ideas and help challenge assumptions about familiar issues.
• It encourages collaboration. Participatory evaluation works best when participants recognize that different people bring different forms of knowledge, experience, and expertise. Collaboration of this kind can strengthen the evaluation while also building relationships that support future community efforts.
• It fits within a broader participatory approach. When assessment and project planning have already involved community members, participants, staff, and other partners, it makes sense for evaluation to follow the same collaborative approach. Planning for evaluation as part of the overall project can strengthen both the work and the evaluation process.

Participatory evaluation also involves challenges. Whether those challenges outweigh the benefits depends on the circumstances, resources, goals, timeline, and community context. Understanding potential barriers can help you decide whether a participatory approach is appropriate and how to design it effectively.

Potential challenges of participatory evaluation include:

• It can take more time than a conventional evaluation process. Meaningful participation requires time for relationship-building, discussion, collaborative decision-making, training, accessibility, and coordination. Bringing together people with different schedules, perspectives, and experiences can require more time than an evaluation conducted primarily by professional evaluators.
• It requires trust among participants. Historical inequities, prior negative experiences with institutions, differences in power, cultural differences, or unresolved conflicts can affect how comfortable people feel participating. Trust develops through transparency, accountability, follow-through, respect, and meaningful opportunities to influence decisions.
• You need to include a broad range of perspectives, not only established leaders. A participatory process can reproduce existing power structures if organizations consult only with people who already hold formal or informal leadership roles. Leaders may provide important perspectives, but they should not automatically be assumed to represent everyone in a community.

Involving only established leaders can exclude important experiences and perspectives, particularly those of people who have had fewer opportunities to participate in formal decision-making. Recruitment should therefore seek a broad range of participants, including people with different experiences, backgrounds, roles, and relationships to the project. This may require additional outreach and relationship-building, but it can make the evaluation more representative, credible, and useful.

• Participants may need accessible training, tools, and ongoing support. People come to participatory evaluation with different levels of experience with research, meetings, data collection, and analysis. Training and support should build on the knowledge participants already have and provide practical opportunities to develop additional skills.
• You need shared commitment from participants. Because participatory evaluators may take on significant responsibilities, people should understand the expectations, have meaningful input into their roles, and receive the resources and support needed to participate.
• Participants may face practical barriers that affect their ability to participate consistently. Work schedules, caregiving responsibilities, health needs, transportation, financial constraints, seasonal employment, and other circumstances can affect when and how people are able to participate. Participatory evaluation should anticipate these barriers rather than treating them as individual shortcomings. Flexible scheduling, transportation assistance, childcare, accessible meeting options, compensation for participants' time, and other forms of support can make participation more equitable and sustainable.
• You may need flexible and accessible ways to collect, document, and communicate information. Participants may have different literacy preferences, languages, communication styles, accessibility needs, or levels of familiarity with evaluation. Written reports may work well for some participants, while others may prefer verbal discussion, visual methods, audio, video, diagrams, mapping, storytelling, or other formats.
• Funders and policymakers may have different expectations about evaluation. Some may prefer external evaluations or particular research designs and may be unfamiliar with participatory approaches. Clear communication about the purpose, methods, rigor, benefits, and limitations of participatory evaluation can help build understanding and support.

Some of these challenges can also create opportunities. Training may strengthen transferable skills. Developing accessible ways to communicate information may improve the evaluation for everyone. Building trust can strengthen relationships beyond the evaluation itself. Helping funders and policymakers understand participatory approaches may also influence how future community work is supported. Even so, meaningful participation requires time and resources, and those realities should be considered when planning an evaluation.

When might you use participatory evaluation?

Participatory evaluation may be especially useful:

• When your project is already committed to a participatory approach. Evaluation can then be designed collaboratively as part of the broader project.
• When meaningful participation and useful results are more important than completing the evaluation as quickly as possible. Participatory evaluation requires time for collaboration, but that investment can lead to richer information and stronger community relationships.
• When funders and organizational partners support participatory approaches. Some funders may require external evaluation, while others may welcome participatory methods. Early communication can help clarify expectations and identify opportunities to combine different evaluation approaches when appropriate.
• When community members have important knowledge that external evaluators or project staff may not have. Historical, political, social, cultural, environmental, and interpersonal factors can influence how a project operates and how its outcomes should be interpreted. Community participants can help evaluators understand these contexts.
• When trust and relationships affect whether people are willing to share information. Community members may be more comfortable discussing experiences with people they know and trust, particularly when topics are sensitive or when institutions have previously failed to listen or follow through.
• When strengthening participant agency, leadership, or transferable skills is part of the project's goals. In these situations, the evaluation process itself can contribute to the broader purpose of the project.
• When you want to strengthen relationships and collaboration within a community. Participatory evaluation can create opportunities for people who may not regularly work together to exchange perspectives, build trust, and develop relationships that may support other community efforts.

Who should be involved in participatory evaluation?

We have referred throughout this section to stakeholders—people and groups who are affected by, involved in, responsible for, or able to influence the project being evaluated. Who those stakeholders are will vary depending on the project's focus, funding, activities, setting, and intended outcomes.

Groups that are often involved include:

• Participants and community members. These are people the project is intended to support or whose lives may be directly affected by it. They might include people living with a particular health condition, members of a particular community, recent immigrants, residents of a neighborhood, people participating in employment programs, or people affected by a community-wide initiative. Their experiences and perspectives are essential to understanding how the project is working.
• Frontline staff and volunteers. These are the people who carry out much of the project's day-to-day work. They may be professionals, people with specialized knowledge, community health workers, peer leaders, mentors, teachers, advocates, organizers, or volunteers. Because they interact closely with participants and implementation processes, they often have valuable insight into how the project functions in practice.
• Administrators and coordinators. These are people who oversee the project or particular aspects of it. They can provide insight into planning, operations, resources, partnerships, decision-making, and organizational constraints.
• External evaluators, when involved. External evaluators may provide methodological expertise, facilitate collaborative processes, support data analysis, or help organizations meet evaluation requirements.
• Community leaders, officials, and organizational partners. Their support may be important to the success of a project, and participation can create opportunities for greater understanding of community priorities and perspectives.
• Others whose lives or work are affected by the project. This group will vary substantially depending on the project and may include people whose employment, property, services, environment, access to resources, or daily experiences may be affected by the project's activities or outcomes.

For example, an environmental initiative or neighborhood plan might affect how local property owners, residents, businesses, community organizations, and other stakeholders use or experience a particular area.

How do you conduct a participatory evaluation?

Participatory evaluation can include aspects of project design as well as evaluation. What you evaluate depends on what the community and project partners want to understand and accomplish. Identifying evaluation questions helps determine what information will be collected and what aspects of the project will receive attention. When those questions emerge from an assessment process in which stakeholders have meaningfully participated, evaluation becomes part of a broader community-based participatory research process.

A participatory evaluation generally involves two interconnected stages. One involves identifying, recruiting, preparing, and supporting stakeholders who will participate in the evaluation. The other involves planning and carrying out the project and its evaluation. This second stage includes six steps:

• Naming and framing the issue, opportunity, or goal
• Developing a theory of practice or logic model
• Deciding what questions to ask and how to gather the information needed to answer them
• Collecting information
• Analyzing and interpreting the information
• Using what is learned to celebrate progress and adjust or improve the project

We will examine both stages in more detail.

Finding and preparing stakeholders to participate as evaluators

Recruiting participants for a participatory evaluation often requires intentional outreach and relationship-building. People may already have many demands on their time, may be unfamiliar with evaluation, or may be unsure whether their participation will genuinely influence decisions.

This is where trust-building becomes especially important. Community members may have good reasons to be cautious about outside organizations or institutions, particularly if previous engagement efforts were extractive, promises were not honored, or community input was collected but not acted upon. Building trust requires transparency, consistency, accountability, and follow-through. Organizations may need to demonstrate over time that participation will be meaningful and that community perspectives will influence decisions.

Once relationships have been established, participants may need different forms of preparation and support depending on their prior experience. Training should build on the skills, knowledge, and lived experience people already bring while providing any additional tools needed to participate confidently in the evaluation.

If the evaluation is part of a broader participatory effort, you may need to decide whether the same people who conducted assessments or helped plan the project should also participate in evaluation. There are advantages and disadvantages to either approach. An existing team may already understand the project, have experience working together, and be familiar with research methods.

At the same time, people who helped design or implement a project may have strong hopes for its success. That perspective can provide valuable insight, but it can also influence how findings are interpreted. Clear evaluation processes, reflection on potential bias, multiple perspectives, and transparent decision-making can help address this concern.

Capacity and workload are also important considerations. Evaluation may require additional time from people who have already contributed substantially to assessment and planning. Some may want to continue, while others may prefer to step back. Bringing in additional participants can distribute the workload and broaden the perspectives represented.

The best approach will depend on the people involved, the community context, available resources, recruitment capacity, and the commitments participants have already made.

Recruit participant evaluators

Recruitment can take many forms. In some situations, a broad invitation may be appropriate. In others, it may make sense to reach out to specific individuals, community organizations, participants, staff, or trusted community connectors. You might also ask community members to identify people whose experiences and perspectives should be represented.

Some basic guidelines for recruitment include:

• Use communication channels and approaches that reach the people you hope to involve
• Make the purpose and expectations as clear as possible
• Use clear, accessible language and provide information in the languages used by participants
• Share recruitment information in places and spaces the intended participants already use
• Reach out personally when appropriate, including through trusted community members and organizations
• Explain what people may gain from participation as well as how their participation may contribute to the project and community
• Make clear that participants are being invited because their knowledge, experiences, and perspectives are valuable
• Be transparent about the time, responsibilities, decision-making authority, and expectations involved
• Work with participants to identify roles and responsibilities that are realistic for them
• Provide or arrange practical supports—such as childcare, transportation, technology access, interpretation, accessibility accommodations, or compensation—when possible
• Ask participants to recommend others whose perspectives should be included

Potential participant evaluators—particularly community members whose participation is not part of their paid employment—should understand the expected commitment before agreeing to participate. The length of an evaluation will vary depending on the project, but some evaluations may continue for many months or longer. Planning should recognize that people's availability may change over time.

Discuss with participants what support they will need in order to participate meaningfully. This may include transportation, childcare, interpretation, accessible meeting formats, technology, flexible scheduling, or compensation for their time and expertise. Meetings should be scheduled in ways that reduce unnecessary barriers. When staff participation is required as part of their employment, evaluation responsibilities should be incorporated into their paid workload rather than treated as additional unpaid labor.

Aim to create an evaluation team that represents a range of stakeholder experiences and perspectives. Recruiting only established leaders may leave out people whose experiences differ from those of formal representatives. Broad participation can strengthen the quality of the evaluation while also creating opportunities for more people to develop leadership and decision-making experience.

Prepare and support participant evaluators

Participants may need different kinds of preparation depending on their experiences, roles, and familiarity with evaluation. Some may benefit from an introduction to participatory evaluation, meeting facilitation, research ethics, interviewing, observation, data interpretation, or other skills. Training should be accessible, practical, and responsive to participants' needs rather than based on assumptions about what they do or do not know.

Training can be organized in many ways. It might take place in short sessions over several weeks or months, during a concentrated workshop or retreat, through mentoring and coaching, or through a combination of approaches. The format should reflect participants' schedules, accessibility needs, learning preferences, and the demands of the project.

Different topics may also call for different learning methods. Facilitation and meeting skills may be strengthened through mentoring and practice. Interviewing may be learned through role-playing and feedback. Community engagement strategies may be best taught or co-developed by community participants themselves. Whenever possible, training should recognize that expertise exists throughout the group rather than flowing only from professional evaluators to community participants.

Some areas in which preparation or training may be helpful include:

• The participatory evaluation process. How participatory evaluation works, its goals, the roles participants may play, how decisions will be made, and what participants can expect.
• Meeting and collaboration skills. Listening, contributing to discussion, facilitating participation, navigating disagreement or conflict, sharing decision-making, and establishing group agreements.
• Interviewing. Building rapport, asking open-ended and follow-up questions, listening carefully, documenting responses accurately, being aware of body language and tone, and conducting individual or group interviews.
• Observation. Understanding different forms of observation, choosing appropriate settings and times, identifying relevant information, recognizing potential bias, and documenting observations accurately.
• Recording information and reporting it to the group. Documenting what participants say or do, the context in which information was collected, relevant environmental or situational factors, and other information needed to interpret findings responsibly.

Participants should have access to recording and reporting methods that work for them. Depending on people's preferences, languages, accessibility needs, and the setting, useful approaches may include written notes, drawings, maps, diagrams, photographs, audio recordings, video, storytelling, or other formats. When audio, photography, or video is used, participants should understand how the material will be used and provide informed permission before recording begins.

• Analyzing information. Critical thinking, interpreting quantitative and qualitative information, recognizing patterns and limitations, considering alternative explanations, and determining what findings may mean for the project.

Planning and implementing the project and its evaluation

A fully participatory approach assumes that stakeholders have meaningful opportunities to contribute not only to evaluation, but also to assessment, planning, implementation, and decision-making. The same people do not necessarily need to participate in every phase. However, when community members and other stakeholders have been involved from the beginning, they often have a deeper understanding of the project's goals, history, assumptions, and structure. That knowledge can strengthen the relevance and interpretation of the evaluation.

Naming and framing the issue or goal to be addressed

Identifying what you are evaluating helps define what the project is intended to address and accomplish. Community members, organizations, project participants, staff, and others with a stake in the work can collaborate to develop a shared understanding of priorities and desired outcomes. By examining community concerns alongside strengths, resources, relationships, and existing efforts, participants can identify where attention and action may be most useful.

Naming an issue or goal means identifying what the community wants to understand, prevent, improve, or change. Framing refers to how that issue is understood and what factors are considered relevant.

Consider community violence involving young people. If the issue is framed primarily as a law-enforcement problem, the responses considered may focus mostly on policing, prosecution, or punishment. A broader public health and community-safety framing may examine factors such as access to firearms, economic opportunity, housing stability, school climate, exposure to trauma, social connection, neighborhood conditions, access to recreation, mental health support, substance use, and relationships with community institutions.

That broader understanding may lead to a wider range of strategies, such as youth employment, mentoring, violence interruption, safe community spaces, behavioral health services, family and peer supports, community-led safety initiatives, firearm injury prevention, and changes to policies or systems that contribute to unequal exposure to violence. Including multiple perspectives can help communities understand both immediate risks and the conditions that shape them.

Developing a theory of practice to address the issue

How can a community effort contribute to meaningful and lasting change? Projects are often less effective when organizations launch disconnected activities or assume that a single strategy will address a complex issue. A stronger approach is to develop a clear plan for understanding the issue, coordinating action, implementing strategies, evaluating progress, adapting the work, and sustaining what is effective.

Whatever you call this plan—a theory of practice, theory of change, logic model, framework, or approach—it should explain how activities are expected to contribute to desired outcomes. It should also reflect community context, existing strengths, available resources, potential barriers, and the roles of different partners.

Once an issue has been identified, one possible theory of practice might include the following:

• Form a coalition of community members, organizations, agencies, and other partners connected to the issue.
• Recruit and prepare a participatory research and evaluation team that includes a range of stakeholder perspectives.
• Gather quantitative and qualitative information about the issue while also identifying community strengths, resources, relationships, and existing efforts that may contribute to solutions.
• Use what is learned to design strategies that reflect the complexity of the issue and the local context.

The result might be a single program or initiative or a coordinated community-wide effort involving several organizations, institutions, media partners, networks, and community members. When multiple partners are involved, coordination, communication, shared decision-making, resource development, and strategies for addressing organizational or systemic barriers should be included in the plan.

• Implement the agreed-upon strategies.
• Conduct monitoring and evaluation that provides ongoing information about implementation, outcomes, equity, and opportunities for improvement.
• Use evaluation findings to adapt and strengthen the work.
• Continue gathering information, evaluating progress, and adjusting strategies as community conditions and priorities change.

Deciding what evaluation questions to ask and how to answer them

Choosing evaluation questions helps determine what the project will pay attention to and what information will be collected. Important effects can be missed if no one asks questions designed to identify them. For this reason, evaluation questions should be selected carefully and developed with input from people who understand the project and the community from different perspectives.

Framing remains important at this stage. Participants should consider the issue in context, examine their assumptions, identify whose perspectives may be missing, and think about the outcomes they hope to see. They should also consider what social, environmental, organizational, economic, or structural conditions may influence those outcomes.

For example, if the goal is only to measure whether incidents of youth violence decrease, evaluation questions might focus primarily on reported incidents or injuries. If the broader goal is to strengthen long-term community safety and well-being, the evaluation may also examine young people's sense of safety and belonging, access to supportive relationships and opportunities, school engagement, exposure to violence, access to behavioral health services, neighborhood conditions, trust in institutions, disparities in outcomes, and other factors connected to community safety.

Broader goals generally require broader evaluation questions. They may also require combining multiple types of evidence and examining not only whether change occurred, but how, for whom, under what conditions, and with what unintended effects.

Collecting information

Collecting information is often one of the most time- and resource-intensive parts of an evaluation.

Depending on the questions being asked, evaluators may use one or more of the following approaches:

• Research using public records, administrative data, news archives, library collections, online sources, organizational records, and other available information
• Individual or group interviews
• Focus groups
• Community information-sharing and dialogue sessions
• Surveys
• Direct or participant observation

Evaluation methods should be adapted to participants' language, literacy preferences, accessibility needs, cultural context, and familiarity with research. Depending on the community and the evaluation questions, maps, drawings, photographs, storytelling, timelines, audio, video, participatory mapping, or other creative approaches may sometimes provide richer information than written questionnaires alone. These methods should be selected collaboratively rather than based on assumptions about what participants can or cannot do.

Analyzing the information you've collected

Once information has been collected, the next step is to make sense of it. What do the numbers suggest? What do people's experiences, observations, and stories reveal? Did the project unfold as planned? If changes were made along the way, what prompted them and what difference did they make?

Some questions may be relatively straightforward. If the project established clear participation or service goals, for example, you may be able to determine whether they were reached. If the goal was to engage 75 people and 82 participated, that tells you something about reach. If 50 participants were expected to complete a program and 61 completed it, that provides another piece of information.

Other findings require deeper interpretation. What if approximately half of participants report that a project was helpful while the other half report that it was not? That result should prompt further questions. Were people's experiences different depending on race, ethnicity, language, disability, geography, income, age, gender, program location, staff relationships, access barriers, or other relevant factors? Did different groups experience different parts of the project? Were the project methods equally accessible and responsive to everyone?

Collecting information often requires substantial effort, but analysis and interpretation are what turn that information into useful knowledge. Strong analysis can help identify ways to improve the project, understand differences in outcomes, recognize unintended effects, demonstrate accountability, and communicate the value and limitations of the work. Participatory analysis also helps ensure that conclusions are informed by people who understand the community and project from different perspectives.

In general, information gathering and analysis should address the three areas discussed earlier: process, implementation, and outcomes. The goal is both to support continuous improvement and to provide accountability to participants, communities, funders, organizational partners, and other stakeholders.

• Process. How well did planning, coordination, communication, participation, staffing, resource development, and decision-making work? Was the planning process meaningfully participatory? Were timelines realistic? Did the team have the resources and support it needed? Were meetings and participation accessible? Did planning and evaluation teams work effectively together? Were responsibilities clear? Did new leaders, partnerships, or opportunities emerge?
• Implementation. Did the project carry out the activities it intended to carry out? Did it reach and engage the people it hoped to reach? Were planned approaches actually used? Were services, activities, or opportunities provided at the intended level? What adaptations were made, and why? Implementation evaluation focuses on what was done and how it was carried out rather than on whether it produced the desired outcomes.
• Outcomes. What changes occurred? Were the expected outcomes observed? Who experienced those outcomes, and were there important differences among groups? What evidence suggests that the project contributed to those changes? What other factors may have influenced the results? Were there unexpected effects, whether beneficial, harmful, or neutral? What can be learned from those results?

Using the information to celebrate what worked and to adjust and improve the project

Accountability is important, but a major purpose of evaluation is also formative: to provide information that helps people strengthen the project as it develops. When reviewing process, implementation, and outcomes, useful questions include: What worked well? What created barriers? Who benefited and who may have been left out? What unintended effects occurred? What should continue? What should change?

Answering these questions may require additional analysis and discussion, but doing so can lead to meaningful improvements. Elements that are ineffective or create unnecessary barriers may need to be changed or discontinued. At the same time, successful strategies should also be examined carefully so they can be strengthened, sustained, and adapted when appropriate.

Remember to recognize progress and contributions. Celebrating accomplishments acknowledges the time, knowledge, effort, and relationships people contributed to the work. It can strengthen shared commitment, reinforce community relationships, and make visible what people have accomplished together.

Participatory research and evaluation are ongoing processes. Once a project has demonstrated positive results, continued learning and adaptation remain important. Community conditions, priorities, resources, and relationships change over time. Evaluation teams may also change, but continued participation from people with different stakeholder perspectives can help keep the work relevant and responsive.

Successful community work rarely means that no further effort is needed. Progress often depends on maintaining effective strategies, strengthening community capacity, monitoring changing conditions, addressing new challenges, and continuing to adapt. Participatory research and evaluation can help communities and organizations understand what is changing and determine how the work should evolve.

In Summary

Participatory evaluation is one component of participatory research. It involves people connected to and affected by a community project in determining what should be evaluated, identifying appropriate questions and methods, collecting and interpreting information, and using what is learned to strengthen the work.

Participatory approaches bring together multiple forms of knowledge, including lived experience, community history, cultural knowledge, professional expertise, organizational experience, and research skills. These perspectives can improve the relevance and usefulness of an evaluation. Meaningful participation can also strengthen relationships, skills, shared decision-making, community leadership, and confidence in collective action.

Conducting a participatory evaluation generally involves:

• Recruiting, preparing, and supporting a diverse stakeholder evaluation team
• Naming and framing the issue, opportunity, or goal
• Developing a theory of practice or logic model to guide the work
• Identifying meaningful evaluation questions and appropriate methods
• Collecting information
• Analyzing and interpreting information collaboratively
• Using what is learned to recognize progress, address challenges, and adjust or improve the work

As with many of the community-building strategies described in the Community Tool Box, the final step is to continue learning and adapting. Participatory research and evaluation should continue for as long as the work can benefit from understanding changing community priorities, conditions, strengths, and outcomes. Ongoing evaluation can help communities and organizations sustain progress, respond to new circumstances, strengthen effective approaches, and remain accountable to the people most affected by the work.

What is participatory evaluation?

___Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work

What are the advantages of using participatory evaluation?

___It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects

___It can get you information you wouldn't get otherwise

___It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff

___It can tell you why something does or doesn't work

___It results in a more effective project

___It empowers stakeholders

___It can provide a voice for those who are often not heard

___It teaches skills that can be used in employment and other areas of life

___It bolsters self-confidence and self-esteem in those who may have little of either

___It demonstrates to people ways in which they can take more control of their lives

___It encourages stakeholder ownership of the project

___It can spark creativity in everyone involved

___It encourages working collaboratively

___It fits into a larger participatory effort

What are the disadvantages of using participatory evaluation?

___It takes more time than conventional process

___It takes the establishment of trust among all participants in the process

___You have to make sure that everyone's involved, not just "leaders" of various groups

___You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills

___You have to get buy-in and commitment from participants

___People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation

___You may have to be creative about how you get, record, and report information

___Funders and policy makers may not understand or believe in participatory evaluation

When would you use participatory evaluation?

___When you're already committed to a participatory process for your project

___When you have the time, or when results are more important than time

___When you can convince funders that it's a good idea

___When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of

___When you need information that it will be difficult for anyone outside the community or population to get

___When part of the goal of the project is to empower participants and help them develop transferable skills

___When you want to bring the community or population together

Who should be involved in participatory evaluation?

___Participants or beneficiaries

___Project line staff and/or volunteers

___Administrators

___Outside evaluators, if they're involved

___Community officials

___Others whose lives are affected by the project

How do you conduct a participatory evaluation?

___Recruit stakeholders as participant evaluators

___Train evaluators

___Name and frame the issue

___Develop a theory of practice to address it

___Determine the evaluation questions

___Collect information

___Analyze the information

___Use your analysis to celebrate what worked and adjust the rest to improve the project

___Stick with it indefinitely

Tool 1: Team Data Collection - How to Do It Well

This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007. We have included her initial e-mail, and then added the summarized reponses to each of her questions.

From: Michelle Garred
Subject: Team data collection: how to do it well?

Dear colleagues,

I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines. Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.

Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well.

Among the many questions are:

• What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
• How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
• How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
• In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
• How would you cite team interviews in a Ph.D. thesis?
• What are the key resources I should review?

What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?

Advantages:

• Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.
• Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.

Pitfalls:

• More personalities and activities to coordinate means more opportunities for things to go wrong.
• Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.
• Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.

Pitfalls can be addressed through training:

• Need for consistent, systematic training of data collectors.
• Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.
• Conduct interviewing practice sessions before beginning actual interviews.
• During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.
• Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.
• Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?
• Demonstrated skills before graduation from training phase

And through regular team meetings:

• Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along.
• Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.
• Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.
• If not together on site, consider on-line conferencing.

How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?

Keep in mind that we will have a team of 5 people conducting semi-structured interviews and we will probably not use audio recording.

• Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.
• Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.
• Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked.  Here's one example of a card storming.
• If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.

How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?

Again, this can be addressed through team training and regular meetings . . .

• Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.
• Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.
• Revisit this issue during regular team meetings during the data collection phase.

And through wise design of the note-taking protocols:

• Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.
• Additionally, you might have team members write up a brief summary within 24 hours of completing the interview.
• Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.

In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?

I understand that team data collection is common in action research, but less common in a Ph.D. thesis. It would help me to hear some different opinions on this issue.

• For quantitative data, one might set a quota of random 10% of data reviewed for QA.
• I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.
• As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
• Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
  • Zuber-Skerritt, Ortrun, and Perry, Chad (2002)  Action research within organisations and university thesis writing. The Learning Organization, 9(4), 171-179.

How would you cite team interviews in a Ph.D. thesis?

• In table format within the document.
• I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
• I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
• When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
• Consider in advance whether you will use the participants' real names or pseudonyms.

Key resources

• See Madeline Church's doctoral thesis, Creating an Uncompromised place to belong - why do I find myself in networks? from the living theory section of Action Research.
• The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews. You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006).  "Finding common ground in team-based qualitative research using the convergent interviewing method." Qualitative Health Research, 16(8), 1145-1157.
  • There's a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006). Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.

• A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about. You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.

  Tool 2: Dabbling in Data

Dabbling in the Data from Public Profit is a practical guide that helps fill this gap. It shares easy-to-use, hands-on ways to explore and analyze data often found in mission-driven work. Whether you are just starting out or want to feel more confident with data, this guide gives you tools and ideas to make data exploration easier and more helpful.

The latest edition, released in late 2024, includes:

• 10 new activities
• More user-friendly tools
• Virtual adaptations
• Playlists for mixed methods analysis

Example 1: True Collaboration for a Win-win in Community-engaged Research

Dr. Marvia Jones, Director of the Kansas City Health Department, spoke on partnerships in community-engaged research at the Life Span Institute's 2024 Collaboratory on January 11th, 2024: 

When we think about the word "partner," we can think about a significant other. If I were to say, "What do you want out of your partnership with your significant other?", you may say things like, "you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year." Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things.

We want someone who is with us, who engages us from the very beginning of a process, so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this?

I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But as a partnership, through our partnership, we have to work through those solutions.

Those are some of the things that I think we look for in a community-based research partnership, having that flexibility and understanding. One of the things that I noticed, as I went from being a researcher to being in practice, is you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?”

What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we do need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information.

So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.