Section 4. Choosing Evaluators

The opening sections of this chapter introduce evaluation: why it matters, how it can be conducted, and whose interests it should address. This section and the next focus on selecting evaluators and developing an evaluation plan. These decisions are closely connected. Begin by clarifying the purpose with staff, community members, and people affected by the initiative, then select evaluators who can help refine the questions and design an appropriate approach.

Why pay attention to the selection of evaluators?

An evaluation should do more than declare that an initiative is doing well or needs improvement. It should examine the work from several perspectives, assess progress toward goals, identify strengths and limitations, and help people decide what to do next. A thoughtful plan needs evaluators with the skills, contextual understanding, and judgment to carry it out and interpret the findings responsibly.

Evaluation can support improvement, learning, and accountability. Useful evidence can help an organization build on strengths, address barriers, respond to changing community priorities, and inform advocacy or funding decisions. More information is not automatically better: the aim is credible information that answers meaningful questions and can guide action.

You may hire external evaluators, draw on staff and community expertise, or combine these approaches. You may also establish a planning team that helps select the evaluators. In each case, the people involved will shape the questions, methods, relationships, and interpretation of findings. Choose them for their ability to support a sound evaluation, rather than their likelihood of producing favorable results.

Organizations have different resources for evaluation. Options include staff-led work, community research teams, independent consultants, university partnerships, and combinations of these. Community members may be experienced evaluators, and professional evaluators may also be local residents. Consider what expertise already exists, what support is needed, and how contributors will be compensated.

A formal planning team is one option, not a requirement for every evaluation. A smaller group may be sufficient for a focused project. Match the scope and process to the available time and resources while maintaining credible methods and respect for participants. A manageable evaluation that answers a useful question can provide a foundation for more extensive work later.

All evaluators bring experiences, interests, assumptions, and relationships that can influence their work. Discuss these openly, whether you are considering consultants, university researchers, staff, students, or community members. Look for people who understand the agreed purpose, select methods that fit the questions, and can report findings honestly, including results the organization or funder may not welcome. Alignment on purpose should support independent judgment rather than require agreement with a preferred conclusion.

When should you choose evaluators?

Involve evaluation expertise early enough to inform planning and collect any baseline information you will need. Waiting until a project ends can limit the questions you can answer. For a new initiative, an initial planning group may help identify priorities and select evaluators. For an existing initiative, begin with its current decisions and available information, then plan opportunities to learn throughout the work.

For example, a new adult literacy program might include evaluation resources in its initial budget and arrange support from a supervised graduate researcher. The researcher could work with learners and staff to define useful questions, observe classes with appropriate permission, review progress information, and conduct interviews during the year. This approach could help the program understand both implementation and change over time. The budget should reflect the work required rather than a fixed percentage assumed to suit every program.

Before finalizing the selection, learn enough about the community, the initiative, and the intended use of findings to identify the expertise required. You do not need to resolve every question first. Evaluators can help assess context and refine the plan, provided community members and staff have meaningful roles in those decisions.

Context of the evaluation

The evaluation's context includes community history, geography, culture, economic conditions, politics, institutional relationships, and available resources. It also includes the initiative's stage of development and people's previous experiences with services or research. Understanding these conditions helps the team choose appropriate methods and avoid judging results without considering how they arose.

An example: Suppose a rural adult literacy program enrolls fewer learners than expected during its first year. Enrollment figures alone might suggest that the program is not meeting community needs. Conversations with residents, however, could reveal that previous programs closed quickly, making people uncertain about whether this one would last. Transportation, scheduling, and outreach may also be barriers. An evaluation that examines these conditions can help the team decide whether to strengthen relationships, adapt delivery, or revise expectations. Later evaluation can test whether those changes improve access and learning.

How should you choose among staff, community, and external evaluators?

Evaluation involves systematically gathering and interpreting information to support judgments and decisions. Relevant expertise can come from formal training, practical evaluation experience, community research, and knowledge of the initiative. The question is which combination of skills and perspectives your evaluation requires.

An external specialist may be useful for a complex design or independent review. Staff and community evaluators may contribute essential knowledge of implementation, relationships, and local priorities. A combined team can bring these strengths together, with clear responsibilities and support for collaboration.

Consider the following factors before deciding who will conduct the evaluation:

Money

Start with a realistic budget for the evaluation's purpose and scope. Include staff time, community participation, training, data collection, analysis, accessibility, and reporting. Ask funders whether evaluation support is available. If resources are limited, consider a smaller evaluation or targeted technical assistance rather than assuming that unpaid labor can cover all the work.

When specialized support is needed, possible sources include local colleges or universities, evaluation consultants, partner organizations, public agencies, and dedicated grants. Students may contribute through supervised coursework, internships, or research partnerships. Agree on the scope, supervision, timeline, data access, and community benefit before starting. Academic requirements should be compatible with the evaluation's purpose, and any additional use of information should be addressed explicitly.

For example, a graduate evaluation course could partner with community organizations on projects that fit a semester. Students and organizations might jointly define questions, develop methods, and produce practical reports, with faculty oversight. A shared learning event could highlight lessons and tools without disclosing sensitive information. Such arrangements work best when expectations are realistic and organizations have a plan for work that continues after the course ends.

Cost is one consideration. Also assess whether the proposed team has the capacity, availability, and working approach needed for the evaluation.

The complexity of the evaluation

Some evaluations address focused descriptive questions: How many residents participated? Did learners improve their reading skills? How did participants experience a recovery-support program? Other questions require more demanding designs: Did the initiative cause the observed change? Which aspects of delivery contributed? How did transportation, housing, or employment conditions influence participation and outcomes? Even a simple-looking question can require careful measurement and attention to missing information. Match the team's methods expertise to the claims you intend to make, and combine that expertise with community knowledge that can help explain the findings.

The type of information needed and how it will be analyzed

Quantitative information may require more than counts or averages. Depending on the questions, the team may need expertise in sampling, comparison designs, measurement, or statistical analysis. Statistical significance does not establish that the program caused a change or indicate the probability that a particular explanation is true. Look for evaluators who can explain the size and practical importance of findings, their uncertainty, and the assumptions behind the analysis. Qualitative information also requires systematic analysis rather than selecting a few memorable stories.

For example, a study of adults returning to education might initially focus on employment changes or access to classes. Systematic review of interviews could reveal that earlier caregiving responsibilities also shaped participants' educational opportunities. Survey information might help describe how common that experience was among respondents, while follow-up conversations could clarify its meaning. Combining methods can uncover overlooked patterns, but software or a statistical association cannot by itself establish what caused people's decisions.

How the evaluation will be used

Will the findings guide program improvement, inform community decisions, support accountability, or help determine future funding? The intended uses affect the questions, level of detail, timeline, and expertise needed. Consider whether participants would feel more comfortable speaking with a community member, a staff evaluator, or someone independent of the initiative. No role guarantees trust or neutrality. Choose arrangements that fit the circumstances and manage possible conflicts of interest.

How the evaluation will establish credibility

Credibility depends on the quality and transparency of the work, not simply on whether evaluators are external or community-based. Community participation can improve relevance and accountability, while independent review can help address particular conflicts or technical questions. Explain who will make decisions, how affected people will contribute, and how findings will be reported. These arrangements should reflect the initiative's mission and principles while protecting the integrity of the evaluation.

What should you look for when choosing evaluators?

Some qualities matter for every evaluator, while others depend on the responsibilities of the team.

Evaluations can use many combinations of expertise. A community researcher may have extensive professional experience, an external evaluator may provide limited technical assistance, and staff may carry out parts of the work with training and supervision. Paid status, credentials, and community membership describe different aspects of a person's role; none alone determines their ability to contribute.

An external evaluator might help a community team design the evaluation, conduct a specialized analysis, or work alongside partners throughout the process. A local team might develop the questions and collect information while a consultant supports interpretation and reporting. Whatever the arrangement, specify responsibilities, decision-making authority, communication, data access, and how knowledge will remain available to the organization and community.

For example, consider an evaluation of community HIV services. A planning group could include people living with HIV, service providers, community advocates, and an evaluator. The group would agree on its purpose and ensure that people using or seeking services have meaningful influence over the questions.

Together, members might decide whether a survey, interviews, service records, or a combination would best address their priorities. They could review questions, choose accessible participation methods, discuss privacy concerns, and agree on how information will be analyzed and shared. The evaluator's role would be to support informed choices and explain tradeoffs.

If the evaluator then conducts the technical analysis, the group can still help interpret findings, identify missing context, and develop recommendations. Participation in interpretation should not require access to identifying information or give any partner authority to suppress unfavorable results.

Characteristics to look for in all evaluators

Commitment to the agreed purpose and honest inquiry

Evaluators may have professional, academic, organizational, or personal interests in the work. A study might contribute to a dissertation, publication, funding decision, or advocacy effort. These interests are not necessarily incompatible with a useful evaluation, but they should be disclosed and discussed. Establish written agreements about scope, methods, deliverables, data use, authorship, and reporting. The agreement should protect the evaluation's purpose and the ability to report credible findings without pressure to reach a predetermined conclusion.

Apply the same expectations to staff, community members, funders, and external evaluators. Ask how they would handle evidence that challenges their views or the initiative's preferred approach. Select people who can listen, consider alternative explanations, and work through disagreement. Different perspectives can strengthen an evaluation when they are addressed openly rather than treated as reasons to exclude someone.

Ability to communicate with a broad range of people

Evaluators may work with people of different ages, languages, identities, beliefs, and experiences. They need to explain the work clearly, listen carefully, and create respectful opportunities for participation. Trust develops through reliable conduct, transparency, and responsiveness; it cannot be assumed from a title or shared background.

Have candidates worked with communities and organizations relevant to your initiative? Can they explain methods without unnecessary jargon and adapt their communication to different audiences? Do they ask about language and accessibility needs? Does the team include relevant language skills or a plan for qualified interpretation and translation?

Explore these questions through interviews, examples of previous work, and conversations with former partners. Ask candidates to explain a method or finding in plain language. The team should be able both to gather useful information respectfully and to present results in forms that community members and decision-makers can understand and use.

Cultural responsiveness and willingness to learn

Evaluators should learn about the community's history, relationships, and priorities without assuming that everyone sharing an identity has the same experiences. Ask how they examine their own assumptions, respond to unfamiliar perspectives, and adapt methods with community input. Knowledge of a particular setting is valuable, but curiosity, humility, and accountability remain important even for evaluators who belong to that community.

Respect for every participant and partner

Look for evaluators who treat people respectfully regardless of their role, education, income, or position in the organization. They should provide meaningful choices about participation, welcome questions and criticism, and respond appropriately to concerns. Their conduct affects both the quality of the evaluation and the organization's relationships with the community.

Commitment to privacy and clear confidentiality protections

Evaluators should explain how they will limit collection of identifying information, protect records, control access, and reduce the risk of identification in reports. Confidentiality concerns how information is protected and shared; anonymity means identities are not collected or linked to responses. Do not promise absolute protection that the evaluation cannot provide. Explain relevant limits in advance, and consider how disclosure could affect participants, especially when the work involves sensitive experiences such as domestic violence.

Informed consent is a communication process, not simply a signed form. Explain what participation involves, how information will be used, foreseeable risks, confidentiality protections and limits, and whom to contact with questions. Provide accessible opportunities to decide freely whether to participate. Determine the applicable review and consent requirements before collecting information; the appropriate documentation depends on the activity and its setting.

Commitment to a useful and credible evaluation

Evaluators should follow through on responsibilities, document important decisions, and communicate emerging issues promptly. The organization and community partners also need to make time for interpretation and action. Agree early on who will use the findings, what decisions they will inform, and how recommendations will be considered. Useful evaluation may confirm strengths, identify changes, or reveal that more information is needed.

Ask evaluators how their work can support future learning. This might include practical data-collection tools, staff and community training, clear documentation, or a schedule for reviewing selected indicators. Ongoing evaluation should fit the initiative's needs and capacity rather than require repeating a large study at every stage.

Community or staff planning and evaluation teams

A planning team helps define the evaluation's purpose, questions, scope, and intended uses. It may also help select evaluators. Its responsibilities can include identifying information sources, choosing methods with technical support, allocating tasks, and agreeing on how findings will inform decisions.

An evaluation team carries out the work, including gathering information, analyzing evidence, interpreting findings, and reporting results. It may include the same people as the planning team or bring in additional expertise. Members may be paid staff, compensated community researchers, volunteers, or external partners. Ensure that the team has appropriate skills and support for the methods it will use.

Assemble a team that can complete the work while including perspectives essential to the evaluation. Organizational position alone should not determine whose contributions carry weight. Team size will depend on the scope, workload, and participation arrangements. A smaller working group can consult more broadly, but affected people should have genuine influence over decisions. Consider the following qualities and arrangements:

Representation and meaningful participation

Depending on the initiative, a planning or evaluation team may include:

• Organizational representatives with different responsibilities, such as program staff, administrators, and governing board members
• People who participate in or are affected by the initiative, including those who have encountered barriers to access
• Community partners with relevant knowledge, such as residents, local businesses, service organizations, schools, and public agencies
• Young people when the initiative affects them, with appropriate support and meaningful roles
• People from relevant language communities and groups underrepresented in previous planning or evaluation efforts
• People with research and evaluation skills, whether developed through community work, employment, formal education, or other experience

Understanding of the evaluation's purpose

Team members should understand what the evaluation is intended to learn and how its questions, methods, and uses connect. Provide orientation and opportunities to revisit the purpose, especially when the team includes people new to evaluation or to the initiative.

Willingness to listen and learn

Every member should recognize both what they contribute and what they need to learn. For example, a school leader can learn from a young person who left school before graduating about barriers that school records do not capture. That young person may also want support with particular evaluation methods. Treat these as complementary contributions, and make room for people to question assumptions regardless of their formal position.

Ability to collaborate, with appropriate support

Familiarity with meetings, technical language, and decision-making processes varies among individuals and should not be inferred from income, occupation, education, or background. Agree on accessible meeting practices, explain unfamiliar terms, and offer several ways to contribute. Provide materials in advance, interpretation or other access support where needed, and fair opportunities to speak. Peer support or shared training can help, but it should respect people's preferences and avoid assigning some members a permanently subordinate role.

External evaluators and specialist teams

When considering a consultant, firm, or university team, assess the people who will actually do the work, including their availability, supervision, and responsibilities. A strong proposal should explain how their expertise fits the questions and how they will collaborate with the organization and community. Review both technical competence and the team's approach to relationships, learning, and accountability.

Knowledge of appropriate evaluation methods

Can candidates explain when quantitative, qualitative, or combined methods would be useful? Do they understand sampling, measurement, analysis, and the limits of different designs? Ask how they would adapt their approach to your questions and resources rather than assuming that their usual method is the best fit.

If you are seeking a university partner, consider departments or centers whose methodological skills and subject knowledge fit the initiative. Relevant contacts may include:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Relevant experience and quality of previous work

Ask about evaluations with similar questions, communities, or constraints. What methods did candidates use, what difficulties arose, and how were findings used? Request appropriate work samples and references, while recognizing that newer evaluators may demonstrate readiness through supervised work and relevant experience.

As when you hire someone for a position, use more than the interview or written proposal to assess fit. Speak with previous clients and community partners about reliability, responsiveness, collaboration, and how the evaluator handled disagreement or unexpected findings.

Review reports or other products that candidates are permitted to share. Are the methods and limitations clear? Are conclusions supported by the evidence? Can intended audiences understand the findings and use them? Ask how community members contributed to interpretation and how the evaluator communicated results to different audiences.

Experience working in partnership with communities

Ask for concrete examples of how candidates shared decisions, responded to community priorities, and addressed differences in power or access. How did they handle a suggestion that changed the design? How did they manage disagreement among partners? Look for a willingness to collaborate while maintaining sound methods and honest reporting.

Ability to listen carefully and check understanding

Evaluators should take others' knowledge seriously and explain where their own expertise can help. Careful listening includes asking follow-up questions, summarizing what they have heard, and checking interpretations rather than assuming they understand someone's meaning. This skill helps the team identify useful questions, gather information respectfully, and make sense of findings in context.

In Summary

The people selected to plan and conduct an evaluation shape its quality, usefulness, and relationships with the community. Choose a combination of skills and perspectives that fits the purpose, methods, resources, and setting. Assess relevant experience, communication, respect for participants, management of conflicts, and willingness to work collaboratively. Clear agreements and meaningful community involvement can help the team produce credible findings and support informed action.