Chapter 36. Introduction to Evaluation | Community Tool Box

This section is adapted from the article "Recommended Framework for Program Evaluation in Public Health Practice," by Bobby Milstein, Scott Wetterhall, and the CDC Evaluation Working Group.

Around the world, there exist many programs and interventions developed to improve conditions in local communities. Communities come together to reduce the level of violence that exists, to work for safe, affordable housing for everyone, or to help more students do well in school, to give just a few examples.

But how do we know whether these programs are working? If they are not effective, and even if they are, how can we improve them to make them better for local communities? And finally, how can an organization make intelligent choices about which promising programs are likely to work best in their community?

Over the past years, there has been a growing trend towards the better use of evaluation to understand and improve practice.The systematic use of evaluation has solved many problems and helped countless community-based organizations do what they do better.

Despite an increased understanding of the need for - and the use of - evaluation, however, a basic agreed-upon framework for program evaluation has been lacking. In 1997, scientists at the United States Centers for Disease Control and Prevention (CDC) recognized the need to develop such a framework. As a result of this, the CDC assembled an Evaluation Working Group comprised of experts in the fields of public health and evaluation. Members were asked to develop a framework that summarizes and organizes the basic elements of program evaluation. This Community Tool Box section describes the framework resulting from the Working Group's efforts.

Before we begin, however, we'd like to offer some definitions of terms that we will use throughout this section.

By evaluation, we mean the systematic investigation of the merit, worth, or significance of an object or effort. Evaluation practice has changed dramatically during the past three decades - new methods and approaches have been developed and it is now used for increasingly diverse projects and audiences.

Throughout this section, the term program is used to describe the object or effort that is being evaluated. It may apply to any action with the goal of improving outcomes for whole communities, for more specific sectors (e.g., schools, work places), or for sub-groups (e.g., youth, people experiencing violence or HIV/AIDS). This definition is meant to be very broad.

Examples of different types of programs include:

• Direct service interventions (e.g., a program that offers free breakfast to improve nutrition for grade school children)
• Community mobilization efforts (e.g., organizing a boycott of California grapes to improve the economic well-being of farm workers)
• Research initiatives (e.g., an effort to find out whether inequities in health outcomes based on race can be reduced)
• Surveillance systems (e.g., whether early detection of school readiness improves educational outcomes)
• Advocacy work (e.g., a campaign to influence the state legislature to pass legislation regarding tobacco control)
• Social marketing campaigns (e.g., a campaign in the Third World encouraging mothers to breast-feed their babies to reduce infant mortality)
• Infrastructure building projects (e.g., a program to build the capacity of state agencies to support community development initiatives)
• Training programs (e.g., a job training program to reduce unemployment in urban neighborhoods)
• Administrative systems (e.g., an incentive program to improve efficiency of health services)

Program evaluation - the type of evaluation discussed in this section - is an essential organizational practice for all types of community health and development work. It is a way to evaluate the specific projects and activities community groups may take part in, rather than to evaluate an entire organization or comprehensive community initiative.

Stakeholders refer to those who care about the program or effort. These may include those presumed to benefit (e.g., children and their parents or guardians), those with particular influence (e.g., elected or appointed officials), and those who might support the effort (i.e., potential allies) or oppose it (i.e., potential opponents). Key questions in thinking about stakeholders are: Who cares? What do they care about?

This section presents a framework that promotes a common understanding of program evaluation. The overall goal is to make it easier for everyone involved in community health and development work to evaluate their efforts.

Why evaluate community health and development programs?

The type of evaluation we talk about in this section can be closely tied to everyday program operations. Our emphasis is on practical, ongoing evaluation that involves program staff, community members, and other stakeholders, not just evaluation experts. This type of evaluation offers many advantages for community health and development professionals.

For example, it complements program management by:

• Helping to clarify program plans
• Improving communication among partners
• Gathering the feedback needed to improve and be accountable for program effectiveness

It's important to remember, too, that evaluation is not a new activity for those of us working to improve our communities. In fact, we assess the merit of our work all the time when we ask questions, consult partners, make assessments based on feedback, and then use those judgments to improve our work. When the stakes are low, this type of informal evaluation might be enough. However, when the stakes are raised - when a good deal of time or money is involved, or when many people may be affected - then it may make sense for your organization to use evaluation procedures that are more formal, visible, and justifiable.

How do you evaluate a specific program?

Before your organization starts with a program evaluation, your group should be very clear about the answers to the following questions:

• What will be evaluated?
• What criteria will be used to judge program performance?
• What standards of performance on the criteria must be reached for the program to be considered successful?
• What evidence will indicate performance on the criteria relative to the standards?
• What conclusions about program performance are justified based on the available evidence?

To clarify the meaning of each, let's look at some of the answers for Drive Smart, a hypothetical program begun to stop drunk driving.

• What will be evaluated?
  • Drive Smart, a program focused on reducing drunk driving through public education and intervention.
• What criteria will be used to judge program performance?
  • The number of community residents who are familiar with the program and its goals
  • The number of people who use "Safe Rides" volunteer taxis to get home
  • The percentage of people who report drinking and driving
  • The reported number of single car night time crashes (This is a common way to try to determine if the number of people who drive drunk is changing)
• What standards of performance on the criteria must be reached for the program to be considered successful?
  • 80% of community residents will know about the program and its goals after the first year of the program
  • The number of people who use the "Safe Rides" taxis will increase by 20% in the first year
  • The percentage of people who report drinking and driving will decrease by 20% in the first year
  • The reported number of single car night time crashes will decrease by 10 % in the program's first two years
• What evidence will indicate performance on the criteria relative to the standards?
  • A random telephone survey will demonstrate community residents' knowledge of the program and changes in reported behavior
  • Logs from "Safe Rides" will tell how many people use their services
  • Information on single-car nighttime crashes will be gathered from police records
• What conclusions about program performance are justified based on the available evidence?
  • Are the changes we have seen in the level of drunk driving due to our efforts, or something else? Or (if no or insufficient change in behavior or outcome,)
  • Should Drive Smart change what it is doing, or have we just not waited long enough to see results?

The following framework provides an organized approach to answer these questions.

A framework for program evaluation

Program evaluation offers a way to understand and improve community health and development practice using methods that are useful, feasible, proper, and accurate. The framework described below is a practical non-prescriptive tool that summarizes in a logical order the important elements of program evaluation.

The framework contains two related dimensions:

• Steps in evaluation practice, and
• Standards for "good" evaluation.

The six connected steps of the framework are actions that should be a part of any evaluation. Although in practice the steps may be encountered out of order, it will usually make sense to follow them in the recommended sequence. That's because earlier steps provide the foundation for subsequent progress. Thus, decisions about how to carry out a given step should not be finalized until prior steps have been thoroughly addressed.

However, these steps are meant to be adaptable, not rigid. Sensitivity to each program's unique context (for example, the program's history and organizational climate) is essential for sound evaluation. They are intended to serve as starting points around which community organizations can tailor an evaluation to best meet their needs.

• Engage stakeholders
• Describe the program
• Focus the evaluation design
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

Understanding and adhering to these basic steps will improve most evaluation efforts.

The second part of the framework is a basic set of standards to assess the quality of evaluation activities. There are 30 specific standards, organized into the following four groups:

• Utility
• Feasibility
• Propriety
• Accuracy

These standards help answer the question, "Will this evaluation be a 'good' evaluation?" They are recommended as the initial criteria by which to judge the quality of the program evaluation efforts.

Engage Stakeholders

Stakeholders are people or organizations that have something to gain or lose from what will be learned from an evaluation, and also in what will be done with that knowledge. Evaluation cannot be done in isolation. Almost everything done in community health and development work involves partnerships - alliances among different organizations, board members, those affected by the problem, and others. Therefore, any serious effort to evaluate a program must consider the different values held by the partners. Stakeholders must be part of the evaluation to ensure that their unique perspectives are understood. When stakeholders are not appropriately involved, evaluation findings are likely to be ignored, criticized, or resisted.

However, if they are part of the process, people are likely to feel a good deal of ownership for the evaluation process and results. They will probably want to develop it, defend it, and make sure that the evaluation really works.

That's why this evaluation cycle begins by engaging stakeholders. Once involved, these people will help to carry out each of the steps that follows.

Three principle groups of stakeholders are important to involve:

• People or organizations involved in program operations may include community members, sponsors, collaborators, coalition partners, funding officials, administrators, managers, and staff.
• People or organizations served or affected by the program may include clients, family members, neighborhood organizations, academic institutions, elected and appointed officials, advocacy groups, and community residents. Individuals who are openly skeptical of or antagonistic toward the program may also be important to involve. Opening an evaluation to opposing perspectives and enlisting the help of potential program opponents can strengthen the evaluation's credibility.

Likewise, individuals or groups who could be adversely or inadvertently affected by changes arising from the evaluation have a right to be engaged. For example, it is important to include those who would be affected if program services were expanded, altered, limited, or ended as a result of the evaluation.

• Primary intended users of the evaluation are the specific individuals who are in a position to decide and/or do something with the results.They shouldn't be confused with primary intended users of the program, although some of them should be involved in this group. In fact, primary intended users should be a subset of all of the stakeholders who have been identified. A successful evaluation will designate primary intended users, such as program staff and funders, early in its development and maintain frequent interaction with them to be sure that the evaluation specifically addresses their values and needs.

The amount and type of stakeholder involvement will be different for each program evaluation. For instance, stakeholders can be directly involved in designing and conducting the evaluation. They can be kept informed about progress of the evaluation through periodic meetings, reports, and other means of communication.

It may be helpful, when working with a group such as this, to develop an explicit process to share power and resolve conflicts. This may help avoid overemphasis of values held by any specific stakeholder.

Describe the Program

A program description is a summary of the intervention being evaluated. It should explain what the program is trying to accomplish and how it tries to bring about those changes. The description will also illustrate the program's core components and elements, its ability to make changes, its stage of development, and how the program fits into the larger organizational and community environment.

How a program is described sets the frame of reference for all future decisions about its evaluation. For example, if a program is described as, "attempting to strengthen enforcement of existing laws that discourage underage drinking," the evaluation might be very different than if it is described as, "a program to reduce drunk driving by teens." Also, the description allows members of the group to compare the program to other similar efforts, and it makes it easier to figure out what parts of the program brought about what effects.

Moreover, different stakeholders may have different ideas about what the program is supposed to achieve and why. For example, a program to reduce teen pregnancy may have some members who believe this means only increasing access to contraceptives, and other members who believe it means only focusing on abstinence.

Evaluations done without agreement on the program definition aren't likely to be very useful. In many cases, the process of working with stakeholders to develop a clear and logical program description will bring benefits long before data are available to measure program effectiveness.

There are several specific aspects that should be included when describing a program.

Statement of need

A statement of need describes the problem, goal, or opportunity that the program addresses; it also begins to imply what the program will do in response. Important features to note regarding a program's need are: the nature of the problem or goal, who is affected, how big it is, and whether (and how) it is changing.

Expectations

Expectations are the program's intended results. They describe what the program has to accomplish to be considered successful. For most programs, the accomplishments exist on a continuum (first, we want to accomplish X... then, we want to do Y...). Therefore, they should be organized by time ranging from specific (and immediate) to broad (and longer-term) consequences. For example, a program's vision, mission, goals, and objectives, all represent varying levels of specificity about a program's expectations.

Activities

Activities are everything the program does to bring about changes. Describing program components and elements permits specific strategies and actions to be listed in logical sequence. This also shows how different program activities, such as education and enforcement, relate to one another. Describing program activities also provides an opportunity to distinguish activities that are the direct responsibility of the program from those that are conducted by related programs or partner organizations. Things outside of the program that may affect its success, such as harsher laws punishing businesses that sell alcohol to minors, can also be noted.

Resources

Resources include the time, talent, equipment, information, money, and other assets available to conduct program activities. Reviewing the resources a program has tells a lot about the amount and intensity of its services. It may also point out situations where there is a mismatch between what the group wants to do and the resources available to carry out these activities. Understanding program costs is a necessity to assess the cost-benefit ratio as part of the evaluation.

Stage of development

A program's stage of development reflects its maturity. All community health and development programs mature and change over time. People who conduct evaluations, as well as those who use their findings, need to consider the dynamic nature of programs. For example, a new program that just received its first grant may differ in many respects from one that has been running for over a decade.

At least three phases of development are commonly recognized: planning, implementation, and effects or outcomes. In the planning stage, program activities are untested and the goal of evaluation is to refine plans as much as possible. In the implementation phase, program activities are being field tested and modified; the goal of evaluation is to see what happens in the "real world" and to improve operations. In the effects stage, enough time has passed for the program's effects to emerge; the goal of evaluation is to identify and understand the program's results, including those that were unintentional.

Context

A description of the program's context considers the important features of the environment in which the program operates. This includes understanding the area's history, geography, politics, and social and economic conditions, and also what other organizations have done. A realistic and responsive evaluation is sensitive to a broad range of potential influences on the program. An understanding of the context lets users interpret findings accurately and assess their generalizability. For example, a program to improve housing in an inner-city neighborhood might have been a tremendous success, but would likely not work in a small town on the other side of the country without significant adaptation.

Logic model

A logic model synthesizes the main program elements into a picture of how the program is supposed to work. It makes explicit the sequence of events that are presumed to bring about change. Often this logic is displayed in a flow-chart, map, or table to portray the sequence of steps leading to program results.

Creating a logic model allows stakeholders to improve and focus program direction. It reveals assumptions about conditions for program effectiveness and provides a frame of reference for one or more evaluations of the program. A detailed logic model can also be a basis for estimating the program's effect on endpoints that are not directly measured. For example, it may be possible to estimate the rate of reduction in disease from a known number of persons experiencing the intervention if there is prior knowledge about its effectiveness.

The breadth and depth of a program description will vary for each program evaluation. And so, many different activities may be part of developing that description. For instance, multiple sources of information could be pulled together to construct a well-rounded description. The accuracy of an existing program description could be confirmed through discussion with stakeholders. Descriptions of what's going on could be checked against direct observation of activities in the field. A narrow program description could be fleshed out by addressing contextual factors (such as staff turnover, inadequate resources, political pressures, or strong community participation) that may affect program performance.

Focus the Evaluation Design

By focusing the evaluation design, we mean doing advance planning about where the evaluation is headed, and what steps it will take to get there. It isn't possible or useful for an evaluation to try to answer all questions for all stakeholders; there must be a focus. A well-focused plan is a safeguard against using time and resources inefficiently.

Depending on what you want to learn, some types of evaluation will be better suited than others. However, once data collection begins, it may be difficult or impossible to change what you are doing, even if it becomes obvious that other methods would work better. A thorough plan anticipates intended uses and creates an evaluation strategy with the greatest chance to be useful, feasible, proper, and accurate.

Among the issues to consider when focusing an evaluation are:

Purpose

Purpose refers to the general intent of the evaluation. A clear purpose serves as the basis for the design, methods, and use of the evaluation. Taking time to articulate an overall purpose will stop your organization from making uninformed decisions about how the evaluation should be conducted and used.

There are at least four general purposes for which a community group might conduct an evaluation:

• To gain insight.This happens, for example, when deciding whether to use a new approach (e.g., would a neighborhood watch program work for our community?) Knowledge from such an evaluation will provide information about its practicality. For a developing program, information from evaluations of similar programs can provide the insight needed to clarify how its activities should be designed.
• To improve how things get done.This is appropriate in the implementation stage when an established program tries to describe what it has done. This information can be used to describe program processes, to improve how the program operates, and to fine-tune the overall strategy. Evaluations done for this purpose include efforts to improve the quality, effectiveness, or efficiency of program activities.
• To determine what the effects of the program are. Evaluations done for this purpose examine the relationship between program activities and observed consequences. For example, are more students finishing high school as a result of the program? Programs most appropriate for this type of evaluation are mature programs that are able to state clearly what happened and who it happened to. Such evaluations should provide evidence about what the program's contribution was to reaching longer-term goals such as a decrease in child abuse or crime in the area. This type of evaluation helps establish the accountability, and thus, the credibility, of a program to funders and to the community.
• To affect those who participate in it. The logic and reflection required of evaluation participants can itself be a catalyst for self-directed change. And so, one of the purposes of evaluating a program is for the process and results to have a positive influence. Such influences may:
  • Empower program participants (for example, being part of an evaluation can increase community members' sense of control over the program);
  • Supplement the program (for example, using a follow-up questionnaire can reinforce the main messages of the program);
  • Promote staff development (for example, by teaching staff how to collect, analyze, and interpret evidence); or
  • Contribute to organizational growth (for example, the evaluation may clarify how the program relates to the organization's mission).

Users

Users are the specific individuals who will receive evaluation findings. They will directly experience the consequences of inevitable trade-offs in the evaluation process. For example, a trade-off might be having a relatively modest evaluation to fit the budget with the outcome that the evaluation results will be less certain than they would be for a full-scale evaluation. Because they will be affected by these tradeoffs, intended users have a right to participate in choosing a focus for the evaluation. An evaluation designed without adequate user involvement in selecting the focus can become a misguided and irrelevant exercise. By contrast, when users are encouraged to clarify intended uses, priority questions, and preferred methods, the evaluation is more likely to focus on things that will inform (and influence) future actions.

Uses

Uses describe what will be done with what is learned from the evaluation. There is a wide range of potential uses for program evaluation. Generally speaking, the uses fall in the same four categories as the purposes listed above: to gain insight, improve how things get done, determine what the effects of the program are, and affect participants. The following list gives examples of uses in each category.

Some specific examples of evaluation uses

• To gain insight:

  • Assess needs and wants of community members
  • Identify barriers to use of the program
  • Learn how to best describe and measure program activities

• To improve how things get done:

  • Refine plans for introducing a new practice
  • Determine the extent to which plans were implemented
  • Improve educational materials
  • Enhance cultural competence
  • Verify that participants' rights are protected
  • Set priorities for staff training
  • Make mid-course adjustments
  • Clarify communication
  • Determine if client satisfaction can be improved
  • Compare costs to benefits
  • Find out which participants benefit most from the program
  • Mobilize community support for the program

• To determine what the effects of the program are:

  • Assess skills development by program participants
  • Compare changes in behavior over time
  • Decide where to allocate new resources
  • Document the level of success in accomplishing objectives
  • Demonstrate that accountability requirements are fulfilled
  • Use information from multiple evaluations to predict the likely effects of similar programs

• To affect participants:

  • Reinforce messages of the program
  • Stimulate dialogue and raise awareness about community issues
  • Broaden consensus among partners about program goals
  • Teach evaluation skills to staff and other stakeholders
  • Gather success stories
  • Support organizational change and improvement

Questions

The evaluation needs to answer specific questions. Drafting questions encourages stakeholders to reveal what they believe the evaluation should answer. That is, what questions are more important to stakeholders? The process of developing evaluation questions further refines the focus of the evaluation.

Methods

The methods available for an evaluation are drawn from behavioral science and social research and development. Three types of methods are commonly recognized. They are experimental, quasi-experimental, and observational or case study designs. Experimental designs use random assignment to compare the effect of an intervention between otherwise equivalent groups (for example, comparing a randomly assigned group of students who took part in an after-school reading program with those who didn't). Quasi-experimental methods make comparisons between groups that aren't equal (e.g. program participants vs. those on a waiting list) or use of comparisons within a group over time, such as in an interrupted time series in which the intervention may be introduced sequentially across different individuals, groups, or contexts. Observational or case study methods use comparisons within a group to describe and explain what happens (e.g., comparative case studies with multiple communities).

No design is necessarily better than another. Evaluation methods should be selected because they provide the appropriate information to answer stakeholders' questions, not because they are familiar, easy, or popular. The choice of methods has implications for what will count as evidence, how that evidence will be gathered, and what kind of claims can be made. Because each method option has its own biases and limitations, evaluations that mix methods are generally more robust.

Over the course of an evaluation, methods may need to be revised or modified. Circumstances that make a particular approach useful can change. For example, the intended use of the evaluation could shift from discovering how to improve the program to helping decide about whether the program should continue or not. Thus, methods may need to be adapted or redesigned to keep the evaluation on track.

Agreements

Agreements summarize the evaluation procedures and clarify everyone's roles and responsibilities. An agreement describes how the evaluation activities will be implemented. Elements of an agreement include statements about the intended purpose, users, uses, and methods, as well as a summary of the deliverables, those responsible, a timeline, and budget.

The formality of the agreement depends upon the relationships that exist between those involved. For example, it may take the form of a legal contract, a detailed protocol, or a simple memorandum of understanding. Regardless of its formality, creating an explicit agreement provides an opportunity to verify the mutual understanding needed for a successful evaluation. It also provides a basis for modifying procedures if that turns out to be necessary.

As you can see, focusing the evaluation design may involve many activities. For instance, both supporters and skeptics of the program could be consulted to ensure that the proposed evaluation questions are politically viable. A menu of potential evaluation uses appropriate for the program's stage of development could be circulated among stakeholders to determine which is most compelling. Interviews could be held with specific intended users to better understand their information needs and timeline for action. Resource requirements could be reduced when users are willing to employ more timely but less precise evaluation methods.

Gather Credible Evidence

Credible evidence is the raw material of a good evaluation. The information learned should be seen by stakeholders as believable, trustworthy, and relevant to answer their questions. This requires thinking broadly about what counts as "evidence." Such decisions are always situational; they depend on the question being posed and the motives for asking it. For some questions, a stakeholder's standard for credibility could demand having the results of a randomized experiment. For another question, a set of well-done, systematic observations such as interactions between an outreach worker and community residents, will have high credibility. The difference depends on what kind of information the stakeholders want and the situation in which it is gathered.

Context matters! In some situations, it may be necessary to consult evaluation specialists. This may be especially true if concern for data quality is especially high. In other circumstances, local people may offer the deepest insights. Regardless of their expertise, however, those involved in an evaluation should strive to collect information that will convey a credible, well-rounded picture of the program and its efforts.

Having credible evidence strengthens the evaluation results as well as the recommendations that follow from them. Although all types of data have limitations, it is possible to improve an evaluation's overall credibility. One way to do this is by using multiple procedures for gathering, analyzing, and interpreting data. Encouraging participation by stakeholders can also enhance perceived credibility. When stakeholders help define questions and gather data, they will be more likely to accept the evaluation's conclusions and to act on its recommendations.

The following features of evidence gathering typically affect how credible it is seen as being:

Indicators

Indicators translate general concepts about the program and its expected effects into specific, measurable parts.

Examples of indicators include:

• The program's capacity to deliver services
• The participation rate
• The level of client satisfaction
• The amount of intervention exposure (how many people were exposed to the program, and for how long they were exposed)
• Changes in participant behavior
• Changes in community conditions or norms
• Changes in the environment (e.g., new programs, policies, or practices)
• Longer-term changes in population health status (e.g., estimated teen pregnancy rate in the county)

Indicators should address the criteria that will be used to judge the program. That is, they reflect the aspects of the program that are most meaningful to monitor. Several indicators are usually needed to track the implementation and effects of a complex program or intervention.

One way to develop multiple indicators is to create a "balanced scorecard," which contains indicators that are carefully selected to complement one another. According to this strategy, program processes and effects are viewed from multiple perspectives using small groups of related indicators. For instance, a balanced scorecard for a single program might include indicators of how the program is being delivered; what participants think of the program; what effects are observed; what goals were attained; and what changes are occurring in the environment around the program.

Another approach to using multiple indicators is based on a program logic model, such as we discussed earlier in the section. A logic model can be used as a template to define a full spectrum of indicators along the pathway that leads from program activities to expected effects. For each step in the model, qualitative and/or quantitative indicators could be developed.

Indicators can be broad-based and don't need to focus only on a program's long -term goals. They can also address intermediary factors that influence program effectiveness, including such intangible factors as service quality, community capacity, or inter -organizational relations. Indicators for these and similar concepts can be created by systematically identifying and then tracking markers of what is said or done when the concept is expressed.

In the course of an evaluation, indicators may need to be modified or new ones adopted. Also, measuring program performance by tracking indicators is only one part of evaluation, and shouldn't be confused as a basis for decision making in itself. There are definite perils to using performance indicators as a substitute for completing the evaluation process and reaching fully justified conclusions. For example, an indicator, such as a rising rate of unemployment, may be falsely assumed to reflect a failing program when it may actually be due to changing environmental conditions that are beyond the program's control.

Sources

Sources of evidence in an evaluation may be people, documents, or observations. More than one source may be used to gather evidence for each indicator. In fact, selecting multiple sources provides an opportunity to include different perspectives about the program and enhances the evaluation's credibility. For instance, an inside perspective may be reflected by internal documents and comments from staff or program managers; whereas clients and those who do not support the program may provide different, but equally relevant perspectives. Mixing these and other perspectives provides a more comprehensive view of the program or intervention.

The criteria used to select sources should be clearly stated so that users and other stakeholders can interpret the evidence accurately and assess if it may be biased. In addition, some sources provide information in narrative form (for example, a person's experience when taking part in the program) and others are numerical (for example, how many people were involved in the program). The integration of qualitative and quantitative information can yield evidence that is more complete and more useful, thus meeting the needs and expectations of a wider range of stakeholders.

Quality

Quality refers to the appropriateness and integrity of information gathered in an evaluation. High quality data are reliable and informative. It is easier to collect if the indicators have been well defined. Other factors that affect quality may include instrument design, data collection procedures, training of those involved in data collection, source selection, coding, data management, and routine error checking. Obtaining quality data will entail tradeoffs (e.g. breadth vs. depth); stakeholders should decide together what is most important to them. Because all data have limitations, the intent of a practical evaluation is to strive for a level of quality that meets the stakeholders' threshold for credibility.

Quantity

Quantity refers to the amount of evidence gathered in an evaluation. It is necessary to estimate in advance the amount of information that will be required and to establish criteria to decide when to stop collecting data - to know when enough is enough. Quantity affects the level of confidence or precision users can have - how sure we are that what we've learned is true. It also partly determines whether the evaluation will be able to detect effects. All evidence collected should have a clear, anticipated use.

Logistics

By logistics, we mean the methods, timing, and physical infrastructure for gathering and handling evidence. People and organizations also have cultural preferences that dictate acceptable ways of asking questions and collecting information, including who would be perceived as an appropriate person to ask the questions. For example, some participants may be unwilling to discuss their behavior with a stranger, whereas others are more at ease with someone they don't know. Therefore, the techniques for gathering evidence in an evaluation must be in keeping with the cultural norms of the community. Data collection procedures should also ensure that confidentiality is protected.

Justify Conclusions

The process of justifying conclusions recognizes that evidence in an evaluation does not necessarily speak for itself. Evidence must be carefully considered from a number of different stakeholders' perspectives to reach conclusions that are well -substantiated and justified. Conclusions become justified when they are linked to the evidence gathered and judged against agreed-upon values set by the stakeholders. Stakeholders must agree that conclusions are justified in order to use the evaluation results with confidence.

The principal elements involved in justifying conclusions based on evidence are:

Standards

Standards reflect the values held by stakeholders about the program. They provide the basis to make program judgments. The use of explicit standards for judgment is fundamental to sound evaluation. In practice, when stakeholders articulate and negotiate their values, these become the standards to judge whether a given program's performance will, for instance, be considered "successful," "adequate," or "unsuccessful."

Analysis and synthesis

Analysis and synthesis are methods to discover and summarize an evaluation's findings. They are designed to detect patterns in evidence, either by isolating important findings (analysis) or by combining different sources of information to reach a larger understanding (synthesis). Mixed method evaluations require the separate analysis of each evidence element, as well as a synthesis of all sources to examine patterns that emerge. Deciphering facts from a given body of evidence involves deciding how to organize, classify, compare, and display information. These decisions are guided by the questions being asked, the types of data available, and especially by input from stakeholders and primary intended users.

Interpretation

Interpretation is the effort to figure out what the findings mean. Uncovering facts about a program's performance isn't enough to make conclusions. The facts must be interpreted to understand their practical significance. For example, saying, "15 % of the people in our area witnessed a violent act last year," may be interpreted differently depending on the situation. For example, if 50% of community members had watched a violent act in the last year when they were surveyed five years ago, the group can suggest that, while still a problem, things are getting better in the community. However, if five years ago only 7% of those surveyed said the same thing, community organizations may see this as a sign that they might want to change what they are doing. In short, interpretations draw on information and perspectives that stakeholders bring to the evaluation. They can be strengthened through active participation or interaction with the data and preliminary explanations of what happened.

Judgements

Judgments are statements about the merit, worth, or significance of the program. They are formed by comparing the findings and their interpretations against one or more selected standards. Because multiple standards can be applied to a given program, stakeholders may reach different or even conflicting judgments. For instance, a program that increases its outreach by 10% from the previous year may be judged positively by program managers, based on standards of improved performance over time. Community members, however, may feel that despite improvements, a minimum threshold of access to services has still not been reached. Their judgment, based on standards of social equity, would therefore be negative. Conflicting claims about a program's quality, value, or importance often indicate that stakeholders are using different standards or values in making judgments. This type of disagreement can be a catalyst to clarify values and to negotiate the appropriate basis (or bases) on which the program should be judged.

Recommendations

Recommendations are actions to consider as a result of the evaluation. Forming recommendations requires information beyond just what is necessary to form judgments. For example, knowing that a program is able to increase the services available to battered women doesn't necessarily translate into a recommendation to continue the effort, particularly when there are competing priorities or other effective alternatives. Thus, recommendations about what to do with a given intervention go beyond judgments about a specific program's effectiveness.

If recommendations aren't supported by enough evidence, or if they aren't in keeping with stakeholders' values, they can really undermine an evaluation's credibility. By contrast, an evaluation can be strengthened by recommendations that anticipate and react to what users will want to know.

Three things might increase the chances that recommendations will be relevant and well-received:

• Sharing draft recommendations
• Soliciting reactions from multiple stakeholders
• Presenting options instead of directive advice

Justifying conclusions in an evaluation is a process that involves different possible steps. For instance, conclusions could be strengthened by searching for alternative explanations from the ones you have chosen, and then showing why they are unsupported by the evidence. When there are different but equally well supported conclusions, each could be presented with a summary of their strengths and weaknesses. Techniques to analyze, synthesize, and interpret findings might be agreed upon before data collection begins.

Ensure Use and Share Lessons Learned

It is naive to assume that lessons learned in an evaluation will necessarily be used in decision making and subsequent action. Deliberate effort on the part of evaluators is needed to ensure that the evaluation findings will be used appropriately. Preparing for their use involves strategic thinking and continued vigilance in looking for opportunities to communicate and influence. Both of these should begin in the earliest stages of the process and continue throughout the evaluation.

The elements of key importance to be sure that the recommendations from an evaluation are used are:

Design

Design refers to how the evaluation's questions, methods, and overall processes are constructed. As discussed in the third step of this framework (focusing the evaluation design), the evaluation should be organized from the start to achieve specific agreed-upon uses. Having a clear purpose that is focused on the use of what is learned helps those who will carry out the evaluation to know who will do what with the findings. Furthermore, the process of creating a clear design will highlight ways that stakeholders, through their many contributions, can improve the evaluation and facilitate the use of the results.

Preparation

Preparation refers to the steps taken to get ready for the future uses of the evaluation findings. The ability to translate new knowledge into appropriate action is a skill that can be strengthened through practice. In fact, building this skill can itself be a useful benefit of the evaluation. It is possible to prepare stakeholders for future use of the results by discussing how potential findings might affect decision making.

For example, primary intended users and other stakeholders could be given a set of hypothetical results and asked what decisions or actions they would make on the basis of this new knowledge. If they indicate that the evidence presented is incomplete or irrelevant and that no action would be taken, then this is an early warning sign that the planned evaluation should be modified. Preparing for use also gives stakeholders more time to explore both positive and negative implications of potential results and to identify different options for program improvement.

Feedback

Feedback is the communication that occurs among everyone involved in the evaluation. Giving and receiving feedback creates an atmosphere of trust among stakeholders; it keeps an evaluation on track by keeping everyone informed about how the evaluation is proceeding. Primary intended users and other stakeholders have a right to comment on evaluation decisions. From a standpoint of ensuring use, stakeholder feedback is a necessary part of every step in the evaluation. Obtaining valuable feedback can be encouraged by holding discussions during each step of the evaluation and routinely sharing interim findings, provisional interpretations, and draft reports.

Follow-up

Follow-up refers to the support that many users need during the evaluation and after they receive evaluation findings. Because of the amount of effort required, reaching justified conclusions in an evaluation can seem like an end in itself. It is not. Active follow-up may be necessary to remind users of the intended uses of what has been learned. Follow-up may also be required to stop lessons learned from becoming lost or ignored in the process of making complex or political decisions. To guard against such oversight, it may be helpful to have someone involved in the evaluation serve as an advocate for the evaluation's findings during the decision -making phase.

Facilitating the use of evaluation findings also carries with it the responsibility to prevent misuse. Evaluation results are always bounded by the context in which the evaluation was conducted. Some stakeholders, however, may be tempted to take results out of context or to use them for different purposes than what they were developed for. For instance, over-generalizing the results from a single case study to make decisions that affect all sites in a national program is an example of misuse of a case study evaluation.

Similarly, program opponents may misuse results by overemphasizing negative findings without giving proper credit for what has worked. Active follow-up can help to prevent these and other forms of misuse by ensuring that evidence is only applied to the questions that were the central focus of the evaluation.

Dissemination

Dissemination is the process of communicating the procedures or the lessons learned from an evaluation to relevant audiences in a timely, unbiased, and consistent fashion. Like other elements of the evaluation, the reporting strategy should be discussed in advance with intended users and other stakeholders. Planning effective communications also requires considering the timing, style, tone, message source, vehicle, and format of information products. Regardless of how communications are constructed, the goal for dissemination is to achieve full disclosure and impartial reporting.

Along with the uses for evaluation findings, there are also uses that flow from the very process of evaluating. These "process uses" should be encouraged. The people who take part in an evaluation can experience profound changes in beliefs and behavior. For instance, an evaluation challenges staff members to act differently in what they are doing, and to question assumptions that connect program activities with intended effects.

Evaluation also prompts staff to clarify their understanding of the goals of the program. This greater clarity, in turn, helps staff members to better function as a team focused on a common end. In short, immersion in the logic, reasoning, and values of evaluation can have very positive effects, such as basing decisions on systematic judgments instead of on unfounded assumptions.

Additional process uses for evaluation include:

• By defining indicators, what really matters to stakeholders becomes clear
• It helps make outcomes matter by changing the reinforcements connected with achieving positive results. For example, a funder might offer "bonus grants" or "outcome dividends" to a program that has shown a significant amount of community change and improvement.

Standards for "good" evaluation

There are standards to assess whether all of the parts of an evaluation are well -designed and working to their greatest potential. The Joint Committee on Educational Evaluation developed "The Program Evaluation Standards" for this purpose. These standards, designed to assess evaluations of educational programs, are also relevant for programs and interventions related to community health and development.

The program evaluation standards make it practical to conduct sound and fair evaluations. They offer well-supported principles to follow when faced with having to make tradeoffs or compromises. Attending to the standards can guard against an imbalanced evaluation, such as one that is accurate and feasible, but isn't very useful or sensitive to the context. Another example of an imbalanced evaluation is one that would be genuinely useful, but is impossible to carry out.

The following standards can be applied while developing an evaluation design and throughout the course of its implementation. Remember, the standards are written as guiding principles, not as rigid rules to be followed in all situations.

The 30 more specific standards are grouped into four categories:

• Utility
• Feasibility
• Propriety
• Accuracy

The utility standards are:

• Stakeholder Identification: People who are involved in (or will be affected by) the evaluation should be identified, so that their needs can be addressed.
• Evaluator Credibility: The people conducting the evaluation should be both trustworthy and competent, so that the evaluation will be generally accepted as credible or believable.
• Information Scope and Selection: Information collected should address pertinent questions about the program, and it should be responsive to the needs and interests of clients and other specified stakeholders.
• Values Identification: The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for judgments about merit and value are clear.
• Report Clarity: Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation. This will help ensure that essential information is provided and easily understood.
• Report Timeliness and Dissemination: Significant midcourse findings and evaluation reports should be shared with intended users so that they can be used in a timely fashion.
• Evaluation Impact: Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the evaluation will be used.

Feasibility Standards

The feasibility standards are to ensure that the evaluation makes sense - that the steps that are planned are both viable and pragmatic.

The feasibility standards are:

• Practical Procedures: The evaluation procedures should be practical, to keep disruption of everyday activities to a minimum while needed information is obtained.
• Political Viability: The evaluation should be planned and conducted with anticipation of the different positions or interests of various groups. This should help in obtaining their cooperation so that possible attempts by these groups to curtail evaluation operations or to misuse the results can be avoided or counteracted.
• Cost Effectiveness: The evaluation should be efficient and produce enough valuable information that the resources used can be justified.

Propriety Standards

The propriety standards ensure that the evaluation is an ethical one, conducted with regard for the rights and interests of those involved. The eight propriety standards follow.

• Service Orientation: Evaluations should be designed to help organizations effectively serve the needs of all of the targeted participants.
• Formal Agreements: The responsibilities in an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that those involved are obligated to follow all conditions of the agreement, or to formally renegotiate it.
• Rights of Human Subjects: Evaluation should be designed and conducted to respect and protect the rights and welfare of human subjects, that is, all participants in the study.
• Human Interactions: Evaluators should respect basic human dignity and worth when working with other people in an evaluation, so that participants don't feel threatened or harmed.
• Complete and Fair Assessment: The evaluation should be complete and fair in its examination, recording both strengths and weaknesses of the program being evaluated. This allows strengths to be built upon and problem areas addressed.
• Disclosure of Findings: The people working on the evaluation should ensure that all of the evaluation findings, along with the limitations of the evaluation, are accessible to everyone affected by the evaluation, and any others with expressed legal rights to receive the results.
• Conflict of Interest: Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.
• Fiscal Responsibility: The evaluator's use of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate.

Accuracy Standards

The accuracy standards ensure that the evaluation findings are considered correct.

There are 12 accuracy standards:

• Program Documentation: The program should be described and documented clearly and accurately, so that what is being evaluated is clearly identified.
• Context Analysis: The context in which the program exists should be thoroughly examined so that likely influences on the program can be identified.
• Described Purposes and Procedures: The purposes and procedures of the evaluation should be monitored and described in enough detail that they can be identified and assessed.
• Defensible Information Sources: The sources of information used in a program evaluation should be described in enough detail that the adequacy of the information can be assessed.
• Valid Information: The information gathering procedures should be chosen or developed and then implemented in such a way that they will assure that the interpretation arrived at is valid.
• Reliable Information: The information gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable.
• Systematic Information: The information from an evaluation should be systematically reviewed and any errors found should be corrected.
• Analysis of Quantitative Information: Quantitative information - data from observations or surveys - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Analysis of Qualitative Information: Qualitative information - descriptive information from interviews and other sources - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Justified Conclusions: The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can understand their worth.
• Impartial Reporting: Reporting procedures should guard against the distortion caused by personal feelings and biases of people involved in the evaluation, so that evaluation reports fairly reflect the evaluation findings.
• Metaevaluation: The evaluation itself should be evaluated against these and other pertinent standards, so that it is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

Applying the framework: Conducting optimal evaluations

There is an ever-increasing agreement on the worth of evaluation; in fact, doing so is often required by funders and other constituents. So, community health and development professionals can no longer question whether or not to evaluate their programs. Instead, the appropriate questions are:

• What is the best way to evaluate?
• What are we learning from the evaluation?
• How will we use what we learn to become more effective?

The framework for program evaluation helps answer these questions by guiding users to select evaluation strategies that are useful, feasible, proper, and accurate.

To use this framework requires quite a bit of skill in program evaluation. In most cases there are multiple stakeholders to consider, the political context may be divisive, steps don't always follow a logical order, and limited resources may make it difficult to take a preferred course of action. An evaluator's challenge is to devise an optimal strategy, given the conditions she is working under. An optimal strategy is one that accomplishes each step in the framework in a way that takes into account the program context and is able to meet or exceed the relevant standards.

This framework also makes it possible to respond to common concerns about program evaluation. For instance, many evaluations are not undertaken because they are seen as being too expensive. The cost of an evaluation, however, is relative; it depends upon the question being asked and the level of certainty desired for the answer. A simple, low-cost evaluation can deliver information valuable for understanding and improvement.

Rather than discounting evaluations as a time-consuming sideline, the framework encourages evaluations that are timed strategically to provide necessary feedback. This makes it possible to make evaluation closely linked with everyday practices.

Another concern centers on the perceived technical demands of designing and conducting an evaluation. However, the practical approach endorsed by this framework focuses on questions that can improve the program.

Finally, the prospect of evaluation troubles many staff members because they perceive evaluation methods as punishing ("They just want to show what we're doing wrong."), exclusionary ("Why aren't we part of it? We're the ones who know what's going on."), and adversarial ("It's us against them.") The framework instead encourages an evaluation approach that is designed to be helpful and engages all interested stakeholders in a process that welcomes their participation.

In Summary

Evaluation is a powerful strategy for distinguishing programs and interventions that make a difference from those that don't. It is a driving force for developing and adapting sound strategies, improving existing programs, and demonstrating the results of investments in time and other resources. It also helps determine if what is being done is worth the cost.

This recommended framework for program evaluation is both a synthesis of existing best practices and a set of standards for further improvement. It supports a practical approach to evaluation based on steps and standards that can be applied in almost any setting. Because the framework is purposefully general, it provides a stable guide to design and conduct a wide range of evaluation efforts in a variety of specific program areas. The framework can be used as a template to create useful evaluation plans to contribute to understanding and improvement. The Magenta Book - Guidance for Evaluation provides additional information on requirements for good evaluation, and some straightforward steps to make a good evaluation of an intervention more feasible, read The Magenta Book - Guidance for Evaluation.

You understand the basic concepts and terms of program evaluation:

___You understand that evaluation is the systematic investigation of the merit, worth, or significance of an object or effort

___You understand that a program is used to describe the object or effort that is being evaluated

___You understand that program evaluation is a way to evaluate the specific projects and activities community groups may take part in

___You understand that stakeholders are those who care about the program or effort

You understand that program evaluation:

___Helps clarify program plans

___Improves communication among partners

___Gathers feedback needed to improve and be accountable for program effectiveness

You know how to evaluate a specific program:

___You know what will be evaluated

___You have decided what criteria will be used to judge program performance

___You have decided what standards of performance must be reached for the program to be considered successful

___You know what evidence will indicate performance

___You know what conclusions about program performance are justified based on the available evidence

___You have engaged stakeholders

___You have described the program

___You have focused the evaluation design

___You understand that possible focuses are: to gain insight, to improve how things get done, to determine what the effects of the program are, and to effect those who participate in it

___You have gathered credible evidence

___You have justified your conclusions

___You understand how to ensure the use of the information and share lessons learned

___You know that there are four categories of evaluation standards: utility, feasibility, propriety, and accuracy

___You understand the standards in each category

___You understand which standards are most applicable to each step of evaluation practice

When writing your evaluation report:

___Provide interim and final reports to intended users in time for intended uses

___Tailor the report content, format, and style for the audience(s) by involving audience members

___Include an executive summary

___Summarize the stakeholders and how they were engaged

___Describe essential features of the program (perhaps in appendices)

___Explain the focus of the evaluation and its limitations

___Include an adequate summary of the evaluation plan and procedures

___Provide all necessary technical information (perhaps in appendices)

___Specify the standards and criteria for evaluative judgments

___Explain the evaluative judgments and how they are supported by the evidence

___List both strengths and weaknesses of the evaluation

___Discuss recommendations for action along with their advantages and disadvantages

___Ensure protections for program clients and other stakeholders

___Anticipate how people or organizations may be affected by the findings

___Present minority opinions or rejoinders where necessary

___Verify that the report is accurate and unbiased

___Organize the report logically and include an appropriate level of detail

___Remove technical jargon

___Use examples, illustrations, graphics, and stories 

Tool 1: Tools Related to the Recommended Framework

See the following Community Tool Box sections for additional information.

 

Engage Stakeholders

To help you understand who stakeholders are:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

Identifying Targets and Agents of Change: Who Can Benefit and Who Can Help

Identifying Community Assets and Resources

To learn how to involve stakeholders:

Involving Key Influentials in the Initiative

Involving People Most Affected by the Problem

Conducting Focus Groups

To work together with a diverse group:

Training for Conflict Resolution

Encouraging Involvement of Potential Opponents as well as Allies

Describe the Program

To fully understand the need, problem, or goal that the program addresses:

Defining and Analyzing the Problem

Analyzing Community Problems

Conducting Needs Assessment Surveys

Conducting Concerns Surveys

Conducting Public Forums and Listening Sessions

To better explain the activities, components, and elements of the program:

Designing Community Interventions

Understanding Risk and Protective Factors: Their Use in Selecting Potential Targets and Promising Strategies for Interventions

Identifying Strategies and Tactics for Reducing Risks

To be able to describe resources and assets for the program:

Identifying Community Assets and Resources

For examples of logic models:

Our Model of Practice: Building Capacity for Community and System Change

Our Evaluation Model: Evaluating Comprehensive Community Initiatives

To look broadly at your program and its context:

An Overview of Strategic Planning or "VMOSA" (Vision, Mission, Objectives, Strategies, Action Plan)

Understanding and Describing the Community

To modify the program to fit the local context:

Adapting Community Interventions for Different Cultures and Communities

To learn to explain the program to others (and so that they see your point of view):

Reframing the Issue

Focus the Evaluation Design

To clarify the purpose:

Developing an Evaluation Plan

To identify potential users and uses:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

To clarify evaluation questions:

Developing an Evaluation Plan

For illustrative evaluation questions:

Our Model of Practice: Building Capacity for Community and System Change

For help identifying specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

For guidance about making agreements:

Identifying Action Steps in Bringing About Community and System Change

Gather Credible Evidence

For support in implementing specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

Justify Conclusions

To see an illustrative process for considering evidence:

Our Model of Practice: Building Capacity for Community and System Change

Ensure use and share lessons learned

To promote the use of what your organization has learned:

Providing Feedback to Improve the Initiative

Conducting a Social Marketing Campaign

Attracting Support for Specific Programs

To share what you have learned with diverse groups:

Making Community Presentations

Communications to Promote Interest

Arranging Celebrations

Tool 2: Evaluation Standards

Use this table to determine how well your evaluation met "good" standards for evaluation.

Standard	Did the evaluation meet this standard? (Yes or No)	Comments:
Utility Standards
1. Stakeholder Identification
2. Evaluator Credibility
3. Information Scope and Selection
4. Values Identification
5. Report Clarity
6. Report Timeliness and Dissemination
7. Evaluation Impact
Feasibility Standards
1. Practical Procedures
2. Political Viability
3. Cost Effectiveness
Propriety Standards
1. Service Orientation
2. Formal Agreements
3. Rights of Human Subjects
4. Human Interactions
5. Complete and Fair Assessment
6. Disclosure of Findings
7. Conflict of Interest
8. Fiscal Responsibility
Accuracy Standards
1. Program Documentation
2. Context Analysis
3. Described Purposes and Procedures
4. Defensible Information Sources
5. Valid Information
6. Reliable Information
7. Systematic Information
8. Analysis of Quantitative Information
9. Analysis of Qualitative Information
10. Justified Conclusions
11. Impartial Reporting
12. Metaevaluation

Tool 3: Steps in Evaluation Practice and the Most Relevant Standards

Codes following each standard designate the conceptual group and number of the standard.  U = Utility; F = Feasibility; P = Propriety; A = Accuracy.

Steps	Most Relevant Standards
Engage Stakeholders	Metaevaluation (A12) Stakeholder identification (U1) Evaluator Credibility (U2) Formal agreements (P2) Rights of human subjects (P3) Human Interactions (P4) Conflict of interest (P7)
Describe the program	Complete and fair assessment (P5) Program documentation (A1) Context analysis (A2)
Focus the evaluation design	Evaluation impact (U7) Practical procedures (F1) Political viability (F2) Cost effectiveness (F3) Service orientation (P1) Complete and fair assessment (P5) Fiscal responsibility (P8) Described purposes and procedures (A3)
Gather credible evidence	Information scope and selection (U3) Defensible information sources (A4) Valid information (A5) Reliable information (A6) Systematic information (A7)
Justify conclusions	Values identification (U4) Analysis of quantitative information (A8) Analysis of qualitative information (A9) Justified conclusions (A10)
Ensure use and share lessons learned	Evaluator credibility (U2) Report clarity (U5) Report timeliness and dissemination (U6) Evaluation impact (U7) Disclosure of findings (P6) Impartial reporting (A11)

 

Della Roberts worked as a nutritionist at the Harperville Hospital. As an African American, she was concerned about obesity among Black children, and about the fact that many of Harperville’s African American neighborhoods didn’t have access to healthy food in stores or restaurants. She felt that the city ought to be doing something to change the situation, but officials didn’t seem to see it as a problem. Della decided to conduct some research to use as a base for advocacy.

Della realized that in order to collect accurate data, she needed to find researchers who would be trusted by people in the neighborhoods she was concerned about. What if she recruited researchers from among the people in those neighborhoods? She contacted two ministers she knew, an African American doctor who practiced in a Black neighborhood, and the director of a community center, as well as using her own family connections. Within two weeks, she had gathered a group of neighborhood residents who were willing to act as researchers. They ranged from high school students to grandparents, and from people who could barely read to others who had taken college courses.

The group met several times at the hospital to work out how they were going to collect information from the community. Della conducted workshops in research methods and in such basic skills as how to record interviews and observations. The group discussed the problem of recording for those who had difficulty writing, and came up with other ways of logging information. They decided they would each interview a given number of residents about their food shopping and eating habits, and that they would also observe people’s buying patterns in neighborhood stores and fast food restaurants. They set a deadline for finishing their data gathering, and went off to learn as much as they could about the food shopping and eating behavior of people in their neighborhoods.

As the data came in, it became clear that people in the neighborhoods would be happy to buy more nutritious food, but it was simply too difficult to get it. They either had to travel long distances on the bus, since many didn’t have cars, or find time after a long work day to drive to another, often unfamiliar, part of the city and spend an evening shopping. Many also had the perception that healthy food was much more expensive, and that they couldn’t afford it.

Ultimately, the data that the group of neighborhood residents had gathered went into a report written by Della and other professionals on the hospital staff. The report helped to convince the city to provide incentives to supermarket chains to locate in neighborhoods where healthy food was hard to find.

The group that Della had recruited had become a community-based participatory research team. Working with Della and others at the hospital, they helped to determine what kind of information would be useful, and then learned how to gather it. Because they were part of the community, they were trusted by residents; because they shared other residents’ experience, they knew what questions to ask and fully understood the answers, as well as what they were seeing when they observed.

This section is about participatory action research: what it is, why it can be effective, who might use it, and how to set up and conduct it.

What is community-based participatory research?

In simplest terms, community-based participatory research (for convenience, we’ll primarily call it CBPR for the rest of this section) enlists those who are most affected by a community issue – typically in collaboration or partnership with others who have research skills – to conduct research on and analyze that issue, with the goal of devising strategies to resolve it. In other words, community-based participatory research adds to or replaces academic and other professional research with research done by community members, so that research results both come from and go directly back to the people who need them most and can make the best use of them.

There are several levels of participatory research. At one end of the spectrum is academic or government research that nonetheless gathers information directly from community members. The community members are those most directly affected by the issue at hand, and they may (or may not) be asked for their opinions about what they need and what they think will help, as well as for specific information. In that circumstance, the community members don’t have any role in choosing what information is sought, in collecting data, or in analyzing the information once it’s collected. (At the same time, this type of participatory research is still a long step from research that is done at second or third hand, where all the information about a group of people is gathered from statistics, census data, and the reports of observers or of human service or health professionals.)

At another level, academic or other researchers recruit or hire members of an affected group – often because they are familiar with and known by the community – to collect data. In this case, the collectors may or may not also help to analyze the information that they have gathered.

A third level of participatory research has academic, government, or other professional researchers recruiting members of an affected group as partners in a research project. The community members work with the researchers as colleagues, participating in the conception and design of the project, data collection, and data analysis. They may participate as well in reporting the results of the project or study. At this level, there is usually – though not always – an assumption that the research group is planning to use its research to take action on an issue that needs to be resolved

The opposite end of the participatory research continuum from the first level described involves community members creating their own research group – although they might seldom think of it as such – to find out about and take action on a community issue that affects them directly.

In this section, we’ll concern ourselves with the latter two types of participatory research – those that involve community members directly in planning and carrying out research, and that lead to some action that can influence the issue studied. This is what is often defined as community-based participatory research. There are certainly scenarios where other types of participatory research are more appropriate, or easier to employ in particular situations, but it’s CBPR that we’ll discuss here.

Employing CBPR for purposes of either evaluation or long-term change can be a good idea for reasons of practicality, personal development, and politics.

On the practical side, community-based participatory research can often get you the best information possible about the issue, for at least reasons including:

• People in an affected population are more liable to be willing to talk and give straight answers to researchers whom they know, or whom they know to be in circumstances similar to their own, than to outsiders with whom they have little in common
• People who have actually experienced the effects of an issue – or an intervention – may have ideas and information about aspects of it that wouldn’t occur to anyone studying it from outside. Thus, action researchers from the community may focus on elements of the issue, or ask questions or follow-ups, that outside researchers wouldn’t, and get crucial information that other researchers might find only by accident, or perhaps not at all
• People who are deeply affected by an issue, or participants in a program, may know intuitively, or more directly, what’s important when they see or hear it. What seems an offhand comment to an outside researcher might reveal its real importance to someone who is part of the same population as person who made the comment.
• Action researchers from the community are on the scene all the time. Their contact both with the issue or intervention and with the population affected by it is constant, and, as a result, they may find information even when they’re not officially engaged in research.
• Findings may receive more community support because community members know that the research was conducted by people in the same circumstances as their own

When you’re conducting an evaluation, these advantages can provide you with a more accurate picture of the intervention or initiative and its effects. When you’re studying a community issue, all these advantages can lead to a true understanding of its nature, its causes, and its effects in the community, and can provide a solid basis for a strategy to resolve it. And that, of course, is the true goal of community research – to identify and resolve an issue or problem, and to improve the quality of life for the community as a whole

In the personal development sphere, CBPR can have profound effects on the development and lives of the community researchers, particularly when those who benefit from an intervention, or who are affected by an issue, are poor or otherwise disadvantaged, lack education or basic skills, and/or feel that the issue is far beyond their influence. By engaging in research, they not only learn new skills, but see themselves in a position of competence, obtain valuable knowledge and information about a subject important to them, and gain the power and the confidence to exercise control over this aspect of their lives.

Two common political results of the CBPR process:

• Through community-based participatory research, citizens can take more control of the direction of their communities
• Community researchers – especially those who are poor or otherwise disadvantaged – come to be viewed differently by professionals and those in positions of power. They have vital information, and the ability to use it, and thus become accepted as contributing members of the community, rather than as voiceless observers or dependents. They have gained a voice, because they understand that they have something to say. Furthermore, the research and other skills and the self-confidence that people acquire in a community-based participatory research process can carry over into other parts of their lives, giving them the ability and the assurance to understand and work to control the forces that affect them. Research skills, discipline, and analytical thinking often translate into job skills, making participatory action researchers more employable. Most important, people who have always seen themselves as bystanders or victims gain the capacity to become activists who can transform their lives and communities.

Community-based participatory research has much in common with the work of the Brazilian political and educational theoretician and activist, Paulo Freire. In Freire’s critical education process, oppressed people are encouraged to look closely at their circumstances, and to understand the nature and causes of their oppressors and oppression. Freire believes that with the right tools – knowledge and critical thinking ability, a concept of their own power, and the motivation to act – they can undo that oppression. Many people see this as the “true” and only reason for supporting action research, but we see many other reasons for doing so, and list some of them both above and below.

Action research is often used to consider social problems – welfare reform or homelessness, for example – but can be turned to any number of areas with positive results.

Some prime examples:

• The environment. It was a community member who first asked the questions and started the probe that uncovered the fact that the Love Canal neighborhood in Niagara Falls, NY, had been contaminated by the dumping of toxic waste.
• Medical/health issues. Action research can be helpful in both undeveloped and developed societies in collecting information about health practices, tracking an epidemic, or mapping the occurrence of a particular condition, to name three of numerous possibilities.
• Political and economic issues. Citizen activists often do their own research to catch corrupt politicians or corporations, trace campaign contributions, etc.

Just as it can be used for different purposes, CBPR can be structured in different ways. The differences have largely to do with who comes up with the idea in the first place, and with who controls, or makes the decisions about, the research. Any of these possibilities might involve a collaboration or partnership, and a community group might well hire or recruit as a volunteer someone with research skills to help guide their work.

Some common scenarios:

• Academic or other researchers devise and construct a study, and employ community people as data collectors and/or analysts.
• A problem or issue is identified by a researcher or other entity (a human service organization, for instance), and community people are recruited to engage in research on it and develop a solution.
• A community based organization or other group gathers community people to define and work on a community issue of their choosing, or to evaluate a community intervention aimed at them or people similar to them.
• A problem is identified by a community member or group, others who are affected and concerned gather around to help, and the resulting group sets out to research and solve the problem on its own.

Why would you use community-based participatory research?

We’ve already alluded to a number of reasons why CBPR could be useful in evaluating a community intervention or initiative or addressing a community issue. We’ll repeat them briefly here, and introduce others as well.

Action research yields better and more nearly complete and accurate information from the community.

• People will speak more freely to peers, especially those they know personally, than to strangers.
• Researchers who are members of the community know the history and relationships surrounding a program or an issue, and can therefore place it in context.
• People experiencing an issue or participating in an intervention know what’s important to them about it – what it disrupts, what parts of their lives it touches, how they have changed as a result, etc. That knowledge helps them to formulate interview questions that get to the heart of what they – as researchers – are trying to learn.

Involving the community in research is more likely to meet community needs.

Action research makes a reasonable resolution or accurate evaluation more probable in two ways. First, by involving the people directly affected by the issue or intervention, it brings to bear the best information available about what’s actually happening. Second, it encourages community buy-in and support for whatever plans or interventions are developed. If people are involved in the planning and implementation of solutions to community issues, they’ll feel they own the process, and work to make it successful. It’s equitable, philosophically consistent for most grassroots and community-based organizations, and practical in that it usually yields the best results

Action research, by involving community members, creates more visibility for the effort in the community.

Researchers are familiar to the community, will talk about what they’re doing (as will their friends and relatives), and will thus spread the word about the effort.

Community members are more likely to accept the legitimacy of the research and buy into its findings if they know it was conducted by people like themselves, perhaps even people they know.

Citizens are more apt to trust both the truthfulness and the motives of their friends and neighbors than those of outsiders.

Action research trains citizen researchers who can turn their skills to other problems as well.

People who discover the power of research to explain conditions in their communities, and to uncover what’s really going on, realize that they can conduct research in other areas than the one covered by their CBPR project. They often become community activists, who work to change the conditions that create difficulty for them and others. Thus, the action research process may benefit the community not only by addressing particular issues, but by – over the long term – creating a core of people dedicated to improving the overall quality of its citizens’ lives.

Involvement in CBPR changes people’s perceptions of themselves and of what they can do.

An action research project can have profound effects on community researchers who are disadvantaged economically, educationally, or in other ways. It can contribute to their personal development, help them develop a voice and a sense of their power to change things, and vastly expand their vision of what’s possible for them and for the community. Such an expanded vision leads to an increased willingness to take action, and to an increase in their control over their lives.

Skills learned in the course of action research carry over into other areas of researchers’ lives.

Both the skills and the confidence gained in a CBPR project can be transferred to employment, education, child-rearing, and other aspects of life, greatly improving people’s prospects and well-being.

A participatory action research process can help to break down racial, ethnic, and class barriers.

 CBPR can remove barriers in two ways. First, action research teams are often diverse, crossing racial, ethnic, and class lines. As people of different backgrounds work together, this encourages tolerance and friendships, and often removes the fear and distrust. In addition, as integral contributors to a research or evaluation effort, community researchers interact with professionals, academics, and community leaders on equal footing. Once again, familiarity breaks down barriers, and allows all groups to see how much the others have to offer. It also allows for people to understand how much they often misjudge others based on preconceptions, and to begin to consider everyone as an individual, rather than as “one of those.”

A member of the Changes Project, a CBPR project that explored the impact of welfare reform on adult literacy and ESOL (English as a Second or Other Language), learners wrote in the final report: “What I learned from working in this project first off is, none of us are so great that change couldn’t help us be better people... I walked into the first meeting thinking I was the greatest thing to hit the pike and found that I, too, had some prejudices that I was not aware of. I thought that no one could ever tell me I wasn’t the perfect person to sit in judgment of others because I never had a negative thought or prejudiced bone in my body. Well, lo and behold, I did, and seeing it through other people’s eyes I found that I, too, had to make some changes in my opinions.

Action research helps people better understand the forces that influence their lives.

Just as Paulo Freire found in his work in Latin America, community researchers, sometimes as a direct result of their research, and sometimes as a side benefit, begin to analyze and understand how larger economic, political, and social forces affect their own lives. This understanding helps them to use and control the effects of those forces, and to gain more control over their own destinies.

Community based action research can move communities toward positive social change.

All of the above rationales described reasons for employing CBPR act to restructure the relationships and the lines of power in a community. They contribute to the mutual respect and understanding among community members and the deep understanding of issues that in turn lead to significant and positive social change.

Who should be involved in community-based participatory research?

The short answer here is people from all sectors of the community, but there are some specific groups that, under most circumstances, are important to include.

• People most affected by the issue or intervention under study. These are the people whose inclusion is most important to a participatory effort – both because it’s their inclusion that makes it participatory, and because of what they bring to it. These folks, as we discussed earlier, are closest to the situation, have better access to the population most concerned, and may have insights others wouldn’t have. In addition, their support is crucial to the planning and implementation of an intervention or initiative. That support is much more likely to be forthcoming if they’ve been involved in research or evaluation.
• Other members of the affected population. People who may not themselves be directly affected by the issue or intervention, but who are trusted by the affected population, can be useful members of a CBPR team.

A businessman from the Portuguese community in a small city was an invaluable member of an action research team examining the need for services in that community. He was quite successful, had graduated from college in the US, and needed no services himself, but his fluency in Portuguese, his credentials as a trusted member of the community, and his understanding of both the culture of the Portuguese residents and the culture of health and human service workers brought a crucial dimension to data gathering, analysis, and general information about the community.

• Decision makers. Involving local officials, legislators, and other decision makers from the very beginning can be crucial, both in securing their support, and in making sure that what they support is in fact what’s needed. If they’re part of the team, and have all the information that it gathers, they become advocates not just for addressing the issue, but for recognizing and implementing the solution or intervention that best meets the actual needs of the population affected.
• Academics with an interest in the issue or intervention in question. Academics who have studied the issue often have important information that can help a CBPR team better understand the data it collects. They usually have research skills as well, and can help to train other team members. At the same time, they can learn a great deal from community-based researchers – about the community and communities in general, about approaching people, about putting assumptions and preconceptions aside – and perhaps, as a result, increase the effectiveness of their own research

It’s important that they be treated, and treat everyone else, as equals. Everyone on a team has to view other members as colleagues, not as superiors or inferiors, or as more or less competent or authoritative. This can be difficult on both sides – i.e. making sure that officials, academics, or other professionals don’t look down on community members, and that community members don’t automatically defer to (or distrust) them. It may take some work to create an environment in which everyone feels equally respected and valued, but it’s worth the effort. Both the quality of the research and the long-term learning by team members will benefit greatly from the effort. (There are some circumstances where actual equality among all team members is not entirely possible. When community members are hired as researchers, for instance, the academic or other researcher who pays the bills has to exercise some control over the process. That doesn’t change the necessity of all team members being viewed as colleagues and treated with respect.)

• Health, human service, and public agency staff and volunteers.  Like the previous two groups, these people have both a lot to offer and – often – a lot to learn that will make them more sensitive and more effective at their jobs in the long run. They may have a perspective on issues in the community that residents lack because of their closeness to the situation. At the same time, they may learn more about the lives of those they work with, and better understand their circumstances and the pressures that shape their lives.
• Community members at large. This category brings us back to the statement at the beginning of this portion of the section that members of all sectors of the community should have the opportunity to be involved. That statement covers the knowledge, skills, and talent that different people bring to the endeavor; the importance of buy-in by all sectors of the community if any long-term change is to be accomplished; and what team members learn and bring back to their families, friends, and neighbors as a result of their involvement.

When should you employ community-based participatory research?

There are times when action research may not be appropriate, and there are times when it’s the best choice. How do you decide?

One criterion is the amount of time you have to do the research on the issue or intervention. Action research may take longer than traditional methods, because of the need for training, and because of the time it often takes for community researchers to adjust to the situation (i.e. to realize that their opinions and intuitions are important, even if they may not always be right, and that their conclusions are legitimate). If your time is limited, CBPR may not be the right option

Another consideration is the type of research that’s necessary. Action research lends itself particularly well to qualitative research. If you’re obligated to deliver complicated, quantitative results to a funder, for instance, you may want to depend on professional researchers or evaluators. Most CBPR isn’t oriented toward producing results couched in terms of statistical procedures. (This isn’t to say that action research teams can’t do quantitative research, but simply that it requires more training, and therefore time, and may require an outside source or an academic team member to crunch the numbers.)

Qualitative research

Relies on information that can’t be expressed in mathematical terms – descriptions, opinions, anecdotes, the comments of those affected by the issue under study, etc. The results of qualitative research are usually expressed as a narrative or set of conclusions, with the analysis backed up by quotes, observation notes, and other non-numerical data.

(Almost anything can be expressed in terms of numbers in some way. Interviewers, for instance, can count the number of references to a particular issue, or even record the number of times that an interviewee squirmed in his chair. Qualitative research, however, relies on elements that can’t be adequately – or, in many cases, at all – described numerically. The number of squirms may say something about how nervous an interviewee is, or it may indicate that he has to go to the bathroom. The interviewer will probably be able to tell the difference, but the numbers won’t.)

Quantitative research

 Depends on numbers – the number of people served by an intervention, for instance, the number that completed the program, the number that achieved some predetermined outcome (lowered blood pressure, employment for a certain period, citizenship), scores on academic or psychological or physical tests, etc. These numbers are usually then processed through one or more statistical operations to tell researchers exactly what they mean. (Some statistics may, for instance, help researchers determine precisely what part of an intervention was responsible for a particular behavior change.)

It may seem that quantitative research is more accurate, but that’s not always the case, especially when the research deals with human beings, who don’t always do what you expect them to. It’s often important to get other information in order to understand exactly what’s going on

Furthermore, sometimes there aren’t any numbers to work with. The Changes Project was looking at the possible effects of a change in the welfare system on adult learners. The project was conducted very early in the change process, in order to try to head off the worst consequences of the new system. There was very little quantitative information available at that point, and most of the project involved collecting information about the personal experiences of learners on welfare.

In other words, neither quantitative nor qualitative methods are necessarily “better,” but sometimes one is better than the other for a specific purpose. Often, a mix of the two will yield the richest and most accurate information.

It’s probably best and most effective to use action research when:

• There’s time to properly train and acclimate community researchers
• The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)
• You need an entry to the community or group from whom the information is being gathered
• You’re concerned with buy-in and support from the community
• Part of the purpose of using CBPR is to have an effect on and empower the community researchers
• Part of the purpose of using CBPR is to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

Once you’ve decided to conduct an action research project, there are a number of steps to take to get it up and running. You have to find and train the participants; determine exactly what information you’re looking for and how to go about finding it; plan and carry out your research; analyze and report on your findings; translate the findings into recommendations; take, or bring about, action based on those recommendations; evaluate the process; and follow up

What follows assumes an ideal action research project with a structure, perhaps one initiated by a health or human service organization. A community group that comes together out of common interest probably would recruit by people already involved pulling in their friends, and probably wouldn’t do any formal training unless they invited a researcher to help them specifically in that way. The nature of your group will help you determine how – or whether – you follow each of the steps below.

Recruit a community research team

How you recruit a team will depend on the purpose of the project as well as on who might be most effective in gaining and analyzing information. A team may already exist, as in the example at the beginning of this section. Or a team may simply be a group that gets together out of common concerns. Many CBPR projects aim for a diverse team, with the idea that a mix of people will both provide the broadest range of benefit and allow for the greatest amount of personal learning for team members. Other projects may specifically draw only from a particular population – a language minority, those served by a certain intervention, those experiencing a particular physical condition.

It often makes sense for at least half the team to be composed of people directly affected by the issue or intervention in question. Those numbers both assure good contact with the population from which information needs to be gathered, and makes it less likely that community researchers will be overwhelmed or intimidated by other (professional) team members or by the task

Recruiting from within an organization or program may be relatively simple, because the pool of potential researchers is somewhat of a captive audience: you know where to find them, and you already have a relationship with them. Recruiting from a more general population, on the other hand, requires attention to some basic rules of communication.

• Use language that your audience can understand, whether that means presenting your message in a language other than English, or presenting it in simple, clear English without any academic or other jargon.
• Use the communication channels that your audience is most likely to pay attention to. An announcement in the church that serves a large proportion of your population, a program newsletter, or word-of-mouth might all be good channels by which to reach a particular population.
• Be culturally sensitive and appropriate. Couch your message in a form that is not only respectful of your audience’s culture, but that also speaks to what is important in that culture.
• Go where your audience is. Meet with groups of people from the population you want to work with, put out information in their neighborhoods or meeting places. Don’t wait for them to come to you.

Given all this, the best recruitment method is still face-to-face contact by someone familiar to the person being recruited.

Orient and train the research team

Orientation and training may be part of the same process, or they might be separate. The two have different purposes. Orientation is meant to give people a chance to ask questions and an overall picture of what is expected.

Orientation might include:

• Introductions all around, and an introductory activity to help team members get to know one another
• Explanation of community-based participatory research, and basic information about this project or evaluation
• Participants’ time commitment and the support available to them, if any. Are child care, transportation, or other support services provided or paid for?
• An opportunity to ask questions, or to discuss any part of the project or evaluation that team members don’t understand or agree with

Especially if the team is diverse, and especially if that diversity is one of education and research experience, an important aspect of the orientation is to start building the team, and to ensure that everyone sees it as a team of colleagues, rather than as one group leading or dominating or – even worse – simply tolerating another. Each person brings different skills and experience to the effort and has something to teach everyone else. Emphasizing that from the beginning may be necessary, not only to keep more educated members from dominating, but also to encourage less educated members not to be afraid to ask questions and give their opinions.

Training is meant to pass on specific information and skills that people will need in order to carry out the work of the research. There are as many models for training as there are teams to be trained. As noted above, orientation might serve as all or part of an introductory training session.   Training can take place all at once – in one or several multi-hour sessions on consecutive days – or over the whole period of the project, with each training piece leading to the activity that it concerns. It might be conducted by one person – who, in turn, could be someone from inside the organization or an outside facilitator – by a series of experts in different areas, or by the team members themselves. (In this last case, team members might, for instance, determine what they need to know, and then decide on and implement an appropriate way to learn it.)

Regardless of how it’s done, here are some general guidelines for training that are usually worth following:

• Find a comfortable space to hold the training
• Provide, or make sure that people bring, food and drink
• Take frequent, short breaks. It’s better for people’s concentration to take a three-minute break every half hour than a 20-minute break every three hours
• Structure the space for maximum participation and interaction - chairs in a circle, room to move around, etc.
• Vary the ways in which material is presented. People learn in a variety of ways – by hearing, by seeing, by discussion, by example (watching others), and by doing.  The more of these methods you can include, the more likely you are to hold people’s attention and engage everyone on the team.
• Use the training to build your team. Training is a golden opportunity for people to get to know and trust one another, and to absorb the guiding principles for the work.

The actual content of the training will, of course, depend on the project you’re undertaking, but general areas should probably include:

• Necessary research skills. These might include interview techniques, Internet searching, constructing a survey, and other basic research and information-gathering methods.
• Important information about the community or the intervention in question.
• Meeting and negotiation skills. Many of the people on your team may not have had the experience of participating in numerous meetings. They need time and support both to develop meeting skills – following discussion, knowing when it’s okay to interrupt, feeling confident enough to express their opinions – and to become comfortable with the meeting process.
• Preparing a report. This doesn’t necessarily mean drafting a formal document. Depending upon the team members, a flow chart, a slide show, a video, or a collage might be informative and powerful ways to convey research results, as might oral testimony or a sound recording.
• Making a presentation. Knowing what to expect, and learning how to make a clear and cogent presentation can make the difference between having your findings and recommendations accepted or rejected.

Determine the questions the research or evaluation is meant to answer

The questions you choose to answer will shape your research. There are many types of answers in either of these cases.

An evaluation can focus on process: What is actually being done, and how does that compare with what the intervention or initiative set out to do? It can focus on outcomes: Is the end result of the intervention what you intended it to be? Or it can try to look at both, and to decide whether the process in fact works to gain the desired outcome. An evaluation may also aim to identify specific elements of the process that have to be changed, or to identify a whole new process to replace one that doesn’t seem to be working

Research on a community issue also may be approached in a number of ways. You may simply be trying to find out whether a certain condition exists in your community, or to what extent it exists. You may be concerned with how, or how much, it affects the community, or what parts of the community it affects. You may be seeking a particular outcome, and the research questions you ask may be designed to help you reach that outcome.

Plan and structure your research activity

Given your time constraints, the capacity of your team, and the questions you’re considering, plan your research.

Your plan should include:

• The kind and amount of information-gathering that best suits your project (e.g., interviews, library research, surveys)
• Who will be responsible for what
• The timeline – i.e., deadlines for completing each phase of the plan
• How and by whom the information will be analyzed
• What the report of the research or evaluation will look like
• When, how, and to whom the report will be presented

Anticipate and prepare contingency plans for problems that might arise

An action research group, like any other, can have internal conflicts, as well as conflicts with external forces. People may disagree, or worse; some people may drop out, or may not do what they promised; people may not understand, or may choose not to follow the procedures you’ve agreed on. There will need to be guidelines to deal with each of these and other potential pitfalls.

Implement your research plan

Now that you've completed your planning, it's time to carry it out.

Prepare and present your report and recommendations

The report, as explained previously, may be a written document, or may be in some alternative form. If it’s an evaluation, it might be presented in one way to the staff of the intervention being evaluated, and in another to funders or the community, depending upon your purposes.

Some possibilities for presentation include:

• A press conference
• A community presentation
• A newspaper or newsletter article
• A written report to funders and/or other interested parties

Take, or try to bring about, appropriate action on the issue or intervention

Action can range from adjusting a single element of an intervention as a result of an evaluation, to writing letters to the editor, advocating with legislators, taking direct action (a demonstration, a lawsuit), and starting a community initiative that grows into a national movement. In most cases, a CBPR effort is meant to lead to some kind of action, even if that action is simply further research.

Follow up

An action research project doesn’t end with the presentation, or even with action. The purpose of the research often has as much to do with the learning of the team members as it does with research results. Even where that’s not the case, the skills and methods that action researchers learn need to be cemented, so they can carry over to other projects.

• Evaluate the research process. This should be a collaborative effort by all team members, and might also include others (those who actually implement an evaluated intervention, for instance). Did things go according to plan? What were the strengths of the process? What were its weaknesses? Was the training understandable and adequate? What other support would have been helpful? What parts of the process should be changed?
• Identify benefits to the community or group that came about (or may come about) as a result of the research process. These may have to do with action, with making the community more aware of particular issues, or with creating more community activists.
• Identify team members’ learning and perceptions of changes in themselves. Some areas to consider are basic and other academic skills; public speaking; meeting skills; self-confidence and self-esteem; ability to influence the world and their own lives; and self-image (seeing themselves as proactive, rather than acted upon, for example).
• Maintain gains by keeping researchers involved. There are a number of ways to keep the momentum of a CBPR team going, including starting another project, if there’s a reason to do so; encouraging team members to be active on other issues they care about (and to suggest some potential areas, and perhaps make introductions that make it easier for them to do so); keeping the group together as a (paid) research consortium; or consulting, as a group, with other organizations interested in conducting action research.

CBPR is not always the right choice for an initiative or evaluation, but it’s always worthy of consideration. If you can employ it in a given situation, the rewards can be great.

Community-based participatory research can serve many purposes. It can supply accurate and appropriate information to guide a community initiative or to evaluate a community intervention. It can secure community buy-in and support for that initiative or intervention. It can enhance participants’ personal development and opportunities. It can empower those who are most affected by conditions or issues in the community to analyze and change them. And, perhaps most important, it can lead to long-term social change that improves the quality of life for everyone.

In Summary

Community-based participatory research is a process conducted by and for the people most affected by the issue or intervention being studied or evaluated. It has multiple purposes, including the empowerment of the participants, the gathering of the best and most accurate information possible, garnering community support for the effort, and social change that leads to the betterment of the community for everyone

As with any participatory process, CBPR can take a great deal of time and effort. The participants are often economically and educationally disadvantaged, lacking basic skills and other resources. Thus, training and support – both technical and personal – are crucial elements in any action research process. With proper preparation, however, participatory action research can yield not only excellent research results, but huge benefits for the community over the long run.

What is community-based participatory research?

___You define community-based participatory research as research conducted by and for those most directly affected by the issue, condition, situation, or intervention being studied or evaluated

Why would you use community-based participatory research?

___It yields better and more nearly complete and accurate information from the community

___Involving the community in research makes a positive outcome that meets community needs more likely

­­___Action research trains citizen researchers who can turn their skills to other problems as well

___Involvement in CBPR changes people’s perceptions of themselves and of what they can do

___Skills learned in the course of action research carry over into other areas of researchers’ lives

___A participatory action research process can help to break down racial, ethnic, and class barriers

___Action research helps people better understand the forces that influence their lives

___Community-based participatory research can move communities toward positive social change

Who should be involved in community-based participatory research?

___People most affected by the issue or intervention under study

___Other members of the affected population

___Decision-makers

___Academics with an interest in the issue or intervention in question

___Health, human service, and public agency staff and volunteers

___Community members at large

When should you employ community-based participatory research?

___There’s time to properly train and acclimate community researchers

___The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)

___You need an entry to the community or group from whom the information is being gathered

___You’re concerned with buy-in and support from the community

___You want to have an effect on and empower the community researchers

___You want to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

___You recruit a community research team

___You orient and train the research team

___The team determines the questions the research or evaluation will try to answer

___The team plans and structures the research activity

___As a team, you carry out your plan

___The team prepares and presents a report and recommendations on its work

___You take, or try to bring about, appropriate action

___You evaluate the process

___You provide an opportunity for team members to reflect on and discuss their learning and the effects of the experience of being involved in a CBPR process

___You maintain gains by keeping team members involved

Example 1: The Changes Project: Holyoke Community College and SABES West

The Changes Project was a three-year participatory action research effort. Funded by a grant from the Office of Educational Research and Improvement, part of the U.S. Department of Education, its goal was to determine the effects on adult learners of the major changes in welfare regulations, immigration laws, and the workplace that took place in the 1990’s.

Background

The project was participatory in two senses:

• The 42 researchers (34 of whom were able to participate for the whole three-year span of the project) were themselves current and former learners and staffers in adult literacy, GED (high school equivalency), and ESOL (English as a Second or Other Language) programs
• All the effort’s data was obtained from 600 learners in the programs from which the researchers came

All the researchers were learners, graduates, or staff members from one of five Western Massachusetts adult education programs.  The Center for New Americans (CNA) in Greenfield, Amherst, and Northampton and the International Language Institute (ILI) of Northampton are independent organizations that work with ESOL learners.  The Mentor Program at Holyoke Community College provides services to adult learners who are making the transition between adult education and higher education.  Read/Write/Now (RWN) Adult and Family Learning Center, sponsored by the Springfield City Library, offers adult and family literacy and GED classes.  The University of Mass. Labor/Management Workplace Education Program (LMWEP) specifically serves University employees with a range of instructional offerings that encompass basic skills, technology, communication, problem-solving, and workplace-specific areas.

Research team members ranged in age from 22 to 55, and in education from adult literacy learners to holders of advanced degrees.  Among them, they spoke six languages; came from six countries on three continents; and claimed Black, Asian, European, and Hispanic heritage.  Twenty-six were parents (ten being single parents), and ten were current or former welfare recipients.  In general, the researchers comprised a group broadly representative of the community of Western Massachusetts adult learning center participants and staff.

Each of the five sites fielded a research team that worked both independently and in conjunction with the other four.  In addition to its other members, ranging in number from 3 to 13, each team included a site research facilitator (SRF) – a program staff member who was both a member and facilitator of the site’s research team.  SRF’s regularly met together with the program coordinator, and also received consultation from a research advisory group and a methodologist.

The research question the project examined was generated by the administrators and staff of 14 western Massachusetts adult learning programs, in response both to the voiced concerns of learners and to their own observations.

Goals

As stated in the Changes Project report, research goals were to learn more about:

• Effects of welfare reform, immigration reform, and the changing requirements of the workplace on adult learners at five adult literacy programs in Western Massachusetts
• Ways in which adult learners respond to and accommodate immigration reform, welfare reform, and the changing requirements of the workplace as they pursue their goals through education
• How practitioners and programs can alter educational services to be more effective, given welfare reform, immigration reform, and the changing requirements of the workplace
• Ways in which policies can be more responsive to the needs of adult learners as they deal with these three issues

The Changes Project began with a challenge: how to mold a large and diverse group of researchers into a team that could work together creatively and effectively.  Changes Project Coordinator Alex Risley Schroeder recalls that this challenge was made even greater by the fact that the project had not been 100% participatory from the beginning.

“We had to write the proposal, and, because of time and other constraints, we did that without the direct participation of learners.  It was something we struggled with throughout the life of the project.  We had teams of learner/researchers, and said to them, ‘Okay, we’ve promised the funders certain things.  Now what can we, as a group, do to make those things reflect what’s important to all of us?’”

There were only two occasions at the beginning of the project when the whole group, all five site teams, got together: an initial orientation/training and a project kick-off.  The real work of team-building actually began when the first set of data came in.

Because of the differing nature of the five adult education programs and their learners, not all teams researched all three areas of concern.  Most of the work on immigration reform, for instance, came out of the research of teams from ESOL programs. The one workplace education program team didn’t research the effects of welfare reform, because all learners in that program were, by definition, employed; it concentrated instead on changes in the workplace.

Data Collection and Information Synthesis

For all teams, information was collected in three phases from learners in their own programs:

• Individual interviews. Teams developed appropriate interview questions for the area they were researching, and conducted a first round of individual interviews.
• More interviews, focus groups, and surveys. Teams conducted a second round of interviews (including some follow-ups – almost all interviews were followed up with second interviews) with revised questions, and also conducted focus groups in each program and administered surveys project-wide.
• More interviews, more focus groups, and three separate surveys, each specifically aimed at learners with firsthand knowledge of one of the three areas of concern.

After each phase of data-gathering, teams struggled with ways to sort out and report what they had found, and with how to make sense of it. There was an enormous amount of material, most of it in the form of people’s stories about their experiences. Each team found its own ways to organize data.

Each phase ended with an Analysis Fest, a coming together of all the teams to try to integrate and analyze what they had found. The meetings were called Analysis Fests, according to Risley Schroeder, to capture both the excitement and the real joy to be found in gaining and making sense of knowledge.  It was in the Analysis Fests, she says, that the five teams really became a coherent unit.

“Not everyone knew the things that were obvious to some of the teams.  What women on welfare went through, for instance, or what it was like to not be able to speak English.  It was a very powerful learning time for all of us, and it gave everyone a chance to get to know others as teams and as individuals.  Really, to make this work, we all had to understand who we were as a large group, as teams, and as individuals.”

The group wrestled with organizing information and stories they had collected. The challenge was compounded by the fact that some researchers had difficulty reading and writing, or with English. It was made yet more difficult because the researchers, at the beginning, thought of themselves as having no expertise, and therefore no tools for the task.

The first Analysis Fest started out print-heavy. There were transcripts of all the interviews, many painstakingly translated into English from Spanish or other languages. The teams were constrained by what they thought “research” was.  In the subsequent Analysis Fests, as researchers relaxed and started to put the material from different teams together, they realized there were themes common to many of the interviews, regardless of the circumstances of the learners who had been interviewed. Those themes became both the core of the project’s ultimate findings and the key to organizing the data.

Those who had the most difficulty with print or with language were also the most creative in getting around that difficulty. They made up representative stories for each of the issues, used pictures and sound as well as – or instead of – print to show what they had found, and developed dramatic interactive role plays to demonstrate the kinds of difficulties learners faced in dealing with welfare and immigration reforms and workplace issues.

One of the most important findings of the project was that adult learners often depended on fragile networks of support of various kinds in order to be able to attend classes and learn successfully.

There were essentially four types of support necessary:

• Practical supports. These include the kinds of supports most people would understand to be necessary for successful completion of an adult education program: food, clothing, shelter, health care, child care, and transportation. But this category also includes information – about welfare recipients’, immigrants’, and workers’ rights, about the laws that govern those areas, about how to find and enroll in programs, about how to study, about how to find other academic and employment resources.
• Inner resources. Hopes, dreams, motivation, focus, determination, natural ability, prior knowledge, spiritual faith.
• Personal supports. These are both emotional and practical, and come not only from family and friends, but from professionals and the community at large. They include advocacy for individuals and programs, as well as respect, encouragement, and belief in people’s ability to change their lives.
• Institutional supports. These include not only the existence, but the responsiveness and appropriateness of programs, organizations, and institutions that provide help and services to those who need them; state and national policies that speak to the real needs of the people at whom they’re aimed; and regulations that accommodate the real needs and circumstances of those they’re meant to help.

The researchers conceived of these supports as threads in a web, all intertwined and interdependent. Cutting any one thread weakens the whole web, and makes an adult learner’s success less likely. The metaphor of the web made the concept of support easier for everyone to see and understand, and also made it clear how important it was that policy makers see that an adult learner’s success is dependent on far more than what goes on in a particular ABE or ESOL program.

Results

Ultimately, the Changes Project accomplished a number of purposes:  

It gathered a large amount of important, first-hand data about the effects – intended and unintended – of welfare and immigration reforms and changes in the workplace on adult learners and on their chances for successful pursuit of their educational goals.

It had a profound effect on the researchers themselves. Both adult learners and staff members found that the experience of participating in the project changed the way they saw themselves and the world.  More specifically:

• Team members increased their ability to communicate and developed job-related skills, and learners specifically increased their literacy and English language skills.
• Team members felt more self-confident and saw themselves as more competent and more able to contribute as a result of their work in the Changes Project.
• Many team members felt they had developed a voice, enabling them to speak up and be heard in both their private and public lives in ways that hadn’t been possible for them previously.
• Their Changes Project experience led many researchers to become active in their communities, and to change their perspective on political action in general and on their own ability to change things in particular.
• Some of the teachers among the team members found that their view of their jobs and of the learners they worked with had changed.  One explained “This project has given me that ability [to be a reflective teacher] because you really have to listen when you’re doing research.  It’s helped me to let go of...being in charge of how the research goes.  Learning to do that has changed me – it’s changed the way I think about how knowledge gets made.  It’s really different when you do it in a group. What you come up with is much richer.”
• The experience of building and belonging to a community that encompassed people of different cultures, ethnicities, and backgrounds stood out as enormously positive for all participants.  Many talked about the excitement of learning about how other cultures viewed the world, and the rewards of learning to get along with people with different perspectives.
• Team members gained valuable technical knowledge about conducting research, including  how to collect and analyze data (as well as how to engage in logical analysis in general), how to adjust research to respond to the culture of those being addressed, and how to develop the interpersonal skills needed to gain information from interviews and focus groups.

The project served as a model, not only for research, but for adult education. Many of the researchers, both teachers and learners, found that their participation had changed their views of learning.  hey realized that it could be fun, practical, and group-centered.  The skills were being learned, perhaps even more effectively than when instruction was focused on them, in the context of work that had a real-world goal.

The research identified peripheral issues that have great consequences for adult learners. The web of support described above is crucial to the success of a majority of learners; yet policy makers are ignorant of it, and often unwittingly unravel it, thereby making it more difficult for learners to succeed.

The Changes Project drafted a series of recommendations for the field, for learners, for workplaces, and for policymakers.

• Recommendations for the field center on how to make programs more effective by accommodating learner needs, including non-academic ones; by strengthening the web of support; by increasing learner participation in the organization’s decision-making; by providing help in transition to next steps – further education, career planning, employment; and by maintaining an organizational atmosphere of mutual respect and support
• The project recommends to learners that they take responsibility for getting all the information they need, for seeking support, and for becoming involved and active in their programs.
• Recommendations to workplaces include their providing support for workers’ occupational and professional development (including career ladders within the workplace, as well as education and training); maintaining an atmosphere of mutual respect and inclusion; recognizing and building on workers’ strengths, particularly those of immigrants; and being sensitive to the cultural diversity of the workforce.
• The Changes Project’s recommendations to policymakers and legislators, although they concern larger issues, are similar in some ways to those for others. They suggest mandating that welfare and immigration caseworkers develop respectful relationships with those they serve; that more effective policies be developed by soliciting the participation of those at whom those policies are aimed; that information and services be provided in whatever languages are necessary; that welfare policies be responsive to the actual needs of recipients; that education and training count toward the welfare work requirement; and that policies encourage fathers to take responsibility for their children.

Participation in the project led to some action on the part of the researchers. This was both perhaps the most promising and the most frustrating part of the effort. One of the greatest pieces of learning for researchers was that action research really does blur the distinctions among research, analysis, and action.  Each of the research teams initiated some action, not so much as a result of, but rather as an integral part of, its research and findings.

• The Changes Project’s research lit a fire under a writing group of parenting teens who had been interviewed by the team, and who wanted to write about their situation.  Their writings were incorporated into a group-composed “Mother’s Day Letter” (entitled “What’s Going to Happen to Us”) that was sent to legislators, the media, and other adult education programs.  As a result. the local paper published a story on the effects of welfare reform on teen parents, and included interviews with several of the writers.
• The CNA team made presentations of their recommendations to the staff and administrators of the program, and also participated in rallies and direct advocacy with legislators for immigrants’ rights at the State House.
• The Read/Write/Now team initiated a letter-writing campaign to their local, state, and national representatives, detailing the real effects of welfare reform on adult-learner welfare recipients.  They also participated in advocacy for economic justice, and, ultimately, self-published a widely-distributed (to legislators, other programs, and the public) book telling the personal stories of many of those learners.  They hope to dispel stereotypes of welfare recipients, and to help people understand the true bind that many adult learners find themselves in.
• The ILI team’s action was centered closer to home.  Struck by the stories of those they interviewed, team members undertook to provide information and assistance to fellow learners in the program.  As did the others, they also engaged in advocacy for immigrants’ rights by attending State House rallies and other events, and by making presentations about their work.

Finally, the UMass Workplace Education team responded to what they learned in two ways:

• Team members created a support and problem-solving group, called Making Work Work, in order to explore and affect workplace issues.This group has met with a group from the University’s Training and Development Unit to design new programs and involve more staff in planning.
• The second important action to grow out of LMWEP’s research is the development of a career ladder for University workers, a result of meetings between the union and University administration initiated as a result of the Changes Project.

While all of this action has helped those involved feel that they now have more of a voice in their own destinies, it is frustrating as well. Except for the LMWEP progress in the workplace, most of the actions taken had little or no effect on the situations they addressed.  Welfare and immigration reform have gone through as originally planned, with virtually no concessions to the needs of people who are being deprived of the education and training that could help them to become financially successful and better citizens.  Legislators and others have listened sympathetically, but the results have shown little sympathy to the poor and/or newly arrived.

All this can be taken as an excuse to stop trying to bring about social change, but it can also be seen as the beginning of something. A number of people who previously thought of themselves as powerless have seen that they can at least struggle for what they believe in or what they need. They have learned, and have passed on, ways of finding and examining information that give them more choices. And they seem to be willing to continue to practice what they’ve learned. It is in circumstances like those created by the Changes Project that the seeds of change take root. The name seems more than fitting.

Example 2: Community implements participatory action research for proposed business deal

When an out-of-state hazardous waste recycling company approached an economically-depressed New England mill town with an offer to build a plant in the community, the town leaders were overjoyed. The plant would mean a boost to the tax base, and the company promised that both the construction contracts and most jobs at the plant – jobs they described as well-paid – would go to local residents.  Without much discussion, the town tentatively agreed to grant permits, and to waive environmental regulations where necessary.

Some citizens were worried, however. The plant was scheduled to be built on land that drained into the aquifer that fed one of the town’s two wells. Although the company claimed that its process was foolproof and waste-free – all the waste they collected was to be recycled into other products, and there would be no chance of any pollution from their operation – it still seemed ill-considered to grant permission for something so dangerous-sounding without learning more about it.

The group of concerned citizens came from diverse backgrounds. One of the leaders was a teacher; another was a single mother on welfare. The group’s membership spanned the social structure of the town. Arrayed against it were town officials, concerned with growth in a period of economic stagnation, and many of the unemployed and underemployed citizens who hoped that the plant would mean jobs for them, and a better life for their families.

The town split down the middle, with officials refusing to budge, and neighbors and long-time friends ceasing to speak. The members of the concerned group realized they had to understand as much as they could about the company and its process, and set out to do research on both. They learned about hazardous waste recycling, and about both the waste the plant would accept, and the products it intended to turn that waste into.  They sought information on the company, discovering that it had operating plants in the Midwest.

Group members called officials in the Midwestern communities where the plants were located, and learned that there had, in fact, been some environmental problems. They also found out that the plant’s proposed methods were still largely untested, and that there were many scientists who believed that there were serious flaws in them. Armed with this information, the group notified the press and called a public meeting.

After a chaotic, but ultimately productive, evening, the group convinced enough people of potential problems that elected officials were pressured into appointing a commission to study the issue of the hazardous waste recycling plant.  Several members of the concerned citizens’ group were appointed to the commission, along with several other community members who favored the plant.

Over the course of several weeks, the commission met with company officials, and continued to do research on the science of hazardous waste recycling, receiving help from chemists and nuclear physicists at the nearby state university. Many of the commission members who had initially favored the plant became suspicious of the company’s motives, and questioned its truthfulness. Finally, the commission decided to send two of its members to one of the company’s other plants.

The commission delegation found a community that felt it had been lied to. There had been some serious leakage problems associated with the plant, and the company had not responded well.  Rather than admitting the problem and trying to fix it, company officials had tried to pretend there was nothing wrong, and accused the town of being alarmist.  When they were finally forced by a regulatory agency to clean up their mess, they had difficulty doing so, and many in town felt they still hadn’t succeeded.

Furthermore, the company had promised the community good jobs for local people, but the jobs hadn’t materialized. The well-paying jobs at the plant required graduate degrees in chemistry and other sciences, and the low-paying jobs local workers were offered were often part-time, and at wages no better than those available at Wal-Mart.

The New England town commission had heard enough. They returned and convinced the town to vote against the permits the plant needed.  Before that could happen, however, the company withdrew its offer, and looked for a site elsewhere in the country. Some people in town were still angry about what they saw as a loss of jobs, but most felt they had dodged a bullet.

A citizens’ group had turned itself into a participatory action research project, and had used the results of its research to take effective action on an issue of public importance.  Many of those involved in the effort are still active in the community. Others felt their moment of “fame” came at too great a cost, and have withdrawn from public participation.  In a small town, that is one of the risks of becoming involved in action research.

Example 3: True Collaboration for a Win-win in Community-engaged Research

“When we think about the word ‘partner,’ we can think about a significant other. If I were to say, ‘what do you want out of your partnership with your significant other?,’ you may say things like, ‘you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year.’ Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things. We want someone who is with us, who engages us from the very beginning of a process--so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this? I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But, as a partnership, we have to work through those solutions. Those are some of the things that we look for in a community-based research partnership, that flexibility and understanding. One of the things I noticed as I went from being a researcher to being in practice is that you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?” What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information. So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.”

Example 4: Compassion Fatigue in Evaluators

Evaluators working with trauma-exposed populations often develop compassion fatigue in the same way caregivers might. Compassion fatigue is the psychological strain one experiences due to sustained empathetic engagement. Dr. Stacy Huff's research analyzes and provides suggestions for managing evaluator compassion fatigue.

Compassion fatigue affects everyone differently. Many factors play a role in an evaluator's likelihood of developing compassion fatigue and their ability to overcome it. Things like the type of evaluation, personal identity, and organizational support all have an effect.

• Type of evaluation: Evaluators working with refugees, Indigenous populations, and those facing systemic oppression are more likely to experience compassion fatigue. They might also face ethical tension due to their proximity to trauma-exposed communities.
• Personal identity: Historically marginalized individuals, such as people of color and women, may face high mental strain from evaluation work. Evaluators who work with groups with whom they have a personal connection are more likely to experience compassion fatigue when hearing stories similar to their lived experiences.
• Organizational support: Many evaluators do not have a support system in place and are thus left to manage this large emotional weight by themselves.

Recommended resources for evaluators to recover from fatigue.

• Understanding your struggle: The Compassion Fatigue Awareness Project provides resources for understanding and preventing compassion fatigue within the workplace.
• Meditation: Huff specifically recommends The Guided Meditation for Compassion Fatigue to manage compassion fatigue.
• Overcoming burnout: The Reducing Compassion Fatigue, Secondary Traumatic Stress, and Burnout workbook is a great place to start!

To read Dr. Huff's blog post: https://aea365.org/blog/drg-tig-week-compassion-fatigue-in-evaluation-t…

Tool: Involving Community Members in Evaluation

When community members are involved in evaluation, all aspects of the project, from identification of the question to dissemination of results, are more likely to reflect the priorities, values, and experiences of the community. In any evaluation or research project, community involvement can be considered along a continuum, where community members have increasing levels of influence and responsibility. This planning framework, which can be used by organizations responsible for conducting or funding evaluation projects, can increase transparency, establish shared expectations, serve as a starting point to further clarify roles, and increase accountability. Read more from the Amherst H Wilder Foundation website.

Download the PDF for the tool here.

 

Tool: Data Walks and Narrative Methods

A Data Walk is a participatory process that engages stakeholders, including community members, in interpreting visual data such as maps, charts, photos, and statistics. Stakeholders should be engaged in the design of Data Walks to select which data to display and identify appropriate spaces to host the walk. This collaborative approach increases ownership, strengthens interpretation, and supports sustainability by positioning community members as co-creators of insight rather than passive recipients of information. 

Evaluators can consider using the Theory, Context, Characteristics, and Methodology (TCCM) framework to organize and reflect on what emerges from the walk. Using a structured reflection process ensures that community narratives are not only heard but thoughtfully integrated into context-sensitive evaluation and decision-making. Read more from the American Evaluation Association website. 

• Who are community leaders, evaluators and funders?

• Why should you understand the interests of these groups?

• When should you understand the interests of these groups?

• What are the interests of these groups?

• How do you determine interests?

You want to know more about how your group is doing, but others you work with want to know whether you are making a difference. Welcome to the world of evaluation. If you are a community initiative, you will want to evaluate your effort. You will need to devote some time and energy to planning the evaluation process. Like many other aspects of community health and development, an evaluation will ultimately be more beneficial if you spend the time and energy searching for ways to successfully begin and complete an evaluation.

One step in the planning process includes understanding and recognizing the interests of stakeholders in the evaluation. The stakeholders include community leaders, evaluators, and funders, and you will want to know how the evaluation will be used by each of them.

The evaluation should respond to the interests of those three stakeholders, and nothing is more productive than designing it together. The evaluation can serve the community leaders' interests, the funders' interests, and the evaluators' interests in a single useful product, if you know what they want before you start. It's important to define the stakeholders interests in using the evaluation so that it can focus on optimally answering questions important to all of them.

What do we mean by needs and interests? Needs and interests are those qualities which community leadership, evaluators, and funders see as important for doing their jobs well. Because each of these stakeholders is looking at the evaluation from a unique perspective, it helps to recognize those differences, and incorporate them into the evaluation.

For starters, let's consider why you'd want to conduct an evaluation in the first place.

There are many basic reasons why stakeholders want an evaluation:

• To be accountable as a public operation
• To assist those who are receiving grants to improve
• To improve a foundation's grantmaking
• To assess the quality or impact of funded programs
• To plan and implement new programs
• To disseminate innovative programs
• To increase knowledge

A stakeholder may want an evaluation for one, two or all of these reasons. Evaluators may want to increase knowledge, funders may want to improve grantmaking and community leaders may want to assess quality. Community leaders may not want to answer more than a phone interview by a student intern, evaluators may be interested in systematic, disciplined inquiry, and funders may look for accountability.

When it comes time for evaluation, you don't have to be specialists in order to make good decisions about what you will do. You should, however, be knowledgeable about uses of evaluations and how they match the many interests involved so that you can make informed choices.

Who are community leaders, evaluators and funders?

Community leaders

May include staff, administrators, committee chairpersons, agency personnel and civic leaders, and trustees of an initiative. They may have little knowledge of evaluation, nor feel they have much time to provide data or read data reports. Yet, the evaluation must be responsive, useful and sensitive to their decision-making requirements. They often are interested in how to improve the functioning of their initiative.

Evaluators

Are often professionals, though anyone can design and implement an evaluation. There are several professional associations that support evaluators and have established standards of practice, such as the American Evaluation Association or its Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. Evaluators can be private consultants, university or foundation staff, or a member of the initiative. Evaluators are often interested in the systematic production of useful, reliable information.

Funders

Are those individuals or organizations who provide financial support for the initiative. They might include program officers or other representatives of government agencies, foundations, or other sources of financial support. Some funders have built a formal evaluation into their regular activities, but they are in the minority. Funders are often interested in whether the use of their funds is having an impact on the problems facing communities.

Why should you understand the interests of these groups?

So, you understand the idea in principle, but why do you need to understand the needs of leaders, evaluators and funders? The information needs of various groups can be very different, so it's important to take into account the kinds of information that will be convincing and useful to the target audiences. Knowing this will help you decide what information is needed and the tools you could use to obtain it.

While you may know your group does good work, chances are good that other important members of the community do not know what you do. Consequently, others who have supported and encouraged your efforts will want to know what has worked, and what hasn't worked; and what should change and what should stay the same. Because these groups or individuals might be instrumental in assisting your work, financially or otherwise, it makes good sense to include their needs in the evaluation process.

Even more important is the requirement that the information used is effectivel. The question is: To whom is it useful? If there's no direction to your information gathering, you can collect just about anything you want, but so what? If it doesn't matter to anyone else, it is meaningless. If I collect information about the number of people that my agency serves, I may find that useful, especially if I'm reimbursed for that number. But what if someone really wanted to know if the efforts of the agencies in town had an impact on a health problem. The number of people my agency serves might not be that useful.

The interests, which helps us determine the information we need, lead us to develop tools to collect it. In other words it is the interests of the stakeholders that shape the inquiring.

• What are the interests of the stakeholders?
• What information will help them?
• What questions will you ask to get that information?
• What tools will help you collect it?

In the long run, including the stakeholders in the process will lead to greater collaboration and organizational capacity to solve community problems. Understanding stakeholders' interests will enable you to employ your resources better. Knowing what everyone wants and needs will help you plan the optimal evaluation.

When should you understand the interests of these groups?

You will want to identify stakeholders from the get-go. By going through a process of stakeholder identification before you begin evaluating, you will be able to obtain their views and incorporate their ideas and needs into the evaluation itself.

Of course, the sooner you identify the needs and interests of those groups, the sooner you will be able to gain understanding of the different issues each group is interested in without wasting time or money. You also have to be watchful so that if interests change, you can adapt to those changes in a timely manner and keep your evaluation valid.

What are the interests of these groups?

How do we find these people?

First and foremost, you and other members of the group will need to sit down, pour a cup of joe, and grab a pencil. Think about the individuals and groups that have needs that should be addressed in the evaluation. You should try to figure out what their interests are.

Of course, some people may ask, "Why them? Our group's interests and needs should be the focus of the evaluation." In one sense, this is true. One of the main purposes of the evaluation process includes providing feedback and ideas for the group itself so members can improve and strengthen their efforts. But remember, everyone is in this together, the community funders and those who will conduct the evaluation. You want the best information possible information that will help you make the best decisions.

But, at the same time, there are other factors to consider.

Over the course of your brainstorming session, you should identify as many stakeholders as you can.

To identify stakeholders, ask yourself these questions:

• Who provides funding for our initiative?
• Who will conduct the evaluation?
• Who do we collaborate with?

Once you know who these people are, find out what they want. To do this, let's take a look at the groups. Then, we'll talk about the specific needs and interests of members of each of these groups.

Community leaders

What will this group need from your evaluation? The information should be:

• Clear and understandable: They may have limited knowledge about the goings-on of your group, or about evaluations. Immediately, then, you know that the evaluation must be clear and understandable.
• Efficient: They probably have a variety of different responsibilities which demand their time and consideration, so they won't want to waste time reading information irrelevant to their needs.
• Responsive: They may include decision-makers that can affect the future of your group. Therefore, your evaluation needs to be responsive to their decision-making requirements.
• Sensitive: They will want to know what the initiative has accomplished, so the evaluation should be sensitive to the activities and accomplishments of the initiative.
• Useful: They will include decision-makers for the initiative, so the evaluation needs to show them how their efforts can be improved.

Evaluators

They will be assessing the effectiveness of the initiative in meeting its goals. What do they need to get out of the evaluation?

• Input: The evaluation team needs to receive input from the initiative's clients --including community leadership, funders, and members of the initiative itself, in order to know what the clients want to learn about the initiative.
• Accurate and complete information: In a similar vein, the evaluators need accurate and complete information in order to answer the questions posed by the stakeholders.
• Cooperation: Finally, the evaluators will need cooperation from participants and officials in order to obtain needed data.

Funders

They will need:

• Clear and timely reports: Because of their responsibilities for making decisions concerning the continuation of financial support, the funders will need information about the progress of the initiative.
• Evidence of community change and impact: Funders will need to be able to measure the success of the initiative and report this to their own trustees or constituents.

How do you determine interests?

Now you know what interests you're looking for, you have to determine a way of finding them. You need to match people with what their interests are, whether it's through a survey, an interview, or some other method. Failure to determine interests is often the source of problems and misunderstandings along the way and can became disastrous if it turns out that different stakeholders had different expectations and priorities.

Some of the questions that you can ask stakeholders to match them with their interests in an evaluation are:

• What are the evaluation's strengths and weaknesses?
• Do you think the evaluation is moving toward its desired outcomes?
• Which kind of implementation problems came up, and how are they being addressed?
• How are staff and clients interacting?
• What is happening that wasn't expected?
• What do you like, dislike, or would like to change in the evaluation process?

From the answers you get, you can determine what each party wants out of the evaluation. You can also group those who have similar interests. For instance, you may find out that a community leader and an evaluator are interested in improving their managing abilities through the evaluation process. You have made a match, and those two will work toward a common goal.

Here are some ways of determining interests and matching them with stakeholders:

• Interviews: Get a representative from each group of stakeholders and ask away. Be direct in your questions so that you can quickly get to the point you're trying to make, that is, what interests this stakeholder has and if they match with some other stakeholders' interests.
• Surveys: You can send out a written questionnaire to assess how the stakeholders rank their interests and which group wants what. The survey must be succinct and direct, asking clear questions about the evaluation in terms of quality and goals. Survey results are easy to utilize and can be helpful for the evaluation presentation.
• Phone surveys: They can save you time and money, if you're doing them locally. You can use the same questions you would use in a written survey, but leave more space for commentary, as people tend to talk more when speaking to a person on the phone. Just be sure your phone surveys don't stray from your objective.
• Brainstorm sessions: Arrange a meeting with stakeholder representatives and brainstorm interests and possibilities for the evaluation's outcome. Bring up problems such as continuity of the program, obtaining funds, coordinating activities, and attracting staff, and let stakeholders have their say. Everybody will come out from the brainstorming session with new ideas and a much better notion of everybody else's ideas.

Besides these methods, you should always conduct a survey after the completion of the evaluation. This will benefit external audiences and decision-makers. Remember, if changes need to be made, don't be afraid to follow through with them.

Remember that decisions about how to improve a program tend to be made in small, incremental steps based on specific findings aimed at making the evaluation a better process for all stakeholders involved.

Now armed with this list of needs and interests, you can find or develop the tools to obtain useful information. The next sections will explore ways to select an evaluation team and present some key questions for the evaluation process. Later, we will be discussing how to evaluate your community initiative!

In Summary

Once you have a clear idea of what each stakeholder really wants, you are very likely to succeed in your evaluation. Be sure to revise frequently the interests of all the stakeholders involved so that you don't lose focus of what you're looking for with your evaluation. The hard part of your evaluation work starts now!

___You understand why it is important to determine the interests of the stakeholders

___You know who the stakeholders are

___You understand the community leaders' interests

___ You understand the funders' interests

___ You understand the evaluators' interests

___You know when you should understand those interests

___You know how to identify stakeholders

___You know the needs of each stakeholder

___You know how to determine interests

___You understand the interests involved in an evaluation

Example 1: Reasons to evaluate

Different organizations will have different reasons to implement an evaluation, and different interests will be at stake. For instance, some organization may want to focus on increasing the quality of its grants programs. Some other organization may be interested in becoming more accountable. Being aware of these interests in fundamental to understand how interests must be balanced. Here we offer a table with hypothetical examples of the kinds of interests that organizations may have.

Foundation / Organization	Interest
The Meyer Memorial Trust request-for-proposal program	Improve grantee programs, improve grantmaking
L.J. and Mary C. Skaggs Foundation program for arts organizations	Be accountable, improve grantmaking
Winthrop Rockfeller Foundation at-risk youth program	Improve grantee program, disseminate successful programs
Cities in Schools program	Increase the state of knowledge
Technology for Literacy project	Be accountable, improve grantmaking, assess quality or impact of program
William Penn Foundation career / vocational program	Be accountable, improve grantee programs, improve grantmaking
Gautreaux program	Increase the state of knowledge
W.K. Kellogg Foundation grants	Be accountable, improve grantee programs, improve grantmaking, assess quality or impact of grants / programs, plan new programs, disseminate successful programs

Example 2: Leading a community

Ultimately, the community is most affected by the results of an evaluation. That's why community leaders play a key role in the interest game of evaluations. Lisa Adkins knows enough about her community. She is the executive director of Youth Friends in Kansas City. Through Youth Friends, students are linked to volunteers with a shared interest, whether it's music or sports, computers or a special hobby, to create an environment in which every young person who needs a caring adult in his or her life has one.

Lisa talked to us about how her interests as a community leader interfere in the evaluation process. However, according to Lisa, conflict of interests has not been a problem in her evaluation process. She said that the work between evaluators, funders and herself as a community leader is usually of a cooperative nature, and everybody works together toward a common goal. She considers the evaluation process a positive experience for her organization, once they get a chance to review their programs and change what needs to be changed. So, evaluations are a constant element in her work

Her main interest, Lisa says, is to receive feedback on the programs Youth Friends provide, and evaluations are a great way of getting a response from the community. Youth Friends can adjust their services as the necessity arises from the community. Lisa points out that she is always the most interested in the evaluation's results since they could change her approach to a certain program and drastically improve it.

Example 3: Evaluating the evaluator

When it comes to figuring out interests in the evaluation game, the evaluator is a key character. Vince Francisco knows this game well. He's worked as an evaluator with diverse, community-based organizations doing hundreds of program evaluations. He has done evaluations of community policing programs, chronic and cardiovascular disease prevention programming, youth development, drug-abuse prevention, alcohol-use prevention, tobacco control initiatives, teen pregnancy prevention, and general youth development initiatives. Here Vince talks about how the evaluator's interests influence the score of the game:

"I try to find out immediately who the key players are and what they want to have answered. As an evaluator, what I bring to the table is a critical set of questions that I ask folks. Sometimes it's just sitting there watching folks in action, and other times it's asking key people very pointed questions about what they want to do, why they're doing it. I work with folks to take it to the next level and make it a little bit more formal in terms of coming up with some evaluation from very different points of view. If we don't understand their program, we can't go anywhere with their evaluation. I get a handle on their program (and I) will help them make their lives easier, help them run their program or initiative better, help them get more money, influence key players in the community to be on their side, the big issues. "

To find the other interests involved in an evaluation, Vince says that he talks to people until he has their roles figured out: "I look for the loudest voices, and then I look for the quietest voices those usually are the key players. The loudest voices usually speak for themselves, and not for the community. And the people who are the absolutely quietest, are the people who really have a voice in the community, but they're being drowned out by the folks who are loud. Very loud people usually get their way and they become key players so I want to know who they are and what their agenda is. And then I find whoever are the quietest people, and they're usually the people who sit in the background. I start talking to people and eventually there are some people that will talk back. I guess I just start talking to folks and (I work with) the ones that end up talking back to me. I combine a lot of talking."

Of course, Vince says, his own interests also influence the outcome of the intervention. To balance out this equation he thinks of the whole group or the whole community as his client. Talking to people, getting information, and shaping them, Vince produces something that works for his clients: "I outline and define the program : What's the initiative? What does it look like? Who are the key players? What are the program components and elements? Who are the targets of change? Who are the agents of change? Once I get a good handle on that, I connect that to some key questions. I collect evaluation data. What evaluation data would people believe as being evidence that these things are actually being implemented, and what would be key evidence that these things are successful? I combine that with appropriate components and elements in a systematically and very logical fashion. I look for ways for economizing procedures for collecting data. I get right to the chase. Don't do a whole lot of surveys that people have to spend time filling out."

Sometimes, however, interests conflict. Vince describes dealing with this kind of situation: "Usually, I'll find a few people that are really on board and very interested in evaluation. A lot of people just hate evaluators and think it's only a bunch of stupid surveys. My goal is to make it simple as possible. So my trick is to go in and sort of navigate the waters. Work with the people who want to work with me at first, and then convince everybody else that this is useful to them." Vince points out that, depending on how political the footfall of the program is, the evaluation may suffer.

Vince doesn't have a recipe to deal with all the interests that are involved in an evaluation: "All I know is that I juggle all the balls and keep them all in the air. And I look at all the different stakeholders. Who is the funding agent, and what is their interest in all this. I have to make sure their interest is represented. I try to give everybody the same exact information, so we're all moving forward together. That's the way I approach these things I try to change. I get everybody talking to each other in a real way and I don't get into the rumor way." Vince says that evaluators have a unique position because they can hold people accountable for their words: "I don't know if (the evaluator) is any more or less important than any other (interests). But without it, you don't do well. If we want to get lasting outcomes, we have to focus on behavior change. We have to focus on changing the systems that are controlling behavior."

Example 4: The funders role

Jeff Usher represents a very important interest in the evaluation process: the funder's. He is a program officer at the Kansas Health Foundation, a philanthropic organization that provides grants to improve the quality of health in Kansas. Here he talks about implementing evaluations and adjusting interests:

"We want to (see) change, we want to make a behavior change or community change. At the foundation, probably 10 to 20 percent of the overall budget will go to evaluation. The evaluation, of course, will be oftentimes both qualitative and quantitative in nature. We want to see the outcomes, whatever it might be; if it's teenage pregnancy, we'd like to see reductions of teenage pregnancies. So we like to see significant outcomes. We normally contract with people that have expertise in that particular area to conduct the evaluation."

Once the outcomes are in, Jeff says that it's time to decide what steps to take: "If it doesn't appear that the initiative is working, we clearly start losing interest in continuing the project in that manner. Oftentimes the evaluation portion of it is after the fact; the initiative is already over with, for the most part. If it comes back and it looks like the initiative was successful and the outcomes were going in the direction that we anticipated, then we may do another round of that initiative. For instance, the School Community Sexual Risk Reduction project, which is in its second phase now. Depending on the initiatives, if it appears that there is something else going on, we may still continue to work with those, or we may want to tweak that initiative a little bit to make it be a little more effective."

Jeff says that balancing interests among funders, community leaders and evaluators is not much of a problem for him: "We don't seem to have much of a problem when working with evaluators and as far as a private foundation, we don't hear a lot of problems." Jeff goes on to say that sometimes he experiences problems with some evaluators regarding the evaluation methods: "We may get some evaluations back that are measuring process rather than outcomes, and we clearly want to see outcomes. As far as balancing the grantee and the evaluator, normally, the evaluator is a grantee too, so in a sense there is even accountability that we want to see from the evaluator, and we would ask for that. Clearly there are certain things we want to see in that evaluation. For instance, we just did a memorandum agreement for an evaluation on a project that we are just going to be kicking off this next year. We want to see a change in the culture of campus, centered around binge drinking or heavy drinking. We outline those particular things in the memorandum when we start negotiating, so we expect to see that."

One of the funder's interests is to keep an eye on how the initiative conducts its business. Jeff says that in every memorandum agreement, they allow his foundation to perform site visits to have a third party come in, audit financial books, and see how the program is going: "Rarely do we follow up on those. The only time we would look at those is when we get a status report, and we ask for status reports from all of the grantees with what we want to see, and they haven't met those objectives. So, for almost all of our major initiatives, we'll have an evaluation component and have someone looking at it. We want to see behavior change, systems change, changes in culture centered around the health and well-being of communities, and individuals in communities."

The opening four sections of this chapter have been largely about theory: what evaluations of community-based initiatives and organizations are about, why they re important, and some essential ideas about how they can be conducted and whose interests they need to serve. This section and the next are about actually starting the process of setting up an evaluation, i.e. choosing the evaluators that will carry it out, and planning what it will look like. The two are very closely connected, but it will probably make sense to choose your evaluators first. Then, in consultation with the organization, the community, and the target population, they'll come up with a plan for helping you get the information you need to become, or to continue to be, as effective as possible.

Why pay attention to the selection of evaluators?

An evaluation is not simply a matter of looking at your organization or initiative and saying, "It's doing okay." It should examine what you're doing from a number of different angles and perspectives, and should give you not only a clear sense of whether or not you're accomplishing your goals, but of where your strengths and weaknesses lie, of how to improve what you're doing, and of new directions to explore. No matter how well your evaluation is planned--and it should be well planned--you'll still need evaluators that have the skills and knowledge to look at your particular situation accurately.

The real reason for an evaluation is its usefulness in improving the organization. The more, and the more accurate, the information you get, and the better it's analyzed, the more useful it will be in helping you identify and build on your organization's strengths and pinpoint and correct the areas it needs to work on. A well-conceived evaluation can also help you adjust to changing needs in the community or target population, learn more about the implications of the issue you're working on, provide a base for advocacy, and help with fundraising. As a result, it's important that your evaluators do the best job they can.

You may be able to hire evaluators, or you may be choosing volunteers from your staff or the community you serve. Or you may simply be choosing a planning team that will go on to select evaluators. In any of these cases, the particular individuals you select will influence the shape of the evaluation you get, and what kinds of results you get from it.

Although this section treats both professional and non-professional evaluators, it is understood that most organizations will either conduct evaluations in-house (i.e. using staff members, participants, and/or organizational records to gather and analyze information), use an individual or team from the community, or find a professional to help with an evaluation without charge. We've tried to cover as many possibilities as we can, so that organizations will be able to consider how best to use the resources they have to get the kind of evaluation they want.

The same is true for community or other planning teams. A planning team is only one of many possibilities, depending again on the resources of an organization and those resources include time. An evaluation that's not perfect, but is useful in some ways is better than no evaluation. A planning team is a great idea, but not if the thought of putting together and working with one prevents an organization from attempting an evaluation. The purpose of this and all Tool Box sections is not to create boxes for you to fit your organization into, but to provide as many options as possible.

If you hire professionals, typically either a consulting firm or a group from a university, they'll have particular ways of doing research, particular prejudices about what an evaluation should cover, and particular interests, all of which may or may not coincide with what you need or what your community wants. Community and in-house evaluators also, depending upon the individuals involved and the groups they represent, will have their own prejudices, interests, and needs which will influence how they view the evaluation process and the evaluation itself. The same goes for graduate students or others who might donate their time to conduct or assist with an evaluation. It's extremely important to select people whose agenda matches your own, or who will put their own agendas aside and respond directly to the needs and desires of the organization or initiative and the community. Only then will you wind up with an evaluation that best serves your purposes.

When should you choose evaluators?

The short answer to this question is "as soon as possible." Evaluations should not start at the end of a project. They should be ongoing, so the information can continually be used to improve what you're doing. A planning team should probably be chosen when the organization or initiative begins, or even before, so that an evaluation can look not only at what the organization has done, but at how it changed over the evaluation period.

Many funders require evaluation from the start in their grants. The grant for an adult literacy program run by a Massachusetts employment training agency, for instance, stipulated that 10% of the money be used for evaluation in the first year of the program. A University of Massachusetts graduate student was hired at the start of the program, and he conducted an evaluation which involved his visiting classrooms several times during the year, meeting regularly with the program staff, and interviewing participants. By the end of the year, he had gained not only a complete picture of the program, but a sense of its movement and growth over time as well. The evaluation he presented was extremely helpful in guiding the program over the next year.

As will become apparent below and in succeeding sections, it's important to know your community and its needs, as well as the needs of your organization or initiative, before you set out to evaluate. If your organization or initiative is just starting out, it may sometimes make more sense to wait a while until you have a clear understanding of the context of your evaluation before beginning to pick evaluators.

Context of the evaluation

Relates to whether an organization with the resources to do so may or may not decide to hire professional evaluators. The context refers to the unique situation of the evaluated organization or initiative and its community, the combination of geographical, historical, political, social, cultural, and other factors that form that situation. An understanding of that context is basic to an understanding of how to approach this particular community to evaluate this particular organization.

An example: An evaluation of an attempt to establish an adult literacy program in a rural area showed that, in the first year, the program failed to attract many students. It looked like a failure, and was in danger of losing its funding. But the staff of the program knew two things that the evaluators and funders didn't: first, that the area's inhabitants were country people, and were suspicious of and slow to accept "outsiders;" and second, that other programs had been started, but had always only lasted a year, because they didn't attract enough students. The staff argued, convincingly, that the missing element in earlier efforts was time, and that by leaving, the earlier programs had confirmed area residents' feelings that the outsiders didn't really care about their problems, and weren't serious about helping. The program continued, and by the end of its second year was thriving, because the local people believed that it would stay. Now, 15 years later, the program has become a fixture in the community, and is always full to capacity.

How should you decide between professional and community or other volunteer evaluators?

Evaluation is actually a type of research: it entails systematically gathering information and then drawing conclusions from it. There are people who engage in research of various kinds: students and academics and many consultants who do research for a living. If evaluation is research, it must make sense to hire trained researchers to conduct evaluations, right?

Well, yes and no. In many cases, it is true that a professional researcher will do the best job. But in other situations, that may not be true at all, for a variety of reasons.

Before you decide who's going to do your evaluation, there are a number of things to consider:

Money

This may be the most important, especially for newer and smaller organizations. Generally, the more grass roots your base is, the less money you have. If you have a funder who requires, or is willing to pay for, evaluation as part of a grant, then hiring professionals is at least an option. If there's no money available and no possibility of getting any, community or other volunteers may be your answer.

In a situation where you believe it's necessary to have professional evaluators, or at least professional guidance, you could try to fundraise what you needed (a difficult alternative, unless you have a lot of lead time and a great community base), or to find a volunteer mentor at a local college or university. Often graduate students, particularly, are willing to consult, or even to conduct the evaluation, without charge, either for the experience, or because the research will fit into their dissertations or some other work they're already doing. Other possible sources of various levels of assistance might include other health or human service agencies or initiatives; local government; or a fundraising drive or proposal (to a local foundation, for instance) specifically aimed at funding an evaluation.

In one instance, the instructor of a graduate course in Program Evaluation solicited proposals from community agencies and organizations for help in getting their programs evaluated. The organizations with successful proposals were each rewarded with a team of grad students who worked on evaluations with them over the course of the semester. At the end, there was a public event to which all agencies were invited, where they learned the overall results, and received copies of "Program Evaluation Tips," written by the grad students. This process produced useful evaluations for the agencies, invaluable hands-on experience for the grad students, and a strengthened connection between the university and the community.

The reality of money aside, if you have the option of considering professionals, the issues below come into play.

The complexity of the evaluation

For some organizations or initiatives, an evaluation may be relatively straightforward. They simply want to measure whether their very specific goals are being met. After a public education initiative, are significantly more children being vaccinated this year than last? Are learners in a literacy program gaining proficiency in reading, writing, and math? What percentage of participants in a substance use treatment program is still substance-free after a given period of time? If these are the types of questions that need to be answered, an evaluation can be fairly simple. But what if the questions asked are more complex? Was it the initiative, or some other factor, or a combination that resulted in increased vaccination rates? What are the environmental factors that influence literacy learners' progress? What do the substance users who backslide have in common? The methods to find the answers to those questions may need to be more complex as well. This issue is muddied, however, by the fact that sometimes a deep knowledge of the community may do more to answer complex questions than a knowledge of research methods. If you have the option of using professionals, you'll have to decide whether or not that's actually the best idea here.

The type of information desired and how it needs to be analyzed

If you're collecting quantitative data (i.e. numbers) it's often not possible to simply look at them and draw conclusions. There are statistical procedures that can be applied to the numbers to tell you what they really mean, how significant they are (i.e. what the odds are that differences or changes were actually caused by what you're measuring), and what they imply about other ways you could operate. If you need this kind of information or if funders or others are specifically asking for it, it would probably be helpful either to hire professional researchers or to find some volunteer professional guidance.

Sometimes conducting such statistical procedures can tell you important things you never expected to find out. A researcher trying to determine what had caused a group of women to return to school in their thirties and forties found that a major factor had been their having to care for younger siblings when they were teenagers. He had barely noticed this in interviews, it had been overshadowed by such more dramatic factors as divorce, substance use, and domestic violence, but computer analysis showed it to be extremely important. In fact, when the researcher went back and talked to several of the women about it, they acknowledged that this caregiving role had been a major reason for their not continuing their educations in the first place, and still loomed large in their lives.

What you're using the evaluation for

Is your evaluation solely to help you become as effective as possible in what you do? Is it to increase your credibility in the community? Is your continued funding dependent on it? Is it a combination of two or all of these? Your answers to these questions will help to determine what kinds of questions you need to ask in the evaluation, what form the answers need to be in, how complex the evaluation needs to be, and whether it's necessary to try to hire researchers or not. The choice of professionals or community volunteers here will depend not only on the factors above, but on your particular situation as well. Professionals may be able to better analyze the information, but may not have access to the same information that people from the community would have (or may in fact have better access, because they're perceived as neutral). Once again, you need to determine what's best for your situation.

How you want the evaluation to be perceived

Some organizations or initiatives may want to demonstrate the extent of their community-based or grassroots orientation by making sure that their evaluation is community-based as well. Others may want to show that their evaluation is objective by hiring evaluators with no connection to the community. Still others, for philosophical or practical reasons, may want to be sure to involve those they serve or benefit in the evaluation process. There are many variations and possibilities here, all legitimate, all right for some organizations and wrong for others. As with anything you do, it's important to be consistent with your mission and principles when you consider how to choose evaluators.

What should you look for in choosing evaluators?

There are characteristics you'd want in professionals, characteristics you'd want in a community planning or evaluation team, and some characteristics you'd want in all of them. We'll examine each of these possibilities separately.

An evaluation can be structured in a huge number of different ways. The split between professional and non-professional can blur in many of these possibilities. One might get a professional evaluator for free, for instance, because of her interests or her commitment to the work of the organization or to the community. While an evaluation could be carried out solely by either a professional or by a community or in-house group, there are many variations on these themes.

A paid evaluator or team probably will work at least to some extent with a group from the organization or the community. A paid evaluator might work with a community or organization planning team to help plan the evaluation, and then either conduct the evaluation itself without further input from the planning team, or withdraw entirely to have the planning team take over arranging and conducting the evaluation. A local planning team might plan the evaluation entirely on its own, then hire a professional to carry it out, or just to analyze the data. Professionals might work alongside a community team through the whole process. Whether and how you choose to use professionals, community or in-house planning teams or evaluators, etc., depends on what makes sense for your organization.

One evaluator, for instance, tells this story: "I was hired to help community members evaluate HIV/AIDS services in the community. The community members here were a mixed group mostly agency folks, but some nonagency people living with HIV /AIDS. I asked for some volunteers to plan the evaluation together with me, and was pleased when about half a dozen came forward.

"When we met, I set most of the agenda, raising the basic questions, such as 'What do we want to evaluate?' and 'How do we want to evaluate it?' We decided on a survey. Then the next questions became what kind of survey we needed, how it should be administered, who should administer it, and how the data should be collected. Our work style was to put each of these main questions on newsprint, to discuss them together, to write down ideas, to decide on the best option, and to move on. We ran systematically and very efficiently down the question list. When we finished, after about three one-hour meetings, the evaluation had basically been planned."

The evaluator then went on to conduct the evaluation and analyze the data on his own before presenting it to the community.

Characteristics to look for in all evaluators

Willingness to leave one's own agenda at the door

Often researchers, particularly those attached to universities, may have their own reasons for embarking on an evaluation. It may fit into a doctoral dissertation, or a book that a professor is writing, or a piece of long-term research that will eventually be published. They may also have very strong prejudices about what kind of research they want to use, or what they expect to find, and a need to prove their prejudices correct. And they may have issues about power and about their standing in relation to that of members of a planning team or organizational staff. If the researchers' needs mesh perfectly with yours, then there's no problem. But if they don't quite fit, there can be a very serious problem. If you're paying for a service, you should get what you're paying for, and not simply what the researcher wants to give you. It's important to be clear about this on all sides at the beginning, and to make sure by writing it into a contract or through some other formal mechanism that the professionals are willing to do what meets your needs, not what meets theirs.

Just as professionals may have research priorities that have nothing to do with the evaluation, community members may have personal or political priorities that also have little to do with the business at hand, but can affect their performance on an evaluation team. As much as possible, it's important that people either be willing to put these priorities aside while they're engaged in the evaluation process, or that you screen out those who will try to impose their particular ways of thinking on everyone, or who will use the process to further their own ends.

Ability to communicate with a broad range of people

Evaluators will have to deal with people from all walks of life, of all political, religious, and philosophical persuasions, and probably of many ethnic, language, and racial groups. If they are to gather accurate information, they will have to be perceived as trustworthy by all of those groups, and will have to be able to generate a certain level of comfort with everyone.

Have they worked with groups that included a broad spectrum of community members Do they have the verbal and interpersonal skills and the patience to explain their methods clearly to people who aren't highly educated and who may never have had any contact with research before (perhaps especially if the evaluators themselves are in that same category)? Can at least some of them speak at least some of the languages of the community?

You can answer at least some of these questions by interviewing potential evaluators before they're hired or chosen. Evaluators need to be able to gather the information needed for the evaluation and to couch the results of the evaluation in terms that the community can understand, and therefore use. The best evaluation plan is worthless if the evaluators' approach means that people can't or won't respond to them, or can't use the results they've come up with.

Cultural sensitivity

Especially in a community that contains residents of many cultures (depending on how you define "culture," that includes virtually every community), mutual respect and some understanding and acceptance of how others see the world is crucial to the functioning of evaluators. Do they understand, or are they willing to learn to understand, the cultures of those in the community? An urban, largely working class community is culturally very different from an upper-middle class suburb; a Haitian neighborhood is different from a Vietnamese or a Puerto Rican one. Evaluators need to respect the cultures of the communities they work with, and not violate them, intentionally or unintentionally.

Ability to treat everyone with the same degree of respect

How evaluators approach people reflects on the organization. If they don't treat everyone respectfully, they're not going to get accurate--or any--information, and they're going to complicate the organization's relationship with its staff, the target population, and the community.

Absolute commitment to keeping all individuals' information confidential

Whether evaluators are paid professionals or not, for ethical, practical, and legal reasons, it is almost always necessary to guarantee that any information gathered in the course of an evaluation will be kept confidential (i.e. used only for the purposes of the evaluation, and not connected to the individual), and that people won't be identified either by name or by other factors that could lead to them. In the case, for instance, of the evaluation of a domestic violence prevention program or women's shelter, confidentiality could be a matter of personal safety. In most instances, it will help evaluators obtain more accurate data. And it will protect the evaluators and the organization from lawsuits that could be brought by individuals injured in some way by the information they provided.

Professional or university researchers often ask informants to read and sign an "informed consent" form that explains exactly what the researchers are doing, what any information will be used for, and how the researchers will protect confidentiality and anonymity. If the researcher violates the terms of the form, the informant might have the grounds for a lawsuit, and the researcher's findings might be questioned. An evaluator, professional or not, who asks for responses from members of the target population or the community might consider using a similar form, both because it helps to explain what the evaluation is about, and because it demonstrates the evaluator's commitment to confidentiality.

Commitment to the evaluation process

For planners and evaluators, whether professionals or otherwise, this means trying to do the best evaluation possible, with an eye toward its actual usefulness for the organization. For the organization, and other community members, commitment means believing enough in the process to take the evaluation seriously and use it to make adjustments to and improve the program, service, or activity. An evaluation, no matter how elegant and informative, is worthless if it's not used.

An important part of a first evaluation is a plan for continued evaluation. To be truly useful, evaluation needs to be ongoing throughout the life of the organization. If evaluators can provide a guide for setting up and conducting evaluations on a regular basis, they'll have performed a service to the organization far beyond the value of the evaluation itself.

Community or in-house planners or evaluators

A word about planning teams vs. evaluation teams: A planning team is meant to plan the evaluation process. If the organization has the option of hiring professional evaluators, a community or staff planning team may decide whether or not to do so, and may serve as the hiring committee for those evaluators. Its main job (which it may do alone or with the help of a hired evaluator) will probably be to examine the possibilities and come up with an actual evaluation plan, detailing what the organization needs to find out, where it can find that information, who will do the information gathering and analysis, and how the information will be used once it's gathered and analyzed.

A community evaluation team is a volunteer team set up to carry out the actual evaluation. In practice, this may be the same group as the planning team, whose function, once the planning is done, is the accomplishment of the task. It's helpful if the evaluation team includes someone with enough understanding of research that the evaluation they conduct will be valid, and its results can be used with some confidence.

With a community planning or evaluation team, you have the opportunity to choose all the members individually. There may be some people who, because of their position in the organization or in the community, have to be on the team. By the same token, some others may not be able to be included. In general, the team should be small enough so that it won't be too difficult for members to meet and to contact one another, and large enough so that there are enough people to do the work. And, in general, the team members should represent the interested stakeholders in the organization or initiative: staff and Board, target population, and community members at large. Characteristics you might look for in those on your planning or evaluation team include:

Representation

The team could include any number of the following possibilities (but, for most organizations, would probably number only four or five):

• A range of organizational representatives (e.g. the director, one or two line staff members line [those who work directly with the target population], and a Board member)
• Members of the target population, ideally individuals who understand and have participated in the organization or initiative
• Community members with a stake in the organization or initiative (business people, health and human service agency representatives, school and local government officials, other citizens with an interest in the issue)
• Youth, if appropriate
• Relevant language and other minority groups
• Someone with research skills and experience, perhaps from a local college or university.

Ability to understand the purpose of the evaluation

It's important to understand not only what the evaluationis supposed to measure, but also how its form and purpose are related.

Willingness to listen and learn

This covers a wide spectrum of people and behaviors. It includes both the local high school principal being able to listen to and take seriously the opinions of a high school dropout, and that dropout's belief that he's capable of learning what he needs to know to function as a contributing member of the team. It also includes everyone's being aware that they have things to learn about the process, one another, and evaluation in general, and, by the same token, that they have valuable knowledge, skills, and information to contribute.

Ability to work in a group

This can be a particularly sticky issue in a group that encompasses several cultures and classes. The middle-class "meeting skills " that most educated people have been unconsciously learning and practicing since junior high or before are often unknown to welfare recipients or migrant workers, simply because they've never been exposed to them. Thus, they may sit silent, confused by the flow of the meeting, unable to contribute, and feeling foolish; or they may speak or act in ways or at times others deem inappropriate. People without meeting skills need support and encouragement in what is often for them an intimidating situation. One answer may be to pair all team members, so that those who need support will have a mentor. Another possibility is to start the process with a training for everyone in group dynamics and meeting skills, so that no one is singled out and the ground rules are clear for all.

Professionals

Unless you're working with an individual consultant, when you work with professionals, it's likely that a team will come already assembled. Thus, you're looking at the whole package: who the team members are, what they know how to do, and how they're likely to interact with your planning team, if you have one, and with your organization and the community. Most professional teams, as mentioned above, will either be private consultants or university researchers. Probably the most important features to look for are their professional experience and skills, and their expertise and style in working with community groups:

Knowledge of different kinds of evaluation techniques

Can they use both quantitative (numbers and statistics) and qualitative (facts, stories, anecdotes, analysis of situations and events, etc.) research, and do they know when each is appropriate? Have they used them in situations like yours, or do they have good ideas about how to do that?

If your team is from higher education, you should consider carefully what department (s) of a university to contact to find evaluators whose professional and scholarly specialties are a good fit with the work you do. Depending upon what your organization or initiative does, and what information you're looking for, some possibilities are:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Past performance

Have they done evaluation before? What kinds of techniques have they used? Do they have good references? Can you see examples of past evaluations they've done?

Just as you probably wouldn't hire someone for a position without checking her references, you shouldn't hire an evaluator or team without checking with those who've employed them before. How well professionals do in an interview and on paper is usually enough to tell you what you need to know about them, but when it's not, it can be disastrous, costing you not only money, but time, results, and relationships in the community.

You should ask to see some of the results of previous evaluations. Are they readable and understandable? How did the candidate explain them to the organization and the community? Knowing what you're getting can help immeasurably in choosing the right evaluators.

Experience working with community groups

Have they worked in partnership with community groups before? Do they understand the roles in this situation, and can they handle theirs well? Have they been able to work with the community to address the appropriate goals? Are they responsive to suggestion, to the needs of the community, and to the context of the evaluation? Do they collaborate, in other words, rather than telling the community group what it needs?

Ability to really listen

It's important that not only is the evaluator able to listen, but also respect others and not assume the "expert" role except where that's appropriate (you are, after all, hiring them for their expertise). If the evaluation is community-based, the evaluators are going to have to listen first to the planning group about what the organization or initiative most needs to know, and then to the community in order to gather that information. Real listening (listening not only for the meaning of what's being said, but for the meaning behind the meaning) is an indispensable skill for all good evaluators.

In Summary

The individual or team you select to plan and/or conduct the evaluation of your organization or initiative will do much to determine the character and usefulness of the evaluation itself. If you work with professionals, you need to consider their credentials, past experience, and level of expertise. Whether you choose professional evaluators, community volunteers, or some combination of the two, you need to think carefully about how their needs and interests fit in with those of the organization and the community, about the range and quality of their communication skills, about their willingness to work as partners with the organization and the community, and about their fit with one another. Once you've put together a good team, you are well on your way to carrying out an accurate and valuable evaluation.

Why should you evaluate?

___To see results of work

___To gain public opinion

___To make the proper adjustments if necessary

What should you consider when you select evaluators?

___Select evaluators carefully for reliability 

Possible evaluators:

___Volunteers

___Community being serviced

___Chosen evaluators by a team

Tool: Interview Questions for evaluation consultant selection

From EvalTalk, the discussion group of the American Evaluation Association (AEA).

• What are the main reasons why you think a program should conduct an evaluation?

• What are the main standards and guiding principles for conducting evaluations?

• How would you determine if a program is ready to undergo an evaluation?

• How do you estimate the resources needed to conduct an evaluation? That is, how do you compute the costs of an evaluation, both in money and time (yourself and others)?

• If you had results from an evaluation that were based on quantitative information that conflicted with results from qualitative information, how would you resolve the conflict?

• If you were to conduct a mailed-out survey to program participants, how would you achieve a representative sample and ensure that your results were generalizable to all participants?

• Frequently evaluations involve various and independent branches and agencies that all have different expectations for the evaluation. If you had a conflict among participants in a program evaluation that could threaten the evaluation, how would you resolve the conflict? What steps would you take to proactively avoid conflict?

• Read any interesting articles on evaluation lately? In what journals? What was interesting about it? What other journals do you see regularly?

• Under what circumstances, if any, would you blow the whistle on a client?

Why should you have an evaluation plan?

After many late nights of hard work, more planning meetings than you care to remember, and many pots of coffee, your initiative has finally gotten off the ground. Congratulations! You have every reason to be proud of yourself and you should probably take a bit of a breather to avoid burnout. Don't rest on your laurels too long, though--your next step is to monitor the initiative's progress. If your initiative is working perfectly in every way, you deserve the satisfaction of knowing that. If adjustments need to be made to guarantee your success, you want to know about them so you can jump right in there and keep your hard work from going to waste. And, in the worst case scenario, you'll want to know if it's an utter failure so you can figure out the best way to cut your losses. For these reasons, evaluation is extremely important.

There's so much information on evaluation out there that it's easy for community groups to fall into the trap of just buying an evaluation handbook and following it to the letter. This might seem like the best way to go about it at first glance-- evaluation is a huge topic and it can be pretty intimidating. Unfortunately, if you resort to the "cookbook" approach to evaluation, you might find you end up collecting a lot of data that you analyze and then end up just filing it away, never to be seen or used again.

Instead, take a little time to think about what exactly you really want to know about the initiative. Your evaluation system should address simple questions that are important to your community, your staff, and (last but never least!) your funding partners. Try to think about financial and practical considerations when asking yourself what sort of questions you want answered. The best way to insure that you have the most productive evaluation possible is to come up with an evaluation plan.

Here are a few reasons why you should develop an evaluation plan:

• It guides you through each step of the process of evaluation
• It helps you decide what sort of information you and your stakeholders really need
• It keeps you from wasting time gathering information that isn't needed
• It helps you identify the best possible methods and strategies for getting the needed information
• It helps you come up with a reasonable and realistic timeline for evaluation
• Most importantly, it will help you improve your initiative!

When should you develop an evaluation plan?

As soon as possible! The best time to do this is before you implement the initiative. After that, you can do it anytime, but the earlier you develop it and begin to implement it, the better off your initiative will be, and the greater the outcomes will be at the end.

Remember, evaluation is more than just finding out if you did your job. It is important to use evaluation data to improve the initiative along the way.

What are the different types of stakeholders and what are their interests in your evaluation?

We'd all like to think that everyone is as interested in our initiative or project as we are, but unfortunately that isn't the case. For community health groups, there are basically three groups of people who might be identified as stakeholders (those who are interested, involved, and invested in the project or initiative in some way): community groups, grantmakers/funders, and university-based researchers. Take some time to make a list of your project or initiative's stakeholders, as well as which category they fall into.

What are the types of stakeholders?

• Community groups: Hey, that's you! Perhaps this is the most obvious category of stakeholders, because it includes the staff and/or volunteers involved in your initiative or project. It also includes the people directly affected by it--your targets and agents of change.
• Grantmakers and funders: Don't forget the folks that pay the bills! Most grantmakers and funders want to know how their money's being spent, so you'll find that they often have specific requirements about things they want you to evaluate. Check out all your current funders to see what kind of information they want you to be gathering. Better yet, find out what sort of information you'll need to have for any future grants you're considering applying for. It can't hurt!
• University-based researchers: This includes researchers and evaluators that your coalition or initiative may choose to bring in as consultants or full partners. Such researchers might be specialists in public health promotion, epidemiologists, behavioral scientists, specialists in evaluation, or some other academic field. Of course, not all community groups will work with university-based researchers on their projects, but if you choose to do so, they should have their own concerns, ideas, and questions for the evaluation. If you can't quite understand why you'd include these folks in your evaluation process, try thinking of them as auto mechanics--if you want them to help you make your car run better, you will of course include them in the diagnostic process. If you went to a mechanic and started ordering him around about how to fix your car without letting him check it out first, he'd probably get pretty annoyed with you. Same thing with your researchers and evaluators: it's important to include them in the evaluation development process if you really want them to help improve your initiative.

Each type of stakeholder will have a different perspective on your organization as well as what they want to learn from the evaluation. Every group is unique, and you may find that there are other sorts of stakeholders to consider with your own organization. Take some time to brainstorm about who your stakeholders are before you being making your evaluation plan.

What do they want to know about the evaluation?

While some information from the evaluation will be of use to all three groups of stakeholders, some will be needed by only one or two of the groups. Grantmakers and funders, for example, will usually want to know how many people were reached and served by the initiative, as well as whether the initiative had the community -level impact it intended to have. Community groups may want to use evaluation results to guide them in decisions about their programs, and where they are putting their efforts. University-based researchers will most likely be interested in proving whether any improvements in community health were definitely caused by your programs or initiatives; they may also want to study the overall structure of your group or initiative to identify the conditions under which success may be reached.

What decisions do they need to make, and how would they use the data to inform those decisions?

You and your stakeholders will probably be making decisions that affect your program or initiative based on the results of your evaluation, so you need to consider what those decisions will be. Your evaluation should yield honest and accurate information for you and your stakeholders; you'll need to be careful not to structure it in such a way that it exaggerates your success, and you'll need to be really careful not to structure it in such a way that it downplays your success!

Consider what sort of decisions you and your stakeholders will be making. Community groups will probably want to use the evaluation results to help them find ways to modify and improve your program or initiative. Grantmakers and funders will most likely be making decisions about how much funding to give you in the future, or even whether to continue funding your program at all (or any related programs). They may also think about whether to impose any requirements on you to get that program (e.g., a grantmaker tells you that your program may have its funding decreased unless you show an increase of services in a given area). University-based researchers will need to decide how they can best assist with plan development and data reporting.

You'll also want to consider how you and your stakeholders plan to balance costs and benefits. Evaluation should take up about 10--15% of your total budget. That may sound like a lot, but remember that evaluation is an essential tool for improving your initiative. When considering how to balance costs and benefits, ask yourself the following questions:

• What do you need to know?
• What is required by the community?
• What is required by funding?

How do you develop an evaluation plan?

There are four main steps to developing an evaluation plan:

• Clarifying program objectives and goals
• Developing evaluation questions
• Developing evaluation methods
• Setting up a timeline for evaluation activities

Clarifying program objectives and goals

The first step is to clarify the objectives and goals of your initiative. What are the main things you want to accomplish, and how have you set out to accomplish them? Clarifying these will help you identify which major program components should be evaluated. One way to do this is to make a table of program components and elements.

Developing evaluation questions

For our purposes, there are four main categories of evaluation questions. Let's look at some examples of possible questions and suggested methods to answer those questions. Later on, we'll tell you a bit more about what these methods are and how they work

• Planning and implementation issues: How well was the program or initiative planned out, and how well was that plan put into practice?
  • Possible questions: Who participates? Is there diversity among participants? Why do participants enter and leave your programs? Are there a variety of services and alternative activities generated? Do those most in need of help receive services? Are community members satisfied that the program meets local needs?
  • Possible methods to answer those questions: monitoring system that tracks actions and accomplishments related to bringing about the mission of the initiative, member survey of satisfaction with goals, member survey of satisfaction with outcomes.
• Assessing attainment of objectives: How well has the program or initiative met its stated objectives?
  • Possible questions: How many people participate? How many hours are participants involved?
  • Possible methods to answer those questions: monitoring system (see above), member survey of satisfaction with outcomes, goal attainment scaling.
• Impact on participants: How much and what kind of a difference has the program or initiative made for its targets of change?
  • Possible questions: How has behavior changed as a result of participation in the program? Are participants satisfied with the experience? Were there any negative results from participation in the program?
  • Possible methods to answer those questions: member survey of satisfaction with goals, member survey of satisfaction with outcomes, behavioral surveys, interviews with key participants.
• Impact on the community: How much and what kind of a difference has the program or initiative made on the community as a whole?
  • Possible questions: What resulted from the program? Were there any negative results from the program? Do the benefits of the program outweigh the costs?
  • Possible methods to answer those questions: Behavioral surveys, interviews with key informants, community-level indicators.

Developing evaluation methods

Once you've come up with the questions you want to answer in your evaluation, the next step is to decide which methods will best address those questions. Here is a brief overview of some common evaluation methods and what they work best for.

Monitoring and feedback system

This method of evaluation has three main elements:

• Process measures: these tell you about what you did to implement your initiative;
• Outcome measures: these tell you about what the results were; and
• Observational system: this is whatever you do to keep track of the initiative while it's happening.

Member surveys about the initiative

When Ed Koch was mayor of New York City, his trademark call of "How am I doing?" was known all over the country. It might seem like an overly simple approach, but sometimes the best thing you can do to find out if you're doing a good job is to ask your members. This is best done through member surveys. There are three kinds of member surveys you're most likely to need to use at some point:

• Member survey of goals: done before the initiative begins - how do your members think you're going to do?
• Member survey of process: done during the initiative - how are you doing so far?
• Member survey of outcomes: done after the initiative is finished - how did you do?

Goal attainment report

If you want to know whether your proposed community changes were truly accomplished-- and we assume you do--your best bet may be to do a goal attainment report. Have your staff keep track of the date each time a community change mentioned in your action plan takes place. Later on, someone compiles this information (e.g., "Of our five goals, three were accomplished by the end of 1997.")

Behavioral surveys

Behavioral surveys help you find out what sort of risk behaviors people are taking part in and the level to which they're doing so. For example, if your coalition is working on an initiative to reduce car accidents in your area, one risk behavior to do a survey on will be drunk driving.

Interviews with key participants

Key participants - leaders in your community, people on your staff, etc. - have insights that you can really make use of. Interviewing them to get their viewpoints on critical points in the history of your initiative can help you learn more about the quality of your initiative, identify factors that affected the success or failure of certain events, provide you with a history of your initiative, and give you insight which you can use in planning and renewal efforts.

Community-level indicators of impact

These are tested-and-true markers that help you assess the ultimate outcome of your initiative. For substance use coalitions, for example, the U.S. Centers for Substance Abuse Prevention (CSAP) and the Regional Drug Initiative in Oregon recommend several proven indicators (e.g., single-nighttime car crashes, emergency transports related to alcohol) which help coalitions figure out the extent of substance use in their communities. Studying community-level indicators helps you provide solid evidence of the effectiveness of your initiative and determine how successful key components have been.

Setting up a timeline for evaluation activities

When does evaluation need to begin?

Right now! Or at least at the beginning of the initiative! Evaluation isn't something you should wait to think about until after everything else has been done. To get an accurate, clear picture of what your group has been doing and how well you've been doing it, it's important to start paying attention to evaluation from the very start. If you're already part of the way into your initiative, however, don't scrap the idea of evaluation altogether--even if you start late, you can still gather information that could prove very useful to you in improving your initiative.

Outline questions for each stage of development of the initiative

We suggest completing a table listing:

• Key evaluation questions (the five categories listed above, with more specific questions within each category)
• Type of evaluation measures to be used to answer them (i.e., what kind of data you will need to answer the question?)
• Type of data collection (i.e., what evaluation methods you will use to collect this data)
• Experimental design (A way of ruling out threats to the validity - e.g., believability - of your data. This would include comparing the information you collect to a similar group that is not doing things exactly the way you are doing things.)

With this table, you can get a good overview of what sort of things you'll have to do in order to get the information you need.

When do feedback and reports need to be provided?

Whenever you feel it's appropriate. Of course, you will provide feedback and reports at the end of the evaluation, but you should also provide periodic feedback and reports throughout the duration of the project or initiative. In particular, since you should provide feedback and reports at meetings of your steering committee or overall coalition, find out ahead of time how often they'd like updates. Funding partners will want to know how the evaluation is going as well.

When should evaluation end?

Shortly after the end of the project - usually when the final report is due. Don't wait too long after the project has been completed to finish up your evaluation - it's best to do this while everything is still fresh in your mind and you can still get access to any information you might need.

What sort of products should you expect to get out of the evaluation?

The main product you'll want to come up with is a report that you can share with everyone involved. What should this report include?

• Effects expected by shareholders: Find out what key people want to know. Be sure to address any information that you know they're going to want to hear about!
• Differences in the behaviors of key individuals: Find out how your coalition's efforts have changed the behaviors of your targets and agents of change. Have any of your strategies caused people to cut down on risky behaviors, or increase behaviors that protect them from risk? Are key people in the community cooperating with your efforts?
• Differences in conditions in the community: Find out what has changed Is the public aware of your coalition or group's efforts? Do they support you? What steps are they taking to help you achieve your goals? Have your efforts caused any changes in local laws or practices?

You'll probably also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team.

What sort of standards should you follow?

Now that you've decided you're going to do an evaluation and have begun working on your plan, you've probably also had some questions about how to ensure that the evaluation will be as fair, accurate, and effective as possible. After all, evaluation is a big task, so you want to get it right. What standards should you use to make sure you do the best possible evaluation? In 1994, the Joint Committee on Standards for Educational Evaluation issued a list of program evaluation standards that are widely used to regulate evaluations of educational and public health programs.The standards the committee outlined are for utility, feasibility, propriety, and accuracy. Consider using evaluation standards to make sure you do the best evaluation possible for your initiative.

You understand that you should have an evaluation plan because:

___It guides you through each step of the process of evaluation

___It helps you decide what sort of information you and your stakeholders really need

___It keeps you from wasting time gathering information that isn't needed

___It helps you identify the best possible methods and strategies for getting the needed information

___It helps you come up with a reasonable and realistic timeline for evaluation

___Most importantly, it will help you improve your initiative!

You understand who your project or initiative's stakeholders are:

___Community groups

___Grantmakers and funders

___University-based researchers

You have taken into consideration:

___What stakeholders want to know about the evaluation

___What decisions stakeholders need to make

___How stakeholders would use the data to inform decisions

When considering how to balance costs and benefits, you have asked yourself the following questions:

___What do you need to know?

___What is required by the community?

___What is required by funding?

You understand these four main steps to developing an evaluation plan:

___Clarify program objectives and goals

___Develop evaluation questions

___Develop evaluation methods

___Set up a timeline for evaluation activities

Clarify the objectives and goals of your initiative:

___Make a table of program components and elements

Develop evaluation questions:

___You understand the four main categories of evaluation questions:

• Planning and implementation issues
• Assessing attainment of objectives
• Impact on participants
• Impact on the community

___You have considered the best possible methods to answer those questions

Develop evaluation methods to best address those questions:

___You understand how to use the monitoring and feedback system's three main elements (process measures, outcome measures, and observational system)

___You understand how to use member surveys about the initiative (member survey of goals, member survey of process, and member survey of outcomes)

___You understand how to use the goal attainment report

___You understand how to use behavioral surveys

___You understand how to use interviews with key participants

___You know how to use community-level indicators of impact

Setting up a timeline for evaluation activities:

___You understand that you should begin right now, or at least at the beginning of the initiative

___You've outlined questions for each stage of development of the initiative

___You've completed a table listing: key evaluation questions, type of evaluation measures to be used to answer them, type of data collection, and experimental design .

___You've determined when you feel it's appropriate to provide feedback and reports

___You will provide feedback and reports at the end of the evaluation

___You will also provide periodic feedback and reports throughout the duration of the project or initiative

___You've decided when the evaluation will end

___You've mapped out a proposed evaluation timeline

Expected evaluation products:

___You will make a report that you can share with everyone involved which includes effects expected by shareholders, differences in the behaviors of key individuals, and differences in conditions in the community

___You've decided whether to also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team

Evaluation standards:

___You've decided what standards you will use to ensure an accurate and useful evaluation

Example 1: Table of Program Components and Elements

Here are a few of the items from the table of program components and elements for a fictional teen pregnancy prevention program:

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery
Risk factor: Unsupervised activities by teens	Provide alternative activities	After-school activities (sports programs, hobby / craft classes, movies, etc.) Weekend "lock-ins", non-alcoholic teen dance club Daytime activities during school holidays	Targets:   Youth   Agents: Youth Teachers Parents	Group sessions Workshops Meetings
Risk factor: Lack of access to contraceptives	Increased access to health services	Extend hours at school health clinics Increase condom distribution by county health department Brown bag program at area pharmacies (teens wishing to buy contraceptives are able to call ahead and pick them up in a plain brown bag)	Targets:   Youth Agents: Health providers	Walk-in clinics Health Department Businesses
Protective factor: Involvement in the faith community	Encourage youth involvement in the faith community	Increase church-sponsored youth activities Institute church-based sexuality education Church-sponsored parent-child communication sessions	Targets: Youth Parents Agents: Youth Parents Religious leaders	Discussions Meetings Educational sessions Workshops

Example 2: Evaluation Questions and Methods

Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (Y.E.S.), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally, and interact with positive role models:

		Type of Data Collection*					Experimental Design**
Key Evaluation Question(s)	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- and Post-test Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1. Who participates? Is there diversity among the participants?	1. Basic demographic data (e.g., age, gender, disability, race, economic status, etc.) on participants.	X	X				X
2. Why do participants enter and leave the program?	2. Information on participants' reasons for entering and leaving the program.	X	X				X
3. Are there a variety of service and alternative activities generated?	3. Frequency of different types of service provided and alternative activities generated.			X	X	X	X
Assessing Attainment of Objectives	Outcome Measures
1. How many people participate?	1. Number of people participating.			X	X	X	X		X
2. How many hours are participants involved in service and activities?	2. Number of hours of participation by type of activity.			X	X	X	X		X
3. How many people are trained in cultural awareness and sensitivity?	3. Number of people trained in cultural awareness and sensitivity.			X	X	X	X		X
Impact on Participants
1. How do behavior and attitudes change as a result of participation in the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X		X	X
2. Does participation affect the incidence of problems?	2. Incidence of problems in living (e.g., substance use, gang involvement).	X	X			X		X
3. Are participants satisfied with the experience?	3. Satisfaction ratings from participants.	X					X
Impact on Community
1.  What resulted from the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X			X
2. Do the benefits of the program outweigh the costs?	2. Cost benefit data.		X	X		X	X
3. Are community members satisfied with the participants and the service they provided?	3. Satisfaction ratings from beneficiaries and community members.	X					X

* Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

• Survey/Scale: i.e., member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys
• Structured Interview: i.e., interview with key participants
• Staff Report/Log: i.e. monitoring system, goal attainment report
• Direct Observation: This refers to the direct observation of behavior by researchers .
• Archival Records: i.e., community-level indicators of impact

**Here are explanations of these terms:

• Case Study: an in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment--good for answering "how" or "why" questions
• Pre-/Posttest Control Group: monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made
• Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative

Example 3: Timeline for Evaluation Activities

This is the timeline for an evaluation being done by Kansas City's Promise on their initiative to promote community and personal caring for youth:

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system	[ - - - ]
Train participants / staff in data entry	[ - - - ]
Maintain system (data collection and reporting)	[- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - ]
Critical events
Identify key people	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Conduct interviews	[ - - ]	[ - - ]	[ - - ]	[ - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Community-Level Indicators
Identify local indicators	[ - - - - ]
Identify comparison site	[ - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Other Indicators (e.g., behavioral surveys, data observations)
Identify	[ - - - - - -	- - - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - -  - - - - -	- - - - - - - - - -]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]

 

Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts

Download PPT Slides >>

 

Table of Contents

General Evaluation Resources
Steps in Community Evaluation (and Related Resources)

1. Design Stakeholder Involvement
2. Identify Resources for Monitoring and Summative Evaluation
3. Describe the Intervention’s Framework, Logic Model, or Theory of Change
4. Focus the Monitoring and Summative Evaluation  Plan
5. Plan for credible methods
6. Synthesize and Generalize
7. Systems and Evaluation

 

General Evaluation Resources

• Institute of Medicine (IOM):
  • ​Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010)
  • Measuring Progress in Obesity Prevention: Workshop Report (2012)
  • Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012)
  • Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013)
• U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
• Community Tool Box (CTB):
  • Toolkit: Evaluating the Initiative
  • A Framework for Program Evaluation: A Gateway to Tools
  • Evaluating Comprehensive Community Initiatives
  • Toubleshooting Guide: We don’t know how to evaluate our program or initiative

Steps in Community Evaluation (and related resources)

 

 

Back to Top

 

1. Design stakeholder involvement
   
   1. Identify stakeholders
   2. Consider the extent of stakeholder involvement
   3. Assess desired outcomes of monitoring and summative evaluation *
   4. Define stakeholder roles in the monitoring and summative evaluation *
   5. Resources:
      • Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Statement On Cultural Competence In Evaluation
      • Community Tool Box:
        • ​Identifying and Analyzing Stakeholders and Their Interests
        • Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
        • Participatory Evaluation
        • Gathering Information: Monitoring Your Progress

 

 

Back to Top

 

2. Identify resources for monitoring and summative evaluation
   
   1. Person-power resources
   2. Data collection resources
   3. Resources:
      • Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Identifying Community Assets and Resources
        • ​Choosing Evaluators

 

Back to Top

 

3. Describe the intervention’s framework, logic model, or theory of change
   
   1. Purpose or mission
   2. Context or conditions
   3. Inputs: resources and barriers
   4. Activities or interventions
   5. Outputs of activities
   6. Intended effects or outcomes
   7. Resources:
      • Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Toolkit: Developing a Framework or Model of Change
        • Developing a Logic Model or Theory of Change

 

 

Back to Top

 

4. Focus the monitoring and summative evaluation plan
   
   1. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
   2. Set priorities by end-user questions, resources, context
   3. What questions will the monitoring and summative evaluation answer?
   4. Ethical implications (benefit outweighs risk)
   5. Resources:
      • Institute of Medicine:
        1. Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        2. Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Guiding Principles For Evaluators
      • Community Tool Box:
        1. Choosing Questions and Planning the Evaluation
        2. Ethical Issues in Community Interventions
        3. Documenting Health Promotion Initiatives Using the PAHO Guide

 

 

Back to Top

 

5. Plan for credible methods
   
   1. Stakeholder agreement on methods
   2. Indicators of success
   3. Credibility of evidence
   4. Resources:
      • Institute of Medicine:
        • Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget)
        • Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • National Collaborative on Childhood Obesity Research (NCCOR):
        • Catalogue of Surveillance Systems
        • ​Measures Registry
      • Community Tool Box:
        • Data Collection: Designing an Observational System
        • Selecting an Appropriate Design for the Evaluation
        • Collecting and Analyzing Data
        • Collecting and Using Archival Data
        • Qualitative Methods to Assess Community Issues
        • Implementing Photovoice in Your Community
        • Community-Level Indicators: Some Examples
      • Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users

 

Back to Top

 

6. Synthesize and generalize
   
   1. Disseminating and compiling studies
   2. Learning more from implementation
   3. Ways to assist generalization
   4. Shared sense-making and cultural competence
   5. Disentangling effects of interventions
   6. Resources:
      • Institute of Medicine:
        • Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support
      • Community Tool Box:
        • Troubleshooting Guide: There is Not Enough Change in the Community or System
        • Troubleshooting Guide: There is Not Enough Improvement in Outcomes
        • Troubleshooting Guide: There are Unintended or Unwanted Outcomes
        • Toolkit: Enhancing Cultural Competence
        • Adapting Community Interventions for Different Cultures and Communities
        • Providing Feedback to Improve the Initiative
        • Communicating Information to Funders for Support and Accountability
        • Refining the Program Intervention Based on Research
        • The County Health Rankings & Roadmaps Take Action Cycle

 

 

Back to Top

 

7. Systems and Evaluation
   • Resources:
     • IOM:
       • Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
       • Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation
       • Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
     • Community Tool Box:
       • Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders
       • Best Change Processes
       • MAPP: Mobilizing for Action through Planning and Partnerships

 

Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box.

 

*Monitoring is the tracking of the implementation of interventions compared to standards of performance.
*Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions.

 

 

Tool 2: Table of Program Components and Elements

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery

 

Tool 3: Evaluation Questions and Methods

		Type of Data Collection *					Experimental Design **
Key Evaluation Questions	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured Interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- Posttest Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Assessing Attainment of Objectives	Outcome Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Participants
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Community
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.

Tool 4: Evaluation Timeline

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system
Train participants / staff  in data entry
Maintain system (data collection and reporting)
Critical events
Identify key people
Conduct interviews
Report results
Community-Level Indicators
Identify local indicators
Identify comparison site
Secure data
Report results
Other Indicators (e.g., behavioral surveys, data observations)
Identify
Secure data
Report results

Experienced community builders know that involving stakeholders - the people directly connected to and affected by their projects - in their work is tremendously important. It gives them the information they need to design, and to adjust or change, what they do to best meet the needs of the community and of the particular populations that an intervention or initiative is meant to benefit. This is particularly true in relation to evaluation.

As we have previously discussed, community-based participatory research can be employed in describing the community, assessing community issues and needs, finding and choosing best practices, and/or evaluation. We consider the topic of participatory evaluation important enough to give it a section of its own, and to show how it fits into the larger participatory research picture.

It's a good idea to build stakeholder participation into a project from the beginning. One of the best ways to choose the proper direction for your work is to involve stakeholders in identifying real community needs, and the ways in which a project will have the greatest impact. One of the best ways to find out what kinds of effects your work is having on the people it's aimed at is to include those on the receiving end of information or services or advocacy on your evaluation team

Often, you can see most clearly what's actually happening through the eyes of those directly involved in it - participants, staff, and others who are involved in taking part in and carrying out a program, initiative, or other project. Previously, we have discussed how you can involve those people in conducting research on the community and choosing issues to address and directions to go in. This section is about how you can involve them in the whole scope of the project, including its evaluation, and how that's likely to benefit the project's final outcomes.

What is participatory evaluation?

When most people think of evaluation, they think of something that happens at the end of a project - that looks at the project after it's over and decides whether it was any good or not. Evaluation actually needs to be an integral part of any project from the beginning. Participatory evaluation involves all the stakeholders in a project - those directly affected by it or by carrying it out - in contributing to the understanding of it, and in applying that understanding to the improvement of the work.

Participatory evaluation, as we shall see, isn't simply a matter of asking stakeholders to take part. Involving everyone affected changes the whole nature of a project from something done for a group of people or a community to a partnership between the beneficiaries and the project implementers. Rather than powerless people who are acted on, beneficiaries become the copilots of a project, making sure that their real needs and those of the community are recognized and addressed. Professional evaluators, project staff, project beneficiaries or participants, and other community members all become colleagues in an effort to improve the community's quality of life.

This approach to planning and evaluation isn't possible without mutual trust and respect. These have to develop over time, but that development is made more probable by starting out with an understanding of the local culture and customs - whether you're working in a developing country or in an American urban neighborhood. Respecting individuals and the knowledge and skills they have will go a long way toward promoting long-term trust and involvement.

The other necessary aspect of any participatory process is appropriate training for everyone involved. Some stakeholders may not even be aware that project research takes place; others may have no idea how to work alongside people from different backgrounds; and still others may not know what to do with evaluation results once they have them. We'll discuss all of these issues - stakeholder involvement, establishing trust, and training - as the section progresses.

The real purpose of an evaluation is not just to find out what happened, but to use the information to make the project better.

In order to accomplish this, evaluation should include examining at least two areas:

• Process. The process of a project includes the planning and logistical activities needed to set up and run it. Did we do a proper assessment beforehand so we would know what the real needs were? Did we use the results of the assessment to identify and respond to those needs in the design of the project?  Did we set up and run the project within the timelines and other structures that we intended? Did we involve the people we intended to? Did we have or get the resources we expected?  Were staff and others trained and prepared to do the work? Did we have the community support we expected?  Did we record what we did accurately and on time?  Did we monitor and evaluate as we intended?
• Implementation. Project implementation is the actual work of running it. Did we do what we intended? Did we serve or affect the number of people we proposed to? Did we use the methods we set out to use? Was the level of our activity what we intended (e.g., did we provide the number of hours of service we intended to)? Did we reach the population(s) we aimed at? What exactly did we provide or do? Did we make intentional or unintentional changes, and why?
• Outcomes. The project's outcomes are its results - what actually happened as a consequence of the project's existence. Did our work have the effects we hoped for? Did it have other, unforeseen effects? Were they positive or negative (or neither)? Do we know why we got the results we did?  What can we change, and how, to make our work more effective?

Many who write about participatory evaluation combine the first two of these areas into process evaluation, and add a third - impact evaluation - in addition to outcome evaluation.  Impact evaluation looks at the long-term results of a project, whether the project continues, or does its work and ends.

Rural development projects in the developing world, for example, often exist simply to pass on specific skills to local people, who are expected to then both practice those skills and teach them to others.  Once people have learned the skills - perhaps particular cultivation techniques, or water purification - the project ends.  If in five or ten years, an impact evaluation shows that the skills the project taught are not only still being practiced, but have spread, then the project's impact was both long-term and positive.

In order for these areas to be covered properly, evaluation has to start at the very beginning of the project, with assessment and planning.

In a participatory evaluation, stakeholders should be involved in:

• Naming and framing the problem or goal to be addressed
• Developing a theory of practice (process, logic model) for how to achieve success
• Identifying the questions to ask about the project and the best ways to ask them - these questions will identify what the project means to do, and therefore what should be evaluated

What's the real goal, for instance, of a program to introduce healthier foods in school lunches?  It could be simply to convince children to eat more fruits, vegetables, and whole grains.  It could be to get them to eat less junk food.  It could be to encourage weight loss in kids who are overweight or obese.  It could simply be to educate them about healthy eating, and to persuade them to be more adventurous eaters. The evaluation questions you ask both reflect and determine your goals for the program.  If you don't measure weight loss, for instance, then clearly that's not what you're aiming at.  If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods?  You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

• Collecting information about the project
• Making sense of that information
• Deciding what to celebrate, and what to adjust or change, based on information from the evaluation

Why would (and why wouldn't) you use participatory evaluation?

Why would you use participatory evaluation?  The short answer is that it's often the most effective way to find out what you need to know, both at the beginning of and throughout the course of a project. In addition, it carries benefits for both individual participants and the community that other methods don't.

Some of the major advantages of participatory evaluation:

• It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects. If stakeholders, including project beneficiaries, are involved from the beginning in determining what needs to be evaluated and why - not to mention what the focus of the project needs to be - you're much more likely to aim your work in the right direction, to correctly determine whether your project is effective or not, and to understand how to change it to make it moreso.
• It can get you information you wouldn't get otherwise. When project direction and evaluation depend, at least in part, on information from people in the community, that information will often be more forthcoming if it's asked for by someone familiar. Community people interviewing their friends and neighbors may get information that an outside person wouldn't be offered.
• It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff. Those implementing the project and those who are directly affected by it are most capable of sorting out the effective from the ineffective.
• It can tell you why something does or doesn't work.Beneficiaries are often able to explain exactly why they didn't respond to a particular technique or approach, thus giving you a better chance to adjust it properly.
• It results in a more effective project. For the reasons just described, you're much more apt to start out in the right direction, and to know when you need to change direction if you haven't. The consequence is a project that addresses the appropriate issues in the appropriate way, and accomplishes what it sets out to do.
• It empowers stakeholders. Participatory evaluation gives those who are often not consulted - line staff and beneficiaries particularly - the chance to be full partners in determining the direction and effectiveness of a project.
• It can provide a voice for those who are often not heard.Project beneficiaries are often low-income people with relatively low levels of education, who seldom have - and often don't think they have a right to - the chance to speak for themselves. By involving them from the beginning in project evaluation, you assure that their voices are heard, and they learn that they have the ability and the right to speak for themselves.
• It teaches skills that can be used in employment and other areas of life. In addition to the development of basic skills and specific research capabilities, participatory evaluation encourages critical thinking, collaboration, problem-solving, independent action, meeting deadlines...all skills valued by employers, and useful in family life, education, civic participation, and other areas.
• It bolsters self-confidence and self-esteem in those who may have little of either. This category can include not only project beneficiaries, but also others who may, because of circumstance, have been given little reason to believe in their own competence or value to society. The opportunity to engage in a meaningful and challenging activity, and to be treated as a colleague by professionals, can make a huge difference for folks who are seldom granted respect or given a chance to prove themselves.
• It demonstrates to people ways in which they can take more control of their lives. Working with professionals and others to complete a complex task with real-world consequences can show people how they can take action to influence people and events.
• It encourages stakeholder ownership of the project. If those involved feel the project is theirs, rather than something imposed on them by others, they'll work hard both in implementing it, and in conducting a thorough and informative evaluation in order to improve it.
• It can spark creativity in everyone involved. For those who've never been involved in anything similar, a participatory evaluation can be a revelation, opening doors to a whole new way of thinking and looking at the world. To those who have taken part in evaluation before, the opportunity to exchange ideas with people who may have new ways of looking at the familiar can lead to a fresh perspective on what may have seemed to be a settled issue.
• It encourages working collaboratively. For participatory evaluation to work well, it has to be viewed by everyone involved as a collaboration, where each participant brings specific tools and skills to the effort, and everyone is valued for what she can contribute. Collaboration of this sort not only leads to many of the advantages described above, but also fosters a more collaborative spirit for the future as well, leading to other successful community projects.
• It fits into a larger participatory effort. When community assessment and the planning of a project have been a collaboration among project beneficiaries, staff, and community members, it only makes sense to include evaluation in the overall plan, and to approach it in the same way as the rest of the project.  In order to conduct a good evaluation, its planning should be part of the overall planning of the project. Furthermore, participatory process generally matches well with the philosophy of community-based or grass roots groups or organizations.

With all these positive aspects, participatory evaluation carries some negative ones as well.  Whether its disadvantages outweigh its advantages depend on your circumstances, but whether you decide to engage in it or not, it's important to understand what kinds of drawbacks it might have.

The significant disadvantages of participatory evaluation include:

• It takes more time than conventional process. Because there are so many people with different perspectives involved, a number of whom have never taken part in planning or evaluation before, everything takes longer than if a professional evaluator or a team familiar with evaluation simply set up and conducted everything.  Decision-making involves a great deal of discussion, gathering people together may be difficult, evaluators need to be trained, etc.
• It takes the establishment of trust among all participants in the process. If you're starting something new (or, all too often, even if the project is ongoing), there are likely to be issues of class distinction, cultural differences, etc., dividing groups of stakeholders.These can lead to snags and slowdowns until they're resolved, which won't happen  overnight. It will take time and a good deal of conscious effort before all stakeholders feel comfortable and confident that their needs and culture are being addressed.
• You have to make sure that everyone's involved, not just "leaders" of various groups. All too often, "participatory" means the participation of an already-existing power structure. Most leaders are actually that - people who are most concerned with the best interests of the group, and whom others trust to represent them and steer them in the direction that best reflects those interests. Sometimes, however, leaders are those who push their way to the front, and try to confirm their own importance by telling others what to do.

By involving only leaders of a population or community, you run the risk of losing - or never gaining - the confidence and perspective of the rest of the population, which may dislike and distrust a leader of the second type, or may simply see themselves shut out of the process.. They may see the participatory evaluation as a function of authority, and be uninterested in taking part in it.  Working to recruit "regular" people as well as, or instead of, leaders may be an important step for the credibility of the process. But it's a lot of work and may be tough to sell.

• You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills. There are really a number of potential disadvantages here. The obvious one is that of time, which we've already raised - training takes time to prepare, time to implement, and time to sink in.  Another is the question of what kind of training participants will respond to. Still another concerns recruitment - will people be willing to put in the time necessary to prepare them for the process, let alone the time for the process itself?
• You have to get buy-in and commitment from participants. Given what evaluators will have to do, they need to be committed to the process, and to feel ownership of it. You have to structure both the training and the process itself to bring about this commitment.
• People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation. Poor people everywhere live on the edge, which means they're engaged in a delicate balancing act. The least tilt to one side or the other - a sick child, too many days of rain in a row - can cause a disruption that may result in an inability to participate on a given day, or at all.  If you're dealing with a rural village that's dependent on agriculture, for instance, an accident of weather can derail the whole process, either temporarily or permanently.
• You may have to be creative about how you get, record, and report information. If some of the participants in an evaluation are non- or semi-literate, or if participants speak a number of different languages (English, Spanish, and Lao, for instance), a way to record information will have to be found that everyone can understand, and that can, in turn, be understood by others outside the group.
• Funders and policy makers may not understand or believe in participatory evaluation. At worst, this can lose you your funding, or the opportunity to apply for funding.  At best, you'll have to spend a good deal of time and effort convincing funders and policy makers that participatory evaluation is a good idea, and obtaining their support for your effort.

Some of these disadvantages could also be seen as advantages: the training people receive blends in with their development of new skills that can be transferred to other areas of life, for instance; coming up with creative ways to express ideas benefits everyone; once funders and policy makers are persuaded of the benefits of participatory process and participatory evaluation, they may encourage others to employ it as well. Nonetheless, all of these potential negatives eat up time, which can be crucial.  If it's absolutely necessary that things happen quickly (which is true not nearly as often as most of us think it is), participatory evaluation is probably not the way to go.

When might you use participatory evaluation?

So when do you use participatory evaluation?  Some of the reasons you might decide it's the best choice for your purposes:

• When you're already committed to a participatory process for your project. Evaluation planning can be included and collaboratively designed as part of the overall project plan.
• When you have the time, or when results are more important than time. As should be obvious from the last part of this section, one of the biggest drawbacks to participatory evaluation is the time it takes. If time isn't what's most important, you can gain the advantages of a participatory evaluation without having to compensate for many of the disadvantages.
• When you can convince funders that it's a good idea. Funders may specify that they want an outside evaluation, or they may simply be dubious about the value of participatory evaluation.  In either case, you may have some persuading to do in order to be able to use a participatory process.  If you can get their support, however, funders may like the fact that participatory evaluation is often less expensive, and that it has added value in the form of empowerment and transferable skills.
• When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of. Political, social, and interpersonal factors in the community can skew the results of an evaluation, and without an understanding of those factors and their history, evaluators may have no idea that what they're finding out is colored in any way. Evaluators who are part of the community can help sort out the influence of these factors, and thus end up with a more accurate evaluation.
• When you need information that it will be difficult for anyone outside the community or population to get. When you know that members of the community or population in question are unwilling to speak freely to anyone from outside, participatory evaluation is a way to raise the chances that you'll get the information you need.
• When part of the goal of the project is to empower participants and help them develop transferable skills. Here, the participatory evaluation, as it should in any case, becomes a part of the project itself and its goals.
• When you want to bring the community or population together. In addition to fostering a collaborative spirit, as we've mentioned, a participatory evaluation can create opportunities for people who normally have little contact to work together and get to know one another. This familiarity can then carry over into other aspects of community life, and even change the social character of the community over the long term.

Who should be involved in participatory evaluation?

We've referred continually to stakeholders - the people who are directly affected by the project being evaluated. Who are the stakeholders? That varies from project to project, depending on the focus, the funding, the intended outcomes, etc.

There are a number of groups that are generally involved, however:

• Participants or beneficiaries. The people whom the project is meant to benefit. That may be a specific group (people with a certain medical condition, for instance), a particular population (recent Southeast Asian immigrants, residents of a particular area), or a whole community.  They may be actively receiving a service (e.g., employment training) or may simply stand to benefit from what the project is doing (violence prevention in a given neighborhood). These are usually the folks with the greatest stake in the project's success, and often the ones with the least experience of evaluation.
• Project line staff and/or volunteers. The people who actually do the work of carrying out the project. They may be professionals, people with specific skills, or community volunteers. They may work directly with project beneficiaries as mentors, teachers, or health care providers; or they may advocate for immigrant rights, identify open space to be preserved, or answer the phone and stuff envelopes. Whoever they are, they often know more about what they're doing than anyone else, and their lives can be affected by the project as much as those of participants or beneficiaries.
• Administrators. The people who coordinate the project or specific aspects of it. Like line staff and volunteers, they know a lot about what's going on, and they're intimately involved with the project every day.
• Outside evaluators, if they're involved. In many cases, outside evaluators are hired to run participatory evaluations. The need for their involvement is obvious.
• Community officials. You may need the support of community leaders, or you may simply want to give them and other participants the opportunity to get to know one another in a context that might lead to better understanding of community needs.
• Others whose lives are affected by the project. The definition of this group varies greatly from project to project. In general, it refers to people whose jobs or other aspects of their lives will be changed either by the functioning of the project itself, or by its outcomes.

An example would be landowners whose potential use of their land would be affected by an environmental initiative or a neighborhood plan.

How do you conduct a participatory evaluation?

Participatory evaluation encompasses elements of designing the project as well as evaluating it. What you evaluate depends on what you want to know and what you're trying to do.  Identifying the actual evaluation questions sets the course of the project just as surely as a standardized testing program guides teaching.  When these questions come out of an assessment in which stakeholders are involved, the evaluation is one phase of a community-based participatory research process.

A participatory evaluation really has two stages: One comprises finding and training stakeholders to act as participant evaluators.  The second - some of which may take place before or during the first stage - encompasses the planning and implementation of the project and its evaluation, and includes six steps:

• Naming and framing the issue
• Developing a theory of practice to address it
• Deciding what questions to ask, and how to ask them to get the information you need
• Collecting information
• Analyzing the information you've collected
• Using the information to celebrate what worked, and to adjust and improve the project

We'll examine both of these stages in detail.

Finding and training stakeholders to act as participant evaluators

Unfortunately, this stage isn't simply a matter of announcing a participatory evaluation and then sitting back while people beat down the doors to be part of it.  In fact, it may be one of the more difficult aspects of conducting a participatory evaluation.

Here's where the trust building we discussed earlier comes into play. The population you're working with may be distrustful of outsiders, or may be used to promises of involvement that turn out to be hollow or simply ignored. They may be used to being ignored in general, and/or offered services and programs that don't speak to their real needs. If you haven't already built a relationship to the point where people are willing to believe that you'll follow through on what you say, now is the time to do it. It may take some time and effort - you may have to prove that you'll still be there in six months - but it's worth it. You're much more likely to have a successful project, let alone a successful evaluation, if you have a relationship of mutual trust and respect.

But let's assume you have that step out of the way, and that you've established good relationships in the community and among the population you're working with, as well as with staff of the project. Let's assume as well that these folks know very little, if anything, about participatory evaluation. That means they'll need training in order to be effective.

If, in fact, your evaluation is part of a larger participatory effort, the question arises as to whether to simply employ the same team that did assessments and/or planned the project, perhaps with some additions, as evaluators.  That course of action has both pluses and minuses.  The team is already assembled, has developed a method of working together, has some training in research methods, etc., so that they can hit the ground running - obviously a plus. 
The fact that they have a big stake in seeing the project be successful can work either way: they may interpret their findings in the best possible light, or even ignore negative information; or they may be eager to see exactly where and how to adjust the work to make it go better.

Another issue is burnout.  Evaluation will mean more time in addition to what an assessment and planning team has already put in. While some may be more than willing to continue, many may be ready for a break (or may be moving on to another phase of their lives).  If the possibility of assembling a new team exists, it will give those who've had enough the chance to gracefully withdraw.

How you handle this question will depend on the attitudes of those involved, how many people you actually have to draw on (if the recruitment of the initial team was really difficult, you may not have a lot of choices), and what people committed to.

Recruit participant evaluators

There are many ways to accomplish this.  In some situations, it makes the most sense to put out a general call for volunteers; in others, to approach specific individuals who are likely - because of their commitment to the project or to the population - to be willing.  Alternatively, you might approach community leaders or stakeholders to suggest possible evaluators.

Some basic guidelines for recruitment include:

• Use communication channels and styles that reach the people you're aiming at
• Make your message as clear as possible
• Use plain English and/or whatever other language(s) the population uses
• Put your message where the audience is
• Approach potential participants individually where possible - if you can find people they know to recruit them, all the better
• Explain what people may gain from participation
• Be clear that they're being asked because they already have the qualities that are necessary for participation
• Encourage people, but also be honest about the amount and extent of what needs to be done
• Work out with participants what they're willing and able to do
• Try to arrange support - child care, for example - to make participation easier
• Ask people you've recruited to recommend - or recruit - others

In general, it's important for potential participant evaluators - particularly those whose connection to the project isn't related to their employment - to understand the commitment involved. An evaluation is likely to last a year, unless the project is considerably shorter than that, and while you might expect and plan for some dropouts, most of the team needs to be available for that long.

In order to make that commitment easier, discuss with participants what kinds of support they'll need in order to fulfill their commitment - child care and transportation, for instance - and try to find ways to provide it. Arrange meetings at times and places that are easiest for them (and keep the number of meetings to a minimum). For participants who are paid project staff, the evaluation should be considered part of their regular work, so that it isn't an extra, unpaid, burden that they feel they can't refuse.

Be careful to try to put together a team that's a cross-section of the stakeholder population. As we've already discussed, if you recruit only "leaders" from among the beneficiary population, for instance, you may create resentment in the rest of the group, not get a true perspective of the thinking or perceptions of that group, and defeat the purpose of the participatory nature of the evaluation as well. Even if the leaders are good representatives of the group, you may want to broaden your recruitment in the hopes of developing more community leadership, and empowering those who may not always be willing to speak out.

Train participant evaluators

Participants, depending on their backgrounds, may need training in a number of areas. They may have very little experience in attending and taking part in meetings, for instance, and may need to start there. They may benefit from an introduction to the idea of participatory evaluation, and how it works.  And they'll almost certainly need some training in data gathering and analysis.

How training gets carried out will vary with the needs and schedules of participants and the project. It may take place in small chunks over a relatively long period of time - weeks or months - might happen all at once in the course of a weekend retreat, or might be some combination. There's no right or wrong way here.  The first option will probably make it possible for more people to take part; the second allows for people to get to know one another and bond as a team, and a combination might allow for both.

By the same token, there are many training methods, any or all of which might be useful with a particular group. Training in meeting skills - knowing when and how to contribute and respond, following discussion, etc. - may best be accomplished through mentoring, rather than instruction. Interviewing skills may best be learned through roleplaying and other experiential techniques. Some training - how to approach local people, for example - might best come from participants themselves.

Some of the areas in which training might be necessary:

• The participatory evaluation process. How participatory evaluation works, its goals, the roles people may play in the process, what to expect.
• Meeting skills. Following discussion, listening skills, handling disagreement or conflict, contributing and responding appropriately, general ground rules and etiquette, etc.
• Interviewing. Putting people at ease, body language and tone of voice, asking open-ended and follow-up questions, recording what people say and other important information, handling interruptions and distractions, group interviews.
• Observation. Direct vs. participant observation, choosing appropriate times and places to observe, relevant information to include, recording observations.
• Recording information and reporting it to the group. What interviewees and those observed say and do, the non-verbal messages they send, who they are (age, situation, etc.), what the conditions were, the date and time, any other factors that influenced the interview or observation.

For people for whom writing isn't comfortable, where writing isn't feasible, or where language is a barrier, there should be alternative recording and reporting methods.  Drawings, maps, diagrams, audio recording, videos, or other imaginative ways of remembering exactly what was said or observed can be substituted, depending on the situation.  In interviews, if audio or video recording is going to be used, it's important to get the interviewee's permission first - before the interviewer shows up with the equipment, so that there are no misunderstandings.

• Analyzing information. Critical thinking, what kinds of things statistics tell you, other things to think about.

Planning and implementing the project and its evaluation

There's an assumption here that all phases of a project will be participatory, so that not only its evaluation, but its planning and the assessment that leads to it also involve stakeholders (not necessarily the same ones who act as evaluators). If stakeholders haven't been involved from the beginning, they don't have the deep understanding of the purposes and structure of a project that they'd have of one they've helped form. The evaluation that results, therefore, is likely to be less perceptive - and therefore less valuable - than one of a project they've been involved in from the start.

Naming and framing the problem or goal to be addressed

Identifying what you're evaluating defines what the project is meant to address and  accomplish. Community representatives and stakeholders, all those with something to gain or lose, work together to develop a shared vision and mission. By collecting information about community concerns and identifying available assets, communities can understand which issues to focus a project on.

Naming a problem or goal refers to identifying the issue that needs to be addressed.  Framing it has to do with the way we look at it.  If youth violence is conceived of as strictly a law enforcement problem, for instance, that framing implies specific ways of solving it: stricter laws, stricter enforcement, zero tolerance for violence, etc. If it's framed as a combination of a number of issues - availability of hand guns, unemployment and drug use among youth, social issues that lead to the formation of gangs, alienation and hopelessness in particular populations, poverty, etc. - then solutions may include employment and recreation programs, mentoring, substance abuse treatment, etc., as well as law enforcement.  The more we know about a problem, and the more different perspectives we can include in our thinking about it, the more accurately we can frame it, and the more likely we are to come up with an effective solution.

Developing a theory of practice to address the problem

How do you conduct a community effort so that it has a good chance of solving the problem at hand? Many communities and organizations answer this question by throwing uncoordinated programs at the problem, or by assuming a certain approach (law enforcement, as in our example, for instance) will take care of it.  In fact, you have to have a plan for creating, implementing, evaluating, adjusting, and maintaining a solution if you want it to work.

Whatever you call this plan - a theory of practice, a logic model, or simply an approach or process - it should be logical, consistent, consider all the areas that need to be coordinated in order for it to work, and give you an overall guideline and a  list of steps to follow in order to carry it out.

Once you've identified an issue, for instance, one possible theory of practice might be:

• Form a coalition of organizations, agencies, and community members concerned with the problem.
• Recruit and train a participatory research team which includes representatives of all stakeholder groups.
• The team collects both statistical and qualitative, first-hand information about the problem, and identifies community assets that might help in addressing it.
• Use the information you have to design a solution that takes into account the problem's complexity and context.

This might be a single program or initiative, or a coordinated, community-wide effort involving several organizations, the media, and individuals.  If it's closer to the latter, that's part of the complexity you have to take into account. Coordination has to be part of your solution, as do ways to get around the bureaucratic roadblocks that might occur and methods to find the financial and personnel resources you need.

• Implement the solution.
• Carry out monitoring and evaluation that will give you ongoing feedback about how well you're meeting objectives, and what you should change to improve your solution.
• Use the information from the evaluation to adjust and improve the solution.
• Go back to # 2 and do as much of it again as you need to until the problem is solved, or - more likely, since many community problems never actually disappear - indefinitely in order to maintain and increase your gains.

Deciding what evaluation questions to ask, and how to ask them to get the information you need

As we've discussed, choosing the evaluation questions essentially guides the work.  What you're really choosing here is what you're going to pay attention to. There could be significant results from your project that you're never aware of, because you didn't look for them - you didn't ask the questions to which those results would have been the answers. That's why it's so important to select questions carefully: they'll determine what you find.

Framing the problem is one element here - putting it in context, looking at it from all sides, stepping back from your own assumptions and biases to get a clearer and broader view of it. Another is envisioning the outcomes you want, and thinking about what needs to change, and how, in order to reach them.

Framing is important in this activity as well.  If you want simply to reduce youth violence, stricter laws and enforcement might seem like a reasonable solution, assuming you're willing to stick with them forever; if you want not only to reduce or eliminate youth violence, but to change the climate that fosters it (i.e., long term social change), the solution becomes much broader and requires, as we pointed out above, much more than law enforcement.  And a broader solution means more, and more complex, evaluation questions.
In the first case, evaluation questions might be limited to some variation of:  "Were there more arrests and convictions of youthful offenders for violent crimes in the time period studied, as compared to the last period for which there were records before the new solution was put in place?"  "Did youthful offenders receive harsher sentences than before?"  "Was there a reduction in violent incidents involving youth?"

Looking at the broader picture, in addition to some of those questions, there might be questions about counseling programs for youthful offenders to change their attitudes and to help ease their transition back to civil society, drug and alcohol treatment, control of handgun sales, changing community attitudes, etc.

Collecting information

This is the largest part, at least in time and effort, of implementing an evaluation.

Various evaluators, depending on the information needed, may conduct any or all of the following:

• Research into census or other public records, as well as news archives, library collections, the Internet, etc.
• Individual and/or group interviews
• Focus groups
• Community information-sharing sessions
• Surveys
• Direct or participant observation

In some cases - particularly with unschooled populations in developing countries - evaluators may have to find creative ways to draw out information.  In some cultures, maps, drawings, representations ("If this rock is the headman's house..."), or even storytelling may be more revealing than the answers to straightforward questions.

Analyzing the information you've collected

Once you've collected all the information you need, the next step is to make sense of it. What do the numbers mean? What do people's stories and opinions tell you about the project?  Did you carry out the process you'd planned?  If not, did it make a difference, positive or negative?

In some cases, these questions are relatively easy to answer. If there were particular objectives for serving people, or for beneficiaries' accomplishments, you can quickly find out whether they were met or not. (We set out to serve 75 people, and we actually served 82. We anticipated that 50 would complete the program, and 61 actually completed.)

In other cases, it's much harder to tell what your information means. What if approximately half of interviewees say the project was helpful to them, and the other half say the opposite? A result like that may leave you doing some detective work. (Is there any ethnic, racial, geographic, or cultural pattern as to who is positive and who is negative?  Whom did each group work with? Where did they experience the project, and how? Did members of each group have specific things in common?)

While collecting the information requires the most work and time, analyzing it is perhaps the most important step in conducting an evaluation. Your analysis tells you what you need to know in order to improve your project, and also gives you the evidence you need to make a case for continued funding and community support. It's important that it be done well, and that it make sense of odd results like that directly above. Here's where good training and good guidance in using critical thinking and other techniques come in.

In general, information-gathering and analysis should cover the three areas we discussed early in the section: process, implementation, and outcomes. The purpose here is both to provide information for improving the project and to provide accountability to funders and the community.

• Process. This concerns the logistics of the project. Was there good coordination and communication? Was the planning process participatory? Was the original timeline for each stage of the project - outreach, assessment, planning, implementation, evaluation - realistic? Were you able to find or hire the right people?  Did you find adequate funding and other resources? Was the space appropriate? Did members of the planning and evaluation teams work well together?  Did the people responsible do what they were expected to do? Did unexpected leaders emerge (in the planning group, for instance)?
• Implementation. Did you do what you set out to do - reach the number of people you expected to, use the methods you intended, provide the amount and kind of service or activity that you planned for?  This part of the evaluation is not meant to assess effectiveness, but only whether the project was carried out as planned - i.e, what you actually did, rather than what you accomplished as a result.  That comes next.
• Outcomes. What were the results of what you did?  Did what you hoped for take place?  If it did, how do you know it was a result of what you did, as opposed to some other factor(s)?  Were there unexpected results?  Were they negative or positive?  Why did this all happen?

Using the information to celebrate what worked, and to adjust and improve the project

While accountability is important - if the project has no effect at all, for example, it's just wasted effort - the real thrust of a good evaluation is formative. That means it's meant to provide information that can help to continue to form the project, reshape it to make it better. As a result, the overall questions when looking at process, implementation, and outcomes are: What worked well?  What didn't?  What changes would improve the project?

Answering these questions requires further analysis, but should allow you to improve the project considerably. In addition to dropping or changing and adjusting those elements of the project that didn't work well, don't neglect those that were successful. Nothing's perfect; even effective approaches can be made better.

Don't forget to celebrate your successes. Celebration recognizes the hard work of everyone involved, and the value of your effort. It creates community support, and strengthens the commitment of those involved. Perhaps most important, it makes clear that people working together can improve the quality of life in the community.

There's a final element to participatory research and evaluation that can't be ignored. Once you've started a project and made it successful, you have to maintain it. The participatory research and evaluation has to continue - perhaps not with the same team(s), but with teams representative of all stakeholders. Conditions change, and projects have to adapt. Research into those conditions and continued evaluation of your work will keep that work fresh and effective.

If your project is successful, you may think your work is done.Think again - community problems are only solved as long as the solutions are actively practiced. The moment you turn your back, the conditions you worked so hard to change can start to return to what existed before The work - supported by participatory research and evaluation - has to go on indefinitely to maintain and increase the gains you've made.

In Summary

Participatory evaluation is a part of participatory research. It involves stakeholders in a community project in setting evaluation criteria for it, collecting and analyzing data, and using the information gained to adjust and improve the project.

Participatory process brings in the all-important multiple perspectives of those most directly affected by the project, who are also most likely to be tied into community history and culture. The information and insights they contribute can be crucial in a project's effectiveness. In addition, their involvement encourages community buy-in, and can result in important gains in skills, knowledge, and self-confidence and self-esteem for the researchers. All in all, participatory evaluation creates a win-win situation.

Conducting a participatory evaluation involves several steps:

• Recruiting and training a stakeholder evaluation team
• Naming and framing the problem
• Developing a theory of practice to guide the process of the work
• Asking the right evaluation questions
• Collecting information
• Analyzing information
• Using the information to celebrate and adjust your work

The final step, as with so many of the community-building strategies and actions described in the Community Tool Box, is to keep at it.  Participatory research in general, and participatory evaluation in particular, has to continue as long as the work continues, in order to keep track of community needs and conditions, and to keep adjusting the project to make it more responsive and effective.  And the work often has to continue indefinitely in order to maintain progress and avoid sliding back into the conditions or attitudes that made the project necessary in the first place.

What is participatory evaluation?

___Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work

What are the advantages of using participatory evaluation?

___It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects

___It can get you information you wouldn't get otherwise

___It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff

___It can tell you why something does or doesn't work

___It results in a more effective project

___It empowers stakeholders

___It can provide a voice for those who are often not heard

___It teaches skills that can be used in employment and other areas of life

___It bolsters self-confidence and self-esteem in those who may have little of either

___It demonstrates to people ways in which they can take more control of their lives

___It encourages stakeholder ownership of the project

___It can spark creativity in everyone involved

___It encourages working collaboratively

___It fits into a larger participatory effort

What are the disadvantages of using participatory evaluation?

___It takes more time than conventional process

___It takes the establishment of trust among all participants in the process

___You have to make sure that everyone's involved, not just "leaders" of various groups

___You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills

___You have to get buy-in and commitment from participants

___People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation

___You may have to be creative about how you get, record, and report information

___Funders and policy makers may not understand or believe in participatory evaluation

When would you use participatory evaluation?

___When you're already committed to a participatory process for your project

___When you have the time, or when results are more important than time

___When you can convince funders that it's a good idea

___When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of

___When you need information that it will be difficult for anyone outside the community or population to get

___When part of the goal of the project is to empower participants and help them develop transferable skills

___When you want to bring the community or population together

Who should be involved in participatory evaluation?

___Participants or beneficiaries

___Project line staff and/or volunteers

___Administrators

___Outside evaluators, if they're involved

___Community officials

___Others whose lives are affected by the project

How do you conduct a participatory evaluation?

___Recruit stakeholders as participant evaluators

___Train evaluators

___Name and frame the issue

___Develop a theory of practice to address it

___Determine the evaluation questions

___Collect information

___Analyze the information

___Use your analysis to celebrate what worked and adjust the rest to improve the project

___Stick with it indefinitely

Tool 1: Team Data Collection - How to Do It Well

This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007. We have included her initial e-mail, and then added the summarized reponses to each of her questions.

From: Michelle Garred
Subject: Team data collection: how to do it well?

Dear colleagues,

I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines. Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.

Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well.

Among the many questions are:

• What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
• How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
• How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
• In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
• How would you cite team interviews in a Ph.D. thesis?
• What are the key resources I should review?

What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?

Advantages:

• Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.
• Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.

Pitfalls:

• More personalities and activities to coordinate means more opportunities for things to go wrong.
• Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.
• Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.

Pitfalls can be addressed through training:

• Need for consistent, systematic training of data collectors.
• Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.
• Conduct interviewing practice sessions before beginning actual interviews.
• During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.
• Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.
• Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?
• Demonstrated skills before graduation from training phase

And through regular team meetings:

• Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along.
• Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.
• Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.
• If not together on site, consider on-line conferencing.

How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?

Keep in mind that we will have a team of 5 people conducting semi-structured interviews and we will probably not use audio recording.

• Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.
• Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.
• Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked.  Here's one example of a card storming.
• If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.

How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?

Again, this can be addressed through team training and regular meetings . . .

• Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.
• Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.
• Revisit this issue during regular team meetings during the data collection phase.

And through wise design of the note-taking protocols:

• Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.
• Additionally, you might have team members write up a brief summary within 24 hours of completing the interview.
• Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.

In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?

I understand that team data collection is common in action research, but less common in a Ph.D. thesis. It would help me to hear some different opinions on this issue.

• For quantitative data, one might set a quota of random 10% of data reviewed for QA.
• I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.
• As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
• Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
  • Zuber-Skerritt, Ortrun, and Perry, Chad (2002)  Action research within organisations and university thesis writing. The Learning Organization, 9(4), 171-179.

How would you cite team interviews in a Ph.D. thesis?

• In table format within the document.
• I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
• I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
• When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
• Consider in advance whether you will use the participants' real names or pseudonyms.

Key resources

• See Madeline Church's doctoral thesis, Creating an Uncompromised place to belong - why do I find myself in networks? from the living theory section of Action Research.
• The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews. You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006).  "Finding common ground in team-based qualitative research using the convergent interviewing method." Qualitative Health Research, 16(8), 1145-1157.
  • There's a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006). Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.

• A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about. You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.

  Tool 2: Dabbling in Data

Dabbling in the Data from Public Profit is a practical guide that helps fill this gap. It shares easy-to-use, hands-on ways to explore and analyze data often found in mission-driven work. Whether you are just starting out or want to feel more confident with data, this guide gives you tools and ideas to make data exploration easier and more helpful.

The latest edition, released in late 2024, includes:

• 10 new activities
• More user-friendly tools
• Virtual adaptations
• Playlists for mixed methods analysis

Example 1: True Collaboration for a Win-win in Community-engaged Research

Dr. Marvia Jones, Director of the Kansas City Health Department, spoke on partnerships in community-engaged research at the Life Span Institute's 2024 Collaboratory on January 11th, 2024: 

When we think about the word "partner," we can think about a significant other. If I were to say, "What do you want out of your partnership with your significant other?", you may say things like, "you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year." Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things.

We want someone who is with us, who engages us from the very beginning of a process, so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this?

I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But as a partnership, through our partnership, we have to work through those solutions.

Those are some of the things that I think we look for in a community-based research partnership, having that flexibility and understanding. One of the things that I noticed, as I went from being a researcher to being in practice, is you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?”

What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we do need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information.

So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.