Section 11. Participatory Evaluation

 

Experienced community builders know that involving stakeholders -- the people directly connected to and affected by their projects -- in their work is tremendously important. It gives them the information they need to design, and to adjust or change, what they do to best meet the needs of the community and of the particular populations that an intervention or initiative is meant to benefit. This is particularly true in relation to evaluation.

As we have previously discussed, community-based participatory research can be employed in describing the community, assessing community issues and needs, finding and choosing best practices, and/or evaluation. We consider the topic of participatory evaluation important enough to give it a section of its own, and to show how it fits into the larger participatory research picture.

It's a good idea to build stakeholder participation into a project from the beginning. One of the best ways to choose the proper direction for your work is to involve stakeholders in identifying real community needs and the ways in which a project will have the greatest impact. One of the best ways to find out what kinds of effects your work is having on the people it's aimed at is to include people receiving or affected by information, services, or advocacy on your evaluation team.

Often, you can see most clearly what's actually happening through the eyes of those directly involved in it -- participants, staff, and others who are involved in taking part in and carrying out a program, initiative, or other project. Previously, we have discussed how you can involve those people in conducting research on the community and choosing issues to address and directions to go in. This section is about how you can involve them in the whole scope of the project, including its evaluation, and how that's likely to benefit the project's final outcomes.

What is participatory evaluation?

When most people think of evaluation, they think of something that happens at the end of a project -- that looks at the project after it's over and decides whether it was any good or not. Evaluation actually needs to be an integral part of any project from the beginning. Participatory evaluation involves all the stakeholders in a project -- those directly affected by it or by carrying it out -- in contributing to the understanding of it, and in applying that understanding to the improvement of the work.

Participatory evaluation, as we shall see, isn't simply a matter of asking stakeholders to take part. Involving everyone affected changes the whole nature of a project from something done for a group of people or a community to a partnership among participants, community members, and those implementing the project. Rather than people with less influence over decisions affecting them being acted upon, participants become copilots of a project, helping make sure that their real needs and those of the community are recognized and addressed. Professional evaluators, project staff, project participants, and other community members all become colleagues in an effort to improve the community's quality of life.

This approach to planning and evaluation isn't possible without mutual trust and respect. These have to develop over time, but that development is made more probable by starting out with an understanding of local culture and customs -- whether you're working in a lower-resource setting or in an American urban neighborhood. Respecting individuals and the knowledge and skills they have will go a long way toward promoting long-term trust and involvement.

The other necessary aspect of any participatory process is appropriate training for everyone involved. Some stakeholders may not even be aware that project research takes place; others may have no idea how to work alongside people from different backgrounds; and still others may not know what to do with evaluation results once they have them. We'll discuss all of these issues -- stakeholder involvement, establishing trust, and training -- as the section progresses.

The real purpose of an evaluation is not just to find out what happened, but to use the information to make the project better. In order to accomplish this, evaluation should include examining at least three areas:

• Process. The process of a project includes the planning and logistical activities needed to set up and run it. Did we do a proper assessment beforehand so we would know what the real needs were? Did we use the results of the assessment to identify and respond to those needs in the design of the project? Did we set up and run the project within the timelines and other structures that we intended? Did we involve the people we intended to? Did we have or get the resources we expected? Were staff and others trained and prepared to do the work? Did we have the community support we expected? Did we record what we did accurately and on time? Did we monitor and evaluate as we intended?
• Implementation. Project implementation is the actual work of running it. Did we do what we intended? Did we serve or affect the number of people we proposed to? Did we use the methods we set out to use? Was the level of our activity what we intended (e.g., did we provide the number of hours of service we intended to)? Did we reach the population(s) we aimed at? What exactly did we provide or do? Did we make intentional or unintentional changes, and why?
• Outcomes. The project's outcomes are its results -- what actually happened as a consequence of the project's existence. Did our work have the effects we hoped for? Did it have other, unforeseen effects? Were they positive or negative (or neither)? Do we know why we got the results we did? What can we change, and how, to make our work more effective?

Many who write about participatory evaluation combine the first two of these areas into process evaluation, and add a third -- impact evaluation -- in addition to outcome evaluation. Impact evaluation looks at the long-term results of a project, whether the project continues or does its work and ends.

Rural development projects in lower-resource settings, for example, often exist simply to pass on specific skills to local people, who are expected to then both practice those skills and teach them to others. Once people have learned the skills -- perhaps particular cultivation techniques or water purification -- the project ends. If, in five or ten years, an impact evaluation shows that the skills the project taught are not only still being practiced, but have spread, then the project's impact was both long-term and positive.

In order for these areas to be covered properly, evaluation has to start at the very beginning of the project, with assessment and planning. In a participatory evaluation, stakeholders should be involved in:

• Naming and framing the problem or goal to be addressed
• Developing a theory of practice (process, logic model) for how to achieve success
• Identifying the questions to ask about the project and the best ways to ask them -- these questions will identify what the project means to do, and therefore what should be evaluated

What's the real goal, for instance, of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss among children for whom excess weight is a health concern. It could simply be to educate them about healthy eating, and to persuade them to be more adventurous eaters. The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure changes in weight, for instance, then clearly that's not what you're aiming at. If you only look at an increase in children's consumption of healthy foods, you're not examining whether their overall eating patterns have changed. Is increased consumption of healthy foods still beneficial if other eating habits remain the same? You answer that question by what you choose to examine.

• Collecting information about the project
• Making sense of that information
• Deciding what to celebrate, and what to adjust or change, based on information from the evaluation

Why would (and why wouldn't) you use participatory evaluation?

Why would you use participatory evaluation? The short answer is that it's often the most effective way to find out what you need to know, both at the beginning of and throughout the course of a project. In addition, it carries benefits for both individual participants and the community that other methods don't.

Some of the major advantages of participatory evaluation:

• It gives you a better perspective on both the initial needs of the people the project is intended to benefit and on its ultimate effects. If stakeholders, including project participants, are involved from the beginning in determining what needs to be evaluated and why -- not to mention what the focus of the project needs to be -- you're much more likely to aim your work in the right direction, to correctly determine whether your project is effective or not, and to understand how to change it to make it more so.
• It can get you information you wouldn't get otherwise. When project direction and evaluation depend, at least in part, on information from people in the community, that information will often be more forthcoming if it's asked for by someone familiar. Community members interviewing their friends and neighbors may get information that an outside person wouldn't be offered.
• It tells you what worked and what didn't from the perspective of those most directly involved -- participants and staff. Those implementing the project and those who are directly affected by it are often well positioned to sort out the effective from the ineffective.

 

• It can tell you why something does or doesn't work. Participants are often able to explain exactly why they didn't respond to a particular technique or approach, thus giving you a better chance to adjust it properly.
• It results in a more effective project. For the reasons just described, you're much more apt to start out in the right direction, and to know when you need to change direction if you haven't. The consequence is a project that addresses the appropriate issues in the appropriate way, and accomplishes what it sets out to do.
• It empowers stakeholders. Participatory evaluation gives people who are often not consulted -- particularly line staff and participants -- the chance to be full partners in determining the direction and effectiveness of a project.
• It creates opportunities for perspectives that are often not included to be heard. Some project participants may have lower incomes or limited access to formal education and may have had fewer opportunities to influence decisions affecting them. By involving them from the beginning in project evaluation, you help ensure that their perspectives are included and that they have meaningful opportunities to speak for themselves.
• It teaches skills that can be used in employment and other areas of life. In addition to the development of basic skills and specific research capabilities, participatory evaluation encourages critical thinking, collaboration, problem-solving, independent action, meeting deadlines...all skills valued by employers, and useful in family life, education, civic participation, and other areas.
• It can strengthen self-confidence and self-esteem. This can include project participants and others who, because of circumstance, may have had few opportunities for their knowledge, abilities, or contributions to be recognized. The opportunity to engage in a meaningful and challenging activity, and to be treated as a colleague by professionals, can make a significant difference.
• It demonstrates ways in which people can take more control of decisions affecting their lives. Working with professionals and others to complete a complex task with real-world consequences can show people how they can take action to influence people and events.
• It encourages stakeholder ownership of the project. If those involved feel the project is theirs, rather than something imposed on them by others, they'll work hard both in implementing it and in conducting a thorough and informative evaluation in order to improve it.
• It can spark creativity in everyone involved. For those who've never been involved in anything similar, a participatory evaluation can be a revelation, opening doors to a whole new way of thinking and looking at the world. To those who have taken part in evaluation before, the opportunity to exchange ideas with people who may have new ways of looking at the familiar can lead to a fresh perspective on what may have seemed to be a settled issue.
• It encourages working collaboratively. For participatory evaluation to work well, it has to be viewed by everyone involved as a collaboration, where each participant brings specific tools and skills to the effort, and everyone is valued for what they can contribute. Collaboration of this sort not only leads to many of the advantages described above, but also fosters a more collaborative spirit for the future as well, leading to other successful community projects.
• It fits into a larger participatory effort. When community assessment and the planning of a project have been a collaboration among project participants, staff, and community members, it only makes sense to include evaluation in the overall plan, and to approach it in the same way as the rest of the project. In order to conduct a good evaluation, its planning should be part of the overall planning of the project. Furthermore, participatory process generally matches well with the philosophy of community-based or grassroots groups or organizations.

With all these positive aspects, participatory evaluation carries some negative ones as well. Whether its disadvantages outweigh its advantages depends on your circumstances, but whether you decide to engage in it or not, it's important to understand what kinds of drawbacks it might have.

The significant disadvantages of participatory evaluation include:

• It takes more time than a conventional process. Because there are so many people with different perspectives involved, a number of whom have never taken part in planning or evaluation before, everything takes longer than if a professional evaluator or a team familiar with evaluation simply set up and conducted everything. Decision-making involves a great deal of discussion, gathering people together may be difficult, evaluators need to be trained, etc.
• It takes the establishment of trust among all participants in the process. If you're starting something new (or, all too often, even if the project is ongoing), differences in social class, culture, and experience may divide groups of stakeholders. These can lead to snags and slowdowns until they're addressed, which won't happen overnight. It will take time and a good deal of conscious effort before all stakeholders feel comfortable and confident that their needs and culture are being addressed.
• You have to make sure that everyone's involved, not just "leaders" of various groups. All too often, "participatory" means the participation of an already-existing power structure. Most leaders are actually that -- people who are concerned with the best interests of the group, and whom others trust to represent them and steer them in the direction that best reflects those interests. Sometimes, however, people who hold leadership positions may not represent the full range of perspectives within a community.

By involving only leaders of a population or community, you run the risk of losing -- or never gaining -- the confidence and perspective of the rest of the population, which may distrust a particular leader or may simply see themselves shut out of the process. They may see the participatory evaluation as a function of authority and be uninterested in taking part in it. Working to recruit other community members as well as leaders may be an important step for the credibility of the process. But it's a lot of work and may be difficult.

• You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills. There are really a number of potential disadvantages here. The obvious one is that of time, which we've already raised -- training takes time to prepare, time to implement, and time to sink in. Another is the question of what kind of training participants will respond to. Still another concerns recruitment -- will people be willing to put in the time necessary to prepare them for the process, let alone the time for the process itself?
• You have to get buy-in and commitment from participants. Given what evaluators will have to do, they need to be committed to the process, and to feel ownership of it. You have to structure both the training and the process itself to bring about this commitment.
• People's lives -- illness, childcare and relationship problems, agricultural work, etc. -- may cause delays or get in the way of the evaluation. People facing economic hardship may have less flexibility when unexpected demands arise. A sick child, relationship difficulties, or too many days of rain in a row during an important agricultural period may disrupt someone's ability to participate on a given day, or at all. If you're working with a rural community that depends heavily on agriculture, for instance, an unexpected change in weather can derail the process temporarily or, in some cases, permanently.
• You may have to be creative about how you get, record, and report information. If some of the participants in an evaluation have limited literacy, or if participants speak a number of different languages (English, Spanish, and Lao, for instance), a way to record information will have to be found that everyone can understand, and that can, in turn, be understood by others outside the group.
• Funders and policymakers may not understand or believe in participatory evaluation. At worst, this can lose you your funding, or the opportunity to apply for funding. At best, you'll have to spend a good deal of time and effort convincing funders and policymakers that participatory evaluation is a good idea, and obtaining their support for your effort.

Some of these disadvantages could also be seen as advantages: the training people receive blends in with their development of new skills that can be transferred to other areas of life, for instance; coming up with creative ways to express ideas benefits everyone; once funders and policymakers are persuaded of the benefits of participatory process and participatory evaluation, they may encourage others to employ it as well. Nonetheless, all of these potential negatives take time, which can be crucial. If it's absolutely necessary that things happen quickly (which is true not nearly as often as most of us think it is), participatory evaluation is probably not the way to go.

When might you use participatory evaluation?

So when do you use participatory evaluation? Some of the reasons you might decide it's the best choice for your purposes:

• When you're already committed to a participatory process for your project. Evaluation planning can be included and collaboratively designed as part of the overall project plan.
• When you have the time, or when results are more important than time. As should be obvious from the last part of this section, one of the biggest drawbacks to participatory evaluation is the time it takes. If time isn't what's most important, you can gain the advantages of a participatory evaluation without having to compensate for many of the disadvantages.
• When you can convince funders that it's a good idea. Funders may specify that they want an outside evaluation, or they may simply be uncertain about the value of participatory evaluation. In either case, you may have some persuading to do in order to be able to use a participatory process. If you can get their support, however, funders may value the fact that participatory evaluation can have added benefits in the form of empowerment and transferable skills.
• When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of. Political, social, and interpersonal factors in the community can shape the results of an evaluation, and without an understanding of those factors and their history, evaluators may have no idea that what they're finding out is being influenced in any way. Evaluators who are part of the community can help sort out the influence of these factors, and thus end up with a more accurate evaluation.
• When you need information that it will be difficult for anyone outside the community or population to get. When you know that members of the community or population in question are unwilling to speak freely to anyone from outside, participatory evaluation is a way to increase the chances that you'll get the information you need.
• When part of the goal of the project is to empower participants and help them develop transferable skills. Here, the participatory evaluation, as it should in any case, becomes a part of the project itself and its goals.
• When you want to bring the community or population together. In addition to fostering a collaborative spirit, as we've mentioned, a participatory evaluation can create opportunities for people who normally have little contact to work together and get to know one another. This familiarity can then carry over into other aspects of community life, and even change the social character of the community over the long term.

Who should be involved in participatory evaluation?

We've referred continually to stakeholders -- the people who are directly affected by the project being evaluated. Who are the stakeholders? That varies from project to project, depending on the focus, the funding, the intended outcomes, etc. There are a number of groups that are generally involved, however:

• Participants or people intended to benefit. These are the people whom the project is meant to benefit. That may be a specific group (people with a certain medical condition, for instance), a particular population (recent immigrants from Southeast Asia, residents of a particular area), or a whole community. They may be actively receiving a service (e.g., employment training) or may simply stand to benefit from what the project is doing (violence prevention in a given neighborhood). These are usually the people with the greatest stake in the project's success, and often those with the least experience with evaluation.
• Project line staff and/or volunteers. The people who actually do the work of carrying out the project. They may be professionals, people with specific skills, or community volunteers. They may work directly with project participants as mentors, teachers, or health care providers; or they may advocate for immigrant rights, identify open space to be preserved, or answer the phone and prepare mailings. Whoever they are, they often know more about what they're doing than anyone else, and their lives can be affected by the project as much as those of participants.
• Administrators. The people who coordinate the project or specific aspects of it. Like line staff and volunteers, they know a lot about what's going on, and they're intimately involved with the project every day.
• Outside evaluators, if they're involved. In many cases, outside evaluators are hired to run participatory evaluations. The need for their involvement is obvious.
• Community officials. You may need the support of community leaders, or you may simply want to give them and other participants the opportunity to get to know one another in a context that might lead to better understanding of community needs.
• Others whose lives are affected by the project. The definition of this group varies greatly from project to project. In general, it refers to people whose jobs or other aspects of their lives will be changed either by the functioning of the project itself, or by its outcomes.

An example would be landowners whose potential use of their land would be affected by an environmental initiative or a neighborhood plan.

How do you conduct a participatory evaluation?

Participatory evaluation encompasses elements of designing the project as well as evaluating it. What you evaluate depends on what you want to know and what you're trying to do. Identifying the actual evaluation questions sets the course of the project just as surely as a standardized testing program guides teaching. When these questions come out of an assessment in which stakeholders are involved, the evaluation is one phase of a community-based participatory research process.

A participatory evaluation really has two stages: One comprises finding and training stakeholders to act as participant evaluators. The second -- some of which may take place before or during the first stage -- encompasses the planning and implementation of the project and its evaluation, and includes six steps:

• Naming and framing the issue
• Developing a theory of practice to address it
• Deciding what questions to ask, and how to ask them to get the information you need
• Collecting information
• Analyzing the information you've collected
• Using the information to celebrate what worked, and to adjust and improve the project

We'll examine both of these stages in detail.

Finding and training stakeholders to act as participant evaluators

Unfortunately, this stage isn't simply a matter of announcing a participatory evaluation and then sitting back while people beat down the doors to be part of it. In fact, it may be one of the more difficult aspects of conducting a participatory evaluation.

Here's where the trust building we discussed earlier comes into play. The population you're working with may be distrustful of outsiders, or may be used to promises of involvement that turn out to be hollow or simply ignored. They may be used to being ignored in general, and/or offered services and programs that don't speak to their real needs. If you haven't already built a relationship to the point where people are willing to believe that you'll follow through on what you say, now is the time to do it. It may take some time and effort -- you may have to prove that you'll still be there in six months -- but it's worth it. You're much more likely to have a successful project, let alone a successful evaluation, if you have a relationship of mutual trust and respect.

But let's assume you have that step out of the way, and that you've established good relationships in the community and among the population you're working with, as well as with staff of the project. Let's assume as well that these people know very little, if anything, about participatory evaluation. That means they'll need training in order to be effective.

If, in fact, your evaluation is part of a larger participatory effort, the question arises as to whether to simply employ the same team that did assessments and/or planned the project, perhaps with some additions, as evaluators. That course of action has both pluses and minuses. The team is already assembled, has developed a method of working together, has some training in research methods, etc., so that they can hit the ground running -- obviously a plus.
The fact that they have a big stake in seeing the project be successful can work either way: they may interpret their findings in the best possible light, or even ignore negative information; or they may be eager to see exactly where and how to adjust the work to make it go better.

Another issue is burnout. Evaluation will mean more time in addition to what an assessment and planning team has already put in. While some may be more than willing to continue, many may be ready for a break (or may be moving on to another phase of their lives). If the possibility of assembling a new team exists, it will give those who've had enough the chance to gracefully withdraw.

How you handle this question will depend on the attitudes of those involved, how many people you actually have to draw on (if the recruitment of the initial team was really difficult, you may not have a lot of choices), and what people committed to.

Recruit participant evaluators

There are many ways to accomplish this. In some situations, it makes the most sense to put out a general call for volunteers; in others, to approach specific individuals who are likely -- because of their commitment to the project or to the population -- to be willing. Alternatively, you might approach community leaders or stakeholders to suggest possible evaluators.

Some basic guidelines for recruitment include:

• Use communication channels and styles that reach the people you're aiming at
• Make your message as clear as possible
• Use plain language and/or whatever other language(s) the population uses
• Put your message where the audience is
• Approach potential participants individually where possible -- if you can find people they know to recruit them, all the better
• Explain what people may gain from participation
• Be clear that they're being asked because they already have qualities that are important for participation
• Encourage people, but also be honest about the amount and extent of what needs to be done
• Work out with participants what they're willing and able to do
• Try to arrange support -- childcare, for example -- to make participation easier
• Ask people you've recruited to recommend -- or recruit -- others

In general, it's important for potential participant evaluators -- particularly those whose connection to the project isn't related to their employment -- to understand the commitment involved. An evaluation is likely to last a year, unless the project is considerably shorter than that, and while you might expect and plan for some dropouts, most of the team needs to be available for that long.

In order to make that commitment easier, discuss with participants what kinds of support they'll need in order to fulfill their commitment -- childcare and transportation, for instance -- and try to find ways to provide it. Arrange meetings at times and places that are easiest for them (and keep the number of meetings to a minimum). For participants who are paid project staff, the evaluation should be considered part of their regular work, so that it isn't an extra, unpaid burden that they feel they can't refuse.

Be careful to try to put together a team that's a cross-section of the stakeholder population. As we've already discussed, if you recruit only "leaders" from among the participant population, for instance, you may create resentment in the rest of the group, fail to get a full perspective of the thinking or perceptions of that group, and defeat the purpose of the participatory nature of the evaluation as well. Even if the leaders are good representatives of the group, you may want to broaden your recruitment in the hopes of developing more community leadership and creating opportunities for people whose perspectives may not always be heard.

Train participant evaluators

Participants, depending on their backgrounds, may need training in a number of areas. They may have very little experience in attending and taking part in meetings, for instance, and may need to start there. They may benefit from an introduction to the idea of participatory evaluation and how it works. And they'll almost certainly need some training in data gathering and analysis.

How training gets carried out will vary with the needs and schedules of participants and the project. It may take place in small chunks over a relatively long period of time -- weeks or months -- might happen all at once in the course of a weekend retreat, or might be some combination. There's no right or wrong way here. The first option will probably make it possible for more people to take part; the second allows for people to get to know one another and bond as a team, and a combination might allow for both.

By the same token, there are many training methods, any or all of which might be useful with a particular group. Training in meeting skills -- knowing when and how to contribute and respond, following discussion, etc. -- may best be accomplished through mentoring, rather than instruction. Interviewing skills may best be learned through roleplaying and other experiential techniques. Some training -- how to approach local people, for example -- might best come from participants themselves.

Some of the areas in which training might be necessary:

• The participatory evaluation process. How participatory evaluation works, its goals, the roles people may play in the process, what to expect.
• Meeting skills. Following discussion, listening skills, handling disagreement or conflict, contributing and responding appropriately, general ground rules and etiquette, etc.
• Interviewing. Putting people at ease, body language and tone of voice, asking open-ended and follow-up questions, recording what people say and other important information, handling interruptions and distractions, group interviews.
• Observation. Direct vs. participant observation, choosing appropriate times and places to observe, relevant information to include, recording observations.
• Recording information and reporting it to the group. What interviewees and those observed say and do, the nonverbal messages they send, who they are (age, situation, etc.), what the conditions were, the date and time, and any other factors that influenced the interview or observation.

For people for whom writing isn't comfortable, where writing isn't feasible, or where language is a barrier, there should be alternative recording and reporting methods. Drawings, maps, diagrams, audio recordings, videos, or other imaginative ways of remembering exactly what was said or observed can be substituted, depending on the situation. In interviews, if audio or video recording is going to be used, it's important to get the interviewee's permission first -- before the interviewer shows up with the equipment, so that there are no misunderstandings.

Planning and implementing the project and its evaluation. There's an assumption here that all phases of a project will be participatory, so that not only its evaluation, but its planning and the assessment that leads to it also involve stakeholders (not necessarily the same ones who act as evaluators). If stakeholders haven't been involved from the beginning, they don't have the deep understanding of the purposes and structure of a project that they'd have of one they've helped form. The evaluation that results, therefore, is likely to be less perceptive -- and therefore less valuable -- than one of a project they've been involved in from the start.

Naming and framing the problem or goal to be addressed. Identifying what you're evaluating defines what the project is meant to address and accomplish. Community representatives and stakeholders, all those with something to gain or lose, work together to develop a shared vision and mission. By collecting information about community concerns and identifying available assets, communities can understand which issues to focus a project on.

Naming a problem or goal refers to identifying the issue that needs to be addressed. Framing it has to do with the way we look at it. If violence involving young people is conceived of as strictly a law enforcement problem, for instance, that framing implies specific ways of solving it: stricter laws, stricter enforcement, zero tolerance for violence, etc. If it's framed as a combination of a number of issues -- availability of handguns, unemployment and substance use among young people, social issues that lead to the formation of gangs, alienation and hopelessness in particular populations, poverty, etc. -- then solutions may include employment and recreation programs, mentoring, substance use treatment, etc., as well as law enforcement. The more we know about a problem, and the more different perspectives we can include in our thinking about it, the more accurately we can frame it, and the more likely we are to come up with an effective solution.

Developing a theory of practice to address the problem. How do you conduct a community effort so that it has a good chance of solving the problem at hand? Many communities and organizations answer this question by throwing uncoordinated programs at the problem, or by assuming a certain approach (law enforcement, as in our example, for instance) will take care of it. In fact, you have to have a plan for creating, implementing, evaluating, adjusting, and maintaining a solution if you want it to work.

Whatever you call this plan -- a theory of practice, a logic model, or simply an approach or process -- it should be logical, consistent, consider all the areas that need to be coordinated in order for it to work, and give you an overall guideline and a list of steps to follow in order to carry it out.

Once you've identified an issue, for instance, one possible theory of practice might be:

• Form a coalition of organizations, agencies, and community members concerned with the problem.
• Recruit and train a participatory research team which includes representatives of all stakeholder groups.
• The team collects both statistical and qualitative, firsthand information about the problem, and identifies community assets that might help in addressing it.
• Use the information you have to design a solution that takes into account the problem's complexity and context.

This might be a single program or initiative, or a coordinated, community-wide effort involving several organizations, the media, and individuals. If it's closer to the latter, that's part of the complexity you have to take into account. Coordination has to be part of your solution, as do ways to get around the bureaucratic roadblocks that might occur and methods to find the financial and personnel resources you need.

• Implement the solution.
• Carry out monitoring and evaluation that will give you ongoing feedback about how well you're meeting objectives, and what you should change to improve your solution.
• Use the information from the evaluation to adjust and improve the solution.
• Go back to #2 and do as much of it again as you need to until the problem is solved, or -- more likely, since many community problems never actually disappear -- indefinitely in order to maintain and increase your gains.

Deciding what evaluation questions to ask, and how to ask them to get the information you need. As we've discussed, choosing the evaluation questions essentially guides the work. What you're really choosing here is what you're going to pay attention to. There could be significant results from your project that you're never aware of, because you didn't look for them -- you didn't ask the questions to which those results would have been the answers. That's why it's so important to select questions carefully: they'll determine what you find.

Framing the problem is one element here -- putting it in context, looking at it from all sides, stepping back from your own assumptions and biases to get a clearer and broader view of it. Another is envisioning the outcomes you want, and thinking about what needs to change, and how, in order to reach them.

Framing is important in this activity as well. If you want simply to reduce violence involving young people, stricter laws and enforcement might seem like a reasonable solution, assuming you're willing to stick with them forever; if you want not only to reduce or eliminate that violence, but to change the climate that fosters it (i.e., long-term social change), the solution becomes much broader and requires, as we pointed out above, much more than law enforcement. And a broader solution means more, and more complex, evaluation questions.
In the first case, evaluation questions might be limited to some variation of: "Were there more arrests and convictions of young people for violent crimes in the time period studied, as compared to the last period for which there were records before the new solution was put in place?" "Did young people convicted of violent offenses receive harsher sentences than before?" "Was there a reduction in violent incidents involving young people?"

Looking at the broader picture, in addition to some of those questions, there might be questions about counseling programs for young people who committed offenses to change their attitudes and help ease their transition back into the community, substance use treatment, control of handgun sales, changing community attitudes, etc.

Collecting information. This is the largest part, at least in time and effort, of implementing an evaluation. Various evaluators, depending on the information needed, may conduct any or all of the following:

• Research into census or other public records, as well as news archives, library collections, online sources, etc.
• Individual and/or group interviews
• Focus groups
• Community information-sharing sessions
• Surveys
• Direct or participant observation

In some cases -- particularly among populations with limited access to formal education in lower-resource settings -- evaluators may have to find creative ways to draw out information. In some cultures, maps, drawings, representations ("If this rock is the headman's house..."), or even storytelling may be more revealing than the answers to straightforward questions.

Analyzing the information you've collected. Once you've collected all the information you need, the next step is to make sense of it. What do the numbers mean? What do people's stories and opinions tell you about the project? Did you carry out the process you'd planned? If not, did it make a difference, positive or negative?

In some cases, these questions are relatively easy to answer. If there were particular objectives for serving people, or for participants' accomplishments, you can quickly find out whether they were met or not. (We set out to serve 75 people, and we actually served 82. We anticipated that 50 would complete the program, and 61 actually completed.)

In other cases, it's much harder to tell what your information means. What if approximately half of interviewees say the project was helpful to them, and the other half say the opposite? A result like that may leave you doing some detective work. (Is there any ethnic, racial, geographic, or cultural pattern as to who is positive and who is negative? Whom did each group work with? Where did they experience the project, and how? Did members of each group have specific things in common?)

While collecting the information requires the most work and time, analyzing it is perhaps the most important step in conducting an evaluation. Your analysis tells you what you need to know in order to improve your project, and also gives you the evidence you need to make a case for continued funding and community support. It's important that it be done well, and that it make sense of odd results like that directly above. Here's where good training and good guidance in using critical thinking and other techniques come in.

In general, information-gathering and analysis should cover the three areas we discussed early in the section: process, implementation, and outcomes. The purpose here is both to provide information for improving the project and to provide accountability to funders and the community.

• Process. This concerns the logistics of the project. Was there good coordination and communication? Was the planning process participatory? Was the original timeline for each stage of the project -- outreach, assessment, planning, implementation, evaluation -- realistic? Were you able to find or hire the right people? Did you find adequate funding and other resources? Was the space appropriate? Did members of the planning and evaluation teams work well together? Did the people responsible do what they were expected to do? Did unexpected leaders emerge (in the planning group, for instance)?
• Implementation. Did you do what you set out to do -- reach the number of people you expected to, use the methods you intended, provide the amount and kind of service or activity that you planned for? This part of the evaluation is not meant to assess effectiveness, but only whether the project was carried out as planned -- i.e., what you actually did, rather than what you accomplished as a result. That comes next.
• Outcomes. What were the results of what you did? Did what you hoped for take place? If it did, how do you know it was a result of what you did, as opposed to some other factor(s)? Were there unexpected results? Were they negative or positive? Why did this all happen?

Using the information to celebrate what worked, and to adjust and improve the project. While accountability is important -- if the project has no effect at all, for example, it's just wasted effort -- the real thrust of a good evaluation is formative. That means it's meant to provide information that can help to continue to form the project, reshape it to make it better. As a result, the overall questions when looking at process, implementation, and outcomes are: What worked well? What didn't? What changes would improve the project?

Answering these questions requires further analysis, but should allow you to improve the project considerably. In addition to dropping or changing and adjusting those elements of the project that didn't work well, don't neglect those that were successful. Nothing's perfect; even effective approaches can be made better.

Don't forget to celebrate your successes. Celebration recognizes the hard work of everyone involved, and the value of your effort. It creates community support, and strengthens the commitment of those involved. Perhaps most important, it makes clear that people working together can improve the quality of life in the community.

There's a final element to participatory research and evaluation that can't be ignored. Once you've started a project and made it successful, you have to maintain it. The participatory research and evaluation has to continue -- perhaps not with the same team(s), but with teams representative of all stakeholders. Conditions change, and projects have to adapt. Research into those conditions and continued evaluation of your work will keep that work fresh and effective.

If your project is successful, you may think your work is done. Think again -- community problems are only solved as long as the solutions are actively practiced. The moment you turn your back, the conditions you worked so hard to change can start to return to what existed before. The work -- supported by participatory research and evaluation -- has to go on indefinitely to maintain and increase the gains you've made.

In Summary

Participatory evaluation is a part of participatory research. It involves stakeholders in a community project in setting evaluation criteria for it, collecting and analyzing data, and using the information gained to adjust and improve the project.

Participatory process brings in the all-important multiple perspectives of those most directly affected by the project, who are also most likely to be connected to community history and culture. The information and insights they contribute can be crucial to a project's effectiveness. In addition, their involvement encourages community buy-in, and can result in important gains in skills, knowledge, self-confidence, and self-esteem for participant evaluators. All in all, participatory evaluation can create benefits for both the project and the people involved.

Conducting a participatory evaluation involves several steps:

• Recruiting and training a stakeholder evaluation team
• Naming and framing the problem
• Developing a theory of practice to guide the process of the work
• Asking the right evaluation questions
• Collecting information
• Analyzing information
• Using the information to celebrate and adjust your work

The final step, as with so many of the community-building strategies and actions described in the Community Tool Box, is to keep at it. Participatory research in general, and participatory evaluation in particular, has to continue as long as the work continues, in order to keep track of community needs and conditions, and to keep adjusting the project to make it more responsive and effective. And the work often has to continue indefinitely in order to maintain progress and avoid sliding back into the conditions or attitudes that made the project necessary in the first place.