| Community Tool Box

This section is adapted from the article "Recommended Framework for Program Evaluation in Public Health Practice," by Bobby Milstein, Scott Wetterhall, and the CDC Evaluation Working Group.

Around the world, there exist many programs and interventions developed to improve conditions in local communities. Communities come together to reduce the level of violence that exists, to work for safe, affordable housing for everyone, or to help more students do well in school, to give just a few examples.

But how do we know whether these programs are working? If they are not effective, and even if they are, how can we improve them to make them better for local communities? And finally, how can an organization make intelligent choices about which promising programs are likely to work best in their community?

Over the past years, there has been a growing trend towards the better use of evaluation to understand and improve practice.The systematic use of evaluation has solved many problems and helped countless community-based organizations do what they do better.

Despite an increased understanding of the need for - and the use of - evaluation, however, a basic agreed-upon framework for program evaluation has been lacking. In 1997, scientists at the United States Centers for Disease Control and Prevention (CDC) recognized the need to develop such a framework. As a result of this, the CDC assembled an Evaluation Working Group comprised of experts in the fields of public health and evaluation. Members were asked to develop a framework that summarizes and organizes the basic elements of program evaluation. This Community Tool Box section describes the framework resulting from the Working Group's efforts.

Before we begin, however, we'd like to offer some definitions of terms that we will use throughout this section.

By evaluation, we mean the systematic investigation of the merit, worth, or significance of an object or effort. Evaluation practice has changed dramatically during the past three decades - new methods and approaches have been developed and it is now used for increasingly diverse projects and audiences.

Throughout this section, the term program is used to describe the object or effort that is being evaluated. It may apply to any action with the goal of improving outcomes for whole communities, for more specific sectors (e.g., schools, work places), or for sub-groups (e.g., youth, people experiencing violence or HIV/AIDS). This definition is meant to be very broad.

Examples of different types of programs include:

• Direct service interventions (e.g., a program that offers free breakfast to improve nutrition for grade school children)
• Community mobilization efforts (e.g., organizing a boycott of California grapes to improve the economic well-being of farm workers)
• Research initiatives (e.g., an effort to find out whether inequities in health outcomes based on race can be reduced)
• Surveillance systems (e.g., whether early detection of school readiness improves educational outcomes)
• Advocacy work (e.g., a campaign to influence the state legislature to pass legislation regarding tobacco control)
• Social marketing campaigns (e.g., a campaign in the Third World encouraging mothers to breast-feed their babies to reduce infant mortality)
• Infrastructure building projects (e.g., a program to build the capacity of state agencies to support community development initiatives)
• Training programs (e.g., a job training program to reduce unemployment in urban neighborhoods)
• Administrative systems (e.g., an incentive program to improve efficiency of health services)

Program evaluation - the type of evaluation discussed in this section - is an essential organizational practice for all types of community health and development work. It is a way to evaluate the specific projects and activities community groups may take part in, rather than to evaluate an entire organization or comprehensive community initiative.

Stakeholders refer to those who care about the program or effort. These may include those presumed to benefit (e.g., children and their parents or guardians), those with particular influence (e.g., elected or appointed officials), and those who might support the effort (i.e., potential allies) or oppose it (i.e., potential opponents). Key questions in thinking about stakeholders are: Who cares? What do they care about?

This section presents a framework that promotes a common understanding of program evaluation. The overall goal is to make it easier for everyone involved in community health and development work to evaluate their efforts.

Why evaluate community health and development programs?

The type of evaluation we talk about in this section can be closely tied to everyday program operations. Our emphasis is on practical, ongoing evaluation that involves program staff, community members, and other stakeholders, not just evaluation experts. This type of evaluation offers many advantages for community health and development professionals.

For example, it complements program management by:

• Helping to clarify program plans
• Improving communication among partners
• Gathering the feedback needed to improve and be accountable for program effectiveness

It's important to remember, too, that evaluation is not a new activity for those of us working to improve our communities. In fact, we assess the merit of our work all the time when we ask questions, consult partners, make assessments based on feedback, and then use those judgments to improve our work. When the stakes are low, this type of informal evaluation might be enough. However, when the stakes are raised - when a good deal of time or money is involved, or when many people may be affected - then it may make sense for your organization to use evaluation procedures that are more formal, visible, and justifiable.

How do you evaluate a specific program?

Before your organization starts with a program evaluation, your group should be very clear about the answers to the following questions:

• What will be evaluated?
• What criteria will be used to judge program performance?
• What standards of performance on the criteria must be reached for the program to be considered successful?
• What evidence will indicate performance on the criteria relative to the standards?
• What conclusions about program performance are justified based on the available evidence?

To clarify the meaning of each, let's look at some of the answers for Drive Smart, a hypothetical program begun to stop drunk driving.

• What will be evaluated?
  • Drive Smart, a program focused on reducing drunk driving through public education and intervention.
• What criteria will be used to judge program performance?
  • The number of community residents who are familiar with the program and its goals
  • The number of people who use "Safe Rides" volunteer taxis to get home
  • The percentage of people who report drinking and driving
  • The reported number of single car night time crashes (This is a common way to try to determine if the number of people who drive drunk is changing)
• What standards of performance on the criteria must be reached for the program to be considered successful?
  • 80% of community residents will know about the program and its goals after the first year of the program
  • The number of people who use the "Safe Rides" taxis will increase by 20% in the first year
  • The percentage of people who report drinking and driving will decrease by 20% in the first year
  • The reported number of single car night time crashes will decrease by 10 % in the program's first two years
• What evidence will indicate performance on the criteria relative to the standards?
  • A random telephone survey will demonstrate community residents' knowledge of the program and changes in reported behavior
  • Logs from "Safe Rides" will tell how many people use their services
  • Information on single-car nighttime crashes will be gathered from police records
• What conclusions about program performance are justified based on the available evidence?
  • Are the changes we have seen in the level of drunk driving due to our efforts, or something else? Or (if no or insufficient change in behavior or outcome,)
  • Should Drive Smart change what it is doing, or have we just not waited long enough to see results?

The following framework provides an organized approach to answer these questions.

A framework for program evaluation

Program evaluation offers a way to understand and improve community health and development practice using methods that are useful, feasible, proper, and accurate. The framework described below is a practical non-prescriptive tool that summarizes in a logical order the important elements of program evaluation.

The framework contains two related dimensions:

• Steps in evaluation practice, and
• Standards for "good" evaluation.

The six connected steps of the framework are actions that should be a part of any evaluation. Although in practice the steps may be encountered out of order, it will usually make sense to follow them in the recommended sequence. That's because earlier steps provide the foundation for subsequent progress. Thus, decisions about how to carry out a given step should not be finalized until prior steps have been thoroughly addressed.

However, these steps are meant to be adaptable, not rigid. Sensitivity to each program's unique context (for example, the program's history and organizational climate) is essential for sound evaluation. They are intended to serve as starting points around which community organizations can tailor an evaluation to best meet their needs.

• Engage stakeholders
• Describe the program
• Focus the evaluation design
• Gather credible evidence
• Justify conclusions
• Ensure use and share lessons learned

Understanding and adhering to these basic steps will improve most evaluation efforts.

The second part of the framework is a basic set of standards to assess the quality of evaluation activities. There are 30 specific standards, organized into the following four groups:

• Utility
• Feasibility
• Propriety
• Accuracy

These standards help answer the question, "Will this evaluation be a 'good' evaluation?" They are recommended as the initial criteria by which to judge the quality of the program evaluation efforts.

Engage Stakeholders

Stakeholders are people or organizations that have something to gain or lose from what will be learned from an evaluation, and also in what will be done with that knowledge. Evaluation cannot be done in isolation. Almost everything done in community health and development work involves partnerships - alliances among different organizations, board members, those affected by the problem, and others. Therefore, any serious effort to evaluate a program must consider the different values held by the partners. Stakeholders must be part of the evaluation to ensure that their unique perspectives are understood. When stakeholders are not appropriately involved, evaluation findings are likely to be ignored, criticized, or resisted.

However, if they are part of the process, people are likely to feel a good deal of ownership for the evaluation process and results. They will probably want to develop it, defend it, and make sure that the evaluation really works.

That's why this evaluation cycle begins by engaging stakeholders. Once involved, these people will help to carry out each of the steps that follows.

Three principle groups of stakeholders are important to involve:

• People or organizations involved in program operations may include community members, sponsors, collaborators, coalition partners, funding officials, administrators, managers, and staff.
• People or organizations served or affected by the program may include clients, family members, neighborhood organizations, academic institutions, elected and appointed officials, advocacy groups, and community residents. Individuals who are openly skeptical of or antagonistic toward the program may also be important to involve. Opening an evaluation to opposing perspectives and enlisting the help of potential program opponents can strengthen the evaluation's credibility.

Likewise, individuals or groups who could be adversely or inadvertently affected by changes arising from the evaluation have a right to be engaged. For example, it is important to include those who would be affected if program services were expanded, altered, limited, or ended as a result of the evaluation.

• Primary intended users of the evaluation are the specific individuals who are in a position to decide and/or do something with the results.They shouldn't be confused with primary intended users of the program, although some of them should be involved in this group. In fact, primary intended users should be a subset of all of the stakeholders who have been identified. A successful evaluation will designate primary intended users, such as program staff and funders, early in its development and maintain frequent interaction with them to be sure that the evaluation specifically addresses their values and needs.

The amount and type of stakeholder involvement will be different for each program evaluation. For instance, stakeholders can be directly involved in designing and conducting the evaluation. They can be kept informed about progress of the evaluation through periodic meetings, reports, and other means of communication.

It may be helpful, when working with a group such as this, to develop an explicit process to share power and resolve conflicts. This may help avoid overemphasis of values held by any specific stakeholder.

Describe the Program

A program description is a summary of the intervention being evaluated. It should explain what the program is trying to accomplish and how it tries to bring about those changes. The description will also illustrate the program's core components and elements, its ability to make changes, its stage of development, and how the program fits into the larger organizational and community environment.

How a program is described sets the frame of reference for all future decisions about its evaluation. For example, if a program is described as, "attempting to strengthen enforcement of existing laws that discourage underage drinking," the evaluation might be very different than if it is described as, "a program to reduce drunk driving by teens." Also, the description allows members of the group to compare the program to other similar efforts, and it makes it easier to figure out what parts of the program brought about what effects.

Moreover, different stakeholders may have different ideas about what the program is supposed to achieve and why. For example, a program to reduce teen pregnancy may have some members who believe this means only increasing access to contraceptives, and other members who believe it means only focusing on abstinence.

Evaluations done without agreement on the program definition aren't likely to be very useful. In many cases, the process of working with stakeholders to develop a clear and logical program description will bring benefits long before data are available to measure program effectiveness.

There are several specific aspects that should be included when describing a program.

Statement of need

A statement of need describes the problem, goal, or opportunity that the program addresses; it also begins to imply what the program will do in response. Important features to note regarding a program's need are: the nature of the problem or goal, who is affected, how big it is, and whether (and how) it is changing.

Expectations

Expectations are the program's intended results. They describe what the program has to accomplish to be considered successful. For most programs, the accomplishments exist on a continuum (first, we want to accomplish X... then, we want to do Y...). Therefore, they should be organized by time ranging from specific (and immediate) to broad (and longer-term) consequences. For example, a program's vision, mission, goals, and objectives, all represent varying levels of specificity about a program's expectations.

Activities

Activities are everything the program does to bring about changes. Describing program components and elements permits specific strategies and actions to be listed in logical sequence. This also shows how different program activities, such as education and enforcement, relate to one another. Describing program activities also provides an opportunity to distinguish activities that are the direct responsibility of the program from those that are conducted by related programs or partner organizations. Things outside of the program that may affect its success, such as harsher laws punishing businesses that sell alcohol to minors, can also be noted.

Resources

Resources include the time, talent, equipment, information, money, and other assets available to conduct program activities. Reviewing the resources a program has tells a lot about the amount and intensity of its services. It may also point out situations where there is a mismatch between what the group wants to do and the resources available to carry out these activities. Understanding program costs is a necessity to assess the cost-benefit ratio as part of the evaluation.

Stage of development

A program's stage of development reflects its maturity. All community health and development programs mature and change over time. People who conduct evaluations, as well as those who use their findings, need to consider the dynamic nature of programs. For example, a new program that just received its first grant may differ in many respects from one that has been running for over a decade.

At least three phases of development are commonly recognized: planning, implementation, and effects or outcomes. In the planning stage, program activities are untested and the goal of evaluation is to refine plans as much as possible. In the implementation phase, program activities are being field tested and modified; the goal of evaluation is to see what happens in the "real world" and to improve operations. In the effects stage, enough time has passed for the program's effects to emerge; the goal of evaluation is to identify and understand the program's results, including those that were unintentional.

Context

A description of the program's context considers the important features of the environment in which the program operates. This includes understanding the area's history, geography, politics, and social and economic conditions, and also what other organizations have done. A realistic and responsive evaluation is sensitive to a broad range of potential influences on the program. An understanding of the context lets users interpret findings accurately and assess their generalizability. For example, a program to improve housing in an inner-city neighborhood might have been a tremendous success, but would likely not work in a small town on the other side of the country without significant adaptation.

Logic model

A logic model synthesizes the main program elements into a picture of how the program is supposed to work. It makes explicit the sequence of events that are presumed to bring about change. Often this logic is displayed in a flow-chart, map, or table to portray the sequence of steps leading to program results.

Creating a logic model allows stakeholders to improve and focus program direction. It reveals assumptions about conditions for program effectiveness and provides a frame of reference for one or more evaluations of the program. A detailed logic model can also be a basis for estimating the program's effect on endpoints that are not directly measured. For example, it may be possible to estimate the rate of reduction in disease from a known number of persons experiencing the intervention if there is prior knowledge about its effectiveness.

The breadth and depth of a program description will vary for each program evaluation. And so, many different activities may be part of developing that description. For instance, multiple sources of information could be pulled together to construct a well-rounded description. The accuracy of an existing program description could be confirmed through discussion with stakeholders. Descriptions of what's going on could be checked against direct observation of activities in the field. A narrow program description could be fleshed out by addressing contextual factors (such as staff turnover, inadequate resources, political pressures, or strong community participation) that may affect program performance.

Focus the Evaluation Design

By focusing the evaluation design, we mean doing advance planning about where the evaluation is headed, and what steps it will take to get there. It isn't possible or useful for an evaluation to try to answer all questions for all stakeholders; there must be a focus. A well-focused plan is a safeguard against using time and resources inefficiently.

Depending on what you want to learn, some types of evaluation will be better suited than others. However, once data collection begins, it may be difficult or impossible to change what you are doing, even if it becomes obvious that other methods would work better. A thorough plan anticipates intended uses and creates an evaluation strategy with the greatest chance to be useful, feasible, proper, and accurate.

Among the issues to consider when focusing an evaluation are:

Purpose

Purpose refers to the general intent of the evaluation. A clear purpose serves as the basis for the design, methods, and use of the evaluation. Taking time to articulate an overall purpose will stop your organization from making uninformed decisions about how the evaluation should be conducted and used.

There are at least four general purposes for which a community group might conduct an evaluation:

• To gain insight.This happens, for example, when deciding whether to use a new approach (e.g., would a neighborhood watch program work for our community?) Knowledge from such an evaluation will provide information about its practicality. For a developing program, information from evaluations of similar programs can provide the insight needed to clarify how its activities should be designed.
• To improve how things get done.This is appropriate in the implementation stage when an established program tries to describe what it has done. This information can be used to describe program processes, to improve how the program operates, and to fine-tune the overall strategy. Evaluations done for this purpose include efforts to improve the quality, effectiveness, or efficiency of program activities.
• To determine what the effects of the program are. Evaluations done for this purpose examine the relationship between program activities and observed consequences. For example, are more students finishing high school as a result of the program? Programs most appropriate for this type of evaluation are mature programs that are able to state clearly what happened and who it happened to. Such evaluations should provide evidence about what the program's contribution was to reaching longer-term goals such as a decrease in child abuse or crime in the area. This type of evaluation helps establish the accountability, and thus, the credibility, of a program to funders and to the community.
• To affect those who participate in it. The logic and reflection required of evaluation participants can itself be a catalyst for self-directed change. And so, one of the purposes of evaluating a program is for the process and results to have a positive influence. Such influences may:
  • Empower program participants (for example, being part of an evaluation can increase community members' sense of control over the program);
  • Supplement the program (for example, using a follow-up questionnaire can reinforce the main messages of the program);
  • Promote staff development (for example, by teaching staff how to collect, analyze, and interpret evidence); or
  • Contribute to organizational growth (for example, the evaluation may clarify how the program relates to the organization's mission).

Users

Users are the specific individuals who will receive evaluation findings. They will directly experience the consequences of inevitable trade-offs in the evaluation process. For example, a trade-off might be having a relatively modest evaluation to fit the budget with the outcome that the evaluation results will be less certain than they would be for a full-scale evaluation. Because they will be affected by these tradeoffs, intended users have a right to participate in choosing a focus for the evaluation. An evaluation designed without adequate user involvement in selecting the focus can become a misguided and irrelevant exercise. By contrast, when users are encouraged to clarify intended uses, priority questions, and preferred methods, the evaluation is more likely to focus on things that will inform (and influence) future actions.

Uses

Uses describe what will be done with what is learned from the evaluation. There is a wide range of potential uses for program evaluation. Generally speaking, the uses fall in the same four categories as the purposes listed above: to gain insight, improve how things get done, determine what the effects of the program are, and affect participants. The following list gives examples of uses in each category.

Some specific examples of evaluation uses

• To gain insight:

  • Assess needs and wants of community members
  • Identify barriers to use of the program
  • Learn how to best describe and measure program activities

• To improve how things get done:

  • Refine plans for introducing a new practice
  • Determine the extent to which plans were implemented
  • Improve educational materials
  • Enhance cultural competence
  • Verify that participants' rights are protected
  • Set priorities for staff training
  • Make mid-course adjustments
  • Clarify communication
  • Determine if client satisfaction can be improved
  • Compare costs to benefits
  • Find out which participants benefit most from the program
  • Mobilize community support for the program

• To determine what the effects of the program are:

  • Assess skills development by program participants
  • Compare changes in behavior over time
  • Decide where to allocate new resources
  • Document the level of success in accomplishing objectives
  • Demonstrate that accountability requirements are fulfilled
  • Use information from multiple evaluations to predict the likely effects of similar programs

• To affect participants:

  • Reinforce messages of the program
  • Stimulate dialogue and raise awareness about community issues
  • Broaden consensus among partners about program goals
  • Teach evaluation skills to staff and other stakeholders
  • Gather success stories
  • Support organizational change and improvement

Questions

The evaluation needs to answer specific questions. Drafting questions encourages stakeholders to reveal what they believe the evaluation should answer. That is, what questions are more important to stakeholders? The process of developing evaluation questions further refines the focus of the evaluation.

Methods

The methods available for an evaluation are drawn from behavioral science and social research and development. Three types of methods are commonly recognized. They are experimental, quasi-experimental, and observational or case study designs. Experimental designs use random assignment to compare the effect of an intervention between otherwise equivalent groups (for example, comparing a randomly assigned group of students who took part in an after-school reading program with those who didn't). Quasi-experimental methods make comparisons between groups that aren't equal (e.g. program participants vs. those on a waiting list) or use of comparisons within a group over time, such as in an interrupted time series in which the intervention may be introduced sequentially across different individuals, groups, or contexts. Observational or case study methods use comparisons within a group to describe and explain what happens (e.g., comparative case studies with multiple communities).

No design is necessarily better than another. Evaluation methods should be selected because they provide the appropriate information to answer stakeholders' questions, not because they are familiar, easy, or popular. The choice of methods has implications for what will count as evidence, how that evidence will be gathered, and what kind of claims can be made. Because each method option has its own biases and limitations, evaluations that mix methods are generally more robust.

Over the course of an evaluation, methods may need to be revised or modified. Circumstances that make a particular approach useful can change. For example, the intended use of the evaluation could shift from discovering how to improve the program to helping decide about whether the program should continue or not. Thus, methods may need to be adapted or redesigned to keep the evaluation on track.

Agreements

Agreements summarize the evaluation procedures and clarify everyone's roles and responsibilities. An agreement describes how the evaluation activities will be implemented. Elements of an agreement include statements about the intended purpose, users, uses, and methods, as well as a summary of the deliverables, those responsible, a timeline, and budget.

The formality of the agreement depends upon the relationships that exist between those involved. For example, it may take the form of a legal contract, a detailed protocol, or a simple memorandum of understanding. Regardless of its formality, creating an explicit agreement provides an opportunity to verify the mutual understanding needed for a successful evaluation. It also provides a basis for modifying procedures if that turns out to be necessary.

As you can see, focusing the evaluation design may involve many activities. For instance, both supporters and skeptics of the program could be consulted to ensure that the proposed evaluation questions are politically viable. A menu of potential evaluation uses appropriate for the program's stage of development could be circulated among stakeholders to determine which is most compelling. Interviews could be held with specific intended users to better understand their information needs and timeline for action. Resource requirements could be reduced when users are willing to employ more timely but less precise evaluation methods.

Gather Credible Evidence

Credible evidence is the raw material of a good evaluation. The information learned should be seen by stakeholders as believable, trustworthy, and relevant to answer their questions. This requires thinking broadly about what counts as "evidence." Such decisions are always situational; they depend on the question being posed and the motives for asking it. For some questions, a stakeholder's standard for credibility could demand having the results of a randomized experiment. For another question, a set of well-done, systematic observations such as interactions between an outreach worker and community residents, will have high credibility. The difference depends on what kind of information the stakeholders want and the situation in which it is gathered.

Context matters! In some situations, it may be necessary to consult evaluation specialists. This may be especially true if concern for data quality is especially high. In other circumstances, local people may offer the deepest insights. Regardless of their expertise, however, those involved in an evaluation should strive to collect information that will convey a credible, well-rounded picture of the program and its efforts.

Having credible evidence strengthens the evaluation results as well as the recommendations that follow from them. Although all types of data have limitations, it is possible to improve an evaluation's overall credibility. One way to do this is by using multiple procedures for gathering, analyzing, and interpreting data. Encouraging participation by stakeholders can also enhance perceived credibility. When stakeholders help define questions and gather data, they will be more likely to accept the evaluation's conclusions and to act on its recommendations.

The following features of evidence gathering typically affect how credible it is seen as being:

Indicators

Indicators translate general concepts about the program and its expected effects into specific, measurable parts.

Examples of indicators include:

• The program's capacity to deliver services
• The participation rate
• The level of client satisfaction
• The amount of intervention exposure (how many people were exposed to the program, and for how long they were exposed)
• Changes in participant behavior
• Changes in community conditions or norms
• Changes in the environment (e.g., new programs, policies, or practices)
• Longer-term changes in population health status (e.g., estimated teen pregnancy rate in the county)

Indicators should address the criteria that will be used to judge the program. That is, they reflect the aspects of the program that are most meaningful to monitor. Several indicators are usually needed to track the implementation and effects of a complex program or intervention.

One way to develop multiple indicators is to create a "balanced scorecard," which contains indicators that are carefully selected to complement one another. According to this strategy, program processes and effects are viewed from multiple perspectives using small groups of related indicators. For instance, a balanced scorecard for a single program might include indicators of how the program is being delivered; what participants think of the program; what effects are observed; what goals were attained; and what changes are occurring in the environment around the program.

Another approach to using multiple indicators is based on a program logic model, such as we discussed earlier in the section. A logic model can be used as a template to define a full spectrum of indicators along the pathway that leads from program activities to expected effects. For each step in the model, qualitative and/or quantitative indicators could be developed.

Indicators can be broad-based and don't need to focus only on a program's long -term goals. They can also address intermediary factors that influence program effectiveness, including such intangible factors as service quality, community capacity, or inter -organizational relations. Indicators for these and similar concepts can be created by systematically identifying and then tracking markers of what is said or done when the concept is expressed.

In the course of an evaluation, indicators may need to be modified or new ones adopted. Also, measuring program performance by tracking indicators is only one part of evaluation, and shouldn't be confused as a basis for decision making in itself. There are definite perils to using performance indicators as a substitute for completing the evaluation process and reaching fully justified conclusions. For example, an indicator, such as a rising rate of unemployment, may be falsely assumed to reflect a failing program when it may actually be due to changing environmental conditions that are beyond the program's control.

Sources

Sources of evidence in an evaluation may be people, documents, or observations. More than one source may be used to gather evidence for each indicator. In fact, selecting multiple sources provides an opportunity to include different perspectives about the program and enhances the evaluation's credibility. For instance, an inside perspective may be reflected by internal documents and comments from staff or program managers; whereas clients and those who do not support the program may provide different, but equally relevant perspectives. Mixing these and other perspectives provides a more comprehensive view of the program or intervention.

The criteria used to select sources should be clearly stated so that users and other stakeholders can interpret the evidence accurately and assess if it may be biased. In addition, some sources provide information in narrative form (for example, a person's experience when taking part in the program) and others are numerical (for example, how many people were involved in the program). The integration of qualitative and quantitative information can yield evidence that is more complete and more useful, thus meeting the needs and expectations of a wider range of stakeholders.

Quality

Quality refers to the appropriateness and integrity of information gathered in an evaluation. High quality data are reliable and informative. It is easier to collect if the indicators have been well defined. Other factors that affect quality may include instrument design, data collection procedures, training of those involved in data collection, source selection, coding, data management, and routine error checking. Obtaining quality data will entail tradeoffs (e.g. breadth vs. depth); stakeholders should decide together what is most important to them. Because all data have limitations, the intent of a practical evaluation is to strive for a level of quality that meets the stakeholders' threshold for credibility.

Quantity

Quantity refers to the amount of evidence gathered in an evaluation. It is necessary to estimate in advance the amount of information that will be required and to establish criteria to decide when to stop collecting data - to know when enough is enough. Quantity affects the level of confidence or precision users can have - how sure we are that what we've learned is true. It also partly determines whether the evaluation will be able to detect effects. All evidence collected should have a clear, anticipated use.

Logistics

By logistics, we mean the methods, timing, and physical infrastructure for gathering and handling evidence. People and organizations also have cultural preferences that dictate acceptable ways of asking questions and collecting information, including who would be perceived as an appropriate person to ask the questions. For example, some participants may be unwilling to discuss their behavior with a stranger, whereas others are more at ease with someone they don't know. Therefore, the techniques for gathering evidence in an evaluation must be in keeping with the cultural norms of the community. Data collection procedures should also ensure that confidentiality is protected.

Justify Conclusions

The process of justifying conclusions recognizes that evidence in an evaluation does not necessarily speak for itself. Evidence must be carefully considered from a number of different stakeholders' perspectives to reach conclusions that are well -substantiated and justified. Conclusions become justified when they are linked to the evidence gathered and judged against agreed-upon values set by the stakeholders. Stakeholders must agree that conclusions are justified in order to use the evaluation results with confidence.

The principal elements involved in justifying conclusions based on evidence are:

Standards

Standards reflect the values held by stakeholders about the program. They provide the basis to make program judgments. The use of explicit standards for judgment is fundamental to sound evaluation. In practice, when stakeholders articulate and negotiate their values, these become the standards to judge whether a given program's performance will, for instance, be considered "successful," "adequate," or "unsuccessful."

Analysis and synthesis

Analysis and synthesis are methods to discover and summarize an evaluation's findings. They are designed to detect patterns in evidence, either by isolating important findings (analysis) or by combining different sources of information to reach a larger understanding (synthesis). Mixed method evaluations require the separate analysis of each evidence element, as well as a synthesis of all sources to examine patterns that emerge. Deciphering facts from a given body of evidence involves deciding how to organize, classify, compare, and display information. These decisions are guided by the questions being asked, the types of data available, and especially by input from stakeholders and primary intended users.

Interpretation

Interpretation is the effort to figure out what the findings mean. Uncovering facts about a program's performance isn't enough to make conclusions. The facts must be interpreted to understand their practical significance. For example, saying, "15 % of the people in our area witnessed a violent act last year," may be interpreted differently depending on the situation. For example, if 50% of community members had watched a violent act in the last year when they were surveyed five years ago, the group can suggest that, while still a problem, things are getting better in the community. However, if five years ago only 7% of those surveyed said the same thing, community organizations may see this as a sign that they might want to change what they are doing. In short, interpretations draw on information and perspectives that stakeholders bring to the evaluation. They can be strengthened through active participation or interaction with the data and preliminary explanations of what happened.

Judgements

Judgments are statements about the merit, worth, or significance of the program. They are formed by comparing the findings and their interpretations against one or more selected standards. Because multiple standards can be applied to a given program, stakeholders may reach different or even conflicting judgments. For instance, a program that increases its outreach by 10% from the previous year may be judged positively by program managers, based on standards of improved performance over time. Community members, however, may feel that despite improvements, a minimum threshold of access to services has still not been reached. Their judgment, based on standards of social equity, would therefore be negative. Conflicting claims about a program's quality, value, or importance often indicate that stakeholders are using different standards or values in making judgments. This type of disagreement can be a catalyst to clarify values and to negotiate the appropriate basis (or bases) on which the program should be judged.

Recommendations

Recommendations are actions to consider as a result of the evaluation. Forming recommendations requires information beyond just what is necessary to form judgments. For example, knowing that a program is able to increase the services available to battered women doesn't necessarily translate into a recommendation to continue the effort, particularly when there are competing priorities or other effective alternatives. Thus, recommendations about what to do with a given intervention go beyond judgments about a specific program's effectiveness.

If recommendations aren't supported by enough evidence, or if they aren't in keeping with stakeholders' values, they can really undermine an evaluation's credibility. By contrast, an evaluation can be strengthened by recommendations that anticipate and react to what users will want to know.

Three things might increase the chances that recommendations will be relevant and well-received:

• Sharing draft recommendations
• Soliciting reactions from multiple stakeholders
• Presenting options instead of directive advice

Justifying conclusions in an evaluation is a process that involves different possible steps. For instance, conclusions could be strengthened by searching for alternative explanations from the ones you have chosen, and then showing why they are unsupported by the evidence. When there are different but equally well supported conclusions, each could be presented with a summary of their strengths and weaknesses. Techniques to analyze, synthesize, and interpret findings might be agreed upon before data collection begins.

Ensure Use and Share Lessons Learned

It is naive to assume that lessons learned in an evaluation will necessarily be used in decision making and subsequent action. Deliberate effort on the part of evaluators is needed to ensure that the evaluation findings will be used appropriately. Preparing for their use involves strategic thinking and continued vigilance in looking for opportunities to communicate and influence. Both of these should begin in the earliest stages of the process and continue throughout the evaluation.

The elements of key importance to be sure that the recommendations from an evaluation are used are:

Design

Design refers to how the evaluation's questions, methods, and overall processes are constructed. As discussed in the third step of this framework (focusing the evaluation design), the evaluation should be organized from the start to achieve specific agreed-upon uses. Having a clear purpose that is focused on the use of what is learned helps those who will carry out the evaluation to know who will do what with the findings. Furthermore, the process of creating a clear design will highlight ways that stakeholders, through their many contributions, can improve the evaluation and facilitate the use of the results.

Preparation

Preparation refers to the steps taken to get ready for the future uses of the evaluation findings. The ability to translate new knowledge into appropriate action is a skill that can be strengthened through practice. In fact, building this skill can itself be a useful benefit of the evaluation. It is possible to prepare stakeholders for future use of the results by discussing how potential findings might affect decision making.

For example, primary intended users and other stakeholders could be given a set of hypothetical results and asked what decisions or actions they would make on the basis of this new knowledge. If they indicate that the evidence presented is incomplete or irrelevant and that no action would be taken, then this is an early warning sign that the planned evaluation should be modified. Preparing for use also gives stakeholders more time to explore both positive and negative implications of potential results and to identify different options for program improvement.

Feedback

Feedback is the communication that occurs among everyone involved in the evaluation. Giving and receiving feedback creates an atmosphere of trust among stakeholders; it keeps an evaluation on track by keeping everyone informed about how the evaluation is proceeding. Primary intended users and other stakeholders have a right to comment on evaluation decisions. From a standpoint of ensuring use, stakeholder feedback is a necessary part of every step in the evaluation. Obtaining valuable feedback can be encouraged by holding discussions during each step of the evaluation and routinely sharing interim findings, provisional interpretations, and draft reports.

Follow-up

Follow-up refers to the support that many users need during the evaluation and after they receive evaluation findings. Because of the amount of effort required, reaching justified conclusions in an evaluation can seem like an end in itself. It is not. Active follow-up may be necessary to remind users of the intended uses of what has been learned. Follow-up may also be required to stop lessons learned from becoming lost or ignored in the process of making complex or political decisions. To guard against such oversight, it may be helpful to have someone involved in the evaluation serve as an advocate for the evaluation's findings during the decision -making phase.

Facilitating the use of evaluation findings also carries with it the responsibility to prevent misuse. Evaluation results are always bounded by the context in which the evaluation was conducted. Some stakeholders, however, may be tempted to take results out of context or to use them for different purposes than what they were developed for. For instance, over-generalizing the results from a single case study to make decisions that affect all sites in a national program is an example of misuse of a case study evaluation.

Similarly, program opponents may misuse results by overemphasizing negative findings without giving proper credit for what has worked. Active follow-up can help to prevent these and other forms of misuse by ensuring that evidence is only applied to the questions that were the central focus of the evaluation.

Dissemination

Dissemination is the process of communicating the procedures or the lessons learned from an evaluation to relevant audiences in a timely, unbiased, and consistent fashion. Like other elements of the evaluation, the reporting strategy should be discussed in advance with intended users and other stakeholders. Planning effective communications also requires considering the timing, style, tone, message source, vehicle, and format of information products. Regardless of how communications are constructed, the goal for dissemination is to achieve full disclosure and impartial reporting.

Along with the uses for evaluation findings, there are also uses that flow from the very process of evaluating. These "process uses" should be encouraged. The people who take part in an evaluation can experience profound changes in beliefs and behavior. For instance, an evaluation challenges staff members to act differently in what they are doing, and to question assumptions that connect program activities with intended effects.

Evaluation also prompts staff to clarify their understanding of the goals of the program. This greater clarity, in turn, helps staff members to better function as a team focused on a common end. In short, immersion in the logic, reasoning, and values of evaluation can have very positive effects, such as basing decisions on systematic judgments instead of on unfounded assumptions.

Additional process uses for evaluation include:

• By defining indicators, what really matters to stakeholders becomes clear
• It helps make outcomes matter by changing the reinforcements connected with achieving positive results. For example, a funder might offer "bonus grants" or "outcome dividends" to a program that has shown a significant amount of community change and improvement.

Standards for "good" evaluation

There are standards to assess whether all of the parts of an evaluation are well -designed and working to their greatest potential. The Joint Committee on Educational Evaluation developed "The Program Evaluation Standards" for this purpose. These standards, designed to assess evaluations of educational programs, are also relevant for programs and interventions related to community health and development.

The program evaluation standards make it practical to conduct sound and fair evaluations. They offer well-supported principles to follow when faced with having to make tradeoffs or compromises. Attending to the standards can guard against an imbalanced evaluation, such as one that is accurate and feasible, but isn't very useful or sensitive to the context. Another example of an imbalanced evaluation is one that would be genuinely useful, but is impossible to carry out.

The following standards can be applied while developing an evaluation design and throughout the course of its implementation. Remember, the standards are written as guiding principles, not as rigid rules to be followed in all situations.

The 30 more specific standards are grouped into four categories:

• Utility
• Feasibility
• Propriety
• Accuracy

The utility standards are:

• Stakeholder Identification: People who are involved in (or will be affected by) the evaluation should be identified, so that their needs can be addressed.
• Evaluator Credibility: The people conducting the evaluation should be both trustworthy and competent, so that the evaluation will be generally accepted as credible or believable.
• Information Scope and Selection: Information collected should address pertinent questions about the program, and it should be responsive to the needs and interests of clients and other specified stakeholders.
• Values Identification: The perspectives, procedures, and rationale used to interpret the findings should be carefully described, so that the bases for judgments about merit and value are clear.
• Report Clarity: Evaluation reports should clearly describe the program being evaluated, including its context, and the purposes, procedures, and findings of the evaluation. This will help ensure that essential information is provided and easily understood.
• Report Timeliness and Dissemination: Significant midcourse findings and evaluation reports should be shared with intended users so that they can be used in a timely fashion.
• Evaluation Impact: Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the evaluation will be used.

Feasibility Standards

The feasibility standards are to ensure that the evaluation makes sense - that the steps that are planned are both viable and pragmatic.

The feasibility standards are:

• Practical Procedures: The evaluation procedures should be practical, to keep disruption of everyday activities to a minimum while needed information is obtained.
• Political Viability: The evaluation should be planned and conducted with anticipation of the different positions or interests of various groups. This should help in obtaining their cooperation so that possible attempts by these groups to curtail evaluation operations or to misuse the results can be avoided or counteracted.
• Cost Effectiveness: The evaluation should be efficient and produce enough valuable information that the resources used can be justified.

Propriety Standards

The propriety standards ensure that the evaluation is an ethical one, conducted with regard for the rights and interests of those involved. The eight propriety standards follow.

• Service Orientation: Evaluations should be designed to help organizations effectively serve the needs of all of the targeted participants.
• Formal Agreements: The responsibilities in an evaluation (what is to be done, how, by whom, when) should be agreed to in writing, so that those involved are obligated to follow all conditions of the agreement, or to formally renegotiate it.
• Rights of Human Subjects: Evaluation should be designed and conducted to respect and protect the rights and welfare of human subjects, that is, all participants in the study.
• Human Interactions: Evaluators should respect basic human dignity and worth when working with other people in an evaluation, so that participants don't feel threatened or harmed.
• Complete and Fair Assessment: The evaluation should be complete and fair in its examination, recording both strengths and weaknesses of the program being evaluated. This allows strengths to be built upon and problem areas addressed.
• Disclosure of Findings: The people working on the evaluation should ensure that all of the evaluation findings, along with the limitations of the evaluation, are accessible to everyone affected by the evaluation, and any others with expressed legal rights to receive the results.
• Conflict of Interest: Conflict of interest should be dealt with openly and honestly, so that it does not compromise the evaluation processes and results.
• Fiscal Responsibility: The evaluator's use of resources should reflect sound accountability procedures and otherwise be prudent and ethically responsible, so that expenditures are accounted for and appropriate.

Accuracy Standards

The accuracy standards ensure that the evaluation findings are considered correct.

There are 12 accuracy standards:

• Program Documentation: The program should be described and documented clearly and accurately, so that what is being evaluated is clearly identified.
• Context Analysis: The context in which the program exists should be thoroughly examined so that likely influences on the program can be identified.
• Described Purposes and Procedures: The purposes and procedures of the evaluation should be monitored and described in enough detail that they can be identified and assessed.
• Defensible Information Sources: The sources of information used in a program evaluation should be described in enough detail that the adequacy of the information can be assessed.
• Valid Information: The information gathering procedures should be chosen or developed and then implemented in such a way that they will assure that the interpretation arrived at is valid.
• Reliable Information: The information gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable.
• Systematic Information: The information from an evaluation should be systematically reviewed and any errors found should be corrected.
• Analysis of Quantitative Information: Quantitative information - data from observations or surveys - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Analysis of Qualitative Information: Qualitative information - descriptive information from interviews and other sources - in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.
• Justified Conclusions: The conclusions reached in an evaluation should be explicitly justified, so that stakeholders can understand their worth.
• Impartial Reporting: Reporting procedures should guard against the distortion caused by personal feelings and biases of people involved in the evaluation, so that evaluation reports fairly reflect the evaluation findings.
• Metaevaluation: The evaluation itself should be evaluated against these and other pertinent standards, so that it is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.

Applying the framework: Conducting optimal evaluations

There is an ever-increasing agreement on the worth of evaluation; in fact, doing so is often required by funders and other constituents. So, community health and development professionals can no longer question whether or not to evaluate their programs. Instead, the appropriate questions are:

• What is the best way to evaluate?
• What are we learning from the evaluation?
• How will we use what we learn to become more effective?

The framework for program evaluation helps answer these questions by guiding users to select evaluation strategies that are useful, feasible, proper, and accurate.

To use this framework requires quite a bit of skill in program evaluation. In most cases there are multiple stakeholders to consider, the political context may be divisive, steps don't always follow a logical order, and limited resources may make it difficult to take a preferred course of action. An evaluator's challenge is to devise an optimal strategy, given the conditions she is working under. An optimal strategy is one that accomplishes each step in the framework in a way that takes into account the program context and is able to meet or exceed the relevant standards.

This framework also makes it possible to respond to common concerns about program evaluation. For instance, many evaluations are not undertaken because they are seen as being too expensive. The cost of an evaluation, however, is relative; it depends upon the question being asked and the level of certainty desired for the answer. A simple, low-cost evaluation can deliver information valuable for understanding and improvement.

Rather than discounting evaluations as a time-consuming sideline, the framework encourages evaluations that are timed strategically to provide necessary feedback. This makes it possible to make evaluation closely linked with everyday practices.

Another concern centers on the perceived technical demands of designing and conducting an evaluation. However, the practical approach endorsed by this framework focuses on questions that can improve the program.

Finally, the prospect of evaluation troubles many staff members because they perceive evaluation methods as punishing ("They just want to show what we're doing wrong."), exclusionary ("Why aren't we part of it? We're the ones who know what's going on."), and adversarial ("It's us against them.") The framework instead encourages an evaluation approach that is designed to be helpful and engages all interested stakeholders in a process that welcomes their participation.

In Summary

Evaluation is a powerful strategy for distinguishing programs and interventions that make a difference from those that don't. It is a driving force for developing and adapting sound strategies, improving existing programs, and demonstrating the results of investments in time and other resources. It also helps determine if what is being done is worth the cost.

This recommended framework for program evaluation is both a synthesis of existing best practices and a set of standards for further improvement. It supports a practical approach to evaluation based on steps and standards that can be applied in almost any setting. Because the framework is purposefully general, it provides a stable guide to design and conduct a wide range of evaluation efforts in a variety of specific program areas. The framework can be used as a template to create useful evaluation plans to contribute to understanding and improvement. The Magenta Book - Guidance for Evaluation provides additional information on requirements for good evaluation, and some straightforward steps to make a good evaluation of an intervention more feasible, read The Magenta Book - Guidance for Evaluation.

You understand the basic concepts and terms of program evaluation:

___You understand that evaluation is the systematic investigation of the merit, worth, or significance of an object or effort

___You understand that a program is used to describe the object or effort that is being evaluated

___You understand that program evaluation is a way to evaluate the specific projects and activities community groups may take part in

___You understand that stakeholders are those who care about the program or effort

You understand that program evaluation:

___Helps clarify program plans

___Improves communication among partners

___Gathers feedback needed to improve and be accountable for program effectiveness

You know how to evaluate a specific program:

___You know what will be evaluated

___You have decided what criteria will be used to judge program performance

___You have decided what standards of performance must be reached for the program to be considered successful

___You know what evidence will indicate performance

___You know what conclusions about program performance are justified based on the available evidence

___You have engaged stakeholders

___You have described the program

___You have focused the evaluation design

___You understand that possible focuses are: to gain insight, to improve how things get done, to determine what the effects of the program are, and to effect those who participate in it

___You have gathered credible evidence

___You have justified your conclusions

___You understand how to ensure the use of the information and share lessons learned

___You know that there are four categories of evaluation standards: utility, feasibility, propriety, and accuracy

___You understand the standards in each category

___You understand which standards are most applicable to each step of evaluation practice

When writing your evaluation report:

___Provide interim and final reports to intended users in time for intended uses

___Tailor the report content, format, and style for the audience(s) by involving audience members

___Include an executive summary

___Summarize the stakeholders and how they were engaged

___Describe essential features of the program (perhaps in appendices)

___Explain the focus of the evaluation and its limitations

___Include an adequate summary of the evaluation plan and procedures

___Provide all necessary technical information (perhaps in appendices)

___Specify the standards and criteria for evaluative judgments

___Explain the evaluative judgments and how they are supported by the evidence

___List both strengths and weaknesses of the evaluation

___Discuss recommendations for action along with their advantages and disadvantages

___Ensure protections for program clients and other stakeholders

___Anticipate how people or organizations may be affected by the findings

___Present minority opinions or rejoinders where necessary

___Verify that the report is accurate and unbiased

___Organize the report logically and include an appropriate level of detail

___Remove technical jargon

___Use examples, illustrations, graphics, and stories 

Tool 1: Tools Related to the Recommended Framework

See the following Community Tool Box sections for additional information.

 

Engage Stakeholders

To help you understand who stakeholders are:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

Identifying Targets and Agents of Change: Who Can Benefit and Who Can Help

Identifying Community Assets and Resources

To learn how to involve stakeholders:

Involving Key Influentials in the Initiative

Involving People Most Affected by the Problem

Conducting Focus Groups

To work together with a diverse group:

Training for Conflict Resolution

Encouraging Involvement of Potential Opponents as well as Allies

Describe the Program

To fully understand the need, problem, or goal that the program addresses:

Defining and Analyzing the Problem

Analyzing Community Problems

Conducting Needs Assessment Surveys

Conducting Concerns Surveys

Conducting Public Forums and Listening Sessions

To better explain the activities, components, and elements of the program:

Designing Community Interventions

Understanding Risk and Protective Factors: Their Use in Selecting Potential Targets and Promising Strategies for Interventions

Identifying Strategies and Tactics for Reducing Risks

To be able to describe resources and assets for the program:

Identifying Community Assets and Resources

For examples of logic models:

Our Model of Practice: Building Capacity for Community and System Change

Our Evaluation Model: Evaluating Comprehensive Community Initiatives

To look broadly at your program and its context:

An Overview of Strategic Planning or "VMOSA" (Vision, Mission, Objectives, Strategies, Action Plan)

Understanding and Describing the Community

To modify the program to fit the local context:

Adapting Community Interventions for Different Cultures and Communities

To learn to explain the program to others (and so that they see your point of view):

Reframing the Issue

Focus the Evaluation Design

To clarify the purpose:

Developing an Evaluation Plan

To identify potential users and uses:

Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?

To clarify evaluation questions:

Developing an Evaluation Plan

For illustrative evaluation questions:

Our Model of Practice: Building Capacity for Community and System Change

For help identifying specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

For guidance about making agreements:

Identifying Action Steps in Bringing About Community and System Change

Gather Credible Evidence

For support in implementing specific evaluation methods:

Some Methods for Evaluating Comprehensive Community Initiatives

Conducting Interviews

Developing Baseline Measures of Behavior

Obtaining Feedback from Constituents: What Changes are Important and Feasible?

Justify Conclusions

To see an illustrative process for considering evidence:

Our Model of Practice: Building Capacity for Community and System Change

Ensure use and share lessons learned

To promote the use of what your organization has learned:

Providing Feedback to Improve the Initiative

Conducting a Social Marketing Campaign

Attracting Support for Specific Programs

To share what you have learned with diverse groups:

Making Community Presentations

Communications to Promote Interest

Arranging Celebrations

Tool 2: Evaluation Standards

Use this table to determine how well your evaluation met "good" standards for evaluation.

Standard	Did the evaluation meet this standard? (Yes or No)	Comments:
Utility Standards
1. Stakeholder Identification
2. Evaluator Credibility
3. Information Scope and Selection
4. Values Identification
5. Report Clarity
6. Report Timeliness and Dissemination
7. Evaluation Impact
Feasibility Standards
1. Practical Procedures
2. Political Viability
3. Cost Effectiveness
Propriety Standards
1. Service Orientation
2. Formal Agreements
3. Rights of Human Subjects
4. Human Interactions
5. Complete and Fair Assessment
6. Disclosure of Findings
7. Conflict of Interest
8. Fiscal Responsibility
Accuracy Standards
1. Program Documentation
2. Context Analysis
3. Described Purposes and Procedures
4. Defensible Information Sources
5. Valid Information
6. Reliable Information
7. Systematic Information
8. Analysis of Quantitative Information
9. Analysis of Qualitative Information
10. Justified Conclusions
11. Impartial Reporting
12. Metaevaluation

Tool 3: Steps in Evaluation Practice and the Most Relevant Standards

Codes following each standard designate the conceptual group and number of the standard.  U = Utility; F = Feasibility; P = Propriety; A = Accuracy.

Steps	Most Relevant Standards
Engage Stakeholders	Metaevaluation (A12) Stakeholder identification (U1) Evaluator Credibility (U2) Formal agreements (P2) Rights of human subjects (P3) Human Interactions (P4) Conflict of interest (P7)
Describe the program	Complete and fair assessment (P5) Program documentation (A1) Context analysis (A2)
Focus the evaluation design	Evaluation impact (U7) Practical procedures (F1) Political viability (F2) Cost effectiveness (F3) Service orientation (P1) Complete and fair assessment (P5) Fiscal responsibility (P8) Described purposes and procedures (A3)
Gather credible evidence	Information scope and selection (U3) Defensible information sources (A4) Valid information (A5) Reliable information (A6) Systematic information (A7)
Justify conclusions	Values identification (U4) Analysis of quantitative information (A8) Analysis of qualitative information (A9) Justified conclusions (A10)
Ensure use and share lessons learned	Evaluator credibility (U2) Report clarity (U5) Report timeliness and dissemination (U6) Evaluation impact (U7) Disclosure of findings (P6) Impartial reporting (A11)

 

Della Roberts worked as a nutritionist at the Harperville Hospital. As an African American, she was concerned about obesity among Black children, and about the fact that many of Harperville’s African American neighborhoods didn’t have access to healthy food in stores or restaurants. She felt that the city ought to be doing something to change the situation, but officials didn’t seem to see it as a problem. Della decided to conduct some research to use as a base for advocacy.

Della realized that in order to collect accurate data, she needed to find researchers who would be trusted by people in the neighborhoods she was concerned about. What if she recruited researchers from among the people in those neighborhoods? She contacted two ministers she knew, an African American doctor who practiced in a Black neighborhood, and the director of a community center, as well as using her own family connections. Within two weeks, she had gathered a group of neighborhood residents who were willing to act as researchers. They ranged from high school students to grandparents, and from people who could barely read to others who had taken college courses.

The group met several times at the hospital to work out how they were going to collect information from the community. Della conducted workshops in research methods and in such basic skills as how to record interviews and observations. The group discussed the problem of recording for those who had difficulty writing, and came up with other ways of logging information. They decided they would each interview a given number of residents about their food shopping and eating habits, and that they would also observe people’s buying patterns in neighborhood stores and fast food restaurants. They set a deadline for finishing their data gathering, and went off to learn as much as they could about the food shopping and eating behavior of people in their neighborhoods.

As the data came in, it became clear that people in the neighborhoods would be happy to buy more nutritious food, but it was simply too difficult to get it. They either had to travel long distances on the bus, since many didn’t have cars, or find time after a long work day to drive to another, often unfamiliar, part of the city and spend an evening shopping. Many also had the perception that healthy food was much more expensive, and that they couldn’t afford it.

Ultimately, the data that the group of neighborhood residents had gathered went into a report written by Della and other professionals on the hospital staff. The report helped to convince the city to provide incentives to supermarket chains to locate in neighborhoods where healthy food was hard to find.

The group that Della had recruited had become a community-based participatory research team. Working with Della and others at the hospital, they helped to determine what kind of information would be useful, and then learned how to gather it. Because they were part of the community, they were trusted by residents; because they shared other residents’ experience, they knew what questions to ask and fully understood the answers, as well as what they were seeing when they observed.

This section is about participatory action research: what it is, why it can be effective, who might use it, and how to set up and conduct it.

What is community-based participatory research?

In simplest terms, community-based participatory research (for convenience, we’ll primarily call it CBPR for the rest of this section) enlists those who are most affected by a community issue – typically in collaboration or partnership with others who have research skills – to conduct research on and analyze that issue, with the goal of devising strategies to resolve it. In other words, community-based participatory research adds to or replaces academic and other professional research with research done by community members, so that research results both come from and go directly back to the people who need them most and can make the best use of them.

There are several levels of participatory research. At one end of the spectrum is academic or government research that nonetheless gathers information directly from community members. The community members are those most directly affected by the issue at hand, and they may (or may not) be asked for their opinions about what they need and what they think will help, as well as for specific information. In that circumstance, the community members don’t have any role in choosing what information is sought, in collecting data, or in analyzing the information once it’s collected. (At the same time, this type of participatory research is still a long step from research that is done at second or third hand, where all the information about a group of people is gathered from statistics, census data, and the reports of observers or of human service or health professionals.)

At another level, academic or other researchers recruit or hire members of an affected group – often because they are familiar with and known by the community – to collect data. In this case, the collectors may or may not also help to analyze the information that they have gathered.

A third level of participatory research has academic, government, or other professional researchers recruiting members of an affected group as partners in a research project. The community members work with the researchers as colleagues, participating in the conception and design of the project, data collection, and data analysis. They may participate as well in reporting the results of the project or study. At this level, there is usually – though not always – an assumption that the research group is planning to use its research to take action on an issue that needs to be resolved

The opposite end of the participatory research continuum from the first level described involves community members creating their own research group – although they might seldom think of it as such – to find out about and take action on a community issue that affects them directly.

In this section, we’ll concern ourselves with the latter two types of participatory research – those that involve community members directly in planning and carrying out research, and that lead to some action that can influence the issue studied. This is what is often defined as community-based participatory research. There are certainly scenarios where other types of participatory research are more appropriate, or easier to employ in particular situations, but it’s CBPR that we’ll discuss here.

Employing CBPR for purposes of either evaluation or long-term change can be a good idea for reasons of practicality, personal development, and politics.

On the practical side, community-based participatory research can often get you the best information possible about the issue, for at least reasons including:

• People in an affected population are more liable to be willing to talk and give straight answers to researchers whom they know, or whom they know to be in circumstances similar to their own, than to outsiders with whom they have little in common
• People who have actually experienced the effects of an issue – or an intervention – may have ideas and information about aspects of it that wouldn’t occur to anyone studying it from outside. Thus, action researchers from the community may focus on elements of the issue, or ask questions or follow-ups, that outside researchers wouldn’t, and get crucial information that other researchers might find only by accident, or perhaps not at all
• People who are deeply affected by an issue, or participants in a program, may know intuitively, or more directly, what’s important when they see or hear it. What seems an offhand comment to an outside researcher might reveal its real importance to someone who is part of the same population as person who made the comment.
• Action researchers from the community are on the scene all the time. Their contact both with the issue or intervention and with the population affected by it is constant, and, as a result, they may find information even when they’re not officially engaged in research.
• Findings may receive more community support because community members know that the research was conducted by people in the same circumstances as their own

When you’re conducting an evaluation, these advantages can provide you with a more accurate picture of the intervention or initiative and its effects. When you’re studying a community issue, all these advantages can lead to a true understanding of its nature, its causes, and its effects in the community, and can provide a solid basis for a strategy to resolve it. And that, of course, is the true goal of community research – to identify and resolve an issue or problem, and to improve the quality of life for the community as a whole

In the personal development sphere, CBPR can have profound effects on the development and lives of the community researchers, particularly when those who benefit from an intervention, or who are affected by an issue, are poor or otherwise disadvantaged, lack education or basic skills, and/or feel that the issue is far beyond their influence. By engaging in research, they not only learn new skills, but see themselves in a position of competence, obtain valuable knowledge and information about a subject important to them, and gain the power and the confidence to exercise control over this aspect of their lives.

Two common political results of the CBPR process:

• Through community-based participatory research, citizens can take more control of the direction of their communities
• Community researchers – especially those who are poor or otherwise disadvantaged – come to be viewed differently by professionals and those in positions of power. They have vital information, and the ability to use it, and thus become accepted as contributing members of the community, rather than as voiceless observers or dependents. They have gained a voice, because they understand that they have something to say. Furthermore, the research and other skills and the self-confidence that people acquire in a community-based participatory research process can carry over into other parts of their lives, giving them the ability and the assurance to understand and work to control the forces that affect them. Research skills, discipline, and analytical thinking often translate into job skills, making participatory action researchers more employable. Most important, people who have always seen themselves as bystanders or victims gain the capacity to become activists who can transform their lives and communities.

Community-based participatory research has much in common with the work of the Brazilian political and educational theoretician and activist, Paulo Freire. In Freire’s critical education process, oppressed people are encouraged to look closely at their circumstances, and to understand the nature and causes of their oppressors and oppression. Freire believes that with the right tools – knowledge and critical thinking ability, a concept of their own power, and the motivation to act – they can undo that oppression. Many people see this as the “true” and only reason for supporting action research, but we see many other reasons for doing so, and list some of them both above and below.

Action research is often used to consider social problems – welfare reform or homelessness, for example – but can be turned to any number of areas with positive results.

Some prime examples:

• The environment. It was a community member who first asked the questions and started the probe that uncovered the fact that the Love Canal neighborhood in Niagara Falls, NY, had been contaminated by the dumping of toxic waste.
• Medical/health issues. Action research can be helpful in both undeveloped and developed societies in collecting information about health practices, tracking an epidemic, or mapping the occurrence of a particular condition, to name three of numerous possibilities.
• Political and economic issues. Citizen activists often do their own research to catch corrupt politicians or corporations, trace campaign contributions, etc.

Just as it can be used for different purposes, CBPR can be structured in different ways. The differences have largely to do with who comes up with the idea in the first place, and with who controls, or makes the decisions about, the research. Any of these possibilities might involve a collaboration or partnership, and a community group might well hire or recruit as a volunteer someone with research skills to help guide their work.

Some common scenarios:

• Academic or other researchers devise and construct a study, and employ community people as data collectors and/or analysts.
• A problem or issue is identified by a researcher or other entity (a human service organization, for instance), and community people are recruited to engage in research on it and develop a solution.
• A community based organization or other group gathers community people to define and work on a community issue of their choosing, or to evaluate a community intervention aimed at them or people similar to them.
• A problem is identified by a community member or group, others who are affected and concerned gather around to help, and the resulting group sets out to research and solve the problem on its own.

Why would you use community-based participatory research?

We’ve already alluded to a number of reasons why CBPR could be useful in evaluating a community intervention or initiative or addressing a community issue. We’ll repeat them briefly here, and introduce others as well.

Action research yields better and more nearly complete and accurate information from the community.

• People will speak more freely to peers, especially those they know personally, than to strangers.
• Researchers who are members of the community know the history and relationships surrounding a program or an issue, and can therefore place it in context.
• People experiencing an issue or participating in an intervention know what’s important to them about it – what it disrupts, what parts of their lives it touches, how they have changed as a result, etc. That knowledge helps them to formulate interview questions that get to the heart of what they – as researchers – are trying to learn.

Involving the community in research is more likely to meet community needs.

Action research makes a reasonable resolution or accurate evaluation more probable in two ways. First, by involving the people directly affected by the issue or intervention, it brings to bear the best information available about what’s actually happening. Second, it encourages community buy-in and support for whatever plans or interventions are developed. If people are involved in the planning and implementation of solutions to community issues, they’ll feel they own the process, and work to make it successful. It’s equitable, philosophically consistent for most grassroots and community-based organizations, and practical in that it usually yields the best results

Action research, by involving community members, creates more visibility for the effort in the community.

Researchers are familiar to the community, will talk about what they’re doing (as will their friends and relatives), and will thus spread the word about the effort.

Community members are more likely to accept the legitimacy of the research and buy into its findings if they know it was conducted by people like themselves, perhaps even people they know.

Citizens are more apt to trust both the truthfulness and the motives of their friends and neighbors than those of outsiders.

Action research trains citizen researchers who can turn their skills to other problems as well.

People who discover the power of research to explain conditions in their communities, and to uncover what’s really going on, realize that they can conduct research in other areas than the one covered by their CBPR project. They often become community activists, who work to change the conditions that create difficulty for them and others. Thus, the action research process may benefit the community not only by addressing particular issues, but by – over the long term – creating a core of people dedicated to improving the overall quality of its citizens’ lives.

Involvement in CBPR changes people’s perceptions of themselves and of what they can do.

An action research project can have profound effects on community researchers who are disadvantaged economically, educationally, or in other ways. It can contribute to their personal development, help them develop a voice and a sense of their power to change things, and vastly expand their vision of what’s possible for them and for the community. Such an expanded vision leads to an increased willingness to take action, and to an increase in their control over their lives.

Skills learned in the course of action research carry over into other areas of researchers’ lives.

Both the skills and the confidence gained in a CBPR project can be transferred to employment, education, child-rearing, and other aspects of life, greatly improving people’s prospects and well-being.

A participatory action research process can help to break down racial, ethnic, and class barriers.

 CBPR can remove barriers in two ways. First, action research teams are often diverse, crossing racial, ethnic, and class lines. As people of different backgrounds work together, this encourages tolerance and friendships, and often removes the fear and distrust. In addition, as integral contributors to a research or evaluation effort, community researchers interact with professionals, academics, and community leaders on equal footing. Once again, familiarity breaks down barriers, and allows all groups to see how much the others have to offer. It also allows for people to understand how much they often misjudge others based on preconceptions, and to begin to consider everyone as an individual, rather than as “one of those.”

A member of the Changes Project, a CBPR project that explored the impact of welfare reform on adult literacy and ESOL (English as a Second or Other Language), learners wrote in the final report: “What I learned from working in this project first off is, none of us are so great that change couldn’t help us be better people... I walked into the first meeting thinking I was the greatest thing to hit the pike and found that I, too, had some prejudices that I was not aware of. I thought that no one could ever tell me I wasn’t the perfect person to sit in judgment of others because I never had a negative thought or prejudiced bone in my body. Well, lo and behold, I did, and seeing it through other people’s eyes I found that I, too, had to make some changes in my opinions.

Action research helps people better understand the forces that influence their lives.

Just as Paulo Freire found in his work in Latin America, community researchers, sometimes as a direct result of their research, and sometimes as a side benefit, begin to analyze and understand how larger economic, political, and social forces affect their own lives. This understanding helps them to use and control the effects of those forces, and to gain more control over their own destinies.

Community based action research can move communities toward positive social change.

All of the above rationales described reasons for employing CBPR act to restructure the relationships and the lines of power in a community. They contribute to the mutual respect and understanding among community members and the deep understanding of issues that in turn lead to significant and positive social change.

Who should be involved in community-based participatory research?

The short answer here is people from all sectors of the community, but there are some specific groups that, under most circumstances, are important to include.

• People most affected by the issue or intervention under study. These are the people whose inclusion is most important to a participatory effort – both because it’s their inclusion that makes it participatory, and because of what they bring to it. These folks, as we discussed earlier, are closest to the situation, have better access to the population most concerned, and may have insights others wouldn’t have. In addition, their support is crucial to the planning and implementation of an intervention or initiative. That support is much more likely to be forthcoming if they’ve been involved in research or evaluation.
• Other members of the affected population. People who may not themselves be directly affected by the issue or intervention, but who are trusted by the affected population, can be useful members of a CBPR team.

A businessman from the Portuguese community in a small city was an invaluable member of an action research team examining the need for services in that community. He was quite successful, had graduated from college in the US, and needed no services himself, but his fluency in Portuguese, his credentials as a trusted member of the community, and his understanding of both the culture of the Portuguese residents and the culture of health and human service workers brought a crucial dimension to data gathering, analysis, and general information about the community.

• Decision makers. Involving local officials, legislators, and other decision makers from the very beginning can be crucial, both in securing their support, and in making sure that what they support is in fact what’s needed. If they’re part of the team, and have all the information that it gathers, they become advocates not just for addressing the issue, but for recognizing and implementing the solution or intervention that best meets the actual needs of the population affected.
• Academics with an interest in the issue or intervention in question. Academics who have studied the issue often have important information that can help a CBPR team better understand the data it collects. They usually have research skills as well, and can help to train other team members. At the same time, they can learn a great deal from community-based researchers – about the community and communities in general, about approaching people, about putting assumptions and preconceptions aside – and perhaps, as a result, increase the effectiveness of their own research

It’s important that they be treated, and treat everyone else, as equals. Everyone on a team has to view other members as colleagues, not as superiors or inferiors, or as more or less competent or authoritative. This can be difficult on both sides – i.e. making sure that officials, academics, or other professionals don’t look down on community members, and that community members don’t automatically defer to (or distrust) them. It may take some work to create an environment in which everyone feels equally respected and valued, but it’s worth the effort. Both the quality of the research and the long-term learning by team members will benefit greatly from the effort. (There are some circumstances where actual equality among all team members is not entirely possible. When community members are hired as researchers, for instance, the academic or other researcher who pays the bills has to exercise some control over the process. That doesn’t change the necessity of all team members being viewed as colleagues and treated with respect.)

• Health, human service, and public agency staff and volunteers.  Like the previous two groups, these people have both a lot to offer and – often – a lot to learn that will make them more sensitive and more effective at their jobs in the long run. They may have a perspective on issues in the community that residents lack because of their closeness to the situation. At the same time, they may learn more about the lives of those they work with, and better understand their circumstances and the pressures that shape their lives.
• Community members at large. This category brings us back to the statement at the beginning of this portion of the section that members of all sectors of the community should have the opportunity to be involved. That statement covers the knowledge, skills, and talent that different people bring to the endeavor; the importance of buy-in by all sectors of the community if any long-term change is to be accomplished; and what team members learn and bring back to their families, friends, and neighbors as a result of their involvement.

When should you employ community-based participatory research?

There are times when action research may not be appropriate, and there are times when it’s the best choice. How do you decide?

One criterion is the amount of time you have to do the research on the issue or intervention. Action research may take longer than traditional methods, because of the need for training, and because of the time it often takes for community researchers to adjust to the situation (i.e. to realize that their opinions and intuitions are important, even if they may not always be right, and that their conclusions are legitimate). If your time is limited, CBPR may not be the right option

Another consideration is the type of research that’s necessary. Action research lends itself particularly well to qualitative research. If you’re obligated to deliver complicated, quantitative results to a funder, for instance, you may want to depend on professional researchers or evaluators. Most CBPR isn’t oriented toward producing results couched in terms of statistical procedures. (This isn’t to say that action research teams can’t do quantitative research, but simply that it requires more training, and therefore time, and may require an outside source or an academic team member to crunch the numbers.)

Qualitative research

Relies on information that can’t be expressed in mathematical terms – descriptions, opinions, anecdotes, the comments of those affected by the issue under study, etc. The results of qualitative research are usually expressed as a narrative or set of conclusions, with the analysis backed up by quotes, observation notes, and other non-numerical data.

(Almost anything can be expressed in terms of numbers in some way. Interviewers, for instance, can count the number of references to a particular issue, or even record the number of times that an interviewee squirmed in his chair. Qualitative research, however, relies on elements that can’t be adequately – or, in many cases, at all – described numerically. The number of squirms may say something about how nervous an interviewee is, or it may indicate that he has to go to the bathroom. The interviewer will probably be able to tell the difference, but the numbers won’t.)

Quantitative research

 Depends on numbers – the number of people served by an intervention, for instance, the number that completed the program, the number that achieved some predetermined outcome (lowered blood pressure, employment for a certain period, citizenship), scores on academic or psychological or physical tests, etc. These numbers are usually then processed through one or more statistical operations to tell researchers exactly what they mean. (Some statistics may, for instance, help researchers determine precisely what part of an intervention was responsible for a particular behavior change.)

It may seem that quantitative research is more accurate, but that’s not always the case, especially when the research deals with human beings, who don’t always do what you expect them to. It’s often important to get other information in order to understand exactly what’s going on

Furthermore, sometimes there aren’t any numbers to work with. The Changes Project was looking at the possible effects of a change in the welfare system on adult learners. The project was conducted very early in the change process, in order to try to head off the worst consequences of the new system. There was very little quantitative information available at that point, and most of the project involved collecting information about the personal experiences of learners on welfare.

In other words, neither quantitative nor qualitative methods are necessarily “better,” but sometimes one is better than the other for a specific purpose. Often, a mix of the two will yield the richest and most accurate information.

It’s probably best and most effective to use action research when:

• There’s time to properly train and acclimate community researchers
• The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)
• You need an entry to the community or group from whom the information is being gathered
• You’re concerned with buy-in and support from the community
• Part of the purpose of using CBPR is to have an effect on and empower the community researchers
• Part of the purpose of using CBPR is to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

Once you’ve decided to conduct an action research project, there are a number of steps to take to get it up and running. You have to find and train the participants; determine exactly what information you’re looking for and how to go about finding it; plan and carry out your research; analyze and report on your findings; translate the findings into recommendations; take, or bring about, action based on those recommendations; evaluate the process; and follow up

What follows assumes an ideal action research project with a structure, perhaps one initiated by a health or human service organization. A community group that comes together out of common interest probably would recruit by people already involved pulling in their friends, and probably wouldn’t do any formal training unless they invited a researcher to help them specifically in that way. The nature of your group will help you determine how – or whether – you follow each of the steps below.

Recruit a community research team

How you recruit a team will depend on the purpose of the project as well as on who might be most effective in gaining and analyzing information. A team may already exist, as in the example at the beginning of this section. Or a team may simply be a group that gets together out of common concerns. Many CBPR projects aim for a diverse team, with the idea that a mix of people will both provide the broadest range of benefit and allow for the greatest amount of personal learning for team members. Other projects may specifically draw only from a particular population – a language minority, those served by a certain intervention, those experiencing a particular physical condition.

It often makes sense for at least half the team to be composed of people directly affected by the issue or intervention in question. Those numbers both assure good contact with the population from which information needs to be gathered, and makes it less likely that community researchers will be overwhelmed or intimidated by other (professional) team members or by the task

Recruiting from within an organization or program may be relatively simple, because the pool of potential researchers is somewhat of a captive audience: you know where to find them, and you already have a relationship with them. Recruiting from a more general population, on the other hand, requires attention to some basic rules of communication.

• Use language that your audience can understand, whether that means presenting your message in a language other than English, or presenting it in simple, clear English without any academic or other jargon.
• Use the communication channels that your audience is most likely to pay attention to. An announcement in the church that serves a large proportion of your population, a program newsletter, or word-of-mouth might all be good channels by which to reach a particular population.
• Be culturally sensitive and appropriate. Couch your message in a form that is not only respectful of your audience’s culture, but that also speaks to what is important in that culture.
• Go where your audience is. Meet with groups of people from the population you want to work with, put out information in their neighborhoods or meeting places. Don’t wait for them to come to you.

Given all this, the best recruitment method is still face-to-face contact by someone familiar to the person being recruited.

Orient and train the research team

Orientation and training may be part of the same process, or they might be separate. The two have different purposes. Orientation is meant to give people a chance to ask questions and an overall picture of what is expected.

Orientation might include:

• Introductions all around, and an introductory activity to help team members get to know one another
• Explanation of community-based participatory research, and basic information about this project or evaluation
• Participants’ time commitment and the support available to them, if any. Are child care, transportation, or other support services provided or paid for?
• An opportunity to ask questions, or to discuss any part of the project or evaluation that team members don’t understand or agree with

Especially if the team is diverse, and especially if that diversity is one of education and research experience, an important aspect of the orientation is to start building the team, and to ensure that everyone sees it as a team of colleagues, rather than as one group leading or dominating or – even worse – simply tolerating another. Each person brings different skills and experience to the effort and has something to teach everyone else. Emphasizing that from the beginning may be necessary, not only to keep more educated members from dominating, but also to encourage less educated members not to be afraid to ask questions and give their opinions.

Training is meant to pass on specific information and skills that people will need in order to carry out the work of the research. There are as many models for training as there are teams to be trained. As noted above, orientation might serve as all or part of an introductory training session.   Training can take place all at once – in one or several multi-hour sessions on consecutive days – or over the whole period of the project, with each training piece leading to the activity that it concerns. It might be conducted by one person – who, in turn, could be someone from inside the organization or an outside facilitator – by a series of experts in different areas, or by the team members themselves. (In this last case, team members might, for instance, determine what they need to know, and then decide on and implement an appropriate way to learn it.)

Regardless of how it’s done, here are some general guidelines for training that are usually worth following:

• Find a comfortable space to hold the training
• Provide, or make sure that people bring, food and drink
• Take frequent, short breaks. It’s better for people’s concentration to take a three-minute break every half hour than a 20-minute break every three hours
• Structure the space for maximum participation and interaction - chairs in a circle, room to move around, etc.
• Vary the ways in which material is presented. People learn in a variety of ways – by hearing, by seeing, by discussion, by example (watching others), and by doing.  The more of these methods you can include, the more likely you are to hold people’s attention and engage everyone on the team.
• Use the training to build your team. Training is a golden opportunity for people to get to know and trust one another, and to absorb the guiding principles for the work.

The actual content of the training will, of course, depend on the project you’re undertaking, but general areas should probably include:

• Necessary research skills. These might include interview techniques, Internet searching, constructing a survey, and other basic research and information-gathering methods.
• Important information about the community or the intervention in question.
• Meeting and negotiation skills. Many of the people on your team may not have had the experience of participating in numerous meetings. They need time and support both to develop meeting skills – following discussion, knowing when it’s okay to interrupt, feeling confident enough to express their opinions – and to become comfortable with the meeting process.
• Preparing a report. This doesn’t necessarily mean drafting a formal document. Depending upon the team members, a flow chart, a slide show, a video, or a collage might be informative and powerful ways to convey research results, as might oral testimony or a sound recording.
• Making a presentation. Knowing what to expect, and learning how to make a clear and cogent presentation can make the difference between having your findings and recommendations accepted or rejected.

Determine the questions the research or evaluation is meant to answer

The questions you choose to answer will shape your research. There are many types of answers in either of these cases.

An evaluation can focus on process: What is actually being done, and how does that compare with what the intervention or initiative set out to do? It can focus on outcomes: Is the end result of the intervention what you intended it to be? Or it can try to look at both, and to decide whether the process in fact works to gain the desired outcome. An evaluation may also aim to identify specific elements of the process that have to be changed, or to identify a whole new process to replace one that doesn’t seem to be working

Research on a community issue also may be approached in a number of ways. You may simply be trying to find out whether a certain condition exists in your community, or to what extent it exists. You may be concerned with how, or how much, it affects the community, or what parts of the community it affects. You may be seeking a particular outcome, and the research questions you ask may be designed to help you reach that outcome.

Plan and structure your research activity

Given your time constraints, the capacity of your team, and the questions you’re considering, plan your research.

Your plan should include:

• The kind and amount of information-gathering that best suits your project (e.g., interviews, library research, surveys)
• Who will be responsible for what
• The timeline – i.e., deadlines for completing each phase of the plan
• How and by whom the information will be analyzed
• What the report of the research or evaluation will look like
• When, how, and to whom the report will be presented

Anticipate and prepare contingency plans for problems that might arise

An action research group, like any other, can have internal conflicts, as well as conflicts with external forces. People may disagree, or worse; some people may drop out, or may not do what they promised; people may not understand, or may choose not to follow the procedures you’ve agreed on. There will need to be guidelines to deal with each of these and other potential pitfalls.

Implement your research plan

Now that you've completed your planning, it's time to carry it out.

Prepare and present your report and recommendations

The report, as explained previously, may be a written document, or may be in some alternative form. If it’s an evaluation, it might be presented in one way to the staff of the intervention being evaluated, and in another to funders or the community, depending upon your purposes.

Some possibilities for presentation include:

• A press conference
• A community presentation
• A newspaper or newsletter article
• A written report to funders and/or other interested parties

Take, or try to bring about, appropriate action on the issue or intervention

Action can range from adjusting a single element of an intervention as a result of an evaluation, to writing letters to the editor, advocating with legislators, taking direct action (a demonstration, a lawsuit), and starting a community initiative that grows into a national movement. In most cases, a CBPR effort is meant to lead to some kind of action, even if that action is simply further research.

Follow up

An action research project doesn’t end with the presentation, or even with action. The purpose of the research often has as much to do with the learning of the team members as it does with research results. Even where that’s not the case, the skills and methods that action researchers learn need to be cemented, so they can carry over to other projects.

• Evaluate the research process. This should be a collaborative effort by all team members, and might also include others (those who actually implement an evaluated intervention, for instance). Did things go according to plan? What were the strengths of the process? What were its weaknesses? Was the training understandable and adequate? What other support would have been helpful? What parts of the process should be changed?
• Identify benefits to the community or group that came about (or may come about) as a result of the research process. These may have to do with action, with making the community more aware of particular issues, or with creating more community activists.
• Identify team members’ learning and perceptions of changes in themselves. Some areas to consider are basic and other academic skills; public speaking; meeting skills; self-confidence and self-esteem; ability to influence the world and their own lives; and self-image (seeing themselves as proactive, rather than acted upon, for example).
• Maintain gains by keeping researchers involved. There are a number of ways to keep the momentum of a CBPR team going, including starting another project, if there’s a reason to do so; encouraging team members to be active on other issues they care about (and to suggest some potential areas, and perhaps make introductions that make it easier for them to do so); keeping the group together as a (paid) research consortium; or consulting, as a group, with other organizations interested in conducting action research.

CBPR is not always the right choice for an initiative or evaluation, but it’s always worthy of consideration. If you can employ it in a given situation, the rewards can be great.

Community-based participatory research can serve many purposes. It can supply accurate and appropriate information to guide a community initiative or to evaluate a community intervention. It can secure community buy-in and support for that initiative or intervention. It can enhance participants’ personal development and opportunities. It can empower those who are most affected by conditions or issues in the community to analyze and change them. And, perhaps most important, it can lead to long-term social change that improves the quality of life for everyone.

In Summary

Community-based participatory research is a process conducted by and for the people most affected by the issue or intervention being studied or evaluated. It has multiple purposes, including the empowerment of the participants, the gathering of the best and most accurate information possible, garnering community support for the effort, and social change that leads to the betterment of the community for everyone

As with any participatory process, CBPR can take a great deal of time and effort. The participants are often economically and educationally disadvantaged, lacking basic skills and other resources. Thus, training and support – both technical and personal – are crucial elements in any action research process. With proper preparation, however, participatory action research can yield not only excellent research results, but huge benefits for the community over the long run.

What is community-based participatory research?

___You define community-based participatory research as research conducted by and for those most directly affected by the issue, condition, situation, or intervention being studied or evaluated

Why would you use community-based participatory research?

___It yields better and more nearly complete and accurate information from the community

___Involving the community in research makes a positive outcome that meets community needs more likely

­­___Action research trains citizen researchers who can turn their skills to other problems as well

___Involvement in CBPR changes people’s perceptions of themselves and of what they can do

___Skills learned in the course of action research carry over into other areas of researchers’ lives

___A participatory action research process can help to break down racial, ethnic, and class barriers

___Action research helps people better understand the forces that influence their lives

___Community-based participatory research can move communities toward positive social change

Who should be involved in community-based participatory research?

___People most affected by the issue or intervention under study

___Other members of the affected population

___Decision-makers

___Academics with an interest in the issue or intervention in question

___Health, human service, and public agency staff and volunteers

___Community members at large

When should you employ community-based participatory research?

___There’s time to properly train and acclimate community researchers

___The research and analysis necessary relies on interviews, experience, knowledge of the community, and an understanding of the issue or intervention from the inside, rather than on academic skills or an understanding of statistics (unless you have the time and resources to teach those skills or the team includes someone who has them)

___You need an entry to the community or group from whom the information is being gathered

___You’re concerned with buy-in and support from the community

___You want to have an effect on and empower the community researchers

___You want to set the stage for long-term social change

How do you institute and carry out community-based participatory research?

___You recruit a community research team

___You orient and train the research team

___The team determines the questions the research or evaluation will try to answer

___The team plans and structures the research activity

___As a team, you carry out your plan

___The team prepares and presents a report and recommendations on its work

___You take, or try to bring about, appropriate action

___You evaluate the process

___You provide an opportunity for team members to reflect on and discuss their learning and the effects of the experience of being involved in a CBPR process

___You maintain gains by keeping team members involved

Example 1: The Changes Project: Holyoke Community College and SABES West

The Changes Project was a three-year participatory action research effort. Funded by a grant from the Office of Educational Research and Improvement, part of the U.S. Department of Education, its goal was to determine the effects on adult learners of the major changes in welfare regulations, immigration laws, and the workplace that took place in the 1990’s.

Background

The project was participatory in two senses:

• The 42 researchers (34 of whom were able to participate for the whole three-year span of the project) were themselves current and former learners and staffers in adult literacy, GED (high school equivalency), and ESOL (English as a Second or Other Language) programs
• All the effort’s data was obtained from 600 learners in the programs from which the researchers came

All the researchers were learners, graduates, or staff members from one of five Western Massachusetts adult education programs.  The Center for New Americans (CNA) in Greenfield, Amherst, and Northampton and the International Language Institute (ILI) of Northampton are independent organizations that work with ESOL learners.  The Mentor Program at Holyoke Community College provides services to adult learners who are making the transition between adult education and higher education.  Read/Write/Now (RWN) Adult and Family Learning Center, sponsored by the Springfield City Library, offers adult and family literacy and GED classes.  The University of Mass. Labor/Management Workplace Education Program (LMWEP) specifically serves University employees with a range of instructional offerings that encompass basic skills, technology, communication, problem-solving, and workplace-specific areas.

Research team members ranged in age from 22 to 55, and in education from adult literacy learners to holders of advanced degrees.  Among them, they spoke six languages; came from six countries on three continents; and claimed Black, Asian, European, and Hispanic heritage.  Twenty-six were parents (ten being single parents), and ten were current or former welfare recipients.  In general, the researchers comprised a group broadly representative of the community of Western Massachusetts adult learning center participants and staff.

Each of the five sites fielded a research team that worked both independently and in conjunction with the other four.  In addition to its other members, ranging in number from 3 to 13, each team included a site research facilitator (SRF) – a program staff member who was both a member and facilitator of the site’s research team.  SRF’s regularly met together with the program coordinator, and also received consultation from a research advisory group and a methodologist.

The research question the project examined was generated by the administrators and staff of 14 western Massachusetts adult learning programs, in response both to the voiced concerns of learners and to their own observations.

Goals

As stated in the Changes Project report, research goals were to learn more about:

• Effects of welfare reform, immigration reform, and the changing requirements of the workplace on adult learners at five adult literacy programs in Western Massachusetts
• Ways in which adult learners respond to and accommodate immigration reform, welfare reform, and the changing requirements of the workplace as they pursue their goals through education
• How practitioners and programs can alter educational services to be more effective, given welfare reform, immigration reform, and the changing requirements of the workplace
• Ways in which policies can be more responsive to the needs of adult learners as they deal with these three issues

The Changes Project began with a challenge: how to mold a large and diverse group of researchers into a team that could work together creatively and effectively.  Changes Project Coordinator Alex Risley Schroeder recalls that this challenge was made even greater by the fact that the project had not been 100% participatory from the beginning.

“We had to write the proposal, and, because of time and other constraints, we did that without the direct participation of learners.  It was something we struggled with throughout the life of the project.  We had teams of learner/researchers, and said to them, ‘Okay, we’ve promised the funders certain things.  Now what can we, as a group, do to make those things reflect what’s important to all of us?’”

There were only two occasions at the beginning of the project when the whole group, all five site teams, got together: an initial orientation/training and a project kick-off.  The real work of team-building actually began when the first set of data came in.

Because of the differing nature of the five adult education programs and their learners, not all teams researched all three areas of concern.  Most of the work on immigration reform, for instance, came out of the research of teams from ESOL programs. The one workplace education program team didn’t research the effects of welfare reform, because all learners in that program were, by definition, employed; it concentrated instead on changes in the workplace.

Data Collection and Information Synthesis

For all teams, information was collected in three phases from learners in their own programs:

• Individual interviews. Teams developed appropriate interview questions for the area they were researching, and conducted a first round of individual interviews.
• More interviews, focus groups, and surveys. Teams conducted a second round of interviews (including some follow-ups – almost all interviews were followed up with second interviews) with revised questions, and also conducted focus groups in each program and administered surveys project-wide.
• More interviews, more focus groups, and three separate surveys, each specifically aimed at learners with firsthand knowledge of one of the three areas of concern.

After each phase of data-gathering, teams struggled with ways to sort out and report what they had found, and with how to make sense of it. There was an enormous amount of material, most of it in the form of people’s stories about their experiences. Each team found its own ways to organize data.

Each phase ended with an Analysis Fest, a coming together of all the teams to try to integrate and analyze what they had found. The meetings were called Analysis Fests, according to Risley Schroeder, to capture both the excitement and the real joy to be found in gaining and making sense of knowledge.  It was in the Analysis Fests, she says, that the five teams really became a coherent unit.

“Not everyone knew the things that were obvious to some of the teams.  What women on welfare went through, for instance, or what it was like to not be able to speak English.  It was a very powerful learning time for all of us, and it gave everyone a chance to get to know others as teams and as individuals.  Really, to make this work, we all had to understand who we were as a large group, as teams, and as individuals.”

The group wrestled with organizing information and stories they had collected. The challenge was compounded by the fact that some researchers had difficulty reading and writing, or with English. It was made yet more difficult because the researchers, at the beginning, thought of themselves as having no expertise, and therefore no tools for the task.

The first Analysis Fest started out print-heavy. There were transcripts of all the interviews, many painstakingly translated into English from Spanish or other languages. The teams were constrained by what they thought “research” was.  In the subsequent Analysis Fests, as researchers relaxed and started to put the material from different teams together, they realized there were themes common to many of the interviews, regardless of the circumstances of the learners who had been interviewed. Those themes became both the core of the project’s ultimate findings and the key to organizing the data.

Those who had the most difficulty with print or with language were also the most creative in getting around that difficulty. They made up representative stories for each of the issues, used pictures and sound as well as – or instead of – print to show what they had found, and developed dramatic interactive role plays to demonstrate the kinds of difficulties learners faced in dealing with welfare and immigration reforms and workplace issues.

One of the most important findings of the project was that adult learners often depended on fragile networks of support of various kinds in order to be able to attend classes and learn successfully.

There were essentially four types of support necessary:

• Practical supports. These include the kinds of supports most people would understand to be necessary for successful completion of an adult education program: food, clothing, shelter, health care, child care, and transportation. But this category also includes information – about welfare recipients’, immigrants’, and workers’ rights, about the laws that govern those areas, about how to find and enroll in programs, about how to study, about how to find other academic and employment resources.
• Inner resources. Hopes, dreams, motivation, focus, determination, natural ability, prior knowledge, spiritual faith.
• Personal supports. These are both emotional and practical, and come not only from family and friends, but from professionals and the community at large. They include advocacy for individuals and programs, as well as respect, encouragement, and belief in people’s ability to change their lives.
• Institutional supports. These include not only the existence, but the responsiveness and appropriateness of programs, organizations, and institutions that provide help and services to those who need them; state and national policies that speak to the real needs of the people at whom they’re aimed; and regulations that accommodate the real needs and circumstances of those they’re meant to help.

The researchers conceived of these supports as threads in a web, all intertwined and interdependent. Cutting any one thread weakens the whole web, and makes an adult learner’s success less likely. The metaphor of the web made the concept of support easier for everyone to see and understand, and also made it clear how important it was that policy makers see that an adult learner’s success is dependent on far more than what goes on in a particular ABE or ESOL program.

Results

Ultimately, the Changes Project accomplished a number of purposes:  

It gathered a large amount of important, first-hand data about the effects – intended and unintended – of welfare and immigration reforms and changes in the workplace on adult learners and on their chances for successful pursuit of their educational goals.

It had a profound effect on the researchers themselves. Both adult learners and staff members found that the experience of participating in the project changed the way they saw themselves and the world.  More specifically:

• Team members increased their ability to communicate and developed job-related skills, and learners specifically increased their literacy and English language skills.
• Team members felt more self-confident and saw themselves as more competent and more able to contribute as a result of their work in the Changes Project.
• Many team members felt they had developed a voice, enabling them to speak up and be heard in both their private and public lives in ways that hadn’t been possible for them previously.
• Their Changes Project experience led many researchers to become active in their communities, and to change their perspective on political action in general and on their own ability to change things in particular.
• Some of the teachers among the team members found that their view of their jobs and of the learners they worked with had changed.  One explained “This project has given me that ability [to be a reflective teacher] because you really have to listen when you’re doing research.  It’s helped me to let go of...being in charge of how the research goes.  Learning to do that has changed me – it’s changed the way I think about how knowledge gets made.  It’s really different when you do it in a group. What you come up with is much richer.”
• The experience of building and belonging to a community that encompassed people of different cultures, ethnicities, and backgrounds stood out as enormously positive for all participants.  Many talked about the excitement of learning about how other cultures viewed the world, and the rewards of learning to get along with people with different perspectives.
• Team members gained valuable technical knowledge about conducting research, including  how to collect and analyze data (as well as how to engage in logical analysis in general), how to adjust research to respond to the culture of those being addressed, and how to develop the interpersonal skills needed to gain information from interviews and focus groups.

The project served as a model, not only for research, but for adult education. Many of the researchers, both teachers and learners, found that their participation had changed their views of learning.  hey realized that it could be fun, practical, and group-centered.  The skills were being learned, perhaps even more effectively than when instruction was focused on them, in the context of work that had a real-world goal.

The research identified peripheral issues that have great consequences for adult learners. The web of support described above is crucial to the success of a majority of learners; yet policy makers are ignorant of it, and often unwittingly unravel it, thereby making it more difficult for learners to succeed.

The Changes Project drafted a series of recommendations for the field, for learners, for workplaces, and for policymakers.

• Recommendations for the field center on how to make programs more effective by accommodating learner needs, including non-academic ones; by strengthening the web of support; by increasing learner participation in the organization’s decision-making; by providing help in transition to next steps – further education, career planning, employment; and by maintaining an organizational atmosphere of mutual respect and support
• The project recommends to learners that they take responsibility for getting all the information they need, for seeking support, and for becoming involved and active in their programs.
• Recommendations to workplaces include their providing support for workers’ occupational and professional development (including career ladders within the workplace, as well as education and training); maintaining an atmosphere of mutual respect and inclusion; recognizing and building on workers’ strengths, particularly those of immigrants; and being sensitive to the cultural diversity of the workforce.
• The Changes Project’s recommendations to policymakers and legislators, although they concern larger issues, are similar in some ways to those for others. They suggest mandating that welfare and immigration caseworkers develop respectful relationships with those they serve; that more effective policies be developed by soliciting the participation of those at whom those policies are aimed; that information and services be provided in whatever languages are necessary; that welfare policies be responsive to the actual needs of recipients; that education and training count toward the welfare work requirement; and that policies encourage fathers to take responsibility for their children.

Participation in the project led to some action on the part of the researchers. This was both perhaps the most promising and the most frustrating part of the effort. One of the greatest pieces of learning for researchers was that action research really does blur the distinctions among research, analysis, and action.  Each of the research teams initiated some action, not so much as a result of, but rather as an integral part of, its research and findings.

• The Changes Project’s research lit a fire under a writing group of parenting teens who had been interviewed by the team, and who wanted to write about their situation.  Their writings were incorporated into a group-composed “Mother’s Day Letter” (entitled “What’s Going to Happen to Us”) that was sent to legislators, the media, and other adult education programs.  As a result. the local paper published a story on the effects of welfare reform on teen parents, and included interviews with several of the writers.
• The CNA team made presentations of their recommendations to the staff and administrators of the program, and also participated in rallies and direct advocacy with legislators for immigrants’ rights at the State House.
• The Read/Write/Now team initiated a letter-writing campaign to their local, state, and national representatives, detailing the real effects of welfare reform on adult-learner welfare recipients.  They also participated in advocacy for economic justice, and, ultimately, self-published a widely-distributed (to legislators, other programs, and the public) book telling the personal stories of many of those learners.  They hope to dispel stereotypes of welfare recipients, and to help people understand the true bind that many adult learners find themselves in.
• The ILI team’s action was centered closer to home.  Struck by the stories of those they interviewed, team members undertook to provide information and assistance to fellow learners in the program.  As did the others, they also engaged in advocacy for immigrants’ rights by attending State House rallies and other events, and by making presentations about their work.

Finally, the UMass Workplace Education team responded to what they learned in two ways:

• Team members created a support and problem-solving group, called Making Work Work, in order to explore and affect workplace issues.This group has met with a group from the University’s Training and Development Unit to design new programs and involve more staff in planning.
• The second important action to grow out of LMWEP’s research is the development of a career ladder for University workers, a result of meetings between the union and University administration initiated as a result of the Changes Project.

While all of this action has helped those involved feel that they now have more of a voice in their own destinies, it is frustrating as well. Except for the LMWEP progress in the workplace, most of the actions taken had little or no effect on the situations they addressed.  Welfare and immigration reform have gone through as originally planned, with virtually no concessions to the needs of people who are being deprived of the education and training that could help them to become financially successful and better citizens.  Legislators and others have listened sympathetically, but the results have shown little sympathy to the poor and/or newly arrived.

All this can be taken as an excuse to stop trying to bring about social change, but it can also be seen as the beginning of something. A number of people who previously thought of themselves as powerless have seen that they can at least struggle for what they believe in or what they need. They have learned, and have passed on, ways of finding and examining information that give them more choices. And they seem to be willing to continue to practice what they’ve learned. It is in circumstances like those created by the Changes Project that the seeds of change take root. The name seems more than fitting.

Example 2: Community implements participatory action research for proposed business deal

When an out-of-state hazardous waste recycling company approached an economically-depressed New England mill town with an offer to build a plant in the community, the town leaders were overjoyed. The plant would mean a boost to the tax base, and the company promised that both the construction contracts and most jobs at the plant – jobs they described as well-paid – would go to local residents.  Without much discussion, the town tentatively agreed to grant permits, and to waive environmental regulations where necessary.

Some citizens were worried, however. The plant was scheduled to be built on land that drained into the aquifer that fed one of the town’s two wells. Although the company claimed that its process was foolproof and waste-free – all the waste they collected was to be recycled into other products, and there would be no chance of any pollution from their operation – it still seemed ill-considered to grant permission for something so dangerous-sounding without learning more about it.

The group of concerned citizens came from diverse backgrounds. One of the leaders was a teacher; another was a single mother on welfare. The group’s membership spanned the social structure of the town. Arrayed against it were town officials, concerned with growth in a period of economic stagnation, and many of the unemployed and underemployed citizens who hoped that the plant would mean jobs for them, and a better life for their families.

The town split down the middle, with officials refusing to budge, and neighbors and long-time friends ceasing to speak. The members of the concerned group realized they had to understand as much as they could about the company and its process, and set out to do research on both. They learned about hazardous waste recycling, and about both the waste the plant would accept, and the products it intended to turn that waste into.  They sought information on the company, discovering that it had operating plants in the Midwest.

Group members called officials in the Midwestern communities where the plants were located, and learned that there had, in fact, been some environmental problems. They also found out that the plant’s proposed methods were still largely untested, and that there were many scientists who believed that there were serious flaws in them. Armed with this information, the group notified the press and called a public meeting.

After a chaotic, but ultimately productive, evening, the group convinced enough people of potential problems that elected officials were pressured into appointing a commission to study the issue of the hazardous waste recycling plant.  Several members of the concerned citizens’ group were appointed to the commission, along with several other community members who favored the plant.

Over the course of several weeks, the commission met with company officials, and continued to do research on the science of hazardous waste recycling, receiving help from chemists and nuclear physicists at the nearby state university. Many of the commission members who had initially favored the plant became suspicious of the company’s motives, and questioned its truthfulness. Finally, the commission decided to send two of its members to one of the company’s other plants.

The commission delegation found a community that felt it had been lied to. There had been some serious leakage problems associated with the plant, and the company had not responded well.  Rather than admitting the problem and trying to fix it, company officials had tried to pretend there was nothing wrong, and accused the town of being alarmist.  When they were finally forced by a regulatory agency to clean up their mess, they had difficulty doing so, and many in town felt they still hadn’t succeeded.

Furthermore, the company had promised the community good jobs for local people, but the jobs hadn’t materialized. The well-paying jobs at the plant required graduate degrees in chemistry and other sciences, and the low-paying jobs local workers were offered were often part-time, and at wages no better than those available at Wal-Mart.

The New England town commission had heard enough. They returned and convinced the town to vote against the permits the plant needed.  Before that could happen, however, the company withdrew its offer, and looked for a site elsewhere in the country. Some people in town were still angry about what they saw as a loss of jobs, but most felt they had dodged a bullet.

A citizens’ group had turned itself into a participatory action research project, and had used the results of its research to take effective action on an issue of public importance.  Many of those involved in the effort are still active in the community. Others felt their moment of “fame” came at too great a cost, and have withdrawn from public participation.  In a small town, that is one of the risks of becoming involved in action research.

Example 3: True Collaboration for a Win-win in Community-engaged Research

“When we think about the word ‘partner,’ we can think about a significant other. If I were to say, ‘what do you want out of your partnership with your significant other?,’ you may say things like, ‘you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year.’ Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things. We want someone who is with us, who engages us from the very beginning of a process--so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this? I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But, as a partnership, we have to work through those solutions. Those are some of the things that we look for in a community-based research partnership, that flexibility and understanding. One of the things I noticed as I went from being a researcher to being in practice is that you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?” What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information. So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.”

Example 4: Compassion Fatigue in Evaluators

Evaluators working with trauma-exposed populations often develop compassion fatigue in the same way caregivers might. Compassion fatigue is the psychological strain one experiences due to sustained empathetic engagement. Dr. Stacy Huff's research analyzes and provides suggestions for managing evaluator compassion fatigue.

Compassion fatigue affects everyone differently. Many factors play a role in an evaluator's likelihood of developing compassion fatigue and their ability to overcome it. Things like the type of evaluation, personal identity, and organizational support all have an effect.

• Type of evaluation: Evaluators working with refugees, Indigenous populations, and those facing systemic oppression are more likely to experience compassion fatigue. They might also face ethical tension due to their proximity to trauma-exposed communities.
• Personal identity: Historically marginalized individuals, such as people of color and women, may face high mental strain from evaluation work. Evaluators who work with groups with whom they have a personal connection are more likely to experience compassion fatigue when hearing stories similar to their lived experiences.
• Organizational support: Many evaluators do not have a support system in place and are thus left to manage this large emotional weight by themselves.

Recommended resources for evaluators to recover from fatigue.

• Understanding your struggle: The Compassion Fatigue Awareness Project provides resources for understanding and preventing compassion fatigue within the workplace.
• Meditation: Huff specifically recommends The Guided Meditation for Compassion Fatigue to manage compassion fatigue.
• Overcoming burnout: The Reducing Compassion Fatigue, Secondary Traumatic Stress, and Burnout workbook is a great place to start!

To read Dr. Huff's blog post: https://aea365.org/blog/drg-tig-week-compassion-fatigue-in-evaluation-t…

Tool: Involving Community Members in Evaluation

When community members are involved in evaluation, all aspects of the project, from identification of the question to dissemination of results, are more likely to reflect the priorities, values, and experiences of the community. In any evaluation or research project, community involvement can be considered along a continuum, where community members have increasing levels of influence and responsibility. This planning framework, which can be used by organizations responsible for conducting or funding evaluation projects, can increase transparency, establish shared expectations, serve as a starting point to further clarify roles, and increase accountability. Read more from the Amherst H Wilder Foundation website.

Download the PDF for the tool here.

 

Tool: Data Walks and Narrative Methods

A Data Walk is a participatory process that engages stakeholders, including community members, in interpreting visual data such as maps, charts, photos, and statistics. Stakeholders should be engaged in the design of Data Walks to select which data to display and identify appropriate spaces to host the walk. This collaborative approach increases ownership, strengthens interpretation, and supports sustainability by positioning community members as co-creators of insight rather than passive recipients of information. 

Evaluators can consider using the Theory, Context, Characteristics, and Methodology (TCCM) framework to organize and reflect on what emerges from the walk. Using a structured reflection process ensures that community narratives are not only heard but thoughtfully integrated into context-sensitive evaluation and decision-making. Read more from the American Evaluation Association website. 

• Who are community leaders, evaluators and funders?

• Why should you understand the interests of these groups?

• When should you understand the interests of these groups?

• What are the interests of these groups?

• How do you determine interests?

You want to know more about how your group is doing, but others you work with want to know whether you are making a difference. Welcome to the world of evaluation. If you are a community initiative, you will want to evaluate your effort. You will need to devote some time and energy to planning the evaluation process. Like many other aspects of community health and development, an evaluation will ultimately be more beneficial if you spend the time and energy searching for ways to successfully begin and complete an evaluation.

One step in the planning process includes understanding and recognizing the interests of stakeholders in the evaluation. The stakeholders include community leaders, evaluators, and funders, and you will want to know how the evaluation will be used by each of them.

The evaluation should respond to the interests of those three stakeholders, and nothing is more productive than designing it together. The evaluation can serve the community leaders' interests, the funders' interests, and the evaluators' interests in a single useful product, if you know what they want before you start. It's important to define the stakeholders interests in using the evaluation so that it can focus on optimally answering questions important to all of them.

What do we mean by needs and interests? Needs and interests are those qualities which community leadership, evaluators, and funders see as important for doing their jobs well. Because each of these stakeholders is looking at the evaluation from a unique perspective, it helps to recognize those differences, and incorporate them into the evaluation.

For starters, let's consider why you'd want to conduct an evaluation in the first place.

There are many basic reasons why stakeholders want an evaluation:

• To be accountable as a public operation
• To assist those who are receiving grants to improve
• To improve a foundation's grantmaking
• To assess the quality or impact of funded programs
• To plan and implement new programs
• To disseminate innovative programs
• To increase knowledge

A stakeholder may want an evaluation for one, two or all of these reasons. Evaluators may want to increase knowledge, funders may want to improve grantmaking and community leaders may want to assess quality. Community leaders may not want to answer more than a phone interview by a student intern, evaluators may be interested in systematic, disciplined inquiry, and funders may look for accountability.

When it comes time for evaluation, you don't have to be specialists in order to make good decisions about what you will do. You should, however, be knowledgeable about uses of evaluations and how they match the many interests involved so that you can make informed choices.

Who are community leaders, evaluators and funders?

Community leaders

May include staff, administrators, committee chairpersons, agency personnel and civic leaders, and trustees of an initiative. They may have little knowledge of evaluation, nor feel they have much time to provide data or read data reports. Yet, the evaluation must be responsive, useful and sensitive to their decision-making requirements. They often are interested in how to improve the functioning of their initiative.

Evaluators

Are often professionals, though anyone can design and implement an evaluation. There are several professional associations that support evaluators and have established standards of practice, such as the American Evaluation Association or its Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. Evaluators can be private consultants, university or foundation staff, or a member of the initiative. Evaluators are often interested in the systematic production of useful, reliable information.

Funders

Are those individuals or organizations who provide financial support for the initiative. They might include program officers or other representatives of government agencies, foundations, or other sources of financial support. Some funders have built a formal evaluation into their regular activities, but they are in the minority. Funders are often interested in whether the use of their funds is having an impact on the problems facing communities.

Why should you understand the interests of these groups?

So, you understand the idea in principle, but why do you need to understand the needs of leaders, evaluators and funders? The information needs of various groups can be very different, so it's important to take into account the kinds of information that will be convincing and useful to the target audiences. Knowing this will help you decide what information is needed and the tools you could use to obtain it.

While you may know your group does good work, chances are good that other important members of the community do not know what you do. Consequently, others who have supported and encouraged your efforts will want to know what has worked, and what hasn't worked; and what should change and what should stay the same. Because these groups or individuals might be instrumental in assisting your work, financially or otherwise, it makes good sense to include their needs in the evaluation process.

Even more important is the requirement that the information used is effectivel. The question is: To whom is it useful? If there's no direction to your information gathering, you can collect just about anything you want, but so what? If it doesn't matter to anyone else, it is meaningless. If I collect information about the number of people that my agency serves, I may find that useful, especially if I'm reimbursed for that number. But what if someone really wanted to know if the efforts of the agencies in town had an impact on a health problem. The number of people my agency serves might not be that useful.

The interests, which helps us determine the information we need, lead us to develop tools to collect it. In other words it is the interests of the stakeholders that shape the inquiring.

• What are the interests of the stakeholders?
• What information will help them?
• What questions will you ask to get that information?
• What tools will help you collect it?

In the long run, including the stakeholders in the process will lead to greater collaboration and organizational capacity to solve community problems. Understanding stakeholders' interests will enable you to employ your resources better. Knowing what everyone wants and needs will help you plan the optimal evaluation.

When should you understand the interests of these groups?

You will want to identify stakeholders from the get-go. By going through a process of stakeholder identification before you begin evaluating, you will be able to obtain their views and incorporate their ideas and needs into the evaluation itself.

Of course, the sooner you identify the needs and interests of those groups, the sooner you will be able to gain understanding of the different issues each group is interested in without wasting time or money. You also have to be watchful so that if interests change, you can adapt to those changes in a timely manner and keep your evaluation valid.

What are the interests of these groups?

How do we find these people?

First and foremost, you and other members of the group will need to sit down, pour a cup of joe, and grab a pencil. Think about the individuals and groups that have needs that should be addressed in the evaluation. You should try to figure out what their interests are.

Of course, some people may ask, "Why them? Our group's interests and needs should be the focus of the evaluation." In one sense, this is true. One of the main purposes of the evaluation process includes providing feedback and ideas for the group itself so members can improve and strengthen their efforts. But remember, everyone is in this together, the community funders and those who will conduct the evaluation. You want the best information possible information that will help you make the best decisions.

But, at the same time, there are other factors to consider.

Over the course of your brainstorming session, you should identify as many stakeholders as you can.

To identify stakeholders, ask yourself these questions:

• Who provides funding for our initiative?
• Who will conduct the evaluation?
• Who do we collaborate with?

Once you know who these people are, find out what they want. To do this, let's take a look at the groups. Then, we'll talk about the specific needs and interests of members of each of these groups.

Community leaders

What will this group need from your evaluation? The information should be:

• Clear and understandable: They may have limited knowledge about the goings-on of your group, or about evaluations. Immediately, then, you know that the evaluation must be clear and understandable.
• Efficient: They probably have a variety of different responsibilities which demand their time and consideration, so they won't want to waste time reading information irrelevant to their needs.
• Responsive: They may include decision-makers that can affect the future of your group. Therefore, your evaluation needs to be responsive to their decision-making requirements.
• Sensitive: They will want to know what the initiative has accomplished, so the evaluation should be sensitive to the activities and accomplishments of the initiative.
• Useful: They will include decision-makers for the initiative, so the evaluation needs to show them how their efforts can be improved.

Evaluators

They will be assessing the effectiveness of the initiative in meeting its goals. What do they need to get out of the evaluation?

• Input: The evaluation team needs to receive input from the initiative's clients --including community leadership, funders, and members of the initiative itself, in order to know what the clients want to learn about the initiative.
• Accurate and complete information: In a similar vein, the evaluators need accurate and complete information in order to answer the questions posed by the stakeholders.
• Cooperation: Finally, the evaluators will need cooperation from participants and officials in order to obtain needed data.

Funders

They will need:

• Clear and timely reports: Because of their responsibilities for making decisions concerning the continuation of financial support, the funders will need information about the progress of the initiative.
• Evidence of community change and impact: Funders will need to be able to measure the success of the initiative and report this to their own trustees or constituents.

How do you determine interests?

Now you know what interests you're looking for, you have to determine a way of finding them. You need to match people with what their interests are, whether it's through a survey, an interview, or some other method. Failure to determine interests is often the source of problems and misunderstandings along the way and can became disastrous if it turns out that different stakeholders had different expectations and priorities.

Some of the questions that you can ask stakeholders to match them with their interests in an evaluation are:

• What are the evaluation's strengths and weaknesses?
• Do you think the evaluation is moving toward its desired outcomes?
• Which kind of implementation problems came up, and how are they being addressed?
• How are staff and clients interacting?
• What is happening that wasn't expected?
• What do you like, dislike, or would like to change in the evaluation process?

From the answers you get, you can determine what each party wants out of the evaluation. You can also group those who have similar interests. For instance, you may find out that a community leader and an evaluator are interested in improving their managing abilities through the evaluation process. You have made a match, and those two will work toward a common goal.

Here are some ways of determining interests and matching them with stakeholders:

• Interviews: Get a representative from each group of stakeholders and ask away. Be direct in your questions so that you can quickly get to the point you're trying to make, that is, what interests this stakeholder has and if they match with some other stakeholders' interests.
• Surveys: You can send out a written questionnaire to assess how the stakeholders rank their interests and which group wants what. The survey must be succinct and direct, asking clear questions about the evaluation in terms of quality and goals. Survey results are easy to utilize and can be helpful for the evaluation presentation.
• Phone surveys: They can save you time and money, if you're doing them locally. You can use the same questions you would use in a written survey, but leave more space for commentary, as people tend to talk more when speaking to a person on the phone. Just be sure your phone surveys don't stray from your objective.
• Brainstorm sessions: Arrange a meeting with stakeholder representatives and brainstorm interests and possibilities for the evaluation's outcome. Bring up problems such as continuity of the program, obtaining funds, coordinating activities, and attracting staff, and let stakeholders have their say. Everybody will come out from the brainstorming session with new ideas and a much better notion of everybody else's ideas.

Besides these methods, you should always conduct a survey after the completion of the evaluation. This will benefit external audiences and decision-makers. Remember, if changes need to be made, don't be afraid to follow through with them.

Remember that decisions about how to improve a program tend to be made in small, incremental steps based on specific findings aimed at making the evaluation a better process for all stakeholders involved.

Now armed with this list of needs and interests, you can find or develop the tools to obtain useful information. The next sections will explore ways to select an evaluation team and present some key questions for the evaluation process. Later, we will be discussing how to evaluate your community initiative!

In Summary

Once you have a clear idea of what each stakeholder really wants, you are very likely to succeed in your evaluation. Be sure to revise frequently the interests of all the stakeholders involved so that you don't lose focus of what you're looking for with your evaluation. The hard part of your evaluation work starts now!

___You understand why it is important to determine the interests of the stakeholders

___You know who the stakeholders are

___You understand the community leaders' interests

___ You understand the funders' interests

___ You understand the evaluators' interests

___You know when you should understand those interests

___You know how to identify stakeholders

___You know the needs of each stakeholder

___You know how to determine interests

___You understand the interests involved in an evaluation

Example 1: Reasons to evaluate

Different organizations will have different reasons to implement an evaluation, and different interests will be at stake. For instance, some organization may want to focus on increasing the quality of its grants programs. Some other organization may be interested in becoming more accountable. Being aware of these interests in fundamental to understand how interests must be balanced. Here we offer a table with hypothetical examples of the kinds of interests that organizations may have.

Foundation / Organization	Interest
The Meyer Memorial Trust request-for-proposal program	Improve grantee programs, improve grantmaking
L.J. and Mary C. Skaggs Foundation program for arts organizations	Be accountable, improve grantmaking
Winthrop Rockfeller Foundation at-risk youth program	Improve grantee program, disseminate successful programs
Cities in Schools program	Increase the state of knowledge
Technology for Literacy project	Be accountable, improve grantmaking, assess quality or impact of program
William Penn Foundation career / vocational program	Be accountable, improve grantee programs, improve grantmaking
Gautreaux program	Increase the state of knowledge
W.K. Kellogg Foundation grants	Be accountable, improve grantee programs, improve grantmaking, assess quality or impact of grants / programs, plan new programs, disseminate successful programs

Example 2: Leading a community

Ultimately, the community is most affected by the results of an evaluation. That's why community leaders play a key role in the interest game of evaluations. Lisa Adkins knows enough about her community. She is the executive director of Youth Friends in Kansas City. Through Youth Friends, students are linked to volunteers with a shared interest, whether it's music or sports, computers or a special hobby, to create an environment in which every young person who needs a caring adult in his or her life has one.

Lisa talked to us about how her interests as a community leader interfere in the evaluation process. However, according to Lisa, conflict of interests has not been a problem in her evaluation process. She said that the work between evaluators, funders and herself as a community leader is usually of a cooperative nature, and everybody works together toward a common goal. She considers the evaluation process a positive experience for her organization, once they get a chance to review their programs and change what needs to be changed. So, evaluations are a constant element in her work

Her main interest, Lisa says, is to receive feedback on the programs Youth Friends provide, and evaluations are a great way of getting a response from the community. Youth Friends can adjust their services as the necessity arises from the community. Lisa points out that she is always the most interested in the evaluation's results since they could change her approach to a certain program and drastically improve it.

Example 3: Evaluating the evaluator

When it comes to figuring out interests in the evaluation game, the evaluator is a key character. Vince Francisco knows this game well. He's worked as an evaluator with diverse, community-based organizations doing hundreds of program evaluations. He has done evaluations of community policing programs, chronic and cardiovascular disease prevention programming, youth development, drug-abuse prevention, alcohol-use prevention, tobacco control initiatives, teen pregnancy prevention, and general youth development initiatives. Here Vince talks about how the evaluator's interests influence the score of the game:

"I try to find out immediately who the key players are and what they want to have answered. As an evaluator, what I bring to the table is a critical set of questions that I ask folks. Sometimes it's just sitting there watching folks in action, and other times it's asking key people very pointed questions about what they want to do, why they're doing it. I work with folks to take it to the next level and make it a little bit more formal in terms of coming up with some evaluation from very different points of view. If we don't understand their program, we can't go anywhere with their evaluation. I get a handle on their program (and I) will help them make their lives easier, help them run their program or initiative better, help them get more money, influence key players in the community to be on their side, the big issues. "

To find the other interests involved in an evaluation, Vince says that he talks to people until he has their roles figured out: "I look for the loudest voices, and then I look for the quietest voices those usually are the key players. The loudest voices usually speak for themselves, and not for the community. And the people who are the absolutely quietest, are the people who really have a voice in the community, but they're being drowned out by the folks who are loud. Very loud people usually get their way and they become key players so I want to know who they are and what their agenda is. And then I find whoever are the quietest people, and they're usually the people who sit in the background. I start talking to people and eventually there are some people that will talk back. I guess I just start talking to folks and (I work with) the ones that end up talking back to me. I combine a lot of talking."

Of course, Vince says, his own interests also influence the outcome of the intervention. To balance out this equation he thinks of the whole group or the whole community as his client. Talking to people, getting information, and shaping them, Vince produces something that works for his clients: "I outline and define the program : What's the initiative? What does it look like? Who are the key players? What are the program components and elements? Who are the targets of change? Who are the agents of change? Once I get a good handle on that, I connect that to some key questions. I collect evaluation data. What evaluation data would people believe as being evidence that these things are actually being implemented, and what would be key evidence that these things are successful? I combine that with appropriate components and elements in a systematically and very logical fashion. I look for ways for economizing procedures for collecting data. I get right to the chase. Don't do a whole lot of surveys that people have to spend time filling out."

Sometimes, however, interests conflict. Vince describes dealing with this kind of situation: "Usually, I'll find a few people that are really on board and very interested in evaluation. A lot of people just hate evaluators and think it's only a bunch of stupid surveys. My goal is to make it simple as possible. So my trick is to go in and sort of navigate the waters. Work with the people who want to work with me at first, and then convince everybody else that this is useful to them." Vince points out that, depending on how political the footfall of the program is, the evaluation may suffer.

Vince doesn't have a recipe to deal with all the interests that are involved in an evaluation: "All I know is that I juggle all the balls and keep them all in the air. And I look at all the different stakeholders. Who is the funding agent, and what is their interest in all this. I have to make sure their interest is represented. I try to give everybody the same exact information, so we're all moving forward together. That's the way I approach these things I try to change. I get everybody talking to each other in a real way and I don't get into the rumor way." Vince says that evaluators have a unique position because they can hold people accountable for their words: "I don't know if (the evaluator) is any more or less important than any other (interests). But without it, you don't do well. If we want to get lasting outcomes, we have to focus on behavior change. We have to focus on changing the systems that are controlling behavior."

Example 4: The funders role

Jeff Usher represents a very important interest in the evaluation process: the funder's. He is a program officer at the Kansas Health Foundation, a philanthropic organization that provides grants to improve the quality of health in Kansas. Here he talks about implementing evaluations and adjusting interests:

"We want to (see) change, we want to make a behavior change or community change. At the foundation, probably 10 to 20 percent of the overall budget will go to evaluation. The evaluation, of course, will be oftentimes both qualitative and quantitative in nature. We want to see the outcomes, whatever it might be; if it's teenage pregnancy, we'd like to see reductions of teenage pregnancies. So we like to see significant outcomes. We normally contract with people that have expertise in that particular area to conduct the evaluation."

Once the outcomes are in, Jeff says that it's time to decide what steps to take: "If it doesn't appear that the initiative is working, we clearly start losing interest in continuing the project in that manner. Oftentimes the evaluation portion of it is after the fact; the initiative is already over with, for the most part. If it comes back and it looks like the initiative was successful and the outcomes were going in the direction that we anticipated, then we may do another round of that initiative. For instance, the School Community Sexual Risk Reduction project, which is in its second phase now. Depending on the initiatives, if it appears that there is something else going on, we may still continue to work with those, or we may want to tweak that initiative a little bit to make it be a little more effective."

Jeff says that balancing interests among funders, community leaders and evaluators is not much of a problem for him: "We don't seem to have much of a problem when working with evaluators and as far as a private foundation, we don't hear a lot of problems." Jeff goes on to say that sometimes he experiences problems with some evaluators regarding the evaluation methods: "We may get some evaluations back that are measuring process rather than outcomes, and we clearly want to see outcomes. As far as balancing the grantee and the evaluator, normally, the evaluator is a grantee too, so in a sense there is even accountability that we want to see from the evaluator, and we would ask for that. Clearly there are certain things we want to see in that evaluation. For instance, we just did a memorandum agreement for an evaluation on a project that we are just going to be kicking off this next year. We want to see a change in the culture of campus, centered around binge drinking or heavy drinking. We outline those particular things in the memorandum when we start negotiating, so we expect to see that."

One of the funder's interests is to keep an eye on how the initiative conducts its business. Jeff says that in every memorandum agreement, they allow his foundation to perform site visits to have a third party come in, audit financial books, and see how the program is going: "Rarely do we follow up on those. The only time we would look at those is when we get a status report, and we ask for status reports from all of the grantees with what we want to see, and they haven't met those objectives. So, for almost all of our major initiatives, we'll have an evaluation component and have someone looking at it. We want to see behavior change, systems change, changes in culture centered around the health and well-being of communities, and individuals in communities."

The opening four sections of this chapter have been largely about theory: what evaluations of community-based initiatives and organizations are about, why they re important, and some essential ideas about how they can be conducted and whose interests they need to serve. This section and the next are about actually starting the process of setting up an evaluation, i.e. choosing the evaluators that will carry it out, and planning what it will look like. The two are very closely connected, but it will probably make sense to choose your evaluators first. Then, in consultation with the organization, the community, and the target population, they'll come up with a plan for helping you get the information you need to become, or to continue to be, as effective as possible.

Why pay attention to the selection of evaluators?

An evaluation is not simply a matter of looking at your organization or initiative and saying, "It's doing okay." It should examine what you're doing from a number of different angles and perspectives, and should give you not only a clear sense of whether or not you're accomplishing your goals, but of where your strengths and weaknesses lie, of how to improve what you're doing, and of new directions to explore. No matter how well your evaluation is planned--and it should be well planned--you'll still need evaluators that have the skills and knowledge to look at your particular situation accurately.

The real reason for an evaluation is its usefulness in improving the organization. The more, and the more accurate, the information you get, and the better it's analyzed, the more useful it will be in helping you identify and build on your organization's strengths and pinpoint and correct the areas it needs to work on. A well-conceived evaluation can also help you adjust to changing needs in the community or target population, learn more about the implications of the issue you're working on, provide a base for advocacy, and help with fundraising. As a result, it's important that your evaluators do the best job they can.

You may be able to hire evaluators, or you may be choosing volunteers from your staff or the community you serve. Or you may simply be choosing a planning team that will go on to select evaluators. In any of these cases, the particular individuals you select will influence the shape of the evaluation you get, and what kinds of results you get from it.

Although this section treats both professional and non-professional evaluators, it is understood that most organizations will either conduct evaluations in-house (i.e. using staff members, participants, and/or organizational records to gather and analyze information), use an individual or team from the community, or find a professional to help with an evaluation without charge. We've tried to cover as many possibilities as we can, so that organizations will be able to consider how best to use the resources they have to get the kind of evaluation they want.

The same is true for community or other planning teams. A planning team is only one of many possibilities, depending again on the resources of an organization and those resources include time. An evaluation that's not perfect, but is useful in some ways is better than no evaluation. A planning team is a great idea, but not if the thought of putting together and working with one prevents an organization from attempting an evaluation. The purpose of this and all Tool Box sections is not to create boxes for you to fit your organization into, but to provide as many options as possible.

If you hire professionals, typically either a consulting firm or a group from a university, they'll have particular ways of doing research, particular prejudices about what an evaluation should cover, and particular interests, all of which may or may not coincide with what you need or what your community wants. Community and in-house evaluators also, depending upon the individuals involved and the groups they represent, will have their own prejudices, interests, and needs which will influence how they view the evaluation process and the evaluation itself. The same goes for graduate students or others who might donate their time to conduct or assist with an evaluation. It's extremely important to select people whose agenda matches your own, or who will put their own agendas aside and respond directly to the needs and desires of the organization or initiative and the community. Only then will you wind up with an evaluation that best serves your purposes.

When should you choose evaluators?

The short answer to this question is "as soon as possible." Evaluations should not start at the end of a project. They should be ongoing, so the information can continually be used to improve what you're doing. A planning team should probably be chosen when the organization or initiative begins, or even before, so that an evaluation can look not only at what the organization has done, but at how it changed over the evaluation period.

Many funders require evaluation from the start in their grants. The grant for an adult literacy program run by a Massachusetts employment training agency, for instance, stipulated that 10% of the money be used for evaluation in the first year of the program. A University of Massachusetts graduate student was hired at the start of the program, and he conducted an evaluation which involved his visiting classrooms several times during the year, meeting regularly with the program staff, and interviewing participants. By the end of the year, he had gained not only a complete picture of the program, but a sense of its movement and growth over time as well. The evaluation he presented was extremely helpful in guiding the program over the next year.

As will become apparent below and in succeeding sections, it's important to know your community and its needs, as well as the needs of your organization or initiative, before you set out to evaluate. If your organization or initiative is just starting out, it may sometimes make more sense to wait a while until you have a clear understanding of the context of your evaluation before beginning to pick evaluators.

Context of the evaluation

Relates to whether an organization with the resources to do so may or may not decide to hire professional evaluators. The context refers to the unique situation of the evaluated organization or initiative and its community, the combination of geographical, historical, political, social, cultural, and other factors that form that situation. An understanding of that context is basic to an understanding of how to approach this particular community to evaluate this particular organization.

An example: An evaluation of an attempt to establish an adult literacy program in a rural area showed that, in the first year, the program failed to attract many students. It looked like a failure, and was in danger of losing its funding. But the staff of the program knew two things that the evaluators and funders didn't: first, that the area's inhabitants were country people, and were suspicious of and slow to accept "outsiders;" and second, that other programs had been started, but had always only lasted a year, because they didn't attract enough students. The staff argued, convincingly, that the missing element in earlier efforts was time, and that by leaving, the earlier programs had confirmed area residents' feelings that the outsiders didn't really care about their problems, and weren't serious about helping. The program continued, and by the end of its second year was thriving, because the local people believed that it would stay. Now, 15 years later, the program has become a fixture in the community, and is always full to capacity.

How should you decide between professional and community or other volunteer evaluators?

Evaluation is actually a type of research: it entails systematically gathering information and then drawing conclusions from it. There are people who engage in research of various kinds: students and academics and many consultants who do research for a living. If evaluation is research, it must make sense to hire trained researchers to conduct evaluations, right?

Well, yes and no. In many cases, it is true that a professional researcher will do the best job. But in other situations, that may not be true at all, for a variety of reasons.

Before you decide who's going to do your evaluation, there are a number of things to consider:

Money

This may be the most important, especially for newer and smaller organizations. Generally, the more grass roots your base is, the less money you have. If you have a funder who requires, or is willing to pay for, evaluation as part of a grant, then hiring professionals is at least an option. If there's no money available and no possibility of getting any, community or other volunteers may be your answer.

In a situation where you believe it's necessary to have professional evaluators, or at least professional guidance, you could try to fundraise what you needed (a difficult alternative, unless you have a lot of lead time and a great community base), or to find a volunteer mentor at a local college or university. Often graduate students, particularly, are willing to consult, or even to conduct the evaluation, without charge, either for the experience, or because the research will fit into their dissertations or some other work they're already doing. Other possible sources of various levels of assistance might include other health or human service agencies or initiatives; local government; or a fundraising drive or proposal (to a local foundation, for instance) specifically aimed at funding an evaluation.

In one instance, the instructor of a graduate course in Program Evaluation solicited proposals from community agencies and organizations for help in getting their programs evaluated. The organizations with successful proposals were each rewarded with a team of grad students who worked on evaluations with them over the course of the semester. At the end, there was a public event to which all agencies were invited, where they learned the overall results, and received copies of "Program Evaluation Tips," written by the grad students. This process produced useful evaluations for the agencies, invaluable hands-on experience for the grad students, and a strengthened connection between the university and the community.

The reality of money aside, if you have the option of considering professionals, the issues below come into play.

The complexity of the evaluation

For some organizations or initiatives, an evaluation may be relatively straightforward. They simply want to measure whether their very specific goals are being met. After a public education initiative, are significantly more children being vaccinated this year than last? Are learners in a literacy program gaining proficiency in reading, writing, and math? What percentage of participants in a substance use treatment program is still substance-free after a given period of time? If these are the types of questions that need to be answered, an evaluation can be fairly simple. But what if the questions asked are more complex? Was it the initiative, or some other factor, or a combination that resulted in increased vaccination rates? What are the environmental factors that influence literacy learners' progress? What do the substance users who backslide have in common? The methods to find the answers to those questions may need to be more complex as well. This issue is muddied, however, by the fact that sometimes a deep knowledge of the community may do more to answer complex questions than a knowledge of research methods. If you have the option of using professionals, you'll have to decide whether or not that's actually the best idea here.

The type of information desired and how it needs to be analyzed

If you're collecting quantitative data (i.e. numbers) it's often not possible to simply look at them and draw conclusions. There are statistical procedures that can be applied to the numbers to tell you what they really mean, how significant they are (i.e. what the odds are that differences or changes were actually caused by what you're measuring), and what they imply about other ways you could operate. If you need this kind of information or if funders or others are specifically asking for it, it would probably be helpful either to hire professional researchers or to find some volunteer professional guidance.

Sometimes conducting such statistical procedures can tell you important things you never expected to find out. A researcher trying to determine what had caused a group of women to return to school in their thirties and forties found that a major factor had been their having to care for younger siblings when they were teenagers. He had barely noticed this in interviews, it had been overshadowed by such more dramatic factors as divorce, substance use, and domestic violence, but computer analysis showed it to be extremely important. In fact, when the researcher went back and talked to several of the women about it, they acknowledged that this caregiving role had been a major reason for their not continuing their educations in the first place, and still loomed large in their lives.

What you're using the evaluation for

Is your evaluation solely to help you become as effective as possible in what you do? Is it to increase your credibility in the community? Is your continued funding dependent on it? Is it a combination of two or all of these? Your answers to these questions will help to determine what kinds of questions you need to ask in the evaluation, what form the answers need to be in, how complex the evaluation needs to be, and whether it's necessary to try to hire researchers or not. The choice of professionals or community volunteers here will depend not only on the factors above, but on your particular situation as well. Professionals may be able to better analyze the information, but may not have access to the same information that people from the community would have (or may in fact have better access, because they're perceived as neutral). Once again, you need to determine what's best for your situation.

How you want the evaluation to be perceived

Some organizations or initiatives may want to demonstrate the extent of their community-based or grassroots orientation by making sure that their evaluation is community-based as well. Others may want to show that their evaluation is objective by hiring evaluators with no connection to the community. Still others, for philosophical or practical reasons, may want to be sure to involve those they serve or benefit in the evaluation process. There are many variations and possibilities here, all legitimate, all right for some organizations and wrong for others. As with anything you do, it's important to be consistent with your mission and principles when you consider how to choose evaluators.

What should you look for in choosing evaluators?

There are characteristics you'd want in professionals, characteristics you'd want in a community planning or evaluation team, and some characteristics you'd want in all of them. We'll examine each of these possibilities separately.

An evaluation can be structured in a huge number of different ways. The split between professional and non-professional can blur in many of these possibilities. One might get a professional evaluator for free, for instance, because of her interests or her commitment to the work of the organization or to the community. While an evaluation could be carried out solely by either a professional or by a community or in-house group, there are many variations on these themes.

A paid evaluator or team probably will work at least to some extent with a group from the organization or the community. A paid evaluator might work with a community or organization planning team to help plan the evaluation, and then either conduct the evaluation itself without further input from the planning team, or withdraw entirely to have the planning team take over arranging and conducting the evaluation. A local planning team might plan the evaluation entirely on its own, then hire a professional to carry it out, or just to analyze the data. Professionals might work alongside a community team through the whole process. Whether and how you choose to use professionals, community or in-house planning teams or evaluators, etc., depends on what makes sense for your organization.

One evaluator, for instance, tells this story: "I was hired to help community members evaluate HIV/AIDS services in the community. The community members here were a mixed group mostly agency folks, but some nonagency people living with HIV /AIDS. I asked for some volunteers to plan the evaluation together with me, and was pleased when about half a dozen came forward.

"When we met, I set most of the agenda, raising the basic questions, such as 'What do we want to evaluate?' and 'How do we want to evaluate it?' We decided on a survey. Then the next questions became what kind of survey we needed, how it should be administered, who should administer it, and how the data should be collected. Our work style was to put each of these main questions on newsprint, to discuss them together, to write down ideas, to decide on the best option, and to move on. We ran systematically and very efficiently down the question list. When we finished, after about three one-hour meetings, the evaluation had basically been planned."

The evaluator then went on to conduct the evaluation and analyze the data on his own before presenting it to the community.

Characteristics to look for in all evaluators

Willingness to leave one's own agenda at the door

Often researchers, particularly those attached to universities, may have their own reasons for embarking on an evaluation. It may fit into a doctoral dissertation, or a book that a professor is writing, or a piece of long-term research that will eventually be published. They may also have very strong prejudices about what kind of research they want to use, or what they expect to find, and a need to prove their prejudices correct. And they may have issues about power and about their standing in relation to that of members of a planning team or organizational staff. If the researchers' needs mesh perfectly with yours, then there's no problem. But if they don't quite fit, there can be a very serious problem. If you're paying for a service, you should get what you're paying for, and not simply what the researcher wants to give you. It's important to be clear about this on all sides at the beginning, and to make sure by writing it into a contract or through some other formal mechanism that the professionals are willing to do what meets your needs, not what meets theirs.

Just as professionals may have research priorities that have nothing to do with the evaluation, community members may have personal or political priorities that also have little to do with the business at hand, but can affect their performance on an evaluation team. As much as possible, it's important that people either be willing to put these priorities aside while they're engaged in the evaluation process, or that you screen out those who will try to impose their particular ways of thinking on everyone, or who will use the process to further their own ends.

Ability to communicate with a broad range of people

Evaluators will have to deal with people from all walks of life, of all political, religious, and philosophical persuasions, and probably of many ethnic, language, and racial groups. If they are to gather accurate information, they will have to be perceived as trustworthy by all of those groups, and will have to be able to generate a certain level of comfort with everyone.

Have they worked with groups that included a broad spectrum of community members Do they have the verbal and interpersonal skills and the patience to explain their methods clearly to people who aren't highly educated and who may never have had any contact with research before (perhaps especially if the evaluators themselves are in that same category)? Can at least some of them speak at least some of the languages of the community?

You can answer at least some of these questions by interviewing potential evaluators before they're hired or chosen. Evaluators need to be able to gather the information needed for the evaluation and to couch the results of the evaluation in terms that the community can understand, and therefore use. The best evaluation plan is worthless if the evaluators' approach means that people can't or won't respond to them, or can't use the results they've come up with.

Cultural sensitivity

Especially in a community that contains residents of many cultures (depending on how you define "culture," that includes virtually every community), mutual respect and some understanding and acceptance of how others see the world is crucial to the functioning of evaluators. Do they understand, or are they willing to learn to understand, the cultures of those in the community? An urban, largely working class community is culturally very different from an upper-middle class suburb; a Haitian neighborhood is different from a Vietnamese or a Puerto Rican one. Evaluators need to respect the cultures of the communities they work with, and not violate them, intentionally or unintentionally.

Ability to treat everyone with the same degree of respect

How evaluators approach people reflects on the organization. If they don't treat everyone respectfully, they're not going to get accurate--or any--information, and they're going to complicate the organization's relationship with its staff, the target population, and the community.

Absolute commitment to keeping all individuals' information confidential

Whether evaluators are paid professionals or not, for ethical, practical, and legal reasons, it is almost always necessary to guarantee that any information gathered in the course of an evaluation will be kept confidential (i.e. used only for the purposes of the evaluation, and not connected to the individual), and that people won't be identified either by name or by other factors that could lead to them. In the case, for instance, of the evaluation of a domestic violence prevention program or women's shelter, confidentiality could be a matter of personal safety. In most instances, it will help evaluators obtain more accurate data. And it will protect the evaluators and the organization from lawsuits that could be brought by individuals injured in some way by the information they provided.

Professional or university researchers often ask informants to read and sign an "informed consent" form that explains exactly what the researchers are doing, what any information will be used for, and how the researchers will protect confidentiality and anonymity. If the researcher violates the terms of the form, the informant might have the grounds for a lawsuit, and the researcher's findings might be questioned. An evaluator, professional or not, who asks for responses from members of the target population or the community might consider using a similar form, both because it helps to explain what the evaluation is about, and because it demonstrates the evaluator's commitment to confidentiality.

Commitment to the evaluation process

For planners and evaluators, whether professionals or otherwise, this means trying to do the best evaluation possible, with an eye toward its actual usefulness for the organization. For the organization, and other community members, commitment means believing enough in the process to take the evaluation seriously and use it to make adjustments to and improve the program, service, or activity. An evaluation, no matter how elegant and informative, is worthless if it's not used.

An important part of a first evaluation is a plan for continued evaluation. To be truly useful, evaluation needs to be ongoing throughout the life of the organization. If evaluators can provide a guide for setting up and conducting evaluations on a regular basis, they'll have performed a service to the organization far beyond the value of the evaluation itself.

Community or in-house planners or evaluators

A word about planning teams vs. evaluation teams: A planning team is meant to plan the evaluation process. If the organization has the option of hiring professional evaluators, a community or staff planning team may decide whether or not to do so, and may serve as the hiring committee for those evaluators. Its main job (which it may do alone or with the help of a hired evaluator) will probably be to examine the possibilities and come up with an actual evaluation plan, detailing what the organization needs to find out, where it can find that information, who will do the information gathering and analysis, and how the information will be used once it's gathered and analyzed.

A community evaluation team is a volunteer team set up to carry out the actual evaluation. In practice, this may be the same group as the planning team, whose function, once the planning is done, is the accomplishment of the task. It's helpful if the evaluation team includes someone with enough understanding of research that the evaluation they conduct will be valid, and its results can be used with some confidence.

With a community planning or evaluation team, you have the opportunity to choose all the members individually. There may be some people who, because of their position in the organization or in the community, have to be on the team. By the same token, some others may not be able to be included. In general, the team should be small enough so that it won't be too difficult for members to meet and to contact one another, and large enough so that there are enough people to do the work. And, in general, the team members should represent the interested stakeholders in the organization or initiative: staff and Board, target population, and community members at large. Characteristics you might look for in those on your planning or evaluation team include:

Representation

The team could include any number of the following possibilities (but, for most organizations, would probably number only four or five):

• A range of organizational representatives (e.g. the director, one or two line staff members line [those who work directly with the target population], and a Board member)
• Members of the target population, ideally individuals who understand and have participated in the organization or initiative
• Community members with a stake in the organization or initiative (business people, health and human service agency representatives, school and local government officials, other citizens with an interest in the issue)
• Youth, if appropriate
• Relevant language and other minority groups
• Someone with research skills and experience, perhaps from a local college or university.

Ability to understand the purpose of the evaluation

It's important to understand not only what the evaluationis supposed to measure, but also how its form and purpose are related.

Willingness to listen and learn

This covers a wide spectrum of people and behaviors. It includes both the local high school principal being able to listen to and take seriously the opinions of a high school dropout, and that dropout's belief that he's capable of learning what he needs to know to function as a contributing member of the team. It also includes everyone's being aware that they have things to learn about the process, one another, and evaluation in general, and, by the same token, that they have valuable knowledge, skills, and information to contribute.

Ability to work in a group

This can be a particularly sticky issue in a group that encompasses several cultures and classes. The middle-class "meeting skills " that most educated people have been unconsciously learning and practicing since junior high or before are often unknown to welfare recipients or migrant workers, simply because they've never been exposed to them. Thus, they may sit silent, confused by the flow of the meeting, unable to contribute, and feeling foolish; or they may speak or act in ways or at times others deem inappropriate. People without meeting skills need support and encouragement in what is often for them an intimidating situation. One answer may be to pair all team members, so that those who need support will have a mentor. Another possibility is to start the process with a training for everyone in group dynamics and meeting skills, so that no one is singled out and the ground rules are clear for all.

Professionals

Unless you're working with an individual consultant, when you work with professionals, it's likely that a team will come already assembled. Thus, you're looking at the whole package: who the team members are, what they know how to do, and how they're likely to interact with your planning team, if you have one, and with your organization and the community. Most professional teams, as mentioned above, will either be private consultants or university researchers. Probably the most important features to look for are their professional experience and skills, and their expertise and style in working with community groups:

Knowledge of different kinds of evaluation techniques

Can they use both quantitative (numbers and statistics) and qualitative (facts, stories, anecdotes, analysis of situations and events, etc.) research, and do they know when each is appropriate? Have they used them in situations like yours, or do they have good ideas about how to do that?

If your team is from higher education, you should consider carefully what department (s) of a university to contact to find evaluators whose professional and scholarly specialties are a good fit with the work you do. Depending upon what your organization or initiative does, and what information you're looking for, some possibilities are:

• Public Health
• Medicine
• Education
• Sociology
• Psychology
• Environmental Sciences
• Urban Studies

Past performance

Have they done evaluation before? What kinds of techniques have they used? Do they have good references? Can you see examples of past evaluations they've done?

Just as you probably wouldn't hire someone for a position without checking her references, you shouldn't hire an evaluator or team without checking with those who've employed them before. How well professionals do in an interview and on paper is usually enough to tell you what you need to know about them, but when it's not, it can be disastrous, costing you not only money, but time, results, and relationships in the community.

You should ask to see some of the results of previous evaluations. Are they readable and understandable? How did the candidate explain them to the organization and the community? Knowing what you're getting can help immeasurably in choosing the right evaluators.

Experience working with community groups

Have they worked in partnership with community groups before? Do they understand the roles in this situation, and can they handle theirs well? Have they been able to work with the community to address the appropriate goals? Are they responsive to suggestion, to the needs of the community, and to the context of the evaluation? Do they collaborate, in other words, rather than telling the community group what it needs?

Ability to really listen

It's important that not only is the evaluator able to listen, but also respect others and not assume the "expert" role except where that's appropriate (you are, after all, hiring them for their expertise). If the evaluation is community-based, the evaluators are going to have to listen first to the planning group about what the organization or initiative most needs to know, and then to the community in order to gather that information. Real listening (listening not only for the meaning of what's being said, but for the meaning behind the meaning) is an indispensable skill for all good evaluators.

In Summary

The individual or team you select to plan and/or conduct the evaluation of your organization or initiative will do much to determine the character and usefulness of the evaluation itself. If you work with professionals, you need to consider their credentials, past experience, and level of expertise. Whether you choose professional evaluators, community volunteers, or some combination of the two, you need to think carefully about how their needs and interests fit in with those of the organization and the community, about the range and quality of their communication skills, about their willingness to work as partners with the organization and the community, and about their fit with one another. Once you've put together a good team, you are well on your way to carrying out an accurate and valuable evaluation.

Why should you evaluate?

___To see results of work

___To gain public opinion

___To make the proper adjustments if necessary

What should you consider when you select evaluators?

___Select evaluators carefully for reliability 

Possible evaluators:

___Volunteers

___Community being serviced

___Chosen evaluators by a team

Tool: Interview Questions for evaluation consultant selection

From EvalTalk, the discussion group of the American Evaluation Association (AEA).

• What are the main reasons why you think a program should conduct an evaluation?

• What are the main standards and guiding principles for conducting evaluations?

• How would you determine if a program is ready to undergo an evaluation?

• How do you estimate the resources needed to conduct an evaluation? That is, how do you compute the costs of an evaluation, both in money and time (yourself and others)?

• If you had results from an evaluation that were based on quantitative information that conflicted with results from qualitative information, how would you resolve the conflict?

• If you were to conduct a mailed-out survey to program participants, how would you achieve a representative sample and ensure that your results were generalizable to all participants?

• Frequently evaluations involve various and independent branches and agencies that all have different expectations for the evaluation. If you had a conflict among participants in a program evaluation that could threaten the evaluation, how would you resolve the conflict? What steps would you take to proactively avoid conflict?

• Read any interesting articles on evaluation lately? In what journals? What was interesting about it? What other journals do you see regularly?

• Under what circumstances, if any, would you blow the whistle on a client?

Why should you have an evaluation plan?

After many late nights of hard work, more planning meetings than you care to remember, and many pots of coffee, your initiative has finally gotten off the ground. Congratulations! You have every reason to be proud of yourself and you should probably take a bit of a breather to avoid burnout. Don't rest on your laurels too long, though--your next step is to monitor the initiative's progress. If your initiative is working perfectly in every way, you deserve the satisfaction of knowing that. If adjustments need to be made to guarantee your success, you want to know about them so you can jump right in there and keep your hard work from going to waste. And, in the worst case scenario, you'll want to know if it's an utter failure so you can figure out the best way to cut your losses. For these reasons, evaluation is extremely important.

There's so much information on evaluation out there that it's easy for community groups to fall into the trap of just buying an evaluation handbook and following it to the letter. This might seem like the best way to go about it at first glance-- evaluation is a huge topic and it can be pretty intimidating. Unfortunately, if you resort to the "cookbook" approach to evaluation, you might find you end up collecting a lot of data that you analyze and then end up just filing it away, never to be seen or used again.

Instead, take a little time to think about what exactly you really want to know about the initiative. Your evaluation system should address simple questions that are important to your community, your staff, and (last but never least!) your funding partners. Try to think about financial and practical considerations when asking yourself what sort of questions you want answered. The best way to insure that you have the most productive evaluation possible is to come up with an evaluation plan.

Here are a few reasons why you should develop an evaluation plan:

• It guides you through each step of the process of evaluation
• It helps you decide what sort of information you and your stakeholders really need
• It keeps you from wasting time gathering information that isn't needed
• It helps you identify the best possible methods and strategies for getting the needed information
• It helps you come up with a reasonable and realistic timeline for evaluation
• Most importantly, it will help you improve your initiative!

When should you develop an evaluation plan?

As soon as possible! The best time to do this is before you implement the initiative. After that, you can do it anytime, but the earlier you develop it and begin to implement it, the better off your initiative will be, and the greater the outcomes will be at the end.

Remember, evaluation is more than just finding out if you did your job. It is important to use evaluation data to improve the initiative along the way.

What are the different types of stakeholders and what are their interests in your evaluation?

We'd all like to think that everyone is as interested in our initiative or project as we are, but unfortunately that isn't the case. For community health groups, there are basically three groups of people who might be identified as stakeholders (those who are interested, involved, and invested in the project or initiative in some way): community groups, grantmakers/funders, and university-based researchers. Take some time to make a list of your project or initiative's stakeholders, as well as which category they fall into.

What are the types of stakeholders?

• Community groups: Hey, that's you! Perhaps this is the most obvious category of stakeholders, because it includes the staff and/or volunteers involved in your initiative or project. It also includes the people directly affected by it--your targets and agents of change.
• Grantmakers and funders: Don't forget the folks that pay the bills! Most grantmakers and funders want to know how their money's being spent, so you'll find that they often have specific requirements about things they want you to evaluate. Check out all your current funders to see what kind of information they want you to be gathering. Better yet, find out what sort of information you'll need to have for any future grants you're considering applying for. It can't hurt!
• University-based researchers: This includes researchers and evaluators that your coalition or initiative may choose to bring in as consultants or full partners. Such researchers might be specialists in public health promotion, epidemiologists, behavioral scientists, specialists in evaluation, or some other academic field. Of course, not all community groups will work with university-based researchers on their projects, but if you choose to do so, they should have their own concerns, ideas, and questions for the evaluation. If you can't quite understand why you'd include these folks in your evaluation process, try thinking of them as auto mechanics--if you want them to help you make your car run better, you will of course include them in the diagnostic process. If you went to a mechanic and started ordering him around about how to fix your car without letting him check it out first, he'd probably get pretty annoyed with you. Same thing with your researchers and evaluators: it's important to include them in the evaluation development process if you really want them to help improve your initiative.

Each type of stakeholder will have a different perspective on your organization as well as what they want to learn from the evaluation. Every group is unique, and you may find that there are other sorts of stakeholders to consider with your own organization. Take some time to brainstorm about who your stakeholders are before you being making your evaluation plan.

What do they want to know about the evaluation?

While some information from the evaluation will be of use to all three groups of stakeholders, some will be needed by only one or two of the groups. Grantmakers and funders, for example, will usually want to know how many people were reached and served by the initiative, as well as whether the initiative had the community -level impact it intended to have. Community groups may want to use evaluation results to guide them in decisions about their programs, and where they are putting their efforts. University-based researchers will most likely be interested in proving whether any improvements in community health were definitely caused by your programs or initiatives; they may also want to study the overall structure of your group or initiative to identify the conditions under which success may be reached.

What decisions do they need to make, and how would they use the data to inform those decisions?

You and your stakeholders will probably be making decisions that affect your program or initiative based on the results of your evaluation, so you need to consider what those decisions will be. Your evaluation should yield honest and accurate information for you and your stakeholders; you'll need to be careful not to structure it in such a way that it exaggerates your success, and you'll need to be really careful not to structure it in such a way that it downplays your success!

Consider what sort of decisions you and your stakeholders will be making. Community groups will probably want to use the evaluation results to help them find ways to modify and improve your program or initiative. Grantmakers and funders will most likely be making decisions about how much funding to give you in the future, or even whether to continue funding your program at all (or any related programs). They may also think about whether to impose any requirements on you to get that program (e.g., a grantmaker tells you that your program may have its funding decreased unless you show an increase of services in a given area). University-based researchers will need to decide how they can best assist with plan development and data reporting.

You'll also want to consider how you and your stakeholders plan to balance costs and benefits. Evaluation should take up about 10--15% of your total budget. That may sound like a lot, but remember that evaluation is an essential tool for improving your initiative. When considering how to balance costs and benefits, ask yourself the following questions:

• What do you need to know?
• What is required by the community?
• What is required by funding?

How do you develop an evaluation plan?

There are four main steps to developing an evaluation plan:

• Clarifying program objectives and goals
• Developing evaluation questions
• Developing evaluation methods
• Setting up a timeline for evaluation activities

Clarifying program objectives and goals

The first step is to clarify the objectives and goals of your initiative. What are the main things you want to accomplish, and how have you set out to accomplish them? Clarifying these will help you identify which major program components should be evaluated. One way to do this is to make a table of program components and elements.

Developing evaluation questions

For our purposes, there are four main categories of evaluation questions. Let's look at some examples of possible questions and suggested methods to answer those questions. Later on, we'll tell you a bit more about what these methods are and how they work

• Planning and implementation issues: How well was the program or initiative planned out, and how well was that plan put into practice?
  • Possible questions: Who participates? Is there diversity among participants? Why do participants enter and leave your programs? Are there a variety of services and alternative activities generated? Do those most in need of help receive services? Are community members satisfied that the program meets local needs?
  • Possible methods to answer those questions: monitoring system that tracks actions and accomplishments related to bringing about the mission of the initiative, member survey of satisfaction with goals, member survey of satisfaction with outcomes.
• Assessing attainment of objectives: How well has the program or initiative met its stated objectives?
  • Possible questions: How many people participate? How many hours are participants involved?
  • Possible methods to answer those questions: monitoring system (see above), member survey of satisfaction with outcomes, goal attainment scaling.
• Impact on participants: How much and what kind of a difference has the program or initiative made for its targets of change?
  • Possible questions: How has behavior changed as a result of participation in the program? Are participants satisfied with the experience? Were there any negative results from participation in the program?
  • Possible methods to answer those questions: member survey of satisfaction with goals, member survey of satisfaction with outcomes, behavioral surveys, interviews with key participants.
• Impact on the community: How much and what kind of a difference has the program or initiative made on the community as a whole?
  • Possible questions: What resulted from the program? Were there any negative results from the program? Do the benefits of the program outweigh the costs?
  • Possible methods to answer those questions: Behavioral surveys, interviews with key informants, community-level indicators.

Developing evaluation methods

Once you've come up with the questions you want to answer in your evaluation, the next step is to decide which methods will best address those questions. Here is a brief overview of some common evaluation methods and what they work best for.

Monitoring and feedback system

This method of evaluation has three main elements:

• Process measures: these tell you about what you did to implement your initiative;
• Outcome measures: these tell you about what the results were; and
• Observational system: this is whatever you do to keep track of the initiative while it's happening.

Member surveys about the initiative

When Ed Koch was mayor of New York City, his trademark call of "How am I doing?" was known all over the country. It might seem like an overly simple approach, but sometimes the best thing you can do to find out if you're doing a good job is to ask your members. This is best done through member surveys. There are three kinds of member surveys you're most likely to need to use at some point:

• Member survey of goals: done before the initiative begins - how do your members think you're going to do?
• Member survey of process: done during the initiative - how are you doing so far?
• Member survey of outcomes: done after the initiative is finished - how did you do?

Goal attainment report

If you want to know whether your proposed community changes were truly accomplished-- and we assume you do--your best bet may be to do a goal attainment report. Have your staff keep track of the date each time a community change mentioned in your action plan takes place. Later on, someone compiles this information (e.g., "Of our five goals, three were accomplished by the end of 1997.")

Behavioral surveys

Behavioral surveys help you find out what sort of risk behaviors people are taking part in and the level to which they're doing so. For example, if your coalition is working on an initiative to reduce car accidents in your area, one risk behavior to do a survey on will be drunk driving.

Interviews with key participants

Key participants - leaders in your community, people on your staff, etc. - have insights that you can really make use of. Interviewing them to get their viewpoints on critical points in the history of your initiative can help you learn more about the quality of your initiative, identify factors that affected the success or failure of certain events, provide you with a history of your initiative, and give you insight which you can use in planning and renewal efforts.

Community-level indicators of impact

These are tested-and-true markers that help you assess the ultimate outcome of your initiative. For substance use coalitions, for example, the U.S. Centers for Substance Abuse Prevention (CSAP) and the Regional Drug Initiative in Oregon recommend several proven indicators (e.g., single-nighttime car crashes, emergency transports related to alcohol) which help coalitions figure out the extent of substance use in their communities. Studying community-level indicators helps you provide solid evidence of the effectiveness of your initiative and determine how successful key components have been.

Setting up a timeline for evaluation activities

When does evaluation need to begin?

Right now! Or at least at the beginning of the initiative! Evaluation isn't something you should wait to think about until after everything else has been done. To get an accurate, clear picture of what your group has been doing and how well you've been doing it, it's important to start paying attention to evaluation from the very start. If you're already part of the way into your initiative, however, don't scrap the idea of evaluation altogether--even if you start late, you can still gather information that could prove very useful to you in improving your initiative.

Outline questions for each stage of development of the initiative

We suggest completing a table listing:

• Key evaluation questions (the five categories listed above, with more specific questions within each category)
• Type of evaluation measures to be used to answer them (i.e., what kind of data you will need to answer the question?)
• Type of data collection (i.e., what evaluation methods you will use to collect this data)
• Experimental design (A way of ruling out threats to the validity - e.g., believability - of your data. This would include comparing the information you collect to a similar group that is not doing things exactly the way you are doing things.)

With this table, you can get a good overview of what sort of things you'll have to do in order to get the information you need.

When do feedback and reports need to be provided?

Whenever you feel it's appropriate. Of course, you will provide feedback and reports at the end of the evaluation, but you should also provide periodic feedback and reports throughout the duration of the project or initiative. In particular, since you should provide feedback and reports at meetings of your steering committee or overall coalition, find out ahead of time how often they'd like updates. Funding partners will want to know how the evaluation is going as well.

When should evaluation end?

Shortly after the end of the project - usually when the final report is due. Don't wait too long after the project has been completed to finish up your evaluation - it's best to do this while everything is still fresh in your mind and you can still get access to any information you might need.

What sort of products should you expect to get out of the evaluation?

The main product you'll want to come up with is a report that you can share with everyone involved. What should this report include?

• Effects expected by shareholders: Find out what key people want to know. Be sure to address any information that you know they're going to want to hear about!
• Differences in the behaviors of key individuals: Find out how your coalition's efforts have changed the behaviors of your targets and agents of change. Have any of your strategies caused people to cut down on risky behaviors, or increase behaviors that protect them from risk? Are key people in the community cooperating with your efforts?
• Differences in conditions in the community: Find out what has changed Is the public aware of your coalition or group's efforts? Do they support you? What steps are they taking to help you achieve your goals? Have your efforts caused any changes in local laws or practices?

You'll probably also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team.

What sort of standards should you follow?

Now that you've decided you're going to do an evaluation and have begun working on your plan, you've probably also had some questions about how to ensure that the evaluation will be as fair, accurate, and effective as possible. After all, evaluation is a big task, so you want to get it right. What standards should you use to make sure you do the best possible evaluation? In 1994, the Joint Committee on Standards for Educational Evaluation issued a list of program evaluation standards that are widely used to regulate evaluations of educational and public health programs.The standards the committee outlined are for utility, feasibility, propriety, and accuracy. Consider using evaluation standards to make sure you do the best evaluation possible for your initiative.

You understand that you should have an evaluation plan because:

___It guides you through each step of the process of evaluation

___It helps you decide what sort of information you and your stakeholders really need

___It keeps you from wasting time gathering information that isn't needed

___It helps you identify the best possible methods and strategies for getting the needed information

___It helps you come up with a reasonable and realistic timeline for evaluation

___Most importantly, it will help you improve your initiative!

You understand who your project or initiative's stakeholders are:

___Community groups

___Grantmakers and funders

___University-based researchers

You have taken into consideration:

___What stakeholders want to know about the evaluation

___What decisions stakeholders need to make

___How stakeholders would use the data to inform decisions

When considering how to balance costs and benefits, you have asked yourself the following questions:

___What do you need to know?

___What is required by the community?

___What is required by funding?

You understand these four main steps to developing an evaluation plan:

___Clarify program objectives and goals

___Develop evaluation questions

___Develop evaluation methods

___Set up a timeline for evaluation activities

Clarify the objectives and goals of your initiative:

___Make a table of program components and elements

Develop evaluation questions:

___You understand the four main categories of evaluation questions:

• Planning and implementation issues
• Assessing attainment of objectives
• Impact on participants
• Impact on the community

___You have considered the best possible methods to answer those questions

Develop evaluation methods to best address those questions:

___You understand how to use the monitoring and feedback system's three main elements (process measures, outcome measures, and observational system)

___You understand how to use member surveys about the initiative (member survey of goals, member survey of process, and member survey of outcomes)

___You understand how to use the goal attainment report

___You understand how to use behavioral surveys

___You understand how to use interviews with key participants

___You know how to use community-level indicators of impact

Setting up a timeline for evaluation activities:

___You understand that you should begin right now, or at least at the beginning of the initiative

___You've outlined questions for each stage of development of the initiative

___You've completed a table listing: key evaluation questions, type of evaluation measures to be used to answer them, type of data collection, and experimental design .

___You've determined when you feel it's appropriate to provide feedback and reports

___You will provide feedback and reports at the end of the evaluation

___You will also provide periodic feedback and reports throughout the duration of the project or initiative

___You've decided when the evaluation will end

___You've mapped out a proposed evaluation timeline

Expected evaluation products:

___You will make a report that you can share with everyone involved which includes effects expected by shareholders, differences in the behaviors of key individuals, and differences in conditions in the community

___You've decided whether to also include specific tools (i.e., brief reports summarizing data), annual reports, quarterly or monthly reports from the monitoring system, and anything else that is mutually agreed upon between the organization and the evaluation team

Evaluation standards:

___You've decided what standards you will use to ensure an accurate and useful evaluation

Example 1: Table of Program Components and Elements

Here are a few of the items from the table of program components and elements for a fictional teen pregnancy prevention program:

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery
Risk factor: Unsupervised activities by teens	Provide alternative activities	After-school activities (sports programs, hobby / craft classes, movies, etc.) Weekend "lock-ins", non-alcoholic teen dance club Daytime activities during school holidays	Targets:   Youth   Agents: Youth Teachers Parents	Group sessions Workshops Meetings
Risk factor: Lack of access to contraceptives	Increased access to health services	Extend hours at school health clinics Increase condom distribution by county health department Brown bag program at area pharmacies (teens wishing to buy contraceptives are able to call ahead and pick them up in a plain brown bag)	Targets:   Youth Agents: Health providers	Walk-in clinics Health Department Businesses
Protective factor: Involvement in the faith community	Encourage youth involvement in the faith community	Increase church-sponsored youth activities Institute church-based sexuality education Church-sponsored parent-child communication sessions	Targets: Youth Parents Agents: Youth Parents Religious leaders	Discussions Meetings Educational sessions Workshops

Example 2: Evaluation Questions and Methods

Here is a sample table of some of the questions from the evaluation plan of Youth Enrichment Services (Y.E.S.), an organization that provides urban young people with services that encourage them to explore, challenge themselves physically and mentally, and interact with positive role models:

		Type of Data Collection*					Experimental Design**
Key Evaluation Question(s)	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- and Post-test Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1. Who participates? Is there diversity among the participants?	1. Basic demographic data (e.g., age, gender, disability, race, economic status, etc.) on participants.	X	X				X
2. Why do participants enter and leave the program?	2. Information on participants' reasons for entering and leaving the program.	X	X				X
3. Are there a variety of service and alternative activities generated?	3. Frequency of different types of service provided and alternative activities generated.			X	X	X	X
Assessing Attainment of Objectives	Outcome Measures
1. How many people participate?	1. Number of people participating.			X	X	X	X		X
2. How many hours are participants involved in service and activities?	2. Number of hours of participation by type of activity.			X	X	X	X		X
3. How many people are trained in cultural awareness and sensitivity?	3. Number of people trained in cultural awareness and sensitivity.			X	X	X	X		X
Impact on Participants
1. How do behavior and attitudes change as a result of participation in the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X		X	X
2. Does participation affect the incidence of problems?	2. Incidence of problems in living (e.g., substance use, gang involvement).	X	X			X		X
3. Are participants satisfied with the experience?	3. Satisfaction ratings from participants.	X					X
Impact on Community
1.  What resulted from the program?	1. Changes in programs, policies, and practices of affiliated organizations, and other outcomes.	X	X	X	X	X			X
2. Do the benefits of the program outweigh the costs?	2. Cost benefit data.		X	X		X	X
3. Are community members satisfied with the participants and the service they provided?	3. Satisfaction ratings from beneficiaries and community members.	X					X

* Please note that the categories of types of data collection use broader terms than we've used to describe evaluation methods; here are some examples of methods that fall under these categories:

• Survey/Scale: i.e., member survey of satisfaction with goals, member survey of satisfaction with process, member survey of satisfaction with outcomes, and behavioral surveys
• Structured Interview: i.e., interview with key participants
• Staff Report/Log: i.e. monitoring system, goal attainment report
• Direct Observation: This refers to the direct observation of behavior by researchers .
• Archival Records: i.e., community-level indicators of impact

**Here are explanations of these terms:

• Case Study: an in-depth, start-to-finish analysis of a person, group, or community that focuses on developmental factors in relation to environment--good for answering "how" or "why" questions
• Pre-/Posttest Control Group: monitoring a group of people who fit into your group's target population who you don't expose to your initiative, so that you can compare them with the people who were part of the initiative, helping you get a better idea of how much real impact you've made
• Time Series: effects are measured from a long series of repeated measurements taken both before and after an initiative

Example 3: Timeline for Evaluation Activities

This is the timeline for an evaluation being done by Kansas City's Promise on their initiative to promote community and personal caring for youth:

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system	[ - - - ]
Train participants / staff in data entry	[ - - - ]
Maintain system (data collection and reporting)	[- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - ]
Critical events
Identify key people	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Conduct interviews	[ - - ]	[ - - ]	[ - - ]	[ - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Community-Level Indicators
Identify local indicators	[ - - - - ]
Identify comparison site	[ - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - - - - - - - -	- - - - - - - - - ]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]
Other Indicators (e.g., behavioral surveys, data observations)
Identify	[ - - - - - -	- - - - - - ]
Secure data	[ - - -	- - - - - - - - - - - - - - -	- - - - - - - - - - -  - - - - -	- - - - - - - - - -]
Report results	[ - - - ]	[ - - - ]	[ - - - ]	[ - - - ]

 

Tool 1: Supports for Implementing an Evaluation Plan for Community-Level Obesity Prevention Efforts

Download PPT Slides >>

 

Table of Contents

General Evaluation Resources
Steps in Community Evaluation (and Related Resources)

1. Design Stakeholder Involvement
2. Identify Resources for Monitoring and Summative Evaluation
3. Describe the Intervention’s Framework, Logic Model, or Theory of Change
4. Focus the Monitoring and Summative Evaluation  Plan
5. Plan for credible methods
6. Synthesize and Generalize
7. Systems and Evaluation

 

General Evaluation Resources

• Institute of Medicine (IOM):
  • ​Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (2010)
  • Measuring Progress in Obesity Prevention: Workshop Report (2012)
  • Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation (2012)
  • Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013)
• U.S. Centers for Disease Control and Prevention (CDC): Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
• Community Tool Box (CTB):
  • Toolkit: Evaluating the Initiative
  • A Framework for Program Evaluation: A Gateway to Tools
  • Evaluating Comprehensive Community Initiatives
  • Toubleshooting Guide: We don’t know how to evaluate our program or initiative

Steps in Community Evaluation (and related resources)

 

 

Back to Top

 

1. Design stakeholder involvement
   
   1. Identify stakeholders
   2. Consider the extent of stakeholder involvement
   3. Assess desired outcomes of monitoring and summative evaluation *
   4. Define stakeholder roles in the monitoring and summative evaluation *
   5. Resources:
      • Institute of Medicine: Improving the Usefulness of Obesity Evaluation Information to Potential Users, Chapter 2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Statement On Cultural Competence In Evaluation
      • Community Tool Box:
        • ​Identifying and Analyzing Stakeholders and Their Interests
        • Understanding Community Leadership, Evaluators, and Funders: What Are Their Interests?
        • Participatory Evaluation
        • Gathering Information: Monitoring Your Progress

 

 

Back to Top

 

2. Identify resources for monitoring and summative evaluation
   
   1. Person-power resources
   2. Data collection resources
   3. Resources:
      • Institute of Medicine: Guiding Principles for Evaluation, Appendix C, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Identifying Community Assets and Resources
        • ​Choosing Evaluators

 

Back to Top

 

3. Describe the intervention’s framework, logic model, or theory of change
   
   1. Purpose or mission
   2. Context or conditions
   3. Inputs: resources and barriers
   4. Activities or interventions
   5. Outputs of activities
   6. Intended effects or outcomes
   7. Resources:
      • Institute of Medicine: Framework for Evaluation, Chapter 3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • Community Tool Box:
        • Toolkit: Developing a Framework or Model of Change
        • Developing a Logic Model or Theory of Change

 

 

Back to Top

 

4. Focus the monitoring and summative evaluation plan
   
   1. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
   2. Set priorities by end-user questions, resources, context
   3. What questions will the monitoring and summative evaluation answer?
   4. Ethical implications (benefit outweighs risk)
   5. Resources:
      • Institute of Medicine:
        1. Specifying Questions and Locating Evidence: An Expanded View, Chapter 5, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        2. Considerations When Designing a Local Monitoring System, Table 8-2, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • American Evaluation Association: Guiding Principles For Evaluators
      • Community Tool Box:
        1. Choosing Questions and Planning the Evaluation
        2. Ethical Issues in Community Interventions
        3. Documenting Health Promotion Initiatives Using the PAHO Guide

 

 

Back to Top

 

5. Plan for credible methods
   
   1. Stakeholder agreement on methods
   2. Indicators of success
   3. Credibility of evidence
   4. Resources:
      • Institute of Medicine:
        • Evaluating Evidence, Chapter 6, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Selected Tools for Evaluating Community Obesity Prevention Initiatives, Table H-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Evaluating Progress in Promoting Health Equity: A Review of Methods and Tools for Measurement, Chapter 5, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Disparities Tables, Appendix E, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
        • Table-Measuring Progress in Obesity Prevention: Indicators and Data Sources, Chapter 4, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (PDF) (Interactive Widget)
        • Evaluations Illustrating Best Practices for Measurement and Design, Appendix H, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • National Collaborative on Childhood Obesity Research (NCCOR):
        • Catalogue of Surveillance Systems
        • ​Measures Registry
      • Community Tool Box:
        • Data Collection: Designing an Observational System
        • Selecting an Appropriate Design for the Evaluation
        • Collecting and Analyzing Data
        • Collecting and Using Archival Data
        • Qualitative Methods to Assess Community Issues
        • Implementing Photovoice in Your Community
        • Community-Level Indicators: Some Examples
      • Book: The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users

 

Back to Top

 

6. Synthesize and generalize
   
   1. Disseminating and compiling studies
   2. Learning more from implementation
   3. Ways to assist generalization
   4. Shared sense-making and cultural competence
   5. Disentangling effects of interventions
   6. Resources:
      • Institute of Medicine:
        • Assembling Evidence and Informing Decisions, Chapter 7, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Opportunities to Generate Evidence, Chapter 8, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
        • Recommended Approaches for Key Outcomes of Community Monitoring and Summative Evaluation, by Level of Evaluation Resources, Table 8-3, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
      • CDC Evaluation Briefs: Disseminating Program Achievements and Evaluation Findings to Garner Support
      • Community Tool Box:
        • Troubleshooting Guide: There is Not Enough Change in the Community or System
        • Troubleshooting Guide: There is Not Enough Improvement in Outcomes
        • Troubleshooting Guide: There are Unintended or Unwanted Outcomes
        • Toolkit: Enhancing Cultural Competence
        • Adapting Community Interventions for Different Cultures and Communities
        • Providing Feedback to Improve the Initiative
        • Communicating Information to Funders for Support and Accountability
        • Refining the Program Intervention Based on Research
        • The County Health Rankings & Roadmaps Take Action Cycle

 

 

Back to Top

 

7. Systems and Evaluation
   • Resources:
     • IOM:
       • Defining the Problem: The Importance of Taking a Systems Perspective, Chapter 4, in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making
       • Answering Questions about Leadership, Prioritization, and Assessment with a Systems Perspective, Chapter 10, in Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation
       • Systems and Evaluation: Placing a Systems Approach in Context, Chapter 9, in Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress
     • Community Tool Box:
       • Working Together for Healthier Communities: A Framework for Collaboration Among Community Partnerships, Support Organizations, and Funders
       • Best Change Processes
       • MAPP: Mobilizing for Action through Planning and Partnerships

 

Sources: BOX 8-1, IOM Report on Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress (2013); Community Tool Box.

 

*Monitoring is the tracking of the implementation of interventions compared to standards of performance.
*Summative Evaluation is the effort to detect changes in output, outcomes, and impacts associated with interventions and to attribute those changes to the interventions.

 

 

Tool 2: Table of Program Components and Elements

Risk / Protective Factors	Program Components (Strategies)	Program Elements (Tactics)	Targets and Agents of Change	Mode / Context of Delivery

 

Tool 3: Evaluation Questions and Methods

		Type of Data Collection *					Experimental Design **
Key Evaluation Questions	Type of Management Information and Evaluation Measure(s)	Survey / Scale	Structured Interview	Self Report / Log	Direct Observation	Archival Records	Case Study	Pre- Posttest Control Group	Time Series
Planning and Implementation Issues	Descriptive and Process Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Assessing Attainment of Objectives	Outcome Measures
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Participants
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.
Impact on Community
1.	1.
2.	2.
3.	3.
4.	4.
5.	5.

Tool 4: Evaluation Timeline

	2010	2011	2012	2013
Monitoring System	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D	J F M A M J J A S O N D
Establish system
Train participants / staff  in data entry
Maintain system (data collection and reporting)
Critical events
Identify key people
Conduct interviews
Report results
Community-Level Indicators
Identify local indicators
Identify comparison site
Secure data
Report results
Other Indicators (e.g., behavioral surveys, data observations)
Identify
Secure data
Report results

Experienced community builders know that involving stakeholders - the people directly connected to and affected by their projects - in their work is tremendously important. It gives them the information they need to design, and to adjust or change, what they do to best meet the needs of the community and of the particular populations that an intervention or initiative is meant to benefit. This is particularly true in relation to evaluation.

As we have previously discussed, community-based participatory research can be employed in describing the community, assessing community issues and needs, finding and choosing best practices, and/or evaluation. We consider the topic of participatory evaluation important enough to give it a section of its own, and to show how it fits into the larger participatory research picture.

It's a good idea to build stakeholder participation into a project from the beginning. One of the best ways to choose the proper direction for your work is to involve stakeholders in identifying real community needs, and the ways in which a project will have the greatest impact. One of the best ways to find out what kinds of effects your work is having on the people it's aimed at is to include those on the receiving end of information or services or advocacy on your evaluation team

Often, you can see most clearly what's actually happening through the eyes of those directly involved in it - participants, staff, and others who are involved in taking part in and carrying out a program, initiative, or other project. Previously, we have discussed how you can involve those people in conducting research on the community and choosing issues to address and directions to go in. This section is about how you can involve them in the whole scope of the project, including its evaluation, and how that's likely to benefit the project's final outcomes.

What is participatory evaluation?

When most people think of evaluation, they think of something that happens at the end of a project - that looks at the project after it's over and decides whether it was any good or not. Evaluation actually needs to be an integral part of any project from the beginning. Participatory evaluation involves all the stakeholders in a project - those directly affected by it or by carrying it out - in contributing to the understanding of it, and in applying that understanding to the improvement of the work.

Participatory evaluation, as we shall see, isn't simply a matter of asking stakeholders to take part. Involving everyone affected changes the whole nature of a project from something done for a group of people or a community to a partnership between the beneficiaries and the project implementers. Rather than powerless people who are acted on, beneficiaries become the copilots of a project, making sure that their real needs and those of the community are recognized and addressed. Professional evaluators, project staff, project beneficiaries or participants, and other community members all become colleagues in an effort to improve the community's quality of life.

This approach to planning and evaluation isn't possible without mutual trust and respect. These have to develop over time, but that development is made more probable by starting out with an understanding of the local culture and customs - whether you're working in a developing country or in an American urban neighborhood. Respecting individuals and the knowledge and skills they have will go a long way toward promoting long-term trust and involvement.

The other necessary aspect of any participatory process is appropriate training for everyone involved. Some stakeholders may not even be aware that project research takes place; others may have no idea how to work alongside people from different backgrounds; and still others may not know what to do with evaluation results once they have them. We'll discuss all of these issues - stakeholder involvement, establishing trust, and training - as the section progresses.

The real purpose of an evaluation is not just to find out what happened, but to use the information to make the project better.

In order to accomplish this, evaluation should include examining at least two areas:

• Process. The process of a project includes the planning and logistical activities needed to set up and run it. Did we do a proper assessment beforehand so we would know what the real needs were? Did we use the results of the assessment to identify and respond to those needs in the design of the project?  Did we set up and run the project within the timelines and other structures that we intended? Did we involve the people we intended to? Did we have or get the resources we expected?  Were staff and others trained and prepared to do the work? Did we have the community support we expected?  Did we record what we did accurately and on time?  Did we monitor and evaluate as we intended?
• Implementation. Project implementation is the actual work of running it. Did we do what we intended? Did we serve or affect the number of people we proposed to? Did we use the methods we set out to use? Was the level of our activity what we intended (e.g., did we provide the number of hours of service we intended to)? Did we reach the population(s) we aimed at? What exactly did we provide or do? Did we make intentional or unintentional changes, and why?
• Outcomes. The project's outcomes are its results - what actually happened as a consequence of the project's existence. Did our work have the effects we hoped for? Did it have other, unforeseen effects? Were they positive or negative (or neither)? Do we know why we got the results we did?  What can we change, and how, to make our work more effective?

Many who write about participatory evaluation combine the first two of these areas into process evaluation, and add a third - impact evaluation - in addition to outcome evaluation.  Impact evaluation looks at the long-term results of a project, whether the project continues, or does its work and ends.

Rural development projects in the developing world, for example, often exist simply to pass on specific skills to local people, who are expected to then both practice those skills and teach them to others.  Once people have learned the skills - perhaps particular cultivation techniques, or water purification - the project ends.  If in five or ten years, an impact evaluation shows that the skills the project taught are not only still being practiced, but have spread, then the project's impact was both long-term and positive.

In order for these areas to be covered properly, evaluation has to start at the very beginning of the project, with assessment and planning.

In a participatory evaluation, stakeholders should be involved in:

• Naming and framing the problem or goal to be addressed
• Developing a theory of practice (process, logic model) for how to achieve success
• Identifying the questions to ask about the project and the best ways to ask them - these questions will identify what the project means to do, and therefore what should be evaluated

What's the real goal, for instance, of a program to introduce healthier foods in school lunches?  It could be simply to convince children to eat more fruits, vegetables, and whole grains.  It could be to get them to eat less junk food.  It could be to encourage weight loss in kids who are overweight or obese.  It could simply be to educate them about healthy eating, and to persuade them to be more adventurous eaters. The evaluation questions you ask both reflect and determine your goals for the program.  If you don't measure weight loss, for instance, then clearly that's not what you're aiming at.  If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods?  You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

• Collecting information about the project
• Making sense of that information
• Deciding what to celebrate, and what to adjust or change, based on information from the evaluation

Why would (and why wouldn't) you use participatory evaluation?

Why would you use participatory evaluation?  The short answer is that it's often the most effective way to find out what you need to know, both at the beginning of and throughout the course of a project. In addition, it carries benefits for both individual participants and the community that other methods don't.

Some of the major advantages of participatory evaluation:

• It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects. If stakeholders, including project beneficiaries, are involved from the beginning in determining what needs to be evaluated and why - not to mention what the focus of the project needs to be - you're much more likely to aim your work in the right direction, to correctly determine whether your project is effective or not, and to understand how to change it to make it moreso.
• It can get you information you wouldn't get otherwise. When project direction and evaluation depend, at least in part, on information from people in the community, that information will often be more forthcoming if it's asked for by someone familiar. Community people interviewing their friends and neighbors may get information that an outside person wouldn't be offered.
• It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff. Those implementing the project and those who are directly affected by it are most capable of sorting out the effective from the ineffective.
• It can tell you why something does or doesn't work.Beneficiaries are often able to explain exactly why they didn't respond to a particular technique or approach, thus giving you a better chance to adjust it properly.
• It results in a more effective project. For the reasons just described, you're much more apt to start out in the right direction, and to know when you need to change direction if you haven't. The consequence is a project that addresses the appropriate issues in the appropriate way, and accomplishes what it sets out to do.
• It empowers stakeholders. Participatory evaluation gives those who are often not consulted - line staff and beneficiaries particularly - the chance to be full partners in determining the direction and effectiveness of a project.
• It can provide a voice for those who are often not heard.Project beneficiaries are often low-income people with relatively low levels of education, who seldom have - and often don't think they have a right to - the chance to speak for themselves. By involving them from the beginning in project evaluation, you assure that their voices are heard, and they learn that they have the ability and the right to speak for themselves.
• It teaches skills that can be used in employment and other areas of life. In addition to the development of basic skills and specific research capabilities, participatory evaluation encourages critical thinking, collaboration, problem-solving, independent action, meeting deadlines...all skills valued by employers, and useful in family life, education, civic participation, and other areas.
• It bolsters self-confidence and self-esteem in those who may have little of either. This category can include not only project beneficiaries, but also others who may, because of circumstance, have been given little reason to believe in their own competence or value to society. The opportunity to engage in a meaningful and challenging activity, and to be treated as a colleague by professionals, can make a huge difference for folks who are seldom granted respect or given a chance to prove themselves.
• It demonstrates to people ways in which they can take more control of their lives. Working with professionals and others to complete a complex task with real-world consequences can show people how they can take action to influence people and events.
• It encourages stakeholder ownership of the project. If those involved feel the project is theirs, rather than something imposed on them by others, they'll work hard both in implementing it, and in conducting a thorough and informative evaluation in order to improve it.
• It can spark creativity in everyone involved. For those who've never been involved in anything similar, a participatory evaluation can be a revelation, opening doors to a whole new way of thinking and looking at the world. To those who have taken part in evaluation before, the opportunity to exchange ideas with people who may have new ways of looking at the familiar can lead to a fresh perspective on what may have seemed to be a settled issue.
• It encourages working collaboratively. For participatory evaluation to work well, it has to be viewed by everyone involved as a collaboration, where each participant brings specific tools and skills to the effort, and everyone is valued for what she can contribute. Collaboration of this sort not only leads to many of the advantages described above, but also fosters a more collaborative spirit for the future as well, leading to other successful community projects.
• It fits into a larger participatory effort. When community assessment and the planning of a project have been a collaboration among project beneficiaries, staff, and community members, it only makes sense to include evaluation in the overall plan, and to approach it in the same way as the rest of the project.  In order to conduct a good evaluation, its planning should be part of the overall planning of the project. Furthermore, participatory process generally matches well with the philosophy of community-based or grass roots groups or organizations.

With all these positive aspects, participatory evaluation carries some negative ones as well.  Whether its disadvantages outweigh its advantages depend on your circumstances, but whether you decide to engage in it or not, it's important to understand what kinds of drawbacks it might have.

The significant disadvantages of participatory evaluation include:

• It takes more time than conventional process. Because there are so many people with different perspectives involved, a number of whom have never taken part in planning or evaluation before, everything takes longer than if a professional evaluator or a team familiar with evaluation simply set up and conducted everything.  Decision-making involves a great deal of discussion, gathering people together may be difficult, evaluators need to be trained, etc.
• It takes the establishment of trust among all participants in the process. If you're starting something new (or, all too often, even if the project is ongoing), there are likely to be issues of class distinction, cultural differences, etc., dividing groups of stakeholders.These can lead to snags and slowdowns until they're resolved, which won't happen  overnight. It will take time and a good deal of conscious effort before all stakeholders feel comfortable and confident that their needs and culture are being addressed.
• You have to make sure that everyone's involved, not just "leaders" of various groups. All too often, "participatory" means the participation of an already-existing power structure. Most leaders are actually that - people who are most concerned with the best interests of the group, and whom others trust to represent them and steer them in the direction that best reflects those interests. Sometimes, however, leaders are those who push their way to the front, and try to confirm their own importance by telling others what to do.

By involving only leaders of a population or community, you run the risk of losing - or never gaining - the confidence and perspective of the rest of the population, which may dislike and distrust a leader of the second type, or may simply see themselves shut out of the process.. They may see the participatory evaluation as a function of authority, and be uninterested in taking part in it.  Working to recruit "regular" people as well as, or instead of, leaders may be an important step for the credibility of the process. But it's a lot of work and may be tough to sell.

• You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills. There are really a number of potential disadvantages here. The obvious one is that of time, which we've already raised - training takes time to prepare, time to implement, and time to sink in.  Another is the question of what kind of training participants will respond to. Still another concerns recruitment - will people be willing to put in the time necessary to prepare them for the process, let alone the time for the process itself?
• You have to get buy-in and commitment from participants. Given what evaluators will have to do, they need to be committed to the process, and to feel ownership of it. You have to structure both the training and the process itself to bring about this commitment.
• People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation. Poor people everywhere live on the edge, which means they're engaged in a delicate balancing act. The least tilt to one side or the other - a sick child, too many days of rain in a row - can cause a disruption that may result in an inability to participate on a given day, or at all.  If you're dealing with a rural village that's dependent on agriculture, for instance, an accident of weather can derail the whole process, either temporarily or permanently.
• You may have to be creative about how you get, record, and report information. If some of the participants in an evaluation are non- or semi-literate, or if participants speak a number of different languages (English, Spanish, and Lao, for instance), a way to record information will have to be found that everyone can understand, and that can, in turn, be understood by others outside the group.
• Funders and policy makers may not understand or believe in participatory evaluation. At worst, this can lose you your funding, or the opportunity to apply for funding.  At best, you'll have to spend a good deal of time and effort convincing funders and policy makers that participatory evaluation is a good idea, and obtaining their support for your effort.

Some of these disadvantages could also be seen as advantages: the training people receive blends in with their development of new skills that can be transferred to other areas of life, for instance; coming up with creative ways to express ideas benefits everyone; once funders and policy makers are persuaded of the benefits of participatory process and participatory evaluation, they may encourage others to employ it as well. Nonetheless, all of these potential negatives eat up time, which can be crucial.  If it's absolutely necessary that things happen quickly (which is true not nearly as often as most of us think it is), participatory evaluation is probably not the way to go.

When might you use participatory evaluation?

So when do you use participatory evaluation?  Some of the reasons you might decide it's the best choice for your purposes:

• When you're already committed to a participatory process for your project. Evaluation planning can be included and collaboratively designed as part of the overall project plan.
• When you have the time, or when results are more important than time. As should be obvious from the last part of this section, one of the biggest drawbacks to participatory evaluation is the time it takes. If time isn't what's most important, you can gain the advantages of a participatory evaluation without having to compensate for many of the disadvantages.
• When you can convince funders that it's a good idea. Funders may specify that they want an outside evaluation, or they may simply be dubious about the value of participatory evaluation.  In either case, you may have some persuading to do in order to be able to use a participatory process.  If you can get their support, however, funders may like the fact that participatory evaluation is often less expensive, and that it has added value in the form of empowerment and transferable skills.
• When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of. Political, social, and interpersonal factors in the community can skew the results of an evaluation, and without an understanding of those factors and their history, evaluators may have no idea that what they're finding out is colored in any way. Evaluators who are part of the community can help sort out the influence of these factors, and thus end up with a more accurate evaluation.
• When you need information that it will be difficult for anyone outside the community or population to get. When you know that members of the community or population in question are unwilling to speak freely to anyone from outside, participatory evaluation is a way to raise the chances that you'll get the information you need.
• When part of the goal of the project is to empower participants and help them develop transferable skills. Here, the participatory evaluation, as it should in any case, becomes a part of the project itself and its goals.
• When you want to bring the community or population together. In addition to fostering a collaborative spirit, as we've mentioned, a participatory evaluation can create opportunities for people who normally have little contact to work together and get to know one another. This familiarity can then carry over into other aspects of community life, and even change the social character of the community over the long term.

Who should be involved in participatory evaluation?

We've referred continually to stakeholders - the people who are directly affected by the project being evaluated. Who are the stakeholders? That varies from project to project, depending on the focus, the funding, the intended outcomes, etc.

There are a number of groups that are generally involved, however:

• Participants or beneficiaries. The people whom the project is meant to benefit. That may be a specific group (people with a certain medical condition, for instance), a particular population (recent Southeast Asian immigrants, residents of a particular area), or a whole community.  They may be actively receiving a service (e.g., employment training) or may simply stand to benefit from what the project is doing (violence prevention in a given neighborhood). These are usually the folks with the greatest stake in the project's success, and often the ones with the least experience of evaluation.
• Project line staff and/or volunteers. The people who actually do the work of carrying out the project. They may be professionals, people with specific skills, or community volunteers. They may work directly with project beneficiaries as mentors, teachers, or health care providers; or they may advocate for immigrant rights, identify open space to be preserved, or answer the phone and stuff envelopes. Whoever they are, they often know more about what they're doing than anyone else, and their lives can be affected by the project as much as those of participants or beneficiaries.
• Administrators. The people who coordinate the project or specific aspects of it. Like line staff and volunteers, they know a lot about what's going on, and they're intimately involved with the project every day.
• Outside evaluators, if they're involved. In many cases, outside evaluators are hired to run participatory evaluations. The need for their involvement is obvious.
• Community officials. You may need the support of community leaders, or you may simply want to give them and other participants the opportunity to get to know one another in a context that might lead to better understanding of community needs.
• Others whose lives are affected by the project. The definition of this group varies greatly from project to project. In general, it refers to people whose jobs or other aspects of their lives will be changed either by the functioning of the project itself, or by its outcomes.

An example would be landowners whose potential use of their land would be affected by an environmental initiative or a neighborhood plan.

How do you conduct a participatory evaluation?

Participatory evaluation encompasses elements of designing the project as well as evaluating it. What you evaluate depends on what you want to know and what you're trying to do.  Identifying the actual evaluation questions sets the course of the project just as surely as a standardized testing program guides teaching.  When these questions come out of an assessment in which stakeholders are involved, the evaluation is one phase of a community-based participatory research process.

A participatory evaluation really has two stages: One comprises finding and training stakeholders to act as participant evaluators.  The second - some of which may take place before or during the first stage - encompasses the planning and implementation of the project and its evaluation, and includes six steps:

• Naming and framing the issue
• Developing a theory of practice to address it
• Deciding what questions to ask, and how to ask them to get the information you need
• Collecting information
• Analyzing the information you've collected
• Using the information to celebrate what worked, and to adjust and improve the project

We'll examine both of these stages in detail.

Finding and training stakeholders to act as participant evaluators

Unfortunately, this stage isn't simply a matter of announcing a participatory evaluation and then sitting back while people beat down the doors to be part of it.  In fact, it may be one of the more difficult aspects of conducting a participatory evaluation.

Here's where the trust building we discussed earlier comes into play. The population you're working with may be distrustful of outsiders, or may be used to promises of involvement that turn out to be hollow or simply ignored. They may be used to being ignored in general, and/or offered services and programs that don't speak to their real needs. If you haven't already built a relationship to the point where people are willing to believe that you'll follow through on what you say, now is the time to do it. It may take some time and effort - you may have to prove that you'll still be there in six months - but it's worth it. You're much more likely to have a successful project, let alone a successful evaluation, if you have a relationship of mutual trust and respect.

But let's assume you have that step out of the way, and that you've established good relationships in the community and among the population you're working with, as well as with staff of the project. Let's assume as well that these folks know very little, if anything, about participatory evaluation. That means they'll need training in order to be effective.

If, in fact, your evaluation is part of a larger participatory effort, the question arises as to whether to simply employ the same team that did assessments and/or planned the project, perhaps with some additions, as evaluators.  That course of action has both pluses and minuses.  The team is already assembled, has developed a method of working together, has some training in research methods, etc., so that they can hit the ground running - obviously a plus. 
The fact that they have a big stake in seeing the project be successful can work either way: they may interpret their findings in the best possible light, or even ignore negative information; or they may be eager to see exactly where and how to adjust the work to make it go better.

Another issue is burnout.  Evaluation will mean more time in addition to what an assessment and planning team has already put in. While some may be more than willing to continue, many may be ready for a break (or may be moving on to another phase of their lives).  If the possibility of assembling a new team exists, it will give those who've had enough the chance to gracefully withdraw.

How you handle this question will depend on the attitudes of those involved, how many people you actually have to draw on (if the recruitment of the initial team was really difficult, you may not have a lot of choices), and what people committed to.

Recruit participant evaluators

There are many ways to accomplish this.  In some situations, it makes the most sense to put out a general call for volunteers; in others, to approach specific individuals who are likely - because of their commitment to the project or to the population - to be willing.  Alternatively, you might approach community leaders or stakeholders to suggest possible evaluators.

Some basic guidelines for recruitment include:

• Use communication channels and styles that reach the people you're aiming at
• Make your message as clear as possible
• Use plain English and/or whatever other language(s) the population uses
• Put your message where the audience is
• Approach potential participants individually where possible - if you can find people they know to recruit them, all the better
• Explain what people may gain from participation
• Be clear that they're being asked because they already have the qualities that are necessary for participation
• Encourage people, but also be honest about the amount and extent of what needs to be done
• Work out with participants what they're willing and able to do
• Try to arrange support - child care, for example - to make participation easier
• Ask people you've recruited to recommend - or recruit - others

In general, it's important for potential participant evaluators - particularly those whose connection to the project isn't related to their employment - to understand the commitment involved. An evaluation is likely to last a year, unless the project is considerably shorter than that, and while you might expect and plan for some dropouts, most of the team needs to be available for that long.

In order to make that commitment easier, discuss with participants what kinds of support they'll need in order to fulfill their commitment - child care and transportation, for instance - and try to find ways to provide it. Arrange meetings at times and places that are easiest for them (and keep the number of meetings to a minimum). For participants who are paid project staff, the evaluation should be considered part of their regular work, so that it isn't an extra, unpaid, burden that they feel they can't refuse.

Be careful to try to put together a team that's a cross-section of the stakeholder population. As we've already discussed, if you recruit only "leaders" from among the beneficiary population, for instance, you may create resentment in the rest of the group, not get a true perspective of the thinking or perceptions of that group, and defeat the purpose of the participatory nature of the evaluation as well. Even if the leaders are good representatives of the group, you may want to broaden your recruitment in the hopes of developing more community leadership, and empowering those who may not always be willing to speak out.

Train participant evaluators

Participants, depending on their backgrounds, may need training in a number of areas. They may have very little experience in attending and taking part in meetings, for instance, and may need to start there. They may benefit from an introduction to the idea of participatory evaluation, and how it works.  And they'll almost certainly need some training in data gathering and analysis.

How training gets carried out will vary with the needs and schedules of participants and the project. It may take place in small chunks over a relatively long period of time - weeks or months - might happen all at once in the course of a weekend retreat, or might be some combination. There's no right or wrong way here.  The first option will probably make it possible for more people to take part; the second allows for people to get to know one another and bond as a team, and a combination might allow for both.

By the same token, there are many training methods, any or all of which might be useful with a particular group. Training in meeting skills - knowing when and how to contribute and respond, following discussion, etc. - may best be accomplished through mentoring, rather than instruction. Interviewing skills may best be learned through roleplaying and other experiential techniques. Some training - how to approach local people, for example - might best come from participants themselves.

Some of the areas in which training might be necessary:

• The participatory evaluation process. How participatory evaluation works, its goals, the roles people may play in the process, what to expect.
• Meeting skills. Following discussion, listening skills, handling disagreement or conflict, contributing and responding appropriately, general ground rules and etiquette, etc.
• Interviewing. Putting people at ease, body language and tone of voice, asking open-ended and follow-up questions, recording what people say and other important information, handling interruptions and distractions, group interviews.
• Observation. Direct vs. participant observation, choosing appropriate times and places to observe, relevant information to include, recording observations.
• Recording information and reporting it to the group. What interviewees and those observed say and do, the non-verbal messages they send, who they are (age, situation, etc.), what the conditions were, the date and time, any other factors that influenced the interview or observation.

For people for whom writing isn't comfortable, where writing isn't feasible, or where language is a barrier, there should be alternative recording and reporting methods.  Drawings, maps, diagrams, audio recording, videos, or other imaginative ways of remembering exactly what was said or observed can be substituted, depending on the situation.  In interviews, if audio or video recording is going to be used, it's important to get the interviewee's permission first - before the interviewer shows up with the equipment, so that there are no misunderstandings.

• Analyzing information. Critical thinking, what kinds of things statistics tell you, other things to think about.

Planning and implementing the project and its evaluation

There's an assumption here that all phases of a project will be participatory, so that not only its evaluation, but its planning and the assessment that leads to it also involve stakeholders (not necessarily the same ones who act as evaluators). If stakeholders haven't been involved from the beginning, they don't have the deep understanding of the purposes and structure of a project that they'd have of one they've helped form. The evaluation that results, therefore, is likely to be less perceptive - and therefore less valuable - than one of a project they've been involved in from the start.

Naming and framing the problem or goal to be addressed

Identifying what you're evaluating defines what the project is meant to address and  accomplish. Community representatives and stakeholders, all those with something to gain or lose, work together to develop a shared vision and mission. By collecting information about community concerns and identifying available assets, communities can understand which issues to focus a project on.

Naming a problem or goal refers to identifying the issue that needs to be addressed.  Framing it has to do with the way we look at it.  If youth violence is conceived of as strictly a law enforcement problem, for instance, that framing implies specific ways of solving it: stricter laws, stricter enforcement, zero tolerance for violence, etc. If it's framed as a combination of a number of issues - availability of hand guns, unemployment and drug use among youth, social issues that lead to the formation of gangs, alienation and hopelessness in particular populations, poverty, etc. - then solutions may include employment and recreation programs, mentoring, substance abuse treatment, etc., as well as law enforcement.  The more we know about a problem, and the more different perspectives we can include in our thinking about it, the more accurately we can frame it, and the more likely we are to come up with an effective solution.

Developing a theory of practice to address the problem

How do you conduct a community effort so that it has a good chance of solving the problem at hand? Many communities and organizations answer this question by throwing uncoordinated programs at the problem, or by assuming a certain approach (law enforcement, as in our example, for instance) will take care of it.  In fact, you have to have a plan for creating, implementing, evaluating, adjusting, and maintaining a solution if you want it to work.

Whatever you call this plan - a theory of practice, a logic model, or simply an approach or process - it should be logical, consistent, consider all the areas that need to be coordinated in order for it to work, and give you an overall guideline and a  list of steps to follow in order to carry it out.

Once you've identified an issue, for instance, one possible theory of practice might be:

• Form a coalition of organizations, agencies, and community members concerned with the problem.
• Recruit and train a participatory research team which includes representatives of all stakeholder groups.
• The team collects both statistical and qualitative, first-hand information about the problem, and identifies community assets that might help in addressing it.
• Use the information you have to design a solution that takes into account the problem's complexity and context.

This might be a single program or initiative, or a coordinated, community-wide effort involving several organizations, the media, and individuals.  If it's closer to the latter, that's part of the complexity you have to take into account. Coordination has to be part of your solution, as do ways to get around the bureaucratic roadblocks that might occur and methods to find the financial and personnel resources you need.

• Implement the solution.
• Carry out monitoring and evaluation that will give you ongoing feedback about how well you're meeting objectives, and what you should change to improve your solution.
• Use the information from the evaluation to adjust and improve the solution.
• Go back to # 2 and do as much of it again as you need to until the problem is solved, or - more likely, since many community problems never actually disappear - indefinitely in order to maintain and increase your gains.

Deciding what evaluation questions to ask, and how to ask them to get the information you need

As we've discussed, choosing the evaluation questions essentially guides the work.  What you're really choosing here is what you're going to pay attention to. There could be significant results from your project that you're never aware of, because you didn't look for them - you didn't ask the questions to which those results would have been the answers. That's why it's so important to select questions carefully: they'll determine what you find.

Framing the problem is one element here - putting it in context, looking at it from all sides, stepping back from your own assumptions and biases to get a clearer and broader view of it. Another is envisioning the outcomes you want, and thinking about what needs to change, and how, in order to reach them.

Framing is important in this activity as well.  If you want simply to reduce youth violence, stricter laws and enforcement might seem like a reasonable solution, assuming you're willing to stick with them forever; if you want not only to reduce or eliminate youth violence, but to change the climate that fosters it (i.e., long term social change), the solution becomes much broader and requires, as we pointed out above, much more than law enforcement.  And a broader solution means more, and more complex, evaluation questions.
In the first case, evaluation questions might be limited to some variation of:  "Were there more arrests and convictions of youthful offenders for violent crimes in the time period studied, as compared to the last period for which there were records before the new solution was put in place?"  "Did youthful offenders receive harsher sentences than before?"  "Was there a reduction in violent incidents involving youth?"

Looking at the broader picture, in addition to some of those questions, there might be questions about counseling programs for youthful offenders to change their attitudes and to help ease their transition back to civil society, drug and alcohol treatment, control of handgun sales, changing community attitudes, etc.

Collecting information

This is the largest part, at least in time and effort, of implementing an evaluation.

Various evaluators, depending on the information needed, may conduct any or all of the following:

• Research into census or other public records, as well as news archives, library collections, the Internet, etc.
• Individual and/or group interviews
• Focus groups
• Community information-sharing sessions
• Surveys
• Direct or participant observation

In some cases - particularly with unschooled populations in developing countries - evaluators may have to find creative ways to draw out information.  In some cultures, maps, drawings, representations ("If this rock is the headman's house..."), or even storytelling may be more revealing than the answers to straightforward questions.

Analyzing the information you've collected

Once you've collected all the information you need, the next step is to make sense of it. What do the numbers mean? What do people's stories and opinions tell you about the project?  Did you carry out the process you'd planned?  If not, did it make a difference, positive or negative?

In some cases, these questions are relatively easy to answer. If there were particular objectives for serving people, or for beneficiaries' accomplishments, you can quickly find out whether they were met or not. (We set out to serve 75 people, and we actually served 82. We anticipated that 50 would complete the program, and 61 actually completed.)

In other cases, it's much harder to tell what your information means. What if approximately half of interviewees say the project was helpful to them, and the other half say the opposite? A result like that may leave you doing some detective work. (Is there any ethnic, racial, geographic, or cultural pattern as to who is positive and who is negative?  Whom did each group work with? Where did they experience the project, and how? Did members of each group have specific things in common?)

While collecting the information requires the most work and time, analyzing it is perhaps the most important step in conducting an evaluation. Your analysis tells you what you need to know in order to improve your project, and also gives you the evidence you need to make a case for continued funding and community support. It's important that it be done well, and that it make sense of odd results like that directly above. Here's where good training and good guidance in using critical thinking and other techniques come in.

In general, information-gathering and analysis should cover the three areas we discussed early in the section: process, implementation, and outcomes. The purpose here is both to provide information for improving the project and to provide accountability to funders and the community.

• Process. This concerns the logistics of the project. Was there good coordination and communication? Was the planning process participatory? Was the original timeline for each stage of the project - outreach, assessment, planning, implementation, evaluation - realistic? Were you able to find or hire the right people?  Did you find adequate funding and other resources? Was the space appropriate? Did members of the planning and evaluation teams work well together?  Did the people responsible do what they were expected to do? Did unexpected leaders emerge (in the planning group, for instance)?
• Implementation. Did you do what you set out to do - reach the number of people you expected to, use the methods you intended, provide the amount and kind of service or activity that you planned for?  This part of the evaluation is not meant to assess effectiveness, but only whether the project was carried out as planned - i.e, what you actually did, rather than what you accomplished as a result.  That comes next.
• Outcomes. What were the results of what you did?  Did what you hoped for take place?  If it did, how do you know it was a result of what you did, as opposed to some other factor(s)?  Were there unexpected results?  Were they negative or positive?  Why did this all happen?

Using the information to celebrate what worked, and to adjust and improve the project

While accountability is important - if the project has no effect at all, for example, it's just wasted effort - the real thrust of a good evaluation is formative. That means it's meant to provide information that can help to continue to form the project, reshape it to make it better. As a result, the overall questions when looking at process, implementation, and outcomes are: What worked well?  What didn't?  What changes would improve the project?

Answering these questions requires further analysis, but should allow you to improve the project considerably. In addition to dropping or changing and adjusting those elements of the project that didn't work well, don't neglect those that were successful. Nothing's perfect; even effective approaches can be made better.

Don't forget to celebrate your successes. Celebration recognizes the hard work of everyone involved, and the value of your effort. It creates community support, and strengthens the commitment of those involved. Perhaps most important, it makes clear that people working together can improve the quality of life in the community.

There's a final element to participatory research and evaluation that can't be ignored. Once you've started a project and made it successful, you have to maintain it. The participatory research and evaluation has to continue - perhaps not with the same team(s), but with teams representative of all stakeholders. Conditions change, and projects have to adapt. Research into those conditions and continued evaluation of your work will keep that work fresh and effective.

If your project is successful, you may think your work is done.Think again - community problems are only solved as long as the solutions are actively practiced. The moment you turn your back, the conditions you worked so hard to change can start to return to what existed before The work - supported by participatory research and evaluation - has to go on indefinitely to maintain and increase the gains you've made.

In Summary

Participatory evaluation is a part of participatory research. It involves stakeholders in a community project in setting evaluation criteria for it, collecting and analyzing data, and using the information gained to adjust and improve the project.

Participatory process brings in the all-important multiple perspectives of those most directly affected by the project, who are also most likely to be tied into community history and culture. The information and insights they contribute can be crucial in a project's effectiveness. In addition, their involvement encourages community buy-in, and can result in important gains in skills, knowledge, and self-confidence and self-esteem for the researchers. All in all, participatory evaluation creates a win-win situation.

Conducting a participatory evaluation involves several steps:

• Recruiting and training a stakeholder evaluation team
• Naming and framing the problem
• Developing a theory of practice to guide the process of the work
• Asking the right evaluation questions
• Collecting information
• Analyzing information
• Using the information to celebrate and adjust your work

The final step, as with so many of the community-building strategies and actions described in the Community Tool Box, is to keep at it.  Participatory research in general, and participatory evaluation in particular, has to continue as long as the work continues, in order to keep track of community needs and conditions, and to keep adjusting the project to make it more responsive and effective.  And the work often has to continue indefinitely in order to maintain progress and avoid sliding back into the conditions or attitudes that made the project necessary in the first place.

What is participatory evaluation?

___Participatory evaluation is an evaluation that involves all the stakeholders in a project - those directly affected by it or by carrying it out - in every phase of evaluating it, and in applying the results of that evaluation to the improvement of the work

What are the advantages of using participatory evaluation?

___It gives you a better perspective on both the initial needs of the project's beneficiaries, and on its ultimate effects

___It can get you information you wouldn't get otherwise

___It tells you what worked and what didn't from the perspective of those most directly involved - beneficiaries and staff

___It can tell you why something does or doesn't work

___It results in a more effective project

___It empowers stakeholders

___It can provide a voice for those who are often not heard

___It teaches skills that can be used in employment and other areas of life

___It bolsters self-confidence and self-esteem in those who may have little of either

___It demonstrates to people ways in which they can take more control of their lives

___It encourages stakeholder ownership of the project

___It can spark creativity in everyone involved

___It encourages working collaboratively

___It fits into a larger participatory effort

What are the disadvantages of using participatory evaluation?

___It takes more time than conventional process

___It takes the establishment of trust among all participants in the process

___You have to make sure that everyone's involved, not just "leaders" of various groups

___You have to train people to understand evaluation and how the participatory process works, as well as teaching them basic research skills

___You have to get buy-in and commitment from participants

___People's lives - illness, child care and relationship problems, getting the crops in, etc. - may cause delays or get in the way of the evaluation

___You may have to be creative about how you get, record, and report information

___Funders and policy makers may not understand or believe in participatory evaluation

When would you use participatory evaluation?

___When you're already committed to a participatory process for your project

___When you have the time, or when results are more important than time

___When you can convince funders that it's a good idea

___When there may be issues in the community or population that outside evaluators (or program providers, for that matter) aren't likely to be aware of

___When you need information that it will be difficult for anyone outside the community or population to get

___When part of the goal of the project is to empower participants and help them develop transferable skills

___When you want to bring the community or population together

Who should be involved in participatory evaluation?

___Participants or beneficiaries

___Project line staff and/or volunteers

___Administrators

___Outside evaluators, if they're involved

___Community officials

___Others whose lives are affected by the project

How do you conduct a participatory evaluation?

___Recruit stakeholders as participant evaluators

___Train evaluators

___Name and frame the issue

___Develop a theory of practice to address it

___Determine the evaluation questions

___Collect information

___Analyze the information

___Use your analysis to celebrate what worked and adjust the rest to improve the project

___Stick with it indefinitely

Tool 1: Team Data Collection - How to Do It Well

This is a summary of responses from e-listserves ARLIST-L (Action Research) and CBPR (Community-Based Participatory Research) to an inquiry posted by Michelle Garred in November 2007. We have included her initial e-mail, and then added the summarized reponses to each of her questions.

From: Michelle Garred
Subject: Team data collection: how to do it well?

Dear colleagues,

I am working on an action research project aimed at field-testing a new method for community-based peacebuilding, in cooperation with a partner agency in the Philippines. Within the agency, there is an established core team assigned to this project, and we collaborate in the research design, data collection, data analysis and dissemination of results. The project has two distinct outcomes: A) a practitioner-focused lessons learned publication to be published by the partner agency and B) my Ph.D. thesis.

Much of our data collection will be conducted by a team, rather than the more traditional approach of data collection by an individual researcher. I am wrestling with some questions about how to do this well.

Among the many questions are:

• What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?
• How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?
• How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?
• In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?
• How would you cite team interviews in a Ph.D. thesis?
• What are the key resources I should review?

What are the methodological advantages of team data collection? What are the pitfalls that we should avoid?

Advantages:

• Data collection - Interview more participants / increase sample size, thus presumably increasing reliability and validity of the data while reducing time and cost to do the study.
• Data analysis - Diversity within the research team can enrich interpretation and lead to fresh perspectives.

Pitfalls:

• More personalities and activities to coordinate means more opportunities for things to go wrong.
• Quality assurance: possibility of inconsistencies in question interpretation; possibility of inadequate, erroneous, or extraneous data.
• Team motivation may requirement payment and/or incentive, and full support of the team’s supervisor.

Pitfalls can be addressed through training:

• Need for consistent, systematic training of data collectors.
• Lead researcher conduct pilot interviews first, to get a handle on how to better train the team.
• Conduct interviewing practice sessions before beginning actual interviews.
• During practice, use some standardized interviewees for training purposes and set a high inter-rater reliability for interviewers to make sure they have demonstrated proper technique before you let them loose on your subject population.
• Make sure all interviewers know how to ask delving follow-up questions to elicit clarity of meaning, depth, examples, etc.
• Ensure interviewing ground rules are clear. For example, what follow-up comments may an interviewer add to help an interviewee understand the question? What words may be used and/or not used?
• Demonstrated skills before graduation from training phase

And through regular team meetings:

• Meet periodically throughout the data collection period. This will help to identify any difficulties, and refine your process as you go along.
• Have regular – daily at first if necessary – debriefing, trouble-shooting meetings with all the data collectors, so all can learn from each others’ problems and solutions.
• Regular quality assurance meetings. Since these are partners and not hired data collectors, you will have to train them in a collaborative and diplomatic way. Group review of the data collected each day or each week may help, serving both for QA and as a support group.
• If not together on site, consider on-line conferencing.

How can we design the written question protocols, the note-taking templates, and the note-taking processes in ways that maximize the depth and consistency of data?

Keep in mind that we will have a team of 5 people conducting semi-structured interviews and we will probably not use audio recording.

• Conduct some preliminary focus groups to clarify the questions and problems. Have the interviewers sit in and/or help with the focus groups as part of their training and debrief with them afterwards.
• Use the focus group data to develop some checklists/checkboxes for anticipated responses so that you can have at least some quantitative data.
• Develop the interview guide together with the interview team. It is possible to use a participatory process such as “card storming” to build team consensus on the questions to be asked.  Here's one example of a card storming.
• If possible, use two-person data collector teams with the second person taking notes and listening/looking for any problems.

How can I encourage and assist my partners in note-taking, when written documentation is not their preferred or strongest skill?

Again, this can be addressed through team training and regular meetings . . .

• Show the trainee interviewers how it’s done right, but also how it’s done badly. They need to see both the model behavior and also what mistakes to avoid.
• Find a way to illustrate to them how faulty memories of interview material can be, and how difficult it is to analyze interview data when there is little depth of content.
• Revisit this issue during regular team meetings during the data collection phase.

And through wise design of the note-taking protocols:

• Ensure the written interview protocol has space for note-taking in answer to each question, and also for noting unprompted comments from the interviewee.
• Additionally, you might have team members write up a brief summary within 24 hours of completing the interview.
• Focus the note-taking on direct recall of interviewee’s words. Interviewers should strive not to conduct implicit analysis at this point. Reserve analysis for a separate follow-up step.

In my Ph.D. thesis, what might be an acceptable proportion of data collected by team versus data collected directly by me?

I understand that team data collection is common in action research, but less common in a Ph.D. thesis. It would help me to hear some different opinions on this issue.

• For quantitative data, one might set a quota of random 10% of data reviewed for QA.
• I would suggest doing as many as you possibly can, up to the average done by the team, so you can know what they’re experience really is.
• As long as you are directing the study, I wouldn’t have a problem with a sharing of the data collection; however, you would need to be integrally involved in all aspects of the study, particularly coding and analysis.
• Ortrun Zuber-Skerritt and Chad Perry suggest that you distinguish between the field research (where others are involved) and the thesis research, which is your reflection on the field research.
  • Zuber-Skerritt, Ortrun, and Perry, Chad (2002)  Action research within organisations and university thesis writing. The Learning Organization, 9(4), 171-179.

How would you cite team interviews in a Ph.D. thesis?

• In table format within the document.
• I assume the interview data and individual interviewer will be de-identified in the thesis itself, even if you know who said what in your field notes and quality assurance activities.
• I think I would explain in the body of the text just how the study was carried out, including the qualifications of the team members. Since they are part of the Human as Instrument section, their backgrounds and experience as researchers would need to be shared to instill a sense of trust on the part of the reader for the qualifications of the data collectors.
• When referencing, check APA, but I think it would involve citing interviews in the text by noting the date of the interview, then citing APA style in the bibliography.
• Consider in advance whether you will use the participants' real names or pseudonyms.

Key resources

• See Madeline Church's doctoral thesis, Creating an Uncompromised place to belong - why do I find myself in networks? from the living theory section of Action Research.
• The following paper describes some of the advantages of using a particular form of team-based qualitative research interviews. You may find it useful: Driedger, S. Michelle; Gallois, Cindy; Sanders, Carrie; and Santesso, Nancy (2006).  "Finding common ground in team-based qualitative research using the convergent interviewing method." Qualitative Health Research, 16(8), 1145-1157.
  • There's a detailed critique in the following book chapter: Rao, Sally, and Perry, Chad (2006). Convergent interviewing: a starting methodology for enterprise research program. In Hine, Damian, and Carson, David, eds., Innovative methodologies in enterprise research, 86-99. Cheltenham, UK: Edward Elgar.

• A good reference is Clifford Geertz's 1973 book, The Interpretation of Cultures. The questions you are asking are actually much deeper and more philosophical than the operational issues you asked about. You are basically working within anthropological practices and rules about fieldnotes and fieldwork rather than traditional interviews like we do in psychology.

  Tool 2: Dabbling in Data

Dabbling in the Data from Public Profit is a practical guide that helps fill this gap. It shares easy-to-use, hands-on ways to explore and analyze data often found in mission-driven work. Whether you are just starting out or want to feel more confident with data, this guide gives you tools and ideas to make data exploration easier and more helpful.

The latest edition, released in late 2024, includes:

• 10 new activities
• More user-friendly tools
• Virtual adaptations
• Playlists for mixed methods analysis

Example 1: True Collaboration for a Win-win in Community-engaged Research

Dr. Marvia Jones, Director of the Kansas City Health Department, spoke on partnerships in community-engaged research at the Life Span Institute's 2024 Collaboratory on January 11th, 2024: 

When we think about the word "partner," we can think about a significant other. If I were to say, "What do you want out of your partnership with your significant other?", you may say things like, "you're with me from beginning to the end, you learn about what motivates me, you learn about what excites me, what I need in order to get through the day or the month or the year." Right? And so, I would say, from the community side, regarding partnership and community-engaged research, we want similar things.

We want someone who is with us, who engages us from the very beginning of a process, so we actually feel like a partner. In that process, do we look at the problem together? How do we discuss what both of us need to get out of this?

I've heard this idea of researchers walking alongside community partners. What that also means is if I, as a practitioner, as a partner, need to take survey data to my governing board, or my officials, and it needs to be presented in a certain way - maybe that is not necessarily how the research originally planned to share the information. Well, now we have a dilemma. But as a partnership, through our partnership, we have to work through those solutions.

Those are some of the things that I think we look for in a community-based research partnership, having that flexibility and understanding. One of the things that I noticed, as I went from being a researcher to being in practice, is you lose some of the hours that you used to have to sit and wonder, to brainstorm together in team meetings and research lab meetings, and conferences, where you can consider, “you know what would be cool to think about?”

What I've realized is that once you're in the job of reporting to a board and reporting to a council, you've got to have results. You've got to deliver. We are on a timeframe. And so, we do need a partner who is doing that thinking for us, and with us, making the space and providing information in bites that are accessible and digestible for us. And understanding we don't have a lot of time to pore over data and records. But we do want useful, timely information.

So, when it comes to partnerships, I like to think of it in terms of a marriage where you're trying to really meet each other's needs, mutually.

• What do we mean by choosing evaluation questions?

• Why is it necessary to choose evaluation questions carefully?

• When should you choose questions and plan the evaluation?

• Who should be involved in the process?

• How do you choose questions and plan the evaluation?

Chapters 36-39 of the Community Tool Box concern the evaluation of community programs.  We've chosen to devote so much space to evaluation because it's one of the most important parts of any effort to improve community life and bring about lasting social change.  It can help you to better understand how well the planning and preparations for your program went, whether you implemented it as you meant to, and what the consequences were.  It can tell you whether you met your original objectives and goals or not, and give you information about what you need to change to be more effective.

Chapter 36 explains how an evaluation works, and gives some guidance as to how to develop it. Chapter 38 will assist you in gathering and analyzing the information you want, and Chapter 39 deals with how to use that information to improve your program and garner community and funding support. Here, in Chapter 37, we look at evaluation from a research point of view - how to plan and structure an evaluation that can help you better understand and improve what you do.

In this section, we'll discuss the first, and perhaps most important, step in evaluation research: deciding exactly what to evaluate. Each of the rest of the sections in the chapter will deal in detail with one of the steps you'll need to take to design, implement, and use the evaluation. The goal of the chapter is to provide guidelines that are useful to grassroots or community-based organizations as well as students or academic researchers.

What do we mean by choosing questions, and why is it necessary?

Every evaluation, like any other research, starts with one or more questions. Sometimes, the questions are simple and easy to answer. (Will we serve something close to the 50 people we expect to?) Often, however, the questions can be complex, and the answers less easy to find. (Which, or which combination, of the three parts of our intervention will affect which of the two behavior changes we seek within participants?) The questions you ask will guide not only your evaluation, but your program as well. By your choice of questions, you're defining what it is you're trying to change.

You choose your evaluation questions by analyzing the community problem or issue you're addressing, and deciding how you want to affect it. Why do you want to ask this particular question in relation to your evaluation? What is it about the issue that is the most pressing to change? What indicators will tell you whether that change is taking place?  Is that all you're concerned with? The answer to each of these and other questions helps to define what it is you're trying to do, and, by extension, how you'll try to do it.

For example, what's the real goal of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss in kids who are overweight or obese.  It could be to educate them about healthy eating, and to persuade them to be more adventurous eaters.

The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure weight loss, for instance, then clearly that's not what you're aiming for. If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods? You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

Things to consider when choosing evaluation questions

What do you want to know?

Academics and other researchers may approach choosing research questions differently from those involved in community programs.  In addition to their practical and social applications, they may choose problems to research simply because they are interesting, or because they tie into other work that they or their colleagues are doing. Community service workers and others directly involved in programs, on the other hand, are concerned specifically with improving what they're doing so they can help to enhance the quality of life for the participants in their programs, and often for the community as a whole.  Since we assume that most people using this chapter of the Tool Box are likely to be practitioners in the community, let's look at some of the reasons they might pick a particular area to evaluate.

If you're running, or about to run, a program to affect a community issue or problem, you might want to know one or more of the following:

Is there a cause-and-effect relationship (i.e., does one action or condition directly cause another) between a particular action and a particular change?  Usually, you'll be concerned with this in terms of your program. (Does our smoking-cessation support group help members to quit smoking?) Sometimes, however, it might be important to look at it in terms of the community. (Does a smoking ban in public buildings, bars, and restaurants lead to a decrease in the number of community residents who smoke?)

If we try this new method, what will happen?

Will the program that worked in the next town, or the one that we read about in a professional journal, work with our population, or with our issue?

Why are you interested?

Some of the same differences between the concerns of researchers and the concerns of practitioners may hold here. Those interested primarily in research may simply be moved by curiosity or by the urge to solve a difficult problem. As a practitioner, on the other hand, you'll want to know the effects of what you're doing on the lives of participants or the community.

Your interest, therefore, might grow from:

• Your experience with an issue and its consequences in a particular population or community
• Your knowledge of promising interventions and their effects on similar issues
• The uniqueness of the issue to your particular community or population
• The similarity of the issue to other issues in your community, or the issue's interaction with other issues

Your interest as a community worker has to be considered in relation to your evaluation and the purpose of your program. Your basic intent is probably to improve things for the population or the community, but in what ways and by what means? Are you trying out some new things in the hope of making an already-successful program more successful? Are you importing a promising practice to see if it works with your population?  Are you trying to solve a particularly difficult professional problem?

A community mediation program found that it was having little success in cases involving adolescents.  After conferring with other similar programs - all of which were struggling with the same issue - mediators in the program devised a number of strategies to try to reach youth. The overall question they were concerned with - "Will these strategies make it possible to mediate successfully where teens are involved?" - was one with real consequences.

Is the issue you're addressing important to the community or to the society?

Media reports about or community attempts to address the issue are clear indicators that it is socially important. If it affects a particular group - violence in a given neighborhood, a high rate of heart disease among middle-aged Black males - it has an obvious impact on the community and society. If your program or intervention has the potential to help resolve the issue in other places, to be used by community workers in other fields, or to be applied in a number of ways, the importance of your analysis increases even further. If addressing the issue can lead to long-term positive social change, then the analysis is vitally important.

All of this affects your evaluation and the questions you ask. If the issue is one of social importance, then your evaluation of your work is socially important as well.  Are you addressing the aspects of your program or intervention that are of the greatest value to participants, the community, and society?  If not, how might you begin to do so?

How does the issue relate to the field?

The real question here is not whether the issue is important to the field - if it's important to the community, that's what matters. However, you should explore whether there's evidence from the field to apply to the issue.  Is what you're doing likely to be more effective than other approaches that have been tried?  If your approach isn't effective, are there other approaches out there that hold more promise? Can the published material about the issue help you understand it better, and give you better ideas about how to address it?

Is the issue general, rather than specific to your population or community?

Consider whether there is evidence that the issue occurs with a variety of populations and under a range of conditions. Also consider whether the observations or methods used to determine the issue's existence are accurate and whether they can be used in different situations and with different groups. Your evaluation may give you valuable information to pass on to practitioners in different fields or different circumstances.

Who might use the results of your evaluation?

If evaluation shows that your program or intervention is successful, that's obviously valuable information, especially if what you're evaluating is innovative and hasn't been tried before. Even if the evaluation turns up major problems with the intervention, that's still important information for others - it tells them what won't work, or what barriers have to be overcome in order to make it work.

Some of those who might use your results include individuals and groups affected by the issue; service providers and others who have to deal with the problem (in the case of youth violence, for instance, this last group might include police, school officials, small business owners, parents, and medical personnel, among others); advocates and community activists; and public officials and other policy makers.

Whose issue is it?

Who has to change in order to address the issue? The focus of the intervention will tell you whom the evaluation should focus on.

Some possibilities:

• Those directly affected by the problem
• Those in direct personal contact with those directly affected: parents, spouses and children, other relatives, friends, neighbors, coworkers
• Those who serve or otherwise deal with those directly affected: medical professionals, police, teachers, social workers, therapists, etc.
• Administrators and others who serve or deal with those indirectly affected: hospital or clinic directors, police chiefs, school principals, agency directors, etc.
• Appointed or elected officials and other policy makers

Why is it necessary to choose evaluation questions carefully?

You know why you're running your program.  Evaluating it should just be a matter of deciding whether things are better when you evaluate than they were before you started, right?  Well, actually...wrong.  It's not that simple. First of all, you need to determine what "things" you are actually looking at (remember the school lunch example?)  Second, you will need to consider how you will determine what you're doing right, and what you need to change. Here's a partial list of reasons why choosing questions beforehand is important.

• It helps you understand what effects different parts of your effort are having. By framing questions carefully, you can evaluate different parts of your effort. If you add an element after the start of the program, for instance, you may be able to see its effect separate from that of the rest of the program...if you focus on examining it.  By the same token, you can look at different possible effects of the program as a whole. (Do adult basic education learners read more as a result of being in a program?  Are they more likely to register to vote? Do their children improve their school performance?)
• It makes you clearly define what it is you're trying to do  What you decide to evaluate defines what you hope to accomplish. Choosing evaluation questions at the start of a program or effort makes clear what you're trying to change, and what you want your results to be.
• It shows you where you need to make changes. Carefully choosing questions and making them specific to your real objectives should tell you exactly where the program is doing well and where the program isn't having the intended effect.
• It highlights unintended consequences. When you find unusual answers to the questions you choose, it often means that your program has had some effects you didn't expect. Sometimes these effects are positive - not only did people in the heart-healthy exercise program gain in fitness, but a majority of them report changing their diet for the better and losing weight as well - sometimes negative - obese children in a healthy eating program actually gained weight, even though they were eating a healthier diet - and sometimes neither. Like the side effects of medication, the unintended consequences of a program can be as important as the program itself. (In the case of the exercise program, the changes in diet might do as much as or more than the exercise to maintain heart health, for instance, and may point toward changing the focus of the program in some way.)
• It guides your future choices. If you find that your program is particularly successful in certain ways and not in others, for example, you may decide to emphasize the successful areas more, or to completely change your approach in the unsuccessful areas. That, in turn, will change the emphasis of future evaluation as well.
• In participant evaluations, evaluation involves stakeholders in setting the course of the program, thus making it more likely that it will meet community needs.
• It provides focus for the evaluation and the program. Choosing evaluation questions carefully keeps you from becoming scattered and trying to do too many things at once, thereby diluting your effectiveness at all of them.
• It determines what needs to be recorded in order to gather data for evaluation. A clear choice of evaluation questions makes the actual gathering of data much easier, since it usually makes obvious what kinds of records must be kept and what areas need to be examined.

When should you choose questions and plan the evaluation?

Evaluation questions, since they help shape your work, should be chosen and the evaluation planned when planning the overall program or effort. That gives you time and room for a participatory process, and gives you the chance to use the evaluation as an integral part of the program. As the program unfolds, you might find yourself adjusting or adding questions to reflect the reality of what is happening, but unless your original questions were misguided (you were wrong about what behavior had to change in order to produce certain results, for instance), they should serve you well.

Now let's discuss reality for many community based and grassroots programs. They're often understaffed and underfunded. Staff members may be underpaid, and may often work many more hours a week than they're paid for, because of their dedication to social justice and social change. Most or all program staff may even be volunteers, with full-time jobs and family responsibilities aside from their work in the program.  Initial evaluation in these circumstances is often anecdotal - i.e., based on participants' comments and stories about their progress and staff members' personal, informal observations.  A formal evaluation will probably wait until there's funding for it, or until someone has the time to coordinate or take charge of it.

In that case, the "when" becomes "as soon as you can."  You may be dealing with a program that has just started, or with one that's been operating for a long time. You may know that changes need to be made, or it may seem that the program is in fact meeting its goals. Whatever the situation, evaluation questions need to be chosen, and an evaluation planned that will give you the information you need to improve your work. Even with a program that's been going on for a while, the questions can still help you define or redefine your work, and will certainly help you improve it over the long term.

Who should be involved in choosing questions and planning the evaluation?

If you've consulted other sections of the Tool Box concerned with evaluation, you probably know that we advocate that all stakeholders be involved in planning the evaluation.  We believe that the best evaluation is participatory. That means that there is representation of the views and knowledge of people affected by the issue to be addressed. The list of potential participants is essentially the same as that under "Whose problem is it?" in the first part of this section: those directly affected and their close contacts; those who work with those directly affected, or who deal directly or indirectly with them and the issue; and public officials. To these groups, we might add other concerned citizens, and those indirectly affected by the issue. (A shop owner may not be a victim of neighborhood violence, but fear of that violence might nonetheless keep customers away from his shop, for instance.)

Evaluations that involve all stakeholders have a number of advantages over those conducted in a vacuum by outside evaluators or agency or program staff. They're more likely to reflect the real needs of the community, and they bring to bear the community's knowledge of its own context - history, relationships, culture, etc. - without which a program and its evaluation can go astray.

Participation can range from simple consultation before the fact to complete involvement in every aspect of an evaluation - assessment, planning, data gathering, analysis, and passing on the information.  In general, the greater the involvement of stakeholders, the better, but in-depth involvement of the stakeholders may not always be possible. There are time disadvantages to participatory evaluation - it takes longer - and there are logistical concerns, as well. Participants may have nothing in their backgrounds to prepare them for research, so training in a number of areas may be necessary, requiring skill, careful planning, and yet more time. The level of participation your evaluation can sustain, therefore, relies to some extent on your time constraints and your capacity to train and support participants.

How do you choose questions and plan the evaluation?

Choosing questions

When you choose evaluation questions, you're really choosing a research problem - what you want to examine with your research. (Evaluation, whether formal or informal, is in fact research.) You have to analyze the issue and your program, consider various ways they can be looked at, and choose the one(s) that most nearly tell you what you want to know about what you're doing.  Are you just trying to determine whether you're reaching the right people in sufficient numbers with your program?  Do you want to know how well an intervention is working with specific populations? What kinds of behavior changes, if any, are taking place as a result?  What the actual outcomes are for the community? Each of these - as well as each of the many other things you might want to know - implies a different set of evaluation questions. To find the questions that best suit your evaluation, there is a series of steps you can follow.

Describe the issue or problem you're addressing

A problem is a difference between some ideal condition (all people 10 years of age or older should be able to read; people should be able to find a decent job) and some actual condition in the community or society (a 25% illiteracy rate among those attending a particular high school; 50% unemployment among minority youths in a particular city). This may mean the absence of some positive factor (qualified teachers and adequate educational facilities; entry-level jobs that are reachable from minority neighborhoods) or the presence of some negative factor (students' difficulty with English; discrimination against minority job applicants), or some combination of these.

To describe the issue or problem:

• Describe the ideal condition, including the positive factors present and the negative factors absent.  What should it look like if everything was as you'd want it to be?
• Describe the actual conditions that constitute the prolem of interest, including the negative conditions present and the positive conditions absent.  What are conditions really like?
• Describe the actual problem in terms of what you're hoping to change.  What positive factors do you want to produce and/or what negative factors do you want to eliminate?

Describe the importance of the problem

To be sure that this is a problem you really should be addressing, consider its importance to those affected and to the community.

• Is the discrepancy between ideal and actual conditions of the kind and size to be considered important?
• What are the consequences (positive and negative) of the problem?
• Who experiences these consequences (i.e. program participants; their families, friends, and peers; service providers, policymakers, and others)? How many people are affected?
• How often and for how long are they affected? What is the intensity of the effect?
• How much does the fact that the problem is experienced to this degree by these people matter to them?

You might also ask whether the effects of the problem matter to society, but in fact, that shouldn't make a difference.  If they matter to the people who experience them, they're important.  Society doesn't always consider a problem important if it's only a problem for a minority, or for a group that's generally ignored (the poor, the homeless).

In light of these factors, decide whether the problem is important to the evaluation.

Describe those who contribute to the problem

Whose behavior, by its presence or absence, contributes to the problem?  Are they in the program participants' personal environment (participants themselves, family, friends), service environment (teachers, police), or broader environment (policymakers, media, general public)? For each of them, consider the types of behavior that, by their presence or absence, contribute to the discrepancy that constitutes the problem.

Assess the importance and feasibility of changing those behaviors

How important is each of these behaviors to solving the problem?  What are the chances that your effort can have any effect on each of them?

Describe the change objective

Based on the above analysis, choose behavior changes to target in specific people. Where you can, specify the desired levels of change in targeted behaviors and outcomes (those changes in conditions that should occur if the problem were to be solved).

For example, a behavior change goal might be an increase in pre-employment capacity - self-presentation, job-seeking, interview skills, interpersonal competence, resume writing, basic skills, etc. - for minority job seekers aged 18-24. Or you might instead or in addition target policy makers, with the goal of having them offer tax incentives to businesses that locate in or close to minority communities.

This is a way of defining your work. If you're planning the evaluation as you plan the program - as you would in the ideal situation - then the questions you're asking the evaluation to examine reflect the problems you're trying to solve, and this kind of analysis is important.  If you're starting an evaluation of a program that has been in place for some time, then you're going to have to do some figuring after the fact about what consequences you think (hope) the program is having, and what they will lead to.  You may be talking about changes in specific participant behaviors, about behaviors that act as indicators of other changes, or about results of another sort (participants gaining employment, for instance, which may have a direct relationship to participant behavior or may have more to do with local economic conditions).

Make sure that the expected changes would constitute a solution or substantial contribution to the problem

If you conclude that they would not result in a substantial contribution, revise your choice of problem and/or your selection of targeted people and actions as necessary. If you think that what you're looking at in an evaluation doesn't address the problem, then you should be looking at something else. If the objectives you've chosen do constitute all or a substantial part of a solution, you've found your questions.

Setting

Now that you've chosen your questions, there may be other factors to consider, such as the settings in which the evaluation will be conducted. If your program is relatively small and/or has only one site, this wouldn't be an issue. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program.

 There are some situations in which the choice setting may be important:

• If your program is very large and/or has multiple sites
• If different sites provide different services, activities, or conditions, or use different methods

Multiple sites

Multiple sites

Can present a challenge for an evaluation, because, although every effort may be made to make the program at all sites exactly the same, it will seldom be so. If the program relies on human interaction - teacher/learner, counselor/counselee, trainer/trainee, doctor/patient, etc. - there will be differences from site to site depending on the people staffing each. (The exception is when the same people staff all sites, providing the same services at each site at different times or on different days.)  Even if all are equally competent, no two staff members or teams will do things in exactly the same way or relate to participants in exactly the same way, and the differences can be reflected in differences in outcomes.  If methods or other factors vary from site to site, that will further complicate the situation.

Furthermore, the physical character of a site can influence not only program effectiveness, but also the recruitment of participants and whether or not they remain in the program long enough for it to have some effect (often called "retention.")  The site's layout, comfort, apparent safety and security, and - often most important - how easy it is to get to, all affect whether participants enroll and stay in the program.

Where you do have the capacity to evaluate all sites, it will be helpful to build into the evaluation a method of comparing them. This will allow you to identify and adopt at all sites methods, conditions, or activities that seem to make one site particularly successful, and to identify and change at all sites methods, conditions, or activities that seem to create barriers to success at others.

If you can't evaluate each site separately, you'll have to decide which one(s) will give you the information that will most help in adjusting and improving your program. If you're most concerned with assessing your overall effectiveness, this may mean evaluating the site(s) closest to the program norm, in terms of methods, conditions, activities, goals, participant/staff interaction, etc. If, on the other hand, your chief consideration is learning whether a particular new or unusual method or situation is working, you may find yourself evaluating the site(s) least like the others.

If sites appear only minimally different, some other considerations that may come into play are:

• The number and character of participants at the site. Participants at a particular site may be experiencing the effects of the issue more severely, or may have a particular important characteristic, such as a language barrier.
• The ability and willingness of participants and staff to support the evaluation research. If staff at a particular site are unable or unwilling to record observations, attendance, and other key information, or if site participants are unable or unwilling to be interviewed or monitored, evaluation at that site might be difficult.
• The stability of the population at the site. If participants at a site come and go at a rapid rate - unless that's the program's intent - it can be difficult to gain information that contributes to an accurate evaluation.
• An exception, of course, occurs here if one point of the evaluation is to find out why participants stay for so short a time, and to try to develop methods or create conditions to assist them to remain in the program long enough to reach their goals.

Sites with different methods, conditions, activities, or services

Programs sometimes are organized so that different methods are used or different services provided at different sites. In other cases, conditions may vary from site to site because of the sites' geographical locations or the available space. The ideal situation is to evaluate all sites and compare the effects of the different methods, conditions, or services. When that's not possible, you'll have to decide what's most important to find out.

If the methods, services, or conditions at a particular site are new or innovative, you may want to evaluate them, rather than those that have a track record. There may be a particular method or service that you want to evaluate, in which case the decision about which site to choose is obvious. The decision should be based on what makes the most sense for your program, and what will give you the best information to improve its effectiveness.

When you have the capacity to choose more than one site to evaluate, it often makes sense to choose two or three sites that are different - especially if each is representative of other sites in the program or of program initiatives - so that you can compare their effectiveness. Even where sites are essentially similar, you'll get more information by evaluating as many as you can.

Participants

Another factor to consider is the participants whose behavior, activity, or circumstances will be evaluated. If your program is relatively small this might not be an issue - the participants will simply be all those in the program. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program, there are some situations in which the choice of participants may be important:

• If your program includes different groups of participants (groups that are in different stages of the program, or that are exposed to different methods or services).
• If groups of participants belong to populations with distinctly different cultures, stemming from race, ethnicity, class, religion, or other factors.

Multiple groups

There are a number of reasons why there might be multiple groups of participants in a program. You might start different groups at different times, either because the program has a rolling start schedule (when there are enough people for a class/training group, one will begin), or because the program is aimed at different groups (for example, 5 year-olds, 8-year-olds, and 14-year-olds). You might also be trying different strategies with different groups.

The Brookline Early Education Project (BEEP), a program aimed at school readiness for children aged pre-birth through 5, recruited pregnant families in three cohorts over the course of three years.  In addition, families in each cohort were assigned to one of three levels of service. Thus, there were actually nine different groups among BEEP participants, even though, by the third year, all were receiving services at the same time.

Once again, if there's no problem in evaluating the whole program, participants will simply include everyone. If that's not possible, there are a number of potential choices:

Evaluate your work with only one group, with the expectation that work with the others will be evaluated in the future. In this case, you'd probably want to choose the one for whom you consider the program most crucial. They might be at greater risk (of heart attack, of school failure, of homelessness, etc.) or might be experiencing the issue at a high level of intensity (daily shooting incidents in the neighborhood, high rates of teen pregnancy, massive unemployment).

Include a small number (2-4) of groups in your evaluation. You might want to choose groups with contrasting characteristics (different ages, for example, or addressed by different strategies). On the other hand, depending on the focus of your evaluation, you might want groups that are essentially similar, to see whether your work is consistent in its effects.

Choose a few participants from each group to focus your evaluation on.  While this won't give you a complete picture, it should give you enough information to tell where your program is accomplishing its goals and where it needs improvement. The differences in the ways participants in different groups respond to the program (assuming there are differences) can also give you ideas for ways to change what you're doing.

Participants from different populations and cultures

Cultural factors can have an enormous effect on participants' responses to a program. They can govern conceptions of social roles, family responsibilities, acceptable and unacceptable behavior, attitudes toward authority (and who constitutes authority), allowable topics of conversation, morality, the role of religion - the list goes on and on. In planning a program that involves members of different populations and cultures, you essentially have three choices:

• Plan your program and implement it in the same way for everyone. If the program involves groups - classes, support groups, etc. - participants' membership is determined not by population group but by when they sign up, what time of day they can attend, what they sign up for, or whatever other criteria make sense logistically.
• Plan your program to be as culturally sensitive as possible, and try to screen out anything that might be offensive to or difficult for any group. In this instance, you might be prepared to respond if participants from a particular population requested a group of their own.
• Divide participants by cultural group and plan different culturally sensitive approaches for each. Your overall approach might be the same for everyone, but the way you apply it might differ by culture.

In any of these instances, it would probably be important to understand how well your approach is working with members of the various populations.  If you can evaluate the whole program, make sure that you include enough members of each group so that you can compare results (and their opinions of the program) among them.  If your evaluation possibilities are limited, then your choices are similar to those for multiple groups of other kinds, and will depend on what exactly is most useful for you.

There are interactions between the choice of sites and the choice of participants here. You may be concerned about the effects of your program on a particular population, which may be largely concentrated at one site.  In that case, if you have limited resources, you may want to evaluate only that site, or that site and one other.

Regardless of other considerations, you may want to set some guidelines about whom you include in the evaluation. How long do people have to be in the program, for instance, before they're included? In other words, what constitutes participation? (This also sets a criterion for who should be counted as a drop-out: anyone who starts, but leaves before meeting the standard for participation.) What about those whose attendance is spotty - a few days here, a few days there, sometimes with weeks in between? Do they have to have attended a certain number of hours to be considered participants?

These issues can be more complex than they seem. People may start and drop out of a program numerous times, and then finally come back and complete it. Many others start programs numerous times, and never complete them. It's usually impossible to tell the difference until someone actually gets to the point of completion, whatever that means for the particular program.

In a reversal of the start-many-times-before-completing scenario, there can be a few people who stay in a program right up till the end and then drop out. This may have to do with the fear of having to cope with success and a change in self-image, or it may simply be a pattern the person has learned to follow, and will have to unlearn before being able to complete the program.

Should any or all of these people be included in or excluded from an evaluation, either before (because of their history in the program) or after the fact? That's a decision you'll have to make, based on what their inclusion or exclusion will tell you. Just be sure that your evaluation clearly describes the criteria that you decide to use for your participants.

If you're an outside evaluator or academic or other independent researcher

Up to this point, we've largely ignored the evaluation difficulties faced by evaluators not directly connected with the organization or institution running the program they're evaluating. If you've been hired or designated by the organization or a funder to evaluate the program, you have to establish trust, both with the organization and its staff and with participants, if you hope to get accurate information to work with. You also have to learn enough in a short period about the community, the organization, the program, and the participants to devise a good evaluation plan, and to analyze the data you and others gather.

If you're an independent researcher - a graduate student, an academic, a journalist - you face even greater obstacles. First, you have to find a place to conduct your research - a program to evaluate - that fits in with your research interests. Then, you have to convince the organization running that program to allow you to do the research.Once you've jumped that hurdle, you're still faced with all the same tasks as an outside evaluator: establishing trust, understanding the context, etc.

Let's look first at the process you as an independent researcher might follow in order to choose and gain access to a setting appropriate to your interests. Once you've gained that access, you've become an outside evaluator, so from that point on, the course of preparing for the evaluation will be the same for both.

Choose a setting

If you're an academic or student, you can probably find an appropriate program by asking colleagues, professors, and other researchers at your institution. If none of them knows of one offhand, someone can almost undoubtedly put you in touch with human service agencies and others who will. Other possible sources of information include the Internet, funders, professional associations, health and human service coalitions, and community organizations. Public funding information is often available on the web, in libraries, or in newspaper archives. The wider you spread your net, the more likely you are to find the program you're looking for.

The right program will obviously vary depending on your research interests, but some questions that will inform your choice include:

• Does the setting include people who are actually experiencing the problem that is of interest to you?
• Is the setting similar to others of this type? (If not, its program might not be useful to others dealing with the issue, even if it works well in its own context.)
• Does the setting provide support for the research? Will staff, participants, and others help with data gathering, be forthcoming about context questions, cooperate with you?
• Does the setting have the resources to maintain the program after your evaluation is done?
• Does the setting permit the changes in operation required by the research?  If the planning of the evaluation and choosing of questions point to doing things differently, can and will the program make the necessary changes?
• Is the setting accessible?
• Accessibility includes not only handicap accessibility, but whether a site is in a neighborhood that feels welcoming or safe to participants, whether it is easily reachable by public transportation or on foot from the areas from which participants are drawn, and whether it is in a building or institution that doesn't feel intimidating or strange (a university campus or building can seem as threatening as a fortress to someone who is insecure about his educational background, for example.) Accessibility can be the determining factor in whether participants consider a program, or whether they stay in it.
• Is the setting stable? Are the program and organization stable enough that you know they'll be able to support their work at the current level, at least until the evaluation is completed?

Once you've found an appropriate setting, you'll have to convince the organization to collaborate with you on an evaluation. The next three steps are directed toward that goal.

Learn as much as you can about the organization you've chosen

Just as you wouldn't go to a job interview without doing some research about the employer, you shouldn't try to gain the cooperation of an organization without knowing something about it - its mission, its goals, whom it serves, who the director and board members are, etc. If someone told you about the organization, she may have, or may know someone who has, much of the information you need. If the organization maintains a website, much of that information will be available there. If it's incorporated, the office of the Secretary of the state of incorporation and/or other state offices will have information about the officers (i.e., the Board of Directors) and other aspects of the organization. Funding agencies may also have information that's a matter of public record, including proposals.

Contact the appropriate person(s) and request an interview

Find out whom (by name as well as position) you should talk to about conducting a research project in the organization you've chosen.

Depending on the organization, this could be the board president, the executive director, or the program director (if the program you're interested in is only part of a larger organization).  In any case, it might be wise to involve the program director even if he's not the final decision-maker, since his cooperation will be crucial for the completion of your research.

• If you can, get a personal introduction. It's always best if you come recommended by someone familiar with the person you need to speak with.
• If you can't get a personal introduction, it's usually best to send a letter requesting a meeting and explaining why, and follow it up with a phone call.
• Before the meeting, send a proposal outlining what you want to do. This should be substantive enough to help the organization decide whether it wants to work with you, but not so specific that it doesn't allow for collaborative planning of the evaluation.

Plan and prepare for the initial meeting

There are several purposes for this meeting, besides the ultimate one of getting permission and support for your project (or at least an agreement to continue to discuss the possibility). They include:

• Establishing your credentials - the experience, educational background, and any other factors that equip you to conduct this evaluation. This might include references from colleagues, professors, or other organizations you've worked with.
• Learning more about the program and the organization
• Explaining what you want to do and why, what form the evaluation results are likely to take, what you'll do with them, who'll have access, etc. This explanation should also cover issues of confidentiality and permission of participants.
• Explaining what you need from the organization and/or program - participation of participants and staff, for instance, any logistical support, access to records, or access to program activities
• Explaining what you're offering in return - your services for a comprehensive formal evaluation, any stipends, equipment or materials, other support services, or whatever else you may have to offer
• Clarifying the organization's needs, and discussing how they fit with your own - and how both can be satisfied

Assuming that your presentation has been convincing, and you're now the program evaluator, the rest of the steps here apply to both independent researchers and outside evaluators.

Find out all you can about the context

This may play out differently for outside evaluators than it does for independent researchers, but it's equally important for both. It means finding out all you can about the community, the organization, the program, and the participants beforehand - the social structure of the community and where participants fit in it, the history of the issue in question, how the organization is viewed, relationships among groups and individuals, community politics, etc.

If you're an outside evaluator, you can pick the brains of program administrators, staff, and participants about the community, the organization, and the issue. Ask them to steer you to others - community leaders, officials, longtime residents, clergy, trusted members of particular groups - who can give you their perspectives as well. If possible, get to know the community physically: walk and/or drive around it, visit businesses, parks, restaurants, the library. Understanding how the issue plays out in the community, the nature of relationships among groups and individuals, and what life is like in the neighborhoods where participants live will help a great deal in analyzing the evaluation of the program.

If you're an independent researcher, learn as much about the context as you can before you contact the program. Websites (for the organization and/or the community) and libraries are two possible sources of information, as are community and organization literature and people who know the community.  Learning about the community, the organization, and the participants beforehand will both help you determine whether this program fits with your research and help you advocate for its cooperation with your project. Once you have that cooperation, you can follow the same path as an outside evaluator (since that's what you are) to learn as much about the context of the program as you can.

Establish trust with program administrators, staff, and participants

This can be the most difficult part of an evaluation for someone from outside the organization. There's no magic bullet or predictable timeline, but there are several things you can do:

• Be yourself.  Don't feel you have to act a certain way: deal with people in the program as you do with friends and acquaintances in other circumstances. People can tell when you're being false, and are unlikely to trust you if you are.
• Treat everyone with equal respect, as colleagues in a research project.
• Don't assume you know more than anyone else just because you're the professional.
• Share freely what you do know, but don't tie yourself to any one process or method, especially in response to an opposite stance from a key individual.
• Ask administrators, staff, and participants what they want from the evaluation, and discuss how the evaluation could provide it.
• Don't be afraid to say "I don't know, but I'll find out," and then do.
• Follow through on whatever you say you'll do. Don't promise anything you can't deliver on, and make deadlines reasonable, so you can meet them.

General tips for all evaluators

These steps apply to everyone, internal evaluators as well as external.

Aim for a participatory evaluation

We've discussed above the involvement of all stakeholders to the extent possible. Involving participants, program staff, and other stakeholders in participatory planning and research can often get you the most accurate data, and may give you entry to people and places you normally might not have. On the other hand, participatory planning and research, as we've explained, takes time and energy. If you have limited time, you may not be able to set up a fully participatory project.  You can, however, still consult with stakeholders, and involve them in ways that don't necessarily involve training or large amounts of your time. They can help you line up interviews with participants or other important informants, for instance, and/or act as informants themselves about community conditions and relationships.

At least the people in charge of the program, and probably those implementing it as well, will expect to be part of the planning of the evaluation. They are, after all, the ones who need to know whether their work is effective, and how to improve it. Involving participants as well, in roles ranging from informants about context to actual researchers, is likely to enrich the quantity and quality of the information you can obtain.

Plan the evaluation, in collaboration with stakeholders

That collaboration should be at the highest level of participation possible, given the nature of the program, the time available, and the capacity of those involved (if program participants are five-year-olds, they probably have relatively little to contribute to evaluation planning...but their parents might want to be involved.)

The actual planning involves ten different areas, each of which will be the subject of one of the remaining sections in this chapter:

• Information gathering and synthesis
• Designing an observational system
• Developing and testing a prototype intervention
• Selecting an appropriate experimental design
• Collecting and analyzing data
• Gathering and interpreting ethnographic information
• Collecting and using archival data
• Encouraging participation throughout the research
• Refining the intervention based on the evaluation
• Preparing the evaluation results for dissemination

Once the planning is done, it's time to get started on conducting the evaluation. And when you're finished - having analyzed the information and planned and made the changes that were needed - it's time to start the process again, so that you can determine whether those changes had the effects you intended.  Evaluation, like so much of community work, is a process that goes on as long as the work itself does.  It's absolutely essential to the continued improvement of your program.

In Summary

Choosing evaluation questions - the areas in your work you'll examine as part of your evaluation of your program - is key to defining exactly what it is you're trying to accomplish. For that reason, those questions should be chosen carefully as part of the planning process for the program itself, so that the questions can guide your work as well as your evaluation of it. The more that stakeholders can be involved in that choice and planning, the more likely you are to create a program that successfully meets its goals serving the community.

Choosing those questions well entails understanding the context of the program - the community, participants, the culture of any groups involved, the history of the issue and of the social structure of the community and the organization - and (if you're an outside evaluator without ties to the program) establishing trust with administrators, staff members, and participants. That trust will enable you to conduct a participatory evaluation that draws on the knowledge and talents of all stakeholders, and to plan an evaluation that fits the goals of the program and accurately analyzes its strengths and weaknesses. With that analysis in hand, you'll be able to make changes to improve the program. Then you're ready to start the whole process again, so you can evaluate the effects of the changes you've made.

What do we mean by choosing questions?

___Evaluation questions are the questions your evaluation is meant to answer about your work

___Evaluation questions help set the direction of the work, as well as assess its effectiveness

___Ideally, choosing evaluation questions is part of the planning of the overall program

Questions to ask yourself as you choose evaluation questions:

___What do you want to know?

___Why are you interested?

___Is the issue you’re addressing important to the community or to the society? 

___How does the issue relate to the field?

___Is the issue general, rather than specific to your population or community?

___Who might use the results of your evaluation? 

___Whose issue is it?

Why is it necessary to choose evaluation questions carefully?

___It helps you understand what effects different parts of your effort are having

___It makes you clearly define what it is you’re trying to do

___It shows you where you need to make changes

___It highlights unintended consequences

___It guides your future choices

___In participant evaluations, it involves stakeholders in setting the course of the program, thus making it more likely that it meets community needs

___It provides focus for the evaluation and the program

___It determines what needs to be recorded in order to gather data for evaluation

When should you choose questions and plan the evaluation?

___If possible, choosing questions and planning the evaluation should be an integral part of planning your program

___If your reality makes that impossible, choosing questions and planning the evaluation should take place as soon as possible after the program starts

Who should be involved in the process?

___To the extent possible, the process should involve all stakeholders, including program participants and beneficiaries

How do you choose questions?

___Describe the issue or problem you’re addressing

___Describe the importance of the problem

___Describe those who contribute to the problem

___Assess the importance and feasibility of changing those behaviors

___Describe the change objective

___Make sure that the expected changes would constitute a solution or substantial contribution to the problem

How do you plan the evaluation?

___Take into account the issues raised by multiple or very different settings

___Take into account the issues raised by participant groups that differ in culture, ability to complete the program, geographical location, and other factors

For outside evaluators, specifically:

___Choose a setting

___Learn as much as you can about the organization you’ve chosen

___Contact the appropriate person(s) and request an interview

___Plan and prepare for the initial meeting

For all evaluators:

___Find out all you can about the context

___Establish trust with program administrators, staff, and participants

___Aim for a participatory evaluation

___Plan the evaluation in collaboration with stakeholders

Consider all the elements of an evaluation in your planning:

___Information gathering and synthesis

___Designing an observational system

___Developing and testing a prototype intervention

___Selecting an appropriate experimental design

___Collecting and analyzing data

___Gathering and interpreting ethnographic information

___Collecting and using archival data

___Encouraging participation throughout the research

___Refining the intervention based on the evaluation

___Preparing the evaluation results for dissemination

Example 1: Using Miro Board for Collaborative Development of Evaluation Goals and Questions

 

Noah Goodman shares the process he used to prepare for and conduct a kick-off meeting to develop evaluation goals and questions for the Right Question Institute.

Integrating Technology in Evaluation TIG Week: Using Miro Boards to Engage Clients in Collaborative Development of Evaluation Goals and Questions by Noah Goodman

Suppose you wanted to design a house that used very little energy, took few resources to build and maintain, and was affordable for most families. You might have some original ideas about how this could be done, but you’d want to find out what ideas others had as well. You’d probably read about earth-bermed houses (houses that are built into a hillside or earth mound), solar panels or windmills for producing electricity, efficient insulating windows, waste-water recycling, and non-toxic building materials that reuse waste wood and metal. You’d talk to people who built or owned energy-efficient houses, to hear about the realities of living green. You’d learn about the barriers to some environmentally-friendly strategies, as well as ways to get around those barriers. There’s a huge amount of information out there, and it would make sense to gather as much of it as possible, so that you could put together the information, incorporate appropriate elements into your design, and get new ideas based on what’s already been done.

The same is true if you’re designing an intervention or program to deal with a community health or other issue, or an evaluation of that program. Others have also undoubtedly tried to address that issue, some with success and some without. Knowing what they did, how they did it, and what the results were can help you decide how to design your effort. You might be able to find a method here, and a technique elsewhere that all fit together into exactly the program that will suit the people and conditions in your community. Or you might realize that something you’d intended to do simply hasn’t worked in a number of other instances, and so wouldn’t be likely to work for you, either.

Gathering and using others’ ideas doesn’t mean that you can’t use your own or come up with something new. New ideas tend to come out of what others have attempted. Most artists start out imitating others before they develop their own styles. Einstein didn’t just chance on relativity; he was familiar with it because others had worked on it. You can usually innovate more effectively if you know what’s been tried.

This section looks at gathering all the information you can about your community issue and about attempts to address it, and putting that information together to design an evaluation to address your questions. Although this chapter is about evaluation, much of the material in these sections applies to planning the intervention (or program) and the evaluation: the two really can’t be separated.

An evaluation is a research project: we are trying to discover what works and under what conditions.  The steps for designing and using an evaluation – the subject of this chapter – are essentially the same as those for designing the program you’re evaluating.  The elements that you borrow from others’ successful efforts, and those that you create yourself, will give you an intervention and related evaluation questions. Although this section talks about program design, it also applies to the design of the evaluation.

What do we mean by information gathering and synthesis?

Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it. The more information you have about the issue itself and the ways it has been approached, the more likely you are to be able to devise an effective program or intervention of your own.

There are obviously many sources of information, and they vary depending on what you’re looking for. In general, you can consult existing sources or look at “natural examples,” examples of actual programs and interventions that have addressed the issue. We’ll touch on where to find both here, and then go into more detail about them later in the section.

• Existing sources. This term refers to published material of various kinds that might shed light either on the issue or on attempts to deal with it.  These can be conveniently divided into scholarly publications, aimed primarily at researchers and the academic community; mass-market sources, written in a popular style and aimed at the general public; and statistical and demographic information published by various research organizations and government agencies.
• Natural examples. These are programs or interventions developed and tried in communities that have addressed your issue. Studying them can tell you what worked for them and what didn’t, and why. By giving you insight into how issues play out in your or other communities, they can provide nuts-and-bolts ideas about how to (or how not to) conduct a successful program or intervention. For the most part, information sources here are the people who are involved in efforts to address issues similar to yours, or those who can steer you to them. Additionally, there are a number of natural examples (such as single case studies) that have been written about descriptively in the literature of community psychology or public health that may be relevant to your work.

Synthesis is from the Greek; it means putting together. Its English meaning is the same: the putting together of something out of two or more different sources. Synthetic fabrics, for instance, are called that because they’re constructed from a number of different chemical building blocks.

In this section, we’re talking about ideas. Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something new that meets the needs of the community and population you’re working with.

Synthesizing in this way requires identifying the functional elements of each idea or program that you’ve looked at that seems to hold lessons for your work. Functional elements are the core components of each program – the methods, framework, activities, techniques, and other aspects – that make up the specific program you’re examining.  Once you’ve separated these parts out, you can put those that meet your needs together with what you’ve learned about the issue and your own ideas to build a program that speaks specifically to your situation.

As we’ve mentioned, the activities of information gathering and synthesis are needed both to create the original program and to develop an evaluation of it that will help you maintain and improve it.  The two really start in the same place, with what you think will address the issue – what shape the program or intervention should take, with whom it should be applied, and what behaviors or conditions it aims to change. This also informs what its short- and long-term goals should be, and by what means you’ll try to achieve those goals.  Once these are determined, they in turn determine your evaluation questions. You can’t construct an evaluation without knowing exactly what you’re trying to evaluate.

Why gather and synthesize information?

If you’re in the process of starting a program to address a community issue, such as violence or early childhood education, you probably know quite a bit about that issue already.  You’ve dealt with it, perhaps, in a variety of ways, and you have some pretty good ideas about what kind of program would work.  Why take the time and trouble, for you and for others engaged in a participatory planning effort, to read a lot of material written by others and to track down people who’ve run programs? If you’re inclined to think this way, there are a lot of good reasons why you should think again. Gathering information beforehand and putting together what you’ve learned could be the most important things you do to make your program effective.  Here’s why:

• It will help you avoid reinventing the wheel. A lot of different organizations have likely approached this issue before you.  Some might have been successful and some might not have, but all of them have probably learned something that would be useful to you in the process. You don’t have to make the same mistakes someone else did if you know about them, and you don’t have to make up something from scratch that may or may not work, when you have a model that has worked.

It’s certainly not a bad thing if you have some of the same good ideas that others have had, but it helps to know that they are good ideas. And there’s a chance that you might have some of the same bad ideas others have had, in which case it helps even more to know that they’re bad ideas.  It will save you a huge amount of trouble, and perhaps be the difference between creating a program that does its job well and one that fails miserably and disappears. Square wheels don’t roll – someone could have told you that.

• It will help you to gain a deep understanding of the issue so that you can address it properly. The first step in figuring out how to deal with an issue is to know what you’re dealing with. The better you understand it – its causes, how it occurs, how people react when they’re affected by it, what its consequences are for individuals and the community, and who can influence it – the more likely it is that you’ll be able to determine how to approach it.
• You need all the tools possible to create the best program you can. Foremost among the tools you need to plan and implement a program or intervention are information, information, and information. Just as with the issue itself, the more you know about what works for whom, how to make things happen, and how to establish or eliminate certain conditions, the more likely that you’ll be able to plan a successful program that addresses all aspects of the issue and leaves nothing to chance. Various kinds of professional and interpersonal skills may help you implement a program, but if what you’re implementing isn’t effective, it doesn’t matter how skillfully you carry it out.
• It’s likely that most solutions aren’t one size fits all. The more information you gather, the greater the variety of approaches, methods, and frameworks you’ll have to choose from. Putting together the right combination will help you to successfully address the particular needs of your community and population.
• It can help you to be culturally sensitive. Not only can you learn more about the culture(s) of the people you’re working with, but you can probably find a number of approaches that have worked with the cultural group you hope will benefit. Perhaps even more important, you can learn to avoid costly mistakes that may take a lot of time and effort – or be impossible – to repair.
• Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program. As we discussed at the beginning of this section, new ideas seldom spring from nowhere.  They’re stimulated by your own experience and the ideas and experience – both good and bad, positive and negative – of others. Look to the experience of other fields, communities, and countries. The more different ideas you’re exposed to, and the more ways you can put them together, the greater chance there is that you’ll come up with something new that’s more effective than what’s gone before.

When should you gather and synthesize information?

Information gathering and synthesis is crucial to the success of the program and to the relevance and effectiveness of the evaluation. It should start at the beginning of any effort, and contribute to the initial planning. It should also go on throughout the life of the program, so that you can continue to adjust by adding or changing program elements to enhance outcomes, and to generate new ideas.

Major adjustments should generally come at the end of an evaluation cycle, when you have solid information about what worked and what didn’t. That doesn’t mean that you can’t make smaller adjustments in the course of the program to improve results along the way.

There’s a tension here between continually changing a program to make it better and obtaining accurate evaluation results. If you change a method or activity in midstream, your evaluation will not be able to give you a clear assessment of its effectiveness.

How much changing you do in the course of a program depends on your intent.  If your first responsibility is to find out what works best, so you can pass it on, then it’s important not to make changes until an evaluation has been completed.  If your primary responsibility is to the current participants in the program, then you should make whatever changes are necessary whenever they’re necessary to ensure the best outcome for them.

There can be ethical issues involved here.  In medical experiments with new therapies or drugs, for example, some participants are given the new treatment and others aren’t (all participants consent to this arrangement, and to not knowing which group they’ll be assigned to.)  If the new treatment proves to be harmful, there is an ethical obligation for the researchers to stop administering it.  If, on the other hand, it quickly proves remarkably effective, researchers usually feel ethically bound to extend it to others in the study as soon as they can prove its positive effects.  Not all programs necessarily pose ethical problems that are as clear-cut as those encountered in medical studies, but ethical issues should always be considered.

Who should gather and synthesize information?

The assumption throughout this chapter is that the whole process – planning, design, implementation, and evaluation – involves multiple stakeholders.

Typical stakeholders in a community program or intervention might include:

• Program participants or beneficiaries
• Program staff and administrators
• Others affected by the program – police, medical staff, teachers, etc.
• Academics or other researchers
• Local officials
• Community activists

Information gathering

In a participatory process, information gathering can be enhanced by a division of labor determined by the skills and experience of the participants. If there are academics or other professional researchers involved, it would probably make the most sense for them – or others with research experience – to review the evaluation literature. Members of the affected population might be the best ones to collect information about the history of the issue in the community, and about how it currently affects people. Program directors and staff would probably have the best contacts in the field, and thus the best chance to find information about other similar programs. Those with Internet access and computer experience might be the logical on-line searchers, or might act as technical support for others to help them find what they’re looking for. Those with knowledge in the law and legislation might be the ones to examine policies.

There’s also the possibility that training could be provided to the whole group, or to various individuals to allow them to pursue various lines of inquiry. There’s no reason, for instance, that people without research experience couldn’t learn to understand and interpret demographic information or contact programs in other places. (There are some limitations here: levels of related education, materials or computers, and/or inability to connect with other people might all figure in to what kind of research it makes sense to ask others to do.)

Synthesis

It is especially important that all participants in the process be involved in putting together the information. Training new participants to synthesize information will pay dividends in the end, because they may be able to see things in the information that aren’t obvious to experienced researchers. They may know things about the community that shed light on which elements of other programs might be appropriate and which might not.

In any case, information gathering and synthesis, like any other part of the process, should reflect the needs, interests, and abilities of all stakeholders.

How do you gather and synthesize information?

There are a number of steps to gathering and putting together the information you need. Most of these can be group activities, part of the participatory process. The actual information gathering can be parceled out to specific individuals or sub-groups.

Decide what you need to know

Not surprisingly, the first step in gathering information is determining what information to gather. There are a number of areas to explore:

• Details about the issue. These might include its immediate and root causes; its general effects on individuals and communities; its consequences; its development through different stages; its history; and the history of attempts to address it.
• How the issue has been dealt with elsewhere. Best practices or approaches for which there is an evidence base; other approaches that have been at least partially effective; and what hasn’t worked, which may give you at least as much important information as what has.
• People who can help. This category encompasses experts in the field and people or organizations that have run or been involved in successful attempts to address the issue.
• Who is affected locally, and how. This really comprises two questions: a) What population groups – geographical, ethnic, cultural, racial, class, etc. – are particularly affected by this issue?  and b) What other groups are affected, but less visibly? These might include those who work with the first group(s) in the community (teachers, for example, or social workers), those who depend on them, and those on whom they depend.
• The importance of the issue to the community. Again, this implies a double question:
  • How important does the community perceive the issue to be? and
  • How much and in what ways does the issue actually affect the community as a whole?
• Community needs related to the issue. What has to be added to or removed from the community in order to improve the situation? What kinds of approaches will the community respond to or reject?
• Other context information. Community history, relationships among groups and individuals that might be relevant to your work, community culture, etc.
• Who, if anyone, has some influence or control over changing the situation. Public officials and other policymakers are often in this position. Business leaders, landlords, government enforcement agencies, schools, employers, hospitals and health personnel, and members of the affected group itself might also be in the position to change the situation (by learning new skills or changing practices).

Determine your likely information sources

As mentioned above, these encompass existing (i.e., published) sources and natural (i.e., experiential) examples. Published sources can be divided into scholarly, mass-market, and statistical, each of which can provide different information and a different perspective on the issue and attempts to address it.  Depending on what you decide you’re looking for, you might use all, or any combination, of these sources.

The single largest storehouse of information available is the Internet. Many scholarly articles are published online and accessible – often free, sometimes for a fee – to anyone who’s interested. Virtually all U.S. laws and regulations at every level of government are easily found, most on several websites. General knowledge on just about anything is widely available, as are lists of best practices and successful organizations and the websites of those organizations. Census data and other similar statistical information are also on view.  Add to these the information provided by such all-encompassing sources as Wikipedia (recently, for all its quirks, found to be just about as accurate across its million-plus entries as the Encyclopedia Britannica), and you have a nearly-bottomless well of fact and opinion to draw from.

As always, you have to be cautious: most of cyberspace is unedited, and the quality of information varies.  If you stick to reasonably reliable sites, you’re likely to find almost whatever you need, or at least directions to it.

Existing sources

Scholarly sources might include:

• Academic and some professional journals
• Books written for the academic market
• Doctoral dissertations - these are accessible to researchers through university libraries and some Internet sources
• Papers and reports delivered at academic and professional conferences - these are often available online, either on the authors’ websites or in e-published conference proceedings
• Occasional articles in respected mass-market scientific magazines, such as Nature or Scientific American
• Newspaper archives
• Direct contact with academics and other researchers who’ve done work on the issue you’re interested in, or who have conducted studies of attempts to deal with it
• Internet listservs and news groups relating to the issue or the field in question

Mass-market sources of information:

• Widely available books, often marketed as “self-help” or “life-changing,” to the public at large
• Articles in popular magazines, both those devoted to science or behavior and those of general interest
• Newspaper stories, often in Sunday magazine sections

Where to find statistical and demographic information:

• Census data - available on the web and at many libraries
• Community reports, such as community report cards, self-studies, and needs assessments, all of which should be obtainable through the appropriate municipal offices, and sometimes on the web as well
• Organizational and agency data, usually a matter of public record if the agency is public or publicly funded

In addition to these sources, the broadcast media often present stories about critical issues or about successful efforts to address them. In most cases, such stories only skim the surface, since they have to fit into short time slots (public broadcasting, on both radio and TV, breaks this mold more than other media outlets).  They can, however, serve as introductions to further research, raising the importance of one or more aspects of an issue, or providing information about effective programs that you can then contact.

Natural examples

Some of the more likely sources of natural examples:

• Program directors
• Friends or colleagues in the field
• Funders (particularly public agencies, because their transactions, including whom they fund and why, are a matter of public record)
• Leaders and members of community coalitions or partnerships
• Officials who coordinate community-wide efforts
• Members of the population most directly affected by the issue at hand
• Current or former participants in or beneficiaries of effective programs
• People who work in collaboration with programs – police, medical staff, teachers, etc.
• Key informants in the community
• Experts – some of them the same academics and other researchers referenced under scholarly sources – who have experience with your issue and efforts to address it
• Your own experience in the community

Don’t be afraid to range far and wide in your search for successful models or new ideas.  Step outside your own field and your own region, and see what’s been done elsewhere.  A model from social work or urban design might work in public health, or vice-versa.  There’s enough overlap among fields that deal with human health and development that you can often find exactly what you need in seemingly odd places.

Devise a plan for collecting information

There are a number of considerations here:

• Who will gather what information? As we’ve discussed, the ideal group is multi-sectoral and diverse in backgrounds and skills.  Information gathering should be assigned according to participants’ skills, interests, and contacts in the community. We’ve suggested, for instance, that scholarly sources might be mined by academics or other experienced researchers, while members of the affected population might be more successful in approaching key informants in the community. This doesn’t mean, however, that in a given group, these and other apparently logical roles couldn’t or shouldn’t be varied, depending on the individuals involved.
• How will the information be gathered? Another issue is just how the information will be gathered. Finding and reading written material is relatively straightforward: it’s in the library or on the web, and you can read and take notes on the relevant parts of it. Getting information directly from other people, however, can be more complicated. Will you engage in formal or informal interviews? In observation? Will you conduct surveys or public meetings? How will you contact people you don’t know – by letter, by phone, through mutual acquaintances? Your information-gathering methods will be determined by how much time you have, exactly what information you need, the depth of the information you need, and the abilities of the participants.
• What adjustments will be made for particular gaps in experience or skills?  People who don’t read, write, and/or speak the language proficiently may have to devise imaginative ways of recording information. Experienced researchers may have to translate scholarly writing for just about everyone in the group who isn’t an academic. In many cases, most or all of the group may need orientation or training before information gathering can begin. You’ll need to work out what the needs are as a group, and devise ways to meet them.
• What’s the timeline for information gathering? While information gathering should continue throughout the life of a project, the initial phase should have a time limit, so that action isn’t delayed for too long. The time limit depends on your time constraints, the seriousness and intensity of the issue, the community’s perception of urgency, and whether there are external time restrictions (student interns who are only available until the end of the summer, for instance.) Having a clear deadline will focus the group’s activities, and boost its efficiency.

Collect information

When your plan is completed, it’s time to put it into practice. You’ll have to conduct any trainings that are necessary, and make sure that all the relevant tasks are assigned appropriately. You may also want to set up regular meetings throughout the information-gathering process, in order to give the group the chance to review progress, make suggestions, and report on what they’ve been finding.  In addition to providing support for those new to research, these meetings, by providing a preview of the results of the process, will save everyone having to digest an overwhelming amount of information all at once.

Synthesize: Take it all apart

The process of synthesis involves breaking the information down into its component parts, sifting through those parts to see which fit together best for your situation, and then integrating them into an approach that is likely to work in your community.

There are usually three major areas to be considered:

• What’s known about the issue itself. What personal and environmental factors contribute to the problem? What are its root causes? Do you have the resources to address them, or are they beyond your scope (e.g., global economic forces or climate change)? Does the issue have a number of different effects, and if so, what are they?  What are the likely consequences for the community as a whole if the issue is not resolved?  (An environmental health risk can not only kill or sicken individuals, but might also affect business productivity, insurance availability and rates, hospital costs, the housing market, or even – as in the case of the Love Canal neighborhood in Niagara Falls, NY – the existence of a neighborhood or community itself.)
• The community context of the issue. What are the specific local effects of the issue. Exactly who is affected?  Exactly how are they affected? What are the consequences for those individuals? For their families, friends, neighbors, and others they have dealings with? For the community as a whole?  What has been the community’s experience with this issue in the past?  How, if at all, has it been addressed?  What local conditions would change if the issue were addressed, and how would they change?  Are there underlying conditions that have to change before the issue can be addressed?  Whose attitudes and/or behaviors need to change to have an effect on the issue (for example, among policy makers, those affected, or specific officials)?
• Successful and unsuccessful attempts to address the issue. These may have been gleaned both from the literature on best practices, and directly or at second hand from those involved in them. Here, it’s important to separate out the elements of various approaches. What specific procedures – methods and intervention components – were used? What kinds of training – feedback, role play, modeling, etc. – were provided to participants?  Was information provided to participants about when, why, and how to act? Were there positive or negative consequences that helped to establish or maintain change (or its opposite)?  Were environmental barriers, policies, or regulations put in place or removed?  What was the overall philosophy behind the approach?  What aspects of the issue did it address? What kind(s) of community was it tried in?  What population groups (in terms of culture, age, social class, etc.) were involved?  Who was the approach to benefit?  What were the specific results in the short term?  In the long term?  What makes a particular program, policy, or practice successful or unsuccessful? What events, if any, were critical, to success (or failure)?  What conditions – organizational features, participant characteristics, broader environmental factors – were critical?  Is there a model successful program? Is there a model unsuccessful program?

The existence of a model unsuccessful program doesn’t indicate that if you do the opposite of everything that program did, you’ll be successful.  Even if it failed spectacularly, much of the program may have been potentially effective, but one or two elements – the way participants were approached, recruited, or treated, a particular method – negated what could have worked.  By the same token, most elements of the program may have been fine, but its basic premise might have been mistaken or ineffective – “Just say no” as a way of preventing AIDS among teens, for instance.  It’s important to try to figure out why the program was unsuccessful.  A true model unsuccessful program is one that did everything wrong, but those are few and far between.

Lisbeth Schorr (Common Purpose) makes a useful distinction between “what works” and conditions under which what works actually works.  Sometimes the presence of a charismatic leader or champion motivates staff and/or participants to succeed. When the leader or original staff members leave, some such programs collapse, while others are able to renew themselves by careful hiring and a faithful implementation of what made it work.

In looking for programs to draw from, you need to understand the intervention components and elements that make those programs work.  Also, try to understand the conditions that allow an intervention to be successful.

Synthesize: Put it back together

Analyze the elements you’ve found to determine which of them would be appropriate for the situation and group you’re working with.

• What has been used specifically with your population in your circumstances?  Have the successful programs you’ve looked at been context-specific (i.e., intended for their specific communities and populations)? Can they be adapted to your context if they weren’t intended for it?
• What can be adapted, if it wasn’t originally aimed at your population?  (Techniques used with children or adolescents that could be modified for use with adults, for instance, or vice-versa.)
• What’s missing?  What aspects of the issue in your community are not addressed by what you’ve found?  Are they important enough that they need to be addressed?
• Does what you’ve found confirm or contradict what you thought you already knew?
• Are there factors in your particular situation that make the issue substantially different for you and your participants than for any other programs or approaches you’ve found out about?  How will you deal with that?
• What does your information tell you about the possibility of successfully addressing the issue’s root causes (e.g., income inequality, social exclusion, lack of power)?
• In general, did most or all successful programs direct their change efforts at the same group of people (policy makers, for example), or was there a variety?  If the latter, what do you think is most likely to work in your community?
• Perhaps most important, what’s your definition of success, and which of the programs you learned about came closest to achieving?  What components and elements of those programs addressed what’s needed in your community?

Answering these questions will give you a good sense of which components of other programs may work for you, and should also fit with what you already know to either give you ideas for new elements that you can add, or confirm (or warn you away from) ideas for new elements that you had already.

We don’t want to imply that simply taking a lot of different program components and playing mix-and-match will provide you with an effective way to address a community issue. You have to start with a clear framework informed by your vision and mission, and put together a program that’s coherent and makes sense.  All the elements have to fit; if they fit well enough, you’ll end up with a whole that’s greater than the sum of its parts. If the elements don’t fit together, or aren’t part of a program with a well-defined framework, the chances are you’ll end up with a mess.

Keep at it

Information gathering and knowledge synthesis should continue throughout the course of the program. While you may wait until the results of an initial evaluation to change something, you should always be looking for improvements and better approaches. No program or effort is perfect: everything can be improved. As long as you keep trying to learn more and grow in your understanding of your work, it will continue to get better. If you become complacent (i.e., you feel you know what you’re doing and can relax), your program may start to lose its effectiveness.

In Summary

Gathering the information that already exists about your issue and attempts to address it is one of the most important aspects of planning a program or evaluation. By putting together what’s known about the issue and the history of the successes and failures of various approaches to it, you can build a program structure that includes your own innovations and elements that have worked for others in similar situations. This synthesis also allows you to avoid ineffective approaches and to incorporate ideas and methods that have been particularly appropriate, culturally or otherwise, to the population and community you’re working with.

Information gathering and synthesis should continue throughout the life of the program. The more information you have, and the more carefully you put it together, the better your chances of implementing a successful program.

What do we mean by information gathering and synthesis?

___Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it

___You can gather information using both existing sources and natural examples

­­___Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something that meets the needs of the community and population you’re working with

___Synthesis involves extracting the functional elements of both the analysis of the issue and approaches to it

___Functional elements are those that are indispensable either to understanding the issue, or to implementing a particular program

Why gather and synthesize information?

___It will help you avoid reinventing the wheel

___It will help you to gain a deep understanding of the issue so that you can address it properly

___You need all the tools possible to create the best program you can

___It’s likely that most solutions aren’t one size fits all

___It can help ensure your program is culturally sensitive

___Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program

When should you gather and synthesize information?

___Information gathering and synthesis should continue throughout the life of the program

Who should gather and synthesize information?

___Information gathering and synthesis is often most effectively conducted by a multi-sectoral participatory group including all stakeholders in the issue

How do you gather and synthesize information?

___Decide what you need to know about the issue itself, successful and unsuccessful attempts to address it in various circumstances, and the local context

___Determine your likely sources for the various types of information you’re seeking

• Existing sources include scholarly, mass-market, and statistical/demographic published information
• Natural sources include some published information about programs, but can best be obtained by direct contact with those involved in planning, implementing, or participating in programs relevant to your issue
• It’s important to pay attention to both successful and unsuccessful attempts to address the issue, and to step outside your own field in search of solutions that work

___Devise a plan for gathering information

• Decide who will gather what information
• Decide how information will be gathered
• Decide what adjustments will be made for gaps in experience or skills
• Set a timeline for the initial information gathering

___Collect information

___Begin synthesis by taking it all apart – extract the functional elements of what you’ve learned

___Complete synthesis by putting the relevant pieces back together as a coherent program that speaks to your community’s needs

___Keep at it by continuing to gather and synthesize information throughout the life of the program

• What do we mean by an observational system?

• Why design an observational system?

• When should you design an observational system?

• Who should design an observational system?

• How do you design an observational system?

The local community health center was starting a program to encourage regular physical activity among people with high blood pressure. The program had one simple objective: to engage participants in 45 minutes of regular, moderate aerobic exercise at least four times a week over the course of six months.  The hope was that this regimen would lower participants’ blood pressure, and lead to weight loss and an overall sense of greater well-being. A related aim was that participants would continue the exercise on their own after the program ended.

The Center had gathered a group of 50 people who were willing to take part. After physical checkups, all the participants attended workshops on diet, the mechanics and dangers of high blood pressure, and how to start and maintain injury-free regular exercise. They also received counseling about the kinds of exercise they might undertake – walking, bicycling, swimming, etc. – and about ways to make exercising pleasant. To help participants to integrate exercise into their lives, the center decided to ask them to exercise at their own convenience, using whatever activities they chose, as long as they maintained the 45-minute, four-day-a-week pattern. They were also asked to keep journals of the frequency and nature of their exercise, and to meet, in groups of five, with Health Center counselors once a month for checks on blood pressure, weight, and progress, as well as support, encouragement, and advice.

The center then had to decide how to evaluate the program. The performance goal of the program – what participants were actually supposed to do – was maintaining the exercise schedule over the six-month period. Since each participant was tending to that individually, it would be hard to actually watch them all exercise whenever they did so. But the center needed to know whether or not they had. The other program goals – lower blood pressure, good weight loss – also had to be observed somehow.  How could the center design a system to find out whether the behavior of physical activity was occurring and whether this resulted in lower blood pressure and weight loss?

Once you’ve determined your evaluation questions and gathered information about what to look for, you have to find a way to look for it.  That’s what observation systems are all about. Like the center in the example, you’ll need to find ways to observe the behavior, the conditions, and the changes – or lack of changes – in them that will answer your evaluation questions. This section is about setting up observation systems to do just that.

What do we mean by designing an observational system?

An observational system is the way you get information about your program – what it and its participants and implementers are actually doing, and what seems to be occurring as a result.  “Observation” here may mean actual observation – watching people, conditions, activity, or results to see what happens – but it may also refer to less direct ways of monitoring a program’s operation and outcomes.  Its varieties include monitoring the behavior of individuals and groups to see the results at different levels.  Some methods of observation that might prove useful in different evaluation situations:

• Direct observation. This is the purest and most verifiable form – watching people or observing conditions or situations firsthand. If you’re involved in an effort to increase the use and neighborhood sense of ownership of a public park, for instance, you might directly observe how much and how people use the park by visiting and observing on different days, in different types of weather, and under different circumstances over a substantial period of time. Direct observers may be “invisible,” as an observer of park activity would probably be, or they may be staff members who work with participants, recording what happens.  In either case, they are taking measures as outside observers, not as participants themselves.
• Participant observation. A participant observer becomes part of the action, and observes as an insider. In the case of the park, a participant observer might be a neighborhood resident directly involved in the effort, or might be someone who becomes part of the life of the park for the purposes of observation. He might jog there daily, or join a weekly volleyball game and get to know others who use the park on a regular basis. His own notes about what is observed in the park might also become part of his recording.
• Self-reports. Some of what you’re trying to achieve may simply not be visible at all, at least not to you. Changes in what people do in private, such as their use of contraceptives, may not be (or should not be) observed directly by an outsider. Similarly, when the goal is to affect changes in the behavior of large numbers of people, such as to promote healthy eating in the community, it will not be feasible to directly observe this for everyone.  In such situations, we ask people to report on their own behavior Thus, an observational system may include interviews, journals, surveys, or other means of first-person reporting. Since such reporting may be subject to bias, we usually try to also use other forms of evidence (e.g., observing weight loss as a product of the behaviors of health nutrition and physical activity).
• Second-hand reports. An observational system may include or depend on the reports of others who have direct experience with the people or conditions you’re concerned with. Teachers, probation officers, park rangers, public health nurses, social workers – even bartenders or hairdressers – might be valuable sources of second-hand information. These reports, like self-reports, may be gathered by interviews, journals, surveys, checklists, and the like.
• Electronic or mechanical observation. The observer in this case isn’t a person (although ultimately people would review its information), but an automatically-operated or always-on camera, audio recorder, heart monitor, pedometer, GPS (global positioning system) tracker, or other piece of equipment.  A camera operated by a tripwire is often used, for example, to study the density of an animal population in a particular area, or along a particular path.
• Tests of various kinds. Depending on what you’re measuring, this category could cover everything from pencil-and-paper tests of academic learning to hands-on skills tests to blood tests and the like. They might also include tests of new program methods and procedures to see if they work before putting them into practice.
• Public and other records. Police reports, census data, employment statistics, public health information – all of these and more could give you information on community-level indicators that will help you determine the outcomes of your work.
• Products or results of behavior. Sometimes it is more practical to observe the product or result of a behavior, rather than the behavior itself. For instance, if interested in environmental pollution, we might observe the amount of debris or toxins on the ground or in the water, rather than the behavior of illegal dumping of toxins or materials. Similarly, an initiative interesting in preventing childhood obesity might use school records of height and weight to measure obesity – in addition to direct observations of school lunches and what youth report on eating surveys.

In addition to specifying what kinds of observation you’ll use, the design of an observational system should also cover when, where, how often, by whom, and under what circumstances observations will take place, as well as just what will be looked at. All of these depend on what you’re observing and what information you hope to gain from your evaluation (back to those evaluation questions again).

Among other considerations, will you look at the process of your effort or program – the steps you took in setting it up, and whether they were faithful to what you intended?  Will you look at what you actually did – the number of participants you had, the methods you used, the time everything took, how long participants stayed, etc.? Are you interested in which parts of what you did were successful and which were not?  And what do you want to know about outcomes – the results of your program?

Designing an observational system entails thinking carefully about what you need to know, and creating a system that is most likely to get you that information as accurately and easily as possible. We’ll discuss the design process in detail, including the issues mentioned in the last two paragraphs, in the “how-to” part of this section.

Why design an observational system?

If you’re serious about evaluation, there are a number of reasons for designing a good observational system:

• It can help you get reliable information. Designing a system that standardizes the methods, times, and other aspects of the observation will mean that the information you get from different observers and places is likely to be accurate and consistent, and therefore more useful to an overall evaluation.
• It can help you find out exactly what you need to know, without wasted effort. You can design a system that examines what you’re interested in, and ignores what you’re not. That means that you don’t have to sort out unnecessary data, and that you’ll have the right means of collecting the data you need to address your evaluation questions.
• It can ensure that observations are made. A consistent system that’s designed and accepted by those who will do the observing, whatever form it takes, makes it far more likely that observations will be made when, where, and how they’re supposed to.
• It can make it easier to analyze your data. A consistent, rational system of observation can give you good information for scientific analysis, whether that analysis is quantitative (based on numbers and statistics) or qualitative (based on narrative and interpretation of the meaning of behavior and events.)
• It can help you avoid haphazard evaluation. A well-designed observational system will allow you to collect information systematically, and not leave you with a mass of disconnected data that are not necessarily related to what you want to know.
• It will make it easier to justify your findings. The more accurate your information, the more reliable the conclusions that can be drawn from it. If your observational system is designed and implemented well, it’s much easier to argue that your information is reliable and accurate, and a good base for the conclusions you reach.
• It can help you gain credibility with funders and policymakers. The people who control funds and policy are particularly concerned with accountability. If you can present them with a useful evaluation based on data collected through a well-designed and reliable observational system, they’ll be more inclined to treat you as a knowledgeable voice in the field.
• It can let you pass on your practices with confidence. A well-designed observational system makes it possible to feel that your evaluation results tell the truth. If those results show that your program is highly effective, you can pass on what you do as a best practice to colleagues in the field and others, without worrying that you may be urging them to use methods or assumptions that might not work very well.
• It can give you the best information possible about what’s working in your program, and what you need to adjust.

When should you design an observational system?

As we’ve discussed, an observational system refers not only to direct observation, but to any method of examining and recording the process, activities, and outcomes of your program. An observational system is intrinsic to your evaluation, since that is what will tell you what actually went on. Therefore, the ideal is to design that system before you actually start implementing the program, so that you can monitor the program throughout its existence.

That’s the ideal. The reality for many community workers – especially those who work in small, community-based organizations – is that evaluation begins whenever the time, energy, and resources are available, which is often months or years after the program has started. Whenever it begins, the observational system should be designed to fit the evaluation questions you’re asking. Because the observations must be consistent and reliable it's well worth taking the time to make sure to design an effective evaluation system.

It’s best if you can observe through a whole program cycle, from beginning to end. Some programs don’t have a cycle, and the observation may focus on the behavior or results of individual participants rather than the program as a whole. In these situations, evaluation may begin as new participants begin the program and are observed from the beginning.

If you’ve been recording events, keeping journals, etc., before you start evaluating, you may have information that you can incorporate into the results of your observation. If your design calls for specific firsthand observations, their definition may be precise enough that similar observations recorded in journals kept by staff may not meet the criteria to be included.  If staff journals or records are part of the system you’re putting in place, however, you may be able to use all or most of the information you have (and, indeed, you could design the observational system so you can.)

The real danger here is that you’ve missed something important already by the time your observational system is operational.  It may be that the early part of the program is crucial, at least for some participants or for some changes, and you’ll be starting to observe after those changes have been made.  Start your observations early so that your system can pick it up.

Who should design an observational system?

We’ve stated many times the Community Tool Box bias toward participatory process, and particularly toward participatory research and evaluation. In the case of an observational system, a system will function best if it’s designed by a group that includes those who will actually be the observers. If they’re part of the planning, they’ll be familiar with the system, know exactly what information it’s meant to observe, and understand their roles with the observational system.

In a community-based or smaller organization, it’s likely that time will be a factor – there are probably too few people already doing too many jobs. If that’s the case, the level and nature of the observational system has to be one that the staff can actually handle, whether they’re the observers, or whether they’re facilitating the observation for outside evaluators or volunteers. If they help to plan and set up the system, they’ll have far more incentive to make sure it works than if it’s imposed on them.

The design of the observational system should specifically include the people who will actually do the observing, who are often either staff members or members of the group that will benefit from the program. In addition to these, it helps to include researchers or others who understand observational systems, and can help to design a system that specifically meets the needs of the evaluation or research project. It might also be beneficial to include individuals from the group(s) that will be observed, to help with cultural issues and provide feedback on their response to the design. The observational design team, therefore, might consist both of members of the overall evaluation planning group, and others specifically recruited to work on an observational system.

The actual short list includes:

• Program staff and administrators
• Support staff (who often do recording or data entry)
• Outside evaluators or research consultants
• Participants or beneficiaries
• Volunteer observers

If, for some reason, the design group doesn’t include anyone with research experience, a training that includes information about different methods of observation, and about which methods are likely to produce which kinds of information, could help greatly to inform the design process. If the group does include researchers, that information could be presented as part of the discussion about design and the various possibilities, rather than in a training or workshop format.

How do you design an observational system?

So, you’ve decided on evaluation questions and planned an evaluation. Now it’s time to determine how you’ll get the data you need to answer your evaluation questions.

Review your evaluation questions

Remember these? You decided what it was you wanted to know, in order to determine whether your program was effective. Let’s go back to the example at the beginning of this section, the local community health center program. The center was starting a physical activity program for people with high blood pressure. Its objective was to have the participants engage in 45 minutes of moderate exercise at least four times a week over six months.

It hoped for several outcomes from this activity:

• That participants’ blood pressure would decrease
• That participants who needed to would lose weight
• That participants would experience a sense of greater well-being
• That participants would continue the exercise routine after the six-month program ended.

Some evaluation questions, therefore, are:

• Did participants engage in the recommended exercise routine for the period of the program?
• Did participants’ blood pressure decrease by the end of the six months?
• Did those participants who needed to lose weight do so by the end of six months?

There could easily be many other questions, a few examples being:

• How well attended were the workshops? Did participants find them helpful? Did participants who attended all or most of the workshops achieve better results than those who didn’t?
• Did participants experience a sense of greater well-being by the end of six months?
• Did participants continue the exercise routine (and maintain their lower blood pressure) after the program ended?

The answers to these questions might be only a few of those that the center wanted, but let’s stick with them for now, and use them as examples as we go through this part of the section. You may be concerned about your own process – how well you actually plan and implement your program.  You may also, like the center, have a specific time frame in which you hope for results. You might also have benchmarks – smaller achievements along the way to a larger goal – that you’re concerned with recording. All of this should  figure into your observational design.

Decide what you need to observe to answer your questions

Depending on the kind of program or effort you’re engaged in, and the nature of your evaluation questions, there’s a broad range of choices here.

Some of the most common:

• Participants’ behavior. This could be anything from the aggressive behaviors of children in a schoolyard or play setting to the degree of welding skill exhibited by participants in an employment-training program to the social interactions of the users of the neighborhood park we discussed earlier. The possibilities are nearly endless.
• Someone else’s behavior. The ultimate test of whether a high school peer mediation training program is working, for instance, may not be the behavior of the mediators on whose training the program focuses, but the behavior of the students with whom they work. There’s also a possibility here of looking at the ways in which participants are treated by program staff and vice versa.
• Conditions. An initiative may aim to change conditions directly – eliminating a crack house where drug dealing occurs, building affordable housing, cleaning up a polluted river – or may be meant to influence those changes by implementing programs or environmental or policy changes..
• Observations of products or results of behavior. When the behavior or event or condition itself isn’t visible or observable, either because it’s private, or because it takes place on a level that can’t be observed directly, you may have to measure its products or results.  It would be virtually impossible to observe directly the rate at which adolescents practiced safe sex, but it would be possible to learn the rate of STD infections among them, and the number of teen pregnancies before, during, and after a safe-sex peer education program.

When products or effects are all you can observe, you have to be sure that you’ve chosen the right ones to look at.  They should be, to the extent possible, obvious results of the behavior or condition you’re interested in, and you should take into account – and try to correct for – any other factors that might have caused them.

• Participants’ knowledge or attitudes. Like participants’ behavior, the possible range here is enormous, from scores on a knowledge test to nearly anything else you might think of.
• Someone else’s knowledge or attitudes. For instance, an advocacy program would be concerned with changing the attitudes of legislators and the public, but might not have direct contact with those whom it hoped to influence. This might use repeated public opinion surveys to assess willingness to support a particular policy change.
• Goal attainment.  Some programs have a particular aim that is their only reason for existence. This might be the passage or repeal of a law, the building of a school, the freedom of one or more political prisoners, etc. The only evaluation question in that case may be whether the goal has been achieved (or to what degree). In this situation, a goal attainment scale can be used to assess the degree of attainment (e.g., from 5 = most favorable outcome, to 1 = least favorable outcome).
• Interactions. The focus of an evaluation might be on the nature of interactions, or on whether particular individuals or groups interact or engage each other.  For instance, if the goal is increasing parent-child interaction, each parties’ talking to and responding to the other might be measure. As above, interactions among program participants, staff, or between participants and staff might be the focus of observations.

All of the above possibilities might have to do with either program goals – i.e., what a program wants to accomplish – or process and implementation – how a program goes about setting up and carrying out its work.

There are also some areas of observation that relate specifically to program process and implementation:

• Planning. Measurement here may focus on who was involved in planning what parts of the program, how the plan was developed, what its content was, satisfaction, etc.
• Timeline. When did the planning, implementation, and evaluation of the program each begin?  How long did each take?  Were deadlines met, and, if not, why not?
• Numbers of participants. How many participants did you have? What was the average amount of time they spent in the program? How many dropped out before completing the program?  How did those numbers compare to what you expected?
• Methods. What methods did you use in the program or intervention? How were they used?
• Program implementation. What did you actually do? This would include the program activity, its frequency and duration, the number of participants it served, where it took place (if that’s relevant), and how it was conducted.

In addition to identifying what you want to look at, you’ll have to define it carefully, so that observers know exactly what to look for. You have to be certain that all observations of a particular behavior, for instance, refer to the same phenomena (e.g., specific features that define whether the behavior occurred), even if observations made by different people.  If they don’t use the same measurement, you can’t really count on the information you get. Setting the limits of observations in each category – what’s included, what’s excluded, and where the boundaries are – will help to eliminate disagreement and make the observation more reliable.

To continue the health center example, let’s look at what the Center needs to observe.  In order to determine whether participants are actually doing their exercise regularly, the center has to find a way to observe people’s behavior (e.g., activity logs, self-reports on how often they engaged in physical activity). To find out whether they’ve lost weight, the center has to observe an outcome of behavior (e.g., weight, body mass index).  To learn whether they’re experiencing an increased sense of well-being, the center has to obtain self-reports.  And to learn whether they continue their exercise past the end of the program, it has to find a way to observe behavior after the program ends.

Decide how the observations will be conducted

Earlier, we discussed some methods of observation. We’ll return to those here, and examine them in greater detail.

• Direct observation. Direct observation involves either the “fly on the wall” approach, where the observer is anonymous and generally unnoticed, or – more often in service organizations of various kinds – is either an (identified) outside evaluator, or a staff member who works with participants and records, sometimes with their help, their behaviors and aspects of the situation  Anonymous observers are particularly good in situations where any people being observed are equally anonymous – conditions, large events, or activity like the use of that neighborhood park we talked about earlier, where the people involved could be anyone.

One common method of direct observation -- whether the observer is a program staffer or a program participant – is through keeping a journal or activity log. The observer writes down or otherwise records, soon after the occurrence, an account of what happened and events related to it, and often reactions to those experiences as well.  The journal or activity log then becomes a picture of the flow of the program, detailing the progress through it of particular participants, adjustments made, and satisfaction.

The nature of journals or report logs will obviously vary with the nature of the program, and not all programs or efforts lend themselves to this kind of observation.  But, especially in situations where several people have written journals or logs that cover the same period and events, they can be a very powerful and revealing means of observation.

• Participant observation. As we’ve explained, participant observers become part of the event, activity, culture, etc. that’s being observed, and experience it firsthand. Thus, in a health or human service program, the observer might be an actual participant (i.e., a  member of the group at whom the program is aimed), or an evaluator who joins participants in their activities. In the case of the park, for instance, a neighborhood resident who already visits the park regularly might volunteer to track how he and others actually use it – when various people or groups come, what they do once they’re there, which parts of the park they frequent, who interacts with whom, etc.

A mico-grant program was designed by a non-governmental organization in a rural village to increase income through creation of small businesses by participants. Members of the staff took part in each workshop, and participated in such activities as training in lending and loans, all the while observing the activities and other participants.  At the end of each day, the staff conducted group discussions where they relayed what they had seen, and asked participants to analyze what they had done.  The staff in this instance functioned as participant observers, and their participation added greatly to their ability to help their client, low-income women from small villages, understand and use their experiences.

• Self-reports. When the object of your observation is participants’ behavior that takes place away from the program (the amount of time participants spend reading to their children, for example), you often have to rely on the observations of participants themselves. This reliance has, as you might expect, both advantages and disadvantages. On the one hand, participants obviously know their own actions well. On the other, they can also leave out things that might be embarrassing or report in ways that think others want to see.  In addition, since they haven’t had prior experience as or been trained as observers, they might miss, or dismiss as unimportant and not report, behaviors or conditions that would be valuable for evaluation purposes.

An obvious remedy would be to train self-reporters as observers, and in some cases –medical trials, for instance – that’s both reasonable and common.  In other situations, however, it would go too far toward telling participants “the right answers,” and thereby possibly changing what they report toward what they think you want to hear.  There is also a huge advantage to self-reports: when they’re honest and represent a real change in behavior or experience for the reporters, they’re far more powerful than anything another person could say about their experience.

Self-reports, at least as defined here, imply an array of possible techniques for data collection.  Individual and group interviews, focus groups, public meetings, surveys and questionnaires, journals, checklists, and even casual conversation might all be ways for participants to convey information for an evaluation.

• Second-hand reports. These are reports about participant behavior or about conditions that come from people associated with those participants or conditions, but not directly connected to your program or effort. They might be service workers, teachers, health professionals youth workers, family members (particularly parents of young children), employers – almost anyone. Second-hand observations have to be viewed with some of the same cautions as those from people who work closely with participants. Observer-participant relationships, sympathy or empathy for participants, or observers’ personal biases can all keep reports from being objective. These observers may also need training.
• Electronic or mechanical observation. There are some circumstances where human observation is impossible or impractical. Observations of speeding are often made electronically, as are observations of health conditions inside the human body (using X-rays, CT or MRI scans, electrocardiographs, etc.)  In these situations, objectivity is no problem, but you have to be sure that whatever equipment you use is working properly, set up correctly, well-maintained, and protected against possible damage. It is also important that whoever interprets the information the equipment provides is trained to do so, and understands the limits and appropriate uses of that information.
• Tests or other similar observation tools. Education and health organizations often use various kinds of tests as observation tools. In a human service context, they are generally used to observe progress in skills, competency levels, or development. In public health or medicine, tests may be used to observe health status (e.g., screenings for elevated blood cholesterol) and the effects of treatment. They can be very useful in all these circumstances, but they’re also very specific, and don’t allow much room for intuition. In addition, the results of tests of skills, knowledge, or intellectual ability may be influenced by nervousness, lack of sleep, personal problems, or other factors that have little to do with actual competency.
• Public records and the like. If you’re using community-level indicators, such as rates of infant mortality or injuries due to motor vehicle crashes, as one way of looking at outcomes, you’ll have to use records, census data, and other similar material to get the data you need.

To continue with our previous example, the local health center would have to use a variety of these observation methods. The beginning and ongoing observations of blood pressure and weight at the monthly counseling sessions would take place with the use of instruments – a blood pressure cuff and a weighing scale – as well as by direct visual observation. (While an obvious reduction in fat may not indicate weight loss if the fat is replaced by muscle, it does indicate an increase in fitness, which may be equally beneficial. Fitness levels could also be mechanically and electronically measured, if the program chose.)

The amount and type of exercise each participant engaged in would be self-observed and self-reported through journals and interviews.  Feelings of well-being would be self-reported, but could also be observed by counselors trained to look for changes over time in posture, self-presentation, and other observable indicators.  Finally, observations of continued exercise would come through one or more follow-up visits some time after the program ended, with interviews, blood pressure and weight measurements, and more direct visual observation of fitness levels.  (Participants might also agree to continue keeping journals for a set period of time after the program ended, thus providing a self-report of their ongoing levels of physical activity.)

Decide when you need to observe

The question here is whether you need to start observing at the very beginning of the program (you almost always do), and how often you should observe throughout the course of the evaluation.

Some of the possibilities:

• Pre- and post- observation. This means making your observations at the beginning and the end of the evaluation period or the program. It’s the equivalent of what many schools do with standardized testing. They test reading scores at the beginning and end of each year, and then compare the two to determine how much the students have advanced. Although this type of observation may tell you whether anything changed during the program, it won’t give you strong evidence how the change took place, what caused it, or how effective your methods were.

This explanation assumes only before-and-after observation.  Most of the possibilities here include before-and-after observation, but add other observations to it.  For most kinds of evaluation, you should start observing at the beginning, or even well before the beginning (to understand whether any changes may in fact be part of an already-existing trend).  If you conduct your first observation partway, you won’t know if changes occurred before then.  A major change may occur toward the beginning in some interventions, toward the end in others, steadily throughout the intervention in still others, and in a few not till after the intervention is over.  It’s important to know just where you started from in order to fully understand what you’re seeing.  It may be that a long intervention is no more effective than a short one, or that a short one makes no difference at all.  You can only tell by knowing where you started from and through repeated or continuous measurement.

If your program or effort is one with a specific, one-time goal – for example, the passage of a law, or the clean-up of a particular space – the temptation may be to evaluate it only by whether you reach your goal or not (i.e., a single observation at the end of the effort).  This would be a mistake, because it wouldn’t take into account the parts of your effort that were successful and why, whether or not you reached your goal.  That’s a piece of information you’ll need the next time – and there will be a next time – you or others in the community take on a similar effort.

• At regular intervals during the evaluation period. You might choose any period from once an hour to once a month or more, depending on what you’re observing. The regularity makes observations easy to schedule, and gives an interval-by-interval picture of what’s going on.
• At irregular intervals during the evaluation period. The reason for this schedule might be logistical (you observe when you can); might have to do with making sure that observations aren’t expected, so that you get a true picture of what you’re looking at; or might be an attempt to look at the program or effort randomly, again to try to get an accurate picture.
• At specific times during the evaluation period. In this case, you might be concerned to see what happens or what participants are doing at different, identifiable times that imply different, identifiable conditions. In observing the use of that neighborhood park we mentioned earlier, you might want to be sure to go on weekdays, on weekends, in the morning, afternoon, and evening, at each of the four seasons, in rain, clouds, sun, and snow, and on days when there were special events in the park, to see who uses the park and how under different conditions and at different times.  If you’re monitoring the process and progress of the program, it’s important to make sure you observe each stage of it – the planning, the preparation, the implementation, the evaluation, and any follow-up – to make sure you get a full picture of what you did and how you did it. This will give you the information you need to analyze in order to make adjustments in how you conduct your work.
• Continuously. When the observer is a staff member working with program participants (or one or more of the participants themselves), it may be possible to make ongoing observations.The observer in this situation might observe directly using checklists, keep a journal, ask participants to keep records, video- or audiotape sessions, or record what happens in some other way, so that there’s an ongoing, day-to-day account of the behavior and what is happening in the environment of the program.

At the local health center, some of the observation – particularly that of monitoring participants’ blood pressure and weight – would be done at regular intervals, during the monthly meetings.  There would also be some continuous observation -- that of participants keeping track of their exercise programs in journals.  And finding out whether participants continued with their routines would be a one- or two-time follow-up – perhaps six months and a year after the program ended.

Define and describe the behaviors, products, conditions, and/or events that observers should be concerned with

If you want to be sure you know what observers are referring to in their reports, you have to be specific about what you want them to look for. The planning group, or a subgroup of it – the ideal would be a group that included a high proportion of people who will actually be researchers and observers – should set out identification standards for each element to be observed. These would explain what it looked like, when it was likely to occur, who would probably be involved, etc. For instance, to observe bullying or interpersonal violence on a playground would require clear definitions of this behavior, examples and non-examples, and scoring instructions. As a result, observers could all start with the same guidance about what they were looking for.

Design training for observers

Unless all the observation is to be done by those directly involved in the planning (not impossible in a small organization), and depending on their previous experience, observers might need to be trained in one or more areas:

• What it’s important to record, and why. People who have no acquaintance with research might not realize how important it is to record such details as the date, time, evaluation length, place, and circumstances of any observation, a description of who was involved and for how long, whether there were unexpected people or conditions present, etc. An early morning observation might provide a different set of observations than a late afternoon or evening one, for example.The presence of other people in a situation or interview – relatives, friends, program staff – can change the character of the behaviors displayed or information offered. It’s crucial, therefore, that observers understand that the context of the observation can be as important as its content.
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed. Careful definitions and descriptions of what’s to be observed won’t make much difference unless those who’ll do the observations are familiar with them.
• Effects of observation. In some cases, the behavior of participants might change as a result of their reactions to being observed. Observers have to be aware of that possibility, and make their own behavior as invisible as they can, so as not to influence participants’ behaviors.

In addition to human observers, the presence of audio or video equipment can often have an effect.  One way to offset it is to wait to start collecting data until participants are used to the presence of the equipment.  It’s also important to get participants’ permission beforehand to use recording equipment.

• Observer bias. Especially in situations where the observers are also program staff, their relationships with participants, or simply with the effort as a whole, may affect their reports or observations. If they particularly like or dislike a participant, that may have some influence on how they interpret or describe that person’s behavior. If they’re heavily invested in the success of the program being evaluated, they may – intentionally or unintentionally – put the best possible light on what they see. Whether or not they’re program staff, observers can also be influenced by their personal assumptions, their cultural, religious, or educational background, or their current psychological states or life circumstances. If they can be helped to recognize these biases and understand why and how they should be acknowledged or eliminated, there’s a better chance that they’ll conduct reliable observations.
• Observer drift. Sometimes, after people have been observing for a while, their observations tend to take on a regularity based on the rules they make up rather than shared definitions based on a standard.  They might tend to rate the behavior of certain participants in ways based more on past experience than on what they see, for instance.

You may also have to correct for observer effects, bias, or drift over the course of the evaluation.  That’s part of devising checks for accuracy and reliability based on a standard.

Devise checks for reliability and accuracy

If your information is to be reliable, it’s important that when two observers record a particular behavior, they mean exactly the same thing. This is a matter of training (see above), and also one of checking, either at the beginning or periodically, to make sure that all observers are seeing things in the same way. A participatory design of the system will help here. If the observers are involved in defining what they’ll all be observing, there’s a much better chance they’ll all see it similarly.

Some ways you can try to ensure agreement among observers:

• Use an external standard. One way to define what you’re looking for is to use a standard that’s used and accepted by all observers. “Behavior X looks like this, occurs in these circumstances, lasts for this long, and has these after-effects or results.”  The use of external standards often employs a checklist or something similar.  The observer checks off components of a behavior or condition, thus documenting what he sees in a way that matches how a different observer would score the event the same situation.  Such standards help assure the continued accuracy of the observational system.

Research teams and laboratories commonly use standards to assure agreement in identifying various conditions.  Each condition is described in detail, with various possible markers, such as measure of blood pressure or environmental toxins.

• Check for inter-rater reliability. Inter-rater reliability is the research term for assessing whether all observers interpret the same things – behaviors, conditions, events – in the same way. One way to address it is to check observers against one another. Two or more are exposed to the same situations or information, and then their scoring, such as of instances of bullying,  are compared. If they all say essentially the same thing, then inter-rater reliability is high, and everything’s fine.Typically, in research, if observers agree 80% or more of the time, observations are deemed reliable.If they disagree about what they saw, then you have to find the source of the disagreement. They may define terms differently, or their backgrounds may bias them toward seeing the same thing in different ways. Whatever the case, you have to uncover differences, and find a way to help all observers see things similarly and accurately.
• Use random third-party checks. A researcher, program director, or someone else who has a clear idea of what information is important and what various conditions or behaviors look like can observe in randomly chosen situations along with a regular observer to see if their observations match reasonably well.  If they disagree once, it may not mean much; but if they consistently disagree, there’s a problem.

Determine how to review and adjust your observational system for the next evaluation

Here’s this section’s version of “keep at it.” Just like your program, your evaluation, including your observational system, should be evaluated and adjusted to be as effective as possible.

Now you’re ready to start collecting and analyzing data. With careful planning and good training, you should be able to get the information you need for your evaluation.

In Summary

In order to conduct an evaluation that allows you to see your program or effort clearly and to adjust and improve it, you have to have a way of collecting accurate and useful information about it. The observational system you use is the way you look at what you’re doing – at your own process, at participants’ behavior and their progress and results, at conditions that affect your effort or that your effort is trying to change – to gain the information that you’ll analyze to evaluate your work. That system has to be feasible within your resources, and has to fit with the nature of your program, so designing it is an important part of your evaluation.

The design of observational systems is best carried out as a participatory process, particularly one involving both researchers or evaluators and those who’ll do the actual data collection. That involvement will give them a clear understanding of the system itself, of what information is needed, and of the pitfalls to data collection that they might encounter along the way. The result should be a more reliable system, and, ultimately, more accurate data for your evaluation.

What do we mean by an observational system?

___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program

___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with

Why design an observational system?

___It can help you get reliable information

___It can help you find out exactly what you need to know, eliminating or reducing wasted effort

___It can ensure that observations are made

___It can make it easier to analyze your data

___It can help you avoid haphazard evaluation

___It will make it easier to justify your findings

___It can help you gain credibility with funders and policy makers

___It can let you pass on your practices with confidence

___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust

When should you design an observational system?

___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning

___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle

Who should design an observational system?

___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation

How do you design an observational system?

___Review your evaluation questions

___Decide what you need to observe to answer your questions:

• Participants’ behavior
• Someone else’s behavior
• Conditions
• Observations of results of behavior
• Participants’ knowledge or attitudes
• Someone else’s knowledge or attitudes
• Goal attainment
• Interactions
• Program process or implementation (e.g., number of participants)

___Decide how the observations will be conducted:

• Direct observation
• Participant observation
• Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
• Second-hand reports, including interviews, journals, surveys, etc.
• Electronic or mechanical observation
• Tests or other similar observation tools
• Public records and the like for community-level indicators

___Decide when you need to observe:

• Pre- and post- observation
• At regular intervals during the evaluation period
• At irregular intervals during the evaluation period
• At specific times during the evaluation period
• Continuously

___Define and describe the behaviors, conditions, and/or events that observers should be concerned with

___Train observers in:

• What it’s important to record, and why
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
• Effects of observation
• Observer bias
• Observer drift

___Devise checks for reliability and accuracy:

• Use an external standard
• Check for inter-rater reliability
• Use random third-party checks

___Adjust the system for the next evaluation

When you hear the word “experiment,” it may call up pictures of people in long white lab coats peering through microscopes. In reality, an experiment is just trying something out to see how or why or whether it works. It can be as simple as putting a different spice in your favorite dish, or as complex as developing and testing a comprehensive effort to improve child health outcomes in a city or state.

Academics and other researchers in public health and the social sciences conduct experiments to understand how environments affect behavior and outcomes, so their experiments usually involve people and aspects of the environment. A new community program or intervention is an experiment, too, one that a governmental or community organization engages in to find out a better way to address a community issue. It usually starts with an assumption about what will work – sometimes called a theory of change - but that assumption is no guarantee. Like any experiment, a program or intervention has to be evaluated to see whether it works and under what conditions.

In this section, we’ll look at some of the ways you might structure an evaluation to examine whether your program is working, and explore how to choose the one that best meets your needs. These arrangements for discovery are known as experimental (or evaluation) designs.

What do we mean by a design for the evaluation?

Every evaluation is essentially a research or discovery project. Your research may be about determining how effective your program or effort is overall, which parts of it are working well and which need adjusting, or whether some participants respond to certain methods or conditions differently from others. If your results are to be reliable, you have to give the evaluation a structure that will tell you what you want to know. That structure – the arrangement of discovery- is the evaluation’s design.

The design depends on what kinds of questions your evaluation is meant to answer.

Some of the most common evaluation (research) questions:

• Does a particular program or intervention – whether an instructional or motivational program, improving access and opportunities, or a policy change – cause a particular change in participants’ or others’ behavior, in physical or social conditions, health or development outcomes, or other indicators of success?
• What component(s) and element(s) of the program or intervention were responsible for the change?
• What are the unintended effects of an intervention, and how did they influence the outcomes?
• If you try a new method or activity, what happens?
• Will the program that worked in another context, or the one that you read about in a professional journal, work in your community, or with your population, or with your issue?

If you want reliable answers to evaluation questions like these, you have to ask them in a way that will show you whether you actually got results, and whether those results were in fact due to your actions or the circumstances you created, or to other factors.  In other words, you have to create a design for your research – or evaluation – to give you clear answers to your questions.  We’ll discuss how to do that later in the section.

Why should you choose a design for your evaluation?

An evaluation may seem simple: if you can see progress toward your goal by the end of the evaluation period, you’re doing OK; if you can’t, you need to change. Unfortunately, it’s not that simple at all. First, how do you measure progress? Second, if there seems to be none, how do you know what you should change in order to increase your effectiveness? Third, if there is progress, how do you know it was caused by ( or contributed to) your program, and not by something else? And finally, even if you’re doing well, how will you decide what you could do better, and what elements of your program can be changed or eliminated without affecting success? A good design for your evaluation will help you answer important questions like these.

Some specific reasons for spending the time to design your evaluation carefully include:

• So your evaluation will be reliable. A good design will give you accurate results. If you design your evaluation well, you can trust it to tell you whether you’re actually having an effect, and why. Understanding your program to this extent makes it easier to achieve and maintain success.
• So you can pinpoint areas you need to work on, as well as those that are successful. A good design can help you understand exactly where the strong and weak points of your program or intervention are, and give you clues as to how they can be further strengthened or changed for the greatest impact.
• So your results are credible. If your evaluation is designed properly, others will take your results seriously. If a well-designed evaluation shows that your program is effective, you’re much more likely to be able to convince others to use similar methods, and to convince funders that your organization is a good investment.
• So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants. Participants’ histories, crucial local or national events, the passage of time, personal crises, and many other factors can influence the outcome of a program or intervention for better or worse. A good evaluation design can help you to identify these, and either correct for them if you can, or devise methods to deal with or incorporate them.
• So you can identify unintended consequences (both positive and negative) and correct for them. A good design can show you all of what resulted from your program or intervention, not just what you expected. If you understand that your work has consequences that are negative as well as positive, or that it has more and/or different positive consequences than you anticipated, you can adjust accordingly.
• So you’ll have a coherent plan and organizing structure for your evaluation. It will be much easier to conduct your evaluation if it has an appropriate design. You’ll know better what you need to do in order to get the information you need. Spending the time to choose and organize an evaluation design will pay off in the time you save later and in the quality of the information you get.

When should you choose a design for your evaluation?

Once you’ve determined your evaluation questions and gathered and organized all the information you can about the issue and ways to approach it, the next step is choosing a design for the evaluation. Ideally, this all takes place at the beginning of the process of putting together a program or intervention.  Your evaluation should be an integral part of your program, and its planning should therefore be an integral part of the program planning.

That’s the ideal; now let’s talk about reality. If you’re reading this, the chances are probably at least 50-50 that you’re connected to an underfunded government agency or to a community-based or non-governmental organization, and that you’re planning an evaluation of a program or intervention that’s been running for some time – months or even years.

Even if that’s true, the same guidelines apply. Choose your questions, gather information, choose a design, and then go on through the steps presented in this chapter. Evaluation is important enough that you won’t really be accomplishing anything by taking shortcuts in planning it. If your program has a cycle, then it probably makes sense to start your evaluation at the beginning of it – the beginning of a year or a program phase, where all participants are starting from the same place, or from the beginning of their involvement.

If that’s not possible – if your program has a rolling admissions policy, or provides a service whenever people need it – and participants are all at different points, that can sometimes present research problems. You may want to evaluate the program’s effects only with new participants, or with  another specific group. On the other hand, if your program operates without a particular beginning and end, you may get the best picture of its effectiveness by evaluating it as it is, starting whenever you’re ready. Whatever the case, your design should follow your information gathering and synthesis.

Who should be involved in choosing a design?

If you’re a regular Tool Box user, and particularly if you’ve been reading this chapter, you know that the Tool Box team generally recommends a participatory process – involving both research and community partners, including all those with an interest in or who are affected with the program in planning and implementation. Choosing a design for evaluation presents somewhat of an exception to this policy, since scientific or evaluation partners may have a much clearer understanding of what is required to conduct research, and of the factors that may interfere with it.

As we’ll see in the “how-to” part of this section, there are a number of considerations that have to be taken into account to gain accurate information that actually tells you what you want to know. Graduate students generally take courses to gain the knowledge they need to conduct research well, and even some veteran researchers have difficulty setting up an appropriate research design. That doesn’t mean a community group can’t learn to do it, but rather that the time they would have to spend on acquiring background knowledge might be too great. Thus, it makes the most sense to assign this task (or at the very least its coordination) to an individual or small group with experience in research and evaluation design. Such a person can not only help you choose among possible designs, but explain what each design entails, in time, resources, and necessary skills, so that you can judge its appropriateness and feasibility for your context.

How do you choose a design for your evaluation?

How do you go about deciding what kind of research design will best serve the purposes of your evaluation?

The answer to that question involves an examination of four areas:

• The nature of the research questions you are trying to answer
• The challenges to the research, and the ways they can be resolved or reduced
• The kinds of research designs that are generally used, and what each design entails
• The possibility of adapting a particular research design to your program or situation – what the structure of your program will support, what participants will consent to, and what your resources and time constraints are

We’ll begin this part of the section with an examination of the concerns research designs should address, go on to considering some common designs and how well they address those concerns, and end with some guidelines for choosing a design that will both be possible to implement and give you the information you need about your program.

Note: in this part of the section, we’re looking at evaluation as a research project.  As a result, we’ll use the term “research” in many places where we could just as easily have said, for the purposes of this section, “evaluation.”  Research is more general, and some users of this section may be more concerned with research in general than evaluation in particular.

Concerns research designs should address

The most important consideration in designing a research project – except perhaps for the value of the research itself – is whether your arrangement will provide you with valid information. If you don’t design and set up your research project properly, your findings won’t give you information that is accurate and likely to hold true with other situations.  In the case of an evaluation, that means that you won’t have a basis for adjusting what you do to strengthen and improve it.

Here’s a far-fetched example that illustrates this point.  If you took children’s heights at age six, then fed them large amounts of a specific food for three years – say carrots – and measured them again at the end of the period, you’d probably find that most of them were considerably taller at nine years than at six.  You might conclude that it was eating carrots that made the children taller because your research design gave you no basis for comparing these children’s growth to that of other children.

There are two kinds of threats to the validity of a piece of research. They are usually referred to as threats to internal validity (whether the intervention produced the change) and threats to external validity (whether the results are likely to apply to other people and situations).

Threats to internal validity

These are threats (or alternative explanations) to your claim that what you did caused changes in the direction you were aiming for. They are generally posed by factors operating at the same time as your program or intervention that might have an effect on the issue you’re trying to address. If you don’t have a way of separating their effects from those of your program, you can’t tell whether the observed changes were caused by your work, or by one or more of these other factors.They’re called threats to internal validity because they’re internal to the study – they have to do with whether your intervention – and not something else – accounted for the difference.

There are several kinds of threats to internal validity:

• History. Both participants’ personal histories – their backgrounds, cultures, experiences, education, etc. – and external events that occur during the research period – a disaster, an election, conflict in the community, a new law – may influence whether or not there’s any change in the outcomes you’re concerned with.
• Maturation. This refers to the natural physical, psychological, and social processes that take place as time goes by. The growth of the carrot-eating children in the example above is a result of maturation, for instance, as might be a decline in risky behavior as someone passed from adolescence to adulthood, the development of arthritis in older people, or participants becoming tired during learning activities towards the end of the day.
• The effects of testing or observation on participants. The mere fact of a program’s existence, or of their taking part in it, may affect participants’ behavior or attitudes, as may the experience of being tested, videotaped, or otherwise observed or measured.
• Changes in measurement. An instrument – a blood pressure cuff or a scale, for instance – can change over time, or different ones may not give the same results. By the same token, observers – those gathering information – may change their standards over time, or two or more observers may disagree on the observations.
• Regression toward the mean. This is a statistical term that refers to the fact that, over time, the very high and very low scores on a measure (a test, for instance) often tend to drift back toward the average for the group. If you start a program with participants who, by definition, have very low or high levels of whatever you’re measuring – reading skill, exposure to domestic violence, particular behavior toward people of other races or backgrounds, etc. – their scores may end up closer to the average over the course of the evaluation period even without any program.
• The selection of participants. Those who choose participants may slant their selection toward a particular group that is more or less likely to change than a cross-section of the population from which the group was selected. (A good example is that of employment training programs that get paid according to the number of people they place in jobs. They’re more likely to select participants who already have all or most of the skills they need to become employed, and neglect those who have fewer skills... and who therefore most need the service.) Selection can play a part when participants themselves choose to enroll in a program (self-selection), since those who decide to participate are probably already motivated to make changes. It may also be a matter of chance: members of a particular group may, simply by coincidence, share a characteristic that will set their results on your measures apart from the norm of the population you’re drawing from.

Selection can also be a problem when two groups being compared are chosen by different standards.  We’ll discuss this further below when we deal with control or comparison groups.

• The loss of data or participants. If too little information is collected about participants, or if too many drop out well before the research period is over, your results may be based on too little data to be reliable. This also arises when two groups are being compared. If their losses of data or participants are significantly different, comparing them may no longer give you valid information.
• The nature of change. Often, change isn’t steady and even. It can involve leaps forward and leaps backward before it gets to a stable place – if it ever does. (Think of looking at the performance of a sports team halfway through the season. No matter what its record is at that moment, you won’t know how well it will finish until the season is over.)  Your measurements may take place over too short a period or come at the wrong times to track the true course of the change or lack of change that’s occurring.
• A combination of the effects of two or more of these. Two or more of these factors may combine to produce or prevent the changes your program aims to produce. A language-study curriculum that is tested only on students who already speak two or more languages runs into problems with both participants’ history – all the students have experience learning languages other than their own – and selection – you’ve chosen students who are very likely to be successful at language learning.

Threats to external validity

These are factors that affect your ability to apply your research results in other circumstances – to increase the chances that your program and its results can be reproduced elsewhere or with other populations. If, for instance, you offer parenting classes only to single mothers, you can’t assume, no matter how successful they appear to be, that the same classes will work as well with men.

Threats to external validity (or generalizability) may be the result of the interactions of other factors with the program or intervention itself, or may be due to particular conditions of the program.

Some examples:

• Interaction of testing or data collection and the program or intervention.  An initial test or observation might change the way participants react to the program, making a difference in final outcomes.  Since you can’t assume that another group will have the same reaction or achieve similar final outcomes as a result, external validity or generalizability of the findings becomes questionable.
• Interaction of selection procedures and the program or intervention.  If the participants selected or self-selected are particularly sensitive to the methods or purpose of the program, it can’t be assumed to be effective with participants who are less sensitive or ready for the program.

Parents who’ve been threatened by the government with the loss of their children due to child abuse may be more receptive to learning techniques for improving their parenting, for example, than parents who are under no such pressure.

• The effects of the research arrangements. Participants may change behavior as a result of being observed, or may react to particular individuals in ways they would be unlikely to react to others.

A classic example here is that of a famous baboon researcher, Irven DeVore, who after years of observing troupes of baboons, realized that they behaved differently when he was there than when he wasn’t.  Although his intent was to observe their natural behavior, his presence itself constituted an intervention, making the behavior of the baboons he was observing different from that of a troupe that was not observed.

• The interference of multiple treatments or interventions. The effects of a particular program can be changed when participants are exposed to it beforehand in a different context, or are exposed to another before or at the same time as the one being evaluated. This may occur when participants are receiving services from different sources, or being treated simultaneously for two or more health issues or other conditions.

Given the range of community programs that exist, there are many possibilities here. Adults might be members of a high school completion class while participating in a substance use recovery program.  A diabetic might be treated with a new drug while at the same time participating in a nutrition and physical activity program to deal with obesity.  Sometimes, the sequence of treatments or services in a single program can have the same effect, with one influencing how participants respond to those that follow, even though each treatment is being evaluated separately.

Common research designs

Many books have been written on the subject of research design. While they contain too much material to summarize here, there are some basic designs that we can introduce. The important differences among them come down to how many measurements you’ll take, when you will take them, and how many groups of what kind will be involved.

Program evaluations generally look for the answers to three basic questions:

• Was there any change – in participants’ or others’ behavior, in physical or social conditions, or in outcomes or indicators of success– during the evaluation period?
• Was whatever change took place – or the lack of change – caused by your program, intervention, or effort?
• What, in your program or outside it, actually caused or prevented the change?

As we’ve discussed, changes and improvement in outcomes may have been caused by some or all of your intervention, or by external factors. Participants’ or the community’s history might have been crucial. Participants may have changed as a result of simply getting older and more mature or more experienced in the world – often an issue when working with children or adolescents. Environmental factors – events, policy change, or conditions in participants’ lives – can often facilitate or prevent change as well. Understanding exactly where the change came from or where the barriers to change reside, gives you the opportunity to adjust your program to take advantage of or combat those factors.

If all you had to do was to measure whatever behavior or condition you wanted to influence at the beginning and end of the evaluation, choosing a design would be an easy task. Unfortunately, it’s not quite that simple – there are those nasty threats to validity to worry about. We have to keep them in mind as we look at some common research designs.

Research designs, in general, differ in one or both of two ways: the number and timing of the measurements they use; and whether they look at single or multiple groups.  We’ll look at single-group designs first, then go on to multiple groups.

Before we go any further, it is helpful to have an understanding of some basic research terms that we will be using in our discussion.

Researchers usually refer to your first measurement(s) or observation(s) – the ones you take before you start your program or intervention – as a baseline measure or baseline observation, because it establishes a baseline – a known level – to which you compare future measurements or observations.

Some other important research terms:

• Independent variables are the program itself and/or the methods or conditions that the researcher – in this case, you – wants to evaluate. They’re called variables because they can change – you might have chosen (and might still choose) other methods.  They’re independent because their existence doesn’t depend on whether something else occurs: you’ve chosen them, and they’ll stay consistent throughout the evaluation period.
• Dependent variables are whatever may or may not change as a result of the presence of the independent variable(s).  In an evaluation, your program or intervention is the independent variable.  (If you’re evaluating a number of different methods or conditions, each of them is an independent variable.)  Whatever you’re trying to change is the dependent variable.  (If you’re aiming at change in more than one behavior or outcome, each type of change is a different dependent variable.)  They’re called dependent variables because changes in them depend on the action of the independent variable...or something else.
• Measures are just that – measurements of the dependent variables.  They usually refer to procedures that have results that can be translated into numbers, and may take the form of community assessments, observations, surveys, interviews, or tests. They may also count incidents or measure the amount of the dependent variable (number or percentage of children who are overweight or obese, violent crimes per 100,000 population, etc.)
• Observations might involve measurement, or they might simply record what happens in specific circumstances: the ways in which people use a space, the kinds of interactions children have in a classroom, the character of the interactions during an assessment.  For convenience, researchers often use “observation” to refer to any kind of measurement and we’ll use the same convention here.

Pre- and post- single-group design

The simplest design is also probably the least accurate and desirable: the pre (before) and post (after) measurement or observation. This consists of simply measuring whatever you’re concerned with in one group – the infant mortality rate, unemployment, water pollution – applying your intervention to that group or community, and then observing again. This type of design assumes that a difference in the two observations will tell you whether there was a change over the period between them, and also assumes that any positive change was caused by the intervention.

In most cases, a pre-post design won’t tell you much, because it doesn’t really address any of the research concerns we’ve discussed. It doesn’t account for the influence of other factors on the dependent variable, and it doesn’t tell you anything about trends of change or the progress of change during the evaluation period – only where participants were at the beginning and where they were at the end. It can help you determine whether certain kinds of things have happened – whether there’s been a reduction in the level of educational attainment or the amount of environmental pollution in a river, for instance – but it won’t tell you why. Despite its limitations, taking measures before and after the intervention is far better than no measures.

Even looking at something as seemingly simple to measure pre and post as blood pressure (in a heart disease prevention program) is questionable.  Blood pressure may be lower at the final observation than at the initial one, but that tells you nothing about how much it may have gone up and down in between. If the readings were taken by different people, the change may be due in part to differences in their skill, or to how relaxed each was able to make participants feel.  Familiarity with the program could also have reduced most participants’ blood pressure from the pre- to the post-measurement, as could some other factor that wasn’t specifically part of the independent variable being evaluated.

Interrupted time series design with a single group (simple time series)

An interrupted time series used repeated measures before and after delayed implementation of the independent variable (e.g., the program, etc.) to help rule out other explanations. This relatively strong design – with comparisons within the group – addresses most threats to internal validity.

The simplest form of this design is to take repeated observations, implement the program or intervention, and observe a number of times during the evaluation period, including at the end. This method is a great improvement over the pre- and post- design in that it tracks the trend of change, and can therefore, help see whether it was actually the independent variable that caused any change. It can also help to identify the influence of external factors such as when the dependent variable shows significant change before the intervention is implemented.

Another possibility for this design is to implement more than one independent variable, either by trying two or more, one after another (often with a break in between), or by adding each to what came before.This gives a picture not only of the progress of change, but can show very clearly what causes change. That gives an evaluator the opportunity not only to adjust the program, but to drop elements that have no effect.

There are a number of variations on the interrupted time series theme, including varying the observation times; implementing the independent variable repeatedly; and implementing one independent variable, then another, then both together to evaluate their interaction.

In any variety of interrupted time series design, it’s important to know what you’re looking for.  In an evaluation of a traffic fatality control program in the United Kingdom that focused on reducing drunk driving, monthly measurements seemed to show only a small decline in fatal accidents. When the statistics for weekends, when there were most likely to be drunk drivers on the road, were separated out, however, they showed that the weekend fatality rate dropped sharply with the implementation of the program, and stayed low thereafter. Had the researchers not realized that that might be the case, the program might have been stopped, and the weekend accident rate would not have been reduced.

Interrupted time series design with multiple groups (multiple baseline/time series)

This has the same possibilities as the single time series design, with the added wrinkle of using repeated measures with one or more other groups (so-called multiple baselines). By using multiple baselines (groups), the external validity or generality of the findings is enhanced – we can see if the effects occur with different groups or under different conditions.

This multiple time series design – typically staggered introduction of the intervention with different groups or communities – gives the researcher more opportunities:

• You can try a method or program with two or more groups from the same
• You can try a particular method or program with different populations, to see if it’s effective with others
• You can vary the timing or intensity of an intervention with different groups
• You can test different interventions at the same time
• You can try the same two or more interventions with each of two groups, but reverse their order to see if sequencing it makes any difference

Again, there are more variations possible here.

Control group design

A common way to evaluate the effects of an independent variable is to use a control group.  This group is usually similar to the participant group, but either receives no intervention at all, or receives a different intervention with the same goal as that offered to the participant group. A control group design is usually the most difficult to set up – you have to find appropriate groups, observe both on a regular basis, etc. – but is generally considered to be the most reliable.

The term control group comes from the attempt to control outside and other influences on the dependent variable. If everything about the two groups except their exposure to the program being evaluated averages out to be the same, then any differences in results must be due to that exposure. The term comparison group is more modest; it typically offers a community watched for similar levels of the problem/goal and relevant characteristics of the community or population (e.g., education, poverty).

The gold standard here is the randomized control group, one that is selected totally at random, either from among the population the program or intervention is concerned with – those at risk for heart disease, unemployed males, young parents – or, if appropriate, the population at large. A random group eliminates the problems of selection we discussed above, as well as issues that might arise from differences in culture, race, or other factors.

A control group that’s carefully chosen will have the same characteristics as the intervention group (the focus of the evaluation). If, for instance, the two groups come from the same pool of people with a particular health condition, and are chosen at random either to be treated in the conventional way or to try a new approach, it can be assumed that – since they were chosen at random from the same population – both groups will be subject, on average, to the same outside influences, and will have the same diversity of backgrounds. Thus, if there is a significant difference in their results, it is fairly safe to assume that the difference comes from the independent variable – the type of intervention, and not something else.

The difficulty for governmental and community-based organizations is to find or create a randomized control group. If the program has a long waiting list, it may be able to create a control by selecting those to first receive the intervention at random. That in itself creates problems, in that people often drop off waiting lists out of frustration or other reasons. Being included in the evaluation may help to keep them, on the other hand, by giving them a closer connection to the program and making them feel valued.

An ESOL (English as a Second or Other Language) program in Boston with a three-year waiting list addressed the problem by offering those on the waiting list a different option.  They received videotapes to use at home, along with biweekly tutoring by advanced students and graduates of the program.  Thus, they became a comparison group with a somewhat different intervention that, as expected, was less effective than the program itself, but was more effective than none, and kept them on the waiting list. It also gave them a head start once they got into the classes, with many starting at a middle rather than at a beginning level.

When there’s no waiting list or similar group to draw from, community organizations often end up using a comparison group - one composed of participants in another place or program and whose members’ characteristics, backgrounds, and experience may or may not be similar to those of the participant group. That circumstance can raise some of the same problems related to selection seen when there is no control group. If the only potential comparisons involve very different groups, it may be better to use a design, such as an interrupted time series design that doesn’t involve a control group at all, where the comparison is within (not between) groups.

Groups may look similar, but may differ in an important way.  Two groups of participants in a substance use intervention program, for instance, may have similar histories, but if one program is voluntary and the other is not, the results aren’t likely to be comparable.  One group will probably be more motivated and less resentful than the other, and composed of people who already know they have a potential problem. The motivation and determination of their participants, rather than the effectiveness of the two programs, may influence the amount of change observed.

This issue may come up in a single-group design as well.  A program that may, on average, seem to be relatively ineffective may prove, on close inspection, to be quite effective with certain participants – those of a specific educational background, for instance, or with particular life experiences. Looking at results with this in mind can be an important part of an evaluation, and give you valuable and usable information.

Choosing a design

This section’s discussion of research designs is in no way complete.  It’s meant to provide an introduction to what’s available.  There are literally thousands of books and articles written on this topic, and you’ll probably want more information.  There are a number of statistical methods that can compensate for less-than-perfect designs, for instance: few community groups have the resources to assemble a randomized control group, or to implement two or more similar programs to see which works better.

Given this, the material that follows is meant only as broad guidelines.  We don’t attempt to be specific about what kind of design you need in what circumstances, but only try to suggest some things to think about in different situations. Help is available from a number of directions: Much can be found on the Internet (see the “Resources” part of this section for a few sites); there are numerous books and articles (the classic text on research design is also cited in “Resources”); and universities are a great resource, both through their libraries and through faculty and graduate students who might be interested in what you’re doing, and be willing to help with your evaluation.  Use any and all of these to find what will work best for you.  Funders may also be willing either to provide technical assistance for evaluations, or to include money in your grant or contract specifically to pay for a professional evaluation.

Your goal in evaluating your effort is to get the most reliable and accurate information possible, given your evaluation questions, the nature of your program, what your participants will consent to, your time constraints, and your resources. The important thing here is not to set up a perfect research study, but to design your evaluation to get real information, and to be able to separate the effects of external factors from the effects of your program.  So how do you go about choosing the best design that will be workable for you? The steps are in the first sentence of this paragraph.

Consider your evaluation questions

What do you need to know? If the intent of your evaluation is simply to see whether something specific happened, it’s possible that a simple pre-post design will do. If, as is more likely, you want to know both whether change has occurred, and if it has, whether it has in fact been caused by your program, you’ll need a design that helps to screen out the effects of external influences and participants’ backgrounds.

For many community programs, a control or comparison group is helpful, but not absolutely necessary. Think carefully about the frequency and timing of your observations and the amount of different kinds of information you can collect. With repeated measures, you can get you quite an accurate picture of the effectiveness of your program from a simple time series design. Single group interrupted time series designs, which are often the most workable for small organizations, can give you a very reliable evaluation if they’re structured well. That generally means obtaining multiple baseline observations (enough to set a trend) before the program begins; observing often and documenting your observations carefully (often with both quantitative – expressed in numbers – and qualitative – expressed in records of incidents and of what participants did and said – data); and  including during intervention and  follow-up observations to see whether effects are maintained.

In many of these situations, a multiple-group interrupted time series design is quite possible, but of a “naturally-occurring” experiment. If your program includes two or more groups or classes, each working toward the same goals, you have the opportunity to stagger the introduction of the intervention across the groups. This comparison with (and across) groups allows you to screen out such factors as the facilitator’s ability and community influences (assuming all participants come from the same general population.)  You could also try different methods or time sequences, to see which works best.

In some cases, the real question is not whether your method or program works, but whether it works better than other methods or programs you could be using. Teaching a skill – for instance, employment training, parenting, diabetes management, conflict resolution – often falls into this category.  Here, you need a comparison of some sort.  While evaluations of some of these – medical treatment, for example – may require a control group, others can be compared to data from the field, to published results of other programs, or, by using community-level indicators, from measurements in other communities.

There are community programs where the bottom line is very simple.  If you’re working to control water pollution, your main concern may be the amount of pollution coming out of effluent pipes, or the amount found in the river.  Your only measure of success may be keeping pollution below a certain level, which means that regular monitoring of water quality is the only evaluation you need.  There are probably relatively few community programs where evaluation is this easy – you might, for instance, want to know which of your pollution-control activities is most effective – but if yours is one, a simple design may be all you need.

Consider the nature of your program

What does your program look like, and what is it meant to do?  Does it work with participants in groups, or individually, for instance? Does it run in cycles – classes or workshops that begin and end on certain dates, or a time-limited program that participants go through only once? Or can participants enter whenever they are ready and stay until they reach their goals?  How much of the work of the program is dependent on staff, and how much do participants do on their own?  How important is the program context – the way staff, participants, and others treat one another, the general philosophy of the program, the physical setting, the organizational culture?  (The culture of an organization consists of accepted and traditional ways of doing things, patterns of relationships, how people dress, how they act toward and communicate with one another, etc.)

• If you work with participants in groups, a multiple-group design – either interrupted time series or control group – might be easier to use.  If you work with participants individually, perhaps a simple time series or a single group design would be appropriate.
• If your program is time-limited – either one-time-only, or with sessions that follow one another – you’ll want a design that fits into the schedule, and that can give you reliable results in the time you have. One possibility is to use a multiple group design, with groups following one another session by session. The program for each group might be adjusted, based on the results for the group before, so that you could test new ideas each session.
• If your program has no clear beginning and end, you’re more likely to need a single group design that considers participants individually, or by the level of their baseline performance. You may also have to compensate for the fact that participants may be entering the program at different levels, or with different goals.

A proverb says that you never step in the same river twice, because the water that flows past a fixed point is always changing. The same is true of most community programs. Someone coming into a program at a particular time may have a totally different experience than a similar person entering at a different time, even though the operation of the program is the same for both. A particular participant may encourage everyone around her, and create an overwhelmingly positive atmosphere different from that experienced by participants who enter the program after she has left, for example. It’s very difficult to control for this kind of difference over time, but it’s important to be aware that it can, and often does, exist, and may affect the results of a program evaluation.

• If the organizational or program context and culture are important, then you’ll probably want to compare your results with participants to those in a control group in a similar situation where those factors are different, or are ignored.

There is, of course, a huge range of possibilities here: nearly any design can be adapted to nearly any situation in the right circumstances. This material is meant only to give you a sense of how to start thinking about the issue of design for an evaluation.

Consider what your participants (and staff) will consent to

In addition to the effect that it might have on the results of your evaluation, you might find that a lot of observation can raise protests from participants who feel their privacy is threatened, or from already-overworked staff members who see adding evaluation to their job as just another burden. You may be able to overcome these obstacles, or you may have to compromise – fewer or different kinds of observations, a less intrusive design – in order to be able to conduct the evaluation at all.

There are other reasons that participants might object to observation, or at least intense observation. Potential for embarrassment, a desire for secrecy (to keep their participation in the program from family members or others), even self-protection (in the case of domestic violence, for instance) can contribute to unwillingness to be a participant in the evaluation. Staff members may have some of the same concerns.

There are ways to deal with these issues, but there’s no guarantee that they’ll work. One is to inform participants at the beginning about exactly what you’re hoping to do, listen to their objections, and meet with them (more than once, if necessary) to come up with a satisfactory approach. Staff members are less likely to complain if they’re involved in planning the evaluation, and thus have some say over the frequency and nature of observations. The same is true for participants.Treating everyone’s concerns seriously and including them in the planning process can go a long way toward assuring cooperation.

Consider your time constraints

As we mentioned above, the important thing here is to choose a design that will give you reasonably reliable information. In general, your design doesn’t have to be perfect, but it does have to be good enough to give you a reasonably good indication that changes are actually taking place, and that they are the result of your program. Just how precise you can be is at least partially controlled by the limits on your time placed by funding, program considerations, and other factors.

Time constraints may also be imposed. Some of the most common:

• Program structure. An evaluation may make the most sense if it’s conducted to correspond with a regular program cycle.
• Funding.  If you are funded only for a pilot project, for example, you’ll have to conduct your evaluation within the time span of the funding, and soon enough to show that your program is successful enough to be refunded. A time schedule for evaluation may be part of your grant or contract, especially if the funder is paying for it.
• Participants’ schedules. A rural education program may need to stop for several months a year to allow participants to plant and tend crops, for instance.
• The seriousness of the issue  A delay in understanding whether a violence prevention program is effective may cost lives.
• The availability of professional evaluators. Perhaps the evaluation team can only work during a particular time frame.

Consider your resources

Strategic planners often advise that groups and organizations consider resources last: otherwise they’ll reject many good ideas because they’re too expensive or difficult, rather than trying to find ways to make them work with the resources at hand. Resources include not only money, but also space, materials and equipment, personnel, and skills and expertise. Often, one of these can substitute for another: a staff person with experience in research can take the place of money that would be used to pay a consultant, for example. A partnership with a nearby university could get you not only expertise, but perhaps needed equipment as well.

The lesson here is to begin by determining the best design possible for your purposes, without regard to resources. You may have to settle for somewhat less, but if you start by aiming for what you want, you’re likely to get a lot closer to it than if you assume you can’t possibly get it.

In Summary

The way you design your evaluation research will have a lot to do with how accurate and reliable your results are, and how well you can use them to improve your program or intervention. The design should be one that best addresses key threats to internal validity (whether the intervention caused the change) and external validity (the ability to generalize your results to other situations, communities, and populations).

Common research designs – such as interrupted time series or control group designs– can be adapted to various situations, and combined in various ways to create a design that is both appropriate and feasible for your program. It may be necessary to seek help from a consultant, a university partner, or simply someone with research experience to help identify a design that fits your needs.

A good design will address your evaluation questions, and take into consideration the nature of your program, what program participants and staff will agree to, your time constraints, and the resources you have available for evaluation. It often makes sense to consider resources last, so that you won’t reject good ideas because they seem too expensive or difficult. Once you’ve chosen a design, you can often find a way around a lack of resources to make it a reality.

What do we mean by a design for the evaluation?

___The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know

___An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors

Why should you choose a design for your evaluation?

___So your evaluation will be reliable

___So you can pinpoint areas you need to work on, as well as those that are successful

___So your results are credible

___So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants

___So you can identify unintended consequences (both positive and negative) and correct for them

___So you’ll have a coherent plan and organizing structure for your evaluation

When should you choose a design for your evaluation?

___In the ideal, when you’re planning the program and evaluation, before you start any implementation

___In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters

Who should be involved in choosing a design for your evaluation?

___It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails

How do you select an appropriate design for your evaluation?

___Take into consideration the necessary research considerations

• Understand threats to internal validity ( whether the intervention produced the change)
  • History
  • Maturation
  • The effects of testing or observation on participants
  • Changes in measurement
  • Regression toward the mean
  • The selection of participants
  • The loss of data or participants
  • The nature of change
  • A combination of the effects of two or more of these
• Understand threats to external validity (generalizability of your findings)
  • Interaction of testing or data collection and the program or intervention
  • Interaction of selection procedures and the program or intervention
  • The effects of the research arrangements
  • The interference of multiple programs or interventions

___Understand common research designs

• Pre- and post- single group design
• Interrupted time series design with a single group (simple time series)
• Interrupted time series with multiple groups (multiple time series)
• Control or comparison group

___Choose a design

___Consider your evaluation questions

___Consider the nature of your program

___Consider what your participants and staff will consent to

___Consider your time constraints

___Consider your resources

In previous sections of this chapter, we’ve discussed studying the issue, deciding on a research design, and creating an observational system for gathering information for your evaluation. Now it’s time to collect your data and analyze it – figuring out what it means – so that you can use it to draw some conclusions about your work. In this section, we’ll examine how to do just that.

What do we mean by collecting data?

Essentially, collecting data means putting your design for collecting information into operation. You’ve decided how you’re going to get information – whether by direct observation, interviews, surveys, experiments and testing, or other methods – and now you and/or other observers have to implement your plan. There’s a bit more to collecting data, however. If you are conducting observations, for example, you’ll have to define what you’re observing and arrange to make observations at the right times, so you actually observe what you need to. You’ll have to record the observations in appropriate ways and organize them so they’re optimally useful.

Recording and organizing data may take different forms, depending on the kind of information you’re collecting. The way you collect your data should relate to how you’re planning to analyze and use it. Regardless of what method you decide to use, recording should be done concurrent with data collection if possible, or soon afterwards, so that nothing gets lost and memory doesn’t fade.

Some of the things you might do with the information you collect include:

• Gathering together information from all sources and observations
• Making photocopies of all recording forms, records, audio or video recordings, and any other collected materials, to guard against loss, accidental erasure, or other problems
• Entering narratives, numbers, and other information into a computer program, where they can be arranged and/or worked on in various ways
• Performing any mathematical or similar operations needed to get quantitative information ready for analysis.  These might, for instance, include entering numerical observations into a chart, table, or spreadsheet, or figuring the mean (average), median (midpoint), and/or mode (most frequently occurring) of a set of numbers.
• Transcribing (making an exact, word-for-word text version of) the contents of audio or video recordings
• Coding data (translating data, particularly qualitative data that isn’t expressed in numbers, into a form that allows it to be processed by a specific software program or subjected to statistical analysis)
• Organizing data in ways that make them easier to work with.  How you do this will depend on your research design and your evaluation questions. You might group observations by the dependent variable (indicator of success) they relate to, by individuals or groups of participants, by time, by activity, etc. You might also want to group observations in several different ways, so that you can study interactions among different variables.

There are two kinds of variables in research.  An independent variable (the intervention) is a condition implemented by the researcher or community to see if it will create change and improvement. This could be a program, method, system, or other action.  A dependent variable is what may change as a result of the independent variable or intervention.  A dependent variable could be a behavior, outcome, or other condition.  A smoking cessation program, for example, is an independent variable that may change group members’ smoking behavior, the primary dependent variable.

What do we mean by analyzing data?

Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc. that can be found within it. That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting.  It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. The point, in terms of your evaluation, is to get an accurate assessment in order to better understand your work and its effects on those you’re concerned with, or in order to better understand the overall situation.

There are two kinds of data you’re apt to be working with, although not all evaluations will necessarily include both. Quantitative data refer to the information that is collected as, or can be translated into, numbers, which can then be displayed and analyzed mathematically. Qualitative data are collected as descriptions, anecdotes, opinions, quotes, interpretations, etc., and are generally either not able to be reduced to numbers, or are considered more valuable or informative if left as narratives. As you might expect, quantitative and qualitative information needs to be analyzed differently.

Quantitative data

Quantitative data are typically collected directly as numbers. Some examples include:

• The frequency (rate, duration) of specific behaviors or conditions
• Test scores (e.g., scores/levels of knowledge, skill, etc.)
• Survey results (e.g., reported behavior, or outcomes to environmental conditions; ratings of satisfaction, stress, etc.)
• Numbers or percentages of people with certain characteristics in a population (diagnosed with diabetes, unemployed, Spanish-speaking, under age 14, grade of school completed, etc.)

Data can also be collected in forms other than numbers, and turned into quantitative data  for analysis. Researchers can count the number of times an event is documented in interviews or records, for instance, or assign numbers to the levels of intensity of an observed event or behavior. For instance, community initiatives often want to document the amount and intensity of environmental changes they bring about – the new programs and policies that result from their efforts. Whether or not this kind of translation is necessary or useful depends on the nature of what you’re observing and on the kinds of questions your evaluation is meant to answer.

Quantitative data is usually subjected to statistical procedures such as calculating the mean or average number of times an event or behavior occurs (per day, month, year). These operations, because numbers are “hard” data and not interpretation, can give definitive, or nearly definitive, answers to different questions. Various kinds of quantitative analysis can indicate changes in a dependent variable related to – frequency, duration, timing (when particular things happen), intensity, level, etc. They can allow you to compare those changes to one another, to changes in another variable, or to changes in another population. They might be able to tell you, at a particular degree of reliability, whether those changes are likely to have been caused by your intervention or program, or by another factor, known or unknown. And they can identify relationships among different variables, which may or may not mean that one causes another.

Qualitative data

Unlike numbers or “hard data,” qualitative information tends to be “soft,” meaning it can’t always be reduced to something definite. That is in some ways a weakness, but it’s also a strength. A number may tell you how well a student did on a test; the look on her face after seeing her grade, however, may tell you even more about the effect of that result on her. That look can’t be translated to a number, nor can a teacher’s knowledge of that student’s history, progress, and experience, all of which go into the teacher’s interpretation of that look. And that interpretation may be far more valuable in helping that student succeed than knowing her grade or numerical score on the test.

Qualitative data can sometimes be changed into numbers, usually by counting the number of times specific things occur in the course of observations or interviews, or by assigning numbers or ratings to dimensions (e.g., importance, satisfaction, ease of use).

The challenges of translating qualitative into quantitative data have to do with the human factor.  Even if most people agree on what 1 (lowest) or 5 (highest) means in regard to rating “satisfaction” with a program, ratings of 2, 3, and 4 may be very different for different people.  Furthermore, the numbers say nothing about why people reported the way they did. One may dislike the program because of the content, the facilitator, the time of day, etc. The same may be true when you’re counting instances of the mention of an event, such as the onset of a new policy or program in a community based on interviews or archival records.  Where one person might see a change in program he considers important another may omit it due to perceived unimportance.

Qualitative data can sometimes tell you things that quantitative data can’t.  It may reveal why certain methods are working or not working, whether part of what you’re doing conflicts with participants’ culture, what participants see as important, etc.  It may also show you patterns – in behavior, physical or social environment, or other factors – that the numbers in your quantitative data don’t, and occasionally even identify variables that researchers weren’t aware of.

It is often helpful to collect both quantitative and qualitative information.

Quantitative analysis is considered to be objective – without any human bias attached to it – because it depends on the comparison of numbers according to mathematical computations.  Analysis of qualitative data is generally accomplished by methods more subjective – dependent on people’s opinions, knowledge, assumptions, and inferences (and therefore biases) – than that of quantitative data.  The identification of patterns, the interpretation of people’s statements or other communication, the spotting of trends – all of these can be influenced by the way the researcher sees the world.  Be aware, however, that quantitative analysis is influenced by a number of subjective factors as well.  What the researcher chooses to measure, the accuracy of the observations, and the way the research is structured to ask only particular questions can all influence the results, as can the researcher’s understanding and interpretation of the subsequent analyses.

Why should you collect and analyze data for your evaluation?

Part of the answer here is that not every organization – particularly small community-based or non-governmental ones – will necessarily have extensive resources to conduct a formal evaluation.  They may have to be content with less formal evaluations, which can still be extremely helpful in providing direction for a program or intervention.  An informal evaluation will involve some data gathering and analysis. This data collection and sensemaking is critical to an initiative and its future success, and has a number of advantages.

• The data can show whether there was any significant change in the dependent variable(s) you hoped to influence. Collecting and analyzing data helps you see whether your intervention brought about the desired results

The term “significance” has a specific meaning when you’re discussing statistics.  The level of significance of a statistical result is the level of confidence you can have in the answer you get.  Generally, researchers don’t consider a result significant unless it shows at least a 95% certainty that it’s correct (called the .05 level of significance, since there’s a 5% chance that it’s wrong).  The level of significance is built into the statistical formulas: once you get a mathematical result, a table (or the software you’re using) will tell you the level of significance.

Thus, if data analysis finds that the independent variable (the intervention) influenced the dependent variable at the .05 level of significance, it means there’s a 95% probability or likelihood that your program or intervention had the desired effect.  The .05 level is generally considered a reasonable result, and the .01 level (99% probability) is considered about as close to certainty as you are likely to get. A 95% level of certainty doesn’t mean that the program works on 95% of participants, or that it will work 95% of the time.  It means that there’s only a 5% possibility that it isn’t actually what’s influencing the dependent variable(s) and causing the changes that it seems to be associated with.

• They can uncover factors that may be associated with changes in the dependent variable(s). Data analyses may help discover unexpected influences; for instance, that the effort was twice as large for those participants who also were a part of a support group. This can be used to identify key aspects of implementation.
• They can show connections between or among various factors that may have an effect on the results of your evaluation. Some types of statistical procedures look for connections (“correlations” is the research term) among variables.  Certain dependent variables may change when others do.  These changes may be similar – i.e., both variables increase or decrease (e.g., as children’s proficiency at reading increases, the amount of reading they do also increases).  Or the opposite may be observed – i.e. the two variables change in opposite directions (as the amount of exercise they engage in increases, peoples’ weight decreases).  Correlations don’t mean that one variable causes another, or that they both have the same cause, but they can provide valuable information about associations to expect in an evaluation.
• They can help shed light on the reasons that your work was effective or, perhaps, less effective than you’d hoped.  By combining quantitative and qualitative analysis, you can often determine not only what worked or didn’t, but why.  The effect of cultural issues, how well methods are used, the appropriateness of your approach for the population – these as well as other factors that influence success can be highlighted by careful data collection and analysis.  This knowledge gives you a basis for adapting and changing what you do to make it more likely you’ll achieve the desired outcomes in the future.
• They can provide you with credible evidence to show stakeholders that your program is successful, or that you’ve uncovered, and are addressing limitations. Stakeholders, such as funders and community boards, want to know their investments are well spent. Showing evidence of intermediate outcomes (e.g. new programs and policies) and longer-term outcomes (e.g., improvements in education or health indicators) is becoming increasingly important to receiving – and retaining – funding.
• Their use shows that you’re serious about evaluation and about improving your work. Being a good trustee or steward of community investment includes regular review of data regarding progress and improvement.
• They can show the field what you’re learning, and thus pave the way for others to implement successful methods and approaches. In that way, you’ll be helping to improve community efforts and, ultimately, quality of life for people who benefit.

When and by whom should data be collected and analyzed?

As far as data collection goes, the “when” part of this question is relatively simple: data collection should start no later than when you begin your work – or before you begin in order to establish a baseline or starting point – and continue throughout.  Ideally, you should collect data for a period of time before you start your program or intervention in order to determine if there are any trends in the data before the onset of the intervention. Additionally, in order to gauge your program’s longer-term effects, you should collect follow-up data for a period of time following the conclusion of the program.

The timing of analysis can be looked at in at least two ways: One is that it’s best to analyze your information when you’ve collected all of it, so you can look at it as a whole. The other is that if you analyze it as you go along, you’ll be able to adjust your thinking about what information you actually need, and to adjust your program to respond to the information you’re getting. Which of these approaches you take depends on your research purposes.  If you’re more concerned with a summative evaluation – finding out whether your approach was effective, you might be more inclined toward the first.  If you’re oriented toward improvement – a formative evaluation – we recommend gathering information along the way. Both approaches are legitimate, but ongoing data collection and review can particularly lead to improvements in your work.

The “who” question can be more complex. If you’re reasonably familiar with statistics and statistical procedures, and you have the resources in time, money, and personnel, it’s likely that you’ll do a somewhat formal study, using standard statistical tests. (There’s a great deal of software – both for sale and free or open-source – available to help you.)

If that’s not the case, you have some choices:

• You can hire or find a volunteer outside evaluator, such as from a nearby college or university, to take care of data collection and/or analysis for you.
• You can conduct a less formal evaluation. Your results may not be as sophisticated as if you subjected them to rigorous statistical procedures, but they can still tell you a lot about your program.  Just the numbers – the number of dropouts (and when most dropped out), for instance, or the characteristics of the people you serve – can give you important and usable information.
• You can try to learn enough about statistics and statistical software to conduct a formal evaluation yourself. (Take a course, for example.)
• You can collect the data and then send it off to someone – a university program, a friendly statistician or researcher, or someone you hire – to process it for you.
• You can collect and rely largely on qualitative data.  Whether this is an option depends to a large extent on what your program is about. You wouldn’t want to conduct a formal evaluation of effectiveness of a new medication using only qualitative data, but you might be able to draw some reasonable conclusions about use or compliance patterns from qualitative information.
• If possible, use a randomized or closely matched control group for comparison. If your control is properly structured, you can draw some fairly reliable conclusions simply by comparing its results to those of your intervention group.  Again, these results won’t be as reliable as if the comparison were made using statistical procedures, but they can point you in the right direction.  It’s fairly easy to tell whether or not there’s a major difference between the numbers for the two or more groups.  If 95% of the students in your class passed the test, and only 60% of those in a similar but uninstructed control group did, you can be pretty sure that your class made a difference in some way, although you may not be able to tell exactly what it was that mattered. By the same token, if 72% of your students passed and 70% of the control group did as well, it seems pretty clear that your instruction had essentially no effect, if the groups were starting from approximately the same place.

Who should actually collect and analyze data also depends on the form of your evaluation.  If you’re doing a participatory evaluation, much of the data collection - and analyzing - will be done by community members or program participants themselves. If you’re conducting an evaluation in which the observation is specialized, the data collectors may be staff members, professionals, highly trained volunteers, or others with specific skills or training (graduate students, for example).  Analysis also could be accomplished by a participatory process. Even where complicated statistical procedures are necessary, participants and/or community members might be involved in sorting out what those results actually mean once the math is done and the results are in. Another way analysis can be accomplished is by professionals or other trained individuals, depending upon the nature of the data to be analyzed, the methods of analysis, and the level of sophistication aimed at in the conclusions.

How do you collect and analyze data?

Whether your evaluation includes formal or informal research procedures, you’ll still have to collect and analyze data, and there are some basic steps you can take to do so.

Implement your measurement system

We've previously discussed designing an observational system to gather information. Now it’s time to put that system in place.

• Clearly define and describe what measurements or observations are needed. The definition and description should be clear enough to enable observers to agree on what they’re observing and reliably record data in the same way.
• Select and train observers. Particularly if this is part of a participatory process, observers need training to know what to record; to recognize key behaviors, events, and conditions; and to reach an acceptable level of inter-rater reliability (agreement among observers).
• Conduct observations at the appropriate times for the appropriate period of time. This may include reviewing archival material; conducting interviews, surveys, or focus groups; engaging in direct observation; etc.
• Record data in the agreed-upon ways. These may include pencil and paper, computer (using a laptop or handheld device in the field, entering numbers into a program, etc.), audio or video, journals, etc.

Organize the data you’ve collected

How you do this depends on what you’re planning to do with it, and on what you’re interested in.

• Enter any necessary data into the computer. This may mean simply typing comments, descriptions, etc., into a word processing program, or entering various kinds of information (possibly including audio and video) into a database, spreadsheet, a GIS (Geographic Information Systems) program, or some other type of software or file.
• Transcribe any audio- or videotapes. This makes them easier to work with and copy, and allows the opportunity to clarify any hard-to-understand passages of speech.
• Score any tests and record the scores appropriately.
• Sort your information in ways appropriate to your interest. This may include sorting by category of observation, by event, by place, by individual, by group, by the time of observation, or by a combination or some other standard.
• When possible, necessary, and appropriate, transform qualitative into quantitative data. This might involve, for example, counting the number of times specific issues were mentioned in interviews, or how often certain behaviors were observed.

Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

We’ve referred several times to statistical procedures that you can apply to quantitative data. If you have the right numbers, you can find out a great deal about whether your program is causing or contributing to change and improvement, what that change is, whether there are any expected or unexpected connections among variables, how your group compares to another you’re measuring, etc.

There are other excellent possibilities for analysis besides statistical procedures, however. A few include:

• Simple counting, graphing and visual inspection of frequency or rates of behavior, events, etc., over time.
• Using visual inspection of patterns over time to identify discontinuities (marked increases, decreases) in the measures over time (sessions, weeks, months).
• Calculating the mean (average), median (midpoint), and/or mode (most frequent) of a series of measurements or observations. What was the average blood pressure, for instance, of people who exercised 30 minutes a day at least five days a week, as opposed to that of people who exercised two days a week or less?
• Using qualitative interviews, conversations, and participant observation to observe (and track changes in) the people or situation. Journals can be particularly revealing in this area because they record people’s experiences and reflections over time.
• Finding patterns in qualitative data.  If many people refer to similar problems or barriers, these may be important in understanding the issue, determining what works or doesn’t work and why, or more.
• Comparing actual results to previously determined goals or benchmarks. One measure of success might be meeting a goal for planning or program implementation, for example.

Take note of any significant or interesting results

Depending on the nature of your research, results may be statistically significant (the 95% or better certainty that we discussed earlier), or simply important or unusual.  They may or may not be socially significant (i.e., large enough to solve the problem).

There are a number of different kinds of results you might be looking for.

• Differences within people or groups. If you have repeated measurements for individuals/groups over time, we can see if there are marked increases/decreases in the (frequency, rate) of behavior (events, etc.) following introduction of the program or intervention. When the effects are seen when and only when the intervention is introduced – and if the intervention is staggered (delayed) across people or groups – this increases our confidence that the intervention, and not something else, is producing the observed effects.
• Differences between or among two or more groups.  If you have one or more randomized control groups in a formal study (groups that are drawn at random from the same population as the group in your program, but are not getting the same program or intervention, or are getting none at all), then the statistical significance of differences between or among the groups should tell you whether your program has any more influence on the dependent variable(s) than what’s experienced by the other groups.
• Results that show statistically significant changes. With or without a control or comparison group, many statistical procedures can tell you whether changes in dependent variables are truly significant (or not likely due to chance). These results may say nothing about the causes of the change (or they may, depending on how you’ve structured your evaluation), but they do tell you what’s happening, and give you a place to start.
• Correlations. Correlation means that there are connections between or among two or more variables. Correlations can sometimes point to important relationships you might not have predicted. Sometimes they can shed light on the issue itself, and sometimes on the effects of a group’s cultural practices. In some cases, they can highlight potential causes of an issue or condition, and thus pave the way for future interventions.

Correlation between variables doesn’t tell you that one necessarily causes the other, but simply that changes in one have a relationship to changes in the other.  Among American teenagers, for instance, there is probably a fairly high correlation between an increase in body size and an understanding of algebra.  This is not because one causes the other, but rather the result of the fact that American schools tend to begin teaching algebra in the seventh, eighth, or ninth grades, a time when many 12-, 13-, and 14-year-olds are naturally experiencing a growth spurt.

On the other hand, correlations can reveal important connections.  A very high correlation between, for instance, the use of a particular medication and the onset of depression might lead to the withdrawal of that medication, or at least a study of its side effects, and increased awareness and caution among doctors who prescribe it.  A very high correlation between gang membership and having a parent with a substance use problem may not reveal a direct cause-and-effect relationship, but may tell you something important about who is more at risk for substance use.

• Patterns. In both quantitative and qualitative information, patterns often emerge: certain health conditions seem to cluster in particular geographical areas; people from a particular group behave in similar ways; etc.  These patterns may not be specifically what you were looking for or expected to find, but they may either be important in themselves or shed light on the areas you’re interested in.  In some cases, you may need to subject them to statistical procedures (regression analysis, for example) to see if, in fact, they’re random, or if they constitute actual patterns.
• Obvious important findings.  Whether as a result of statistical analysis, or of examination of your data and application of logic, some findings may stand out.  If 70% of a group of overweight participants in a healthy eating and physical activity program lowered their weight and blood pressure significantly, compared to only 20% of a similar group not in the program, you can probably assume that program may have been effective.  If there’s no change whatsoever in education outcomes after two years of your education program, then you’re either running an ineffective program, or you’re simply not reaching those who are most likely to have poorer outcomes (which can also be interpreted to mean you’re running an ineffective program.)

Not all important findings will necessarily tell you whether your program worked, or what is the most effective method.  It might be obvious from your data collection, for instance, that, while violence or roadway injuries may not be seen as a problem citywide, they are much higher in one or more particular areas, or that the rates of diabetes are markedly higher for particular groups or those living in areas with greater disparities of income. If you have the resources, it’s wise to look at the results of your research in a number of different ways, both to find out how to improve your program, and to learn what else you might do to affect the issue.

Interpret the results

Once you’ve organized your results and run them through whatever statistical or other analysis you’ve planned for, it’s time to figure out what they mean for your evaluation. Probably the most common question that evaluation research is directed toward is whether the program being evaluated works or makes a difference. In research terms, that often translates to “What were the effects of the independent variable (the program, intervention, etc.) on the dependent variable(s) (the behavior, conditions, or other factors it was meant to change)?”  There are a number of possible answers to this question:

• Your program had exactly the effects on the dependent variable(s) you expected and hoped it would. Statistics or other analysis showed clear positive effects at a high level of significance for the people in your program and – if you used a multiple-group design – none, or far fewer, of the same effects for a similar control group and/or for a group that received a different intervention with the same purpose. Your early childhood education program, for instance, greatly increased development outcomes for children in the community, and also contributed to an increase in the percentage of children succeeding in school.
• Your program had no effect.  Your program produced no significant results on the dependent variable, whether alone or compared to other groups. This would mean no change as a result of your program or intervention.
• Your program had a negative effect. For instance, intimate partner violence increased (or at least appeared to) as a result of your intervention. (It is relatively common for reported events, such as violence or injury, to increase when the intervention results in improved surveillance and ease of reporting).
• Your program had the effects you hoped for and other effects as well.
  • These effects might be positive. Your youth violence prevention program, for instance, might have resulted in greatly reduced violence among teens, and might also have resulted in significantly improved academic performance for the kids involved.
  • These effects might be neutral. The same youth violence prevention program might somehow result in youth watching TV more often after school.
  • These effects might be negative. (These effects are usually called unintended consequences.) Youth violence might decrease significantly, but the incidence of teen pregnancies or alcohol consumption among youth in the program might increase significantly at the same time.
  • These effects might be multiple, or mixed.For instance, a program to reduce HIV/AIDS might lower rates of unprotected sex but might also increase conflict and instances of partner violence. Your program had no effect or a negative effect and other effects as well. As with programs with positive effects, these might be positive, neutral, or negative; single or multiple; or consistent or mixed.

If your analysis gives you a clear indication that what you’re doing is accomplishing your purposes, interpretation is relatively simple: You should keep doing it, while trying out ways to make it even more effective, or while aiming at other related issues as well.

As we discuss elsewhere in the Community Tool Box, good programs are dynamic -- constantly striving to improve, rather than assuming that what they’re doing is as good as it can be.

If your analysis shows that your program is ineffective or negative, however – or, for that matter, if a positive analysis leaves you wondering how to make your successful efforts still more successful – interpretation becomes more complex. Are you using an absolutely wrong approach? Are you using an approach that could be effective, but is poorly implement? Is there a particular contributing factor you’re failing to take into account?  Are there barriers to success – of culture, experience, personal characteristics, systematic discrimination – present in the population from which participants are drawn?  Are there particular components or elements you can change to make your program more effective, or should you start again from scratch?  What should you address to make a good program better?

Careful and insightful interpretation of your data may allow you to answer questions like these. You may be able to use correlations, for instance, to generate hypotheses about your results. If positive or negative changes in particular variables are consistently associated with positive or negative changes in other variables, the two may be connected. (The word “may” is important here. The two may be connected, but they may not, or both may be related to a third variable that you’re not aware of or that you consider trivial.) Such a connection can point the way toward a factor (e.g., access to support) that is causing the changes in both variables, and that must be addressed to make your program successful. Correlations may also indicate patterns in your data, or may lead to an unexpected way of looking at the issue you’re addressing.

You can often use qualitative data to understand the meaning of an intervention, and people’s reactions to the results.The observation that participants are continually suffering from a variety of health problems may be traced, through qualitative data, to nutrition problems (due either to poverty or ignorance) or to lack of access to health services, or to cultural restrictions (some Muslim women may be unwilling – or unable because of family prohibition – to accept care and treatment from male doctors, for example).

Once you have organized your data, both statistical results and anything that can’t be analyzed statistically need to be analyzed logically. This may not give you convincing information but it will almost undoubtedly give you some ideas to follow up on, and some indications of connections and avenues you might not yet have considered. It will also show you some additional results – people reacting differently than before to the program, for example. The numbers can tell you whether there is change, but they can’t always tell you what causes it or why (although they sometimes can), or why some people benefit while others don’t. Those are often matters for logical analysis, or critical thinking.
 
Analyzing and interpreting the data you’ve collected brings you, in a sense, back to the beginning. You can use the information you’ve gained to adjust and improve your program or intervention, evaluate it again, and use that information to adjust and improve it further, for as long as it runs. You have to keep up the process to ensure that you’re doing the best work you can and encouraging changes in individuals, systems, and policies that make for a better and healthier community.

You have to become a cultural detective to understand your initiative, and, in some ways, every evaluation is an anthropological study.

In Summary

The heart of evaluation research is gathering information about the program or intervention you’re evaluating and analyzing it to determine what it tells you about the effectiveness of what you’re doing, as well as about how you can maintain and improve that effectiveness.

Collecting quantitative data – information expressed in numbers – and subjecting it to a visual inspection or formal statistical analysis can tell you whether your work is having the desired effect, and may be able to tell you why or why not as well.  It can also highlight connections (correlations) among variables, and call attention to factors you may not have considered.

Collecting and analyzing qualitative data – interviews, descriptions of environmental factors, or events, and circumstances – can provide insight into how participants experience the issue you’re addressing, what barriers and advantages they experience, and what you might change or add to improve what you do.

Once you’ve gained the knowledge that your information provides, it’s time to start the process again. Use what you’ve learned to continue to evaluate what you do by collecting and analyzing data, and continually improve your program.

What do we mean by collecting data?

___Collecting data means putting your design for a measurement system into operation

___Collecting data involves gathering information through observation, interviews, testing, surveys, and/or other means; recording it in appropriate ways; and organizing it so that it’s easier to work with

What do we mean by analyzing data?

___Analyzing data involves examining the information you’ve collected in ways that reveal the relationships, patterns, trends, etc. that can be found within it

___Data can be quantitative (collected as numbers) or qualitative (collected as narrative information, records, journal notes, etc.)

___Quantitative data is usually analyzed by subjecting it to one or more graphical displays or statistical operations that demonstrate the significance of relationships among variables

___Data are also usually analyzed logically, by looking for patterns and relationships revealed within them

___Qualitative data can sometimes be turned into quantitative data by, for instance, counting (e.g., the number of times a behavior occurs in various circumstances) or by rating on a number scale such dimensions as importance, satisfaction, or quality (e.g., the quality of housing or quality of life in neighborhoods)

___A combination of quantitative and qualitative data often yields the best overall picture

Why should you collect and analyze data for your evaluation?

___This can show whether or not there was actually any significant change in the dependent variable(s) you hoped to influence

___This can show connections between or among various factors that may have an effect on the results of your evaluation

___This can imply or show the reasons that your work was effective or ineffective

___This can provide you with credible evidence to show funders and the community that your program is successful, or that you’ve uncovered, and are fixing, the elements that are barriers to success

___This can show that you’re serious about evaluation and about improving your work

___This can show the field that what you’re doing works well, and thus pave the way for others to use similar methods and approaches as best practices

When and by whom should data be collected and analyzed?

___Data collection should start no later than when you begin your work and continue throughout

___If you want to see if any change is part of a long-term trend, you should collect data for some time before your program actually starts

___If you want to understand long-term effects, you should collect data on participants for some time after they leave your program

___Data should be collected and analyzed by people who are capable of doing so

___Data collection and analysis can be done by anyone from community members who have been trained to professionals with experience in conducting studies

How do you collect and analyze data?

___Implement the observational system you’ve planned

___Organize the data you’ve collected

___Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

___Take note of any significant or interesting results

___Interpret the results

Example #1: Santa Monica’s Use of Data to Lift All Boats: A Deeper Look

Four years ago, Santa Monica took a data-driven approach to become a true City of Wellbeing where everyone thrives. The city used a Wellbeing Index to produce a data-based guide for steering policy. The city used existing information from city departments and fresh input from surveys. The Wellbeing index sets a framework for priorities and promotes initiatives on health and wellness.

Contributed by Lia Thompson, University of Kansas, Community Tool Box Intern.

 

 

• What is ethnographic information?

• Why might ethnographic information be important to evaluation?

• When might you want to collect ethnographic information?

• Who should collect and interpret ethnographic information?

• How do you gather ethnographic information?

• How do you interpret ethnographic information?

Someone watching a baseball game or cricket match for the first time might be baffled by seeing a batter perform a series of actions before each pitch – touching his helmet, knocking the dirt off his shoes, stepping out of the batters’ box. Any long-time fan, however, knows what’s happening: first, many players have superstitious or calming rituals they perform when batting; second, some of the batter’s motions may be acknowledgements of signals from coaches; and third, there is an intense psychological battle between hitter and pitcher that plays itself out in stalling, distracting motions and other actions meant to make it difficult for the opposition. Without this knowledge, the first-time observer sees only a series of random, seemingly purposeless actions and could be left with the impression that players are afflicted with a strange disease.

Understanding why people do what they do can be tremendously important in trying to understand a culture. When anthropologists want to understand a culture, they often immerse themselves in it, so they can study it from the inside. Margaret Mead popularized this type of research in the 1920s in Samoa, where she lived for nine months as part of an island community. A researcher who is embedded in a culture in this way can gain insight not only into what happens – how people go about their daily lives, what rituals they perform, what their celebrations are like, etc. – but also how members of the culture view what happens. They can begin to piece together the culture’s structure – the beliefs and assumptions that underlie everything people do and the world view that creates and supports them.

Depending on what you’re trying to change, your evaluation may have to look at more than just the behavior of individuals. The whole context of the community may be involved if, for instance, you’re attempting to address social exclusion based on race or gender or to alter a culture of violence. Deep knowledge of a community, or a segment of it, can’t necessarily be gained by administering tests or counting the number of times individuals perform certain actions. Rather, it takes an understanding of the community’s whole world view and what leads to it – whether religion, social or political factors, economics, some combination, or something else entirely. That kind of deep knowledge may best be gained by immersion in the culture of the community you’re serving. This section will explore the use of ethnographic research – the kind of cultural immersion we’ve been describing – as a method for evaluation.

What is ethnographic information?

Ethnographic information is data about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view as much as possible. It’s meant to provide an understanding of the culture from the inside – to describe and explain its assumptions, customs, and accepted behavior from the point of view of those who live by them.

These can include community norms, health conditions and knowledge, political realities, religion, economics, and world view.

The terms “culture,” “group,” and “community” are all used in this section to refer to a group that can be seen as having its own set of norms and practices and the formal or informal rules that guide them. By that definition, we may be referring to a culture as large as that of a whole ethnic or religious group (e.g., Latinos, Hindus, Navajos) or as small as that of a single elementary school classroom. It may encompass people of many backgrounds (e.g., English as a Second or Other Language students; low-income families) or people from a single neighborhood or informal settlement. No culture or group or community is completely homogeneous, of course, – not everyone thinks and believes exactly the same things as everyone else – but you can generally find basic cultural elements on which most people agree, and those are what define the culture.

Community norms. These encompass what the community or group considers normal and acceptable in everyday behavior and activity. We can look at community norms through a number of filters:

• Gender roles. These define what men and women are each expected to do, how they’re expected to act, dress, etc., and how they are expected to relate to one another in different circumstances.
• Social structure. Social structure establishes who is accorded higher status in the community, how one gains status, how social groups are made up and what forms they take, and who are the social arbiters – those who make the rules or set the standards. It also defines who is considered lower status, who is socially excluded, and how possible it is to improve one’s status through merit, entrepreneurship, illegal activity, education, etc. It also determines whether and how the community’s world is divided up into social classes or hierarchies, and whether some are perceived to have higher status than others.
• Behavior. What is considered acceptable behavior and language in public? This encompasses, for example, how people greet one another (and how much this depends on status, gender, age, etc.). It also refers to how they respond to real or perceived compliments and insults (and what are considered compliments and insults), ritual or other behavior unique to the population, and behavior accepted or condoned (or encouraged) by the community. Most communities and groups also set standards for behavior that is so unacceptable or harmful to the group, whether public or private, that it is punished in official ways, including exclusion from the group.

In some communities and cultures, behaviors that people in other communities and cultures have come to consider unhealthy or antisocial are seen as acceptable – if sometimes regrettable – or normal. Domestic and other physical violence, alcoholism, racism and racial discrimination, smoking, traffic violations – even slavery – are examples of illegal, unethical, unhealthy, or undesirable behaviors that have been or still are tolerated in some communities.

Understanding the level of tolerance or encouragement for these kinds of behaviors is absolutely necessary for understanding how to change them. An ethnographic investigation can show you how people view your efforts, and what you can do to make it more likely that they’ll respond to your program or initiative.

Health conditions and knowledge. What do the folks you’re concerned with know about health and health conditions in general and in the community, and what healthy (or unhealthy) practices do they engage in? Some areas to examine:

• Knowledge about and practices related to prenatal, infant, and child health.
• The understanding and frequency of the effects of drugs, alcohol, and tobacco, and of addiction in general.
• Awareness of chronic or other health conditions that are common in the community, and of environmental threats or benefits to health.
• Knowledge about nutrition and physical activity, and their relationship to health, as well as practices that reflect that knowledge.
• Engagement in regular health screenings, vaccinations, and other health protection strategies.
• Relationships with a regular primary care provider whom people see regularly.
• The percentage of the population with access to health care.

In the developing world, how many people have access to clean drinking water, sanitation, and basic medications? Do they know how to treat dysentery and other common diseases in order to prevent infant mortality? How far must they go to see a health care provider, and will they consider doing so?

 

Power and Political realities. Who holds positions of power? Who are the opinion leaders and policymakers? How is power exercised – formally or informally, fairly or ruthlessly, for self-interest or the public good? How are civic goals or actions accomplished (or are they at all)? Who is or feels powerless? Whose concerns are most often attended to? Are laws enforced differently for some people than for others? The answers to these and similar questions can help paint a clear picture of the power relationships and political realities of a community or group.

Religion. Faith can be a complex issue, since the extent of its influence, depending upon the population, can range from none to all-encompassing. How large a role does religion play in the daily life of the community? How does religion affect daily life? Asking more specific questions along these lines can help you develop a picture of religion’s role in the community:

• Are most people in the community of one faith, or is it made up of members of a number of different faiths, as well as many with no religious affiliation?
• What roles do religious institutions play – spiritual guide, social center, political adviser, information source, community leader (in the person of the clergy), or some or all of the above?
• If most of the community subscribes to a particular religion, how inclusive and tolerant is it of different beliefs?
• What are the major elements of the belief system (e.g. tolerance, alms-giving)?
• Do people try to convert others?
• Do they understand and believe in the separation of church and state?

Economics. What is the community or group’s economic situation?

• Is there a problem with unemployment or underemployment?
• Is the community dependent on a single industry, occupation, or employer (e.g., manufacturing, farming, a military base)?
• Is there discrimination in hiring or promotion? Against or in favor of whom?
• How does the community view education in relation to economic opportunity, and how good is its education system?
• How strong is its work ethic?
• How easy is it to advance economically?
• Is decent housing affordable and available?
• Are most people able to provide for their families’ basic needs, or is poverty a serious problem? Homelessness? Hunger?

“Basic needs” is a relative term. In a village in a developing country, it may mean one or two meals a day, a simple one-room shelter, access to water and firewood or other cooking fuel, one change of clothing (which may or may not include shoes), and occasional basic health care. In the developed world, it usually means three substantial meals a day; a solid, enclosed, multi-room dwelling that includes a modern kitchen and bathroom; decent clothes for all weather conditions; regular access to health care; and even such amenities as a telephone, a TV set, and – at least in rural areas – a vehicle. The definition of “basic needs” (beyond what it takes simply to survive) depends on what is normal for the rest of the society.

• How large are the disparities in income and wealth? What groups experience greater disparities?

World view. This idea overlaps with several of the categories above and encompasses the way individuals in a community (and, therefore, often the community as a whole) see themselves and their lives. A “world view” is really the set of assumptions about the “rightness” of things that everyone applies to the world around her. Some of the most important elements of world view are:

• Attitudes toward authority. There are three elements here: how authority is defined; which definition, if any, is considered legitimate; and how authority is viewed and treated. The definitions can vary from authority that comes from official position (judges, police, teachers, politicians, bureaucrats) to authority enforced by physical or economic power (warlords, gang leaders, employers and supervisors) to authority conferred by knowledge or respect and position in the social structure (elders, doctors, business leaders, clergy, academics, individuals known for their integrity and intelligence). Who is considered legitimate authority depends on loyalty and solidarity (you give your allegiance to “one of your own”). So the gang leader of the same ethnic background as most people in the neighborhood may wield more authority than the police. It also relates to who has real power (that same gang leader can have you beaten – or worse – if you disobey) and who can best advance the community’s interests. For instance, many Palestinians voted for Hamas in the 2006 election not specifically because they supported the violent overthrow of Israel, but because they believed that Hamas could improve economic conditions for Palestinians in the West Bank and Gaza. The basic view of authority might range from unquestioning respect (authority is always right and must be trusted and obeyed) to profound lack of respect (authority is always self-serving and wrong and must be resisted actively or passively).

Obedience to authority has been of tremendous importance as a topic in history. The obvious example is that of Nazi Germany in the 1930s and ’40s, where much of the population blindly followed an authoritarian government into world war, wholesale murder, and genocide. We have since apparently seen many other examples, where blind obedience to authority – or the resistance to that obedience – has led to violence, overthrow of governments, and yet more genocide. Street and gang violence can be a result of ignoring authority altogether. The questions of how a culture views authority and whom it considers to be legitimate authority are extremely significant, and may have a great deal to do with your evaluation and the success of your initiative.

• Sense of efficacy and control of the environment. Do people see their lives as totally or mostly under their own control, or out of their control entirely? Do they feel their actions make a difference, or that they can effect change in their lives or in the community? If not with them, where do they think control lies? With politicians? With the wealthy? With those who exercise control by force? If the population is made up of many young people (or socially excluded groups), what is their sense of control or lack of control?

Control might be seen to reside in a person or a group of people in “official” authority (the President, the landlord, the school principal, the boss, parents, police); in an institution or set of institutions (government, welfare, the university, the church, the army); in a set of legal, ethical, or religious principles (the Constitution, Islam, the law, the Hippocratic Oath); in the conventions of a group of which the individual is a member (Latinos, the football or soccer team, upper class, the Catholic Church, etc.); or in the decisions and actions of the individual herself. The way an individual sees her ability to control her life depends on a number of factors, including family background and upbringing, education, emotional issues, and life experience. Obviously, perspectives on this issue will differ among individuals even in a highly homogeneous community, but community norms generally reflect the views of most community members.

• Relationships and trust. Do members of the population see most others as generally trustworthy, or is trust reserved only for family or for those within a small inner circle? Are marital (or, among youth, sexual) relationships viewed differently by men and women, and, if so, how? How are marital relationships viewed in general (positive, negative, mutually supportive, exploitative, etc.)? Are children indulged, beaten, expected to work, treasured, seen as burdens? Are there spoken or unspoken rules for economic and social relationships, and relationships among people of different economic and social classes?
• Obligations and rights. Do most members of the community or group feel obligated to consider the general interest as well as their own? Do they see the group as a community, or does their sense of obligation extend only to family and/or close friends? What do they expect from the community or group? What rights do they feel they can exercise, both in the larger society and in the community?​

The issue of rights and obligations can vary tremendously with the nature of the group or community in question, and with the nature of the society that group is part of. In a farming village in a developing country, people may grow up with a sense of obligation to others because everyone in the village is dependent on everyone else. If one family’s crops fail, the village may feel obligated to feed them, knowing that the next year, it could be one of the helping families that is without food. The same might be true, at least to a certain extent, in small towns in the more industrialized world.

In large cities, neighborhoods may be like villages and small towns, if many residents have grown up there or lived there for many years, or they may be impersonal places where people seldom know their neighbors and feel very little sense of obligation to them. More often, if people feel an obligation at all, it’s to their particular group, however it’s defined, and/or to the society as a whole. Obligation becomes more an expression of principle, whether religious or ethical/philosophical, than a personal connection to a group of others with whom one lives.

The same may be true for rights. In the Global North, we take for granted a certain set of political rights under the law – rights to property, to political choice (voting), to fair and equal treatment under the law, etc. When those rights don’t exist or are often violated, people may define their rights more locally, in terms of the obligations owed to them by those in their group or community (I helped your family get through a bad time last year, and I have a right to your help this year.)

• Tolerance and inclusiveness. Is the community or group inclusive – does it welcome those from other communities and cultures and generally accept their differences? Does it discriminate against or distrust “outsiders”? Are there laws or customs that guarantee everyone’s rights, and are they honored or enforced? Are behaviors that are different tolerated? Who is socially excluded, and what basis (e.g. racial or ethnic group, social class).
• Sense of identification. How do members of the population identify themselves, and with whom do they identify and claim solidarity? Family? Ethnic group? Peer group (teens, musicians, political party, close friends, sports team)? Community (neighborhood, municipality, state, region)? As individuals, or simply as part of the human race?

You may not need to know about all these aspects of the community unless the issue you’re dealing with is so rooted in the culture (racism or gang violence, for example) that you really can’t understand the context of your evaluation without understanding the culture as a whole. It’s more likely that you’re concerned with a specific situation, and you’ll need to focus on gathering ethnographic information as it relates to that situation. (How do young people who leave school view education, for instance? What can you learn from an ethnographic investigation that can make it more attractive to them?)

The major points about ethnographic information are these:

• Ethnographic information is obtained directly from those who live in the community of interest. In the case of gathering information for an evaluation, that means participants in a program or initiative, or the specific group in which they’re members (Black men over 40, Hispanic teens, single mothers, non-English speakers, unemployed workers, etc.) It’s not exactly the same as simple qualitative information, although some researchers see it that way, but rather a specific and more focused instance of qualitative information gathering. (See Chapter 3, Section 15, Qualitative Methods to Assess Community Issues, for more about qualitative data.)
• The gathering of ethnographic information takes place in a natural, rather than an artificial setting. That means it happens in the environment where people in the group normally spend their time. If you’re running a program, you want to understand participants’ normal lives, not just the ways they behave and function in the program.
• Ethnographic information is meant to help you understand a culture from the point of view of its members.
• Ethnographic information is simply descriptive, not judgmental. Valuing may come later, but the point of the research is simply to understand how the individuals you’re concerned with see the world, and how they understand and view what they do. That can give you the entry you need to devise or revise programs in order to effect change and improvement.

Why might ethnographic information be important to evaluation?

In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

The U.S. Census Bureau used ethnographic studies to understand what populations were undercounted and why. The bureau had long acknowledged the difficulty of counting people who were homeless but knew that many others went uncounted as well. By using interviews, observation, surveys, and other methods, investigators found that undercounting resulted from several factors:

• Record-keeping errors (uncounted residences listed as unoccupied or nonexistent, which in fact did exist and were occupied)
• Different definitions of residence (some older or young family members might have their care shared by several siblings or cousins and might live with different people at different times so that their permanent residence was difficult to determine)
• Attempts to avoid losing benefits (men living with women receiving public assistance would simply not be reported)
• General distrust of government, especially among those who were in the country illegally
• Difficulties with reading or understanding the census form
• Difficulties with English (no accommodations for other languages)

As a result, the U.S. Census Bureau decentralized some of the data collection function, put more collectors on the streets, simplified and translated the form, and made other adjustments trying to include as many as possible of those previously uncounted. None of this information would have been found and none of the adjustments to address the issues it demonstrated would have been made without an ethnographic approach in the areas where undercounting was most common.

Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them, as opposed to what staff members and others predict or think about how they’re experiencing them. Successful programs learn how their participants are actually experiencing the program (e.g. delays, how people are treated, time and effort to participate). An ethnographic study can help you understand if and why there’s a gap between how you view your program or initiative and how participants actually see it.

Ethnographic information helps clarify what needs to be addressed in order for participants to respond to your approach. If there are differences in the ways you and the participants see your work, an understanding of their point of view can help you make adjustments that will either make clearer to them what the intent of the program is, or adjust the program so it starts closer to what they perceive and moves toward your shared goals.

Ethnographic information can clarify the issue and its effect on, and importance to, the population of interest. Seeing the issue you’re concerned with from the perspective of current and potential participants – its seriousness, its effect on their lives, its causes, its possible alternatives – can guide the way you frame your program for greatest effectiveness.

Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future. A true picture of the culture of the community or group can inform all your work, and allow you to participate in that culture, to be a better advocate, and to plan and run more effective initiatives in the future.

When might you want to collect ethnographic information?

Ethnographic information is useful, but an ethnographic study, even a very specific one, takes some effort. Especially if you’re part of a small community-based or non-governmental organization, you may not want to spend staff or volunteer time on gathering information that isn’t really necessary. When might ethnographic information in fact be necessary?

• When you’re engaged with a population or cultural group that you’re not familiar with or part of. Ethnographic information can prevent you from unintentionally causing harm, or from ignoring “obvious” factors that everyone in the population takes for granted and that might be key to the success or failure of your program.
• When you’re working with a clearly-defined group that has had a chance to develop its own culture. This might be a class, a church, a gang, a club, a team … any group that has been together long enough to have created norms for itself that might be even slightly different from those of the larger society. The more mainstream the group, the easier it is to miss these norms and to misunderstand how to approach them.
• When an understanding of the context and culture of the community is fundamental to what you’re doing. If you’re addressing an issue, such as racism or violence, it’s essential to understand the real attitudes of the people you’re working with – not only what they think, but why they think it, how they react to various aspects of the issues, what the history is for them, etc. If you don’t understand that at least part of the attraction of a gang is that it functions as a family for kids who often have no real connection or belonging to their actual family, you’re not likely to be able to make any headway in working with gang members.
• When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group. Understanding exactly how that population sees that situation (e.g., a job training program, or an initiative to increase physical activity among children) will make it easier to understand what is and is not working and why.

Who should collect ethnographic information?

This depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc. In some circumstances, anyone trained to do so can become a participant or non-participant observer; in others, you might do best to get the cooperation of one or more key individuals in the community. When outsiders are universally mistrusted, the information might best be gathered by actual members of the community who are trained in interviewing and other data collection techniques.

How do you gather ethnographic information?

Now that you’ve decided to gather and use ethnographic information in your planning and evaluation, how do you go about getting it? The first step is to decide what kinds of information you need. Then you’ll have to determine how to get it, gain the trust of the group you want to learn about, plan your field study, and carry it out.

1. Decide what kinds of information you need. As we’ve discussed, you might want to understand the whole context and content of the group’s culture, or you may only be concerned with a very specific topic. If you’re dealing with a distinct cultural group – one defined by ethnicity or religion, for instance, whose culture has developed over centuries – gaining an understanding of the whole culture may take years. If, on the other hand, the group is part of the general culture – a middle school classroom, a workplace – but has developed a culture of its own within that confined physical and psychological space, a useful ethnographic study may only take days or weeks. In most cases, when you’re gathering ethnographic data for an evaluation, you’ll be looking at culture in relation to a very specific topic.

To conduct ethnographic research on a specific topic, you have to make sure you’ve defined that topic well enough so that you can zero in on it in your observations and conversations. What exactly is it that you need to understand about your population that will inform your evaluation? One question often asked is “How do participants (or members of the population that participants are drawn from) view the program or initiative?” Other related questions might be asked as well: Is the program and its activities culturally appropriate and respectful? Are participants’ goals similar to those of the organization, or are they very different? Are there cultural advantages or barriers that make it more or less likely that the program will succeed or that individuals will be able to participate? What would enhance those advantages or remove those barriers? Are cultural differences between participants and staff or volunteers – or cultural ignorance on one or both sides – getting in the way of program effectiveness?

Suppose, for example, you want to find out how an ESOL (English as a Second or Other Language) program was viewed by students. In this case, you’d be dealing not with a coherent ethnic or religious culture, but with the culture of the classroom, unless all the students came from the same background.

Some questions you might ask:

• How do the ESOL students view education (as something for the upper classes, as necessary for them and their children, as an imposition, as time away from earning a living, as a ticket to a better life, as part of becoming part of the country)?
• How do they view teachers (great respect, fear, intimidation, scorn, confusion)?
• How do they approach their learning (enthusiastically, with a strong work ethic, with no knowledge or experience of how to do it, collaboratively, with fear and trepidation, hopefully)?
• What are the cultural barriers to learning? The cultural advantages? (In many families, for example, family support may be an extremely important factor.)
• What arguments for learning and techniques of teaching do students respond to?
• What do they discuss among themselves?
• Are there intercultural dynamics in the classroom, rooted in people’s cultures, that enhance or detract from the learning environment (gender discrimination or racial or ethnic prejudice, for instance)?

Answers to these and other questions might lead to such adjustments as restructuring classes by language group or ethnicity, changing the vocabulary and topics discussed in the course of lessons, changing the atmosphere in the classroom to be more or less teacher-centered, assigning more or less independent work and more or less group work, changing the teacher-student ratio, etc.

Knowing what kinds of questions you want to answer will help you plan whom you need to gather information from, how long it might take, and – perhaps most important – how you should go about it. (We’ll look at that in more detail when we discuss planning a field study.)

2. Determine what you have the resources to do. If you’re part of a small non-governmental organization (NGO) serving Haitian immigrants, it’s unlikely that you’ll have the money or the time to spend two years in Haiti. You may have the resources to spend a good bit of time in your participants’ neighborhoods, however, learning about their assumptions and their lives. That will probably be far more useful ethnographic information for your evaluation than a years-long study of Haitian culture, anyway.

3. Gain the trust of the group you’re engaged with. If you’re an anthropologist looking for a site to conduct a field study, you’ll have to find an appropriate situation, perhaps get permission to travel there (if you’re venturing onto tribal lands, for instance), find the people you’re looking for, and convince them not only to let you stay but to answer your questions, show and explain to you their ways and customs, and help you understand their lives.

In the case of an evaluation, you won’t have to choose your field study site – it’s the program you’re evaluating and/or the community from which your participants come. But no matter whether you’re an outside evaluator or you’re gathering ethnographic information from your program’s participants’ community (Hispanics, laid-off workers, Haitian immigrants, migrant laborers), you’ll have to establish enough trust so that people will tell and show you what you want to know.

This is easier if you have a presence in the community of concern – if your program has been running for a while and has the community’s support and respect, for instance. You may already know and have the trust and support of many in the community who can vouch for you and what you’re doing.

Often the best strategy is to enter the community under the auspices of someone community members trust – a clergy person, a community leader or tribal leader, a respected long-time community member or elder, a coach, etc. This is often how a new program is introduced to a community.

An ESOL program in a community with a large Portuguese population received a boost when the outreach worker helping to establish it made contact with the president of the local soccer club. The club’s headquarters and teams were central to the social life of the Portuguese community, and when the club president began promoting the program, the community immediately accepted and flocked to it.

If you can’t do that, then you’ll have to gain entry on your own. Find out where you can meet community members – sports events, festivals, bars and cafes, parks, shopping places, etc. Choose a place that’s comfortable for you to start with, since you’re far more likely to make others comfortable with you if you feel comfortable yourself. (If you don’t normally spend time in bars, you may not want to start with bars, for example, unless that’s the only place to meet the population you’re interested in. If you’re a sports fan or athlete, the local soccer field or recreational center may be the best place for you to start.) Also make sure it’s a place that feels safe and comfortable to people you’re likely to approach, so they don’t feel threatened or rudely intruded upon. Once you start meeting people, they’ll introduce you to others.

4. Plan your field study. The core of ethnographic research is the field study. This is essentially your observational system (see Section 3 of this chapter). It involves going into the normal environment of the population of interest and observing and interviewing members there, rather than in an artificial environment such as an office or laboratory, or under artificial conditions such as a test or an experiment. This could involve anything from spending time in the ESOL classroom we’ve talked about to living for years in a West African village to working for a year or two in an auto assembly plant. Most ethnographic information for evaluations is likely to be the kind that focuses on a specific topic and that can be collected over the course of the evaluation – usually a year or less.

By this point, you’ve already done some of the work planning your field study. The basic steps are:

• Decide on your questions. You may have already set these out when you determined what information you needed. They’ll probably be related to, but not the same as, your evaluation questions. They’re meant to inform your evaluation questions by providing such information as how participants perceive the program and their (and your) role in it. Questions might be focused on a specific topic (See “How do ESOL students perceive their classes?” above), or might be more general and relate to the larger elements of the culture. They should make sense in terms of the group you’re concerned with (i.e., they should be able to be answered by observation and/or interviews and other methods of data collection) and in terms of the evaluation (they should give you information that helps you look at what you’re doing and adjust it for the better).
• Determine what method(s) you’ll use to gather information. There are a number of ways to gain insight into the culture of a community or group:
  • Individual or group interviews can range from casual conversation to relatively formal, recorded semi-structured or structured interviews.
  • Participant or non-participant direct observation.
  • Casual interaction
  • Participation in community events, celebrations, work, etc.
  • Surveys. If you use them, they should be open-ended (e.g., Tell me about…), rather than agree/disagree.
  • Third-party informants (others with knowledge).
  • Documents – texts, AV, budgets, posters, brochures, manuals, art, etc. – by, about, important to (religious texts, for example), and directed at (instructions, policies) the group.
  • Demographic data and public records – census, court records, public health data, etc.
  • For more detail on several of these data collection methods, see Chapter 3, Section 15, Qualitative Methods to Assess Community Issues.
• Decide whom you’ll need to contact for information, and how. These may be specific people, chosen for their position in the community, appropriateness for the research at hand, or particular characteristics; or they may simply be members of the general population – whoever you happen to be able to make contact with. Will you contact targeted individuals directly, by mail, e-mail, personal approach, or phone? Will you contact people at random – striking up conversations at community events, approaching people in public places, knocking on doors? Or, will you work through intermediaries – community leaders, clergy, translators, etc.?

Ethical considerations: Ethically, your first obligation is to those whom you’re studying. The essentials include informing people about the existence and purpose of your research and obtaining their permission to be included; giving them the choice of anonymity and honoring their decision; giving them access to the final report if they wish; and being careful not to exploit or harm them.

While ethnographic research is by its nature subjective, you also have ethical obligations to be as truthful as possible about what you see, hear, and experience, and to paint as clear a picture as you can of the culture in question. Your interpretation of your data will undoubtedly be colored by your assumptions, experience, and background (and by the norms of your own culture), but your description of the group and its individual members should be impeccably accurate with biases filtered out. (See Chapter 19, Section 5, Ethical Issues in Community Interventions, for a more detailed discussion of research ethics.)

5. Carry out your fieldwork and keeping fieldnotes. An ethnographic study requires fieldwork – going to where the population you want to find out about is and observing and interacting with them in their own environment. That might be as simple as walking down the street to a particular corner or as complex as flying into a remote Arctic village on a seaplane. Whatever the character of your study, the elements of it will be the same.

Fieldwork is really carrying out the details of your plan – establishing trust, contacting people, and gathering information using the methods you worked out beforehand. You might find yourself changing some of your strategies and adjusting to the conditions and the people as you learn more about them. Your understanding of the culture will evolve as you get deeper into it and observe, listen, and participate more. This evolution should be reflected in your fieldnotes, both as they describe what you’ve seen, heard and taken part in, and as they describe your own reactions to your experience.

Fieldnotes are notes taken about your conversations, observations, and other data collection. They should be taken during or as soon as possible after each visit, conversation, experience, etc., before memory fades or changes. Where possible, audio and/or video recording is helpful to get an accurate rendering of what was said and done.

Elizabeth Chiseri-Strater and Bonnie Stone Sunstein (FieldWorking: Reading and Writing Research, 1997. P. 73. Blair Press: Upper Saddle River, NJ.) propose a list of elements that should be included in all fieldnotes:

• Date, time, and place of observation.
• Specific facts, numbers, details of what happens at the site.
• Sensory impressions: sights, sounds, textures, smells, tastes.
• Personal responses to the fact of recording fieldnotes.
• Specific words, phrases, summaries of conversations, and insider language.
• Questions about people or behaviors at the site for future investigation.
• Page numbers to help keep observations in order.

Fieldnotes come in four categories, which should be separated to reflect different forms and functions (from How to Do Ethnographic Research: A Simplified Guide, by Barbara Hall (U. of Pennsylvania):

• Jottings are the brief words or phrases written down while at the fieldsite or in a situation about which more complete notes will be written later. Usually recorded in a small notebook, jottings are intended to help you remember things you want to include when you write the full-fledged notes. They’re the kind of notes that newspaper reporters take when they’re talking to sources about a story. Although not all research situations are appropriate for writing jottings all the time, they do help a great deal when sitting down to write afterwards.
• Description of everything you can remember regarding the occasion you are writing about - a meal, a ritual, a meeting, a sequence of events, etc. While it is useful to focus primarily on things you did or observed that relate to the guiding question, some amount of general information is also helpful. This information might help in writing an overall description of the site later, but it may also help to link related factors to one another or to point out useful research directions later.
• Analysis of what you learned in the setting regarding your guiding question and other related points. This is how you will make links between details and the larger things you are learning about how culture works in this context. What themes can you begin to identify regarding your guiding question? What questions do you have to help focus your observation on subsequent visits? Can you begin to draw preliminary connections or potential conclusions based on what you learned?
• Personal reflection on what you learned. What was it like for you to be doing this research in this context? What felt comfortable for you about being in this situation and what felt uncomfortable? In what ways did you connect with informants, and in what ways didn't you? While this is extremely important information, be especially careful to separate it from analysis.

In evaluation, this information can help you identify your biases, and may make it clearer why your program is less effective than you’d like, or how it’s challenging or in conflict with some of the participants’ important beliefs or assumptions. (Challenging beliefs and assumptions isn’t necessarily wrong, but denying them – even if you find them unacceptable or clearly misguided in some way – is usually counterproductive. Challenging is asking people to reassess their actions or assumptions; denying is telling them they’re wrong or stupid.) If you’re ignoring your participants’ culture, or presenting things in ways that they’re likely, for cultural reasons, to understand differently, you may not be reaching them, and may, in fact, be driving them away.

These notes are the basis for the study, so their timeliness, completeness, and accuracy are extremely important.

How do you interpret ethnographic information?

The ethnographic information you’ve gathered won’t do you any good unless you use it to understand the dynamics of your program and the results of your evaluation. You can begin by organizing your fieldnotes and whatever other information you’ve collected.

Organize the final version of your completed notes. Categorize them by patterns, by demographics (gender, age, education level, socio-economic status, etc.), by topic or theme– whatever makes sense in relation to what you were trying to find out. Sometimes, this may mean categorizing them in more than one way. Often, just the act of putting your notes together in particular ways can reveal patterns or highlight issues you weren’t previously aware of.

Starting from the point of view of your original interest, write up a description of the culture as you understand it to be seen by its members. Depending on your purposes, this might range from a complete description of the culture to a very narrow description of how members of that culture – a village, an office, an agency, a consumer sector, a country – perceive a particular aspect of your program.

This type of description can encompass great complexity if you’ve gotten a true picture of the culture you’ve been studying. Many elements may fit together in patterns you didn’t expect, and these may stand out as you put all your information together or organize it in certain ways. The goal here is to understand how the elements of the culture blend and influence one another, and how those elements and the culture as a whole affect participants’ performance, responses, behavior, communication, etc., in your program or initiative. That understanding can lead to adjustments that will make the program more successful with the group intended to benefit.

Let’s look at that ESOL program again. If most of the class members are from a culture where women defer to men as a matter of course, it may be difficult to persuade women in a coed class to participate actively. This is a particular problem in such a class, because the teaching method is based in part on encouraging as much speaking in English as possible from the beginning. If women are reluctant to speak out – even if their grasp of the new language is improving – their progress will be slower.

Understanding the underlying reasons for women’s silence in class allows the instructor to respond to the real situation, rather than simply assuming that the women don’t get it, or that calling on them will solve the problem. One solution is to break the class up into single-gender groups for part of each lesson. (This could be explained either forthrightly – to give the women a chance to speak more – or as a way to let men and women discuss what interests each.) Another is to explain that in this country, at least in public, customs are different, and that women need to learn to speak out so that they’ll be able to hold their own in shopping, jobs, and other daily tasks. Whatever the method, knowing the reasons for the women’s silence makes devising a solution possible.

Reexamine the analyses from your field notes. What did you find out about the culture’s perceptions, norms, and customs as they interacted with the focus of your questions? Seeing the study whole, do you still agree with most of your initial analyses? Set aside those ideas you think are probably incorrect or misinformed, and reformulate analyses to replace them that reflect your subsequent learning, if you haven’t already.

Try to understand the answer to your original question from the point of view of those you’re concerned with.

• Do local people see the question in the same way you do?
• Are there factors you didn’t know about that affect their perspective on the answer to the question?
• Are there factors that they perceive or act on in a different way than you expected?
• How important to the group is what’s important to you?
• Do you and they have the same expectations for outcomes? If so, do they have the same expectations about how to get to those outcomes? If not, are your and their expectations in opposition or totally different? Are they starting with the same view of reality that you have?
• Do some of their perceptions, cultural norms, or culturally-rooted behaviors present barriers to the success of your program that you’re either already working to counter or could work to counter?
• Do some of their perceptions, norms, or culturally-rooted behaviors support the success of your program in ways that you do already or could take advantage of?
• Does the program have unintended or unexpected consequences – positive or negative – as a result of cultural factors?

Some possible examples from our ESOL class: Are the younger members of the class intentionally holding back in answering questions or even in demonstrating their developing English skills so as not to show up or embarrass older class members whom they have been taught to defer to out of respect? Are class members specifically slowing down the rate of learning/instruction to that of the slowest learners so as not to embarrass anyone? Are there conflicts caused by cultural differences along these lines?

Translate what you’ve learned about participants into supporting information or answers to your evaluation questions. Some of the questions you might want to pose for yourself include:

• How do the culture, attitudes, and norms of the population you’re engaged with affect the effectiveness of the program for them?
• Are your methods appropriate for the culture you’re working with?
• Are you creating rather than eliminating barriers with some of your activities or methods, or, indeed, with your program as a whole?
• How can you make things easier and more rewarding for participants?
• What are you doing that is most effective? Can you understand from your ethnographic data why it is effective?

Continue to gather ethnographic information to guide your work. Just as evaluation continues as long as your program or initiative lasts, your ethnographic investigation should continue as well, so that you can use the information to continue to improve your program and adapt it to the culture of the community.

In Summary

Ethnography – the attempt to understand a culture from the perspective of its members – can be extremely helpful in conducting a valid evaluation of a program or initiative. Ethnographic information can tell you why participants behave or respond in certain ways, what parts of your program conflict with or are reinforced by participants’ cultural norms, and why particular program elements are more or less effective. This knowledge can help you adjust what you’re doing to be more effective and more valuable for participants.

The culture you’re engaged with might be as small and self-contained as that of a classroom or as large as a major world religion or ethnicity. Whatever the case, an ethnographic study involves approaching the culture from the inside by embedding yourself in it to the extent possible. That may mean spending many days in that classroom, observing and talking to the teacher and students, or spending years in another country, absorbing the language, customs, and behavior that define it as a distinct culture. You’ll have to decide whether an ethnographic study is necessary for your evaluation, and, if so, what resources you can devote to it.

Planning and conducting an ethnographic study requires deciding on the information you need and the resources you have to devote to it; gaining access to the group you’re interested in; choosing your specific research questions, the methods you’ll use to collect information, and whom you’ll collect it from; and then carrying out your plan, taking careful notes and/or audio or video recordings of your observations and conversations throughout. Organizing and analyzing your notes and other information can then give you a picture of the group’s culture that can help you better understand your evaluation results and improve your program. Continuing to gather ethnographic data will result in a continuing growth of your understanding and knowledge, and, ultimately, can mean a more successful program.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

What is ethnographic information?

__ Ethnographic information is information about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view.

__ Elements of ethnographic information include:

• Community norms
• Health conditions and knowledge
• Power and political realities
• Religion
• Economics
• World view

__ Ethnographic information is obtained directly from those who live it.

__ The gathering of ethnographic information takes place in the environment where members of the culture normally spend their time (natural setting).

__ Ethnographic information is meant to help you understand a culture from the point of view of its members.

­­__ Ethnographic information is simply descriptive, not judgmental.

Why might ethnographic information be important to evaluation?

__ In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

__ Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them.

__ Ethnographic information helps to clarify what needs to be addressed in order for participants to respond to your approach.

__ Ethnographic information can clarify the issue and its effect on and importance to the population of interest.

__ Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future.

When might you want to collect ethnographic information?

__ When you’re engaged with a population or cultural group that you’re not familiar with or part of.

__ When you’re working with a clearly-defined group that has had a chance to develop its own culture.

__ When an understanding of the context and culture of the community is fundamental to what you’re doing.

__ When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group.

Who should collect and interpret ethnographic information?

__ That depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc.

How do you gather ethnographic information?

__ Decide what kinds of information you need.

__ Determine what you have the resources to do.

__ Gain the trust of the group you’re concerned with

__ Plan your field study

• Decide on your questions.
• Decide on your methods of data collection.
• Decide whom you’ll need to contact and how.
• Work out ethical issues.

__ Carry out your field study, taking careful field notes, including jottings, descriptions, analyses, and personal reflections on what you learned.

How do you interpret ethnographic information?

__ Organize your data.

__ Write as complete a description as possible of the culture or the element(s) of the culture you’re concerned with from the perspective of its members.

__ Reexamine the analyses from your field notes.

__ Try to understand the answer to your original question from the point of view of those you’re concerned with.

__ Translate what you’ve learned about participants into answers to, or supporting information for answers to, your evaluation questions.

__ Continue to gather ethnographic information to guide your work.

You’re evaluating your teen pregnancy prevention program, and you’d like to know whether it will result in a reduction in pregnancy rates among young girls in the community. There are statistics on pregnancy rates for the state and county collected by the Public Health Service, but none at the community level, and none that separate the rates for girls under 16, the population you’re most concerned with.  You could do a community survey to try to find out the local rate, but there are two problems with that idea. The first is that you have neither the time nor the resources to conduct the survey, and the second is that you’re unlikely to get an accurate picture– there may be embarrassment in asking the questions or other reasons.

There might be another way of getting the information, however. A number of other agencies in the community work with youth, and one of them, or a combination, might have the figures you’re looking for. Rather than generating or collecting it yourself, you can save a great deal of time and trouble by using data that already exist.

We have previously discussed ways to find existing information to help you conduct a community assessment of assets and needs, but now our goal is somewhat different, because we’re seeking data that you can use to evaluate your work.This means that it needs to be in a form that can be analyzed, and may have to refer to a very specific population, issue, and/or method. As a result, it may be harder to find, and may have to be converted in some way once you do find it.  In many situations, however, using existing information still can be much easier than collecting the data yourself. In this section, we’ll try to help you make the use of archival data as easy as possible.

What are archival data?

Archival data refer to information that already exists in someone else’s files. Originally generated for reporting or research purposes, it’s often kept because of legal requirements, for reference, or as an internal record. In general, because it’s the result of completed activities, it’s not subject to change and is therefore sometimes known as fixed data.

Some researchers make a distinction between archival and secondary data. They see archival data as information specifically collected for bureaucratic procedures and the like – applications, reports, etc. – that can then be made usable for other purposes.  Secondary data refer to research information, collected as a result of studies and similar efforts, that can then be used by others either as comparison data or as part of new research.  For the purposes of this section, we’ll include both of these types of data in our discussion, and not distinguish between them.

Sources of archival data

Archival data can exist almost anywhere that information is collected.

Some of the most common sources (we’ll look at these and others later in more detail) are:

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

Archives are often stored as paper files or on electronic storage – computer disks, CDs, DVDs, etc. – and may include photographs and audio and video recordings as well. It may also take the form of encoded information expressed in numbers, or in computer language. Computer files, of course, may include various media and text, all in the same place.

Many organizations have archives so large that they store most of the material off-site, either with a data storage firm, or in their own or a rented facility. Some archives are made available on a website maintained by the government or other organization.

Types of archival data you might look for

As explained above, much of the data you’re likely to use for evaluation purposes will probably be more focused than data you’d use for an assessment of the level of a problem. Evaluation information would be more likely than assessment data to come in the form of study results, for example, than as narrative history or original documents. There’s a good deal of overlap: census data, for instance, could be used in both assessment and evaluation. In general, however, the possibilities below would refer to the types of data available, including information for a specific population or geographic area on:

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population

Why collect and use archival data?

There are sometimes good reasons for using original data, including that the information you need just isn’t available elsewhere. Additionally, if a researcher collects original data, he or she has more control what data are collected.

On the other hand, if the information you need, or something very close to it, already exists, there are several good reasons to find and use it.

• It’s easier and less time-consuming than collecting all the data yourself. This is probably the most obvious and most common reason for taking advantage of archival data. Especially if you’re looking for a large amount of information or information about a large group of people, you may be able to save yourself an enormous amount of time and trouble by using archival data.
• Archival data may have already been processed by people with more statistical expertise. Unless you’re a statistician or a health or human service researcher with an advanced degree (and often not even then), the chances are that you don’t have a flawless grasp of data analysis. You can hire someone or find a volunteer to help you, but if the hard work has already been done, it will make your work that much easier.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The archival data you find may be more sweeping or more specific than what you’d be able to gather. It may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Archival data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for  In cases like these, the use of pre-existing data might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• It may eliminate the need to correct for problems, such as improper sampling, lack of inter-rater reliability, or observer bias.
• Archival data allows the possibility of looking at the effects of your work over time. Is the change in your population part of a trend that seems to be reflected in data from a similar population or the entire state or nation? You may not have the capacity to collect data over a long enough period to answer such questions, but if the data already exist, it makes longer-term analysis possible.
• Archival data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exists elsewhere.

When should you collect and use archival data?

• When it’s available. This is the key question. If you know the data exist and you can get access, use it. If the data doesn’t exist, if finding it would take more time and effort than it’s worth, or if you have no access to it, then it’s not possible.
• When it’s relevant. As with its availability, the relevance of the data to what you’re trying to find out is a key issue. All the archival data in the world won’t do you any good if it doesn’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, archival data may be the only source of the information you need.
• When it can inform your evaluation. There are large amounts of archival data available almost everywhere. The mere fact that it exists doesn’t mean that it will do you any good. You have to be selective about what you gather and use. Make sure it is actually what you need, that it refers to the population and/or other elements of your program that will make it truly useful to you.  If not, you’ve not only wasted your effort, but the resulting evaluation won’t give you a realistic picture of your work and how to improve it.

How do you collect and use archival data?

As you search out and collect archival data, there are several questions you should ask.

What information are you looking for and why?

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants.

Some possibilities include:

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• General information on the population and/or the community you’re working with. You may want to see how well various characteristics of your participants match those of the general population or you may simply want to understand the context of the evaluation better. You may also be looking for information to choose community-level indicators. Community-level indicators are specific, measurable units that help us determine success of an initiative or intervention in the community; examples of community-level indicators include program participation rates, services delivered, levels of crime, or new cases of HIV/AIDS.

Community-level indicators show trends for the community as a whole.  You can often use them to find out whether your efforts have had any effect in the community.  If, for example, you are conducting a program to reduce alcohol use among youth, one indicator of success might be a reduction in the number of weekend and nighttime one-car crashes involving teens. In many cases, you might choose community-level indicators according to the data that’s available.  For example, if there are reliable figures for exactly the kinds of car crashes described, then using those figures as a community-level indicator would probably make sense.

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include:

• Demographics:
  • Demographic statistics – age, race, gender, ethnicity
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level Other characteristics –primary language spoken, etc.
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Substance use and abuse
  • Sexual behavior Various other behaviors – work habits, consumer patterns, etc.
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Development outcomes (e.g. those completing primary education, high school; those with disabilities)
  • General health and well being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins)

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes.
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

One way to increase the reliability of data where you are dealing with small data sets is to combine multi-year data (for instance, results of cancer deaths in a community for three years instead of one).  A drawback to this option is that looking at multi-year data limits the ability to monitor program interventions and identify new trends.  Rolling year averages (e.g., looking at data for 1997-2000 one year, and 1998-2001 the following year) may overcome this drawback and should is an option that should be considered.

Another way to decrease the possibility of statistical instability is to expand the geographic area you are investigating by looking at regional health assessments conducted by collaborating neighboring jurisdictions, or in the example above, expanding from county to state.  A drawback to this option is that you may then be examining results for a geographical area that does not necessarily apply to your assessment.   Analyzing data at the regional level may also mask interesting local variations in the data.

Who is likely to have collected that information?

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

Where should you look for archival data?

The question here is not only where to find archival information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble.

Your own archives

Unless it’s brand new, your own organization should have an archive of administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information.

The Internet

Most public documents are either on the Web or can be found and/or ordered through a website. The place to start is usually the website of the government agency most likely to have collected the data. The Resources portion of this section contains a list of U.S. government websites. In the U.S., states and most cities and towns have websites as well, with links to state or municipal agencies and departments. (The URL’s for all state websites take the same form: http:// www .[state abbreviation].gov.  Municipal websites can easily be found by searching the name and state of the community.) States or provinces and communities in most of the developed, and much of the developing, world have websites as well.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

As always when using the Internet, you should be cautious about where you find your information. There are enormous numbers of reliable websites...and huge numbers of unreliable ones as well.  If you’re not sure of a website or of the information you get from it, try to find that information elsewhere as well. In general, you can rely on websites when you know where they get their information, and when you trust the reputation and integrity of the site’s owner.

Go directly to the source

Often, the best way to find information from health and human service organizations, schools, and businesses, as well as from advocates and community activists, is to go to them directly.  If you do, be prepared to explain exactly what you’re looking for, what you plan to use it for, and what you can offer in return. Unless the organization is willing to let you comb through its files – confidentiality is often a barrier to that – someone will have to spend some time finding what you need.  It’s only fair to offer something in return, whether it’s payment, data analysis services, advocacy for the cause, or something else the other organization needs.

If you’re asking another similar organization for data so you can use it as a comparison or control group, the request has to be extremely tactful.  In a sense, you’ll be telling the staff of that organization that you expect your results to be better than theirs.  Depending upon how they see their work – and how they perceive you and your organization – they may take this as an opportunity to find better methods to serve their participants, or as a grave insult.  If it’s the latter, they’re hardly likely to agree to the use of their data.  You’ll have to frame the request in the right way, and offer a good exchange as well.  It will help if you’re dealing with an organization with which you already have a good relationship of mutual respect.

Libraries

Librarians have always been world-class experts at finding what library users needed.  With current technology, they’ve become even better. Many have an encyclopedic knowledge of not only what’s available in the library itself, but what’s on the web as well.  They may be familiar with sources of archival information you’d never think of, and be able to help you find what you need quickly and with minimum effort.  When in doubt, head to (or communicate with) an available library.

What are you planning to do with the data once you have it?

This question has to do not only with what form you need the data in, but also just what data you actually need. If you’re planning to use it as a comparison to the group participating in the program you’re evaluating – whether as a formal control group or as baseline data – you’ll need information on the variables you’re planning to look at, as they relate to the population you’re working with, or at least a population that’s reasonably similar. If, for example, you’re evaluating a chronic disease prevention program intended to benefit Latinos, and you’ve found archival data on physical activity and nutrition among Native Americans, you can’t compare your results with those of the archival data because the groups are likely to be too different.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the archival data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending.

Using archival data

It’s difficult to imagine evaluating a program or approach without actually collecting your own data on participants.  You might be able to find data on those participants from an earlier time, which you can then use as a baseline. You might be able to find appropriate data on a similar group that you can use as a comparison or control. But you can’t find data elsewhere on what those participants are currently experiencing, and that’s what you’re evaluating in almost every case.

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find archival data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both.

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use archival data:

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. Along with a clearer picture of the community goes a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved as a result of your work.

There are two kinds of variables (things that may change) in research.  An independent variable or intervention is a program, treatment, method, system, or other action or condition set up by the researcher to see if it will create change and improvement.  A dependent variable is a behavior, condition, or other element that may change as a result of the independent variable.  A violence prevention program, for example, is an independent variable that may change community members’ engagement in violent behavior and associated injuries, the dependent variables.

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding.

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group.

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use archival data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change.The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find archival data that tells you whether the effects of your program continued, kept growing, or trailed off.

In Summary

Most government agencies and departments, community-based health and human service providers, advocacy organizations, universities, and many other entities keep archival records of information. You may be able to use these as part of the data for your evaluation, saving time and trouble. Especially for small organizations with limited resources, the use of archival data can make it possible to produce an evaluation that provides the information needed to accurately assess a program’s effectiveness and make the changes necessary to improve it.

What are archival data?

___Archival data is data that already exists as a result of administrative procedures or past studies or evaluations

___Be aware of sources of archival data:

• Public records
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

___Be aware of the possible types of data available:

• Demographics
• Behavior
• Health and development outcomes
• Attitudes – racial, political, social, etc.
• Knowledge and awareness of issues
• Environmental conditions or factors affecting the population and/or your work

Why collect and use archival data?

___It’s easier and less time-consuming than collecting all the data yourself

___Archival data may have already been processed by people with more statistical expertise than you, making it easier to use in analysis

___Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources

___It’s quite possible that you can find more information than you’d be able to gather if you did it yourself

___Archival data could touch on important areas you might not have thought of, or identify patterns or relationships you wouldn’t have looked for

___It may eliminate the need to correct for such problems as lack of inter-rater reliability or observer bias

___Archival data allows the possibility of looking at the effects of your work over time

___Archival data can make it possible for small organizations with limited resources to nonetheless conduct thorough evaluation studies

When should you collect and use archival data?

___When it’s available

___When it’s relevant

___When you don’t have the time and/or resources to collect the data yourself

___When it can truly inform your evaluation

How do you collect and use archival data?

___Determine what information you’re looking for and why

Possible categories of information you’d be interested in:

___Data on past participants

___General information on the population and/or the community you’re working with

___Specific information on appropriate characteristics of the population you’re working with

___Cultural information

___Data on a similar group that can be used as a control or comparison

___Results of previous studies

___Determine who is likely to have collected that information:

• Government departments and agencies, the census, and other entities whose data are likely to be available in public records
• Research organizations
• Academia
• News media
• Foundations and other private funders
• Hospitals and other health care providers
• Mental health providers
• Human service and other community-based organizations
• Advocates and watchdog organizations
• Community activists
• Community economic development organizations
• Businesses and corporations

___Decide where you should look for archival data

• Your own archives
• The Internet
• The original source
• Libraries

___Decide what you plan to do with the data once you have it

Use archival data:

___To better understand the context of your evaluation

___To identify areas to address

___To establish a baseline against which to measure your results

___To identify already-existing trends that may affect the results of your evaluation study

___To establish a standard of comparison against which to measure your efforts

___To act as a control or comparison group

___To provide data for a longitudinal study

Tool 1: Common Secondary Data Sources Used to Address Gender-Based Violence

This tool from Strategic Prevention Solutions provides descriptions of common secondary data sources useful for addressing gender-based violence, along with common challenges with secondary sources.

Whenever you begin a new job or start a project, you will probably want to evaluate your work. You might ask yourself: "Is my work meeting my expectations and those of my boss, my administrator, or our target audience?"

In a similar fashion, community initiatives need to be "evaluated" along the way. Asking the community if your action plans are actually working is one important example of how to begin an evaluation. If the results are positive, that suggests your initiative is heading in the right direction, and you probably should keep heading there. However, evaluation is also essential in helping one see what "course corrections" might need to be made if the initiative is not heading in the direction you planned.

This chapter will explain the importance of evaluation, and will offer step-by -step methods to evaluate your group's work. Then, when you discover that your efforts are beginning to make a positive difference, you can continue on with even more energy and enthusiasm!

What does it mean to evaluate a community initiative?

Basically, it means to determine the value of the work. You have developed and implemented an initiative in your community, and you want to know how well it's working. Evaluation provides you with this feedback.

In many avenues of life, we get feedback right away. You hit the brakes, the car stops. You shoot a basketball; it either goes in or it doesn't. That's instant evaluation, and it's completely understandable. There's no ambiguity. But with more complex events, such as social interventions, the results are not always as clear. (Sometimes they are--you schedule an event and people are lining up around the block to get in--that's rapid and clear evaluation; but it's rare). That's why you need to put more energy and thought into finding out how you did. And that's basically what evaluation is all about--giving you information on the value of your work.

In this chapter, we'll talk about evaluation as a means of obtaining feedback, data and information about your group and its activities. By using this information, you can decide what aspects of your action plan work, and what areas need improvement. When you evaluate your program, then, you are gathering information to help you draw conclusions about your project and the efforts of your group. After you have drawn conclusions from the information, you can make any necessary changes to your goals and/or action plan.

The sections in this chapter will focus specifically on ways to gather these valuable data. The next chapter will discuss methods to best use the data collected to strengthen and improve the initiative. Resistance to evaluation is common. To gain the greatest benefits from evaluation, you must overcome some common misconceptions about it.

Fears of evaluation

Evaluation can be frightening to many people. Generally, these fears fall into three types--"I don't know how;" "I don't have time;" or "The results might be negative and hurt us." All of these are valid concerns. But they shouldn't be so discouraging as to outweigh the benefits of doing an evaluation. Here are some responses to these concerns:

I don't know how to do an evaluation.

Well, that's why we're here. We'll help you through the steps of how to plan an evaluation of your program, how to do the evaluation itself, and then how to use it to help your group.

I don't have the time.

Evaluation can take some time to do. That's absolutely true. However, doing an evaluation now will save you a lot more time "down the road," as it will point out potential problems while they're still small problems, instead of waiting until a disaster occurs.

And there is another time-saving benefit. Once you begin to record the kinds of information you will need for your evaluation, it will become just a regular part of your routine operations. It will take less time later on. The hardest part is the beginning.

The results may be negative or hurt us.

This is a possibility. However, it's unlikely to happen if you use evaluation from early on and don't let little problems grow into big problems. And remember that any negative results you may find should actually be helpful to you, at least in the long run. They will help you improve the quality of your program or initiative-- which should be one of your own goals from the start.

Why should you evaluate a community initiative?

Being successful demands careful attention during the beginning, middle, and end of a project. If a violinist wants to learn a new piece of music for an upcoming concert, for example, she would prepare by practicing for many hours each day. But, if she never asks her teacher to listen to her play, she may be playing the music too slowly, too fast, too softly, or too loudly. If she never knows the proper way to play the piece--if she never gets any feedback--all of the practice in the world won't help her sound in tune and in time on the night of the performance.

Like the violinist, community groups need to pay careful attention to feedback during the beginning, middle, and end of their projects. An initiative can devote a great deal of time and energy to working on meeting its goals. But, if the work isn't heading in the right direction, all of those long hours and hard work can lead to frustration instead of a feeling of success. Evaluation tells the group how it's doing and helps identify any necessary changes along the way that will help you stay "in tune" with your own goals and the needs of the community.

There are many reasons why evaluations are valuable. Let's look at a few examples of ways in which evaluation can benefit a community group.

• Success is reinforcing - it brings more resources your way. There is an old adage that says, "Nothing succeeds like success." It stands to reason that the more successful your group's work proves to be, the more support and encouragement you might receive from members of the community and maybe even from funders. Evaluation can document your success, with facts, figures, and examples. If volunteer hours in your organization increased by 100% last year, or if every single child in your community was up-to-date on immunizations, those types of achievements can bring new resources your way. In other words, if evaluation can provide concrete examples of your group's successes, that can only be advantageous. In this way, evaluation can help you "toot your own horn."
• Failure is instructive. Even if your work falls short of its goals - and even if your program falls flat on its face - that knowledge can be helpful too. It may be painful in the short run; it might hurt. Yet negative feedback, or a negative evaluation, can really help you in the longer-range scheme of things. At least you know what the truth is, and where you stand. You have fewer illusions. And once you have dusted off, you will probably learn from the evaluation you received. Chances are you won't make the same mistakes again. You are now in a better position to make improvement.
• Evaluation can make you feel good. Being able to see your successes and the value of your work will obviously boost your spirits and motivate you to continue with your work. Again, this works both ways, but even negative aspects should be seen as an opportunity to learn about what works, and not as a failure!
• Evaluation raises the chances of further action. Once you have completed your first evaluation, you know what has worked and what has not for your group. You can modify the tactics that didn't work as well as planned, and reinforce those areas that were successful. So now you can take further action with an even greater chance of success! Others will notice this success and may join or help your group, further increasing your chances that your program will make a positive impact.
• The evaluation can help you understand important aspects of the initiative. You've just finished your first evaluation. The results may indicate some part of your initiative worked really well. For example, the free cholesterol screening at the local health clinic you sponsored was jam-packed with people waiting to be tested. On the other hand, no one is showing up to your monthly Healthy Cooking classes. Maybe it's because the classes are only held during the day, or maybe the classroom is located too far away from most of the people in your community. There are many possible reasons your project might not work as planned. Evaluation will help you understand why things worked, or didn't work, as they did.

When should you evaluate a community initiative?

Evaluation should take place during several stages of your group's life.

When your plan is in action:

• Determine baselines for behaviors you wish to change. If you want to know how much change your program has brought about, you'll need to know what was happening before your group got started.
• Focus on the impact your work is having on the community.
• Continue revising and updating action plans.
• Keep the group strong and focused on the goals at hand. You may want to use a survey that appraises your community goals, and use the feedback to change your planned priorities.

When some of your action plans are complete:

• Use the evaluation to help the group continue to measure its impact on the community, and to create plans for continuing helpful programs for the future. For example, community level indicators will tell you if your interventions are having an impact on the bottom line.

How do you begin an evaluation?

Suppose you know you want to conduct an evaluation. But where do you begin? The process of evaluating can sound overwhelming. However, remember that what you've already accomplished may have seemed hard to imagine at the beginning. Evaluation can be broken down into several parts to make it more manageable.

Before we jump into specifics, here is a general thought to share with your group members: your evaluation should address questions that are important to you, members of your community, and those who are providing financial support. Of course, finances and time constraints might limit the kind of information that can be collected. Try to ask the questions that will ultimately help your group succeed. Because each group is unique, the evaluation should be as well. Nonetheless, below are some questions of interest to many groups. Would they be of interest to yours?

• How much did the community participate?
• What programs, policies or practices have changed in the community?
• Have people's behaviors changed? If so, what kinds, and how?
• Are those changes due to your efforts?

Tips for implementing a successful evaluation

You have to want to evaluate.

The first step is internal. You have to be motivated to do the evaluation, or else it will be half-hearted, if it happens at all. And you need to be clear on the purpose of the evaluation. Why do you want to do it? For example, if you project was to reduce smoking among adolescents, you may want to find out if your program has lead to stricter enforcement of laws regulating the sale of tobacco to minors.

Your evaluation should be grounded in your stated objectives or goals.
When you planned your initiative, you should have clearly identified what you intended to accomplish. What specific outcomes are you trying to achieve? Having well-defined objectives is valuable because they serve as the foundation for your evaluation.
For example, if your objective is to reduce the percentage of local high school students who use tobacco to 5% by May 2032, the evaluation standard is straightforward. You simply measure the percentage of students who use tobacco at that time and compare it to your target. Clear objectives make it easier to determine whether your initiative has been successful.

For each of your objectives, you need to identify criteria, or indicators, which will provide reliable and valid measures for each of your objectives.

You will need to develop measures that tell you what is really happening. Once again, your measures (or indicators) should be based on your objectives.

For example, if your objective is to reduce youth violence in your local high school, then some possible measures might be hospital admissions records for violence-related injuries, attendance at the school's conflict resolution training seminar, or police records of arrests of youths for assault or carrying concealed weapons.

If your objective was increasing the availability of heart healthy foods in your community, some measures might be the number of restaurants who have a low fat menu section, the percentage of milk that is skim in local grocery stores, or the amount of shelf space used to display lean meat versus higher fat cuts.

You need to collect data on each of these indicators.

Sometimes you can find the indicators you need from existing sources. For example, if you were interested in increasing library borrowing, or in reducing false fire alarms, you could gather existing data from the library or fire department. But sometimes data on your chosen indicators may not be available. Suppose, for instance, your number one issue was speeding along residential streets. The key information here unfortunately may not exist. In those cases, the local police might be willing to help collect it; or you and your group might need to collect it yourselves.

Either way, if you can assemble "before" and "after" statistics on your chosen indicators, you can use them to help determine whether your program or initiative made a positive difference. Did borrowing go up? Did traffic slow down? The data here will do a lot of the talking.

Use the results to adjust the program or intervention as necessary.

Are you meeting the objectives you had planned? If so, no adjustment may be needed. If you are not meeting those objectives, the data may indicate what changes need to be made to get back on track. For instance, in our example on high school smokers, if the percentage of high school students who quit smoking is not very high, you might want to change the content of the high school tobacco education program now being given or add a new program to help them quit.

In Summary

Ask yourself, "What questions do I want to answer?" That's a key first step. Now, how do you answer them? Each of the following sections in this chapter will thoroughly explain methods that you can use to evaluate your initiative.

What is evaluation?

___You understand that evaluating means determining the value of your work

What are some objections to evaluation and how can you resolve those objections?

___"I don't know how to do an evaluation."

• Use a resource such as the Community Tool Box to find guide once the steps you should take to plan and implement the evaluation, and use the results.

___"I don't have time."

• Although it is true that evaluation takes time, it will save time in the long run, by pointing out potential problems before they become disasters

___"The results may be negative or hurt us."

• If you begin evaluating early, you should be able to prevent any major problems
• If you do find problems, you can use that to improve the quality of your program

Why should you evaluate your program?

___Success is reinforcing

___Failure is instructive

___Evaluation can make you feel good

___Evaluation raises the chances of further action

___Evaluation can help you understand important aspects of the initiative

___You should evaluate your initiative when your plan is in action, or when some of your plans have been completed

How should you do an evaluation?

___You want to evaluate

___You have evaluated in terms of objectives or goals

___You have identified criteria or indicators that will provide ways to measure your objectives

___You have collected data on each of these indicators

___You have used the results to adjust the program or intervention as necessary

 

 

 

What does it mean to monitor your progress?

Monitoring your community initiative can help you weigh your actions against the results to see if you are meeting the goals of the community and your initiative. In a sense, monitoring data helps you understand how well the initiative is functioning. That is, monitoring can help pinpoint where the actions of the initiative are not producing the desired effects. Additionally, the monitoring system can help you.

• Better understand the initiative
• Make decisions concerning the programming of the initiative
• Promote awareness of accomplishments
• Recruit support
• Secure funding

Despite the scary sound of "monitoring system," you have probably already observed examples of monitoring in a variety of ways. Political candidates monitor the status of their campaigns by conducting polls and analyzing the results. Teachers monitor the progress of their students by giving tests at the beginning and end of the school year to see if they have mastered the secrets of long division. You might monitor your utility bills by keeping track of the monthly increases and decreases. Monitoring has a wide variety of applications. As a member of a community initiative, monitoring means a way of tracking major events and accomplishments of the initiative.

There are three key parts to the monitoring system:

• Process and outcome measures
• Observational monitoring system
• Regular feedback on performance

In this section, we'll explain what we mean by process and outcome measures, and the observational system. We'll tackle ways to provide feedback to members of your coalition and your community in later sections.

Process measures

Process measures are the activities that take place during the initiative which help you determine how well things are going.

Process measures can include many aspects of your initiative, such as:

• Members who participate: the number and type of participants, frequency of attendance, and turnover rate of the members.
  • Example: At the last general meeting of the initiative, 17 people attended. Of these folks, 10 were regular members, and 7 were sitting in on a meeting for the first time.
• Planning products: written objectives, by-laws, or committees that contribute to the initiative.
  • Example: The action plan for the coalition was approved by the coalition and will be implemented immediately.
• Media coverage: by radio, television, and print media.
  • Example: Several five minute radio spots describing one of your group's projects aired on a local FM radio station.
• Financial resources: grants or donations. Financial resources also can include "in-kind" services, such as free advertising or products that an individual or business might offer instead of money.
  • Example: $8000 was received at a $50 per plate lunch that was held to raise money for local drug and alcohol abuse efforts.
• Services that are ultimately provided: classes, programs, workshops, publications or other services or communications provided for the community by the initiative.
  • Example: Nutrition education workshops were conducted by child care providers.
• Community actions: actions taken to encourage change in the community.
  • Example: Merchants were asked to display signs describing the penalty for selling alcohol to minors and the need for proper identification.

Later, you will incorporate process measures into event logs to better measure the progress of your group's work. We'll walk through an example of an event log later in this section.

Outcome measures

While process measures document the specific methods you use to create change, outcome measures explain the overall impact that occurs as a result of these individual actions. Outcome measures highlight the changes that happen in the community as a result of the work done by your initiative.

These include:

• Changes in programs, such as a new or modified service program.
  • Example: A parenting class was implemented by the initiative.
• Changes in policies, such as a new or modified policy.
  • Example: A city ordinance was passed requiring owners of cigarette vending machines to place on every machine a sign that states "No cigarette sales to minors." The legislation was introduced at the urging of the Law Enforcement and Government Committee.
• Changes in practices, such as a new or modified practice.
  • Example: Merchants displayed signs describing the penalty for selling alcohol to minors and the need for proper identification.

As you can see, the monitoring system involves several components that will help your group determine how it's doing, what it's doing correctly, and what can be improved. Sound interesting? Read on for more!

Why do you need to monitor your progress?

Now you might be thinking to yourself, "You want me to gather information about our initiative. But what will I do with all of those numbers and comments? Why is this important for the success of our group?" Don't worry; the data you gather by using the monitoring system can help the group in a variety of ways.

• Data can tell you where the initiative places its emphasis. For example, the monitoring system might reveal that your initiative focuses on services, rather than change. If change is what you want, the monitoring system will help you detect this at an early date.
• Data can point out which groups in the community are affected by your initiative. Is your initiative producing a lot of change in the schools, but little change in the criminal justice system? Is this what you and the community want and need? Who is being targeted? With the results of the monitoring system, you might be able to better answer these questions.
• Data can reveal which strategies are being addressed. The monitoring system can determine whether your initiative is offering information without following through with peer support or access to other resources.
• Data can be used by the staff to achieve a variety of results. Staff and leadership can use the data to promote community awareness of the initiative's activities and accomplishments, recruit community support, and secure financial resources.

How do you monitor your progress?

Before you can start analyzing your data, you first have to collect it. What follows is a guide to creating the observational monitoring system that you will need to collect the data.

Collect data by completing event logs and other forms on a regular basis

Event logs are written accounts of the major activities of the initiative. They might also be used to record any changes in the community brought about by the initiative, such as new programs, policies, or practices related to the initiative's goals and mission.

The event log might include important information such as:

• The month/day/year of the event
• A description of the event itself, including:
  • Why it was important
  • What happened as a result
• A description of the details of the event, including:
  • Who was involved
  • What organizations contributed people and resources
  • What community sector or objective this relates to
  • If this is the first time this event happened

For example, a group called the Adolescent Pregnancy Prevention Initiative in Franklin County, Kansas, participated in an event to help bring about changes in the community that are related to reducing adolescent pregnancy. At the next coalition meeting, Maggie B., a coalition member who helped organize the event, completed the information in the event log. She wrote: "Shannon P. and Gary M. participated in a meeting at Ottawa High School on February 14, 1996. They met with the high school principal to get permission to organize a Teen Action Committee in the school. The Teen Action Committee was approved by the principal. This was important because it will create leadership opportunities for youth."

Maggie recorded this information in a chart where she identified details such as the month, day, and year of the event, a description of the event itself, and a description of the details of the event.

Have several people, such as project staff and active members, complete the logs

The people who fill out logs will be those members who are taking action on behalf of the initiative. Ideally, everyone who is doing something for the initiative will complete event logs. Event logs should be completed no later than one week after the event took place. That way, the important details will be fresh in everyone's mind! The completed event logs can then be given to the people in the group who are in charge of evaluation.

Gather your information carefully and accurately

We've found that data collection in our own work takes about two hours to a week to complete. Of course, if your initiative is working on a particularly involved project one week, you might spend twice or even three times that amount of time gathering data. Carefully gathering your information will mean that the results that are generated will be more accurate, and therefore more valuable to your initiative.

Organize the data so that it can be used

Once the information has been gathered, it needs to be turned over to the people in your group who will put the data into some kind of organized form so that the initiative can use it in a beneficial way. This step in the process might take a good deal of time; but, don't be scared! As you have heard many times already, the benefits you receive from the evaluation process will only be helpful if you spend the time to carefully analyze the results. While this might take a few hours, in the end, you'll be glad you did it!

Summarize the data and distribute that summary to the group

Those who organize the information that is gathered will then take time to summarize the event logs. The results will then be distributed to members of the group.

In Summary

Congratulations! You've just finished learning one of the most important ways to evaluate a community initiative. Keep reading to see examples of event logs, and other monitoring charts. But don't stop there! More exciting sections await you and your initiative. The next chapters will introduce the constituent survey of goals, process, and outcomes, and other methods to measure the progress and success of your initiative. Remember, monitoring your initiative helps your group be the best that it can be!

You understand that monitoring my community initiative can help you:

___Weigh actions against the results to see if we're meeting the goals of the community and the initiative

___Understand the initiative

___Make decisions concerning the programming of the initiative

___Promote awareness of accomplishments

___Recruit support

___Secure funding

You understand these 3 key parts of a monitoring system:

___Process and outcome measures

___Observational system

___Regular feedback on performance

You understand that process measures can be used to monitor the following:

___Members who participate

___Planning products

___Media coverage

___Financial resources

___Services that are ultimately provided

___Community actions

You understand that outcome measures can be used to monitor the following:

___Changes in programs

___Changes in policies

___Changes in practices

You understand that the data you gather by using the monitoring system can help the group in the following ways:

___Data can tell you where the initiative places its emphasis

___Data can point out which groups in the community are affected by your initiative

___Data can reveal which strategies are being addressed

___Data can be used by the staff to achieve a variety of results

Data collection checklist:

___Have key project staff and active members of the initiative keep event logs of all events

The event logs include the following information:

___The month/day/year of the event

___A description of the event itself, including:

• Why it was important
• What happened as a result

___A description of the details of the event, including:

• Who was involved
• What organizations contributed people and resources
• What community sector or objective this relates to
• If this is the first time this event happened

___Complete event logs no later than one week after the event has taken place

___Give completed event logs to the people in the group who are in charge of evaluation

___Turn over the information gathered to the people in your group who will put the data into some kind of organized form so that the initiative can use it in a beneficial way

___Those who organize the information that is gathered summarize the event logs

___The results are distributed to members of the group

Example: Event Log for an Adolescent Pregnancy Prevention Initiative

Site: Franklin County

Recorder: Maggie B.

Date	Event	Description
(MM/DD/YYYY)	Give the broad information about the event. Include why it is important and what happened as a result.	Who was involved? What organizations were the collaborators? What community sector or objective does this relate to? Was this the first time this event happened?
2/14/2013	Met with principal of Ottawa High School to get permission to organize a Teen Action Committee in the school. The Teen Action Committee was approved. This was important because it will create leadership opportunities for youth.	Shannon P. and Gary M. Ottawa High School Schools Yes
2/21/2013	Central Heights School District board meeting was held to discuss administering a sexuality questionnaire. Survey was authorized for use at Central Heights. This is important because it is a new practice to gather information about knowledge, attitudes, and behaviors related to sexuality.	Midge R. and School Board members  USD 288 Schools Yes

 

Tool 1: Event Log

The event log form is designed to help you record major activities of your group's initiative, such as community actions. They are also used to record any changes in the community facilitated by the initiative.

Event log for: ______________

Site: ____________________                        Recorder: ____________________

For this form, please describe:

• Actions taken to bring about changes in the community related to your initiative
• Changes in programs, policies, and practices related to your intiative

Date	Event	Description
(MM/DD/YYYY)	Give broad information about the event. Include why it is important and what happened as a result.	Who was involved? What organizations were the collaborators? What community sector or objective does this relate to? Was this the first time the event happened?

Tool 2: Observational codes for the monitoring system

These tables provide a list of key events, definitions, and examples of items using the observational code for the monitoring system as well as general instructions on how to code key events on the event logs.

Codes for Internal Events

Code	Definition	Examples
Planning Product (PP)	Products or residuals of planning activities within the group	Statement of objectives Formation of committees or task forces with established members Grant application
Resources Generated (RG)	Acquisition of funding through grants, donations, or gifts in-kind	Materials received People's time (except volunteers) Money

Codes for External Events

Code	Definition	Examples
Community Action (CA)	Actions taken to bring about changes in the community related to the initiative's goals	Actions that attempt to modify or create a new program, policy, or practice Petitions
Community Change (CC)	Changes in programs, policies, or practices of governmental bodies - agencies, businesses, and other sectors of the community that are related to the initiative's goals	A new or modified service program A new or modified policy (e.g., tobacco control) A new or modified practice (e.g., regarding office hours)
Services Provided (SP)	Events that are designed to inform or educate people	Classes Workshops Communications such as bill stuffers
Media (M)	Each instance of radio time, television time, newspaper article, brochure, or newsletter	Radio TV (e.g., Public Service Announcements)
Other (O)	Items for which no other code applies, or no definition has been created