Chapter 39. Using Evaluation to Understand and Improve the Initiative | Community Tool Box

If you’re reading this section, the chances are you already know how important evaluation is to your effort. It can help you in a number of ways – in improving your methods, in appealing to funders, in supporting staff – but none is more important than understanding and improving the effort.

Often, evaluations fail to examine whether efforts are responsible for, or related to, changes and improvements in the community. They may examine the process or output (e.g., number of people trained), but they often fail to tell a story about whether conditions changed and whether this led to more distant outcomes.

It’s obviously important to know whether your methods are successful, and whether your intervention or initiative accomplished what it set out to do. In addition, however, you can evaluate whether your work changed community conditions, and whether it seems to be leading toward changes in the community or system. In this section, we’ll discuss how an evaluation can help you make those determinations.

What do we mean by answering evaluation questions?

Evaluation questions refer to what stakeholders – the community and funders, for instance – want to know about the functioning of the program or initiative. Here are some illustrative evaluation questions:

• Are participants satisfied with the program? (Process and implementation issue)
• How well is the initiative meeting its stated objectives? (Attainment of objectives)
• How much and what kind of difference did it make for the participants? (Impact on participants)
• How much and what kind of difference did the initiative make on the community? (Impact on the community)

So what’s a key evaluation question? That depends on what you set out to do (mission and objectives) and how you plan to get there (the logic of how the process and activities will produce immediate and longer-term outcomes). The path from here to there – from assessment to change in the environment to longer-term outcomes – is known as a logic model. Key evaluation questions focus on critical aspects of the logic model; for instance, “Is the initiative bringing about change in the environment?” or “Is that change in the community (the intervention) associated with improvement in outcomes for people in the community?

A note here: The Center for Community Health and Development at the University of Kansas, the source of the Community Tool Box, consults on evaluation, and has developed a Community Check Box system to support participatory research and evaluation of community health and development efforts. Much of the material in this section comes from the thinking that underlies this monitoring and evaluation system.

The documentation system makes it possible to answer key evaluation questions that get at the heart of an effort’s overall effectiveness. We’ll use this monitoring and evaluation system to illustrate how to address evaluation questions.

Regardless of your aims, your work can be considered on two levels. The first is whether, how, and why you’ve been able to accomplish the specific goals you’ve set for your effort – successful advocacy for the passage of a new regulation, the improved delivery of services, a change in people’s behavior, improvement in environmental conditions, etc. The second is the wider influence that your work has in the community or system, and how conditions have changed.

A decent comprehensive evaluation might examine how you planned and implemented your effort (process), whether you actually took the specific actions you set out to take, such as serving the number of people you expected in the ways you planned to (impact), and whether your actions had the results you hoped for (outcomes.)

A typical evaluation may measure ultimate outcomes but may fail to document changes in the community resulting from your work such as other similar programs starting, more funding becoming available for your issue, or more media attention. Furthermore, a specific effort may give rise to more general and sweeping changes even when it doesn’t initially attempt to. A campaign to reduce youth violence may achieve its immediate purpose, for instance, but may also change community programs and resources available to address the needs of adolescents and result in significant changes in the ways that community members engage youth. Sometimes, even an apparently unsuccessful effort can have a broader community effect by increasing awareness and understanding of a community problem or strength, or by changing the behavior of key community members or leaders. These broader effects can be harder to track than the more immediate achievement of the specific goals and objectives of an effort.

Some definitions before we start the next phase of discussion:

We usually think of a community as a group of people who live in one place – a city, a neighborhood, a town, a village, a rural area. We also use the term, however, to refer to a group of people with one or more common characteristic and common interest. Using that definition, we speak of the business community, the Latino community, or the school community. In this section, the word is generally used to mean community of place, because that’s the community in which organizations generally try to create longer-term change.

System refers to the arrangements or infrastructure that supports key functions – for instance, the public health system, the legislative system, the education system, or the legal system. The essential features of a system – preparation of the workforce, capacity to monitor and diagnose problems – also affect the conditions under which people live their lives.

At both the specific and general levels, it’s important to know whether any changes in the community or system you’re concerned with are actually taking place, what they are, and whether and how your work has influenced them (and vice versa).

Figure 1 below displays a framework (or logic model) for collaborative action, as well as evaluation questions that reflect potential associations along the logic model’s pathway.

For the logic of collaborative action, Fawcett and Schultz (2008) recommend four key evaluation questions:

1. Is your effort serving as a catalyst for community change related to its mission? In chemistry, a catalyst is a substance that, while remaining unchanged itself, changes the speed, intensity, or nature of a reaction. Is your effort, while doing its intended job, setting in motion other forces in the community (new or modified programs, policies, and practices) that lead to longer-term change related to the issue you’re concerned with?
2. What factors or processes are associated with the rate of community or system change? What encourages or gets in the way of change? Can you identify specific community events, actions by individuals or by your or another organization, or other factors that accelerate or impede the process of change in a community or system?
3. How are community or system changes contributing to efforts to promote community health and development or other mission-related goals? Are the changes you observe in the community in fact helping to bring about the realization of your goals – or related ones – or are they having little or no effect (or even a negative effect)?
4. Are community or system changes associated with improvements in population-level outcomes? Can you see connections between changes in the community (programs or policies for instance) and the kinds of positive changes in measuring success (e.g., fewer youth violence arrests, reductions in cigarette sales, fewer teens killed or injured in driving accidents) that you hope your effort will bring about?

Figure 1. Framework for collaborative community action and related evaluation questions (Source: Fawcett and Schultz 2008; Institute of Medicine, 2003).

Note (in Figure 1) that we can overlay evaluation questions on this logic model to address key (presumed) relationships along the path (see Fawcett and Schultz, 2008, in print resources).

Source: Adapted from the Institute of Medicine’s framework for collaborative public health action in communities (2003).

Answering these questions sets your effort in the context of the community and shows you whether it’s actually having the effects you want in a broader sphere, or whether it’s only influencing individual successful participants. Questions like these are not always easy to answer. Later in the section, we’ll discuss them more fully, as well as how an evaluation can be structured to answer them.

Why answer these questions?

In many cases, organizations are funded for several years at a time at best. This requires focus on fulfilling the conditions of the grant or contract, which often hinge on relatively short-term goals. There are, however, a number of compelling reasons for answering evaluation questions:

• To improve your work. The foremost reason for any evaluation is to make your effort more effective. By understanding exactly what affects progress on your issue, and the interaction between your actions and community system change (an intermediate outcome), you can see how to adjust your work to get the longer-term results you seek.
• To understand what affects the work in what ways. What actions or events seem to slow down the pace of change (e.g., loss of leadership), and what speeds it up (e.g., action planning)? What effects do small and large changes in the organization and the community have on your work?
• To see how to accelerate results. Once you understand what makes change move more quickly, you can take steps to make sure that those change-accelerating processes, events, or conditions become an integral part of your work.
• To understand specifically how broader actions, events, or conditions – a crisis or concentrated poverty – affect the work. Identifying and understanding these factors that affect the success or failure of an effort is critical – for the work underway and for future efforts.
• To understand what works to bring about community change, and adjust accordingly. The chances are you’re doing your work in order to permanently change some condition in the community or to improve the quality of life for a particular group or for the community as a whole. You hope that, eventually, the goals you’re working toward – the elimination of domestic violence, reductions in pollution, a decrease the incidence of diabetes, better schools for all children – become the community norms.
  
  Using an evaluation to answer key questions will provide information both about which small changes seem to lead to larger ones and about how to generate those and larger changes as well. That, again, will allow you to adjust your effort in order to make it more effective.
• To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effect. The loss of a competent leader, an increase or decrease in funding, the end of one phase of an effort and the beginning of another – all of these and many other factors can affect the rate and direction of change in the community. Knowing the results of events can help you determine how to best manage them.
• To show the community the value of your work. If you can show that your work has a positive influence on the community, you have a better chance of gaining broad support. That support, in turn, can lead to funding and in-kind help, volunteers, and community participation in ongoing social change.

Who should be involved in answering these questions?

The ideal is to engage in a participatory evaluation that draws in members of all stakeholder groups. This not only leads to buy-in and support on the parts of many sectors, but also provides multiple perspectives on such issues as how participants are treated, how the effort is viewed in the community, how non-participants’ lives are affected by your work, whether there are changes in the social climate, etc. You may want to conduct the evaluation as a participatory action research project, or you may simply want to involve as many individuals and sectors as possible in providing information and being kept current about the findings of the evaluation. Among the possible stakeholders:

• Participants in or beneficiaries of the effort. They may have a great deal to offer, both in terms of identifying the real needs of the population and/or the community and in reaching other members of the population of concern.
• Residents of a focused geographic area, whether or not they’re members of the population of concern. If your focus is on a neighborhood, a small rural community, or other limited area, you may want to involve all residents. Such involvement is likely to bring with it support for your effort and may increase understanding about the issue and serve to improve relations among these residents and the group you’re working with.
• Professionals and volunteers carrying out the work. The more deeply involved these people are, the better equipped and the more eager they’ll be to do their jobs well.
• Those whose jobs or relationships bring them into contact and involvement with the population and/or issue with which you’re concerned. These might include health care providers, teachers, family members of substance users, etc. Since these people may have to help in carrying out the effort, and since they may be asked to change their behavior as a result, it’s both important to the success of the effort and a matter of fairness to include them
• Funders and local officials. Funders have a vested interest in information about the effort, and their participation can also serve to help them understand the real impact of your work. Including local officials in an evaluation can forestall problems and help to gain community support. It can also lead to changes in official policy over the longer term.

​When should you set up and use an evaluation system to answer key questions about the effort?

As we’ve discussed throughout the Community Tool Box chapters on evaluation (Chapters 36-39), it should be planned and embarked upon at the beginning of the effort. This is especially true for the kind of evaluation we’ll describe in this section, because its purpose is to monitor and evaluate what is changing – organizational and community changes, longer-term outcomes, and to examine the interactions among them. If you don’t start the evaluation at the very beginning of the effort with community assessment and planning, and instead wait until the implementation phase, you’re at risk of missing something important.

Once you’ve started the evaluation, you should be collecting information constantly, recording not only what happens but when it happens, so that you can make connections. For this type of evaluation to be truly useful, those connections have to be made frequently, not just once or twice a year. The point here is to see what is happening, reflect on its meaning, and use the information to strengthen and speed up positive change, or to address and prevent negative consequences. In either case, acting swiftly can be important. You can do that if you have the information in hand.

How do you use the evaluation system to answer key questions about the effort?

Once you’ve assembled a participatory team to gather and analyze data, you’re faced with some important questions: What kinds of information do you need in order to answer the key questions that relate to your logic model, and how do you get it? Equally important, once you have the information, how do you sift through it to extract the answers you’re looking for? There is a general method of examining the data that should be useful even as you analyze each of your questions individually.

An important element to using your evaluation well is setting up the data so you can best see patterns and relationships. A particularly good way to record the data is on a graph, as in the illustration below. This graph shows the total number of community changes – new programs (e.g., an after-school program for youth) or new policies (e.g., a policy change that improves ways to health services) – related to the group’s mission over a five-year period. Some changes might be relatively minor – a single business changes its practice for protecting coworkers – while others might be as important as a new law or a drastic change in policy that completely alters the way the community addresses the issue. [Note: with a cumulative graph, each change is added to the prior one; so a flat line shows no change; and the steeper the line, the higher the rate of change.]

Figure 2. Cumulative number of community changes (new or modified programs and policies) brought about by the South Side Youth Development. Initiative from 2008-2014. [Source: Example graph from the KU Center for Community Health and Development's Community Check Box]

Other information – specific inputs or events – could be pinpointed on the graph in order to give a clearer view of potential influences on the rate of changes, as in the graph below. Here, added to a graph showing information similar to that in the one above, are the points in time at which various events in the progress of the effort took place.

From left to right, these are:

• The forming of a planning team
• The hiring of a new director for the effort
• The implementation of a social marketing action plan
• The hiring of new staff
• Action planning for the initiative
• A loss of leadership (probably when the director left to take another position.)

These events, singly or in combination, might have contributed to the sustained high rate of change or to decreased rates.

Figure 3. Cumulative number of community/system changes – and associated events – brought about by the Lincoln Coalition for Healthy living from 1999-2006.

By comparing the progress of the effort, as indicated by the total number and rate of community changes to the timing of recorded inputs and events, you can see that most of the events seem to have some connection with a speed-up or slow-down in the rate and number of community changes. There was, for example, a dramatic increase in the rate of changes when new staff members were hired and an equally dramatic decrease in that rate when the effort lost its leadership. This is the kind of information seldom captured by a typical evaluation, and yet it’s extremely important in understanding how to adjust your work for greater effectiveness and how to correct problems.

Other elements that might be recorded on a graph – individually or together – include statistics for population-level outcomes, rates or levels of change for different populations or neighborhoods, or for different approaches, statistics for comparison communities not involved in the effort, etc. The particular measures depend on your logic model, related research/evaluation questions, and indicators of success.

To get the information you need, you’ll have to decide what’s important to measure. Then, once you have obtained the data – not always an easy task– the next step is to analyze it, looking for patterns and connections. Let’s look at each of the key questions considered for the logic model in Figure 1 and discuss what kinds of data will help you examine it. The overriding question is how to adjust your work to make it more effective. Answering key questions for your logic model is really a way to ask the larger question: “What can we do to make our effort work as well as possible to accomplish our goals and realize our vision?”

What follows is an illustration of how evaluation questions can be answered with data, asking those key questions identified is the logic model often used by the Center for Community Health and Development (see Figure 1).

1. Is the initiative serving as a catalyst for community/system change related to its mission? Has your group’s effort brought about changes in the community or systems (e.g., public health, public safety, welfare, environmental protection, etc.) related to the mission? What specific changes related to the issue have taken place in the community, and when?

Some types of changes related to your work that you might want to document include:

• New or modified programs. These might be programs meant to address aspects of your issue or related issues; workshops or public awareness programs on the topic; community-wide events such as marches against violence or vaccination fairs; citizen initiatives, etc. They could be the product of formal bodies – the Chamber of Commerce, health or human service organizations – or of informal parents’ groups, neighbors, “friends of...” groups, and the like.
• New or modified practices. These refer to changes in how people do their work or deliver services. For instance, the clinic might change its hours to assure better access or the community might start a new partnership to address discrimination against immigrants.
• New or modified policies. New laws or regulations, changes in agency or corporate policies, etc. Other policy changes may be more informal; for instance, increased city services or store openings in previously underserved neighborhood, or changes in the way police or city officials respond to intimate partner violence.

Of course, depending on the project, other valued events may be tracked – these might include organizational changes, media coverage, resources generated, measures of outcome (addressed in later questions) and other key events.

Keeping track of the timing of changes is very important here, because it may indicate when those changes are associated with activities you’re doing, and when they result from other factors, such as a change in leadership (see Figure 3). Marked increases or decreases in the rate or amount of change that occur just after key events in the effort may point toward the effectiveness or ineffectiveness of these activities.

Changes often take time to show their effects or have to build to a certain level before their effects can be felt. Where there’s a trend toward change in a particular direction, even if it doesn’t appear to have any great immediate effect, it might be wise to track it over time to see whether its different changes in the environment – taken together – eventually influence indicators of success.

Another consideration is the presence or absence of patterns in the rate and type of change. Are there patterns in community or system changes that correspond with the work of the initiative (e.g., action planning), or with other identifiable factors (e.g., funding to hire a community mobilizer)? Recognizing these patterns allows you to adjust your work.

It wouldn’t be surprising, for instance, to find a more common pattern of community change in community health initiatives; for instance, lower rates during planning, higher and sustained rates after action plans and staff are in place; and lower rates near the end of funding as the project focuses on sustainability. Community groups take time to understand how to bring about change and integrate it into the efforts. Rapid change might require a period of support and relative stability. Once that has taken place, the community may focus on sustaining more important programs and policies.

As change levels off, you might be tempted to step up activity to counter that trend. That might include additional action planning or bringing on staff responsible for change efforts.

Other questions to ask here are whether the changes in the community or system are actually likely to improve outcomes (Are they great enough to have an effect? Is that effect likely to change the situation for the better?), and how you can obtain more accurate or more nearly complete data about what is occurring.

2. What factors or processes are associated with the existence and/or the rate of community or system change? There are a number of potential factors that you may want to monitor and analyze.

• The processes by which you conduct the assessment, planning, implementation, and evaluation aspects of the effort. The particular focus should be on use of evidence-based processes for community change and improvement (e.g., developing and using action plans; arranging for community mobilization; documentation and feedback.) That also includes examining who’s involved (Does adding certain people or sectors have an effect, for instance?), whether everyone fulfills her responsibilities, whether deadlines are met, how people are involved, whether the effort plays out as planned, etc. It also includes examining how everyone involved – staff, administrators, volunteers, planning team, board – fits into and advances or impedes the process. (See “Promising Approaches” for promoting community health and development.)
• People served or benefited. Here you might examine whether the people affected by the effort were those you planned for; whether you served or benefited the number of people you expected to; the amount of service or benefit that each participant received (if that’s possible to determine); how many participants dropped out, and why (and whether there were patterns to be found there.) (See the “Solve a Problem” troubleshooting guide for “There is not enough community participation.”)
• Significant program events. These might include:
  • Changes in leadership.
  • Staffing changes.
  • Changes in method or direction, such as a shift in mission to address a newly-stated community need. This kind of event can also include unintentional changes that result from staff turnover, loss of focus, lack of staff engagement, or failure to do the work properly.
  • Milestones of the effort: the completion or revision of an action plan; the beginning, revamping, or completion of a program; the arrival of a new group of participants; the completion of staff or volunteer training; the accomplishment of a particular objective, etc.
  • Increases or cuts in funding.
  • Unforeseen circumstances, such as safety or security problems; financial changes (a cut or increase in funding, difficulty in managing money); measures taken to respond to financial changes (layoffs, shifts in hours or location of services, changes in the number or qualifications of participants).
  • Changes in broad conditions – This might include natural disasters, epidemics, etc.
  • Community events, such as:
    • A change of local political leadership or government administration.
    • Racial or ethnic conflict.
    • An economic windfall (a corporation sets up a new facility in town, creating new jobs and a huge boost to the tax base).
    • An economic downturn (the town’s biggest employer moves all its business to another country).
    • A community tragedy or celebration (recent deaths due to violence; honoring of community heroes).

It also might consist of changes in how state and local governments conduct business, the priorities of key funders, or other aspects of the broader system that might have positive or negative effects on your work.

As noted in Figure 3, combining graphs and comparing the timing of the process and events with patterns of community and system changes can reveal connections among them. These connections, in turn, can suggest processes or activities you might take that could advance positive change or prevent negative consequences.

3. How are community/system changes contributing to efforts to promote community health and development? What kinds of effects are community and system changes – whether caused by your effort or not – having on the issue you’re concerned with and on community indicators of health and development in general?

One way to find out is to examine those changes in relation to various dimensions or aspects of your work. How are community changes distributed among and within different aspects of your effort? Do changes cluster around one or more particular aspects; for instance, in one section of the community new program (e.g., mostly in schools). Are most of the changes you see in the community related to population (e.g., Latinos) for instance? Are most of the changes related to particular goals (e.g., preventing violence, increasing childhood immunizations)? Should you make adjustments in your work to alter the situation in such cases, and, if so, what should the adjustments be? You can ask these questions in each case to examine the relationship of documented change to particular dimensions of your work, including:

• Group your goals or aims. These are the longer-term goals and objectives of your effort. If your program has the mission of promoting heart health in the community, for example, young groups’ goals might include promoting physical activity, reducing tobacco use, increasing healthy nutrition, and assuring access to health screening.
• The strategies of intervention you’re using. These might take such forms as providing information, enhancing services or support, or encouraging policy change.
• Risk and protective factors. These are the personal, social, and environmental factors that make it more or less likely that individuals and groups might engage in various types of behavior or experience a specific problem or condition. Belonging to a close and loving family might be a protective factor against youth violence, for example; an alcoholic parent might be a risk factor for substance use.
• The expected duration of change. Is your effort a one-time event, such as a one-day health fair? Or is it ongoing, such as a problem change or extension of health services to reach a specific population? How long do you expect the change in program or policy to be in place?
• The populations benefited. Depending on the nature of your effort, the populations might range from a small number of individuals from a specific group (Latino women at risk for diabetes) to most of the residents of a diverse community (families with children in school).
• The sectors addressed. These might include government (in the case of advocacy for changes in laws, regulations, or official policy), health organizations (if you’re addressing health promotion or access to health care), education (school improvement, school policy change), labor (creating more trade apprenticeships for particular underrepresented groups), business (workplace and environmental concerns), etc.
• The ecological level addressed. This refers to whether your effort focuses on individuals (as in an adult literacy program), relationships (an anti-discrimination initiative), the community as a whole (enhancing overall community quality of life and health – working to establish community-wide bike lanes, for example, or promoting an ongoing series of free concerts), or the broader society (national health care policy).
• Place. This might be a particular neighborhood, city or town, county, or rural area.

How are the community or system changes you’ve tracked related to each of these aspects of your effort? What are you seeing in the distribution of community/system changes related to particular goals, strategies, populations, places, etc.? What meaning do you give to those patterns – e.g., they reflect opportunities, barriers, key partner’s interests, etc. What are the implications for adjustment (if any)?

Figure 4. Distribution of community accomplishments (N = 125) brought about by the initiative by primary sector of the community in which the change occurred.

This pie chart makes it easier to see what pattern of change is occurring, to reflect on what it means (e.g., we encountered opposition from some sections) and to make adjustments. (See the “Solve a Problem” troubleshooting guides for supports for making adjustments to common problems in the work.)

4. Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort? Community-level indicators of an issue, as we discussed earlier, are markers of success at the level of the community as a whole, rather than for particular individuals. Some of those indicators can be found through checking available statistical information, including, among many others:

• Census data (e.g., percentage of population with different levels of education).
• Public health statistics (e.g., rates of infant mortality, percentage of children immunizations).
• Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.
• Statewide data and data from other communities (for comparison purposes).
• Police and court files (e.g., number of motor vehicle accidents).
• Educational data – standardized test averages, truancy, dropouts, primary or secondary school completion rates, incidence of school violence.
• Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc.

In addition to archival data, other sources of information can be used; including from observation, surveys, and other typical data-collection methods. Are there differences in reports by youth from one data-gathering period to the next of the prevalence of alcohol and drug use, smoking, or unprotected sex? Do observations show that there are more people, from one period to the next, using a public park, or out on the street at night? Are teachers seeing fewer incidents of violence in the schoolyard or in school in general? Are doctors seeing more or fewer cases of diabetes or TB?

The population-level indicators that tell you what you want to know aren’t always obvious. If you’re concerned about teenage drinking, for instance, you will probably want to examine statistics about traffic violations and nighttime single-vehicle crashes even where alcohol isn’t specifically mentioned as a factor.

Once again, by graphing your information, you can see relationships between population-level outcomes and community changes (the unfolding of the intervention.)

Figure 5 below provides an illustration. It displays the relationship between community changes (the unfolding of the intervention) to prevent child sexual abuse and the associated improvement in reported cases of child sexual abuse in a state (the population-level indicator of success).

Figure 5. An observed association between the unfolding of community changes (the intervention) and a decrease in reported cases of child sexual abuse in Minnesota.

Here, there is definitely a trend toward fewer child sexual abuse cases – a 28% drop in four years – as the number of community changes (the intervention) unfolds. Of course, this is simply an association; without suitable comparisons it does not demonstrate a cause-and-effect relationship, but this observed pattern is suggestive of a promising approach.

It’s possible that the number of cases will continue to fall even if the number of changes doesn’t continue to grow. The community may have reached a tipping point, where the changes in the environment lead to a combination of increased vigilance, by bystanders, increased reporting and follow-up, certain arrest and prosecution for perpetrators, and child-molester registration, all resulting in a decline in cases. Whether the community has reached this point can only be determined by tracking the potential relationship between changes and reported cases over time.

5. Did the effort lead to improvements in population-level outcomes?

To determine whether your initiative brought about community changes that had a positive influence on community-level indicators, you’ll need a graph similar to that in Figure 5.

You may use multiple graphs, one for each community-level indicator you’re concerned with (e.g., number of drunk driving deaths, incidents of youth violence, number of teen pregnancies, percentage of children entering school with all vaccinations, etc.) Each of these will compare the statistics for that indicator over time with the documented unfolding of your intervention. Once again, where improvements in the indicator coincide with or follow closely that implementation it’s possible that your work has had the desired effect.

You’ll also have to take into consideration whether there are other plausible causes for improvements in outcomes. Sometimes a significant community event unrelated to your initiative – another program, a major policy change, widespread media coverage – can influence the outcomes. The action may be independent of anything your effort is doing, even if it occurs at the same time. Thus, major community events and broader incidents should be tracked and graphed along with changes and the progress of your effort, so you can see whether you’ve actually had the influence you hope for. In addition, experiential designs can be used – such as interrupted time series designs across different communities – to try to rule out other plausible explanations of observed effects on outcomes.

Begin early and keep at it indefinitely

If evaluations that address the key questions are to be useful, they must begin early and be sustained. Collecting, recording, and analyzing data has to go on all the time, starting before the planning and implementation of the effort and continuing for as long as the effort itself continues. A case can be made that evaluation research should continue even after the effort has ended, in order to judge its lasting effects in the community. Whether that’s possible or not, understanding the real results of your work and continually adjusting it to be more effective is an important part of any successful initiative.

To sum up

There are some key questions that can help you understand the relationship between what your initiative does (process and activities), charges in community and systems (intermediate outcomes), and (longer-term) outcomes.

We recommend several key evaluation questions:

• Is the initiative serving as a catalyst for community/system change related to its mission?
• What factors or processes are associated with the rate of community/system change?
• How are community/system changes contributing to efforts to promote community health and development?
• Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort?
• Did the effort lead to improvements in population-level outcomes?

Answering these questions involves careful thinking about what data will be useful, and then careful collection, recording, and analysis of those data. Graphing what you’ve recorded and comparing or combining graphs can often make analysis easier. The results should give you a great deal of information about exactly what works and what doesn’t, where you need to make adjustments, and what kinds of changes you should be trying to bring about in the community to address the issue.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

Here you will find a checklist summarizing the important points of the section:

What do we mean by answering evaluation questions?

__ Evaluation questions refer to what stakeholders – the community and funders, for example – want to know about the functioning of the program or initiative.

Some illustrative evaluation questions:

__ Are participants satisfied with the program? (Process and Implementation Issue)

__ How well is the initiative meeting its stated objectives? (Attainment of Objectives)

__ How much and what kind of difference did it make for participants? (Impact on Participants)

__ How much and what kind of difference did it make on the community? (Impact on the Community)

Why answer the key questions?

__ To improve your work.     

__ To understand what affects the work in what ways.

__ To see how to accelerate results.

__ To understand specifically how broader actions, events, or conditions – e.g., a crisis or concentrated poverty – affect the work.

__ To understand what works to bring about community change, and adjust accordingly.

__ To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effects.

__ To show the community the value of your work.

Who should be involved in answering these questions?

__ This type of evaluation works best as a participatory effort.

Those who might be involved include:

__ Participants in or beneficiaries of the effort.

__ Residents of a geographic area you’re focused on.

__ Professionals and volunteers carrying out the work.

__ Those whose jobs or relationships bring them into contact and involvement with the population and/or issue you’re concerned with.

__ Funders and local officials.

When should you set up and use an evaluation system to answer key questions about the effort?

__ Start at the very beginning of the effort, so you can record the whole of your process – outreach, planning, implementation, and evaluation.

__ Continue gathering, recording, analyzing, and using data throughout the course of the effort.

How do you use an evaluation system to answer key questions about the effort?

__ Decide what information you need to answer each question.

__ Decide how to gather that information.

__ Devise a method for recording and setting up your data that makes it easy to analyze.

__ Graphing is particularly good because it allows you to easily compare different sets of data.

__ Consider each question separately:

Is the initiative serving as a catalyst for community/system change related to its mission?

Changes to look for include:

__ New or modified programs.

__ New or modified practices.

__ New or modified policies.

What factors or processes are associated with the rate of community or system change?

__ The processes by which you conduct the assessment, planning, implementation, and evaluation aspects of the effort.

__ People served or benefited.

__ Significant program events. These might include:

• Changes in leadership.
• Staffing changes.
• Changes in method or direction.
• Milestones of the effort.
• Increases or cuts in funding.
• Unforeseen circumstances.

__ Community events, such as:

• A change of local political leadership or government administration.
• Racial or ethnic conflict.
• An economic windfall.
• An economic downturn.
• A community tragedy or celebration.

__ Changes in broader conditions.

How are community/system changes contributing to efforts to promote community health and development?

__ Examine community changes in relation to various aspects of your work.

• Group goals or aims.
• The strategies of intervention you’re using.
• Risk and protective factors.
• The expected duration of change.
• The populations benefited.
• The sectors addressed.
• The ecological level addressed.
• Place.

Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort?

__ Community-level indicators of an issue are markers of success at the level of the community as a whole, rather than for particular individuals.

__ Population-level outcomes can be found or inferred by consulting publicly available statistics and records, including:

• Census data.
• Public health statistics.
• Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.
• Statewide data and data from other communities (for comparison purposes).
• Police and court files.
• Educational data – standardized test averages, truancy, dropouts, high school completion rates, incidence of school violence.
• Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc.

__ You can also collect other sources of data, using observation, surveys, and other methods.

__ Graphing and comparing sets of data on community-level indicators can show connections among them.

Did the effort lead to improvements in population-level outcomes?

__ The effort may have led to improvements by influencing changes in the community or system (intermediate outcome) and (more distant) community-level indicators of success. that brought them about.

__ Comparing data on population-level outcomes, community changes, and the timing of various phases and events of your effort can help you understand the connections among them.

__ Begin early and continue to gather, record, analyze, and use data indefinitely to understand how to adjust your effort for greatest effectiveness.

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What is providing feedback?

• Why provide feedback?

• When should you provide feedback?

• How should you provide feedback?

What is providing feedback?

When we talk about providing feedback from your evaluation, we mean presenting the data on accomplishments that you're getting from your evaluation to those involved in the initiative – line staff and volunteers, as well as administrators and board members. Depending on the methods of the evaluation and the type of data involved, some of it may be presented in the form of tables and graphs, some as narrative, and some perhaps as portfolios or even audio or videotape.

Feedback is a two-way street. The folks engaged in the day-to-day operation of the organization and its programs can benefit from learning what seems to be working well and what doesn’t, and from understanding what needs to be changed to improve results. At the same time, administrators and board members need to understand where the weaknesses in the organization are that keep staff and volunteers from being as effective as possible.

Is there enough support? Are there enough supplies? Is funding adequate to get the work done? Is the organizational climate one of mutual respect that’s conducive to staff and volunteers doing their best, and to participants feeling welcomed and supported? These are questions that staff and volunteers – and participants – can answer at the same time that the organization grapples with monitoring results. To be really useful, feedback has to travel in both directions.

Providing feedback should be done on an ongoing basis so that all can be kept up-to-date on what they're doing well and what can stand improvement. It can also be done at the end of an evaluation (although not exclusively so).

In this section, we'll mainly be discussing the tracking of activities and accomplishments that may lead to longer-term outcomes, usually referred to as monitoring.

Why provide feedback?

Here are a few reasons to consider providing feedback from time to time:

• To help community leadership assess progress towards meeting the initiative's goals
• To help see areas wherein the members of the initiative may want to put more energy
• To help detect when too much energy or effort is spent in areas less central to the mission
• To provide the opportunity to celebrate small accomplishments
• To help the initiative focus on the "big picture" by seeing cumulative accomplishments over time
• To provide funders the opportunity to help re-direct the initiative towards activities more directly related to the mission
• To provide funders the opportunity to see and reward the accomplishments of the initiative

When should you provide feedback?

Your staff should get feedback at regular intervals, especially early in the initiative's development, so that they can continually adjust their efforts to improve the initiative's results. You might want to have feedback sessions once a month for the first year and then after that do them on a quarterly (four times a year) basis. Giving people feedback often gives them the chance to see what changes have resulted from adjustments they made after the last feedback session, so that you can all work together to continuously improve your efforts.

How should you provide feedback?

Begin with an overall statement summing up how the initiative is doing

This statement should be affirming – you don't want to start in on areas that need to be approved right off the bat – and it should be very specific about what is going well. For example, "We've received 35 letters from parents praising the school safety committee's work at Uptown Middle School" is a much more specific statement than "A lot of parents are pleased with the school safety committee."

Perhaps the data you've gathered will indicate that everything your group has been doing is perfect. If this is the case, presenting feedback to your group will probably be a breeze. However, it's more likely that you'll have at least a few areas, or even – perish the thought – a lot of areas that need improvement. You want the people involved in your initiative to understand what they could be doing better, but at the same time you will have to be very careful to keep them from feeling discouraged, angry at themselves, resentful, or insulted by less-than-glowing feedback. How do you do this?

Present the data to the group as a good thing

It's important to present the feedback as a good thing, even if the feedback isn't all that positive - after all, you're giving them information that will help them do a better job. Try to communicate the value of a group that really wants to know how it's doing. Show them how using data gathered for your evaluation can help them do that. Provide a shared vision of the initiative as a catalyst for change: "We can make a difference and we will make a difference, and here's how we can do it!" Communicating your optimism that the group will ultimately succeed in having an impact is very important. In other words, start off with the good news, and then present any bad news in a way that encourages everyone to work hard to change it.

Show graphs

We suggest doing it in this order:

• Process Measures - number of people involved with program, member satisfaction, amount of activity to make changes in the community relative to the groups goals
• Intermediate Outcomes - the number of changes in the community (e.g., new or modified programs, policies, practices), information about the context of the initiative and critical events
• Ultimate Outcomes/Impact Data - data from surveys of behavior change among people targeted in the intervention, community-level impact data

Show non-graphic material

As we mentioned above, there may be a good deal of material – narratives, examples of participant work or learning, participant comments – that isn’t best provided in graphic form. This kind of material also needs to be discussed differently, in that what it communicates may not be as clear-cut as the data presented in graphs.

Like graphic data, non-graphic data can be examined in relation to process, intermediate outcomes, and long-term impact. Unlike graphic data, it might take some creativity to understand the trends or conclusions it demonstrates. (A trend is movement in a given direction that isn’t yet definite or far-reaching enough to be definitely said to have reached or failed to reach a goal.)

Much of this data might be best understood in discussion. What specific actions, methods, or conditions do participant comments refer to, and how could those be changed, if they need to be? What does participant work show about the effectiveness of your work? What do anecdotes illustrate about challenges, and about things that need to be adjusted or eliminated?

Most of this is qualitative data, data that can’t or shouldn’t (because it would lose the subtlety of what it can tell you) be expressed in numerical form.

Present the information

Here are some tips on how to do this:

• Begin with positive information
• Start by asking the group how they would interpret the data - if your interpretation is different, or if there are differing ideas among the group members, discuss why that might be so, and how to resolve the differences.
• Give a more detailed example from accomplishments that have happened recently – within the last few months, if possible
• Point out what is positive about the data
• Discuss with the group any trends in the data, and what those trends mean
• Discuss any suggested re-direction of efforts
• Review examples listed on the graph or in the qualitative information
• Affirm the initiative by noting evidence of progress
• Discuss any comments or questions group members may still have

If the measure doesn't show recent growth:

• Have the group discuss, using the data, why they think this might be so
• Point out any previous times that growth occurred to keep folks from getting discouraged - many of these measures will be cyclical in nature – resources generated, for example, often are recorded in spurts associated with funding deadlines. If it seems appropriate to point this out, do so.
• Ask the group for ideas about how to adjust the work in order to address the issue, if it needs to be addressed
• Ask if there were additional activities that were not recorded. List items that have been recorded over the life of the initiative. You may find things that didn't get recorded, and this could make a big difference in the results.

Summarize the data by discussing strengths of the initiative

End on a positive note! You want your staff and volunteers to come away from this presentation with the resolve to work hard to improve the initiative, and you want to bolster their confidence that they can accomplish this – so don't let them leave feeling bad about their efforts up until now. Review the graphs, and be sure to show the graph most directly related to the mission of the project (most likely, this will be the graph of community change).

In Summary

Let's recap the main things to keep in mind when you prepare feedback for your staff:

• Present this data as a gift, informing your audience about how the initiative is doing
• Focus on the positive – be affirming!
• Convey optimism about the prospects for success (if this is appropriate)
• Convey the need for changes or adjustment (if this is appropriate)
• Convey a shared vision of the initiative as an effective catalyst for change

Remember to keep it positive and give lots of encouragement to the people working on the initiative. Feedback sessions are meant to make your staff and volunteers want to work harder and make the initiative better, not to dishearten them. Good luck!

What does it mean to provide feedback to improve your intiative?

___Providing feedback from your evaluation involves presenting the data on accomplishments that you're getting from your evaluation to those involved in the initiative - your staff and volunteers

___Most of this information is usually, but not always, presented in graph form

___Providing feedback should be done on an ongoing basis so that your staff and volunteers can be kept up-to-date on what they're doing well and what can stand improvement

___Providing feedback can also be done at the end of an evaluation (although not exclusively so)

___Your staff should get feedback at regular intervals, especially early in the initiative's development, so that they can continually adjust their efforts to improve the initiative's results

What are some reasons to consider providing feedback?

___To help community leadership assess progress towards meeting the initiative's goals

___To help see areas wherein the members of the initiative may want to put more energy

___To help detect when too much energy or effort is spent in areas less central to the mission

___To provide the opportunity to celebrate small accomplishments

___To help the initiative focus on the "big picture" by seeing cumulative accomplishments over time

___To provide funders the opportunity to help re-direct the initiative towards activities more directly related to the mission

___To provide funders the opportunity to see and reward the accomplishments of the initiative

How should you provide feedback?

___You began with an affirming and specific overall statement summing up how the initiative is doing

___You've presented the data to the group as a good thing, even if the feedback isn't all that positive

___You've tried to communicate the value of a group that really wants to know how it's doing and provided a shared vision of the initiative as a catalyst for change

___You've shown graphs and qualitative material in this order:

• Process Measures
• Intermediate Outcomes
• Ultimate Outcomes/Impact Data

___You’ve asked the group for its interpretation of the data

___You've introduced the information and provided a brief definition, example, and explanation of why it is important

___You've given a more detailed example from accomplishments that have happened recently - within the last few months, if possible

___You've pointed out what is positive about the data

___You've made note of any trends in the data, and what those trends mean

___You've discussed any suggested re-direction of efforts

___You've reviewed examples listed on the graph

___You've affirmed the initiative by noting evidence of progress

___You've asked for comments or questions from the group

How should you present the information If the measure hasn't shown recent growth?

___Ask the group for its ideas about why this is so

___Point out any previous times that growth occurred to keep people from getting discouraged

___If the measures are cyclical in nature and it seems appropriate, point this out

___When meeting with the initiative's leadership, ask if there were additional activities that were not recorded and remind leaders of the definition and list items that have been recorded over the life of the initiative

___Summarize the data by discussing strengths of the initiative

Example 1: Sharing Information through Participatory Evaluation—Enroll Wyandotte: A Community Mobilization Effort for the Affordable Care Act

Background

The passage of the Affordable Care Act (ACA) created for the first time the opportunity for Americans to receive health insurance that previously could not afford it. Healthcare insurance marketplaces were established to ensure healthcare access to those in need. State-level support for ACA marketplace enrollment varied considerably from the development of state-managed health exchanges to erecting barriers to impede marketplace access. In Kansas, local efforts were required to assure access to healthcare through community mobilization and partnerships. Wyandotte County (Kansas City), Kansas (pop. 160,384) is a low-income and culturally diverse area that ranks 96 out of 98 in the county health rankings. One in four residents are without health insurance, a rate twice that of the national average, with an equal number of residents below the national poverty level.

Context

Enroll Wyandotte, a local ACA exchange navigation effort, engaged multiple partnerships across a different sectors to support enrollment. Collaborative partnerships included government health organizations, public institutions, hospitals and health organizations, community and cultural organizations, community health coalitions, and universities.

Efforts of the Initiative

The initiative employed a community mobilization approach to assist Wyandotte County residents with obtaining health insurance through the ACA marketplace exchanges. This effort consisted of four broad components: a) Providing information and enhancing skills (eg, one-on-one education through the walk-in enrollment center, door-to-door education referral campaign to identify eligible community residents); b) Modifying access, barriers, exposures, and opportunities (eg, one-on-one enrollment assistance through the walk-in enrollment center, conducted community enrollment events in collaboration with community partners); c) Enhancing services and support (eg, hired a full-time navigator/project coordinator, trained a workforce of university volunteers as Certified Application Assistors); and d) Modifying policies and broader systems (eg, established collaborative partnerships with community organizations, provided testimony to senate committees on legislation that would have required expensive certification of volunteers trained as Certified Application Assistors).

Participatory Evaluation

A participatory evaluation involved the leadership of the Enroll Wyandotte effort and staff of the Center for Community Health and Development at the University of Kansas. The partners developed a monitoring and evaluation plan that identified some key indicators for success: a) Community/organizational changes (ie, new or modified programs, policies, or practices to facilitate enrollment in the ACA; eg, expanded enrollment hours at the health department; new sites at local public libraries, community organizations); b) Services provided (eg, media communications to promote awareness, direct assistance in enrollment sites); and c) Created accounts (the total number of family accounts created within the ACA website at three enrollment sites).

Measurement and Results

The KU Center for Community Health and Development designed an online documentation and support system (the Community Check Box) that included prompts and supports to: a) Capture instances of community/organizational changes and services provided (ie, what was done when and by whom toward what goal; b) Code by type of activity (ie, using definitions of community/organizational change and services provided); c) Characterize the activity (eg, by goal, strategy used, sector in which implemented); and d) Communicate progress (eg, graphs of the unfolding of community/organizational changes over time). The results showed that activities that created lasting changes in the environment (eg, programs, policies, and practices) and services provided were associated with increased enrollment. More than 500 new accounts were created as a result of the work of the initiative.

Providing Feedback through Participatory Sensemaking

Staff from the KU Center for Community Health and Development reviewed documented activities captured using the Community Check Box to assure data quality and to provide feedback on a monthly basis. The Check Box also supported participatory sensemaking by pairing graphs of activities/outcomes with reflection questions: a) What are we seeing? b) What does it mean? and c) What are the implications for adjustment? Enroll Wyandotte and KU Center for Community Health and Development staff met on two occasions to review the data and apply these questions. Participatory sensemaking provided an opportunity to evaluate data patterns to check for data completeness and identify key events (eg, when grants were received, when essential community/organizational changes occurred) that were essential to implementation of the marketplace navigation initiative. For example, the illustrating the unfolding of services provided over time showed a marked increase at the end of December, 2013. Enroll Wyandotte staff identified that this was associated with the hiring of a project coordinator made possible by community foundation funding and collaboration with community partners. Additionally, the use of sensemaking included the leveraging of additional funding from grantmakers and community partners through the use of data from the evaluation. In this initiative, providing feedback through participatory sensemaking helped guide systematic reflection of the data to make adjustments for continuous improvement and to communicate an objective story to community partners, grantmakers, and other stakeholders.

This example is based on the following article:

Fawcett, S. B., Sepers, C. E., Jr., Jones, J., Jones, L., & McKain, W. (Submitted for publication) Participatory evaluation of a community mobilization effort to enroll residents of Wyandotte County, Kansas through the Affordable Care Act.

A Community Health Center conducted an evaluation of its program to promote physical activity among those with higher risk for heart disease. The evaluation showed mixed results. A small number of participants (15%) had very good outcomes. They had marked increases in physical activity and improved nutrition. Their fitness improved and they lost weight. As predicted, their blood pressure dropped, their pulse rates went down, and they reported feeling more energized. They reported high levels of satisfaction with the program and results.

A large majority of the original group (70%) exercised, but not as regularly as hoped. The health benefits for this group varied, with several reducing blood pressure at least slightly, and the rest maintaining the levels they had entered with.

A final group (15%) consisted of dropouts – several participants left the program, most within a short time – and other people who simply never managed to exercise on any schedule at all. There was virtually no change in their weight, blood pressure, or sense of well-being...except for a small number that had relatively positive results.

What could the Community Health Center do with these results? It knew that, while the intervention apparently worked if people stuck with it, the program was only partially successful. How could it use the evaluation to improve the program, and so improve the health of those it served?

This chapter so far has discussed the elements of conducting a research-based evaluation. But evaluation itself is only a means to an end: a tool to help you see what is happening so you can improve the effectiveness of your work. In this section, we’ll examine how you can use your research – the results of your evaluation – to do just that.

What do we mean by refining the intervention?

Data allow you and other group members to critically reflect on your work and look for opportunities to improve.

Some key reflection questions that you and your group might consider:

• What are we seeing? (e.g., amount and kind of activities implemented; results shown – increases, decreases, trends)
• What does it mean? (e.g., was the introduction of the intervention associated with changes)
• What are the implications for improvement? (e.g., do the results suggest that the intervention should be sustained, altered, discontinued; what changes are suggested)

The reflection questions you ask will depend on the nature of your intervention, but the above set of questions is a good starting point. Consider holding a meeting or brief retreat where the evaluation results can be presented through graphs and charts, and key questions can be discussed. Such a meeting might benefit from an experienced facilitator to keep the process moving toward consensus for specific recommendations on how to improve.

Refining the intervention is the process of making your work more effective by using data collected from your evaluation.

Depending on what you’ve learned from this data, you might want to:

• Increase or strengthen your intervention in certain areas or with particular groups
• Change or eliminate elements of the intervention that didn’t work well
• Adjust your intervention to changing conditions or needs in the community

It will be important for you to meet with other members of your group to review the data, identify key areas for improvement, and brainstorm and come to consensus on how to address issues that have been raised. Careful attention to your evaluation results can help inform which courses of action you should take to improve your efforts.

To continue with our example from above, the Community Health Center staff and selected participants met to review the results. They felt that the evaluation had shown that if people exercised regularly, they could lower their blood pressure, lose weight, and improve their overall health.

A key implication of the findings was how to help people establish and stay with an exercise routine.

Further dialogue about results of the evaluation left the Community Health Center with additional questions:

• How can we increase the number of participants who actually adopt and continue regular exercise and other healthy behaviors?
• Why did some people who didn’t exercise regularly reduce their blood pressure, and should we add another component (e.g., healthy nutrition) to our program?
• What other factors, if any, besides exercise seem to help participants exercise regularly and lower their blood pressure (e.g., wellness group, medication)?

By focusing on the key reflection questions – What are we seeing? What does it mean? What are the implications for improvement? – the center should be able to refine their program to get even better results for more participants.

Why should you use your evaluation research to refine the intervention?

Refining the intervention is the primary purpose of an evaluation. If you find out that your intervention wasn’t effective, you have three choices: you can quit; you can blindly try another approach; or you can use your evaluation research to guide you towards a more effective intervention.

Using evaluation results is vital: it points you in the direction that your research tells you is apt to be most helpful. Using research to help you choose your course of action also establishes you as a credible and practical organization, one that’s concerned with what works. That kind of reputation is likely to increase your opportunities for getting funding and other resources, and to gain and sustain your community support. Most importantly, it helps the group succeed in addressing the important problems or goals of your community.

When should you refine the intervention?

The short answer to this question is “constantly.” Monitoring and evaluation should go on throughout the life of the program or project, and should be used to adapt and adjust what you do on an ongoing basis. In practical terms, it’s wise to reevaluate your work regularly – once a year is typical – and make any major changes at that time. Of course, you can and should make minor adjustments throughout the year, based on your monitoring and on feedback from participants, staff, and others who implement or experience the intervention.

There are, in addition, some specific times when adjusting your work can be especially helpful:

• When what you’re doing isn’t working. If it’s obvious that your work isn’t having the desired effect, it’s time to consider what you need to change.

Make sure that you allow enough time for a program or intervention to have an effect before you make a judgment that it isn’t working. Nothing happens overnight, and the more difficult the issue you’re addressing, the longer it’s likely to take to influence intended outcomes. You have to walk a line between cutting a program off before it’s had time to work and letting it go on after it’s shown itself to be ineffective.

• When participants are dropping out at a high rate. What are you doing – or what are the external factors – that might be causing participants to leave your program? How can you change the intervention to assure that people experience it long enough for them to benefit?
• Between sessions of a time-limited or sequential program. Some programs – like the exercise program used as an example – are only designed to run for a limited period, but may run again and again, with new participants each time. If such a program is continually evaluated, you’ll get – and should use – information each time that will help you make the next round of the program better.
• When funders or participants ask you to adjust some aspect(s) of your program. Your evaluation research should be helpful in determining how to respond to the funder’s or participants’ requests.
• When funding or other resources are reduced. You may be faced with eliminating parts of your program, cutting numbers of participants, or other unpleasant choices. Your evaluation research can help you find the best way to make cuts without losing your effectiveness, by keeping intact the elements of the program that make the most difference.
• When the issue or goal changes. Sometimes there is a shift in priority issues for the community following a rise in unemployment or violence. Your research can tell you that, and suggest ways of dealing with the change in conditions.

Who should be involved in refining the intervention?

The best plan here is to involve a number of stakeholders, depending to some extent on who has been involved in the planning and evaluation of the effort.

Some people who definitely should take part:

• Participants. These are the folks who experience both the intervention itself and its effects, and they are likely to have ideas about what would make it better, easier for them to participate, or more relevant for them. Participants should be your partners in refining programs and interventions, since they have an inside perspective on whether they are working.
• Staff members, paid or volunteer. Like participants, staff members have a unique perspective on the intervention. Not only do they see the way it works every day, but they’ll also have to carry out any changes. If they can claim ownership of those changes by participating in the planning process for them, they’re far more likely to understand them properly and to be eager to make them work.
• People who are directly or indirectly involved in supporting the work. Depending upon the nature of your issue, these might include educators, government officials, health professionals, employers, funders, or others. Since their contribution is needed to make any changes successful, it’s important that they have input into the planning of those changes. They’ll need to understand and support them if the adjusted intervention is to go well.
• Those who led and participated in the evaluation. They’ll have a good handle on what the evaluation showed, and a grasp of what might need changing and how.

For example, the Community Health Center put together a team to look at the evaluation results and make some recommendations for changes in the program. The team included a variety of participants who had experienced different outcomes, a health care provider, a Center board member, and a staff member from the university that conducted the evaluation. They went over some of the research that the Center had used in developing the program, and carefully studied participant interviews and other evaluation material, as well as the records kept by program staff.

How do you refine an intervention based on research?

Changes in interventions should be focused on one or more of the three aspects of evaluation: Process (both your process – activities implemented, doing what you intended, etc. – and participants’ process – what did they actually do?), impact, and outcomes. You have to examine each of these separately, and ultimately integrate them to decide what adjustments you need to make in your intervention.

Each aspect of the evaluation builds on what comes before. In order to have the impact you want, you have to put together and run your program well, and that’s a matter of process. If your process didn’t go properly, then you haven’t really conducted the program you planned for. If you didn’t get the impact you hoped for, it may be due to the fact that you simply didn’t do what you planned, and the first adjustments should be to the process, to ensure that the intervention is implemented as intended.

Similarly, to get the outcomes you intend, the program has to have an impact on the appropriate risk and protective factors or other environmental conditions. If the program had the impact you envisioned, but not the outcomes, then adjustments need to take place at the impact level, perhaps in the risk and protective factors and/or conditions that influence outcomes.

Process

An evaluation of the process of your effort compares what you planned to do with what you actually did.

Process has a number of elements to which evaluation might be applied. They encompass both logistics (the handling of details, such as finding space and buying materials) and program implementation (methods, program structure, etc.).

These elements can include:

• Community participation. Were you able to involve members and sectors of the community that you intended to? Were you able to make good contacts and establish relationships within the priority population?
• Community assessment. Did you conduct an assessment of the situation in the way you planned? Did it give you the information you needed?
• Program planning. Was the planning participatory? Did it include research into best practices and successful interventions? Did it result in an approach that everyone felt would work?
• Staff hiring and/or volunteer recruitment. Did you hire staff and/or recruit volunteers that were the right people for the jobs?
• Staff and/or volunteer training. Were staff and/or volunteers oriented and trained before they started, so that they knew what they were doing when they began work? Was there ongoing training?
• Outreach to and recruitment of potential participants. Was outreach successful to engage those from the groups intended? Were you able to recruit the number and type of participants intended?
• Implementation strategy. Here, you’re determining both what you actually did in implementing the program, and what participants actually did. Did you structure the program as planned? Did you use the methods you intended to? Did you arrange the amount and intensity of services, other activities, or conditions as intended? Did you obtain and use the materials and equipment you expected to? Did relationships develop as envisioned? Did participants actually do or experience what you had intended?
• Evaluation strategy. Did you conduct the evaluation as planned? Did you gather data related to process, impact, and outcome?
• Timelines and benchmarks. Did you complete or start each of these elements in the time you planned for? Did you complete key milestones or accomplishments as planned?

If all or most things went as planned, and any that didn’t were trivial, you’ve essentially done what you set out to do. If they didn’t, there are a number of possible reasons for changes in the intended process:

• It took more time than you expected to complete one or more important tasks (finding and hiring key staff is a typical one here)
• It was harder than you expected to accomplish a particular task. This may be a matter of time spent, but it may also mean that you simply didn’t have the skills or personnel to do what you needed to
• Something you had good reason to expect didn’t happen (e.g., funding or support that you expected didn’t come through)
• Someone or some organization you depended on didn’t come through (e.g., a hired staff member became ill and did not finish the work on time)
• More participants dropped out than you anticipated
• More people participated than you anticipated
• Partway through, you found that the methods you had planned didn’t work well, and you had to make adjustments
• A funder or community advisor board asked you to change some of what you were doing
• Partway through, you became aware of a new method that seemed to be extremely effective, and you switched to implement it
• You discovered a more successful way of doing things in the course of the work, and adopted it
• You underestimated the resources necessary to carry out your original plan, and had to scale back (or look for more funding/volunteer help/space/other support)
• You encountered opposition
• You encountered unexpected difficulties (someone quit, materials/equipment weren’t available from the supplier)
• You encountered disaster (e.g., the site burned down, the program coordinator became ill, a staff member got arrested or misused your funds)
• You simply didn’t pay attention to following the plan, and/or didn’t do your job as an organization

The deviation from your plan may have had made very little difference at all, or it may have made all the difference. Some differences might be positive – a delay might make it possible to find a more stable funding source; a change in method might make for a more effective program – but they’re still differences. It’s worthwhile to understand what changed to make sense of the evaluation results and to make any needed adjustments.

Perhaps you implemented your process according to plan, and your program ran as intended. By contrast, the process may have been filled with difficulties – opposition, no community support, difficulty recruiting participants, missed deadlines. Does that mean that all the work you put into planning was unnecessary?

It’s likely that the answer is the opposite. If you were able to carry off implementing a program regardless of the fact that your plans were disrupted, it’s a good bet that the clear vision of what you wanted to do kept you on track.

Taking a close look at how you managed to overcome the obstacles in your way will help you understand how to avoid them in the future. (Avoiding all obstacles is unusual in any community work. The key is learning how to anticipate and overcome them.)

It’s also possible that the process leading up to the program went as planned, but the implementation didn’t turn out as expected. In that case, it was probably your plan that was at fault.

Some possible problems:

• You didn’t assess the situations or take some important aspects of preparation into account
• You didn’t properly understand some aspect(s) of what you had to do to be successful
• You didn’t properly communicate some aspect(s) of what you had to do to staff, participants, funders, or the community
• You underestimated the amount of money or other resources you would need
• You didn’t have proper fiscal control
• You ignored something important (treating participants with respect, for instance)
• You didn’t involve the community enough
• You didn’t factor in enough time for some aspect(s) of what you had to do (i.e., you planned for a given time period and carried that out, but it was too short)
• You didn’t provide some important support for participants (travel, child care, stipend), and a large number dropped out as a result

Finding out why your plan didn’t produce the intervention you expected can be helpful. Understanding what you need to plan for, and how to do it, can make your future work both more efficient and more effective.

Impact

Your program or initiative’s impact is the effect it had on the environmental conditions, events, or behaviors that it aimed to change (increase, decrease, sustain.)

In most – but not all – cases, the immediate impact of the program is not the same as the eventual intended results. Generally, a program aims only to influence one or more particular behaviors or conditions – risk or protective factors. The assumption is that such influence will then lead to a longer-term change, which is the ultimate goal of the program.

The intended impact of the Health Center’s exercise program, for example, is the adoption by participants’ regular exercise, a protective factor in reducing risk for chronic diseases. The goals of the program, however, are actually better heart health, and, ultimately, a longer and higher-quality life. Impact is the intermediate step – the influence you have on a behavior or other factor that will in turn lead to the intended results.
 

Your process might have gone perfectly – you might have done exactly what you set out to do – and might still have had no impact on the risk and protective factors you targeted. By the same token, you may have ended up running a program markedly different from the one you planned, and still have had the impact you hoped for. The results of the process evaluation will tell you how closely you stuck to your plan in setting up and running your program. The results of your impact evaluation will tell you whether your program made the changes or intended results.

Your program worked as you planned if the behaviors and risk and/or protective factors changed in the ways you intended. The big question that remains in this case is whether the changes your program influenced led to the ultimate outcomes you were working toward. We’ll consider that when we look at outcomes a little later in the section.

In all these cases, evaluation should involve feedback from both participants who had good results and those who didn’t. What worked particularly well for those who had success? What were barriers to those for whom the program didn’t work well? It’s not always easy to get participants to describe the positives and negatives – but it’s the best way to find out.

If your program actually had a negative impact on the targeted behaviors or risk and/or protective factors – the intervention aimed to increase childhood immunizations, and fewer children were immunized, for example – it is important to look more deeply into what is happening.

Some possibilities:

• You failed to communicate your message, or its importance
• You underestimated or ignored cultural influences that were powerful enough that your methods failed to overcome them
• You didn’t take into account cultural influences in participants’ lives that made it difficult to achieve intended results - these factors could include poverty, or competing demands for time, among many others
• The cultural incompetence of the organization or some staff members worked against your goals
• The structure and/or methods of the program led to unanticipated negative consequences
• The program was seen by participants as something that was being imposed on them, and they had little influence on its design or implementation

Just as you might find that your process went well and your program still didn’t influence the risk and protective factors you meant to, it’s possible that you created exactly the changes you intended in risk and protective factors, and the program still didn’t achieve the outcomes intended. We’ll look at outcomes to consider that situation.

Outcomes

The outcomes of an intervention are the changes that actually took place as a result of it. The goal of an intervention is usually not just a change in behavior or circumstances, but the changes in community health and development that occur as a result of that immediate change. A tobacco control program, for instance, aims to help participants avoid or quit smoking: that’s its impact. Its real goals – the hoped-for outcomes of the program – are reduced rates of heart disease, lung cancer, and other smoking-related diseases for participants and their family members.

The ultimate outcomes may take years to assess, but others – like the blood pressure goals of the Health Center exercise program, or the results of a job training course – can be determined at or soon after the end of the intervention. Outcomes are the true measure of the success of the intervention, because they are the reason it was conducted in the first place. However, the impact made – such as changes in community programs of policies– can be an important intermediate outcome since it can take years to see changes in longer-term outcomes.

The program produced the intended outcomes

If the program produced the outcomes you intended, congratulations: you’ve achieved the goals of your effort. This isn’t the time to consider your work complete, however. How can you make the intervention even better and more effective?

• Can you expand or strengthen parts of the program that worked particularly well?
• Are there evidence-based methods or best practices out there that could make your work even more effective?
• Would targeting more or different behaviors or risk and protective factors lead to greater success?
• How can you reach people who dropped out early or who didn’t really benefit from your work?
• How can you improve your outreach? Are there marginalized or other groups you’re not reaching?
• Can you add services – either directly aimed at program outcomes or related services such as transportation – that would improve results for participants?
• Can you improve the efficiency of your process, saving time and/or money without compromising your effectiveness or sacrificing important elements of your program?

Good interventions are dynamic: they keep changing and experimenting, always reaching for something better. Programs can always be improved.

The program only produced some of the intended outcomes

If the intervention produced only some, or some lower level, of the desired outcomes, you may be headed in the right direction. The program may also have greater effects in the long run, as participants incorporate the changes they’ve made into their everyday lives.

Some possible reasons for the program’s effect not being as great as planned:

• You didn’t target sufficient risk and/or protective factors
• The program’s message didn’t reach participants or speak to them in a powerful way
• There were intervening factors – attendance or lack of support services – that made the program less effective than it could’ve been
• Particular parts of the program didn’t work well
• Particular parts of the program weren’t implemented well
• You overestimated what was possible in the time available
• The program didn’t approach participants in the right way – it was too formal, the language used posed barriers for some, etc.
• The program wasn’t culturally adapted for the population
• There were conflicts among participants or between participants and staff

For example, let's say that the Health Center’s exercise program wasn’t by any means a failure, but it was only modestly successful. How migth the Health Center use its evaluation information to improve the results for program participants?

First, the Center could examine what participants said about the program. What enabled the members of the most successful group to exercise? Why weren’t members of the much larger group able to establish regular effective exercise routines? And for members of the third group – those who didn’t exercise at all or dropped out quickly – what might have gotten them more motivated?

Perhaps those in the first group attended all the sessions and found exercise partners who challenged one another to do a little more (or to eat a little better.) Perhaps those in the other groups did not locate partners.

Based on the evaluation, the program’s designers decided that they should arrange for exercise partners or groups for everyone. It seemed from the evaluation that both the social situation and the challenge that exercising with others presented made exercise more likely and more fun, and promoted a more vigorous workout. They also decided to develop a much more formal nutrition component to the program, and to incorporate a buddy system into that component as well, in the hopes that participants could help one another develop recipes and stick to a reasonable eating plan.

The program produced no outcomes

If the program produced no outcomes at all, you may have to make big changes.

It can be very difficult to admit that you’ve been taking the wrong direction, especially after investing a lot of time and effort in planning and implementing a program. It’s tempting to believe that if you just work harder, or recruit different participants, or use better materials, you’ll get the results you want. It takes courage conclude that the results call for a major re-design in the effort.

The program may have produced unintended outcomes, either positive or negative. If they’re positive, you might want to understand how they came about so that you can continue to produce them. If they’re negative, you’ll probably want to learn more so you can seek to eliminate them. Most of the reasons for unintended outcomes are similar to those for lack of outcomes.

A positive unintended outcome in a youth violence prevention program, for example, might be better school performance; a negative example in the same program might be an increase in school dropout. Teens in the program might improve their school performance because they admire a staff member with college education, and want either to be like him or to impress him. Or they may see college and an escape from the neighborhood as their best way out of the cycle of violence.

Those who drop out of school as a result of the program may also do so because they see it as a way to avoid violence: school – or the trip to and from school – may be especially dangerous because of the presence of youth from other neighborhoods or rival gangs. Conversely, they may see dropping out of school in favor of work as a non-violent road to financial success, as opposed to dealing drugs or other similar violence-prone activities.

Given all this, how do you approach your evaluation research to decide what you need to refine and how? A good general approach is to work backward from outcomes – asking “but why?” – regarding why each previous phase failed to produce the results you wanted.

Using the "But why?" Method to examine outcomes

• Examine the outcomes. If your intervention achieved the intended outcomes, it has done its job. Now you can consider how to maintain these effects or refine your program (see above). You should still examine the results for process and impact, and make changes where they’ll gain you greater effectiveness or efficiency. But chances are the program doesn’t need major changes, unless you want to enlarge your goals, or unless you’ve found an alternative approach that could lead to even more impressive outcomes.
• Examine the impact. If your evaluation research shows no outcomes, or outcomes that fall short of what you intended, the next area to examine is the impact of your program on the targeted behaviors and risk/protective factors.

If the program had the impact you expected, but no outcomes, perhaps you’ve chosen the wrong behaviors or factors to target, and need to rethink your problem analysis and related intervention. There are other plausible explanations: your intervention wasn’t in place long enough, the effects are delayed, your measures are insensitive to what is being achieved, etc.

• Examine the process. The next step here is to understand how well you planned, prepared for, and implemented your intervention. If the reasoning and assumptions behind your planning were accurate, and if you set up and implemented your program based on them, you should have the impact you were aiming for, and that impact should lead to the outcomes you intended. If your program didn’t go as planned, that could be a good part, if not all, of the reason for your lack of outcomes. Your process evaluation can show you where you need to adjust and improve your implementation to have a better chance to get the intended results.

If your program did go as planned – you met your deadlines and did what you intended to do in the way you intended to do it – and you failed to achieve your goals, there’s a good chance that your planning was the problem. You may have aimed at insufficient risk and/or protective factors, as mentioned above, or you may have chosen ineffective methods to influence the right ones...or both. There are other possibilities that could be picked up by a process evaluation as well, many of which have already been suggested – treatment of participants, language or other communication issues, lack of cultural competence, etc. Identifying and correcting such problems can help a program reach success.

• Keep making adjustments. Make your adjustments and refinements, run and evaluate the intervention, and make further adjustments and refinements to improve your work. This should be a continual cycle for the life of your program.

In Summary

The purpose of an evaluation and the research that goes into it is not just to tell you whether or not your intervention has been a success. The real value of evaluation research lies in its ability to help you identify and correct problems – as well as to celebrate progress. Evaluation can pinpoint the strengths of your program, and help you to protect and enhance those strengths and make them even stronger.

By examining the three elements of an intervention – process, impact, and outcomes – your evaluation can tell you whether you did what you had planned; whether what you did had the influence you expected on the behaviors and factors you intended to influence; and whether the changes in those factors led to the intended outcomes. That knowledge can show you what you might change to improve your program, as well as the overall effectiveness of the intervention. And, the information can be used to celebrate the accomplishments you are making along the way.

What do we mean by refining the intervention?

___Refining the intervention is the process of making your work more effective by using evaluation information

___Key reflection questions include:

• What are we seeing?
• What does it mean?
• What are the implications for improvement?

Why should you use your evaluation research to refine the intervention?

___Understanding and improving your work is one of the main purposes of evaluation

When should you refine the intervention?

___Throughout the life of an intervention or program

___When it’s clear that what you’re doing isn’t working

___When participants are dropping out at a high rate

___Between sessions of a time-limited or sequential program

___When funders or participants ask you to adjust some aspect(s) of your program

___When funding or other resources are reduced

___When the goal or issue changes character

Who should be involved in refining the intervention?

___Participants

___Staff members, paid or volunteer

___People who are directly or indirectly involved in implementing or supporting your work

___Those who led and participated in the evaluation

How do you refine an intervention based on research?

___Determine whether your process – the planning, set-up and implementation of the program – went as planned

___Determine whether your program had the desired impact on the behaviors and risk and protective factors you targeted

___If there was no impact, or only a partial impact (or a negative impact), that could affect the program’s outcomes

___Determine whether your program achieved its desired outcomes

___If the program achieved its desired outcomes, consider how you can expand and strengthen the areas that worked well, and make the program more efficient and effective

___If the program failed to achieve the desired outcomes, work backwards – “but why” – to find where the problems lie

___If the program failed to have the intended impact on the targeted behaviors and risk and protective factors, that could explain the fact that you didn’t meet your outcome goals

___If the program had the intended impact on the targeted behaviors and factors, but failed to achieve the desired outcomes, you may have targeted the wrong behaviors or factors

___If some or all of the process was different from what was planned, that might affect the program’s impact on the targeted factors

___If the planning process or reasoning was faulty, it could result in targeting the wrong factors, or in using ineffective methods that target the right factors

___Once you’ve determined where and how to make adjustments and done it, repeat the evaluation and refinement cycle for the life of the intervention

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• Why inform people about your evaluation findings?

• What are some key audiences for the data?

• How do you communicate evaluation findings?

Why inform people about your evaluation findings?

There are many reasons you should inform people about your evaluation findings. For one thing, you want the name and purpose of your coalition or initiative to be recognizable to your community, and you want them to have a positive impression of you. Releasing your evaluation findings not only lets people know you exist, but it also lets them know some of what you've been doing to help your community. If the findings are positive, it shows that you've been getting results. If they are less than encouraging, you can use this information to make the case that your group or coalition needs more public support. Sharing your evaluation findings should also stir public interest, and provoke thinking and discussion about the issues you're working on. The main reasons for making your evaluation findings known publicly are to expose the issue and encourage the public to take action.

There are three main levels of the public that you can tell about your evaluation findings: local, regional/state, and national. Some reasons for sharing your evaluation results with each of these levels:

Reasons for informing the public at the local level

• To help raise awareness about the issue
• To help attract volunteers, funding, and in-kind resources from local concerned citizens and agencies
• To promote awareness of the efforts of volunteers and collaborators
• To help lobby for local ordinances or program changes to address issues of concern
• To provide accountability to the community, trustees, and funders

Reasons for informing the public at the state level

• To create a "name" for your initiative in the state, which makes it more competitive when seeking state resources
• To help establish a statewide network of persons and agencies with similar goals
• To help lobby for legislative changes to address the issues of concern
• To help the initiative garner recognition and resources from the state and region

Reasons for informing the public at the national level

• To create a "name" for the initiative nationwide, which makes you more competitive when seeking resources from the state or federal government or from large private foundations
• To help tap into nationwide networks of persons and agencies with similar goals and wide expertise
• To help the initiative garner recognition and resources from across the country
• To encourage community partnerships to work on the problem or issue

Three tips for making sure your findings aren't ignored

• Give your information to the right people!
• Address issues which those people think are important
• Be sure the information is presented in time to be useful and in a way that's clearly understood

What are some key audiences for the data?

Who do you share this data with? Staff, volunteers, supporters in the community, and funders are all supporting groups that should be up-to-date on your group's efforts and successes. You should also share your evaluation findings with your target population--the people you are trying to help. If you work with a heart disease prevention program and your evaluation shows an increased risk factor for members of a particular ethnic group, for example, you will want to let members of that group know in order to spark their interest and make them more receptive to whatever work you will be doing in their community. Your target group needs to know what and how they will need to change to improve their health. Finally, the general public should be kept informed.

Key groups to share your evaluation findings with

Local

• Civic organizations
• Business groups
• Grassroots organizations
• School boards
• Parent-teacher groups
• Church organizations
• The local press
• Health organizations
• Elected and appointed local government officials
• Grantmakers

State/regional

• State and regional professional conferences,
• Regional professional training workshops
• Grassroots and advocacy organizations
• Church conferences
• Grantmakers

National

• Professional conferences
• Professional training workshops
• Grassroots and advocacy organizations
• Church conferences
• Grantmakers

What about difficult audiences?

• Anticipate their questions, concerns, and objections. Think ahead of time about what this particular audience won't like about what you have to say, and come up with calm, measured, logical, and thorough responses for them. If possible, sit down with someone who knows your audience well to get their feedback on what questions or concerns they may raise and how they might react to your answers. During the presentation, this will help you keep from getting flustered and defensive.
• Have a primary figure in your initiative present the findings. Having a director in your program deliver the data can lend a greater amount of authority to the presentation.
• Have someone else give out the information. With a particularly hostile or uninformed audience, or an audience that you don't normally have access to, you may wish to let someone outside your group or initiative relay the evaluation findings - a member of that group, for example.
• Reinforce the data repeatedly
• Keep your cool

Presenting evaluation results to the press

• Be honest with reporters. It's essential for any spokesperson for your coalition or group must have credibility with the press. Answer any questions simply and candidly, and if you can't, explain why.
• Write your own press releases. Give reporters a clear, brief press release with all of the important information, as well as contact details, in advance in order to clear up misconceptions and help the press better prepare for your presentation.
• Train your reporters. If your evaluation results are going to include a lot of complicated statistical information, consider doing an in -house training session for the press. These workshops should be unbiased, and they should be broad enough that the reporters can use the information learned for working with other groups, not just yours. Consider doing an informational workshop or some one-on-one presentations to explain such concepts as:
  • What do percentile scores mean?
  • What are some common methods of gathering evaluation data?
  • What are community-level indicators?

How do you communicate evaluation findings?

Develop a general presentation format that can be lengthened or shortened depending on the amount of time available, including compelling descriptions and visuals of:

• The issue(s) of concern
• The initiative's goals, strategies, and methods for reaching those goals
• Data on activities (e.g., services provided)
• Data on accomplishments (e.g., community changes)
• Data on outcomes (i.e. behavioral measures and community-level indicators)

Keep your visuals simple to cut down on problems interpreting data. The first and last visuals should contain your message or your primary findings--whatever it is that you most want your audience to remember afterwards.

Identify different avenues of getting the word out about your evaluation results, such as:

• Word of mouth
• Presentations
• Newspapers and newsletters
• Radio - both public service announcements and local news or call-in shows
• Television coverage
• Professional journals

Why might you use different formats for presenting your evaluation results?

There are many different types of reports that are suitable for different types of audiences. Some groups might not be interested in or know how to interpret the statistical details, but would still like to know the general findings of your report. Funding agents, grantmakers, advisory boards, and program staff will most likely want detailed, explicit information. You should always take into consideration the type of group you're presenting the information to and tailor your presentation to that audience.

Here are some ideas of what types of reports work for what audiences:

• Technical reports: This is a detailed report on a single issue, such as a small study on one or two sample groups. It can be given at a staff meeting or as part of a larger report.
  • Best for: Funding agencies, program administrators, advisory committees
• Executive summary: A few pages, usually at the beginning or end of a longer report, which outlines a study's major findings and recommendations.
  • Best for: Funding agencies, program administrators, board members and trustees, program staff, advisory committees, political bodies, program service providers (technicians, teachers, etc.)
• Technical professional paper: A detailed article that summarizes information for a scientific or technical audience. It usually contains information about what is done, how to do it, what worked and what did not work, and why.
  • Best for: Program administrators, advisory committees, organizations interested in program content
• Popular article: An article written with the target audience of the medium in mind. Some magazines and papers target specific populations. It normally contains more information than a press release, but focuses on two or three quick points.
  • Best for: Program administrators, board members and trustees, program staff, political bodies, community groups, current clients, potential clients, program service providers, organizations interested in program content
• News release and/or press conference: A gathering with the media done for the purpose of releasing specific information.
  • Best for: Program administrators, the media, wide distribution of simplified information
• Public meeting: A gathering that's open to the general public where more general evaluation findings are released in a clear, simple manner, usually with some time set aside for open discussion.
  • Best for: Community groups, current clients, the media
• Media appearance: Different from a press release in that this incorporates some sort of staged event--for example, a local author doing a public reading to highlight awareness about a study on adult literacy.
  • Best for: Current clients, the media
• Staff workshop: A more interactive, working presentation for your group or coalition's staff and volunteers.
  • Best for: Program administrators, program staff, program service providers
• Brochures/posters: Brief, simply-worded printed materials that can be distributed and mailed to various outlets in the community. Needs to focus on one quick point.
  • Best for: Potential clients
• Memo: A short letter circulated internally among program staff.
  • Best for: Program administrators, program staff, program service providers
• Personal discussion: Sitting down face-to-face to discuss evaluation findings with an individual or small group.
  • Best for: Funding agencies, program administrators, program staff, program service providers

Possible goals of your presentation

What do you want from the group you're presenting to? How you can best present the evaluation findings to get those results?

A few things you might be hoping to get back from your audience:

• Money and in-kind resources for your initiative
• Volunteers for project activities
• Influence in changing a program, policy, or practice
• Input on how to make the initiative more responsive
• Overcoming resistance to the initiative
• Ideas on how the initiative can become more effective

Steps in developing your presentation

• Understand your primary users and audiences. What information do they need and why do they need it? Try to understand the audience's viewpoint, and be sure to get the report to them in time for it to be useful to them.
• Review the results of your evaluation with program staff before you write up your evaluation report. This gives you a chance to get your staff's input on the meaning of the findings, as well as the opportunity to talk about any ambiguous data that has come up.
• Brief any important political figures before you release your report to the public. This is especially important if your evaluation findings make it clear that there will be a need for any changes in policies. Policymakers or agency officials may wish to make a public response to your findings as well. For example, if your evaluation shows that your recreation program for families with developmentally disabled children is highly effective but underfunded, area human service agencies may wish to let people know that they plan to increase funding for the project in the next budget year.
• Your final report can just be a short document summarizing the evaluation findings with a technical appendix for those who are interested.
• If you decide to do an oral presentation, you should make up a small number of charts and tables--six to ten should be plenty--illustrating the most important findings. Make up one version--printed on posterboard or done as an overhead projection-- to show during the presentation itself as well as copies that your audience can keep afterwards. You may also wish to prepare a single sheet summarizing the overall results for the audience to take home.
• Your report should begin with the reasons the evaluation was done, what questions were asked, and why those were the questions chosen. Explain what your group or coalition wanted to learn from the evaluation and what methods were used to conduct the evaluation.
• Depending on your audience, you may want to simply highlight the results, or you may want to go into more detail about what you found. Be sure to explain what sort of implications the results have for your group or initiative. If the evaluation findings have led you to any particular conclusions about what your group should do in the future, talk about them.

Why should you try to let others know about your initiative?

___To let the public to know you exist

___To let the public know some of what you've been doing to help your community

___To stir public interest

___To expose the issue and encourage the public to take action

Why should you inform the public at the local level?

___To help raise awareness about the issue

___To help attract volunteers, funding, and in-kind resources from local concerned citizens and agencies

___To promote awareness of the efforts of volunteers and collaborators

___To help lobby for local ordinances or program changes to address issues of concern

___To provide accountability to the community, trustees, and funders

Why should you inform the public at the state level?

___To create a "name" for your initiative in the state, which makes it more competitive when seeking state resources

___To help establish a statewide network of persons and agencies with similar goals

___To help lobby for legislative changes to address the issues of concern

___To help the initiative garner recognition and resources from the state and region

Why should you inform the public at the national level?

___To create a "name" for the initiative nationwide, which makes you more competitive when seeking resources from the state or federal government or from large private foundations

___To help tap into nationwide networks of persons and agencies with similar goals and wide expertise

___To help the initiative garner recognition and resources from across the country

___To encourage community partnerships to work on the problem or issue

How can you ensure that your findings aren't ignored?

___Give your information to the right people!

___Address issues which those people think are important

___Be sure the information has been presented in time to be useful and in a way that's clearly understood

What are some key audiences for sharing your data?

___Staff

___Volunteers

___Supporters in the community

___Funders

___Your target population

___The general public

What are some key local audiences for sharing your data?

___Civic organizations

___Business groups

___Grassroots organizations

___School boards

___Parent-teacher groups

___Church organizations

___The local press

___Health organizations

___Elected and appointed local government officials

___Grantmakers

What are some key state/regional audiences for sharing your data?

___State and regional professional conferences,

___Regional professional training workshops

___Grassroots and advocacy organizations

___Church conferences

___Grantmakers

What are some key national audiences for sharing your data?

___Professional conferences

___Professional training workshops

___Grassroots and advocacy organizations

___Church conferences

___Grantmakers

How should you deal with difficult audiences?

___Anticipate their questions, concerns, and objections

___If appropriate, have a primary figure in your initiative present the findings

___Have someone else give out the information

___Reinforce the data repeatedly

___Keep your cool

How should you present your evaluation findings to the press?

___Be honest with reporters

___Write your own press releases

___Train your reporters

What should you include in your presentation?

___The issue(s) of concern

___The initiative's goals, strategies, and methods for reaching those goals

___Data on activities (e.g., services provided)

___Data on accomplishments (e.g., community changes)

___Data on outcomes (i.e. behavioral measures and community-level indicators)

___Keep your visuals simple to cut down on problems interpreting data

What are some different avenues of getting the word out about your evaluation results?

___Word of mouth

___Presentations

___Newspapers and newsletters

___Radio - both public service announcements and local news or call-in shows

___Television coverage

___Professional journals

What are the different formats for presenting your evaluation results?

___Technical reports

___Executive summary

___Technical professional paper

___Popular article

___News release and/or press conference

___Public meeting

___Media appearance

___Staff workshop

___Brochures/posters

___Memo

___Personal discussion

What are possible goals of your presentation?

___Money and in-kind resources for your initiative

___Volunteers for project activities

___Influence in changing a program, policy, or practice

___Input on how to make the initiative more responsive

___Overcoming resistance to the initiative

___Ideas on how the initiative can become more effective

What are the steps for developing your presentation?

___Understand your primary users and audiences

___Review the results of your evaluation with program staff before writing up your evaluation report

___Take time to brief any important political figures before releasing your report to the public

___Know that your final report can just be a short document summarizing the evaluation findings with a technical appendix for those who are interested

___If you decide to do an oral presentation, make a small number of charts and tables illustrating the most important findings

___Begin your report with the reasons the evaluation was done, what questions were asked, and why those were the questions chosen

___Explain what your group or coalition wanted to learn from the evaluation and what methods were used to conduct the evaluation

___Explain what sort of implications the results have for your group or initiative

Tool: Evaluation Report Outline

This tool gives you an overview of the different parts of an evaluation report and what each part should include.

Front Cover

The front cover should include:

• Program title and location
• Name(s) of evaluator(s)
• Period covered by the report
• Date of the report

Lay out your front cover neatly and make it look nice - the front cover is the first thing your audience sees and it makes an important impression.

Section I - Summary (or Executive Summary)

This is a brief (two to three pages) overview of the evaluation outlining major findings and recommendations. Some folks are too busy to read any further than the summary, so make sure that this is as complete and clear as possible.

The summary should include:

• What was evaluated?
• Why was the evaluation done?
• What are the major findings and recommendations?

And, if space permits:

• What audience is the report aimed at?
• What decisions, if any, need to be made or have been made based on the results of the evaluation?
• Who else might find the report to be of interest or importance?

Section II - Background Information About the Program

Presumably, most of the people reading your evaluation report will at least be somewhat familiar with the program, but that's not necessarily the case. And even people who are familiar with the program may have some misconceptions, so take the time to make your goals, strategic plan, organizational structure, and other essential program elements clear.

Typically, this section will include:

• Origins of the program
• Program goals
• Clients involved with the program
• Administrative/organizational structure
• Program activities and services
• Materials used and produced by the program
• Program staff

Section III - Description of the Evaluation

This part explains why an evaluation was done and what you hoped to learn from it. It should also explain anything the evaluation was not intended to do.

Here are some of the questions that should be answered by this section:

• Who requested the evaluation?
• Was the evaluation meant to satisfy any particular audience and, if so, which one(s)?
• Were there any restrictions to the evaluation in terms of money, time, or other resources?
• Was any particular kind of evaluation design used and, if so, why?
• What was the timetable for collecting data?
• For each measure, what sort of data was collected?
• What sort of methods were used to gather data, and why were these particular methods chosen?
• How did the evaluators ensure accuracy?

Section IV - Results of the Evaluation

This part will explain what your findings were in detail.

This section may include:

• All data collected - analyzed, recorded, and organized in understandable forms (charts, tables, graphs, etc.)
• Excerpts from interviews
• Testimonials from participants and clients
• Questionnaire results
• Test scores
• Anecdotal evidence

Section V - Discussion of Results

Here is your chance to go into more detail --the why of your evaluation results.

This part should answer the following questions:

• How sure are you that your program or initiative caused these results?
• Were there any other factors that could have contributed to the results?
• How are the results different from what they would have been if your program didn't exist?
• What do the evaluators feel are the strengths and weaknesses of your program?

Section VI - Costs and Benefits

This part of the report is optional - if you choose to include it, it gives you a chance to justify your program's budget and financial choices.

If you include it, here's what we suggest you include:

• Costs associated with the initiative (not only financial costs, but costs in terms of resources, energy, results, and staff/volunteer hours)
• Methods used to come up with the budget
• Benefits from the program (both financial and non-financial)

Section VII - Conclusions

After writing all this stuff up, it may be tempting to dash off a quickie conclusion to this report, but avoid that temptation!

This is a very important piece of the big pie, because this is where you make your recommendations:

• What major conclusions about the initiative can be reached as a result of this evaluation?
• Is there anything you feel should not be judged at this time, and if so, why?
• Based on the evaluation results, what recommendations can you make for the program?
• If the evaluation gives you any idea of what the future holds for the initiative, what would that be?
• What worked well about the evaluation? What didn't work so well?
• What recommendations do you have for anyone doing future evaluations with the program?