Section 1. Using an Evaluation System to Answer Key Questions About Your Initiative

Learn how to determine whether your methods are successful, and whether your intervention or initiative accomplished what it set out to do.

 

If you’re reading this section, the chances are you already know how important evaluation is to your effort. It can help you in a number of ways – in improving your methods, in appealing to funders, in supporting staff – but none is more important than understanding and improving the effort.

Often, evaluations fail to examine whether efforts are responsible for, or related to, changes and improvements in the community. They may examine the process or output (e.g., number of people trained), but they often fail to tell a story about whether conditions changed and whether this led to more distant outcomes.

It’s obviously important to know whether your methods are successful, and whether your intervention or initiative accomplished what it set out to do. In addition, however, you can evaluate whether your work changed community conditions, and whether it seems to be leading toward changes in the community or system. In this section, we’ll discuss how an evaluation can help you make those determinations.

What do we mean by answering evaluation questions?

Evaluation questions refer to what stakeholders – the community and funders, for instance – want to know about the functioning of the program or initiative. Here are some illustrative evaluation questions:

• Are participants satisfied with the program? (Process and implementation issue)
• How well is the initiative meeting its stated objectives? (Attainment of objectives)
• How much and what kind of difference did it make for the participants? (Impact on participants)
• How much and what kind of difference did the initiative make on the community? (Impact on the community)

So what’s a key evaluation question? That depends on what you set out to do (mission and objectives) and how you plan to get there (the logic of how the process and activities will produce immediate and longer-term outcomes). The path from here to there – from assessment to change in the environment to longer-term outcomes – is known as a logic model. Key evaluation questions focus on critical aspects of the logic model; for instance, “Is the initiative bringing about change in the environment?” or “Is that change in the community (the intervention) associated with improvement in outcomes for people in the community?

A note here: The Center for Community Health and Development at the University of Kansas, the source of the Community Tool Box, consults on evaluation, and has developed a Community Check Box system to support participatory research and evaluation of community health and development efforts. Much of the material in this section comes from the thinking that underlies this monitoring and evaluation system.

The documentation system makes it possible to answer key evaluation questions that get at the heart of an effort’s overall effectiveness. We’ll use this monitoring and evaluation system to illustrate how to address evaluation questions.

Regardless of your aims, your work can be considered on two levels. The first is whether, how, and why you’ve been able to accomplish the specific goals you’ve set for your effort – successful advocacy for the passage of a new regulation, the improved delivery of services, a change in people’s behavior, improvement in environmental conditions, etc. The second is the wider influence that your work has in the community or system, and how conditions have changed.

A decent comprehensive evaluation might examine how you planned and implemented your effort (process), whether you actually took the specific actions you set out to take, such as serving the number of people you expected in the ways you planned to (impact), and whether your actions had the results you hoped for (outcomes.)

A typical evaluation may measure ultimate outcomes but may fail to document changes in the community resulting from your work such as other similar programs starting, more funding becoming available for your issue, or more media attention. Furthermore, a specific effort may give rise to more general and sweeping changes even when it doesn’t initially attempt to. A campaign to reduce youth violence may achieve its immediate purpose, for instance, but may also change community programs and resources available to address the needs of adolescents and result in significant changes in the ways that community members engage youth. Sometimes, even an apparently unsuccessful effort can have a broader community effect by increasing awareness and understanding of a community problem or strength, or by changing the behavior of key community members or leaders. These broader effects can be harder to track than the more immediate achievement of the specific goals and objectives of an effort.

Some definitions before we start the next phase of discussion:

We usually think of a community as a group of people who live in one place – a city, a neighborhood, a town, a village, a rural area. We also use the term, however, to refer to a group of people with one or more common characteristic and common interest. Using that definition, we speak of the business community, the Latino community, or the school community. In this section, the word is generally used to mean community of place, because that’s the community in which organizations generally try to create longer-term change.

System refers to the arrangements or infrastructure that supports key functions – for instance, the public health system, the legislative system, the education system, or the legal system. The essential features of a system – preparation of the workforce, capacity to monitor and diagnose problems – also affect the conditions under which people live their lives.

At both the specific and general levels, it’s important to know whether any changes in the community or system you’re concerned with are actually taking place, what they are, and whether and how your work has influenced them (and vice versa).

Figure 1 below displays a framework (or logic model) for collaborative action, as well as evaluation questions that reflect potential associations along the logic model’s pathway.

For the logic of collaborative action, Fawcett and Schultz (2008) recommend four key evaluation questions:

1. Is your effort serving as a catalyst for community change related to its mission? In chemistry, a catalyst is a substance that, while remaining unchanged itself, changes the speed, intensity, or nature of a reaction. Is your effort, while doing its intended job, setting in motion other forces in the community (new or modified programs, policies, and practices) that lead to longer-term change related to the issue you’re concerned with?
2. What factors or processes are associated with the rate of community or system change? What encourages or gets in the way of change? Can you identify specific community events, actions by individuals or by your or another organization, or other factors that accelerate or impede the process of change in a community or system?
3. How are community or system changes contributing to efforts to promote community health and development or other mission-related goals? Are the changes you observe in the community in fact helping to bring about the realization of your goals – or related ones – or are they having little or no effect (or even a negative effect)?
4. Are community or system changes associated with improvements in population-level outcomes? Can you see connections between changes in the community (programs or policies for instance) and the kinds of positive changes in measuring success (e.g., fewer youth violence arrests, reductions in cigarette sales, fewer teens killed or injured in driving accidents) that you hope your effort will bring about?

Figure 1. Framework for collaborative community action and related evaluation questions (Source: Fawcett and Schultz 2008; Institute of Medicine, 2003).

Note (in Figure 1) that we can overlay evaluation questions on this logic model to address key (presumed) relationships along the path (see Fawcett and Schultz, 2008, in print resources).

Source: Adapted from the Institute of Medicine’s framework for collaborative public health action in communities (2003).

Answering these questions sets your effort in the context of the community and shows you whether it’s actually having the effects you want in a broader sphere, or whether it’s only influencing individual successful participants. Questions like these are not always easy to answer. Later in the section, we’ll discuss them more fully, as well as how an evaluation can be structured to answer them.

Why answer these questions?

In many cases, organizations are funded for several years at a time at best. This requires focus on fulfilling the conditions of the grant or contract, which often hinge on relatively short-term goals. There are, however, a number of compelling reasons for answering evaluation questions:

• To improve your work. The foremost reason for any evaluation is to make your effort more effective. By understanding exactly what affects progress on your issue, and the interaction between your actions and community system change (an intermediate outcome), you can see how to adjust your work to get the longer-term results you seek.
• To understand what affects the work in what ways. What actions or events seem to slow down the pace of change (e.g., loss of leadership), and what speeds it up (e.g., action planning)? What effects do small and large changes in the organization and the community have on your work?
• To see how to accelerate results. Once you understand what makes change move more quickly, you can take steps to make sure that those change-accelerating processes, events, or conditions become an integral part of your work.
• To understand specifically how broader actions, events, or conditions – a crisis or concentrated poverty – affect the work. Identifying and understanding these factors that affect the success or failure of an effort is critical – for the work underway and for future efforts.
• To understand what works to bring about community change, and adjust accordingly. The chances are you’re doing your work in order to permanently change some condition in the community or to improve the quality of life for a particular group or for the community as a whole. You hope that, eventually, the goals you’re working toward – the elimination of domestic violence, reductions in pollution, a decrease the incidence of diabetes, better schools for all children – become the community norms.

  Using an evaluation to answer key questions will provide information both about which small changes seem to lead to larger ones and about how to generate those and larger changes as well. That, again, will allow you to adjust your effort in order to make it more effective.

• To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effect. The loss of a competent leader, an increase or decrease in funding, the end of one phase of an effort and the beginning of another – all of these and many other factors can affect the rate and direction of change in the community. Knowing the results of events can help you determine how to best manage them.
• To show the community the value of your work. If you can show that your work has a positive influence on the community, you have a better chance of gaining broad support. That support, in turn, can lead to funding and in-kind help, volunteers, and community participation in ongoing social change.

Who should be involved in answering these questions?

The ideal is to engage in a participatory evaluation that draws in members of all stakeholder groups. This not only leads to buy-in and support on the parts of many sectors, but also provides multiple perspectives on such issues as how participants are treated, how the effort is viewed in the community, how non-participants’ lives are affected by your work, whether there are changes in the social climate, etc. You may want to conduct the evaluation as a participatory action research project, or you may simply want to involve as many individuals and sectors as possible in providing information and being kept current about the findings of the evaluation. Among the possible stakeholders:

• Participants in or beneficiaries of the effort. They may have a great deal to offer, both in terms of identifying the real needs of the population and/or the community and in reaching other members of the population of concern.
• Residents of a focused geographic area, whether or not they’re members of the population of concern. If your focus is on a neighborhood, a small rural community, or other limited area, you may want to involve all residents. Such involvement is likely to bring with it support for your effort and may increase understanding about the issue and serve to improve relations among these residents and the group you’re working with.
• Professionals and volunteers carrying out the work. The more deeply involved these people are, the better equipped and the more eager they’ll be to do their jobs well.
• Those whose jobs or relationships bring them into contact and involvement with the population and/or issue with which you’re concerned. These might include health care providers, teachers, family members of substance users, etc. Since these people may have to help in carrying out the effort, and since they may be asked to change their behavior as a result, it’s both important to the success of the effort and a matter of fairness to include them
• Funders and local officials. Funders have a vested interest in information about the effort, and their participation can also serve to help them understand the real impact of your work. Including local officials in an evaluation can forestall problems and help to gain community support. It can also lead to changes in official policy over the longer term.

​When should you set up and use an evaluation system to answer key questions about the effort?

As we’ve discussed throughout the Community Tool Box chapters on evaluation (Chapters 36-39), it should be planned and embarked upon at the beginning of the effort. This is especially true for the kind of evaluation we’ll describe in this section, because its purpose is to monitor and evaluate what is changing – organizational and community changes, longer-term outcomes, and to examine the interactions among them. If you don’t start the evaluation at the very beginning of the effort with community assessment and planning, and instead wait until the implementation phase, you’re at risk of missing something important.

Once you’ve started the evaluation, you should be collecting information constantly, recording not only what happens but when it happens, so that you can make connections. For this type of evaluation to be truly useful, those connections have to be made frequently, not just once or twice a year. The point here is to see what is happening, reflect on its meaning, and use the information to strengthen and speed up positive change, or to address and prevent negative consequences. In either case, acting swiftly can be important. You can do that if you have the information in hand.

How do you use the evaluation system to answer key questions about the effort?

Once you’ve assembled a participatory team to gather and analyze data, you’re faced with some important questions: What kinds of information do you need in order to answer the key questions that relate to your logic model, and how do you get it? Equally important, once you have the information, how do you sift through it to extract the answers you’re looking for? There is a general method of examining the data that should be useful even as you analyze each of your questions individually.

An important element to using your evaluation well is setting up the data so you can best see patterns and relationships. A particularly good way to record the data is on a graph, as in the illustration below. This graph shows the total number of community changes – new programs (e.g., an after-school program for youth) or new policies (e.g., a policy change that improves ways to health services) – related to the group’s mission over a five-year period. Some changes might be relatively minor – a single business changes its practice for protecting coworkers – while others might be as important as a new law or a drastic change in policy that completely alters the way the community addresses the issue. [Note: with a cumulative graph, each change is added to the prior one; so a flat line shows no change; and the steeper the line, the higher the rate of change.]

Figure 2. Cumulative number of community changes (new or modified programs and policies) brought about by the South Side Youth Development. Initiative from 2008-2014. [Source: Example graph from the KU Center for Community Health and Development's Community Check Box]

Other information – specific inputs or events – could be pinpointed on the graph in order to give a clearer view of potential influences on the rate of changes, as in the graph below. Here, added to a graph showing information similar to that in the one above, are the points in time at which various events in the progress of the effort took place.

From left to right, these are:

• The forming of a planning team
• The hiring of a new director for the effort
• The implementation of a social marketing action plan
• The hiring of new staff
• Action planning for the initiative
• A loss of leadership (probably when the director left to take another position.)

These events, singly or in combination, might have contributed to the sustained high rate of change or to decreased rates.

Figure 3. Cumulative number of community/system changes – and associated events – brought about by the Lincoln Coalition for Healthy living from 1999-2006.

By comparing the progress of the effort, as indicated by the total number and rate of community changes to the timing of recorded inputs and events, you can see that most of the events seem to have some connection with a speed-up or slow-down in the rate and number of community changes. There was, for example, a dramatic increase in the rate of changes when new staff members were hired and an equally dramatic decrease in that rate when the effort lost its leadership. This is the kind of information seldom captured by a typical evaluation, and yet it’s extremely important in understanding how to adjust your work for greater effectiveness and how to correct problems.

Other elements that might be recorded on a graph – individually or together – include statistics for population-level outcomes, rates or levels of change for different populations or neighborhoods, or for different approaches, statistics for comparison communities not involved in the effort, etc. The particular measures depend on your logic model, related research/evaluation questions, and indicators of success.

To get the information you need, you’ll have to decide what’s important to measure. Then, once you have obtained the data – not always an easy task– the next step is to analyze it, looking for patterns and connections. Let’s look at each of the key questions considered for the logic model in Figure 1 and discuss what kinds of data will help you examine it. The overriding question is how to adjust your work to make it more effective. Answering key questions for your logic model is really a way to ask the larger question: “What can we do to make our effort work as well as possible to accomplish our goals and realize our vision?”

What follows is an illustration of how evaluation questions can be answered with data, asking those key questions identified is the logic model often used by the Center for Community Health and Development (see Figure 1).

1. Is the initiative serving as a catalyst for community/system change related to its mission? Has your group’s effort brought about changes in the community or systems (e.g., public health, public safety, welfare, environmental protection, etc.) related to the mission? What specific changes related to the issue have taken place in the community, and when?

Some types of changes related to your work that you might want to document include:

• New or modified programs. These might be programs meant to address aspects of your issue or related issues; workshops or public awareness programs on the topic; community-wide events such as marches against violence or vaccination fairs; citizen initiatives, etc. They could be the product of formal bodies – the Chamber of Commerce, health or human service organizations – or of informal parents’ groups, neighbors, “friends of...” groups, and the like.
• New or modified practices. These refer to changes in how people do their work or deliver services. For instance, the clinic might change its hours to assure better access or the community might start a new partnership to address discrimination against immigrants.
• New or modified policies. New laws or regulations, changes in agency or corporate policies, etc. Other policy changes may be more informal; for instance, increased city services or store openings in previously underserved neighborhood, or changes in the way police or city officials respond to intimate partner violence.

Of course, depending on the project, other valued events may be tracked – these might include organizational changes, media coverage, resources generated, measures of outcome (addressed in later questions) and other key events.

Keeping track of the timing of changes is very important here, because it may indicate when those changes are associated with activities you’re doing, and when they result from other factors, such as a change in leadership (see Figure 3). Marked increases or decreases in the rate or amount of change that occur just after key events in the effort may point toward the effectiveness or ineffectiveness of these activities.

Changes often take time to show their effects or have to build to a certain level before their effects can be felt. Where there’s a trend toward change in a particular direction, even if it doesn’t appear to have any great immediate effect, it might be wise to track it over time to see whether its different changes in the environment – taken together – eventually influence indicators of success.

Another consideration is the presence or absence of patterns in the rate and type of change. Are there patterns in community or system changes that correspond with the work of the initiative (e.g., action planning), or with other identifiable factors (e.g., funding to hire a community mobilizer)? Recognizing these patterns allows you to adjust your work.

It wouldn’t be surprising, for instance, to find a more common pattern of community change in community health initiatives; for instance, lower rates during planning, higher and sustained rates after action plans and staff are in place; and lower rates near the end of funding as the project focuses on sustainability. Community groups take time to understand how to bring about change and integrate it into the efforts. Rapid change might require a period of support and relative stability. Once that has taken place, the community may focus on sustaining more important programs and policies.

As change levels off, you might be tempted to step up activity to counter that trend. That might include additional action planning or bringing on staff responsible for change efforts.

Other questions to ask here are whether the changes in the community or system are actually likely to improve outcomes (Are they great enough to have an effect? Is that effect likely to change the situation for the better?), and how you can obtain more accurate or more nearly complete data about what is occurring.

2. What factors or processes are associated with the existence and/or the rate of community or system change? There are a number of potential factors that you may want to monitor and analyze.

• The processes by which you conduct the assessment, planning, implementation, and evaluation aspects of the effort. The particular focus should be on use of evidence-based processes for community change and improvement (e.g., developing and using action plans; arranging for community mobilization; documentation and feedback.) That also includes examining who’s involved (Does adding certain people or sectors have an effect, for instance?), whether everyone fulfills her responsibilities, whether deadlines are met, how people are involved, whether the effort plays out as planned, etc. It also includes examining how everyone involved – staff, administrators, volunteers, planning team, board – fits into and advances or impedes the process. (See “Promising Approaches” for promoting community health and development.)
• People served or benefited. Here you might examine whether the people affected by the effort were those you planned for; whether you served or benefited the number of people you expected to; the amount of service or benefit that each participant received (if that’s possible to determine); how many participants dropped out, and why (and whether there were patterns to be found there.) (See the “Solve a Problem” troubleshooting guide for “There is not enough community participation.”)
• Significant program events. These might include:
  • Changes in leadership.
  • Staffing changes.
  • Changes in method or direction, such as a shift in mission to address a newly-stated community need. This kind of event can also include unintentional changes that result from staff turnover, loss of focus, lack of staff engagement, or failure to do the work properly.
  • Milestones of the effort: the completion or revision of an action plan; the beginning, revamping, or completion of a program; the arrival of a new group of participants; the completion of staff or volunteer training; the accomplishment of a particular objective, etc.
  • Increases or cuts in funding.
  • Unforeseen circumstances, such as safety or security problems; financial changes (a cut or increase in funding, difficulty in managing money); measures taken to respond to financial changes (layoffs, shifts in hours or location of services, changes in the number or qualifications of participants).
  • Changes in broad conditions – This might include natural disasters, epidemics, etc.
  • Community events, such as:
    • A change of local political leadership or government administration.
    • Racial or ethnic conflict.
    • An economic windfall (a corporation sets up a new facility in town, creating new jobs and a huge boost to the tax base).
    • An economic downturn (the town’s biggest employer moves all its business to another country).
    • A community tragedy or celebration (recent deaths due to violence; honoring of community heroes).

It also might consist of changes in how state and local governments conduct business, the priorities of key funders, or other aspects of the broader system that might have positive or negative effects on your work.

As noted in Figure 3, combining graphs and comparing the timing of the process and events with patterns of community and system changes can reveal connections among them. These connections, in turn, can suggest processes or activities you might take that could advance positive change or prevent negative consequences.

3. How are community/system changes contributing to efforts to promote community health and development? What kinds of effects are community and system changes – whether caused by your effort or not – having on the issue you’re concerned with and on community indicators of health and development in general?

One way to find out is to examine those changes in relation to various dimensions or aspects of your work. How are community changes distributed among and within different aspects of your effort? Do changes cluster around one or more particular aspects; for instance, in one section of the community new program (e.g., mostly in schools). Are most of the changes you see in the community related to population (e.g., Latinos) for instance? Are most of the changes related to particular goals (e.g., preventing violence, increasing childhood immunizations)? Should you make adjustments in your work to alter the situation in such cases, and, if so, what should the adjustments be? You can ask these questions in each case to examine the relationship of documented change to particular dimensions of your work, including:

• Group your goals or aims. These are the longer-term goals and objectives of your effort. If your program has the mission of promoting heart health in the community, for example, young groups’ goals might include promoting physical activity, reducing tobacco use, increasing healthy nutrition, and assuring access to health screening.
• The strategies of intervention you’re using. These might take such forms as providing information, enhancing services or support, or encouraging policy change.
• Risk and protective factors. These are the personal, social, and environmental factors that make it more or less likely that individuals and groups might engage in various types of behavior or experience a specific problem or condition. Belonging to a close and loving family might be a protective factor against youth violence, for example; an alcoholic parent might be a risk factor for substance use.
• The expected duration of change. Is your effort a one-time event, such as a one-day health fair? Or is it ongoing, such as a problem change or extension of health services to reach a specific population? How long do you expect the change in program or policy to be in place?
• The populations benefited. Depending on the nature of your effort, the populations might range from a small number of individuals from a specific group (Latino women at risk for diabetes) to most of the residents of a diverse community (families with children in school).
• The sectors addressed. These might include government (in the case of advocacy for changes in laws, regulations, or official policy), health organizations (if you’re addressing health promotion or access to health care), education (school improvement, school policy change), labor (creating more trade apprenticeships for particular underrepresented groups), business (workplace and environmental concerns), etc.
• The ecological level addressed. This refers to whether your effort focuses on individuals (as in an adult literacy program), relationships (an anti-discrimination initiative), the community as a whole (enhancing overall community quality of life and health – working to establish community-wide bike lanes, for example, or promoting an ongoing series of free concerts), or the broader society (national health care policy).
• Place. This might be a particular neighborhood, city or town, county, or rural area.

How are the community or system changes you’ve tracked related to each of these aspects of your effort? What are you seeing in the distribution of community/system changes related to particular goals, strategies, populations, places, etc.? What meaning do you give to those patterns – e.g., they reflect opportunities, barriers, key partner’s interests, etc. What are the implications for adjustment (if any)?

Figure 4. Distribution of community accomplishments (N = 125) brought about by the initiative by primary sector of the community in which the change occurred.

This pie chart makes it easier to see what pattern of change is occurring, to reflect on what it means (e.g., we encountered opposition from some sections) and to make adjustments. (See the “Solve a Problem” troubleshooting guides for supports for making adjustments to common problems in the work.)

4. Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort? Community-level indicators of an issue, as we discussed earlier, are markers of success at the level of the community as a whole, rather than for particular individuals. Some of those indicators can be found through checking available statistical information, including, among many others:

• Census data (e.g., percentage of population with different levels of education).
• Public health statistics (e.g., rates of infant mortality, percentage of children immunizations).
• Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.
• Statewide data and data from other communities (for comparison purposes).
• Police and court files (e.g., number of motor vehicle accidents).
• Educational data – standardized test averages, truancy, dropouts, primary or secondary school completion rates, incidence of school violence.
• Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc.

In addition to archival data, other sources of information can be used; including from observation, surveys, and other typical data-collection methods. Are there differences in reports by youth from one data-gathering period to the next of the prevalence of alcohol and drug use, smoking, or unprotected sex? Do observations show that there are more people, from one period to the next, using a public park, or out on the street at night? Are teachers seeing fewer incidents of violence in the schoolyard or in school in general? Are doctors seeing more or fewer cases of diabetes or TB?

The population-level indicators that tell you what you want to know aren’t always obvious. If you’re concerned about teenage drinking, for instance, you will probably want to examine statistics about traffic violations and nighttime single-vehicle crashes even where alcohol isn’t specifically mentioned as a factor.

Once again, by graphing your information, you can see relationships between population-level outcomes and community changes (the unfolding of the intervention.)

Figure 5 below provides an illustration. It displays the relationship between community changes (the unfolding of the intervention) to prevent child sexual abuse and the associated improvement in reported cases of child sexual abuse in a state (the population-level indicator of success).

Figure 5. An observed association between the unfolding of community changes (the intervention) and a decrease in reported cases of child sexual abuse in Minnesota.

Here, there is definitely a trend toward fewer child sexual abuse cases – a 28% drop in four years – as the number of community changes (the intervention) unfolds. Of course, this is simply an association; without suitable comparisons it does not demonstrate a cause-and-effect relationship, but this observed pattern is suggestive of a promising approach.

It’s possible that the number of cases will continue to fall even if the number of changes doesn’t continue to grow. The community may have reached a tipping point, where the changes in the environment lead to a combination of increased vigilance, by bystanders, increased reporting and follow-up, certain arrest and prosecution for perpetrators, and child-molester registration, all resulting in a decline in cases. Whether the community has reached this point can only be determined by tracking the potential relationship between changes and reported cases over time.

5. Did the effort lead to improvements in population-level outcomes?

To determine whether your initiative brought about community changes that had a positive influence on community-level indicators, you’ll need a graph similar to that in Figure 5.

You may use multiple graphs, one for each community-level indicator you’re concerned with (e.g., number of drunk driving deaths, incidents of youth violence, number of teen pregnancies, percentage of children entering school with all vaccinations, etc.) Each of these will compare the statistics for that indicator over time with the documented unfolding of your intervention. Once again, where improvements in the indicator coincide with or follow closely that implementation it’s possible that your work has had the desired effect.

You’ll also have to take into consideration whether there are other plausible causes for improvements in outcomes. Sometimes a significant community event unrelated to your initiative – another program, a major policy change, widespread media coverage – can influence the outcomes. The action may be independent of anything your effort is doing, even if it occurs at the same time. Thus, major community events and broader incidents should be tracked and graphed along with changes and the progress of your effort, so you can see whether you’ve actually had the influence you hope for. In addition, experiential designs can be used – such as interrupted time series designs across different communities – to try to rule out other plausible explanations of observed effects on outcomes.

Begin early and keep at it indefinitely

If evaluations that address the key questions are to be useful, they must begin early and be sustained. Collecting, recording, and analyzing data has to go on all the time, starting before the planning and implementation of the effort and continuing for as long as the effort itself continues. A case can be made that evaluation research should continue even after the effort has ended, in order to judge its lasting effects in the community. Whether that’s possible or not, understanding the real results of your work and continually adjusting it to be more effective is an important part of any successful initiative.

To sum up

There are some key questions that can help you understand the relationship between what your initiative does (process and activities), charges in community and systems (intermediate outcomes), and (longer-term) outcomes.

We recommend several key evaluation questions:

• Is the initiative serving as a catalyst for community/system change related to its mission?
• What factors or processes are associated with the rate of community/system change?
• How are community/system changes contributing to efforts to promote community health and development?
• Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort?
• Did the effort lead to improvements in population-level outcomes?

Answering these questions involves careful thinking about what data will be useful, and then careful collection, recording, and analysis of those data. Graphing what you’ve recorded and comparing or combining graphs can often make analysis easier. The results should give you a great deal of information about exactly what works and what doesn’t, where you need to make adjustments, and what kinds of changes you should be trying to bring about in the community to address the issue.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: http://ctb.ku.edu/.