Chapter 38. Some Methods for Evaluating Comprehensive Community Initiatives | Community Tool Box

Whenever you begin a new job or start a project, you will probably want to evaluate your work. You might ask yourself: "Is my work meeting my expectations and those of my boss, my administrator, or our target audience?"

In a similar fashion, community initiatives need to be "evaluated" along the way. Asking the community if your action plans are actually working is one important example of how to begin an evaluation. If the results are positive, that suggests your initiative is heading in the right direction, and you probably should keep heading there. However, evaluation is also essential in helping one see what "course corrections" might need to be made if the initiative is not heading in the direction you planned.

This chapter will explain the importance of evaluation, and will offer step-by -step methods to evaluate your group's work. Then, when you discover that your efforts are beginning to make a positive difference, you can continue on with even more energy and enthusiasm!

What does it mean to evaluate a community initiative?

Basically, it means to determine the value of the work. You have developed and implemented an initiative in your community, and you want to know how well it's working. Evaluation provides you with this feedback.

In many avenues of life, we get feedback right away. You hit the brakes, the car stops. You shoot a basketball; it either goes in or it doesn't. That's instant evaluation, and it's completely understandable. There's no ambiguity. But with more complex events, such as social interventions, the results are not always as clear. (Sometimes they are--you schedule an event and people are lining up around the block to get in--that's rapid and clear evaluation; but it's rare). That's why you need to put more energy and thought into finding out how you did. And that's basically what evaluation is all about--giving you information on the value of your work.

In this chapter, we'll talk about evaluation as a means of obtaining feedback, data and information about your group and its activities. By using this information, you can decide what aspects of your action plan work, and what areas need improvement. When you evaluate your program, then, you are gathering information to help you draw conclusions about your project and the efforts of your group. After you have drawn conclusions from the information, you can make any necessary changes to your goals and/or action plan.

The sections in this chapter will focus specifically on ways to gather these valuable data. The next chapter will discuss methods to best use the data collected to strengthen and improve the initiative. Resistance to evaluation is common. To gain the greatest benefits from evaluation, you must overcome some common misconceptions about it.

Fears of evaluation

Evaluation can be frightening to many people. Generally, these fears fall into three types--"I don't know how;" "I don't have time;" or "The results might be negative and hurt us." All of these are valid concerns. But they shouldn't be so discouraging as to outweigh the benefits of doing an evaluation. Here are some responses to these concerns:

I don't know how to do an evaluation.

Well, that's why we're here. We'll help you through the steps of how to plan an evaluation of your program, how to do the evaluation itself, and then how to use it to help your group.

I don't have the time.

Evaluation can take some time to do. That's absolutely true. However, doing an evaluation now will save you a lot more time "down the road," as it will point out potential problems while they're still small problems, instead of waiting until a disaster occurs.

And there is another time-saving benefit. Once you begin to record the kinds of information you will need for your evaluation, it will become just a regular part of your routine operations. It will take less time later on. The hardest part is the beginning.

The results may be negative or hurt us.

This is a possibility. However, it's unlikely to happen if you use evaluation from early on and don't let little problems grow into big problems. And remember that any negative results you may find should actually be helpful to you, at least in the long run. They will help you improve the quality of your program or initiative-- which should be one of your own goals from the start.

Why should you evaluate a community initiative?

Being successful demands careful attention during the beginning, middle, and end of a project. If a violinist wants to learn a new piece of music for an upcoming concert, for example, she would prepare by practicing for many hours each day. But, if she never asks her teacher to listen to her play, she may be playing the music too slowly, too fast, too softly, or too loudly. If she never knows the proper way to play the piece--if she never gets any feedback--all of the practice in the world won't help her sound in tune and in time on the night of the performance.

Like the violinist, community groups need to pay careful attention to feedback during the beginning, middle, and end of their projects. An initiative can devote a great deal of time and energy to working on meeting its goals. But, if the work isn't heading in the right direction, all of those long hours and hard work can lead to frustration instead of a feeling of success. Evaluation tells the group how it's doing and helps identify any necessary changes along the way that will help you stay "in tune" with your own goals and the needs of the community.

There are many reasons why evaluations are valuable. Let's look at a few examples of ways in which evaluation can benefit a community group.

• Success is reinforcing - it brings more resources your way. There is an old adage that says, "Nothing succeeds like success." It stands to reason that the more successful your group's work proves to be, the more support and encouragement you might receive from members of the community and maybe even from funders. Evaluation can document your success, with facts, figures, and examples. If volunteer hours in your organization increased by 100% last year, or if every single child in your community was up-to-date on immunizations, those types of achievements can bring new resources your way. In other words, if evaluation can provide concrete examples of your group's successes, that can only be advantageous. In this way, evaluation can help you "toot your own horn."
• Failure is instructive. Even if your work falls short of its goals - and even if your program falls flat on its face - that knowledge can be helpful too. It may be painful in the short run; it might hurt. Yet negative feedback, or a negative evaluation, can really help you in the longer-range scheme of things. At least you know what the truth is, and where you stand. You have fewer illusions. And once you have dusted off, you will probably learn from the evaluation you received. Chances are you won't make the same mistakes again. You are now in a better position to make improvement.
• Evaluation can make you feel good. Being able to see your successes and the value of your work will obviously boost your spirits and motivate you to continue with your work. Again, this works both ways, but even negative aspects should be seen as an opportunity to learn about what works, and not as a failure!
• Evaluation raises the chances of further action. Once you have completed your first evaluation, you know what has worked and what has not for your group. You can modify the tactics that didn't work as well as planned, and reinforce those areas that were successful. So now you can take further action with an even greater chance of success! Others will notice this success and may join or help your group, further increasing your chances that your program will make a positive impact.
• The evaluation can help you understand important aspects of the initiative. You've just finished your first evaluation. The results may indicate some part of your initiative worked really well. For example, the free cholesterol screening at the local health clinic you sponsored was jam-packed with people waiting to be tested. On the other hand, no one is showing up to your monthly Healthy Cooking classes. Maybe it's because the classes are only held during the day, or maybe the classroom is located too far away from most of the people in your community. There are many possible reasons your project might not work as planned. Evaluation will help you understand why things worked, or didn't work, as they did.

When should you evaluate a community initiative?

Evaluation should take place during several stages of your group's life.

When your plan is in action:

• Determine baselines for behaviors you wish to change. If you want to know how much change your program has brought about, you'll need to know what was happening before your group got started.
• Focus on the impact your work is having on the community.
• Continue revising and updating action plans.
• Keep the group strong and focused on the goals at hand. You may want to use a survey that appraises your community goals, and use the feedback to change your planned priorities.

When some of your action plans are complete:

• Use the evaluation to help the group continue to measure its impact on the community, and to create plans for continuing helpful programs for the future. For example, community level indicators will tell you if your interventions are having an impact on the bottom line.

How do you begin an evaluation?

Suppose you know you want to conduct an evaluation. But where do you begin? The process of evaluating can sound overwhelming. However, remember that what you've already accomplished may have seemed hard to imagine at the beginning. Evaluation can be broken down into several parts to make it more manageable.

Before we jump into specifics, here is a general thought to share with your group members: your evaluation should address questions that are important to you, members of your community, and those who are providing financial support. Of course, finances and time constraints might limit the kind of information that can be collected. Try to ask the questions that will ultimately help your group succeed. Because each group is unique, the evaluation should be as well. Nonetheless, below are some questions of interest to many groups. Would they be of interest to yours?

• How much did the community participate?
• What programs, policies or practices have changed in the community?
• Have people's behaviors changed? If so, what kinds, and how?
• Are those changes due to your efforts?

Tips for implementing a successful evaluation

You have to want to evaluate.

The first step is internal. You have to be motivated to do the evaluation, or else it will be half-hearted, if it happens at all. And you need to be clear on the purpose of the evaluation. Why do you want to do it? For example, if you project was to reduce smoking among adolescents, you may want to find out if your program has lead to stricter enforcement of laws regulating the sale of tobacco to minors.

Your evaluation should be grounded in your stated objectives or goals.
When you planned your initiative, you should have clearly identified what you intended to accomplish. What specific outcomes are you trying to achieve? Having well-defined objectives is valuable because they serve as the foundation for your evaluation.
For example, if your objective is to reduce the percentage of local high school students who use tobacco to 5% by May 2032, the evaluation standard is straightforward. You simply measure the percentage of students who use tobacco at that time and compare it to your target. Clear objectives make it easier to determine whether your initiative has been successful.

For each of your objectives, you need to identify criteria, or indicators, which will provide reliable and valid measures for each of your objectives.

You will need to develop measures that tell you what is really happening. Once again, your measures (or indicators) should be based on your objectives.

For example, if your objective is to reduce youth violence in your local high school, then some possible measures might be hospital admissions records for violence-related injuries, attendance at the school's conflict resolution training seminar, or police records of arrests of youths for assault or carrying concealed weapons.

If your objective was increasing the availability of heart healthy foods in your community, some measures might be the number of restaurants who have a low fat menu section, the percentage of milk that is skim in local grocery stores, or the amount of shelf space used to display lean meat versus higher fat cuts.

You need to collect data on each of these indicators.

Sometimes you can find the indicators you need from existing sources. For example, if you were interested in increasing library borrowing, or in reducing false fire alarms, you could gather existing data from the library or fire department. But sometimes data on your chosen indicators may not be available. Suppose, for instance, your number one issue was speeding along residential streets. The key information here unfortunately may not exist. In those cases, the local police might be willing to help collect it; or you and your group might need to collect it yourselves.

Either way, if you can assemble "before" and "after" statistics on your chosen indicators, you can use them to help determine whether your program or initiative made a positive difference. Did borrowing go up? Did traffic slow down? The data here will do a lot of the talking.

Use the results to adjust the program or intervention as necessary.

Are you meeting the objectives you had planned? If so, no adjustment may be needed. If you are not meeting those objectives, the data may indicate what changes need to be made to get back on track. For instance, in our example on high school smokers, if the percentage of high school students who quit smoking is not very high, you might want to change the content of the high school tobacco education program now being given or add a new program to help them quit.

In Summary

Ask yourself, "What questions do I want to answer?" That's a key first step. Now, how do you answer them? Each of the following sections in this chapter will thoroughly explain methods that you can use to evaluate your initiative.

What is evaluation?

___You understand that evaluating means determining the value of your work

What are some objections to evaluation and how can you resolve those objections?

___"I don't know how to do an evaluation."

• Use a resource such as the Community Tool Box to find guide once the steps you should take to plan and implement the evaluation, and use the results.

___"I don't have time."

• Although it is true that evaluation takes time, it will save time in the long run, by pointing out potential problems before they become disasters

___"The results may be negative or hurt us."

• If you begin evaluating early, you should be able to prevent any major problems
• If you do find problems, you can use that to improve the quality of your program

Why should you evaluate your program?

___Success is reinforcing

___Failure is instructive

___Evaluation can make you feel good

___Evaluation raises the chances of further action

___Evaluation can help you understand important aspects of the initiative

___You should evaluate your initiative when your plan is in action, or when some of your plans have been completed

How should you do an evaluation?

___You want to evaluate

___You have evaluated in terms of objectives or goals

___You have identified criteria or indicators that will provide ways to measure your objectives

___You have collected data on each of these indicators

___You have used the results to adjust the program or intervention as necessary

What does it mean to monitor your progress?

Monitoring your community initiative can help you weigh your actions against the results to see if you are meeting the goals of the community and your initiative. In a sense, monitoring data helps you understand how well the initiative is functioning. That is, monitoring can help pinpoint where the actions of the initiative are not producing the desired effects. Additionally, the monitoring system can help you.

• Better understand the initiative
• Make decisions concerning the programming of the initiative
• Promote awareness of accomplishments
• Recruit support
• Secure funding

Despite the scary sound of "monitoring system," you have probably already observed examples of monitoring in a variety of ways. Political candidates monitor the status of their campaigns by conducting polls and analyzing the results. Teachers monitor the progress of their students by giving tests at the beginning and end of the school year to see if they have mastered the secrets of long division. You might monitor your utility bills by keeping track of the monthly increases and decreases. Monitoring has a wide variety of applications. As a member of a community initiative, monitoring means a way of tracking major events and accomplishments of the initiative.

There are three key parts to the monitoring system:

• Process and outcome measures
• Observational monitoring system
• Regular feedback on performance

In this section, we'll explain what we mean by process and outcome measures, and the observational system. We'll tackle ways to provide feedback to members of your coalition and your community in later sections.

Process measures

Process measures are the activities that take place during the initiative which help you determine how well things are going.

Process measures can include many aspects of your initiative, such as:

• Members who participate: the number and type of participants, frequency of attendance, and turnover rate of the members.
  • Example: At the last general meeting of the initiative, 17 people attended. Of these folks, 10 were regular members, and 7 were sitting in on a meeting for the first time.
• Planning products: written objectives, by-laws, or committees that contribute to the initiative.
  • Example: The action plan for the coalition was approved by the coalition and will be implemented immediately.
• Media coverage: by radio, television, and print media.
  • Example: Several five minute radio spots describing one of your group's projects aired on a local FM radio station.
• Financial resources: grants or donations. Financial resources also can include "in-kind" services, such as free advertising or products that an individual or business might offer instead of money.
  • Example: $8000 was received at a $50 per plate lunch that was held to raise money for local drug and alcohol abuse efforts.
• Services that are ultimately provided: classes, programs, workshops, publications or other services or communications provided for the community by the initiative.
  • Example: Nutrition education workshops were conducted by child care providers.
• Community actions: actions taken to encourage change in the community.
  • Example: Merchants were asked to display signs describing the penalty for selling alcohol to minors and the need for proper identification.

Later, you will incorporate process measures into event logs to better measure the progress of your group's work. We'll walk through an example of an event log later in this section.

Outcome measures

While process measures document the specific methods you use to create change, outcome measures explain the overall impact that occurs as a result of these individual actions. Outcome measures highlight the changes that happen in the community as a result of the work done by your initiative.

These include:

• Changes in programs, such as a new or modified service program.
  • Example: A parenting class was implemented by the initiative.
• Changes in policies, such as a new or modified policy.
  • Example: A city ordinance was passed requiring owners of cigarette vending machines to place on every machine a sign that states "No cigarette sales to minors." The legislation was introduced at the urging of the Law Enforcement and Government Committee.
• Changes in practices, such as a new or modified practice.
  • Example: Merchants displayed signs describing the penalty for selling alcohol to minors and the need for proper identification.

As you can see, the monitoring system involves several components that will help your group determine how it's doing, what it's doing correctly, and what can be improved. Sound interesting? Read on for more!

Why do you need to monitor your progress?

Now you might be thinking to yourself, "You want me to gather information about our initiative. But what will I do with all of those numbers and comments? Why is this important for the success of our group?" Don't worry; the data you gather by using the monitoring system can help the group in a variety of ways.

• Data can tell you where the initiative places its emphasis. For example, the monitoring system might reveal that your initiative focuses on services, rather than change. If change is what you want, the monitoring system will help you detect this at an early date.
• Data can point out which groups in the community are affected by your initiative. Is your initiative producing a lot of change in the schools, but little change in the criminal justice system? Is this what you and the community want and need? Who is being targeted? With the results of the monitoring system, you might be able to better answer these questions.
• Data can reveal which strategies are being addressed. The monitoring system can determine whether your initiative is offering information without following through with peer support or access to other resources.
• Data can be used by the staff to achieve a variety of results. Staff and leadership can use the data to promote community awareness of the initiative's activities and accomplishments, recruit community support, and secure financial resources.

How do you monitor your progress?

Before you can start analyzing your data, you first have to collect it. What follows is a guide to creating the observational monitoring system that you will need to collect the data.

Collect data by completing event logs and other forms on a regular basis

Event logs are written accounts of the major activities of the initiative. They might also be used to record any changes in the community brought about by the initiative, such as new programs, policies, or practices related to the initiative's goals and mission.

The event log might include important information such as:

• The month/day/year of the event
• A description of the event itself, including:
  • Why it was important
  • What happened as a result
• A description of the details of the event, including:
  • Who was involved
  • What organizations contributed people and resources
  • What community sector or objective this relates to
  • If this is the first time this event happened

For example, a group called the Adolescent Pregnancy Prevention Initiative in Franklin County, Kansas, participated in an event to help bring about changes in the community that are related to reducing adolescent pregnancy. At the next coalition meeting, Maggie B., a coalition member who helped organize the event, completed the information in the event log. She wrote: "Shannon P. and Gary M. participated in a meeting at Ottawa High School on February 14, 1996. They met with the high school principal to get permission to organize a Teen Action Committee in the school. The Teen Action Committee was approved by the principal. This was important because it will create leadership opportunities for youth."

Maggie recorded this information in a chart where she identified details such as the month, day, and year of the event, a description of the event itself, and a description of the details of the event.

Have several people, such as project staff and active members, complete the logs

The people who fill out logs will be those members who are taking action on behalf of the initiative. Ideally, everyone who is doing something for the initiative will complete event logs. Event logs should be completed no later than one week after the event took place. That way, the important details will be fresh in everyone's mind! The completed event logs can then be given to the people in the group who are in charge of evaluation.

Gather your information carefully and accurately

We've found that data collection in our own work takes about two hours to a week to complete. Of course, if your initiative is working on a particularly involved project one week, you might spend twice or even three times that amount of time gathering data. Carefully gathering your information will mean that the results that are generated will be more accurate, and therefore more valuable to your initiative.

Organize the data so that it can be used

Once the information has been gathered, it needs to be turned over to the people in your group who will put the data into some kind of organized form so that the initiative can use it in a beneficial way. This step in the process might take a good deal of time; but, don't be scared! As you have heard many times already, the benefits you receive from the evaluation process will only be helpful if you spend the time to carefully analyze the results. While this might take a few hours, in the end, you'll be glad you did it!

Summarize the data and distribute that summary to the group

Those who organize the information that is gathered will then take time to summarize the event logs. The results will then be distributed to members of the group.

In Summary

Congratulations! You've just finished learning one of the most important ways to evaluate a community initiative. Keep reading to see examples of event logs, and other monitoring charts. But don't stop there! More exciting sections await you and your initiative. The next chapters will introduce the constituent survey of goals, process, and outcomes, and other methods to measure the progress and success of your initiative. Remember, monitoring your initiative helps your group be the best that it can be!

You understand that monitoring my community initiative can help you:

___Weigh actions against the results to see if we're meeting the goals of the community and the initiative

___Understand the initiative

___Make decisions concerning the programming of the initiative

___Promote awareness of accomplishments

___Recruit support

___Secure funding

You understand these 3 key parts of a monitoring system:

___Process and outcome measures

___Observational system

___Regular feedback on performance

You understand that process measures can be used to monitor the following:

___Members who participate

___Planning products

___Media coverage

___Financial resources

___Services that are ultimately provided

___Community actions

You understand that outcome measures can be used to monitor the following:

___Changes in programs

___Changes in policies

___Changes in practices

You understand that the data you gather by using the monitoring system can help the group in the following ways:

___Data can tell you where the initiative places its emphasis

___Data can point out which groups in the community are affected by your initiative

___Data can reveal which strategies are being addressed

___Data can be used by the staff to achieve a variety of results

Data collection checklist:

___Have key project staff and active members of the initiative keep event logs of all events

The event logs include the following information:

___The month/day/year of the event

___A description of the event itself, including:

• Why it was important
• What happened as a result

___A description of the details of the event, including:

• Who was involved
• What organizations contributed people and resources
• What community sector or objective this relates to
• If this is the first time this event happened

___Complete event logs no later than one week after the event has taken place

___Give completed event logs to the people in the group who are in charge of evaluation

___Turn over the information gathered to the people in your group who will put the data into some kind of organized form so that the initiative can use it in a beneficial way

___Those who organize the information that is gathered summarize the event logs

___The results are distributed to members of the group

Example: Event Log for an Adolescent Pregnancy Prevention Initiative

Site: Franklin County

Recorder: Maggie B.

Tool 1: Event Log

The event log form is designed to help you record major activities of your group's initiative, such as community actions. They are also used to record any changes in the community facilitated by the initiative.

Event log for: ______________

Site: ____________________                        Recorder: ____________________

For this form, please describe:

• Actions taken to bring about changes in the community related to your initiative
• Changes in programs, policies, and practices related to your intiative

Tool 2: Observational codes for the monitoring system

These tables provide a list of key events, definitions, and examples of items using the observational code for the monitoring system as well as general instructions on how to code key events on the event logs.

Codes for Internal Events

Codes for External Events

What does it mean to rate community goals?

Rating community goals that have been developed through a planning process is a way to discover if the community believes its goals are desirable and realistic. The goals should be appraised by people both in and outside your group. They should be asked to explain which goals they believe are most important and most feasible for the community. Asking a variety of individuals to express their opinions about what goals would best serve the community will increase the likelihood that actions taken later will have the support and backing of the community. Goals can be any change in a policy, program, or practice that may have an impact on the issue your group has chosen to resolve.

Why rate community goals?

When your group rates the community's goals, it is listening to the community --What should the goals be? How should these goals be reached? Your group then refines these goals, clarifying and carefully defining them. Your group then asks the community to rate the revised goals to ensure they reflect the community's views, and what they think will work effectively.

For example, let's say you and some neighbors decide to develop a program to reduce cardiovascular disease in your community. To understand the basics, you read some helpful books on health promotion, community development and cardiovascular disease. You collect information from community members on the types of programs people want the most. But, suppose everyone's vision isn't the same! Maybe a group of people thinks you should work on building a community walking-trail. Others don't think this. They have lived in this part of the country for many years longer than you. Their experience tells them past similar programs aimed at increasing physical activity in their community have failed, and only aerobics classes have worked in the past, for a variety of reasons. Others have different suggestions. To satisfy everyone's dreams, you would probably try to incorporate as many suggestions as possible into your grand plan.

You then organize everyone's view into goals and present the results to your community. The community members vote on the goals they think are most important, and most easily accomplished. Based on the votes, you prioritize the goals. You sit down to create a five-year plan or design describing the steps you will take in order to see a once passive, apathetic community come alive with activity and an interest in doing something about reducing heart disease.

In a similar way, your group can benefit from asking both members of your group and members of the community what they think might work best to solve a problem in your community.

Appraising community goals can help the community and your group:

• Reach consensus among members
• Continue to set priorities
• Validate the choices they have made about their goals

When should you rate community goals?

In this step of the evaluation process, the early bird gets the worm. That is, the best time to rate goals would be before you set your action plan in stone, and certainly before you begin to take action. That way, your group can make any needed adjustments before beginning work toward meeting its objectives.

How do you rate community goals?

Do you want to hear people's opinions? Do you want to know what the community thinks about the goals your group has developed? Do you want to feel confident that your group's plans have a solid chance of producing change in the community. If you do, you are ready to conduct a survey of members of your group and community who have something to say about the issue.

However, before you begin, you should review the reasons for doing the survey. That is, what do you hope to learn by asking for input about the importance and feasibility of the community's goals?

• Verify the importance of the goals
• Prioritize the goals
• Revise the goals

Develop a survey that lists and explains your objectives

At this point, members or staff of your group will create a survey that asks for importance and feasibility ratings about proposed goals. A small, or fledgling initiative may only have a dozen or so goals, while a large, extensive, long term initiative might have hundreds of goals.

For example, if your group wants to reduce the rate of teenage pregnancy, one of the goals listed on the survey might be: "Distribute written material to schools and the target area concerning the problem of teen pregnancy."

You will ask the person filling out the survey to rate the importance and feasibility of each goal. How important is this goal to accomplishing your mission of reducing teen pregnancy? How easy will it be to reach this goal?

The importance of a goal to your mission is related to how much of an impact it will have on the problem. Distributing brochures on the legal consequences of selling tobacco to minors to convenience stores in your community will probably help reduce smoking among high school students more than the goal of creating a biking/hiking trail.

Feasibility, on the other hand, is asking people to rate your chances of accomplishing the goal. That is, how many barriers and obstacles such as the amount of resources available, stand in the way of you reaching the goal.

For example, the goal of establishing a peer education group in a local high school to help reduce teen pregnancy is probably more feasible than distributing free birth control to any high school students that ask for it. The latter is, of course, highly controversial and could meet political opposition.

Members of your group who are responsible for evaluation or members of an outside evaluation team (if you have the resources to hire them) should propose a format for the survey.

These people would then help design and develop it. However, here's an easy way to set up the survey:

For each goal, give the respondent a scale of 1 through 5, with "1" being "Very Unimportant" and "5" being "Very Important" Then the respondent can evaluate each goal according to both it's importance and it's feasibility.

The figure below illustrates a sample Survey of Goals used in a school-community initiative to prevent adolescent pregnancy.

Survey of goals for school/community initiative to prevent adolescent pregnancy 

 Looks easy enough? Try creating your own!

Conduct a survey of constituents

Now is the time to distribute the survey to your constituents or those persons most interested in the problem:

• Members of your group
• Outside experts
• Funders
• Those people affected by the problem
• Other interested community members

These are the individuals who will provide different perspectives and ideas about the plans your group is proposing. By sampling a diversity of voices, you bring to bear on the goals vast experiences and community knowledge.

Ideally, you might distribute the surveys when you have a captive audience--at meetings or similar events. Mailing the surveys is also an option, although you should keep in mind that return rates might be low. Also, do not forget to include a return address so people will know where to return the survey! Stamped envelopes help too... If you have the resources.

Review the ratings from the returned surveys

Ideally, the information that you receive should be broken down into two categories:

• How is this goal important to the mission?
• What is the feasibility, or possibility, of this goal becoming a reality?

By examining the information returned on the survey, you can better evaluate the importance and the feasibility of your group's plans. How do you go about evaluating this information? First, someone in the group will be responsible for averaging the ratings for importance and for feasibility of each proposed goal.

Let's look at an example of what the results might look like. Let's say 10 people returned a survey that included the four goals from our previous example. The responses were averaged and the results compiled in the table below.

Sample Importance and Feasibility Ratings from Survey of Goals

But what do these numbers mean? It might be useful, at this point, to order the goals by their importance rating. The ratings are important to you because they tell you how the community feels about its goals. Does a 3.0 for importance mean the goal should be included in the action plan? Not necessarily. There's still one more step to help you prioritize your goals even further.

Use the survey results to finalize the plan

Doing the survey will prove useful only if your group uses the results to select and implement community-supported goals and to reevaluate those that are viewed less favorably.

There are several ways to order the goals, based on the community responses to them:

• Must Do: These are goals the community thinks are important and feasible. You will want to pursue these right away.
• Important to Try: These are goals the community thinks are important, but will be difficult to accomplish. You should try to do these, but keep in mind these goals will require extra effort to accomplish.
• Easy to Do: These are goals that the community thinks are easy to accomplish, but are also are not all that important. You should do these if you need to increase your group's credibility.
• Last Resort: These goals are of low importance to your community, and are difficult to do. The only time you would want to do these is if you know something the community doesn't; because you won't get much support and even if you do succeed it may not matter.

Matrix for Prioritizing Goals

How do you decide which importance and feasibility ratings are high or low?

This is a somewhat arbitrary decision. You can divide them in half and lump them into high and low categories..

Using the ratings of importance and feasibility from our earlier example of importance and feasibility ratings, the first goal of "Distribute written materials in schools and the target area concerning the problem of teen pregnancy." received an average importance rating of 1.5 and a feasibility rating of 1.7. These results would put this goal in the Last Resort quadrant, as it has low feasibility and low importance. Therefore, you would probably want to drop this goal as it would be too difficult to accomplish, and most likely would have little meaning or effect on your community if you were successful.

The second goal, "Provide support group training for youth and adults," on the other hand, received high feasibility and importance scores of 4.5 and 4.1, respectively. This goal, with high feasibility and importance, falls into the Must Do quadrant of the matrix. You would certainly want to pursue this goal quickly, as you will most likely be successful and your success will lend credibility to your group.

So what about the third goal of "Working with USD 259 to facilitate parent/teacher interaction and involvement in sexual risk reduction activities," which had an importance rating of 1.8 but a feasibility rating of 3.9 This falls into the Easy to Do category. That is, the goal is not that important to the community, but it would be easy for you to do. What should you do? You may want to go ahead and do this as your success will build your group's credibility.

Finally, the fourth goal, of "Recruit and train teachers and students involved in drama clubs, journalism and other clubs to begin youth presentations and support group activities about teen sexuality" received an importance rating of 3.7 and feasibility rating of 1.6. This places the goal in the Important to Try quadrant of the matrix. You probably should go ahead and try to reach this goal, as it will definitely be important to your mission. However, this will probably not be an easy task, so keep in mind the extra effort it will take to accomplish the goal.

In Summary

Appraising community goals allows you to get a better feel for the likelihood your constituents will join your group in pursuing a goal.

The entire process (from developing the questions to final summary of the data) can take a long time, depending on the size of your group. For small groups turn around time might only be a few weeks, while large community work groups might take months. From our experience, staff or members of your group could spend from a few hours to over 10 hours, again depending on your size, in developing questions, distributing appraisals, and tabulating results. Ultimately, all of your careful preparation will result in a plan that has the potential to be tremendously useful in developing consensus and direction in your group.

___You understand what it means to appraise community goals

___You understand why you should appraise community goals

___You understand when you should appraise community goals

___You understand how to appraise community goals

___You understand how to survey participants about community goals

___You know how to develop a survey that lists and explains the different objectives that your group hopes to achieve

___You know how to conduct a survey of constituents

___You know how to review the ratings from the returned surveys

___You know how to use the survey results to finalize the plan

Tool: Survey of Goals

Conducting the survey of goals will help your group develop a list of community changes to be sought. It may also be useful in your future strategic planning efforts.

Project:

The purpose of this survey is to help set priorities for_________________________ . We would like your evaluation of the importance and feasibility of each of the proposed changes.

In the first column (Importance), please circle the number that shows how important you feel the listed change is in preventing or reducing __________________________ . In the second column (Feasibility), please circle the number that shows how easy (feasible) it is to reach the objective.

• What is the member survey of process?

• Why should you use the member survey of process?

• When should you use the member survey of process?

• How should you conduct the member survey of process?

What is the member survey of process?

How many times have you visited a restaurant and seen a mini-questionnaire tucked between the salt and pepper shakers on your table? These surveys usually ask you to rate the service, the food, the atmosphere, and the overall quality of the meal.

With that in mind, picture this scenario. Last weekend, you went to a fine Italian restaurant to celebrate a friend's birthday. When you arrived, everything from the simple yet beautiful candles on your table to the friendly service met or surpassed your expectations.

But maybe you wished the waiter or waitress would have been better informed about the evening's specials, or about the different varieties of wine. After finishing your meal, you filled out one of the surveys where you offered suggestions for improvement while still complimenting the many enjoyable aspects of the meal. If the restaurant truly wants to please its customers, it will look at your comments and try to incorporate them into its operation.

With the member survey of process, groups ask members of the group to evaluate their "service" in the hopes of improving weak spots and continuing on with those methods that are proving to be successful. In other words, the member survey of process asks members of the group to indicate their satisfaction with the group's day-to-day operations. Like the restaurant owner who wants to please her customers, a community group can use the survey of process to better meet the needs of the community.

You may be asking yourself, "But, how is this different from the member survey of goals?" Although the two sound similar, they are actually quite different.

• The member survey of goals asks community members to respond to action plans during the planning stages of the group
• The member survey of process gathers opinions from members of the coalition once it has begun work in the community

The survey of process might ask questions about such areas as planning, leadership, services, community involvement, and progress towards accomplishing goals.

Why should you do this type of survey?

The importance of the survey of process

The survey of process reminds members of your group that you care about their ideas, suggestions, hopes, and dreams. If members feel like they can actively participate in all aspects of the group--including sharing their feelings of satisfaction or disappointment--their desire to devote time and energy to the group will probably increase. In this way, both your group and its members benefit from the member survey of process. While your group learns more about its strengths and weaknesses, members feel like an important and necessary part of the work that you do.

When should you use the member survey of process?

Timing the member survey of process

We recommend that this survey be done annually. That way, you can continually receive the input of your members concerning the work done by your group. Our experience with coalitions is that this often takes a real time commitment from staff, and that the amount of time that will be devoted to this survey will vary depending upon the nature of the group.

How should you conduct the member survey of process?

Like many other aspects of evaluation, the member survey of process involves several steps that can easily be broken down into manageable parts.

Develop a survey

You'll want to develop a survey that will help you understand the level of satisfaction that members of the community feel towards your initiative. For this, you'll need some members of the coalition to propose a survey form and help design the survey. You might hold a small meeting of those who might be interested in developing a survey and brainstorm possible areas for questions. Another thing you could do is send out a preliminary draft of the survey to your most active members and ask them to comment, add, and take out questions.

Develop the questions which you will want to ask on your survey. You will want to prepare the questions in such a way that you can determine the satisfaction and the overall approval with the different aspects of the group.

You might ask questions concerning both planning and implementation and leadership.

For example, one of the questions under planning and implementation might read:

Concerning the follow through of the group's leaders on various projects, I felt:

• Very dissatisfied
• Somewhat dissatisfied
• Somewhat satisfied
• Satisfied
• Very satisfied

If a respondent to the survey checked "Very satisfied" for this question, your group would know that this individual's satisfaction was high for this particular area of the group. Then, after having looked over the entire survey, you might see that the respondent answered "Satisfied" or "Very satisfied" for every question. This would lead you to believe that his or her overall approval for the coalition's activities was strong.

Try to come up with questions that assess satisfaction with the following aspects of the group:

• Planning and implementation
• Leadership
• Services
• Community involvement with the group
• Progress and outcome

Obtain the ratings

Your survey might address any or all of the above issues. Depending upon the nature of your group, you might feel like you could use more input in one area than another. In any case, you should cater your survey to meet the needs of your group. At the end of this section is a sample survey to help you better understand the ways to break down the questions.

After the surveys have been written, members of the group can distribute them to all the other members of the group. Sometimes this can be done through the mail, but there are other options depending upon the size of your group and what works best. One option would be to distribute them at meetings and have a "drop off" box at later meetings and events.

Assess satisfaction

Once the respondents have completed and returned the anonymous surveys to the evaluation team, you can begin to look at the information to determine what changes might need to happen in the structure of the group. One way to help accomplish this includes tabulating the results of the survey.

Use the data to improve the functioning of the initiative

Once again, the most brilliant research in the world won't help anyone if it is not used to make changes for the good of the coalition and of the community. Because this survey focuses on the thoughts and ideas of members, it is crucial that these members know that their suggestions are being heard and used by the group.

To help complete the member survey of process, feel free to use and adopt example of the sample cover letter, a sample survey, and a sample collection of results. Hopefully, these samples will help you create your own successful survey.

 Now, take a breather and pat yourself on the back for the great work you've done as you learn about evaluation your group! And then get ready for just one more survey: the member survey of outcomes.

What does it mean to rate member satisfaction?

___You understand that the member survey of goals asks community members to respond to action plans during the planning stages of the group

___You understand that the member survey of process gathers opinions from members of the coalition once it has begun work in the community

How do you conduct the member survey of process?

___You understand the four steps of the member survey of process: developing a survey; obtaining the ratings; assessing satisfaction with different aspects of the group; using the data to improve the functioning of the initiative

___Hold a small meeting of those who might be interested in developing a survey and brainstorm possible areas for questions

___Send out a preliminary draft of the survey to your most active members and ask them to comment, add, and take out questions

___Develop the questions to assess satisfaction with aspects of the group, including planning and implementation, leadership, services, community involvement with the group, progress and outcome

___After the surveys have been written, have members of the group distribute them to all the other members of the group

___After the respondents have completed and returned the anonymous surveys to the evaluation team, start looking at the information to determine what changes might need to happen in the structure of the group

___Tabulate the results of the survey

Example 1: Sample cover letter for member satisfaction survey

Date

Dear Member of (name of the initiative):

The purpose of the attached member satisfaction questionnaire is to get your feedback on how well (the initiative) is doing. As you know, (the initiative's) mission is to ___________________________________ in ____________________ (city/county).

Please complete each question by circling the letter that best shows your satisfaction with that aspect of the initiative. We welcome additional comments and suggestions you have for improving (the initiative).

For your anonymity and convenience, please use the enclosed envelope to return your completed questionnaire to our evaluators, the (name them).

Thanks in advance for your valuable advice and feedback.

Best regards,

______________________________

Executive Director

Example 2: Sample member survey of satisfaction

Annual Member Satisfaction Survey
 

Date
Please return by: (date)

We welcome your feedback on how well (the initiative) is doing. For each item, please circle the number that best shows your satisfaction with that aspect of the initiative. Provide additional comments if you wish.

Example 3: Sample Memorandum of Results

Date: ____________________________________

To: (Leadership, Board of Directors of the group, members of the group, members of the community)

cc: Program officer, Funding source

From: The Evaluation Team

Re: Results of the Annual Member Satisfaction Survey for ______________________ .

Enclosed please find the results of the Annual Member satisfaction Survey. We had a good response - ___people (__%) completed the survey. This report includes the average rating and the range of responses for each question, and a summary of the comments for each section.

Generally, respondents were satisfied with the different aspects of functioning outlined in the survey. However, the range of responses was from 1 to 5 for each issue. The highest ratings were noted for the strength of competence of staff, the strength and competence of leadership, and the clarity of the vision of the initiative. The lowest rating was noted for item 20, the group's contribution to the goal of ___________________. Ninety-three percent of the members who completed the survey felt that the community was better off today because of (the initiative).

Many of the members who completed the survey also provided comments at the end of each section. The comments, in addition to the ratings of each question, may provide some helpful feedback in planning for future activities of (the initiative ).

If you have any questions, please feel free to call us. We will also further discuss the survey findings at our next meeting.

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What is a constituent survey of outcomes?

• Why should we use a constituent survey of outcomes?

• When should we conduct this survey?

• How do we conduct this survey?

What is a constituent survey of outcomes?

A constituent survey of outcomes will help you to examine the greater impact of your group's work. It poses questions to members of the initiative and experts in the community. The people who fill out the survey will express their opinions about the importance of the group's accomplishments for the community as a whole, and for the mission of the initiative.

If your group's focus is on reducing the risk of cardiovascular disease in your community, one of your goals might be to sponsor an employee health fair at the local public schools where you offer free screening and information to reduce the risk for heart disease.

On the survey of outcomes, you might ask respondents to rate this objective as being either "Very Important," or "Very Unimportant." Then, after averaging the responses, you would have a better idea of how this action benefited the community, in the eyes of community members.

Why should you conduct this survey?

As with the constituent survey of process, this survey gives members of the community and of the initiative an opportunity to voice their opinions and express their ideas about the outcome of your group's work. Additionally, the survey gives those who are close to the group more chances to offer suggestions for improvement.

If the group hoping to raise awareness of heart disease found out from its survey that many people benefited from the campaign to reach school employees, it could continue on with this type of community outreach. But, if another goal didn't produce the same type of positive feedback, then the group could think about re-adjusting its action plans to incorporate another type of outreach.

As with all of the aspects of the evaluation, the constituent survey of outcomes is a useful way to get a more objective, unbiased opinion about the work your group is pursuing.

When should you conduct the constituent survey of outcomes?

We recommend conducting this survey every several years, this will increase your feedback and allow you to revise your vision as necessary. The last year of the grant period might be a good time, as well. Sometimes the place that you are receiving funding from will require you to do this survey.

How do you conduct a constituent survey of outcomes?

If you've already conducted the constituent surveys of goals and process, you probably feel like an old pro by now. You'll be happy to hear that you can continue to practice your finely tuned surveying skills with the constituent survey of outcomes!

The constituent survey of outcomes involves four different steps.

Develop a survey

Remember the folks who put together the constituent survey of goals and the constituent survey of process? You'll need their expertise and creativity once again. These individuals should design a survey that lists the accomplishments that your group has proudly achieved. Looking back through event logs and the goal attainment report will help refresh your memory.

Conduct a survey of constituents

Are you ready to distribute the surveys? Great! You'll want to share these surveys with members who have some knowledge of the group and the areas you are evaluating and with outside experts with specialized knowledge in the area. Once you have determined who should receive the questionnaires, they should be mailed, or perhaps distributed at a general meeting of your initiative. Don't forget to clearly state a return address!

Use the data to determine which changes have had the greatest effect

Once you have collected all of the surveys, you will want to find the means of the respondents' answers. If, for each of the questions, you asked the respondent to answer "1" for "Very Unimportant" and "5" for "Very Important" to describe the impact on the community, you would determine an average for the numbers, and then find a range of the numbers.

To find the mean, or the average, response, simply add together all of the ratings, then divide by the total number of respondents. If, for one of the questions, ten people respond with ratings of 3, 4, 4, 5, 3, 2, 5, 4, 4, 3, you would add up all of these numbers and then divide that total by ten. The mean response in this case would be 3.70 (or 37/10). The range of numbers runs from 2-5, two being the lowest and five the highest score.

Use the survey results to refine the group's choices for action

Okay, so now you have a whole batch of numbers. Where do you go from there? That answer, in most situations, lies in your hands. The results of this survey may encourage you to continue on in the same direction, or it may challenge you to rethink some of your plans. In either case, the survey of outcomes is yet another tool to help you work better in your group and in the community.

You understand the process of conducting constituent surveys of outcomes:

___You have decided to whom you will distribute the survey

___You have determined that it is a good time to receive feedback

___A survey has been developed

___This survey lists all community changes that the initiative has helped create

___You have mailed or distributed your survey that has a return address or envelope

___The data from the returned surveys has been compiled

___You have performed basic statistics, the mean and the range, that demonstrate early on simple trends

___You have had your results analyzed

___You have used the survey to refine and rewrite choices for action if necessary

Example: Sample Memorandum of Results

The following is a sample of results from a constituent survey of outcomes. It is in reference to a substance use initiative.

Memorandum of Results

Date: _________________________

To: (Leadership and Board of Directors of the Initiative)

CC: Program Officer, Funding source

From: Evaluation Team

RE: Results of the Outcome Survey for ____________________________

Enclosed please find the results of the Outcome Survey for _____________________. The purpose of the survey was to assess the significance of community changes resulting from the initiative's efforts according to its constituents. We had a reasonable response rate (13%), with 103 completed questionnaires from the 800 coalition members and policy makers to whom the survey was mailed. The attached report include the average rating and the range of responses for each question listed in order of occurrence on the survey. we conclude with written comments related to (the initiative's) accomplishments, a critical review of community changes, and recommendations for improvement.

In general, respondents rated the community changes facilitated by ________________ as "important," an average rating of 3.79 (with 5 the highest). Further, no community change received a rating below 3 (neither unimportant nor important). Taken together, the contribution of the reported community changes to the mission was rated as 4.21 (between important and very important).

The three top rated community changes involved changes in resource allocations or policy: City Council approves the addition of 32 additional police officers for the city, (Mean + 4.39, Range + 1-5), the new, "drive-by" shooting ordinance, approved by the City Council, increases the penalty to a maximum of one year (Mean + 4.38, range + 1-5), and based upon the effort of _______________________, the Governor signed the "drive-by" shooting bill to make shooting a firearm at a home a felony and revisions to the state "drive-by" shooting law (Mean + 4.37, Range + 1-5).

Among the lowest ranked community changes involved small, short-term, or symbolic efforts: _______________________ helped in the development of a job description for the new executive director position at SRS Family and Youth Commission (Mean = 3.06, Range = 1-5), mini-grant awarded to the Holy Savior School to give kids the opportunity to create a project to present at a fair in the Spring of 1992 (Mean + 3.12, Range + 1-5), __________________________ agrees to cosponsor the Great American Smoke-out (mean + 3.21, Range + 1-5), and commitment from the mayor to use drug-free life style sticker slogan on all city vehicles (Mean + 3.21, range + 1-5).

Most of the respondents also provided written comments. The comments are attached and are grouped according to accomplishments, a critical review of community changes, and recommendations for improvement.

You should be proud of these accomplishments. your constituents have noted the significant contributions of these many changes to _______________________'s mission of reducing substance use among adolescents. If you have any questions, please feel free to call us.

Tool 1: Cover Letter for Constituent Survey of Outcomes

This is a good beginning to your survey, it informs the person filling it out what your intentions are.

Cover Letter for Constituent Survey of Outcomes

Date:

Dear Members of (the initiative):

The purpose of the attached outcome survey is to get your feedback on how (the initiative) is doing. As you know, the mission of (the initiative) is to ________________________________ in ____________________________________ (city / county).

Please complete each question by circling the number that illustrates how important each community change is to the mission of (the initiative).

To protect anonymity, please use the enclosed self-addressed envelope and return your completed survey to our evaluators, the (name them).

Thanks in advance for your valuable advice and feedback.

Best regards,

_____________________

Executive Director

Enclosure

Tool 2: Sample Outcome Survey

This survey template can give you an idea of how to set up your own.

                                                             Outcome Survey

Please return by:

Staff and members of (the initiative) have been involved in efforts that resulted in a variety of community changes related to the mission of _________________.

This survey lists (#) community changes that resulted from the initiative's efforts. For each survey item, please circle the number that best describes how important each community change is to the initiative's mission of ______________________. Use the following scale to rate your answers.

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What is a goal attainment report?

• Why conduct a goal attainment report?

• When should you use the goal attainment report?

• How do you create a goal attainment report?

What is a goal attainment report?

For the busy student, keeping track of time can be as challenging as passing a calculus exam, or analyzing that chapter of nineteenth century British poetry. To help stay organized and on target, many people carry date books with them at all times. That way, the student can constantly check to see when he must finish the first leg of that huge group project for his business communications course. Then, when he has completed the necessary steps, he can look back to determine if he stayed on task and completed the work he needed to finish.

Like a date book, the goal attainment report allows your group to look back and check its progress. If you work with a substance use prevention initiative, one of your goals might have been to conduct a series of workshops on substance prevention at the local high school. You wanted to hold all three of the workshops between the months of August and December, and you had specific speakers and topics that you wanted to present.

Come January, then, you would want to look through your list of goals. If you held the workshops, brought in the speakers, and met your deadlines, you could report that this goal had, in fact, been accomplished. This is the purpose of the goal attainment report.

Why conduct the goal attainment report?

Let's once again consider the situation of the busy student who, armed with his date book, knows that he must complete a series of small deadlines in order to finish a big project for his business communications class. By reviewing the steps he has successfully finished, he will not only stay focused on the task at hand, but he will also feel less scared and more confident about his ability to complete the ultimate goal of finishing the project.

In this way, reviewing the goals that your group has or has not completed can not only keep you focused on the big picture, but it can also help you feel proud and confident that you have taken steps, however small, in the right direction of your goals.

Additionally, the goal attainment report can help you:

• Show progress in meeting objectives over time
• Help the initiative stay focused on its action plan
• Communicate the good work you've been doing to the public

When should you use the goal attainment report?

Depending upon the completeness of records that have been kept in the past, the gathering and recording process could take a few minutes or several hours. Generally, we recommend that data be collected and summarized every six months.

How do you create a goal attainment report?

There are five main steps to completing the goal attainment report. Let's look at each step closely:

Write down all goals

This step is easy. Simply write down all of the goals that you defined when you wrote your action plan. This will give you a better sense of where you have already been in terms of completeness, and also where you still need to go.

Identify completed goals

Every six months or so, members of the coalition mark a completion date next to each community change that you had completed in the action plan. If you had planned on holding a substance use awareness program at the local high school on October 5, 1998, and you completed this task, you would write the date of completion next to this goal in the action plan.

Compute percent of goals completed

The members of the evaluation team would then calculate the percentage of community changes that were actually met. Numbers provide quantitative, or numerical, evidence of the group's work in the community. These numbers can then be passed on to members of the community, leaders of the group, and those who are providing funding.

If your math is a bit rusty, this simple calculation might help you balance these numbers. To calculate the percentage of goals met, simply divide the number of goals met by the number of set goals. If your group had established 16 goals for the year, and 13 were completed, you would divide 13 by 16 (13/16) which would give you 81%. A solid B! That's definitely above average.

Communicate your results

Now, armed with these numbers, you can go to others who might be interested in your work to show them all the goals that have been accomplished.

Use the report to inspire positive change

For your purposes, these data can provide valuable insight into the work your group has done. Maybe the goal attainment report will reveal the fact that your group has met all of its goals on time and with flying. Or, perhaps the report will indicate to you the need to rework your action steps to make them more realistic with the resources that are available to you.

If the report highlights weaknesses, that is certainly understandable. In fact, that is the main purpose of this step of the evaluation process. If, after completing the report, your group draws some conclusions that will help strengthen your action plan, then it is worthwhile. In other words, low numbers should inspire positive change, and hopefully not just dissatisfaction.

In Summary

Try to think of it this way. When you plant a garden, some of your flowers will thrive and prosper while others may not grow at all. The reasons for this can be varied; the climate and the soil may be better for some plants and not for others. Try not to let the goal attainment report become a measurement of failures. Instead, let it be a way to help you rediscover the best conditions for your group to blossom and grow!

What are the reasons for making a goal attainment report?

___Show progress

___Stay focused

___Communicate you're good work to the public

How do you implement the process of writing a goal attainment report?

___Determine how often to gather data, usually every six months

___Write down all of your goals using your action plan

___Identify the goals you have completed

___Compute the percentage of goals completed

___Communicate your results

___Use the report to inspire positive change

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What are behavioral surveys?

• Why would you need to use behavioral surveys?

• When should you use behavioral surveys?

• How do you conduct a behavioral survey?

What are behavioral surveys?

As you have probably already guessed after reading some of the other sections in this chapter, asking people questions can be immensely valuable for gaining insight and information into various questions, puzzles, and problems that may exist in your community.

Another type of survey, the behavioral survey, asks people to respond to questions about certain actions or behaviors that affect their physical, emotional, or mental well-being. These behaviors might include cigarette use, unprotected sexual activity, or habits that might increase the chance for cardiovascular diseases.

Unlike the constituent surveys of goals, process, and outcomes, behavioral surveys do not try to determine what people think; rather, they focus on what people do. But, one important distinction to make with behavioral surveys is this: these surveys will tell you what people say they do. Consequently, the surveys must be taken as self-reports. That is, your group should recognize that the results will be subjective accounts of individual actions. This doesn't diminish the value of behavioral surveys; rather, it simply must be taken into consideration when you analyze the data.

Why would you need to use behavioral surveys?

Behavioral surveys provide information that is crucial to the success of the initiative. Without knowing exactly what the extent of the problem is in your community, improving it would be quite a challenge! Behavioral surveys help clear up these questions by providing you with data that can help you move in the right direction towards improving your community.

Also, the data you receive from these surveys can be used to push the issue to the forefront of public awareness. Maybe individuals in your community don't know that a large percentage of the seventh-graders at the local junior high school smoke at least one cigarette each day. Seeing this kind of information on paper and supported by numbers may give it the validity that will convince more people of the need for change. The behavioral survey is an excellent way to obtain this kind of evidence.

Finally, behavioral surveys offer yet another way to evaluate the success of the initiative. Of course, it is important to keep in mind that behavioral surveys focus on self-reported behavior. That's to say, these surveys must be recognized as being potentially-biased reports of what is going on in people's lives. Because some people might exaggerate or minimize their behaviors, these surveys must be examined in this context. However, that in no way makes behavioral surveys less valuable. It just means the results should be interpreted carefully.

When conducted at regular intervals, behavioral surveys can reinforce the fact, for example, that your group's work has helped decrease the number of self-reported teenage smokers in your community. Seeing the proof in numbers can give you a feeling of satisfaction, and renew your determination to keep working with continued spirit and zest!

When should you use behavioral surveys?

We recommend that surveys be conducted at regular intervals. That might mean every three months, every six months, or once a year. That will depend upon the resources your group has. Regardless, the surveys should be distributed regularly so you can keep you finger on the pulse of the community. Read on for specific details on how to conduct the survey.

How do you conduct a behavioral survey?

Are you ready to go? Let's get to it! The behavioral survey requires four steps.

Obtaining behavioral data

The first step in this process involves defining your objectives. You already did this on a grander scale when you determined objectives for your initiative in general. Now you will want to repeat this process as you try to determine, "What do we want to discover by conducting a behavioral survey?" After you have determined what your goals are for the behavioral survey, then you can begin to write the questions.

Here are some questions that might help you determine these objectives.

• What do we hope to learn from this survey?
• Who do we want to survey?
• How will the information we will obtain help us achieve our goals and objectives for the group?

Next, put these questions to good use. To conduct a behavioral survey, you must first decide whether your group wants to write, design, and distribute the survey yourself. First, you will have to come up with questions related to your group's objectives. There are many ways to conduct behavioral surveys. One way is to use a measurement tool called the Likert scale. This scale uses questions that ask individuals to rate their activities. Below are some questions that might appear on a behavioral survey using the Likert scale.

Following are some examples of topics to consider and possible questions for several different areas of focus.

If you are focusing on substance use, you might ask questions about the use of cigarettes, smokeless tobacco, alcohol, marijuana, and cocaine.

For example, if you are focusing on substance use, you might ask questions about the use of:

• Cigarettes
• Smokeless tobacco
• Alcohol
• Marijuana
• Cocaine

A possible question would be:

How often have you used marijuana in the last 30 days?

• Never
• One to three times
• Once a week
• More than three times a week
• At least once a day

If you are focusing on adolescent pregnancy, you might ask about:

• Sexual activity
• Use of contraceptives

A possible question would be:

How often have you had sexual intercourse without using some method of birth control in the last 30 days?

• I used birth control every time I had sexual intercourse
• I sometimes used birth control
• I usually didn't use birth control
• I never used birth control
• I am not sexually active

If you are focusing on tobacco control, you might ask about:

• Use of cigarettes
• Quit attempts

A possible question would be:

How many cigarettes have you smoked during the past 30 days?

• None
• Less than one cigarette per day
• One to five cigarettes per day
• About one-half pack per day
• About one pack per day
• About one and one-half packs per day
• Two packs or more per day

If you are focusing on cardiovascular risk reduction, you might ask about consumption of:

• Higher fat foods
• Fruits and vegetables
• Breads and grains

A possible question would be:

Yesterday I ate:

___Three to five servings of fruits and vegetables
___One to three servings of fruits and vegetables
___No fruits and vegetables

With these examples in mind, now you can write a survey that will address the specific needs of your group and your community. But then what?

If your target population is the adolescents in your community, you would want to distribute the surveys to local elementary, junior high, and high schools. If you're going to be researching a potentially sensitive subject (for example, sexual activity or drug use), think about presenting a finished copy of the questions to the school before you begin working with the students. You may need to receive permission from the principal, teachers, and parents before you begin work. This way, you will have the help and support of community members as you conduct your survey.

If you simply want to gain an understanding of the behaviors of the general population, you might conduct a random phone survey to reach a sampling of the population.

Another great resource might be local school districts, health departments, or other community organizations. Maybe they have already conducted a behavioral survey of their own. If so, asking if you can use their data might help you feed two birds with one seed.

In order to obtain a more national perspective, you may want to contact the Centers for Disease Control and Prevention (CDC) in Atlanta to obtain information about adult health issues and a variety of youth issues such as substance use, adolescent pregnancy, and youth violence. For more information about the CDC, see Resources at the end of this section.

Tabulating the data

Now that you have an assortment of numbers in your hands, it's time to calculate the data. Evaluators or others in your group will want to determine the percentage of people who engage in risk behaviors in your community, perhaps compared to others. Then, this information can be used to benefit your group by helping you better understand the extent of an issue in your community.

Plotting the data and providing feedback

By plotting the data in a chart form, you will have a visual representation of the problem. Also, by plotting the data, you'll be able to see the trends of the problem over time. These trends might include changes in data across different age groups, and changes in data across time. For example, you might graph the regular use of substances (such as cigarettes, smokeless tobacco, alcohol, marijuana, and cocaine) by high school seniors in the community across different years. Finally, you'll want to provide feedback to your group and to the community.

Using the data

Now, what should you do with this data? You have several possibilities! First, you will probably want to distribute the data to members of the group and to leaders of the community. Then, these data can help serve other important purposes.

Some of the important options include:

• Using the data to assess risk. That is, the information you gather can help you determine where the greatest risk lies, and how you can continue to address that need in your community. If, after several years of working towards ending teen pregnancy, your behavioral surveys reveal an increase in unprotected sexual activity, you'd know to continue focusing your objectives on addressing this issue. More importantly, however, you could change your approach so that you can begin doing work that lowers the incidence of unprotected sexual activity.
• Raising public awareness about the issue. The results from these behavioral surveys can be yet another way to push important issues to the forefront of your community's agenda. Again, sometimes attaching a number to a problem helps people put it into a perspective that is easier to understand. If your research shows that the number of teenage smokers continues to rise in your community's junior high and high schools, this would be a terrific time to spread the word. Who knows? With everyone in the community working together towards ending a problem, greater progress might be made.
• Evaluating the effects of the initiative. In addition to all of the other benefits of behavioral analysis, you can also use the information you've obtained to evaluate the initiative. Like the other evaluation methods, behavioral surveys can help you recognize the weaker points in your action plan so you can try to strengthen and improve them.

As always, these evaluation methods should be seen as springboards for continued improvement. And, they will help you stay inspired to keep up the great work you've done and that you will continue to do!

What are behavioral surveys?

___You understand that behavioral surveys ask people to respond to questions about certain actions or behaviors that affect their physical, emotional, or mental well-being

___You understand that behavioral surveys do not try to determine what people think; rather, they focus on what people do

___You understand that your group should recognize that the results will be subjective accounts of individual actions

What are the three basic uses of behavioral surveys?

___Behavioral surveys can shed light on exactly what the extent of the problem is in your community

___The data you receive from these surveys can be used to push the issue to the forefront of public awareness

___Behavioral surveys offer yet another way to evaluate the success of the initiative

What are the four basic steps for conducting a behavioral survey?

___Obtaining behavioral data

___Tabulating the data

___Plotting the data and providing feedback

___Using the data to determine the level of risk, raise public awareness, or evaluate your group's level of effectiveness

Conducting behavioral surveys:

___You will conduct behavioral surveys at regular intervals

___You have defined your objectives for the survey

___You have written questions related to your group's objectives that addresses the specific needs of your group and your community

___If appropriate, you've considered conducting a random phone survey to reach a sampling of the population

___If appropriate, you've checked with other area groups to see if any of them have already conducted a behavioral survey of their own that you can use

___If appropriate, you've considered contacting the Centers for Disease Control and Prevention (CDC) in Atlanta for information

___You have calculated the data to determine the percentage of people who engage in risk behaviors in your community and the extent of an issue in your community

___You've plotted the data in a chart form to have a visual representation of the problem and see the trends of the problem over time

___You've used the data to provide feedback to people involved in the initiative

___You've decided whether to use the data to help you determine where the greatest risk lies, and how you can continue to address that need in your community

___You've decided whether to use the data to raise public awareness about the issue

___You've decided whether to use the data to evaluate the effects of the initiative

Tool: General Resources for specific community concerns 

Substance Use

Organizations:

Local school district, regional prevention center, local and state health department, law enforcement agency, treatment service providers 

Surveys:

Youth Risk Behavior Survey (YRBS) (Available from the Office of Adolescent Health, Centers for Disease Control and Prevention [CDC}, 4770 Buford Highway, NE, Atlanta, GA 30333.) 

Adolescent Pregnancy

Organizations:

Local health department, local school districts, family planning organizations

Surveys:

• Youth Risk Behavior Survey (YRBS)
• Adolescent Curriculum Evaluation (Available from Dr. Murray Vincent, School of Public Health, University of South Carolina, Columbia, SC 29208.) 

Youth Violence

Organizations:

Local school districts, Center for Disease Prevention 

Surveys:

Youth Risk Behavior Survey (YRBS)

Cardiovascular Disease

Organizations:

Local and state health departments 

Surveys:

Behavioral Risk Factor Survey (BRFS) (Available from state health department or U.S. Centers for Disease Control and Prevention [CDC].) 

Injury Control and Prevention

Organizations:

Local and state law enforcement agencies; state department of transportation 

Surveys:

Behavior Risk Factor Survey (BRFS) (Available from state health department or U.S. Centers for Disease Control and Prevention [CDC].) 

Tobacco Control

Organizations:

Local and state health departments

Surveys:

Behavior Risk Factor Survey (BRFS) (Available from state health department or U.S. Centers for Disease Control and Prevention [CDC].) 

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What do we mean by conducting interviews with key participants?

• Why should you interview key participants and identify critical events?

• When should you conduct interviews with key participants?

• How should you conduct interviews with key participants?

What do we mean by conducting interviews with key participants?

To better understand a subject, it is usually helpful to seek information from an expert in the field. The media often rely on experts to give facts, statistics, and opinions to support their news stories and articles. You can do the same. But while reporters must sometimes search far and wide to find people with the proper credentials, you only need look as far as the leadership and active members of your group. They are your experts.

Like other methods of evaluation, interviewing allows you to learn more about the effectiveness of your group's actions. With the information you gain from the interviewing process, you can continue to improve upon your work with the community.

For your purposes, interviews involve asking key participants to respond to a variety of questions about the initiative. Key participants can include both leaders of the group and members who play an especially active role in the group's work. You can use interviews to hear members' concerns and insights, and also to put a finger on critical events in the life cycle of the group. Additionally, interviews can allow participants to share their specific experiences with your group's work. That is, participants can explain what they learned, how they felt about the activities, what could be changed to improve the activities, and other, more personal thoughts. In this way, interviews can shed light on both the personal experiences of individuals and on the critical events of the group.

But, you might be asking yourself, what exactly are critical events? Well, we're glad you asked! Critical events are the important events that happen within your group, within your community, or within the world that have an effect on your group's mission and action plan.

Critical events might include:

• Applying for grant funds
• Brainstorming, writing, editing, and completing a draft of your action plan
• Fulfilling any or all of your group's objectives
• Facing a crisis in your community or within your group
• Changing the direction of your mission
• Any other major event

A great way to pinpoint these critical events is to interview those who organized and participated in them. You might still be wondering, however, why you need to identify the group's critical events. How will this be helpful?

Why should you interview key participants and analyze critical events?

The main purpose of interviewing is to gain a better understanding of the initiative. In an interview, you have the luxury of asking in-depth questions that can result in more detailed responses and remarks. You can also ask the respondent to clarify certain responses or to provide more details about his thoughts and ideas. Interviewing allows you the freedom and opportunity to thoroughly "pick the brain" of the key participant.

As we already mentioned, interviews also help you nail down the critical events in your group. But, why is this important?

Critical events provide qualitative information

In some ways, gathering quantitative information can be done very easily. Other sections in this chapter describe some of these methods in detail. By using monitoring system and event logs, for example, you can note exactly what activities happened, when they happened, and who they involved. These methods give you quantitative, or measurable, information about the events your group has completed.

Determining critical events through interviews, however, lets you understand more about the event than just when it happened and who helped. Gathering critical events in this manner supplies not only factual information, but also gives key participants the chance to tell you how they felt about the event. Qualitative, or quality-oriented, information can be obtained by asking value-oriented questions.

These might include:

• Was the event important?
• Was the event significant?
• Was the event fair?
• Was the event effective?
• Did the event mesh with local values?

By attaching these kinds of adjectives to questions you might ask, will gain more of a qualitative, or quality-oriented, perspective of the critical events in your group.

Quantitative information is important. But the qualitative information described here is, too. When both methods are combined--generally speaking, a good practice-- the accuracy of the overall conclusions is strengthened.

Critical events can provide a history of the initiative

As with the monitoring system, tracking critical events helps you trace the development of your group's work and achievements. Keeping track of the history of the initiative gives you yet another way to see progress being made and hurdles being crossed.

Critical events help identify factors that affected the initiative's success or failure

By identifying the critical events for your group, you can then see which specific actions may have helped you fulfill your objectives and which actions may have hindered your progress. Then, you can try to repeat those actions that helped you achieve your goals, and learn from those actions that were not as effective.

Critical events can provide you with insights for planning and renewal efforts

You have examined the information you have gained through pinpointing critical events. Now you can begin to use what you have learned to continue making plans for the future and to reenergize your present efforts. Knowing you have fulfilled many of your objectives can only help you stay focused on continuing your hard work. Use what you've learned to help your group and the community improve!

When should you conduct interviews with key participants?

We recommend that you conduct interviews with key participants at least every few years, or perhaps at the end of a grant period. If you have both the time and the resources, conducting interviews more frequently could be extremely beneficial.

How should you conduct interviews with key participants?

In preparation for your role as the next Dan Rather or Oprah, you can practice your interviewing skills with key participants. Not only will you sharpen your interviewing instincts, you'll also help with the evaluation process.

There are four main steps to conducting key participant interviews.

Determining who will be interviewed

One of the best ways to go about finding participants for your interviews is to ask members of the group who they feel has made notable contributions of time and energy to the group.

During a meeting of the coalition, you might ask members to reflect upon who they want interviewed, then write three of these names on a piece of paper. The evaluation team can then make note of the names of those who were most often mentioned. After narrowing it down to four or five individuals, let the interviews begin!

Conducting the interviews

Now you're ready to "rock and roll!" But first, you need questions. There are some key questions that should be asked in order to determine the critical events and gain some personal insight into the initiative.

These questions might include:

• What are the critical events in the history of the initiative? (You might need to explain what you mean by a "critical event.")
• Why was the event important?
• What was the context for the event?
• What key actions, actors, and other resources were required?
• What barriers and resistance were encountered?
• What were the consequences of the event for the initiative and the community?
• What are some overall lessons for the initiative?
• What future directions should be taken by the initiative?

Sharing these questions with the participant before the interview will give her time to reflect on her answers. That way, no one will be too surprised on the day of the interview. Additionally, the participant's answers are likely to be clear, thoughtful, and better focused.

Once you have decided on questions, you are ready to conduct the interviews. Interviewers, who might be members of the evaluation team or other members of the initiative, will conduct face-to-face or phone interviews with the key participants. In general, meeting with the person face-to-face works best. Those conducting the interview might want to bring along a recorder to document the conversation. Or, he or she can simply jot down notes while the person talks. Again, using an audio recorder will ensure that you get the most accurate responses. Plus, the interviewer will have the freedom to pay attention to the participant instead of having to focus on taking good notes.

Remember, the more comfortable the interviewed person feels, the more likely he or she will provide honest, heartfelt responses. You can help create this comfortable environment by being exceptionally friendly, supportive, and open to hearing the other person's ideas and comments.

Putting the data together

After the final words have been spoken and the last notes written down, members of the evaluation team should generally make a report of the interviews. This report should include summaries of the comments made by the participants who reported on a particular event. Also, the report might include a list of the critical events as an easy way to review the initiative's history.

Using the data to better understand the initiative, to provide a history, and to aid in renewal efforts

As you've probably inferred, the data from these interviews should be used to continually improve and strengthen your group's work. As you will have data that gives you a better sense of how people really feel about the initiative, you can use this knowledge to make still more improvements to your action plan and objectives.

In Summary

Interviewing key participants provides yet another valuable way for your initiative to measure its progress as you work towards change in the community. Interviews offer more qualitative, or quality-oriented, measurements because they ask individuals about their own personal experiences while seeking suggestions concerning the work of the initiative. As one method of evaluation, interviews can provide valuable insights into the performance of the initiative. Using this method can help your group improve its work and the work that is done with the community.

You understand that critical events might include:

___Applying for grant funds

___Brainstorming, writing, editing, and completing a draft of your action plan

___Fulfilling any or all of your group's objectives

___Facing a crisis in your community or within your group

___Changing the direction of your mission

___Any other major event

You understand why it is important to interview key participants regarding key events:

___Critical events provide qualitative information

___Critical events can provide a history of the initiative

___Critical events help identify factors that affected the initiative's success or failure

___Critical events can provide you with insights for planning and renewal efforts

___You understand the importance of conducting interviews with key participants at least every few years, or at the end of a grant period

You know how to conduct interviews with key participants:

___Determine who will be interviewed

___Conduct the interviews

___Put the information together (perhaps in the form of a report)

___Use the data to better understand the initiative, provide a history, and aid in renewal efforts

Example: Sample Interview about critical event

Here are some examples of responses from an interview in which the participant discussed the critical event of action planning with her group.

Rationale

The action plan kept us on a time line. Things were no longer pie in the sky. We put up time lines that gave us something to go against. The action plans are very much a working document.

Context / Conditions

The coalition had spent a year talking in generalities. We were ready to get down to action. We were holding press conferences that made a commitment to the public to do something. We took all the change objectives and laid out our action plan by task forces. Each task force met to further develop the action plan. Most of us were not used to being in a true collaboration. The diversity of the group required that we figure out how people fit. If we didn't define roles, we would lose people.

Key actions and actors

Judy Johnston, Paula Marmet, Steve Coen, Mary Clark, Tom Ryan, Maggie Chamberlain, and David Banks were key leaders. They forced us to decide what we were going to do. The monthly brainstorming was very helpful. The survey of importance and feasibility was very critical. The survey gave all the task force members and partners a chance to get involved.

Barriers and Resistance

Since people had experienced other types of strategic planning, some eyes rolled where we started. Some thought it would not produce anything useful. Eventually people believed that this planning process would be different.

Key resources

The task forces still had control over what would be pursued. Steve Coen was very helpful in providing technical support.

Consequences for the coalition

The strategic plan made the coalition members feel that we were finally underway. It helped with public perception. We now had substance. This is what we are and what we do. It brought the coalition together on its priorities. For example, the emphasis on early nutrition made it easier for people to see that children were a priority for the coalition. It also made it much easier to talk with potential funders.

Consequences for the community

Strategic planning got programs in place. Without it, we might not have begun to work on the childcare curriculum. The action plan put programs out there that weren't there before. Action planning also helped us to pull in new partners. As the plan grew, additional partners become obvious. Partners could see how they could be involved with the coalition and the mission.

Tool: Template for conducting interview about critical events

This form can be used to gather important information when interviewing group members.

Background Information

Initiative:

Key Participant:

Participant's position / involvement with the initiative

Date and Time:

Location:

Interviewer:

Critical Events

List all critical events, and then for each, ask about the following aspects:

Rationale: Why was this event particularly important?

Context / Conditions: What was going on at the time of the event? What made the conditions right for this to happen?

Key actions and actors: What key actions brought about the critical event? Who were the key actors?

Barriers and resistance: Were the group's actions met with barriers or resistance? What types of barriers? Who resisted?

Key resources: What key resources (e.g., people, financial resources, political influence) were used to bring about the critical event? How were these resources used to overcome barriers and resistance?

Consequences for the initiative: What were the consequences or results of the critical event for the initiative?

Consequences for the community: What were the consequences for the community?

Overall lessons: Overall, what lessons have you learned from your involvement with the initiative? What lessons have you learned from the initiative's attempts to define and act on its mission?

Future directions: What issues does the initiative face in the future? What challenges should be addressed?

This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.

• What are community-level indicators?

• Why should you use community-level indicators?

• When should we evaluate community-level indicators?

• How do we find community-level indicators?

What are community-level indicators?

For community initiatives, community-level indicators can be seen as one ultimate outcome of any initiative. While the monitoring system, constituent surveys, goal attainment reports, and interviews can help pinpoint the success of individual events, community-level indicators try to determine how these individual events affect the community as a whole.

One way to discover the big picture effect is to review proven "indicators" recommended by experts in your group's field of interest. For a group focusing on substance use, for just one example, one indicator might be the number of emergency medical transports related to alcohol use. By examining these numbers, your group will have a better sense of the ultimate effect of your group's work on the community at large.

Why should you use community-level indicators?

While the other evaluation tools can offer you valuable insights into the effectiveness of specific actions and objectives of the initiative, community-level indicators provide the major "big picture" perspective. But the benefits don't stop there!

Community-level indicators provide "bottom-line" evidence of the impact of the initiative.

"Bottom-line" evidence describes the ultimate effect of your group's work on the initiative. Community-level indicators give you an objective way to measure and assess the bottom-line evidence. Examples might include single-nighttime vehicle crashes (as a way to track efforts to reduce substance use) or the number of hospital admissions for violence-related injuries as a way to account for efforts to reduce youth violence). In addition, community-level indicators help you measure progress towards the goal. Measurements are ideally taken at multiple stops along the road, so that you (and the community) can see how you are doing and adjust your course, if need be.

Community-level indicators help determine the effects of key components of the initiative.

If you decided to implement several bold ideas in your action plan, community-level indicators can help you determine the effectiveness of these ideas, and of specific objectives. That way, you can see which ideas and strategies helped you achieve some of your goals, and which might need some reworking in a future action plan.

Community-level indicators help push issues to the forefront of the public agenda.

If the community-level indicators reveal negative results (i.e. the number of single nighttime-vehicle crashes stays the same even after you have implemented several of your objectives), this can be the perfect time to remind the community that the issue still exists. It still needs to be addressed through continued action. Although negative results may not be the results you want to see, try to envision them as a golden opportunity to push your issues to the forefront of the public agenda.

Community-level indicators that show positive results can help secure more support for the initiative.

But, if your community-level indicators suggest that your work has been resulting in many positive changes for the community, you can use these glowing reports to gain more support for your initiative and the good work that you do. This might include financial support, which can be used to improve and extend the work of the initiative. So get out there and tell it to the world; your group deserves recognition and support for all of its hard work and contributions!

When should we evaluate community-level indicators?

The sooner you start to evaluate your community-level indicators, the sooner you'll get a better understanding of how your community is affected by different variants. This way, you can plan in advance so that the effects are the ones that you desire.

It's advisable to start measuring community-level indicator as soon as you start a new project or event, so that you can change things around as necessary. To get a broader picture is always valuable and the sooner the better.

Of course, after you conclude your event or project, run a detailed community-level indicator survey so that you can measure the results and effects of your activity, and learn more for the future.

How do we find community-level indicators?

There are five main steps to evaluating community-level indicators.

Select indicators that are available, accurate, possible to collect, and sensitive to the initiative

The key to success here starts with knowing the right questions to ask. First, your group will want to hold several meetings with staff and also with groups of community members to identify which community-level indicators will best serve your needs.

Let's say, for example, that your group wants to focus on reducing the rate of teenagers who smoke marijuana in your community. One of the community-level indicators includes the number of juvenile arrests for drug offenses.

Contact relevant local and state agencies to find data

Once you have decided upon the appropriate community-level indicators, members of the staff can pool their knowledge of local, regional, state, and federal sources who might be good resources for information. These sources might include health departments, law enforcement agencies, or transportation offices. Your exact choices will be determined by what you want to study.

Let's continue on with the above example. If you know that the community- level indicator you want to measure is the number of juvenile arrests for drug offenses, you could contact the local police department. Then, gather information about the number of arrests for drug offenses for community members between the ages of 14 -20 over the last several years.

Compile, summarize, and graph the data

Members of the evaluation team put together all of the crucial information in a report, and also in graph, chart or table form.

How do we graph the data from the number of drug arrests? One way would be to graph the numbers of arrested people at six-month intervals. Then, you can compare your group's presence in the community with the changes in the numbers. If, for example, you began implementing your action steps one year ago, you would want to compare the levels of drug arrests before your work with the levels during and after your presence.

Present the data to community leadership, trustees, and funders

Data obtained from the community-level indicators can then be used to inform and educate your organization and the community. As a tool to encourage public support, community-level indicators can highlight the positive results of your group. Feel free to spread the word far and wide about the results you have obtained. Post your information everywhere; let the community know what is going on in their world.

Use the data to elevate the issue on the public agenda, and to redirect the initiative's efforts

If the community-level indicators yield negative results, this can provide a chance to push the issue to the top of the public agenda. Additionally, data, regardless of the results, can give the group a chance to redirect its work in a more effective direction.

If, for example, your research shows that levels of drug arrests among juveniles continues to hold steady, this would be an excellent opportunity to remind the community that drug abuse exists and continues to be a significant issue in the community. In this case, you might want to distribute your results to local schools, local community leaders, and other groups or organizations that focus on substance use.

Finally, remember that results can be viewed in a variety of ways. Positive results might mean that your group needs to redirect its energy towards other areas if you have achieved your original goals. Negative results should not be viewed as failure. Instead, these results should simply mean that another strategy might work better. Then, rather than continuing to focus your energy in the wrong direction, you can try to find a more effective way to work towards change.

In Summary

You deserve a round of applause! You have successfully worked your way through all of the evaluation steps! Good work! Keep reading to find some helpful examples of community-level indicators. Then, prepare to enter the wonderful world of feedback!

___You know what community-level indicators are

___You understand why you should use community-level indicators

___You know that they provide bottom-line evidence of the impact of your initiative

___You know that they help determine the effects of key components

___You know that they help push issues to the forefront of the public agenda

___You know that they can help secure more support

___You know when you should evaluate community-level indicators

___You know how to find community-level indicators

___You have selected indicators that are available

___You have contacted relevant agencies to find data

___You have compiled, summarized and graphed data

___You have presented the data to the community

___You have used the data to bring attention to your issue

All examples are from Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J.A., Richter, K.P., Lewis, R.K., Williams, E.L., Harris, K.J., Berkley, J.Y., Fisher, J.L., & Lopez, C.M. (1994). Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development. Lawrence, KS: Work Group on Health Promotion and Community Development, University of Kansas.

Example 1: Recommended Community-Level Indicators for Substance Use Coalitions

• Number of single-car nighttime vehicle accidents
• Number of drug positives from urine samples of arrested people
• Number of arrests for drug possession
• Cost and purity of street drugs
• Number of drug positives from urine samples of pregnant women at the time of delivery
• Number of alcohol or drug (AOD)-related emergency room episodes
• Number of AOD-related deaths
• Number of individuals on waiting lists for and admissions to in-patient and out-patient AOD program service
• Number of referrals and admissions to mental health centers for AOD problems
• Incidence of AOD-related birth outcomes (e.g., fetal alcohol syndrome, positive drug toxicology)
• Incidence of drug-related sexually transmitted diseases (STDs), including HIV transmission in AIDS cases
• Incidence of AOD-related medical conditions (e.g., cirrhosis of the liver, hepatitis)
• Number of drug positives from urine samples of job applicants and employees
• Aggregate per capita consumption of alcohol, based on alcohol tax revenue data
• Student alcohol and drug abuse
• Juvenile arrests for drug offenses
• Adult arrests for drug offenses
• Marijuana plants seized
• Positives in pre-employment drug testing
• Employees with employee assistance programs
• People treated for substance use problems
• Teen alcohol-involved traffic deaths
• Adult alcohol-involved traffic deaths
• Drug overdose deaths

Example 2: Recommended community-level indicators for adolescent pregnancy prevention initiatives

The estimated pregnancy rate for females aged 15-19 is the most commonly used indicator for adolescent pregnancy. Data may also be available for females ages 10-19 and 10-14. Currently, the majority of the pregnancies occur within the 15 to 19 year old age group.

The formula for calculating the estimated pregnancy rate is as follows: (live births + fetal deaths + reported abortion) / population of females age 15-19 x 1000.) This formula can also be used to calculate the estimated pregnancy rate for females of different ages. Epidemiological data such as these rely on the accuracy of both the numerator and denominator. Limitations of each are described below.

Abortion data include only those reported by hospitals and clinics participating in state health department report systems. Because of their sensitivity, abortion data tend to be underreported. For example, in a typical county in 1993, 6.6% of adolescent pregnancies reportedly resulted in an abortion. Yet, nationally, it is estimated that 36% of adolescent pregnancies result in abortions.

Population estimates for the age group may come from different sources. Estimates are available from the Census Bureau and as well as from state offices such as health departments and university research institutes. Population estimates may come from different sources over a given time period. For example, population estimates for years 1 through 5 may have been provide by the state Census Bureau and for years 6 through 10 by a local research institute.

It is important to know the data that are needed to compute the estimated pregnancy rate for a given age group and their limitations. Because of the difficulty in securing data on abortions and population size with absolute certainty, the estimated pregnancy rate for adolescents age 15 through 19 remains an "estimate." Issues around reporting by hospitals in a given area and the accuracy of recording place of residence of the mother may also come into play when exploring the accuracy of estimated pregnancy rate as an indicator. Changes in data collection that may have occurred over the period of the prevention initiative, and a few years prior, should also be explored with the state health department's statistician. This way, you can give the initiative a sense of whether changes in the estimated pregnancy rate could be due to a change in reporting rather than as a result of the initiative.

Example 3: Recommended Community-Level Indicators for Tobacco Control Initiatives

Per capita consumption of tobacco products represents the most objective data available on population levels of tobacco consumption. Data are available from the state health department (or department of revenue) based on the excise taxes that are imposed on tobacco products. Excise taxes are collected at the level of tobacco distributors. This information is available on a monthly basis.

Variability from month to month may be an artifact of this measure. It may reflect patterns of stocking at the retail level. When systematic seasonal variations are adjusted for, however, collective sales data provide the best available estimate of total tobacco use.

The main limitation of these data is that they do not provide information on the behavior of consumers. A change in consumption rate is a composite of many individuals ? uptake and quitting behavior. For example, a drop in cigarette sales may be the result of the same people smoking fewer cigarettes or fewer people smoking. Nor do consumption data indicate what changes in special populations, such as young women or ethnic minorities, might be reflected in the data. Despite these limitations, tobacco consumption remains the best community-level indicator.

Example 4: Recommended Community-Level Indicators for Injury Prevention and Control Initiatives

• Deaths due to unintentional injuries
• Hospitalizations due to unintentional injury
• Deaths caused by motor vehicle crashes
• Deaths from falls and fall-related injuries
• Drowning deaths
• Hip fractures among older adults
• Emergency room admission for non-fatal poisonings
• Hospitalizations due to non-fatal head injuries
• Hospitalizations due to non-fatal spinal injuries
• Secondary injuries associated with injuries to the head and spinal cord
• Deaths due to farm-related injuries
• Hospitalizations due to farm-related injuries
• Local sources of other measures may include coroner and police reports, data from walk-in clinics and emergency rooms, child care centers, schools, nursing homes, social service agencies, senior services, large businesses, and insurance companies

Example 5: Recommended Community-Level Indicators for Violence Prevention Initiatives

• Uniform Crime Reports, Federal Bureau of Investigation / Department of Justice : Published annually; includes crimes such as rape, assault, homicide, and robbery
• National Crime Survey and National Victimization Survey: Administered annually ; includes questions about spouse, child and elder abuse
• National Center for Health Statistics Mortality Data and Centers for Disease Control Mortality Data (Center for Disease Control and Prevention, Atlanta, GA), which includes data collected based on the ninth edition of the international Classification of Diseases; summarizes the deaths in the United States by cause and demographics
• State Crime Data (for example, from the Kansas Bureau of Investigation): Published annually; includes the same information given to the FBI for their Uniform Crime Reports, but is broken down by county and metropolitan areas
• State Incident Based Reporting System (for example, from the Kansas Bureau of Investigation): Unpublished data; includes information provided by police agencies through "Standard Offense Reports" and "Standard Arrest Reports"
• State Juvenile Justice Information System (for example, from the Kansas Bureau of Investigation): Unpublished data; includes the same information provided by the Incident Based Reporting System as well as information from Child in Need of Care or Child in Need of Services reporting
• Local sources of data may include coroner reports and local police reports, information collected by the school district on the incidence and prevalence of assaults, and information collected by local women's shelters and social service agencies about spouse abuse

Example 6: Sustainable Seattle's Fourth Set of Regional Sustainability Indicators

After the release of its 1998 Indicators, Sustainable Seattle decided to reassess its indicator program.  The board was concerned that merely publishing the indicators, as Sustainable Seattle did in 1993, 1995, and 1998, was not sufficient and that a successful program should include programs which support actions by citizens, businesses and policy makers to affect the trends documented by the indicators. It has taken several years for the organization to build the professional staff capacity to support an expanded indicator program.

The Payneside Health-for-All Coalition wanted to better understand the effectiveness of its effort to make health services more accessible to everyone in the community. The Coalition selected some community-level indicators to help them understand if progress was being made. They got information about the percent of adults seen by a doctor in the past year for a routine checkup, the percent who did not have a medical home, number of community health clinic visits for people of various ages, and how many visits to the sliding-scale clinics were of patients who could afford to pay little or nothing. They also tracked the number of community residents who received free care at local hospitals, the number of seniors who received subsidized home care, and other statistics. As a result, Health-for-All learned that a larger number of citizens were, in fact, seeking regular health care, and that the Coalition’s work had borne fruit. Using community-level indicators had helped the Coalition provide community members with better access to services.

In the previous section, Gathering and Using Community-Level Indicators, we introduced community-level indicators, and briefly discussed why and when they can be used and how to gather them. The section included a number of examples of possible community-level indicators for different issues: substance use, teen pregnancy prevention, tobacco control, injury prevention, and violence prevention. In this section, we’ll explain further how to identify and choose community-level indicators to fit your needs, and give some guidance as to what kinds of indicators you might look for in a given field.

What are community-level indicators?

Community-level indicators are measures that refer to population groups rather than individuals. (They indicate what’s happening at the community level, rather than the individual level.) Community-level indicators offer objective measures of outcomes. An example would be monitoring smoking through the community-level indicator of sales of cigarettes in the community, rather than by polling people to find out how many cigarettes each person smoked daily.

There are many places to find information on community-level indicators. For health-related indicators, some excellent places to start are Health Indicators Warehouse (HIW), which provides a wide variety of health indicators with national, state, and community data; and County Health Rankings, which provides rankings of each county within each state according to its health outcomes and health determinants. And for demographic indicators, The U.S. Census Bureau is always a good place to start. There are many other helpful websites that can guide your choices of community-level indicators, listed in the Internet Resources section below.

Community-level indicators can range from the very specific and focused – the rate of nighttime, single-vehicle crashes as an indicator of alcohol and substance use – to the more subtle and indirect – the amount of shelf space devoted to alcoholic beverages in local supermarkets. All the community-level indicators you use, however, should have some things in common.

They should be:

• Relevant
• Available
• Usable in practice
• Statistically measurable
• Logically or scientifically defensible - You must be able to convince people that the link between your indicators and the issue they concern is real.
• Reliable
• Leading - “Leading indicators” tell you what’s coming. If they change, it usually means that the rest of that category will make the same changes soon. If you can identify leading community-level indicators, you can use them to predict trends relating to your issue.
• Policy-relevant

Some general considerations

• As we’ve already mentioned, good community-level indicators can take several forms. Sometimes, what seem to be the most obvious aren’t as useful as they seem. For instance, smoking is a major cause of lung cancer, so measuring lung cancer rates in a community ought to tell us about smoking rates. It does, but it tells us about smoking rates 20 or 30 years ago, because that’s how long it may take many cancers to develop. The same is true for heart attacks and cholesterol – the current condition of your heart reflects your eating and exercise habits of 20 years ago as much as your current ones. It’s important, therefore, to pick indicators that tell you about current conditions (unless what you want to know is how many people are currently affected, or what happened 20 years ago).
• A second consideration is that unusual connections may give you important information. An increase in the shelf space that supermarkets devote to healthy foods says something about people’s concerns about diet, for instance, and may reflect the adoption of healthier lifestyles.

How might you use community-level indicators?

Community assessment. Community-level indicators can be useful in community assessments for different purposes. An assessment to identify community issues and problems, for instance, might rely on such indicators as the incidence of a disease or medical condition either in the community at large, or in a particular social, ethnic, or geographic group. The number of children receiving free or reduced-price school lunches, or the number of schools instituting breakfast programs can speak volumes about the community’s economic problems, or about child hunger. 

Community-level indicators can also provide information about the effects of community projects and initiatives. A reduction in the number of single vehicle nighttime crashes might be evidence that a recent substance use prevention program is working; a reduction in the number of people who are unable to obtain or have to delay medical care due to costs might be evidence that the new clinic in the area is helping a previously underserved population.

Accountability. Community-level indicators, by providing evidence of progress, can help maintain accountability for those who are running initiatives and projects. A community initiative to provide more affordable housing might be measured, for instance, by the number of affordable housing units available, or the average per-room or per-square-foot rental cost for residential property in the community. Levels of economic prosperity can be measured by looking at the community’s unemployment rate, average wage for workers, etc.

Evaluation. You’re most likely to use community-level indicators to measure your progress toward the outcomes you’re aiming at. A substance-abuse initiative, for example, might look at admissions to treatment programs for signs of progress. Outcomes might be measured by such indicators as drug-and-alcohol-related arrests, emergency-room admissions related to substance use, underage drunk-driving incidents, or single vehicle, nighttime crashes.

Policy change. Community-level indicators can help determine where policy change is needed, and whether a change in policy is having the desired effect. Many states and communities, for example, have reformed property tax laws to help seniors and lower-income people stay in the homes they have owned for many years. These reforms came about because of (community-level) indications that these groups were being forced out of their homes by high taxes. Checking the records of home sales in the community two years before and two years after such reforms might tell you whether fewer seniors and lower-income residents are selling their homes, or at least whether the neighborhoods formerly most affected are becoming more stable.

How do you choose community-level indicators?

Decide on just what your goals are in using community-level indicators. What exactly is it you want to find out? For instance, if your initiative is working to increase health care access for those in your community, your goals might be to determine the number of uninsured people who have to delay getting medical care because of high costs, and then as your initiative progresses, determine whether your initiative has reduced this number.

Determine what kinds of information you’d need to achieve your goals. You’d want to assess the nature of the problem as it currently exists, which means looking at issues such as how many people in the community are uninsured, the percentage of uninsured people who are receiving annual checkups and recommended cancer screenings, etc.  The same issues could be examined later to find out how successful you were, and what else might need to be done.

Choose community-level indicators that reflect the information you want, and that can be measured, so that they can be compared, and give you real information to work with. Some community-level indicators you might choose in this case are the number people who have a primary care physician or stable source of ongoing care, the number of families using the emergency room as their primary care provider (as reported by emergency room intake data), and the number of people who have to delay receiving treatment or obtaining prescriptions because of the costs. All of these might be compared to measurements from previous years, and then analyzed again in the future to determine if there has been progress.

Measure the indicators and analyze the results. You might find that there are high numbers of uninsured people in the community, and they are not receiving the recommended annual checkups and cancer screenings. Then your efforts should focus on disseminating information about the importance of such screenings and information about the availability and affordability of those services.

By continuing to use the indicators you’ve chosen, and adding others that make sense, you can evaluate your efforts, and make changes where needed.

Examples of community-level indicators

Here we’ll examine a number of areas – health, human services and education, community development, public safety, the environment – and point out some general types of community-level indicators that could be used as starting points. You might use or adapt some of these indicators for your own work. Additionally, if you have community-level indicators that are not listed here and you would like to share them with us, please email us at toolbox@ku.edu.

Many of these indicators are most useful when compared to past figures or figures from other similar communities, when monitored over time, or when looked at as a percentage or proportion of the population. These figures are often stated as “per 100,000 (or 1,000) people”. Thus, if there are 20 cases of lung cancer in a community of 10,000, the number might be stated as “a rate of 200 cases per 100,000.” In that way, the figures can be reasonably compared with those of communities of any size.

Health

Health covers a range of areas – wellness promotion and health maintenance, disease and injury prevention, detecting and addressing medical issues unique to the community, and providing health services to all who need them.

Wellness promotion and health maintenance. Research shows that healthy eating, regular exercise, and stress reduction contribute to a longer and better-quality life. In addition, most people benefit from regular health maintenance and monitoring – visits to health professionals, care about exposure to dangerous conditions or substances, etc. Some community-level indicators include:

• Percentage of adults people with a Primary Care Provider
• Percentage of people who have had a Primary Care Provider checkup in the past 12 months
• Percentage of adults who receive regular dental checkups
• Percentage of adults receiving annual vision screenings
• Percentage of children receiving regular medical checkups
• Percentage of children who have regular dental visits
• Vision screening for children (percent)
• Average number of prenatal doctor visits for pregnant women
• Shelf space devoted to healthy and/or organic products and foods in supermarkets
• Shelf space in stores devoted to tobacco
• Number of liquor stores in the community
• The mileage of bike paths and hiking/running trails, compared to road miles in the community

Disease and injury prevention. Unlike most wellness and health maintenance activities, disease and injury prevention involves taking specific measures to prevent or minimize the effects of specific diseases and conditions. Some community level indicators include:

• The percentage of the over-65 population inoculated for flu
• The percentage of children registering for school with proof of all recommended inoculations
• The number of people receiving regular cancer screenings – mammograms, Pap smears, prostate antigen tests, colonoscopies, etc.
• Condom sales, especially among teens. These might be related to the incidence of sexually-transmitted infections (STI’s), including HIV, and might also relate to teen pregnancy rates
• The percentage of schools providing sex education
• The number of homeless people tested for TB and other infectious diseases
• Number of single vehicle, nighttime crashes

Detecting and addressing health issues unique to the community. Any community may have unique health needs, stemming from the prevalence of a particular disease or condition, or from a threat to the community’s health. The causes here may range from the environmental – pollution of drinking water, poor air quality, chemicals in the soil, etc. – to the genetic – a large population of an ethnic group that’s susceptible to a certain disease – to the cultural – a population at risk of heart attack because of a traditional diet high in saturated fat. Examining some community-level indicators can help you find and deal with these kinds of conditions.

• Results of screenings for blood pressure and cholesterol, HIV/AIDS, TB, various cancers, neurological conditions, diabetes, and other chronic conditions, both for the community as a whole and for specific populations
• The number of hospital admissions and/or deaths resulting from particular diseases or conditions – certain cancers, diabetes, HIV-related illnesses, etc
• The number of miscarriages, stillbirths, and birth defects, both for the community as a whole and for specific populations
• Infant mortality rate, low-birthweight babies, babies with fetal alcohol syndrome, etc.
• The incidence of obesity in children
• The frequency of emergency room visits for asthma attacks, particularly among children
• The average number of cavities per childhood dental visit
• Number of children’s visits to mental health facilities

Providing health services to all who need them. A community may have excellent health services, but if many people are unable to take advantage of them – because of cost, difficulty of access, or some other reason – then they do those people no good. Community-based indicators can tell you whether there’s a large segment of your community that’s denied proper health care.

• Number of practicing primary care providers
• Patient to primary care provider ratio
• Percentage of people who have a usual primary care provider
• Proportion of insured persons with coverage for clinical preventive services
• Proportion of persons who receive appropriate evidence-based clinical preventive services
• Percentage of uninsured families in the community. The number of free or sliding-fee-scale clinics, and their rate of use
• Percent of dental care providers who accept Medicaid
• The number of families using the emergency room as their primary care provider (as reported by emergency room intake data)
• Proportion of individuals who are unable to obtain or delay obtaining necessary medical care, dental care, or prescription medicines
• Proportion of hospital emergency department visits in which the wait time to see an emergency department clinician exceeds the recommended timeframe
• The number of elders receiving in-home health and care services
• Average waiting time to obtain non-emergency medical services – for regular checkups, follow-ups, chronic conditions, etc.
• Handicapped accessibility of health facilities – hospitals, clinics, medical buildings and practices, dentists’ offices, optometrists, etc.
• Availability of ambulance and emergency medical services, particularly in rural areas
• Availability of interpreters in various languages at health facilities

Human services and education

Human services includes things such as welfare; homeless and emergency aid services; job training and retraining; youth development; community-based mediation; counseling and other mental health services; violence prevention; services for seniors, and those with physical or developmental disabilities; providing support to new immigrants and language minorities; and safeguarding citizens’ civil and legal rights.

Education encompasses not only public education (including special education for those with learning disabilities, or with physical, developmental, or psychological handicaps), and state-sponsored and -funded post-secondary systems, but also adult literacy and adult education (including language classes for immigrants), pre-school, after-school programs, alternative schools, and, to an increasing extent, distance learning (Internet, TV, or video-based instruction).

Human services aimed at children and youth. There are several areas to focus on: kids’ economic conditions, their school achievement, employment, family status, use of services, etc. Some types of indicators to consider:

• Number of families that are below the poverty line; within 150% of poverty; 200% of poverty; number of children eligible for free school lunches, subsidized child care, and other programs dependent on income

The U.S. federal poverty guidelines are updated annually. They are meant to define poverty, but don’t really represent the level at which a family is able to get by on its income alone. Although it sets separate guidelines for Alaska and Hawaii, which are known, because of their isolation, to be more expensive, the government doesn’t make allowances for people living in an expensive urban area, or for the need for transportation in rural areas, for instance. The guidelines are usually low, and a great number of families that must struggle to pay for both food and shelter are far above them. As a result, eligibility for many federal and state services is set at 150% or 200% of poverty.

• Family status: percentages of children living with two parents, one parent, other family members, in state custody (foster homes and institutions)
• Number of children under 18 involved with the juvenile court system: complaints, arrests, involvement in violent crimes, delinquency petitions, court referrals because of uncontrollable behavior.

This last goes by different names in different states, but boils down to parents or guardians referring children – usually young teens – to the court system because the kids are beyond their parents’ control, and are doing things likely to harm themselves or others.

• Family status: percentages of children living with two parents, one parent, other family members, in state custody (foster homes and institutions)
• Number of children under 18 involved with the juvenile court system: complaints, arrests, involvement in violent crimes, delinquency petitions, court referrals because of uncontrollable behavior
• Number of children under 18 in residential or lock-up programs
• Frequency of child abuse and neglect – physical and sexual assault, abandonment, negligent parenting, etc. – (available from emergency room reports, police files, state child welfare departments, and relevant human service agencies)
• Number of families with children seen by various agencies – emergency services, food bank, free health clinics, mental health services, rehabilitation, fuel assistance, child care, families-at-risk, pregnant and parenting teens, etc.
• Employment rate of youth 16 to 18
• Number of 16-and-under youth seeking family planning services
• Number of under-21 referrals to drug and alcohol treatment programs
• Availability of outreach and services for homeless youth, and numbers served

Emergency and similar services. Emergency services are essentially those services necessary to sustain health and/or life for people without the resources to continue to do so for themselves. Community-level indicators can measure both the extent of the need for these services and their effectiveness.

• The extent of homelessness in the community  (determined by admissions to shelters, winter counts of sleepers in the open, soup-kitchen meals served)
• Number of people on welfare or disability income
• Patronage of food banks and other food-distribution centers, surplus food eligibility, number of people receiving food stamps, number of people fed at the Salvation Army
• Number of families on fuel assistance
• Length of waiting lists for various types of subsidized housing, availability of affordable housing in the community, average percentage of income paid for shelter by people on public assistance
• Admissions to shelters and other services – counseling, placement in a new area, job training – for victims of domestic violence
• Number of people seen by emergency mental health services

“Second-level” human services. These are services that involve quality of life, rather than immediate physical survival or health, but, as a result, often address the underlying causes of problems, where emergency services, of necessity, may only address the symptoms.

• The percentage of people in the community who are unemployed
• The number of adults enrolled in or on a waiting list for adult literacy, basic education, or majority-language-learning programs
• The percentage of adults in the community lacking high school diplomas
• Number of child care slots available for low-income families
• Availability of public transportation
• Availability of independent or assisted living facilities or services, or other housing dedicated to seniors and people with disabilities
• Availability and use of community mediation – mediation covers divorce and family mediation, landlord-tenant disputes, small claims, conflicts among neighbors or organizations, etc. Community mediation is meant both to demonstrate better ways to resolve conflict, and to relieve the burden on an overworked court system.
• Number of complaints of discrimination reported and/or investigated, including discrimination in the workplace, housing, the granting of bank loans, etc.
• Availability of affordable (usually sliding fee-scale) community mental health services
• Availability and use of transition services – housing, employment, and support – for those recently released from prison
• Handicapped accessibility to various services and agencies

Education. From pre-school to college, as well as non-traditional education. There’s usually an emphasis here on publicly-funded or otherwise universally-available education, so that private schools and colleges are not part of the mix, except insofar as they offer scholarships or the equivalent. Many private schools in smaller towns, for instance, offer free tuition to local youth who can meet admission standards.

• Number of slots in affordable pre-school programs – Head Start, subsidized pre-school slots for low-income families, school district early education programs
• School status: dropout rate, number of children in grades 3-12 more than one grade level behind in reading or math
• Availability of bilingual and other programs aimed at helping immigrant children
• Class size, age and condition of public school buildings, teacher pay scale, amount spent per pupil, and other indications of community commitment to education
• Private schools and colleges offering scholarships or free tuition earmarked for local youth
• The percentage of children attending private or parochial schools
• Availability of alternative high schools and charter schools
• Number of local graduates attending college
• Community scholarships for post-secondary education available to residents
• Handicapped accessibility to educational facilities

Community development

Community development includes a number of areas, spanning the economic, the social/demographic, and the cultural.

Economic. You may be looking at the economic development of the community as a whole or that of specific groups within it. Community-level indicators to consider:

• The number of new businesses starting annually
• The number of business failures or bankruptcies each year
• The average income level
• The average education level
• The unemployment rate
• Local revenue from taxes and fees
• The percentage of businesses and industries that are locally owned
• The average commute to work for local residents
• The default rate on local business and mortgage loans
• Average housing prices
• The percentage of residents who own their own homes
• The percentage of people who are below the poverty line, homeless, or receiving public assistance
• The percentage of children receiving free school lunches

Social/demographic. The social indicators include how people relate to one another in the community, and what people do as groups to participate in the community and hold it together. Demographic indicators describe the community’s residents – their ethnic, gender, age, and racial diversity, their geographic distribution, etc. Social indicators are important in understanding how people are connected to one another, as well as how they sometimes discriminate against or distrust one another. Demographic indicators help to understand population shifts, changes in the nature of the community, and what services might be needed for particular groups.

• Number of service clubs, fraternal organizations, churches, and other gathering points
• Percent of citizens voting
• The number of single-parent families, and of children living with caregivers other than their parents
• The diversity of the population – number and size of various ethnic and racial groups, number of ethnic societies and organizations, number of kids under 18 – and population trends – how these numbers have changed in the past year, five years, ten years
• Where various groups live, if they aren’t dispersed evenly throughout the community, and whether different groups mix socially and otherwise
• The number of community activist and citizen advocacy groups and organizations (block associations, groups organized for a specific task, watchdog or oversight groups, policy change advocates, etc.)

Cultural. “Cultural” is used here to refer to both “culture” in the sense of arts, entertainment, and knowledge (which are, in fact, the passing on of the valued products of the majority culture), and “culture” in the sense of the customs and world view of the various ethnic, racial, and religious groups in the community. The availability of culture in both senses adds tremendously to citizens’ quality of life.

• Number of performance centers in the community – theaters, concert halls, clubs, etc.
• Number of low-cost or free performances (music, dance, theater, film, readings) available to the community
• Availability of affordable learning opportunities in the arts for children
• The number of free and/or affordable classes in the arts and other subjects for adults
• Number and character of museums in the community (and the number of visitors they receive)
• Availability of the public library – hours, number of branches, accessibility, etc. – and its use
• The number of after-school and summer recreation programs for children and youth
• Number of ethnic and religious festivals and other such multi-cultural events

Public safety

Since 2001, public safety, in many cities, at least, has come to mean more than police protection or earthquake-proof buildings. It also means safety from terrorism, which, in turn, raises issues of privacy and civil rights when “protection” becomes too vigorous. As with health, the most effective way to ensure public safety is to take preventive measures long before any threat exists.

In the examples of community-level indicators below, we offer measures of prevention, of the balance between safety and civil liberties, and of traditional public safety activities.

• The existence of emergency management plans, earthquake-, hurricane-, or tornado-proofing in buildings (in areas where that’s necessary), flood control measures, and other protections against natural disasters
• The number of fire stations and/or firefighters available to respond to fires and other emergencies, as well as the average response times in various neighborhoods
• The existence of neighborhood watch programs, good street lighting, police on foot patrol, etc.
• Relationship between the police and the minority and/or low-income community, as reported by members of that community, particularly youth
• The extent to which the ethnic/racial composition of the police force mirrors that of the community as a whole
• Equal police presence and enforcement of laws in all neighborhoods
• The existence of police/community relations boards or something similar
• Number of reports of police brutality, racial/ethnic profiling, or excessive force, and the number of those reports actually investigated
• Number of homicides annually in the community
• Number of violent crimes (rape, armed robbery, criminal or domestic assault, etc.) annually in the community
• Number of homicides and violent crimes in which victims and/or perpetrators are under 18
• Number of arrests of youth under 18
• Crime rate by neighborhood
• The number of police

The environment

The state of both the natural and the “built” (i.e., man-made) environment has a great deal to do with the quality of life in a community. Open space and wild places provide not only beauty, opportunities for exercise, and an escape from crowds and noise, but also sources of oxygen, fresh water, and wildlife habitat. Rundown buildings and streets full of trash can have a depressing effect on those who live there, and make life seem bleak and change impossible. Air, water, and soil pollution can threaten both wildlife and human health.

• The amount and condition of open space in the community (including public parks, conservation land and other protected areas, wildlife refuges, state forest, farmland, and protected or unprotected wild areas)
• Amount of protected land in the community
• Efforts to preserve and restore historic buildings, monuments, spaces, etc.
• The quality and adequacy of drinking water in the community (measurable pollutants, threats to or problems with the water source, size of the water supply and its ability to meet future needs, etc.)
• Air quality in the community
• Efforts by local government to reduce its effect on the environment (e.g., mandated use of low-emissions vehicles on government business, low-emissions or electric vehicles used for public transportation and garbage pickup)
• Availability and ease of recycling of paper, plastic, hazardous waste, and metal for both households and business/industry
• Level of regulation and enforcement of environmental standards for business and industry
• Local sponsorship of or support for public art (e.g., sculpture in public spaces, murals painted by teenagers in neighborhoods)

These are some examples of community-level indicators, but there are thousands more possibilities, depending upon your issue and your community.

In Summary

Community-level indicators – measures that show what the conditions are for the community or a large part of the community, as opposed to specific individuals – can be useful in evaluation, assessment, accountability, and policy change. You can find community-level indicators – either gleaned from available information, such as census data, or collected locally by observation and other methods – that will help you understand issues and trends for just about anything that affects the community.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: www.ctb.ku.edu

What are community-level indicators?

___Community-level indicators are measures taken from objective information, observation, or other methods that look at what actually happens

What can you use community-level indicators for?

___Evaluation

___Assessment

___Accountability

___Policy making

How do you choose community-level indicators?

___Decide on your goals

___Determine the information you need

___Choose measurable community-level indicators that reflect that information

___Measure indicators, and analyze the results

___Continue to use indicators, and add others as necessary, to monitor issues, evaluate success, and change approach, methods, and focus where appropriate, based on what the indicators tell you

Example 1: Using Community-Level Indicators for Anti-Displacement Work

All-In Cities has compiled a list of community-level indicators they use to gauge the impact of their anti-displacement efforts which can be found here.

They look for positive changes in these indicators:

• Increased number of low-income people and people of color who are in long-term affordable homes.
• Increase in business and home ownership by people of color.
• Increased political participation of people from communities impacted by displacement, including increasing public resources for impacted communities to build power and increase impact in decision-making and political spheres.
• Reduction in evictions and foreclosures.
• Reduction in people who are unhoused.
• Reduction in displacement of local businesses owned by people of color.
• Reduction in cultural displacement.
• Reduction in housing-cost burdens, particularly for women, people of color, Indigenous women, marginalized LGBTQ people, and women of color who are most burdened.