Table of Contents >
Chapter 38. Some Methods for Evaluating Comprehensive Communi... >
Section 9. Gathering and Using Community-Level Indicators >
Gathering and Using Community-Level Indicators |
Main Section | Contributed by Aimee Whitman Edited by Bill Berkowitz and Jerry Schultz |
This section is based on an article in the Work Group Evaluation Handbook: Evaluating and Supporting Community Initiatives for Health and Development by Stephen B. Fawcett, Adrienne Paine-Andrews, Vincent T. Francisco, Jerry Schultz, Kimber P. Richter, R.K. Lewis, E.L. Williams, K.J. Harris, Jannette Berkley, Jacqueline L. Fisher, and Christine M. Lopez of the Work Group on Health Promotion and Community Development, University of Kansas, Lawrence, Kansas.
What are community-level indicators?
Why should you use community-level indicators?
When should we evaluate community-level indicators?
How do we find community-level indicators?
What are community-level indicators?
Imagine for a few moments that you play basketball for a Division I college. Despite the presence of some solid players, your team finishes with an average record, and does not make any of the brackets come NCAA tournament time.
Next summer, the team practices long and hard on defense, offense, and free-throws. Then, during regular season play, everyone plays consistently well, and you match the numbers you had hoped to reach at the beginning of the season. And, come March, your team earns a spot in one of the brackets to compete in the Big Dance.
In this scenario, you might say that one of your team objectives when you began the next season was to start hitting 50% from the field and to shoot 75% from the charity stripe. The improvement of the team as a whole would be equivalent to a community-level indicator.
For community initiatives, community-level indicators can be seen as one ultimate outcome of any initiative. While the monitoring system, constituent surveys, goal attainment reports, and interviews can help pinpoint the success of individual events, community-level indicators try to determine how these individual events affect the community as a whole.
One way to discover the big picture effect is to review proven "indicators" recommended by experts in your group's field of interest. For a group focusing on substance abuse, for just one example, one indicator might be the number of emergency medical transports related to alcohol use. By examining these numbers, your group will have a better sense of the ultimate effect of your group's work on the community at large.
Why should you use community-level indicators?
While the other evaluation tools can offer you valuable insights into the effectiveness of specific actions and objectives of the initiative, community-level indicators provide the major "big picture" perspective. But the benefits don't stop there!
- Community-level indicators provide "bottom-line" evidence of the impact of the initiative.
"Bottom-line" evidence describes the ultimate effect of your group's work on the initiative. Community-level indicators give you an objective way to measure and assess the bottom-line evidence. Examples might include single-nighttime vehicle crashes (as a way to track efforts to reduce substance abuse) or the number of hospital admissions for violence-related injuries as a way to account for efforts to reduce youth violence). In addition, community-level indicators help you measure progress towards the goal. Measurements are ideally taken at multiple stops along the road, so that you (and the community) can see how you are doing and adjust your course, if need be.
- Community-level indicators help determine the effects of key components of the initiative.
If you decided to implement several bold ideas in your action plan, community-level indicators can help you determine the effectiveness of these ideas, and of specific objectives. That way, you can see which ideas and strategies helped you achieve some of your goals, and which might need some reworking in a future action plan.
- Community-level indicators help push issues to the forefront of the public agenda.
If the community-level indicators reveal negative results (i.e. the number of single nighttime-vehicle crashes stays the same even after you have implemented several of your objectives), this can be the perfect time to remind the community that the issue still exists. It still needs to be addressed through continued action. Although negative results may not be the results you want to see, try to envision them as a golden opportunity to push your issues to the forefront of the public agenda.
- Community-level indicators that show positive results can help secure more support for the initiative.
But, if your community-level indicators suggest that your work has been resulting in many positive changes for the community, you can use these glowing reports to gain more support for your initiative and the good work that you do. This might include financial support, which can be used to improve and extend the work of the initiative. So get out there and tell it to the world; your group deserves recognition and support for all of its hard work and contributions!
When should we evaluate community-level indicators?
The sooner you start to evaluate your community-level indicators, the sooner you'll get a better understanding of how your community is affected by different variants. This way, you can plan in advance so that the effects are the ones that you desire.
It's advisable to start measuring community-level indicator as soon as you start a new project or event, so that you can change things around as necessary. To get a broader picture is always valuable and the sooner the better.
Of course, after you conclude your event or project, run a detailed community-level indicator survey so that you can measure the results and effects of your activity, and learn more for the future.
How do we find community-level indicators?
It's time to dip your toes in the crystal blue waters of community-level indicators. Soon you'll be swimming with the big fish as you navigate the waves of availability, accuracy, feasibility and sensitivity. Let's dive in!
There are five main steps to evaluating community-level indicators. They involve:
1. Selecting indicators that meet the criteria of being:
- Available;
- Accurate;
- Possible to collect;
- Relevant to the initiative.
2. Securing data from relevant local and state agencies
3. Summarizing and graphing the data
4. Presenting the data to community leadership, trustees, and funders
5. Using the data to push the issue on the public agenda and to redirect the initiative's efforts
Now, let's get our feet wet!
1. Select indicators that are available, accurate, possible to collect, and sensitive to the initiative.
The key to success here starts with knowing the right questions to ask. First, your group will want to hold several meetings with staff and also with groups of community members to identify which community-level indicators will best serve your needs.
In the Examples section, you will find a variety of recommended community-level indicators for:
- Substance abuse coalitions
- Adolescent pregnancy prevention initiatives
- Tobacco control initiatives
- Injury control initiatives
- Violence prevention initiatives
Let's say, for example, that your group wants to focus on reducing the rate of teenagers who smoke marijuana in your community. One of the community-level indicators includes the number of juvenile arrests for drug offenses. (See Examples)
2. Contact relevant local and state agencies to find data.
Once you have decided upon the appropriate community-level indicators, members of the staff can pool their knowledge of local, regional, state, and federal sources who might be good resources for information. These sources might include health departments, law enforcement agencies, or transportation offices. Your exact choices will be determined by what you want to study.
Let's continue on with the above example. If you know that the community- level indicator you want to measure is the number of juvenile arrests for drug offenses, you could contact the local police department. Then, gather information about the number of arrests for drug offenses for community members between the ages of 14 -20 over the last several years.
3. Compile, summarize, and graph the data.
Members of the evaluation team put together all of the crucial information in a report, and also in graph, chart or table form.
How do we graph the data from the number of drug arrests? One way would be to graph the numbers of arrested people at six-month intervals. Then, you can compare your group's presence in the community with the changes in the numbers. If, for example, you began implementing your action steps one year ago, you would want to compare the levels of drug arrests before your work with the levels during and after your presence.
4. Present the data to community leadership, trustees, and funders.
Data obtained from the community-level indicators can then be used to inform and educate your organization and the community. As a tool to encourage public support, community-level indicators can highlight the positive results of your group. Feel free to spread the word far and wide about the results you have obtained. Post your information everywhere; let the community know what is going on in their world.
5. Use the data to elevate the issue on the public agenda, and to redirect the initiative's efforts.
If the community-level indicators yield negative results, this can provide a chance to push the issue to the top of the public agenda. Additionally, data, regardless of the results, can give the group a chance to redirect its work in a more effective direction.
If, for example, your research shows that levels of drug arrests among juveniles continues to hold steady, this would be an excellent opportunity to remind the community that drug abuse exists and continues to be a significant issue in the community. In this case, you might want to distribute your results to local schools, local community leaders, and other groups or organizations that focus on substance abuse.
Finally, remember that results can be viewed in a variety of ways. Positive results might mean that your group needs to redirect its energy towards other areas if you have achieved your original goals. Negative results should not be viewed as failure. Instead, these results should simply mean that another strategy might work better. Then, rather than continuing to focus your energy in the wrong direction, you can try to find a more effective way to work towards change.
In Summary
You deserve a round of applause! You have successfully worked your way through all of the evaluation steps! Good work! Keep reading to find some helpful examples of community-level indicators. Then, prepare to enter the wonderful world of Feedback! (Want to know more? Check out Chapter 39: Using Evaluation to Understand and Improve the Initiative!)
We encourage the reproduction of this material, but ask that you credit the
Community Tool Box: http://ctb.ku.edu
Resources
Berkowitz, William.(1982) Community impact: creating grassroots change in hard times. Cambridge, MA: Schenkman Publishing Company, Inc.
Cox, Fred and eds. (1984) Tactics and techniques of community practice. Itasca, IL: F.E. Peacock Publishers, Inc.
Fawcett, S.B., Paine-Andrews, A., Francisco, V.T., Schultz, J.A., Richter, K.P., Lewis, R.K., Williams, E.L., Harris, K.J., Berkley, J.Y., Fisher, J.L., & Lopez, C.M. (1994). Work group evaluation handbook: evaluating and supporting community initiatives for health and development, Lawrence, KS: Work Group on Health Promotion and Community Development, University of Kansas.
Fetterman, David and eds. (1996) Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: SAGE Publications, Inc.
Francisco, V. and Wolff, T. (1990). Evaluating coalition efforts. Amherst, MA: AHEC Community Partners.
Pietrzak, J., Ramler, M., Renner, T., Ford, L., and Gilbert, N. (1990) Practical program evaluation: examples from child abuse prevention. Newbury Park, CA: SAGE Publications, Inc.
Rutman, Leonard, ed. (1984) Evaluation research methods: a basic guide. Newbury Park, CA: SAGE Publications, Inc.
Internet Resources
Healthy Indicators Warehouse (HIW)
Provides a single, user-friendly, source for national, state, and community health indicators, facilitates harmonization of indicators across initiatives, and links indicators with evidence-based interventions.
The U.S. Census Bureau
Selected Stories from the 2004 Community Indicators Conference
“A Community Indicators Report: Selected Stories from the 2004 Community Indicators Conference” of the Association for Community Health Improvement’s Health Research and Educational Trust.
Health Canada
A look at studies on social capital from Health Canada, showing how community-level indicators have been used in various studies to both define and measure the concept.
The International Institute of Sustainable Development’s Compendium of Sustainable Development Indicator Initiatives
A listing of organizations and projects using community-level indicators to work toward sustainable development in a variety of fields (environment, population, economics, etc.) and locations around the globe.
Good Health Counts: Measurement and Evaluation for Health Equity
The Prevention Institute's Module 7: Good Health Counts: Measurement and Evaluation for Health Equity.
National Neighborhood Indicators
The Urban Institute’s National Neighborhood Indicators Project. An example of how community-level indicators can be used to drive community change. See more about the idea. There are a number of websites listing community-level indicators for various issues, many in specific states, counties, cities, etc. The following is a sampling:
A Community Indicators Report
Selected Stories from the 2004 Community Indicators Conference.
Community Indicators of Alcohol and Drug Abuse Risk
June 2004, for all counties in California.
State Indicator Report on Physical Activity
The U.S. Centers for Disease Control. A list of emerging community-level indicators for physical activity. 2010.
Identifying the Community's Critical Health Issues
Hawaii State Health Planning and Development Agency. The “Tools” section on p.16 contains a list of community-level indicators for various issues.
Livable Tucson
A Tucson, AZ, initiative. This page is a list of goals, each of which, when clicked on, displays, among other things, a list of key indicators for that goal.
San Diego County's Report Card
San Diego (CA) County’s report card on Child and Family Health and Well Being, which includes, starting on p.10, an exhaustive list of community-level indicators and explanations of how they’re used. Warning: if you have a slow Internet connection, this site takes a long time to download.
Sustainable Seattle
An explanation of how Sustainable Seattle, a Seattle, WA, development initiative, chose indicators, and a pdf of the indicators it used.
Community Health Status Indicators (CHSI)
A web-based tool providing local public health agencies access to county health status profiles for improving community health by identifying resources and setting priorities. Visit the CHSI home page to read about the data sources, definitions, and notes.
County Health Rankings
An interactive website providing access to 50 state reports with rankings of each county within each state according to its health outcomes and health determinants. The County Health Rankings are a key component of the Mobilizing Action Toward Community Health (MATCH) project. MATCH is a collaboration between the Robert Wood Johnson Foundation and the University of Wisconsin Population Health Institute. Read about the project.
DATA2010 An interactive database system containing the most recent monitoring data for the Healthy People 2010 objectives. Statistical tables from eighteen focus areas with state level data are now available.
America’s Children in Brief: Key National Indicators of Well-Being, 2010
A compendium of indicators illustrating both the promises and the difficulties confronting our Nation’s young people.
Chronic Disease Indicators
A set of 90 cross-cutting, important, uniform, and available indicators developed by consensus and drawn from many data sources.