Section 5. Our Evaluation Model: Evaluating Comprehensive Community Initiatives

Use this model to evaluate comprehensive community initiatives working to improve quality of life in the community.

 

 

Throughout the world, people and organizations come together to address issues that matter to them. For example, some community partnerships have formed to reduce substance use, teen pregnancy, or violence. Other community-based efforts attempt to lower risks for HIV/AIDS, cardiovascular diseases, or injuries. Alliances among community people have also focused on promoting urban economic development, access to decent housing, and quality education.

These initiatives try to improve the quality of life for everyone in a community. Often, they do this in two ways. Initiatives use universal approaches -- that is, they try to reach everyone who could possibly be affected by the concern. They also use targeted approaches, which try to affect conditions for people who are at higher risk for the problem. Through these two approaches, initiatives try to change people's behavior, such as using illegal drugs, being physically active, or caring for children. They also might go deeper and try to change the conditions, such as the availability of drugs, or opportunity for drugs or daycare, under which these behaviors occur.

Community health promotion is a process that includes many things at many levels. For example, efforts use multiple strategies, such as providing information about the problem or improving people's access to assistance. They also operate at multiple levels, including individuals, families and organizations, and through a variety of community sectors, such as schools, businesses, and religious organizations. All of this works together to make small but widespread changes in the health of the community. The goal is to promote healthy behaviors by making them easier to do and more likely to meet with positive reinforcement.

There are a lot of different models that describe how to best promote community health and development. Some of the more popular models include the Healthy Cities/Healthy Communities Model, the PRECEDE/PROCEED model, and the Planned Approach to Community Health [PATCH]. Similarly, our University of Kansas (U.S.A.) Center for Community Health and Development's model of Building Capacity for Community Change is outlined elsewhere. While how things should be done differs in each model, the basic goal of these and other community approaches is the same. They aim to increase opportunities for community members to work together to improve their quality of life.

Unfortunately, only modest information on the effectiveness of community-based initiatives exists. That's because evaluation practice hasn't fully caught up with a recent shift towards community control of programs. Although there are models for studying community health efforts, community initiatives are often evaluated using research methods borrowed from clinical trials and other researcher-controlled techniques. While these methods work very well in the fields for which they were developed, they're not necessarily a "good fit" for evaluating community work. It's like trying to put a square peg into a round hole -- with a lot of work, you might be able to do it, but it will never be as smooth as you want.

New ideas about community evaluation have their roots in several different models and traditions. These include:

• Action anthropology, which refers to the use of research to empower communities.
• Qualitative research, which highlights the value of the experience as an important part of understanding the effort. That is, it looks at other things besides statistics as important.
• Community-based participatory research (CBPR), which uses dialogue among community and scientific partners to produce knowledge and guide the actions taken by a group or community.
• Empowerment evaluation, which aims to assess the effort's worth while improving the community's desire and ability to take care of its own problems.

These and other types of research actively involve community members in designing and conducting the evaluation. They all have two primary goals: understanding what is going on, and empowering communities to take care of themselves. What is different between these methods is the various balances they strike between these two ends.

In this section, we'll look at models, methods, and applications of community evaluation in understanding and improving comprehensive community initiatives. We'll start with a look at some of the reasons why community groups should evaluate their efforts. Then, we'll describe some of the major challenges to evaluation. We'll also describe a model of community initiatives as catalysts for change. Then, we'll discuss some principles, assumptions, and values that guide community evaluation and outline a "logic model" for our KU Center for Community Health and Development's system of evaluation. We'll also make some specific recommendations to practitioners and policymakers about how these issues can be addressed. Finally, we'll end with a discussion examining some of the broad issues and opportunities in community evaluation.

Why evaluate?

There are many good reasons for a community group to evaluate its efforts. When done properly, evaluation can improve efforts to promote health and development at any level -- from a small local nonprofit group to a statewide or even national effort. Evaluation offers the following advantages for groups of almost any size:

• Collecting information about how things are done and the results help us understand how community initiatives develop, offering lessons other groups can profit from.
• Providing ongoing feedback can improve community work by encouraging continuous adjustments of programs, policies, and other interventions.
• By involving community members, people who haven't had a voice may gain the opportunity to better understand and improve local efforts.
• Finally, evaluation can help hold groups accountable to the community and to the grantmakers who provide funding. It can also help hold grantmakers accountable to the communities that they serve.

Challenges to community participatory evaluation

Although there are a lot of advantages to evaluating community efforts, that doesn't mean it's an easy thing to do. There are some serious challenges that make it difficult to do a meaningful evaluation of community work. They are:

• The causes of a lot of community problems, such as substance use or violence, aren't very well understood. Without this better understanding of the causes, it's hard to decide what needs to be done and if the work has been successful.
• Some of the more important things to evaluate, such as the ability of the group to successfully accomplish its goals or the quality of life of community members, can be very difficult to measure.
• Community initiatives are very complex. They include doing a lot of things on many levels with a lot of different people. Because of this, it is daunting to describe what's been done thoroughly enough for another community to try to do the same thing.
• For some community issues, such as child abuse or domestic violence, researchers haven't yet come up with valid ways to determine if efforts are working. Because of this, it's difficult to evaluate the effectiveness of efforts.
• For many issues, it takes a long time to move the bottom line. For example, if a group is trying to reduce HIV/AIDS in the community, they won't know if they have really affected the number of people who contract HIV for years and years. Therefore, it's necessary to identify markers along the path -- measures of intermediate outcome, such as changes in the community or system, which give community members an idea of whether or not they are going in the right direction.
• It's very hard to estimate how strong a community-driven intervention is -- will it make a large impact, or just a ripple? Evaluators will need to collect precise information on what happened, who it happened to, and for how long the intervention occurred.
• Information collected on individuals can't always be generalized to come to a conclusion about the community as a whole.
• Because there aren't always suitable experimental designs or fitting comparisons (for example, it's hard to say that two towns are exactly alike), it is not always possible to say that the results were really because of the community initiative, and not because of something else that was going on.
• Because community initiatives change with time and circumstances, what they do gets modified as well. Since they are so malleable, it can be difficult to assess the generality of effects, and decide if a given program is good in general or just worked in one particular circumstance.
• People see things differently. Evaluators, especially those in the field of participatory evaluation, must guard against potential confusion resulting from conflicting ways of looking at things when interviewing different people about the same event.
• To evaluate a community initiative fully and well takes a lot of time and work. For an already overburdened organization, it may not be feasible to do all of this properly.
• As we discussed above, there are two primary goals of evaluation: understanding what is going on and empowering communities to take care of themselves. It can be very difficult to try and attain both of these goals at the same time. Often, one seems to need to give.

Despite the challenges that evaluation poses, our belief is that it is a very worthwhile pursuit. In order to minimize these challenges, the KU Center for Community Health and Development has developed a model and some principles that may provide guidance for people trying to evaluate the work done in their community.

A model: Community initiatives as catalysts for change

Although different community groups have different missions, many of them use the same logic model or framework: that of a community initiative as a catalyst for change. This type of community initiative tries to transform specific parts of the community. They change programs, policies, and practices to make healthy behaviors more likely for large numbers of people.

Below, we offer a model of what occurs in a comprehensive community initiative and its results.

• Supporting collaborative planning
• Documenting community implementation, action, and change
• Assessing community adaptation, institutionalization, and capacity
• Evaluating more distal outcomes
• Promoting dissemination

This model is nonlinear -- that is, community partnerships don't just do one thing at a time. Instead, they take part in many interrelated activities that occur simultaneously. A new initiative to reduce the risks for youth violence, for example, may be refining its action plan while pursuing relatively easy changes in the community, such as posting billboards that warn people of the results of gang-related violence.

The components of the model are also interrelated -- that is, they can't be taken separately. They are all part of the same puzzle. For example, collaborative planning should decide what needs to happen in the community. That, in turn, should guide community action and change. Important community actions may be adapted to fit local conditions, and then kept going through policy changes, public funding, or other means of institutionalization.

Also important in this model is the idea that success breeds success. If a community is able to successfully bring about changes, their capacity to create even more community changes related to the group's mission should improve. This, in turn, may affect more distal outcomes -- the long term goals the group is working for. Finally, successful comprehensive initiatives or their components (e.g. public awareness, changes, policies) may be picked up and adapted by other communities addressing similar concerns.

The goals and expectations of community initiatives vary. A community may have a single, narrowly defined mission, such as increasing children's immunizations against disease. It may also have much broader goals that involve several different objectives. For example, members of an initiative may wish to work on two problems, such as reducing child abuse and domestic violence, which share common risk and protective factors.

Some communities have a relatively free hand in deciding what to do. Other partnerships may be required by grantmakers to use "tried and true" strategies or interventions. Some initiatives try hybrid approaches that combine the use of these "tried and true" methods with the role of a catalyst. They do this by implementing core components, such as sexuality education and peer support for preventing adolescent pregnancy, along with developing new community changes, such as enhancing access to contraceptives, that are related to the group's desired outcomes.

Different initiatives will modify programs to make them work well in their community. For example, different groups might want to develop supervised alternative activities for teens to make their taking part in risky behavior, such as unsafe sex or drug abuse, less likely. However, different communities may start any one of a variety of interventions, such as expanding recreational opportunities, offering summer jobs, or developing community gardens.

Adapting interventions to fit community needs has several advantages. First of all, it creates an approach that "belongs" to community members -- it's something they are proud of, that they feel they created -- it's really theirs . Second, because it has been modified to fit the community's needs, the program or policy is more likely to remain in existence. Finally, through changing interventions to fit local needs, community members improve their ability to take care of their own problems.

If a comprehensive community initiative (or a program or policy that is part of it) proves to be successful over a long period, it may be used as an example that other communities can follow. For example, comprehensive interventions for reducing risks for cardiovascular diseases, or specific parts of the intervention such as increasing access to lower fat foods, might be held up as examples for other groups. Leaders of nonprofit organizations need to know what works, what makes it work, and what doesn't work. That way, local efforts can learn from other community-based projects and demonstrations, and adopt some of what experience and research suggest are the "best practices" in the field.

Principles, assumptions, and values of community evaluation

When we look at the process of supporting and evaluating community initiatives, we need to look at what our ideas are based on. The following principles, assumptions, and values serve as the foundation for these processes. You'll notice that they reflect the challenges of addressing both of the major aims of evaluation: understanding community initiatives while empowering the community to address its concerns.

Principles, Assumptions, and Values of Community Evaluation

Community initiatives often function as catalysts for change in which community members and organizations work together to improve the quality of life.

Community initiatives are complex and ever-changing, and they must be analyzed on multiple levels.

Community initiatives help launch interventions that are planned and implemented by community members.

Community evaluation must understand and reflect the issue, and the context in which it is happening.

Community evaluation should involve people from throughout the community.

Community evaluation information should be linked to questions of importance to key stakeholders.

Community evaluation should better community member's ability to understand what's going on, improve practices, and increase self-determination.

Community evaluation should begin early and be ongoing.

Community evaluation results, if positive, should be used to help sustain and promote widespread adoption of the community initiative and/or its components.

Community evaluation should be coupled with technical assistance to provide total support.

Community initiatives engage community members and organizations as catalysts for change: they transform the community to have a better quality of life. Community evaluation is based on the premise that community initiatives are very complex. To be effective, they need many levels of intervention. Researchers try to understand the issue, the history of the initiative, and the community in which it operates. Ideally, local initiatives are planned and implemented with the involvement of many community members, including those from diverse backgrounds. Because of this, community evaluation is a participatory process involving a lot of collaboration and negotiation among many different people.

Evaluation should take place from the beginning of an initiative. That way, it can offer ongoing information and feedback to better understand and improve the initiative. Evaluation priorities (that is, what to evaluate) should be based on what's of most importance to community members, grantmakers, and the field.

If done properly, evaluation results should actually help sustain and renew the community initiative. The information gathered in evaluation can be used to obtain resources such as grant money, show how to improve, and offer an opportunity to celebrate accomplishments. If the initiative seems to be effective, information from community evaluation can be used to promote its widespread adoption. Finally, evaluation should be coupled with technical assistance to provide a complete support system for the initiative. Evaluation without support can actually hurt the initiative. It can be seen as criticism, and leave members of the group frustrated and unsure of what to do next.

Logic model for community evaluation

The figure below describes the logic model for the KU Center for Community Health and Development's Community Evaluation System. A logic model is simply a way of thinking about something in a rational order -- one thought naturally follows another, and you build on ideas as you go. It has its roots in the catalyst model we described above, and tries to show the ideal situation -- what might occur in a fully implemented community evaluation.

 

Initiative phases and evaluation activities

Our ideas about evaluation and support for community initiatives are based on the model of community initiatives as catalysts for change we described earlier. The five parts are:

1. Supporting collaborative planning
2. Documenting community implementation, action, and change
3. Assessing community adaptation, institutionalization, and capacity
4. Evaluating more distal outcomes
5. Promoting dissemination

Supporting collaborative planning

Agenda setting -- that is, deciding what issues and options should be acted upon--is one of the most important things a community initiative can do. Agendas shape the choice of which issues should be addressed. The topics for the agenda might be decided on by using assessment tools, such as gathering information about community concerns. Media advocacy--understanding how to use the media to effectively get the word out--may also assist agenda-building efforts.

Supporting collaborative planning, when done comprehensively, will include all of the following:

• Involving many people in planning efforts, including those from diverse backgrounds
• Clarifying the group's vision, mission, objectives, and strategies
• Developing an action plan that identifies specific community changes to be sought (and later documented) throughout the community
• Identify local concerns, and gather information about them
• Identifying local resources that can help solve the problem

Documenting community implementation, action, and change

Community evaluation documents what gets done by community initiatives, and lets all of the members of the initiative know about these changes. That way, community members can improve on what they have done. This includes documentation of:

• Local efforts and accomplishments
• Community and system changes: Changes in programs, policies, and practices that are related to the mission
• How things got done
• How many changes occurred in the community and where they happened (This is also known as intermediate outcomes)

Assessing community adaptation, institutionalization, and capacity

Community evaluators also look at how the interventions get changed, and whether or not these adjustments to fit the community actually work. They also determine if efforts to sustain the initiative are effective. Finally, evaluators try to measure if efforts to improve the community's capacity to address current (and future) issues have been effective.

Evaluating more distal outcomes

Of course, the ultimate goal of most community initiatives is to move the bottom line--to have fewer people contract HIV/AIDS or be victims of violence, to give two examples. To see if this has happened, community evaluators use quantitative methods. One such method is the use of behavioral surveys. For example, a teen pregnancy prevention project might survey students about reported abstinence or unprotected sexual activity. Another quantitative method is finding archival records of outcomes. For example, the project above might use estimated rates of teen pregnancy from the health department.

They also use qualitative methods, such as interviews with participants, to better understand the meaning and value of efforts. Used together, quantitative and qualitative information weave a rich tapestry of understanding around the initiative's efforts, and offer a solid understanding of the community-level outcomes. They are much more powerful together than either could be alone.

Unfortunately, it usually takes so long to see if the initiative has really moved the bottom line that this information isn't useful for making the day-to-day improvements initiatives need. This is why we recommend documenting intermediate outcomes such as changes in the community or broader system. Measuring community changes--new or modified programs, policies, or practices -- assists in detecting patterns to see if the initiative is helping to create a healthier environment.

Promoting dissemination

Finally, evaluators help community initiatives spread the word about effectiveness to important audiences, such as community boards and grantmakers. Evaluators help provide and interpret data about what works, what makes it work, and what doesn't work. Ways to get the word out may include presentations, professional articles, workshops and training, handbooks, media reports and on the Internet.

So, how does all of this work together? To give a quick recap:

Identifying local concerns helps communities decide on and develop strategies and tactics. These, in turn, may guide implementation of interventions, actions, and changes. Important parts may be adapted to work better in the local community, and important changes may be sustained. This should improve the community's ability to address current (and future) issues. It may also help obtain the initiative's long-term goals, and at the same time improve researchers' understanding of how to get things done. This may help promote adoption of the entire initiative or its more effective components by other communities. All of these steps may influence each other and help decide what the community will do next.

Recommendations for practitioners and policymakers

Research and experience in the field provide us with recommendations for community evaluation. These 34 specific recommendations are grouped into categories that follow the five phases of the catalyst and logic models:

• Supporting collaborative planning
• Documenting community implementation, action, and change
• Assessing community adaptation, institutionalization, and capacity
• Evaluating more distal outcomes
• Promoting dissemination

These recommendations are directed to a wide audience that includes both practitioners, especially members of community initiatives, and policymakers, including elected and appointed officials and grantmakers.

Recommendations for Supporting Collaborative Planning

• Policymakers should support, and practitioners assist, community members in identifying local concerns and collecting information that documents these problems.
• Policymakers should support, and practitioners assist, community members in strategic planning. This includes identifying a vision and developing a mission, objectives, strategies, and action steps.
• Practitioners should develop and share information regarding factors that put people at risk for (or protect them against) local concerns.
• Practitioners should use community members' knowledge of what's going on and build on this understanding by assisting with the interpretation and analysis of available information.
• Practitioners and policymakers should involve community members in developing an evaluation plan for the initiative.
• Practitioners should develop a "give and take" relationship with members of community initiatives. That is, they can provide providing technical assistance and resources for the initiative, and in turn ask for information and data.
• Information should be shared among practitioners, community members, and other key stakeholders.

Recommendations for Documenting and Supporting Community Implementation, Action, and Change

• Practitioners and policymakers should help community members choose interventions and prioritize goals using local and expert knowledge of what is important and what is feasible.
• Practitioners and policymakers should encourage community initiatives to be a catalyst for change. They should focus their efforts on transforming the environment (i.e., by changing programs, policies, and practices), rather than focusing only on individual behavior.
• Practitioners should highlight the products of planning, such as forming committees or completing grant applications, rather than the process it took to do it (e.g., how much time was spent, the number of meetings that took place).
• Practitioners should provide technical support and constructive feedback to help the initiative understand (and do!) what works in their community.
• Practitioners should evaluate progress made in moving the "bottom line," or indicators population-level outcomes.
• Practitioners should also evaluate and share information about the process with community members. For example, they might look at and explain the amount of media coverage, number of community members and organizations participating, resources generated, and services provided.
• Policymakers should request, and practitioners provide, a way to measure changes in the community such as knowing how many new or modified programs, policies, or practices that the group has brought about. These data on community (systems) change will help show how the environment is changing to improve community health and development.

Recommendations for Assessing and Supporting Community Adaptations, Institutionalization, and Capacity

• Policymakers should allow, and practitioners support, the reinvention or adaptation of interventions to be more effective in the local community.
• Practitioners should collect information on what happens and what makes it happen to see if the group's work is effective.
• Policymakers should encourage community groups to look at things over the long haul. This means helping with long-range planning, providing training, and fading funding over time. For example, a grant may give the most money in the first year, less money in year two, and even less in year three. This might allow the initiative to have the initial support it needs, and then prompt the group to look for more sustainable funding. All of this should help to promote the institutionalization of the initiative.
• Practitioners should conduct periodic assessments to see how many of the group's community or systems changes have been sustained. This helps determine the level of institutionalization of the initiative.
• Practitioners should collect information on rates of community change over time and across concerns (that is, changes that occurred in the community for different missions, such as substance use and child abuse). High rates of change over time and across different areas of local concern provide an indication of "community capacity."
• Practitioners should collect and share information on community members who become "community champions"--that is, who do great things for the initiative and the community as a whole.
• Policymakers should request, and practitioners should provide, regular reports on what's happening.
• Practitioners should provide feedback on how and where community changes have occurred to help understand and improve efforts to address community issues.
• Policymakers should provide grant funding that improves the ability of a diverse team of leaders to successfully carry out the initiative. For example, they might provide training on grant writing or leadership development.

Recommendations for Evaluating and Influencing More Distal Outcomes

• Practitioners should record what people say has happened related to risk and protective factors (for example, "I don't smoke") and statistical evidence that will back up or contradict what people are saying (for example, the number of cases of lung cancer).
• Practitioners should study how "health promoting" the environment is and how it changes over time. This may include studying rates of community or systems changes and their relationship to changes in the bottom line.
• Practitioners should develop consistent, practical methods for collecting information on relevant behavior and related outcomes in a comparison community. (A comparison community is one similar to that you are studying, but in which no systematic intervention occurs.) For example, if you are conducting a comprehensive initiative in an urban neighborhood, you might use another urban neighborhood that is nearby as a comparison.
• Policymakers should encourage, and practitioners support, community members and outside experts to evaluate the importance of the initiative's achievements. This can help increase accountability to community members and other stakeholders.
• Practitioners should use qualitative methods to improve understanding of what gets done and how it happens. These methods might include interviews with participants about barriers, resources, and lessons they have learned about the works.
• Policymakers should provide funding that is based on showing positive results. For example, annual renewal of grants might be based on evidence of high rates of community or systems change; bonuses could be given for groups that have done outstanding work; and outcome dividends for those showing improvement in community-level outcomes.
• Practitioners should share information on what has happened, why and how it happened, and the resulting changes in the community. This should be shared early and regularly to a broad cross section of people, including staff, community members, board members, and grantmakers.
• Practitioners should collaborate with initiative members to develop meaningful ways to present evaluation data to key stakeholders.

Recommendations for Promoting Dissemination

• Practitioners, community members, and staff should present data at local, state, national, and international venues to create a larger audience for their efforts.
• Practitioners and policymakers should share information about effective programs, and encourage other communities to adopt them.
• Policymakers and practitioners should use traditional methods such as the newspaper and storytelling, and modern methods such as the Internet, to get the word out about successful interventions, promising practices, and lessons they have learned.

In Summary

Community evaluation offers two overarching benefits. First, it helps us better understand the community initiative, and second, it improves the community's ability to address issues that matter to local people. This evaluation perspective joins the traditional research purpose of determining worth with ideas of empowerment.

In community evaluation, community members, grantmakers, and evaluators work together to pick the best strategies for the community. The specific mix chosen is determined by several things: the issue to be addressed, the interests and needs of those involved, the resources available for the evaluation, and what the initiative is doing. The evaluation is designed very carefully to answer the following: How well does this help us understand and contribute to our ability to improve our community?

For example, an injury prevention initiative might work with the local clinic to assess risk behavior with surveys and determine how many deaths and injuries occurred that were related to violence, motor vehicle crashes, or other causes. Evaluation might be very different for a child welfare initiative, however, which might find it too expensive to watch parents and children interact, or not be able to afford a behavioral survey. Instead, it might collect information on the number of children living below the poverty level or other measurements of children's well-being.

Ideally, community evaluation is an early and central part of the initiative's support system. At the beginning, it helps the group decide on goals and strategies. Later, the evaluation team can document the community's progress towards its goals.

Communities often have a local support system, which might include things such as financial resources or service networks, which help make it possible for the initiative to make a difference in the community. Community evaluation can help communities recognize their own abilities to bring about change, and then to act on that knowledge. The community is in a partnership with the evaluation team, with both working together to understand and improve the initiative.

Communities identify and mobilize existing resources to bring about changes, and members also help document them. By documenting these community or systems changes, community evaluation can prompt community members and leadership to discover where change is (and should be) occurring.

When communities are not making things happen, however, the role of the community evaluation team may shift to making the initiative accountable for its actions. When not much happens over a long period, for example, evaluation information can be used to encourage leaders of the initiative to change what's going on. In extreme cases, community initiatives may be encouraged to change the leadership of the initiative. Finally, renewal of funding -- and bonuses and dividends -- can be based on evidence of progress, with intermediate and longer-term outcomes.

Detecting community capacity -- the community's ability to improve things that matter to local people -- is a particularly important challenge for community evaluation. For example, an initiative trying to prevent substance use that causes many important community changes over a long period, and that then really moves the bottom line, might be said to have greater community capacity than a community whose changes didn't stick. If members of the same initiative later take on a new concern such as preventing youth violence and do so effectively, we might be further convinced of improved community capacity.

Successful community partnerships develop, adopt, or adapt interventions and promising practices that will work in their community. How interventions are adapted and implemented becomes almost as important for researchers as what happened as a result of the intervention.

Relationships between scientists and communities seem to be changing. This may reflect a minor revolution in traditional modes of science and practice. In the late 1980's, community-based grantmaking emerged as a new (or re-discovered) way to distribute resources. It awards grants to the communities to address their concerns themselves instead of to research scientists to design and implement interventions. This researcher-controlled earlier way of doing business didn't address the multiple goals of community initiatives -- improving understanding, capacity, and self-determination. Because of this, there was a lot of unhappiness with traditional research and evaluation. Challenges about their purposes helped bring about the new community-based approaches to evaluation that we have discussed in this section.

The community evaluation system described in this chapter gives a framework and a logic model for examining and improving community initiatives.The methods include providing support, documentation, and feedback. We believe that this approach to evaluation can help local people make a positive difference in their communities.