Here you will find a checklist summarizing the important points of the section:
What do we mean by answering evaluation questions?
__ Evaluation questions refer to what stakeholders – the community and funders, for example – want to know about the functioning of the program or initiative.
Some illustrative evaluation questions:
__ Are participants satisfied with the program? (Process and Implementation Issue)
__ How well is the initiative meeting its stated objectives? (Attainment of Objectives)
__ How much and what kind of difference did it make for participants? (Impact on Participants)
__ How much and what kind of difference did it make on the community? (Impact on the Community)
Why answer the key questions?
__ To improve your work.
__ To understand what affects the work in what ways.
__ To see how to accelerate results.
__ To understand specifically how broader actions, events, or conditions – e.g., a crisis or concentrated poverty – affect the work.
__ To understand what works to bring about community change, and adjust accordingly.
__ To understand how to address specific events and changes within your organization or effort so they will have the most positive or least negative effects.
__ To show the community the value of your work.
Who should be involved in answering these questions?
__ This type of evaluation works best as a participatory effort.
Those who might be involved include:
__ Participants in or beneficiaries of the effort.
__ Residents of a geographic area you’re focused on.
__ Professionals and volunteers carrying out the work.
__ Those whose jobs or relationships bring them into contact and involvement with the population and/or issue you’re concerned with.
__ Funders and local officials.
When should you set up and use an evaluation system to answer key questions about the effort?
__ Start at the very beginning of the effort, so you can record the whole of your process – outreach, planning, implementation, and evaluation.
__ Continue gathering, recording, analyzing, and using data throughout the course of the effort.
How do you use an evaluation system to answer key questions about the effort?
__ Decide what information you need to answer each question.
__ Decide how to gather that information.
__ Devise a method for recording and setting up your data that makes it easy to analyze.
__ Graphing is particularly good because it allows you to easily compare different sets of data.
__ Consider each question separately:
Is the initiative serving as a catalyst for community/system change related to its mission?
Changes to look for include:
__ New or modified programs.
__ New or modified practices.
__ New or modified policies.
What factors or processes are associated with the rate of community or system change?
__ The processes by which you conduct the assessment, planning, implementation, and evaluation aspects of the effort.
__ People served or benefited.
__ Significant program events. These might include:
- Changes in leadership.
- Staffing changes.
- Changes in method or direction.
- Milestones of the effort.
- Increases or cuts in funding.
- Unforeseen circumstances.
__ Community events, such as:
- A change of local political leadership or government administration.
- Racial or ethnic conflict.
- An economic windfall.
- An economic downturn.
- A community tragedy or celebration.
__ Changes in broader conditions.
How are community/system changes contributing to efforts to promote community health and development?
__ Examine community changes in relation to various aspects of your work.
- Group goals or aims.
- The strategies of intervention you’re using.
- Risk and protective factors.
- The expected duration of change.
- The populations benefited.
- The sectors addressed.
- The ecological level addressed.
- Place.
Are community/system changes related to improvements in population-level outcomes that reflect the objectives of your effort?
__ Community-level indicators of an issue are markers of success at the level of the community as a whole, rather than for particular individuals.
__ Population-level outcomes can be found or inferred by consulting publicly available statistics and records, including:
- Census data.
- Public health statistics.
- Records of health and human service organizations. Although most of these organizations shield individual participants with confidentiality, their general records – number of people served, units of service, general outcomes, etc. – are often open to scrutiny.
- Statewide data and data from other communities (for comparison purposes).
- Police and court files.
- Educational data – standardized test averages, truancy, dropouts, high school completion rates, incidence of school violence.
- Environmental statistics – pollution rates, bad air days, amount of open space, water quality, etc.
__ You can also collect other sources of data, using observation, surveys, and other methods.
__ Graphing and comparing sets of data on community-level indicators can show connections among them.
Did the effort lead to improvements in population-level outcomes?
__ The effort may have led to improvements by influencing changes in the community or system (intermediate outcome) and (more distant) community-level indicators of success. that brought them about.
__ Comparing data on population-level outcomes, community changes, and the timing of various phases and events of your effort can help you understand the connections among them.
__ Begin early and continue to gather, record, analyze, and use data indefinitely to understand how to adjust your effort for greatest effectiveness.