<<< Back
by Bob Williams
Bob Williams is a professional evaluation consultant based in New Zealand. He explains the origin of the following tips: "Every so often I get phoned up by someone wanting to undertake an evaluation for the first time. One day as an exercise, I decided to jot down everything I tend to say to them."
Bob Williams' home page is available at:
http://users.actrix.co.nz/bobwill/
Go to Free Resources/Writings for this and other tips. Reprinted with permission.
Ten Tips When Thinking About Undertaking an Evaluation
1. Evaluation can be thought of as seeking to answer accurately, validly and usefully the following three questions:
-
What happened?
-
So what?
- Now what?
2. Evaluation can also be thought of as seeking to answer accurately, validly and usefully the following question:
"Who, in what circumstances, and in what way, did or didn't benefit from the thing you are evaluating, who learned what from that, and what does this imply for the future?"
3. Note that the question "why" is missing from #1 and #2. If you want an answer to the question "why" things happened, then think of the most you are prepared to spend on the research and multiply that figure by 10. Think of the time when you want the results and add 12 months.
4. If #3 is not feasible, then how will your organization react to many different possible explanations for the events observed in the evaluation?
5. If #3 is not feasible, then you will be restricted to three kinds of "causal" conclusions:
- Assigned causality (i.e., "I did A because of B")
-
Observed causality (i.e., "We could see that B was a direct result of A happening")
-
Assumed causality (i.e., "Research and professional judgment indicate that if A happens then it is likely that B will also happen)
6. Design the evaluation backwards not forwards. Before you decide what the evaluation should look at or even whether to have an evaluation, answer the following questions:
How does your organization learn?
- What brings about change in your organization?
- What kind of information and analysis is acceptable to your organization?
- Under these circumstances what kind of evaluation would your organization be able to use?
- Who might and might not benefit from the evaluation; how could that affect the data you collect and the acceptance of any conclusions?
- Taking account of these factors, what kind of evaluation is ethical, valid, feasible and desirable?
7. In trying to decide the objectives of an evaluation, avoid if possible a version that just places "did" in front of the program or organizational objectives. If you are interested only in whether it reached the formal stated objectives, think of these two statements: "The operation was a success, but the patient died" and "The operation was a failure, but the patient lived." Watch out for the law of unintended consequences-you can't learn from what you don't look at. You don't always find your dropped car keys by looking where the streetlight shines.
8. If you are interested in assessing whether your project or organization had any sustained impact, think of the last time you experienced something like the project yourself. Ask yourself how much you did or didn't change, and write a list of all the things that affected that. If the list is longer than three items then see #3.
9. If you want to know the cost benefits of any complex program that involves people, then don't call an "evaluator" call an economist. If the economist doesn't start asking questions you find difficult to answer, then call your lawyer.
10. A young evaluator was working with a long established community. She did all the right things about deciding which questions were valid to ask, accurate ways of collecting the data, and appropriate ways of analyzing things. But in the end she could make no sense of what she collected. Throwing up her arms, she decided to approach the community's elders, whose wisdom she respected and trusted. Perhaps they could provide an interpretation that made sense. "You have done a good job" they said "there is nothing wrong with your data, or your attempts to understand it; but we have lived in this area a long time, and in that time have learned that there are some things here you are just not meant to know."