Example 1: Reasons to evaluate
Different organizations will have different reasons to implement an evaluation, and different interests will be at stake. For instance, some organization may want to focus on increasing the quality of its grants programs. Some other organization may be interested in becoming more accountable. Being aware of these interests in fundamental to understand how interests must be balanced. Here we offer a table with hypothetical examples of the kinds of interests that organizations may have.
|Foundation / Organization
|The Meyer Memorial Trust request-for-proposal program
|Improve grantee programs, improve grantmaking
|L.J. and Mary C. Skaggs Foundation program for arts organizations
|Be accountable, improve grantmaking
|Winthrop Rockfeller Foundation at-risk youth program
|Improve grantee program, disseminate successful programs
|Cities in Schools program
|Increase the state of knowledge
|Technology for Literacy project
|Be accountable, improve grantmaking, assess quality or impact of program
|William Penn Foundation career / vocational program
|Be accountable, improve grantee programs, improve grantmaking
|Increase the state of knowledge
|W.K. Kellogg Foundation grants
|Be accountable, improve grantee programs, improve grantmaking, assess quality or impact of grants / programs, plan new programs, disseminate successful programs
Example 2: Leading a community
Ultimately, the community is most affected by the results of an evaluation. That's why community leaders play a key role in the interest game of evaluations. Lisa Adkins knows enough about her community. She is the executive director of Youth Friends in Kansas City. Through Youth Friends, students are linked to volunteers with a shared interest, whether it's music or sports, computers or a special hobby, to create an environment in which every young person who needs a caring adult in his or her life has one.
Lisa talked to us about how her interests as a community leader interfere in the evaluation process. However, according to Lisa, conflict of interests has not been a problem in her evaluation process. She said that the work between evaluators, funders and herself as a community leader is usually of a cooperative nature, and everybody works together toward a common goal. She considers the evaluation process a positive experience for her organization, once they get a chance to review their programs and change what needs to be changed. So, evaluations are a constant element in her work
Her main interest, Lisa says, is to receive feedback on the programs Youth Friends provide, and evaluations are a great way of getting a response from the community. Youth Friends can adjust their services as the necessity arises from the community. Lisa points out that she is always the most interested in the evaluation's results since they could change her approach to a certain program and drastically improve it.
Example 3: Evaluating the evaluator
When it comes to figuring out interests in the evaluation game, the evaluator is a key character. Vince Francisco knows this game well. He's worked as an evaluator with diverse, community-based organizations doing hundreds of program evaluations. He has done evaluations of community policing programs, chronic and cardiovascular disease prevention programming, youth development, drug-abuse prevention, alcohol-use prevention, tobacco control initiatives, teen pregnancy prevention, and general youth development initiatives. Here Vince talks about how the evaluator's interests influence the score of the game:
"I try to find out immediately who the key players are and what they want to have answered. As an evaluator, what I bring to the table is a critical set of questions that I ask folks. Sometimes it's just sitting there watching folks in action, and other times it's asking key people very pointed questions about what they want to do, why they're doing it. I work with folks to take it to the next level and make it a little bit more formal in terms of coming up with some evaluation from very different points of view. If we don't understand their program, we can't go anywhere with their evaluation. I get a handle on their program (and I) will help them make their lives easier, help them run their program or initiative better, help them get more money, influence key players in the community to be on their side, the big issues. "
To find the other interests involved in an evaluation, Vince says that he talks to people until he has their roles figured out: "I look for the loudest voices, and then I look for the quietest voices those usually are the key players. The loudest voices usually speak for themselves, and not for the community. And the people who are the absolutely quietest, are the people who really have a voice in the community, but they're being drowned out by the folks who are loud. Very loud people usually get their way and they become key players so I want to know who they are and what their agenda is. And then I find whoever are the quietest people, and they're usually the people who sit in the background. I start talking to people and eventually there are some people that will talk back. I guess I just start talking to folks and (I work with) the ones that end up talking back to me. I combine a lot of talking."
Of course, Vince says, his own interests also influence the outcome of the intervention. To balance out this equation he thinks of the whole group or the whole community as his client. Talking to people, getting information, and shaping them, Vince produces something that works for his clients: "I outline and define the program : What's the initiative? What does it look like? Who are the key players? What are the program components and elements? Who are the targets of change? Who are the agents of change? Once I get a good handle on that, I connect that to some key questions. I collect evaluation data. What evaluation data would people believe as being evidence that these things are actually being implemented, and what would be key evidence that these things are successful? I combine that with appropriate components and elements in a systematically and very logical fashion. I look for ways for economizing procedures for collecting data. I get right to the chase. Don't do a whole lot of surveys that people have to spend time filling out."
Sometimes, however, interests conflict. Vince describes dealing with this kind of situation: "Usually, I'll find a few people that are really on board and very interested in evaluation. A lot of people just hate evaluators and think it's only a bunch of stupid surveys. My goal is to make it simple as possible. So my trick is to go in and sort of navigate the waters. Work with the people who want to work with me at first, and then convince everybody else that this is useful to them." Vince points out that, depending on how political the footfall of the program is, the evaluation may suffer.
Vince doesn't have a recipe to deal with all the interests that are involved in an evaluation: "All I know is that I juggle all the balls and keep them all in the air. And I look at all the different stakeholders. Who is the funding agent, and what is their interest in all this. I have to make sure their interest is represented. I try to give everybody the same exact information, so we're all moving forward together. That's the way I approach these things I try to change. I get everybody talking to each other in a real way and I don't get into the rumor way." Vince says that evaluators have a unique position because they can hold people accountable for their words: "I don't know if (the evaluator) is any more or less important than any other (interests). But without it, you don't do well. If we want to get lasting outcomes, we have to focus on behavior change. We have to focus on changing the systems that are controlling behavior."
Example 4: The funders role
Jeff Usher represents a very important interest in the evaluation process: the funder's. He is a program officer at the Kansas Health Foundation, a philanthropic organization that provides grants to improve the quality of health in Kansas. Here he talks about implementing evaluations and adjusting interests:
"We want to (see) change, we want to make a behavior change or community change. At the foundation, probably 10 to 20 percent of the overall budget will go to evaluation. The evaluation, of course, will be oftentimes both qualitative and quantitative in nature. We want to see the outcomes, whatever it might be; if it's teenage pregnancy, we'd like to see reductions of teenage pregnancies. So we like to see significant outcomes. We normally contract with people that have expertise in that particular area to conduct the evaluation."
Once the outcomes are in, Jeff says that it's time to decide what steps to take: "If it doesn't appear that the initiative is working, we clearly start losing interest in continuing the project in that manner. Oftentimes the evaluation portion of it is after the fact; the initiative is already over with, for the most part. If it comes back and it looks like the initiative was successful and the outcomes were going in the direction that we anticipated, then we may do another round of that initiative. For instance, the School Community Sexual Risk Reduction project, which is in its second phase now. Depending on the initiatives, if it appears that there is something else going on, we may still continue to work with those, or we may want to tweak that initiative a little bit to make it be a little more effective."
Jeff says that balancing interests among funders, community leaders and evaluators is not much of a problem for him: "We don't seem to have much of a problem when working with evaluators and as far as a private foundation, we don't hear a lot of problems." Jeff goes on to say that sometimes he experiences problems with some evaluators regarding the evaluation methods: "We may get some evaluations back that are measuring process rather than outcomes, and we clearly want to see outcomes. As far as balancing the grantee and the evaluator, normally, the evaluator is a grantee too, so in a sense there is even accountability that we want to see from the evaluator, and we would ask for that. Clearly there are certain things we want to see in that evaluation. For instance, we just did a memorandum agreement for an evaluation on a project that we are just going to be kicking off this next year. We want to see a change in the culture of campus, centered around binge drinking or heavy drinking. We outline those particular things in the memorandum when we start negotiating, so we expect to see that."
One of the funder's interests is to keep an eye on how the initiative conducts its business. Jeff says that in every memorandum agreement, they allow his foundation to perform site visits to have a third party come in, audit financial books, and see how the program is going: "Rarely do we follow up on those. The only time we would look at those is when we get a status report, and we ask for status reports from all of the grantees with what we want to see, and they haven't met those objectives. So, for almost all of our major initiatives, we'll have an evaluation component and have someone looking at it. We want to see behavior change, systems change, changes in culture centered around the health and well-being of communities, and individuals in communities."