Chapter 37. Operations in Evaluating Community Interventions | Community Tool Box

• What do we mean by choosing evaluation questions?

• Why is it necessary to choose evaluation questions carefully?

• When should you choose questions and plan the evaluation?

• Who should be involved in the process?

• How do you choose questions and plan the evaluation?

Chapters 36-39 of the Community Tool Box concern the evaluation of community programs.  We've chosen to devote so much space to evaluation because it's one of the most important parts of any effort to improve community life and bring about lasting social change.  It can help you to better understand how well the planning and preparations for your program went, whether you implemented it as you meant to, and what the consequences were.  It can tell you whether you met your original objectives and goals or not, and give you information about what you need to change to be more effective.

Chapter 36 explains how an evaluation works, and gives some guidance as to how to develop it. Chapter 38 will assist you in gathering and analyzing the information you want, and Chapter 39 deals with how to use that information to improve your program and garner community and funding support. Here, in Chapter 37, we look at evaluation from a research point of view - how to plan and structure an evaluation that can help you better understand and improve what you do.

In this section, we'll discuss the first, and perhaps most important, step in evaluation research: deciding exactly what to evaluate. Each of the rest of the sections in the chapter will deal in detail with one of the steps you'll need to take to design, implement, and use the evaluation. The goal of the chapter is to provide guidelines that are useful to grassroots or community-based organizations as well as students or academic researchers.

What do we mean by choosing questions, and why is it necessary?

Every evaluation, like any other research, starts with one or more questions. Sometimes, the questions are simple and easy to answer. (Will we serve something close to the 50 people we expect to?) Often, however, the questions can be complex, and the answers less easy to find. (Which, or which combination, of the three parts of our intervention will affect which of the two behavior changes we seek within participants?) The questions you ask will guide not only your evaluation, but your program as well. By your choice of questions, you're defining what it is you're trying to change.

You choose your evaluation questions by analyzing the community problem or issue you're addressing, and deciding how you want to affect it. Why do you want to ask this particular question in relation to your evaluation? What is it about the issue that is the most pressing to change? What indicators will tell you whether that change is taking place?  Is that all you're concerned with? The answer to each of these and other questions helps to define what it is you're trying to do, and, by extension, how you'll try to do it.

For example, what's the real goal of a program to introduce healthier foods in school lunches? It could be simply to convince children to eat more fruits, vegetables, and whole grains. It could be to get them to eat less junk food. It could be to encourage weight loss in kids who are overweight or obese.  It could be to educate them about healthy eating, and to persuade them to be more adventurous eaters.

The evaluation questions you ask both reflect and determine your goals for the program. If you don't measure weight loss, for instance, then clearly that's not what you're aiming for. If you only look at an increase in children's consumption of healthy foods, you're ignoring the fact that if they don't cut down on something else (junk food, for instance), they'll simply gain weight.  Is that still better than not eating the healthy foods? You answer that question by what you choose to examine - if it is better, you may not care what else the children are eating; if it's not, then you will care.

Things to consider when choosing evaluation questions

What do you want to know?

Academics and other researchers may approach choosing research questions differently from those involved in community programs.  In addition to their practical and social applications, they may choose problems to research simply because they are interesting, or because they tie into other work that they or their colleagues are doing. Community service workers and others directly involved in programs, on the other hand, are concerned specifically with improving what they're doing so they can help to enhance the quality of life for the participants in their programs, and often for the community as a whole.  Since we assume that most people using this chapter of the Tool Box are likely to be practitioners in the community, let's look at some of the reasons they might pick a particular area to evaluate.

If you're running, or about to run, a program to affect a community issue or problem, you might want to know one or more of the following:

Is there a cause-and-effect relationship (i.e., does one action or condition directly cause another) between a particular action and a particular change?  Usually, you'll be concerned with this in terms of your program. (Does our smoking-cessation support group help members to quit smoking?) Sometimes, however, it might be important to look at it in terms of the community. (Does a smoking ban in public buildings, bars, and restaurants lead to a decrease in the number of community residents who smoke?)

If we try this new method, what will happen?

Will the program that worked in the next town, or the one that we read about in a professional journal, work with our population, or with our issue?

Why are you interested?

Some of the same differences between the concerns of researchers and the concerns of practitioners may hold here. Those interested primarily in research may simply be moved by curiosity or by the urge to solve a difficult problem. As a practitioner, on the other hand, you'll want to know the effects of what you're doing on the lives of participants or the community.

Your interest, therefore, might grow from:

• Your experience with an issue and its consequences in a particular population or community
• Your knowledge of promising interventions and their effects on similar issues
• The uniqueness of the issue to your particular community or population
• The similarity of the issue to other issues in your community, or the issue's interaction with other issues

Your interest as a community worker has to be considered in relation to your evaluation and the purpose of your program. Your basic intent is probably to improve things for the population or the community, but in what ways and by what means? Are you trying out some new things in the hope of making an already-successful program more successful? Are you importing a promising practice to see if it works with your population?  Are you trying to solve a particularly difficult professional problem?

A community mediation program found that it was having little success in cases involving adolescents.  After conferring with other similar programs - all of which were struggling with the same issue - mediators in the program devised a number of strategies to try to reach youth. The overall question they were concerned with - "Will these strategies make it possible to mediate successfully where teens are involved?" - was one with real consequences.

Is the issue you're addressing important to the community or to the society?

Media reports about or community attempts to address the issue are clear indicators that it is socially important. If it affects a particular group - violence in a given neighborhood, a high rate of heart disease among middle-aged Black males - it has an obvious impact on the community and society. If your program or intervention has the potential to help resolve the issue in other places, to be used by community workers in other fields, or to be applied in a number of ways, the importance of your analysis increases even further. If addressing the issue can lead to long-term positive social change, then the analysis is vitally important.

All of this affects your evaluation and the questions you ask. If the issue is one of social importance, then your evaluation of your work is socially important as well.  Are you addressing the aspects of your program or intervention that are of the greatest value to participants, the community, and society?  If not, how might you begin to do so?

How does the issue relate to the field?

The real question here is not whether the issue is important to the field - if it's important to the community, that's what matters. However, you should explore whether there's evidence from the field to apply to the issue.  Is what you're doing likely to be more effective than other approaches that have been tried?  If your approach isn't effective, are there other approaches out there that hold more promise? Can the published material about the issue help you understand it better, and give you better ideas about how to address it?

Is the issue general, rather than specific to your population or community?

Consider whether there is evidence that the issue occurs with a variety of populations and under a range of conditions. Also consider whether the observations or methods used to determine the issue's existence are accurate and whether they can be used in different situations and with different groups. Your evaluation may give you valuable information to pass on to practitioners in different fields or different circumstances.

Who might use the results of your evaluation?

If evaluation shows that your program or intervention is successful, that's obviously valuable information, especially if what you're evaluating is innovative and hasn't been tried before. Even if the evaluation turns up major problems with the intervention, that's still important information for others - it tells them what won't work, or what barriers have to be overcome in order to make it work.

Some of those who might use your results include individuals and groups affected by the issue; service providers and others who have to deal with the problem (in the case of youth violence, for instance, this last group might include police, school officials, small business owners, parents, and medical personnel, among others); advocates and community activists; and public officials and other policy makers.

Whose issue is it?

Who has to change in order to address the issue? The focus of the intervention will tell you whom the evaluation should focus on.

Some possibilities:

• Those directly affected by the problem
• Those in direct personal contact with those directly affected: parents, spouses and children, other relatives, friends, neighbors, coworkers
• Those who serve or otherwise deal with those directly affected: medical professionals, police, teachers, social workers, therapists, etc.
• Administrators and others who serve or deal with those indirectly affected: hospital or clinic directors, police chiefs, school principals, agency directors, etc.
• Appointed or elected officials and other policy makers

Why is it necessary to choose evaluation questions carefully?

You know why you're running your program.  Evaluating it should just be a matter of deciding whether things are better when you evaluate than they were before you started, right?  Well, actually...wrong.  It's not that simple. First of all, you need to determine what "things" you are actually looking at (remember the school lunch example?)  Second, you will need to consider how you will determine what you're doing right, and what you need to change. Here's a partial list of reasons why choosing questions beforehand is important.

• It helps you understand what effects different parts of your effort are having. By framing questions carefully, you can evaluate different parts of your effort. If you add an element after the start of the program, for instance, you may be able to see its effect separate from that of the rest of the program...if you focus on examining it.  By the same token, you can look at different possible effects of the program as a whole. (Do adult basic education learners read more as a result of being in a program?  Are they more likely to register to vote? Do their children improve their school performance?)
• It makes you clearly define what it is you're trying to do  What you decide to evaluate defines what you hope to accomplish. Choosing evaluation questions at the start of a program or effort makes clear what you're trying to change, and what you want your results to be.
• It shows you where you need to make changes. Carefully choosing questions and making them specific to your real objectives should tell you exactly where the program is doing well and where the program isn't having the intended effect.
• It highlights unintended consequences. When you find unusual answers to the questions you choose, it often means that your program has had some effects you didn't expect. Sometimes these effects are positive - not only did people in the heart-healthy exercise program gain in fitness, but a majority of them report changing their diet for the better and losing weight as well - sometimes negative - obese children in a healthy eating program actually gained weight, even though they were eating a healthier diet - and sometimes neither. Like the side effects of medication, the unintended consequences of a program can be as important as the program itself. (In the case of the exercise program, the changes in diet might do as much as or more than the exercise to maintain heart health, for instance, and may point toward changing the focus of the program in some way.)
• It guides your future choices. If you find that your program is particularly successful in certain ways and not in others, for example, you may decide to emphasize the successful areas more, or to completely change your approach in the unsuccessful areas. That, in turn, will change the emphasis of future evaluation as well.
• In participant evaluations, evaluation involves stakeholders in setting the course of the program, thus making it more likely that it will meet community needs.
• It provides focus for the evaluation and the program. Choosing evaluation questions carefully keeps you from becoming scattered and trying to do too many things at once, thereby diluting your effectiveness at all of them.
• It determines what needs to be recorded in order to gather data for evaluation. A clear choice of evaluation questions makes the actual gathering of data much easier, since it usually makes obvious what kinds of records must be kept and what areas need to be examined.

When should you choose questions and plan the evaluation?

Evaluation questions, since they help shape your work, should be chosen and the evaluation planned when planning the overall program or effort. That gives you time and room for a participatory process, and gives you the chance to use the evaluation as an integral part of the program. As the program unfolds, you might find yourself adjusting or adding questions to reflect the reality of what is happening, but unless your original questions were misguided (you were wrong about what behavior had to change in order to produce certain results, for instance), they should serve you well.

Now let's discuss reality for many community based and grassroots programs. They're often understaffed and underfunded. Staff members may be underpaid, and may often work many more hours a week than they're paid for, because of their dedication to social justice and social change. Most or all program staff may even be volunteers, with full-time jobs and family responsibilities aside from their work in the program.  Initial evaluation in these circumstances is often anecdotal - i.e., based on participants' comments and stories about their progress and staff members' personal, informal observations.  A formal evaluation will probably wait until there's funding for it, or until someone has the time to coordinate or take charge of it.

In that case, the "when" becomes "as soon as you can."  You may be dealing with a program that has just started, or with one that's been operating for a long time. You may know that changes need to be made, or it may seem that the program is in fact meeting its goals. Whatever the situation, evaluation questions need to be chosen, and an evaluation planned that will give you the information you need to improve your work. Even with a program that's been going on for a while, the questions can still help you define or redefine your work, and will certainly help you improve it over the long term.

Who should be involved in choosing questions and planning the evaluation?

If you've consulted other sections of the Tool Box concerned with evaluation, you probably know that we advocate that all stakeholders be involved in planning the evaluation.  We believe that the best evaluation is participatory. That means that there is representation of the views and knowledge of people affected by the issue to be addressed. The list of potential participants is essentially the same as that under "Whose problem is it?" in the first part of this section: those directly affected and their close contacts; those who work with those directly affected, or who deal directly or indirectly with them and the issue; and public officials. To these groups, we might add other concerned citizens, and those indirectly affected by the issue. (A shop owner may not be a victim of neighborhood violence, but fear of that violence might nonetheless keep customers away from his shop, for instance.)

Evaluations that involve all stakeholders have a number of advantages over those conducted in a vacuum by outside evaluators or agency or program staff. They're more likely to reflect the real needs of the community, and they bring to bear the community's knowledge of its own context - history, relationships, culture, etc. - without which a program and its evaluation can go astray.

Participation can range from simple consultation before the fact to complete involvement in every aspect of an evaluation - assessment, planning, data gathering, analysis, and passing on the information.  In general, the greater the involvement of stakeholders, the better, but in-depth involvement of the stakeholders may not always be possible. There are time disadvantages to participatory evaluation - it takes longer - and there are logistical concerns, as well. Participants may have nothing in their backgrounds to prepare them for research, so training in a number of areas may be necessary, requiring skill, careful planning, and yet more time. The level of participation your evaluation can sustain, therefore, relies to some extent on your time constraints and your capacity to train and support participants.

How do you choose questions and plan the evaluation?

Choosing questions

When you choose evaluation questions, you're really choosing a research problem - what you want to examine with your research. (Evaluation, whether formal or informal, is in fact research.) You have to analyze the issue and your program, consider various ways they can be looked at, and choose the one(s) that most nearly tell you what you want to know about what you're doing.  Are you just trying to determine whether you're reaching the right people in sufficient numbers with your program?  Do you want to know how well an intervention is working with specific populations? What kinds of behavior changes, if any, are taking place as a result?  What the actual outcomes are for the community? Each of these - as well as each of the many other things you might want to know - implies a different set of evaluation questions. To find the questions that best suit your evaluation, there is a series of steps you can follow.

Describe the issue or problem you're addressing

A problem is a difference between some ideal condition (all people 10 years of age or older should be able to read; people should be able to find a decent job) and some actual condition in the community or society (a 25% illiteracy rate among those attending a particular high school; 50% unemployment among minority youths in a particular city). This may mean the absence of some positive factor (qualified teachers and adequate educational facilities; entry-level jobs that are reachable from minority neighborhoods) or the presence of some negative factor (students' difficulty with English; discrimination against minority job applicants), or some combination of these.

To describe the issue or problem:

• Describe the ideal condition, including the positive factors present and the negative factors absent.  What should it look like if everything was as you'd want it to be?
• Describe the actual conditions that constitute the prolem of interest, including the negative conditions present and the positive conditions absent.  What are conditions really like?
• Describe the actual problem in terms of what you're hoping to change.  What positive factors do you want to produce and/or what negative factors do you want to eliminate?

Describe the importance of the problem

To be sure that this is a problem you really should be addressing, consider its importance to those affected and to the community.

• Is the discrepancy between ideal and actual conditions of the kind and size to be considered important?
• What are the consequences (positive and negative) of the problem?
• Who experiences these consequences (i.e. program participants; their families, friends, and peers; service providers, policymakers, and others)? How many people are affected?
• How often and for how long are they affected? What is the intensity of the effect?
• How much does the fact that the problem is experienced to this degree by these people matter to them?

You might also ask whether the effects of the problem matter to society, but in fact, that shouldn't make a difference.  If they matter to the people who experience them, they're important.  Society doesn't always consider a problem important if it's only a problem for a minority, or for a group that's generally ignored (the poor, the homeless).

In light of these factors, decide whether the problem is important to the evaluation.

Describe those who contribute to the problem

Whose behavior, by its presence or absence, contributes to the problem?  Are they in the program participants' personal environment (participants themselves, family, friends), service environment (teachers, police), or broader environment (policymakers, media, general public)? For each of them, consider the types of behavior that, by their presence or absence, contribute to the discrepancy that constitutes the problem.

Assess the importance and feasibility of changing those behaviors

How important is each of these behaviors to solving the problem?  What are the chances that your effort can have any effect on each of them?

Describe the change objective

Based on the above analysis, choose behavior changes to target in specific people. Where you can, specify the desired levels of change in targeted behaviors and outcomes (those changes in conditions that should occur if the problem were to be solved).

For example, a behavior change goal might be an increase in pre-employment capacity - self-presentation, job-seeking, interview skills, interpersonal competence, resume writing, basic skills, etc. - for minority job seekers aged 18-24. Or you might instead or in addition target policy makers, with the goal of having them offer tax incentives to businesses that locate in or close to minority communities.

This is a way of defining your work. If you're planning the evaluation as you plan the program - as you would in the ideal situation - then the questions you're asking the evaluation to examine reflect the problems you're trying to solve, and this kind of analysis is important.  If you're starting an evaluation of a program that has been in place for some time, then you're going to have to do some figuring after the fact about what consequences you think (hope) the program is having, and what they will lead to.  You may be talking about changes in specific participant behaviors, about behaviors that act as indicators of other changes, or about results of another sort (participants gaining employment, for instance, which may have a direct relationship to participant behavior or may have more to do with local economic conditions).

Make sure that the expected changes would constitute a solution or substantial contribution to the problem

If you conclude that they would not result in a substantial contribution, revise your choice of problem and/or your selection of targeted people and actions as necessary. If you think that what you're looking at in an evaluation doesn't address the problem, then you should be looking at something else. If the objectives you've chosen do constitute all or a substantial part of a solution, you've found your questions.

Setting

Now that you've chosen your questions, there may be other factors to consider, such as the settings in which the evaluation will be conducted. If your program is relatively small and/or has only one site, this wouldn't be an issue. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program.

 There are some situations in which the choice setting may be important:

• If your program is very large and/or has multiple sites
• If different sites provide different services, activities, or conditions, or use different methods

Multiple sites

Multiple sites

Can present a challenge for an evaluation, because, although every effort may be made to make the program at all sites exactly the same, it will seldom be so. If the program relies on human interaction - teacher/learner, counselor/counselee, trainer/trainee, doctor/patient, etc. - there will be differences from site to site depending on the people staffing each. (The exception is when the same people staff all sites, providing the same services at each site at different times or on different days.)  Even if all are equally competent, no two staff members or teams will do things in exactly the same way or relate to participants in exactly the same way, and the differences can be reflected in differences in outcomes.  If methods or other factors vary from site to site, that will further complicate the situation.

Furthermore, the physical character of a site can influence not only program effectiveness, but also the recruitment of participants and whether or not they remain in the program long enough for it to have some effect (often called "retention.")  The site's layout, comfort, apparent safety and security, and - often most important - how easy it is to get to, all affect whether participants enroll and stay in the program.

Where you do have the capacity to evaluate all sites, it will be helpful to build into the evaluation a method of comparing them. This will allow you to identify and adopt at all sites methods, conditions, or activities that seem to make one site particularly successful, and to identify and change at all sites methods, conditions, or activities that seem to create barriers to success at others.

If you can't evaluate each site separately, you'll have to decide which one(s) will give you the information that will most help in adjusting and improving your program. If you're most concerned with assessing your overall effectiveness, this may mean evaluating the site(s) closest to the program norm, in terms of methods, conditions, activities, goals, participant/staff interaction, etc. If, on the other hand, your chief consideration is learning whether a particular new or unusual method or situation is working, you may find yourself evaluating the site(s) least like the others.

If sites appear only minimally different, some other considerations that may come into play are:

• The number and character of participants at the site. Participants at a particular site may be experiencing the effects of the issue more severely, or may have a particular important characteristic, such as a language barrier.
• The ability and willingness of participants and staff to support the evaluation research. If staff at a particular site are unable or unwilling to record observations, attendance, and other key information, or if site participants are unable or unwilling to be interviewed or monitored, evaluation at that site might be difficult.
• The stability of the population at the site. If participants at a site come and go at a rapid rate - unless that's the program's intent - it can be difficult to gain information that contributes to an accurate evaluation.
• An exception, of course, occurs here if one point of the evaluation is to find out why participants stay for so short a time, and to try to develop methods or create conditions to assist them to remain in the program long enough to reach their goals.

Sites with different methods, conditions, activities, or services

Programs sometimes are organized so that different methods are used or different services provided at different sites. In other cases, conditions may vary from site to site because of the sites' geographical locations or the available space. The ideal situation is to evaluate all sites and compare the effects of the different methods, conditions, or services. When that's not possible, you'll have to decide what's most important to find out.

If the methods, services, or conditions at a particular site are new or innovative, you may want to evaluate them, rather than those that have a track record. There may be a particular method or service that you want to evaluate, in which case the decision about which site to choose is obvious. The decision should be based on what makes the most sense for your program, and what will give you the best information to improve its effectiveness.

When you have the capacity to choose more than one site to evaluate, it often makes sense to choose two or three sites that are different - especially if each is representative of other sites in the program or of program initiatives - so that you can compare their effectiveness. Even where sites are essentially similar, you'll get more information by evaluating as many as you can.

Participants

Another factor to consider is the participants whose behavior, activity, or circumstances will be evaluated. If your program is relatively small this might not be an issue - the participants will simply be all those in the program. However, if you don't have the resources - whether finances, time, or personnel - to evaluate the whole program, there are some situations in which the choice of participants may be important:

• If your program includes different groups of participants (groups that are in different stages of the program, or that are exposed to different methods or services).
• If groups of participants belong to populations with distinctly different cultures, stemming from race, ethnicity, class, religion, or other factors.

Multiple groups

There are a number of reasons why there might be multiple groups of participants in a program. You might start different groups at different times, either because the program has a rolling start schedule (when there are enough people for a class/training group, one will begin), or because the program is aimed at different groups (for example, 5 year-olds, 8-year-olds, and 14-year-olds). You might also be trying different strategies with different groups.

The Brookline Early Education Project (BEEP), a program aimed at school readiness for children aged pre-birth through 5, recruited pregnant families in three cohorts over the course of three years.  In addition, families in each cohort were assigned to one of three levels of service. Thus, there were actually nine different groups among BEEP participants, even though, by the third year, all were receiving services at the same time.

Once again, if there's no problem in evaluating the whole program, participants will simply include everyone. If that's not possible, there are a number of potential choices:

Evaluate your work with only one group, with the expectation that work with the others will be evaluated in the future. In this case, you'd probably want to choose the one for whom you consider the program most crucial. They might be at greater risk (of heart attack, of school failure, of homelessness, etc.) or might be experiencing the issue at a high level of intensity (daily shooting incidents in the neighborhood, high rates of teen pregnancy, massive unemployment).

Include a small number (2-4) of groups in your evaluation. You might want to choose groups with contrasting characteristics (different ages, for example, or addressed by different strategies). On the other hand, depending on the focus of your evaluation, you might want groups that are essentially similar, to see whether your work is consistent in its effects.

Choose a few participants from each group to focus your evaluation on.  While this won't give you a complete picture, it should give you enough information to tell where your program is accomplishing its goals and where it needs improvement. The differences in the ways participants in different groups respond to the program (assuming there are differences) can also give you ideas for ways to change what you're doing.

Participants from different populations and cultures

Cultural factors can have an enormous effect on participants' responses to a program. They can govern conceptions of social roles, family responsibilities, acceptable and unacceptable behavior, attitudes toward authority (and who constitutes authority), allowable topics of conversation, morality, the role of religion - the list goes on and on. In planning a program that involves members of different populations and cultures, you essentially have three choices:

• Plan your program and implement it in the same way for everyone. If the program involves groups - classes, support groups, etc. - participants' membership is determined not by population group but by when they sign up, what time of day they can attend, what they sign up for, or whatever other criteria make sense logistically.
• Plan your program to be as culturally sensitive as possible, and try to screen out anything that might be offensive to or difficult for any group. In this instance, you might be prepared to respond if participants from a particular population requested a group of their own.
• Divide participants by cultural group and plan different culturally sensitive approaches for each. Your overall approach might be the same for everyone, but the way you apply it might differ by culture.

In any of these instances, it would probably be important to understand how well your approach is working with members of the various populations.  If you can evaluate the whole program, make sure that you include enough members of each group so that you can compare results (and their opinions of the program) among them.  If your evaluation possibilities are limited, then your choices are similar to those for multiple groups of other kinds, and will depend on what exactly is most useful for you.

There are interactions between the choice of sites and the choice of participants here. You may be concerned about the effects of your program on a particular population, which may be largely concentrated at one site.  In that case, if you have limited resources, you may want to evaluate only that site, or that site and one other.

Regardless of other considerations, you may want to set some guidelines about whom you include in the evaluation. How long do people have to be in the program, for instance, before they're included? In other words, what constitutes participation? (This also sets a criterion for who should be counted as a drop-out: anyone who starts, but leaves before meeting the standard for participation.) What about those whose attendance is spotty - a few days here, a few days there, sometimes with weeks in between? Do they have to have attended a certain number of hours to be considered participants?

These issues can be more complex than they seem. People may start and drop out of a program numerous times, and then finally come back and complete it. Many others start programs numerous times, and never complete them. It's usually impossible to tell the difference until someone actually gets to the point of completion, whatever that means for the particular program.

In a reversal of the start-many-times-before-completing scenario, there can be a few people who stay in a program right up till the end and then drop out. This may have to do with the fear of having to cope with success and a change in self-image, or it may simply be a pattern the person has learned to follow, and will have to unlearn before being able to complete the program.

Should any or all of these people be included in or excluded from an evaluation, either before (because of their history in the program) or after the fact? That's a decision you'll have to make, based on what their inclusion or exclusion will tell you. Just be sure that your evaluation clearly describes the criteria that you decide to use for your participants.

If you're an outside evaluator or academic or other independent researcher

Up to this point, we've largely ignored the evaluation difficulties faced by evaluators not directly connected with the organization or institution running the program they're evaluating. If you've been hired or designated by the organization or a funder to evaluate the program, you have to establish trust, both with the organization and its staff and with participants, if you hope to get accurate information to work with. You also have to learn enough in a short period about the community, the organization, the program, and the participants to devise a good evaluation plan, and to analyze the data you and others gather.

If you're an independent researcher - a graduate student, an academic, a journalist - you face even greater obstacles. First, you have to find a place to conduct your research - a program to evaluate - that fits in with your research interests. Then, you have to convince the organization running that program to allow you to do the research.Once you've jumped that hurdle, you're still faced with all the same tasks as an outside evaluator: establishing trust, understanding the context, etc.

Let's look first at the process you as an independent researcher might follow in order to choose and gain access to a setting appropriate to your interests. Once you've gained that access, you've become an outside evaluator, so from that point on, the course of preparing for the evaluation will be the same for both.

Choose a setting

If you're an academic or student, you can probably find an appropriate program by asking colleagues, professors, and other researchers at your institution. If none of them knows of one offhand, someone can almost undoubtedly put you in touch with human service agencies and others who will. Other possible sources of information include the Internet, funders, professional associations, health and human service coalitions, and community organizations. Public funding information is often available on the web, in libraries, or in newspaper archives. The wider you spread your net, the more likely you are to find the program you're looking for.

The right program will obviously vary depending on your research interests, but some questions that will inform your choice include:

• Does the setting include people who are actually experiencing the problem that is of interest to you?
• Is the setting similar to others of this type? (If not, its program might not be useful to others dealing with the issue, even if it works well in its own context.)
• Does the setting provide support for the research? Will staff, participants, and others help with data gathering, be forthcoming about context questions, cooperate with you?
• Does the setting have the resources to maintain the program after your evaluation is done?
• Does the setting permit the changes in operation required by the research?  If the planning of the evaluation and choosing of questions point to doing things differently, can and will the program make the necessary changes?
• Is the setting accessible?
• Accessibility includes not only handicap accessibility, but whether a site is in a neighborhood that feels welcoming or safe to participants, whether it is easily reachable by public transportation or on foot from the areas from which participants are drawn, and whether it is in a building or institution that doesn't feel intimidating or strange (a university campus or building can seem as threatening as a fortress to someone who is insecure about his educational background, for example.) Accessibility can be the determining factor in whether participants consider a program, or whether they stay in it.
• Is the setting stable? Are the program and organization stable enough that you know they'll be able to support their work at the current level, at least until the evaluation is completed?

Once you've found an appropriate setting, you'll have to convince the organization to collaborate with you on an evaluation. The next three steps are directed toward that goal.

Learn as much as you can about the organization you've chosen

Just as you wouldn't go to a job interview without doing some research about the employer, you shouldn't try to gain the cooperation of an organization without knowing something about it - its mission, its goals, whom it serves, who the director and board members are, etc. If someone told you about the organization, she may have, or may know someone who has, much of the information you need. If the organization maintains a website, much of that information will be available there. If it's incorporated, the office of the Secretary of the state of incorporation and/or other state offices will have information about the officers (i.e., the Board of Directors) and other aspects of the organization. Funding agencies may also have information that's a matter of public record, including proposals.

Contact the appropriate person(s) and request an interview

Find out whom (by name as well as position) you should talk to about conducting a research project in the organization you've chosen.

Depending on the organization, this could be the board president, the executive director, or the program director (if the program you're interested in is only part of a larger organization).  In any case, it might be wise to involve the program director even if he's not the final decision-maker, since his cooperation will be crucial for the completion of your research.

• If you can, get a personal introduction. It's always best if you come recommended by someone familiar with the person you need to speak with.
• If you can't get a personal introduction, it's usually best to send a letter requesting a meeting and explaining why, and follow it up with a phone call.
• Before the meeting, send a proposal outlining what you want to do. This should be substantive enough to help the organization decide whether it wants to work with you, but not so specific that it doesn't allow for collaborative planning of the evaluation.

Plan and prepare for the initial meeting

There are several purposes for this meeting, besides the ultimate one of getting permission and support for your project (or at least an agreement to continue to discuss the possibility). They include:

• Establishing your credentials - the experience, educational background, and any other factors that equip you to conduct this evaluation. This might include references from colleagues, professors, or other organizations you've worked with.
• Learning more about the program and the organization
• Explaining what you want to do and why, what form the evaluation results are likely to take, what you'll do with them, who'll have access, etc. This explanation should also cover issues of confidentiality and permission of participants.
• Explaining what you need from the organization and/or program - participation of participants and staff, for instance, any logistical support, access to records, or access to program activities
• Explaining what you're offering in return - your services for a comprehensive formal evaluation, any stipends, equipment or materials, other support services, or whatever else you may have to offer
• Clarifying the organization's needs, and discussing how they fit with your own - and how both can be satisfied

Assuming that your presentation has been convincing, and you're now the program evaluator, the rest of the steps here apply to both independent researchers and outside evaluators.

Find out all you can about the context

This may play out differently for outside evaluators than it does for independent researchers, but it's equally important for both. It means finding out all you can about the community, the organization, the program, and the participants beforehand - the social structure of the community and where participants fit in it, the history of the issue in question, how the organization is viewed, relationships among groups and individuals, community politics, etc.

If you're an outside evaluator, you can pick the brains of program administrators, staff, and participants about the community, the organization, and the issue. Ask them to steer you to others - community leaders, officials, longtime residents, clergy, trusted members of particular groups - who can give you their perspectives as well. If possible, get to know the community physically: walk and/or drive around it, visit businesses, parks, restaurants, the library. Understanding how the issue plays out in the community, the nature of relationships among groups and individuals, and what life is like in the neighborhoods where participants live will help a great deal in analyzing the evaluation of the program.

If you're an independent researcher, learn as much about the context as you can before you contact the program. Websites (for the organization and/or the community) and libraries are two possible sources of information, as are community and organization literature and people who know the community.  Learning about the community, the organization, and the participants beforehand will both help you determine whether this program fits with your research and help you advocate for its cooperation with your project. Once you have that cooperation, you can follow the same path as an outside evaluator (since that's what you are) to learn as much about the context of the program as you can.

Establish trust with program administrators, staff, and participants

This can be the most difficult part of an evaluation for someone from outside the organization. There's no magic bullet or predictable timeline, but there are several things you can do:

• Be yourself.  Don't feel you have to act a certain way: deal with people in the program as you do with friends and acquaintances in other circumstances. People can tell when you're being false, and are unlikely to trust you if you are.
• Treat everyone with equal respect, as colleagues in a research project.
• Don't assume you know more than anyone else just because you're the professional.
• Share freely what you do know, but don't tie yourself to any one process or method, especially in response to an opposite stance from a key individual.
• Ask administrators, staff, and participants what they want from the evaluation, and discuss how the evaluation could provide it.
• Don't be afraid to say "I don't know, but I'll find out," and then do.
• Follow through on whatever you say you'll do. Don't promise anything you can't deliver on, and make deadlines reasonable, so you can meet them.

General tips for all evaluators

These steps apply to everyone, internal evaluators as well as external.

Aim for a participatory evaluation

We've discussed above the involvement of all stakeholders to the extent possible. Involving participants, program staff, and other stakeholders in participatory planning and research can often get you the most accurate data, and may give you entry to people and places you normally might not have. On the other hand, participatory planning and research, as we've explained, takes time and energy. If you have limited time, you may not be able to set up a fully participatory project.  You can, however, still consult with stakeholders, and involve them in ways that don't necessarily involve training or large amounts of your time. They can help you line up interviews with participants or other important informants, for instance, and/or act as informants themselves about community conditions and relationships.

At least the people in charge of the program, and probably those implementing it as well, will expect to be part of the planning of the evaluation. They are, after all, the ones who need to know whether their work is effective, and how to improve it. Involving participants as well, in roles ranging from informants about context to actual researchers, is likely to enrich the quantity and quality of the information you can obtain.

Plan the evaluation, in collaboration with stakeholders

That collaboration should be at the highest level of participation possible, given the nature of the program, the time available, and the capacity of those involved (if program participants are five-year-olds, they probably have relatively little to contribute to evaluation planning...but their parents might want to be involved.)

The actual planning involves ten different areas, each of which will be the subject of one of the remaining sections in this chapter:

• Information gathering and synthesis
• Designing an observational system
• Developing and testing a prototype intervention
• Selecting an appropriate experimental design
• Collecting and analyzing data
• Gathering and interpreting ethnographic information
• Collecting and using archival data
• Encouraging participation throughout the research
• Refining the intervention based on the evaluation
• Preparing the evaluation results for dissemination

Once the planning is done, it's time to get started on conducting the evaluation. And when you're finished - having analyzed the information and planned and made the changes that were needed - it's time to start the process again, so that you can determine whether those changes had the effects you intended.  Evaluation, like so much of community work, is a process that goes on as long as the work itself does.  It's absolutely essential to the continued improvement of your program.

In Summary

Choosing evaluation questions - the areas in your work you'll examine as part of your evaluation of your program - is key to defining exactly what it is you're trying to accomplish. For that reason, those questions should be chosen carefully as part of the planning process for the program itself, so that the questions can guide your work as well as your evaluation of it. The more that stakeholders can be involved in that choice and planning, the more likely you are to create a program that successfully meets its goals serving the community.

Choosing those questions well entails understanding the context of the program - the community, participants, the culture of any groups involved, the history of the issue and of the social structure of the community and the organization - and (if you're an outside evaluator without ties to the program) establishing trust with administrators, staff members, and participants. That trust will enable you to conduct a participatory evaluation that draws on the knowledge and talents of all stakeholders, and to plan an evaluation that fits the goals of the program and accurately analyzes its strengths and weaknesses. With that analysis in hand, you'll be able to make changes to improve the program. Then you're ready to start the whole process again, so you can evaluate the effects of the changes you've made.

What do we mean by choosing questions?

___Evaluation questions are the questions your evaluation is meant to answer about your work

___Evaluation questions help set the direction of the work, as well as assess its effectiveness

___Ideally, choosing evaluation questions is part of the planning of the overall program

Questions to ask yourself as you choose evaluation questions:

___What do you want to know?

___Why are you interested?

___Is the issue you’re addressing important to the community or to the society? 

___How does the issue relate to the field?

___Is the issue general, rather than specific to your population or community?

___Who might use the results of your evaluation? 

___Whose issue is it?

Why is it necessary to choose evaluation questions carefully?

___It helps you understand what effects different parts of your effort are having

___It makes you clearly define what it is you’re trying to do

___It shows you where you need to make changes

___It highlights unintended consequences

___It guides your future choices

___In participant evaluations, it involves stakeholders in setting the course of the program, thus making it more likely that it meets community needs

___It provides focus for the evaluation and the program

___It determines what needs to be recorded in order to gather data for evaluation

When should you choose questions and plan the evaluation?

___If possible, choosing questions and planning the evaluation should be an integral part of planning your program

___If your reality makes that impossible, choosing questions and planning the evaluation should take place as soon as possible after the program starts

Who should be involved in the process?

___To the extent possible, the process should involve all stakeholders, including program participants and beneficiaries

How do you choose questions?

___Describe the issue or problem you’re addressing

___Describe the importance of the problem

___Describe those who contribute to the problem

___Assess the importance and feasibility of changing those behaviors

___Describe the change objective

___Make sure that the expected changes would constitute a solution or substantial contribution to the problem

How do you plan the evaluation?

___Take into account the issues raised by multiple or very different settings

___Take into account the issues raised by participant groups that differ in culture, ability to complete the program, geographical location, and other factors

For outside evaluators, specifically:

___Choose a setting

___Learn as much as you can about the organization you’ve chosen

___Contact the appropriate person(s) and request an interview

___Plan and prepare for the initial meeting

For all evaluators:

___Find out all you can about the context

___Establish trust with program administrators, staff, and participants

___Aim for a participatory evaluation

___Plan the evaluation in collaboration with stakeholders

Consider all the elements of an evaluation in your planning:

___Information gathering and synthesis

___Designing an observational system

___Developing and testing a prototype intervention

___Selecting an appropriate experimental design

___Collecting and analyzing data

___Gathering and interpreting ethnographic information

___Collecting and using archival data

___Encouraging participation throughout the research

___Refining the intervention based on the evaluation

___Preparing the evaluation results for dissemination

Example 1: Using Miro Board for Collaborative Development of Evaluation Goals and Questions

 

Noah Goodman shares the process he used to prepare for and conduct a kick-off meeting to develop evaluation goals and questions for the Right Question Institute.

Integrating Technology in Evaluation TIG Week: Using Miro Boards to Engage Clients in Collaborative Development of Evaluation Goals and Questions by Noah Goodman

Suppose you wanted to design a house that used very little energy, took few resources to build and maintain, and was affordable for most families. You might have some original ideas about how this could be done, but you’d want to find out what ideas others had as well. You’d probably read about earth-bermed houses (houses that are built into a hillside or earth mound), solar panels or windmills for producing electricity, efficient insulating windows, waste-water recycling, and non-toxic building materials that reuse waste wood and metal. You’d talk to people who built or owned energy-efficient houses, to hear about the realities of living green. You’d learn about the barriers to some environmentally-friendly strategies, as well as ways to get around those barriers. There’s a huge amount of information out there, and it would make sense to gather as much of it as possible, so that you could put together the information, incorporate appropriate elements into your design, and get new ideas based on what’s already been done.

The same is true if you’re designing an intervention or program to deal with a community health or other issue, or an evaluation of that program. Others have also undoubtedly tried to address that issue, some with success and some without. Knowing what they did, how they did it, and what the results were can help you decide how to design your effort. You might be able to find a method here, and a technique elsewhere that all fit together into exactly the program that will suit the people and conditions in your community. Or you might realize that something you’d intended to do simply hasn’t worked in a number of other instances, and so wouldn’t be likely to work for you, either.

Gathering and using others’ ideas doesn’t mean that you can’t use your own or come up with something new. New ideas tend to come out of what others have attempted. Most artists start out imitating others before they develop their own styles. Einstein didn’t just chance on relativity; he was familiar with it because others had worked on it. You can usually innovate more effectively if you know what’s been tried.

This section looks at gathering all the information you can about your community issue and about attempts to address it, and putting that information together to design an evaluation to address your questions. Although this chapter is about evaluation, much of the material in these sections applies to planning the intervention (or program) and the evaluation: the two really can’t be separated.

An evaluation is a research project: we are trying to discover what works and under what conditions.  The steps for designing and using an evaluation – the subject of this chapter – are essentially the same as those for designing the program you’re evaluating.  The elements that you borrow from others’ successful efforts, and those that you create yourself, will give you an intervention and related evaluation questions. Although this section talks about program design, it also applies to the design of the evaluation.

What do we mean by information gathering and synthesis?

Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it. The more information you have about the issue itself and the ways it has been approached, the more likely you are to be able to devise an effective program or intervention of your own.

There are obviously many sources of information, and they vary depending on what you’re looking for. In general, you can consult existing sources or look at “natural examples,” examples of actual programs and interventions that have addressed the issue. We’ll touch on where to find both here, and then go into more detail about them later in the section.

• Existing sources. This term refers to published material of various kinds that might shed light either on the issue or on attempts to deal with it.  These can be conveniently divided into scholarly publications, aimed primarily at researchers and the academic community; mass-market sources, written in a popular style and aimed at the general public; and statistical and demographic information published by various research organizations and government agencies.
• Natural examples. These are programs or interventions developed and tried in communities that have addressed your issue. Studying them can tell you what worked for them and what didn’t, and why. By giving you insight into how issues play out in your or other communities, they can provide nuts-and-bolts ideas about how to (or how not to) conduct a successful program or intervention. For the most part, information sources here are the people who are involved in efforts to address issues similar to yours, or those who can steer you to them. Additionally, there are a number of natural examples (such as single case studies) that have been written about descriptively in the literature of community psychology or public health that may be relevant to your work.

Synthesis is from the Greek; it means putting together. Its English meaning is the same: the putting together of something out of two or more different sources. Synthetic fabrics, for instance, are called that because they’re constructed from a number of different chemical building blocks.

In this section, we’re talking about ideas. Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something new that meets the needs of the community and population you’re working with.

Synthesizing in this way requires identifying the functional elements of each idea or program that you’ve looked at that seems to hold lessons for your work. Functional elements are the core components of each program – the methods, framework, activities, techniques, and other aspects – that make up the specific program you’re examining.  Once you’ve separated these parts out, you can put those that meet your needs together with what you’ve learned about the issue and your own ideas to build a program that speaks specifically to your situation.

As we’ve mentioned, the activities of information gathering and synthesis are needed both to create the original program and to develop an evaluation of it that will help you maintain and improve it.  The two really start in the same place, with what you think will address the issue – what shape the program or intervention should take, with whom it should be applied, and what behaviors or conditions it aims to change. This also informs what its short- and long-term goals should be, and by what means you’ll try to achieve those goals.  Once these are determined, they in turn determine your evaluation questions. You can’t construct an evaluation without knowing exactly what you’re trying to evaluate.

Why gather and synthesize information?

If you’re in the process of starting a program to address a community issue, such as violence or early childhood education, you probably know quite a bit about that issue already.  You’ve dealt with it, perhaps, in a variety of ways, and you have some pretty good ideas about what kind of program would work.  Why take the time and trouble, for you and for others engaged in a participatory planning effort, to read a lot of material written by others and to track down people who’ve run programs? If you’re inclined to think this way, there are a lot of good reasons why you should think again. Gathering information beforehand and putting together what you’ve learned could be the most important things you do to make your program effective.  Here’s why:

• It will help you avoid reinventing the wheel. A lot of different organizations have likely approached this issue before you.  Some might have been successful and some might not have, but all of them have probably learned something that would be useful to you in the process. You don’t have to make the same mistakes someone else did if you know about them, and you don’t have to make up something from scratch that may or may not work, when you have a model that has worked.

It’s certainly not a bad thing if you have some of the same good ideas that others have had, but it helps to know that they are good ideas. And there’s a chance that you might have some of the same bad ideas others have had, in which case it helps even more to know that they’re bad ideas.  It will save you a huge amount of trouble, and perhaps be the difference between creating a program that does its job well and one that fails miserably and disappears. Square wheels don’t roll – someone could have told you that.

• It will help you to gain a deep understanding of the issue so that you can address it properly. The first step in figuring out how to deal with an issue is to know what you’re dealing with. The better you understand it – its causes, how it occurs, how people react when they’re affected by it, what its consequences are for individuals and the community, and who can influence it – the more likely it is that you’ll be able to determine how to approach it.
• You need all the tools possible to create the best program you can. Foremost among the tools you need to plan and implement a program or intervention are information, information, and information. Just as with the issue itself, the more you know about what works for whom, how to make things happen, and how to establish or eliminate certain conditions, the more likely that you’ll be able to plan a successful program that addresses all aspects of the issue and leaves nothing to chance. Various kinds of professional and interpersonal skills may help you implement a program, but if what you’re implementing isn’t effective, it doesn’t matter how skillfully you carry it out.
• It’s likely that most solutions aren’t one size fits all. The more information you gather, the greater the variety of approaches, methods, and frameworks you’ll have to choose from. Putting together the right combination will help you to successfully address the particular needs of your community and population.
• It can help you to be culturally sensitive. Not only can you learn more about the culture(s) of the people you’re working with, but you can probably find a number of approaches that have worked with the cultural group you hope will benefit. Perhaps even more important, you can learn to avoid costly mistakes that may take a lot of time and effort – or be impossible – to repair.
• Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program. As we discussed at the beginning of this section, new ideas seldom spring from nowhere.  They’re stimulated by your own experience and the ideas and experience – both good and bad, positive and negative – of others. Look to the experience of other fields, communities, and countries. The more different ideas you’re exposed to, and the more ways you can put them together, the greater chance there is that you’ll come up with something new that’s more effective than what’s gone before.

When should you gather and synthesize information?

Information gathering and synthesis is crucial to the success of the program and to the relevance and effectiveness of the evaluation. It should start at the beginning of any effort, and contribute to the initial planning. It should also go on throughout the life of the program, so that you can continue to adjust by adding or changing program elements to enhance outcomes, and to generate new ideas.

Major adjustments should generally come at the end of an evaluation cycle, when you have solid information about what worked and what didn’t. That doesn’t mean that you can’t make smaller adjustments in the course of the program to improve results along the way.

There’s a tension here between continually changing a program to make it better and obtaining accurate evaluation results. If you change a method or activity in midstream, your evaluation will not be able to give you a clear assessment of its effectiveness.

How much changing you do in the course of a program depends on your intent.  If your first responsibility is to find out what works best, so you can pass it on, then it’s important not to make changes until an evaluation has been completed.  If your primary responsibility is to the current participants in the program, then you should make whatever changes are necessary whenever they’re necessary to ensure the best outcome for them.

There can be ethical issues involved here.  In medical experiments with new therapies or drugs, for example, some participants are given the new treatment and others aren’t (all participants consent to this arrangement, and to not knowing which group they’ll be assigned to.)  If the new treatment proves to be harmful, there is an ethical obligation for the researchers to stop administering it.  If, on the other hand, it quickly proves remarkably effective, researchers usually feel ethically bound to extend it to others in the study as soon as they can prove its positive effects.  Not all programs necessarily pose ethical problems that are as clear-cut as those encountered in medical studies, but ethical issues should always be considered.

Who should gather and synthesize information?

The assumption throughout this chapter is that the whole process – planning, design, implementation, and evaluation – involves multiple stakeholders.

Typical stakeholders in a community program or intervention might include:

• Program participants or beneficiaries
• Program staff and administrators
• Others affected by the program – police, medical staff, teachers, etc.
• Academics or other researchers
• Local officials
• Community activists

Information gathering

In a participatory process, information gathering can be enhanced by a division of labor determined by the skills and experience of the participants. If there are academics or other professional researchers involved, it would probably make the most sense for them – or others with research experience – to review the evaluation literature. Members of the affected population might be the best ones to collect information about the history of the issue in the community, and about how it currently affects people. Program directors and staff would probably have the best contacts in the field, and thus the best chance to find information about other similar programs. Those with Internet access and computer experience might be the logical on-line searchers, or might act as technical support for others to help them find what they’re looking for. Those with knowledge in the law and legislation might be the ones to examine policies.

There’s also the possibility that training could be provided to the whole group, or to various individuals to allow them to pursue various lines of inquiry. There’s no reason, for instance, that people without research experience couldn’t learn to understand and interpret demographic information or contact programs in other places. (There are some limitations here: levels of related education, materials or computers, and/or inability to connect with other people might all figure in to what kind of research it makes sense to ask others to do.)

Synthesis

It is especially important that all participants in the process be involved in putting together the information. Training new participants to synthesize information will pay dividends in the end, because they may be able to see things in the information that aren’t obvious to experienced researchers. They may know things about the community that shed light on which elements of other programs might be appropriate and which might not.

In any case, information gathering and synthesis, like any other part of the process, should reflect the needs, interests, and abilities of all stakeholders.

How do you gather and synthesize information?

There are a number of steps to gathering and putting together the information you need. Most of these can be group activities, part of the participatory process. The actual information gathering can be parceled out to specific individuals or sub-groups.

Decide what you need to know

Not surprisingly, the first step in gathering information is determining what information to gather. There are a number of areas to explore:

• Details about the issue. These might include its immediate and root causes; its general effects on individuals and communities; its consequences; its development through different stages; its history; and the history of attempts to address it.
• How the issue has been dealt with elsewhere. Best practices or approaches for which there is an evidence base; other approaches that have been at least partially effective; and what hasn’t worked, which may give you at least as much important information as what has.
• People who can help. This category encompasses experts in the field and people or organizations that have run or been involved in successful attempts to address the issue.
• Who is affected locally, and how. This really comprises two questions: a) What population groups – geographical, ethnic, cultural, racial, class, etc. – are particularly affected by this issue?  and b) What other groups are affected, but less visibly? These might include those who work with the first group(s) in the community (teachers, for example, or social workers), those who depend on them, and those on whom they depend.
• The importance of the issue to the community. Again, this implies a double question:
  • How important does the community perceive the issue to be? and
  • How much and in what ways does the issue actually affect the community as a whole?
• Community needs related to the issue. What has to be added to or removed from the community in order to improve the situation? What kinds of approaches will the community respond to or reject?
• Other context information. Community history, relationships among groups and individuals that might be relevant to your work, community culture, etc.
• Who, if anyone, has some influence or control over changing the situation. Public officials and other policymakers are often in this position. Business leaders, landlords, government enforcement agencies, schools, employers, hospitals and health personnel, and members of the affected group itself might also be in the position to change the situation (by learning new skills or changing practices).

Determine your likely information sources

As mentioned above, these encompass existing (i.e., published) sources and natural (i.e., experiential) examples. Published sources can be divided into scholarly, mass-market, and statistical, each of which can provide different information and a different perspective on the issue and attempts to address it.  Depending on what you decide you’re looking for, you might use all, or any combination, of these sources.

The single largest storehouse of information available is the Internet. Many scholarly articles are published online and accessible – often free, sometimes for a fee – to anyone who’s interested. Virtually all U.S. laws and regulations at every level of government are easily found, most on several websites. General knowledge on just about anything is widely available, as are lists of best practices and successful organizations and the websites of those organizations. Census data and other similar statistical information are also on view.  Add to these the information provided by such all-encompassing sources as Wikipedia (recently, for all its quirks, found to be just about as accurate across its million-plus entries as the Encyclopedia Britannica), and you have a nearly-bottomless well of fact and opinion to draw from.

As always, you have to be cautious: most of cyberspace is unedited, and the quality of information varies.  If you stick to reasonably reliable sites, you’re likely to find almost whatever you need, or at least directions to it.

Existing sources

Scholarly sources might include:

• Academic and some professional journals
• Books written for the academic market
• Doctoral dissertations - these are accessible to researchers through university libraries and some Internet sources
• Papers and reports delivered at academic and professional conferences - these are often available online, either on the authors’ websites or in e-published conference proceedings
• Occasional articles in respected mass-market scientific magazines, such as Nature or Scientific American
• Newspaper archives
• Direct contact with academics and other researchers who’ve done work on the issue you’re interested in, or who have conducted studies of attempts to deal with it
• Internet listservs and news groups relating to the issue or the field in question

Mass-market sources of information:

• Widely available books, often marketed as “self-help” or “life-changing,” to the public at large
• Articles in popular magazines, both those devoted to science or behavior and those of general interest
• Newspaper stories, often in Sunday magazine sections

Where to find statistical and demographic information:

• Census data - available on the web and at many libraries
• Community reports, such as community report cards, self-studies, and needs assessments, all of which should be obtainable through the appropriate municipal offices, and sometimes on the web as well
• Organizational and agency data, usually a matter of public record if the agency is public or publicly funded

In addition to these sources, the broadcast media often present stories about critical issues or about successful efforts to address them. In most cases, such stories only skim the surface, since they have to fit into short time slots (public broadcasting, on both radio and TV, breaks this mold more than other media outlets).  They can, however, serve as introductions to further research, raising the importance of one or more aspects of an issue, or providing information about effective programs that you can then contact.

Natural examples

Some of the more likely sources of natural examples:

• Program directors
• Friends or colleagues in the field
• Funders (particularly public agencies, because their transactions, including whom they fund and why, are a matter of public record)
• Leaders and members of community coalitions or partnerships
• Officials who coordinate community-wide efforts
• Members of the population most directly affected by the issue at hand
• Current or former participants in or beneficiaries of effective programs
• People who work in collaboration with programs – police, medical staff, teachers, etc.
• Key informants in the community
• Experts – some of them the same academics and other researchers referenced under scholarly sources – who have experience with your issue and efforts to address it
• Your own experience in the community

Don’t be afraid to range far and wide in your search for successful models or new ideas.  Step outside your own field and your own region, and see what’s been done elsewhere.  A model from social work or urban design might work in public health, or vice-versa.  There’s enough overlap among fields that deal with human health and development that you can often find exactly what you need in seemingly odd places.

Devise a plan for collecting information

There are a number of considerations here:

• Who will gather what information? As we’ve discussed, the ideal group is multi-sectoral and diverse in backgrounds and skills.  Information gathering should be assigned according to participants’ skills, interests, and contacts in the community. We’ve suggested, for instance, that scholarly sources might be mined by academics or other experienced researchers, while members of the affected population might be more successful in approaching key informants in the community. This doesn’t mean, however, that in a given group, these and other apparently logical roles couldn’t or shouldn’t be varied, depending on the individuals involved.
• How will the information be gathered? Another issue is just how the information will be gathered. Finding and reading written material is relatively straightforward: it’s in the library or on the web, and you can read and take notes on the relevant parts of it. Getting information directly from other people, however, can be more complicated. Will you engage in formal or informal interviews? In observation? Will you conduct surveys or public meetings? How will you contact people you don’t know – by letter, by phone, through mutual acquaintances? Your information-gathering methods will be determined by how much time you have, exactly what information you need, the depth of the information you need, and the abilities of the participants.
• What adjustments will be made for particular gaps in experience or skills?  People who don’t read, write, and/or speak the language proficiently may have to devise imaginative ways of recording information. Experienced researchers may have to translate scholarly writing for just about everyone in the group who isn’t an academic. In many cases, most or all of the group may need orientation or training before information gathering can begin. You’ll need to work out what the needs are as a group, and devise ways to meet them.
• What’s the timeline for information gathering? While information gathering should continue throughout the life of a project, the initial phase should have a time limit, so that action isn’t delayed for too long. The time limit depends on your time constraints, the seriousness and intensity of the issue, the community’s perception of urgency, and whether there are external time restrictions (student interns who are only available until the end of the summer, for instance.) Having a clear deadline will focus the group’s activities, and boost its efficiency.

Collect information

When your plan is completed, it’s time to put it into practice. You’ll have to conduct any trainings that are necessary, and make sure that all the relevant tasks are assigned appropriately. You may also want to set up regular meetings throughout the information-gathering process, in order to give the group the chance to review progress, make suggestions, and report on what they’ve been finding.  In addition to providing support for those new to research, these meetings, by providing a preview of the results of the process, will save everyone having to digest an overwhelming amount of information all at once.

Synthesize: Take it all apart

The process of synthesis involves breaking the information down into its component parts, sifting through those parts to see which fit together best for your situation, and then integrating them into an approach that is likely to work in your community.

There are usually three major areas to be considered:

• What’s known about the issue itself. What personal and environmental factors contribute to the problem? What are its root causes? Do you have the resources to address them, or are they beyond your scope (e.g., global economic forces or climate change)? Does the issue have a number of different effects, and if so, what are they?  What are the likely consequences for the community as a whole if the issue is not resolved?  (An environmental health risk can not only kill or sicken individuals, but might also affect business productivity, insurance availability and rates, hospital costs, the housing market, or even – as in the case of the Love Canal neighborhood in Niagara Falls, NY – the existence of a neighborhood or community itself.)
• The community context of the issue. What are the specific local effects of the issue. Exactly who is affected?  Exactly how are they affected? What are the consequences for those individuals? For their families, friends, neighbors, and others they have dealings with? For the community as a whole?  What has been the community’s experience with this issue in the past?  How, if at all, has it been addressed?  What local conditions would change if the issue were addressed, and how would they change?  Are there underlying conditions that have to change before the issue can be addressed?  Whose attitudes and/or behaviors need to change to have an effect on the issue (for example, among policy makers, those affected, or specific officials)?
• Successful and unsuccessful attempts to address the issue. These may have been gleaned both from the literature on best practices, and directly or at second hand from those involved in them. Here, it’s important to separate out the elements of various approaches. What specific procedures – methods and intervention components – were used? What kinds of training – feedback, role play, modeling, etc. – were provided to participants?  Was information provided to participants about when, why, and how to act? Were there positive or negative consequences that helped to establish or maintain change (or its opposite)?  Were environmental barriers, policies, or regulations put in place or removed?  What was the overall philosophy behind the approach?  What aspects of the issue did it address? What kind(s) of community was it tried in?  What population groups (in terms of culture, age, social class, etc.) were involved?  Who was the approach to benefit?  What were the specific results in the short term?  In the long term?  What makes a particular program, policy, or practice successful or unsuccessful? What events, if any, were critical, to success (or failure)?  What conditions – organizational features, participant characteristics, broader environmental factors – were critical?  Is there a model successful program? Is there a model unsuccessful program?

The existence of a model unsuccessful program doesn’t indicate that if you do the opposite of everything that program did, you’ll be successful.  Even if it failed spectacularly, much of the program may have been potentially effective, but one or two elements – the way participants were approached, recruited, or treated, a particular method – negated what could have worked.  By the same token, most elements of the program may have been fine, but its basic premise might have been mistaken or ineffective – “Just say no” as a way of preventing AIDS among teens, for instance.  It’s important to try to figure out why the program was unsuccessful.  A true model unsuccessful program is one that did everything wrong, but those are few and far between.

Lisbeth Schorr (Common Purpose) makes a useful distinction between “what works” and conditions under which what works actually works.  Sometimes the presence of a charismatic leader or champion motivates staff and/or participants to succeed. When the leader or original staff members leave, some such programs collapse, while others are able to renew themselves by careful hiring and a faithful implementation of what made it work.

In looking for programs to draw from, you need to understand the intervention components and elements that make those programs work.  Also, try to understand the conditions that allow an intervention to be successful.

Synthesize: Put it back together

Analyze the elements you’ve found to determine which of them would be appropriate for the situation and group you’re working with.

• What has been used specifically with your population in your circumstances?  Have the successful programs you’ve looked at been context-specific (i.e., intended for their specific communities and populations)? Can they be adapted to your context if they weren’t intended for it?
• What can be adapted, if it wasn’t originally aimed at your population?  (Techniques used with children or adolescents that could be modified for use with adults, for instance, or vice-versa.)
• What’s missing?  What aspects of the issue in your community are not addressed by what you’ve found?  Are they important enough that they need to be addressed?
• Does what you’ve found confirm or contradict what you thought you already knew?
• Are there factors in your particular situation that make the issue substantially different for you and your participants than for any other programs or approaches you’ve found out about?  How will you deal with that?
• What does your information tell you about the possibility of successfully addressing the issue’s root causes (e.g., income inequality, social exclusion, lack of power)?
• In general, did most or all successful programs direct their change efforts at the same group of people (policy makers, for example), or was there a variety?  If the latter, what do you think is most likely to work in your community?
• Perhaps most important, what’s your definition of success, and which of the programs you learned about came closest to achieving?  What components and elements of those programs addressed what’s needed in your community?

Answering these questions will give you a good sense of which components of other programs may work for you, and should also fit with what you already know to either give you ideas for new elements that you can add, or confirm (or warn you away from) ideas for new elements that you had already.

We don’t want to imply that simply taking a lot of different program components and playing mix-and-match will provide you with an effective way to address a community issue. You have to start with a clear framework informed by your vision and mission, and put together a program that’s coherent and makes sense.  All the elements have to fit; if they fit well enough, you’ll end up with a whole that’s greater than the sum of its parts. If the elements don’t fit together, or aren’t part of a program with a well-defined framework, the chances are you’ll end up with a mess.

Keep at it

Information gathering and knowledge synthesis should continue throughout the course of the program. While you may wait until the results of an initial evaluation to change something, you should always be looking for improvements and better approaches. No program or effort is perfect: everything can be improved. As long as you keep trying to learn more and grow in your understanding of your work, it will continue to get better. If you become complacent (i.e., you feel you know what you’re doing and can relax), your program may start to lose its effectiveness.

In Summary

Gathering the information that already exists about your issue and attempts to address it is one of the most important aspects of planning a program or evaluation. By putting together what’s known about the issue and the history of the successes and failures of various approaches to it, you can build a program structure that includes your own innovations and elements that have worked for others in similar situations. This synthesis also allows you to avoid ineffective approaches and to incorporate ideas and methods that have been particularly appropriate, culturally or otherwise, to the population and community you’re working with.

Information gathering and synthesis should continue throughout the life of the program. The more information you have, and the more carefully you put it together, the better your chances of implementing a successful program.

What do we mean by information gathering and synthesis?

___Information gathering refers to gathering information about the issue you’re facing and the ways other organizations and communities have addressed it

___You can gather information using both existing sources and natural examples

­­___Synthesis here refers to analyzing what you’ve learned from your information gathering, and constructing a coherent program or approach by taking ideas from a number of sources and putting them together to create something that meets the needs of the community and population you’re working with

___Synthesis involves extracting the functional elements of both the analysis of the issue and approaches to it

___Functional elements are those that are indispensable either to understanding the issue, or to implementing a particular program

Why gather and synthesize information?

___It will help you avoid reinventing the wheel

___It will help you to gain a deep understanding of the issue so that you can address it properly

___You need all the tools possible to create the best program you can

___It’s likely that most solutions aren’t one size fits all

___It can help ensure your program is culturally sensitive

___Knowing what’s been done in a variety of other circumstances and understanding the issue from a number of different viewpoints may give you new insights and new ideas for your program

When should you gather and synthesize information?

___Information gathering and synthesis should continue throughout the life of the program

Who should gather and synthesize information?

___Information gathering and synthesis is often most effectively conducted by a multi-sectoral participatory group including all stakeholders in the issue

How do you gather and synthesize information?

___Decide what you need to know about the issue itself, successful and unsuccessful attempts to address it in various circumstances, and the local context

___Determine your likely sources for the various types of information you’re seeking

• Existing sources include scholarly, mass-market, and statistical/demographic published information
• Natural sources include some published information about programs, but can best be obtained by direct contact with those involved in planning, implementing, or participating in programs relevant to your issue
• It’s important to pay attention to both successful and unsuccessful attempts to address the issue, and to step outside your own field in search of solutions that work

___Devise a plan for gathering information

• Decide who will gather what information
• Decide how information will be gathered
• Decide what adjustments will be made for gaps in experience or skills
• Set a timeline for the initial information gathering

___Collect information

___Begin synthesis by taking it all apart – extract the functional elements of what you’ve learned

___Complete synthesis by putting the relevant pieces back together as a coherent program that speaks to your community’s needs

___Keep at it by continuing to gather and synthesize information throughout the life of the program

• What do we mean by an observational system?

• Why design an observational system?

• When should you design an observational system?

• Who should design an observational system?

• How do you design an observational system?

The local community health center was starting a program to encourage regular physical activity among people with high blood pressure. The program had one simple objective: to engage participants in 45 minutes of regular, moderate aerobic exercise at least four times a week over the course of six months.  The hope was that this regimen would lower participants’ blood pressure, and lead to weight loss and an overall sense of greater well-being. A related aim was that participants would continue the exercise on their own after the program ended.

The Center had gathered a group of 50 people who were willing to take part. After physical checkups, all the participants attended workshops on diet, the mechanics and dangers of high blood pressure, and how to start and maintain injury-free regular exercise. They also received counseling about the kinds of exercise they might undertake – walking, bicycling, swimming, etc. – and about ways to make exercising pleasant. To help participants to integrate exercise into their lives, the center decided to ask them to exercise at their own convenience, using whatever activities they chose, as long as they maintained the 45-minute, four-day-a-week pattern. They were also asked to keep journals of the frequency and nature of their exercise, and to meet, in groups of five, with Health Center counselors once a month for checks on blood pressure, weight, and progress, as well as support, encouragement, and advice.

The center then had to decide how to evaluate the program. The performance goal of the program – what participants were actually supposed to do – was maintaining the exercise schedule over the six-month period. Since each participant was tending to that individually, it would be hard to actually watch them all exercise whenever they did so. But the center needed to know whether or not they had. The other program goals – lower blood pressure, good weight loss – also had to be observed somehow.  How could the center design a system to find out whether the behavior of physical activity was occurring and whether this resulted in lower blood pressure and weight loss?

Once you’ve determined your evaluation questions and gathered information about what to look for, you have to find a way to look for it.  That’s what observation systems are all about. Like the center in the example, you’ll need to find ways to observe the behavior, the conditions, and the changes – or lack of changes – in them that will answer your evaluation questions. This section is about setting up observation systems to do just that.

What do we mean by designing an observational system?

An observational system is the way you get information about your program – what it and its participants and implementers are actually doing, and what seems to be occurring as a result.  “Observation” here may mean actual observation – watching people, conditions, activity, or results to see what happens – but it may also refer to less direct ways of monitoring a program’s operation and outcomes.  Its varieties include monitoring the behavior of individuals and groups to see the results at different levels.  Some methods of observation that might prove useful in different evaluation situations:

• Direct observation. This is the purest and most verifiable form – watching people or observing conditions or situations firsthand. If you’re involved in an effort to increase the use and neighborhood sense of ownership of a public park, for instance, you might directly observe how much and how people use the park by visiting and observing on different days, in different types of weather, and under different circumstances over a substantial period of time. Direct observers may be “invisible,” as an observer of park activity would probably be, or they may be staff members who work with participants, recording what happens.  In either case, they are taking measures as outside observers, not as participants themselves.
• Participant observation. A participant observer becomes part of the action, and observes as an insider. In the case of the park, a participant observer might be a neighborhood resident directly involved in the effort, or might be someone who becomes part of the life of the park for the purposes of observation. He might jog there daily, or join a weekly volleyball game and get to know others who use the park on a regular basis. His own notes about what is observed in the park might also become part of his recording.
• Self-reports. Some of what you’re trying to achieve may simply not be visible at all, at least not to you. Changes in what people do in private, such as their use of contraceptives, may not be (or should not be) observed directly by an outsider. Similarly, when the goal is to affect changes in the behavior of large numbers of people, such as to promote healthy eating in the community, it will not be feasible to directly observe this for everyone.  In such situations, we ask people to report on their own behavior Thus, an observational system may include interviews, journals, surveys, or other means of first-person reporting. Since such reporting may be subject to bias, we usually try to also use other forms of evidence (e.g., observing weight loss as a product of the behaviors of health nutrition and physical activity).
• Second-hand reports. An observational system may include or depend on the reports of others who have direct experience with the people or conditions you’re concerned with. Teachers, probation officers, park rangers, public health nurses, social workers – even bartenders or hairdressers – might be valuable sources of second-hand information. These reports, like self-reports, may be gathered by interviews, journals, surveys, checklists, and the like.
• Electronic or mechanical observation. The observer in this case isn’t a person (although ultimately people would review its information), but an automatically-operated or always-on camera, audio recorder, heart monitor, pedometer, GPS (global positioning system) tracker, or other piece of equipment.  A camera operated by a tripwire is often used, for example, to study the density of an animal population in a particular area, or along a particular path.
• Tests of various kinds. Depending on what you’re measuring, this category could cover everything from pencil-and-paper tests of academic learning to hands-on skills tests to blood tests and the like. They might also include tests of new program methods and procedures to see if they work before putting them into practice.
• Public and other records. Police reports, census data, employment statistics, public health information – all of these and more could give you information on community-level indicators that will help you determine the outcomes of your work.
• Products or results of behavior. Sometimes it is more practical to observe the product or result of a behavior, rather than the behavior itself. For instance, if interested in environmental pollution, we might observe the amount of debris or toxins on the ground or in the water, rather than the behavior of illegal dumping of toxins or materials. Similarly, an initiative interesting in preventing childhood obesity might use school records of height and weight to measure obesity – in addition to direct observations of school lunches and what youth report on eating surveys.

In addition to specifying what kinds of observation you’ll use, the design of an observational system should also cover when, where, how often, by whom, and under what circumstances observations will take place, as well as just what will be looked at. All of these depend on what you’re observing and what information you hope to gain from your evaluation (back to those evaluation questions again).

Among other considerations, will you look at the process of your effort or program – the steps you took in setting it up, and whether they were faithful to what you intended?  Will you look at what you actually did – the number of participants you had, the methods you used, the time everything took, how long participants stayed, etc.? Are you interested in which parts of what you did were successful and which were not?  And what do you want to know about outcomes – the results of your program?

Designing an observational system entails thinking carefully about what you need to know, and creating a system that is most likely to get you that information as accurately and easily as possible. We’ll discuss the design process in detail, including the issues mentioned in the last two paragraphs, in the “how-to” part of this section.

Why design an observational system?

If you’re serious about evaluation, there are a number of reasons for designing a good observational system:

• It can help you get reliable information. Designing a system that standardizes the methods, times, and other aspects of the observation will mean that the information you get from different observers and places is likely to be accurate and consistent, and therefore more useful to an overall evaluation.
• It can help you find out exactly what you need to know, without wasted effort. You can design a system that examines what you’re interested in, and ignores what you’re not. That means that you don’t have to sort out unnecessary data, and that you’ll have the right means of collecting the data you need to address your evaluation questions.
• It can ensure that observations are made. A consistent system that’s designed and accepted by those who will do the observing, whatever form it takes, makes it far more likely that observations will be made when, where, and how they’re supposed to.
• It can make it easier to analyze your data. A consistent, rational system of observation can give you good information for scientific analysis, whether that analysis is quantitative (based on numbers and statistics) or qualitative (based on narrative and interpretation of the meaning of behavior and events.)
• It can help you avoid haphazard evaluation. A well-designed observational system will allow you to collect information systematically, and not leave you with a mass of disconnected data that are not necessarily related to what you want to know.
• It will make it easier to justify your findings. The more accurate your information, the more reliable the conclusions that can be drawn from it. If your observational system is designed and implemented well, it’s much easier to argue that your information is reliable and accurate, and a good base for the conclusions you reach.
• It can help you gain credibility with funders and policymakers. The people who control funds and policy are particularly concerned with accountability. If you can present them with a useful evaluation based on data collected through a well-designed and reliable observational system, they’ll be more inclined to treat you as a knowledgeable voice in the field.
• It can let you pass on your practices with confidence. A well-designed observational system makes it possible to feel that your evaluation results tell the truth. If those results show that your program is highly effective, you can pass on what you do as a best practice to colleagues in the field and others, without worrying that you may be urging them to use methods or assumptions that might not work very well.
• It can give you the best information possible about what’s working in your program, and what you need to adjust.

When should you design an observational system?

As we’ve discussed, an observational system refers not only to direct observation, but to any method of examining and recording the process, activities, and outcomes of your program. An observational system is intrinsic to your evaluation, since that is what will tell you what actually went on. Therefore, the ideal is to design that system before you actually start implementing the program, so that you can monitor the program throughout its existence.

That’s the ideal. The reality for many community workers – especially those who work in small, community-based organizations – is that evaluation begins whenever the time, energy, and resources are available, which is often months or years after the program has started. Whenever it begins, the observational system should be designed to fit the evaluation questions you’re asking. Because the observations must be consistent and reliable it's well worth taking the time to make sure to design an effective evaluation system.

It’s best if you can observe through a whole program cycle, from beginning to end. Some programs don’t have a cycle, and the observation may focus on the behavior or results of individual participants rather than the program as a whole. In these situations, evaluation may begin as new participants begin the program and are observed from the beginning.

If you’ve been recording events, keeping journals, etc., before you start evaluating, you may have information that you can incorporate into the results of your observation. If your design calls for specific firsthand observations, their definition may be precise enough that similar observations recorded in journals kept by staff may not meet the criteria to be included.  If staff journals or records are part of the system you’re putting in place, however, you may be able to use all or most of the information you have (and, indeed, you could design the observational system so you can.)

The real danger here is that you’ve missed something important already by the time your observational system is operational.  It may be that the early part of the program is crucial, at least for some participants or for some changes, and you’ll be starting to observe after those changes have been made.  Start your observations early so that your system can pick it up.

Who should design an observational system?

We’ve stated many times the Community Tool Box bias toward participatory process, and particularly toward participatory research and evaluation. In the case of an observational system, a system will function best if it’s designed by a group that includes those who will actually be the observers. If they’re part of the planning, they’ll be familiar with the system, know exactly what information it’s meant to observe, and understand their roles with the observational system.

In a community-based or smaller organization, it’s likely that time will be a factor – there are probably too few people already doing too many jobs. If that’s the case, the level and nature of the observational system has to be one that the staff can actually handle, whether they’re the observers, or whether they’re facilitating the observation for outside evaluators or volunteers. If they help to plan and set up the system, they’ll have far more incentive to make sure it works than if it’s imposed on them.

The design of the observational system should specifically include the people who will actually do the observing, who are often either staff members or members of the group that will benefit from the program. In addition to these, it helps to include researchers or others who understand observational systems, and can help to design a system that specifically meets the needs of the evaluation or research project. It might also be beneficial to include individuals from the group(s) that will be observed, to help with cultural issues and provide feedback on their response to the design. The observational design team, therefore, might consist both of members of the overall evaluation planning group, and others specifically recruited to work on an observational system.

The actual short list includes:

• Program staff and administrators
• Support staff (who often do recording or data entry)
• Outside evaluators or research consultants
• Participants or beneficiaries
• Volunteer observers

If, for some reason, the design group doesn’t include anyone with research experience, a training that includes information about different methods of observation, and about which methods are likely to produce which kinds of information, could help greatly to inform the design process. If the group does include researchers, that information could be presented as part of the discussion about design and the various possibilities, rather than in a training or workshop format.

How do you design an observational system?

So, you’ve decided on evaluation questions and planned an evaluation. Now it’s time to determine how you’ll get the data you need to answer your evaluation questions.

Review your evaluation questions

Remember these? You decided what it was you wanted to know, in order to determine whether your program was effective. Let’s go back to the example at the beginning of this section, the local community health center program. The center was starting a physical activity program for people with high blood pressure. Its objective was to have the participants engage in 45 minutes of moderate exercise at least four times a week over six months.

It hoped for several outcomes from this activity:

• That participants’ blood pressure would decrease
• That participants who needed to would lose weight
• That participants would experience a sense of greater well-being
• That participants would continue the exercise routine after the six-month program ended.

Some evaluation questions, therefore, are:

• Did participants engage in the recommended exercise routine for the period of the program?
• Did participants’ blood pressure decrease by the end of the six months?
• Did those participants who needed to lose weight do so by the end of six months?

There could easily be many other questions, a few examples being:

• How well attended were the workshops? Did participants find them helpful? Did participants who attended all or most of the workshops achieve better results than those who didn’t?
• Did participants experience a sense of greater well-being by the end of six months?
• Did participants continue the exercise routine (and maintain their lower blood pressure) after the program ended?

The answers to these questions might be only a few of those that the center wanted, but let’s stick with them for now, and use them as examples as we go through this part of the section. You may be concerned about your own process – how well you actually plan and implement your program.  You may also, like the center, have a specific time frame in which you hope for results. You might also have benchmarks – smaller achievements along the way to a larger goal – that you’re concerned with recording. All of this should  figure into your observational design.

Decide what you need to observe to answer your questions

Depending on the kind of program or effort you’re engaged in, and the nature of your evaluation questions, there’s a broad range of choices here.

Some of the most common:

• Participants’ behavior. This could be anything from the aggressive behaviors of children in a schoolyard or play setting to the degree of welding skill exhibited by participants in an employment-training program to the social interactions of the users of the neighborhood park we discussed earlier. The possibilities are nearly endless.
• Someone else’s behavior. The ultimate test of whether a high school peer mediation training program is working, for instance, may not be the behavior of the mediators on whose training the program focuses, but the behavior of the students with whom they work. There’s also a possibility here of looking at the ways in which participants are treated by program staff and vice versa.
• Conditions. An initiative may aim to change conditions directly – eliminating a crack house where drug dealing occurs, building affordable housing, cleaning up a polluted river – or may be meant to influence those changes by implementing programs or environmental or policy changes..
• Observations of products or results of behavior. When the behavior or event or condition itself isn’t visible or observable, either because it’s private, or because it takes place on a level that can’t be observed directly, you may have to measure its products or results.  It would be virtually impossible to observe directly the rate at which adolescents practiced safe sex, but it would be possible to learn the rate of STD infections among them, and the number of teen pregnancies before, during, and after a safe-sex peer education program.

When products or effects are all you can observe, you have to be sure that you’ve chosen the right ones to look at.  They should be, to the extent possible, obvious results of the behavior or condition you’re interested in, and you should take into account – and try to correct for – any other factors that might have caused them.

• Participants’ knowledge or attitudes. Like participants’ behavior, the possible range here is enormous, from scores on a knowledge test to nearly anything else you might think of.
• Someone else’s knowledge or attitudes. For instance, an advocacy program would be concerned with changing the attitudes of legislators and the public, but might not have direct contact with those whom it hoped to influence. This might use repeated public opinion surveys to assess willingness to support a particular policy change.
• Goal attainment.  Some programs have a particular aim that is their only reason for existence. This might be the passage or repeal of a law, the building of a school, the freedom of one or more political prisoners, etc. The only evaluation question in that case may be whether the goal has been achieved (or to what degree). In this situation, a goal attainment scale can be used to assess the degree of attainment (e.g., from 5 = most favorable outcome, to 1 = least favorable outcome).
• Interactions. The focus of an evaluation might be on the nature of interactions, or on whether particular individuals or groups interact or engage each other.  For instance, if the goal is increasing parent-child interaction, each parties’ talking to and responding to the other might be measure. As above, interactions among program participants, staff, or between participants and staff might be the focus of observations.

All of the above possibilities might have to do with either program goals – i.e., what a program wants to accomplish – or process and implementation – how a program goes about setting up and carrying out its work.

There are also some areas of observation that relate specifically to program process and implementation:

• Planning. Measurement here may focus on who was involved in planning what parts of the program, how the plan was developed, what its content was, satisfaction, etc.
• Timeline. When did the planning, implementation, and evaluation of the program each begin?  How long did each take?  Were deadlines met, and, if not, why not?
• Numbers of participants. How many participants did you have? What was the average amount of time they spent in the program? How many dropped out before completing the program?  How did those numbers compare to what you expected?
• Methods. What methods did you use in the program or intervention? How were they used?
• Program implementation. What did you actually do? This would include the program activity, its frequency and duration, the number of participants it served, where it took place (if that’s relevant), and how it was conducted.

In addition to identifying what you want to look at, you’ll have to define it carefully, so that observers know exactly what to look for. You have to be certain that all observations of a particular behavior, for instance, refer to the same phenomena (e.g., specific features that define whether the behavior occurred), even if observations made by different people.  If they don’t use the same measurement, you can’t really count on the information you get. Setting the limits of observations in each category – what’s included, what’s excluded, and where the boundaries are – will help to eliminate disagreement and make the observation more reliable.

To continue the health center example, let’s look at what the Center needs to observe.  In order to determine whether participants are actually doing their exercise regularly, the center has to find a way to observe people’s behavior (e.g., activity logs, self-reports on how often they engaged in physical activity). To find out whether they’ve lost weight, the center has to observe an outcome of behavior (e.g., weight, body mass index).  To learn whether they’re experiencing an increased sense of well-being, the center has to obtain self-reports.  And to learn whether they continue their exercise past the end of the program, it has to find a way to observe behavior after the program ends.

Decide how the observations will be conducted

Earlier, we discussed some methods of observation. We’ll return to those here, and examine them in greater detail.

• Direct observation. Direct observation involves either the “fly on the wall” approach, where the observer is anonymous and generally unnoticed, or – more often in service organizations of various kinds – is either an (identified) outside evaluator, or a staff member who works with participants and records, sometimes with their help, their behaviors and aspects of the situation  Anonymous observers are particularly good in situations where any people being observed are equally anonymous – conditions, large events, or activity like the use of that neighborhood park we talked about earlier, where the people involved could be anyone.

One common method of direct observation -- whether the observer is a program staffer or a program participant – is through keeping a journal or activity log. The observer writes down or otherwise records, soon after the occurrence, an account of what happened and events related to it, and often reactions to those experiences as well.  The journal or activity log then becomes a picture of the flow of the program, detailing the progress through it of particular participants, adjustments made, and satisfaction.

The nature of journals or report logs will obviously vary with the nature of the program, and not all programs or efforts lend themselves to this kind of observation.  But, especially in situations where several people have written journals or logs that cover the same period and events, they can be a very powerful and revealing means of observation.

• Participant observation. As we’ve explained, participant observers become part of the event, activity, culture, etc. that’s being observed, and experience it firsthand. Thus, in a health or human service program, the observer might be an actual participant (i.e., a  member of the group at whom the program is aimed), or an evaluator who joins participants in their activities. In the case of the park, for instance, a neighborhood resident who already visits the park regularly might volunteer to track how he and others actually use it – when various people or groups come, what they do once they’re there, which parts of the park they frequent, who interacts with whom, etc.

A mico-grant program was designed by a non-governmental organization in a rural village to increase income through creation of small businesses by participants. Members of the staff took part in each workshop, and participated in such activities as training in lending and loans, all the while observing the activities and other participants.  At the end of each day, the staff conducted group discussions where they relayed what they had seen, and asked participants to analyze what they had done.  The staff in this instance functioned as participant observers, and their participation added greatly to their ability to help their client, low-income women from small villages, understand and use their experiences.

• Self-reports. When the object of your observation is participants’ behavior that takes place away from the program (the amount of time participants spend reading to their children, for example), you often have to rely on the observations of participants themselves. This reliance has, as you might expect, both advantages and disadvantages. On the one hand, participants obviously know their own actions well. On the other, they can also leave out things that might be embarrassing or report in ways that think others want to see.  In addition, since they haven’t had prior experience as or been trained as observers, they might miss, or dismiss as unimportant and not report, behaviors or conditions that would be valuable for evaluation purposes.

An obvious remedy would be to train self-reporters as observers, and in some cases –medical trials, for instance – that’s both reasonable and common.  In other situations, however, it would go too far toward telling participants “the right answers,” and thereby possibly changing what they report toward what they think you want to hear.  There is also a huge advantage to self-reports: when they’re honest and represent a real change in behavior or experience for the reporters, they’re far more powerful than anything another person could say about their experience.

Self-reports, at least as defined here, imply an array of possible techniques for data collection.  Individual and group interviews, focus groups, public meetings, surveys and questionnaires, journals, checklists, and even casual conversation might all be ways for participants to convey information for an evaluation.

• Second-hand reports. These are reports about participant behavior or about conditions that come from people associated with those participants or conditions, but not directly connected to your program or effort. They might be service workers, teachers, health professionals youth workers, family members (particularly parents of young children), employers – almost anyone. Second-hand observations have to be viewed with some of the same cautions as those from people who work closely with participants. Observer-participant relationships, sympathy or empathy for participants, or observers’ personal biases can all keep reports from being objective. These observers may also need training.
• Electronic or mechanical observation. There are some circumstances where human observation is impossible or impractical. Observations of speeding are often made electronically, as are observations of health conditions inside the human body (using X-rays, CT or MRI scans, electrocardiographs, etc.)  In these situations, objectivity is no problem, but you have to be sure that whatever equipment you use is working properly, set up correctly, well-maintained, and protected against possible damage. It is also important that whoever interprets the information the equipment provides is trained to do so, and understands the limits and appropriate uses of that information.
• Tests or other similar observation tools. Education and health organizations often use various kinds of tests as observation tools. In a human service context, they are generally used to observe progress in skills, competency levels, or development. In public health or medicine, tests may be used to observe health status (e.g., screenings for elevated blood cholesterol) and the effects of treatment. They can be very useful in all these circumstances, but they’re also very specific, and don’t allow much room for intuition. In addition, the results of tests of skills, knowledge, or intellectual ability may be influenced by nervousness, lack of sleep, personal problems, or other factors that have little to do with actual competency.
• Public records and the like. If you’re using community-level indicators, such as rates of infant mortality or injuries due to motor vehicle crashes, as one way of looking at outcomes, you’ll have to use records, census data, and other similar material to get the data you need.

To continue with our previous example, the local health center would have to use a variety of these observation methods. The beginning and ongoing observations of blood pressure and weight at the monthly counseling sessions would take place with the use of instruments – a blood pressure cuff and a weighing scale – as well as by direct visual observation. (While an obvious reduction in fat may not indicate weight loss if the fat is replaced by muscle, it does indicate an increase in fitness, which may be equally beneficial. Fitness levels could also be mechanically and electronically measured, if the program chose.)

The amount and type of exercise each participant engaged in would be self-observed and self-reported through journals and interviews.  Feelings of well-being would be self-reported, but could also be observed by counselors trained to look for changes over time in posture, self-presentation, and other observable indicators.  Finally, observations of continued exercise would come through one or more follow-up visits some time after the program ended, with interviews, blood pressure and weight measurements, and more direct visual observation of fitness levels.  (Participants might also agree to continue keeping journals for a set period of time after the program ended, thus providing a self-report of their ongoing levels of physical activity.)

Decide when you need to observe

The question here is whether you need to start observing at the very beginning of the program (you almost always do), and how often you should observe throughout the course of the evaluation.

Some of the possibilities:

• Pre- and post- observation. This means making your observations at the beginning and the end of the evaluation period or the program. It’s the equivalent of what many schools do with standardized testing. They test reading scores at the beginning and end of each year, and then compare the two to determine how much the students have advanced. Although this type of observation may tell you whether anything changed during the program, it won’t give you strong evidence how the change took place, what caused it, or how effective your methods were.

This explanation assumes only before-and-after observation.  Most of the possibilities here include before-and-after observation, but add other observations to it.  For most kinds of evaluation, you should start observing at the beginning, or even well before the beginning (to understand whether any changes may in fact be part of an already-existing trend).  If you conduct your first observation partway, you won’t know if changes occurred before then.  A major change may occur toward the beginning in some interventions, toward the end in others, steadily throughout the intervention in still others, and in a few not till after the intervention is over.  It’s important to know just where you started from in order to fully understand what you’re seeing.  It may be that a long intervention is no more effective than a short one, or that a short one makes no difference at all.  You can only tell by knowing where you started from and through repeated or continuous measurement.

If your program or effort is one with a specific, one-time goal – for example, the passage of a law, or the clean-up of a particular space – the temptation may be to evaluate it only by whether you reach your goal or not (i.e., a single observation at the end of the effort).  This would be a mistake, because it wouldn’t take into account the parts of your effort that were successful and why, whether or not you reached your goal.  That’s a piece of information you’ll need the next time – and there will be a next time – you or others in the community take on a similar effort.

• At regular intervals during the evaluation period. You might choose any period from once an hour to once a month or more, depending on what you’re observing. The regularity makes observations easy to schedule, and gives an interval-by-interval picture of what’s going on.
• At irregular intervals during the evaluation period. The reason for this schedule might be logistical (you observe when you can); might have to do with making sure that observations aren’t expected, so that you get a true picture of what you’re looking at; or might be an attempt to look at the program or effort randomly, again to try to get an accurate picture.
• At specific times during the evaluation period. In this case, you might be concerned to see what happens or what participants are doing at different, identifiable times that imply different, identifiable conditions. In observing the use of that neighborhood park we mentioned earlier, you might want to be sure to go on weekdays, on weekends, in the morning, afternoon, and evening, at each of the four seasons, in rain, clouds, sun, and snow, and on days when there were special events in the park, to see who uses the park and how under different conditions and at different times.  If you’re monitoring the process and progress of the program, it’s important to make sure you observe each stage of it – the planning, the preparation, the implementation, the evaluation, and any follow-up – to make sure you get a full picture of what you did and how you did it. This will give you the information you need to analyze in order to make adjustments in how you conduct your work.
• Continuously. When the observer is a staff member working with program participants (or one or more of the participants themselves), it may be possible to make ongoing observations.The observer in this situation might observe directly using checklists, keep a journal, ask participants to keep records, video- or audiotape sessions, or record what happens in some other way, so that there’s an ongoing, day-to-day account of the behavior and what is happening in the environment of the program.

At the local health center, some of the observation – particularly that of monitoring participants’ blood pressure and weight – would be done at regular intervals, during the monthly meetings.  There would also be some continuous observation -- that of participants keeping track of their exercise programs in journals.  And finding out whether participants continued with their routines would be a one- or two-time follow-up – perhaps six months and a year after the program ended.

Define and describe the behaviors, products, conditions, and/or events that observers should be concerned with

If you want to be sure you know what observers are referring to in their reports, you have to be specific about what you want them to look for. The planning group, or a subgroup of it – the ideal would be a group that included a high proportion of people who will actually be researchers and observers – should set out identification standards for each element to be observed. These would explain what it looked like, when it was likely to occur, who would probably be involved, etc. For instance, to observe bullying or interpersonal violence on a playground would require clear definitions of this behavior, examples and non-examples, and scoring instructions. As a result, observers could all start with the same guidance about what they were looking for.

Design training for observers

Unless all the observation is to be done by those directly involved in the planning (not impossible in a small organization), and depending on their previous experience, observers might need to be trained in one or more areas:

• What it’s important to record, and why. People who have no acquaintance with research might not realize how important it is to record such details as the date, time, evaluation length, place, and circumstances of any observation, a description of who was involved and for how long, whether there were unexpected people or conditions present, etc. An early morning observation might provide a different set of observations than a late afternoon or evening one, for example.The presence of other people in a situation or interview – relatives, friends, program staff – can change the character of the behaviors displayed or information offered. It’s crucial, therefore, that observers understand that the context of the observation can be as important as its content.
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed. Careful definitions and descriptions of what’s to be observed won’t make much difference unless those who’ll do the observations are familiar with them.
• Effects of observation. In some cases, the behavior of participants might change as a result of their reactions to being observed. Observers have to be aware of that possibility, and make their own behavior as invisible as they can, so as not to influence participants’ behaviors.

In addition to human observers, the presence of audio or video equipment can often have an effect.  One way to offset it is to wait to start collecting data until participants are used to the presence of the equipment.  It’s also important to get participants’ permission beforehand to use recording equipment.

• Observer bias. Especially in situations where the observers are also program staff, their relationships with participants, or simply with the effort as a whole, may affect their reports or observations. If they particularly like or dislike a participant, that may have some influence on how they interpret or describe that person’s behavior. If they’re heavily invested in the success of the program being evaluated, they may – intentionally or unintentionally – put the best possible light on what they see. Whether or not they’re program staff, observers can also be influenced by their personal assumptions, their cultural, religious, or educational background, or their current psychological states or life circumstances. If they can be helped to recognize these biases and understand why and how they should be acknowledged or eliminated, there’s a better chance that they’ll conduct reliable observations.
• Observer drift. Sometimes, after people have been observing for a while, their observations tend to take on a regularity based on the rules they make up rather than shared definitions based on a standard.  They might tend to rate the behavior of certain participants in ways based more on past experience than on what they see, for instance.

You may also have to correct for observer effects, bias, or drift over the course of the evaluation.  That’s part of devising checks for accuracy and reliability based on a standard.

Devise checks for reliability and accuracy

If your information is to be reliable, it’s important that when two observers record a particular behavior, they mean exactly the same thing. This is a matter of training (see above), and also one of checking, either at the beginning or periodically, to make sure that all observers are seeing things in the same way. A participatory design of the system will help here. If the observers are involved in defining what they’ll all be observing, there’s a much better chance they’ll all see it similarly.

Some ways you can try to ensure agreement among observers:

• Use an external standard. One way to define what you’re looking for is to use a standard that’s used and accepted by all observers. “Behavior X looks like this, occurs in these circumstances, lasts for this long, and has these after-effects or results.”  The use of external standards often employs a checklist or something similar.  The observer checks off components of a behavior or condition, thus documenting what he sees in a way that matches how a different observer would score the event the same situation.  Such standards help assure the continued accuracy of the observational system.

Research teams and laboratories commonly use standards to assure agreement in identifying various conditions.  Each condition is described in detail, with various possible markers, such as measure of blood pressure or environmental toxins.

• Check for inter-rater reliability. Inter-rater reliability is the research term for assessing whether all observers interpret the same things – behaviors, conditions, events – in the same way. One way to address it is to check observers against one another. Two or more are exposed to the same situations or information, and then their scoring, such as of instances of bullying,  are compared. If they all say essentially the same thing, then inter-rater reliability is high, and everything’s fine.Typically, in research, if observers agree 80% or more of the time, observations are deemed reliable.If they disagree about what they saw, then you have to find the source of the disagreement. They may define terms differently, or their backgrounds may bias them toward seeing the same thing in different ways. Whatever the case, you have to uncover differences, and find a way to help all observers see things similarly and accurately.
• Use random third-party checks. A researcher, program director, or someone else who has a clear idea of what information is important and what various conditions or behaviors look like can observe in randomly chosen situations along with a regular observer to see if their observations match reasonably well.  If they disagree once, it may not mean much; but if they consistently disagree, there’s a problem.

Determine how to review and adjust your observational system for the next evaluation

Here’s this section’s version of “keep at it.” Just like your program, your evaluation, including your observational system, should be evaluated and adjusted to be as effective as possible.

Now you’re ready to start collecting and analyzing data. With careful planning and good training, you should be able to get the information you need for your evaluation.

In Summary

In order to conduct an evaluation that allows you to see your program or effort clearly and to adjust and improve it, you have to have a way of collecting accurate and useful information about it. The observational system you use is the way you look at what you’re doing – at your own process, at participants’ behavior and their progress and results, at conditions that affect your effort or that your effort is trying to change – to gain the information that you’ll analyze to evaluate your work. That system has to be feasible within your resources, and has to fit with the nature of your program, so designing it is an important part of your evaluation.

The design of observational systems is best carried out as a participatory process, particularly one involving both researchers or evaluators and those who’ll do the actual data collection. That involvement will give them a clear understanding of the system itself, of what information is needed, and of the pitfalls to data collection that they might encounter along the way. The result should be a more reliable system, and, ultimately, more accurate data for your evaluation.

What do we mean by an observational system?

___An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program

___It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with

Why design an observational system?

___It can help you get reliable information

___It can help you find out exactly what you need to know, eliminating or reducing wasted effort

___It can ensure that observations are made

___It can make it easier to analyze your data

___It can help you avoid haphazard evaluation

___It will make it easier to justify your findings

___It can help you gain credibility with funders and policy makers

___It can let you pass on your practices with confidence

___Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust

When should you design an observational system?

___If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning

___If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle

Who should design an observational system?

___Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation

How do you design an observational system?

___Review your evaluation questions

___Decide what you need to observe to answer your questions:

• Participants’ behavior
• Someone else’s behavior
• Conditions
• Observations of results of behavior
• Participants’ knowledge or attitudes
• Someone else’s knowledge or attitudes
• Goal attainment
• Interactions
• Program process or implementation (e.g., number of participants)

___Decide how the observations will be conducted:

• Direct observation
• Participant observation
• Self-reports, including individual and group interviews, focus groups, journals, surveys, etc.
• Second-hand reports, including interviews, journals, surveys, etc.
• Electronic or mechanical observation
• Tests or other similar observation tools
• Public records and the like for community-level indicators

___Decide when you need to observe:

• Pre- and post- observation
• At regular intervals during the evaluation period
• At irregular intervals during the evaluation period
• At specific times during the evaluation period
• Continuously

___Define and describe the behaviors, conditions, and/or events that observers should be concerned with

___Train observers in:

• What it’s important to record, and why
• The definitions and descriptions of the behaviors, conditions, events, or situations to be observed
• Effects of observation
• Observer bias
• Observer drift

___Devise checks for reliability and accuracy:

• Use an external standard
• Check for inter-rater reliability
• Use random third-party checks

___Adjust the system for the next evaluation

When you hear the word “experiment,” it may call up pictures of people in long white lab coats peering through microscopes. In reality, an experiment is just trying something out to see how or why or whether it works. It can be as simple as putting a different spice in your favorite dish, or as complex as developing and testing a comprehensive effort to improve child health outcomes in a city or state.

Academics and other researchers in public health and the social sciences conduct experiments to understand how environments affect behavior and outcomes, so their experiments usually involve people and aspects of the environment. A new community program or intervention is an experiment, too, one that a governmental or community organization engages in to find out a better way to address a community issue. It usually starts with an assumption about what will work – sometimes called a theory of change - but that assumption is no guarantee. Like any experiment, a program or intervention has to be evaluated to see whether it works and under what conditions.

In this section, we’ll look at some of the ways you might structure an evaluation to examine whether your program is working, and explore how to choose the one that best meets your needs. These arrangements for discovery are known as experimental (or evaluation) designs.

What do we mean by a design for the evaluation?

Every evaluation is essentially a research or discovery project. Your research may be about determining how effective your program or effort is overall, which parts of it are working well and which need adjusting, or whether some participants respond to certain methods or conditions differently from others. If your results are to be reliable, you have to give the evaluation a structure that will tell you what you want to know. That structure – the arrangement of discovery- is the evaluation’s design.

The design depends on what kinds of questions your evaluation is meant to answer.

Some of the most common evaluation (research) questions:

• Does a particular program or intervention – whether an instructional or motivational program, improving access and opportunities, or a policy change – cause a particular change in participants’ or others’ behavior, in physical or social conditions, health or development outcomes, or other indicators of success?
• What component(s) and element(s) of the program or intervention were responsible for the change?
• What are the unintended effects of an intervention, and how did they influence the outcomes?
• If you try a new method or activity, what happens?
• Will the program that worked in another context, or the one that you read about in a professional journal, work in your community, or with your population, or with your issue?

If you want reliable answers to evaluation questions like these, you have to ask them in a way that will show you whether you actually got results, and whether those results were in fact due to your actions or the circumstances you created, or to other factors.  In other words, you have to create a design for your research – or evaluation – to give you clear answers to your questions.  We’ll discuss how to do that later in the section.

Why should you choose a design for your evaluation?

An evaluation may seem simple: if you can see progress toward your goal by the end of the evaluation period, you’re doing OK; if you can’t, you need to change. Unfortunately, it’s not that simple at all. First, how do you measure progress? Second, if there seems to be none, how do you know what you should change in order to increase your effectiveness? Third, if there is progress, how do you know it was caused by ( or contributed to) your program, and not by something else? And finally, even if you’re doing well, how will you decide what you could do better, and what elements of your program can be changed or eliminated without affecting success? A good design for your evaluation will help you answer important questions like these.

Some specific reasons for spending the time to design your evaluation carefully include:

• So your evaluation will be reliable. A good design will give you accurate results. If you design your evaluation well, you can trust it to tell you whether you’re actually having an effect, and why. Understanding your program to this extent makes it easier to achieve and maintain success.
• So you can pinpoint areas you need to work on, as well as those that are successful. A good design can help you understand exactly where the strong and weak points of your program or intervention are, and give you clues as to how they can be further strengthened or changed for the greatest impact.
• So your results are credible. If your evaluation is designed properly, others will take your results seriously. If a well-designed evaluation shows that your program is effective, you’re much more likely to be able to convince others to use similar methods, and to convince funders that your organization is a good investment.
• So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants. Participants’ histories, crucial local or national events, the passage of time, personal crises, and many other factors can influence the outcome of a program or intervention for better or worse. A good evaluation design can help you to identify these, and either correct for them if you can, or devise methods to deal with or incorporate them.
• So you can identify unintended consequences (both positive and negative) and correct for them. A good design can show you all of what resulted from your program or intervention, not just what you expected. If you understand that your work has consequences that are negative as well as positive, or that it has more and/or different positive consequences than you anticipated, you can adjust accordingly.
• So you’ll have a coherent plan and organizing structure for your evaluation. It will be much easier to conduct your evaluation if it has an appropriate design. You’ll know better what you need to do in order to get the information you need. Spending the time to choose and organize an evaluation design will pay off in the time you save later and in the quality of the information you get.

When should you choose a design for your evaluation?

Once you’ve determined your evaluation questions and gathered and organized all the information you can about the issue and ways to approach it, the next step is choosing a design for the evaluation. Ideally, this all takes place at the beginning of the process of putting together a program or intervention.  Your evaluation should be an integral part of your program, and its planning should therefore be an integral part of the program planning.

That’s the ideal; now let’s talk about reality. If you’re reading this, the chances are probably at least 50-50 that you’re connected to an underfunded government agency or to a community-based or non-governmental organization, and that you’re planning an evaluation of a program or intervention that’s been running for some time – months or even years.

Even if that’s true, the same guidelines apply. Choose your questions, gather information, choose a design, and then go on through the steps presented in this chapter. Evaluation is important enough that you won’t really be accomplishing anything by taking shortcuts in planning it. If your program has a cycle, then it probably makes sense to start your evaluation at the beginning of it – the beginning of a year or a program phase, where all participants are starting from the same place, or from the beginning of their involvement.

If that’s not possible – if your program has a rolling admissions policy, or provides a service whenever people need it – and participants are all at different points, that can sometimes present research problems. You may want to evaluate the program’s effects only with new participants, or with  another specific group. On the other hand, if your program operates without a particular beginning and end, you may get the best picture of its effectiveness by evaluating it as it is, starting whenever you’re ready. Whatever the case, your design should follow your information gathering and synthesis.

Who should be involved in choosing a design?

If you’re a regular Tool Box user, and particularly if you’ve been reading this chapter, you know that the Tool Box team generally recommends a participatory process – involving both research and community partners, including all those with an interest in or who are affected with the program in planning and implementation. Choosing a design for evaluation presents somewhat of an exception to this policy, since scientific or evaluation partners may have a much clearer understanding of what is required to conduct research, and of the factors that may interfere with it.

As we’ll see in the “how-to” part of this section, there are a number of considerations that have to be taken into account to gain accurate information that actually tells you what you want to know. Graduate students generally take courses to gain the knowledge they need to conduct research well, and even some veteran researchers have difficulty setting up an appropriate research design. That doesn’t mean a community group can’t learn to do it, but rather that the time they would have to spend on acquiring background knowledge might be too great. Thus, it makes the most sense to assign this task (or at the very least its coordination) to an individual or small group with experience in research and evaluation design. Such a person can not only help you choose among possible designs, but explain what each design entails, in time, resources, and necessary skills, so that you can judge its appropriateness and feasibility for your context.

How do you choose a design for your evaluation?

How do you go about deciding what kind of research design will best serve the purposes of your evaluation?

The answer to that question involves an examination of four areas:

• The nature of the research questions you are trying to answer
• The challenges to the research, and the ways they can be resolved or reduced
• The kinds of research designs that are generally used, and what each design entails
• The possibility of adapting a particular research design to your program or situation – what the structure of your program will support, what participants will consent to, and what your resources and time constraints are

We’ll begin this part of the section with an examination of the concerns research designs should address, go on to considering some common designs and how well they address those concerns, and end with some guidelines for choosing a design that will both be possible to implement and give you the information you need about your program.

Note: in this part of the section, we’re looking at evaluation as a research project.  As a result, we’ll use the term “research” in many places where we could just as easily have said, for the purposes of this section, “evaluation.”  Research is more general, and some users of this section may be more concerned with research in general than evaluation in particular.

Concerns research designs should address

The most important consideration in designing a research project – except perhaps for the value of the research itself – is whether your arrangement will provide you with valid information. If you don’t design and set up your research project properly, your findings won’t give you information that is accurate and likely to hold true with other situations.  In the case of an evaluation, that means that you won’t have a basis for adjusting what you do to strengthen and improve it.

Here’s a far-fetched example that illustrates this point.  If you took children’s heights at age six, then fed them large amounts of a specific food for three years – say carrots – and measured them again at the end of the period, you’d probably find that most of them were considerably taller at nine years than at six.  You might conclude that it was eating carrots that made the children taller because your research design gave you no basis for comparing these children’s growth to that of other children.

There are two kinds of threats to the validity of a piece of research. They are usually referred to as threats to internal validity (whether the intervention produced the change) and threats to external validity (whether the results are likely to apply to other people and situations).

Threats to internal validity

These are threats (or alternative explanations) to your claim that what you did caused changes in the direction you were aiming for. They are generally posed by factors operating at the same time as your program or intervention that might have an effect on the issue you’re trying to address. If you don’t have a way of separating their effects from those of your program, you can’t tell whether the observed changes were caused by your work, or by one or more of these other factors.They’re called threats to internal validity because they’re internal to the study – they have to do with whether your intervention – and not something else – accounted for the difference.

There are several kinds of threats to internal validity:

• History. Both participants’ personal histories – their backgrounds, cultures, experiences, education, etc. – and external events that occur during the research period – a disaster, an election, conflict in the community, a new law – may influence whether or not there’s any change in the outcomes you’re concerned with.
• Maturation. This refers to the natural physical, psychological, and social processes that take place as time goes by. The growth of the carrot-eating children in the example above is a result of maturation, for instance, as might be a decline in risky behavior as someone passed from adolescence to adulthood, the development of arthritis in older people, or participants becoming tired during learning activities towards the end of the day.
• The effects of testing or observation on participants. The mere fact of a program’s existence, or of their taking part in it, may affect participants’ behavior or attitudes, as may the experience of being tested, videotaped, or otherwise observed or measured.
• Changes in measurement. An instrument – a blood pressure cuff or a scale, for instance – can change over time, or different ones may not give the same results. By the same token, observers – those gathering information – may change their standards over time, or two or more observers may disagree on the observations.
• Regression toward the mean. This is a statistical term that refers to the fact that, over time, the very high and very low scores on a measure (a test, for instance) often tend to drift back toward the average for the group. If you start a program with participants who, by definition, have very low or high levels of whatever you’re measuring – reading skill, exposure to domestic violence, particular behavior toward people of other races or backgrounds, etc. – their scores may end up closer to the average over the course of the evaluation period even without any program.
• The selection of participants. Those who choose participants may slant their selection toward a particular group that is more or less likely to change than a cross-section of the population from which the group was selected. (A good example is that of employment training programs that get paid according to the number of people they place in jobs. They’re more likely to select participants who already have all or most of the skills they need to become employed, and neglect those who have fewer skills... and who therefore most need the service.) Selection can play a part when participants themselves choose to enroll in a program (self-selection), since those who decide to participate are probably already motivated to make changes. It may also be a matter of chance: members of a particular group may, simply by coincidence, share a characteristic that will set their results on your measures apart from the norm of the population you’re drawing from.

Selection can also be a problem when two groups being compared are chosen by different standards.  We’ll discuss this further below when we deal with control or comparison groups.

• The loss of data or participants. If too little information is collected about participants, or if too many drop out well before the research period is over, your results may be based on too little data to be reliable. This also arises when two groups are being compared. If their losses of data or participants are significantly different, comparing them may no longer give you valid information.
• The nature of change. Often, change isn’t steady and even. It can involve leaps forward and leaps backward before it gets to a stable place – if it ever does. (Think of looking at the performance of a sports team halfway through the season. No matter what its record is at that moment, you won’t know how well it will finish until the season is over.)  Your measurements may take place over too short a period or come at the wrong times to track the true course of the change or lack of change that’s occurring.
• A combination of the effects of two or more of these. Two or more of these factors may combine to produce or prevent the changes your program aims to produce. A language-study curriculum that is tested only on students who already speak two or more languages runs into problems with both participants’ history – all the students have experience learning languages other than their own – and selection – you’ve chosen students who are very likely to be successful at language learning.

Threats to external validity

These are factors that affect your ability to apply your research results in other circumstances – to increase the chances that your program and its results can be reproduced elsewhere or with other populations. If, for instance, you offer parenting classes only to single mothers, you can’t assume, no matter how successful they appear to be, that the same classes will work as well with men.

Threats to external validity (or generalizability) may be the result of the interactions of other factors with the program or intervention itself, or may be due to particular conditions of the program.

Some examples:

• Interaction of testing or data collection and the program or intervention.  An initial test or observation might change the way participants react to the program, making a difference in final outcomes.  Since you can’t assume that another group will have the same reaction or achieve similar final outcomes as a result, external validity or generalizability of the findings becomes questionable.
• Interaction of selection procedures and the program or intervention.  If the participants selected or self-selected are particularly sensitive to the methods or purpose of the program, it can’t be assumed to be effective with participants who are less sensitive or ready for the program.

Parents who’ve been threatened by the government with the loss of their children due to child abuse may be more receptive to learning techniques for improving their parenting, for example, than parents who are under no such pressure.

• The effects of the research arrangements. Participants may change behavior as a result of being observed, or may react to particular individuals in ways they would be unlikely to react to others.

A classic example here is that of a famous baboon researcher, Irven DeVore, who after years of observing troupes of baboons, realized that they behaved differently when he was there than when he wasn’t.  Although his intent was to observe their natural behavior, his presence itself constituted an intervention, making the behavior of the baboons he was observing different from that of a troupe that was not observed.

• The interference of multiple treatments or interventions. The effects of a particular program can be changed when participants are exposed to it beforehand in a different context, or are exposed to another before or at the same time as the one being evaluated. This may occur when participants are receiving services from different sources, or being treated simultaneously for two or more health issues or other conditions.

Given the range of community programs that exist, there are many possibilities here. Adults might be members of a high school completion class while participating in a substance use recovery program.  A diabetic might be treated with a new drug while at the same time participating in a nutrition and physical activity program to deal with obesity.  Sometimes, the sequence of treatments or services in a single program can have the same effect, with one influencing how participants respond to those that follow, even though each treatment is being evaluated separately.

Common research designs

Many books have been written on the subject of research design. While they contain too much material to summarize here, there are some basic designs that we can introduce. The important differences among them come down to how many measurements you’ll take, when you will take them, and how many groups of what kind will be involved.

Program evaluations generally look for the answers to three basic questions:

• Was there any change – in participants’ or others’ behavior, in physical or social conditions, or in outcomes or indicators of success– during the evaluation period?
• Was whatever change took place – or the lack of change – caused by your program, intervention, or effort?
• What, in your program or outside it, actually caused or prevented the change?

As we’ve discussed, changes and improvement in outcomes may have been caused by some or all of your intervention, or by external factors. Participants’ or the community’s history might have been crucial. Participants may have changed as a result of simply getting older and more mature or more experienced in the world – often an issue when working with children or adolescents. Environmental factors – events, policy change, or conditions in participants’ lives – can often facilitate or prevent change as well. Understanding exactly where the change came from or where the barriers to change reside, gives you the opportunity to adjust your program to take advantage of or combat those factors.

If all you had to do was to measure whatever behavior or condition you wanted to influence at the beginning and end of the evaluation, choosing a design would be an easy task. Unfortunately, it’s not quite that simple – there are those nasty threats to validity to worry about. We have to keep them in mind as we look at some common research designs.

Research designs, in general, differ in one or both of two ways: the number and timing of the measurements they use; and whether they look at single or multiple groups.  We’ll look at single-group designs first, then go on to multiple groups.

Before we go any further, it is helpful to have an understanding of some basic research terms that we will be using in our discussion.

Researchers usually refer to your first measurement(s) or observation(s) – the ones you take before you start your program or intervention – as a baseline measure or baseline observation, because it establishes a baseline – a known level – to which you compare future measurements or observations.

Some other important research terms:

• Independent variables are the program itself and/or the methods or conditions that the researcher – in this case, you – wants to evaluate. They’re called variables because they can change – you might have chosen (and might still choose) other methods.  They’re independent because their existence doesn’t depend on whether something else occurs: you’ve chosen them, and they’ll stay consistent throughout the evaluation period.
• Dependent variables are whatever may or may not change as a result of the presence of the independent variable(s).  In an evaluation, your program or intervention is the independent variable.  (If you’re evaluating a number of different methods or conditions, each of them is an independent variable.)  Whatever you’re trying to change is the dependent variable.  (If you’re aiming at change in more than one behavior or outcome, each type of change is a different dependent variable.)  They’re called dependent variables because changes in them depend on the action of the independent variable...or something else.
• Measures are just that – measurements of the dependent variables.  They usually refer to procedures that have results that can be translated into numbers, and may take the form of community assessments, observations, surveys, interviews, or tests. They may also count incidents or measure the amount of the dependent variable (number or percentage of children who are overweight or obese, violent crimes per 100,000 population, etc.)
• Observations might involve measurement, or they might simply record what happens in specific circumstances: the ways in which people use a space, the kinds of interactions children have in a classroom, the character of the interactions during an assessment.  For convenience, researchers often use “observation” to refer to any kind of measurement and we’ll use the same convention here.

Pre- and post- single-group design

The simplest design is also probably the least accurate and desirable: the pre (before) and post (after) measurement or observation. This consists of simply measuring whatever you’re concerned with in one group – the infant mortality rate, unemployment, water pollution – applying your intervention to that group or community, and then observing again. This type of design assumes that a difference in the two observations will tell you whether there was a change over the period between them, and also assumes that any positive change was caused by the intervention.

In most cases, a pre-post design won’t tell you much, because it doesn’t really address any of the research concerns we’ve discussed. It doesn’t account for the influence of other factors on the dependent variable, and it doesn’t tell you anything about trends of change or the progress of change during the evaluation period – only where participants were at the beginning and where they were at the end. It can help you determine whether certain kinds of things have happened – whether there’s been a reduction in the level of educational attainment or the amount of environmental pollution in a river, for instance – but it won’t tell you why. Despite its limitations, taking measures before and after the intervention is far better than no measures.

Even looking at something as seemingly simple to measure pre and post as blood pressure (in a heart disease prevention program) is questionable.  Blood pressure may be lower at the final observation than at the initial one, but that tells you nothing about how much it may have gone up and down in between. If the readings were taken by different people, the change may be due in part to differences in their skill, or to how relaxed each was able to make participants feel.  Familiarity with the program could also have reduced most participants’ blood pressure from the pre- to the post-measurement, as could some other factor that wasn’t specifically part of the independent variable being evaluated.

Interrupted time series design with a single group (simple time series)

An interrupted time series used repeated measures before and after delayed implementation of the independent variable (e.g., the program, etc.) to help rule out other explanations. This relatively strong design – with comparisons within the group – addresses most threats to internal validity.

The simplest form of this design is to take repeated observations, implement the program or intervention, and observe a number of times during the evaluation period, including at the end. This method is a great improvement over the pre- and post- design in that it tracks the trend of change, and can therefore, help see whether it was actually the independent variable that caused any change. It can also help to identify the influence of external factors such as when the dependent variable shows significant change before the intervention is implemented.

Another possibility for this design is to implement more than one independent variable, either by trying two or more, one after another (often with a break in between), or by adding each to what came before.This gives a picture not only of the progress of change, but can show very clearly what causes change. That gives an evaluator the opportunity not only to adjust the program, but to drop elements that have no effect.

There are a number of variations on the interrupted time series theme, including varying the observation times; implementing the independent variable repeatedly; and implementing one independent variable, then another, then both together to evaluate their interaction.

In any variety of interrupted time series design, it’s important to know what you’re looking for.  In an evaluation of a traffic fatality control program in the United Kingdom that focused on reducing drunk driving, monthly measurements seemed to show only a small decline in fatal accidents. When the statistics for weekends, when there were most likely to be drunk drivers on the road, were separated out, however, they showed that the weekend fatality rate dropped sharply with the implementation of the program, and stayed low thereafter. Had the researchers not realized that that might be the case, the program might have been stopped, and the weekend accident rate would not have been reduced.

Interrupted time series design with multiple groups (multiple baseline/time series)

This has the same possibilities as the single time series design, with the added wrinkle of using repeated measures with one or more other groups (so-called multiple baselines). By using multiple baselines (groups), the external validity or generality of the findings is enhanced – we can see if the effects occur with different groups or under different conditions.

This multiple time series design – typically staggered introduction of the intervention with different groups or communities – gives the researcher more opportunities:

• You can try a method or program with two or more groups from the same
• You can try a particular method or program with different populations, to see if it’s effective with others
• You can vary the timing or intensity of an intervention with different groups
• You can test different interventions at the same time
• You can try the same two or more interventions with each of two groups, but reverse their order to see if sequencing it makes any difference

Again, there are more variations possible here.

Control group design

A common way to evaluate the effects of an independent variable is to use a control group.  This group is usually similar to the participant group, but either receives no intervention at all, or receives a different intervention with the same goal as that offered to the participant group. A control group design is usually the most difficult to set up – you have to find appropriate groups, observe both on a regular basis, etc. – but is generally considered to be the most reliable.

The term control group comes from the attempt to control outside and other influences on the dependent variable. If everything about the two groups except their exposure to the program being evaluated averages out to be the same, then any differences in results must be due to that exposure. The term comparison group is more modest; it typically offers a community watched for similar levels of the problem/goal and relevant characteristics of the community or population (e.g., education, poverty).

The gold standard here is the randomized control group, one that is selected totally at random, either from among the population the program or intervention is concerned with – those at risk for heart disease, unemployed males, young parents – or, if appropriate, the population at large. A random group eliminates the problems of selection we discussed above, as well as issues that might arise from differences in culture, race, or other factors.

A control group that’s carefully chosen will have the same characteristics as the intervention group (the focus of the evaluation). If, for instance, the two groups come from the same pool of people with a particular health condition, and are chosen at random either to be treated in the conventional way or to try a new approach, it can be assumed that – since they were chosen at random from the same population – both groups will be subject, on average, to the same outside influences, and will have the same diversity of backgrounds. Thus, if there is a significant difference in their results, it is fairly safe to assume that the difference comes from the independent variable – the type of intervention, and not something else.

The difficulty for governmental and community-based organizations is to find or create a randomized control group. If the program has a long waiting list, it may be able to create a control by selecting those to first receive the intervention at random. That in itself creates problems, in that people often drop off waiting lists out of frustration or other reasons. Being included in the evaluation may help to keep them, on the other hand, by giving them a closer connection to the program and making them feel valued.

An ESOL (English as a Second or Other Language) program in Boston with a three-year waiting list addressed the problem by offering those on the waiting list a different option.  They received videotapes to use at home, along with biweekly tutoring by advanced students and graduates of the program.  Thus, they became a comparison group with a somewhat different intervention that, as expected, was less effective than the program itself, but was more effective than none, and kept them on the waiting list. It also gave them a head start once they got into the classes, with many starting at a middle rather than at a beginning level.

When there’s no waiting list or similar group to draw from, community organizations often end up using a comparison group - one composed of participants in another place or program and whose members’ characteristics, backgrounds, and experience may or may not be similar to those of the participant group. That circumstance can raise some of the same problems related to selection seen when there is no control group. If the only potential comparisons involve very different groups, it may be better to use a design, such as an interrupted time series design that doesn’t involve a control group at all, where the comparison is within (not between) groups.

Groups may look similar, but may differ in an important way.  Two groups of participants in a substance use intervention program, for instance, may have similar histories, but if one program is voluntary and the other is not, the results aren’t likely to be comparable.  One group will probably be more motivated and less resentful than the other, and composed of people who already know they have a potential problem. The motivation and determination of their participants, rather than the effectiveness of the two programs, may influence the amount of change observed.

This issue may come up in a single-group design as well.  A program that may, on average, seem to be relatively ineffective may prove, on close inspection, to be quite effective with certain participants – those of a specific educational background, for instance, or with particular life experiences. Looking at results with this in mind can be an important part of an evaluation, and give you valuable and usable information.

Choosing a design

This section’s discussion of research designs is in no way complete.  It’s meant to provide an introduction to what’s available.  There are literally thousands of books and articles written on this topic, and you’ll probably want more information.  There are a number of statistical methods that can compensate for less-than-perfect designs, for instance: few community groups have the resources to assemble a randomized control group, or to implement two or more similar programs to see which works better.

Given this, the material that follows is meant only as broad guidelines.  We don’t attempt to be specific about what kind of design you need in what circumstances, but only try to suggest some things to think about in different situations. Help is available from a number of directions: Much can be found on the Internet (see the “Resources” part of this section for a few sites); there are numerous books and articles (the classic text on research design is also cited in “Resources”); and universities are a great resource, both through their libraries and through faculty and graduate students who might be interested in what you’re doing, and be willing to help with your evaluation.  Use any and all of these to find what will work best for you.  Funders may also be willing either to provide technical assistance for evaluations, or to include money in your grant or contract specifically to pay for a professional evaluation.

Your goal in evaluating your effort is to get the most reliable and accurate information possible, given your evaluation questions, the nature of your program, what your participants will consent to, your time constraints, and your resources. The important thing here is not to set up a perfect research study, but to design your evaluation to get real information, and to be able to separate the effects of external factors from the effects of your program.  So how do you go about choosing the best design that will be workable for you? The steps are in the first sentence of this paragraph.

Consider your evaluation questions

What do you need to know? If the intent of your evaluation is simply to see whether something specific happened, it’s possible that a simple pre-post design will do. If, as is more likely, you want to know both whether change has occurred, and if it has, whether it has in fact been caused by your program, you’ll need a design that helps to screen out the effects of external influences and participants’ backgrounds.

For many community programs, a control or comparison group is helpful, but not absolutely necessary. Think carefully about the frequency and timing of your observations and the amount of different kinds of information you can collect. With repeated measures, you can get you quite an accurate picture of the effectiveness of your program from a simple time series design. Single group interrupted time series designs, which are often the most workable for small organizations, can give you a very reliable evaluation if they’re structured well. That generally means obtaining multiple baseline observations (enough to set a trend) before the program begins; observing often and documenting your observations carefully (often with both quantitative – expressed in numbers – and qualitative – expressed in records of incidents and of what participants did and said – data); and  including during intervention and  follow-up observations to see whether effects are maintained.

In many of these situations, a multiple-group interrupted time series design is quite possible, but of a “naturally-occurring” experiment. If your program includes two or more groups or classes, each working toward the same goals, you have the opportunity to stagger the introduction of the intervention across the groups. This comparison with (and across) groups allows you to screen out such factors as the facilitator’s ability and community influences (assuming all participants come from the same general population.)  You could also try different methods or time sequences, to see which works best.

In some cases, the real question is not whether your method or program works, but whether it works better than other methods or programs you could be using. Teaching a skill – for instance, employment training, parenting, diabetes management, conflict resolution – often falls into this category.  Here, you need a comparison of some sort.  While evaluations of some of these – medical treatment, for example – may require a control group, others can be compared to data from the field, to published results of other programs, or, by using community-level indicators, from measurements in other communities.

There are community programs where the bottom line is very simple.  If you’re working to control water pollution, your main concern may be the amount of pollution coming out of effluent pipes, or the amount found in the river.  Your only measure of success may be keeping pollution below a certain level, which means that regular monitoring of water quality is the only evaluation you need.  There are probably relatively few community programs where evaluation is this easy – you might, for instance, want to know which of your pollution-control activities is most effective – but if yours is one, a simple design may be all you need.

Consider the nature of your program

What does your program look like, and what is it meant to do?  Does it work with participants in groups, or individually, for instance? Does it run in cycles – classes or workshops that begin and end on certain dates, or a time-limited program that participants go through only once? Or can participants enter whenever they are ready and stay until they reach their goals?  How much of the work of the program is dependent on staff, and how much do participants do on their own?  How important is the program context – the way staff, participants, and others treat one another, the general philosophy of the program, the physical setting, the organizational culture?  (The culture of an organization consists of accepted and traditional ways of doing things, patterns of relationships, how people dress, how they act toward and communicate with one another, etc.)

• If you work with participants in groups, a multiple-group design – either interrupted time series or control group – might be easier to use.  If you work with participants individually, perhaps a simple time series or a single group design would be appropriate.
• If your program is time-limited – either one-time-only, or with sessions that follow one another – you’ll want a design that fits into the schedule, and that can give you reliable results in the time you have. One possibility is to use a multiple group design, with groups following one another session by session. The program for each group might be adjusted, based on the results for the group before, so that you could test new ideas each session.
• If your program has no clear beginning and end, you’re more likely to need a single group design that considers participants individually, or by the level of their baseline performance. You may also have to compensate for the fact that participants may be entering the program at different levels, or with different goals.

A proverb says that you never step in the same river twice, because the water that flows past a fixed point is always changing. The same is true of most community programs. Someone coming into a program at a particular time may have a totally different experience than a similar person entering at a different time, even though the operation of the program is the same for both. A particular participant may encourage everyone around her, and create an overwhelmingly positive atmosphere different from that experienced by participants who enter the program after she has left, for example. It’s very difficult to control for this kind of difference over time, but it’s important to be aware that it can, and often does, exist, and may affect the results of a program evaluation.

• If the organizational or program context and culture are important, then you’ll probably want to compare your results with participants to those in a control group in a similar situation where those factors are different, or are ignored.

There is, of course, a huge range of possibilities here: nearly any design can be adapted to nearly any situation in the right circumstances. This material is meant only to give you a sense of how to start thinking about the issue of design for an evaluation.

Consider what your participants (and staff) will consent to

In addition to the effect that it might have on the results of your evaluation, you might find that a lot of observation can raise protests from participants who feel their privacy is threatened, or from already-overworked staff members who see adding evaluation to their job as just another burden. You may be able to overcome these obstacles, or you may have to compromise – fewer or different kinds of observations, a less intrusive design – in order to be able to conduct the evaluation at all.

There are other reasons that participants might object to observation, or at least intense observation. Potential for embarrassment, a desire for secrecy (to keep their participation in the program from family members or others), even self-protection (in the case of domestic violence, for instance) can contribute to unwillingness to be a participant in the evaluation. Staff members may have some of the same concerns.

There are ways to deal with these issues, but there’s no guarantee that they’ll work. One is to inform participants at the beginning about exactly what you’re hoping to do, listen to their objections, and meet with them (more than once, if necessary) to come up with a satisfactory approach. Staff members are less likely to complain if they’re involved in planning the evaluation, and thus have some say over the frequency and nature of observations. The same is true for participants.Treating everyone’s concerns seriously and including them in the planning process can go a long way toward assuring cooperation.

Consider your time constraints

As we mentioned above, the important thing here is to choose a design that will give you reasonably reliable information. In general, your design doesn’t have to be perfect, but it does have to be good enough to give you a reasonably good indication that changes are actually taking place, and that they are the result of your program. Just how precise you can be is at least partially controlled by the limits on your time placed by funding, program considerations, and other factors.

Time constraints may also be imposed. Some of the most common:

• Program structure. An evaluation may make the most sense if it’s conducted to correspond with a regular program cycle.
• Funding.  If you are funded only for a pilot project, for example, you’ll have to conduct your evaluation within the time span of the funding, and soon enough to show that your program is successful enough to be refunded. A time schedule for evaluation may be part of your grant or contract, especially if the funder is paying for it.
• Participants’ schedules. A rural education program may need to stop for several months a year to allow participants to plant and tend crops, for instance.
• The seriousness of the issue  A delay in understanding whether a violence prevention program is effective may cost lives.
• The availability of professional evaluators. Perhaps the evaluation team can only work during a particular time frame.

Consider your resources

Strategic planners often advise that groups and organizations consider resources last: otherwise they’ll reject many good ideas because they’re too expensive or difficult, rather than trying to find ways to make them work with the resources at hand. Resources include not only money, but also space, materials and equipment, personnel, and skills and expertise. Often, one of these can substitute for another: a staff person with experience in research can take the place of money that would be used to pay a consultant, for example. A partnership with a nearby university could get you not only expertise, but perhaps needed equipment as well.

The lesson here is to begin by determining the best design possible for your purposes, without regard to resources. You may have to settle for somewhat less, but if you start by aiming for what you want, you’re likely to get a lot closer to it than if you assume you can’t possibly get it.

In Summary

The way you design your evaluation research will have a lot to do with how accurate and reliable your results are, and how well you can use them to improve your program or intervention. The design should be one that best addresses key threats to internal validity (whether the intervention caused the change) and external validity (the ability to generalize your results to other situations, communities, and populations).

Common research designs – such as interrupted time series or control group designs– can be adapted to various situations, and combined in various ways to create a design that is both appropriate and feasible for your program. It may be necessary to seek help from a consultant, a university partner, or simply someone with research experience to help identify a design that fits your needs.

A good design will address your evaluation questions, and take into consideration the nature of your program, what program participants and staff will agree to, your time constraints, and the resources you have available for evaluation. It often makes sense to consider resources last, so that you won’t reject good ideas because they seem too expensive or difficult. Once you’ve chosen a design, you can often find a way around a lack of resources to make it a reality.

What do we mean by a design for the evaluation?

___The design of the evaluation is the arrangement that will make sure it reliably tells you what you need to know

___An appropriate design will show you whether you actually got results, and whether those results were likely due to your actions or the circumstances you created, or to other factors

Why should you choose a design for your evaluation?

___So your evaluation will be reliable

___So you can pinpoint areas you need to work on, as well as those that are successful

___So your results are credible

___So you can identify factors unrelated to what you’re doing that have an effect – positive or negative – on your results and on the lives of participants

___So you can identify unintended consequences (both positive and negative) and correct for them

___So you’ll have a coherent plan and organizing structure for your evaluation

When should you choose a design for your evaluation?

___In the ideal, when you’re planning the program and evaluation, before you start any implementation

___In reality, for many organizations, at the beginning of a program cycle, or as a new group of participants enters

Who should be involved in choosing a design for your evaluation?

___It’s extremely helpful to include someone with research experience to help you decide on an appropriate design, and to help you understand what each possible design entails

How do you select an appropriate design for your evaluation?

___Take into consideration the necessary research considerations

• Understand threats to internal validity ( whether the intervention produced the change)
  • History
  • Maturation
  • The effects of testing or observation on participants
  • Changes in measurement
  • Regression toward the mean
  • The selection of participants
  • The loss of data or participants
  • The nature of change
  • A combination of the effects of two or more of these
• Understand threats to external validity (generalizability of your findings)
  • Interaction of testing or data collection and the program or intervention
  • Interaction of selection procedures and the program or intervention
  • The effects of the research arrangements
  • The interference of multiple programs or interventions

___Understand common research designs

• Pre- and post- single group design
• Interrupted time series design with a single group (simple time series)
• Interrupted time series with multiple groups (multiple time series)
• Control or comparison group

___Choose a design

___Consider your evaluation questions

___Consider the nature of your program

___Consider what your participants and staff will consent to

___Consider your time constraints

___Consider your resources

In previous sections of this chapter, we’ve discussed studying the issue, deciding on a research design, and creating an observational system for gathering information for your evaluation. Now it’s time to collect your data and analyze it – figuring out what it means – so that you can use it to draw some conclusions about your work. In this section, we’ll examine how to do just that.

What do we mean by collecting data?

Essentially, collecting data means putting your design for collecting information into operation. You’ve decided how you’re going to get information – whether by direct observation, interviews, surveys, experiments and testing, or other methods – and now you and/or other observers have to implement your plan. There’s a bit more to collecting data, however. If you are conducting observations, for example, you’ll have to define what you’re observing and arrange to make observations at the right times, so you actually observe what you need to. You’ll have to record the observations in appropriate ways and organize them so they’re optimally useful.

Recording and organizing data may take different forms, depending on the kind of information you’re collecting. The way you collect your data should relate to how you’re planning to analyze and use it. Regardless of what method you decide to use, recording should be done concurrent with data collection if possible, or soon afterwards, so that nothing gets lost and memory doesn’t fade.

Some of the things you might do with the information you collect include:

• Gathering together information from all sources and observations
• Making photocopies of all recording forms, records, audio or video recordings, and any other collected materials, to guard against loss, accidental erasure, or other problems
• Entering narratives, numbers, and other information into a computer program, where they can be arranged and/or worked on in various ways
• Performing any mathematical or similar operations needed to get quantitative information ready for analysis.  These might, for instance, include entering numerical observations into a chart, table, or spreadsheet, or figuring the mean (average), median (midpoint), and/or mode (most frequently occurring) of a set of numbers.
• Transcribing (making an exact, word-for-word text version of) the contents of audio or video recordings
• Coding data (translating data, particularly qualitative data that isn’t expressed in numbers, into a form that allows it to be processed by a specific software program or subjected to statistical analysis)
• Organizing data in ways that make them easier to work with.  How you do this will depend on your research design and your evaluation questions. You might group observations by the dependent variable (indicator of success) they relate to, by individuals or groups of participants, by time, by activity, etc. You might also want to group observations in several different ways, so that you can study interactions among different variables.

There are two kinds of variables in research.  An independent variable (the intervention) is a condition implemented by the researcher or community to see if it will create change and improvement. This could be a program, method, system, or other action.  A dependent variable is what may change as a result of the independent variable or intervention.  A dependent variable could be a behavior, outcome, or other condition.  A smoking cessation program, for example, is an independent variable that may change group members’ smoking behavior, the primary dependent variable.

What do we mean by analyzing data?

Analyzing information involves examining it in ways that reveal the relationships, patterns, trends, etc. that can be found within it. That may mean subjecting it to statistical operations that can tell you not only what kinds of relationships seem to exist among variables, but also to what level you can trust the answers you’re getting.  It may mean comparing your information to that from other groups (a control or comparison group, statewide figures, etc.), to help draw some conclusions from the data. The point, in terms of your evaluation, is to get an accurate assessment in order to better understand your work and its effects on those you’re concerned with, or in order to better understand the overall situation.

There are two kinds of data you’re apt to be working with, although not all evaluations will necessarily include both. Quantitative data refer to the information that is collected as, or can be translated into, numbers, which can then be displayed and analyzed mathematically. Qualitative data are collected as descriptions, anecdotes, opinions, quotes, interpretations, etc., and are generally either not able to be reduced to numbers, or are considered more valuable or informative if left as narratives. As you might expect, quantitative and qualitative information needs to be analyzed differently.

Quantitative data

Quantitative data are typically collected directly as numbers. Some examples include:

• The frequency (rate, duration) of specific behaviors or conditions
• Test scores (e.g., scores/levels of knowledge, skill, etc.)
• Survey results (e.g., reported behavior, or outcomes to environmental conditions; ratings of satisfaction, stress, etc.)
• Numbers or percentages of people with certain characteristics in a population (diagnosed with diabetes, unemployed, Spanish-speaking, under age 14, grade of school completed, etc.)

Data can also be collected in forms other than numbers, and turned into quantitative data  for analysis. Researchers can count the number of times an event is documented in interviews or records, for instance, or assign numbers to the levels of intensity of an observed event or behavior. For instance, community initiatives often want to document the amount and intensity of environmental changes they bring about – the new programs and policies that result from their efforts. Whether or not this kind of translation is necessary or useful depends on the nature of what you’re observing and on the kinds of questions your evaluation is meant to answer.

Quantitative data is usually subjected to statistical procedures such as calculating the mean or average number of times an event or behavior occurs (per day, month, year). These operations, because numbers are “hard” data and not interpretation, can give definitive, or nearly definitive, answers to different questions. Various kinds of quantitative analysis can indicate changes in a dependent variable related to – frequency, duration, timing (when particular things happen), intensity, level, etc. They can allow you to compare those changes to one another, to changes in another variable, or to changes in another population. They might be able to tell you, at a particular degree of reliability, whether those changes are likely to have been caused by your intervention or program, or by another factor, known or unknown. And they can identify relationships among different variables, which may or may not mean that one causes another.

Qualitative data

Unlike numbers or “hard data,” qualitative information tends to be “soft,” meaning it can’t always be reduced to something definite. That is in some ways a weakness, but it’s also a strength. A number may tell you how well a student did on a test; the look on her face after seeing her grade, however, may tell you even more about the effect of that result on her. That look can’t be translated to a number, nor can a teacher’s knowledge of that student’s history, progress, and experience, all of which go into the teacher’s interpretation of that look. And that interpretation may be far more valuable in helping that student succeed than knowing her grade or numerical score on the test.

Qualitative data can sometimes be changed into numbers, usually by counting the number of times specific things occur in the course of observations or interviews, or by assigning numbers or ratings to dimensions (e.g., importance, satisfaction, ease of use).

The challenges of translating qualitative into quantitative data have to do with the human factor.  Even if most people agree on what 1 (lowest) or 5 (highest) means in regard to rating “satisfaction” with a program, ratings of 2, 3, and 4 may be very different for different people.  Furthermore, the numbers say nothing about why people reported the way they did. One may dislike the program because of the content, the facilitator, the time of day, etc. The same may be true when you’re counting instances of the mention of an event, such as the onset of a new policy or program in a community based on interviews or archival records.  Where one person might see a change in program he considers important another may omit it due to perceived unimportance.

Qualitative data can sometimes tell you things that quantitative data can’t.  It may reveal why certain methods are working or not working, whether part of what you’re doing conflicts with participants’ culture, what participants see as important, etc.  It may also show you patterns – in behavior, physical or social environment, or other factors – that the numbers in your quantitative data don’t, and occasionally even identify variables that researchers weren’t aware of.

It is often helpful to collect both quantitative and qualitative information.

Quantitative analysis is considered to be objective – without any human bias attached to it – because it depends on the comparison of numbers according to mathematical computations.  Analysis of qualitative data is generally accomplished by methods more subjective – dependent on people’s opinions, knowledge, assumptions, and inferences (and therefore biases) – than that of quantitative data.  The identification of patterns, the interpretation of people’s statements or other communication, the spotting of trends – all of these can be influenced by the way the researcher sees the world.  Be aware, however, that quantitative analysis is influenced by a number of subjective factors as well.  What the researcher chooses to measure, the accuracy of the observations, and the way the research is structured to ask only particular questions can all influence the results, as can the researcher’s understanding and interpretation of the subsequent analyses.

Why should you collect and analyze data for your evaluation?

Part of the answer here is that not every organization – particularly small community-based or non-governmental ones – will necessarily have extensive resources to conduct a formal evaluation.  They may have to be content with less formal evaluations, which can still be extremely helpful in providing direction for a program or intervention.  An informal evaluation will involve some data gathering and analysis. This data collection and sensemaking is critical to an initiative and its future success, and has a number of advantages.

• The data can show whether there was any significant change in the dependent variable(s) you hoped to influence. Collecting and analyzing data helps you see whether your intervention brought about the desired results

The term “significance” has a specific meaning when you’re discussing statistics.  The level of significance of a statistical result is the level of confidence you can have in the answer you get.  Generally, researchers don’t consider a result significant unless it shows at least a 95% certainty that it’s correct (called the .05 level of significance, since there’s a 5% chance that it’s wrong).  The level of significance is built into the statistical formulas: once you get a mathematical result, a table (or the software you’re using) will tell you the level of significance.

Thus, if data analysis finds that the independent variable (the intervention) influenced the dependent variable at the .05 level of significance, it means there’s a 95% probability or likelihood that your program or intervention had the desired effect.  The .05 level is generally considered a reasonable result, and the .01 level (99% probability) is considered about as close to certainty as you are likely to get. A 95% level of certainty doesn’t mean that the program works on 95% of participants, or that it will work 95% of the time.  It means that there’s only a 5% possibility that it isn’t actually what’s influencing the dependent variable(s) and causing the changes that it seems to be associated with.

• They can uncover factors that may be associated with changes in the dependent variable(s). Data analyses may help discover unexpected influences; for instance, that the effort was twice as large for those participants who also were a part of a support group. This can be used to identify key aspects of implementation.
• They can show connections between or among various factors that may have an effect on the results of your evaluation. Some types of statistical procedures look for connections (“correlations” is the research term) among variables.  Certain dependent variables may change when others do.  These changes may be similar – i.e., both variables increase or decrease (e.g., as children’s proficiency at reading increases, the amount of reading they do also increases).  Or the opposite may be observed – i.e. the two variables change in opposite directions (as the amount of exercise they engage in increases, peoples’ weight decreases).  Correlations don’t mean that one variable causes another, or that they both have the same cause, but they can provide valuable information about associations to expect in an evaluation.
• They can help shed light on the reasons that your work was effective or, perhaps, less effective than you’d hoped.  By combining quantitative and qualitative analysis, you can often determine not only what worked or didn’t, but why.  The effect of cultural issues, how well methods are used, the appropriateness of your approach for the population – these as well as other factors that influence success can be highlighted by careful data collection and analysis.  This knowledge gives you a basis for adapting and changing what you do to make it more likely you’ll achieve the desired outcomes in the future.
• They can provide you with credible evidence to show stakeholders that your program is successful, or that you’ve uncovered, and are addressing limitations. Stakeholders, such as funders and community boards, want to know their investments are well spent. Showing evidence of intermediate outcomes (e.g. new programs and policies) and longer-term outcomes (e.g., improvements in education or health indicators) is becoming increasingly important to receiving – and retaining – funding.
• Their use shows that you’re serious about evaluation and about improving your work. Being a good trustee or steward of community investment includes regular review of data regarding progress and improvement.
• They can show the field what you’re learning, and thus pave the way for others to implement successful methods and approaches. In that way, you’ll be helping to improve community efforts and, ultimately, quality of life for people who benefit.

When and by whom should data be collected and analyzed?

As far as data collection goes, the “when” part of this question is relatively simple: data collection should start no later than when you begin your work – or before you begin in order to establish a baseline or starting point – and continue throughout.  Ideally, you should collect data for a period of time before you start your program or intervention in order to determine if there are any trends in the data before the onset of the intervention. Additionally, in order to gauge your program’s longer-term effects, you should collect follow-up data for a period of time following the conclusion of the program.

The timing of analysis can be looked at in at least two ways: One is that it’s best to analyze your information when you’ve collected all of it, so you can look at it as a whole. The other is that if you analyze it as you go along, you’ll be able to adjust your thinking about what information you actually need, and to adjust your program to respond to the information you’re getting. Which of these approaches you take depends on your research purposes.  If you’re more concerned with a summative evaluation – finding out whether your approach was effective, you might be more inclined toward the first.  If you’re oriented toward improvement – a formative evaluation – we recommend gathering information along the way. Both approaches are legitimate, but ongoing data collection and review can particularly lead to improvements in your work.

The “who” question can be more complex. If you’re reasonably familiar with statistics and statistical procedures, and you have the resources in time, money, and personnel, it’s likely that you’ll do a somewhat formal study, using standard statistical tests. (There’s a great deal of software – both for sale and free or open-source – available to help you.)

If that’s not the case, you have some choices:

• You can hire or find a volunteer outside evaluator, such as from a nearby college or university, to take care of data collection and/or analysis for you.
• You can conduct a less formal evaluation. Your results may not be as sophisticated as if you subjected them to rigorous statistical procedures, but they can still tell you a lot about your program.  Just the numbers – the number of dropouts (and when most dropped out), for instance, or the characteristics of the people you serve – can give you important and usable information.
• You can try to learn enough about statistics and statistical software to conduct a formal evaluation yourself. (Take a course, for example.)
• You can collect the data and then send it off to someone – a university program, a friendly statistician or researcher, or someone you hire – to process it for you.
• You can collect and rely largely on qualitative data.  Whether this is an option depends to a large extent on what your program is about. You wouldn’t want to conduct a formal evaluation of effectiveness of a new medication using only qualitative data, but you might be able to draw some reasonable conclusions about use or compliance patterns from qualitative information.
• If possible, use a randomized or closely matched control group for comparison. If your control is properly structured, you can draw some fairly reliable conclusions simply by comparing its results to those of your intervention group.  Again, these results won’t be as reliable as if the comparison were made using statistical procedures, but they can point you in the right direction.  It’s fairly easy to tell whether or not there’s a major difference between the numbers for the two or more groups.  If 95% of the students in your class passed the test, and only 60% of those in a similar but uninstructed control group did, you can be pretty sure that your class made a difference in some way, although you may not be able to tell exactly what it was that mattered. By the same token, if 72% of your students passed and 70% of the control group did as well, it seems pretty clear that your instruction had essentially no effect, if the groups were starting from approximately the same place.

Who should actually collect and analyze data also depends on the form of your evaluation.  If you’re doing a participatory evaluation, much of the data collection - and analyzing - will be done by community members or program participants themselves. If you’re conducting an evaluation in which the observation is specialized, the data collectors may be staff members, professionals, highly trained volunteers, or others with specific skills or training (graduate students, for example).  Analysis also could be accomplished by a participatory process. Even where complicated statistical procedures are necessary, participants and/or community members might be involved in sorting out what those results actually mean once the math is done and the results are in. Another way analysis can be accomplished is by professionals or other trained individuals, depending upon the nature of the data to be analyzed, the methods of analysis, and the level of sophistication aimed at in the conclusions.

How do you collect and analyze data?

Whether your evaluation includes formal or informal research procedures, you’ll still have to collect and analyze data, and there are some basic steps you can take to do so.

Implement your measurement system

We've previously discussed designing an observational system to gather information. Now it’s time to put that system in place.

• Clearly define and describe what measurements or observations are needed. The definition and description should be clear enough to enable observers to agree on what they’re observing and reliably record data in the same way.
• Select and train observers. Particularly if this is part of a participatory process, observers need training to know what to record; to recognize key behaviors, events, and conditions; and to reach an acceptable level of inter-rater reliability (agreement among observers).
• Conduct observations at the appropriate times for the appropriate period of time. This may include reviewing archival material; conducting interviews, surveys, or focus groups; engaging in direct observation; etc.
• Record data in the agreed-upon ways. These may include pencil and paper, computer (using a laptop or handheld device in the field, entering numbers into a program, etc.), audio or video, journals, etc.

Organize the data you’ve collected

How you do this depends on what you’re planning to do with it, and on what you’re interested in.

• Enter any necessary data into the computer. This may mean simply typing comments, descriptions, etc., into a word processing program, or entering various kinds of information (possibly including audio and video) into a database, spreadsheet, a GIS (Geographic Information Systems) program, or some other type of software or file.
• Transcribe any audio- or videotapes. This makes them easier to work with and copy, and allows the opportunity to clarify any hard-to-understand passages of speech.
• Score any tests and record the scores appropriately.
• Sort your information in ways appropriate to your interest. This may include sorting by category of observation, by event, by place, by individual, by group, by the time of observation, or by a combination or some other standard.
• When possible, necessary, and appropriate, transform qualitative into quantitative data. This might involve, for example, counting the number of times specific issues were mentioned in interviews, or how often certain behaviors were observed.

Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

We’ve referred several times to statistical procedures that you can apply to quantitative data. If you have the right numbers, you can find out a great deal about whether your program is causing or contributing to change and improvement, what that change is, whether there are any expected or unexpected connections among variables, how your group compares to another you’re measuring, etc.

There are other excellent possibilities for analysis besides statistical procedures, however. A few include:

• Simple counting, graphing and visual inspection of frequency or rates of behavior, events, etc., over time.
• Using visual inspection of patterns over time to identify discontinuities (marked increases, decreases) in the measures over time (sessions, weeks, months).
• Calculating the mean (average), median (midpoint), and/or mode (most frequent) of a series of measurements or observations. What was the average blood pressure, for instance, of people who exercised 30 minutes a day at least five days a week, as opposed to that of people who exercised two days a week or less?
• Using qualitative interviews, conversations, and participant observation to observe (and track changes in) the people or situation. Journals can be particularly revealing in this area because they record people’s experiences and reflections over time.
• Finding patterns in qualitative data.  If many people refer to similar problems or barriers, these may be important in understanding the issue, determining what works or doesn’t work and why, or more.
• Comparing actual results to previously determined goals or benchmarks. One measure of success might be meeting a goal for planning or program implementation, for example.

Take note of any significant or interesting results

Depending on the nature of your research, results may be statistically significant (the 95% or better certainty that we discussed earlier), or simply important or unusual.  They may or may not be socially significant (i.e., large enough to solve the problem).

There are a number of different kinds of results you might be looking for.

• Differences within people or groups. If you have repeated measurements for individuals/groups over time, we can see if there are marked increases/decreases in the (frequency, rate) of behavior (events, etc.) following introduction of the program or intervention. When the effects are seen when and only when the intervention is introduced – and if the intervention is staggered (delayed) across people or groups – this increases our confidence that the intervention, and not something else, is producing the observed effects.
• Differences between or among two or more groups.  If you have one or more randomized control groups in a formal study (groups that are drawn at random from the same population as the group in your program, but are not getting the same program or intervention, or are getting none at all), then the statistical significance of differences between or among the groups should tell you whether your program has any more influence on the dependent variable(s) than what’s experienced by the other groups.
• Results that show statistically significant changes. With or without a control or comparison group, many statistical procedures can tell you whether changes in dependent variables are truly significant (or not likely due to chance). These results may say nothing about the causes of the change (or they may, depending on how you’ve structured your evaluation), but they do tell you what’s happening, and give you a place to start.
• Correlations. Correlation means that there are connections between or among two or more variables. Correlations can sometimes point to important relationships you might not have predicted. Sometimes they can shed light on the issue itself, and sometimes on the effects of a group’s cultural practices. In some cases, they can highlight potential causes of an issue or condition, and thus pave the way for future interventions.

Correlation between variables doesn’t tell you that one necessarily causes the other, but simply that changes in one have a relationship to changes in the other.  Among American teenagers, for instance, there is probably a fairly high correlation between an increase in body size and an understanding of algebra.  This is not because one causes the other, but rather the result of the fact that American schools tend to begin teaching algebra in the seventh, eighth, or ninth grades, a time when many 12-, 13-, and 14-year-olds are naturally experiencing a growth spurt.

On the other hand, correlations can reveal important connections.  A very high correlation between, for instance, the use of a particular medication and the onset of depression might lead to the withdrawal of that medication, or at least a study of its side effects, and increased awareness and caution among doctors who prescribe it.  A very high correlation between gang membership and having a parent with a substance use problem may not reveal a direct cause-and-effect relationship, but may tell you something important about who is more at risk for substance use.

• Patterns. In both quantitative and qualitative information, patterns often emerge: certain health conditions seem to cluster in particular geographical areas; people from a particular group behave in similar ways; etc.  These patterns may not be specifically what you were looking for or expected to find, but they may either be important in themselves or shed light on the areas you’re interested in.  In some cases, you may need to subject them to statistical procedures (regression analysis, for example) to see if, in fact, they’re random, or if they constitute actual patterns.
• Obvious important findings.  Whether as a result of statistical analysis, or of examination of your data and application of logic, some findings may stand out.  If 70% of a group of overweight participants in a healthy eating and physical activity program lowered their weight and blood pressure significantly, compared to only 20% of a similar group not in the program, you can probably assume that program may have been effective.  If there’s no change whatsoever in education outcomes after two years of your education program, then you’re either running an ineffective program, or you’re simply not reaching those who are most likely to have poorer outcomes (which can also be interpreted to mean you’re running an ineffective program.)

Not all important findings will necessarily tell you whether your program worked, or what is the most effective method.  It might be obvious from your data collection, for instance, that, while violence or roadway injuries may not be seen as a problem citywide, they are much higher in one or more particular areas, or that the rates of diabetes are markedly higher for particular groups or those living in areas with greater disparities of income. If you have the resources, it’s wise to look at the results of your research in a number of different ways, both to find out how to improve your program, and to learn what else you might do to affect the issue.

Interpret the results

Once you’ve organized your results and run them through whatever statistical or other analysis you’ve planned for, it’s time to figure out what they mean for your evaluation. Probably the most common question that evaluation research is directed toward is whether the program being evaluated works or makes a difference. In research terms, that often translates to “What were the effects of the independent variable (the program, intervention, etc.) on the dependent variable(s) (the behavior, conditions, or other factors it was meant to change)?”  There are a number of possible answers to this question:

• Your program had exactly the effects on the dependent variable(s) you expected and hoped it would. Statistics or other analysis showed clear positive effects at a high level of significance for the people in your program and – if you used a multiple-group design – none, or far fewer, of the same effects for a similar control group and/or for a group that received a different intervention with the same purpose. Your early childhood education program, for instance, greatly increased development outcomes for children in the community, and also contributed to an increase in the percentage of children succeeding in school.
• Your program had no effect.  Your program produced no significant results on the dependent variable, whether alone or compared to other groups. This would mean no change as a result of your program or intervention.
• Your program had a negative effect. For instance, intimate partner violence increased (or at least appeared to) as a result of your intervention. (It is relatively common for reported events, such as violence or injury, to increase when the intervention results in improved surveillance and ease of reporting).
• Your program had the effects you hoped for and other effects as well.
  • These effects might be positive. Your youth violence prevention program, for instance, might have resulted in greatly reduced violence among teens, and might also have resulted in significantly improved academic performance for the kids involved.
  • These effects might be neutral. The same youth violence prevention program might somehow result in youth watching TV more often after school.
  • These effects might be negative. (These effects are usually called unintended consequences.) Youth violence might decrease significantly, but the incidence of teen pregnancies or alcohol consumption among youth in the program might increase significantly at the same time.
  • These effects might be multiple, or mixed.For instance, a program to reduce HIV/AIDS might lower rates of unprotected sex but might also increase conflict and instances of partner violence. Your program had no effect or a negative effect and other effects as well. As with programs with positive effects, these might be positive, neutral, or negative; single or multiple; or consistent or mixed.

If your analysis gives you a clear indication that what you’re doing is accomplishing your purposes, interpretation is relatively simple: You should keep doing it, while trying out ways to make it even more effective, or while aiming at other related issues as well.

As we discuss elsewhere in the Community Tool Box, good programs are dynamic -- constantly striving to improve, rather than assuming that what they’re doing is as good as it can be.

If your analysis shows that your program is ineffective or negative, however – or, for that matter, if a positive analysis leaves you wondering how to make your successful efforts still more successful – interpretation becomes more complex. Are you using an absolutely wrong approach? Are you using an approach that could be effective, but is poorly implement? Is there a particular contributing factor you’re failing to take into account?  Are there barriers to success – of culture, experience, personal characteristics, systematic discrimination – present in the population from which participants are drawn?  Are there particular components or elements you can change to make your program more effective, or should you start again from scratch?  What should you address to make a good program better?

Careful and insightful interpretation of your data may allow you to answer questions like these. You may be able to use correlations, for instance, to generate hypotheses about your results. If positive or negative changes in particular variables are consistently associated with positive or negative changes in other variables, the two may be connected. (The word “may” is important here. The two may be connected, but they may not, or both may be related to a third variable that you’re not aware of or that you consider trivial.) Such a connection can point the way toward a factor (e.g., access to support) that is causing the changes in both variables, and that must be addressed to make your program successful. Correlations may also indicate patterns in your data, or may lead to an unexpected way of looking at the issue you’re addressing.

You can often use qualitative data to understand the meaning of an intervention, and people’s reactions to the results.The observation that participants are continually suffering from a variety of health problems may be traced, through qualitative data, to nutrition problems (due either to poverty or ignorance) or to lack of access to health services, or to cultural restrictions (some Muslim women may be unwilling – or unable because of family prohibition – to accept care and treatment from male doctors, for example).

Once you have organized your data, both statistical results and anything that can’t be analyzed statistically need to be analyzed logically. This may not give you convincing information but it will almost undoubtedly give you some ideas to follow up on, and some indications of connections and avenues you might not yet have considered. It will also show you some additional results – people reacting differently than before to the program, for example. The numbers can tell you whether there is change, but they can’t always tell you what causes it or why (although they sometimes can), or why some people benefit while others don’t. Those are often matters for logical analysis, or critical thinking.
 
Analyzing and interpreting the data you’ve collected brings you, in a sense, back to the beginning. You can use the information you’ve gained to adjust and improve your program or intervention, evaluate it again, and use that information to adjust and improve it further, for as long as it runs. You have to keep up the process to ensure that you’re doing the best work you can and encouraging changes in individuals, systems, and policies that make for a better and healthier community.

You have to become a cultural detective to understand your initiative, and, in some ways, every evaluation is an anthropological study.

In Summary

The heart of evaluation research is gathering information about the program or intervention you’re evaluating and analyzing it to determine what it tells you about the effectiveness of what you’re doing, as well as about how you can maintain and improve that effectiveness.

Collecting quantitative data – information expressed in numbers – and subjecting it to a visual inspection or formal statistical analysis can tell you whether your work is having the desired effect, and may be able to tell you why or why not as well.  It can also highlight connections (correlations) among variables, and call attention to factors you may not have considered.

Collecting and analyzing qualitative data – interviews, descriptions of environmental factors, or events, and circumstances – can provide insight into how participants experience the issue you’re addressing, what barriers and advantages they experience, and what you might change or add to improve what you do.

Once you’ve gained the knowledge that your information provides, it’s time to start the process again. Use what you’ve learned to continue to evaluate what you do by collecting and analyzing data, and continually improve your program.

What do we mean by collecting data?

___Collecting data means putting your design for a measurement system into operation

___Collecting data involves gathering information through observation, interviews, testing, surveys, and/or other means; recording it in appropriate ways; and organizing it so that it’s easier to work with

What do we mean by analyzing data?

___Analyzing data involves examining the information you’ve collected in ways that reveal the relationships, patterns, trends, etc. that can be found within it

___Data can be quantitative (collected as numbers) or qualitative (collected as narrative information, records, journal notes, etc.)

___Quantitative data is usually analyzed by subjecting it to one or more graphical displays or statistical operations that demonstrate the significance of relationships among variables

___Data are also usually analyzed logically, by looking for patterns and relationships revealed within them

___Qualitative data can sometimes be turned into quantitative data by, for instance, counting (e.g., the number of times a behavior occurs in various circumstances) or by rating on a number scale such dimensions as importance, satisfaction, or quality (e.g., the quality of housing or quality of life in neighborhoods)

___A combination of quantitative and qualitative data often yields the best overall picture

Why should you collect and analyze data for your evaluation?

___This can show whether or not there was actually any significant change in the dependent variable(s) you hoped to influence

___This can show connections between or among various factors that may have an effect on the results of your evaluation

___This can imply or show the reasons that your work was effective or ineffective

___This can provide you with credible evidence to show funders and the community that your program is successful, or that you’ve uncovered, and are fixing, the elements that are barriers to success

___This can show that you’re serious about evaluation and about improving your work

___This can show the field that what you’re doing works well, and thus pave the way for others to use similar methods and approaches as best practices

When and by whom should data be collected and analyzed?

___Data collection should start no later than when you begin your work and continue throughout

___If you want to see if any change is part of a long-term trend, you should collect data for some time before your program actually starts

___If you want to understand long-term effects, you should collect data on participants for some time after they leave your program

___Data should be collected and analyzed by people who are capable of doing so

___Data collection and analysis can be done by anyone from community members who have been trained to professionals with experience in conducting studies

How do you collect and analyze data?

___Implement the observational system you’ve planned

___Organize the data you’ve collected

___Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate

___Take note of any significant or interesting results

___Interpret the results

Example #1: Santa Monica’s Use of Data to Lift All Boats: A Deeper Look

Four years ago, Santa Monica took a data-driven approach to become a true City of Wellbeing where everyone thrives. The city used a Wellbeing Index to produce a data-based guide for steering policy. The city used existing information from city departments and fresh input from surveys. The Wellbeing index sets a framework for priorities and promotes initiatives on health and wellness.

Contributed by Lia Thompson, University of Kansas, Community Tool Box Intern.

 

 

• What is ethnographic information?

• Why might ethnographic information be important to evaluation?

• When might you want to collect ethnographic information?

• Who should collect and interpret ethnographic information?

• How do you gather ethnographic information?

• How do you interpret ethnographic information?

Someone watching a baseball game or cricket match for the first time might be baffled by seeing a batter perform a series of actions before each pitch – touching his helmet, knocking the dirt off his shoes, stepping out of the batters’ box. Any long-time fan, however, knows what’s happening: first, many players have superstitious or calming rituals they perform when batting; second, some of the batter’s motions may be acknowledgements of signals from coaches; and third, there is an intense psychological battle between hitter and pitcher that plays itself out in stalling, distracting motions and other actions meant to make it difficult for the opposition. Without this knowledge, the first-time observer sees only a series of random, seemingly purposeless actions and could be left with the impression that players are afflicted with a strange disease.

Understanding why people do what they do can be tremendously important in trying to understand a culture. When anthropologists want to understand a culture, they often immerse themselves in it, so they can study it from the inside. Margaret Mead popularized this type of research in the 1920s in Samoa, where she lived for nine months as part of an island community. A researcher who is embedded in a culture in this way can gain insight not only into what happens – how people go about their daily lives, what rituals they perform, what their celebrations are like, etc. – but also how members of the culture view what happens. They can begin to piece together the culture’s structure – the beliefs and assumptions that underlie everything people do and the world view that creates and supports them.

Depending on what you’re trying to change, your evaluation may have to look at more than just the behavior of individuals. The whole context of the community may be involved if, for instance, you’re attempting to address social exclusion based on race or gender or to alter a culture of violence. Deep knowledge of a community, or a segment of it, can’t necessarily be gained by administering tests or counting the number of times individuals perform certain actions. Rather, it takes an understanding of the community’s whole world view and what leads to it – whether religion, social or political factors, economics, some combination, or something else entirely. That kind of deep knowledge may best be gained by immersion in the culture of the community you’re serving. This section will explore the use of ethnographic research – the kind of cultural immersion we’ve been describing – as a method for evaluation.

What is ethnographic information?

Ethnographic information is data about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view as much as possible. It’s meant to provide an understanding of the culture from the inside – to describe and explain its assumptions, customs, and accepted behavior from the point of view of those who live by them.

These can include community norms, health conditions and knowledge, political realities, religion, economics, and world view.

The terms “culture,” “group,” and “community” are all used in this section to refer to a group that can be seen as having its own set of norms and practices and the formal or informal rules that guide them. By that definition, we may be referring to a culture as large as that of a whole ethnic or religious group (e.g., Latinos, Hindus, Navajos) or as small as that of a single elementary school classroom. It may encompass people of many backgrounds (e.g., English as a Second or Other Language students; low-income families) or people from a single neighborhood or informal settlement. No culture or group or community is completely homogeneous, of course, – not everyone thinks and believes exactly the same things as everyone else – but you can generally find basic cultural elements on which most people agree, and those are what define the culture.

Community norms. These encompass what the community or group considers normal and acceptable in everyday behavior and activity. We can look at community norms through a number of filters:

• Gender roles. These define what men and women are each expected to do, how they’re expected to act, dress, etc., and how they are expected to relate to one another in different circumstances.
• Social structure. Social structure establishes who is accorded higher status in the community, how one gains status, how social groups are made up and what forms they take, and who are the social arbiters – those who make the rules or set the standards. It also defines who is considered lower status, who is socially excluded, and how possible it is to improve one’s status through merit, entrepreneurship, illegal activity, education, etc. It also determines whether and how the community’s world is divided up into social classes or hierarchies, and whether some are perceived to have higher status than others.
• Behavior. What is considered acceptable behavior and language in public? This encompasses, for example, how people greet one another (and how much this depends on status, gender, age, etc.). It also refers to how they respond to real or perceived compliments and insults (and what are considered compliments and insults), ritual or other behavior unique to the population, and behavior accepted or condoned (or encouraged) by the community. Most communities and groups also set standards for behavior that is so unacceptable or harmful to the group, whether public or private, that it is punished in official ways, including exclusion from the group.

In some communities and cultures, behaviors that people in other communities and cultures have come to consider unhealthy or antisocial are seen as acceptable – if sometimes regrettable – or normal. Domestic and other physical violence, alcoholism, racism and racial discrimination, smoking, traffic violations – even slavery – are examples of illegal, unethical, unhealthy, or undesirable behaviors that have been or still are tolerated in some communities.

Understanding the level of tolerance or encouragement for these kinds of behaviors is absolutely necessary for understanding how to change them. An ethnographic investigation can show you how people view your efforts, and what you can do to make it more likely that they’ll respond to your program or initiative.

Health conditions and knowledge. What do the folks you’re concerned with know about health and health conditions in general and in the community, and what healthy (or unhealthy) practices do they engage in? Some areas to examine:

• Knowledge about and practices related to prenatal, infant, and child health.
• The understanding and frequency of the effects of drugs, alcohol, and tobacco, and of addiction in general.
• Awareness of chronic or other health conditions that are common in the community, and of environmental threats or benefits to health.
• Knowledge about nutrition and physical activity, and their relationship to health, as well as practices that reflect that knowledge.
• Engagement in regular health screenings, vaccinations, and other health protection strategies.
• Relationships with a regular primary care provider whom people see regularly.
• The percentage of the population with access to health care.

In the developing world, how many people have access to clean drinking water, sanitation, and basic medications? Do they know how to treat dysentery and other common diseases in order to prevent infant mortality? How far must they go to see a health care provider, and will they consider doing so?

 

Power and Political realities. Who holds positions of power? Who are the opinion leaders and policymakers? How is power exercised – formally or informally, fairly or ruthlessly, for self-interest or the public good? How are civic goals or actions accomplished (or are they at all)? Who is or feels powerless? Whose concerns are most often attended to? Are laws enforced differently for some people than for others? The answers to these and similar questions can help paint a clear picture of the power relationships and political realities of a community or group.

Religion. Faith can be a complex issue, since the extent of its influence, depending upon the population, can range from none to all-encompassing. How large a role does religion play in the daily life of the community? How does religion affect daily life? Asking more specific questions along these lines can help you develop a picture of religion’s role in the community:

• Are most people in the community of one faith, or is it made up of members of a number of different faiths, as well as many with no religious affiliation?
• What roles do religious institutions play – spiritual guide, social center, political adviser, information source, community leader (in the person of the clergy), or some or all of the above?
• If most of the community subscribes to a particular religion, how inclusive and tolerant is it of different beliefs?
• What are the major elements of the belief system (e.g. tolerance, alms-giving)?
• Do people try to convert others?
• Do they understand and believe in the separation of church and state?

Economics. What is the community or group’s economic situation?

• Is there a problem with unemployment or underemployment?
• Is the community dependent on a single industry, occupation, or employer (e.g., manufacturing, farming, a military base)?
• Is there discrimination in hiring or promotion? Against or in favor of whom?
• How does the community view education in relation to economic opportunity, and how good is its education system?
• How strong is its work ethic?
• How easy is it to advance economically?
• Is decent housing affordable and available?
• Are most people able to provide for their families’ basic needs, or is poverty a serious problem? Homelessness? Hunger?

“Basic needs” is a relative term. In a village in a developing country, it may mean one or two meals a day, a simple one-room shelter, access to water and firewood or other cooking fuel, one change of clothing (which may or may not include shoes), and occasional basic health care. In the developed world, it usually means three substantial meals a day; a solid, enclosed, multi-room dwelling that includes a modern kitchen and bathroom; decent clothes for all weather conditions; regular access to health care; and even such amenities as a telephone, a TV set, and – at least in rural areas – a vehicle. The definition of “basic needs” (beyond what it takes simply to survive) depends on what is normal for the rest of the society.

• How large are the disparities in income and wealth? What groups experience greater disparities?

World view. This idea overlaps with several of the categories above and encompasses the way individuals in a community (and, therefore, often the community as a whole) see themselves and their lives. A “world view” is really the set of assumptions about the “rightness” of things that everyone applies to the world around her. Some of the most important elements of world view are:

• Attitudes toward authority. There are three elements here: how authority is defined; which definition, if any, is considered legitimate; and how authority is viewed and treated. The definitions can vary from authority that comes from official position (judges, police, teachers, politicians, bureaucrats) to authority enforced by physical or economic power (warlords, gang leaders, employers and supervisors) to authority conferred by knowledge or respect and position in the social structure (elders, doctors, business leaders, clergy, academics, individuals known for their integrity and intelligence). Who is considered legitimate authority depends on loyalty and solidarity (you give your allegiance to “one of your own”). So the gang leader of the same ethnic background as most people in the neighborhood may wield more authority than the police. It also relates to who has real power (that same gang leader can have you beaten – or worse – if you disobey) and who can best advance the community’s interests. For instance, many Palestinians voted for Hamas in the 2006 election not specifically because they supported the violent overthrow of Israel, but because they believed that Hamas could improve economic conditions for Palestinians in the West Bank and Gaza. The basic view of authority might range from unquestioning respect (authority is always right and must be trusted and obeyed) to profound lack of respect (authority is always self-serving and wrong and must be resisted actively or passively).

Obedience to authority has been of tremendous importance as a topic in history. The obvious example is that of Nazi Germany in the 1930s and ’40s, where much of the population blindly followed an authoritarian government into world war, wholesale murder, and genocide. We have since apparently seen many other examples, where blind obedience to authority – or the resistance to that obedience – has led to violence, overthrow of governments, and yet more genocide. Street and gang violence can be a result of ignoring authority altogether. The questions of how a culture views authority and whom it considers to be legitimate authority are extremely significant, and may have a great deal to do with your evaluation and the success of your initiative.

• Sense of efficacy and control of the environment. Do people see their lives as totally or mostly under their own control, or out of their control entirely? Do they feel their actions make a difference, or that they can effect change in their lives or in the community? If not with them, where do they think control lies? With politicians? With the wealthy? With those who exercise control by force? If the population is made up of many young people (or socially excluded groups), what is their sense of control or lack of control?

Control might be seen to reside in a person or a group of people in “official” authority (the President, the landlord, the school principal, the boss, parents, police); in an institution or set of institutions (government, welfare, the university, the church, the army); in a set of legal, ethical, or religious principles (the Constitution, Islam, the law, the Hippocratic Oath); in the conventions of a group of which the individual is a member (Latinos, the football or soccer team, upper class, the Catholic Church, etc.); or in the decisions and actions of the individual herself. The way an individual sees her ability to control her life depends on a number of factors, including family background and upbringing, education, emotional issues, and life experience. Obviously, perspectives on this issue will differ among individuals even in a highly homogeneous community, but community norms generally reflect the views of most community members.

• Relationships and trust. Do members of the population see most others as generally trustworthy, or is trust reserved only for family or for those within a small inner circle? Are marital (or, among youth, sexual) relationships viewed differently by men and women, and, if so, how? How are marital relationships viewed in general (positive, negative, mutually supportive, exploitative, etc.)? Are children indulged, beaten, expected to work, treasured, seen as burdens? Are there spoken or unspoken rules for economic and social relationships, and relationships among people of different economic and social classes?
• Obligations and rights. Do most members of the community or group feel obligated to consider the general interest as well as their own? Do they see the group as a community, or does their sense of obligation extend only to family and/or close friends? What do they expect from the community or group? What rights do they feel they can exercise, both in the larger society and in the community?​

The issue of rights and obligations can vary tremendously with the nature of the group or community in question, and with the nature of the society that group is part of. In a farming village in a developing country, people may grow up with a sense of obligation to others because everyone in the village is dependent on everyone else. If one family’s crops fail, the village may feel obligated to feed them, knowing that the next year, it could be one of the helping families that is without food. The same might be true, at least to a certain extent, in small towns in the more industrialized world.

In large cities, neighborhoods may be like villages and small towns, if many residents have grown up there or lived there for many years, or they may be impersonal places where people seldom know their neighbors and feel very little sense of obligation to them. More often, if people feel an obligation at all, it’s to their particular group, however it’s defined, and/or to the society as a whole. Obligation becomes more an expression of principle, whether religious or ethical/philosophical, than a personal connection to a group of others with whom one lives.

The same may be true for rights. In the Global North, we take for granted a certain set of political rights under the law – rights to property, to political choice (voting), to fair and equal treatment under the law, etc. When those rights don’t exist or are often violated, people may define their rights more locally, in terms of the obligations owed to them by those in their group or community (I helped your family get through a bad time last year, and I have a right to your help this year.)

• Tolerance and inclusiveness. Is the community or group inclusive – does it welcome those from other communities and cultures and generally accept their differences? Does it discriminate against or distrust “outsiders”? Are there laws or customs that guarantee everyone’s rights, and are they honored or enforced? Are behaviors that are different tolerated? Who is socially excluded, and what basis (e.g. racial or ethnic group, social class).
• Sense of identification. How do members of the population identify themselves, and with whom do they identify and claim solidarity? Family? Ethnic group? Peer group (teens, musicians, political party, close friends, sports team)? Community (neighborhood, municipality, state, region)? As individuals, or simply as part of the human race?

You may not need to know about all these aspects of the community unless the issue you’re dealing with is so rooted in the culture (racism or gang violence, for example) that you really can’t understand the context of your evaluation without understanding the culture as a whole. It’s more likely that you’re concerned with a specific situation, and you’ll need to focus on gathering ethnographic information as it relates to that situation. (How do young people who leave school view education, for instance? What can you learn from an ethnographic investigation that can make it more attractive to them?)

The major points about ethnographic information are these:

• Ethnographic information is obtained directly from those who live in the community of interest. In the case of gathering information for an evaluation, that means participants in a program or initiative, or the specific group in which they’re members (Black men over 40, Hispanic teens, single mothers, non-English speakers, unemployed workers, etc.) It’s not exactly the same as simple qualitative information, although some researchers see it that way, but rather a specific and more focused instance of qualitative information gathering. (See Chapter 3, Section 15, Qualitative Methods to Assess Community Issues, for more about qualitative data.)
• The gathering of ethnographic information takes place in a natural, rather than an artificial setting. That means it happens in the environment where people in the group normally spend their time. If you’re running a program, you want to understand participants’ normal lives, not just the ways they behave and function in the program.
• Ethnographic information is meant to help you understand a culture from the point of view of its members.
• Ethnographic information is simply descriptive, not judgmental. Valuing may come later, but the point of the research is simply to understand how the individuals you’re concerned with see the world, and how they understand and view what they do. That can give you the entry you need to devise or revise programs in order to effect change and improvement.

Why might ethnographic information be important to evaluation?

In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

The U.S. Census Bureau used ethnographic studies to understand what populations were undercounted and why. The bureau had long acknowledged the difficulty of counting people who were homeless but knew that many others went uncounted as well. By using interviews, observation, surveys, and other methods, investigators found that undercounting resulted from several factors:

• Record-keeping errors (uncounted residences listed as unoccupied or nonexistent, which in fact did exist and were occupied)
• Different definitions of residence (some older or young family members might have their care shared by several siblings or cousins and might live with different people at different times so that their permanent residence was difficult to determine)
• Attempts to avoid losing benefits (men living with women receiving public assistance would simply not be reported)
• General distrust of government, especially among those who were in the country illegally
• Difficulties with reading or understanding the census form
• Difficulties with English (no accommodations for other languages)

As a result, the U.S. Census Bureau decentralized some of the data collection function, put more collectors on the streets, simplified and translated the form, and made other adjustments trying to include as many as possible of those previously uncounted. None of this information would have been found and none of the adjustments to address the issues it demonstrated would have been made without an ethnographic approach in the areas where undercounting was most common.

Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them, as opposed to what staff members and others predict or think about how they’re experiencing them. Successful programs learn how their participants are actually experiencing the program (e.g. delays, how people are treated, time and effort to participate). An ethnographic study can help you understand if and why there’s a gap between how you view your program or initiative and how participants actually see it.

Ethnographic information helps clarify what needs to be addressed in order for participants to respond to your approach. If there are differences in the ways you and the participants see your work, an understanding of their point of view can help you make adjustments that will either make clearer to them what the intent of the program is, or adjust the program so it starts closer to what they perceive and moves toward your shared goals.

Ethnographic information can clarify the issue and its effect on, and importance to, the population of interest. Seeing the issue you’re concerned with from the perspective of current and potential participants – its seriousness, its effect on their lives, its causes, its possible alternatives – can guide the way you frame your program for greatest effectiveness.

Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future. A true picture of the culture of the community or group can inform all your work, and allow you to participate in that culture, to be a better advocate, and to plan and run more effective initiatives in the future.

When might you want to collect ethnographic information?

Ethnographic information is useful, but an ethnographic study, even a very specific one, takes some effort. Especially if you’re part of a small community-based or non-governmental organization, you may not want to spend staff or volunteer time on gathering information that isn’t really necessary. When might ethnographic information in fact be necessary?

• When you’re engaged with a population or cultural group that you’re not familiar with or part of. Ethnographic information can prevent you from unintentionally causing harm, or from ignoring “obvious” factors that everyone in the population takes for granted and that might be key to the success or failure of your program.
• When you’re working with a clearly-defined group that has had a chance to develop its own culture. This might be a class, a church, a gang, a club, a team … any group that has been together long enough to have created norms for itself that might be even slightly different from those of the larger society. The more mainstream the group, the easier it is to miss these norms and to misunderstand how to approach them.
• When an understanding of the context and culture of the community is fundamental to what you’re doing. If you’re addressing an issue, such as racism or violence, it’s essential to understand the real attitudes of the people you’re working with – not only what they think, but why they think it, how they react to various aspects of the issues, what the history is for them, etc. If you don’t understand that at least part of the attraction of a gang is that it functions as a family for kids who often have no real connection or belonging to their actual family, you’re not likely to be able to make any headway in working with gang members.
• When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group. Understanding exactly how that population sees that situation (e.g., a job training program, or an initiative to increase physical activity among children) will make it easier to understand what is and is not working and why.

Who should collect ethnographic information?

This depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc. In some circumstances, anyone trained to do so can become a participant or non-participant observer; in others, you might do best to get the cooperation of one or more key individuals in the community. When outsiders are universally mistrusted, the information might best be gathered by actual members of the community who are trained in interviewing and other data collection techniques.

How do you gather ethnographic information?

Now that you’ve decided to gather and use ethnographic information in your planning and evaluation, how do you go about getting it? The first step is to decide what kinds of information you need. Then you’ll have to determine how to get it, gain the trust of the group you want to learn about, plan your field study, and carry it out.

1. Decide what kinds of information you need. As we’ve discussed, you might want to understand the whole context and content of the group’s culture, or you may only be concerned with a very specific topic. If you’re dealing with a distinct cultural group – one defined by ethnicity or religion, for instance, whose culture has developed over centuries – gaining an understanding of the whole culture may take years. If, on the other hand, the group is part of the general culture – a middle school classroom, a workplace – but has developed a culture of its own within that confined physical and psychological space, a useful ethnographic study may only take days or weeks. In most cases, when you’re gathering ethnographic data for an evaluation, you’ll be looking at culture in relation to a very specific topic.

To conduct ethnographic research on a specific topic, you have to make sure you’ve defined that topic well enough so that you can zero in on it in your observations and conversations. What exactly is it that you need to understand about your population that will inform your evaluation? One question often asked is “How do participants (or members of the population that participants are drawn from) view the program or initiative?” Other related questions might be asked as well: Is the program and its activities culturally appropriate and respectful? Are participants’ goals similar to those of the organization, or are they very different? Are there cultural advantages or barriers that make it more or less likely that the program will succeed or that individuals will be able to participate? What would enhance those advantages or remove those barriers? Are cultural differences between participants and staff or volunteers – or cultural ignorance on one or both sides – getting in the way of program effectiveness?

Suppose, for example, you want to find out how an ESOL (English as a Second or Other Language) program was viewed by students. In this case, you’d be dealing not with a coherent ethnic or religious culture, but with the culture of the classroom, unless all the students came from the same background.

Some questions you might ask:

• How do the ESOL students view education (as something for the upper classes, as necessary for them and their children, as an imposition, as time away from earning a living, as a ticket to a better life, as part of becoming part of the country)?
• How do they view teachers (great respect, fear, intimidation, scorn, confusion)?
• How do they approach their learning (enthusiastically, with a strong work ethic, with no knowledge or experience of how to do it, collaboratively, with fear and trepidation, hopefully)?
• What are the cultural barriers to learning? The cultural advantages? (In many families, for example, family support may be an extremely important factor.)
• What arguments for learning and techniques of teaching do students respond to?
• What do they discuss among themselves?
• Are there intercultural dynamics in the classroom, rooted in people’s cultures, that enhance or detract from the learning environment (gender discrimination or racial or ethnic prejudice, for instance)?

Answers to these and other questions might lead to such adjustments as restructuring classes by language group or ethnicity, changing the vocabulary and topics discussed in the course of lessons, changing the atmosphere in the classroom to be more or less teacher-centered, assigning more or less independent work and more or less group work, changing the teacher-student ratio, etc.

Knowing what kinds of questions you want to answer will help you plan whom you need to gather information from, how long it might take, and – perhaps most important – how you should go about it. (We’ll look at that in more detail when we discuss planning a field study.)

2. Determine what you have the resources to do. If you’re part of a small non-governmental organization (NGO) serving Haitian immigrants, it’s unlikely that you’ll have the money or the time to spend two years in Haiti. You may have the resources to spend a good bit of time in your participants’ neighborhoods, however, learning about their assumptions and their lives. That will probably be far more useful ethnographic information for your evaluation than a years-long study of Haitian culture, anyway.

3. Gain the trust of the group you’re engaged with. If you’re an anthropologist looking for a site to conduct a field study, you’ll have to find an appropriate situation, perhaps get permission to travel there (if you’re venturing onto tribal lands, for instance), find the people you’re looking for, and convince them not only to let you stay but to answer your questions, show and explain to you their ways and customs, and help you understand their lives.

In the case of an evaluation, you won’t have to choose your field study site – it’s the program you’re evaluating and/or the community from which your participants come. But no matter whether you’re an outside evaluator or you’re gathering ethnographic information from your program’s participants’ community (Hispanics, laid-off workers, Haitian immigrants, migrant laborers), you’ll have to establish enough trust so that people will tell and show you what you want to know.

This is easier if you have a presence in the community of concern – if your program has been running for a while and has the community’s support and respect, for instance. You may already know and have the trust and support of many in the community who can vouch for you and what you’re doing.

Often the best strategy is to enter the community under the auspices of someone community members trust – a clergy person, a community leader or tribal leader, a respected long-time community member or elder, a coach, etc. This is often how a new program is introduced to a community.

An ESOL program in a community with a large Portuguese population received a boost when the outreach worker helping to establish it made contact with the president of the local soccer club. The club’s headquarters and teams were central to the social life of the Portuguese community, and when the club president began promoting the program, the community immediately accepted and flocked to it.

If you can’t do that, then you’ll have to gain entry on your own. Find out where you can meet community members – sports events, festivals, bars and cafes, parks, shopping places, etc. Choose a place that’s comfortable for you to start with, since you’re far more likely to make others comfortable with you if you feel comfortable yourself. (If you don’t normally spend time in bars, you may not want to start with bars, for example, unless that’s the only place to meet the population you’re interested in. If you’re a sports fan or athlete, the local soccer field or recreational center may be the best place for you to start.) Also make sure it’s a place that feels safe and comfortable to people you’re likely to approach, so they don’t feel threatened or rudely intruded upon. Once you start meeting people, they’ll introduce you to others.

4. Plan your field study. The core of ethnographic research is the field study. This is essentially your observational system (see Section 3 of this chapter). It involves going into the normal environment of the population of interest and observing and interviewing members there, rather than in an artificial environment such as an office or laboratory, or under artificial conditions such as a test or an experiment. This could involve anything from spending time in the ESOL classroom we’ve talked about to living for years in a West African village to working for a year or two in an auto assembly plant. Most ethnographic information for evaluations is likely to be the kind that focuses on a specific topic and that can be collected over the course of the evaluation – usually a year or less.

By this point, you’ve already done some of the work planning your field study. The basic steps are:

• Decide on your questions. You may have already set these out when you determined what information you needed. They’ll probably be related to, but not the same as, your evaluation questions. They’re meant to inform your evaluation questions by providing such information as how participants perceive the program and their (and your) role in it. Questions might be focused on a specific topic (See “How do ESOL students perceive their classes?” above), or might be more general and relate to the larger elements of the culture. They should make sense in terms of the group you’re concerned with (i.e., they should be able to be answered by observation and/or interviews and other methods of data collection) and in terms of the evaluation (they should give you information that helps you look at what you’re doing and adjust it for the better).
• Determine what method(s) you’ll use to gather information. There are a number of ways to gain insight into the culture of a community or group:
  • Individual or group interviews can range from casual conversation to relatively formal, recorded semi-structured or structured interviews.
  • Participant or non-participant direct observation.
  • Casual interaction
  • Participation in community events, celebrations, work, etc.
  • Surveys. If you use them, they should be open-ended (e.g., Tell me about…), rather than agree/disagree.
  • Third-party informants (others with knowledge).
  • Documents – texts, AV, budgets, posters, brochures, manuals, art, etc. – by, about, important to (religious texts, for example), and directed at (instructions, policies) the group.
  • Demographic data and public records – census, court records, public health data, etc.
  • For more detail on several of these data collection methods, see Chapter 3, Section 15, Qualitative Methods to Assess Community Issues.
• Decide whom you’ll need to contact for information, and how. These may be specific people, chosen for their position in the community, appropriateness for the research at hand, or particular characteristics; or they may simply be members of the general population – whoever you happen to be able to make contact with. Will you contact targeted individuals directly, by mail, e-mail, personal approach, or phone? Will you contact people at random – striking up conversations at community events, approaching people in public places, knocking on doors? Or, will you work through intermediaries – community leaders, clergy, translators, etc.?

Ethical considerations: Ethically, your first obligation is to those whom you’re studying. The essentials include informing people about the existence and purpose of your research and obtaining their permission to be included; giving them the choice of anonymity and honoring their decision; giving them access to the final report if they wish; and being careful not to exploit or harm them.

While ethnographic research is by its nature subjective, you also have ethical obligations to be as truthful as possible about what you see, hear, and experience, and to paint as clear a picture as you can of the culture in question. Your interpretation of your data will undoubtedly be colored by your assumptions, experience, and background (and by the norms of your own culture), but your description of the group and its individual members should be impeccably accurate with biases filtered out. (See Chapter 19, Section 5, Ethical Issues in Community Interventions, for a more detailed discussion of research ethics.)

5. Carry out your fieldwork and keeping fieldnotes. An ethnographic study requires fieldwork – going to where the population you want to find out about is and observing and interacting with them in their own environment. That might be as simple as walking down the street to a particular corner or as complex as flying into a remote Arctic village on a seaplane. Whatever the character of your study, the elements of it will be the same.

Fieldwork is really carrying out the details of your plan – establishing trust, contacting people, and gathering information using the methods you worked out beforehand. You might find yourself changing some of your strategies and adjusting to the conditions and the people as you learn more about them. Your understanding of the culture will evolve as you get deeper into it and observe, listen, and participate more. This evolution should be reflected in your fieldnotes, both as they describe what you’ve seen, heard and taken part in, and as they describe your own reactions to your experience.

Fieldnotes are notes taken about your conversations, observations, and other data collection. They should be taken during or as soon as possible after each visit, conversation, experience, etc., before memory fades or changes. Where possible, audio and/or video recording is helpful to get an accurate rendering of what was said and done.

Elizabeth Chiseri-Strater and Bonnie Stone Sunstein (FieldWorking: Reading and Writing Research, 1997. P. 73. Blair Press: Upper Saddle River, NJ.) propose a list of elements that should be included in all fieldnotes:

• Date, time, and place of observation.
• Specific facts, numbers, details of what happens at the site.
• Sensory impressions: sights, sounds, textures, smells, tastes.
• Personal responses to the fact of recording fieldnotes.
• Specific words, phrases, summaries of conversations, and insider language.
• Questions about people or behaviors at the site for future investigation.
• Page numbers to help keep observations in order.

Fieldnotes come in four categories, which should be separated to reflect different forms and functions (from How to Do Ethnographic Research: A Simplified Guide, by Barbara Hall (U. of Pennsylvania):

• Jottings are the brief words or phrases written down while at the fieldsite or in a situation about which more complete notes will be written later. Usually recorded in a small notebook, jottings are intended to help you remember things you want to include when you write the full-fledged notes. They’re the kind of notes that newspaper reporters take when they’re talking to sources about a story. Although not all research situations are appropriate for writing jottings all the time, they do help a great deal when sitting down to write afterwards.
• Description of everything you can remember regarding the occasion you are writing about - a meal, a ritual, a meeting, a sequence of events, etc. While it is useful to focus primarily on things you did or observed that relate to the guiding question, some amount of general information is also helpful. This information might help in writing an overall description of the site later, but it may also help to link related factors to one another or to point out useful research directions later.
• Analysis of what you learned in the setting regarding your guiding question and other related points. This is how you will make links between details and the larger things you are learning about how culture works in this context. What themes can you begin to identify regarding your guiding question? What questions do you have to help focus your observation on subsequent visits? Can you begin to draw preliminary connections or potential conclusions based on what you learned?
• Personal reflection on what you learned. What was it like for you to be doing this research in this context? What felt comfortable for you about being in this situation and what felt uncomfortable? In what ways did you connect with informants, and in what ways didn't you? While this is extremely important information, be especially careful to separate it from analysis.

In evaluation, this information can help you identify your biases, and may make it clearer why your program is less effective than you’d like, or how it’s challenging or in conflict with some of the participants’ important beliefs or assumptions. (Challenging beliefs and assumptions isn’t necessarily wrong, but denying them – even if you find them unacceptable or clearly misguided in some way – is usually counterproductive. Challenging is asking people to reassess their actions or assumptions; denying is telling them they’re wrong or stupid.) If you’re ignoring your participants’ culture, or presenting things in ways that they’re likely, for cultural reasons, to understand differently, you may not be reaching them, and may, in fact, be driving them away.

These notes are the basis for the study, so their timeliness, completeness, and accuracy are extremely important.

How do you interpret ethnographic information?

The ethnographic information you’ve gathered won’t do you any good unless you use it to understand the dynamics of your program and the results of your evaluation. You can begin by organizing your fieldnotes and whatever other information you’ve collected.

Organize the final version of your completed notes. Categorize them by patterns, by demographics (gender, age, education level, socio-economic status, etc.), by topic or theme– whatever makes sense in relation to what you were trying to find out. Sometimes, this may mean categorizing them in more than one way. Often, just the act of putting your notes together in particular ways can reveal patterns or highlight issues you weren’t previously aware of.

Starting from the point of view of your original interest, write up a description of the culture as you understand it to be seen by its members. Depending on your purposes, this might range from a complete description of the culture to a very narrow description of how members of that culture – a village, an office, an agency, a consumer sector, a country – perceive a particular aspect of your program.

This type of description can encompass great complexity if you’ve gotten a true picture of the culture you’ve been studying. Many elements may fit together in patterns you didn’t expect, and these may stand out as you put all your information together or organize it in certain ways. The goal here is to understand how the elements of the culture blend and influence one another, and how those elements and the culture as a whole affect participants’ performance, responses, behavior, communication, etc., in your program or initiative. That understanding can lead to adjustments that will make the program more successful with the group intended to benefit.

Let’s look at that ESOL program again. If most of the class members are from a culture where women defer to men as a matter of course, it may be difficult to persuade women in a coed class to participate actively. This is a particular problem in such a class, because the teaching method is based in part on encouraging as much speaking in English as possible from the beginning. If women are reluctant to speak out – even if their grasp of the new language is improving – their progress will be slower.

Understanding the underlying reasons for women’s silence in class allows the instructor to respond to the real situation, rather than simply assuming that the women don’t get it, or that calling on them will solve the problem. One solution is to break the class up into single-gender groups for part of each lesson. (This could be explained either forthrightly – to give the women a chance to speak more – or as a way to let men and women discuss what interests each.) Another is to explain that in this country, at least in public, customs are different, and that women need to learn to speak out so that they’ll be able to hold their own in shopping, jobs, and other daily tasks. Whatever the method, knowing the reasons for the women’s silence makes devising a solution possible.

Reexamine the analyses from your field notes. What did you find out about the culture’s perceptions, norms, and customs as they interacted with the focus of your questions? Seeing the study whole, do you still agree with most of your initial analyses? Set aside those ideas you think are probably incorrect or misinformed, and reformulate analyses to replace them that reflect your subsequent learning, if you haven’t already.

Try to understand the answer to your original question from the point of view of those you’re concerned with.

• Do local people see the question in the same way you do?
• Are there factors you didn’t know about that affect their perspective on the answer to the question?
• Are there factors that they perceive or act on in a different way than you expected?
• How important to the group is what’s important to you?
• Do you and they have the same expectations for outcomes? If so, do they have the same expectations about how to get to those outcomes? If not, are your and their expectations in opposition or totally different? Are they starting with the same view of reality that you have?
• Do some of their perceptions, cultural norms, or culturally-rooted behaviors present barriers to the success of your program that you’re either already working to counter or could work to counter?
• Do some of their perceptions, norms, or culturally-rooted behaviors support the success of your program in ways that you do already or could take advantage of?
• Does the program have unintended or unexpected consequences – positive or negative – as a result of cultural factors?

Some possible examples from our ESOL class: Are the younger members of the class intentionally holding back in answering questions or even in demonstrating their developing English skills so as not to show up or embarrass older class members whom they have been taught to defer to out of respect? Are class members specifically slowing down the rate of learning/instruction to that of the slowest learners so as not to embarrass anyone? Are there conflicts caused by cultural differences along these lines?

Translate what you’ve learned about participants into supporting information or answers to your evaluation questions. Some of the questions you might want to pose for yourself include:

• How do the culture, attitudes, and norms of the population you’re engaged with affect the effectiveness of the program for them?
• Are your methods appropriate for the culture you’re working with?
• Are you creating rather than eliminating barriers with some of your activities or methods, or, indeed, with your program as a whole?
• How can you make things easier and more rewarding for participants?
• What are you doing that is most effective? Can you understand from your ethnographic data why it is effective?

Continue to gather ethnographic information to guide your work. Just as evaluation continues as long as your program or initiative lasts, your ethnographic investigation should continue as well, so that you can use the information to continue to improve your program and adapt it to the culture of the community.

In Summary

Ethnography – the attempt to understand a culture from the perspective of its members – can be extremely helpful in conducting a valid evaluation of a program or initiative. Ethnographic information can tell you why participants behave or respond in certain ways, what parts of your program conflict with or are reinforced by participants’ cultural norms, and why particular program elements are more or less effective. This knowledge can help you adjust what you’re doing to be more effective and more valuable for participants.

The culture you’re engaged with might be as small and self-contained as that of a classroom or as large as a major world religion or ethnicity. Whatever the case, an ethnographic study involves approaching the culture from the inside by embedding yourself in it to the extent possible. That may mean spending many days in that classroom, observing and talking to the teacher and students, or spending years in another country, absorbing the language, customs, and behavior that define it as a distinct culture. You’ll have to decide whether an ethnographic study is necessary for your evaluation, and, if so, what resources you can devote to it.

Planning and conducting an ethnographic study requires deciding on the information you need and the resources you have to devote to it; gaining access to the group you’re interested in; choosing your specific research questions, the methods you’ll use to collect information, and whom you’ll collect it from; and then carrying out your plan, taking careful notes and/or audio or video recordings of your observations and conversations throughout. Organizing and analyzing your notes and other information can then give you a picture of the group’s culture that can help you better understand your evaluation results and improve your program. Continuing to gather ethnographic data will result in a continuing growth of your understanding and knowledge, and, ultimately, can mean a more successful program.

We encourage the reproduction of this material, but ask that you credit the Community Tool Box: /.

What is ethnographic information?

__ Ethnographic information is information about a particular culture or group gathered specifically from members of that culture or group, defining and using their own perspective and world view.

__ Elements of ethnographic information include:

• Community norms
• Health conditions and knowledge
• Power and political realities
• Religion
• Economics
• World view

__ Ethnographic information is obtained directly from those who live it.

__ The gathering of ethnographic information takes place in the environment where members of the culture normally spend their time (natural setting).

__ Ethnographic information is meant to help you understand a culture from the point of view of its members.

­­__ Ethnographic information is simply descriptive, not judgmental.

Why might ethnographic information be important to evaluation?

__ In many circumstances, ethnographic information explains why approaches work or don’t work in ways that quantitative information can’t.

__ Ethnographic information gives real insight into the ways participants or beneficiaries of programs and initiatives experience them.

__ Ethnographic information helps to clarify what needs to be addressed in order for participants to respond to your approach.

__ Ethnographic information can clarify the issue and its effect on and importance to the population of interest.

__ Ethnographic information can help you gain a clearer and more complex understanding of the culture you’re working with, so you can make better plans and adjustments in the future.

When might you want to collect ethnographic information?

__ When you’re engaged with a population or cultural group that you’re not familiar with or part of.

__ When you’re working with a clearly-defined group that has had a chance to develop its own culture.

__ When an understanding of the context and culture of the community is fundamental to what you’re doing.

__ When you’re addressing, as is often the case in an evaluation, a focused, clearly-defined situation that involves a specific population group.

Who should collect and interpret ethnographic information?

__ That depends on such factors as how much time you have, whether you already have a foothold in the community (or are part of it), the size of the group you’re concerned with, your financial resources, etc.

How do you gather ethnographic information?

__ Decide what kinds of information you need.

__ Determine what you have the resources to do.

__ Gain the trust of the group you’re concerned with

__ Plan your field study

• Decide on your questions.
• Decide on your methods of data collection.
• Decide whom you’ll need to contact and how.
• Work out ethical issues.

__ Carry out your field study, taking careful field notes, including jottings, descriptions, analyses, and personal reflections on what you learned.

How do you interpret ethnographic information?

__ Organize your data.

__ Write as complete a description as possible of the culture or the element(s) of the culture you’re concerned with from the perspective of its members.

__ Reexamine the analyses from your field notes.

__ Try to understand the answer to your original question from the point of view of those you’re concerned with.

__ Translate what you’ve learned about participants into answers to, or supporting information for answers to, your evaluation questions.

__ Continue to gather ethnographic information to guide your work.

Existing data can help you evaluate whether an intervention is working without requiring you to collect every piece of information yourself. When relevant, reliable data are already available, using them can save time and resources while strengthening your evaluation. It is important to have data to evaluate the effectiveness of an intervention.

To support an evaluation, existing data must be relevant to your population, issue, intervention, and evaluation questions. They also need to be available in a form you can analyze. You data may need to clean, combine, or otherwise prepare the data before use. This section explains how to find, assess, and use existing data thoughtfully.

WHAT Is EXISTING DATA?

Existing data is information that an organization, agency, researcher, or other source has already collected for reporting, program administration, research, or another purpose. Examples include administrative and program records, public datasets and dashboards, surveys, research studies, and community assessments. Some sources are updated regularly, while others provide a snapshot from a particular point in time.

 

COMMON SOURCES OF EXISTING DATA 

Some of the most common sources of data that may be useful for your evaluation include: 

• Public records from governmental agencies
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry 

Existing data is often available through online dashboards, open-data portals, reports, electronic record systems, downloadable files, or direct requests to the organization from which the data originated. Some information will still require a formal data request or data-sharing agreement, particularly when it contains confidential or sensitive information. 

TYPES OF EXISTING DATA YOU MIGHT LOOK FOR 

Depending on your evaluation questions, you might look for data about a specific population or geographic area, including: 

• Knowledge and awareness of issues
• Demographics of the population (e.g., age, education, income)
• Behavior
• Health and development outcomes
• Environmental conditions or risk/protection factors affecting the population 

WHY COLLECT AND USE EXISTING DATA?  

There are sometimes good reasons to collect new data, including when the information you need is not available elsewhere. When you collect new data, you also have more control over who is included, what is measured, and how the information is gathered. However, primary data collection can be time and resource intensive. 

If the information you need, or something very close to it, already exists, there are several good reasons to find and use it. 

• Existing data can save substantial time and effort, especially when you need information about a large population, geographic area, or time period. They may also provide broader coverage or greater detail than your organization could collect on its own.
• Existing datasets may already have been cleaned, organized, analyzed, or documented by people with specialized expertise. This work can reduce the technical burden on your team, although you should still review the methods, definitions, and limitations before using the results.
• Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources.
• It’s quite possible that you can find more information than you’d be able to gather if you did it yourself. The existing data you find may be more sweeping or more specific than what you’d be able to gather. They may involve more people than you’d be able to, cover a larger geographic area, or provide more detail.
• Existing data could touch on important areas you have not considered, or identify patterns or relationships you wouldn’t have looked for.  In cases like these, the use of data that already exist might change your whole view of your work, and help bring you to a level of effectiveness you wouldn’t have reached otherwise.
• The data may have been collected using established methods and systems. This can save time and strengthen your evaluation, but it does not guarantee that the data are free from bias, missing information, inconsistent definitions, or other limitations. You still need to understand how the data were collected.
• Existing data can help you examine change over time. Historical information can show whether an observed change is part of a longer trend and whether similar patterns appear in comparable communities, the state, or the nation. This perspective may be difficult to obtain through short-term data collection alone.
• Existing data can make it possible for small organizations with limited resources to conduct thorough evaluation studies. Most small community-based organizations simply have neither the money nor the personnel to gather large amounts of data – but, there’s no need to when the data you need exist elsewhere. 

WHEN SHOULD YOU COLLECT AND USE EXISTING DATA? 

• When it’s available and accessible. This is the key question. If you know the data exist and can obtain appropriate access, they may be useful. If the data do not exist, are too difficult to locate, or cannot be shared because of privacy or other restrictions, you may need another approach.
• When it’s relevant. As with availability, the relevance of the data to what you’re trying to find out is a key issue. All the existing data in the world won’t do you any good if they don’t help you answer your evaluation questions.
• When you don’t have the time and/or resources to collect the data yourself. Whether it’s a matter of the size and scope of your organization, time pressure from a funder to produce an evaluation, or some other factor, existing data may be the only source of the information you need.
• When the data match your evaluation needs. Data should align with the population, place, time period, measures, and program elements you are studying. Select only sources that can help answer your evaluation questions; otherwise, the information may provide an incomplete or misleading picture. 

HOW DO YOU FIND AND USE EXISTING DATA? 

As you search for and use existing data, there are several questions you should ask. 

Are the data a good fit for your evaluation? 

Before using an existing source, look beyond whether it is easy to obtain. Ask whether it can answer your evaluation question and whether it can be meaningfully compared with any data you collect yourself. 

• Who and what do the data represent? Make sure the population, geographic area, and issue are relevant to your program.
• When were the data collected or updated? Older data can be useful for trends and context, but may not reflect current conditions.
• How were the data collected and defined? Review the data source’s methods, measures, and definitions. Two sources may use the same term—for example, “service received” or “graduation”—but mean different things.
• Are the data complete and sufficiently detailed? Consider missing data, small numbers, the level of detail available, and whether results can be broken down in ways that are useful and appropriate.
• What limitations or potential biases should you keep in mind? Administrative data reflect the people who use a service and the way staff record information; survey and research data may be affected by sampling, response rates, or measurement choices.
• Can you use the data ethically and securely? Follow data-sharing agreements and privacy requirements. Whenever possible, use aggregate or de-identified data rather than information that could identify individuals. 

WHAT INFORMATION DO YOU NEED, AND WHY? 

To answer this question, you first might think about what information you need for your evaluation that doesn’t necessarily require gathering data on current participants. 

Some possibilities include: 

• Data on past participants. You may want to compare the results for current participants with data on past participants, especially if you’ve changed your methods or the population has changed significantly.
• Community-level indicators are specific, measurable signs of conditions or change across a community. Examples include program participation, services delivered, crime rates, or new cases of HIV. These indicators can help you track community trends and assess whether an initiative may be contributing to change. 

Community-level indicators describe trends for the community as a whole, rather than changes among individual participants. For example, an initiative to reduce alcohol use among young people might track weekend and nighttime single-vehicle crashes involving teenagers. Choose indicators because they are meaningful to the initiative and community—not only because data happen to be available. 

• Specific information on appropriate characteristics of the population you’re working with. This may be used to compare your participants to the population they’re part of, as well as to track specific differences that might be a result of your program.

Both general and specific information might include several categories to choose from. These categories include: 

• Demographics:
  • Demographic characteristics, such as age, race, ethnicity, gender identity, household composition, education, income, employment, housing, disability status, primary language, and geographic location
  • Geographical location and distribution, population density, etc.
  • Economics – income, employment, living conditions
  • Housing – household size and ownership
  • Education level  
  • Other characteristics –primary language spoken, etc. 
• Behavior:
  • Health-related – tobacco use, physical activity, diet, etc.
  • Alcohol and other substance use
  • Sexual behavior  
  • Various other behaviors – work habits, consumer patterns, etc. 
• Health and Development Outcomes:
  • Incidence (new cases) and prevalence (existing cases) of specific health conditions (e.g., diabetes, injuries, infant mortality)
  • Developmental and educational outcomes, such as completion of primary or secondary education and equitable participation by people with disabilities
  • General health and well-being characteristics of the population or community Access to health and human services (e.g. those with access to clean water and sanitation
  • Knowledge and awareness of issues (e.g. survey data on public concern with violence)
  • Environmental conditions or risk/protective factors (e.g. exposures to pollution or toxins) 

• Cultural information. Norms, customs, celebrations, beliefs, and practices related to culture. If you can see whether and where your group’s efforts are fitting with participants’ cultures, it will help you to determine whether that’s an issue, and where you might need to make changes. 
• Data on a similar group that can be used as a control or comparison. This might be a group from the same population that signed up for but did not experience the program, or another comparable community in a different place.
• Results of previous studies. You’d probably be most interested in studies that looked at the same issue and population group you’re addressing. These can provide a standard of comparison, as well as some sense of what kinds of results might be reasonable to expect.

Suppression of Statistics

An issue that you should be aware of and prepared to encounter during your research is suppression of statistics.  When research deals with small populations or data pools, in order to protect the privacy of individuals, it is sometimes necessary to suppress data. In other words, when the number of cases in a category - i.e., females in Wyandotte County who died from lung cancer in 2004 - is small enough that disclosing the data might allow a specific individual to be identified, steps are taken to protect the privacy of individuals.

The most common method of preventing the identification of specific individuals is through cell suppression. This means not providing counts in individual cells where doing so would potentially allow identification of a specific person. Cell suppression can also be done by combining cells from different small groups to create larger groupings that reduce the risk of identifying individuals.

The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to chronic liver disease and cirrhosis in Wyandotte County, KS in 2010. Because the numbers for Black and Other individuals are small enough that it might be possible to identify individuals from those statistics, the data is suppressed, as indicated by the #.

Death Statistics for Chronic liver disease & cirrhosis for Wyandotte County, 2010
White	Black	Other	All races
14	#	#	20
# Indicates numbers below 6

The National Association of County and City Health Officials (NACCHO) has a useful tip sheet that explores this and other challenges of data collection and analysis in jurisdictions with small populations and provides useful information for overcoming these challenges.

In addition to the question of confidentiality, low numbers in a given category can also be an issue when considering the stability of data. In other words, when there are low numbers or incidences in the data you are researching, it is more difficult to accurately calculate rates and it can give an inaccurate picture of the categories you are researching.  For instance, if the number of lung cancer deaths in 2004 was 20, and in 2005 it was 30, statistically that is a 50% rise over one year, which is quite a substantial fluctuation; however, it may be that it is simply a normal variation in reporting. Because the numbers reported are so small, even minor changes can seem substantial, and this can result in unreliable or unstable data. The table below, from the Kansas Department of Health and Environment's Bureau of Epidemiology and Public Health Informatics, shows a break-down by race of deaths due to breast cancer in 2010 in Wyandotte County, KS. Because the numbers available for White, Black, and Other are too small to allow for an accurate, reliable calculation of the rates for that year, the information is suppressed, indicated by the @.@ symbols.

Death statistics for malignant neoplasms of breast, Wyandotte County, 2010
	White	Black	Other	All Races
Number	15	6	#	23
Rate	@.@	@.@	@.@	15.5
@.@ indicates numerator too small for rate calculation # indicates numbers below 6

However, there are a couple of strategies that can be used to help avoid or address these problems of instability.

Combining several years of data can produce more stable estimates when annual counts are small. Rolling averages—for example, comparing 1997–2000 with 1998–2001—can help show change over time while reducing year-to-year variation. The tradeoff is that multi-year estimates may make recent changes or program effects harder to detect.

Expanding the geographic area can also increase the number of observations and improve statistical stability. For example, you might use regional or state data when county-level counts are too small. However, broader estimates may not reflect local conditions and can hide meaningful differences among communities. 

Who is likely to have collected that information?

Public agencies, research organizations, universities, schools, health and human service providers, funders, advocacy groups, community organizations, and businesses may all maintain useful data. Start with the source most likely to collect information about your topic, population, or geographic area.

In some cases, you might know for certain that it exists; in others, you’ll have to search around. Some places to start:

Public records.

Government records at all levels – including federal state, county, and local. Copies of publicly-funded studies (after publication), financial information, crime statistics, demographic information, and much more are available in public records.

Some you might be most interested in:

• Census Bureau. In most developed countries, the census covers a broad range of demographic, economic, and geographic information.
• Federal and state departments and ministries. From environmental data to farming practices and subsidies to poverty statistics to public health issues, the federal government is a vast storehouse of information.
• Various levels of the court system. In the U.S., where civil and criminal trials and their results are public, their records are public also.
• Police records. Arrests, domestic disputes, injury reports, and other information can be found in police reports.
• Securities Exchange Commission and other business regulators. The SEC and other regulators require businesses to file various information, usually annually, including annual financial reports and environmental statements, all of it public.
• County commissions, agencies, and authorities. County Extension Services in the U.S. (part of the U.S. Department of Agriculture) can be particularly helpful.
• City and town clerks’ offices.

Sometimes, government agencies are reluctant to share information, even though it’s public.  The Freedom of Information Act (FOIA) deals with this issue in the U.S.  It allows for access to a wide range of federal government records.  Similar laws at the state level do the same for state documents.

• Research organizations. Think tanks, independent oversight organizations, and research organizations all issue reports on various topics, often backed up by studies.

Some of these organizations aren’t, and don’t pretend to be, politically neutral.  They have agendas, conservative or liberal, and some of them interpret their research in light of those agendas.  It’s important to be aware of the bias of any archival data that you use if you want reliable data.  However, many organizations with a political stance nonetheless try to make their studies as objective as possible.

• Academia. Much research in health, human services, social issues, education, the environment, and the sciences is conducted by universities and institutions connected to them. This includes theses and dissertations for advanced degrees, as well as the results of funded research web search engines, such as Google scholar, can help locate research information.
• News media. Newspapers, magazines, and radio and TV outlets all keep archives, often going back to the founding of the publication or station. These are often available to the public – sometimes on line – either free or for a fee. Although they are unlikely to contain detailed study results, they often have summaries of important studies, and may serve to point you in the right direction to find what you need.
• Foundations and other private funders.These organizations fund studies of all kinds, and many publish or otherwise make available the results as a condition of funding.
• Hospitals and other health care providers are sometimes university-related, and may conduct studies of various health issues. They also may collect, as an administrative necessity, demographic and other statistics on their patients, as well as information on the frequency, geographical location, and intensity of various medical conditions.
• Mental health providers may have data on particular types of conditions, or on who is most at risk for particular behaviors or conditions (e.g. depression).
• Human service and other non-governmental community-based organizations. The information most likely to be gleaned from these organizations is administrative, and to cover such areas as demographics and the location and character of community issues.

Depending on its nature, some of the research carried out or administrative data gathered by universities, health and mental health providers, and human service organizations may have some restrictions on them because of confidentiality. These restrictions usually only cover access to individual records and identification of study participants, and generally don’t pose a barrier to obtaining aggregate results of studies, assessments, or surveys with no identification of individuals.

• Advocates and watchdog organizations may collect data (either locally, statewide, or nationally) on businesses, on the environment, and on other particular issues – nearly anything that pertains to their causes – and they’re usually willing to share it.
• Community activists. These folks tend to focus on specific issues, but if your issues are similar to theirs, they may have a great deal of information that’s useful to you.
• Community economic development organizations are likely to have economic data, land-use maps and patterns (perhaps including population distribution by race, ethnicity, age, etc.), environmental information, and other similar material you might find useful.
• Businesses and corporations, particularly large ones, often collect information on their workforces, economics and economic trends, and similar topics.

WHERE SHOULD YOU LOOK FOR EXISTING DATA? 

The question here is not only where to find existing information, but where to find it most quickly and easily. Some of this material will be published, some only available from the organizations that collected it. Looking in the right place first can save you a lot of time and trouble. 

Your own records and data 

Unless it’s brand new, your own organization should have administrative records, past evaluations, assessments, and other data that might be helpful to you. Don’t ignore this obvious and easily accessible source of information. 

Online data portals and websites 

Many public documents and datasets are available online. Start with the website or open-data portal of the government agency, organization, or institution most likely to have collected the data. In the U.S., federal, state, tribal, county, and municipal agencies often provide searchable reports, dashboards, or downloadable files. Similar sources are available in many other countries.

Many of the other sources of information mentioned above are likely to have websites also. Whether their data is available on those sites is another matter, and depends to some extent on what kind of information you’re seeking. Watchdog organizations and some think tanks are likely to post at least some of the results of their research on websites because they want it to be as public as possible. Community economic development organizations likewise usually have informative websites, since they’re trying to attract businesses and residents to an area.

Health providers and academics, on the other hand, may post their research on a website, but only after it’s been published in a journal or book, or presented at a conference. That means that you’re not apt to find very recent data (from the past year, for example). Local health and human service providers and schools rarely conduct formal research, and rarely post any administrative data on their websites, for two reasons: confidentiality, to which we’ve already referred, and the fact that most of that data are intended for internal use, and therefore not seen as useful to anyone outside the organization.  Business websites generally include material only of interest to potential customers. Community activists may or may not have websites at all.

Assess online data sources carefully. Favor information from the organization that originally collected or maintains the data, and look for clear documentation of methods, definitions, update schedules, and limitations. If a source is unfamiliar or a finding seems uncertain, compare it with other credible sources before using it. 

Go directly to the source

When data is not publicly available, contact the organization that maintains them. Explain what information you need, how you plan to use it, the level of detail required, and how you will protect privacy. Be prepared to discuss a data-sharing agreement, reasonable costs, or a mutually beneficial exchange of information or expertise. 

When requesting data from another organization for comparison, frame the conversation around shared learning rather than competition. Explain why the comparison is useful, how you will account for differences between programs or populations, and how findings will be interpreted and shared. An existing relationship built on trust and mutual respect can make this collaboration easier. 

Libraries

Libraries and librarians can help you identify credible datasets, government reports, scholarly research, archives, and specialized databases. They may also help you develop search terms, locate hard-to-find sources, and assess whether a source is authoritative. 

What are you planning to do with the data once you have it?

Plan how you will use the data before requesting or downloading them. Determine which variables, time periods, population groups, and level of detail you need. If you intend to compare existing data with information from your program, make sure the groups, definitions, measures, and collection periods are sufficiently similar to support a meaningful comparison.

If you’re planning to subject your data to statistical analysis, you’ll want information that either is, or can be made, quantitative. If the information you’re collecting on your participants is largely qualitative, then the existing data should be qualitative as well. Furthermore, the information you get either should determine or should match the way you collect your own data, so that there’s a reasonable comparison, assuming a comparison is what you’re intending. 

USING EXISTING DATA 

In many evaluations, you will combine existing data with new information collected from participants. Existing program or administrative records may also provide current information about participants, depending on what is recorded, how quickly the data are updated, and whether you have appropriate access. You may be able to find earlier data on participants to use as a baseline or data on a similar group to use as a comparison or control. 

The “almost” here refers to a situation where you’re evaluating a program in retrospect – looking back at it after it’s underway or been completed.  It may be possible in that case to find existing data that will allow you to determine the program’s effectiveness in terms of process, outcomes, or both. 

Although you’ll probably collect information on the participants in the program you’re evaluating, there are a number of ways you might use existing data: 

• To better understand the context of your evaluation. These might be ethnographic data (see Section 6 of this chapter), oral histories, assessment information, interviews, etc. You’d use it to get a clearer picture of the community in a number of ways, and to help you interpret the results of your evaluation.  It might, for instance, give you insight into why a particular approach did or didn’t work, or why some participants stayed in the program while others didn’t.
• To identify areas to address. A clearer picture of the community allows for a deeper understanding of the community’s needs and concerns.
• To establish a baseline against which to measure your results. For this purpose, you’d need recent information about where the population you’re working with stands on the dependent variables or outcomes you’re concerned with. That would tell you where the participants started from (on average), so that you could see from the measures you used in your evaluation whether and how much they might have improved because of your work. 

In an evaluation, the intervention is the program, policy, service, or other action expected to contribute to change. The outcomes are the behaviors, conditions, or other results that may change. For example, a violence-prevention program is the intervention, while changes in violent behavior and related injuries are outcomes. 

• To identify already-existing trends that may affect the results of your evaluation study. The fact that there’s been a change in participants between the beginning and end of your evaluation doesn’t necessarily mean that you’ve caused it. Among other things, it may be part of an ongoing trend toward change that started well before your program did, and may continue after it.  Archival data might show such a trend over a number of measures of your dependent variable in the population your participants come from.
• To establish a standard of comparison against which to measure your efforts. There are two ways that you could use archival data for this purpose.  One is to use census, statewide, and/or community-wide data to compare with that of the population you’re working with. That comparison can give you a sense of how serious the issue is for your group, compared to the general public.  The second way is to use similar data to compare your outcomes with the data on the larger population. This might work especially well when you’re using community-level indicators (e.g., rate of injuries, percentage of girls completing different education levels).

You might find, for example, that even though community-level indicators moved in the right direction – the sale of tobacco products went down, say – they still compared unfavorably with the state or national averages for the same indicators.  That knowledge might be important in future goal-setting and in using your evaluation results to gain community support or funding. 

• To act as a control or comparison group. One of the best ways to learn whether or not your program had an effect is to compare the participants you’re working with to those in another group that received no program or a different one. The best alternative here is to create a group from the same population as participants – so that all participants will have approximately the same background, environmental influences, cultural norms, etc. – and to conduct the same observations on both groups at the same times, so that the only difference between them is the program that one of them is exposed to. In practice, creating or finding a perfect control group is often difficult.  Archival data may be able to provide a reasonable alternative, in the form of data collected on a comparison group or population similar to that of participants in your program.

Often, the most likely possibility is a group that was part of another program with the same goal as yours, but using different methods. This has the advantage not only of providing a control, but of letting you infer whether your approach works as well as, not as well as, or better than that of the comparison group. 

• To provide data for a longitudinal study.  If you think your program might have a long-term effect, or if you think it will interact with the effects of past events, circumstances, or programs, you might want to conduct a longitudinal study – one that looks at participants over a longer period of time – for your evaluation. You may not have the time or resources to collect data over a period of years, but you may be able to find archival information that allows you to draw some conclusions about long-term effects.

There are at least two circumstances where you might be able to use existing data for a longitudinal perspective.  The first is one in which you’re looking at the effect of an issue on the population for a length of time before your program began. This might make it easier to see program results in context, and to understand whether the program broke a cycle and started real change. The second circumstance is when you’re looking back at the effects of a program that was completed some time ago.  In some circumstances, the effects of a program multiply or accelerate over time. Particularly if your program was aimed at changes throughout the community (reducing intimate partner violence, for instance), you may be able to find existing data that tells you whether the effects of your program continued, kept growing, or trailed off. 

In Summary

Existing data can help you understand community conditions, establish a baseline, identify trends, compare outcomes, and examine change over time. Useful sources include your own records as well as data maintained by public agencies, community organizations, health and human service providers, universities, funders, and other institutions. Select data that fit your evaluation questions, examine their quality and limitations, and follow all privacy and data-sharing requirements. Used thoughtfully, existing data can make evaluation more feasible and informative—especially for organizations with limited time and resources.

What are archival data?

___Archival data is data that already exists as a result of administrative procedures or past studies or evaluations

___Be aware of sources of archival data:

• Public records
• Research organizations
• Health and human service organizations
• Schools and education departments
• Academic and similar institutions
• Business and industry

___Be aware of the possible types of data available:

• Demographics
• Behavior
• Health and development outcomes
• Attitudes – racial, political, social, etc.
• Knowledge and awareness of issues
• Environmental conditions or factors affecting the population and/or your work

Why collect and use archival data?

___It’s easier and less time-consuming than collecting all the data yourself

___Archival data may have already been processed by people with more statistical expertise than you, making it easier to use in analysis

___Even with raw data, the basic organization and preparation (transcription of interviews, entry of numbers into a spreadsheet or specific software, etc.) may have already been done, again saving time and resources

___It’s quite possible that you can find more information than you’d be able to gather if you did it yourself

___Archival data could touch on important areas you might not have thought of, or identify patterns or relationships you wouldn’t have looked for

___It may eliminate the need to correct for such problems as lack of inter-rater reliability or observer bias

___Archival data allows the possibility of looking at the effects of your work over time

___Archival data can make it possible for small organizations with limited resources to nonetheless conduct thorough evaluation studies

When should you collect and use archival data?

___When it’s available

___When it’s relevant

___When you don’t have the time and/or resources to collect the data yourself

___When it can truly inform your evaluation

How do you collect and use archival data?

___Determine what information you’re looking for and why

Possible categories of information you’d be interested in:

___Data on past participants

___General information on the population and/or the community you’re working with

___Specific information on appropriate characteristics of the population you’re working with

___Cultural information

___Data on a similar group that can be used as a control or comparison

___Results of previous studies

___Determine who is likely to have collected that information:

• Government departments and agencies, the census, and other entities whose data are likely to be available in public records
• Research organizations
• Academia
• News media
• Foundations and other private funders
• Hospitals and other health care providers
• Mental health providers
• Human service and other community-based organizations
• Advocates and watchdog organizations
• Community activists
• Community economic development organizations
• Businesses and corporations

___Decide where you should look for archival data

• Your own archives
• The Internet
• The original source
• Libraries

___Decide what you plan to do with the data once you have it

Use archival data:

___To better understand the context of your evaluation

___To identify areas to address

___To establish a baseline against which to measure your results

___To identify already-existing trends that may affect the results of your evaluation study

___To establish a standard of comparison against which to measure your efforts

___To act as a control or comparison group

___To provide data for a longitudinal study

Tool 1: Common Secondary Data Sources Used to Address Gender-Based Violence

This tool from Strategic Prevention Solutions provides descriptions of common secondary data sources useful for addressing gender-based violence, along with common challenges with secondary sources.